Ruud Questions: Ralph Tegtmeier aka Fantomaster

Ruud Hein

15 years ago

The Man in Black fled across the desert and the Gunslinger followed

When thinking how to introduce Fantomaster, those words came to mind. His is a nickname that evokes the legendary imagery of Clint Eastwood, of one who took the red pill.

I was pleasantly surprised to come across him on Twitter; more so when I realized how smart he is (he even retweets me on occasion, so like, there!).

Seeing some of his ideas and opinions bubble around I just had to ask him for an interview.

""Vell, Zaphod's just zis guy, you know?"

You've said "links simply aren't everything if you want to achieve rankings" yet the majority of SEO these days is built on links, links, links. If links aren't everything, what are those link-focussed people missing?

For one, the good old classic SEO rules still apply and have actually never stopped doing so - build web sites with good, useful information or content; make your pages light and fast to load; use keywords in your title tags and text headers; use meta tags: sparingly perhaps, but intelligently as well; attend to usability as much for your human visitors as for the search engine spiders hitting your site, i.e. don't make things difficult to discern for either of them; make sure you optimize your on site navigation and linking structure.

There's lots more but this little overview should suffice to illustrate my point that on site factors are still all important: for your human visitors as much as for the crawlers.

For example, whenever we conduct an on site link analysis of just about any fairly large corporate sites, there'll always be hundreds if not thousands, sometimes even tens of thousands of pages that are either entirely orphaned or bleeding link juice the wrong way.

In some cases we've detected that up to 25% of a site's pages weren't linked to from anywhere at all - and mind you, these are big corporations deploying only best-of-breed expensive content management systems, not some cheapo amateur outfits with only a couple of hundred pages!

As for off site ranking factors, RSS feeds, for example are still extremely useful though sadly underrated. People seem to be so obsessed with an utterly simplistic view of link building, it's quite pathetic. All they've heard about is that "nofollow" links are useless (which is plain untrue) or that "you must have high PR links or else" - which is just another stupid myth, even though it has spawned a multi million dollar industry of link selling and brokering etc.

I'll gladly concede that the Web is getting ever more complex which makes it harder to fathom its interdependencies and to properly gauge the potential synergies they offer. But on the upside there's such a huge, unprecedented amount of opportunities to leverage these very relationships that I feel this is actually a very good thing.

So if all you can think of in terms of organic SEO is "links, links, links", you're actually missing out on about 90% of all those other possibilities and traffic conduits out there. That's like restricting your offline advertising efforts to classifieds only: yes, such a one-sided approach may actually work out nicely in some areas, but chances are you're still leaving a lot of money on the table by ignoring other advertising platforms, by not investing in proper public relations activities etc.

This is not to say that links are unimportant in any way, on the contrary. Yes, you do need links to achieve good search engine rankings, and yes, you're right in putting a lot of effort into getting those links. But don't make the mistake of getting all smug about it, conning yourself that this is all there is to it. You really couldn't be more mistaken.

You advise that link development should be a consistent process showing no sudden or strange cyclic, account-based dips and spikes.

It's what I consider a common sense approach to link development but what I find interesting is that you explain that in the search engine's mind such spikes are only reasonable and plausible if they coincide with a corresponding spike in search queries for that particular topic.

Apart from Query Deserves Freshness, in which other ways do you think query data is folded back into both ranking and filtering?

One familiar major way in which both search query and link spikes will occur is the news. If a topic starts hitting it big with the news outlets, it's only natural to expect lots of people chiming in both in terms of their search query activity as well as their publishing and linking efforts.

Let's say you're running a blog featuring op-eds on economic subjects. And let's assume that half a year back you or one of your guest commentators published a cutting edge analysis of the U.S. prime rate sector along with a pretty shrewd and detailed prediction that it was bound to impact the entire banking system and the world's economy in a very negative way. Come the credit crunch, lots of authority sites will report on this very topic at great length. Now suppose a major site like WSJ happens to link to your piece - this will funnel a lot of attention your way, and on the Web "attention" will typically translate into incoming links, probably creating a huge spike.

If the search engines regard this as a "natural" rather than an artificial or manipulated spike, how would they actually determine that? There's a number of possible ways they may be doing it: analyzing the news sites' topical focus, their human users' search queries, RSS feeds activity, social networks buzz, etc. Unfortunately, this is nothing you can reliably plan in advance as a rule - which in turn makes it all the more "organic".

On a slightly less dramatic scale you'll find seasonal topicality. E.g. you wouldn't expect a lot of new links suddenly pointing to your Easter egg coloring site during the Christmas season, would you? So there are some fairly obvious exceptions to that rule. However, generally speaking links won't come your way in sudden spikes only to peter out again soon after. Rather, it will probably be an ongoing, fairly consistent process which you should mimic to the best of your ability.

We do know about a lot of search engine patents addressing this issue, though whether they're actually making use of them in any meaningful manner is nothing the engines will tell us about openly. We can, however, venture some educated guesses. One of these being that they're leveraging user behavior data - quite a rat's nest once you dig into the mathematics involved here, and definitely anything but perfect. But it does stand to reason that the search engines will make use of it any way they can if it actually helps them achieve better results.

The way I see it this is one out of many reasons why Google, for example, is rolling out such a boatload of blatant spyware: whether it's their notorious toolbar, their AdSense tracking technology, their Web Accelerator, Gmail, personalized search, their Webmaster Tools, the Chrome browser or their data storage facilities allowing you to host all your documents on their servers ranging from your personal or corporate correspondence through spreadsheets and health records - all of these are prime data funnels allowing them to discern public trends on the fly.

Now while I'm not exactly on record for being the world's most ardent Google lover, I have never claimed that they are stupid or clueless in terms of technology and data mining. They certainly would have to be stupid, however, not to make ample use of all this data deluge they're so obviously keen on grabbing. To what extent they're actually doing so, and in which precise manner, is probably anybody's guess outside the 'Plex. For us as SEOs this amounts to a great challenge in terms of reverse engineering their ranking process, of course.

You've mentioned that maintaining the IP list of bots is not purely automated: critical steps are done manually.
How involved are you with the process and how much time do you spend on it every day?

We would love to automate the entire process but we've found out the hard way that there's no reliable way of doing so. And so have many others, by the way. Sure, we've created a lot of scripts that will analyze traffic logs and alert us to suspected spider activities, but at the end of the day it requires a great deal of manual checking out by highly experienced staff.

For our fantomas spiderSpy(TM) database we're monitoring over 35k third party sites' traffic on a daily basis. It's hard to accurately gauge the amount of human labor time spent on this because it tends to come in shoves. Ok, so we update our database every six hours and have been doing so since 1999. However, sometimes no new search engine spiders will crop up for days or even weeks on end. Then, all of a sudden, Yahoo! or Google may roll out 800 new spiders within a couple of hours (yeah, you read right: I'm not exaggerating!) which is when things tend to get rather frantic.

You see, it's not as easy as simply defining entire IP ranges in one fell swoop: more often than not, some IPs within any given range are obviously reserved for human editors, i.e. non-spiders, and it requires a lot of expertise to suss them out reliably.

Two general, rule of thumb, best practices type questions regarding cloaking.

First; the majority of web sites do or do not need cloaking -- which is it, in your opinion?

It's a question of a) how competitive your niche or industry is in the first place, and b) how hard hitting your competitors are. Even Tim Mayer of Yahoo! has admitted publicly that there are industries where you don't stand a snowball's chance in hell if you don't adopt hard core black hat SEO techniques. Take pharma, adult content or online gambling (the "other PPC" aka "pills, porn and casinos"): these are classic and pretty well known areas where you'd better acquaint yourself with cloaking or IP delivery technology if you don't want to go belly up in no time.

There are lots of other, albeit less notorious niches where the same applies: online finance in general and credit repair or payday loans in particular are one prominent example. So are travel, ringtones, webhosting, domain registrations, and many many others.

On the whole, what with an ever more competitive online market, the number of niches requiring cloaking as a boost for organic search rankings and qualified traffic is actually dramatically on the increase.

Still, if you are working in a low competition environment, cloaking may actually not be for you. It does require some pretty good reliable and sophisticated tools and these don't come cheap.

Another obvious factor to consider is your overall ROI or profit margin, of course.

Second and two-fold; if my SEO company recommends my business to employ cloaking, how worried should I be that A) this company is a scam, and B) that I'll be found out and destroyed by Google?

First off, it's a question of full disclosure or due diligence. If your SEO agency "forgets" to mention the risks you'd incur by going for cloaking or even pretends that there's absolutely no risk involved, I'd say you would be justified in rating them a scam outfit.

This said, the risks involved, while quite real, are generally overrated. Yes, cloaked sites may be found out and they may get banned by search engines: it can and does happen, so you'd better be prepared for it. But for one it's actually a pretty rare occurrence. More importantly, however, there's a lot of fairly uncomplicated security measures you can adopt to buffer the impact or render it negligible.

One practical example: Don't mix cloaked and non-cloaked content on your money site. Should you get penalized for cloaking by the search engines, you'd lose your money site's indexing and ranking. Rather, build self-contained cloaked-only sites, what we term Shadow Domains(TM), and redirect your human visitors from there to your money site. If possible, clone your money site, hosted on a different server, and use it as a redirection target. That way, even if a search engine editor should track it down, following your SDs' redirection trail, you would simply set up a new clone somewhere else and start from scratch.

Again, there's lots more but this example shows that there's plenty of very effective things you can do to reduce the risks and improve your risk/gain ratio dramatically.

Most importantly, though: don't try to pull the cheap stunt on yourself. "Poor man's cloaking" such as delivery by UserAgent, JavaScript based operations etc. are simply too risky to be worth the effort at all. Only use software that offers genuine IP delivery because there's no other reliable approach.

And always work from a reliable, regularly updated list of verified search engine spiders - the free lists available out there are hopelessly dated and inaccurate. This is quite critical, of course, because unless your cloaking application can positivelys determine a search engine spider for what it is, you're running the risk of detection.

Information retrieval has come a long way. Given a clean index filled with good documents only, most IR systems should be able to give a user a set of relevant results. But a web-index is filled not only with great on-topic content but also garbage and plain spam. Search engines meanwhile don't give the best ever results; they approximate a top 10, or 100, or 1000 of possible candidates and rank those. What do you see in the future then; pay for inclusion indexes? Premium search and free search?

While I don't see general, all-purpose search going away anytime soon, chances are there'll be a growing demand for more focused, laser targeted and specialized search options in future. Ther's actually quite a lot of this happening already, even though we as SEOs seem to ignore it. For example, as a professional lawyer or a doctor or a physicist you'll probably sign up for specialist portals or databases that cater to your profession's specific needs only. Most of these verticals are human edited and monitored which is an entirely viable business model because registration is generally not free and actually quite expensive.

But these fairly smallish clienteles aside, there's plenty of scope for medium size markets to be catered to as well. Whether you're involved in researching music rights, late Elisabethan poetry, alternative non-insulin based diabetes therapies or airplane spare parts - merely googling these topics will generally be a pretty frustrating experience even today. Even if you're not a professional researcher subscribing to highly demanding standards you're likely to be sorely disappointed with what crawler based engines have to offer you. So as demand for compact quality information grows, so will platforms addressing this need - specificity will rule.

Eventually, we'll see a lot more of this cropping up, segmenting the search market as a whole and obviously stealing significant chunks of traffic from the all-purpose, one-for-all engines like Google, Yahoo! and MSN/Live. Let's not forget that Web search, for all its 15 years of history is really still in its infancy as, for that matter, is IR as a whole.

There's still a great deal of work to be done to actually make all the world's information available in a meaningful, cost and time efficient and reliable manner. Currently, search engines like Google, Yahoo! etc. are merely servicing pretty unsophisticated mass markets who are content with gobbling up their shoddy results for the simple reason of convenience and because they are free. But with further specialization on the rise in economy and commerce and society as a whole, people in the not too distant future will probably consider today's search technology to have been an incredibly primitive and error prone affair they will probably make a lot of corny jokes about...

On to the economy and its victims. Is the web a place for people, for regular folks, to look at and say "hey, I can make a living here" or is setting up something that pays a monthly income on the web akin to starting your own shop "in the real world"?

I guess this depends on how prone you are to being hyped. Essentially, the Internet as a marketplace has been overloaded with ridiculous expectations and promises pushed from many different quarters. Sure, it offers a lot of opportunities the world hasn't seen before, and if you know how to make the best of it, it can make you incredibly rich in an equally incredibly short time.

And yes, this has actually happened over and again - but as always in life, if you mistake the singular exception for the general rule, grief is only one small step away. Generally speaking, building a viable online business is just as much of a challenge as is setting up shop in a brick and mortar environment.

It's also governed by the very same economic laws: don't spend more than you have; buy low, sell high; listen to your customers; watch out for your competition; don't give in to smug routines etc. None of this has fundamentally changed because it's ingrained in the way the world of commerce (or, if you like, capitalism) simply works.

So while some ventures may prove to be a roaring success in no time (which, incidentally, tends to happen in the "real world" as well, so it's nothing entirely tied to the Web), most all operations will have to practise relentless perseverance until they will actually turn a profit. There's no easy way out and no guarantees of success, no matter what some marketing bozos may claim when trying to pitch their overpriced stuff.

And the sooner you come to terms with it, the better your chances of success. It's still very much about toil and market savvy and shrewd risk assessment, mingled with a decent quantity of sheer luck. Merely because you're using a computer and some digital download platform instead of high street shopping premises and a warehouse doesn't change any of that.

Once a person like the one above has gone ahead and setup a blog or an Adsense site, a freemium idea or subscription content -- what is the single most effective next step they should take to start to have a chance of getting people in via search ... with the constraint of that question being limited budget?

Once you've optimized your site for SEO and conducted some decent link building, go and create a buzz, make people talk about you, preferably in a positive way. Underpromise and overdeliver as much as you reasonably can. Get involved in conversations - there's plenty of free platforms available for that. Don't be an obnoxious pitcher of your own stuff, help people out with advice and create lots of relationships. Let them spread the word for you: after all, word of mouth is still the most cost effective kind of promotion. Linkbait can work miracles if you do it right, too. At the end of the day, all of this should translate into plenty of inbound, one-way links and in the search engines taking note.

On our fantomNews blog at we're getting a pretty consistent volume of traffic even when I refrain from active blogging due to time constraints. I actually didn't blog for two consecutive years once without any dramatic slump in traffic. Why? Because we've obviously created plenty of useful content people are interested in.

Plus, publishing our SEO cartoons consistently week by week has proven a prime traffic magnet as well. Those cartoons are arguably one of the best investments we've ever made in terms of keeping the buzz level up. This is no "secret sauce": it's pure common sense - give people what they need or enjoy or even get worked up about, and they'll honor it.