Instead of painstakingly grabbing the absolute best matches for your query to then rank those with infinite precision, one time saving strategy has search engines go for "close enough".

Painstaking Precision

Given all the time, money and resources in the world, here's what we'd normally do.

Word by word you go through a search. You look in your documents and see which has word one…. word two… word three…. You get the picture.

You give some plus points for every time a searched word appears in a document. How many points? Depends on the TFxIDF score for that specific word.

You add up the scoring into a sum, measured by relevance again. Do the same for the query itself (treating it like a very short document basically). In short: you're calculating their vector space scores.

Measure the mathematical similarity between document 1 and the query, document 2 and the query, document 3 and… Yup.

And then you can't just slap all those on the screen. You have to tailor to the searcher's need and pick and sort the top scoring documents!

Now you could sort all your scored documents at once or just go for the top number of documents needed; say the first or next 10 because the searchers has that set as maximum results per page.

Instead of doing this HUGE sorting routine you throw all values together in a big black hat (mathematicians call this a "heap"), come up with the top 10 documents or so and only then sort them.

Perceived Relevance

What struck me as funny is that this high precision, high-cost way of doing things doesn't necessarily mean you get the most bang for you buck, the best quality results for your searcher's patience.

No, the mathematical similarity between our search and those documents is something we perceive as relevant.

That's a low payback to work for when the cost is so high; comparing a huge number of documents, calculating mathematical similarities…. grabbing the top of the heap…

The perception of relevance is something a search engine can use though by going for "good enough".

The Inexact Sort Of Top 10-ish Documents You Might Want

Instead of calculating the top 10 with high precision, why not grab a bundle of documents that will most probably be in that top 10?

Just grab a bunch of documents that are in the race to be the answer to the searcher's query and take the top 10 of that bunch!

Even though this top 10 is not The Top 10 we would have found using our Painstaking Precision method, it will contain many documents that would have been in that top 10 or near it.

It's like having a bowl of M&M's and wanting to eat red ones. You could sort them out painstakingly and then go for the red ones… or you could grab in that area where you see most of the red ones seem to be.

In order of appearance, images courtesy of westpark, Irina Souiki and jacalynsnana

Post from: Search Engine People SEO Blog

How Search Really Works: Grabbing Most Red M&M's

Another way we can assess the relevance of a document is by term weighting.

From the keyword density myth we know that true term weighting is done collection wide.

By looking at the number of documents in the index that a term appears in we can make a measurement of information: how good, how special… how meaningful is this word?

The word the would not be special at all, appearing in way too many documents. Its worth would be close to zero.

But klebenleiben ("the reluctance to stop talking about a certain subject" …)would be very special indeed! Because it appears in only 18 documents among millions, its worth, its weight, would automatically be very high.

The measure is called inverse document frequency.

This measure is our weight; it is what we use to judge the relevance of a document with.

Term Frequency Times

We do so by counting the number of times a word appears in a document. We normalize that count; we adjust it so that the length of a document doesn't matter that much anymore.

We then multiply it by our weight measurement: TF x IDF. Term Frequency times Inverse Document Frequency.

In other words, a high count of a rare word = a high score for that document, for that word. But… a high count of a common word = not so high score for that document, for that word.

Vectors

A vector is a line of a certain length into a certain direction.

Both the length and the direction of the line represent important information.

Vectors enable us to represent, to talk about, size and direction when position is irrelevant. Wind speed, velocity, force, acceleration; all these are good candidates to be represented as a vector.

TFxIDF scores are perfectly suited to be represented as vectors.

Vector Space

Think of the words that make up our index as axes of a space.

Of course in a real index this space would consists of thousands upon thousands of axes…

Documents as Vectors

For each word in our document we can draw a line (vector) which shows its TFxIDF score for a certain term.

Queries as Vectors

Every word in a query can also be shown as a vector.

By looking at documents that are "near" our query we can rank (sort) documents in our result set.

TFxIDF Vector Space Ranking

If a document is close to our query it answers our query.

But better yet: documents close to ours are similar documents. They're talking about roughly the same thing.

This makes TFxIDF vector space ranking extremely useful to find sets of similar documents through "closeness".

Post from: Search Engine People SEO Blog

How Search Really Works: Relevance (2) – Vector Space

Search is always boolean: yes or no. True or false.

Either the words are in the document or not.

But as you see, not all documents are "born alike". Some are about our topic, some just mention it.

What we need, what we want, is not just a big list of results — we want a relevant list of results, preferably sorted so that the best bet appears on top.

Boolean Zone Scoring

Zone scoring uses multiplication values (weights) to calculate the "relevance" of the occurrences of our search term based on how it appears in which zone of the document.

Document zones we're all familiar with are header/title, body/content, footer.

These weights are generally machine-learned by running test queries on a clean, non-spammed, non-gamed index. Relevance judges gauge how relevant the test results are.

Next week: Term Weight Scoring

Post from: Search Engine People SEO Blog

How Search Really Works: Relevance (1)

While human beings can scan a page and see if the whole phrase "a grandiloquent dictionary" appears on it, a search engine can't.

A search engine needs to:

1. Lookup the occurrences for each word in the phrase
2. See if the positions of words in the document fit the phrase

As a search engine isn't smart it needs to work smart.

Leverage Keyword Frequency

By storing the frequency with which a word appears in the whole index we can right away cut down to the smallest set from which to draw results.

Instead of selecting 15,570,000,000 documents in which "a" occurs and then checking which have the words grandiloquent and dictionary we can immediately limit the set to 222,000 documents; those documents that contain the relatively rare grandiloquent.

Post from: Search Engine People SEO Blog

How Search Really Works: Simple Query Optimization

Memory is much faster than looking things up.

In order for a search engine in high demand to serve its users efficiently it should keep things in memory instead of looking it up on a disk.

Traditionally large scale search engines will keep their complete dictionary in memory and the posting list on disk.

Inefficient Storage

Obviously the more you can keep in memory and the more information can be read back with one disk action, the better.

Unfortunately computer information gets inefficiently stored in boxes with fixed dimension: if a box is 10 characters wide a 4 character word still takes up 1 box of 10 characters.

Compression

The solution is to squeeze information together so the least amount of space contains the maximum amount of information.

Big String Compressed Dictionary

could become:

By adding the length of each word to every entry we can make the list of words hundreds of characters shorter.

Gap Compressed Posting List

could become:

By storing the difference between the document ID's (the gaps) we can save hundreds of characters.

The same could be done for storing the gaps between the index numbers for the occurrence positions in each document.

This "compressed representation encodes occurrences of a term as a pointer to the next occurrence of the term to facilitate rapid enumeration of the occurrences of the term".

You could search "occurrences of the terms in the set of documents by following pointers through the compressed representation".

Partial Decompression

An index stored this way is completely lossless: it retains all information from document identifier to document positional identifier.

By starting with the least frequently used term in the search it is very easy to unravel to do a partial decompression of this index by

"identifying occurrences of the terms in the set of documents"

(the dictionary) – to then use;

"the corresponding term identifiers for the terms in the search request to look up a term offset table for a pointer to a first occurrence of the terms in the compressed representation of the set of documents"

(the very first posting document ID);

"and following a chain of pointers starting at the first occurrence to identify other occurrences of the terms in the compressed representation of the set of documents"

(the gap compressed document ID list)

Recommended reading:

• New Google Approach to Indexing and Stopwords

Post from: Search Engine People SEO Blog

How Search Really Works: The Compressed Index

Oversimplified: we have at least a few pages in our index, have extracted every single word from those pages and have written down in an index where in which pages those words occur.

Want to talk numbers? We have some very precise ones for the English language.

Google says;

"We processed 1,024,908,267,229 words of running text and are publishing the counts for all 1,176,470,663 five-word sequences that appear at least 40 times. There are 13,588,391 unique words, after discarding words that appear less than 200 times."

And that's just a part of their index…

Now comes the fun…

I have to sort what?!

That list of words in the index (the dictionary as they call it) together with the document ID numbers they have as a pointer and the positional information needs to be sorted.

Uhuh. Sorted.

Let's say each of the above mentioned unique words (13,588,391) is 5 characters long. That's 67 MegaByte right there. Say each unique word is found in one unique document and the document pointer is 5 numbers wide: that's another 67 MegaByte to store the occurrence of each unique word in one document each. Imagine the word the which most probably appears at least once in every document as well…

As you see, the memory requirements are huge and we haven't even started factoring in the storage requirements for the in-document positional pointers for the positional inverted index we know search engines use.

And once we do — we still need to factor in temporary memory to actually do something with that list; like sorting it…

Bit by Bit

The only way to handle this is to work with chunks of data which you combine later on.

A chunk, or block, is read into memory, sorted, written back. At one point you can start to merge the pre-sorted blocks and write them back into one sorted super-index.

In a small setup this is one machine reading and writing blocks but in a large scale setup this is a whole bunch of machines working with chunks of chunks.

Recognize This?

In such an index you can't randomly insert new or updated documents or remove deleted ones. You would have to re-sort on every update.

So what do you do?

You sort your index and use it: this is your main index. New stuff you find on the web goes into another, more temporary index. Call it the supplemental index. In order to deliver complete and up to date results, when people search you have to return results from both indexes.

Every once in a while you'll need to merge the new stuff from the supplemental index into the new one. If you find a lot of new stuff every day you'll need some kind of priority setup which says these entries in the supplemental index are worth the CPU time of merging them back in the main index and these are not … yet.

Of course back in the old days you would have just gone out and re-index everything thoroughly…

Post from: Search Engine People SEO Blog

How Search Really Works: Recognize This Index?

Last week we saw how an inverted index (where a list of words points to a list of documents in which they appear) is insanely useful for doing AND queries.

But what if you're not looking for any document that has the words search AND people AND engine but you're looking for Search Engine People?

Well, if document 42 in our example reads "the engine was found after a search by some people" or "people use a search engine such as Google" than a traditional inverted index would think it's spot-on for your search. Ai….

(Extended) Biword Inverted Index

One way to go about this would be to somehow generate an inverted list of phrases.

Problem: if you index phrases of 2-words length, 3+ word phrase searches become just another AND query, combining the parts of the phrase. "Long John" AND "Silver".

Indexing phrases of 3-words length simply moves the problem to 4+ word phrase searches…. etc. etc.

Problem: the inverted index becomes huge, listing every word in every document and every 2 (3? 4?) word phrase in every document….

Positional Inverted Index

The only real solution is to store not only the incidence of occurrence of a word in a document but to store the exact position(s) of the word in that document.

In this example document 42 is identified for "search engine people" because the words appear in that order: they appear in positions 1, 2 and 3.

Advantage: because the positional index is similar in construction as the traditional inverted index it inherits the same advantage. That is, when doing an AND query it can jump ahead whenever one of the words doesn't occur in the document it is looking at.

Advantage: simply by looking at words occurring in the right order, any phrase of any length can be found even though it isn't indexed as such.

Advantage: by having precise position information we can do proximity queries.

Advantage: phrase matching and query word proximity can also be used to rank search results.

The Winner

Although a positional index is at least 2-4 times (or up to 50%) larger than a traditional inverted index the payoff is so large that this is the type of index in use by commercial search engines — for phrases. … In general….

Frequently searched phrases are still better stored in a biwords index; less frequently searched phrases are better processed with a positional inverted index.

Index Type & SEO

The fun (warning: geek talking!) is of course that knowing this kind of stuff implicitly explains you things.

For example, knowing that in order for a positional inverted index to really work all words, including so-called "stop words", need to be indexed makes it less surprising that stop words are dead.

Positional indexing and retrieval also makes it not only logic but expected that shop in new york and shop new york give different results.

Different results = different thinking, different SEO… different opportunities.

It's all in the index

Post from: Search Engine People SEO Blog

How Search Really Works: "The" Index (2)

If a search engine would search "live" through the documents it knows about for the occurrence of the word we're looking for it could take its time and then simply report where it found our word.

In this example our search engine has only one index: the documents itself.

However, time is something a search engine doesn't have; the query needs to be answered now.

What we need is a real index!

Boolean Index – Talk about the Matrix

The problem with a boolean index, where we put a little flag (1) or not (0) for every word for every document is that it quickly grows way and way too large.

Three documents with amongst them just four words take 12 1's or 0's — apart from the bits and bytes we need to store the word. Now imagine a matrix where one of the sides is 13,940,000,000 columns wide…

The Inverted Index

In the inverted index we record only the places (documents) where a word does occur.

It's called inverted because instead of the documents providing the occurrences of a word, the word points to which documents it occurs in.

Sorted by document pointer, the inverted index is extremely efficient in performing AND queries.

Let's reshuffle our example a little bit to make this visually clear:

If we search for documents that contain the words "search compression" and we down these rows at the same time, as soon as one row makes a jump to a higher document ID, you can jump forward in the other row as well: no use checking the intermediate ones as you now know that those won't have both words.

Knowing only about yes/no occurrences, an inverted index is horrible at phrase and proximity matching:

To be continued…

Post from: Search Engine People SEO Blog

How Search Really Works: "The" Index (1)

What is Keyword Density?

Keyword Density is a function, a calculation, of keyword frequency.

It's calculated as number of occurrences divided by number of words and is usually expressed as a percentage.

What is Keyword Density Used For?

Nothing much, really.

Keyword density can help in readability calculations.

Keyword density is also sometimes used as a simplified manner to introduce local keyword weight but should never be confused with it.

Why don't Search Engines use Keyword Density?

Search engines deal with calculations that say something about words in a document in relation to the index it appears in.

Keyword density says something about words in a document in relation to the document itself. It doesn't help you to compare and thus sort or rank a set of documents.

Frequency <> Relevance

The fact is that frequency in and of itself doesn't equate to relevance.

The word the is the most commonly used English word: it appears with the highest frequency. If a search engine would calculate relevance as frequency, all documents in its index would have the as their topic subject.

Likewise the word time is the most commonly used English noun. This would make a multitude of documents relevant to time before anything else.

Keyword Weight

To make sense of word occurrences in a document a search engine has to see those words in the context of its index.

This is done by calculating the overall importance of words both in the document and in the index.

This importance is called term weight.

To calculate the importance of a word in a document, 3 variables are needed:

• local weight: a calculation based on keyword frequency in this document. This variable can be calculated in many ways but not as a straightforward count of how many times the word appears in the document.
• global weight: calculated based upon number of documents in index divided by number of documents with the keyword.
• normalization: a calculation designed to remove the unfair advantages and disadvantages of document length. Usually you work to express the end values between 0 and 1.

None of the search engines have ever disclosed which published or unpublished scales they use for local weight or global weight.

What we're looking to achieve is to get high values for terms (words/phrases) that occur a lot of times in the relevant documents but infrequently in the index as a whole.

Keyword Density Myth Summary

Search engines use term weight to rank documents by relevance.

Term weight is calculated from the result of two other calculations: local weight and global weight.

Without knowing the function used for local weight we can't calculate it — but we do know that it's not just pure keyword frequency.

Without knowing the size of the index, the number of documents relevant to the term, and the function used for global weight we can't calculate it.

Using keyword density as a guesstimator of weight or relevance is therefore utterly useless. It's like giving you the height of a three dimensional object based on which you have to not only return its volume but also tell whether it is larger or smaller than any other unseen object in a collection you don't know about.

Hungry for more? I recommend The Keyword Density of Non-Sense.

Post from: Search Engine People SEO Blog

How Search Really Works: The Keyword Density Myth

Left to their own devices, people will assign keywords (tag or link) as they please.

They paint a rich picture of the linked content.

Keyword stuffing is the unnatural repetitive use of a specific word or phrase.

In your content….

..or your links…

Post from: Search Engine People SEO Blog

How Search Really Works: Keyword Stuffing