How Search Really Works: Relevance (2) – Vector Space

This post is part of an ongoing series: How Search Really Works.
Previously: Relevance (1)

Another way we can assess the relevance of a document is by term weighting.

From the keyword density myth we know that true term weighting is done collection wide.

By looking at the number of documents in the index that a term appears in we can make a measurement of information: how good, how special... how meaningful is this word?

The word the would not be special at all, appearing in way too many documents. Its worth would be close to zero.

But klebenleiben ("the reluctance to stop talking about a certain subject" ...)would be very special indeed! Because it appears in only 18 documents among millions, its worth, its weight, would automatically be very high.

The measure is called inverse document frequency.

This measure is our weight; it is what we use to judge the relevance of a document with.

Term Frequency Times

We do so by counting the number of times a word appears in a document. We normalize that count; we adjust it so that the length of a document doesn't matter that much anymore.

We then multiply it by our weight measurement: TF x IDF. Term Frequency times Inverse Document Frequency.

In other words, a high count of a rare word = a high score for that document, for that word. But... a high count of a common word = not so high score for that document, for that word.

Vectors

A vector is a line of a certain length into a certain direction.

Both the length and the direction of the line represent important information.

Vectors enable us to represent, to talk about, size and direction when position is irrelevant. Wind speed, velocity, force, acceleration; all these are good candidates to be represented as a vector.

TFxIDF scores are perfectly suited to be represented as vectors.

Vector Space

Think of the words that make up our index as axes of a space.

vector-space

Of course in a real index this space would consists of thousands upon thousands of axes...

Documents as Vectors

For each word in our document we can draw a line (vector) which shows its TFxIDF score for a certain term.

vector-space-documents

Queries as Vectors

Every word in a query can also be shown as a vector.

vector-space-documents-queries

By looking at documents that are "near" our query we can rank (sort) documents in our result set.

TFxIDF Vector Space Ranking

If a document is close to our query it answers our query.

But better yet: documents close to ours are similar documents. They're talking about roughly the same thing.

This makes TFxIDF vector space ranking extremely useful to find sets of similar documents through "closeness".

About the Author: Ruud Hein

My paid passion at Search Engine People sees me applying my passions and knowledge to a wide array of problems, ones I usually experience as challenges. People who know me know I love coffee.

Ruud Hein

Additional Posts

Friday Funnies: Regrettable Choice

Fiddle Dee Dee – Is Yahoo Gone With The Wind?

Yahoo – Ever Corner a Wild Animal?

Read previous post:
Friday Funnies: Regrettable Choice

OK, I'll admit it, I read the comments on YouTube...but I don't think there is any truth to the rumor...Read...

Close