This post is part of an ongoing series: How Search Really Works.
Previously: Relevance (1)
Another way we can assess the relevance of a document is by term weighting.
From the keyword density myth we know that true term weighting is done collection wide.
By looking at the number of documents in the index that a term appears in we can make a measurement of information: how good, how special… how meaningful is this word?
The word the would not be special at all, appearing in way too many documents. Its worth would be close to zero.







