Comments on: How Search Really Works: Relevance (2) - Vector Space http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html Canada's Search and Social Media Authority Sat, 19 Jul 2008 23:20:25 +0000 http://wordpress.org/?v=2.2.3 By: SEOs and their IDF Myths: Part 2 « IR Thoughts http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html#comment-2370 SEOs and their IDF Myths: Part 2 « IR Thoughts Thu, 03 Jul 2008 13:24:25 +0000 http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html#comment-2370 [...] Others have claimed that it is not possible to evaluate the IDF of a phrase. Even some that plan to teach IR have claimed that calling log(N/n) “inverse document frequency” is an “insult to students”. Before making a fool of themselves they should read Robertson and Sparck Jones legacy papers on the topic. [...] […] Others have claimed that it is not possible to evaluate the IDF of a phrase. Even some that plan to teach IR have claimed that calling log(N/n) “inverse document frequency” is an “insult to students”. Before making a fool of themselves they should read Robertson and Sparck Jones legacy papers on the topic. […]

]]> By: Vector Space Models and Search Engines « IR Thoughts http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html#comment-1473 Vector Space Models and Search Engines « IR Thoughts Mon, 21 Apr 2008 15:36:31 +0000 http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html#comment-1473 [...] That said, today’s post is in reaction to the article at http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html [...] […] That said, today’s post is in reaction to the article at http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html […]

]]> By: How Search Engines Do Not Work « IR Thoughts http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html#comment-1474 How Search Engines Do Not Work « IR Thoughts Thu, 17 Apr 2008 12:40:37 +0000 http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html#comment-1474 [...] 1. http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html [...] […] 1. http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html […]

]]> By: Malte Landwehr http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html#comment-1469 Malte Landwehr Wed, 16 Apr 2008 21:06:29 +0000 http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html#comment-1469 An excellent analysis of how to weight terms by their frequency. But I doubt that the two dimensional space is enough to represent the complexity needed to maintain an index of millions of documents. An excellent analysis of how to weight terms by their frequency. But I doubt that the two dimensional space is enough to represent the complexity needed to maintain an index of millions of documents.

]]> By: Dev Basu http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html#comment-1470 Dev Basu Mon, 14 Apr 2008 21:22:14 +0000 http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html#comment-1470 As usual Ruud this is a great post. It's always interesting to learn the inner workings of an SE :) As usual Ruud this is a great post. It’s always interesting to learn the inner workings of an SE :)

]]> By: Ruud Hein http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html#comment-1471 Ruud Hein Sat, 12 Apr 2008 03:20:14 +0000 http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html#comment-1471 Good indeed to point that out. Doing any of this at run time is extremely costly. There are cost reducing procedures; working with top N documents or leader/follower samples. Yet I too think that this isn't used at run time (read: query time) because the TFxIDF vector space model is geared towards words. The IDF of a words is computed; not of phrases. All in all it doesn't deliver enough bang for its buck. Worse: it's typically a model for a clean index. Boosting TF for a high IDF word is too easy when you have search access to the whole collection. It's interesting though to see how this model can find related documents. Good indeed to point that out. Doing any of this at run time is extremely costly. There are cost reducing procedures; working with top N documents or leader/follower samples.

Yet I too think that this isn’t used at run time (read: query time) because the TFxIDF vector space model is geared towards words. The IDF of a words is computed; not of phrases. All in all it doesn’t deliver enough bang for its buck.

Worse: it’s typically a model for a clean index. Boosting TF for a high IDF word is too easy when you have search access to the whole collection.

It’s interesting though to see how this model can find related documents.

]]> By: Hamlet Batista http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html#comment-1472 Hamlet Batista Fri, 11 Apr 2008 22:08:39 +0000 http://www.searchenginepeople.com/blog/how-search-really-works-relevance-2-vector-space.html#comment-1472 Hi Rudd, Excellent post as usual. It is important to mention that vector space model for ranking is not currently practical for the top search engines due to the size of their index (and the corresponding size of the document vectors). While they use huge matrices for computing the importance of the links (PageRank), the process is done offline and is query-independent. Computing such vectors are query time would be prohibitively expensive in times and resources. Cheers Hi Rudd,

Excellent post as usual. It is important to mention that vector space model for ranking is not currently practical for the top search engines due to the size of their index (and the corresponding size of the document vectors). While they use huge matrices for computing the importance of the links (PageRank), the process is done offline and is query-independent. Computing such vectors are query time would be prohibitively expensive in times and resources.

Cheers

]]>