|  Chirag Shah | Last update: December 19, 2008 | |
| Assignment-8: Evaluation-2 |
|---|
| Assigned on: 10/15/2008, Due on: 10/19/2008 |
| 1. Build an index using Krovetz stemmer and stop words removal, taking the TREC file given from the LA Times (with over 130,000 documents). Run topic 402 with its title, description, and narrative parts as queries. Calculate (1) bpref for each query, (2) MAP, (3) GMAP, (4) reciprocal rank for each query, and (5) MRR using 'trec_eval'. [5 points] |
| 2. Build an index using Porter stemmer and stop words removal, taking the first 10 countries' descriptions from the CIA World Factbook as documents. Use independence as a query and rank all the documents (or as many as you can) using TFIDF as well as Okapi retrieval models. Compare these rank-lists using (1) Pearson's covariance, (2) Spearman's Rho test, and (3) Kendall's Tau test. [3 points] What can you say based on these values? Give your comments in 2-4 sentences. [2 points] |
| | Chirag Shah | Last update: December 19, 2008 | |