Maximum Entropy Method

IR Evaluation Measures

The efficacy of retrieval systems is evaluated by a number of

performance measures such as average precision, R-precision, and precisions at standard cutoffs.  Broadly speaking, these measures can be classified as either system-oriented measures of overall performance (e.g., average precision and R-precision) or user-oriented measures of specific performance (e.g., precision-at-cutoff 10).  Different measures evaluate different aspects of retrieval performance, and much thought and analysis has been devoted to analyzing the quality of various different performance measures.

We consider the problem of analyzing the quality of various measures of retrieval performance and propose a model based on the maximum entropy method for evaluating the quality of a performance measure. While measures such as average precision at relevant documents, R-precision, and 11pt average precision are known to be good measures of overall performance, other measures such as precisions at specific cutoffs are not.  Our goal in this work is to develop a model within which one can numerically assess the overall quality of a given measure based on the reduction in uncertainty of a system's performance one gains by learning the value of the measure.  As such, our evaluation model is primarily concerned with assessing the relative merits of system-oriented measures, but it can be applied to other classes of measures as well.