pytrec_eval

0.5 verified Fri May 01 auth: no python

Python bindings for the trec_eval information retrieval evaluation tool, providing standard IR metrics like MAP, NDCG, P@k, etc. Supports Python 3+. Current version 0.5 is the latest stable release. Updates are infrequent.

pip install pytrec-eval

Common errors

error trec_eval: error while loading shared libraries: libtrec_eval.so: cannot open shared object file: No such file or directory ↓

cause trec_eval C library not installed or not in library path.

fix

Install trec_eval (e.g., apt install trec_eval on Debian/Ubuntu) and ensure LD_LIBRARY_PATH includes the library directory.

error TypeError: string indices must be integers ↓

cause Query or document IDs are integers, but pytrec_eval expects string keys.

fix

Cast all IDs to strings: {str(qid): {str(docid): score}}

Warnings

gotcha trec_eval binary must be installed separately and on PATH. pytrec_eval does not ship the binary. ↓

fix Install trec_eval via system package manager or build from source, and ensure trec_eval is in PATH.

gotcha Input dictionaries must have string keys for query and document IDs. Integer keys may cause type errors. ↓

fix Use str(query_id) and str(doc_id) when building qrel and run dictionaries.

breaking Version 0.5 dropped Python 2 support. Older code using Python 2 will break. ↓

fix Upgrade to Python 3.

Imports

pytrec_eval
```
import pytrec_eval
```
Standard import, no submodule.

Quickstart

Evaluate a run against qrels using MAP and NDCG.

import pytrec_eval

qrel = {
    'q1': {
        'd1': 1,
        'd2': 0,
    }
}

run = {
    'q1': {
        'd1': 0.9,
        'd2': 0.1,
    }
}

evaluator = pytrec_eval.RelevanceEvaluator(qrel, {'map', 'ndcg'})
results = evaluator.evaluate(run)
print(results)