{"id":5444,"library":"pytrec-eval-terrier","title":"pytrec-eval-terrier","description":"pytrec-eval-terrier provides Python bindings for common Information Retrieval evaluation measures, leveraging the highly optimized `trec_eval` C library. It simplifies the process of evaluating ranking performance for search systems. The current version is 0.5.10, and releases occur periodically, often tied to Python version support or minor bug fixes.","status":"active","version":"0.5.10","language":"en","source_language":"en","source_url":"https://github.com/terrierteam/pytrec_eval","tags":["information retrieval","evaluation","trec","ranking","search","performance"],"install":[{"cmd":"pip install pytrec-eval-terrier","lang":"bash","label":"Install stable version"}],"dependencies":[],"imports":[{"note":"The PyPI package is 'pytrec-eval-terrier', but the imported module is 'pytrec_eval'.","wrong":"import pytrec_eval_terrier","symbol":"pytrec_eval","correct":"import pytrec_eval"},{"symbol":"RelevanceEvaluator","correct":"from pytrec_eval import RelevanceEvaluator"}],"quickstart":{"code":"import pytrec_eval\n\n# Example QRELS (Query Relevance Judgments)\n# Format: {query_id: {doc_id: relevance_score}}\nqrels = {\n    'q1': {'d1': 1, 'd2': 0, 'd3': 1},\n    'q2': {'d4': 1, 'd5': 0}\n}\n\n# Example RUNS (System Rankings)\n# Format: {query_id: {doc_id: score}}\nruns = {\n    'q1': {'d1': 0.9, 'd2': 0.8, 'd4': 0.7},\n    'q2': {'d4': 0.95, 'd6': 0.85}\n}\n\n# Define measures to evaluate\nmeasures = pytrec_eval.supported_measures\n# Or a specific set: measures = {'map', 'ndcg_cut.10', 'recip_rank'}\n\n# Instantiate the evaluator\nevaluator = pytrec_eval.RelevanceEvaluator(qrels, measures)\n\n# Evaluate the runs\nresults = evaluator.evaluate(runs)\n\n# Print results for a specific query and measure\nprint(f\"MAP for q1: {results['q1']['map']:.4f}\")\nprint(f\"NDCG@10 for q2: {results['q2']['ndcg_cut_10']:.4f}\")\n\n# Print average results across all queries\nagg_results = pytrec_eval.compute_aggregated_results(evaluator, results, measures)\nprint(f\"Average MAP: {agg_results['map']:.4f}\")\n","lang":"python","description":"This quickstart demonstrates how to use `pytrec_eval` to evaluate a set of runs against relevance judgments. It shows how to prepare QRELs and runs as dictionaries, select evaluation measures, instantiate the `RelevanceEvaluator`, and compute both per-query and aggregated results."},"warnings":[{"fix":"Upgrade to Python 3.8 or newer. Ensure your CI/CD pipelines use a compatible Python version.","message":"Minimum required Python version was upgraded to 3.8. Users on Python 3.7 or older will encounter errors or build failures with versions 0.5.7 and newer.","severity":"breaking","affected_versions":">=0.5.7"},{"fix":"Always use `import pytrec_eval` or `from pytrec_eval import ...` in your Python code, not `pytrec_eval_terrier`.","message":"The PyPI package name is `pytrec-eval-terrier`, but the Python module to import is `pytrec_eval`.","severity":"gotcha","affected_versions":"All"},{"fix":"Ensure `build-essential` (Debian/Ubuntu) or `Development Tools` (CentOS/RHEL) and `python3-dev` (or equivalent) packages are installed on your system if you encounter build errors during `pip install`.","message":"When installing on systems without pre-built wheels (e.g., specific Linux distributions, older Python versions, or ARM architectures), compilation from source may require a C/C++ compiler (like GCC or Clang) and Python development headers (`python3-dev` or similar).","severity":"gotcha","affected_versions":"All"},{"fix":"Carefully ensure your QRELs and run files are parsed into the expected nested dictionary format before passing them to `RelevanceEvaluator`. Consult the official documentation for examples.","message":"Input QRELs and runs must adhere to specific dictionary formats: `{query_id: {doc_id: score}}`. Incorrectly structured inputs will lead to evaluation errors or unexpected results.","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"last_verified":"2026-04-13T00:00:00.000Z","next_check":"2026-07-12T00:00:00.000Z"}