ROUGE Metric

Common errors

• FileNotFoundError: [Errno 2] No such file or directory: 'ROUGE-1.5.5.pl'

  cause This error typically occurs when using `rouge_metric.PerlRouge` on a system (especially Windows) where the ROUGE-1.5.5.pl script or its Perl interpreter (e.g., Strawberry Perl) is not installed or not correctly added to the system's PATH.
  fix Install Strawberry Perl (on Windows) and ensure its binary folder is added to your system's PATH environment variable. Alternatively, use the pure Python implementation `rouge_metric.Rouge` if Perl script compatibility is not strictly required.

• ModuleNotFoundError: No module named 'rouge_metric'

  cause The `rouge-metric` package is not installed in your current Python environment.
  fix Run `pip install rouge-metric` in your terminal to install the library.

• TypeError: evaluate() missing 1 required positional argument: 'reference'

  cause The `evaluate` method of the `Rouge` class requires both a hypothesis and a reference string as input.
  fix Ensure you pass both the `hypothesis` and `reference` strings (or tokenized lists for `evaluate_from_tokens`) to the evaluation method, e.g., `rouge.evaluate(hypothesis_text, reference_text)`.

Warnings

• gotcha ROUGE metrics primarily rely on n-gram overlap and do not capture semantic meaning or contextual understanding. This can lead to high scores for syntactically similar but semantically divergent texts. It is recommended to complement ROUGE with other metrics (e.g., BERTScore) or human evaluation for a comprehensive assessment.
  Fix: Combine ROUGE scores with semantic similarity metrics (e.g., BERTScore) and qualitative human evaluation to get a more robust assessment of text quality.
• gotcha The multi-document evaluation results from the pure Python implementation (rouge_metric.Rouge) may be slightly different from those produced by the official ROUGE-1.5.5.pl Perl script (accessed via rouge_metric.PerlRouge) because the Python implementation does not use bootstrap resampling.
  Fix: If exact replication of ROUGE-1.5.5.pl multi-document scores is critical, use the `PerlRouge` wrapper or adjust expectations for slight differences in multi-document scenarios with `Rouge`.
• gotcha The pure Python implementation (rouge_metric.Rouge) expects pre-tokenized sentences (lists of tokens). Preprocessing steps like tokenization, stemming, and stopword removal are left to the client, which can impact scores if not handled consistently.
  Fix: Ensure consistent and appropriate preprocessing (tokenization, stemming, etc.) is applied to both hypothesis and reference texts before passing them to `rouge_metric.Rouge`.
• deprecated The `PerlRouge` wrapper, which calls the official ROUGE-1.5.5.pl script, is primarily intended for English corpora. For non-English summaries, it is recommended to use the pure Python implementation (`rouge_metric.Rouge`).
  Fix: For non-English text evaluation, use `rouge_metric.Rouge` and ensure your input tokens are appropriately preprocessed for the target language. For English, `PerlRouge` can still be used for official ROUGE-1.5.5.pl compatibility.

Install

• pip install rouge-metric Install stable version from PyPI
• pip install git+https://github.com/li-plus/rouge-metric.git@master Install latest version from GitHub

Imports

• Rouge

  from rouge_metric import Rouge

  For the pure Python implementation of ROUGE.
• PerlRouge

  from rouge_metric import PerlRouge

  For the Python wrapper around the official ROUGE-1.5.5.pl Perl script.

Quickstart

This quickstart demonstrates how to use the pure Python ROUGE implementation to evaluate a single hypothesis against a single reference string, and also how to evaluate tokenized input against multiple references. The output includes ROUGE-N, ROUGE-L, ROUGE-W, ROUGE-S, and ROUGE-SU scores.

from rouge_metric import Rouge

hypothesis = 'The cat sat on the mat.'
reference = 'The cat was on the mat.'

rouge = Rouge()
scores = rouge.evaluate(hypothesis, reference)

print(scores)

# Example with multiple references (list of lists of tokens)
hyp_tokens = ['the', 'cat', 'sat', 'on', 'the', 'mat']
ref1_tokens = ['the', 'cat', 'was', 'on', 'the', 'mat']
ref2_tokens = ['a', 'feline', 'was', 'resting', 'on', 'the', 'rug']

scores_multi_ref = rouge.evaluate_from_tokens(hyp_tokens, [ref1_tokens, ref2_tokens])
print(scores_multi_ref)