sacreBLEU

2.6.0 · active · verified Thu Apr 09

sacreBLEU is a Python library providing hassle-free computation of shareable, comparable, and reproducible BLEU, chrF, and TER scores for machine translation evaluation. It is actively maintained, with the current version being 2.6.0, and releases typically occur several times a year to add new test sets, tokenizers, or minor features.

Warnings

breaking The default output format for the CLI utility changed from single-line to JSON. Tools or scripts parsing sacrebleu's standard output will need to adapt.
Fix: Update scripts to parse JSON output. Use the `--format text` option to revert to the old single-line format for compatibility, e.g., `sacrebleu -t wmt14 -l en-de --format text`.
breaking Python 3.5, 3.6, and 3.8 support have been dropped in recent major versions. The library now requires Python >=3.9.
Fix: Upgrade your Python environment to 3.9 or higher. For older Python versions, use an earlier sacrebleu version (e.g., sacrebleu<2.0 for Python <3.6, sacrebleu<2.6 for Python 3.8).
breaking The default smoothing method and floor value for `corpus_bleu()` and `sentence_bleu()` changed, potentially yielding different scores compared to earlier versions.
Fix: Review your code to ensure the desired smoothing method (`floor`, `add-k`, `exp`) and `smooth_value` are explicitly passed to `corpus_bleu()` or `sentence_bleu()` to ensure consistent scoring behavior.
gotcha Some language-specific tokenizers require extra dependencies that are not installed by default. For example, Japanese (`-tok ja`) and Korean (`-tok ko-mecab`) tokenizers need additional packages.
Fix: Install with optional extras, e.g., `pip install sacrebleu[ja]` for Japanese. For Korean tokenizers like `ko-mecab`, you might need to manually install `mecab-ko`.

Install

pip install sacrebleu Install sacreBLEU

Imports

corpus_bleu

import sacrebleu
score = sacrebleu.corpus_bleu(hyp, [ref])

BLEU, CHRF, TER
```
from sacrebleu.metrics import BLEU, CHRF, TER
```
Newer object-oriented API for metrics introduced after v1.x.

Quickstart

Calculates the corpus BLEU score for a given hypothesis against one or more reference translations. Outputs the raw score and the formatted string.

import sacrebleu

# Example hypothesis and reference sentences
hypothesis = "The cat sat on the mat."
references = [
    "The cat is on the mat.",
    "A cat sat on the mat."
]

# Calculate corpus BLEU score
# Note: sacrebleu expects lists of sentences, even for a single hypothesis/reference
bleu_score = sacrebleu.corpus_bleu([hypothesis], [references])

print(f"BLEU score: {bleu_score.score:.2f}")
print(f"BLEU string: {bleu_score.format()}")

view raw JSON →