BERTScore

0.3.13 · active · verified Sat Apr 11

BERTScore is a Python library that provides a PyTorch implementation of the BERTScore metric, a robust evaluation metric for text generation tasks. It leverages pre-trained BERT embeddings to compute a similarity score between generated and reference texts, addressing limitations of traditional metrics like BLEU. The library is actively maintained with regular updates and is currently at version 0.3.13.

Warnings

breaking BERTScore often requires specific versions of the `transformers` library. Incompatible versions can lead to `KeyError`, `AttributeError`, or incorrect scores due to changes in the underlying Hugging Face API.
Fix: Refer to the `bert-score` release notes and test with the `transformers` version specified as compatible. If encountering issues, try pinning `transformers` to a known working version (e.g., `pip install transformers<4.17.0` for `bert-score==0.3.12` or `pip install transformers>=4.17.0` for `bert-score==0.3.13`).
breaking The command-line option for rescaling with a baseline was changed from `--rescale-with-baseline` to `--rescale_with_baseline` for consistency with other options.
Fix: Update CLI calls to use `--rescale_with_baseline` instead of `--rescale-with-baseline`.
gotcha Using the `--use_fast_tokenizer` option (or `use_fast_tokenizer=True` in Python) with Hugging Face transformers can lead to different BERTScore results due to subtle differences in tokenizer implementations compared to default slow tokenizers.
Fix: Be aware that enabling fast tokenizers may alter scores. For reproducibility, explicitly set `use_fast_tokenizer` to `True` or `False` based on your desired behavior and consistently use the same setting. Compare scores with and without this option if consistency with previous results is critical.
gotcha The default or recommended model for BERTScore has evolved. While RoBERTa is often the default, models like DeBERTa (e.g., `microsoft/deberta-xlarge-mnli`) have shown higher correlation with human scores on some datasets.
Fix: Consider specifying a `model_type` like `microsoft/deberta-xlarge-mnli` (or other recommended models from the Hugging Face Model Hub) explicitly to potentially achieve better evaluation correlations. Refer to the official documentation or release notes for the latest recommendations and model performance benchmarks.

Install

pip install bert-score Install latest version

Imports

score
```
from bert_score import score
```

Quickstart

This example calculates BERTScore (Precision, Recall, and F1) between a list of candidate sentences and reference sentences. The `lang` parameter specifies the language model to use, and `verbose=True` provides detailed output during computation.

from bert_score import score

cands = ["The cat sat on the mat.", "The dog ate the food."]
refs = [["The cat was on the mat."], ["A dog consumed the meal."]]

P, R, F1 = score(cands, refs, lang="en", verbose=True)

print(f"Precision: {P.mean().item():.3f}")
print(f"Recall: {R.mean().item():.3f}")
print(f"F1 Score: {F1.mean().item():.3f}")

view raw JSON →