Hugging Face Evaluate

Warnings

• breaking The `use_auth_token` parameter has been deprecated across the Hugging Face ecosystem, including `evaluate`. It has been replaced by `token` for authentication.
  Fix: Update your code to use `token='your_huggingface_token'` instead of `use_auth_token=True` or `use_auth_token='your_token'` when loading metrics or interacting with the Hub.
• breaking As of v0.4.6, `evaluate` removed support for the deprecated `HfFolder` class from `huggingface_hub`. This change adds support for `huggingface_hub>=1.0`.
  Fix: Ensure `huggingface_hub` is updated to version `1.0.0` or higher to maintain compatibility and access the latest authentication mechanisms. Older `huggingface_hub` versions might cause issues with token loading.
• gotcha Many evaluation metrics and evaluators within the `evaluate` library have external dependencies that are not installed by default. These include `nltk`, `scikit-learn`, `transformers`, `datasets`, `jiwer`, etc.
  Fix: When `evaluate.load()` reports a `ModuleNotFoundError`, check the metric's documentation on the Hugging Face Hub to identify and install the required extra packages (e.g., `pip install transformers` for 'perplexity' or `pip install evaluate[full]` for a broad set of common dependencies).
• gotcha The `evaluate` library uses caching mechanisms for loaded metrics and sometimes for computation results. While beneficial for performance, this can lead to unexpected behavior if you modify metric parameters or source data without properly clearing or understanding the cache.
  Fix: If you encounter unexpected results, try restarting your environment or manually managing the cache (`evaluate.config.set_caching_path()`, `evaluate.config.HF_EVALUATE_OFFLINE`, `evaluate.config.set_verbosity()`). Refer to the official documentation for advanced cache management options if needed.

Install

• pip install evaluate Install core library
• pip install evaluate[full] Install with common optional dependencies

Imports

• load

  import evaluate
  metric = evaluate.load("accuracy")

  Most users start by importing the `evaluate` module and then calling `evaluate.load()`.

Quickstart

This quickstart demonstrates how to load a metric (e.g., 'accuracy') using `evaluate.load()` and then compute its results with sample predictions and references. While 'accuracy' doesn't require authentication, the commented section illustrates how an `HF_TOKEN` environment variable would be used with the `token` parameter for metrics or evaluators requiring access to private resources on the Hugging Face Hub. Remember that many metrics require specific input formats (e.g., text for BERTScore) and may need additional `pip install` commands for their dependencies.

import evaluate

# Load an evaluation metric
accuracy_metric = evaluate.load("accuracy")

# Prepare dummy predictions and references
predictions = [0, 1, 0, 1, 0]
references = [0, 1, 1, 0, 0]

# Compute the metric
results = accuracy_metric.compute(predictions=predictions, references=references)
print(f"Accuracy results: {results}")

# Load a metric from the Hub that requires a token (e.g., for private models/datasets)
# TOKEN = os.environ.get('HF_TOKEN', '')
# if TOKEN:
#     # Example for a metric that might need a token, like an 'evaluator'
#     # Note: 'accuracy' itself doesn't require a token for basic use.
#     # Let's simulate loading an evaluator that might need one.
#     # For actual evaluators or private resources, `token=TOKEN` would be passed.
#     # For this example, we'll just demonstrate the token parameter concept.
#     # evaluator = evaluate.load("text_classification", model_or_pipeline="username/my-private-model", token=TOKEN)
#     # print("Evaluator loaded with token (concept shown).")
#     pass # Placeholder, as 'accuracy' doesn't use token for compute

# Example with specific configuration (e.g., for BERTScore)
# bertscore_metric = evaluate.load("bertscore")
# predictions_text = ["The cat sat on the mat.", "The dog ate the bone."]
# references_text = [["A cat was on the mat."], ["A dog consumed the bone."]]
# bertscore_results = bertscore_metric.compute(predictions=predictions_text, references=references_text, lang="en")
# print(f"BERTScore results (first example): {bertscore_results['f1'][0]}")