Massive Text Embedding Benchmark (MTEB)

Warnings

• breaking MTEB v2 introduced a large-scale refactor with breaking changes, particularly affecting direct usage of `mteb.MTEB` class and `mteb.load_results` functions. Past minor/patch releases also occasionally introduced breaking changes.
  Fix: Refer to the official MTEB documentation for the updated API, especially focusing on `mteb.evaluate`, `mteb.get_model`, and `mteb.get_tasks`. Ensure your code aligns with the new functional interface rather than directly instantiating the `MTEB` class.
• gotcha Evaluating high-performing or large multilingual models on MTEB can be computationally very expensive, requiring significant GPU resources and time, especially for tasks with large document collections like retrieval.
  Fix: Start with smaller task subsets or mini-benchmarks to estimate resource usage. Consider using optimized models or distributed evaluation setups. MTEB also offers caching mechanisms to speed up repeated evaluations.
• gotcha Models excelling on the general MTEB leaderboard might underperform on domain-specific data. The benchmark datasets may not perfectly reflect unique domain, user behavior, or query patterns.
  Fix: Always perform additional evaluations on your specific domain data to validate model suitability. MTEB can be extended with custom tasks to facilitate this.
• deprecated Directly submitting model results to the MTEB leaderboard by adding metadata to Hugging Face model cards is no longer supported.
  Fix: Follow the updated submission guidelines on the MTEB GitHub repository or documentation to ensure results are correctly associated with the model implementation.
• gotcha When evaluating existing models, it is recommended to use `mteb.get_model("{model_name}")` instead of directly using `SentenceTransformer("{model_name}")`. This ensures consistent and reproducible results as it loads the model as MTEB implemented it, accounting for specific normalizations, quantizations, or prompts.
  Fix: Replace `model = SentenceTransformer(model_name)` with `model = mteb.get_model(model_name)`. MTEB's function will fall back to `SentenceTransformer` if the model isn't specifically registered.

Install

• pip install mteb Install core library
• uv add mteb Faster installation with uv

Imports

• MTEB

  from mteb import MTEB

  The MTEB class was part of a major refactor in v2; direct use is less common than 'mteb.evaluate' or 'mteb.get_model' now.
• evaluate

  import mteb
  results = mteb.evaluate(model, tasks=tasks)

  The recommended way to run evaluations.
• get_model

  import mteb
  model = mteb.get_model('sentence-transformers/all-MiniLM-L6-v2')

  Recommended for loading existing models as implemented in MTEB for reproducibility.
• get_tasks

  import mteb
  tasks = mteb.get_tasks(tasks=['Banking77Classification.v2'])

  Used to select specific benchmark tasks.
• SentenceTransformer

  from sentence_transformers import SentenceTransformer

  Used to load models that are not yet directly implemented in MTEB's registry.

Quickstart

This quickstart demonstrates how to load a pre-trained Sentence Transformer model and evaluate it on a specific MTEB task using the `mteb.evaluate` function. It showcases how to select tasks and retrieve the evaluation results.

import mteb
from sentence_transformers import SentenceTransformer

# Select a model to evaluate
model_name = "sentence-transformers/all-MiniLM-L6-v2"
# It's recommended to use mteb.get_model for reproducibility if the model is in MTEB's registry
# Otherwise, SentenceTransformer can be used directly
model = mteb.get_model(model_name) # Will fall back to SentenceTransformer if not registered in MTEB

# Select tasks to run (e.g., a specific classification task)
tasks = mteb.get_tasks(tasks=["Banking77Classification.v2"], languages=["eng"])

# Evaluate the model on the selected tasks
print(f"Running evaluation for {model_name} on {len(tasks)} tasks...")
results = mteb.evaluate(model, tasks=tasks)

print("Evaluation complete. Results:")
for task_name, task_results in results.items():
    print(f"Task: {task_name}")
    print(f"  Main score: {task_results['main_score']:.4f}")
    # Example of accessing detailed metrics
    if 'accuracy' in task_results['mteb_results']:
        print(f"  Accuracy: {task_results['mteb_results']['accuracy']:.4f}")

# To save results to a specific folder
# output_folder = f"./results/{model_name.replace('/', '_')}"
# results = mteb.evaluate(model, tasks=tasks, output_folder=output_folder)
# print(f"Results saved to: {output_folder}")