{"library":"mteb","title":"Massive Text Embedding Benchmark (MTEB)","description":"MTEB (Massive Text Embedding Benchmark) is a Python framework for evaluating embeddings and retrieval systems across diverse NLP tasks, including classification, clustering, retrieval, reranking, and semantic textual similarity. It supports over 1000 languages and various modalities like text and image, with continuous expansion. As of version 2.12.16, it aims to provide a standardized, comprehensive, and reproducible way to compare embedding models. The library maintains a frequent release cadence with minor updates often occurring weekly.","language":"python","status":"active","last_verified":"Fri May 15","install":{"commands":["pip install mteb"],"cli":{"name":"mteb","version":""}},"imports":["from mteb import MTEB","import mteb\nresults = mteb.evaluate(model, tasks=tasks)","import mteb\nmodel = mteb.get_model('sentence-transformers/all-MiniLM-L6-v2')","import mteb\ntasks = mteb.get_tasks(tasks=['Banking77Classification.v2'])","from sentence_transformers import SentenceTransformer"],"auth":{"required":false,"env_vars":[]},"quickstart":{"code":"import mteb\nfrom sentence_transformers import SentenceTransformer\n\n# Select a model to evaluate\nmodel_name = \"sentence-transformers/all-MiniLM-L6-v2\"\n# It's recommended to use mteb.get_model for reproducibility if the model is in MTEB's registry\n# Otherwise, SentenceTransformer can be used directly\nmodel = mteb.get_model(model_name) # Will fall back to SentenceTransformer if not registered in MTEB\n\n# Select tasks to run (e.g., a specific classification task)\ntasks = mteb.get_tasks(tasks=[\"Banking77Classification.v2\"], languages=[\"eng\"])\n\n# Evaluate the model on the selected tasks\nprint(f\"Running evaluation for {model_name} on {len(tasks)} tasks...\")\nresults = mteb.evaluate(model, tasks=tasks)\n\nprint(\"Evaluation complete. Results:\")\nfor task_name, task_results in results.items():\n    print(f\"Task: {task_name}\")\n    print(f\"  Main score: {task_results['main_score']:.4f}\")\n    # Example of accessing detailed metrics\n    if 'accuracy' in task_results['mteb_results']:\n        print(f\"  Accuracy: {task_results['mteb_results']['accuracy']:.4f}\")\n\n# To save results to a specific folder\n# output_folder = f\"./results/{model_name.replace('/', '_')}\"\n# results = mteb.evaluate(model, tasks=tasks, output_folder=output_folder)\n# print(f\"Results saved to: {output_folder}\")","lang":"python","description":"This quickstart demonstrates how to load a pre-trained Sentence Transformer model and evaluate it on a specific MTEB task using the `mteb.evaluate` function. It showcases how to select tasks and retrieve the evaluation results.","tag":null,"tag_description":null,"last_tested":null,"results":[]},"compatibility":{"tag":null,"tag_description":null,"last_tested":"2026-05-15","installed_version":null,"pypi_latest":"2.12.30","is_stale":null,"summary":{"python_range":"3.10–3.9","success_rate":0,"avg_install_s":null,"avg_import_s":null,"wheel_type":null},"results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"mteb","exit_code":1,"wheel_type":null,"failure_reason":"build_error","import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"mteb","exit_code":1,"wheel_type":null,"failure_reason":"timeout","import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"mteb","exit_code":1,"wheel_type":null,"failure_reason":"build_error","import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"mteb","exit_code":1,"wheel_type":null,"failure_reason":"timeout","import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"mteb","exit_code":1,"wheel_type":null,"failure_reason":"build_error","import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"mteb","exit_code":1,"wheel_type":null,"failure_reason":"timeout","import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"mteb","exit_code":1,"wheel_type":null,"failure_reason":"build_error","import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"mteb","exit_code":1,"wheel_type":null,"failure_reason":"timeout","import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"mteb","exit_code":1,"wheel_type":null,"failure_reason":"build_error","import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"mteb","exit_code":1,"wheel_type":null,"failure_reason":"timeout","import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null}]}}