{"id":8897,"library":"clip-benchmark","title":"CLIP Benchmark","description":"CLIP-benchmark is a Python library designed to evaluate CLIP-like models on a standard set of datasets for various tasks, including zero-shot classification, zero-shot retrieval, linear probing, and captioning. It supports models like OpenCLIP, Japanese CLIP, and NLLB CLIP, and integrates with datasets from torchvision, TensorFlow datasets, and VTAB. The library is currently active, with version 1.6.2, and focuses on reproducible evaluation results.","status":"active","version":"1.6.2","language":"en","source_language":"en","source_url":"https://github.com/mehdidc/clip_benchmark","tags":["CLIP","benchmark","evaluation","vision-language models","zero-shot learning","machine learning"],"install":[{"cmd":"pip install clip-benchmark","lang":"bash","label":"Install stable version"}],"dependencies":[{"reason":"Required for evaluating OpenCLIP models.","package":"open_clip_torch","optional":false},{"reason":"Core deep learning framework dependency.","package":"torch","optional":false},{"reason":"Required for datasets from torchvision.","package":"torchvision","optional":true},{"reason":"Required for datasets from TensorFlow.","package":"tensorflow-datasets","optional":true},{"reason":"Used by some models and for TensorFlow dataset support.","package":"timm","optional":true}],"imports":[{"note":"While the primary interaction is via CLI, 'run_benchmark' is the core function for programmatic evaluation, typically invoked indirectly by the 'clip_benchmark' command.","symbol":"run_benchmark","correct":"from clip_benchmark.cli import run_benchmark"}],"quickstart":{"code":"clip_benchmark eval --dataset=cifar10 --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64","lang":"bash","description":"This command runs a zero-shot classification benchmark on the CIFAR-10 dataset using an OpenCLIP pre-trained model (ViT-B-32-quickgelu trained on LAION-400m) and saves the results to `result.json`."},"warnings":[{"fix":"Refer to the documentation for examples on using templated paths, especially for Hugging Face WebDatasets.","message":"The `--dataset_root` and `--output` arguments support templating (e.g., `wds_{dataset_cleaned}`). Ensure you understand the templating syntax when specifying paths to avoid unexpected file locations or dataset loading issues.","severity":"gotcha","affected_versions":">=1.0.0"},{"fix":"Consult the `clip-benchmark` documentation on 'Evaluating on a webdataset' for instructions on preparing and using WebDatasets.","message":"When working with WebDatasets, the conversion process may require specific tools (e.g., `webdataset` utilities) and manual uploading to platforms like Hugging Face Hub, which is not fully automated by the library itself.","severity":"gotcha","affected_versions":">=1.0.0"},{"fix":"Install necessary optional dependencies based on the datasets you plan to benchmark. Check the library's `README` or `datasets/builder.py` for specific requirements.","message":"Some dataset types (e.g., TensorFlow Datasets, VTAB) require additional installations beyond `clip-benchmark` itself. For instance, TensorFlow Datasets may need `tfds-nightly` and `timm`, while VTAB requires its dedicated package.","severity":"gotcha","affected_versions":">=1.0.0"},{"fix":"Refer to `clip_benchmark/models/open_clip.py` as an example for structuring your model loading function and ensure it returns the (model, transform, tokenizer) tuple as expected.","message":"When adding support for new custom CLIP models, you must define a specific model loading function and integrate it into `clip_benchmark/models/__init__.py`'s `TYPE2FUNC` mapping. This requires understanding the internal model loading mechanism.","severity":"gotcha","affected_versions":">=1.0.0"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Ensure `clip-benchmark` is installed correctly (`pip install clip-benchmark`) and that your shell's PATH includes the directory where pip installs executables (e.g., `~/.local/bin` on Linux/macOS or `Scripts` folder in Python install dir on Windows).","cause":"The `clip-benchmark` command-line interface tool is not found in your system's PATH. This usually happens if pip's script directory isn't included in PATH, or if the installation was incomplete.","error":"command 'clip_benchmark' not found"},{"fix":"Install the `open_clip_torch` dependency: `pip install open_clip_torch`.","cause":"You are attempting to benchmark an OpenCLIP model, but the `open_clip_torch` package, which provides the OpenCLIP implementation, is not installed.","error":"ModuleNotFoundError: No module named 'open_clip_torch'"},{"fix":"Install the required package: `pip install tensorflow-datasets tfds-nightly timm`.","cause":"You are trying to use a TensorFlow dataset (e.g., `tfds/imagenet_v2`) but the `tensorflow-datasets` library is not installed.","error":"Failed to load dataset 'tfds/imagenet_v2': module 'tensorflow_datasets' not found"}]}