OpenCLIP

Warnings

• gotcha When using `timm`-based image encoders (e.g., ConvNeXt, SigLIP, EVA), ensure you have the latest `timm` library installed. Older versions may result in 'Unknown model' errors.
  Fix: pip install -U timm
• breaking The default activation function for models changed from `QuickGELU` to `torch.nn.GELU` in newer PyTorch versions. For OpenCLIP pretrained weights, using model definitions with a `-quickgelu` postfix (e.g., 'ViT-B-32-quickgelu') is necessary to match the original training and avoid an accuracy drop, especially during fine-tuning.
  Fix: Specify model definitions with a `-quickgelu` postfix when loading OpenCLIP pretrained weights (e.g., `open_clip.create_model_and_transforms('ViT-B-32-quickgelu', ...)`).
• gotcha Mismatch between installed `torch` and `open-clip-torch` versions can lead to `ModuleNotFoundError` or other runtime issues. Ensure compatible versions are installed, often by following PyTorch's installation instructions for your CUDA version before installing OpenCLIP.
  Fix: Verify `torch` and `open-clip-torch` compatibility. Downgrade `open_clip_torch` or upgrade `torch` as needed. Consult OpenCLIP's GitHub issues for known compatibility pairs.
• gotcha For optimal performance and consistency with original CLIP, OpenCLIP is designed to be used within a mixed-precision context (e.g., `torch.autocast('cuda')`) as OpenAI's original models utilized mixed-precision. Without it, there might be slight numerical differences in embeddings or reduced performance on GPU.
  Fix: Wrap inference calls with `with torch.no_grad(), torch.autocast('cuda'):` for GPU inference.
• gotcha If you are using models that rely on transformer tokenizers (e.g., certain text encoders), the `transformers` library must be installed separately, as it is an optional dependency for `open-clip-torch`.
  Fix: pip install transformers

Install

• pip install open_clip_torch Base Installation
• pip install open_clip_torch[training] With Training Dependencies
• pip install -U timm Update timm (Recommended for ConvNeXt, SigLIP, EVA encoders)
• pip install transformers Install transformers (If using transformer-based tokenizers)

Imports

• open_clip

  import open_clip

• create_model_and_transforms

  model, _, preprocess = open_clip.create_model_and_transforms(...)

• get_tokenizer

  tokenizer = open_clip.get_tokenizer(...)

  While 'from open_clip import tokenizer' works, the recommended pattern is `open_clip.get_tokenizer()` to retrieve the correct tokenizer instance for a given model.

Quickstart

This quickstart demonstrates how to load a pre-trained OpenCLIP model, preprocess a dummy image and text, then compute the zero-shot similarity probabilities between the image and the given text labels. It includes loading the model, tokenizer, and performing inference with feature normalization.

import torch
from PIL import Image
import open_clip
import io
import base64

# Create a dummy image (in a real scenario, load from file or URL)
dummy_image_data = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAAAAYAAjCB0C8AAAAASUVORK5CYII="
image = Image.open(io.BytesIO(base64.b64decode(dummy_image_data))).convert('RGB')

# 1. Load model and preprocessing transforms
model, _, preprocess = open_clip.create_model_and_transforms(
    'ViT-B-32',
    pretrained='laion2b_s34b_b79k'
)
model.eval() # Set model to evaluation mode

# 2. Get tokenizer
tokenizer = open_clip.get_tokenizer('ViT-B-32')

# 3. Prepare inputs
image_input = preprocess(image).unsqueeze(0) # Add batch dimension
text_input = tokenizer(["a diagram", "a dog", "a cat"])

# 4. Run inference
with torch.no_grad(): # Disable gradient computation for inference
    image_features = model.encode_image(image_input)
    text_features = model.encode_text(text_input)

    # Normalize features
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    # Compute similarity scores
    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probabilities:", text_probs)

# Optional: Interpret results
labels = ["a diagram", "a dog", "a cat"]
top_prob, top_idx = text_probs[0].max(dim=0)
print(f"Predicted: {labels[top_idx]} ({top_prob.item():.1%} confidence)")