CLIP (Contrastive Language-Image Pre-training)

Warnings

• gotcha The `openai-clip` package on PyPI is an unofficial wrapper around OpenAI's official CLIP GitHub repository. For the most up-to-date and officially supported version, it is recommended to install directly from the OpenAI CLIP GitHub repository.
  Fix: Use `pip install git+https://github.com/openai/CLIP.git` instead of `pip install openai-clip`.
• breaking Newer versions of `setuptools` (81+) cause build failures due to the removal of `pkg_resources`, which `clip` (from OpenAI's GitHub) might still implicitly use.
  Fix: Pin `setuptools` to a version below 81 (`pip install 'setuptools<81'`) or ensure your Python environment uses compatible versions if encountering build errors during installation.
• gotcha CLIP's performance can be sensitive to the phrasing of text prompts ('prompt engineering'). Slight variations in wording can significantly impact classification accuracy.
  Fix: Experiment with different prompt templates (e.g., 'a photo of {class}', 'this is a {class}') and systematically test prompt variations for your specific use case. Consider ensembling predictions from multiple prompts.
• gotcha CLIP may struggle with tasks requiring precise spatial reasoning, counting, or very fine-grained classification (e.g., distinguishing between similar car models or flower species). It also exhibits poor generalization to images not well-represented in its pre-training data.
  Fix: Be aware of these limitations. For highly specific or fine-grained tasks, fine-tuning CLIP on domain-specific data or using task-specific models might be necessary. Augmenting prompts or using retrieval-augmented approaches can sometimes help.
• gotcha Models trained on internet-scale data like CLIP can inherit and exhibit social biases present in the training datasets.
  Fix: Exercise caution when deploying CLIP in sensitive applications. Conduct thorough bias evaluations for your specific use cases and consider ethical implications. Techniques like debiasing embeddings or careful prompt engineering can mitigate some issues.

Install

• pip install openai-clip Unofficial PyPI package
• pip install git+https://github.com/openai/CLIP.git Official (GitHub source) - Recommended
• conda install --yes -c pytorch pytorch=1.7.1 torchvision cudatoolkit=11.0 && pip install ftfy regex tqdm && pip install git+https://github.com/openai/CLIP.git Official (GitHub source) with PyTorch and dependencies (CUDA 11.0)

Imports

• clip

  import clip

• CLIPModel, CLIPProcessor

  from transformers import CLIPModel, CLIPProcessor

  For using CLIP via Hugging Face Transformers, which is an alternative and often recommended approach for ease of use.

Quickstart

This quickstart loads a pre-trained CLIP model (ViT-B/32) and its associated preprocessing function. It then processes a dummy image and a list of text labels, computes the image and text features, and calculates the similarity scores to predict the most relevant text snippet for the image. It runs on CUDA if available, otherwise falls back to CPU.

import torch
import clip
from PIL import Image
import os

# Ensure 'CLIP.png' exists or replace with a valid image path
# For demonstration, let's create a dummy image file if it doesn't exist
if not os.path.exists('CLIP.png'):
    try:
        from io import BytesIO
        import base64
        dummy_image_b64 = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAAAAYAAjCB0C8AAAAASUVORK5CYII="
        img_data = base64.b64decode(dummy_image_b64)
        with open('CLIP.png', 'wb') as f:
            f.write(img_data)
    except ImportError:
        print("Pillow not installed or cannot create dummy image. Please provide a real image path.")
        exit()

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

image = preprocess(Image.open("CLIP.png")).unsqueeze(0).to(device)
text = clip.tokenize(["a diagram", "a dog", "a cat"]).to(device)

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    
    logits_per_image, logits_per_text = model(image, text)
    probs = logits_per_image.softmax(dim=-1).cpu().numpy()

print("Label probs:", probs)
# Expected output for a blank image and these labels might be somewhat uniform or biased, 
# but demonstrates the process. For a real image, probabilities would be skewed.