LAION CLAP

Common errors

• ModuleNotFoundError: No module named 'clap_module'

  cause Attempting to import `CLAP` from `clap_module.model` after installing `laion-clap` from PyPI.
  fix Use the correct import path: `from laion_clap import CLAP`.

• RuntimeError: CUDA out of memory. Tried to allocate XXX MiB (GPU XXX; XXX MiB total capacity; XXX MiB already allocated; XXX MiB free; XXX MiB reserved in total by PyTorch)

  cause The input audio/text batch size or model size exceeds the available GPU memory.
  fix Reduce the batch size for text or audio inputs. If using a large CLAP model version, consider a smaller one or upgrading your GPU.

• soundfile.LibsndfileError: Error opening 'path/to/audio.wav': File not found.

  cause The specified audio file path is incorrect or the file does not exist. This error typically occurs when `soundfile` attempts to load a non-existent file.
  fix Double-check the file path. Ensure the file exists and is accessible. Verify any relative paths are correct based on the current working directory.

• KeyError: 'CLAP_512' (or similar model name)

  cause Attempting to load a CLAP model version that does not exist or whose name is misspelled.
  fix Verify the correct model version names from the `laion-clap` documentation or GitHub repository. Common versions include 'CLAP_512', 'CLAP_630k'. Ensure the library version supports the requested model.

Warnings

• gotcha The CLAP model downloads its pre-trained weights (approx. 600MB-1.5GB depending on the version) to a cache directory on the first initialization. This can be slow and requires an active internet connection. Ensure sufficient disk space and network connectivity.
  Fix: Ensure stable internet connection. The cache directory is typically `~/.cache/torch/hub/checkpoints/`. You can pre-download if needed, but it's usually handled automatically.
• gotcha When processing audio, CLAP typically expects a specific sampling rate (e.g., 48000 Hz) and mono channel. Providing audio with different characteristics without resampling can lead to suboptimal embeddings or errors.
  Fix: Use the `resample=True` argument in `get_audio_embeddings` or manually resample/mixdown audio to 48kHz mono before passing it to the model. Libraries like `torchaudio` or `librosa` are useful for this.
• gotcha Using the CLAP model on CPU can be significantly slower than using a GPU (CUDA). For larger batches or real-time applications, a CUDA-enabled GPU is highly recommended.
  Fix: Ensure PyTorch is installed with CUDA support (`pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118` for CUDA 11.8). Verify CUDA availability with `torch.cuda.is_available()` and pass `use_cuda=True` during model initialization.
• deprecated Older examples or internal code might refer to `clap_module.model.CLAP`. When installing via `pip install laion-clap`, this import path is incorrect.
  Fix: Always use `from laion_clap import CLAP` after installing the `laion-clap` PyPI package.

Install

• pip install laion-clap Basic installation
• pip install laion-clap[full] Full installation with all audio dependencies

Imports

• CLAP
  wrong:

  from clap_module.model import CLAP

  correct:

  from laion_clap import CLAP

  When installed via `pip install laion-clap`, the main class `CLAP` is exposed directly from the top-level `laion_clap` package, not from an internal module like `clap_module.model`.

Quickstart

This quickstart demonstrates how to initialize the CLAP model, generate embeddings for both text and (dummy) audio inputs, and calculate the similarity between them. The model weights are downloaded automatically on the first run. For actual audio files, use libraries like `soundfile`, `torchaudio`, or `librosa` to load them into PyTorch tensors before passing them to `get_audio_embeddings`.

import torch
from laion_clap import CLAP

# Determine device (CUDA if available, otherwise CPU)
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using device: {device}")

# Initialize the CLAP model (default 'CLAP_512' model is loaded)
# This will download model weights on first run (can be ~600MB).
model = CLAP(version='CLAP_512', use_cuda=torch.cuda.is_available())

# --- Text Embedding Example ---
text_data = [
    "A clear audio recording of a dog barking.",
    "The sound of waves crashing on the shore."
]
text_embeddings = model.get_text_embeddings(text_data)
print(f"Text embeddings shape: {text_embeddings.shape}")

# --- Audio Embedding Example ---
# For a runnable quickstart without needing actual audio files, 
# we generate dummy audio data. In a real scenario, you'd load files.
# CLAP expects audio at 48kHz sampling rate, mono channel.

sample_rate = 48000
duration_seconds = 5
# Generate a batch of 2 mono audio tensors (2 x 5 seconds at 48kHz)
dummy_audio = torch.randn(2, sample_rate * duration_seconds)

# Move audio to the correct device
audio_data_tensors = [d.to(device) for d in dummy_audio]

# Get audio embeddings. `resample=True` is often helpful to handle 
# potential mismatches in sample rates, though here our dummy data matches.
audio_embeddings = model.get_audio_embeddings(audio_data_tensors, resample=True)
print(f"Audio embeddings shape: {audio_embeddings.shape}")

# --- Similarity Calculation ---
# Normalize embeddings for cosine similarity
text_embeddings_norm = text_embeddings / text_embeddings.norm(dim=-1, keepdim=True)
audio_embeddings_norm = audio_embeddings / audio_embeddings.norm(dim=-1, keepdim=True)

similarity = torch.matmul(text_embeddings_norm, audio_embeddings_norm.T)
print(f"\nSimilarity scores (text x audio):\n{similarity.cpu().numpy()}")
# Expected: High similarity for text[0] with audio[0], text[1] with audio[1] (if embeddings were meaningful)