pyctcdecode

Common errors

• ModuleNotFoundError: No module named 'kenlm'

  cause The `pyctcdecode[kenlm]` extra was not installed, or its installation failed during compilation, preventing the import of the `kenlm` module required for language model functionality.
  fix Install with KenLM support using `pip install pyctcdecode[kenlm]`. If this fails, review system dependencies for KenLM (Boost, Zlib, Bzip2, CMake) as described in the warnings.

• error: Boost library not found or not configured. Set BOOST_ROOT or BOOST_INCLUDEDIR to point to the Boost install.

  cause During `pyctcdecode[kenlm]` installation, the underlying KenLM C++ compilation process failed to locate the necessary Boost library, which is a critical dependency.
  fix Install Boost development headers on your system (e.g., `sudo apt-get install libboost-all-dev` on Debian/Ubuntu, or `brew install boost` on macOS). Ensure `cmake` is also installed. If Boost is in a non-standard location, you might need to manually set environment variables like `BOOST_ROOT` or `BOOST_INCLUDEDIR`.

• IndexError: list index out of range

  cause This often occurs when the `alphabet` passed to the decoder does not correctly map to the CTC model's output dimension, or more commonly, if the blank token is not positioned as the very first element in the alphabet list.
  fix Verify that your `labels` list used to create the `alphabet` has the `BLANK_TOKEN` (or an empty string) as its first element, and that the total length of your alphabet exactly matches the last dimension of your CTC model's output logits.

Warnings

• gotcha The KenLM C++ library, an optional but highly recommended dependency for language model integration, can be challenging to install due to its native compilation requirements (e.g., Boost, Zlib, Bzip2, CMake).
  Fix: For common Linux distributions, ensure development headers for `boost`, `zlib`, `bzip2`, and `cmake` are installed. For example: `sudo apt-get install libboost-all-dev liblzma-dev libbz2-dev cmake`. Then retry `pip install pyctcdecode[kenlm]` or consult KenLM's specific build instructions if issues persist.
• gotcha The CTC blank token MUST be the first element in the alphabet list provided to `get_alphabet` or `Alphabet` constructor. Incorrect positioning will lead to incorrect decoding results or runtime errors.
  Fix: Always construct your alphabet list by placing `BLANK_TOKEN` (or an empty string) as the first element. Example: `labels = [BLANK_TOKEN] + list('abc')`. Ensure the total length of the alphabet matches your model's CTC output dimension.
• gotcha Loading a KenLM language model requires a valid `.arpa` file. Incorrect file paths, malformed `.arpa` files, or extremely large language models can cause `FileNotFoundError`, `MemoryError`, or excessively slow initialization.
  Fix: Ensure the `.arpa` file exists and is accessible. Verify its format using KenLM's tools if possible. For very large models, consider reducing the beam width or using a smaller language model for initial testing to mitigate performance issues.

Install

• pip install pyctcdecode Base installation
• pip install pyctcdecode[kenlm] Installation with KenLM support

Imports

• BeamSearchDecoderCTC

  from pyctcdecode import BeamSearchDecoderCTC

• LanguageModel

  from pyctcdecode import LanguageModel

• Alphabet

  from pyctcdecode.alphabet import Alphabet

• BLANK_TOKEN

  from pyctcdecode.alphabet import BLANK_TOKEN

• get_alphabet

  from pyctcdecode.alphabet import get_alphabet

Quickstart

This quickstart demonstrates how to initialize `BeamSearchDecoderCTC` with a custom alphabet and decode dummy CTC logits. It shows the basic usage without a language model. For real-world applications, integrating with a KenLM language model is highly recommended for improved accuracy.

from pyctcdecode import BeamSearchDecoderCTC
from pyctcdecode.alphabet import BLANK_TOKEN, get_alphabet
import numpy as np

# Define your model's alphabet. The BLANK_TOKEN must be the first element.
# This example uses a common alphabet for English speech recognition.
labels = [BLANK_TOKEN] + list("abcdefghijklmnopqrstuvwxyz '")
alphabet = get_alphabet(labels)

# Create dummy CTC output (logits) for demonstration.
# In a real scenario, these would come from your deep learning model.
# Shape: (time_steps, alphabet_size)
time_steps = 50
logits = np.random.rand(time_steps, len(labels)).astype(np.float32)

# Initialize the decoder without a language model.
# For better accuracy, integrate with a KenLM language model (see warnings).
decoder = BeamSearchDecoderCTC(alphabet)

# Decode the logits. The decode method returns a list of hypotheses.
# We take the first (most probable) one.
hypotheses = decoder.decode(logits)
decoded_text = hypotheses[0]

print(f"Decoded text (example): {decoded_text}")