Tokenizers: Fast State-of-the-Art Tokenizers

0.22.2 · active · verified Sat Mar 28

Tokenizers is a Python library providing fast and versatile tokenization tools, optimized for both research and production environments. The current version is 0.22.2, released on January 5, 2026. The library is actively maintained with regular updates to enhance performance and add features.

Warnings

breaking Python 3.13 compatibility issues during installation
Fix: Use Python 3.12 or earlier for installation; Python 3.13 is not supported due to PyO3 compatibility issues.
gotcha Ensure correct import path to avoid ImportError
Fix: Use 'from tokenizers import Tokenizer' to import the Tokenizer class.

Install

pip install tokenizers Install Tokenizers

Imports

Tokenizer
```
from tokenizers import Tokenizer
```
Ensure correct import path to avoid ImportError

Quickstart

A simple example demonstrating how to load a pretrained tokenizer and tokenize a sample text.

from tokenizers import Tokenizer

# Load a pretrained tokenizer
tokenizer = Tokenizer.from_pretrained('bert-base-uncased')

# Tokenize a text
output = tokenizer.encode('Hello, world!')
print(output.tokens)

view raw JSON →