RustBPE Tokenizer

Common errors

• ModuleNotFoundError: No module named 'rustbpe'

  cause The `rustbpe` package is not installed in the current Python environment.
  fix Run `pip install rustbpe` to install the package.

• TypeError: train_from_iterator() missing 1 required positional argument: 'vocab_size'

  cause The `train_from_iterator` method requires `vocab_size` to be explicitly provided, which determines the final size of the tokenizer's vocabulary.
  fix Ensure `vocab_size` is passed as a keyword argument, e.g., `tokenizer.train_from_iterator(data_iterator, vocab_size=32768)`.

• error: can't find Rust compiler

  cause This error occurs when attempting to install `rustbpe` from source (e.g., if a pre-compiled wheel isn't available for your system or Python version) without a Rust toolchain installed.
  fix Install Rust by following instructions on `rustup.rs` (e.g., `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`) and then retry `pip install rustbpe`. Ensure `maturin` is also installed if building directly from the git repository.

Warnings

• gotcha RustBPE is optimized for training BPE tokenizers and exporting them to the tiktoken format for inference. While it offers encoding/decoding, its primary value proposition lies in the training aspect, distinguishing it from libraries solely focused on inference.
  Fix: Understand that rustbpe provides the 'missing tiktoken training code' and plan your workflow accordingly, potentially using tiktoken for production inference.
• gotcha The library is in its initial `0.1.0` release. While tested, the API may be subject to changes and refinements in subsequent minor versions as the project matures.
  Fix: Pin your dependency to a specific minor version (`rustbpe==0.1.*`) in production environments and review release notes for breaking changes when updating.
• gotcha RustBPE defaults to GPT-4 style regex pre-tokenization. If you need to match tokenization behavior of older GPT models (e.g., GPT-2/3) or other tokenizer types, this default pattern might yield different token splits.
  Fix: Review the tokenizer's pre-tokenization regex if exact compatibility with non-GPT-4 style models is critical. The library's source code indicates the specific pattern used.
• gotcha The author notes that while the Python reference code is expert-level and equality tests pass, the underlying Rust implementation had significant AI assistance and might not be optimally structured by a Rust expert.
  Fix: While functional, be aware that future performance or maintainability improvements might arise from Rust-specific code optimizations. Report any observed inefficiencies or bugs on the GitHub issues page.

Install

• pip install rustbpe Install stable version

Imports

• Tokenizer

  import rustbpe
  tokenizer = rustbpe.Tokenizer()

  The primary class for BPE operations is directly available under the 'rustbpe' module.

Quickstart

This quickstart demonstrates how to initialize the `rustbpe.Tokenizer`, train it on a small dataset, and then use it to encode and decode text, including batch operations. It also shows the (optional) export capability to the tiktoken format.

import rustbpe
import os

# Create a tokenizer instance
tokenizer = rustbpe.Tokenizer()

# Prepare some sample training data
training_texts = [
    "hello world",
    "this is a test sentence",
    "rustbpe is fast and efficient"
]

# Train the tokenizer
# vocab_size is a crucial parameter defining the output vocabulary size
tokenizer.train_from_iterator(training_texts, vocab_size=256) # Small vocab for example

# Encode text
text_to_encode = "hello rustbpe, how are you today?"
ids = tokenizer.encode(text_to_encode)
print(f"Encoded IDs: {ids}")

# Decode IDs back to text
decoded_text = tokenizer.decode(ids)
print(f"Decoded Text: {decoded_text}")

# Batch encode multiple texts (uses parallelization)
batch_texts = ["text one", "text two", "text three"]
all_ids = tokenizer.batch_encode(batch_texts)
print(f"Batch Encoded IDs: {all_ids}")

# Optional: Export to tiktoken format (requires tiktoken to be installed)
# if os.environ.get('ENABLE_TIKTOKEN_EXPORT', 'false').lower() == 'true':
#     import tiktoken
#     tiktoken_tokenizer = tokenizer.export_to_tiktoken()
#     print("Tokenizer exported to tiktoken format.")

print(f"Vocabulary size: {tokenizer.vocab_size}")