PyTorch Tokenizers

Common errors

• ModuleNotFoundError: No module named 'pytorch_tokenizers'

  cause The `pytorch-tokenizers` package is not installed in your current Python environment.
  fix Run `pip install pytorch-tokenizers` to install the package.

• FileNotFoundError: No such file or directory: 'your_model.model'

  cause The `SentencePieceTokenizer.from_file()` method was called with a path to a SentencePiece model file that does not exist or is inaccessible.
  fix Verify that the path to your `.model` file is correct and that the file exists at that location. Ensure proper file permissions if the file is present but still inaccessible.

• RuntimeError: The sentencepiece model file is not found or corrupted. (pytorch_tokenizers.cpp)

  cause The SentencePiece model file (`.model`) provided to `SentencePieceTokenizer.from_file()` is either empty, corrupted, or not a valid SentencePiece model.
  fix Check the integrity of your SentencePiece model file. Try regenerating it if you have the original training data or re-downloading it from its source. Ensure it's a valid `.model` file generated by the `sentencepiece` library.

Warnings

• breaking PyTorch-Tokenizers maintains tight version alignment with PyTorch and ExecuTorch. Major version changes or significant updates in these upstream libraries may introduce incompatibilities or require an update to `pytorch-tokenizers` for continued functionality.
  Fix: Always check the release notes of `pytorch-tokenizers` and its associated PyTorch/ExecuTorch versions. Upgrade `pytorch-tokenizers` to match your PyTorch/ExecuTorch installation.
• gotcha The `pytorch-tokenizers` library is primarily an internal dependency for PyTorch's on-device AI efforts (like ExecuTorch). As such, it lacks extensive standalone documentation and examples compared to general-purpose tokenization libraries (e.g., Hugging Face's `tokenizers` or `torchtext`). Users expecting a feature-rich, standalone API similar to these other libraries might find the direct usage less intuitive.
  Fix: Consult the `pytorch-tokenizers` GitHub repository's `README.md` and related ExecuTorch documentation for usage patterns. Do not assume feature parity or API identicality with other tokenizer libraries.
• gotcha Tokenizers provided by `pytorch-tokenizers`, such as `SentencePieceTokenizer`, require pre-trained model files (e.g., `.model` for SentencePiece) to be instantiated. These model files are not distributed with the `pytorch-tokenizers` Python package itself.
  Fix: Ensure you have access to the necessary model files. These are typically generated by training SentencePiece on a corpus or obtained alongside pre-trained language models from sources like Hugging Face. The quickstart demonstrates generating a dummy file for local testing.

Install

• pip install pytorch-tokenizers Install stable version

Imports

• SentencePieceTokenizer
  wrong:

  from torchtext.transforms import SentencePieceTokenizer

  correct:

  from pytorch_tokenizers import SentencePieceTokenizer

  This import is for `torchtext`, a separate library with its own SentencePiece tokenizer implementation. `pytorch-tokenizers` provides its own distinct implementation.
• SentencePieceTokenizer
  wrong:

  from transformers import SentencePieceTokenizer

  correct:

  from pytorch_tokenizers import SentencePieceTokenizer

  This import is for Hugging Face's `transformers` library, which is a different tokenizer ecosystem. `pytorch-tokenizers` is a distinct PyTorch-specific implementation.

Quickstart

This quickstart demonstrates how to initialize and use the `SentencePieceTokenizer` from `pytorch-tokenizers`. Note that `SentencePieceTokenizer` requires a pre-trained SentencePiece model file (`.model`). For a runnable example, we temporarily generate a dummy model using the `sentencepiece` library. In practical applications, you would typically load an existing model file.

import os
import tempfile
import sentencepiece as spm # Required for generating dummy model
from pytorch_tokenizers import SentencePieceTokenizer

# 1. Create a dummy SentencePiece model file for demonstration
#    In real-world scenarios, you would use an existing pre-trained model.
model_prefix = os.path.join(tempfile.gettempdir(), 'm_test')
model_file = f'{model_prefix}.model'
vocab_file = f'{model_prefix}.vocab'

# Ensure clean slate for temporary files
if os.path.exists(model_file): os.remove(model_file)
if os.path.exists(vocab_file): os.remove(vocab_file)

text_data = "Hello world. This is a test sentence. SentencePiece is great!"
with open(f'{model_prefix}.txt', 'w') as f:
    f.write(text_data)

spm.SentencePieceTrainer.train(
    input=f'{model_prefix}.txt',
    model_prefix=model_prefix,
    vocab_size=10,
    model_type='bpe'
)

# 2. Instantiate the SentencePieceTokenizer from the created model file
tokenizer = SentencePieceTokenizer.from_file(model_file)

# 3. Encode text
input_text = "This is a sample text for tokenization."
encoded_tokens = tokenizer.encode(input_text)
print(f"Original text: {input_text}")
print(f"Encoded token IDs: {encoded_tokens}")

# 4. Decode tokens
decoded_text = tokenizer.decode(encoded_tokens)
print(f"Decoded text: {decoded_text}")

# Clean up temporary files
os.remove(f'{model_prefix}.txt')
os.remove(model_file)
os.remove(vocab_file)