A Fast, spec compliant Python 3.14+ tokenizer

0.4.1 · active · verified Sat Mar 28

Pytokens is an open-source Python library providing a fast, spec-compliant tokenizer for Python 3.14+ that is also capable of running on older Python versions (>=3.8). Currently at version 0.4.1, it appears to be actively maintained with recent releases focusing on packaging and development improvements.

Warnings

gotcha Pytokens is compiled with mypyc by default for performance. This means the installed module might be a compiled extension (.so or .pyd) rather than pure Python code, which can affect debugging or introspection.
Fix: To disable mypyc compilation (e.g., for local development or debugging), set the environment variable PYTOKENS_USE_MYPYC=0 before installation (e.g., PYTOKENS_USE_MYPYC=0 pip install pytokens).
gotcha While pytokens aims for Python 3.14+ spec compliance and runs on older Python versions (>=3.8), be aware that its tokenization behavior adheres to the 3.14+ specification. This might introduce subtle differences compared to the native tokenizer behavior of older Python interpreters.
Fix: Ensure your environment or tests account for the 3.14+ tokenizer specification if running on older Python versions, especially if precise tokenizer behavior is critical for your application.

Install

pip install pytokens Install with pip

Imports

pytokens
```
import pytokens
```

Quickstart

Pytokens is primarily designed for command-line usage to tokenize Python files. This example demonstrates tokenizing a simple Python file. The output will be a stream of tokens (e.g., NAME, STRING, OP, etc.) representing the input file's content.

echo "print('Hello, World!')" > example.py
python -m pytokens example.py

view raw JSON →