A Fast, spec compliant Python 3.14+ tokenizer

library 0.4.1 ·python

✓ verified Jun 9, 2026 ✓ install

Pytokens is an open-source Python library providing a fast, spec-compliant tokenizer for Python 3.14+ that is also capable of running on older Python versions (>=3.8). Currently at version 0.4.1, it appears to be actively maintained with recent releases focusing on packaging and development improvements.

Traffic · last 30 days ↓33% vs prev 7d · indexed Sat Mar 28 · updated Sat Jun 27

total hits 13

actors 4 distinct systems

last hit 1h ago ClaudeBot

Amazonbot

4

Script

2

ClaudeBot

1

Humans

2

top countries 🇺🇸 United States · 🇨🇦 Canada · 🇩🇪 Germany · 🇸🇬 Singapore

Resources

githubgithub.com/tusharsadhwani/pytokens ↗

packagepypi.org/project/pytokens/ ↗

API endpoints

full doc /v1/registry/pytokens

install /v1/registry/pytokens/install

compatibility /v1/registry/pytokens/compatibility