tiktoken
JSON →Fast BPE tokenizer from OpenAI, written in Rust. Used to count tokens and encode/decode text for OpenAI models. 3-6x faster than comparable Python tokenizers. Does NOT call any API — purely local computation. Requires a Rust compiler at build time on platforms without pre-built wheels. Package name and import name are both 'tiktoken'.
Traffic · last 30 days ↓55% vs prev 7d
total hits 118
actors 12 distinct systems
last hit 17h ago human
top countries 🇺🇸 United States · 🇸🇬 Singapore · 🇨🇦 Canada · 🇩🇪 Germany · 🇬🇧 United Kingdom
Resources
packagepypi.org/project/tiktoken/ ↗
API endpoints
full doc /v1/registry/tiktoken
install /v1/registry/tiktoken/install
compatibility /v1/registry/tiktoken/compatibility