tiktoken
JSON →Fast BPE tokenizer from OpenAI, written in Rust. Used to count tokens and encode/decode text for OpenAI models. 3-6x faster than comparable Python tokenizers. Does NOT call any API — purely local computation. Requires a Rust compiler at build time on platforms without pre-built wheels. Package name and import name are both 'tiktoken'.
Traffic · last 30 days ↑188% vs prev 7d
total hits 102
actors 12 distinct systems
last hit 9h ago human
top countries 🇺🇸 United States · 🇨🇦 Canada · 🇸🇬 Singapore · 🇬🇧 United Kingdom · 🇨🇳 China
Resources
packagepypi.org/project/tiktoken/ ↗
API endpoints
full doc /v1/registry/tiktoken
install /v1/registry/tiktoken/install
compatibility /v1/registry/tiktoken/compatibility