Tokie
JSON →A blazingly fast tokenizer library that is 50x faster and 10x smaller than alternatives, providing 100% accurate encoding/decoding for large language models. Current version 0.0.10 targets Python >=3.10, with weekly release cadence.
A blazingly fast tokenizer library that is 50x faster and 10x smaller than alternatives, providing 100% accurate encoding/decoding for large language models. Current version 0.0.10 targets Python >=3.10, with weekly release cadence.