tiktoken-rs for Node.js
JSON →tiktoken-rs-node provides performant NAPI Rust bindings for the `tiktoken-rs` library, which implements OpenAI's `tiktoken` BPE tokenizer for encoding and decoding text to and from token integers. As of version 1.0.6, it offers a zero-copy encode mechanism and avoids WebAssembly (WASM), differentiating it from other Node.js `tiktoken` implementations that often rely on WASM. The package is actively maintained with frequent patch releases, indicating ongoing support for its current stable API. Its primary use case is for applications requiring efficient tokenization of text for large language models, particularly in Node.js environments where high performance and minimal overhead are critical. It supports a range of encodings including `cl100k_base`, `gpt2`, and `o200k_base`.
Traffic · last 30 days ↑0% vs prev 7d
top countries 🇺🇸 United States · 🇨🇦 Canada · 🇫🇷 France · 🇩🇪 Germany