{"id":12143,"library":"tiktoken-rs-node","title":"tiktoken-rs for Node.js","description":"tiktoken-rs-node provides performant NAPI Rust bindings for the `tiktoken-rs` library, which implements OpenAI's `tiktoken` BPE tokenizer for encoding and decoding text to and from token integers. As of version 1.0.6, it offers a zero-copy encode mechanism and avoids WebAssembly (WASM), differentiating it from other Node.js `tiktoken` implementations that often rely on WASM. The package is actively maintained with frequent patch releases, indicating ongoing support for its current stable API. Its primary use case is for applications requiring efficient tokenization of text for large language models, particularly in Node.js environments where high performance and minimal overhead are critical. It supports a range of encodings including `cl100k_base`, `gpt2`, and `o200k_base`.","status":"active","version":"1.0.6","language":"javascript","source_language":"en","source_url":"https://github.com/tmb/tiktoken-rs-node","tags":["javascript","typescript"],"install":[{"cmd":"npm install tiktoken-rs-node","lang":"bash","label":"npm"},{"cmd":"yarn add tiktoken-rs-node","lang":"bash","label":"yarn"},{"cmd":"pnpm add tiktoken-rs-node","lang":"bash","label":"pnpm"}],"dependencies":[],"imports":[{"note":"The library primarily uses named exports. While CommonJS `require` works, ESM `import` is the recommended pattern, especially when using TypeScript.","wrong":"const getEncoding = require('tiktoken-rs-node');","symbol":"getEncoding","correct":"import { getEncoding } from 'tiktoken-rs-node';"},{"note":"The `Tiktoken` class itself is not directly exported for instantiation. `getEncoding()` returns an instance of `Tiktoken`. Import its type for strong TypeScript typing.","wrong":"import { Tiktoken } from 'tiktoken-rs-node';","symbol":"Tiktoken","correct":"import type { Tiktoken } from 'tiktoken-rs-node';"},{"note":"For CommonJS environments, destructuring `require` is the correct way to access `getEncoding`.","symbol":"CommonJS","correct":"const { getEncoding } = require('tiktoken-rs-node');"}],"quickstart":{"code":"import { getEncoding } from 'tiktoken-rs-node';\n\n// Load the encoding once per application startup for optimal performance.\n// Supported encodings: 'o200k_base', 'cl100k_base', 'p50k_edit', 'p50k_base', 'r50k_base', 'gpt2'\nconst encodingName = 'cl100k_base';\nconst encoding = getEncoding(encodingName);\n\nconst textToEncode = \"This is a sample sentence for tokenization using tiktoken-rs-node.\";\n\n// Encode the text into an array of token integers\nconst tokens = encoding.encode(textToEncode);\nconsole.log(`Original text: \"${textToEncode}\"`);\nconsole.log(`Encoded tokens (${encodingName}): [${tokens.join(', ')}]`);\nconsole.log(`Number of tokens: ${tokens.length}`);\n\n// Decode the tokens back into a string\nconst decodedString = encoding.decode(tokens);\nconsole.log(`Decoded string: \"${decodedString}\"`);\n\n// Example with a different encoding\nconst gpt2Encoding = getEncoding('gpt2');\nconst gpt2Tokens = gpt2Encoding.encode(\"Hello, world!\");\nconsole.log(`GPT-2 Encoded tokens: [${gpt2Tokens.join(', ')}]`);","lang":"typescript","description":"This quickstart demonstrates how to load an `tiktoken` encoding, tokenize a string, and then decode the tokens back into a string using the `tiktoken-rs-node` library."},"warnings":[{"fix":"Store the result of `getEncoding` in a module-level or application-level variable and reuse it across your application.","message":"Calling `getEncoding` multiple times with the same encoding name can lead to unnecessary overhead. For long-running processes, it is highly recommended to call `getEncoding` only once per-startup and reuse the returned `Tiktoken` instance.","severity":"gotcha","affected_versions":">=1.0.0"},{"fix":"Check the project's GitHub releases for supported platforms. If prebuilt binaries are unavailable, ensure a Rust toolchain is installed in your build environment. Alternatively, consider using `npm install --build-from-source` if available.","message":"As a Node.js NAPI binding, `tiktoken-rs-node` includes prebuilt binaries for common platforms. If your deployment environment (e.g., specific Docker image, serverless function, or CPU architecture) is not supported by the prebuilt binaries, you may encounter installation failures requiring local compilation, which needs a Rust toolchain.","severity":"breaking","affected_versions":">=1.0.0"},{"fix":"Refer to the `tiktoken-rs-node` documentation or `tiktoken-rs` project for a list of valid encoding names.","message":"Using an unsupported encoding name with `getEncoding` will result in a runtime error. Ensure you use one of the officially supported encoding names (e.g., 'cl100k_base', 'gpt2', 'o200k_base').","severity":"gotcha","affected_versions":">=1.0.0"}],"env_vars":null,"last_verified":"2026-04-19T00:00:00.000Z","next_check":"2026-07-18T00:00:00.000Z","problems":[{"fix":"Run `npm install tiktoken-rs-node` again. Check the installation logs for errors related to NAPI or Rust compilation. If running in a specific environment, verify Rust toolchain presence or ensure prebuilt binaries exist for your platform. Verify the import statement `import { getEncoding } from 'tiktoken-rs-node';`.","cause":"The package either failed to install correctly (e.g., prebuilt binaries missing, build from source failed) or the import path is incorrect.","error":"Error: Cannot find module 'tiktoken-rs-node' or its corresponding type declarations."},{"fix":"Ensure `getEncoding` was called with a valid string encoding name (e.g., 'cl100k_base'). Add error handling around `getEncoding` calls. Verify `encoding` is not `undefined` or `null` before calling methods.","cause":"The `encoding` object returned by `getEncoding` is not the expected `Tiktoken` instance, or `getEncoding` itself failed to return a valid object. This might happen if `getEncoding` was called with an invalid name.","error":"TypeError: encoding.encode is not a function"}],"ecosystem":"npm"}