{"id":15131,"library":"js-tiktoken","title":"JS Tiktoken","description":"js-tiktoken is a pure JavaScript port of OpenAI's tiktoken library, providing a BPE (Byte Pair Encoding) tokenizer primarily for use with OpenAI's models. Currently at version 1.0.21, the library undergoes frequent patch releases, mainly to incorporate new OpenAI models and their corresponding tokenizer configurations. Its key differentiators include being a pure JavaScript implementation, making it suitable for web browsers, edge environments, and Node.js applications where Python dependencies are not feasible. It also offers a \"lite\" mode, allowing developers to load only specific encoding ranks to significantly reduce bundle size, or to dynamically fetch encoding data from a CDN, addressing concerns about the full library's potentially large footprint.","status":"active","version":"1.0.21","language":"javascript","source_language":"en","source_url":"https://github.com/dqbd/tiktoken","tags":["javascript","typescript"],"install":[{"cmd":"npm install js-tiktoken","lang":"bash","label":"npm"},{"cmd":"yarn add js-tiktoken","lang":"bash","label":"yarn"},{"cmd":"pnpm add js-tiktoken","lang":"bash","label":"pnpm"}],"dependencies":[],"imports":[{"note":"Primarily an ESM-first package. For CommonJS, named exports like `getEncoding` are properties of the `require` result, but native ESM is recommended.","wrong":"const getEncoding = require('js-tiktoken').getEncoding;","symbol":"getEncoding","correct":"import { getEncoding } from 'js-tiktoken';"},{"note":"A named export for retrieving an encoding based on a model name. Requires correct ESM import syntax for optimal usage.","wrong":"const { encodingForModel } = require('js-tiktoken');","symbol":"encodingForModel","correct":"import { encodingForModel } from 'js-tiktoken';"},{"note":"The `Tiktoken` class for the bundle-size-optimized 'lite' version must be imported from the `js-tiktoken/lite` subpath. Importing from the main package will not expose this specific class for the lite usage pattern.","wrong":"import { Tiktoken } from 'js-tiktoken';","symbol":"Tiktoken","correct":"import { Tiktoken } from 'js-tiktoken/lite';"}],"quickstart":{"code":"import assert from 'node:assert';\nimport { getEncoding, encodingForModel } from 'js-tiktoken';\n\n// Basic usage: Get an encoding directly\nconst enc = getEncoding('gpt2');\nconst encodedTokens = enc.encode('hello world');\nconsole.log(`'gpt2' tokens for 'hello world': ${encodedTokens}`);\nassert(enc.decode(encodedTokens) === 'hello world');\n\n// Model-specific usage: Get encoding for a known model\nconst modelName = 'gpt-4'; // Or 'gpt-3.5-turbo', 'text-embedding-ada-002', etc.\ntry {\n  const modelEnc = encodingForModel(modelName);\n  const text = 'This is an example sentence for GPT-4 tokenization.';\n  const tokens = modelEnc.encode(text);\n  console.log(`\\n'${modelName}' tokens: ${tokens.length}, tokens array: [${tokens.slice(0, 5)}..., ${tokens.slice(-5)}]`);\n  const decoded = modelEnc.decode(tokens);\n  console.log(`'${modelName}' decoded: ${decoded}`);\n} catch (error) {\n  console.error(`\\nError getting encoding for model '${modelName}':`, error.message);\n}\n","lang":"typescript","description":"Demonstrates how to obtain an encoding using both `getEncoding` for a specific scheme and `encodingForModel` for a model name, then encode and decode text, verifying the round trip. Includes error handling for unknown models."},"warnings":[{"fix":"For web applications or environments with strict bundle size limits, use the `js-tiktoken/lite` import path and load encoding ranks manually from `js-tiktoken/ranks/...` or dynamically fetch them from a CDN. Example: `import { Tiktoken } from 'js-tiktoken/lite'; import o200k_base from 'js-tiktoken/ranks/o200k_base'; const enc = new Tiktoken(o200k_base);`","message":"Importing the main `js-tiktoken` package directly (e.g., `import { getEncoding } from 'js-tiktoken';`) will bundle *all* OpenAI tokenizer data, which can significantly increase your application's bundle size, especially for web environments.","severity":"gotcha","affected_versions":">=1.0.0"},{"fix":"Migrate projects to use native ESM imports (`import ... from '...'`) for optimal compatibility, tree-shaking benefits, and future-proofing. If strictly bound to CommonJS, consider dynamic `import()` or ensure your build setup correctly handles ESM-to-CJS conversion.","message":"`js-tiktoken` is built as an ESM-first package. While CommonJS `require` might partially work for some exports through transpilation or Node.js interoperability, it is not the primary pattern, and can lead to unexpected behavior or larger bundle sizes in certain setups.","severity":"gotcha","affected_versions":">=1.0.0"},{"fix":"Always refer to the official OpenAI documentation or `js-tiktoken`'s source for the exact and up-to-date list of supported model names. Implement error handling (e.g., `try...catch`) around calls to `encodingForModel` to gracefully manage unknown model names.","message":"Model names used with `encodingForModel` are frequently updated and can be case-sensitive. Using an unrecognized or incorrectly cased model name will result in an `Error: Unknown encoding`.","severity":"gotcha","affected_versions":">=1.0.0"}],"env_vars":null,"last_verified":"2026-04-21T00:00:00.000Z","next_check":"2026-07-20T00:00:00.000Z","problems":[{"fix":"Ensure your build tool is configured to process and include static JSON files. For example, in Webpack, you might need a `json-loader`. Alternatively, fetch the rank data dynamically from a CDN as outlined in the `js-tiktoken/lite` documentation example.","cause":"You are likely using the `js-tiktoken/lite` import strategy but your bundler (e.g., Webpack, Rollup, Vite) is not configured to correctly handle direct imports of the raw JSON rank data files from `js-tiktoken/ranks/`.","error":"TypeError: Cannot find module 'js-tiktoken/ranks/o200k_base'"},{"fix":"Use native ESM imports: `import { getEncoding } from 'js-tiktoken';`. If you must use CommonJS, ensure you are correctly accessing the named export, potentially via `const { getEncoding } = require('js-tiktoken');` though full ESM compatibility is recommended.","cause":"This error typically occurs when trying to access a named export like `getEncoding` directly from a CommonJS `require` call in a package that is primarily ESM, or when destructuring is applied incorrectly.","error":"TypeError: getEncoding is not a function"},{"fix":"Verify the exact spelling and casing of the model name. Consult the `js-tiktoken` documentation or the underlying `tiktoken` library's list of supported models, as these are frequently updated with new OpenAI releases. Consider adding a `try...catch` block to handle cases where a model name might become deprecated or is not yet supported.","cause":"The model name provided to `getEncoding` or `encodingForModel` does not match any of the currently supported models or encoding schemes within the `js-tiktoken` library.","error":"Error: Unknown encoding 'some-unrecognized-model-name'"}],"ecosystem":"npm"}