JS Tiktoken

Common errors

• TypeError: Cannot find module 'js-tiktoken/ranks/o200k_base'

  cause You are likely using the `js-tiktoken/lite` import strategy but your bundler (e.g., Webpack, Rollup, Vite) is not configured to correctly handle direct imports of the raw JSON rank data files from `js-tiktoken/ranks/`.
  fix Ensure your build tool is configured to process and include static JSON files. For example, in Webpack, you might need a `json-loader`. Alternatively, fetch the rank data dynamically from a CDN as outlined in the `js-tiktoken/lite` documentation example.

• TypeError: getEncoding is not a function

  cause This error typically occurs when trying to access a named export like `getEncoding` directly from a CommonJS `require` call in a package that is primarily ESM, or when destructuring is applied incorrectly.
  fix Use native ESM imports: `import { getEncoding } from 'js-tiktoken';`. If you must use CommonJS, ensure you are correctly accessing the named export, potentially via `const { getEncoding } = require('js-tiktoken');` though full ESM compatibility is recommended.

• Error: Unknown encoding 'some-unrecognized-model-name'

  cause The model name provided to `getEncoding` or `encodingForModel` does not match any of the currently supported models or encoding schemes within the `js-tiktoken` library.
  fix Verify the exact spelling and casing of the model name. Consult the `js-tiktoken` documentation or the underlying `tiktoken` library's list of supported models, as these are frequently updated with new OpenAI releases. Consider adding a `try...catch` block to handle cases where a model name might become deprecated or is not yet supported.

Warnings

• gotcha Importing the main `js-tiktoken` package directly (e.g., `import { getEncoding } from 'js-tiktoken';`) will bundle *all* OpenAI tokenizer data, which can significantly increase your application's bundle size, especially for web environments.
  Fix: For web applications or environments with strict bundle size limits, use the `js-tiktoken/lite` import path and load encoding ranks manually from `js-tiktoken/ranks/...` or dynamically fetch them from a CDN. Example: `import { Tiktoken } from 'js-tiktoken/lite'; import o200k_base from 'js-tiktoken/ranks/o200k_base'; const enc = new Tiktoken(o200k_base);`
• gotcha `js-tiktoken` is built as an ESM-first package. While CommonJS `require` might partially work for some exports through transpilation or Node.js interoperability, it is not the primary pattern, and can lead to unexpected behavior or larger bundle sizes in certain setups.
  Fix: Migrate projects to use native ESM imports (`import ... from '...'`) for optimal compatibility, tree-shaking benefits, and future-proofing. If strictly bound to CommonJS, consider dynamic `import()` or ensure your build setup correctly handles ESM-to-CJS conversion.
• gotcha Model names used with `encodingForModel` are frequently updated and can be case-sensitive. Using an unrecognized or incorrectly cased model name will result in an `Error: Unknown encoding`.
  Fix: Always refer to the official OpenAI documentation or `js-tiktoken`'s source for the exact and up-to-date list of supported model names. Implement error handling (e.g., `try...catch`) around calls to `encodingForModel` to gracefully manage unknown model names.

Install

• npm install js-tiktoken npm
• yarn add js-tiktoken yarn
• pnpm add js-tiktoken pnpm

Imports

• getEncoding
  wrong:

  const getEncoding = require('js-tiktoken').getEncoding;

  correct:

  import { getEncoding } from 'js-tiktoken';

  Primarily an ESM-first package. For CommonJS, named exports like `getEncoding` are properties of the `require` result, but native ESM is recommended.
• encodingForModel
  wrong:

  const { encodingForModel } = require('js-tiktoken');

  correct:

  import { encodingForModel } from 'js-tiktoken';

  A named export for retrieving an encoding based on a model name. Requires correct ESM import syntax for optimal usage.
• Tiktoken
  wrong:

  import { Tiktoken } from 'js-tiktoken';

  correct:

  import { Tiktoken } from 'js-tiktoken/lite';

  The `Tiktoken` class for the bundle-size-optimized 'lite' version must be imported from the `js-tiktoken/lite` subpath. Importing from the main package will not expose this specific class for the lite usage pattern.

Quickstart

Demonstrates how to obtain an encoding using both `getEncoding` for a specific scheme and `encodingForModel` for a model name, then encode and decode text, verifying the round trip. Includes error handling for unknown models.

import assert from 'node:assert';
import { getEncoding, encodingForModel } from 'js-tiktoken';

// Basic usage: Get an encoding directly
const enc = getEncoding('gpt2');
const encodedTokens = enc.encode('hello world');
console.log(`'gpt2' tokens for 'hello world': ${encodedTokens}`);
assert(enc.decode(encodedTokens) === 'hello world');

// Model-specific usage: Get encoding for a known model
const modelName = 'gpt-4'; // Or 'gpt-3.5-turbo', 'text-embedding-ada-002', etc.
try {
  const modelEnc = encodingForModel(modelName);
  const text = 'This is an example sentence for GPT-4 tokenization.';
  const tokens = modelEnc.encode(text);
  console.log(`\n'${modelName}' tokens: ${tokens.length}, tokens array: [${tokens.slice(0, 5)}..., ${tokens.slice(-5)}]`);
  const decoded = modelEnc.decode(tokens);
  console.log(`'${modelName}' decoded: ${decoded}`);
} catch (error) {
  console.error(`\nError getting encoding for model '${modelName}':`, error.message);
}