tiktoken-rs for Node.js

Common errors

• Error: Cannot find module 'tiktoken-rs-node' or its corresponding type declarations.

  cause The package either failed to install correctly (e.g., prebuilt binaries missing, build from source failed) or the import path is incorrect.
  fix Run `npm install tiktoken-rs-node` again. Check the installation logs for errors related to NAPI or Rust compilation. If running in a specific environment, verify Rust toolchain presence or ensure prebuilt binaries exist for your platform. Verify the import statement `import { getEncoding } from 'tiktoken-rs-node';`.

• TypeError: encoding.encode is not a function

  cause The `encoding` object returned by `getEncoding` is not the expected `Tiktoken` instance, or `getEncoding` itself failed to return a valid object. This might happen if `getEncoding` was called with an invalid name.
  fix Ensure `getEncoding` was called with a valid string encoding name (e.g., 'cl100k_base'). Add error handling around `getEncoding` calls. Verify `encoding` is not `undefined` or `null` before calling methods.

Warnings

• gotcha Calling `getEncoding` multiple times with the same encoding name can lead to unnecessary overhead. For long-running processes, it is highly recommended to call `getEncoding` only once per-startup and reuse the returned `Tiktoken` instance.
  Fix: Store the result of `getEncoding` in a module-level or application-level variable and reuse it across your application.
• breaking As a Node.js NAPI binding, `tiktoken-rs-node` includes prebuilt binaries for common platforms. If your deployment environment (e.g., specific Docker image, serverless function, or CPU architecture) is not supported by the prebuilt binaries, you may encounter installation failures requiring local compilation, which needs a Rust toolchain.
  Fix: Check the project's GitHub releases for supported platforms. If prebuilt binaries are unavailable, ensure a Rust toolchain is installed in your build environment. Alternatively, consider using `npm install --build-from-source` if available.
• gotcha Using an unsupported encoding name with `getEncoding` will result in a runtime error. Ensure you use one of the officially supported encoding names (e.g., 'cl100k_base', 'gpt2', 'o200k_base').
  Fix: Refer to the `tiktoken-rs-node` documentation or `tiktoken-rs` project for a list of valid encoding names.

Install

• npm install tiktoken-rs-node npm
• yarn add tiktoken-rs-node yarn
• pnpm add tiktoken-rs-node pnpm

Imports

• getEncoding
  wrong:

  const getEncoding = require('tiktoken-rs-node');

  correct:

  import { getEncoding } from 'tiktoken-rs-node';

  The library primarily uses named exports. While CommonJS `require` works, ESM `import` is the recommended pattern, especially when using TypeScript.
• Tiktoken
  wrong:

  import { Tiktoken } from 'tiktoken-rs-node';

  correct:

  import type { Tiktoken } from 'tiktoken-rs-node';

  The `Tiktoken` class itself is not directly exported for instantiation. `getEncoding()` returns an instance of `Tiktoken`. Import its type for strong TypeScript typing.
• CommonJS

  const { getEncoding } = require('tiktoken-rs-node');

  For CommonJS environments, destructuring `require` is the correct way to access `getEncoding`.

Quickstart

This quickstart demonstrates how to load an `tiktoken` encoding, tokenize a string, and then decode the tokens back into a string using the `tiktoken-rs-node` library.

import { getEncoding } from 'tiktoken-rs-node';

// Load the encoding once per application startup for optimal performance.
// Supported encodings: 'o200k_base', 'cl100k_base', 'p50k_edit', 'p50k_base', 'r50k_base', 'gpt2'
const encodingName = 'cl100k_base';
const encoding = getEncoding(encodingName);

const textToEncode = "This is a sample sentence for tokenization using tiktoken-rs-node.";

// Encode the text into an array of token integers
const tokens = encoding.encode(textToEncode);
console.log(`Original text: "${textToEncode}"`);
console.log(`Encoded tokens (${encodingName}): [${tokens.join(', ')}]`);
console.log(`Number of tokens: ${tokens.length}`);

// Decode the tokens back into a string
const decodedString = encoding.decode(tokens);
console.log(`Decoded string: "${decodedString}"`);

// Example with a different encoding
const gpt2Encoding = getEncoding('gpt2');
const gpt2Tokens = gpt2Encoding.encode("Hello, world!");
console.log(`GPT-2 Encoded tokens: [${gpt2Tokens.join(', ')}]`);