tiktoken

Warnings

• breaking get_encoding() takes an encoding name, NOT a model name. tiktoken.get_encoding('gpt-4o') raises KeyError. tiktoken.get_encoding('cl100k_base') is correct. This is the single most common error for new users.
  Fix: Use tiktoken.encoding_for_model('gpt-4o') to look up by model name, or pass the correct encoding name to get_encoding().
• breaking gpt-4o and all o-series models (o1, o3, o4-mini) use o200k_base, NOT cl100k_base. Code that hardcodes cl100k_base for all OpenAI models silently produces wrong token counts for newer models — undercounting or overcounting by up to 10-15% depending on content.
  Fix: Always use tiktoken.encoding_for_model(model_name) to get the correct encoding. Never hardcode cl100k_base as a universal default.
• gotcha On first use, tiktoken downloads encoding vocab files (~1MB each) from OpenAI's CDN. In firewalled/offline environments this silently hangs or raises a connection error, not an ImportError.
  Fix: Pre-warm the cache in a networked environment: tiktoken.get_encoding('o200k_base') and tiktoken.get_encoding('cl100k_base'). Set TIKTOKEN_CACHE_DIR to a writable path. Vocab files are then reused from disk.
• gotcha Token count from enc.encode(text) counts tokens in raw text only. Chat completions add 3 overhead tokens per message and 3 tokens for the assistant reply primer. Omitting this overhead causes off-by-N errors in context window management.
  Fix: Use the OpenAI Cookbook's num_tokens_from_messages() pattern which adds per-message overhead. Don't use raw encode() length for chat token budgeting.
• gotcha Building from source requires Rust. On platforms or Python versions without a pre-built wheel, pip install tiktoken triggers a Rust compile. If Rust is not installed, the install fails with a cryptic error about 'cargo' not found.
  Fix: Install Rust via rustup (https://rustup.rs) before pip install if on an unsupported platform. Or use a Docker image with tiktoken pre-installed.
• gotcha Special tokens like <|endoftext|> are not encoded by default. enc.encode('<|endoftext|>') raises ValueError. Must explicitly allow them.
  Fix: enc.encode('<|endoftext|>', allowed_special={'<|endoftext|>'}) or enc.encode(text, allowed_special='all') to allow all special tokens.

Install

• pip install tiktoken Standard install
• pip install tiktoken --no-binary tiktoken Build from source (requires Rust toolchain)

Imports

• get_encoding

  import tiktoken; enc = tiktoken.get_encoding('o200k_base')

  get_encoding() takes an encoding name (e.g. 'o200k_base'), NOT a model name. Passing a model name raises KeyError.
• encoding_for_model

  import tiktoken; enc = tiktoken.encoding_for_model('gpt-4o')

  encoding_for_model() takes a model name and returns the correct encoding. Preferred over hardcoding encoding names.

Quickstart

encoding_for_model() is safer than hardcoding encoding names — it handles model→encoding mapping automatically and stays correct as OpenAI adds new models. Token counts for chat completions must add per-message overhead (3 tokens per message) to get accurate billing estimates.

import tiktoken

# Get encoding by model name (recommended)
enc = tiktoken.encoding_for_model('gpt-4o')  # returns o200k_base

# Or get encoding directly by name
enc = tiktoken.get_encoding('o200k_base')

# Encode text → list of token integers
tokens = enc.encode('Hello, world!')
print(tokens)        # [9906, 11, 1917, 0]
print(len(tokens))   # 4

# Decode tokens → string
text = enc.decode(tokens)
print(text)  # 'Hello, world!'

# Count tokens for a chat message (accounts for message overhead)
def count_chat_tokens(messages: list[dict], model: str = 'gpt-4o') -> int:
    enc = tiktoken.encoding_for_model(model)
    tokens_per_message = 3  # every message has <|im_start|>, role, <|im_end|>
    tokens_per_name = 1
    total = 0
    for msg in messages:
        total += tokens_per_message
        for key, value in msg.items():
            total += len(enc.encode(value))
            if key == 'name':
                total += tokens_per_name
    total += 3  # reply is primed with <|im_start|>assistant
    return total

messages = [{'role': 'user', 'content': 'How many tokens is this?'}]
print(count_chat_tokens(messages))  # ~12