OpenAI Messages Token Helper

0.1.13 verified Mon Apr 27 auth: no python

A helper library for estimating token counts for OpenAI Chat Completions messages. Currently at version 0.1.13, it supports GPT-3.5, GPT-4, and GPT-4-turbo models. Active development with monthly releases.

pip install openai-messages-token-helper

Common errors

error ModuleNotFoundError: No module named 'openai_messages_token_helper' ↓

cause Package installed as 'openai-messages-token-helper' but import requires underscores.

fix

pip install openai-messages-token-helper # and import as from openai_messages_token_helper import ...

error TypeError: messages_to_tokens() missing 1 required positional argument: 'model' ↓

cause The 'model' argument is required but was omitted.

fix

messages_to_tokens(messages, model="gpt-4")

Warnings

gotcha The token count is an approximation and may differ from actual API usage. Do not rely on exact counts for billing or critical limits. ↓

fix Use for estimation only; always handle token limit errors in your code.

gotcha Model parameter must exactly match the model identifier used in the API (e.g., 'gpt-4', 'gpt-3.5-turbo'). Using an unrecognized model may fall back to a default tokenizer and give inaccurate results. ↓

fix Always specify the correct model string. Check OpenAI documentation for supported model names.

Imports

messages_to_tokens

wrong

from openai_messages_token_helpers import ...

correct

from openai_messages_token_helper import messages_to_tokens

Module name uses underscores, not hyphens.

build_messages

wrong

from openai_messages_token_helper import build_message

correct

from openai_messages_token_helper import build_messages

Function is plural: build_messages.

Quickstart

Quick estimate tokens for a list of messages.

import os
from openai import AzureOpenAI
from openai_messages_token_helper import messages_to_tokens

client = AzureOpenAI(
    api_key=os.environ.get('AZURE_OPENAI_KEY', ''),
    api_version="2024-02-01",
    azure_endpoint=os.environ.get('AZURE_OPENAI_ENDPOINT', '')
)
system_message = "You are a helpful assistant."
user_message = "Hello!"
tokens = messages_to_tokens([
    {"role": "system", "content": system_message},
    {"role": "user", "content": user_message}
], model="gpt-4")
print(f"Token count: {tokens}")