TOON (Token-Oriented Object Notation) Python Library

1.6.0 · active · verified Thu Apr 16

Toonify is a Python implementation of TOON (Token-Oriented Object Notation), a compact, human-readable serialization format designed specifically for use with Large Language Models (LLMs). It aims to reduce token consumption by 30-60% compared to JSON, while preserving data structure, type safety, and human readability. The library provides functions to encode Python dictionaries and Pydantic models into TOON strings and decode them back. It is actively maintained with a regular release cadence.

Common errors

```
ModuleNotFoundError: No module named 'toon'
```
cause The `toonify` package is installed, but the Python interpreter cannot find the `toon` module during import. This often happens if the package wasn't installed in the active environment or there's a typo.

fix Ensure the library is installed in your active environment: `pip install toonify`. If using Pydantic, use `pip install toonify[pydantic]`.
```
AttributeError: module 'toonify' has no attribute 'encode'
```
cause Attempting to call `toonify.encode()` or `toonify.decode()`. The main serialization functions are located within the `toon` submodule, not directly under the top-level `toonify` package.

fix Correct the import statement to `from toon import encode, decode` (or `encode_pydantic`, `decode_to_pydantic`) and use the functions directly, e.g., `encode(data)`.
```
ValueError: Invalid TOON string format: expected a valid key or header at line X, column Y
```
cause The input TOON string provided to `decode()` is malformed, violating the TOON specification. Common causes include incorrect indentation, missing delimiters, or improper headers for tabular data.

fix Carefully review the TOON string for syntax errors, paying close attention to indentation, delimiters (comma, tab, pipe), and the correct format for object/array headers (e.g., `products[2]{sku,name,price}:`). Use a TOON linter if available or refer to the TOON specification.

Warnings

gotcha Prior to v1.5.0, `toonify` had issues preserving nested arrays and objects when converting to tabular format, potentially leading to data loss or incorrect structure upon decoding.
Fix: Upgrade to `toonify` version 1.5.0 or newer. Ensure your data structures are well-formed and test round-trip conversions.
gotcha TOON is highly optimized for uniform, tabular-like data structures. While it supports arbitrary JSON data models, its token efficiency benefits are maximized with structured, repetitive data, such as lists of objects with identical fields. Deeply nested or highly irregular data may see fewer benefits.
Fix: Evaluate your data structure; TOON provides significant savings for tabular data. For highly irregular data, traditional JSON might still be a suitable choice.
gotcha The `toonify` library uses `from toon import ...` for its core functions, not `from toonify import ...`. Importing directly from `toonify` will lead to `AttributeError` or `ModuleNotFoundError` for the main `encode` and `decode` functions.
Fix: Always use `from toon import encode, decode` (or `encode_pydantic`, `decode_to_pydantic`) for the core serialization functions.
gotcha There are other Python libraries and APIs named 'Toonify' that relate to image cartoonification. Ensure you are using the correct library for Token-Oriented Object Notation (TOON) serialization, identifiable by its PyPI slug `toonify` and its focus on LLM data.
Fix: Verify the project description and import statements (`from toon import ...`) to confirm you are using the correct serialization library.

Install

pip install toonify Base Installation
pip install toonify[pydantic] With Pydantic Support

Imports

encode, decode
```
from toon import encode, decode
```
Main functions for converting Python objects to TOON strings and vice-versa.
encode_pydantic, decode_to_pydantic
```
from toon import encode_pydantic, decode_to_pydantic
```
Functions specifically for converting Pydantic models to/from TOON strings. Requires 'toonify[pydantic]' installation.

Quickstart

This quickstart demonstrates the basic usage of the `encode` and `decode` functions to convert a Python dictionary containing nested data into a TOON string and then back into a Python dictionary. It showcases how `toonify` automatically uses a compact tabular format for uniform arrays.

from toon import encode, decode

data = {
    "products": [
        {"sku": "LAP-001", "name": "Gaming Laptop", "price": 1299.99},
        {"sku": "MOU-042", "name": "Wireless Mouse", "price": 29.99},
    ]
}

# Encode Python dict to TOON
toon_string = encode(data)
print("Encoded TOON string:\n" + toon_string)
# Expected output for data above:
# products[2]{sku,name,price}:
# LAP-001,Gaming Laptop,1299.99
# MOU-042,Wireless Mouse,29.99

# Decode TOON back to Python
result = decode(toon_string)
assert result == data
print("Decoded Python object:" + str(result))

view raw JSON →