Chonkie

Warnings

• breaking Chonkie v1.5.0 dropped support for Python 3.9. Users on Python 3.9 must upgrade their Python version to 3.10 or newer to use Chonkie v1.5.0 or later.
  Fix: Upgrade your Python environment to 3.10 or higher (e.g., `python -m pip install --upgrade python` or use a new virtual environment).
• gotcha Many advanced chunkers (e.g., OpenAI, TeraflopAI, LangChain-based) and visualization tools have optional dependencies that are not installed with `pip install chonkie`. Attempting to use these features without the correct dependencies will result in `ModuleNotFoundError`.
  Fix: Install Chonkie with the relevant extras, for example `pip install 'chonkie[llm]'` for LLM-related features, or `pip install 'chonkie[all]'` for all extras.
• gotcha Before v1.5.5, importing the `chonkie` library could fail with a `ModuleNotFoundError` if `openai` was not installed, even if you did not intend to use OpenAI-specific features. This was due to non-lazy imports.
  Fix: Upgrade to Chonkie v1.5.5 or newer, which includes a fix for lazy imports, or ensure `openai` is installed if using older versions.
• gotcha The `TeraflopAIChunker` (introduced in v1.6.2) requires an API key for its service. Without a valid API key, initialization or chunking attempts will fail.
  Fix: Obtain an API key from TeraflopAI and provide it during chunker initialization (e.g., `TeraflopAIChunker(api_key="your_key")`).
• gotcha Chonkie migrated its performance-critical components from Cython to Rust in v1.5.4. While this is largely an internal change, it might affect build environments or specific performance characteristics for advanced users compiling from source.
  Fix: Ensure your environment has Rust toolchain if building from source. For most users installing via `pip`, pre-built wheels should handle this transparently.

Install

• pip install chonkie Core library
• pip install 'chonkie[all]' With all optional dependencies (e.g., LLMs, embedding)
• pip install 'chonkie[llm]' With LLM-related dependencies

Imports

• RecursiveChunker

  from chonkie import RecursiveChunker

• TeraflopAIChunker

  from chonkie import TeraflopAIChunker

  New in v1.6.2, requires `teraflopai` dependency (part of `[llm]` or `[all]` extras) and an API key.
• Visualizer

  from chonkie import Visualizer

  Allows visualizing chunking results.
• FastChunker

  from chonkie import FastChunker

• LateChunker

  from chonkie import LateChunker

Quickstart

This quickstart demonstrates how to use the `RecursiveChunker` to break down a sample text into smaller pieces. It's a common and flexible chunking strategy. An commented-out example for `TeraflopAIChunker` is also included, highlighting the need for an API key and optional dependencies.

import os
from chonkie import RecursiveChunker

# Instantiate a chunker. RecursiveChunker is a common choice.
chunker = RecursiveChunker(chunk_size=500, chunk_overlap=50)

text = (
    "Chonkie is a highly efficient and flexible text chunking library in Python. "
    "It provides various strategies for breaking down long documents into smaller, "
    "manageable chunks, which is crucial for many NLP applications like RAG. "
    "The library supports different chunking methods, including recursive, semantic, "
    "and AI-driven approaches, and can handle various input formats like raw text and HTML. "
    "Recent versions have introduced features like HTML table support and CLI tools." 
)

# Chunk the text
chunks = chunker.chunk(text)

print(f"Original text length: {len(text)} characters")
print(f"Number of chunks: {len(chunks)}")
for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1} (length {len(chunk)}): {chunk[:100]}...")

# Example with TeraflopAIChunker (requires API key and 'llm' extra)
# from chonkie import TeraflopAIChunker
# teraflop_api_key = os.environ.get('TERAFLOPAI_API_KEY', 'YOUR_TERAFLOPAI_API_KEY')
# if teraflop_api_key != 'YOUR_TERAFLOPAI_API_KEY':
#     try:
#         ai_chunker = TeraflopAIChunker(api_key=teraflop_api_key)
#         ai_chunks = ai_chunker.chunk(text)
#         print(f"\nAI Chunker chunks: {len(ai_chunks)}")
#     except Exception as e:
#         print(f"Could not use TeraflopAIChunker: {e}")