Chonkie Core

0.10.1 · active · verified Fri Apr 17

Chonkie Core is a high-performance Python library for semantic text chunking, powered by a Rust backend for speed. It offers various strategies including delimiter-based, size-constrained, and Savitzky-Golay filter-based semantic splitting. The library is actively developed, with frequent releases adding new features and optimizations.

Common errors

```
ModuleNotFoundError: No module named 'memchunk'
```
cause The package name was changed from `memchunk` to `chonkie-core` in version `0.5.0`.

fix Replace all `import memchunk` statements with `import chonkie_core` (or `from chonkie_core import ...`). Ensure you have `chonkie-core` installed via `pip install chonkie-core`.
```
AttributeError: 'Chunker' object has no attribute 'patterns'
```
cause The `.patterns()` method for multi-byte delimiters was added to the Python bindings in `chonkie-core` version `0.10.1`.

fix Upgrade your `chonkie-core` installation to `0.10.1` or newer: `pip install --upgrade chonkie-core`.
```
TypeError: argument 'text': 'int' object cannot be interpreted as a string, expected str
```
cause You are passing an integer or another non-string/non-bytes type as the primary text input to a chunking function.

fix Ensure the `text` argument passed to `Chunker` or `chunk` is a string (`str`) type. For example, `chunk(str(my_int_var))`.

Warnings

breaking The package and module name was changed from `memchunk` to `chonkie-core` in version `0.5.0`. Existing `import memchunk` statements will fail.
Fix: Update all import statements from `memchunk` to `chonkie_core` (e.g., `from memchunk import Chunker` becomes `from chonkie_core import Chunker`).
breaking The `.patterns()` API for multi-byte delimiter support was introduced for Python in `v0.10.1`. Attempting to use it on earlier versions will result in an `AttributeError`.
Fix: Upgrade to `chonkie-core>=0.10.1` to use the `.patterns()` API. For earlier versions, only ASCII delimiters via `.delimiters()` are supported.
gotcha The `chunk` and `Chunker` functions primarily operate on string (`str`) inputs. While they can sometimes handle bytes, unexpected behavior or type errors can occur if non-string/bytes types are passed directly.
Fix: Ensure the `text` argument passed to `Chunker` or `chunk` is a `str`. If you have byte data, decode it first (e.g., `my_bytes.decode('utf-8')`).
gotcha The Savitzky-Golay filter module, including `savgol_filter` and related functions for semantic chunking, was introduced in `v0.9.0`. These functions internally leverage NumPy for efficient array operations, providing zero-copy performance.
Fix: Upgrade to `chonkie-core>=0.9.0` to access the Savitzky-Golay filter module. Although NumPy is not a direct dependency of `chonkie-core` itself, it's highly recommended for optimal performance when using these functions.

Install

pip install chonkie-core Install latest version

Imports

Chunker
wrong:
```
from memchunk import Chunker
```
correct:
```
from chonkie_core import Chunker
```
Package renamed from 'memchunk' to 'chonkie-core' in v0.5.0
chunk
wrong:
```
from memchunk import chunk
```
correct:
```
from chonkie_core import chunk
```
Package renamed from 'memchunk' to 'chonkie-core' in v0.5.0
chunk_offsets
wrong:
```
from memchunk import chunk_offsets
```
correct:
```
from chonkie_core import chunk_offsets
```
Package renamed from 'memchunk' to 'chonkie-core' in v0.5.0
merge_splits
wrong:
```
from memchunk import merge_splits
```
correct:
```
from chonkie_core import merge_splits
```
Package renamed from 'memchunk' to 'chonkie-core' in v0.5.0. Introduced in v0.8.0.
split_at_delimiters
wrong:
```
from memchunk import split_at_delimiters
```
correct:
```
from chonkie_core import split_at_delimiters
```
Package renamed from 'memchunk' to 'chonkie-core' in v0.5.0. Introduced in v0.6.0.
savgol_filter
wrong:
```
from memchunk import savgol_filter
```
correct:
```
from chonkie_core import savgol_filter
```
Package renamed from 'memchunk' to 'chonkie-core' in v0.5.0. Savitzky-Golay module introduced in v0.9.0.

Quickstart

Demonstrates basic text chunking using the Chunker class for customizable splitting, the `chunk` convenience function, and retrieving character offsets with `chunk_offsets`. It includes examples using both ASCII and multi-byte delimiters/patterns.

from chonkie_core import Chunker, chunk, chunk_offsets

text = "This is the first sentence. This is the second sentence! And this is the third sentence, with a comma. Finally, the last one. Here is some Japanese: これは日本語のテキストです。句読点も含まれます。"

# Using Chunker class with delimiters and patterns
print("--- Using Chunker ---")
chunks_obj = list(Chunker(text, delimiters="\n.?!", patterns=["。", "，", "！"]))
for c in chunks_obj:
    print(f"'{c.text}' (len: {len(c.text)})\nOffset range: {c.offset_range}")

# Using convenience function `chunk`
print("\n--- Using chunk function ---")
for c in chunk(text, delimiters=".", patterns=["。"]): # The 'chunk' function returns Chunk objects
    print(f"'{c.text}' (len: {len(c.text)})\nOffset range: {c.offset_range}")

# Getting offsets directly
print("\n--- Using chunk_offsets function ---")
offsets = chunk_offsets(text, delimiters=".", patterns=["。"])
print(f"Offsets: {offsets}")

view raw JSON →