StringZilla

4.6.0 · active · verified Fri Apr 10

StringZilla is a Python library that significantly accelerates string operations like search, hashing, sorting, and processing, leveraging SIMD (Single Instruction, Multiple Data) and SWAR (SIMD Within A Register) for performance. It's designed to handle large textual datasets (100+ GB) efficiently, offering a `Str` class similar to Python's built-in `str` and a `File` class for memory-mapping files. Currently at version 4.6.0, it sees frequent updates with multiple patch and minor releases per month.

Warnings

gotcha Users often install `stringzilla` (the base package) but might expect parallel performance. For multi-CPU backends, `stringzillas-cpus` is required, and for Nvidia GPU, `stringzillas-cuda` must be installed separately. These are distinct packages, and `stringzilla` only provides serial algorithms.
Fix: Ensure you install `stringzillas-cpus` or `stringzillas-cuda` if parallel processing is desired: `pip install stringzillas-cpus`.
gotcha Optimal performance and certain advanced features (e.g., specific case-folding and case-insensitive search paths) heavily rely on modern CPU extensions like AVX-512, Arm Neon, or SVE. Running on older hardware or virtualized environments without these features may result in lower performance than expected or fallback to less optimized scalar implementations.
Fix: Consult `stringzilla.__capabilities__` to understand the detected hardware features. Performance will scale with available SIMD instructions.
gotcha The `stringzillas` (parallel) components are explicitly noted as being in 'beta' and 'under active development, and are likely to break in subsequent releases'. Users of these parallel backends should anticipate potential API changes or breaking changes between minor versions.
Fix: Pin `stringzillas` versions carefully in production environments and review release notes for breaking changes upon upgrade.
gotcha StringZilla aims for full Unicode 17.0 compliance, particularly for case-folding and case-insensitive searches. Ensure that your input data is correctly encoded in UTF-8 to leverage these features accurately. Incorrect encodings can lead to unexpected search results or behavior.
Fix: Always provide UTF-8 encoded strings to StringZilla functions for full Unicode compliance. Use `str.encode('utf-8')` if necessary.

Install

pip install stringzilla For serial algorithms
pip install stringzillas-cpus For parallel multi-CPU backends
pip install stringzillas-cuda For parallel Nvidia GPU backends

Imports

Str
```
from stringzilla import Str
```
Main string class for in-memory byte arrays.
File
```
from stringzilla import File
```
Memory-maps files for immutable, shared access.
Strs
```
from stringzilla import Strs
```
Class for collections of strings, similar to `list[str]`, used in split operations.

Quickstart

Demonstrates basic usage of `Str` for in-memory string operations and `File` for memory-mapped file handling, including length, substring checks, and finding substrings.

from stringzilla import Str, File

# Using Str for in-memory strings
text_str = Str('hello stringzilla and the world!')
print(f"Length of text_str: {len(text_str)}")
print(f"Does 'stringzilla' exist? {'stringzilla' in text_str}")
print(f"Index of 'world': {text_str.find('world')}")

# Example with File (requires a dummy file)
# To run this, create a file named 'example.txt' with some content
import os
with open('example.txt', 'w') as f:
    f.write('This is a test file for StringZilla.\n')
    f.write('It demonstrates memory-mapped file usage.')

file_str = File('example.txt')
print(f"Length of file_str: {len(file_str)}")
print(f"Does 'test file' exist? {'test file' in file_str}")
print(f"Lines in file_str: {len(file_str.splitlines())}")

# Clean up the dummy file
os.remove('example.txt')

view raw JSON →