JSON Stream Rust Tokenizer

0.5.1 · active · verified Sat Apr 11

A faster tokenizer for the `json-stream` Python library, this package ports `json-stream`'s internal tokenizer to Rust using PyO3, offering a significant parsing speedup (4-10x on CPython). As of `json-stream` version 2.0+, it is automatically detected and used by default, making explicit installation or usage generally unnecessary. It targets Python versions <3.15,>=3.8 and is actively maintained.

Warnings

gotcha Starting with `json-stream` version 2.0, this tokenizer is used automatically if available. Explicitly installing `json-stream-rs-tokenizer` or passing `RustTokenizer` to `json_stream.load()` is usually unnecessary unless you are on an older `json-stream` version or need a specific configuration.
Fix: Ensure `json-stream` is updated. If you are using `json-stream` 2.0+, simply installing `json-stream-rs-tokenizer` is sufficient for it to be used implicitly.
breaking Installation from source requires a Rust toolchain. If a prebuilt wheel is not available for your platform and Python version, the installation process will attempt to build from source. A failed Rust build might still report a successful Python package installation, but `RustTokenizer` will not be importable.
Fix: Ensure a Rust toolchain is installed and accessible in your environment (`curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`). Increase pip verbosity (`pip install -vv json-stream-rs-tokenizer`) to see detailed build logs and diagnose Rust compilation errors.
gotcha When installed in editable/development mode, the Rust library might be compiled in debug mode, which can make it *slower* than the pure-Python tokenizer.
Fix: To ensure release mode compilation, verify installation commands use `--release`. Check `pip install -v` output for Rust compilation flags.
gotcha For PyPy, the performance improvement from `json-stream-rs-tokenizer` is significantly lower (1.0-1.5x) compared to CPython (4-10x).
Fix: Be aware of this limitation if deploying on PyPy; the benefits might not be as pronounced.
gotcha When parsing mixed data with `json-stream.load(..., correct_cursor=True)`, keeping track of the exact stream position for un-seekable streams incurs a significant performance cost.
Fix: Avoid `correct_cursor=True` for un-seekable streams unless absolutely necessary for mixed data parsing. Consider refactoring data formats or using seekable streams if possible.

Install

pip install json-stream-rs-tokenizer Install latest version

Imports

RustTokenizer
```
from json_stream_rs_tokenizer import RustTokenizer
```
The Rust tokenizer is exposed directly from `json_stream_rs_tokenizer`, not an internal `json_stream` submodule.
load
```
from json_stream_rs_tokenizer import load
```
This provides a convenience wrapper around `json_stream.load` that pre-configures the Rust tokenizer.

Quickstart

This quickstart demonstrates how to explicitly use the `RustTokenizer` with `json-stream`'s `load` function and how to use the convenience `load` wrapper provided by this package.

from io import StringIO
from json_stream import load as json_stream_load
from json_stream_rs_tokenizer import RustTokenizer

# Example JSON data
json_buf = StringIO('{ "a": [1,2,3,4], "b": [5,6,7] }')

# Explicitly use the Rust tokenizer with json_stream.load
d = json_stream_load(json_buf, tokenizer=RustTokenizer)

print(f"Keys: {list(d.keys())}") # Output: Keys: ['a', 'b']
for k, l in d.items():
    print(f"{k}: {' '.join(str(n) for n in l)}")
# Expected output for 'a': a: 1 2 3 4

# Alternatively, use the convenience wrapper from json_stream_rs_tokenizer
json_buf_alt = StringIO('{ "x": "hello", "y": "world" }')
from json_stream_rs_tokenizer import load as rs_load
data_alt = rs_load(json_buf_alt)
print(f"Value of x: {data_alt['x']}") # Output: Value of x: hello

view raw JSON →