JSON Stream

Warnings

• gotcha When reading in default 'transient' mode (e.g., `json_stream.load(file_obj)`), data is discarded after it's been read. Attempting to access previously consumed elements (e.g., `data['items'][0]` then `data['items'][0]` again if `items` is a large list) will raise a `TransientAccessException`.
  Fix: If you need to re-read or persist parts of the JSON document, use `json_stream.load(file_obj, persistent=True)` for the entire document, or explicitly convert desired subsections to standard Python types (e.g., `list(data['items'])`) before moving past them in the stream. Be aware that `persistent=True` will load the entire document into memory, negating memory benefits for very large files.
• gotcha When streaming from network responses, using `requests.get(url, stream=True).json()` still reads the entire JSON payload into memory before parsing. The `requests` library's `.json()` method is not stream-aware in this context.
  Fix: Use `json_stream.requests.load(response)` after making a `requests.get(url, stream=True)` call. This leverages `json-stream`'s streaming capabilities correctly. Ensure `stream=True` is passed to `requests.get()`.
• gotcha The standard library's `json.dump()` or `json.dumps()` functions, when given a regular Python `dict` or `list`, will build the entire data structure in memory first, even if using `json-stream` for other parts of your application.
  Fix: To achieve streaming output, you *must* wrap your top-level Python generators for dictionaries and lists with `json_stream.writer.streamable_dict()` and `json_stream.writer.streamable_list()`, respectively, before passing them to `json.dump()` or `json.dumps()`.
• gotcha While `json-stream` excels at memory efficiency, be mindful of common JSON syntax errors (e.g., trailing commas, single quotes instead of double quotes for keys/strings, unquoted keys, comments) which are not permitted in strict JSON and can lead to `JSONDecodeError`.
  Fix: Ensure generated or received JSON adheres to the strict JSON specification. Tools like `json.tool` (built-in Python module) or online JSON validators can help identify issues. Python string literals using single quotes for keys or values are a common source of error if directly copied into JSON strings without conversion to double quotes.

Install

• pip install json-stream Install `json-stream`
• pip install json-stream[rs] Install with Rust tokenizer for speedups
• pip install json-stream[requests] Install with `requests` integration

Imports

• load

  import json_stream
  data = json_stream.load(file_object)

• streamable_dict

  from json_stream.writer import streamable_dict

• streamable_list

  from json_stream.writer import streamable_list

Quickstart

This quickstart demonstrates both streaming JSON decoding and encoding. For decoding, `json_stream.load()` reads from a file-like object, allowing you to access elements as they are parsed without loading the entire structure into memory. For encoding, `streamable_dict` and `streamable_list` wrap Python generators, enabling JSON serialization of large or dynamically generated data structures without constructing the full object graph upfront before `json.dumps()` or `json.dump()` is called.

import json_stream
from json_stream.writer import streamable_dict, streamable_list
import io
import json

# --- Reading JSON (Decoding) ---
json_data_str = '{"name": "Alice", "items": [1, 2, 3], "settings": {"active": true}}'

# Simulate a file-like object for streaming
json_stream_input = io.StringIO(json_data_str)

# Load the stream in transient mode (default)
data = json_stream.load(json_stream_input)

# Access data - values are loaded as accessed
name = data['name']
first_item = data['items'][0]
setting_active = data['settings']['active']

print(f"Decoded Name: {name}")
print(f"Decoded First Item: {first_item}")
print(f"Decoded Setting Active: {setting_active}")

# --- Writing JSON (Encoding) ---
def generate_items():
    for i in range(3):
        yield i + 1

def generate_data():
    yield 'id', 123
    yield 'status', 'processed'
    yield 'results', streamable_list(generate_items())

# Use streamable_dict for the top-level object
streaming_output = streamable_dict(generate_data())

# Dump to a string (or file) using the standard json module
# The streamable_dict/list objects adapt to json.dump/dumps
encoded_json = json.dumps(streaming_output)
print(f"Encoded JSON: {encoded_json}")

# Expected output for writing is a complete JSON string after dumps() is called.