{"id":4063,"library":"json-stream","title":"JSON Stream","description":"json-stream is a Python library (version 2.5.0, actively maintained) designed for efficient streaming JSON encoding and decoding. It allows processing JSON data in chunks, rather than loading the entire document into memory, which significantly reduces memory consumption and latency for large files or network streams. It provides a Pythonic dict/list-like interface for reading and uses generators for writing, making it suitable for web applications, data pipelines, and other scenarios requiring optimized JSON handling.","status":"active","version":"2.5.0","language":"en","source_language":"en","source_url":"https://github.com/daggaz/json-stream","tags":["json","streaming","parser","encoder","memory efficiency","large files"],"install":[{"cmd":"pip install json-stream","lang":"bash","label":"Install `json-stream`"},{"cmd":"pip install json-stream[rs]","lang":"bash","label":"Install with Rust tokenizer for speedups"},{"cmd":"pip install json-stream[requests]","lang":"bash","label":"Install with `requests` integration"}],"dependencies":[{"reason":"Optional Rust-based tokenizer for significant parsing speedups.","package":"json-stream-rs-tokenizer","optional":true},{"reason":"Optional integration for streaming JSON data from URLs.","package":"requests","optional":true},{"reason":"Optional integration for streaming JSON data from URLs.","package":"httpx","optional":true}],"imports":[{"symbol":"load","correct":"import json_stream\ndata = json_stream.load(file_object)"},{"symbol":"streamable_dict","correct":"from json_stream.writer import streamable_dict"},{"symbol":"streamable_list","correct":"from json_stream.writer import streamable_list"}],"quickstart":{"code":"import json_stream\nfrom json_stream.writer import streamable_dict, streamable_list\nimport io\nimport json\n\n# --- Reading JSON (Decoding) ---\njson_data_str = '{\"name\": \"Alice\", \"items\": [1, 2, 3], \"settings\": {\"active\": true}}'\n\n# Simulate a file-like object for streaming\njson_stream_input = io.StringIO(json_data_str)\n\n# Load the stream in transient mode (default)\ndata = json_stream.load(json_stream_input)\n\n# Access data - values are loaded as accessed\nname = data['name']\nfirst_item = data['items'][0]\nsetting_active = data['settings']['active']\n\nprint(f\"Decoded Name: {name}\")\nprint(f\"Decoded First Item: {first_item}\")\nprint(f\"Decoded Setting Active: {setting_active}\")\n\n# --- Writing JSON (Encoding) ---\ndef generate_items():\n    for i in range(3):\n        yield i + 1\n\ndef generate_data():\n    yield 'id', 123\n    yield 'status', 'processed'\n    yield 'results', streamable_list(generate_items())\n\n# Use streamable_dict for the top-level object\nstreaming_output = streamable_dict(generate_data())\n\n# Dump to a string (or file) using the standard json module\n# The streamable_dict/list objects adapt to json.dump/dumps\nencoded_json = json.dumps(streaming_output)\nprint(f\"Encoded JSON: {encoded_json}\")\n\n# Expected output for writing is a complete JSON string after dumps() is called.","lang":"python","description":"This quickstart demonstrates both streaming JSON decoding and encoding. For decoding, `json_stream.load()` reads from a file-like object, allowing you to access elements as they are parsed without loading the entire structure into memory. For encoding, `streamable_dict` and `streamable_list` wrap Python generators, enabling JSON serialization of large or dynamically generated data structures without constructing the full object graph upfront before `json.dumps()` or `json.dump()` is called."},"warnings":[{"fix":"If you need to re-read or persist parts of the JSON document, use `json_stream.load(file_obj, persistent=True)` for the entire document, or explicitly convert desired subsections to standard Python types (e.g., `list(data['items'])`) before moving past them in the stream. Be aware that `persistent=True` will load the entire document into memory, negating memory benefits for very large files.","message":"When reading in default 'transient' mode (e.g., `json_stream.load(file_obj)`), data is discarded after it's been read. Attempting to access previously consumed elements (e.g., `data['items'][0]` then `data['items'][0]` again if `items` is a large list) will raise a `TransientAccessException`.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Use `json_stream.requests.load(response)` after making a `requests.get(url, stream=True)` call. This leverages `json-stream`'s streaming capabilities correctly. Ensure `stream=True` is passed to `requests.get()`.","message":"When streaming from network responses, using `requests.get(url, stream=True).json()` still reads the entire JSON payload into memory before parsing. The `requests` library's `.json()` method is not stream-aware in this context.","severity":"gotcha","affected_versions":"All versions using `requests`"},{"fix":"To achieve streaming output, you *must* wrap your top-level Python generators for dictionaries and lists with `json_stream.writer.streamable_dict()` and `json_stream.writer.streamable_list()`, respectively, before passing them to `json.dump()` or `json.dumps()`.","message":"The standard library's `json.dump()` or `json.dumps()` functions, when given a regular Python `dict` or `list`, will build the entire data structure in memory first, even if using `json-stream` for other parts of your application.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure generated or received JSON adheres to the strict JSON specification. Tools like `json.tool` (built-in Python module) or online JSON validators can help identify issues. Python string literals using single quotes for keys or values are a common source of error if directly copied into JSON strings without conversion to double quotes.","message":"While `json-stream` excels at memory efficiency, be mindful of common JSON syntax errors (e.g., trailing commas, single quotes instead of double quotes for keys/strings, unquoted keys, comments) which are not permitted in strict JSON and can lead to `JSONDecodeError`.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-11T00:00:00.000Z","next_check":"2026-07-10T00:00:00.000Z"}