{"id":1796,"library":"zstd","title":"Python ZSTD Bindings","description":"The 'zstd' library provides fast Python bindings to Yann Collet's Zstandard (zstd) lossless compression algorithm. It offers a compelling balance of speed and compression ratio, making it a popular choice for real-time compression and large-scale data processing. The library is actively maintained with frequent releases, currently at version 1.5.7.3, and focuses on performance optimizations and bug fixes.","status":"active","version":"1.5.7.3","language":"en","source_language":"en","source_url":"https://github.com/sergey-dryabzhinsky/python-zstd","tags":["compression","zstd","data-processing","performance"],"install":[{"cmd":"pip install zstd","lang":"bash","label":"Install 'zstd' from PyPI"}],"dependencies":[],"imports":[{"note":"The PyPI package `zstd` (or `python-zstd`) exposes its API directly under the `zstd` module name. `zstandard` is a separate, distinct Python library for Zstandard.","wrong":"import zstandard","symbol":"zstd","correct":"import zstd"}],"quickstart":{"code":"import zstd\n\noriginal_data = b\"This is some data that will be compressed using Zstandard. It\\'s a fairly long string to demonstrate compression efficiency.\" * 100\n\n# Compress data\n# level can range from -100 (ultra-fast) to 22 (slowest, best compression)\n# threads can be 0 (auto-tune) or a specific number\ncompressed_data = zstd.compress(original_data, level=3, threads=0)\n\nprint(f\"Original size: {len(original_data)} bytes\")\nprint(f\"Compressed size: {len(compressed_data)} bytes\")\nprint(f\"Compression ratio: {len(compressed_data) / len(original_data):.2f}\")\n\n# Decompress data\ndecompressed_data = zstd.decompress(compressed_data)\n\nassert original_data == decompressed_data\nprint(\"Decompression successful! Data matches original.\")\n\n# Example of streaming compression for large data\ncctx = zstd.ZstdCompressor(level=1)\ndctx = zstd.ZstdDecompressor()\n\nchunk_size = len(original_data) // 5\ncompressed_chunks = []\n\n# Stream compress\nfor i in range(0, len(original_data), chunk_size):\n    chunk = original_data[i:i + chunk_size]\n    compressed_chunk = cctx.compress(chunk)\n    compressed_chunks.append(compressed_chunk)\n# Important: flush the compressor to finalize the frame\ncompressed_chunks.append(cctx.flush())\n\nstreaming_compressed_data = b''.join(compressed_chunks)\n\n# Stream decompress\nstreaming_decompressed_data = dctx.decompress(streaming_compressed_data)\n\nassert original_data == streaming_decompressed_data\nprint(\"Streaming decompression successful! Data matches original.\")","lang":"python","description":"Demonstrates basic one-shot compression and decompression of bytes data using `zstd.compress()` and `zstd.decompress()`. It also includes an example of streaming compression/decompression for larger datasets, highlighting the use of `ZstdCompressor` and `ZstdDecompressor` and the importance of flushing."},"warnings":[{"fix":"For Python 3.14+, explicitly use `from backports import zstd` if you intend to use the PyPI `zstd` package, or adapt your code to use the standard library's `from compression import zstd` module. For cross-version compatibility, use conditional imports (`if sys.version_info >= (3, 14): from compression import zstd else: import zstd`).","message":"Starting with Python 3.14, a new `compression.zstd` module will be added to the standard library (PEP 784). This may cause import ambiguities or conflicts if the third-party `zstd` package is also installed and `import zstd` is used directly, as the standard library module might take precedence or behave differently.","severity":"breaking","affected_versions":"Python 3.14+"},{"fix":"For large datasets, use the streaming APIs provided by `zstd.ZstdCompressor` and `zstd.ZstdDecompressor` which process data in chunks, keeping memory usage manageable. Ensure files are opened in binary mode ('rb', 'wb') for streaming I/O.","message":"Using `zstd.compress()` or `zstd.decompress()` with extremely large datasets (e.g., multi-gigabyte files) can lead to significant memory consumption as the entire input/output must fit in memory simultaneously. This can result in `MemoryError`.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Always call `compressor.flush()` after the last `compressor.compress()` call to finalize the frame. Using `ZstdCompressor` as a context manager (e.g., `with cctx.stream_writer(fh) as compressor: ...`) can help ensure flushing is handled automatically upon exiting the block.","message":"When using `ZstdCompressor` for incremental compression, it is crucial to call its `flush()` method after providing all input data to ensure that all buffered compressed data is emitted and the Zstandard frame is properly finalized. Failing to do so can result in incomplete or corrupted compressed data that cannot be decompressed correctly by other tools or even the same library.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Review any code that calls `zstd.ZSTD_min_compression_level()` and adapt it to expect a standard integer value for the compression level. Most users are unlikely to be directly affected unless interacting with this specific low-level function.","message":"In version 1.5.7.1, the `ZSTD_min_compression_level()` function was fixed to return a 'real number' (likely a standard integer representing the level) instead of a 'shifted int value'. If previous code relied on the specific bitwise representation or shifted value returned by this function, its behavior will change.","severity":"breaking","affected_versions":"1.5.7.1 and later"}],"env_vars":null,"last_verified":"2026-04-09T00:00:00.000Z","next_check":"2026-07-08T00:00:00.000Z"}