Stream ZIP

Warnings

• breaking The signature for the `async_stream_zip` function changed in version `0.0.81`. If you were using the asynchronous interface, verify the updated argument order and names.
  Fix: Consult the `stream-zip` documentation or changelog for `v0.0.81` to update the `async_stream_zip` call with the corrected signature, specifically regarding parameters like `chunk_size`.
• gotcha Client code must explicitly choose between `ZIP_32` and `ZIP_64` compression methods for each member file. The library cannot automatically determine if `ZIP_64` (for files > 4GiB) is needed during streaming, as this information is required before the compressed data is processed. Incorrect choice can lead to `ZipOverflowError` for large files.
  Fix: For files potentially exceeding 4GiB (compressed or uncompressed size, or total archive size), use `ZIP_64`. Otherwise, `ZIP_32` offers broader compatibility. Be mindful of these limits for individual files and the overall archive.
• gotcha Using `NO_COMPRESSION_32` or `NO_COMPRESSION_64` for uncompressed files will buffer the *entire content* of those files in memory. This negates the streaming benefit for uncompressed large files, as their full size and CRC32 must be known before their data in the ZIP stream.
  Fix: Avoid `NO_COMPRESSION_*` for large files if memory efficiency is critical. If uncompressed data must be included, consider if another streaming approach or a different file format is more suitable, or accept the memory footprint for these specific files.
• gotcha Due to the ZIP file format specification, small bits of metadata for each member file (like names) must be placed at the end of the ZIP archive. `stream-zip` buffers this metadata in memory until it can be output. For archives with a very large number of small files, this could still lead to increased memory usage for metadata.
  Fix: Be aware of potential memory spikes if creating ZIPs with an extremely high count of member files, even if each individual file's content is streamed efficiently.

Install

• pip install stream-zip Install stable version

Imports

• stream_zip

  from stream_zip import stream_zip

  The primary function for creating ZIP streams.
• ZIP_32

  from stream_zip import ZIP_32

  Constant for specifying the ZIP compression method, limited to 4GiB. Also `ZIP_64` for larger files.
• async_stream_zip

  from stream_zip import async_stream_zip

  The signature of `async_stream_zip` changed in v0.0.81, affecting how arguments like `chunk_size` are passed. Refer to documentation for the updated signature if using versions prior to 0.0.81.

Quickstart

This quickstart demonstrates how to use `stream_zip` to create a ZIP archive containing multiple files and a directory. The `stream_zip` function takes an iterable of member file definitions, where each member file is a tuple specifying its name, modification time, mode, compression method (`ZIP_32` or `ZIP_64`), and an iterable of binary content chunks. The function returns an iterable that yields chunks of the resulting ZIP file.

from datetime import datetime
from stat import S_IFREG
from stream_zip import ZIP_32, stream_zip, ZIP_64

def generate_file_content(data):
    yield data.encode('utf-8')

def member_files():
    modified_at = datetime.now()
    mode = S_IFREG | 0o600 # Regular file, owner read/write

    # Example 1: Small file using ZIP_32 (default)
    yield (
        'my-file-1.txt',
        modified_at,
        mode,
        ZIP_32,
        generate_file_content('This is some content for file 1.')
    )

    # Example 2: Potentially larger file or for explicit 64-bit support
    yield (
        'my-file-2.json',
        modified_at,
        mode,
        ZIP_64,
        generate_file_content('{"key": "value", "data": [1, 2, 3]}')
    )

    # Example 3: An empty directory
    yield (
        'my-directory/',
        modified_at,
        S_IFREG | 0o755, # Directory permissions
        ZIP_32,
        ()
    )

# Stream the ZIP file chunks
zipped_chunks = stream_zip(member_files())

# In a real application, you would send these chunks directly as an HTTP response
# or write them to a file. For this example, we'll print them.
# You might also use io.BytesIO to collect them into a single bytes object for testing.

# Example of consuming chunks (e.g., writing to a file)
# with open('output.zip', 'wb') as f:
#     for chunk in zipped_chunks:
#         f.write(chunk)

print("Generated ZIP chunks (first few bytes of each):")
for i, chunk in enumerate(zipped_chunks):
    print(f"Chunk {i}: {len(chunk)} bytes (starts with: {chunk[:20]})")
    if i > 5: # Limit output for demonstration
        print("...")
        break