mgzip

0.2.5 verified Fri May 01 auth: no python

A multi-threading implementation of Python's gzip module, providing drop-in replacements for gzip.open, gzip.compress, and gzip.decompress with parallel compression/decompression. Current version 0.2.5 supports Python 3.8-3.14. Active development with regular releases.

pip install mgzip

Common errors

error AttributeError: '_MultiGzipReader' object has no attribute '_read_exact' ↓

cause Python 3.12+ removed internal gzip._read_exact method.

fix

Upgrade mgzip to >=0.2.3.

error AttributeError: 'Decompress' object has no attribute 'unconsumed_tail' ↓

cause Python 3.12+ changed zlib decompressobj API.

fix

Upgrade mgzip to >=0.2.3.

error ResourceWarning: unclosed <socket> for multiprocessing pool ↓

cause Multiprocessing pool not cleaned properly in mgzip <0.2.2.

fix

Upgrade mgzip to >=0.2.2.

error TypeError: 'thread' is an invalid keyword argument for this function ↓

cause Using 'threads' or 'num_threads' instead of 'thread'.

fix

Use thread=<N> as keyword argument.

Warnings

gotcha The parameter for number of threads is 'thread' (singular), not 'threads' or 'num_workers'. ↓

fix Use keyword argument 'thread' when opening files.

gotcha mgzip may be slower than gzip for small files or low block counts due to threading overhead. ↓

fix Consider using standard gzip for files < 1 MB.

deprecated The old class name 'MultiGzipFile' is deprecated; use 'open' function instead. ↓

fix Replace 'MultiGzipFile' with mgzip.open() or mgzip.MultiGzipFile is still available but not recommended.

breaking Python 3.12+ removed some internal attributes (e.g., _read_exact, unconsumed_tail); mgzip 0.2.3+ fixed this, but older versions break on Python 3.12+. ↓

fix Upgrade to mgzip >=0.2.3.

Imports

open
wrong
```
import gzip (then using gzip.open)
```
correct
```
from mgzip import open
```
Using gzip.open instead of mgzip.open won't use multithreading.
compress
```
from mgzip import compress
```
decompress
```
from mgzip import decompress
```

Quickstart

Basic usage of mgzip.open for writing and reading gzip files with multiple threads.

from mgzip import open

# Multithreaded gzip compression
with open('example.txt.gz', 'wt', compresslevel=5, thread=4) as f:
    f.write('Hello, world!')

# Multithreaded decompression
with open('example.txt.gz', 'rt', thread=4) as f:
    print(f.read())