compress-pickle

2.1.0 · active · verified Wed Apr 15

compress-pickle is a Python library (version 2.1.0) that thinly wraps the standard `pickle` package with various standard compression libraries (gzip, bz2, lzma, zipfile, and optionally lz4). It provides an interface similar to `pickle.dump`, `pickle.load`, `pickle.dumps`, and `pickle.loads` to seamlessly serialize and deserialize Python objects to disk or file-like objects in a compressed manner. The library has an infrequent release cadence, with its last major update in September 2021.

Warnings

breaking Python 3.5 support was dropped starting from version 2.0.0. If you need to support Python 3.5, you must install `compress-pickle==1.1.1`.
Fix: Upgrade Python to 3.6+ or pin `compress-pickle` to version `1.1.1` for Python 3.5 environments.
gotcha Like the underlying `pickle` module, `compress-pickle` is not secure against maliciously constructed data. Deserializing data from untrusted sources can lead to arbitrary code execution.
Fix: Only unpickle data from trusted sources. Consider alternative serialization formats like JSON for untrusted data.
gotcha For very small Python objects, the overhead introduced by compression might cause the compressed file size to be larger than the uncompressed pickled data.
Fix: Evaluate file sizes for your specific use case. Compression is generally beneficial for larger objects.
gotcha By default, `dump` and `load` infer the compression protocol from the file extension (e.g., '.pkl.gz' for gzip). If the file extension does not match the desired compression or is missing, explicitly set the `compression` argument (e.g., `compression='gzip'`).
Fix: Always use appropriate file extensions or explicitly pass the `compression` argument to `dump` and `load`.
gotcha The `pickle` protocol used can affect compatibility. Objects pickled with newer Python versions or higher protocols may not be readable by older Python versions or lower protocols. `compress-pickle` inherits this behavior.
Fix: Be mindful of the Python versions used for pickling and unpickling. Specify a compatible `pickler_kwargs={'protocol': N}` if cross-version compatibility is critical.

Install

pip install compress-pickle Basic installation
pip install compress-pickle[lz4] With optional lz4 support

Imports

dump
```
from compress_pickle import dump
```
Used for serializing and compressing an object to a file-like object.
load
```
from compress_pickle import load
```
Used for decompressing and deserializing an object from a file-like object.
dumps
```
from compress_pickle import dumps
```
Used for serializing and compressing an object to a bytes object.
loads
```
from compress_pickle import loads
```
Used for decompressing and deserializing an object from a bytes object.

Quickstart

This quickstart demonstrates how to use `compress_pickle.dump` to serialize and compress a Python dictionary to a file, and `compress_pickle.load` to decompress and deserialize it back. The compression method (gzip in this case) is automatically inferred from the file extension '.pkl.gz'.

from compress_pickle import dump, load
import os

data = {'key': 'value', 'numbers': [1, 2, 3]}
filename = 'my_compressed_data.pkl.gz'

# Dump (serialize and compress) the data
dump(data, filename)
print(f"Data saved to {filename} with inferred gzip compression.")

# Load (decompress and deserialize) the data
loaded_data = load(filename)
print(f"Data loaded successfully: {loaded_data}")

# Clean up the created file
os.remove(filename)

view raw JSON →