Lilcom

1.8.2 · active · verified Mon Apr 13

Lilcom is a Python library that provides lossy compression for sequence data stored in NumPy arrays. It efficiently compresses floating-point or 16-bit integer NumPy arrays into byte strings, typically used in machine learning applications for storing training data and models. The current version is 1.8.2, and it has a fairly active release cadence, with updates addressing compatibility and functionality.

Warnings

gotcha Lilcom provides *lossy* compression. The decompressed data will not be bit-for-bit identical to the original array. The amount of error is controlled by the `tick_power` argument (default -8), which determines the step size for discretized values.
Fix: Be aware of the lossy nature and choose an appropriate `tick_power` for your application's accuracy requirements. `2**(tick_power-1)` gives the maximum error per element.
breaking The method for controlling compression accuracy changed. Older versions of Lilcom might have used `bits_per_sample`. Current versions (e.g., 1.8.2 and recent GitHub README) use `tick_power` to specify the quantization step size.
Fix: If migrating from older code, replace `bits_per_sample` with `tick_power`. The `tick_power` argument (default: -8) is the power of 2 used for the step size between discretized values.
breaking Lilcom requires Python 3.6 or newer. It is not compatible with Python 2.x. Attempts to install or run on unsupported Python versions will fail.
Fix: Ensure your environment uses Python 3.6 or higher. Upgrade Python if necessary.
gotcha The underlying algorithm is highly vulnerable to transmission errors. Even a single bit error in the compressed byte string can make the entire file or sequence unreadable during decompression. This is acceptable for its target machine learning applications where data integrity is often handled at a higher level or re-generated.
Fix: Implement robust error handling, retransmission, or checksums if storing or transmitting compressed data in environments prone to bit errors and where data recovery is critical.
gotcha If installing `lilcom` from source (e.g., if pre-compiled wheels are not available for your specific platform/Python version), a C++ compiler (like g++ or clang) is required on your system.
Fix: Ensure a C++ compiler is installed and accessible in your system's PATH before attempting a source installation (e.g., `apt-get install build-essential` on Debian/Ubuntu or Xcode Command Line Tools on macOS).

Install

pip install lilcom Install latest version

Imports

lilcom
```
import lilcom
```
compress
```
lilcom.compress(...)
```
decompress
```
lilcom.decompress(...)
```

Quickstart

This quickstart demonstrates how to compress a NumPy array using `lilcom.compress()` and then decompress it using `lilcom.decompress()`. Note that `lilcom` is a lossy compression utility, so the decompressed array will not be exactly identical to the original. The `tick_power` argument controls the precision of the compression.

import numpy as np
import lilcom

# Create a sample NumPy array
a = np.random.randn(300, 500).astype(np.float32)

# Compress the array (default tick_power=-8, controls accuracy)
a_compressed = lilcom.compress(a)

# Decompress the array, specifying original dtype
a_decompressed = lilcom.decompress(a_compressed, dtype=a.dtype)

print(f"Original array shape: {a.shape}, dtype: {a.dtype}")
print(f"Compressed data size: {len(a_compressed)} bytes")
print(f"Decompressed array shape: {a_decompressed.shape}, dtype: {a_decompressed.dtype}")
print(f"Max absolute error: {np.max(np.abs(a - a_decompressed)):.2e}")

view raw JSON →