LZFSE Python Bindings

0.4.2 · active · verified Sat Apr 11

The `lzfse` library provides Python bindings to Apple's LZFSE (Lempel–Ziv Finite State Entropy) compression algorithm. LZFSE is an open-source lossless data compression algorithm developed by Apple, designed for faster decompression and higher energy efficiency compared to zlib, particularly optimized for modern micro-architectures like arm64. The Python bindings are currently at version 0.4.2 (as of April 19, 2024), with infrequent releases often tied to updates in the underlying C reference implementation or improvements to the Python packaging.

Warnings

breaking The current `lzfse` package is a fork and rename of an earlier project, `pyliblzfse`. Codebases written for `pyliblzfse` might use `import liblzfse`, which will break with the `lzfse` package. The correct import is now `import lzfse`.
Fix: Update import statements from `import liblzfse` to `import lzfse`.
gotcha On platforms where pre-built wheels are not available (e.g., some Linux distributions, or when specific Python versions are not covered), installing `lzfse` might attempt to build from source. This requires a C/C++ compiler and potentially system-specific development headers. On Windows, this often means installing 'Desktop Development with C++' and the Windows 10/11 SDK via Visual Studio Build Tools, failing with errors like 'stddef.h: No such file or directory'.
Fix: Ensure a compatible C/C++ build environment is installed on your system. For Windows, install the 'Desktop Development with C++' workload and the Windows 10/11 SDK using the Visual Studio Build Tools installer. For Linux, ensure `build-essential` or similar developer tools are installed.
gotcha The underlying LZFSE C reference implementation has known limitations and potential issues, including degraded compression performance for inputs larger than 2GiB and documented memory-related crashes (e.g., double free when compressing empty files, malloc errors). As the Python library is a binding, these issues can manifest when using `lzfse`.
Fix: Be aware of these limitations when processing very large files or unusual inputs. Consider pre-processing large data into smaller chunks if 2GiB+ performance degradation is a concern. Refer to the upstream LZFSE C library's GitHub issues for more details and potential workarounds.

Install

pip install lzfse Install from PyPI

Imports

lzfse
```
import lzfse
```
The original project was named 'pyliblzfse' and imported as `liblzfse`. The current `lzfse` package on PyPI is a fork and rename, which should be imported directly as `lzfse`.

Quickstart

This example demonstrates basic LZFSE compression and decompression of a byte string, including error handling for corrupted data.

import lzfse

original_data = b"This is some data to be compressed using LZFSE! " * 100

# Compress data
compressed_data = lzfse.compress(original_data)
print(f"Original size: {len(original_data)} bytes")
print(f"Compressed size: {len(compressed_data)} bytes")

# Decompress data
decompressed_data = lzfse.decompress(compressed_data)
print(f"Decompressed size: {len(decompressed_data)} bytes")

# Verify data integrity
assert original_data == decompressed_data
print("Data compressed and decompressed successfully!")

# Handle decompression errors
corrupted_data = b'\xde\xad\xbe\xef' + compressed_data[4:] # Corrupt header
try:
    lzfse.decompress(corrupted_data)
except lzfse.error as e:
    print(f"Caught expected error during decompression: {e}")

view raw JSON →