{"id":3857,"library":"xopen","title":"xopen: Open Compressed Files Transparently","description":"This Python module provides an `xopen` function that works like Python's built-in `open` function but also transparently deals with compressed files. `xopen` selects the most efficient method for reading or writing a compressed file, often leveraging external tools like `pigz` for parallel processing. It supports gzip (.gz), bzip2 (.bz2), xz (.xz), and optionally Zstandard (.zst) formats. The library is actively maintained, currently at version 2.0.2, and has a consistent release cadence.","status":"active","version":"2.0.2","language":"en","source_language":"en","source_url":"https://github.com/pycompression/xopen","tags":["compression","gzip","bzip2","xz","zstandard","file-io","transparent-compression"],"install":[{"cmd":"pip install xopen","lang":"bash","label":"Install xopen"}],"dependencies":[{"reason":"Provides highly optimized gzip (de)compression when threads=0 or external tools are not preferred/available. It's a direct dependency for optimal gzip performance.","package":"python-isal","optional":false},{"reason":"Enables support for Zstandard (.zst) compressed files.","package":"zstandard","optional":true}],"imports":[{"symbol":"xopen","correct":"from xopen import xopen"}],"quickstart":{"code":"from xopen import xopen\nimport os\n\n# Create a dummy gzipped file\nwith open('example.txt', 'w') as f:\n    f.write('Hello, world!\\nThis is a test.')\nimport gzip\nwith open('example.txt', 'rb') as f_in:\n    with gzip.open('example.txt.gz', 'wb') as f_out:\n        f_out.write(f_in.read())\nos.remove('example.txt') # Clean up uncompressed file\n\n# Open for reading (auto-detects gzip)\nwith xopen('example.txt.gz', mode='rt') as f:\n    content = f.read()\n    print(f'Read content: {content.strip()}')\n\n# Open for writing (creates xz compressed file)\noutput_file = 'output.txt.xz'\nwith xopen(output_file, mode='wt', compresslevel=3) as f:\n    f.write('This is compressed with xz.\\nAnother line.')\n\n# Verify writing (optional: requires another xopen to read it back)\nwith xopen(output_file, mode='rt') as f:\n    written_content = f.read()\n    print(f'Written content to {output_file}: {written_content.strip()}')\n\n# Clean up generated files\nos.remove('example.txt.gz')\nos.remove(output_file)","lang":"python","description":"Demonstrates reading from a gzipped file (auto-detected) and writing to an xz-compressed file with a specified compression level. `xopen` behaves like the built-in `open` but handles compression transparently based on file extension or magic numbers."},"warnings":[{"fix":"Upgrade Python to 3.8 or later, or downgrade `xopen` to `<1.8.0`.","message":"xopen v1.8.0 and later (including 2.x) dropped support for Python 3.7. Users on older Python versions must either upgrade Python or pin `xopen` to a version prior to 1.8.0.","severity":"breaking","affected_versions":">=1.8.0"},{"fix":"Explicitly specify the correct `encoding` parameter (e.g., `encoding='latin-1'` or `encoding=locale.getpreferredencoding(False)`) when working with non-UTF-8 text files.","message":"Unlike Python's built-in `open()`, `xopen()` defaults the `encoding` parameter to 'utf-8' when opening files in text mode ('rt', 'wt', 'at'). If your files are not UTF-8 encoded, this can lead to decoding/encoding errors.","severity":"gotcha","affected_versions":">=1.3.0"},{"fix":"Understand your performance requirements. For maximum speed on multi-core systems, leave `threads=None`. For predictable single-threaded behavior (e.g., in resource-constrained environments or when external tools are undesirable), use `threads=0`.","message":"The `threads` parameter affects (de)compression performance and backend choice. `threads=None` (default) uses up to 4 CPU cores (potentially with external tools like `pigz`). `threads=0` forces single-threaded operation using only Python-based backends (`python-isal` or standard library modules). Behavior and performance will vary significantly.","severity":"gotcha","affected_versions":"All"},{"fix":"Ensure `xopen` is updated to at least v2.0.0. If using `compresslevel=0` with gzip, test thoroughly, especially if external gzip utilities are involved.","message":"For gzip files, `compresslevel=0` (no compression) could previously cause crashes if the external `gzip` application backend was used, as it lacked a `--0` flag. `xopen` now attempts to defer to other backends in this scenario, but unexpected behavior might still occur with specific system configurations or older `xopen` versions.","severity":"gotcha","affected_versions":"<2.0.0 (fixed in 2.0.0, but still a consideration for older deployments)"}],"env_vars":null,"last_verified":"2026-04-11T00:00:00.000Z","next_check":"2026-07-10T00:00:00.000Z"}