Indexed Gzip

1.10.3 · active · verified Thu Apr 16

The `indexed-gzip` project is a Python extension providing fast random access to gzip files by building an index of seek points. It acts as a drop-in replacement for Python's built-in `gzip.GzipFile` class, significantly improving performance for `seek` operations on large gzipped files. It is currently at version 1.10.3 and is actively maintained with regular releases.

Common errors

```
IOError: No write support for IndexedGzipFile
```
cause Attempting to open an `IndexedGzipFile` in write mode ('w', 'wb', 'a', etc.) or calling write methods on it.

fix The `IndexedGzipFile` is strictly read-only. For writing gzipped files, use Python's built-in `gzip` module (e.g., `gzip.GzipFile('file.gz', 'wb')`) or another appropriate library. Once written, the file can be opened with `indexed-gzip` for efficient random access.
```
TypeError: argument of type 'Path' is not iterable
```
cause Passing a `pathlib.Path` object directly as the `filename` argument to `IndexedGzipFile` in an older version (<1.10.2) that did not explicitly support `pathlib.Path`.

fix Upgrade `indexed-gzip` to version 1.10.2 or later. Alternatively, convert the `pathlib.Path` object to a string before passing it: `igzip.IndexedGzipFile(str(my_path_obj))`.
```
Extremely slow seek() operations when using `gzip.GzipFile` on large files.
```
cause The standard `gzip.GzipFile` class must decompress from the beginning of the file up to the desired seek point, making random access inefficient, especially for large files.

fix Replace `gzip.GzipFile` with `indexed_gzip.IndexedGzipFile`. This library builds an internal index allowing for much faster random `seek` operations. `import indexed_gzip as igzip` and use `igzip.IndexedGzipFile` instead of `gzip.GzipFile`.

Warnings

breaking The `IndexedGzipFile` class currently does not support writing data. It is a read-only interface. Attempting to open in write mode or call write methods will result in an error.
Fix: Use Python's built-in `gzip` module or another library for writing gzipped files. Then, open the saved file with `indexed-gzip` for fast random access.
gotcha The `spacing` parameter during `IndexedGzipFile` initialization (or implicitly during index building) controls the density of seek points. A smaller `spacing` improves seek performance but increases memory usage for the index, and vice-versa. The default is 1MB.
Fix: Tune the `spacing` parameter (e.g., `IndexedGzipFile(filename, spacing=65536)`) based on your application's read patterns and memory constraints. For very fine-grained seeking, a smaller value might be beneficial, while for large files with infrequent seeks, a larger value saves memory.
deprecated Prior to version 1.10.2, passing `pathlib.Path` objects directly to `IndexedGzipFile` for the filename argument might not have been fully supported, potentially leading to errors or unexpected behavior.
Fix: Upgrade to `indexed-gzip` version 1.10.2 or newer. If upgrading is not possible, convert `pathlib.Path` objects to strings using `str(path_obj)` before passing them to `IndexedGzipFile`.
gotcha A bug in versions prior to 1.10.0 could occur when CRC validation was disabled, particularly on GZIP streams where the stream footer contained bytes matching the GZIP magic bytes `0x1f8b`.
Fix: Upgrade to `indexed-gzip` version 1.10.0 or newer to avoid this specific data corruption/read error. If unable to upgrade, ensure CRC validation is enabled (`crc_check=True`, default) or be aware of potential issues with malformed (but technically valid) GZIP files.

Install

pip install indexed-gzip Install via pip

Imports

IndexedGzipFile
```
from indexed_gzip import IndexedGzipFile
```
Commonly imported as 'igzip' for brevity: `import indexed_gzip as igzip` then `igzip.IndexedGzipFile`.

Quickstart

This example demonstrates how to open a gzip file with `IndexedGzipFile`, perform random `seek` and `read` operations, and explicitly build a full index. It shows how `indexed-gzip` acts as a drop-in replacement for `gzip.GzipFile` for read operations, but with significantly improved performance for non-sequential access.

import indexed_gzip as igzip
import os

# Create a dummy gzip file for demonstration
dummy_data = b"This is some sample data for a gzipped file.\nRepeat this line many times to make it bigger.\n" * 10000
with open('test_file.gz', 'wb') as f:
    import gzip
    g = gzip.GzipFile(fileobj=f, mode='wb')
    g.write(dummy_data)
    g.close()

# Open the indexed gzip file
try:
    with igzip.IndexedGzipFile('test_file.gz') as fobj:
        print(f"Original file size: {len(dummy_data)} bytes")
        
        # Seek to an arbitrary position
        fobj.seek(15000)
        data = fobj.read(100)
        print(f"Read 100 bytes from offset 15000: {data.decode(errors='ignore')[:50]}...")
        
        # Seek to another position
        fobj.seek(5000)
        data = fobj.read(50)
        print(f"Read 50 bytes from offset 5000: {data.decode(errors='ignore')[:50]}...")
        
        # Build a full index explicitly (optional, often done on demand)
        fobj.build_full_index()
        print(f"Index built with {fobj.tell()} bytes processed.")

finally:
    # Clean up the dummy file
    if os.path.exists('test_file.gz'):
        os.remove('test_file.gz')

view raw JSON →