Numcodecs

Warnings

• breaking Numcodecs versions 0.16.x and higher require Python 3.11 or newer. Installations on older Python versions will fail.
  Fix: Upgrade your Python environment to 3.11 or later, or pin numcodecs to an older compatible version (e.g., `<0.16.0`).
• gotcha Many high-performance codecs (e.g., Blosc, Zstd, LZ4, Snappy) are implemented as optional dependencies. If you try to use them without their respective packages installed, an `ImportError` or `ValueError` will occur.
  Fix: Install numcodecs with the required extras, for example, `pip install "numcodecs[blosc]"` for Blosc, or `pip install "numcodecs[all]"` to include all optional codecs.
• breaking Compatibility with Zarr v3 has evolved across recent numcodecs versions. Changes in Zarr3 configuration serialization (v0.15.1) and handling for Zarr 3.1.0 (v0.16.2) may affect users relying on specific Zarr3 features or metadata structures.
  Fix: Ensure you are using a numcodecs version compatible with your Zarr-Python version. Refer to the Zarr-Python documentation and numcodecs changelog for specific compatibility matrices. Update numcodecs to the latest stable version for the best Zarr v3 compatibility.
• gotcha Prior to v0.16.3, there was a known issue with Zstd decompression leading to negative size errors on 32-bit platforms.
  Fix: Upgrade numcodecs to version 0.16.3 or newer if you are using Zstd compression on a 32-bit system.

Install

• pip install numcodecs Base installation
• pip install "numcodecs[all]" Install with all optional codecs

Imports

• numcodecs

  import numcodecs

• Blosc

  from numcodecs.blosc import Blosc

  Specific codecs like Blosc, Zstd, GZip reside in their own submodules, not directly under the top-level numcodecs namespace.
• Zstd

  from numcodecs.zstd import Zstd

  Specific codecs like Blosc, Zstd, GZip reside in their own submodules, not directly under the top-level numcodecs namespace.
• GZip

  from numcodecs.gzip import GZip

  Specific codecs like Blosc, Zstd, GZip reside in their own submodules, not directly under the top-level numcodecs namespace.

Quickstart

This quickstart demonstrates how to instantiate a codec (Blosc or GZip as a fallback), encode a NumPy array into bytes, and then decode it back. It also shows how to globally register and retrieve codecs, which is often used in frameworks like Zarr.

import numpy as np
import numcodecs

# Define some data to encode
data = np.arange(10000, dtype='i4').reshape(100, 100)

# Choose a codec. Blosc is common, but requires 'pip install "numcodecs[blosc]"'.
# If Blosc isn't installed, GZip is a good fallback.
try:
    codec = numcodecs.blosc.Blosc(cname='lz4', clevel=5, shuffle=numcodecs.blosc.SHUFFLE)
    print("Using Blosc codec.")
except ImportError:
    print("Blosc not installed. Falling back to GZip codec.")
    codec = numcodecs.gzip.GZip(level=5)

# Encode the data
encoded_data = codec.encode(data.tobytes())
print(f"Original data shape: {data.shape}, dtype: {data.dtype}")
print(f"Original bytes: {data.nbytes}")
print(f"Encoded bytes: {len(encoded_data)}")

# Decode the data
decoded_bytes = codec.decode(encoded_data)

# Reconstruct the numpy array
decoded_data = np.frombuffer(decoded_bytes, dtype=data.dtype).reshape(data.shape)

# Verify that the decoded data matches the original
assert np.array_equal(data, decoded_data)
print("Data successfully encoded and decoded!")

# You can also register codecs globally for retrieval by ID
numcodecs.register_codec(codec)
retrieved_codec = numcodecs.get_codec({'id': codec.codec_id, **codec.get_config()})
assert retrieved_codec.codec_id == codec.codec_id
print(f"Codec '{retrieved_codec.codec_id}' registered and retrieved successfully.")