Blosc2

Warnings

• breaking Buffers generated with C-Blosc2 are generally not format-compatible with C-Blosc1 (i.e., forward compatibility is not supported). While C-Blosc2 is backward compatible with the C-Blosc1 API and in-memory format, users upgrading or sharing data between versions should be aware of this limitation.
  Fix: Ensure all consumers of Blosc2-generated data are using Blosc2. For C-Blosc1 compatibility, consider defining `BLOSC1_COMPAT` during C-Blosc2 compilation if using the C API directly.
• breaking The `NDArray.size` property changed its behavior in version 3.11.0. It now returns the number of elements in the array (Array API standard compliant) instead of the size of the array in bytes. Code relying on `NDArray.size` for byte size will need to be updated.
  Fix: For the size in bytes, use `ndarray.nbytes`. Update any code that used `.size` expecting byte count.
• breaking The `blosc2.concatenate()` function was renamed to `blosc2.concat()` in version 3.5.0 to align with the Array API. While `concatenate` is still available for backward compatibility, it will be removed in a future release.
  Fix: Migrate code to use `blosc2.concat()` for future compatibility.
• gotcha When specifying compression codecs or filters, users must pass members of the `blosc2.Codec` and `blosc2.Filter` enums, respectively, not string literals. Passing strings will result in an `AttributeError`.
  Fix: Always use `blosc2.Codec.CODEC_NAME` (e.g., `blosc2.Codec.BLOSCLZ`) and `blosc2.Filter.FILTER_NAME` (e.g., `blosc2.Filter.SHUFFLE`).
• gotcha For in-memory tasks, Blosc2's overhead can sometimes make it slower than pure NumPy/Numexpr, especially on x86 CPUs. However, it consistently outperforms them for on-disk operations or on modern ARM architectures (e.g., Apple Silicon) due to its efficient use of compression and cache optimization.
  Fix: Evaluate performance for specific workloads and hardware. Blosc2 shines in I/O-bound scenarios and for out-of-core computations. Consider the trade-offs between compression overhead and I/O savings.
• gotcha When using Blosc2 as an HDF5 filter, it is important not to activate the shuffle filter directly within HDF5. Blosc2 uses an internal SIMD shuffle that is much faster and should be handled by Blosc2 itself for optimal performance.
  Fix: Configure Blosc2's compression parameters directly, ensuring its internal shuffle is used when writing HDF5 datasets with the Blosc2 filter.

Install

• pip install blosc2 --upgrade Install from PyPI
• conda install -c conda-forge python-blosc2 Install with Conda

Imports

• blosc2

  import blosc2

• NDArray

  import blosc2
  array = blosc2.zeros((10, 10))

  NDArray is generally accessed as `blosc2.NDArray` or through creation functions like `blosc2.zeros` or `blosc2.asarray`.
• TreeStore

  from blosc2 import TreeStore

• Codec

  blosc2.Codec.BLOSCLZ

  Compression codecs should be passed using the `blosc2.Codec` enum members, not string literals.
• Filter

  blosc2.Filter.SHUFFLE

  Compression filters should be passed using the `blosc2.Filter` enum members, not string literals.

Quickstart

This quickstart demonstrates creating a compressed NDArray, performing a simple computation, and decompressing it. It also includes an example of using `blosc2.TreeStore` for hierarchical data persistence, converting NumPy arrays and Blosc2 NDArrays to a file-based storage format.

import blosc2
import numpy as np

# Create a Blosc2 NDArray from a NumPy array
data = np.arange(1_000_000, dtype=np.float64)
ndarray = blosc2.asarray(data)
print(f"Original data size: {data.nbytes / (1024**2):.2f} MB")
print(f"Compressed data size: {ndarray.nbytes / (1024**2):.2f} MB")

# Perform a computation (e.g., sum) on the compressed array
computed_sum = ndarray.sum()
print(f"Sum of array elements: {computed_sum}")

# Decompress the array back to a NumPy array
decompressed_data = ndarray[:]
assert np.allclose(data, decompressed_data)
print("Data compressed and decompressed successfully.")

# Example of using TreeStore for hierarchical data storage
with blosc2.TreeStore("my_data.b2z", mode="w") as ts:
    ts["/group1/dataset_a"] = np.random.rand(100, 100)
    ts["/group2/dataset_b"] = blosc2.zeros((50, 50), dtype=np.int32)
    print("Data stored in TreeStore 'my_data.b2z'.")

with blosc2.TreeStore("my_data.b2z", mode="r") as ts:
    ds_a = ts["/group1/dataset_a"]
    ds_b = ts["/group2/dataset_b"]
    print(f"Read dataset_a shape: {ds_a.shape}")
    print(f"Read dataset_b dtype: {ds_b.dtype}")

os.remove("my_data.b2z")