h5netcdf: NetCDF4 via h5py

Warnings

• breaking With h5py version 3.0+, the default behavior for decoding variable-length strings changed from automatically decoding to UTF-8 strings to returning arrays of bytes. To restore the automatic decoding behavior that matches the legacy h5py API and netCDF4-python, explicitly set `decode_vlen_strings=True` in the `h5netcdf.File` constructor.
  Fix: Pass `decode_vlen_strings=True` to `h5netcdf.File()` constructor when opening files with variable-length strings, or handle byte arrays directly.
• breaking The `track_order` parameter's default behavior changed in h5netcdf 1.1.0 to `True` (if h5py >= 3.7.0 is detected) for *newly created* netCDF4 files. This ensures compatibility with netCDF4-c. However, files created with older versions of h5netcdf (e.g., 1.0.2 and older, except for 0.13.0) where `track_order=False` was effectively or explicitly set, will continue to open with order tracking disabled in newer h5netcdf versions, potentially leading to interoperability issues if external netCDF4-c tools expect ordered dimensions/variables.
  Fix: For new files, ensure `h5py >= 3.7.0` is installed (h5netcdf will default to `track_order=True`). For existing files created with older versions, be aware that order tracking might be disabled upon reopening; if compatibility with netCDF4-c append operations is critical, recreation with `track_order=True` might be necessary, or explicitly setting the parameter if using older h5py versions.
• gotcha By default, `h5netcdf` raises a `CompatibilityError` if you attempt to write HDF5 features (like certain data types or arbitrary filters) that are not considered valid NetCDF4 by other tools. While these are valid HDF5, they break NetCDF compatibility. In versions prior to 0.7.3, this was merely a warning.
  Fix: To allow writing these non-NetCDF4 compliant HDF5 features, pass `invalid_netcdf=True` to the `h5netcdf.File()` constructor. Be aware that such files may not be readable by other netCDF tools.
• gotcha If you access variables in an HDF5 file that have no dimension scale associated with one of their axes, `h5netcdf` will raise a `ValueError`. This often occurs with non-NetCDF HDF5 files.
  Fix: When opening the file, set `phony_dims='sort'` in `h5netcdf.File()` to instruct `h5netcdf` to invent phony dimensions, mimicking standard NetCDF behavior. Alternatively, `phony_dims='access'` can defer phony dimension creation to access time, but with different naming conventions.
• gotcha When using the new API, automatic resizing of unlimited dimensions with array indexing (e.g., `variable[i, :] = data`) is *not* available. This differs from the `netCDF4-python` library's behavior.
  Fix: Manually resize dimensions using `group.resize_dimension(dimension, size)` before writing data that would exceed the current dimension size.
• gotcha Repeated access to properties that rely on the underlying `_h5ds` HDF5 dataset object can be costly in terms of performance, as `_h5ds` is created on demand. This can impact workflows that frequently query properties like `variable.shape` in a loop.
  Fix: Cache the `_h5ds` object or its relevant properties if they are accessed repeatedly in a performance-critical loop. For example, store `variable.shape` in a local variable if it's constant for the loop's duration.
• gotcha If you initialize `h5netcdf.File` by passing an existing `h5py.File` object (e.g., `h5netcdf.File(h5py_file_obj)`), closing the `h5netcdf.File` wrapper will *not* close the underlying `h5py.File` object. However, if the file is opened by path (e.g., `h5netcdf.File('mydata.nc')`), closing the `h5netcdf.File` *will* close the underlying HDF5 file.
  Fix: When wrapping an `h5py.File` object, ensure you explicitly close the original `h5py.File` object when it's no longer needed to release resources.

Install

• pip install h5netcdf Basic install (requires h5py to be installed separately for versions >= 1.8.0)
• pip install h5netcdf[h5py] Install with h5py backend (recommended)
• pip install h5netcdf[pyfive] Install with pyfive pure-Python backend
• pip install h5netcdf[h5pyd] Install with h5pyd backend for HDF5 REST API

Imports

• File

  from h5netcdf import File

  Main class for interacting with NetCDF4 files in the new API.
• Dataset

  from h5netcdf.legacyapi import Dataset

  Entry point for the legacy API, designed for netCDF4-python users.

Quickstart

This quickstart demonstrates how to create a NetCDF4 file, define dimensions, create a variable, write data, add an attribute, create a group, and then read the data back using the `h5netcdf.File` (new API) interface.

import h5netcdf
import numpy as np
import os

file_path = 'my_test_data.nc'

# Write data using the new API
with h5netcdf.File(file_path, 'w') as f:
    f.dimensions = {'x': 5, 'y': 3}
    var = f.create_variable('temperature', ('x', 'y'), 'f4')
    var[:] = np.random.rand(5, 3)
    var.units = 'Kelvin'
    f.create_group('forecast_data')

print(f"Successfully wrote data to {file_path}")

# Read data
with h5netcdf.File(file_path, 'r') as f:
    print(f"Dimensions: {list(f.dimensions.keys())}")
    temp = f.variables['temperature']
    print(f"Variable 'temperature' shape: {temp.shape}")
    print(f"Variable 'temperature' units: {temp.units}")
    print(f"First few values: {temp[:2, :2]}")
    if 'forecast_data' in f.groups:
        print("Group 'forecast_data' exists.")

# Clean up
os.remove(file_path)