Numbagg: Fast N-dimensional Aggregation

0.9.4 · active · verified Tue Apr 14

Numbagg provides fast N-dimensional aggregation functions accelerated by Numba's just-in-time (JIT) compiler and NumPy's generalized universal function (gufunc) machinery. It aims to outperform libraries like pandas, bottleneck, and NumPy for certain operations, especially with parallelization. The library is currently at version 0.9.4 and maintains an active development pace with regular updates.

Warnings

gotcha Numbagg is currently considered experimental and not yet ready for production use, as stated on its PyPI page. While robust, its API or internal workings may evolve.
Fix: Monitor official releases and change logs for updates on stability and production readiness.
gotcha The first call to any Numbagg function incurs a significant performance penalty due to Numba's Just-In-Time (JIT) compilation. Subsequent calls to the same function with compatible argument types will be much faster.
Fix: Consider a warm-up run with dummy data if predictable performance from the first invocation is critical in a performance-sensitive application.
deprecated Numbagg's grouped calculation functions (e.g., `numbagg.grouped.group_nanmean`) might be deprecated in favor of using `flox` with `numbagg` as a backend. This is an ongoing discussion within the xarray community, with `flox` potentially offering better support for nD array grouped by 1D labels.
Fix: For new projects or complex grouped aggregations, consider evaluating `flox` (especially with its `numbagg` engine option) instead of directly using `numbagg.grouped` functions. Monitor `numbagg` and `xarray` (and `flox`) repositories for updates on this transition.
gotcha The `numbagg.decorators` module, used internally for creating JIT-compiled aggregation functions, is not part of Numbagg's public API. Its functions and signatures may change without prior notice.
Fix: Avoid direct imports from `numbagg.decorators` for application code. Instead, use the top-level functions directly exposed by the `numbagg` package.

Install

pip install numbagg Install latest version

Imports

nansum

import numbagg
result = numbagg.nansum(...)

move_mean

import numbagg
result = numbagg.move_mean(...)

ndreduce
```
from numbagg.decorators import ndreduce
```
The `decorators` module is explicitly noted as *not* part of Numbagg's public API and is subject to change at any time. Direct imports from `numbagg` are preferred for stable functions.

Quickstart

This quickstart demonstrates how to use `numbagg` for basic array aggregation (nansum) and moving window calculations (move_mean) on NumPy arrays. Note that the first call to any numbagg function will incur JIT compilation overhead.

import numbagg
import numpy as np

a = np.array([1, 2, np.nan, 4, 5])
b = np.random.rand(10, 5)

# Calculate sum, ignoring NaNs
sum_result = numbagg.nansum(a)
print(f"nansum(a): {sum_result}")

# Calculate moving mean with a window of 3
moving_mean_result = numbagg.move_mean(b, window=3, axis=1)
print(f"move_mean(b, window=3, axis=1, shape): {moving_mean_result.shape}")

view raw JSON →