Fast numerical expression evaluator for NumPy

2.14.1 · active · verified Sun Apr 05

NumExpr is a Python library that provides a fast numerical expression evaluator for NumPy. It accelerates array operations by avoiding memory allocation for intermediate results, leading to better cache utilization and reduced memory access. It also leverages multi-threading to utilize multiple CPU cores and supports Intel's Math Kernel Library (MKL) for further performance gains, especially with transcendental functions. The current version is 2.14.1, and it maintains a regular release cadence.

Warnings

gotcha Breaking down expressions into multiple NumPy operations that create intermediate arrays negates NumExpr's primary performance and memory benefits.
Fix: Always pass the entire complex expression as a single string to `numexpr.evaluate()` to allow the library to optimize the entire computation and avoid temporary memory allocations.
gotcha NumExpr's internal casting rules and type promotion might differ slightly from NumPy's in specific cases (e.g., `int8`/`uint8` upcasting to `int32`, `uint32` to `int64`, and `abs()` on complex numbers returning a complex result).
Fix: Be aware of potential type discrepancies for specific operations and explicitly cast arrays if exact NumPy-like type behavior is required. For example, use `real(abs(cplx))` for complex absolute values if a float result is expected.
gotcha Incorrectly configuring threadpool settings (e.g., `NUMEXPR_MAX_THREADS`, `NUMEXPR_NUM_THREADS`, or `OMP_NUM_THREADS` environment variables) can lead to oversubscription and degraded performance, especially when combined with other parallel processing libraries or Python's threading.
Fix: Set `NUMEXPR_MAX_THREADS` to a reasonable value, typically equal to or slightly less than the number of physical CPU cores. Adjust dynamic thread usage via `numexpr.set_num_threads()` if needed, and avoid oversubscribing when other parallel libraries are active.
gotcha NumExpr introduces parsing and compilation overhead. For small arrays or very simple expressions, native NumPy operations can be faster due to this overhead.
Fix: Reserve `numexpr` for large arrays (typically larger than your CPU's L1 cache) and complex multi-operator expressions where its optimization benefits outweigh the overhead. Benchmarking is recommended for critical sections.
breaking NumExpr 2.10.0 introduced *experimental* support for NumPy 2.0.0. While intended for forward compatibility, users adopting NumPy 2.x may encounter edge cases or instabilities.
Fix: Report any issues encountered when using NumExpr with NumPy 2.x to the NumExpr GitHub repository. Consider pinning NumPy to a 1.x version if stability is critical and NumPy 2.x features are not strictly required.

Install

pip install numexpr Install with pip

Imports

numexpr
```
import numexpr as ne
```
Commonly imported as 'ne' for brevity, similar to 'np' for NumPy.
evaluate
```
ne.evaluate('expression')
```
The `evaluate` function is a direct method of the `numexpr` module, not a submodule.

Quickstart

This quickstart demonstrates how to import `numexpr` and use its primary `evaluate` function. It takes a string expression, which is then parsed and efficiently computed on large NumPy arrays, leveraging NumExpr's optimizations and multi-threading. The example uses common mathematical and logical operations.

import numpy as np
import numexpr as ne

# Create large arrays for demonstrating performance benefits
a = np.arange(1_000_000, dtype=np.float64)
b = np.arange(1_000_000, 0, -1, dtype=np.float64)

# Evaluate a complex expression using numexpr
result_ne = ne.evaluate("sin(a) + arcsinh(a/b) + (a * b - 4.1 * a) > 2.5 * b")

print(result_ne)
print(f"Result type: {result_ne.dtype}")

# You can also set the number of threads dynamically
# print(f"Current numexpr threads: {ne.nthreads}")
# ne.set_num_threads(4)
# print(f"New numexpr threads: {ne.nthreads}")

view raw JSON →