pytest-benchmark

5.2.3 · active · verified Sun Apr 05

pytest-benchmark is a plugin for pytest that provides a `benchmark` fixture for benchmarking code. It automatically groups tests into rounds calibrated to the chosen timer, offering sensible defaults and automatic calibration for micro-benchmarks. As of version 5.2.3, it is actively maintained with regular major releases approximately once a year and several minor/patch releases.

Warnings

breaking Dropped support for Python 3.8 in version 5.0.0. Projects using Python 3.8 or older must remain on an earlier version of pytest-benchmark.
Fix: Upgrade to Python 3.9+ or pin pytest-benchmark to <5.0.0.
breaking Version 5.1.0 and later require pytest version 8.1 or newer due to internal hook handling changes. Older pytest versions will lead to errors.
Fix: Upgrade pytest to version 8.1 or newer, or pin pytest-benchmark to <5.1.0.
gotcha Benchmarking internal functions or wrapping code in a `@benchmark` decorator within the test function (e.g., `benchmark(lambda: my_func())`) can introduce unnecessary function call overhead, leading to less accurate micro-benchmarks. Pass the function reference directly instead.
Fix: Prefer `benchmark(my_function, *args, **kwargs)` over `benchmark(lambda: my_function(*args, **kwargs))` for better accuracy, especially with very fast functions.
gotcha High standard deviation in benchmark results (flaky benchmarks) often indicates external factors (e.g., other running services, VMs, CPU turbo boost) or non-deterministic test functions (I/O, external resources, side-effects). PyPy's GC and JIT can also add overhead, requiring sufficient warmup.
Fix: Ensure a stable, isolated environment. Minimize I/O and external dependencies in benchmarked code. Use `--benchmark-warmup` and `--benchmark-warmup-iterations` for JIT-heavy interpreters like PyPy. Consider `median` or `IQR` over `mean`/`stddev` for outlier-prone results.
gotcha In 'pedantic' mode, if a `setup` function is provided to `benchmark.pedantic()`, you cannot simultaneously use the `args`, `kwargs`, or `iterations` parameters. The setup function is expected to return the arguments for the target function if needed.
Fix: When using a `setup` function with `benchmark.pedantic`, have the `setup` function return the arguments tuple/dict, and do not pass `args`, `kwargs`, or `iterations` separately to `pedantic`.
gotcha Using the `benchmark` fixture more than once within a single test function is disallowed and will raise an exception (since v3.0.0). Each benchmarkable unit should be in its own test function.
Fix: Create separate test functions for each distinct piece of code you want to benchmark. Use `pytest.mark.parametrize` for benchmarking the same function with different inputs.

Install

pip install pytest-benchmark Install stable version
pip install 'pytest-benchmark[histogram]' Install with histogram support

Imports

benchmark
```
def test_my_function(benchmark):
    # benchmark is a pytest fixture, no explicit import needed
```
The 'benchmark' fixture is automatically provided by pytest-benchmark to test functions; no direct import statement for 'pytest_benchmark' is typically required within test modules.

Quickstart

Define a test function that accepts the `benchmark` fixture. Pass the function you wish to benchmark as the first argument to `benchmark()`. You can also pass positional and keyword arguments to the benchmarked function. Running `pytest` will execute the benchmarks and display a summary table.

import time

def something(duration=0.000001):
    """ Function that needs some serious benchmarking. """
    time.sleep(duration)
    return 123

def test_my_stuff(benchmark):
    # benchmark something
    result = benchmark(something)
    assert result == 123

# To run: save the above as a Python file (e.g., test_benchmarks.py) and run `pytest` from your terminal.
# Use `pytest --benchmark-autosave --benchmark-json=report.json` for advanced features.

view raw JSON →