{"library":"pytest-benchmark","title":"pytest-benchmark","description":"pytest-benchmark is a plugin for pytest that provides a `benchmark` fixture for benchmarking code. It automatically groups tests into rounds calibrated to the chosen timer, offering sensible defaults and automatic calibration for micro-benchmarks. As of version 5.2.3, it is actively maintained with regular major releases approximately once a year and several minor/patch releases.","status":"active","version":"5.2.3","language":"en","source_language":"en","source_url":"https://github.com/ionelmc/pytest-benchmark","tags":["pytest","benchmarking","performance","testing","fixture"],"install":[{"cmd":"pip install pytest-benchmark","lang":"bash","label":"Install stable version"},{"cmd":"pip install 'pytest-benchmark[histogram]'","lang":"bash","label":"Install with histogram support"}],"dependencies":[{"reason":"Core testing framework dependency.","package":"pytest>=8.1","optional":false},{"reason":"Used for CPU information in benchmark reports.","package":"py-cpuinfo","optional":false},{"reason":"Optional, for generating histogram plots.","package":"pygal","optional":true},{"reason":"Optional, for generating histogram plots.","package":"pygaljs","optional":true},{"reason":"Optional, for storing benchmark results in Elasticsearch.","package":"elasticsearch","optional":true},{"reason":"Optional, for aspect mode functionality.","package":"aspectlib","optional":true}],"imports":[{"note":"The 'benchmark' fixture is automatically provided by pytest-benchmark to test functions; no direct import statement for 'pytest_benchmark' is typically required within test modules.","symbol":"benchmark","correct":"def test_my_function(benchmark):\n    # benchmark is a pytest fixture, no explicit import needed"}],"quickstart":{"code":"import time\n\ndef something(duration=0.000001):\n    \"\"\" Function that needs some serious benchmarking. \"\"\"\n    time.sleep(duration)\n    return 123\n\ndef test_my_stuff(benchmark):\n    # benchmark something\n    result = benchmark(something)\n    assert result == 123\n\n# To run: save the above as a Python file (e.g., test_benchmarks.py) and run `pytest` from your terminal.\n# Use `pytest --benchmark-autosave --benchmark-json=report.json` for advanced features.","lang":"python","description":"Define a test function that accepts the `benchmark` fixture. Pass the function you wish to benchmark as the first argument to `benchmark()`. You can also pass positional and keyword arguments to the benchmarked function. Running `pytest` will execute the benchmarks and display a summary table."},"warnings":[{"fix":"Upgrade to Python 3.9+ or pin pytest-benchmark to <5.0.0.","message":"Dropped support for Python 3.8 in version 5.0.0. Projects using Python 3.8 or older must remain on an earlier version of pytest-benchmark.","severity":"breaking","affected_versions":">=5.0.0"},{"fix":"Upgrade pytest to version 8.1 or newer, or pin pytest-benchmark to <5.1.0.","message":"Version 5.1.0 and later require pytest version 8.1 or newer due to internal hook handling changes. Older pytest versions will lead to errors.","severity":"breaking","affected_versions":">=5.1.0"},{"fix":"Prefer `benchmark(my_function, *args, **kwargs)` over `benchmark(lambda: my_function(*args, **kwargs))` for better accuracy, especially with very fast functions.","message":"Benchmarking internal functions or wrapping code in a `@benchmark` decorator within the test function (e.g., `benchmark(lambda: my_func())`) can introduce unnecessary function call overhead, leading to less accurate micro-benchmarks. Pass the function reference directly instead.","severity":"gotcha","affected_versions":"<all>"},{"fix":"Ensure a stable, isolated environment. Minimize I/O and external dependencies in benchmarked code. Use `--benchmark-warmup` and `--benchmark-warmup-iterations` for JIT-heavy interpreters like PyPy. Consider `median` or `IQR` over `mean`/`stddev` for outlier-prone results.","message":"High standard deviation in benchmark results (flaky benchmarks) often indicates external factors (e.g., other running services, VMs, CPU turbo boost) or non-deterministic test functions (I/O, external resources, side-effects). PyPy's GC and JIT can also add overhead, requiring sufficient warmup.","severity":"gotcha","affected_versions":"<all>"},{"fix":"When using a `setup` function with `benchmark.pedantic`, have the `setup` function return the arguments tuple/dict, and do not pass `args`, `kwargs`, or `iterations` separately to `pedantic`.","message":"In 'pedantic' mode, if a `setup` function is provided to `benchmark.pedantic()`, you cannot simultaneously use the `args`, `kwargs`, or `iterations` parameters. The setup function is expected to return the arguments for the target function if needed.","severity":"gotcha","affected_versions":"<all>"},{"fix":"Create separate test functions for each distinct piece of code you want to benchmark. Use `pytest.mark.parametrize` for benchmarking the same function with different inputs.","message":"Using the `benchmark` fixture more than once within a single test function is disallowed and will raise an exception (since v3.0.0). Each benchmarkable unit should be in its own test function.","severity":"gotcha","affected_versions":">=3.0.0"}],"env_vars":null,"last_verified":"2026-04-05T00:00:00.000Z","next_check":"2026-07-04T00:00:00.000Z"}