pytest-codspeed

Warnings

• breaking Upgrading to v4.0.0 introduced `CodSpeedHQ/instrument-hooks`. This may cause slight performance changes in tiny microbenchmarks compared to previous versions due to changes in how instrument states are controlled.
  Fix: Review existing benchmarks after upgrading and adjust any baselines if slight performance shifts are observed, particularly for very small, fast functions.
• gotcha Running benchmarks locally with the `--codspeed` flag will display results in the console but will *not* produce performance reports or integrate with the CodSpeed dashboard. For full performance reporting, regression tracking, and consistent measurements, benchmarks must be run in a CI/CD environment using the official CodSpeed runner (e.g., GitHub Action).
  Fix: Integrate CodSpeedHQ/action into your CI/CD workflow to enable performance reporting and historical tracking. Local runs are primarily for functional verification of benchmarks.
• gotcha When using the `walltime` instrument (enabled via `--codspeed-mode walltime`), avoid running multiple benchmark processes in parallel (e.g., with `pytest-xdist`). Parallel execution can lead to noisy and inconsistent measurements. For optimal consistency in walltime mode, use isolated machines like CodSpeed Hosted Macro Runners.
  Fix: Configure your CI environment to run benchmarks in a single process when using `walltime` mode, or use CodSpeed's dedicated macro runners if available.
• gotcha Python 3.9 or newer is required. Older Python versions are not supported.
  Fix: Ensure your project's Python environment is 3.9 or later. Upgrade Python if necessary.
• gotcha Benchmarks are still tests: always include assertions in your benchmark functions to verify that the code being measured is producing the expected output. This ensures that you're not just measuring speed, but the speed of correct functionality.
  Fix: Add `assert` statements to your benchmark functions to validate their behavior, just like any other unit test.

Install

• pip install pytest-codspeed Install with pip

Imports

• pytest.mark.benchmark

  import pytest
  
  @pytest.mark.benchmark
  def test_my_function(): ...

  Used as a decorator to mark an entire test function for benchmarking.
• benchmark fixture

  def test_my_function(benchmark):
      result = benchmark(lambda: my_code_to_measure())
      assert result is not None

  The `benchmark` fixture is automatically provided by pytest-codspeed for fine-grained control over which parts of a test function are measured.

Quickstart

This example demonstrates two common ways to create benchmarks with pytest-codspeed: using the `@pytest.mark.benchmark` decorator to measure an entire test function, and using the `benchmark` fixture for precise control over the code block to be measured within a test. Benchmarks can be run locally using `pytest --codspeed`.

import pytest

def sum_squares(arr):
    """Sum the squares of the numbers in an array."""
    total = 0
    for x in arr:
        total += x * x
    return total

@pytest.mark.benchmark
def test_sum_squares_full_function():
    """Benchmark an entire test function using the decorator."""
    # This entire function's execution will be measured
    assert sum_squares(range(1000)) == 332833500

def mean(data):
    return sum(data) / len(data)

def test_mean_performance_fine_grained(benchmark):
    """Benchmark a specific part of a test function using the fixture."""
    data = list(range(1_000_000)) # Setup not measured
    
    # Only the lambda function's execution will be measured
    result = benchmark(lambda: mean(data))
    assert result == 499999.5

# To run locally: pytest your_test_file.py --codspeed