pytest-flakefinder

1.1.0 · active · verified Sat Apr 11

pytest-flakefinder is a plugin for the pytest testing framework that helps identify flaky tests by running them multiple times within a single pytest invocation. The current version, 1.1.0, was last released in October 2022 and is maintained by Dropbox, offering features to configure the number of runs and set a maximum execution time.

Warnings

gotcha Running tests many times with `--flake-runs` without parallelization (e.g., using `pytest-xdist`) can significantly increase execution time.
Fix: For larger test suites or higher flake counts, install `pytest-xdist` (`pip install pytest-xdist`) and run pytest with `-n auto` (e.g., `pytest -n auto --flake-finder`).
gotcha The `--flake-max-minutes` option will only skip tests that *start* after the timeout has been reached. It does not interrupt currently running tests.
Fix: Account for individual test run times when setting `--flake-max-minutes`. For strict test timeouts, consider `pytest-timeout` in conjunction with `pytest-flakefinder`.
gotcha pytest-flakefinder is designed to *find* flakiness, not to *fix* or *mitigate* it during regular CI runs. Its purpose is diagnostic.
Fix: Once flakes are identified, investigate and fix the underlying causes (e.g., race conditions, external dependencies, state pollution). For automatic retries in CI, plugins like `pytest-rerunfailures` are often used instead.

Install

pip install pytest-flakefinder Install core library

Quickstart

After installation, pytest-flakefinder integrates directly with the `pytest` command-line interface. You enable it with `--flake-finder` and can customize the number of repetitions with `--flake-runs` or set a time limit with `--flake-max-minutes`. Place the example code in a file named `test_example.py` and run the `pytest` commands from your terminal.

import random

def test_might_be_flaky():
    # This test has a 30% chance of failing to simulate flakiness
    if random.random() < 0.3:
        assert False, "This test failed flakily!"
    else:
        assert True

# To run all tests 100 times to find flakes:
# pytest --flake-finder --flake-runs=100

# To run tests with a maximum time limit of 5 minutes:
# pytest --flake-finder --flake-max-minutes=5

# Combine with pytest-xdist for parallel execution (requires prior installation of pytest-xdist):
# pytest -n auto --flake-finder --flake-runs=100

view raw JSON →