Flaky

3.8.1 · active · verified Thu Apr 09

Flaky is a plugin for pytest that automatically reruns flaky tests. It helps improve test reliability by providing mechanisms to retry tests that exhibit intermittent or sporadic failures. Currently at version 3.8.1, the library maintains an active development pace with regular minor and patch releases.

Warnings

breaking The 'flaky' library's `@flaky` decorator can conflict with `@pytest.mark.flaky` provided by the 'pytest-rerunfailures' plugin if both are installed. It's recommended to choose one plugin for rerunning tests.
Fix: Decide which plugin best suits your needs and remove the other. If you must use `flaky`, ensure you import `from flaky import flaky` and use `@flaky`.
gotcha Using the `--force-flaky` command-line option will apply flaky behavior (default `max_runs=1`, `min_passes=1`, implying 1 retry if not specified otherwise by `--max-runs` and `--min-passes`) to *all* tests in the run, overriding their deterministic nature. While useful for debugging, this can mask genuine failures.
Fix: Use `--force-flaky` judiciously and be aware of its global impact. Explicitly mark individual tests with `@flaky(...)` for fine-grained control.
gotcha The 'flaky' plugin is compatible with `pytest-xdist` but explicitly does *not* work with `pytest-xdist`'s `--boxed` option. Using `--boxed` will result in unexpected behavior or errors.
Fix: Avoid using the `--boxed` option when running tests with `flaky` and `pytest-xdist`.
gotcha Doctests cannot be marked as flaky using this plugin. The `@flaky` decorator is intended for standard pytest functions.
Fix: Do not attempt to apply the `@flaky` decorator to doctests. Consider converting doctests to standard pytest functions if flaky behavior is required.
gotcha The `max_runs` and `min_passes` parameters for the `@flaky` decorator control the retry logic. `max_runs` is the total number of attempts (initial run + retries), and `min_passes` is the number of successful attempts required. Misunderstanding these can lead to tests passing after fewer or more retries than intended.
Fix: Always explicitly define `max_runs` and `min_passes` to match your desired retry behavior (e.g., `max_runs=3, min_passes=1` for up to 2 retries after 1 failure). Default behavior for `@flaky()` without arguments is generally one retry.

Install

pip install flaky Install stable version

Imports

flaky
```
from flaky import flaky
```
This imports the main decorator provided by the 'flaky' library.

Quickstart

This quickstart demonstrates how to mark a pytest function as flaky using the `@flaky` decorator. The test will automatically be re-run up to a specified number of times (`max_runs`) until it achieves a minimum number of passes (`min_passes`).

import pytest
from flaky import flaky
import random

@flaky(max_runs=3, min_passes=1)
def test_something_that_sometimes_fails():
    # Simulate a flaky condition that might pass on retry
    if random.random() < 0.6: # 60% chance of passing
        assert True
    else:
        assert False

# To run this test, save it as a Python file (e.g., test_flaky_example.py)
# and execute from your terminal: pytest test_flaky_example.py
# The test will be run up to 3 times (max_runs) until it passes once (min_passes).

view raw JSON →