Pytest Runtime Xfail

1.1.1 · active · verified Fri Apr 17

pytest-runtime-xfail is a pytest plugin that provides a `runtime_xfail` fixture, enabling tests to be marked as 'xfail' (expected to fail) dynamically during test execution. This is particularly useful for tests that might fail under specific, conditional circumstances (e.g., different operating systems, library versions, or data states) that cannot be determined at test collection time. The current version is 1.1.1, and its release cadence is generally low, with updates as needed for bug fixes or compatibility.

Common errors

```
NameError: name 'runtime_xfail' is not defined
```
cause Attempting to use `runtime_xfail` without requesting it as a fixture in the test function signature.

fix Add `runtime_xfail` as an argument to your test function: `def test_my_feature(runtime_xfail):`
```
TypeError: runtime_xfail() missing 1 required positional argument: 'reason'
```
cause Calling `runtime_xfail()` without providing a reason string.

fix Provide a descriptive string for why the test is being xfailed: `runtime_xfail('Specific condition not met')`
```
Test still fails instead of being marked as XFAIL
```
cause The `runtime_xfail()` call was placed after an assertion that failed, or the condition for calling `runtime_xfail` was not met when an assertion failed.

fix Ensure `runtime_xfail()` is called *before* any assertion that it is intended to mark as xfail. Verify your conditional logic correctly triggers `runtime_xfail` when the test is expected to fail.

Warnings

gotcha Calling `runtime_xfail()` without a reason string will raise a TypeError.
Fix: Always provide a string argument to `runtime_xfail()` explaining why the test is being marked as xfail, e.g., `runtime_xfail('Known issue with X on Y platform')`.
gotcha If an assertion fails *before* `runtime_xfail()` is called within a test, the test will be marked as FAILED, not XFAIL. `runtime_xfail` needs to be called proactively.
Fix: Place the `runtime_xfail()` call at the beginning of the conditional block, before any code that might cause the test to fail if the condition is met. Ensure `runtime_xfail` is called *before* any potentially failing assertions it is meant to 'cover'.
gotcha Mixing `pytest-runtime-xfail` with the static `@pytest.mark.xfail` can be confusing. While `runtime_xfail` takes precedence if called, using `@pytest.mark.xfail(run=False)` will prevent the test body (and thus `runtime_xfail`) from executing at all.
Fix: Prefer `runtime_xfail` for conditions determined only at runtime. For conditions known at collection time (e.g., specific Python version or installed library), `@pytest.mark.xfail` or `@pytest.mark.skipif` are often more appropriate. Understand the interaction to avoid unexpected test outcomes.

Install

pip install pytest-runtime-xfail Install stable version

Imports

runtime_xfail
wrong:
```
from pytest_runtime_xfail import runtime_xfail
```
correct:
```
def test_example(runtime_xfail):
```
`runtime_xfail` is provided as a pytest fixture; it should be requested directly in your test function signature, not imported from the package.

Quickstart

This quickstart demonstrates how to use the `runtime_xfail` fixture within a test function. The test requests `runtime_xfail` as an argument, and then calls it conditionally. If `runtime_xfail` is called, the test will be marked as 'XFAIL' (expected to fail) with the provided reason, even if subsequent assertions fail.

import platform

def test_feature_on_linux_only(runtime_xfail):
    """This test expects to pass on Linux, fail on others."""
    if platform.system() != "Linux":
        runtime_xfail(f"This feature is not supported on {platform.system()}")
    
    # Simulate a test that would fail on non-Linux systems
    assert platform.system() == "Linux", "Expected to be on Linux"

def test_conditional_bug(runtime_xfail):
    """This test demonstrates runtime xfail for a known bug under a condition."""
    # Imagine some condition based on a runtime value or environment
    is_bug_present = True # This would be dynamically determined

    if is_bug_present:
        runtime_xfail("Known bug #123 causing failure under this condition")
    
    # Simulate a failing assertion if the bug is present
    assert False, "This assertion will fail if bug is present and not xfailed"

view raw JSON →