pytest-xdist

Warnings

• gotcha The `-s` or `--capture=no` option for disabling pytest's output capture does not work with `pytest-xdist`. This means you will not see real-time stdout/stderr output from workers, and debugging with tools like `PDB` is generally not supported for tests running in parallel workers.
  Fix: For debugging, run tests without `-n` (i.e., sequentially). For inspecting output, rely on logging or `--capture=fd` (file descriptor capture) where available and aggregated at the end of the run.
• gotcha Tests using `pytest.mark.parametrize` with unordered iterables (e.g., `set`) can lead to inconsistent test collection order across workers, causing `pytest-xdist` to fail.
  Fix: Ensure parameterized values have a consistent order by converting them to lists or sorting them (e.g., `list({'a', 'b'})` or `sorted({'a', 'b'})`).
• deprecated The `--looponfail` and `--rsyncdir` command-line arguments and their corresponding configuration variables (`looponfailroots`, `rsyncdirs`) are deprecated. They are scheduled for removal in `pytest-xdist 4.0`.
  Fix: Migrate away from using `--looponfail` (which is already deprecated). For `--rsyncdir`, consider alternative methods for syncing code to remote environments, as modern CI/CD practices often handle this differently. The `--boxed` argument was also deprecated and removed in 3.0.0, use `pytest-forked` and `--forked` instead.
• gotcha pytest's `scope='session'` fixtures are executed once per *worker process*, not once for the entire `pytest-xdist` run across all workers. This can lead to unexpected resource setup/teardown behavior and state issues if not managed carefully.
  Fix: For fixtures that truly need to run once per entire test session across all workers, explicit inter-process communication or shared storage (e.g., temporary files) is required, often with a 'first worker initializes, others wait' pattern. Consider using plugins like `pytest-shared-session-scope`.
• breaking Users have reported that `--exitfirst` (`-x`) (which stops the test session on the first failure) does not function as expected with `pytest-xdist` versions 3.4.0 and higher. Tests might continue running in other workers despite a failure.
  Fix: If strict `--exitfirst` behavior is critical for your workflow, you might need to pin `pytest-xdist` to version `3.3.1` or earlier. Alternatively, check the latest `pytest-xdist` documentation or issue tracker for updates or workarounds regarding this behavior.

Install

• pip install pytest-xdist Standard installation
• pip install pytest-xdist[psutil] With psutil for logical CPU detection

Imports

• pytest-xdist

  pytest -n auto

  pytest-xdist is a pytest plugin and primarily exposes its functionality via command-line arguments to the 'pytest' executable. Direct Python imports of its core functionality by users are generally not required for standard usage.

Quickstart

After installation, `pytest-xdist` is typically used directly from the command line by adding the `-n` or `--numprocesses` option to your `pytest` command. The most common use is `pytest -n auto` to automatically detect and utilize all available physical CPU cores, or `pytest -n <NUM>` to specify a fixed number of worker processes. This example demonstrates how you would invoke it programmatically via `subprocess`, though direct command-line execution is the primary quickstart.

# Assuming you have pytest and pytest-xdist installed and tests in a 'tests/' directory
# Create a simple test file, e.g., tests/test_example.py
# import time
# def test_fast():
#     assert True
# def test_medium():
#     time.sleep(0.5)
#     assert True
# def test_slow():
#     time.sleep(1)
#     assert True

import os
import subprocess

def run_pytest_xdist(num_processes):
    cmd = ['pytest', '-n', str(num_processes)]
    print(f"Running: {' '.join(cmd)}")
    result = subprocess.run(cmd, capture_output=True, text=True)
    print(result.stdout)
    if result.stderr:
        print("STDERR:", result.stderr)
    return result.returncode

# Example usage: run tests on all available CPU cores
# A real-world scenario would just be `pytest -n auto` from the command line
if __name__ == '__main__':
    # Ensure pytest is installed for this example to run locally in a script
    try:
        import pytest
    except ImportError:
        print("pytest is not installed. Please install with 'pip install pytest'.")
        exit(1)

    print("\n--- Running tests with pytest -n auto ---")
    # In a real shell, you would simply type: pytest -n auto
    # For programmatic execution, we demonstrate the subprocess call
    exit_code = run_pytest_xdist('auto')
    if exit_code == 0:
        print("Tests passed successfully with -n auto.")
    else:
        print(f"Tests failed with -n auto. Exit code: {exit_code}")

    print("\n--- Running tests with pytest -n 2 ---")
    # In a real shell, you would simply type: pytest -n 2
    exit_code = run_pytest_xdist(2)
    if exit_code == 0:
        print("Tests passed successfully with -n 2.")
    else:
        print(f"Tests failed with -n 2. Exit code: {exit_code}")