pytest-harvest

1.10.5 · active · verified Thu Apr 16

pytest-harvest is a pytest plugin that allows you to store data created during your pytest tests execution and retrieve it at the end of the session. This is particularly useful for applicative benchmarking purposes, collecting custom metrics, or generating reports. The current version is 1.10.5, and it maintains a regular release cadence, often to ensure compatibility with new pytest versions and address bug fixes.

Common errors

```
AttributeError: 'str' object has no attribute 'iter_parents'
```
cause Incompatibility between older `pytest-harvest` versions and `pytest>=8.1` due to internal `pathlib` API changes.

fix Upgrade `pytest-harvest` to version `1.10.5` or higher: `pip install --upgrade pytest-harvest`.
```
ModuleNotFoundError: No module named 'pandas'
```
cause Attempting to use `module_results_df` or `session_results_df` fixtures when `pandas` is not installed.

fix Install the `pandas` library: `pip install pandas`.
```
pytest-xdist: results not consolidated or missing across workers/master node
```
cause Issues with `pytest-xdist` hook execution or data transfer, especially in older `pytest-harvest` versions or specific Python/xdist combinations.

fix Ensure `pytest-harvest>=1.10.4` is installed. Verify `pytest-xdist` is also up-to-date and compatible with your `pytest` version. Check `pytest_sessionfinish` hook for data retrieval.
```
KeyError: 'custom_data_field' or attribute error when accessing collected data
```
cause Incorrectly accessing collected data via `request.node.add_pytest_stats` or `request.node.add_report_section` where the key/attribute name does not match retrieval logic in `get_session_synthesis_dct` or result fixtures.

fix When using `request.node.add_pytest_stats(key=value)`, data is accessible as `result_object.key`. For `request.node.add_report_section('call', 'category', value)`, access might involve parsing `result_object.sections`.

Warnings

breaking pytest-harvest versions prior to 1.10.5 are incompatible with pytest>=8.1 due to changes in pytest internals, leading to `AttributeError` or signature mismatches.
Fix: Upgrade to pytest-harvest>=1.10.5 to ensure compatibility with pytest 8.1 and later: `pip install --upgrade pytest-harvest`.
gotcha Using result fixtures like `module_results_df` or `session_results_df` without `pandas` installed will raise an `ImportError`. `pytest-harvest` uses lazy loading for `pandas`.
Fix: Explicitly install pandas if you intend to use DataFrame result fixtures: `pip install pandas`.
gotcha `pytest-xdist` integration, especially with older `pytest` versions or `python 3.5`, had known issues with result consolidation or `pathlib` object handling.
Fix: Ensure you are using `pytest-harvest>=1.10.4` and the latest compatible `pytest-xdist` version for robust distributed testing result collection.
gotcha Test IDs containing `::` were not properly handled by `get_session_synthesis_dct` and related fixtures, leading to incomplete or incorrect results.
Fix: Upgrade to `pytest-harvest>=1.10.1` to correctly parse test IDs with `::` delimiters.
gotcha When using `pytest-cases` with `lazy_value` parameters, `module_results_df` in `pytest < 5.3` might wrongly insert values as integers instead of objects.
Fix: Upgrade `pytest-harvest` to `>=1.10.0` and `pytest-cases` to `>=2.3.0` to resolve this specific interaction issue.

Install

pip install pytest-harvest Install pytest-harvest

Imports

get_session_synthesis_dct
```
from pytest_harvest import get_session_synthesis_dct
```
Used to retrieve all collected results from the session at the end of the test run, typically in a `pytest_sessionfinish` hook.
module_results_dct
```
def test_something(module_results_dct):
```
This is a pytest fixture provided by `pytest-harvest` to store and access results within a test module context. It's injected, not explicitly imported.
session_results_df
```
def test_something(session_results_df):
```
This is a pytest fixture provided by `pytest-harvest` to store and access results as a pandas DataFrame within a test session context. Requires `pandas` to be installed. It's injected, not explicitly imported.
saved_shared_data
```
def test_something(saved_shared_data):
```
A pytest fixture for easily sharing data across tests within a session. It's injected, not explicitly imported.

Quickstart

This quickstart demonstrates how to define a custom metric using a fixture, store its value during test execution with `request.node.add_pytest_stats`, and then retrieve all collected data at the end of the session using `get_session_synthesis_dct` within a `pytest_sessionfinish` hook. Run `pytest -s` from your terminal to see the results printed.

import pytest
from pytest_harvest import get_session_synthesis_dct

# conftest.py
@pytest.fixture
def custom_metric_value():
    return 'important_data'

def pytest_sessionfinish(session):
    print("\n--- Session Synthesis Results ---")
    results = get_session_synthesis_dct(session)
    for test_id, res in results.items():
        custom = getattr(res, 'custom_metric_value', 'N/A')
        print(f"Test: {test_id}, Outcome: {res.outcome}, Custom: {custom}")
    print("---------------------------------")

# test_example.py
def test_success(request, custom_metric_value):
    request.node.add_pytest_stats(custom_metric_value=custom_metric_value)
    assert True

def test_failure(request, custom_metric_value):
    request.node.add_pytest_stats(custom_metric_value=custom_metric_value + '_failed')
    pytest.fail("Intentionally failed test")

# To run these tests and see the output:
# 1. Save the conftest.py content as 'conftest.py' in your test directory.
# 2. Save the test_example.py content as 'test_example.py' in the same directory.
# 3. Run from your terminal: pytest -s

view raw JSON →