nbval

0.11.0 · active · verified Mon Apr 13

nbval is a pytest plugin designed to validate Jupyter notebooks. It allows users to execute notebooks as part of their test suite and compare cell outputs against previously saved outputs, ensuring reproducibility and correctness. The current version is 0.11.0, and it has a sporadic but active release cadence, typically with minor releases every 6-12 months.

Warnings

breaking The `--nbval-sanitize` command-line option was removed in version 0.10.0. This option was previously used to sanitize notebook outputs before comparison but was deprecated in earlier versions.
Fix: Remove `--nbval-sanitize` from your pytest command-line arguments or `pytest.ini` configuration. If sanitation is still required, consider pre-processing notebooks or using `nbval`'s cell metadata options for more granular control over output comparison (e.g., `nbval-ignore_data_priority`).
breaking Support for Python 3.6 was dropped in version 0.9.0. Users on Python 3.6 will not be able to install or run `nbval` versions 0.9.0 or newer.
Fix: Upgrade your Python environment to Python 3.7 or newer. The current minimum requirement is Python 3.7.
gotcha By default, `nbval` performs a strict diff on notebook outputs. This can lead to very verbose output in case of failures, especially with slight differences in timestamps, floating-point precision, or minor formatting changes. This behavior is controlled by the `nbval_diff` option.
Fix: To get a simpler pass/fail indication without detailed diffs, you can set `nbval_diff=False` in your `pytest.ini` or use `--no-nbval-diff` on the command line. For more controlled output validation, explore options like `--nbval-lax` or specific cell metadata for ignoring certain outputs.
gotcha nbval's primary function is *output validation*, not just execution and failure on any runtime error. It compares the newly generated outputs against the stored outputs in the notebook. If the notebook is expected to raise an exception or has no outputs for a cell, this needs to be explicitly indicated using cell metadata (e.g., `raises-exception`, `skip-execution`).
Fix: Ensure you understand how `nbval` handles outputs. For cells that are expected to fail, add `"raises-exception": true` to their cell metadata. For cells with no relevant output or to skip execution, use `"skip-execution": true` or `"skip-compare": true`. Use `--nbval-lax` for more permissive output comparison, or `--nbval-errors` to fail on any executed code error regardless of output comparison.

Install

pip install nbval pytest Install nbval and pytest

Imports

nbval functionality

nbval is primarily used via the pytest CLI (e.g., `pytest --nbval`) and typically does not require direct Python imports for its core validation features. pytest automatically discovers and loads the plugin upon installation.

As a pytest plugin, direct imports from `nbval` are generally not needed for basic usage. Configuration is primarily done via pytest command-line options or `pytest.ini`.

Quickstart

This quickstart script demonstrates how to programmatically create a simple Jupyter notebook and then run `pytest` with the `nbval` plugin to validate its output. It shows the minimal setup required to get started with notebook testing using `nbval`.

import pytest
import json
import os

# 1. Create a dummy Jupyter notebook file for testing
notebook_content = {
    "cells": [
        {
            "cell_type": "code",
            "execution_count": 1,
            "metadata": {},
            "outputs": [
                {"name": "stdout", "output_type": "stream", "text": ["Hello from nbval!\n"]}
            ],
            "source": ["print('Hello from nbval!')"]
        }
    ],
    "metadata": {
        "kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},
        "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7"}
    },
    "nbformat": 4,
    "nbformat_minor": 5
}

notebook_filename = "example.ipynb"
with open(notebook_filename, "w") as f:
    json.dump(notebook_content, f, indent=4)

print(f"Created {notebook_filename}. Running nbval tests...")

# 2. Run pytest programmatically with nbval
#    The --nbval flag enables the plugin, and the notebook filename specifies the target.
exit_code = pytest.main(["--nbval", notebook_filename])

# 3. Report results and clean up
# os.remove(notebook_filename) # Uncomment to clean up the notebook file after run

if exit_code == 0:
    print(f"\nnbval tests passed for {notebook_filename}!")
else:
    print(f"\nnbval tests failed for {notebook_filename} with exit code {exit_code}.")

view raw JSON →