Laboratory: Sure-footed Refactoring with Experiments

1.0.2 · active · verified Thu Apr 16

Laboratory is a Python library (version 1.0.2) that enables confident refactoring of critical code paths by running 'experiments' in production. Inspired by GitHub's Scientist, it executes new code (candidate) alongside existing code (control) in a randomized order, compares return values, records timing, and logs exceptions, providing a feedback loop for verification. The library is stable and addresses a timeless engineering problem, though its release cadence is slow.

Common errors

```
TypeError: 'NoneType' object is not callable
```
cause This error often occurs if either the `control()` or `candidate()` function or their arguments are not correctly passed to the `Experiment` instance before `conduct()` is called.

fix Ensure `experiment.control(your_function, args=(arg1, arg2))` and `experiment.candidate(another_function, args=(arg1, arg2))` are called with valid, callable functions and `args` are provided as a tuple or list.
```
Mismatched results are occurring, but my application isn't alerting me.
```
cause By default, `laboratory` reports mismatches internally but doesn't trigger alerts or re-raise exceptions. You need to explicitly handle how mismatches are reported.

fix To get immediate feedback, use `experiment = laboratory.Experiment(raise_on_mismatch=True)`. For production, implement a custom `publish` method in a subclass of `Experiment` to send mismatch reports to your alerting system (e.g., Slack, PagerDuty).

Warnings

gotcha Exceptions raised by the candidate function are caught and recorded, but by default, they do not halt the experiment or propagate. This allows for safe testing in production but means a buggy candidate won't immediately stop execution unless explicitly configured.
Fix: Initialize `Experiment(raise_on_mismatch=True)` to re-raise exceptions immediately for easier debugging in non-production environments. Implement robust reporting for candidate exceptions in production.
gotcha Laboratory only collects experiment data (return values, timing, exceptions, mismatches) internally. It does NOT automatically publish these results to any external monitoring or logging system. You must implement a custom publisher to act on the experiment data.
Fix: Subclass `laboratory.Experiment` and override the `publish()` method to send `Observation` data to your preferred metrics (e.g., StatsD) or logging system (e.g., Sentry, Prometheus).
gotcha The `conduct()` method of an `Experiment` instance always returns the value produced by the 'control' function. It will never return the candidate's value, even if the candidate's result is deemed 'correct' or identical.
Fix: This is intended behavior for safe refactoring. If you need to access the candidate's return value, you would do so via your custom `publish` implementation, which receives the `Observation` objects.

Install

pip install laboratory Install stable version

Imports

Experiment
```
from laboratory import Experiment
```

Quickstart

This quickstart demonstrates how to set up and run a basic experiment. You define a 'control' function (your existing code) and a 'candidate' function (your new code). The `Experiment` class runs both, returning the control's result. Under the hood, it compares results, records performance, and logs exceptions from the candidate, which can then be published to a metrics system (not shown in this basic example).

import laboratory
import time

def old_function(value):
    time.sleep(0.01)
    return value * 2

def new_function(value):
    # Simulate a refactored version, possibly faster or different
    time.sleep(0.005)
    return value + value

experiment = laboratory.Experiment()
experiment.control(old_function, args=(5,))
experiment.candidate(new_function, args=(5,))

# Conduct the experiment. By default, it returns the control's value.
# Candidate's return value, timing, and exceptions are recorded internally.
result = experiment.conduct()

print(f"Experiment conducted. Control result: {result}")

# To access detailed observation data (requires a publisher to be configured)
# For demonstration, we'll manually inspect the internal result, normally
# this would be sent to a metrics/logging system.
# For a real scenario, you'd typically subclass Experiment to implement
# a publisher. Example below is illustrative of what data is collected.
# print(f"Control value: {experiment._observations[0].value}") # Not a public API, for illustration
# print(f"Candidate value: {experiment._observations[1].value}") # Not a public API, for illustration
# print(f"Mismatched: {experiment._result.mismatched}") # Not a public API, for illustration

view raw JSON →