{"id":8261,"library":"laboratory","title":"Laboratory: Sure-footed Refactoring with Experiments","description":"Laboratory is a Python library (version 1.0.2) that enables confident refactoring of critical code paths by running 'experiments' in production. Inspired by GitHub's Scientist, it executes new code (candidate) alongside existing code (control) in a randomized order, compares return values, records timing, and logs exceptions, providing a feedback loop for verification. The library is stable and addresses a timeless engineering problem, though its release cadence is slow.","status":"active","version":"1.0.2","language":"en","source_language":"en","source_url":"https://github.com/joealcorn/laboratory","tags":["refactoring","experimentation","a/b-testing","production-testing","scientist-pattern","observability"],"install":[{"cmd":"pip install laboratory","lang":"bash","label":"Install stable version"}],"dependencies":[],"imports":[{"symbol":"Experiment","correct":"from laboratory import Experiment"}],"quickstart":{"code":"import laboratory\nimport time\n\ndef old_function(value):\n    time.sleep(0.01)\n    return value * 2\n\ndef new_function(value):\n    # Simulate a refactored version, possibly faster or different\n    time.sleep(0.005)\n    return value + value\n\nexperiment = laboratory.Experiment()\nexperiment.control(old_function, args=(5,))\nexperiment.candidate(new_function, args=(5,))\n\n# Conduct the experiment. By default, it returns the control's value.\n# Candidate's return value, timing, and exceptions are recorded internally.\nresult = experiment.conduct()\n\nprint(f\"Experiment conducted. Control result: {result}\")\n\n# To access detailed observation data (requires a publisher to be configured)\n# For demonstration, we'll manually inspect the internal result, normally\n# this would be sent to a metrics/logging system.\n# For a real scenario, you'd typically subclass Experiment to implement\n# a publisher. Example below is illustrative of what data is collected.\n# print(f\"Control value: {experiment._observations[0].value}\") # Not a public API, for illustration\n# print(f\"Candidate value: {experiment._observations[1].value}\") # Not a public API, for illustration\n# print(f\"Mismatched: {experiment._result.mismatched}\") # Not a public API, for illustration\n","lang":"python","description":"This quickstart demonstrates how to set up and run a basic experiment. You define a 'control' function (your existing code) and a 'candidate' function (your new code). The `Experiment` class runs both, returning the control's result. Under the hood, it compares results, records performance, and logs exceptions from the candidate, which can then be published to a metrics system (not shown in this basic example)."},"warnings":[{"fix":"Initialize `Experiment(raise_on_mismatch=True)` to re-raise exceptions immediately for easier debugging in non-production environments. Implement robust reporting for candidate exceptions in production.","message":"Exceptions raised by the candidate function are caught and recorded, but by default, they do not halt the experiment or propagate. This allows for safe testing in production but means a buggy candidate won't immediately stop execution unless explicitly configured.","severity":"gotcha","affected_versions":"1.0.0+"},{"fix":"Subclass `laboratory.Experiment` and override the `publish()` method to send `Observation` data to your preferred metrics (e.g., StatsD) or logging system (e.g., Sentry, Prometheus).","message":"Laboratory only collects experiment data (return values, timing, exceptions, mismatches) internally. It does NOT automatically publish these results to any external monitoring or logging system. You must implement a custom publisher to act on the experiment data.","severity":"gotcha","affected_versions":"1.0.0+"},{"fix":"This is intended behavior for safe refactoring. If you need to access the candidate's return value, you would do so via your custom `publish` implementation, which receives the `Observation` objects.","message":"The `conduct()` method of an `Experiment` instance always returns the value produced by the 'control' function. It will never return the candidate's value, even if the candidate's result is deemed 'correct' or identical.","severity":"gotcha","affected_versions":"1.0.0+"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Ensure `experiment.control(your_function, args=(arg1, arg2))` and `experiment.candidate(another_function, args=(arg1, arg2))` are called with valid, callable functions and `args` are provided as a tuple or list.","cause":"This error often occurs if either the `control()` or `candidate()` function or their arguments are not correctly passed to the `Experiment` instance before `conduct()` is called.","error":"TypeError: 'NoneType' object is not callable"},{"fix":"To get immediate feedback, use `experiment = laboratory.Experiment(raise_on_mismatch=True)`. For production, implement a custom `publish` method in a subclass of `Experiment` to send mismatch reports to your alerting system (e.g., Slack, PagerDuty).","cause":"By default, `laboratory` reports mismatches internally but doesn't trigger alerts or re-raise exceptions. You need to explicitly handle how mismatches are reported.","error":"Mismatched results are occurring, but my application isn't alerting me."}]}