LangWatch Scenario

0.7.26 verified Fri May 01 auth: no python

LangWatch Scenario is an end-to-end agent testing library for Python (>=3.10) that lets you define, run, and evaluate LLM agent scenarios. It supports async execution, built-in judges, red-teaming, and integration with LangWatch observability. Current version: 0.7.26, released 2026-04-28. Active development with frequent releases.

pip install langwatch-scenario

Common errors

error ModuleNotFoundError: No module named 'langwatch.scenario' ↓

cause Incorrect import path; the package is langwatch_scenario, not langwatch.scenario.

fix

pip install langwatch-scenario and use from langwatch_scenario import ScenarioRunner

error RuntimeWarning: coroutine 'ScenarioRunner.run' was never awaited ↓

cause Calling async run() without await in a non-async context.

fix

Wrap in asyncio.run(): result = asyncio.run(runner.run('scenario-name'))

error langwatch_scenario.exceptions.AuthenticationError: Invalid API key ↓

cause Missing or incorrect LANGWATCH_API_KEY environment variable or api_key parameter.

fix

Set LANGWATCH_API_KEY environment variable or pass api_key='your_key' to ScenarioRunner.

Warnings

gotcha The package import uses underscore: from langwatch_scenario import ..., not from langwatch.scenario. ↓

fix Use underscore: pip install langwatch-scenario, import langwatch_scenario.

gotcha Async functions are required: run() and arun() are async. Calling them without await leads to RuntimeWarning. ↓

fix Use asyncio.run() in scripts or await inside async functions.

gotcha Relicensed from AGPLv3 to Apache 2.0 in v0.7.25 – ensure your project's license is compatible. ↓

fix No code changes needed, but update license notices in your project if you were complying with AGPL terms.

Imports

ScenarioRunner
wrong
```
from langwatch.scenario import ScenarioRunner
```
correct
```
from langwatch_scenario import ScenarioRunner
```
Package name uses underscore, not dot; wrong import causes ModuleNotFoundError.

EventReporter

wrong

from langwatch_scenario.reporter import EventReporter

correct

from langwatch_scenario import EventReporter

EventReporter is exposed at top-level, not in a submodule.

Quickstart

Create a ScenarioRunner with your API key and run a predefined scenario asynchronously.

from langwatch_scenario import ScenarioRunner
import asyncio

async def main():
    runner = ScenarioRunner(api_key=os.environ.get('LANGWATCH_API_KEY', ''))
    result = await runner.run('example-scenario')
    print(result)

if __name__ == '__main__':
    asyncio.run(main())