{"id":9634,"library":"cwltest","title":"Common Workflow Language Testing Framework (cwltest)","description":"cwltest is the Common Workflow Language (CWL) testing framework. It provides tools to validate CWL workflows against defined test cases. As of its latest version 2.6.20251216093331, it supports Python versions 3.10 to 3.14. The library has a rapid release cadence, often with multiple releases per month, using a date-based versioning scheme.","status":"active","version":"2.6.20251216093331","language":"en","source_language":"en","source_url":"https://github.com/common-workflow-language/cwltest","tags":["workflow","testing","cwl","common workflow language","pytest","bioinformatics"],"install":[{"cmd":"pip install cwltest","lang":"bash","label":"Install cwltest"},{"cmd":"pip install cwltest cwltool","lang":"bash","label":"Install cwltest with default CWL runner"}],"dependencies":[{"reason":"Requires Python >=3.10 and <3.15.","package":"python","optional":false},{"reason":"Often used as a pytest plugin for running CWL tests. Version compatibility is frequently updated.","package":"pytest","optional":true},{"reason":"The default CWL reference runner used by cwltest to execute workflows during testing. While not a direct pip dependency of cwltest, it is required for cwltest to run actual CWL tests.","package":"cwltool","optional":true}],"imports":[{"note":"For programmatic execution of the cwltest suite. Most users interact via the 'cwltest' command-line tool or as a pytest plugin.","symbol":"main","correct":"from cwltest.cwltest import main"}],"quickstart":{"code":"import subprocess\nimport tempfile\nfrom pathlib import Path\nimport os\n\n# Create a dummy CWL workflow file\ncwl_content = \"\"\"\ncwlVersion: v1.0\nclass: CommandLineTool\nbaseCommand: echo\ninputs:\n  message:\n    type: string\n    inputBinding:\n      position: 1\noutputs:\n  output: \n    type: stdout\nstdout: output.txt\n\"\"\"\n\n# Create a dummy CWL test file\ntest_content = \"\"\"\n- doc: Example test for echo tool\n  tool: workflow.cwl\n  job:\n    message: Hello, CWL!\n  output:\n    output: Hello, CWL!\\n\n- doc: Another test\n  tool: workflow.cwl\n  job:\n    message: Another message\n  output:\n    output: Another message\\n\n\"\"\"\n\nwith tempfile.TemporaryDirectory() as tmpdir:\n    tmp_path = Path(tmpdir)\n    workflow_path = tmp_path / \"workflow.cwl\"\n    test_path = tmp_path / \"test_example.yml\"\n\n    workflow_path.write_text(cwl_content)\n    test_path.write_text(test_content)\n\n    print(f\"Created dummy workflow: {workflow_path}\")\n    print(f\"Created dummy test file: {test_path}\")\n\n    print(\"\\nRunning cwltest...\")\n    try:\n        # cwltest uses cwltool by default to execute the workflow\n        # Ensure cwltool is installed for this example to fully function:\n        # pip install cwltest cwltool\n        cmd = [\"cwltest\", \"--test\", str(test_path), \"--basedir\", str(tmp_path)]\n        result = subprocess.run(cmd, capture_output=True, text=True, check=True)\n        print(\"cwltest output:\")\n        print(result.stdout)\n        print(\"Tests passed successfully!\")\n    except subprocess.CalledProcessError as e:\n        print(f\"cwltest failed with error: {e}\")\n        print(f\"Stderr: {e.stderr}\")\n        print(f\"Stdout: {e.stdout}\")\n    except FileNotFoundError:\n        print(\"Error: 'cwltest' command not found. Ensure cwltest is installed and in your PATH.\")\n        print(\"Note: You might also need 'cwltool' installed for cwltest to execute CWL workflows.\")","lang":"python","description":"This quickstart demonstrates how to use the `cwltest` command-line tool to run tests against a simple CWL workflow. It creates a dummy CWL workflow and a test definition file, then executes `cwltest`. For `cwltest` to actually run the CWL workflow during testing, a CWL runner like `cwltool` must be installed alongside `cwltest`."},"warnings":[{"fix":"Upgrade your Python environment to version 3.10, 3.11, 3.12, 3.13, or 3.14.","message":"Support for Python 3.9 has been removed in recent versions. Users on Python 3.9 will encounter installation errors or runtime failures.","severity":"breaking","affected_versions":">=2.6.20251216093331"},{"fix":"Refer to the cwltest GitHub releases and `pyproject.toml` for supported `pytest` versions. Upgrade or downgrade `pytest` if you encounter `InvalidPluginWarning` or test runner issues.","message":"cwltest frequently updates its internal pytest dependency and can have compatibility issues with very old or very new pytest versions. Check release notes if encountering plugin loading issues.","severity":"gotcha","affected_versions":"All versions, especially when pytest is upgraded/downgraded independently."},{"fix":"Be aware that new releases might introduce changes (including deprecations or minor breaking changes) more frequently than a strict semantic versioning scheme might imply for a 'patch' or 'minor' release.","message":"cwltest uses a date-based versioning scheme (e.g., `2.6.YYYYMMDDHHMMSS`) rather than standard semantic versioning (Major.Minor.Patch). This indicates frequent releases, but the 'Minor' component (`2.6` here) does not strictly adhere to typical breaking change expectations.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure that `cwltool` (or your preferred CWL runner) is installed in the same environment as `cwltest` for tests to run successfully (e.g., `pip install cwltest cwltool`).","message":"While `cwltest` provides the testing framework, it requires a separate CWL runner (like `cwltool`) to execute the actual CWL workflows defined in your tests. `cwltest` defaults to using `cwltool`.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-17T00:00:00.000Z","next_check":"2026-07-16T00:00:00.000Z","problems":[{"fix":"Ensure `cwltest` is installed in your active environment (`pip install cwltest`) and that your shell's PATH includes the script directory for your Python installation (e.g., `~/.local/bin` or the virtual environment's `bin/Scripts` folder).","cause":"The `cwltest` executable is not in your system's PATH, or the package was not installed correctly.","error":"cwltest: command not found"},{"fix":"Run `pip install cwltest` in your desired Python environment. If using a virtual environment, ensure it is activated. Check for typos in your import statements.","cause":"`cwltest` is not installed in the currently active Python environment, or you are trying to import it incorrectly.","error":"ModuleNotFoundError: No module named 'cwltest'"},{"fix":"Upgrade your Python environment to a supported version (3.10, 3.11, 3.12, 3.13, or 3.14). Alternatively, downgrade `cwltest` to an older version that supported Python 3.9 (e.g., `pip install cwltest==2.5.20240425111257`).","cause":"You are attempting to use a recent version of `cwltest` (>=2.6.x) with Python 3.9, which is no longer supported.","error":"ERROR: cwltest.cwltest.UnsupportedPythonVersion: Python 3.9 is not supported, please upgrade to Python 3.10 or newer"},{"fix":"Install `cwltool` in the same environment as `cwltest`: `pip install cwltool`. Ensure that the `cwltool` executable is accessible via your system's PATH.","cause":"cwltest tried to invoke the `cwltool` command (its default runner) but couldn't find it. `cwltool` is not installed or not in the PATH.","error":"FileNotFoundError: [Errno 2] No such file or directory: 'cwltool'"}]}