Common Workflow Language Testing Framework (cwltest)

Common errors

• cwltest: command not found

  cause The `cwltest` executable is not in your system's PATH, or the package was not installed correctly.
  fix Ensure `cwltest` is installed in your active environment (`pip install cwltest`) and that your shell's PATH includes the script directory for your Python installation (e.g., `~/.local/bin` or the virtual environment's `bin/Scripts` folder).

• ModuleNotFoundError: No module named 'cwltest'

  cause `cwltest` is not installed in the currently active Python environment, or you are trying to import it incorrectly.
  fix Run `pip install cwltest` in your desired Python environment. If using a virtual environment, ensure it is activated. Check for typos in your import statements.

• ERROR: cwltest.cwltest.UnsupportedPythonVersion: Python 3.9 is not supported, please upgrade to Python 3.10 or newer

  cause You are attempting to use a recent version of `cwltest` (>=2.6.x) with Python 3.9, which is no longer supported.
  fix Upgrade your Python environment to a supported version (3.10, 3.11, 3.12, 3.13, or 3.14). Alternatively, downgrade `cwltest` to an older version that supported Python 3.9 (e.g., `pip install cwltest==2.5.20240425111257`).

• FileNotFoundError: [Errno 2] No such file or directory: 'cwltool'

  cause cwltest tried to invoke the `cwltool` command (its default runner) but couldn't find it. `cwltool` is not installed or not in the PATH.
  fix Install `cwltool` in the same environment as `cwltest`: `pip install cwltool`. Ensure that the `cwltool` executable is accessible via your system's PATH.

Warnings

• breaking Support for Python 3.9 has been removed in recent versions. Users on Python 3.9 will encounter installation errors or runtime failures.
  Fix: Upgrade your Python environment to version 3.10, 3.11, 3.12, 3.13, or 3.14.
• gotcha cwltest frequently updates its internal pytest dependency and can have compatibility issues with very old or very new pytest versions. Check release notes if encountering plugin loading issues.
  Fix: Refer to the cwltest GitHub releases and `pyproject.toml` for supported `pytest` versions. Upgrade or downgrade `pytest` if you encounter `InvalidPluginWarning` or test runner issues.
• gotcha cwltest uses a date-based versioning scheme (e.g., `2.6.YYYYMMDDHHMMSS`) rather than standard semantic versioning (Major.Minor.Patch). This indicates frequent releases, but the 'Minor' component (`2.6` here) does not strictly adhere to typical breaking change expectations.
  Fix: Be aware that new releases might introduce changes (including deprecations or minor breaking changes) more frequently than a strict semantic versioning scheme might imply for a 'patch' or 'minor' release.
• gotcha While `cwltest` provides the testing framework, it requires a separate CWL runner (like `cwltool`) to execute the actual CWL workflows defined in your tests. `cwltest` defaults to using `cwltool`.
  Fix: Ensure that `cwltool` (or your preferred CWL runner) is installed in the same environment as `cwltest` for tests to run successfully (e.g., `pip install cwltest cwltool`).

Install

• pip install cwltest Install cwltest
• pip install cwltest cwltool Install cwltest with default CWL runner

Imports

• main

  from cwltest.cwltest import main

  For programmatic execution of the cwltest suite. Most users interact via the 'cwltest' command-line tool or as a pytest plugin.

Quickstart

This quickstart demonstrates how to use the `cwltest` command-line tool to run tests against a simple CWL workflow. It creates a dummy CWL workflow and a test definition file, then executes `cwltest`. For `cwltest` to actually run the CWL workflow during testing, a CWL runner like `cwltool` must be installed alongside `cwltest`.

import subprocess
import tempfile
from pathlib import Path
import os

# Create a dummy CWL workflow file
cwl_content = """
cwlVersion: v1.0
class: CommandLineTool
baseCommand: echo
inputs:
  message:
    type: string
    inputBinding:
      position: 1
outputs:
  output: 
    type: stdout
stdout: output.txt
"""

# Create a dummy CWL test file
test_content = """
- doc: Example test for echo tool
  tool: workflow.cwl
  job:
    message: Hello, CWL!
  output:
    output: Hello, CWL!\n
- doc: Another test
  tool: workflow.cwl
  job:
    message: Another message
  output:
    output: Another message\n
"""

with tempfile.TemporaryDirectory() as tmpdir:
    tmp_path = Path(tmpdir)
    workflow_path = tmp_path / "workflow.cwl"
    test_path = tmp_path / "test_example.yml"

    workflow_path.write_text(cwl_content)
    test_path.write_text(test_content)

    print(f"Created dummy workflow: {workflow_path}")
    print(f"Created dummy test file: {test_path}")

    print("\nRunning cwltest...")
    try:
        # cwltest uses cwltool by default to execute the workflow
        # Ensure cwltool is installed for this example to fully function:
        # pip install cwltest cwltool
        cmd = ["cwltest", "--test", str(test_path), "--basedir", str(tmp_path)]
        result = subprocess.run(cmd, capture_output=True, text=True, check=True)
        print("cwltest output:")
        print(result.stdout)
        print("Tests passed successfully!")
    except subprocess.CalledProcessError as e:
        print(f"cwltest failed with error: {e}")
        print(f"Stderr: {e.stderr}")
        print(f"Stdout: {e.stdout}")
    except FileNotFoundError:
        print("Error: 'cwltest' command not found. Ensure cwltest is installed and in your PATH.")
        print("Note: You might also need 'cwltool' installed for cwltest to execute CWL workflows.")