Taskcluster Taskgraph Generator

20.0.0 · active · verified Mon Apr 13

Taskgraph is a Python library designed to generate complex dependency graphs for the Taskcluster CI service. It serves as the recommended approach for configuring CI tasks when projects outgrow simple .taskcluster.yml configurations, powering large-scale CI systems like Firefox's with tens of thousands of tasks. The library is currently at version 20.0.0 and maintains an active release cadence with frequent minor and patch updates.

Warnings

breaking The schema validation backend was migrated from `voluptuous` to `msgspec` in version 19.1.0. If you have custom schemas or directly interacted with `taskgraph.util.schema.Schema`, your code will require updates.
Fix: Migrate custom schemas to use the new `msgspec`-based `Schema` or explicitly import `LegacySchema` for continued `voluptuous` compatibility. Refer to the migration guide for detailed steps.
breaking In version 18.0.0, support for Mercurial sparse profiles was removed from `run-task`, and `run-task` no longer automatically cleans up the `fetches` directory. Additionally, the default root directory for configuration changed, and `config.yml` and the `ci` directory were renamed/moved.
Fix: Review `run-task` scripts and worker configurations to ensure proper cleanup if required. Update paths for `root_dir`, `config.yml`, and `taskcluster/ci` (now `taskcluster/kinds`) in custom logic or configuration.
breaking Version 16.0.0 removed the `--tag` flag from `taskgraph build-image`. Also, `taskgraph load-task` now requires the `-i/--interactive` flag to pause task execution.
Fix: Remove `--tag` from `taskgraph build-image` commands and handle image tagging manually. Add `-i` or `--interactive` to `taskgraph load-task` calls where pausing execution is desired.
breaking From version 6.x to 7.x, Python 3.8 or higher became a requirement. Several core modules and decorators were renamed (e.g., `taskgraph.transforms.job` to `taskgraph.transforms.run`, `run_job_using` to `run_task_using`).
Fix: Ensure your environment uses Python 3.8+. Update all references to the renamed modules and decorators in your transform files and custom logic.
gotcha Integrating Taskgraph into a CI system requires a 'Decision Task' defined in a `.taskcluster.yml` file, a `taskcluster` directory with kind definitions and transforms, and proper permissions (scopes) configured within your Taskcluster deployment, potentially including a GitHub App integration.
Fix: Follow the official Taskgraph 'Getting Started' and 'Configuring your Project' documentation to set up the `.taskcluster.yml`, `taskcluster` directory structure, and obtain necessary Taskcluster permissions.

Install

pip install taskcluster-taskgraph Install stable release

Imports

TransformSequence
```
from taskgraph.transforms.base import TransformSequence
```
Common import for defining custom transforms, which modify task definitions.
LegacySchema
```
from taskgraph.util.schema import LegacySchema
```
After 19.1.0, 'Schema' was migrated to msgspec. Use 'LegacySchema' for voluptuous-based schemas or migrate to msgspec.

Quickstart

The primary interaction with taskcluster-taskgraph is via its command-line interface. The `taskgraph init` command bootstraps a new project with the necessary configuration files and directory structure. You can then use `taskgraph full` to locally generate and view the task graph, and `taskgraph morphed -J` to inspect the final JSON definitions of tasks.

# 1. Initialize a new Taskgraph setup in your repository
# This creates a .taskcluster.yml and a taskcluster/ directory structure.
mkdir my-repo && cd my-repo
taskgraph init

# 2. Generate the task graph locally to see defined tasks
# This processes your task definitions and transforms.
taskgraph full

# To see the full JSON definition of tasks, use the -J/--json flag:
taskgraph morphed -J

view raw JSON →