{"id":9311,"library":"snakemake-interface-scheduler-plugins","title":"Snakemake Scheduler Plugin Interface","description":"This package provides a stable interface for interactions between Snakemake and its custom scheduler plugins. It offers abstract base classes and settings definitions that plugin developers must implement to create compliant Snakemake schedulers. The library is currently at version 2.0.2 and aims for a stable API to minimize breaking changes for plugin developers, supporting Snakemake workflows (version 8.0+) with Python 3.11 and newer.","status":"active","version":"2.0.2","language":"en","source_language":"en","source_url":"https://github.com/snakemake/snakemake-interface-scheduler-plugins","tags":["snakemake","workflow","scheduler","plugin","interface","HPC"],"install":[{"cmd":"pip install snakemake-interface-scheduler-plugins","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"This package provides an interface for Snakemake plugins; Snakemake is the runtime environment that utilizes these plugins. Plugin support is available in Snakemake 8.0 and later.","package":"snakemake>=8.0","optional":false}],"imports":[{"symbol":"SchedulerSettingsBase","correct":"from snakemake_interface_scheduler_plugins.settings import SchedulerSettingsBase"},{"symbol":"SchedulerBase","correct":"from snakemake_interface_scheduler_plugins.base import SchedulerBase"},{"symbol":"SchedulerDAGInterface","correct":"from snakemake_interface_scheduler_plugins.interfaces.dag import SchedulerDAGInterface"},{"symbol":"SchedulerJobInterface","correct":"from snakemake_interface_scheduler_plugins.interfaces.jobs import SchedulerJobInterface"}],"quickstart":{"code":"from typing import Dict, Mapping, Optional, Union, Sequence\nfrom dataclasses import dataclass, field\nfrom snakemake_interface_scheduler_plugins.settings import SchedulerSettingsBase\nfrom snakemake_interface_scheduler_plugins.base import SchedulerBase\nfrom snakemake_interface_scheduler_plugins.interfaces.dag import SchedulerDAGInterface\nfrom snakemake_interface_scheduler_plugins.interfaces.jobs import SchedulerJobInterface\n\n# Define settings for your scheduler plugin. These will appear in the Snakemake CLI\n# as --scheduler-<plugin-name>-<param-name>. All fields should be Optional.\n@dataclass\nclass SchedulerSettings(SchedulerSettingsBase):\n    my_custom_param: Optional[int] = field(\n        default=None,\n        metadata={\n            \"help\": \"A custom parameter for my scheduler plugin.\",\n            \"env_var\": False # Use True for sensitive info like passwords\n        }\n    )\n\n# Implement the core scheduler logic\nclass Scheduler(SchedulerBase):\n    def __init__(\n        self, workflow: 'snakemake.workflow.Workflow',\n        dag: SchedulerDAGInterface, \n        dryrun: bool,\n        scheduler_settings: SchedulerSettings\n    ):\n        super().__init__(workflow, dag, dryrun, scheduler_settings)\n        self.scheduler_settings = scheduler_settings\n        print(f\"Initializing MyCustomScheduler with param: {self.scheduler_settings.my_custom_param}\")\n\n    def _choose_single_job(self, jobs: Sequence[SchedulerJobInterface]) -> Optional[SchedulerJobInterface]:\n        # Example: always pick the first job available\n        if jobs:\n            print(f\"MyCustomScheduler choosing job: {jobs[0].jobid}\")\n            return jobs[0]\n        return None\n\n    def _get_selected_jobs(\n        self, jobs: Sequence[SchedulerJobInterface]\n    ) -> Optional[Sequence[SchedulerJobInterface]]:\n        # Example: return all available jobs, or None to fall back to Snakemake's greedy scheduler\n        if self.scheduler_settings.my_custom_param is not None and self.scheduler_settings.my_custom_param < 0:\n            print(\"MyCustomScheduler falling back to greedy scheduler.\")\n            return None # Indicate fallback to Snakemake's internal greedy scheduler\n        \n        chosen_job = self._choose_single_job(jobs)\n        if chosen_job: \n            return [chosen_job]\n        return []\n\n# Note: This is a skeleton for plugin development. To use, a Snakemake plugin package\n# named `snakemake-scheduler-plugin-<name>` must be created and published to PyPI,\n# which then exposes this `Scheduler` class and `SchedulerSettings` via entry points.\n","lang":"python","description":"This quickstart provides the fundamental Python class structure and required imports for developing a Snakemake scheduler plugin. A real plugin would extend `SchedulerBase` and `SchedulerSettingsBase`, implementing methods like `_get_selected_jobs` to define scheduling logic. Plugins must follow a specific naming convention (`snakemake-scheduler-plugin-<name>`) and be published to PyPI for Snakemake to discover them. The `snakedeploy scaffold-snakemake-plugin` command is recommended to set up a new plugin project."},"warnings":[{"fix":"Review your plugin's implementation against the updated `snakemake-interface-scheduler-plugins` `2.0.0+` API. Specifically, check methods interacting with `SchedulerJobInterface` and `SchedulerDAGInterface` to ensure they align with the new property locations.","message":"Version 2.0.0 introduced breaking changes by moving properties from the general job interface to a single job interface. Plugin implementations built against older versions will likely fail due to changed method signatures or property access.","severity":"breaking","affected_versions":">=2.0.0"},{"fix":"Ensure your plugin package name on PyPI and its internal entry points strictly follow the `snakemake-scheduler-plugin-<name>` format, replacing `<name>` with a descriptive, non-forbidden identifier. Use `snakedeploy scaffold-snakemake-plugin` to ensure correct setup.","message":"Snakemake scheduler plugins must adhere to a strict naming convention: `snakemake-scheduler-plugin-<name>`. The names `greedy`, `ilp`, and `milp` are forbidden as they are reserved for Snakemake's internal schedulers. Failure to follow this will prevent Snakemake from discovering and loading your plugin.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure your development and deployment environment uses Python 3.11 or a later 3.x version. Update your `conda` or `virtualenv` to a compatible Python interpreter.","message":"The Snakemake scheduler plugin interface requires Python 3.11 or newer (but less than 4.0). Older Python versions are not supported, and attempting to install or run with them will result in environment or runtime errors.","severity":"gotcha","affected_versions":"<=1.x.x, >=2.0.0"},{"fix":"Ensure your Snakemake installation is version 8.0 or higher. Upgrade Snakemake if necessary (e.g., `pip install --upgrade snakemake`).","message":"Scheduler plugin support in Snakemake itself is available from version 8.0 onwards. Attempting to use plugins with older Snakemake versions may lead to unexpected behavior or plugin not being recognized.","severity":"gotcha","affected_versions":"All versions of the interface"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Check the Snakemake logs for more context. Ensure that required solver binaries (e.g., CBC for ILP) are installed and accessible in your environment. If using a custom scheduler, debug its `_get_selected_jobs` method to ensure it consistently returns valid job sequences or `None` for fallback, not causing unexpected states. Sometimes, this occurs with older Snakemake versions or specific scheduler plugin versions; consider updating Snakemake or reviewing plugin compatibility.","cause":"This error often indicates an issue with the underlying ILP solver used by Snakemake's internal scheduler, which can be exposed when a custom scheduler fails or defers to the default. It can be a symptom of a misconfigured Snakemake environment or a resource issue.","error":"Pulp: Error while trying to execute, use msg=True for more details"},{"fix":"This specific issue was resolved in Snakemake v9.14.5. Upgrade your Snakemake installation to version 9.14.5 or newer (e.g., `pip install --upgrade snakemake`). If the problem persists, ensure your scheduler plugin is compatible with your Snakemake version.","cause":"This assertion error indicates that a scheduler plugin (or Snakemake's internal scheduler) is attempting an operation that should only be performed by the main Snakemake process, but it's running in a subprocess. This was a known issue with Snakemake v9.14 in combination with certain scheduler plugins.","error":"AssertionError: assert self.workflow.is_main_process"},{"fix":"First, ensure both Snakemake and your scheduler plugin are up-to-date. If using a custom plugin, thoroughly debug its `_get_selected_jobs` implementation. If the issue persists with default schedulers or specific plugins, you might temporarily switch to Snakemake's `greedy` scheduler (`--scheduler greedy`) or consider downgrading Snakemake to a known stable version if a recent update introduced the regression.","cause":"This can happen when the scheduler logic, either custom or built-in, fails to correctly determine job dependencies or available jobs, leading to an incorrect execution order or jobs being missed. This might be due to a bug in the scheduler, an incompatibility between Snakemake and the scheduler plugin, or unexpected workflow states.","error":"Failed to solve the job scheduling wth snakemake on SLURM scheduler / scheduler skips the first Job (Job 0) and directly jumps to Job1"}]}