Snakemake Scheduler Plugin Interface

Common errors

• Pulp: Error while trying to execute, use msg=True for more details

  cause This error often indicates an issue with the underlying ILP solver used by Snakemake's internal scheduler, which can be exposed when a custom scheduler fails or defers to the default. It can be a symptom of a misconfigured Snakemake environment or a resource issue.
  fix Check the Snakemake logs for more context. Ensure that required solver binaries (e.g., CBC for ILP) are installed and accessible in your environment. If using a custom scheduler, debug its `_get_selected_jobs` method to ensure it consistently returns valid job sequences or `None` for fallback, not causing unexpected states. Sometimes, this occurs with older Snakemake versions or specific scheduler plugin versions; consider updating Snakemake or reviewing plugin compatibility.

• AssertionError: assert self.workflow.is_main_process

  cause This assertion error indicates that a scheduler plugin (or Snakemake's internal scheduler) is attempting an operation that should only be performed by the main Snakemake process, but it's running in a subprocess. This was a known issue with Snakemake v9.14 in combination with certain scheduler plugins.
  fix This specific issue was resolved in Snakemake v9.14.5. Upgrade your Snakemake installation to version 9.14.5 or newer (e.g., `pip install --upgrade snakemake`). If the problem persists, ensure your scheduler plugin is compatible with your Snakemake version.

• Failed to solve the job scheduling wth snakemake on SLURM scheduler / scheduler skips the first Job (Job 0) and directly jumps to Job1

  cause This can happen when the scheduler logic, either custom or built-in, fails to correctly determine job dependencies or available jobs, leading to an incorrect execution order or jobs being missed. This might be due to a bug in the scheduler, an incompatibility between Snakemake and the scheduler plugin, or unexpected workflow states.
  fix First, ensure both Snakemake and your scheduler plugin are up-to-date. If using a custom plugin, thoroughly debug its `_get_selected_jobs` implementation. If the issue persists with default schedulers or specific plugins, you might temporarily switch to Snakemake's `greedy` scheduler (`--scheduler greedy`) or consider downgrading Snakemake to a known stable version if a recent update introduced the regression.

Warnings

• breaking Version 2.0.0 introduced breaking changes by moving properties from the general job interface to a single job interface. Plugin implementations built against older versions will likely fail due to changed method signatures or property access.
  Fix: Review your plugin's implementation against the updated `snakemake-interface-scheduler-plugins` `2.0.0+` API. Specifically, check methods interacting with `SchedulerJobInterface` and `SchedulerDAGInterface` to ensure they align with the new property locations.
• gotcha Snakemake scheduler plugins must adhere to a strict naming convention: `snakemake-scheduler-plugin-<name>`. The names `greedy`, `ilp`, and `milp` are forbidden as they are reserved for Snakemake's internal schedulers. Failure to follow this will prevent Snakemake from discovering and loading your plugin.
  Fix: Ensure your plugin package name on PyPI and its internal entry points strictly follow the `snakemake-scheduler-plugin-<name>` format, replacing `<name>` with a descriptive, non-forbidden identifier. Use `snakedeploy scaffold-snakemake-plugin` to ensure correct setup.
• gotcha The Snakemake scheduler plugin interface requires Python 3.11 or newer (but less than 4.0). Older Python versions are not supported, and attempting to install or run with them will result in environment or runtime errors.
  Fix: Ensure your development and deployment environment uses Python 3.11 or a later 3.x version. Update your `conda` or `virtualenv` to a compatible Python interpreter.
• gotcha Scheduler plugin support in Snakemake itself is available from version 8.0 onwards. Attempting to use plugins with older Snakemake versions may lead to unexpected behavior or plugin not being recognized.
  Fix: Ensure your Snakemake installation is version 8.0 or higher. Upgrade Snakemake if necessary (e.g., `pip install --upgrade snakemake`).

Install

• pip install snakemake-interface-scheduler-plugins Install latest version

Imports

• SchedulerSettingsBase

  from snakemake_interface_scheduler_plugins.settings import SchedulerSettingsBase

• SchedulerBase

  from snakemake_interface_scheduler_plugins.base import SchedulerBase

• SchedulerDAGInterface

  from snakemake_interface_scheduler_plugins.interfaces.dag import SchedulerDAGInterface

• SchedulerJobInterface

  from snakemake_interface_scheduler_plugins.interfaces.jobs import SchedulerJobInterface

Quickstart

This quickstart provides the fundamental Python class structure and required imports for developing a Snakemake scheduler plugin. A real plugin would extend `SchedulerBase` and `SchedulerSettingsBase`, implementing methods like `_get_selected_jobs` to define scheduling logic. Plugins must follow a specific naming convention (`snakemake-scheduler-plugin-<name>`) and be published to PyPI for Snakemake to discover them. The `snakedeploy scaffold-snakemake-plugin` command is recommended to set up a new plugin project.

from typing import Dict, Mapping, Optional, Union, Sequence
from dataclasses import dataclass, field
from snakemake_interface_scheduler_plugins.settings import SchedulerSettingsBase
from snakemake_interface_scheduler_plugins.base import SchedulerBase
from snakemake_interface_scheduler_plugins.interfaces.dag import SchedulerDAGInterface
from snakemake_interface_scheduler_plugins.interfaces.jobs import SchedulerJobInterface

# Define settings for your scheduler plugin. These will appear in the Snakemake CLI
# as --scheduler-<plugin-name>-<param-name>. All fields should be Optional.
@dataclass
class SchedulerSettings(SchedulerSettingsBase):
    my_custom_param: Optional[int] = field(
        default=None,
        metadata={
            "help": "A custom parameter for my scheduler plugin.",
            "env_var": False # Use True for sensitive info like passwords
        }
    )

# Implement the core scheduler logic
class Scheduler(SchedulerBase):
    def __init__(
        self, workflow: 'snakemake.workflow.Workflow',
        dag: SchedulerDAGInterface, 
        dryrun: bool,
        scheduler_settings: SchedulerSettings
    ):
        super().__init__(workflow, dag, dryrun, scheduler_settings)
        self.scheduler_settings = scheduler_settings
        print(f"Initializing MyCustomScheduler with param: {self.scheduler_settings.my_custom_param}")

    def _choose_single_job(self, jobs: Sequence[SchedulerJobInterface]) -> Optional[SchedulerJobInterface]:
        # Example: always pick the first job available
        if jobs:
            print(f"MyCustomScheduler choosing job: {jobs[0].jobid}")
            return jobs[0]
        return None

    def _get_selected_jobs(
        self, jobs: Sequence[SchedulerJobInterface]
    ) -> Optional[Sequence[SchedulerJobInterface]]:
        # Example: return all available jobs, or None to fall back to Snakemake's greedy scheduler
        if self.scheduler_settings.my_custom_param is not None and self.scheduler_settings.my_custom_param < 0:
            print("MyCustomScheduler falling back to greedy scheduler.")
            return None # Indicate fallback to Snakemake's internal greedy scheduler
        
        chosen_job = self._choose_single_job(jobs)
        if chosen_job: 
            return [chosen_job]
        return []

# Note: This is a skeleton for plugin development. To use, a Snakemake plugin package
# named `snakemake-scheduler-plugin-<name>` must be created and published to PyPI,
# which then exposes this `Scheduler` class and `SchedulerSettings` via entry points.