Hydra Submitit Launcher

1.2.0 · active · verified Thu Apr 16

The Hydra Submitit Launcher provides a plugin for Hydra applications, enabling them to launch jobs on various backend environments like Slurm, AWS Batch, or locally via Submitit. It integrates seamlessly with Hydra's configuration system. The current version is 1.2.0, with releases typically aligning with updates to Submitit or Hydra itself.

Common errors

```
Error creating plugin 'hydra_submitit_launcher'. Check that it is correctly installed.
```
cause The `hydra-submitit-launcher` package is either not installed in the active Python environment, or Hydra cannot find its entry points.

fix Run `pip install hydra-submitit-launcher`. Ensure your Python environment is active and the package is installed where Hydra can find it. If using a virtual environment, ensure it's activated.
```
Could not instantiate launcher `submitit_slurm`.
```
cause This error often occurs when the specific Submitit backend (e.g., `submitit_slurm`) specified in the config is not properly set up, or the required `submitit` extra dependencies (e.g., `submitit[slurm]`) are not installed.

fix Verify the launcher name (`submitit_slurm`, `submitit_local`, `submitit_aws`) in your Hydra config is spelled correctly. If using `submitit_slurm`, ensure you've installed `submitit` with the slurm extra: `pip install submitit[slurm]`.
```
omegaconf.errors.KeyError: 'launcher' when trying to configure submitit options, e.g., `hydra.launcher.submitit_local.timeout_min`.
```
cause The `hydra/launcher` override in the `defaults` section of your Hydra config is missing or incorrectly placed, preventing the launcher from being selected and its configuration subtree created.

fix Add or correct the `defaults` entry: `defaults: - override hydra/launcher: submitit_local` (or `submitit_slurm`). This must be present at the top level of your main Hydra config file.

Warnings

breaking The behavior for distinguishing job timeouts from preemptions has changed in version 1.2.0 due to upstream Slurm regressions (versions 19.04-20.02). Relying on Slurm's exact signals for preemption detection might require review, as the launcher now implements its own logic.
Fix: Review job monitoring and error handling logic, especially if it directly interprets Slurm exit codes or signals for preemption. Adapt to the launcher's new detection method for timeouts vs. preemptions, or consult Submitit documentation for best practices.
gotcha Paths containing spaces or special characters may not be correctly handled by Submitit backends, leading to job failures or unexpected behavior during resource staging. While 1.2.0 includes fixes for quoting, complex path scenarios can still be problematic.
Fix: Upgrade to `hydra-submitit-launcher>=1.2.0`. As a best practice, avoid special characters and spaces in job-related paths. Thoroughly test job submission with non-trivial paths if their use is unavoidable.
gotcha If the `hydra/launcher` default is not correctly overridden in your Hydra configuration, Hydra will default to its `basic` launcher, or raise an error if an incorrect launcher name is specified, preventing Submitit from being used.
Fix: Ensure your `conf/config.yaml` or equivalent explicitly sets `defaults: - override hydra/launcher: submitit_slurm` (or `submitit_local`, etc.) within the global defaults to activate the plugin. This line must be present at the top level of your primary config file.

Install

pip install hydra-submitit-launcher Install launcher

Imports

SubmititLauncher
wrong:
```
from hydra.launcher import SubmititLauncher
```
correct:
```
from hydra_plugins.hydra_submitit_launcher.submitit_launcher import SubmititLauncher
```
While rarely directly imported by user applications (it's a plugin that Hydra auto-discovers), this is the correct path if introspection or direct instantiation is needed. Common mistakes include trying to import from `hydra` core or an incorrect plugin path.

Quickstart

This example demonstrates how to set up a basic Hydra application to be launched using `submitit_local`. Users need to define a Hydra configuration file (e.g., `conf/config.yaml`) that specifies the `submitit_local` launcher. The `@hydra.main` decorator will then use this configuration to launch the `my_app` function as a Submitit job. For demonstration purposes, `submitit_local` is used; for production, `submitit_slurm` or other backends would be chosen after installing their respective dependencies (e.g., `pip install submitit[slurm]`).

import hydra
from omegaconf import DictConfig
import os

# To run this quickstart:
# 1. Ensure hydra-submitit-launcher is installed: `pip install hydra-submitit-launcher`
# 2. Create a directory named 'conf' in the same location as this script.
# 3. Create a file 'conf/config.yaml' with the following content:
#    ---
#    # conf/config.yaml
#    # @package _global_
#    defaults:
#      - override hydra/launcher: submitit_local
#      - _self_
#
#    hydra:
#      launcher:
#        submitit_local:
#          timeout_min: 1
#          cpus_per_task: 1
#          mem_gb: 1
#          nodes: 1
#    ---
# 4. Save this Python code as a .py file (e.g., my_app.py).
# 5. Run from your terminal: `python my_app.py`

@hydra.main(config_path='conf', config_name='config', version_base='1.3')
def my_app(cfg: DictConfig) -> None:
    print(f"Hello from a Submitit-launched job!")
    print(f"Working directory : {os.getcwd()}")
    print(f"Config: {cfg.pretty()}")
    # The actual job logic would go here. Submitit handles serialization
    # of the function and its arguments to the remote/local executor.

if __name__ == "__main__":
    my_app()

view raw JSON →