Hydra Joblib Launcher

1.2.0 verified Fri May 01 auth: no python

Provides a Joblib-based launcher for Hydra applications, enabling parallel execution of joblib tasks (e.g., using loky or threading backend). Current version 1.2.0 is compatible with Hydra 1.1 and 1.2. Low release cadence; last updated 2021.

pip install hydra-joblib-launcher

Common errors

error ModuleNotFoundError: No module named 'hydra_joblib_launcher' ↓

cause Incorrect import path; using 'hydra_joblib_launcher' instead of 'hydra_plugins.hydra_joblib_launcher'.

fix

Use from hydra_plugins.hydra_joblib_launcher.joblib_launcher import JoblibLauncher.

error KeyError: 'hydra.joblib' ↓

cause Missing or misconfigured joblib launcher config in your Hydra configuration.

fix

Add - joblib to your defaults list or set hydra.joblib.backend: loky in your config.

error AttributeError: 'NoneType' object has no attribute 'backend' ↓

cause The launcher is not activated because the config key `hydra.joblib` is not defined.

fix

Ensure your config includes hydra.joblib.backend and hydra.joblib.n_jobs.

Warnings

breaking Hydra 1.3 changed the launcher API; hydra-joblib-launcher 1.2.0 is not compatible with Hydra 1.3. ↓

fix Stay on hydra-core<1.3 or use the built-in Hydra joblib launcher (experimental) from Hydra 1.3.

gotcha The plugin must be installed and the config key 'hydra.joblib' must be set in your Hydra config to activate the launcher, else it defaults to the SequentialLauncher. ↓

fix Add `hydra.joblib: {backend: 'loky', n_jobs: 4}` to your config.

gotcha JoblibLauncher does not support dynamic n_jobs changes; n_jobs must be set before the job starts. ↓

fix Configure n_jobs in the config YAML, not at runtime.

Imports

JoblibLauncher

wrong

from hydra_joblib_launcher import JoblibLauncher

correct

from hydra_plugins.hydra_joblib_launcher.joblib_launcher import JoblibLauncher

Module lives under hydra_plugins namespace.

Quickstart

Example Hydra app using joblib launcher (launcher config set in YAML).

import hydra
from omegaconf import DictConfig

@hydra.main(config_name="config", config_path=".", version_base=None)
def my_app(cfg: DictConfig):
    # joblib launcher is specified in config
    print(f"Working on {cfg.job_id}")

if __name__ == "__main__":
    my_app()