Submitit Job Submission Library

Warnings

• breaking Submitit 1.2.0 changed how job preemption vs. timeout is detected for Slurm jobs. This was in response to a regression in Slurm versions (between 19.04 and 20.02) and might alter the behavior or reporting for long-running or pre-empted jobs compared to older `submitit` versions.
  Fix: Review job logs and `submitit`'s internal state for jobs that are expected to be pre-empted or timed out. Adapt your application's logic if it relied on the previous detection mechanism. Ensure your Slurm environment is consistent.
• breaking Submitit 1.2.0 introduced fixes for quoting paths in various internal operations. If your code or Slurm configurations previously relied on specific (and possibly incorrect) path handling, this update might cause previously working but malformed paths to now fail explicitly or behave differently due to correct quoting.
  Fix: Inspect any file paths or directory paths passed to `submitit` or configured in Slurm scripts. Ensure they are correctly formatted and do not contain characters that might have been misinterpreted before the quoting fixes.
• gotcha The `submitit.AutoExecutor` dynamically chooses between `SlurmExecutor` and `LocalExecutor` based on the environment (e.g., presence of `SLURM_JOB_ID` or Slurm executables). This can lead to unexpected local execution when a Slurm environment is assumed but not active, potentially consuming local resources or not fulfilling HPC requirements.
  Fix: For production or critical workloads, explicitly use `submitit.SlurmExecutor` or `submitit.LocalExecutor` to guarantee the desired execution environment. Always test `AutoExecutor` behavior in your target environments to understand its fallback logic.
• gotcha Submitit extensively uses `cloudpickle` for serializing functions and their arguments across processes. Complex objects, lambda functions capturing intricate state, or non-picklable resources (e.g., open file handles, database connections) passed to submitted functions will often lead to serialization errors.
  Fix: Ensure all arguments passed to `executor.submit()` are standard Python types or custom classes that are fully `cloudpickle`-compatible. For complex setups, consider passing file paths to data or configuration instead of directly passing objects, allowing the job to load resources within its own context.

Install

• pip install submitit Install submitit

Imports

• AutoExecutor

  from submitit import AutoExecutor

  The primary entry point for automatically selecting between Slurm or Local executors.
• SlurmExecutor

  from submitit import SlurmExecutor

  Used for explicitly submitting jobs to a Slurm cluster.
• LocalExecutor

  from submitit import LocalExecutor

  Used for explicitly running jobs on the local machine (useful for debugging).

Quickstart

This quickstart demonstrates how to use `submitit.AutoExecutor` to submit Python functions as jobs. It showcases configuring an executor, defining a simple function, submitting multiple jobs in a batch, and retrieving their results. The `AutoExecutor` intelligently switches between Slurm and local execution based on the environment.

import submitit
import time
import os

def my_function(x):
    time.sleep(0.1) # Simulate some work
    print(f"Hello from job! Input: {x}, PID: {os.getpid()}")
    return x * x

# Configure a log folder for submitit to store job information
# The %j placeholder will be replaced by the Slurm job ID
log_folder = os.path.join(os.getcwd(), "submitit_logs", "%j")

# Use AutoExecutor, which selects SlurmExecutor if a Slurm environment
# is detected, otherwise falls back to LocalExecutor.
executor = submitit.AutoExecutor(folder=log_folder)

# Set Slurm parameters (these are ignored by LocalExecutor)
executor.update_parameters(timeout_min=5, slurm_array_parallelism=2)

# Submit jobs within a batch context
with executor.batch():
    jobs = []
    for i in range(5):
        job = executor.submit(my_function, i)
        jobs.append(job)
    
    print(f"Submitted {len(jobs)} jobs. Waiting for results...")
    
    # Retrieve results (blocks until all jobs are complete)
    outputs = [job.result() for job in jobs]
    print(f"All jobs completed. Results: {outputs}")