{"id":3284,"library":"submitit","title":"Submitit Job Submission Library","description":"Submitit is a Python 3.8+ toolbox developed by Facebook Incubator for submitting jobs to Slurm clusters, as well as providing a local executor for testing. It simplifies the process of dispatching Python functions to compute nodes, managing job states, and retrieving results. The current version is 1.5.4, and it sees active maintenance with occasional releases.","status":"active","version":"1.5.4","language":"en","source_language":"en","source_url":"https://github.com/facebookincubator/submitit","tags":["slurm","job-scheduler","hpc","distributed-computing","facebook"],"install":[{"cmd":"pip install submitit","lang":"bash","label":"Install submitit"}],"dependencies":[],"imports":[{"note":"The primary entry point for automatically selecting between Slurm or Local executors.","symbol":"AutoExecutor","correct":"from submitit import AutoExecutor"},{"note":"Used for explicitly submitting jobs to a Slurm cluster.","symbol":"SlurmExecutor","correct":"from submitit import SlurmExecutor"},{"note":"Used for explicitly running jobs on the local machine (useful for debugging).","symbol":"LocalExecutor","correct":"from submitit import LocalExecutor"}],"quickstart":{"code":"import submitit\nimport time\nimport os\n\ndef my_function(x):\n    time.sleep(0.1) # Simulate some work\n    print(f\"Hello from job! Input: {x}, PID: {os.getpid()}\")\n    return x * x\n\n# Configure a log folder for submitit to store job information\n# The %j placeholder will be replaced by the Slurm job ID\nlog_folder = os.path.join(os.getcwd(), \"submitit_logs\", \"%j\")\n\n# Use AutoExecutor, which selects SlurmExecutor if a Slurm environment\n# is detected, otherwise falls back to LocalExecutor.\nexecutor = submitit.AutoExecutor(folder=log_folder)\n\n# Set Slurm parameters (these are ignored by LocalExecutor)\nexecutor.update_parameters(timeout_min=5, slurm_array_parallelism=2)\n\n# Submit jobs within a batch context\nwith executor.batch():\n    jobs = []\n    for i in range(5):\n        job = executor.submit(my_function, i)\n        jobs.append(job)\n    \n    print(f\"Submitted {len(jobs)} jobs. Waiting for results...\")\n    \n    # Retrieve results (blocks until all jobs are complete)\n    outputs = [job.result() for job in jobs]\n    print(f\"All jobs completed. Results: {outputs}\")","lang":"python","description":"This quickstart demonstrates how to use `submitit.AutoExecutor` to submit Python functions as jobs. It showcases configuring an executor, defining a simple function, submitting multiple jobs in a batch, and retrieving their results. The `AutoExecutor` intelligently switches between Slurm and local execution based on the environment."},"warnings":[{"fix":"Review job logs and `submitit`'s internal state for jobs that are expected to be pre-empted or timed out. Adapt your application's logic if it relied on the previous detection mechanism. Ensure your Slurm environment is consistent.","message":"Submitit 1.2.0 changed how job preemption vs. timeout is detected for Slurm jobs. This was in response to a regression in Slurm versions (between 19.04 and 20.02) and might alter the behavior or reporting for long-running or pre-empted jobs compared to older `submitit` versions.","severity":"breaking","affected_versions":">=1.2.0"},{"fix":"Inspect any file paths or directory paths passed to `submitit` or configured in Slurm scripts. Ensure they are correctly formatted and do not contain characters that might have been misinterpreted before the quoting fixes.","message":"Submitit 1.2.0 introduced fixes for quoting paths in various internal operations. If your code or Slurm configurations previously relied on specific (and possibly incorrect) path handling, this update might cause previously working but malformed paths to now fail explicitly or behave differently due to correct quoting.","severity":"breaking","affected_versions":">=1.2.0"},{"fix":"For production or critical workloads, explicitly use `submitit.SlurmExecutor` or `submitit.LocalExecutor` to guarantee the desired execution environment. Always test `AutoExecutor` behavior in your target environments to understand its fallback logic.","message":"The `submitit.AutoExecutor` dynamically chooses between `SlurmExecutor` and `LocalExecutor` based on the environment (e.g., presence of `SLURM_JOB_ID` or Slurm executables). This can lead to unexpected local execution when a Slurm environment is assumed but not active, potentially consuming local resources or not fulfilling HPC requirements.","severity":"gotcha","affected_versions":"All"},{"fix":"Ensure all arguments passed to `executor.submit()` are standard Python types or custom classes that are fully `cloudpickle`-compatible. For complex setups, consider passing file paths to data or configuration instead of directly passing objects, allowing the job to load resources within its own context.","message":"Submitit extensively uses `cloudpickle` for serializing functions and their arguments across processes. Complex objects, lambda functions capturing intricate state, or non-picklable resources (e.g., open file handles, database connections) passed to submitted functions will often lead to serialization errors.","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"last_verified":"2026-04-11T00:00:00.000Z","next_check":"2026-07-10T00:00:00.000Z"}