Accelerate

Warnings

• breaking accelerate config must be run before first use. Without a config file, Accelerate falls back to single-process CPU mode silently — multi-GPU training simply won't use multiple GPUs.
  Fix: Run accelerate config once after install, or programmatically: from accelerate.utils import write_basic_config; write_basic_config(). For CI: set ACCELERATE_CONFIG_FILE env var pointing to a pre-built config.
• breaking Python 3.9 support dropped in 1.13.0. Accelerate now requires Python >=3.10.
  Fix: Upgrade Python to 3.10+. Pin accelerate<1.13.0 for Python 3.9 environments.
• breaking Accelerator() initialized outside the training function raises ValueError when using notebook_launcher for multi-GPU. Silently falls back to 1 GPU without error if no notebook_launcher is used.
  Fix: Always initialize Accelerator() inside the training function passed to notebook_launcher. Never create it at notebook cell level or module level when using multi-GPU in notebooks.
• breaking accelerator.load_state() fails with PyTorch 2.6+ due to torch.load weights_only=True default flip. Optimizer states with custom objects (omegaconf.ListConfig, etc.) raise UnpicklingError.
  Fix: Use torch.serialization.add_safe_globals([ListConfig]) to allowlist custom types, or pass weights_only=False to the underlying load call if the checkpoint source is trusted.
• breaking DeepSpeed integration: only one nn.Module per Accelerator instance is supported. Passing multiple models to accelerator.prepare() with DeepSpeed raises AssertionError.
  Fix: With DeepSpeed, create a separate Accelerator instance per model, or merge models before wrapping.
• gotcha accelerate launch ignores Python script argument ordering. Flags intended for the script must come after --, otherwise they are parsed as accelerate launch flags.
  Fix: Use: accelerate launch script.py --my-arg value. If ambiguous: accelerate launch -- script.py --my-arg value.
• gotcha loss.backward() instead of accelerator.backward(loss) silently bypasses mixed precision gradient scaling. Training proceeds but gradients are wrong under fp16/bf16 — numerical instability or NaN loss.
  Fix: Replace all loss.backward() calls with accelerator.backward(loss) throughout the training loop.

Install

• pip install accelerate Standard
• accelerate config Required first-time setup (interactive CLI)
• python -c "from accelerate.utils import write_basic_config; write_basic_config(mixed_precision='fp16')" Non-interactive basic config

Imports

• Accelerator

  from accelerate import Accelerator
  
  def training_function():
      # Accelerator MUST be initialized inside the training function for notebook_launcher
      accelerator = Accelerator(mixed_precision='fp16')
      model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
      
      for batch in dataloader:
          optimizer.zero_grad()
          loss = model(batch)
          accelerator.backward(loss)  # NOT loss.backward()
          optimizer.step()

  For notebook_launcher (Colab/Jupyter multi-GPU), Accelerator() must be initialized INSIDE the training function, never at module/notebook level.
• accelerator.backward

  loss = criterion(outputs, targets)
  accelerator.backward(loss)

  Always use accelerator.backward(loss) instead of loss.backward(). Direct loss.backward() bypasses Accelerate's mixed precision gradient scaling and gradient accumulation logic.

Quickstart

Core Accelerate pattern. Run with: accelerate launch train.py

from accelerate import Accelerator
import torch
import torch.nn as nn

def train():
    accelerator = Accelerator(mixed_precision='bf16')
    
    model = nn.Linear(10, 1)
    optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
    dataloader = ...  # your DataLoader
    
    # prepare() handles device placement and distributed wrapping
    model, optimizer, dataloader = accelerator.prepare(
        model, optimizer, dataloader
    )
    
    model.train()
    for batch in dataloader:
        optimizer.zero_grad()
        outputs = model(batch['input'])
        loss = nn.functional.mse_loss(outputs, batch['target'])
        accelerator.backward(loss)  # not loss.backward()
        optimizer.step()
    
    # Save on main process only
    accelerator.wait_for_everyone()
    if accelerator.is_main_process:
        accelerator.save_model(model, 'output/')