{"id":9893,"library":"lion-pytorch","title":"Lion Optimizer for PyTorch","description":"lion-pytorch provides an efficient and high-performance implementation of the Lion optimizer for PyTorch. Based on the paper 'Symbolic Discovery of Optimization Algorithms', Lion often outperforms AdamW and other adaptive optimizers, especially in large-scale models, due to its sign-based update mechanism. The library is actively maintained, currently at version 0.2.4, and requires Python 3.9+.","status":"active","version":"0.2.4","language":"en","source_language":"en","source_url":"https://github.com/lucidrains/lion-pytorch","tags":["pytorch","optimizer","deep-learning","machine-learning"],"install":[{"cmd":"pip install lion-pytorch","lang":"bash","label":"Install Lion PyTorch"}],"dependencies":[{"reason":"Required for PyTorch model and tensor operations.","package":"torch","optional":false}],"imports":[{"symbol":"Lion","correct":"from lion_pytorch import Lion"}],"quickstart":{"code":"import torch\nfrom torch import nn\nfrom lion_pytorch import Lion\n\n# 1. Define a simple PyTorch model\nclass SimpleModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.linear = nn.Linear(10, 2)\n        self.relu = nn.ReLU()\n        self.output = nn.Linear(2, 1)\n\n    def forward(self, x):\n        return self.output(self.relu(self.linear(x)))\n\nmodel = SimpleModel()\n\n# 2. Instantiate the Lion optimizer\n# Note: Lion often requires a smaller learning rate than AdamW (e.g., 1e-4)\noptimizer = Lion(model.parameters(), lr=1e-4, weight_decay=1e-2)\n\n# 3. Create dummy data and target\ninputs = torch.randn(32, 10) # 32 samples, 10 features\ntargets = torch.randn(32, 1) # 32 samples, 1 target value\n\n# 4. Define a loss function\ncriterion = nn.MSELoss()\n\n# 5. Training loop (one step for quickstart demonstration)\noptimizer.zero_grad()           # Clear gradients from previous step\noutputs = model(inputs)         # Forward pass\nloss = criterion(outputs, targets) # Calculate loss\nloss.backward()                 # Backward pass (compute gradients)\noptimizer.step()                # Update model parameters\n\nprint(f\"Loss after one optimization step: {loss.item():.4f}\")","lang":"python","description":"This quickstart demonstrates how to initialize a PyTorch model and the Lion optimizer, then perform a single forward and backward pass to update the model parameters. It highlights the typical workflow for integrating Lion into a PyTorch training loop."},"warnings":[{"fix":"Experiment with learning rates in the range of 1e-5 to 1e-4 as a starting point, lower than typical AdamW values. Use a learning rate scheduler for better convergence.","message":"Lion often requires a significantly lower learning rate (e.g., 1/3 to 1/10) compared to AdamW for optimal performance and stability. Directly applying learning rates common for AdamW may lead to issues like NaN losses or poor convergence.","severity":"gotcha","affected_versions":">=0.1.0"},{"fix":"Treat Lion as a new optimizer requiring independent hyperparameter search. Do not assume previous AdamW settings will transfer directly.","message":"While powerful, Lion's optimal performance often requires re-tuning hyperparameters (especially `lr` and `betas`) specific to your task and model, rather than using it as a direct drop-in replacement with existing AdamW settings.","severity":"gotcha","affected_versions":">=0.1.0"},{"fix":"Remove the `eps` argument from the `Lion` optimizer's constructor. Only provide `betas` if customizing beyond the default `(0.9, 0.99)`.","message":"Unlike optimizers like Adam, Lion uses only two `betas` values for momentum and update, and does *not* accept an `eps` (epsilon) parameter. Attempting to pass `eps` will result in a `TypeError`.","severity":"gotcha","affected_versions":">=0.1.0"}],"env_vars":null,"last_verified":"2026-04-17T00:00:00.000Z","next_check":"2026-07-16T00:00:00.000Z","problems":[{"fix":"Run `pip install lion-pytorch` to install the library.","cause":"The `lion-pytorch` library is not installed in your current Python environment.","error":"ModuleNotFoundError: No module named 'lion_pytorch'"},{"fix":"Remove the `eps` argument from the `Lion` optimizer's constructor.","cause":"You are attempting to pass an `eps` parameter to the `Lion` optimizer, which it does not support. This parameter is common in optimizers like Adam/AdamW.","error":"TypeError: Lion.__init__() got an unexpected keyword argument 'eps'"},{"fix":"Ensure you pass `model.parameters()` to the optimizer: `optimizer = Lion(model.parameters(), lr=...)`.","cause":"The `Lion` optimizer constructor requires an iterable of model parameters (e.g., `model.parameters()`) as its first argument.","error":"TypeError: Lion.__init__() missing 1 required positional argument: 'params'"}]}