PyTorch Ranger Optimizer

0.1.1 · maintenance · verified Thu Apr 16

Ranger is a synergistic PyTorch optimizer that combines Rectified Adam (RAdam) and LookAhead techniques to improve training stability and convergence in deep learning models. The PyPI package, `pytorch-ranger`, provides an implementation of this optimizer, though its last update was in March 2020. More recent developments and features are primarily found in the original author's GitHub repository or the `Ranger21` project.

Common errors

```
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu
```
cause Your model and input data are on different devices (e.g., one on GPU, one on CPU), leading to a device mismatch during computation.

fix Ensure both your model and all input tensors are explicitly moved to the same device (e.g., `model.to('cuda')`, `inputs.to('cuda')`).
```
RuntimeError: The size of tensor a (X) must match the size of tensor b (Y) at non-singleton dimension Z
```
cause A shape mismatch occurred between tensors, often when the output of one layer doesn't match the expected input shape of the next, or when input data doesn't align with the model's first layer.

fix Carefully inspect the `forward` method of your model and the shapes of your input data. Use `tensor.shape` or `print(tensor.size())` to debug tensor dimensions.
```
Loss is not decreasing or model is not learning effectively, despite seemingly correct setup.
```
cause This can stem from various issues, including an incorrect learning rate, inappropriate learning rate schedule for Ranger, or using an outdated `pytorch-ranger` version with subtle bugs.

fix Review your learning rate schedule (Ranger benefits from specific patterns). Consider if your `pytorch-ranger` version is too old for your PyTorch version, leading to unpatched issues. Double-check your loss function and data preprocessing.

Warnings

deprecated The `pytorch-ranger` PyPI package (v0.1.1) has not been updated since March 2020. It may lack critical bug fixes, performance improvements, and newer features (like Gradient Centralization v2) present in the original author's more actively maintained GitHub repository's `ranger.py` file or the `Ranger21` project.
Fix: For the latest features and fixes, consider copying the `ranger.py` file directly from the `lessw2020/Ranger-Deep-Learning-Optimizer` GitHub repository or exploring the `lessw2020/Ranger21` project.
gotcha Older versions of Ranger (likely including `pytorch-ranger==0.1.1`) may have issues with optimizer state management, specifically, 'save and then load may leave first run weights stranded in memory, slowing down future runs'.
Fix: Ensure you are using the most up-to-date `ranger.py` from the GitHub repository if you frequently save and load models, as this issue was addressed in later revisions of the original project.
gotcha Ranger, like other advanced optimizers, can be sensitive to learning rate schedules. Suboptimal schedules can lead to slow convergence or unstable training, even though Ranger aims for stability.
Fix: The author often recommends specific learning rate strategies, such as a 75% flat learning rate followed by a step-down or cosine annealing for the final 25% of training. Experiment with different schedules.

Install

pip install pytorch-ranger Install with pip

Imports

Ranger
wrong:
```
from ranger import Ranger
```
correct:
```
from pytorch_ranger import Ranger
```
The PyPI package `pytorch-ranger` uses `pytorch_ranger` as its top-level module, not `ranger`, unless you manually copied the `ranger.py` file from the original GitHub repository.

Quickstart

This quickstart demonstrates how to initialize a `Ranger` optimizer with a simple PyTorch model's parameters and perform a single forward and backward pass, followed by an optimization step.

import torch
import torch.nn as nn
from pytorch_ranger import Ranger

# 1. Define a simple PyTorch model
class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(10, 1)

    def forward(self, x):
        return self.linear(x)

model = SimpleModel()

# 2. Define dummy data and target
inputs = torch.randn(32, 10) # Example batch of 32 samples, 10 features
targets = torch.randn(32, 1) # Example batch of 32 targets

# 3. Instantiate the Ranger optimizer
# Pass model.parameters() to the optimizer
optimizer = Ranger(model.parameters(), lr=0.001)

# 4. Define a loss function
criterion = nn.MSELoss()

# 5. Perform a single training step (in a real scenario, this would be in a loop)
optimizer.zero_grad() # Zero the gradients
outputs = model(inputs) # Forward pass
loss = criterion(outputs, targets) # Compute loss
loss.backward() # Backward pass (compute gradients)
optimizer.step() # Update model parameters

print(f"Loss after one step: {loss.item():.4f}")

view raw JSON →