Unsloth

2026.4.4 · active · verified Sat Apr 11

Unsloth is a library that enables 2-5X faster training, reinforcement learning, and finetuning of large language models (LLMs) on consumer GPUs, often reducing VRAM usage. As of version 2026.4.4, it continues to provide significant optimizations for Hugging Face Transformers models. Releases follow a calendar-based versioning scheme, indicating frequent updates.

Warnings

gotcha Unsloth performance critically depends on matching your CUDA toolkit version (if using NVIDIA GPUs) with the correct `[cuXXX]` installation extra (e.g., `[cu121]` for CUDA 12.1). Mismatching can lead to significant performance degradation or runtime errors.
Fix: Always install Unsloth with the specific CUDA extra that matches your PyTorch/system CUDA installation (e.g., `pip install "unsloth[cu121]"`). Verify your `torch.version.cuda`.
gotcha Unsloth is primarily designed and optimized for GPU acceleration. While it might run on a CPU, performance will be severely degraded and practically unusable for meaningful LLM finetuning. Ensure you have a compatible NVIDIA GPU with sufficient VRAM.
Fix: Ensure your environment has a compatible NVIDIA GPU and that PyTorch is configured to use it (e.g., `torch.cuda.is_available()` should return `True`).
gotcha Unsloth often relies on specific versions of `transformers` and `peft` libraries for optimal compatibility and performance. New releases of these dependencies might introduce breaking changes or require updates to Unsloth itself.
Fix: Check the Unsloth GitHub README or release notes for recommended `transformers` and `peft` versions. Regularly update Unsloth and its dependencies, or pin versions to known-working configurations.
breaking The arguments and default values for `FastLanguageModel.from_pretrained` and `FastLanguageModel.get_peft_model` can change between major Unsloth releases, especially concerning quantization, LoRA configuration, and gradient checkpointing.
Fix: Refer to the latest Unsloth documentation or GitHub README for the correct API signatures and recommended parameters when upgrading. Pay attention to changes in `dtype`, `load_in_4bit`, `r`, `target_modules`, and `use_gradient_checkpointing`.

Install

pip install "unsloth[cu121]" --upgrade Recommended for CUDA 12.1+
pip install "unsloth[cu118]" --upgrade Recommended for CUDA 11.8
pip install unsloth --upgrade Base install (requires manual PyTorch/CUDA setup)

Imports

FastLanguageModel
```
from unsloth import FastLanguageModel
```
PatchModel
```
from unsloth import FastLanguageModel, PatchModel
```
PatchModel is often imported alongside FastLanguageModel for convenience.

Quickstart

This quickstart demonstrates how to load a pre-trained LLM using Unsloth's `FastLanguageModel.from_pretrained` with 4-bit quantization and then prepare it for PEFT (LoRA) finetuning using `get_peft_model`. It sets up the model for efficient training on a GPU.

from unsloth import FastLanguageModel
import torch

# 1. Load a pre-trained model and tokenizer
max_seq_length = 2048 # Max sequence length for your model
dtype = None # None for auto detection (bfloat16 preferred, float16 as fallback)
load_in_4bit = True # Use 4-bit quantization

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/mistral-7b-instruct-v0.2-bnb-4bit", # or "mistralai/Mistral-7B-Instruct-v0.2"
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)

# 2. Prepare the model for training (add LoRA adapters)
model = FastLanguageModel.get_peft_model(
    model,
    r = 16, # LoRA attention dimension
    target_modules = [
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_alpha = 16, # Alpha for LoRA scaling
    lora_dropout = 0.05, # Dropout for LoRA layers
    bias = "none", # Only under the LoRA layers
    use_gradient_checkpointing = "current-device",
    random_state = 3407,
    use_rslora = False,
    loftq_config = None,
)

# 3. Define a simple dataset (for demonstration)
# In a real scenario, you would load your actual dataset here
# from datasets import Dataset
# dataset = Dataset.from_dict({"text": ["...", "..."]})

print("Model and tokenizer loaded and prepared for finetuning!")
print(f"Using dtype: {model.dtype}, Quantization: {model.quantization_method}")
# Further steps would involve data preparation and using Hugging Face Trainer

view raw JSON →