TRL

Warnings

• breaking tokenizer= parameter deprecated since TRL 0.12 and being removed. All trainers (SFTTrainer, DPOTrainer, etc.) now use processing_class= instead. Passing tokenizer= raises DeprecationWarning now, TypeError in future release.
  Fix: Replace tokenizer=tokenizer with processing_class=tokenizer in all trainer constructors.
• breaking Training args like max_seq_length, packing, dataset_text_field have moved from SFTTrainer constructor to SFTConfig. Passing them directly to SFTTrainer raises TypeError in recent versions.
  Fix: Use SFTConfig(max_length=512, ...) and pass as args=SFTConfig(...) to SFTTrainer.
• breaking TRL has extremely high API churn — major parameter renames, removals, and behavioral changes in almost every minor release. Code written for 0.8 likely fails on 0.15+. Pin versions in production.
  Fix: Pin exact TRL version in requirements. Read the GitHub releases page before upgrading: https://github.com/huggingface/trl/releases
• gotcha DPOTrainer with PEFT does not keep a separate reference model in memory — it disables the adapter to recover reference behavior. Passing ref_model= with a PEFT model wastes memory and raises a warning about sync_ref_model incompatibility.
  Fix: With PEFT models, omit ref_model entirely. DPOTrainer handles reference behavior automatically via adapter disable/enable.
• gotcha GRPOTrainer reward_funcs must be a list of callables, not a single callable. Passing a single function raises TypeError.
  Fix: Wrap single reward function in a list: reward_funcs=[my_reward_fn].
• gotcha SFTTrainer dataset format: conversational datasets (with 'messages' column) are handled differently from text datasets (with 'text' column). Mixing formats or using wrong column name causes silent empty-loss training.
  Fix: For chat format datasets use a 'messages' column with OpenAI-style message dicts. For plain text use a 'text' column. Check dataset_text_field in SFTConfig if using a custom column name.

Install

• pip install trl Standard
• pip install trl[peft] With PEFT/LoRA support
• pip install trl[quantization] With bitsandbytes quantization
• pip install trl[vllm] With vLLM for GRPO online generation

Imports

• SFTTrainer

  from trl import SFTConfig, SFTTrainer
  from datasets import load_dataset
  
  trainer = SFTTrainer(
      model='Qwen/Qwen2.5-0.5B',
      args=SFTConfig(output_dir='output', max_length=512),
      train_dataset=load_dataset('trl-lib/Capybara', split='train'),
      processing_class=tokenizer,  # not tokenizer=
  )

  tokenizer= parameter renamed to processing_class= in TRL 0.12. Training args like max_seq_length moved to SFTConfig, not passed directly to SFTTrainer.
• DPOTrainer

  from trl import DPOConfig, DPOTrainer
  
  trainer = DPOTrainer(
      model=model,
      args=DPOConfig(output_dir='output', beta=0.1),
      train_dataset=dataset,
      processing_class=tokenizer,
      # With PEFT: no ref_model needed — adapter is disabled to recover reference behavior
  )

  When using PEFT with DPOTrainer, do not pass ref_model. TRL automatically recovers reference behavior by disabling the adapter. Passing ref_model with PEFT wastes memory and may conflict.

Quickstart

SFT then DPO pipeline. Use SFTConfig/DPOConfig for all training args.

from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

# SFT — minimal setup
trainer = SFTTrainer(
    model='Qwen/Qwen2.5-0.5B',
    args=SFTConfig(output_dir='sft_output', num_train_epochs=1),
    train_dataset=load_dataset('trl-lib/Capybara', split='train'),
)
trainer.train()

# DPO — after SFT
from trl import DPOConfig, DPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('sft_output')
tokenizer = AutoTokenizer.from_pretrained('sft_output')

trainer = DPOTrainer(
    model=model,
    args=DPOConfig(output_dir='dpo_output', beta=0.1),
    train_dataset=load_dataset('trl-lib/ultrafeedback_binarized', split='train'),
    processing_class=tokenizer,
)
trainer.train()