MLX Language Models

0.31.2 · active · verified Sat Apr 11

mlx-lm provides tools for loading, fine-tuning, and generating text with Large Language Models (LLMs) on Apple Silicon, leveraging the MLX framework. It offers seamless integration with the Hugging Face Hub for model access. The library is actively developed, with frequent patch releases, currently at version 0.31.2.

Warnings

gotcha MLX (and thus mlx-lm) is primarily optimized for Apple Silicon (macOS devices with M-series chips). While it can run on CPU, performance will be significantly slower, and larger models might exceed memory limits.
Fix: Ensure you are running on Apple Silicon for best performance. For non-Apple Silicon, manage model size and expectations accordingly.
breaking Compatibility with the `transformers` library can be sensitive across `mlx-lm` versions. Significant changes to `transformers` (e.g., transition to v5) have required corresponding `mlx-lm` updates.
Fix: Always check release notes for `mlx-lm` regarding `transformers` compatibility. It's recommended to keep both `mlx-lm` and `transformers` updated to their latest compatible versions or pin versions if specific behavior is needed.
gotcha Not all Hugging Face models can be directly loaded or will perform optimally with `mlx-lm`. Many require a conversion step to the MLX format, especially for quantization or specific architectures.
Fix: Prefer models from the `mlx-community` on Hugging Face Hub, which are pre-converted. For other models, use the `mlx_lm.convert` tool to convert them to MLX format before loading.
gotcha Batch generation and KV caching mechanisms have received numerous improvements and fixes across versions. Older versions might exhibit inefficiencies, incorrect behavior with varying prompt lengths, or issues with specific cache strategies.
Fix: For critical applications involving batch inference, ensure you are on the latest `mlx-lm` version to benefit from bug fixes and performance enhancements in batching and caching.

Install

pip install mlx-lm Install mlx-lm

Imports

load
```
from mlx_lm import load
```
generate
```
from mlx_lm import generate
```
convert
```
from mlx_lm.convert import convert
```

Quickstart

This quickstart demonstrates how to load an MLX-optimized LLM and tokenizer from the Hugging Face Hub and use it for streaming text generation. Ensure you have an Apple Silicon device for optimal performance.

import mlx_lm as lm

# Load a model and its tokenizer from Hugging Face Hub (MLX community variants are optimized)
# Replace 'mlx-community/Phi-3-mini-4k-instruct-8bit' with your desired model
model, tokenizer = lm.load("mlx-community/Phi-3-mini-4k-instruct-8bit")

# Define a prompt for text generation
prompt_text = "Write a short story about a cat who learns to fly:"

# Generate text
response_stream = lm.generate(
    model=model,
    tokenizer=tokenizer,
    prompt=prompt_text,
    verbose=False, # Set to True for detailed generation info
    temp=0.7,
    max_tokens=200,
    stream=True # Stream tokens as they are generated
)

print("Generated text:")
for token in response_stream:
    print(token, end="")
print()

view raw JSON →