Garak - LLM vulnerability scanner

0.14.1 verified Fri May 01 auth: no python

Garak is a framework for probing large language models (LLMs) for security vulnerabilities. It automates the process of generating adversarial inputs and evaluating model outputs for issues like jailbreak, hallucination, data leakage, toxicity, and more. Current version is 0.14.1, requiring Python >=3.10. Active development with monthly releases.

pip install garak

Common errors

error ModuleNotFoundError: No module named 'garak' ↓

cause Garak not installed or installed in a different environment.

fix

Run pip install garak in the correct Python environment (Python >=3.10).

error garak: error: the following arguments are required: --model_type ↓

cause CLI invoked without specifying the model type.

fix

Provide --model_type (e.g., openai, huggingface, test) and optionally --model_name.

error openai.error.AuthenticationError: No API key found ↓

cause OpenAI API key not set when using OpenAI generator.

fix

Set environment variable OPENAI_API_KEY before running garak, or use a different model type.

error yaml.scanner.ScannerError: mapping values are not allowed here ↓

cause Malformed YAML config file (e.g., missing colon or space).

fix

Check the YAML file syntax. Use --config with a valid .yml file or remove config flag to use defaults.

error KeyError: 'failure' in report parsing ↓

cause Script expects 'failure' key but garak v0.12+ uses 'attack_success'.

fix

Update parsing code to target 'attack_success' instead of 'failure'.

Warnings

breaking In v0.14.0, CLI option `--generate_autodan` was removed. Scripts using this flag will break. ↓

fix Remove `--generate_autodan` from command lines. Use other probe options instead.

breaking In v0.12.0, the `failure` concept was renamed to `attack success`. Output reports and APIs using 'failure' need updating. ↓

fix Replace references to 'failure' with 'attack_success' in custom reporting or parsing scripts.

deprecated JSON config support (v0.14.0) is experimental; YAML config remains default and recommended for stability. ↓

fix Continue using YAML config files unless you specifically need JSON. Monitor future releases.

gotcha Garak requires a model to be accessible via API or local. Running without a valid model will throw an error. ↓

fix Set environment variables like OPENAI_API_KEY or use local generators (e.g., `--model_type test` for testing).

gotcha Probe performance can be slow on large models; set `--max_workers` to control parallelism. ↓

fix Use `--max_workers 1` to reduce load or increase for faster scanning on capable hardware.

Imports

garak
```
import garak
```
Garak exposes a CLI; programmatic usage uses submodules like garak.probes, garak.generators, etc.

Quickstart

Basic import test. Garak is primarily used via CLI; this verifies the installation.

import garak
from garak.probes.base import Probe

# Quick scan with default config
# Run from command line: garak --model_type openai --model_name gpt-3.5-turbo
print("Garak imported successfully - run CLI for full functionality")