Torch-Fidelity: Generative Model Metrics

0.4.0 · active · verified Wed Apr 15

Torch-fidelity is a PyTorch library offering precise, efficient, and extensible implementations of popular generative model evaluation metrics, including Inception Score (ISC), Fréchet Inception Distance (FID), Kernel Inception Distance (KID), Perceptual Path Length (PPL), and Precision and Recall (PRC). It aims for epsilon-exact numerical fidelity with reference TensorFlow implementations. The library is actively maintained, with its latest version being 0.4.0, and has a steady release cadence with significant updates, like new metrics and feature extractors.

Warnings

breaking In version 0.4.0, several KID-related API and CLI parameters were renamed for clarity. Specifically, `kid_degree`, `kid_gamma`, and `kid_coef0` are now `kid_kernel_poly_degree`, `kid_kernel_poly_gamma`, and `kid_kernel_poly_coef0` respectively.
Fix: Update your API calls and CLI arguments to use the new parameter names: `kid_kernel_poly_degree`, `kid_kernel_poly_gamma`, `kid_kernel_poly_coef0`.
breaking In version 0.3.0, the `calculate_metrics` function's `input1` and `input2` arguments changed from positional to keyword-only arguments. Several CLI arguments were also renamed, e.g., `--datasets-downloaded` to `--no-datasets-download` and `--cache-input1-name` to `--input1-cache-name`.
Fix: Ensure `input1` and `input2` are passed as keyword arguments (e.g., `input1=generator, input2='cifar10-train'`) and update CLI arguments according to the changelog.
gotcha The `feature_extractor_compile` option is experimental and might negatively impact numerical precision, leading to discrepancies compared to reference values.
Fix: Use `feature_extractor_compile=False` (default) if numerical precision is critical for your evaluation. Only enable it if you have thoroughly validated its impact on your specific use case.
gotcha KID (Kernel Inception Distance) metric can mathematically produce negative values. This is expected behavior and not an indication of an error.
Fix: Understand that negative KID values are normal and reflect the mathematical properties of the metric.
gotcha Using lossy image formats (like JPG/JPEG) for inputs can affect metric precision and may trigger warnings from `torch-fidelity`.
Fix: Prefer lossless image formats (e.g., PNG) for evaluating generative models to maintain the highest numerical precision for metrics.
gotcha `torch-fidelity`'s InceptionV3 implementation intentionally differs from `torchvision`'s and uses custom bilinear interpolation to ensure numerical fidelity with original TensorFlow implementations. This is crucial for comparing results with existing literature.
Fix: Be aware that direct comparison of InceptionV3 features with `torchvision`'s model might yield minor differences. For precise comparisons with TF-based results, rely on `torch-fidelity`'s built-in InceptionV3.
gotcha The default cache (`fidelity_cache`) and dataset (`fidelity_datasets`) root directories, usually located under `$HOME` or `$ENV_TORCH_HOME`, can consume significant disk space.
Fix: To manage disk usage, specify alternative locations using the `--cache-root` and `--datasets-root` CLI arguments or the corresponding `cache_root` and `datasets_root` keyword arguments in `calculate_metrics`. Caching can be disabled with `--no-cache` or `cache=False` (not recommended for efficiency).

Install

pip install torch-fidelity Latest stable release

Imports

torch_fidelity
```
import torch_fidelity
```

calculate_metrics

from torch_fidelity import calculate_metrics

GenerativeModelModuleWrapper

from torch_fidelity.generative_model_module_wrapper import GenerativeModelModuleWrapper

Quickstart

This quickstart demonstrates how to calculate Inception Score (ISC), Fréchet Inception Distance (FID), and Kernel Inception Distance (KID) using `torch-fidelity`'s Python API. It involves defining a dummy generative model, wrapping it with `GenerativeModelModuleWrapper`, and then passing it along with a reference input (like a pre-registered dataset 'cifar10-train') to the `calculate_metrics` function. The results are returned as a dictionary.

import torch
import torch.nn as nn
from torch_fidelity import calculate_metrics
from torch_fidelity.generative_model_module_wrapper import GenerativeModelModuleWrapper

# Dummy generator model for demonstration
class DummyGenerator(nn.Module):
    def __init__(self, z_size, img_size, img_channels):
        super().__init__()
        self.img_size = img_size
        self.img_channels = img_channels
        self.main = nn.Sequential(
            nn.Linear(z_size, 256),
            nn.ReLU(),
            nn.Linear(256, img_channels * img_size * img_size),
            nn.Sigmoid()
        )

    def forward(self, z):
        img = self.main(z)
        return img.view(-1, self.img_channels, self.img_size, self.img_size)

# Configuration
z_size = 128
img_size = 32
img_channels = 3
num_samples = 10000 # Number of samples to generate for metrics
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Instantiate and wrap the generator
generator = DummyGenerator(z_size, img_size, img_channels).to(device)
wrapped_generator = GenerativeModelModuleWrapper(
    generator, z_size, 'normal', 0, num_samples=num_samples, samples_batch_size=32
)

# Calculate metrics
# For FID/KID, you need a second input, e.g., a real dataset name or directory path.
# Here, we use a registered input 'cifar10-train' for demonstration.
metrics_dict = calculate_metrics(
    input1=wrapped_generator,
    input2='cifar10-train',
    cuda=True if device.type == 'cuda' else False,
    isc=True,
    fid=True,
    kid=True,
    verbose=False,
    save_cpu_ram=True # Optional: reduce GPU memory if needed
)

print(metrics_dict)

view raw JSON →