Clean-FID: Consistent FID Score Calculation

0.1.35 · active · verified Mon Apr 13

Clean-FID is a Python library built on PyTorch for calculating the Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) between image distributions. It addresses common inconsistencies in FID scores caused by differing or incorrect implementations of low-level image processing steps, such as resizing and quantization, in other libraries. The current version is 0.1.35, and it is actively maintained with periodic releases.

Warnings

gotcha FID scores from other libraries might be inconsistent. `clean-fid` aims to provide a 'clean' and reproducible FID by correctly implementing image resizing and quantization steps, which are often sources of discrepancy in other implementations (e.g., PyTorch-FID, TensorFlow-FID).
Fix: Always use `clean-fid` for new evaluations or for comparing with 'clean' results. Be aware that scores from other libraries might not be directly comparable without using `clean-fid`'s legacy modes.
gotcha Using `mode='legacy_pytorch'` or `mode='legacy_tensorflow'` will reproduce potentially inconsistent FID scores from older implementations. While useful for backward compatibility and comparison with existing literature, these modes do not reflect the 'clean' FID standard.
Fix: For new, robust, and comparable evaluations, prefer the default `mode='clean'` (or omit the `mode` argument as 'clean' is default). Only use `legacy` modes when explicitly needing to match a specific prior work's (potentially flawed) FID calculation.
gotcha JPEG compression, even if perceptually subtle, can significantly impact FID scores. If your generated or real images are compressed (e.g., for storage efficiency), it can lead to different FID values compared to uncompressed images.
Fix: Ensure consistent image compression (or lack thereof) across all image sets (real, generated, and any pre-computed statistics) for meaningful FID comparisons. When comparing with other works, verify their image compression practices.
breaking The `clip` dependency for CLIP-FID is installed directly from GitHub (`clip @ git+https://github.com/openai/CLIP.git`). This direct Git installation can sometimes cause issues with dependency resolvers or break if the upstream repository changes in an incompatible way.
Fix: If encountering installation issues with `clip`, try installing it manually before `clean-fid` or ensure your environment supports direct Git installs. Monitor the `openai/CLIP` repository for breaking changes if you rely on CLIP-FID functionality.

Install

pip install clean-fid Install stable version

Imports

fid
```
from cleanfid import fid
```
The primary interface for `clean-fid` is the `fid` module, which exposes functions like `compute_fid` and `compute_kid`.

Quickstart

This quickstart demonstrates how to compute the FID score between two folders of images using `clean-fid`. Replace `path_to_real_images` and `path_to_generated_images` with actual directories containing your image files. The library also supports computing FID against pre-computed statistics for standard datasets or using a generative model directly.

import os
from cleanfid import fid

# Create dummy image folders for demonstration
if not os.path.exists('path_to_real_images'):
    os.makedirs('path_to_real_images', exist_ok=True)
    # In a real scenario, populate this folder with real images
    # For this example, we'll just create a dummy file
    with open('path_to_real_images/dummy_real.txt', 'w') as f: pass

if not os.path.exists('path_to_generated_images'):
    os.makedirs('path_to_generated_images', exist_ok=True)
    # In a real scenario, populate this folder with generated images
    # For this example, we'll just create a dummy file
    with open('path_to_generated_images/dummy_gen.txt', 'w') as f: pass

# Compute FID between two image folders
fdir1 = "path_to_real_images"
fdir2 = "path_to_generated_images"

# Note: For actual FID computation, these folders need to contain actual images.
# The dummy files above will cause an error when clean-fid tries to load images.
# This is a placeholder for a runnable example structure.

# Example of computing FID (will likely fail with dummy folders but shows API)
try:
    score = fid.compute_fid(fdir1, fdir2)
    print(f"Computed FID: {score}")
except Exception as e:
    print(f"Error computing FID (expected with dummy data): {e}")

# Example with pre-computed dataset statistics (requires actual dataset name like 'FFHQ')
# score_with_stats = fid.compute_fid(fdir2, dataset_name="FFHQ", dataset_res=1024, dataset_split="trainval70k")
# print(f"Computed FID with FFHQ stats: {score_with_stats}")

view raw JSON →