{"id":6474,"library":"torch-fidelity","title":"Torch-Fidelity: Generative Model Metrics","description":"Torch-fidelity is a PyTorch library offering precise, efficient, and extensible implementations of popular generative model evaluation metrics, including Inception Score (ISC), Fréchet Inception Distance (FID), Kernel Inception Distance (KID), Perceptual Path Length (PPL), and Precision and Recall (PRC). It aims for epsilon-exact numerical fidelity with reference TensorFlow implementations. The library is actively maintained, with its latest version being 0.4.0, and has a steady release cadence with significant updates, like new metrics and feature extractors.","status":"active","version":"0.4.0","language":"en","source_language":"en","source_url":"https://www.github.com/toshas/torch-fidelity","tags":["generative models","metrics","FID","ISC","KID","PPL","PRC","PyTorch","GANs","diffusion models","evaluation","reproducibility"],"install":[{"cmd":"pip install torch-fidelity","lang":"bash","label":"Latest stable release"}],"dependencies":[{"reason":"Core deep learning framework.","package":"torch","optional":false},{"reason":"Used for standard datasets and transformations.","package":"torchvision","optional":true},{"reason":"Required for CLIP feature extractor dependencies.","package":"ftfy","optional":true},{"reason":"Required for CLIP feature extractor dependencies.","package":"regex","optional":true},{"reason":"Required for CLIP feature extractor dependencies.","package":"setuptools","optional":true},{"reason":"Required for CLIP feature extractor dependencies.","package":"clean-fid","optional":true}],"imports":[{"symbol":"torch_fidelity","correct":"import torch_fidelity"},{"symbol":"calculate_metrics","correct":"from torch_fidelity import calculate_metrics"},{"symbol":"GenerativeModelModuleWrapper","correct":"from torch_fidelity.generative_model_module_wrapper import GenerativeModelModuleWrapper"}],"quickstart":{"code":"import torch\nimport torch.nn as nn\nfrom torch_fidelity import calculate_metrics\nfrom torch_fidelity.generative_model_module_wrapper import GenerativeModelModuleWrapper\n\n# Dummy generator model for demonstration\nclass DummyGenerator(nn.Module):\n    def __init__(self, z_size, img_size, img_channels):\n        super().__init__()\n        self.img_size = img_size\n        self.img_channels = img_channels\n        self.main = nn.Sequential(\n            nn.Linear(z_size, 256),\n            nn.ReLU(),\n            nn.Linear(256, img_channels * img_size * img_size),\n            nn.Sigmoid()\n        )\n\n    def forward(self, z):\n        img = self.main(z)\n        return img.view(-1, self.img_channels, self.img_size, self.img_size)\n\n# Configuration\nz_size = 128\nimg_size = 32\nimg_channels = 3\nnum_samples = 10000 # Number of samples to generate for metrics\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n\n# Instantiate and wrap the generator\ngenerator = DummyGenerator(z_size, img_size, img_channels).to(device)\nwrapped_generator = GenerativeModelModuleWrapper(\n    generator, z_size, 'normal', 0, num_samples=num_samples, samples_batch_size=32\n)\n\n# Calculate metrics\n# For FID/KID, you need a second input, e.g., a real dataset name or directory path.\n# Here, we use a registered input 'cifar10-train' for demonstration.\nmetrics_dict = calculate_metrics(\n    input1=wrapped_generator,\n    input2='cifar10-train',\n    cuda=True if device.type == 'cuda' else False,\n    isc=True,\n    fid=True,\n    kid=True,\n    verbose=False,\n    save_cpu_ram=True # Optional: reduce GPU memory if needed\n)\n\nprint(metrics_dict)","lang":"python","description":"This quickstart demonstrates how to calculate Inception Score (ISC), Fréchet Inception Distance (FID), and Kernel Inception Distance (KID) using `torch-fidelity`'s Python API. It involves defining a dummy generative model, wrapping it with `GenerativeModelModuleWrapper`, and then passing it along with a reference input (like a pre-registered dataset 'cifar10-train') to the `calculate_metrics` function. The results are returned as a dictionary."},"warnings":[{"fix":"Update your API calls and CLI arguments to use the new parameter names: `kid_kernel_poly_degree`, `kid_kernel_poly_gamma`, `kid_kernel_poly_coef0`.","message":"In version 0.4.0, several KID-related API and CLI parameters were renamed for clarity. Specifically, `kid_degree`, `kid_gamma`, and `kid_coef0` are now `kid_kernel_poly_degree`, `kid_kernel_poly_gamma`, and `kid_kernel_poly_coef0` respectively.","severity":"breaking","affected_versions":"0.4.0+"},{"fix":"Ensure `input1` and `input2` are passed as keyword arguments (e.g., `input1=generator, input2='cifar10-train'`) and update CLI arguments according to the changelog.","message":"In version 0.3.0, the `calculate_metrics` function's `input1` and `input2` arguments changed from positional to keyword-only arguments. Several CLI arguments were also renamed, e.g., `--datasets-downloaded` to `--no-datasets-download` and `--cache-input1-name` to `--input1-cache-name`.","severity":"breaking","affected_versions":"0.3.0+"},{"fix":"Use `feature_extractor_compile=False` (default) if numerical precision is critical for your evaluation. Only enable it if you have thoroughly validated its impact on your specific use case.","message":"The `feature_extractor_compile` option is experimental and might negatively impact numerical precision, leading to discrepancies compared to reference values.","severity":"gotcha","affected_versions":"0.4.0+"},{"fix":"Understand that negative KID values are normal and reflect the mathematical properties of the metric.","message":"KID (Kernel Inception Distance) metric can mathematically produce negative values. This is expected behavior and not an indication of an error.","severity":"gotcha","affected_versions":"All"},{"fix":"Prefer lossless image formats (e.g., PNG) for evaluating generative models to maintain the highest numerical precision for metrics.","message":"Using lossy image formats (like JPG/JPEG) for inputs can affect metric precision and may trigger warnings from `torch-fidelity`.","severity":"gotcha","affected_versions":"All"},{"fix":"Be aware that direct comparison of InceptionV3 features with `torchvision`'s model might yield minor differences. For precise comparisons with TF-based results, rely on `torch-fidelity`'s built-in InceptionV3.","message":"`torch-fidelity`'s InceptionV3 implementation intentionally differs from `torchvision`'s and uses custom bilinear interpolation to ensure numerical fidelity with original TensorFlow implementations. This is crucial for comparing results with existing literature.","severity":"gotcha","affected_versions":"All"},{"fix":"To manage disk usage, specify alternative locations using the `--cache-root` and `--datasets-root` CLI arguments or the corresponding `cache_root` and `datasets_root` keyword arguments in `calculate_metrics`. Caching can be disabled with `--no-cache` or `cache=False` (not recommended for efficiency).","message":"The default cache (`fidelity_cache`) and dataset (`fidelity_datasets`) root directories, usually located under `$HOME` or `$ENV_TORCH_HOME`, can consume significant disk space.","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"last_verified":"2026-04-15T00:00:00.000Z","next_check":"2026-07-14T00:00:00.000Z"}