Neural Networks Compression Framework

Warnings

• breaking NNCFGraph, a core internal representation, was migrated from `nx.DiGraph` to `nx.MultiDiGraph` in v3.1.0 to support models with parallel/multi-edges. This can break code that directly interacts with NNCF's internal graph structure.
  Fix: Review any code that directly manipulates `NNCFGraph` objects and adapt it for `nx.MultiDiGraph` semantics.
• breaking The `nncf.CompressWeightsMode.CB4_F8E4M3` mode option was renamed to `nncf.CompressWeightsMode.CB4`.
  Fix: Update references to `nncf.CompressWeightsMode.CB4_F8E4M3` to `nncf.CompressWeightsMode.CB4`.
• breaking The `nncf.CompressWeightsMode.E2M1` mode option was renamed to `nncf.CompressWeightsMode.MXFP4`.
  Fix: Update references to `nncf.CompressWeightsMode.E2M1` to `nncf.CompressWeightsMode.MXFP4`.
• deprecated The TensorFlow backend is deprecated and will be removed in future releases. It is recommended to use PyTorch models for training-aware optimization and OpenVINO IR, PyTorch, or ONNX for post-training methods.
  Fix: Migrate TensorFlow-based NNCF workflows to PyTorch, OpenVINO IR, or ONNX backends.
• deprecated Several experimental NNCF methods including NAS, Structural Pruning, AutoML, Knowledge Distillation, Mixed-Precision Quantization, and Movement Sparsity are deprecated and will be removed in future releases.
  Fix: Consult NNCF documentation for alternative or officially supported compression methods.
• gotcha When using Quantization-Aware Training with NNCF, it is generally recommended to turn off Dropout layers (and similar layers like DropConnect) during training to prevent accuracy degradation.
  Fix: Ensure Dropout layers are disabled in your model's training pipeline when applying NNCF QAT.
• gotcha Users may encounter 'CUDA out of memory' errors during compression-aware training due to the increased GPU memory footprint of NNCF-compressed models. Additionally, `gcc`, `nvcc`, `ninja`, or `cl.exe` errors can occur if CUDA development tools are not properly installed or configured in the PATH/PYTHONPATH for PyTorch.
  Fix: Reduce batch size for NNCF training runs or ensure CUDA development tools (e.g., `nvcc` compiler) are installed and accessible in your environment variables.

Install

• pip install nncf Basic Install
• pip install nncf[openvino] For OpenVINO Backend
• pip install nncf[torch] For PyTorch Backend
• pip install nncf[tensorflow] For TensorFlow Backend (Deprecated)

Imports

• quantize

  import nncf
  quantized_model = nncf.quantize(model, calibration_dataset)

  The primary API for Post-Training Quantization across supported frameworks.
• compress_weights

  import nncf
  compressed_model = nncf.compress_weights(model)

  Used for data-free or data-aware weight compression, especially for LLMs.
• prune

  import nncf
  pruned_model = nncf.prune(model, config)

  Unified API for pruning algorithms, currently for PyTorch.
• NNCFConfig

  from nncf import NNCFConfig

• ModelType

  from nncf import ModelType

  Used in `nncf.quantize` to specify model architecture for better optimization.
• QuantizationPreset

  from nncf import QuantizationPreset

  Defines quantization modes (e.g., symmetric/asymmetric) for `nncf.quantize`.
• AdvancedQuantizationParameters

  from nncf.quantization.advanced_parameters import AdvancedQuantizationParameters

• IgnoredScope

  from nncf import IgnoredScope

• get_config

  from nncf.torch import get_config

  For saving PyTorch model compression configurations.
• load_from_config

  from nncf.torch import load_from_config

  For loading PyTorch model compression configurations.

Quickstart

This quickstart demonstrates how to perform 8-bit Post-Training Quantization (PTQ) on a pre-trained PyTorch model and convert it to an OpenVINO Intermediate Representation (IR) format using NNCF. It involves loading a model, creating a dummy calibration dataset, defining a transformation, and then applying `nncf.quantize`.

import nncf
import openvino as ov
import torch
from torchvision import datasets, transforms, models
import os

# 1. Load a pre-trained PyTorch model
model = models.resnet18(weights=models.ResNet18_Weights.IMAGENET1K_V1)
model.eval()

# 2. Convert PyTorch model to OpenVINO Model
# Create a dummy input for tracing
dummy_input = torch.randn(1, 3, 224, 224)
ov_model = ov.convert_model(model, example_input=dummy_input)

# 3. Prepare a calibration dataset (example with random data)
# In a real scenario, use representative data from your dataset
class RandomDataset(torch.utils.data.Dataset):
    def __init__(self, size=300):
        self.size = size
    def __len__(self):
        return self.size
    def __getitem__(self, idx):
        return torch.randn(3, 224, 224), 0 # dummy label

calibration_dataset = RandomDataset()

# 4. Define a transformation function for the calibration dataset
def transform_fn(data_item):
    return data_item[0].numpy() # NNCF expects NumPy array for OpenVINO PTQ

# 5. Apply Post-Training Quantization (PTQ)
print("Applying Post-Training Quantization...")
quantized_ov_model = nncf.quantize(
    ov_model,
    nncf.Dataset(calibration_dataset, transform_fn)
)

# 6. Save the quantized OpenVINO model
output_dir = "./quantized_model"
os.makedirs(output_dir, exist_ok=True)
model_path = os.path.join(output_dir, "resnet18_quantized.xml")
ov.save_model(quantized_ov_model, model_path)
print(f"Quantized model saved to {model_path}")

# To load and use the quantized model:
# core = ov.Core()
# loaded_model = core.read_model(model_path)
# compiled_model = core.compile_model(loaded_model, "CPU")
# # Inference goes here
# print("Model loaded and compiled for inference.")