NVIDIA cuDNN Frontend

Warnings

• breaking The `nvidia-cudnn-frontend` Python package requires a compatible installation of the NVIDIA CUDA Toolkit and the cuDNN native C++ library on the system. Installing the Python package alone is insufficient for functionality.
  Fix: Ensure that the NVIDIA CUDA Toolkit and a supported version of the cuDNN backend library are correctly installed and configured on your system (e.g., `LD_LIBRARY_PATH` or system paths). Consult the official cuDNN documentation for installation instructions and compatibility matrix.
• gotcha Specific `nvidia-cudnn-frontend` versions are often recommended for particular `cuDNN` backend and `CUDA Toolkit` versions. Mismatched versions can lead to runtime errors, performance issues, or inability to leverage new features.
  Fix: Always refer to the release notes of your `nvidia-cudnn-frontend` version to identify the recommended `cuDNN` backend and `CUDA Toolkit` versions. For example, v1.22.1 is recommended for cuDNN 9.20.0 and later. Ensure your system setup matches these recommendations.
• breaking In version 1.18.0, the library internally transitioned away from using the older `v0.x API` and now directly calls the cuDNN backend API. This change might break compatibility for users who were relying on or interacting with internal `v0.x` API constructs.
  Fix: Review your code for any dependencies on internal `v0.x` API elements of `cudnn_frontend`. Update your code to use the officially exposed public API functions and methods, which now directly map to the backend calls.
• gotcha Version 1.19.1 was released to address issues with `pybind11` versions and restore `cuda-12` toolkit support accidentally dropped in 1.19.0. Older or incompatible `pybind11` installations can cause installation or runtime failures.
  Fix: If encountering issues, ensure you are using `nvidia-cudnn-frontend` v1.19.1 or later. If you manage `pybind11` versions explicitly, ensure compatibility or allow `pip` to manage it for `nvidia-cudnn-frontend`.

Install

• pip install nvidia-cudnn-frontend Install latest version

Imports

• cudnn_frontend

  import cudnn_frontend

Quickstart

This quickstart demonstrates how to initialize cuDNN frontend, define input tensors, create a convolution operation within a graph, build and execute the graph, and retrieve the output using PyTorch tensors on a CUDA-enabled GPU.

import cudnn_frontend
import torch

# Ensure CUDA device is available
if not torch.cuda.is_available():
    raise RuntimeError("CUDA is not available. This library requires a CUDA-enabled GPU.")

# Example: Create and execute a simple convolution graph
# Define input and weight tensors on CUDA
x = torch.randn(1, 1, 28, 28, device="cuda", dtype=torch.float32)
w = torch.randn(16, 1, 3, 3, device="cuda", dtype=torch.float32)

# Create a cuDNN frontend graph
graph = cudnn_frontend.create_graph({"fp8_mode": False}) # fp8_mode can be set to True for FP8 operations

# Make input tensors for the graph from PyTorch tensors
X = graph.make_input_tensor(
    "X", cudnn_frontend.DataType.FLOAT, x.shape, x.stride()
)
W = graph.make_input_tensor(
    "W", cudnn_frontend.DataType.FLOAT, w.shape, w.stride()
)

# Define a convolution operation
Y = graph.make_convolution(
    X, W, padding=[1, 1], stride=[1, 1], dilation=[1, 1]
)
# Mark the output tensor
Y.set_output()

# Build the operation graph, create execution plans, and check support
graph.build_operation_graph()
graph.create_execution_plans(
    [cudnn_frontend.heur_mode.A, cudnn_frontend.heur_mode.FALLBACK]
)
graph.check_support()
graph.build_plans()

# Allocate the output tensor on CUDA
y_out = torch.empty(Y.get_output_tensors()[0].get_dim(), device="cuda", dtype=torch.float32)

# Execute the graph
graph.execute([x, w], [y_out])

print("Graph execution successful!")
print(f"Output tensor shape: {y_out.shape}")