CuPy (CUDA 12.x)

Warnings

• breaking CuPy v14 updates its type promotion rules and casting behavior to align with NumPy v2 semantics. Code relying on NumPy v1 specific behaviors in earlier CuPy versions (v13 and prior) may behave differently.
  Fix: Review code for type-sensitive operations, especially when mixing dtypes or interacting with NumPy arrays, and adjust for NumPy v2 compatibility. Test thoroughly.
• breaking CuPy v14 drops support for CUDA 11 and Python 3.9. Additionally, all cuDNN-related functionality has been completely removed from CuPy. Users requiring cuDNN should consider external libraries like cuDNN Frontend.
  Fix: Upgrade to Python 3.10+ and a compatible CUDA Toolkit version (12.x for `cupy-cuda12x`). If cuDNN is needed, integrate an alternative Python binding for cuDNN.
• gotcha CuPy uses a memory pool for GPU allocations. This means GPU memory might not be immediately released back to the system even after arrays go out of scope, which can cause utilities like `nvidia-smi` to report higher memory usage than expected.
  Fix: While generally beneficial for performance, for long-running processes or when strict memory limits are needed, use `cp.get_default_memory_pool().free_all_blocks()` to explicitly release unused cached memory. Monitor fragmentation with `cp.cuda.get_memory_info()`.
• gotcha Frequent data transfers between CPU (host) and GPU (device) are a major performance bottleneck due to PCIe bandwidth limitations. Avoid 'round-tripping' arrays in hot loops.
  Fix: Keep data on the GPU as much as possible for computations. Perform all necessary operations on CuPy arrays directly. Transfer data to CPU only when the final result is needed or for visualization/storage.
• gotcha The first time a CuPy kernel is called for specific shapes and data types, it may experience a brief pause for JIT compilation. Subsequent calls with the same parameters will use the cached, pre-compiled kernel.
  Fix: This is expected behavior and generally not a problem in long-running applications. For benchmarking or time-critical initial runs, consider a warm-up execution or factor in the first-call compilation time.
• gotcha Installing a `cupy-cudaXX` package that does not match your system's CUDA Toolkit version or compatible driver can lead to import failures or runtime errors.
  Fix: Ensure the `cupy-cudaXX` package suffix matches your installed CUDA version (e.g., `cupy-cuda12x` for CUDA 12.x). If using PyPI `[ctk]` extras, only a compatible CUDA driver is required. Refer to CuPy's installation matrix for exact compatibility.
• breaking In CuPy v13+, the default behavior for transferring NumPy arrays backed by pinned memory from CPU to GPU (`cupy.array()`, `cupy.asarray()`) changed from blocking to asynchronous. This can improve performance but may introduce data races if the source array is modified on the CPU before the asynchronous transfer completes.
  Fix: If explicit blocking is required to prevent data races, use the `blocking=True` argument when calling `cp.array()` or `cp.asarray()`.

Install

• pip install cupy-cuda12x Base installation for CUDA 12.x
• pip install "cupy-cuda12x[ctk]" Installation with CUDA Toolkit component wheels (Python-managed CUDA dependencies)

Imports

• cupy

  import cupy as cp

  Standard alias, similar to 'import numpy as np'.
• cupyx.scipy

  import cupyx.scipy as sp

  For SciPy-compatible routines on GPU.

Quickstart

This quickstart demonstrates basic CuPy array creation, GPU computation, and data transfer between the GPU and CPU. It also includes a check for GPU availability and emphasizes `cp.cuda.Stream.null.synchronize()` for accurate benchmarking, as GPU operations are often asynchronous.

import cupy as cp
import numpy as np

# Check if a GPU is available
if cp.cuda.is_available():
    print("GPU is available. Current device:", cp.cuda.Device().id)

    # Create a CuPy array on the GPU
    x_gpu = cp.arange(10, dtype=cp.float32).reshape(2, 5)
    print("CuPy array on GPU:\n", x_gpu)

    # Perform an operation on the GPU
    y_gpu = x_gpu * 2 + 1
    print("Result of GPU operation:\n", y_gpu)

    # Transfer the result back to CPU (NumPy array)
    y_cpu = cp.asnumpy(y_gpu)
    print("Result on CPU (NumPy array):\n", y_cpu)

    # Example: Dot product
    a_gpu = cp.array([[1, 2], [3, 4]], dtype=cp.float32)
    b_gpu = cp.array([[5, 6], [7, 8]], dtype=cp.float32)
    c_gpu = a_gpu @ b_gpu
    print("\nMatrix multiplication on GPU:\n", c_gpu)

    # For accurate performance timings, ensure GPU operations complete
    cp.cuda.Stream.null.synchronize()
else:
    print("No GPU available. CuPy will not be functional.")