{"id":8708,"library":"tensorrt-cu12","title":"NVIDIA TensorRT for CUDA 12","description":"TensorRT is a high-performance deep learning inference optimizer and runtime from NVIDIA. The `tensorrt-cu12` package provides the Python bindings specifically compiled for CUDA Toolkit 12.x. As of its latest version `10.16.1.11`, it supports optimizing and deploying trained deep learning models for faster inference on NVIDIA GPUs. Releases are frequent, typically aligning with major TensorRT core library and CUDA toolkit updates.","status":"active","version":"10.16.1.11","language":"en","source_language":"en","source_url":"https://github.com/nvidia/tensorrt","tags":["deep-learning","inference","gpu","nvidia","cuda","optimization","onnx"],"install":[{"cmd":"pip install tensorrt-cu12","lang":"bash","label":"Install for CUDA 12.x"}],"dependencies":[{"reason":"Often used for low-level CUDA interactions, especially if converting `pycuda` code. Not a direct dependency but a common companion for advanced use cases.","package":"cuda-python","optional":true}],"imports":[{"note":"The standard convention is to import `tensorrt` and alias it as `trt` for brevity.","wrong":"import trt","symbol":"tensorrt","correct":"import tensorrt as trt"},{"note":"Core classes like Logger, Builder, Network are directly under the `tensorrt` top-level package.","wrong":"from tensorrt.infer import Logger","symbol":"Logger","correct":"from tensorrt import Logger"}],"quickstart":{"code":"import tensorrt as trt\nimport os\n\n# A basic example: creating a TensorRT builder and network\n# Note: A real application would involve loading an ONNX/UFF model and building an engine.\n\n# Create a logger to track verbose output and errors\nTRT_LOGGER = trt.Logger(trt.Logger.WARNING)\n\ntry:\n    # Create a builder\n    builder = trt.Builder(TRT_LOGGER)\n    print(f\"TensorRT Builder created successfully. Max batch size: {builder.max_batch_size}\")\n\n    # Create an empty network definition. EXPLICIT_BATCH is crucial for modern TensorRT.\n    network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))\n    print(\"Network created with EXPLICIT_BATCH flag.\")\n\n    # Example: Add an input layer (simplified, a real model would have specific shapes)\n    input_tensor = network.add_input(name='input_tensor', dtype=trt.float32, shape=(1, 3, 224, 224))\n    print(f\"Added input tensor with shape {input_tensor.shape}\")\n\n    # In a real scenario, you'd parse a model, e.g., using trt.OnnxParser(network, TRT_LOGGER)\n    # and then configure the builder for engine creation and serialization.\n\nexcept Exception as e:\n    print(f\"An error occurred: {e}\")\n    print(\"Please ensure you have a compatible NVIDIA GPU and the correct CUDA/cuDNN installations.\")\n\n# Clean up resources (important for complex applications)\n# Note: In a production script, `del` might not be strictly necessary if objects go out of scope,\n# but it's good practice for clarity or long-running processes.\n# Also, ensure network and builder are valid objects before attempting to delete.\nif 'network' in locals() and network is not None: del network\nif 'builder' in locals() and builder is not None: del builder\nif 'TRT_LOGGER' in locals() and TRT_LOGGER is not None: del TRT_LOGGER\n","lang":"python","description":"This quickstart demonstrates the foundational steps of initializing TensorRT components: a logger, a builder, and a network. It serves as a basic sanity check for TensorRT installation and API access. A complete workflow would involve parsing an existing model (e.g., ONNX, UFF), configuring optimization profiles, and building an inference engine."},"warnings":[{"fix":"Upgrade Python to 3.10+ and ensure system CUDA is 12.x compatible. For CUDA 11.x, consider using `tensorrt-cu11` or an older `tensorrt` version.","message":"TensorRT 10.13.2 (and subsequent versions) dropped support for Python versions older than 3.10 and CUDA 11.x. Users on older environments must upgrade their Python interpreter or use an older `tensorrt-cu12` package version.","severity":"breaking","affected_versions":">=10.13.2"},{"fix":"Manually install NVIDIA CUDA Toolkit 12.x and cuDNN for CUDA 12.x from NVIDIA's developer website. Ensure `nvcc --version` shows CUDA 12.x and that the library paths are correctly set up for your OS.","message":"The `tensorrt-cu12` package requires a matching system-wide NVIDIA CUDA Toolkit and cuDNN installation (version 12.x) to be present and correctly configured (e.g., via `LD_LIBRARY_PATH` on Linux). These are *not* installed by `pip`.","severity":"gotcha","affected_versions":"all"},{"fix":"Review custom plugins or CUDA-interfacing code that uses `pycuda`. Consider migrating to `cuda-python` for future compatibility and to align with NVIDIA's recommended practices.","message":"TensorRT 10.14 deprecated `pycuda` usages in its samples and shifted towards `cuda-python`. While not a direct breaking change for the core API, it indicates a shift in recommended practices for low-level CUDA interaction.","severity":"deprecated","affected_versions":">=10.14"},{"fix":"Update custom TensorRT plugins to implement `IPluginV3` (or higher) to ensure future compatibility. Refer to the TensorRT developer guide for specific plugin API migration details.","message":"Several standard plugins (e.g., `cropAndResizeDynamic`, `DecodeBbox3DPlugin`) have been migrated from `IPluginV2` to `IPluginV3`, with `IPluginV2` versions being deprecated and scheduled for removal. Custom plugins implementing `IPluginV2` might need updates.","severity":"breaking","affected_versions":">=10.12"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Run `pip install tensorrt-cu12` to install the package.","cause":"The `tensorrt-cu12` package is not installed in the current Python environment or the environment is not activated.","error":"ModuleNotFoundError: No module named 'tensorrt'"},{"fix":"Ensure all necessary plugin libraries (.so or .dll files) are in a path accessible by TensorRT (e.g., `LD_LIBRARY_PATH` on Linux), and that custom plugins are built against the correct TensorRT version. For built-in plugins, verify your TensorRT installation is complete.","cause":"This error typically indicates that a required TensorRT plugin (either a built-in one or a custom one) is not available, is incompatible with the loaded engine/network, or its library could not be loaded.","error":"tensorrt.infer.NMSPlugin_TRT.NMSPlugin: no matching engine found for requested plugin"},{"fix":"Install NVIDIA CUDA Toolkit 12.x and ensure its `lib` directory (e.g., `/usr/local/cuda-12.X/lib64`) is included in your system's `LD_LIBRARY_PATH` (Linux) or `PATH` (Windows).","cause":"The system cannot find the CUDA runtime library, indicating either CUDA Toolkit 12.x is not installed or its library paths are not correctly configured for your system.","error":"libcudart.so.12.X: cannot open shared object file: No such file or directory"},{"fix":"Double-check the arguments passed to TensorRT API calls, especially for `builder.create_network()` and `builder.build_engine()`. Ensure the network object is properly created with valid flags (e.g., `EXPLICIT_BATCH`) and populated before being used by the builder.","cause":"This error often occurs when passing an incorrect type or `None` to a TensorRT function expecting a specific object, particularly when initializing the network or builder, or during engine building.","error":"TypeError: argument 'network' (unambiguous type name missing)"}]}