{"id":7781,"library":"tensorrt-cu12-bindings","title":"TensorRT CUDA 12 Bindings","description":"TensorRT-cu12-bindings provides Python bindings for NVIDIA's TensorRT, a high-performance deep learning inference optimizer and runtime. This specific package targets CUDA 12.x environments. It is actively developed by NVIDIA, with frequent releases aligning with major TensorRT and CUDA versions, typically every few months.","status":"active","version":"10.16.1.11","language":"en","source_language":"en","source_url":"https://github.com/nvidia/tensorrt","tags":["deep learning","inference","gpu","nvidia","cuda","optimization","performance"],"install":[{"cmd":"pip install tensorrt-cu12-bindings","lang":"bash","label":"Install for CUDA 12.x"}],"dependencies":[{"reason":"Used for low-level CUDA API interactions in newer versions.","package":"cuda-python","optional":false},{"reason":"Standard for numerical operations and array handling.","package":"numpy","optional":false},{"reason":"Requires a compatible NVIDIA CUDA Toolkit (12.x) and driver installed on the system.","package":"NVIDIA CUDA Toolkit (system-level)","optional":false}],"imports":[{"symbol":"tensorrt","correct":"import tensorrt as trt"}],"quickstart":{"code":"import tensorrt as trt\nimport numpy as np\n\n# 1. Create a logger (TRT_LOGGER = trt.Logger(trt.Logger.INFO) for more verbose output)\nTRT_LOGGER = trt.Logger(trt.Logger.WARNING)\n\n# 2. Create builder, network, and configuration\nbuilder = trt.Builder(TRT_LOGGER)\n# Explicit batch is required for some features (e.g., dynamic shapes)\nnetwork = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))\nconfig = builder.create_builder_config()\n\n# Configure builder options\n# max_workspace_size: The maximum GPU memory size (in bytes) that TensorRT can use for temporary buffers.\nconfig.max_workspace_size = 1 << 20  # 1 MiB (adjust as needed for larger models)\n\n# 3. Define the network: a simple identity layer for demonstration\n# Input shape (batch_size, channels, height, width)\ninput_shape = (1, 3, 224, 224)\ninput_tensor = network.add_input(name=\"input_tensor\", dtype=trt.float32, shape=input_shape)\n\n# Add an identity layer as a simple example operation\nidentity_layer = network.add_identity(input_tensor)\noutput_tensor = identity_layer.get_output(0)\n\n# Mark the output tensor\nnetwork.mark_output(output_tensor)\noutput_tensor.name = \"output_tensor\"\n\n# 4. Build the engine\nprint(f\"Building TensorRT engine with input shape {input_shape}...\")\nengine = builder.build_engine(network, config)\n\nif engine:\n    print(\"TensorRT engine built successfully!\")\n    # Example: serialize the engine to disk\n    # with open(\"my_identity_engine.trt\", \"wb\") as f:\n    #     f.write(engine.serialize())\n    # print(\"Engine serialized to my_identity_engine.trt\")\nelse:\n    print(\"Failed to build TensorRT engine.\")\n\n# Cleanup\ndel network, builder, config, engine\n","lang":"python","description":"This quickstart demonstrates how to initialize the TensorRT builder, define a simple identity network with an explicit batch dimension, configure the builder, and build a TensorRT engine. This is the fundamental process for optimizing and compiling deep learning models for NVIDIA GPUs."},"warnings":[{"fix":"Verify your NVIDIA CUDA Toolkit and driver versions precisely match the `cuXX` suffix of the installed package. Upgrade or downgrade system CUDA components as necessary.","message":"The `tensorrt-cuXX-bindings` packages are tightly coupled with specific CUDA versions (e.g., `cu12` for CUDA 12.x). Using a mismatched CUDA driver or toolkit version on your system will lead to runtime failures or import errors.","severity":"breaking","affected_versions":"All `tensorrt-cuXX-bindings` packages."},{"fix":"For full compatibility and access to samples/demos, ensure you are running a CUDA 12.x (or newer) environment and Python 3.10 or newer.","message":"TensorRT 10.13.2 dropped official support for CUDA 11.X. Additionally, official samples and demos now require Python 3.10 or newer.","severity":"breaking","affected_versions":"TensorRT 10.13.2 and later."},{"fix":"If you maintain custom plugins, review the TensorRT documentation for plugin migration guides and update them to `IPluginV3` or later to ensure forward compatibility.","message":"Custom TensorRT plugins (e.g., those implementing `IPluginV2`) are being migrated to `IPluginV3`. Older plugin versions may be deprecated and removed in future releases.","severity":"breaking","affected_versions":"TensorRT 10.11 and later."},{"fix":"Update any custom Python code that directly uses `pycuda` for CUDA context management or memory operations to use `cuda-python` for alignment with official practices and future compatibility.","message":"Official TensorRT Python samples and tools have transitioned from `pycuda` to `cuda-python` for low-level CUDA interactions.","severity":"deprecated","affected_versions":"TensorRT 10.14 and later."},{"fix":"Access all TensorRT samples, demos, and associated scripts directly from the official NVIDIA TensorRT GitHub repository (github.com/nvidia/tensorrt/tree/main/samples).","message":"Starting with TensorRT 10.14, samples and demos are no longer included directly within the `tensorrt-cuXX-bindings` Python packages.","severity":"gotcha","affected_versions":"TensorRT 10.14 and later."}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Ensure you have installed the correct package using `pip install tensorrt-cu12-bindings` in your active Python environment.","cause":"The `tensorrt-cuXX-bindings` package is not installed or the Python environment is incorrect.","error":"No module named 'tensorrt'"},{"fix":"Upgrade your NVIDIA GPU drivers to a version compatible with CUDA 12.x. Check NVIDIA's official documentation for driver requirements for specific CUDA versions.","cause":"The NVIDIA CUDA driver installed on your system is too old or incompatible with the CUDA runtime version linked by the `tensorrt-cu12-bindings` package.","error":"[TRT] ERROR: [checkRuntime.cpp::checkRuntime::0] CUDA driver version is insufficient for CUDA runtime version"},{"fix":"After defining the computational graph, ensure you call `network.mark_output(output_tensor)` for all desired output tensors.","cause":"The TensorRT network definition was created but no output tensor was explicitly marked using `network.mark_output()`.","error":"[TRT] ERROR: Network must have at least one output."},{"fix":"Review your network definition for unsupported operations, ensure input dimensions are valid, and consider increasing `config.max_workspace_size` if memory is an issue. Check TensorRT documentation for layer support.","cause":"TensorRT's builder could not find an implementation for a specific layer or operation on the target hardware, possibly due to unsupported operations, invalid input shapes, or insufficient `max_workspace_size`.","error":"[TRT] ERROR: [builder.cpp::buildEngine::] Error Code 4: Internal Error (Could not find any implementation for node...)"},{"fix":"Check the return values of TensorRT API calls, especially after defining layers or adding inputs/outputs. This often indicates invalid parameters passed to the function or an internal TensorRT error preventing object creation.","cause":"A TensorRT API call (e.g., `network.add_input`, `network.add_convolution`) failed to create the intended object and returned `None`, but the subsequent code tried to operate on it as if it were a valid object.","error":"TypeError: argument of type 'NoneType' is not iterable (or similar NoneType errors after API calls)"}]}