{"id":8435,"library":"polygraphy","title":"Polygraphy","description":"Polygraphy is a Deep Learning Inference Prototyping and Debugging Toolkit developed by NVIDIA. It helps users run, compare, and debug inference across various deep learning frameworks and backends, especially NVIDIA TensorRT. It provides Python APIs and a command-line interface for tasks like model conversion, correctness checking, and performance profiling. Polygraphy is tightly integrated with the TensorRT ecosystem and typically receives updates in conjunction with TensorRT releases.","status":"active","version":"0.49.26","language":"en","source_language":"en","source_url":"https://github.com/NVIDIA/TensorRT/tree/main/tools/Polygraphy","tags":["deep-learning","inference","tensorrt","onnx","onnx-runtime","debugging","profiling","nvidia","model-conversion"],"install":[{"cmd":"pip install polygraphy[all]","lang":"bash","label":"Full installation with all common backends"},{"cmd":"pip install polygraphy","lang":"bash","label":"Minimal installation"}],"dependencies":[{"reason":"Required for ONNX model manipulation and parsing.","package":"onnx","optional":true},{"reason":"Required for the ONNX Runtime backend (`OnnxrtRunner`). Installed via `polygraphy[onnxrt]` or `polygraphy[all]`.","package":"onnxruntime","optional":true},{"reason":"Required for the TensorRT backend (`TrtRunner`). Installed via `polygraphy[trt]` or `polygraphy[all]`. Often requires specific CUDA/Python wheels.","package":"nvidia-tensorrt","optional":true},{"reason":"Required for PyTorch framework integration. Installed via `polygraphy[torch]` or `polygraphy[all]`.","package":"torch","optional":true},{"reason":"Required for TensorFlow framework integration. Installed via `polygraphy[tf]` or `polygraphy[all]`.","package":"tensorflow","optional":true}],"imports":[{"symbol":"Comparator","correct":"from polygraphy.comparator import Comparator"},{"symbol":"TrtRunner","correct":"from polygraphy.backend.trt import TrtRunner"},{"symbol":"OnnxrtRunner","correct":"from polygraphy.backend.onnxrt import OnnxrtRunner"},{"note":"SizingInput is part of the comparator module, not backend.trt","wrong":"from polygraphy.backend.trt import SizingInput","symbol":"SizingInput","correct":"from polygraphy.comparator import SizingInput"},{"symbol":"G_LOGGER","correct":"from polygraphy.logger import G_LOGGER"}],"quickstart":{"code":"import numpy as np\nfrom polygraphy.comparator import Comparator, SizingInput\nfrom polygraphy.backend.trt import TrtRunner, TrtConfig\nfrom polygraphy.backend.onnxrt import OnnxrtRunner\nfrom polygraphy.logger import G_LOGGER\nimport os\n\n# --- Dummy ONNX Model Creation (for runnable quickstart) ---\n# In a real scenario, you would load your own ONNX model.\n# This creates a simple identity model for demonstration purposes.\nonnx_model_path = \"identity.onnx\"\nif not os.path.exists(onnx_model_path):\n    import onnx\n    graph = onnx.helper.make_graph(\n        [onnx.helper.make_node(\"Identity\", [\"input_0\"], [\"output_0\"])],\n        \"identity_graph\",\n        [onnx.helper.make_tensor_value_info(\"input_0\", onnx.TensorProto.FLOAT, [1, 3, 224, 224])],\n        [onnx.helper.make_tensor_value_info(\"output_0\", onnx.TensorProto.FLOAT, [1, 3, 224, 224])],\n    )\n    onnx_model = onnx.helper.make_model(graph, producer_name=\"polygraphy-quickstart\")\n    onnx.save(onnx_model, onnx_model_path)\n# --- End Dummy Model Creation ---\n\nG_LOGGER.severity = G_LOGGER.INFO # Set logging severity\n\n# Define input data for the model\ninput_data = {\n    \"input_0\": np.random.rand(1, 3, 224, 224).astype(np.float32)\n}\n\nwith Comparator() as c:\n    # Add a TensorRT runner\n    # Input shapes are required for TensorRT\n    c.add_runner(TrtRunner(\n        TrtConfig(),\n        input_shapes=[SizingInput(\"input_0\", (1, 3, 224, 224))]\n    ))\n\n    # Add an ONNX-Runtime runner\n    c.add_runner(OnnxrtRunner())\n\n    # Run the comparison\n    c.run(onnx_model_path, data=input_data)\n\n    # Results can be accessed via c.get_comparison_results()\n    print(\"Polygraphy comparison completed for\", onnx_model_path)\n\n# Clean up the dummy model\nos.remove(onnx_model_path)","lang":"python","description":"This quickstart demonstrates how to use Polygraphy's `Comparator` to run and compare inference on a dummy ONNX model using both TensorRT and ONNX Runtime backends. It highlights the need to specify input shapes for the TensorRT runner and how to set up basic logging."},"warnings":[{"fix":"Verify your system meets NVIDIA TensorRT requirements and that `nvidia-tensorrt` is correctly installed for your CUDA version. Refer to TensorRT documentation for setup. Consider using `OnnxrtRunner` for CPU-only model validation if GPU is not available.","message":"Polygraphy's core functionality, especially with the TensorRT backend, requires a compatible NVIDIA GPU, CUDA toolkit, and cuDNN installed. Without these, `TrtRunner` will fail or report errors.","severity":"gotcha","affected_versions":"All"},{"fix":"Ensure your Polygraphy version is compatible with your TensorRT installation. Check the official Polygraphy and TensorRT documentation for version compatibility matrices. Often, upgrading both simultaneously is the safest approach.","message":"Polygraphy is typically released alongside specific TensorRT versions. Mismatched Polygraphy and TensorRT versions can lead to unexpected behavior, `PolygraphyException` errors, or failed engine builds.","severity":"breaking","affected_versions":"All"},{"fix":"Upgrade to Python 3.10 or newer for full compatibility and best experience with the latest TensorRT releases and Polygraphy.","message":"TensorRT 10.13.2 and later (released mid-2025) dropped support for Python versions older than 3.10 for samples and demos. While Polygraphy's `requires_python` is `>=3.6`, using older Python versions (e.g., 3.6, 3.7, 3.8) with recent TensorRT backends may lead to unexpected issues or lack of features.","severity":"deprecated","affected_versions":">=0.49.x in conjunction with TensorRT 10.13.2+"},{"fix":"When initializing `TrtRunner`, pass a list of `SizingInput` objects or `TensorMetadata` to specify model input shapes. For dynamic shapes, use `SizingInput('input_name', (min_batch, C, H, W), (opt_batch, C, H, W), (max_batch, C, H, W))`.","message":"TensorRT requires explicit input shapes for engine building, even when dynamic axes are present. If your model has dynamic input shapes (e.g., batch size), you must provide `input_shapes` or `input_metadata` when using `TrtRunner`.","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Install the ONNX Runtime backend: `pip install polygraphy[onnxrt]`. For all common backends, use `pip install polygraphy[all]`.","cause":"The ONNX Runtime backend is not installed, or only the minimal `polygraphy` package was installed.","error":"polygraphy.exception.PolygraphyException: Could not find any ONNX backend. Make sure you have onnxruntime installed. Install it with: pip install polygraphy[onnxrt]"},{"fix":"Ensure your `nvidia-tensorrt` package matches your CUDA toolkit version. You might need to uninstall and reinstall `nvidia-tensorrt` using the specific wheel file corresponding to your CUDA version and Python. Refer to NVIDIA's TensorRT installation guide for details.","cause":"Your TensorRT Python package (`nvidia-tensorrt`) is compiled for a different CUDA toolkit version than what is installed on your system or configured in your environment.","error":"[ERROR] [TRT] ... This version of TensorRT was compiled for CUDA <X.Y> but was linked against CUDA <A.B> ..."},{"fix":"Inspect the TensorRT error message carefully for clues. Try converting the model with the command-line tool `polygraphy convert --trt --output=engine.plan model.onnx` to get more detailed error reports. Check if custom TensorRT plugins are required for specific operations, or if the model needs simplification/re-export to be TensorRT-compatible.","cause":"The ONNX model is not fully compatible with TensorRT (e.g., unsupported operations, incorrect input/output definitions, issues with dynamic shapes requiring optimization profiles, or plugin issues).","error":"polygraphy.exception.PolygraphyException: TrtRunner failed to build engine: [TensorRT] ERROR: ... (e.g., 'Unsupported engine field: ...', 'Error Code 7: Internal Error ...')"},{"fix":"Ensure that parameters expecting integer dimensions (e.g., `max_shapes`) are provided as plain integers or tuples of integers, not NumPy arrays. For dynamic input shapes in `TrtRunner`, use `SizingInput(..., min_shapes, opt_shapes, max_shapes)` with tuples for dimensions.","cause":"Often occurs when passing a NumPy array where a Python integer or tuple is expected, particularly when defining input shapes for dynamic models without correctly using `SizingInput`.","error":"TypeError: 'numpy.ndarray' object cannot be interpreted as an integer"}]}