ONNX Runtime

Warnings

• breaking The `onnxruntime` and `onnxruntime-gpu` packages are mutually exclusive. Only one should be installed in a given Python environment. Installing both can lead to unexpected behavior or errors.
  Fix: Uninstall any conflicting packages (`pip uninstall onnxruntime onnxruntime-gpu`) before installing the desired version.
• breaking Since ONNX Runtime 1.10, execution providers (like CUDAExecutionProvider for GPU) must be explicitly specified when creating an `InferenceSession`. If not specified, it defaults to `CPUExecutionProvider` only. Older code that relied on implicit GPU usage will break or silently fall back to CPU.
  Fix: Pass a list of providers to `ort.InferenceSession(model_path, providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])`. Use `ort.get_available_providers()` to see what's available.
• gotcha Input data shapes and data types must precisely match the ONNX model's expected inputs, otherwise `onnxruntime` will raise `INVALID_ARGUMENT` errors. Common mistakes include incorrect dimensions or using `float64` instead of the expected `float32`.
  Fix: Inspect `session.get_inputs()` to verify expected `shape` and `type`. Ensure NumPy arrays are created with the correct `dtype` (e.g., `np.float32`). Reshape inputs using `np.reshape` or `np.expand_dims` as needed.
• gotcha When using `onnxruntime-gpu`, correct installation of the CUDA Toolkit and cuDNN libraries matching your `onnxruntime-gpu` version is crucial. Mismatched versions are a frequent cause of 'DLL not found' or 'Failed to create session' errors.
  Fix: Consult the ONNX Runtime documentation for the exact CUDA/cuDNN version requirements for your `onnxruntime-gpu` package. Ensure they are installed and discoverable in your system's PATH/LD_LIBRARY_PATH.
• deprecated The `generate()` API for generative AI models underwent breaking changes from ONNX Runtime GenAI 0.5.2 to 0.6.0, notably replacing `params.input_ids = input_tokens` with `generator.append_tokens(input_tokens)` and removing `generator.compute_logits()`.
  Fix: Update `onnxruntime-genai` code to use the new `append_tokens` method and remove `compute_logits` calls, as detailed in the migration guide.
• gotcha For pre- and post-processing steps using custom ONNX operators, the `onnxruntime-extensions` package must be separately installed and its custom operators registered with the `InferenceSession` via `session_options.register_custom_ops_library()`.
  Fix: Install `pip install onnxruntime-extensions` and use `so = ort.SessionOptions(); so.register_custom_ops_library(get_library_path()); sess = ort.InferenceSession(model, sess_options=so)` where `get_library_path` comes from `onnxruntime_extensions`.

Install

• pip install onnxruntime CPU Version (default)
• pip install onnxruntime-gpu GPU Version (CUDA 12.x)
• pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-11/pypi/simple/ GPU Version (CUDA 11.8)

Imports

• InferenceSession

  from onnxruntime import InferenceSession

• SessionOptions

  from onnxruntime import SessionOptions

• get_available_providers

  from onnxruntime import get_available_providers

• PyOrtFunction

  from onnxruntime_extensions import PyOrtFunction

  For using custom operators provided by `onnxruntime-extensions`.

Quickstart

This quickstart demonstrates how to load an ONNX model, inspect its inputs and outputs, prepare sample input data using NumPy, and perform inference using `onnxruntime.InferenceSession`. It also shows how to configure execution providers for CPU or GPU inference. A `model.onnx` file is required for this code to run; a comment in the code suggests how to create a simple dummy model using the `onnx` library.

import onnxruntime as ort
import numpy as np
import os

# NOTE: This example assumes you have an ONNX model file named 'model.onnx'.
# You can typically export models from frameworks like PyTorch or TensorFlow to ONNX format.
# For a runnable example, you'd need to create a dummy model:
# e.g., using ONNX library:
# import onnx
# from onnx import TensorProto
# from onnx.helper import make_model, make_node, make_graph, make_tensor_value_info
# X = make_tensor_value_info('input', TensorProto.FLOAT, [None, 2])
# Y = make_tensor_value_info('output', TensorProto.FLOAT, [None, 2])
# node = make_node('Add', ['input', 'input'], ['output'])
# graph = make_graph([node], 'simple-graph', [X], [Y])
# onnx_model = make_model(graph)
# onnx.save(onnx_model, 'model.onnx')

model_path = os.environ.get('ONNX_MODEL_PATH', 'model.onnx')

# 1. Create an InferenceSession
# For GPU, add providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
session = ort.InferenceSession(model_path, providers=ort.get_available_providers())

print("Model inputs:")
for input_meta in session.get_inputs():
    print(f"  Name: {input_meta.name}, Shape: {input_meta.shape}, Type: {input_meta.type}")

print("\nModel outputs:")
for output_meta in session.get_outputs():
    print(f"  Name: {output_meta.name}, Shape: {output_meta.shape}, Type: {output_meta.type}")

# 2. Prepare input data (example for a model expecting a float32 array)
# Assuming the first input expects a 2D float32 array, e.g., shape (1, 2)
input_name = session.get_inputs()[0].name
input_shape = [dim if isinstance(dim, int) else 1 for dim in session.get_inputs()[0].shape] # Handle dynamic shapes
input_data = np.random.randn(*input_shape).astype(np.float32)

# 3. Run inference
outputs = session.run(None, {input_name: input_data})

# 4. Process outputs
print(f"\nOutput data type: {outputs[0].dtype}")
print(f"Output shape: {outputs[0].shape}")
print(f"Output data (first 5 elements): {outputs[0].flatten()[:5]}")