Triton Inference Server Python Client

2.67.0 · active · verified Sun Apr 05

The `tritonclient` library provides Python APIs for interacting with NVIDIA Triton Inference Server. It supports both HTTP/REST and gRPC protocols, allowing applications to send inference requests, retrieve server and model status, manage models, and perform other tasks. Currently at version 2.67.0 (released March 27, 2026), it is actively maintained with a release cadence that generally aligns with the broader Triton Inference Server project.

Warnings

gotcha When using `InferInput` or `InferRequestedOutput`, ensure you import them from the correct protocol submodule (`tritonclient.http` or `tritonclient.grpc`) corresponding to the client you are using. Mixing them will lead to errors.
Fix: Explicitly import `InferInput` and `InferRequestedOutput` from `tritonclient.http` or `tritonclient.grpc` as appropriate for your client instance.
gotcha For BYTES tensors (variable-length binary data/strings), it is recommended to use `numpy.object_` for the dtype of the NumPy array. While `numpy.bytes_` is supported for backward compatibility, `numpy.object_` is the preferred and more robust type.
Fix: When creating NumPy arrays for BYTES tensors, set the dtype to `np.object_`.
gotcha The gRPC client (`tritonclient.grpc.InferenceServerClient`) has known limitations where it does not support timeouts for model configuration and model metadata requests. The HTTP client may also not correctly respect timeouts under 1 second.
Fix: Be aware of these limitations. For gRPC, ensure your server is responsive to avoid indefinite waits. For HTTP, consider increasing timeout values if experiencing unexpected delays with short timeouts.
gotcha Avoid using `tritonclient.utils.cuda_shared_memory` APIs in multithreaded environments. There are known issues that can lead to instability until fixed by the underlying CuPy library.
Fix: If shared memory is required in multithreaded Python clients, consider using system shared memory (`tritonclient.utils.shared_memory`) or investigate alternatives until the CUDA shared memory issue is resolved in CuPy.
gotcha When communicating with decoupled models (models that can return multiple responses over time), the order of responses received by the streaming gRPC client may not always match the order in which they were sent by the backend for different requests.
Fix: If your application relies on response order for decoupled models, ensure you handle responses asynchronously and correlate them using request IDs or other unique identifiers.
gotcha Triton Client PIP wheels for ARM SBSA are not available on PyPI. Installing `tritonclient` via `pip` on ARM SBSA systems may result in an incorrect Jetson version of the library being installed, leading to compatibility issues.
Fix: For ARM SBSA, obtain the correct client wheel file directly from the ARM SBSA SDK image and install it manually, rather than relying on `pip install tritonclient`.

Install

pip install tritonclient Base installation
pip install tritonclient[all] With all optional dependencies (e.g., CUDA shared memory)

Imports

InferenceServerClient

from tritonclient.http import InferenceServerClient

InferenceServerClient

from tritonclient.grpc import InferenceServerClient

InferInput
```
from tritonclient.http import InferInput
```
InferInput is specific to HTTP or gRPC client, depending on which one you're using. Import from the correct submodule (http or grpc).
InferRequestedOutput
```
from tritonclient.grpc import InferRequestedOutput
```
InferRequestedOutput is specific to HTTP or gRPC client, depending on which one you're using. Import from the correct submodule (http or grpc).

InferenceServerException

from tritonclient.utils import InferenceServerException

ProtocolType

from tritonclient.utils import ProtocolType

Quickstart

This quickstart demonstrates how to initialize an HTTP client, check server readiness, prepare input tensors using NumPy, send an inference request to a hypothetical 'simple_model', and process the returned output. Remember to replace `TRITON_SERVER_URL`, `MODEL_NAME`, `MODEL_VERSION`, `INPUT_NAME`, and `OUTPUT_NAME` with your actual server and model details. For gRPC, import `tritonclient.grpc` instead and use `tritonclient.grpc.InferenceServerClient`.

import numpy as np
import tritonclient.http as tritonhttp
import os

TRITON_SERVER_URL = os.environ.get('TRITON_SERVER_URL', 'localhost:8000')
MODEL_NAME = 'simple_model'
MODEL_VERSION = '1'
INPUT_NAME = 'input_0'
OUTPUT_NAME = 'output_0'

def main():
    try:
        # Create a Triton HTTP client
        client = tritonhttp.InferenceServerClient(url=TRITON_SERVER_URL)

        # Check server readiness
        if not client.is_server_ready():
            print(f"Triton server at {TRITON_SERVER_URL} is not ready.")
            return
        print(f"Triton server at {TRITON_SERVER_URL} is ready.")

        # Prepare input data (e.g., a simple numpy array)
        input_data = np.random.rand(1, 16).astype(np.float32)
        
        # Create InferInput object
        infer_input = tritonhttp.InferInput(INPUT_NAME, input_data.shape, 'FP32')
        infer_input.set_data_from_numpy(input_data, binary_data=True)

        # Create InferRequestedOutput object
        infer_output = tritonhttp.InferRequestedOutput(OUTPUT_NAME, binary_data=True)

        # Send inference request
        response = client.infer(
            model_name=MODEL_NAME,
            inputs=[infer_input],
            outputs=[infer_output],
            model_version=MODEL_VERSION
        )

        # Get output as numpy array
        output_data = response.as_numpy(OUTPUT_NAME)
        print(f"Inference successful! Output shape: {output_data.shape}")
        print(f"First 5 output values: {output_data.flatten()[:5]}")

    except Exception as e:
        print(f"An error occurred: {e}")

if __name__ == '__main__':
    main()

view raw JSON →