TensorFlow Serving API

Warnings

• breaking Version compatibility between `tensorflow-serving-api` and the `tensorflow_model_server` binary is crucial. Mismatched versions can lead to protobuf deserialization errors or other unexpected client/server communication failures.
  Fix: Always align the `tensorflow-serving-api` Python package version with the version of the `tensorflow_model_server` (e.g., from Docker image `tensorflow/serving:2.19.1`). Consult the official TensorFlow Serving GitHub releases for version information.
• gotcha This library (`tensorflow-serving-api`) provides only the *client* API. It does not include the `tensorflow_model_server` itself, which is the actual server component that loads and serves your models. The server must be installed and run separately (typically via Docker or `apt-get`).
  Fix: Ensure `tensorflow_model_server` is running and accessible before attempting to connect with this API. For example, using Docker: `docker run -p 8500:8500 --name tfserving_test -v "$(pwd)/my_model_dir:/models/my_model" -e MODEL_NAME=my_model -t tensorflow/serving`.
• gotcha Models must be exported in the TensorFlow `SavedModel` format and stored in a versioned directory structure (e.g., `model_name/1/`, `model_name/2/`) for the `tensorflow_model_server` to load them correctly. Incorrect directory structure is a common source of 'Model not found' or 'No versions of servable found' errors.
  Fix: When saving your model, place it in a subdirectory named with an integer version number (e.g., `tf.saved_model.save(model, '/path/to/models/my_model/1')`). The server automatically picks up the highest version.
• gotcha When sending gRPC requests, input data (e.g., NumPy arrays) must be correctly converted into Protobuf `TensorProto` format. Incorrect type mapping or shape can lead to prediction errors on the server side.
  Fix: Use `tensorflow.python.framework.tensor_util.make_tensor_proto` or `np.ndarray._to_proto()` with appropriate data types (e.g., `np.float32`) to ensure compatibility with your model's input signature.
• gotcha The documentation specifically for the `tensorflow-serving-api` Python client can be sparse. Many guides focus on the server setup or REST API, requiring users of the gRPC Python client to infer usage from examples or C++ API definitions.
  Fix: Refer to official TensorFlow Serving examples on GitHub (e.g., `tensorflow/serving/tensorflow_serving/example`) and community blogs for comprehensive usage patterns, particularly for advanced scenarios or specific data types.

Install

• pip install tensorflow-serving-api Install latest version

Imports

• grpc

  import grpc

• prediction_service_pb2_grpc

  from tensorflow_serving.apis import prediction_service_pb2_grpc

• predict_pb2

  from tensorflow_serving.apis import predict_pb2

• get_model_status_pb2

  from tensorflow_serving.apis import get_model_status_pb2

• model_service_pb2_grpc

  from tensorflow_serving.apis import model_service_pb2_grpc

• TensorProto

  from tensorflow.core.framework import tensor_pb2

  TensorProto is part of the core TensorFlow framework, not directly within tensorflow_serving.apis.

Quickstart

This quickstart demonstrates how to construct and send a gRPC prediction request to a running TensorFlow Serving instance using the `tensorflow-serving-api` library. It covers setting up the gRPC channel and stub, creating a `PredictRequest`, populating it with input data converted to `TensorProto` format, and handling the response. Ensure that a TensorFlow Serving server is running and accessible at the specified address and port, with your model loaded, before attempting to run this client code. Replace `your_model`, `input_tensor_name`, and `output_tensor_name` with your actual model's details.

import grpc
import numpy as np
from tensorflow_serving.apis import prediction_service_pb2_grpc, predict_pb2
from tensorflow.core.framework import tensor_pb2
from tensorflow.python.framework import dtypes # for tf.float32, etc. (usually for older TF, or explicit type)

# NOTE: This quickstart assumes a TensorFlow Serving server is already running.
# For example, via Docker: 
# docker run -p 8500:8500 --name tfserving_test \
#  --mount type=bind,source=/path/to/your/model,target=/models/your_model \
#  -e MODEL_NAME=your_model -t tensorflow/serving & 

# Configuration for your model server
SERVER_ADDRESS = 'localhost:8500'
MODEL_NAME = 'your_model'
SIGNATURE_NAME = 'serving_default'

def make_prediction(input_data: np.ndarray):
    """Sends a prediction request to TensorFlow Serving via gRPC."""
    channel = grpc.insecure_channel(SERVER_ADDRESS)
    stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)

    request = predict_pb2.PredictRequest()
    request.model_spec.name = MODEL_NAME
    request.model_spec.signature_name = SIGNATURE_NAME

    # Convert NumPy array to TensorProto
    tensor_proto = tensor_pb2.TensorProto()
    tensor_proto.CopyFrom(np.asarray(input_data).astype(np.float32)._to_proto())
    
    # TensorFlow 2.x often requires explicit type, e.g., using a dtypes enum
    # tensor_proto.dtype = dtypes.float32.as_datatype_enum # Example for explicit type
    
    request.inputs['input_tensor_name'].CopyFrom(tensor_proto) # Replace 'input_tensor_name' with your model's input name

    try:
        response = stub.Predict(request, 10.0) # 10-second timeout
        print("Prediction successful:")
        # Process response.outputs to get results
        # Example: print(response.outputs['output_tensor_name'])
        for key, val in response.outputs.items():
            print(f"  Output '{key}': {np.array(val.float_val) if val.float_val else val}")

    except grpc.RpcError as e:
        print(f"Error making prediction: {e.code()} - {e.details()}")

if __name__ == '__main__':
    # Example usage: a simple 1x5 float array as input
    sample_input = np.array([[1.0, 2.0, 3.0, 4.0, 5.0]])
    make_prediction(sample_input)