Azure Machine Learning Inference Server (HTTP)

1.5.1 · active · verified Thu Apr 16

The `azureml-inference-server-http` library provides the core HTTP server runtime for deploying machine learning models on Azure Machine Learning. It hosts user-defined model scripts (`score.py`) by loading `init()` and `run()` functions, allowing models to be exposed via a FastAPI-based REST API. As of version 1.5.1, it continues to evolve primarily through internal improvements and stability updates, with a focus on seamless integration into the Azure ML ecosystem.

Common errors

```
Failed to find 'init' function in entry script.
```
cause The `score.py` file does not contain a globally defined `init()` function, which is required by the inference server.

fix Ensure your `score.py` contains `def init():` at the module level. This function is called once when the server starts.
```
Failed to find 'run' function in entry script.
```
cause The `score.py` file does not contain a globally defined `run(raw_data)` function, which is required for processing inference requests.

fix Ensure your `score.py` contains `def run(raw_data):` at the module level. This function is called for each incoming inference request.
```
ModuleNotFoundError: No module named 'your_custom_package'
```
cause A Python package imported by your `score.py` (or its dependencies) is not present in the environment where the inference server is running.

fix For local testing, `pip install your_custom_package` in your active environment. For Azure ML deployments, ensure 'your_custom_package' is listed in your `conda_env.yml` (or `requirements.txt`) file specified during deployment.
```
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```
cause The `run(raw_data)` function received input that is not valid JSON, or `raw_data` was not correctly parsed as a string.

fix Ensure your client sends a valid JSON string in the request body (e.g., `Content-Type: application/json`). Inside your `run` function, add robust error handling for `json.loads(raw_data)`.

Warnings

gotcha This library is primarily a server runtime, not a client library for direct programmatic interaction with ML models. Users define `init()` and `run()` functions in a `score.py` file that this server loads and executes.
Fix: Focus on correctly structuring your `score.py` script and its environment, rather than attempting to import and use internal server components directly.
gotcha Local testing requires setting specific environment variables, most notably `AZUREML_ENTRY_SCRIPT`, to point the server to your `score.py` file.
Fix: Before running `python -m azureml.inference.server.http.http_server`, ensure `export AZUREML_ENTRY_SCRIPT=your_score_file.py` (or `set AZUREML_ENTRY_SCRIPT=your_score_file.py` on Windows) is executed.
breaking The exact signature requirements for `init()` and `run()` in `score.py` have been stable, but future versions might introduce subtle changes or new optional parameters.
Fix: Always refer to the official Azure ML documentation for the recommended `score.py` template and function signatures for your target Azure ML SDK and server version to ensure compatibility.
gotcha All Python package dependencies for your model code within `score.py` must be explicitly managed and installed into the server's environment. `pip install azureml-inference-server-http` does not install your model's dependencies.
Fix: When deploying to Azure ML, provide a `conda_env.yml` file. For local testing, manually `pip install` all necessary packages into your Python environment *before* running the server.

Install

pip install azureml-inference-server-http Install core server components

Imports

HttpServer
wrong:
```
from azureml.inference.server.http import HttpServer
```
correct:
```
Users define `init()` and `run()` functions in a `score.py` file.
```
This library provides the server runtime. User interaction is primarily by defining a `score.py` script with `init()` and `run()` functions, not by directly importing server components into application code.

Quickstart

This quickstart demonstrates how to create a minimal `score.py` file, which is the standard interface for models running on `azureml-inference-server-http`. It also provides the necessary shell commands to run this server locally for testing. The server exposes a `/score` endpoint for inference requests.

# Create a file named 'score.py'
# ---
# import json
# 
# def init():
#     global model
#     # In a real scenario, load your model here, e.g., from a file.
#     model = {"status": "initialized"}
# 
# def run(raw_data):
#     try:
#         data = json.loads(raw_data)
#         prediction = f"Model received input: {data.get('input', 'no input')} and is {model['status']}"
#         return json.dumps({"output": prediction})
#     except Exception as e:
#         return json.dumps({"error": str(e)})
# ---
# To run locally, save the above to 'score.py' and execute in your terminal:
# export AZUREML_ENTRY_SCRIPT=score.py
# python -m azureml.inference.server.http.http_server
# 
# Then, send a request to http://localhost:5001/score
# Example with curl:
# curl -X POST -H "Content-Type: application/json" -d '{"input": "example data"}' http://localhost:5001/score

view raw JSON →