{"id":7417,"library":"mlserver","title":"MLServer","description":"MLServer is an open-source inference server for machine learning models, designed to serve any ML framework through a standard V2 inference protocol. It aims to provide a lightweight and performant solution for deploying models and supports both REST and gRPC endpoints. The current version is 1.7.1, and it is actively developed and maintained by SeldonIO with a regular release cadence.","status":"active","version":"1.7.1","language":"en","source_language":"en","source_url":"https://github.com/SeldonIO/MLServer","tags":["inference","model-serving","machine-learning","mlops","rest","grpc","seldon"],"install":[{"cmd":"pip install mlserver","lang":"bash","label":"Install core MLServer"},{"cmd":"pip install mlserver[all]","lang":"bash","label":"Install core MLServer with common runtimes (optional)"}],"dependencies":[],"imports":[{"note":"The primary base class for implementing custom ML models. The import path changed from `mlserver.model` to `mlserver` in 1.x.","wrong":"from mlserver.model import MLModel","symbol":"MLModel","correct":"from mlserver import MLModel"},{"note":"Required for defining input payloads according to the V2 inference protocol.","symbol":"InferenceRequest","correct":"from mlserver.types import InferenceRequest"},{"note":"Required for defining output payloads according to the V2 inference protocol.","symbol":"InferenceResponse","correct":"from mlserver.types import InferenceResponse"},{"note":"The entry point for MLServer's command-line interface, used to start the server.","symbol":"cli.main","correct":"from mlserver.cli import main"}],"quickstart":{"code":"from mlserver import MLModel\nfrom mlserver.types import InferenceRequest, InferenceResponse, ResponseOutput\nimport numpy as np\n\nclass MyModel(MLModel):\n    async def load(self):\n        # In a real scenario, load your model artifacts here\n        self.model = lambda x: x * 2 # A simple dummy function\n        self.ready = True\n\n    async def predict(self, request: InferenceRequest) -> InferenceResponse:\n        input_data = request.inputs[0].data.__root__\n        input_array = np.array(input_data).astype(np.float32)\n        output_array = self.model(input_array)\n        \n        return InferenceResponse(\n            outputs=[\n                ResponseOutput(\n                    name=\"output-0\",\n                    shape=output_array.shape,\n                    datatype=\"FP32\",\n                    data=output_array.tolist(),\n                )\n            ]\n        )\n\n# To run this model:\n# 1. Save the above code as `model.py` in an empty directory.\n# 2. Open your terminal in that directory.\n# 3. Ensure mlserver and numpy are installed: `pip install mlserver numpy`\n# 4. Run the MLServer: `mlserver start .`\n#\n# You can then send an inference request (e.g., using curl in a new terminal):\n# curl -X POST 'http://localhost:8080/v2/models/MyModel/infer' \\\n#      -H 'Content-Type: application/json' \\\n#      -d '{\n#            \"inputs\": [\n#              {\n#                \"name\": \"input-0\",\n#                \"shape\": [1, 2],\n#                \"datatype\": \"FP32\",\n#                \"data\": [10.0, 20.0]\n#              }\n#            ]\n#          }'","lang":"python","description":"This quickstart defines a simple `MyModel` that doubles its input. Save the code as `model.py`, then use the `mlserver start .` command to run the server. An example `curl` command is provided to demonstrate how to send an inference request to the running server."},"warnings":[{"fix":"Update `predict(self, payload: InferenceRequest)` to `predict(self, request: InferenceRequest)`.","message":"The `predict` method signature in `MLModel` changed between 0.x and 1.x. The parameter `payload` was renamed to `request` for clarity.","severity":"breaking","affected_versions":"0.x to 1.x"},{"fix":"Refer to the official migration guide for updated configuration parameters and environment variable names (e.g., `MLSERVER_MODEL_NAME` vs `MODEL_NAME`).","message":"MLServer's configuration (`settings.py`) and environment variable prefixes underwent significant changes in 1.x, simplifying the overall configuration schema.","severity":"breaking","affected_versions":"0.x to 1.x"},{"fix":"Install the required runtime: `pip install mlserver-sklearn` (replace `sklearn` with your framework).","message":"To serve models from specific frameworks (e.g., Scikit-learn, TensorFlow, XGBoost), you must install the corresponding MLServer runtime package (e.g., `mlserver-sklearn`, `mlserver-tensorflow`). The core `mlserver` package does not include these by default.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure all `ResponseOutput` objects in your `predict` method have correct and consistent `name`, `shape`, and `datatype` fields matching the data being sent/returned.","message":"MLServer strictly adheres to the V2 Inference Protocol. Inputs and outputs in `InferenceRequest` and `InferenceResponse` must correctly specify `name`, `shape`, and `datatype` fields, especially for custom models.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Ensure your model class is named `MyModel` (or matches `MLSERVER_MODEL_NAME` env var) and correctly subclasses `mlserver.MLModel`. Verify `model.py` is in the directory you're running `mlserver start` from, or specify its path.","cause":"The `model.py` file does not define a class that inherits from `mlserver.MLModel`, or the class is not discoverable (e.g., typo in class name, wrong file path).","error":"No MLModel class found in module 'model'"},{"fix":"Review the structure of your `InferenceRequest` to ensure it precisely matches the V2 protocol. Pay close attention to the `inputs` array's contents and their types.","cause":"The incoming `InferenceRequest` JSON payload does not conform to the V2 Inference Protocol specification (e.g., missing required fields like `name`, `shape`, `datatype` for inputs, or incorrect data types/shapes).","error":"RequestValidationError: 1 validation error for InferenceRequest"},{"fix":"Install the specific runtime package for TensorFlow: `pip install mlserver-tensorflow`.","cause":"You are attempting to serve a TensorFlow model, but the `mlserver-tensorflow` runtime library is not installed.","error":"ModuleNotFoundError: No module named 'mlserver_tensorflow'"},{"fix":"Add a unique `name` string to each `ResponseOutput` object within your `InferenceRequest` and `InferenceResponse` (e.g., `name='input-0'`, `name='output-0'`).","cause":"The V2 Inference Protocol mandates that all inputs and outputs have a 'name' field, which is missing in your request or model's response.","error":"Input '...' missing 'name' field"}]}