Matrice Inference Utilities

Common errors

• ModuleNotFoundError: No module named 'matrice_inference.base_inference'

  cause The `matrice-inference` package is not installed or the import path is incorrect.
  fix Ensure `matrice-inference` is installed in your active environment: `pip install matrice-inference`. Double-check the import statement for typos: `from matrice_inference.base_inference import BaseInferenceService`.

• TypeError: Can't instantiate abstract class MyService with abstract methods predict, warmup

  cause You have subclassed `BaseInferenceService` but have not implemented one or both of the required abstract methods (`warmup` and `predict`) or have not defined them as `async`.
  fix Implement both `async def warmup(self):` and `async def predict(self, request: RequestType) -> ResponseType:` in your service class, ensuring they are `async` functions.

• pydantic.v2.errors.PydanticImportError: Pydantic V1 is installed, but Matrice Inference requires Pydantic V2.

  cause Your Python environment has a conflicting version of Pydantic. `matrice-inference` specifically requires Pydantic v2 (e.g., `pydantic>=2.7.0,<2.8.0`).
  fix Upgrade Pydantic to the required v2 range: `pip install "pydantic>=2.7.0,<2.8.0" --upgrade`. If other packages prevent this, consider using a separate virtual environment or a dependency management tool to resolve conflicts.

• RuntimeError: Service not warmed up.

  cause Your custom `predict` method in `MyService` was called before the `warmup` method completed, or before the service's `is_ready` flag was set to True. The `create_app` function automatically sets up a startup event to call `warmup`, but manual calls or improper `is_ready` logic can lead to this.
  fix Ensure your `warmup` method correctly sets the service's ready state (e.g., `self.is_ready = True`) and that `create_app` is used, allowing FastAPI to manage the startup lifecycle. Do not call `predict` manually before the service has fully started.

Warnings

• breaking As a pre-1.0 library (version 0.1.x), `matrice-inference` does not guarantee API stability between minor releases. Expect frequent breaking changes to class signatures, function parameters, or module structures without explicit warnings in a changelog.
  Fix: Refer to the GitHub repository's latest source code for up-to-date API usage. Pin dependencies to exact versions (e.g., `matrice-inference==0.1.166`) and test thoroughly before upgrading.
• gotcha The `BaseInferenceService` is an abstract base class requiring all abstract methods (`warmup` and `predict`) to be implemented as `async` functions in your concrete service class. Forgetting `async` or not implementing a method will result in a `TypeError`.
  Fix: Ensure your service class properly subclasses `BaseInferenceService` and implements both `async def warmup(self):` and `async def predict(self, request: RequestType) -> ResponseType:` methods.
• gotcha `matrice-inference` has strict Pydantic v2 dependencies (e.g., `pydantic>=2.7.0,<2.8.0`). If your project uses Pydantic v1 or an incompatible Pydantic v2 range due to other dependencies, you will encounter `PydanticImportError` or other runtime issues.
  Fix: Use a dedicated virtual environment. Ensure all project dependencies are compatible with the required Pydantic v2 range. If conflicts arise, consider using tools like `pipdeptree` or `poetry` to resolve dependency graph issues, or temporarily isolate `matrice-inference` in a microservice.
• gotcha This library is primarily an internal utility for Matrice.ai. Its documentation for external users is minimal, and some assumptions about the operational environment or pre-existing infrastructure might not be explicitly stated, leading to unexpected behavior in different setups.
  Fix: Carefully review the source code for implicit configurations or dependencies. Be prepared to debug unexpected startup or runtime issues that might stem from environment mismatches. If possible, consult the maintainers for clarification on external use.

Install

• pip install matrice-inference Install stable version

Imports

• BaseInferenceService

  from matrice_inference.base_inference import BaseInferenceService

  This is the abstract base class for defining your custom inference logic.
• create_app

  from matrice_inference.api.app import create_app

  Function to create a FastAPI application based on your custom inference service.
• InferenceConfig

  from matrice_inference.config import InferenceConfig

  Pydantic model for service configuration.

Quickstart

This quickstart demonstrates how to create a basic inference service using `matrice-inference`. It involves defining custom request/response models, implementing `BaseInferenceService` with `warmup` and `predict` methods, configuring the service, and finally using `create_app` to generate a runnable FastAPI application. Save the code as `main.py` and run it with `uvicorn main:app --host 0.0.0.0 --port 8000`. You can then interact with the generated API at `http://localhost:8000/docs`.

import uvicorn
from fastapi import FastAPI
from pydantic import BaseModel
from typing import Any, TypeVar, Generic, Awaitable
import asyncio # Required for async in warmup/predict

from matrice_inference.api.app import create_app
from matrice_inference.base_inference import BaseInferenceService
from matrice_inference.config import InferenceConfig

# Define your custom request and response models
class MyInferenceRequest(BaseModel):
    text: str
    upper_case: bool = False

class MyInferenceResponse(BaseModel):
    processed_text: str
    original_length: int

# Implement your inference service
class MyService(BaseInferenceService[MyInferenceRequest, MyInferenceResponse]):
    def __init__(self, config: InferenceConfig):
        super().__init__(config)
        self.is_ready = False
        print(f"Service '{config.service_name}' initialized.")

    async def warmup(self):
        """Simulate loading a model."""
        print("Warming up MyService...")
        await asyncio.sleep(0.01) # Simulate async I/O
        self.is_ready = True
        print("MyService is ready.")

    async def predict(self, request: MyInferenceRequest) -> MyInferenceResponse:
        """Perform actual inference."""
        if not self.is_ready:
            raise RuntimeError("Service not warmed up.")

        processed_text = request.text
        if request.upper_case:
            processed_text = request.text.upper()

        return MyInferenceResponse(
            processed_text=processed_text,
            original_length=len(request.text)
        )

# Create a minimal configuration
inference_config = InferenceConfig(
    service_name="MyUpperCaseService",
    model_name="text_processor",
    model_version="1.0.0"
)

# Instantiate your service
my_service = MyService(inference_config)

# Create the FastAPI application
app: FastAPI = create_app(
    inference_service=my_service,
    request_model=MyInferenceRequest,
    response_model=MyInferenceResponse
)

# To run this, save as `main.py` and execute in your terminal:
# uvicorn main:app --host 0.0.0.0 --port 8000
# Then open http://localhost:8000/docs in your browser to test the API.