SGLang Router

Warnings

• breaking The metrics architecture has been redesigned, significantly changing metric names and structures. Users relying on Prometheus dashboards or alerting rules will need to update them when upgrading to newer versions of SGLang (which sglang-router interacts with).
  Fix: Update Prometheus dashboards and alerting rules to reflect the new 6-layer metrics architecture (protocol, router, worker, streaming, circuit breaker, policy) and unified error codes.
• breaking Worker resource management now uses UUIDs instead of network endpoints for identification. This is a breaking change for systems that directly manage or monitor workers based on their network addresses.
  Fix: Update any custom worker management or monitoring logic to use UUIDs for worker identification.
• gotcha Installing `sglang-router` in editable mode (`pip install -e .`) can lead to performance degradation. This is generally suitable for development but not recommended for performance testing or production environments.
  Fix: For performance-critical scenarios, always build and install the wheel package (`python -m build && pip install --force-reinstall dist/*.whl`) rather than using an editable install.
• gotcha Under high concurrent load (~32,768 connections), SGLang, including its router, may experience performance degradation and request failures due to underlying file descriptor limits. This is often a system-level limitation.
  Fix: Increase system-level file descriptor limits (ulimit) for the process running the SGLang Router and workers. Implement proper connection management, capacity limits, and graceful degradation strategies.
• breaking Migrating from SGLang backend versions 0.3.x to 0.5.x (which sglang-router 0.3.2 will likely interact with) may require configuration updates due to backward-incompatible changes in the SGLang core.
  Fix: Review SGLang migration guides and release notes when upgrading SGLang backend versions to ensure `sglang-router` configurations remain compatible.
• gotcha The project is actively evolving from a simple load balancer to a more comprehensive 'SGLang Model Gateway', which aims to provide a full OpenAI API server experience with advanced features like native tool calling, session management, and direct gRPC communication. This ongoing architectural shift may introduce new paradigms and potentially breaking changes in future major versions for users interacting with these advanced features.
  Fix: Stay informed about the official SGLang documentation and GitHub releases for updates on the Model Gateway's features and any migration paths.

Install

• pip install sglang-router Install stable version

Imports

• Router

  from sglang_router import Router

Quickstart

This quickstart demonstrates how to programmatically initialize the SGLang Router by providing a list of SGLang worker URLs. The router acts as a load balancer for these workers. For actual deployment and to handle incoming requests, the router is often launched as a separate process using the `python -m sglang_router.launch_router` command, or integrated into an ASGI application. Ensure that SGLang worker instances are running and accessible at the specified URLs for the router to function correctly.

import os
from sglang_router import Router

# NOTE: This quickstart assumes SGLang worker instances are running
# at the specified URLs (e.g., http://localhost:8000).
# Replace with actual worker URLs if available.
worker_urls = [
    os.environ.get('SGLANG_WORKER_URL_1', 'http://localhost:8000'),
    os.environ.get('SGLANG_WORKER_URL_2', 'http://localhost:8001')
]

try:
    # Initialize the SGLang Router
    # By default, it runs in regular HTTP routing mode.
    router = Router(worker_urls=worker_urls)
    print(f"SGLang Router initialized with workers: {worker_urls}")

    # In a real application, you would typically start the router
    # (e.g., in a separate thread or process) and then send requests to it.
    # For demonstration, we just show initialization.

    # Example of running the router process (requires a running event loop or main function)
    # This part is conceptual as `Router` doesn't expose a simple `run()` method directly
    # in this programmatic interface; it's often launched via `python -m`.
    print("Router instance created. To run, typically use 'python -m sglang_router.launch_router'\n"+
          "or integrate into an ASGI app. Refer to SGLang documentation for full deployment.")

except Exception as e:
    print(f"Error initializing SGLang Router: {e}")
    print("Ensure SGLang worker instances are running and accessible at the provided URLs.")