Prometheus Remote Writer

Common errors

• Error sending samples to remote storage

  cause This is a generic error often found in Prometheus logs (or client-side exceptions) indicating a failure to transmit data. Common reasons include incorrect remote write URL, invalid authentication credentials, network connectivity issues, or the remote endpoint being down or overloaded.
  fix Verify the `url` and `headers` (especially authentication tokens) provided to the `RemoteWriter` instance. Check network connectivity from the client to the remote endpoint. Ensure the remote endpoint (e.g., Mimir, Thanos, Grafana Cloud) is operational and accessible.

• 429 Too Many Requests

  cause The remote storage backend is rate-limiting the incoming requests from your client. This means the client is sending data faster than the remote storage can accept it or exceeding configured ingestion limits.
  fix Reduce the volume or rate of metrics being sent. Review and adjust the ingestion limits on your remote storage solution. Consider aggregating metrics or applying `write_relabel_configs` on the Prometheus server (if applicable) to drop high-cardinality or less critical metrics.

• connection refused

  cause The client attempted to connect to the remote write endpoint, but the connection was actively refused. This typically indicates that the target server is not running, the port is incorrect, or a firewall is blocking the connection.
  fix Double-check the `url` in `RemoteWriter` to ensure the hostname and port are correct. Verify that the remote write endpoint service is running and accessible on the specified port. Check any firewalls or network security groups that might be preventing the connection.

• authentication failed

  cause The credentials (e.g., API key, bearer token, username/password) provided in the `headers` or `auth` parameters are incorrect or unauthorized by the remote storage endpoint.
  fix Review the authentication configuration for your remote storage and ensure the exact credentials are being passed correctly in the `RemoteWriter`'s `headers` or `auth` parameters. Pay attention to case sensitivity and token expiration.

Warnings

• breaking The underlying Prometheus Remote Write protocol has a v2.0 specification that introduces breaking changes, specifically deprecating `prometheus.WriteRequest` in favor of `io.prometheus.write.v2.Request`. Future versions of this library or Prometheus itself might transition, requiring updates or explicit configuration for v2.0 endpoints.
  Fix: Monitor official documentation for `prometheus-remote-writer` for v2.0 support. If interacting with a v2.0-only endpoint, ensure the library version supports it or configure it explicitly if options become available.
• gotcha Prometheus, and by extension clients pushing to it via remote write, often drops data when a remote endpoint returns a `429 Too Many Requests` status code, rather than implementing retries. This can lead to silent data loss during periods of high load or transient outages at the remote storage.
  Fix: Implement client-side rate limiting or backoff mechanisms, or ensure your remote storage backend has sufficient capacity and robust handling for spikes. Monitor your remote write queue metrics on the Prometheus server or client side to detect dropped samples.
• gotcha Prometheus remote write queues can saturate, potentially leading to 'Out of Memory' (OOM) errors or dropped samples, especially if the remote endpoint is slow, unavailable, or the client's internal buffering is not adequately managed.
  Fix: Monitor the health and memory usage of the application sending metrics. Ensure the remote storage is healthy and responsive. If possible, configure queue capacity and timeout parameters in your client or Prometheus configuration to prevent excessive buffering.

Install

• pip install prometheus-remote-writer Install stable version

Imports

• RemoteWriter

  from prometheus_remote_writer import RemoteWriter

Quickstart

This quickstart demonstrates how to initialize the `RemoteWriter` with a target URL and an authorization header, then construct and send a list of time-series data points. Timestamps should be in milliseconds. Ensure `PROMETHEUS_REMOTE_WRITE_URL` and `PROMETHEUS_REMOTE_WRITE_TOKEN` environment variables are set or replace the placeholder values.

import os
import time
from prometheus_remote_writer import RemoteWriter

# Configure the remote write endpoint URL
REMOTE_WRITE_URL = os.environ.get('PROMETHEUS_REMOTE_WRITE_URL', 'https://example.com/prometheus/api/v1/write')
# Configure authentication (e.g., Bearer token)
AUTH_TOKEN = os.environ.get('PROMETHEUS_REMOTE_WRITE_TOKEN', 'YOUR_STATIC_TOKEN_HERE')

# Create a RemoteWriter instance
writer = RemoteWriter(
    url=REMOTE_WRITE_URL,
    headers={'Authorization': f'Bearer {AUTH_TOKEN}'}
)

# Prepare some sample data to send
# Prometheus Remote Write expects timestamps in milliseconds
current_time_ms = int(time.time() * 1000)

data_points = [
    {
        'metric': {'__name__': 'my_app_cpu_usage_total', 'host': 'server1', 'region': 'us-east-1'},
        'values': [9.5],
        'timestamps': [current_time_ms]
    },
    {
        'metric': {'__name__': 'my_app_memory_utilization_bytes', 'host': 'server1', 'region': 'us-east-1'},
        'values': [1024 * 1024 * 500],
        'timestamps': [current_time_ms]
    }
]

try:
    # Send the data
    writer.send(data_points)
    print(f"Successfully sent {len(data_points)} data points to {REMOTE_WRITE_URL}")
except Exception as e:
    print(f"Failed to send data: {e}")