{"id":9635,"library":"dagster-prometheus","title":"Dagster Prometheus Integration","description":"Dagster-prometheus is a library that provides an integration for Dagster with Prometheus, allowing users to expose Dagster run and asset metrics, as well as custom metrics from their ops and assets. It is part of the Dagster ecosystem and typically releases new versions in lockstep with Dagster core releases. The current version is 0.29.0, corresponding to Dagster core 1.13.0.","status":"active","version":"0.29.0","language":"en","source_language":"en","source_url":"https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-prometheus","tags":["data-orchestration","observability","metrics","prometheus","etl","data-pipeline"],"install":[{"cmd":"pip install dagster-prometheus","lang":"bash","label":"Install dagster-prometheus"}],"dependencies":[{"reason":"Core data orchestration platform, required for defining jobs and resources.","package":"dagster"},{"reason":"Python client for Prometheus, used internally by the resource to expose metrics.","package":"prometheus_client"}],"imports":[{"symbol":"make_prometheus_resource","correct":"from dagster_prometheus import make_prometheus_resource"}],"quickstart":{"code":"import os\nfrom dagster import Definitions, job, op\nfrom dagster_prometheus import make_prometheus_resource\n\n# The Prometheus resource starts an HTTP server and exposes the\n# prometheus_client registry for custom metrics.\nprometheus_resource = make_prometheus_resource(\n    port=int(os.environ.get(\"PROMETHEUS_PORT\", \"8000\")),\n    prefix=os.environ.get(\"PROMETHEUS_PREFIX\", \"dagster_\"),\n)\n\n@op\ndef emit_custom_metrics_op(context):\n    \"\"\"\n    An op that emits custom Prometheus metrics using the provided registry.\n    The HTTP server will be running as long as the resource is active (e.g., during a job run).\n    \"\"\"\n    registry = context.resources.prometheus_resource.registry\n    \n    # Create and increment a custom counter\n    my_counter = registry.get_or_create_metric(\n        \"my_custom_run_counter\",\n        \"A counter for runs of my_metrics_job\",\n        \"counter\"\n    )\n    my_counter.inc()\n    \n    # Create and set a custom gauge with labels\n    my_gauge = registry.get_or_create_metric(\n        \"my_custom_value_gauge\",\n        \"A gauge for a custom value\",\n        \"gauge\",\n        labelnames=[\"job_name\", \"op_name\"]\n    )\n    # Example value derived from run_id, ensures a different value each time for demo\n    example_value = float(int(context.run_id.split('-')[0], 16) % 100)\n    my_gauge.labels(job_name=context.job_name, op_name=context.op.name).set(example_value)\n\n    context.log.info(f\"Emitted custom metrics to port {context.resources.prometheus_resource.port}\")\n\n\n@job(resource_defs={\"prometheus_resource\": prometheus_resource})\ndef my_metrics_job():\n    emit_custom_metrics_op()\n\ndefs = Definitions(jobs=[my_metrics_job])\n\n# To run this example locally:\n# 1. Save the code as a Python file (e.g., `metrics_repo.py`).\n# 2. Run `dagster dev -f metrics_repo.py` in your terminal.\n# 3. In the Dagster UI (typically localhost:3000), navigate to the Launchpad for `my_metrics_job` and click 'Launch'.\n# 4. While the job is running (or if `dagster dev` is active), open your browser to `http://localhost:8000/metrics`\n#    (or the port specified in PROMETHEUS_PORT if set) to view the exposed metrics.\n#    You will see Dagster's default metrics along with 'dagster_my_custom_run_counter_total' and 'dagster_my_custom_value_gauge'.","lang":"python","description":"This example demonstrates how to define a Prometheus resource using `make_prometheus_resource`, attach it to a job, and then use the resource's exposed Prometheus registry within an op to emit custom metrics. The resource automatically starts an HTTP server for Prometheus to scrape metrics."},"warnings":[{"fix":"Consult the Dagster 1.0 migration guide for updating resource definitions. Ensure `dagster-prometheus` is updated to a version compatible with your Dagster core version.","message":"Dagster's resource API underwent significant changes with the 1.0 release. If you are upgrading from Dagster core versions prior to 1.0, your existing resource definitions, including those for `dagster-prometheus`, will likely need to be updated to the new resource definition syntax.","severity":"breaking","affected_versions":"All `dagster-prometheus` versions used with Dagster core < 1.0"},{"fix":"For persistent metrics, consider running the Prometheus resource in a long-lived Dagster deployment, or implement a Prometheus Pushgateway if metrics need to be pushed from ephemeral jobs.","message":"The Prometheus HTTP server started by `make_prometheus_resource` is bound to the lifespan of the Dagster process that initializes the resource (e.g., a specific job run process, or the `dagster dev` process). For continuous metric exposure in a production environment, ensure the Dagster process (e.g., `dagster-webserver`, `dagster-daemon`, or a persistent run process) is long-lived.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Always install `dagster-prometheus` alongside a compatible version of `dagster`. Generally, installing the latest `dagster-prometheus` with the latest `dagster` core version is recommended. For example, `dagster-prometheus==0.29.0` is designed for `dagster==1.13.0`.","message":"`dagster-prometheus` is a library within the Dagster monorepo. While its own versioning (0.x.y) is separate from Dagster core (1.x.y), it is developed and released in tandem. Using mismatched versions of `dagster-prometheus` and `dagster` core can lead to unexpected behavior or `ImportError`s.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-17T00:00:00.000Z","next_check":"2026-07-16T00:00:00.000Z","problems":[{"fix":"Run `pip install dagster-prometheus` to install the package.","cause":"The `dagster-prometheus` library has not been installed in your Python environment.","error":"ModuleNotFoundError: No module named 'dagster_prometheus'"},{"fix":"Provide a different port to `make_prometheus_resource` (e.g., `make_prometheus_resource(port=8001)`), or ensure no other application is listening on the default port.","cause":"The port specified for the Prometheus HTTP server (default 8000) is already being used by another process on your machine.","error":"OSError: [Errno 98] Address already in use"},{"fix":"Ensure that the `port` argument is provided when calling `make_prometheus_resource`, for example: `make_prometheus_resource(port=8000)`.","cause":"The `make_prometheus_resource` function requires the `port` argument to be explicitly provided or configured, especially if you're using a config system that overrides defaults.","error":"dagster._core.errors.DagsterInvalidConfigError: Error in config for resource \"prometheus_resource\": Field \"port\" is missing and not optional."}]}