A Prometheus exporter for Dagster that polls GraphQL instead of pushing to Pushgateway

Dagster doesn't expose a /metrics endpoint out of the box. The officially documented way to get Dagster metrics into Prometheus is the dagster-prometheus resource, which pushes metrics to a Pushgateway from inside a run. That works, but it has a structural blind spot: the push only happens if code inside the run gets to call it. A run that's OOM-killed, or crashes before reaching the push call, never reports anything. The failure is silent from Prometheus's point of view. A run sitting in QUEUED because of a run-queue concurrency limit hasn't started user code yet, so it can't push either — you can't see queue backlog forming. Prometheus's own docs are explicit that Pushgateway isn't meant to be a general substitute for pull-based scraping, only for short-lived batch jobs that genuinely can't be scraped. So I built dagster-prometheus-exporter: a small standalone Go binary that polls Dagster's GraphQL API on an interval and derives metrics from whatever state Dagster itself already has — including runs that never got a chance to push anything. Architecture Dagster (GraphQL) Prometheus --> Grafana The exporter keeps in-memory state, not an external store. Scraping (writing state) and serving /metrics (reading state) are decoupled, so if a GraphQL call is slow or fails, /metrics still serves the last-known state instead of breaking. Completed runs are fetched incrementally (watermark-based, not a full rescan every time), so the cost doesn't grow with total run history. What it currently covers Metric What it answers dagster_active_runs How many runs are queued/starting/started, per job dagster_active_run_duration_seconds How long the oldest active run in a job has been stuck there — useful for spotting stalls dagster_completed_runs_total / dagster_last_run_info / dagster_last_run_duration_seconds Success/failure counts and timing for completed runs dagster_run_queue_concurrency_key_backlog Queue backlog per dagster/concurrency_key tag — this is the one that needed the most digging, since instance.concurrencyLimits in the GraphQL schema looks like the answer but is actually a separate op-pool concurrency mechanism and reports 0 regardless of run-queue backlog. Ended up reading each queued run's own tags instead. dagster_code_location_load_error Whether a code location is currently failing to load (e.g. broken import) — independent of job-level metrics, since a broken location can't be inferred from run counts dagster_schedule_status / dagster_schedule_last_tick_status Whether a schedule is running, and its last tick outcome dagster_sensor_status / dagster_sensor_last_tick_status Same, for sensors dagster_exporter_build_info Which version/commit is actually running — handy once you have more than one exporter pod Full label reference is in the README. Trying it out git clone https://github.com/HirofumiTsuda/dagster-prometheus-exporter.git cd dagster-prometheus-exporter docker compose up --build brings up Dagster + the exporter + Prometheus + a pre-provisioned Grafana dashboard together. For an existing Dagster deployment, there's a published image and a Helm chart: docker run -p 9101:9101 -e DAGSTER_GRAPHQL_ENDPOINT=http://dagster:3000/graphql \ ghcr.io/hirofumitsuda/dagster-prometheus-exporter:0.2.0 helm install my-dagster-exporter oci://ghcr.io/hirofumitsuda/charts/dagster-prometheus-exporter \ --version 0.1.3 \ --set env.DAGSTER_GRAPHQL_ENDPOINT=http://dagster-webserver.dagster.svc.cluster.local/graphql What's not covered yet Asset materialization status. I've looked into the GraphQL shape for it (assetNodes + assetsLatestInfo) and the same kind of gotcha as the concurrency backlog shows up: AssetNode.assetMaterializations only records successful materializations, so detecting a failed one means cross-referencing assetsLatestInfo.latestRun.status instead. Tracked in #56 if anyone wants to compare notes. Repo: https://github.com/HirofumiTsuda/dagster-prometheus-exporter — issues and PRs welcome.