Observability phase A: span metrics + health/API/CI dashboards (KSE-0183)

founder commented

2026-07-18 08:43:35 +00:00

Owner

First increment of KSE-0183 (total observability). Adds spanmetrics+servicegraph connectors to the otel-collector (RED metrics + service map from existing API spans), Prometheus scrapes for the collector and Garage, and three OpenTofu dashboards: Service Health Matrix, API Throughput, Namespace CI.

Closes the named gaps (API throughput, Namespace CI have boards). Remaining per KSE-0183: PostHog board (post clean rebuild), Service Map + Blob/Data Plane boards, alerting, phase B client OTel.

Go-live needs a forge deploy + cloud-tofu run.

🤖 Generated with Claude Code

First increment of KSE-0183 (total observability). Adds spanmetrics+servicegraph connectors to the otel-collector (RED metrics + service map from existing API spans), Prometheus scrapes for the collector and Garage, and three OpenTofu dashboards: Service Health Matrix, API Throughput, Namespace CI. Closes the named gaps (API throughput, Namespace CI have boards). Remaining per KSE-0183: PostHog board (post clean rebuild), Service Map + Blob/Data Plane boards, alerting, phase B client OTel. Go-live needs a forge deploy + cloud-tofu run. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

founder added 1 commit

2026-07-18 08:43:35 +00:00

observability: KSE-0183 phase A — span metrics, health/API/CI dashboards

Release: If Needed / Check (Release Needed) (push) Successful in 6s

Details

Namespace: Build Images / Refresh macOS runner lane (push) Successful in 7s

Details

Namespace: Build Images / Refresh Linux runner image (push) Failing after 16s

Details

Infra: Cloud OpenTofu / Infra (Cloud OpenTofu) (push) Successful in 1m3s

Details

Deploy: API / Deploy (API) (push) Successful in 3m6s

Details

Deploy: Web / Deploy (Web) (push) Successful in 6m19s

Details

3d9dd365b0

Anchor total observability (KSE-0183): one Grafana across client->edge->API->
forge for logs, metrics, traces, and rolled-up health.

Phase A (no client changes, all data already on the forge):
- otel-collector: add spanmetrics + servicegraph connectors and a Prometheus
  exporter (127.0.0.1:8889). RED metrics and the service map are now derived
  from the OTLP spans the API already emits, without needing /metrics.
- Prometheus: scrape otel-spanmetrics (8889) and the Garage blob plane (3903).
- Three OpenTofu-managed dashboards: Service Health Matrix (up/down + RPS +
  errors + logs for every scraped service), API Throughput (RED by route from
  spanmetrics + Fly edge), and Namespace CI (runners, builds, failures).

Named gaps this closes: API throughput and Namespace CI now have boards; the
service map and Garage metrics are wired. Remaining per KSE-0183: PostHog
health board (after the clean rebuild), Traces/Service Map + Blob/Data Plane
boards, alerting, and phase B client instrumentation.

Go-live depends on a forge deploy (collector/scrapes) and cloud-tofu run
(dashboards) — both gated on forge deploy-pipeline health.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

founder requested reviews from infra, release-runner, agent, manager, cmo, runner, security-agent, windows, forge, assistant, ios, android, web, devex, cdo, billing, linux

2026-07-18 08:43:35 +00:00

founder merged commit 3d9dd365b0 into main

2026-07-18 08:43:47 +00:00

Rows
Columns

Observability phase A: span metrics + health/API/CI dashboards (KSE-0183) #3