key-store/key.store
15
0
Fork 0
Code Pull requests 18 Releases 180 Activity Actions 5

observability: chain-disk, restart-loop and Zcash chunk-store alerts (2026-08-26 chain disk-full outage) #241

Open
founder wants to merge 2 commits from claude/chain-health-alerts into main
pull from: claude/chain-health-alerts
merge into: key-store:main
Conversation 1 Commits 2 Files changed 7 +618 -5
founder commented 2026-09-07 06:32:40 +00:00
Owner
Copy link

Why

The 1.8 TB chain disk (/var/lib/key-store-chain, nvme1n1p1) filled to 100% around 2026-08-26 during a Reth resync. podman-key-store-zcash-mainnet-zebra, podman-key-store-ethereum-mainnet-{reth,lighthouse} and key-store-bitcoind crash-looped for ~9 days, key-store-zcash-chunk-producer logged rpc getblockcount: io: Connection refused on every pass, and https://zec.key.store/v1/mainnet/manifest.json stayed frozen at finalized_through 3459999. No alert fired, because:

  • the OpenTofu disk rules (HostDiskLow/HostDiskCritical) only watch /;
  • a crash-looping unit never sits in state=failed long enough for CriticalUnitFailed (for: 3m);
  • AnyUnitFailed selects type="service", a label value node_exporter never emits (it is the service Type=: oneshot/simple/notify…), so it has never been able to fire — two units are failed on the forge today with no alert;
  • nothing read the manifest at all.

What

All rules are Grafana-managed, Nix-provisioned next to the existing forge-health groups (services."key.store".grafana.alerting.chainHealth, default on; folder "key.store forge" → root policy → key.store dispatcher webhook), so they go live with the forge switch rather than a founder-gated cloud-tofu apply.

Rule Fires when Severity
Chain disk usage high > 85% used for 10 min (node_filesystem_*{mountpoint="/var/lib/key-store-chain"}) warning
Chain disk free space low < 100 GiB available for 5 min critical
Chain unit in a restart loop increase(node_systemd_service_restart_total[10m]) > 5 for any podman-key-store-*, key-store-bitcoind, key-store-zcash-chunk-producer critical
Zcash chunk producer passes failing (Loki) pass failed / follower pass failed in every 5-min window for 15 min warning
Zcash chunk-store manifest stale manifest generated_at older than 6 h (a pass rewrites it every 30 s) critical
Zcash chunk-store finalization stalled finalized_through unchanged for 30 h warning
Zcash chunk-store layer-1b follower lagging finalized_through - verified_through > 1000 for 6 h warning
Zcash chunk-store probe failing manifest unreadable/unparsable for 30 min warning

Chunk-store probe (services."key.store".zcash.chunks.probe, default on): key-store-zcash-chunks-probe.timer runs services/observability/zcash-chunks-manifest-textfile.sh every minute, reads the served manifest.json into key_store_zcash_chunks_manifest_* node_exporter textfile gauges and keeps its own finalized/verified "advanced at" clocks under StateDirectory. On a read failure it re-emits the last good gauges with probe_success 0, so the staleness clocks keep running instead of going NoData; OnFailure=key-store-failure-notice@ covers a broken probe.

Why not "finalized_through not advancing for > 6 h" literally: a compact range is 1000 blocks (~21 h at 75 s blocks), so finalized_through steps roughly once a day and a 6 h rule on it alone would page daily. The manifest is rewritten on every pass, so generated_at is the 6 h freshness signal (it would have caught the incident within 6 h; the 15-min "pass failed" rule within 15 min; the restart-loop rule within 10 min; the disk rules days earlier). The 30 h finalization rule and the follower-lag rule cover the cases where passes still complete but ranges or layer 1b stop.

Also:

  • https://zec.key.store/v1/mainnet/manifest.json added to the blackbox targets (serving path + TLS via the existing PublicEndpointDown / TlsCertificateExpiring).
  • infra/grafana/alerts.tf: AnyUnitFailed selects name=~".+[.]service" instead of the never-matching type="service". This part needs a founder-gated cloud-tofu (stack grafana) apply; everything else is Nix.
  • Docs: docs/infrastructure/forge-chain-nodes.md § Alerts (rule table, probe, runbook); services/observability/graphs-on-forge.md.
  • The hourly key-store-chain-storage-check df unit is kept as the journal-side record; the 85% page now comes from node_exporter.

Validation

  • nix-instantiate --parse on the changed modules; nix eval of the probe unit/timer, alerting.chainHealth, and the blackbox target list.
  • The provisioned YAML rendered (writeText stub) and parsed: 7 rules in key-store-chain-health, 1 in key-store-chain-health-logs, all with condition: C, data, for, severity and summary.
  • Every PromQL/LogQL expression parsed and evaluated against the forge Prometheus/Loki (/api/v1/query), including the corrected AnyUnitFailed selector (matches 410 service units; the old one matched 0).
  • Probe script exercised locally against the live manifest (first run, advance detection, missing file, invalid JSON, fresh state) and on the forge from the built store path into a temp dir: probe_success 1, generated_at/tip_height/finalized_through 3473999/verified_through 3472999 as served.
  • tofu fmt -check clean.
  • Forge bash /root/drybranch.sh chain-health claude/chain-health-alerts → key-store-dry-chain-health.service Result=success, new generation /nix/store/c3m3sb7risvx3q1vlgi8b7dhnzav9n9x-nixos-system-keystore-forge-26.05.20260907.51f5794c; the built system contains key-store-zcash-chunks-probe.{service,timer} and the grafana-env script installs the two new rule files next to the forge-health ones.

After merge

deploy-forge-host (switch) restarts grafana-env → grafana (provisioning re-read) and starts the probe timer; the manifest gauges appear within ~4 minutes. cloud-tofu grafana apply (founder) for the AnyUnitFailed fix.

🤖 Generated with Claude Code

## Why The 1.8 TB chain disk (`/var/lib/key-store-chain`, nvme1n1p1) filled to 100% around 2026-08-26 during a Reth resync. `podman-key-store-zcash-mainnet-zebra`, `podman-key-store-ethereum-mainnet-{reth,lighthouse}` and `key-store-bitcoind` crash-looped for ~9 days, `key-store-zcash-chunk-producer` logged `rpc getblockcount: io: Connection refused` on every pass, and https://zec.key.store/v1/mainnet/manifest.json stayed frozen at `finalized_through` 3459999. **No alert fired**, because: - the OpenTofu disk rules (`HostDiskLow`/`HostDiskCritical`) only watch `/`; - a crash-looping unit never sits in `state=failed` long enough for `CriticalUnitFailed` (`for: 3m`); - `AnyUnitFailed` selects `type="service"`, a label value node_exporter never emits (it is the service `Type=`: `oneshot`/`simple`/`notify`…), so it has never been able to fire — two units are failed on the forge today with no alert; - nothing read the manifest at all. ## What All rules are Grafana-managed, Nix-provisioned next to the existing forge-health groups (`services."key.store".grafana.alerting.chainHealth`, default on; folder "key.store forge" → root policy → `key.store dispatcher` webhook), so they go live with the forge switch rather than a founder-gated `cloud-tofu` apply. | Rule | Fires when | Severity | | --- | --- | --- | | Chain disk usage high | `> 85%` used for 10 min (`node_filesystem_*{mountpoint="/var/lib/key-store-chain"}`) | warning | | Chain disk free space low | `< 100 GiB` available for 5 min | critical | | Chain unit in a restart loop | `increase(node_systemd_service_restart_total[10m]) > 5` for any `podman-key-store-*`, `key-store-bitcoind`, `key-store-zcash-chunk-producer` | critical | | Zcash chunk producer passes failing (Loki) | `pass failed` / `follower pass failed` in every 5-min window for 15 min | warning | | Zcash chunk-store manifest stale | manifest `generated_at` older than 6 h (a pass rewrites it every 30 s) | critical | | Zcash chunk-store finalization stalled | `finalized_through` unchanged for 30 h | warning | | Zcash chunk-store layer-1b follower lagging | `finalized_through - verified_through > 1000` for 6 h | warning | | Zcash chunk-store probe failing | manifest unreadable/unparsable for 30 min | warning | **Chunk-store probe** (`services."key.store".zcash.chunks.probe`, default on): `key-store-zcash-chunks-probe.timer` runs `services/observability/zcash-chunks-manifest-textfile.sh` every minute, reads the served `manifest.json` into `key_store_zcash_chunks_manifest_*` node_exporter textfile gauges and keeps its own finalized/verified "advanced at" clocks under `StateDirectory`. On a read failure it re-emits the last good gauges with `probe_success 0`, so the staleness clocks keep running instead of going NoData; `OnFailure=key-store-failure-notice@` covers a broken probe. **Why not "finalized_through not advancing for > 6 h" literally:** a compact range is 1000 blocks (~21 h at 75 s blocks), so `finalized_through` steps roughly once a day and a 6 h rule on it alone would page daily. The manifest is rewritten on every pass, so `generated_at` is the 6 h freshness signal (it would have caught the incident within 6 h; the 15-min "pass failed" rule within 15 min; the restart-loop rule within 10 min; the disk rules days earlier). The 30 h finalization rule and the follower-lag rule cover the cases where passes still complete but ranges or layer 1b stop. Also: - `https://zec.key.store/v1/mainnet/manifest.json` added to the blackbox targets (serving path + TLS via the existing `PublicEndpointDown` / `TlsCertificateExpiring`). - `infra/grafana/alerts.tf`: `AnyUnitFailed` selects `name=~".+[.]service"` instead of the never-matching `type="service"`. This part needs a founder-gated `cloud-tofu` (stack `grafana`) apply; everything else is Nix. - Docs: `docs/infrastructure/forge-chain-nodes.md` § Alerts (rule table, probe, runbook); `services/observability/graphs-on-forge.md`. - The hourly `key-store-chain-storage-check` df unit is kept as the journal-side record; the 85% page now comes from node_exporter. ## Validation - `nix-instantiate --parse` on the changed modules; `nix eval` of the probe unit/timer, `alerting.chainHealth`, and the blackbox target list. - The provisioned YAML rendered (writeText stub) and parsed: 7 rules in `key-store-chain-health`, 1 in `key-store-chain-health-logs`, all with `condition: C`, data, `for`, severity and summary. - Every PromQL/LogQL expression parsed and evaluated against the forge Prometheus/Loki (`/api/v1/query`), including the corrected `AnyUnitFailed` selector (matches 410 service units; the old one matched 0). - Probe script exercised locally against the live manifest (first run, advance detection, missing file, invalid JSON, fresh state) and on the forge from the built store path into a temp dir: `probe_success 1`, `generated_at`/`tip_height`/`finalized_through 3473999`/`verified_through 3472999` as served. - `tofu fmt -check` clean. - Forge `bash /root/drybranch.sh chain-health claude/chain-health-alerts` → `key-store-dry-chain-health.service` `Result=success`, new generation `/nix/store/c3m3sb7risvx3q1vlgi8b7dhnzav9n9x-nixos-system-keystore-forge-26.05.20260907.51f5794c`; the built system contains `key-store-zcash-chunks-probe.{service,timer}` and the `grafana-env` script installs the two new rule files next to the forge-health ones. ## After merge `deploy-forge-host` (switch) restarts `grafana-env` → `grafana` (provisioning re-read) and starts the probe timer; the manifest gauges appear within ~4 minutes. `cloud-tofu grafana apply` (founder) for the `AnyUnitFailed` fix. 🤖 Generated with [Claude Code](https://claude.com/claude-code) 
The 1.8 TB chain disk (/var/lib/key-store-chain) filled to 100% around
2026-08-26 during a Reth resync; Zebra, Reth, Lighthouse and bitcoind
crash-looped for nine days, key-store-zcash-chunk-producer logged
"rpc getblockcount: io: Connection refused" on every pass and
zec.key.store served a manifest frozen at finalized_through 3459999. No
alert fired: the OpenTofu disk rules only watch "/", a crash-looping
unit never sits in state=failed long enough for CriticalUnitFailed,
AnyUnitFailed selects type="service" (a label value node_exporter never
emits), and nothing read the manifest.

- grafana module: alerting.chainHealth (default on) provisions two
  Grafana-managed groups next to forge-health, same folder / contact
  point / root policy: key-store-chain-health (chain disk > 85% used
  [warning] and < 100 GiB free [critical]; > 5 automatic restarts in
  10 min for podman-key-store-* / key-store-bitcoind /
  key-store-zcash-chunk-producer [critical]; manifest not rewritten for
  6 h [critical]; finalized_through not advancing for 30 h [warning];
  layer-1b follower more than one range behind for 6 h [warning]; probe
  failing 30 min [warning]) and key-store-chain-health-logs ("pass
  failed" in every 5-min window for 15 min, Loki [warning]).
- zcash module: chunks.probe (default on) -- key-store-zcash-chunks-probe
  timer runs services/observability/zcash-chunks-manifest-textfile.sh
  every minute: reads the served manifest.json into
  key_store_zcash_chunks_manifest_* textfile gauges and keeps its own
  finalized/verified "advanced at" clocks (finalized_through steps once
  per 1000-block range, ~21 h at 75 s blocks, so a 6 h rule on it alone
  would page daily; the manifest itself is rewritten every 30 s pass,
  hence generated_at is the 6 h freshness signal). On a read failure it
  re-emits the last good gauges with probe_success 0 so the staleness
  clocks keep running instead of going NoData.
- forge host: OnFailure=key-store-failure-notice@ on the probe;
  https://zec.key.store/v1/mainnet/manifest.json added to the blackbox
  targets (serving path + TLS through the existing PublicEndpointDown /
  TlsCertificateExpiring rules).
- infra/grafana/alerts.tf: AnyUnitFailed selects name=~".+[.]service"
  instead of type="service" (founder-gated cloud-tofu apply; two units
  are failed on the forge today with no alert).
- docs: forge-chain-nodes.md "Alerts" (rule table, probe, runbook);
  graphs-on-forge.md alerting summary.

Verified: nix-instantiate --parse; nix eval of the probe unit/timer,
chainHealth options and blackbox targets; the provisioned YAML rendered
(writeText stub) and parsed, 8 rules; every PromQL/LogQL expression
parsed against the forge Prometheus/Loki; the probe script exercised
against the live manifest (first run, advance, missing file, invalid
JSON, fresh state); tofu fmt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The "Zcash chunk-store layer-1b follower lagging" rule assumed
verified_through trails finalized_through by exactly one range and fired
only above 1000 blocks. With the producer fix in PR #242 the follower
verifies a range within a minute of its finalization and publishes the
state ahead of the next range as `next_state`, so verified_through equals
finalized_through in steady state; any lag held for 6 h now means the
follower has stopped. Merge after PR #242 so the lowered threshold never
observes the old producer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
founder commented 2026-09-07 23:10:01 +00:00
Author
Owner
Copy link

Pushed a follow-up commit lowering the layer-1b follower lagging threshold from gt 1000 to gt 0 (6 h) and updating the Alerts row: with #242 the producer verifies a range within a minute of its finalization, so verified_through == finalized_through is the steady state and any sustained lag means the follower stopped.

Merge order: #242 first, then this PR, so the lowered threshold never observes the old producer (which trailed by exactly one range and would have tripped it after 6 h).

Pushed a follow-up commit lowering the **layer-1b follower lagging** threshold from `gt 1000` to `gt 0` (6 h) and updating the Alerts row: with #242 the producer verifies a range within a minute of its finalization, so `verified_through == finalized_through` is the steady state and any sustained lag means the follower stopped. **Merge order: #242 first, then this PR**, so the lowered threshold never observes the old producer (which trailed by exactly one range and would have tripped it after 6 h).
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin claude/chain-health-alerts:claude/chain-health-alerts
git switch claude/chain-health-alerts
Sign in to join this conversation.
Reviewers
No reviewers
windows
runner
assistant
devex
android
agent
cmo
linux
cdo
billing
manager
forge
web
release-runner
ios
security-agent
infra
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
key-store/key.store!241
No description provided.