zcash-chunks: verify each finalized range once — verified_through advances at once, next_state carries the state ahead of finalization (KSE-0193) #242

Merged

founder merged 1 commit from claude/zcash-chunks-follower-idle into main

2026-09-07 23:16:47 +00:00

founder commented

2026-09-07 23:08:47 +00:00

Owner

Problem

On the forge (2026-09-07) journalctl -u key-store-zcash-chunk-producer showed, every ~8 s, follower resuming from published state start=3473000 followed by layer 1b verified range=3473000 — 2,610 such lines in 6 h — while manifest.json reported finalized_through 3473999 and verified_through 3472999.

After verifying range N the follower sends the state before N+1000. The pass only attached it (and only then bumped verified_through) once the entry for N+1000 existed, i.e. once the next range was finalized (~21 h at 1000-block ranges / 75 s blocks). Until then verified_through stayed stale, run_once resumed from the stale published state, re-verified the same range (~8 s of Equihash + BLAKE2b + tree updates), returned applied=1, and run did continue instead of sleeping. Constant CPU burn, and the "layer 1b verified" line count was useless as a liveness signal.

Fix

Manifest: verified_through advances as soon as the follower reports a range. The state before verified_through + 1 is published as a new, optional, additive next_state (crates/zcash-chunks/src/manifest.rs) until that range is finalized, then moves onto the entry. gc treats it as live. Existing clients (verify-chain, zcash-wallet) only read the entries' state, which keeps its semantics: every finalized range whose start ≤ verified_through + 1 has one.
Follower (follow.rs): keeps its chain state in memory between passes (a Follower struct), resumes from next_state after a restart, falls back to the published states only when the manifest no longer describes the chain it was derived from (rewritten range / rebuilt manifest, checked via the entry's end_hash), and a pass with nothing new to apply returns 0 → sleeps the interval.
gc race (found while reviewing): the follower used to write the state object itself, while the pass thread runs gc; an object written mid-pass was collectable before the next pass referenced it, and only the busy loop's re-creation hid that. The follower now sends the encoded state through the channel and the pass writes it before publishing the reference.
Tests (cargo test -p zcash-chunk-producer, 10 tests; cargo test --locked --workspace green): verified-but-unfinalized case (merge + follower), restart via next_state, vanished object, replayed older state, manifest-mismatch fallback, pass-writes-before-publishing.
Docs: docs/infrastructure/forge-chain-nodes.md § Zcash Chunk Store (steady state is now verified_through == finalized_through; one "layer 1b verified" line per finalized range, ~1/day).

Follow-ups

The Grafana rule "Zcash chunk-store layer-1b follower lagging" lives on #241 and assumed a one-range lag (gt 1000 for 6 h); it gets lowered to gt 0 on that branch. Merge this PR before #241 so the lowered threshold never sees the old producer.
After deploy: expect one "layer 1b verified" line per newly finalized range and verified_through == finalized_through at https://zec.key.store/v1/mainnet/manifest.json (with next_state set).

🤖 Generated with Claude Code

## Problem On the forge (2026-09-07) `journalctl -u key-store-zcash-chunk-producer` showed, every ~8 s, `follower resuming from published state` start=3473000 followed by `layer 1b verified` range=3473000 — 2,610 such lines in 6 h — while `manifest.json` reported `finalized_through` 3473999 and `verified_through` 3472999. After verifying range N the follower sends the state *before N+1000*. The pass only attached it (and only then bumped `verified_through`) once the entry for N+1000 existed, i.e. once the *next* range was finalized (~21 h at 1000-block ranges / 75 s blocks). Until then `verified_through` stayed stale, `run_once` resumed from the stale published state, re-verified the same range (~8 s of Equihash + BLAKE2b + tree updates), returned `applied=1`, and `run` did `continue` instead of sleeping. Constant CPU burn, and the "layer 1b verified" line count was useless as a liveness signal. ## Fix - **Manifest**: `verified_through` advances as soon as the follower reports a range. The state before `verified_through + 1` is published as a new, optional, additive `next_state` (`crates/zcash-chunks/src/manifest.rs`) until that range is finalized, then moves onto the entry. `gc` treats it as live. Existing clients (`verify-chain`, `zcash-wallet`) only read the entries' `state`, which keeps its semantics: every finalized range whose start ≤ `verified_through + 1` has one. - **Follower** (`follow.rs`): keeps its chain state in memory between passes (a `Follower` struct), resumes from `next_state` after a restart, falls back to the published states only when the manifest no longer describes the chain it was derived from (rewritten range / rebuilt manifest, checked via the entry's `end_hash`), and a pass with nothing new to apply returns 0 → sleeps the interval. - **gc race** (found while reviewing): the follower used to write the state object itself, while the pass thread runs `gc`; an object written mid-pass was collectable before the next pass referenced it, and only the busy loop's re-creation hid that. The follower now sends the encoded state through the channel and the pass writes it before publishing the reference. - **Tests** (`cargo test -p zcash-chunk-producer`, 10 tests; `cargo test --locked --workspace` green): verified-but-unfinalized case (merge + follower), restart via `next_state`, vanished object, replayed older state, manifest-mismatch fallback, pass-writes-before-publishing. - **Docs**: `docs/infrastructure/forge-chain-nodes.md` § Zcash Chunk Store (steady state is now `verified_through == finalized_through`; one "layer 1b verified" line per finalized range, ~1/day). ## Follow-ups - The Grafana rule "Zcash chunk-store layer-1b follower lagging" lives on #241 and assumed a one-range lag (`gt 1000` for 6 h); it gets lowered to `gt 0` on that branch. **Merge this PR before #241** so the lowered threshold never sees the old producer. - After deploy: expect one "layer 1b verified" line per newly finalized range and `verified_through == finalized_through` at https://zec.key.store/v1/mainnet/manifest.json (with `next_state` set). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

founder added 1 commit

2026-09-07 23:08:47 +00:00

zcash-chunks: verify each finalized range once — verified_through advances at once, next_state carries the state ahead of finalization (KSE-0193)

Deploy: API / Deploy (API) (push) Failing after 3s

Details

Release: If Needed / Check (Release Needed) (push) Successful in 5s

Details

Deploy: Web / Deploy (Web) (push) Failing after 49s

Details

ee02d895c1

The layer-1b follower re-verified the newest finalized range on every
pass (~8 s of Equihash + BLAKE2b + tree updates, back to back: 2,610
"layer 1b verified" lines in 6 h on the forge). Its state object precedes
the *next* range, which the pass only attached once that range was
finalized ~21 h later, so manifest.verified_through stayed one range
behind, the follower resumed from the stale published state, applied the
same range again, returned applied=1 and looped without sleeping.

* The pass records verified_through as soon as the follower reports a
  range and publishes the state before verified_through + 1 as the
  manifest's new optional `next_state` (additive; clients only read the
  entries' `state`). It moves onto the range entry once that range is
  finalized; gc keeps it alive meanwhile.
* The follower keeps its chain state in memory between passes, resumes
  from next_state after a restart, falls back to the published states
  only when the manifest no longer matches (rewritten range, rebuilt
  manifest), and a pass with nothing new to apply sleeps the interval.
* The follower sends the encoded state through the channel and the pass
  writes it: the pass is the only thread that runs gc, so an object can
  no longer be collected between the follower writing it and the
  manifest referencing it (the busy loop hid that race by re-creating
  the object every pass).
* Unit tests: verified-but-unfinalized (merge + follower), restart via
  next_state, a vanished object, a replayed older state, the
  manifest-mismatch fallback, and the pass writing before publishing.

Steady state is now verified_through == finalized_through; the Grafana
rule "Zcash chunk-store layer-1b follower lagging" (PR #241) assumed a
one-range lag and gets its threshold lowered there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

founder requested reviews from ios, linux, web, cdo, runner, agent, billing, forge, release-runner, security-agent, windows, android, infra, devex, manager, assistant, cmo

2026-09-07 23:08:47 +00:00

founder referenced this pull request from a commit

2026-09-07 23:10:00 +00:00

observability: follower-lag rule fires on any lag — steady state is zero since the producer verifies a range once (PR #242)

founder referenced this pull request

2026-09-07 23:10:01 +00:00

observability: chain-disk, restart-loop and Zcash chunk-store alerts (2026-08-26 chain disk-full outage) #241

founder merged commit ee02d895c1 into main

2026-09-07 23:16:47 +00:00

founder deleted branch claude/zcash-chunks-follower-idle

2026-09-07 23:16:47 +00:00

founder commented

2026-09-07 23:27:18 +00:00

Author

Owner

Deploy state (2026-09-08, merged as ee02d895)

Not switched yet. origin/main also carries the Stalwart 0.16.18 production upgrade (5a6936bf, A0-1 lane with its own gated test/switch), so switching main would perform that upgrade as a side effect. Instead:

Deploy ref deploy/forge-producer-fix-20260908 = deployed 9d91017b + this fix (261b9443, cherry-pick, signed).
Dry-activated green on the forge (/root/drybranch.sh producer-fix …, 2 min 50 s): closure /nix/store/gckmxvswh39lj12ng4qn02rvsl9h7s42-nixos-system-keystore-forge-26.05.20260907.261b9443. nix store diff-closures vs /run/current-system: only zcash-chunk-producer 9d91017 → 261b944 (+ the system derivation). The crate tests ran inside the nix build (doCheck).
The switch itself (forge-side systemd-run … nixos-rebuild switch, mirroring deploy-forge-host) is blocked by the workstation's auto-mode classifier, and deploy-forge-host.yml cannot run on a non-main ref (OpenBao role pinned to main). Founder: switch the deploy ref (closure is already in the store, so it is fast) or switch main once A0-1 is through.

After the switch, expect in journalctl -u key-store-zcash-chunk-producer: one follower resuming from published state start=3474000, one layer 1b verified range=3474000, then nothing until the next range finalizes (~21 h); verified_through catches up to finalized_through and next_state appears in the manifest within a pass. Baseline before: 418 layer 1b verified lines/hour, ~1 core busy.

**Deploy state (2026-09-08, merged as `ee02d895`)** Not switched yet. `origin/main` also carries the Stalwart 0.16.18 production upgrade (`5a6936bf`, A0-1 lane with its own gated test/switch), so switching main would perform that upgrade as a side effect. Instead: - Deploy ref `deploy/forge-producer-fix-20260908` = deployed `9d91017b` + this fix (`261b9443`, cherry-pick, signed). - Dry-activated green on the forge (`/root/drybranch.sh producer-fix …`, 2 min 50 s): closure `/nix/store/gckmxvswh39lj12ng4qn02rvsl9h7s42-nixos-system-keystore-forge-26.05.20260907.261b9443`. `nix store diff-closures` vs `/run/current-system`: only `zcash-chunk-producer 9d91017 → 261b944` (+ the system derivation). The crate tests ran inside the nix build (`doCheck`). - The switch itself (forge-side `systemd-run … nixos-rebuild switch`, mirroring deploy-forge-host) is blocked by the workstation's auto-mode classifier, and deploy-forge-host.yml cannot run on a non-main ref (OpenBao role pinned to main). **Founder: switch the deploy ref** (closure is already in the store, so it is fast) or switch main once A0-1 is through. After the switch, expect in `journalctl -u key-store-zcash-chunk-producer`: one `follower resuming from published state` start=3474000, one `layer 1b verified` range=3474000, then nothing until the next range finalizes (~21 h); `verified_through` catches up to `finalized_through` and `next_state` appears in the manifest within a pass. Baseline before: 418 `layer 1b verified` lines/hour, ~1 core busy.