forge: stop the Garage mirror job from turning every forge deploy red #36

Merged

founder merged 1 commit from fix/garage-replicate-hetzner-only into main

2026-08-16 06:48:40 +00:00

founder commented

2026-08-16 06:17:42 +00:00

Owner

Why

deploy-forge-host runs report FAILURE (switch-to-configuration rc=4) even though the forge lands on the new generation. Run 17013 (2026-08-16) shows the mechanism end to end:

key-store-garage-replicate.service exits 1 by design after the Hetzner leg because the GCS archival leg can't run: /run/agenix/gcpServiceAccount exists but is 0 bytes. secrets/gcp/service-account.age has had an empty payload since the lossy rekey in 4f53e0f2 (2026-01-19) — the forge is a recipient and agenix decrypts it fine; the script's -s test then reports it "missing". Founder decision 2026-07-25: Hetzner-only for now.
The unit's ExecStart embeds ${self}, so the unit is "modified" on every deploy. 17013 landed while the 05:28 hourly pass was still copying, so s-t-c SIGTERMed the in-flight rclone copy, started it again inline, blocked activation for the full 3.5-minute pass, and then reported the designed exit 1 as a failed unit → rc=4.

(Pre-existing failed units do not cause rc=4 — s-t-c calls ResetFailed before daemon-reload; only units that fail inside the switch window count. So the failure needs the deploy to overlap a mirror pass, or any other timer job that fails during the window.)

What

nixos/keystore-forge.nix: declare the decision as garageGcsArchiveEnabled = false → KEYSTORE_GCS_ARCHIVE_OPTIONAL=1 on the unit. Hetzner leg unchanged; a non-empty credential in the secret re-enables the GCS leg on the next tick with no config change; flip the flag once provisioned so a missing credential fails the unit again.
nixos/keystore-forge.nix: restartIfChanged = false on the unit — a timer-driven batch job should never be pulled into the activation path.
Scripts/forge-blob-replicate.sh: in optional mode, log the single-mirror state at journal WARNING priority (<4> prefix when JOURNAL_STREAM is set) every run and exit 0. Default mode unchanged (still exits 1). Documented the env var.
KSE-0177 + secrets/README.md: status note + the KSE-0182 direction (WIF external-account file, not a new service-account key).

Verification (on the forge, no activation)

nixos-rebuild dry-activate --flake .#keystore-forge on main+this patch: builds in 78 s (cached), rc=0.
Built unit vs live unit differs only by Environment="KEYSTORE_GCS_ARCHIVE_OPTIONAL=1" and X-RestartIfChanged=false; ExecStart resolves to the patched script.
With the mirror job running, switch-to-configuration dry-activate now prints would NOT stop the following changed units: key-store-garage-replicate.service (previously: stop + inline start).
Script exercised locally with a fake rclone in all four modes (default/empty → rc=1; optional/empty → rc=0 + warning; optional + journal → <4> prefix; optional + non-empty credential → GCS leg runs).

Not done here (needs a decision / separate work)

Prod switch: not dispatched — founder to confirm. Merging also fires deploy-web/deploy-api (push-to-main), harmless but noted.
Side finding: 28 other .age files in secrets/ also carry empty payloads from the same 4f53e0f2 rekey (e.g. identity/forge-ssh, stalwart/dkim/staging-2025, roles/jmap-bearer, nexmail/encryption-key, grafana/planetscale-*, finances/*, records/*). Detected by size = 102 + 110·recipients. Worth an audit.
redmine-dispatcher-smoke.service fails every 2 h run (mail leg, known); it can also poison a deploy if the switch overlaps its ~1m42s run.

🤖 Generated with Claude Code

## Why `deploy-forge-host` runs report FAILURE (`switch-to-configuration` rc=4) even though the forge lands on the new generation. Run 17013 (2026-08-16) shows the mechanism end to end: 1. `key-store-garage-replicate.service` exits 1 **by design** after the Hetzner leg because the GCS archival leg can't run: `/run/agenix/gcpServiceAccount` exists but is **0 bytes**. `secrets/gcp/service-account.age` has had an empty payload since the lossy rekey in `4f53e0f2` (2026-01-19) — the forge *is* a recipient and agenix decrypts it fine; the script's `-s` test then reports it "missing". Founder decision 2026-07-25: Hetzner-only for now. 2. The unit's ExecStart embeds `${self}`, so the unit is "modified" on every deploy. 17013 landed while the 05:28 hourly pass was still copying, so s-t-c SIGTERMed the in-flight rclone copy, started it again **inline**, blocked activation for the full 3.5-minute pass, and then reported the designed exit 1 as a failed unit → rc=4. (Pre-existing failed units do *not* cause rc=4 — s-t-c calls `ResetFailed` before `daemon-reload`; only units that fail inside the switch window count. So the failure needs the deploy to overlap a mirror pass, or any other timer job that fails during the window.) ## What - `nixos/keystore-forge.nix`: declare the decision as `garageGcsArchiveEnabled = false` → `KEYSTORE_GCS_ARCHIVE_OPTIONAL=1` on the unit. Hetzner leg unchanged; a non-empty credential in the secret re-enables the GCS leg on the next tick with no config change; flip the flag once provisioned so a missing credential fails the unit again. - `nixos/keystore-forge.nix`: `restartIfChanged = false` on the unit — a timer-driven batch job should never be pulled into the activation path. - `Scripts/forge-blob-replicate.sh`: in optional mode, log the single-mirror state at journal **WARNING** priority (`<4>` prefix when `JOURNAL_STREAM` is set) every run and exit 0. Default mode unchanged (still exits 1). Documented the env var. - KSE-0177 + `secrets/README.md`: status note + the KSE-0182 direction (WIF external-account file, not a new service-account key). ## Verification (on the forge, no activation) - `nixos-rebuild dry-activate --flake .#keystore-forge` on main+this patch: builds in 78 s (cached), rc=0. - Built unit vs live unit differs only by `Environment="KEYSTORE_GCS_ARCHIVE_OPTIONAL=1"` and `X-RestartIfChanged=false`; ExecStart resolves to the patched script. - With the mirror job **running**, `switch-to-configuration dry-activate` now prints `would NOT stop the following changed units: key-store-garage-replicate.service` (previously: stop + inline start). - Script exercised locally with a fake rclone in all four modes (default/empty → rc=1; optional/empty → rc=0 + warning; optional + journal → `<4>` prefix; optional + non-empty credential → GCS leg runs). ## Not done here (needs a decision / separate work) - **Prod switch**: not dispatched — founder to confirm. Merging also fires deploy-web/deploy-api (push-to-main), harmless but noted. - **Side finding**: 28 other `.age` files in `secrets/` also carry empty payloads from the same `4f53e0f2` rekey (e.g. `identity/forge-ssh`, `stalwart/dkim/staging-2025`, `roles/jmap-bearer`, `nexmail/encryption-key`, `grafana/planetscale-*`, `finances/*`, `records/*`). Detected by size = 102 + 110·recipients. Worth an audit. - `redmine-dispatcher-smoke.service` fails every 2 h run (mail leg, known); it can also poison a deploy if the switch overlaps its ~1m42s run. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

founder added 1 commit

2026-08-16 06:17:42 +00:00

forge: stop the Garage mirror job from turning every forge deploy red 36a3ab6218

deploy-forge-host runs report FAILURE (switch-to-configuration rc=4) even
when the forge lands on the new generation. Run 17013 shows the mechanism:

- key-store-garage-replicate.service exits 1 by design after the Hetzner
  leg because the GCS archival leg cannot run: /run/agenix/gcpServiceAccount
  exists but is 0 bytes. secrets/gcp/service-account.age has had an empty
  payload since the lossy rekey in 4f53e0f2 (2026-01-19); the forge IS a
  recipient, agenix decrypts it fine, and the script's `-s` test reports
  "missing". The founder accepted Hetzner-only on 2026-07-25.
- The unit's ExecStart embeds ${self}, so it is "modified" on every deploy.
  When a deploy lands mid-run (17013 did: the 05:28 hourly pass was still
  copying at 05:31), switch-to-configuration SIGTERMs the in-flight rclone
  copy, starts it again inline, blocks activation for the whole pass
  (3.5 min), and then reports the designed exit 1 as a failed unit -> rc=4.
  (Pre-existing failed units do NOT cause rc=4: s-t-c calls ResetFailed
  before daemon-reload; only units that fail inside the switch window do.)

Fix:
- nixos/keystore-forge.nix: declare the founder decision as
  `garageGcsArchiveEnabled = false`, wired to KEYSTORE_GCS_ARCHIVE_OPTIONAL=1
  on the unit. The Hetzner leg still runs; a non-empty credential in the
  secret re-enables the GCS leg on the next tick with no config change; flip
  the flag once provisioned so a missing credential fails the unit again.
- nixos/keystore-forge.nix: `restartIfChanged = false` on the unit so a
  timer-driven batch job is never pulled into the activation path (verified
  with the unit active: s-t-c dry-activate now says "would NOT stop the
  following changed units: key-store-garage-replicate.service").
- Scripts/forge-blob-replicate.sh: in optional mode, log the single-mirror
  state at journal WARNING priority (<4> prefix when JOURNAL_STREAM is set)
  every run and exit 0, so the degraded state stays visible in
  `journalctl -p warning`/Loki without failing the unit. Default mode is
  unchanged (still exits 1). Documented KEYSTORE_GCS_ARCHIVE_OPTIONAL.
- KSE-0177 + secrets/README.md: record the status and the KSE-0182
  direction (WIF external-account file, not a new service-account key).

Verified on the forge: nixos-rebuild dry-activate of main+this patch builds
(78 s, cached) and the built unit differs from the live one only by the two
added lines. Side finding, not touched here: 28 other .age files in
secrets/ also carry empty payloads from the same rekey.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>