hq: keep key-store-codex-login active (exited) to stop switch start-limit failures #277

Closed

infra wants to merge 6 commits from infra/hq-codex-login-remain into main

pull from: infra/hq-codex-login-remain

merge into: key-store:main

key-store:main

key-store:codex/zulip-ios-push

key-store:codex/founder-activity-rollups

key-store:codex/resident-chat-responsiveness

key-store:codex/common-hub-rollout

key-store:infra/office-sight-52

key-store:codex/subscriptions-hub-rollout

key-store:manager/role-prompt-corrections

key-store:codex/hub-everything

key-store:devex/kse-0199-steer-subscriptions

key-store:assistant/kse-check-next

key-store:infra/hq-codex-login-remain-clean

key-store:infra/sms-sender-routing-40

key-store:codex/assistant-wakeups

key-store:infra/goalert-oncall

key-store:linux/kse-0045-typography

key-store:security/security-txt-rfc9116

key-store:windows-campus-build

key-store:manager/campus-build

key-store:cmo-campus-build

key-store:web/guest-entry-spawn-401

key-store:android-campus-build

key-store:codex/resident-task-lifecycle

key-store:ios/swiftlint-gate-repair

key-store:linux/campus-build

key-store:billing-campus-build

key-store:cdo/office-launcher-arrival-states

key-store:codex/assistant-founder-routing

key-store:billing/vendor-register

key-store:cdo/kse-0045-reality-check

key-store:devex/check-workflows-lane

key-store:cdo/campus-build-home

key-store:codex/assistant-communications

key-store:codex/office-renderer-and-build-feedback

key-store:infra/campus-home

key-store:ios/campus-home

key-store:devex/campus-build

key-store:campus-security-agent-20260913

key-store:codex/minecraft-campus-and-guest-entry

key-store:devex/runner-namespace-smoke-reliability

key-store:linux/flatpak-metadata-validation

key-store:cmo-homepage-collect-minimum

key-store:android/ci-unit-tests-2

key-store:codex/resident-credential-acceptance

key-store:ios/apple-swiftlint-strict-gate

key-store:codex/resident-execution-acceptance

key-store:codex/zulip12-startup

key-store:codex/free-resident-inference

key-store:lately-files/fix-dead10cc

key-store:fix/agent-image-after-prune

key-store:lately-files/fix-nse-macos-entitlements

key-store:lately-files/fix-sign-digest

key-store:lately-files/phase23-landing

key-store:lately-files/fix-remove-account

key-store:deploy/forge-producer-fix-20260908

key-store:claude/chain-health-alerts

key-store:api-funding-secrets

key-store:lately-files/a0-1

key-store:lately-files/ci-fix

key-store:web/ci-bun-test-gate

key-store:cmo-support-and-not-found-pages

key-store:android/ci-unit-tests

key-store:codex/windows-agent-core-pr-gate

key-store:lately-files/phase1

key-store:zcash-fund-page

key-store:codex/inbox-native-oidc

key-store:codex/inbox-rollout-verification

key-store:codex/inbox-authentik-permissions

key-store:forge-chain-nodes-doc

key-store:forge-park-ethereum

key-store:zcash-funding

key-store:zcash-broadcast-rewrite

key-store:zcash-overlay

key-store:claude/lately-oidc-issuer-from-jmap-autodiscovery

key-store:claude/kse-0191-play-freshness

key-store:phase-f/wasm-core

key-store:phase-e/ios

key-store:phase2/d-overlay

key-store:phase2/c-adapters

key-store:phase2/g-runtime

key-store:phase2/ext-ts

key-store:phase1/transport

key-store:phase1/persist

key-store:phase1/peer-policy

key-store:phase0/lint

key-store:claude/kse-0191-play-reports-wif

key-store:claude/kse-0191-clickhouse-readonly

key-store:phase0/integrated

key-store:phase0/cg-core

key-store:claude/kse-0191-caddy-ingest-reload

key-store:phase0/a-core

key-store:phase0/33-keying

key-store:claude/kse-0191-hq-openbao-policy

key-store:agent-spine/phase0-base

key-store:claude/kse-0191-fmt-and-deploy-api-policy

key-store:claude/total-observability-v2

key-store:lately-calendar-and-mobile

key-store:claude/w6-android-02-optional

key-store:claude/jolly-mendel-809f3f

key-store:sqlite3mc/ci-and-docs

key-store:sqlite3mc/keystore-persistence

key-store:kse-0187-embedded-agent-core

key-store:push/integration

key-store:auth/shared-credentials

key-store:claude/rbe-executor-image-pin

key-store:claude/caches-d3-d4

key-store:claude/bazel-scope-d7

key-store:claude/forgejo-nsc-pin-92cd5bb0

key-store:claude/nix-drop-pin-oidc-bridge-off

key-store:stalwart-016

key-store:claude/nix-cache-harmonia

key-store:claude/namespace-oidc-trust

key-store:claude/namespace-runners-rightsize

key-store:publish-assets-distributables

key-store:publish-skip-ostree-repos

key-store:lately-sparkle-publish

key-store:lately-sparkle-embed

key-store:release/apple-20260816

key-store:winrunner-launcher-arg

key-store:winrunner-auto-dispatch

key-store:sparkle-appcast-xcode27

key-store:claude/flatpak-garage-remote

key-store:sparkle-region-and-key

key-store:claude/flatpak-share-nix

key-store:claude/flatpak-slim-shell

key-store:claude/flatpak-experimental-ungate

key-store:claude/epic-hellman-ad90c4

key-store:claude/kse-0186

key-store:claude/release-cancel-storm

key-store:lately-macos-default-on

key-store:claude/cranky-wu-a15bdd

key-store:ios-joint-build-lately

key-store:lately-ios-min-os-18

key-store:publish-lately-dmg-optional

key-store:windows-native-gate

key-store:claude/android-bridge-unwrapped-swift-driver

key-store:claude/routing-smoke-seed-stalwart

key-store:dmg-no-hide-extensions

key-store:lately-notification-arm64

key-store:lately-macos-preflight-fixes

key-store:lately-macro-plugin-linux

key-store:dispatch-vendor-hash

key-store:dispatch-mail-intake-deadlock

key-store:asc-profile-regenerate

key-store:lately-macos-target

key-store:lately-macos-gate

key-store:hq-codex-apikey-and-models

key-store:apple-release-build-fixes

key-store:hq-watchdog-no-steer

key-store:lately-web-core

key-store:swiftlint-zero

key-store:authentik-smoke-gawk

key-store:swiftlint-mechanical-fixes

key-store:swiftlint-exclude-vendored

key-store:codex-models-5-6

key-store:office-backups-bridge-and-codex-apikey

key-store:fix-inbox-smoke-jmap-host-and-gawk

key-store:mail-forwardemail-dns-and-sender

key-store:codex/release-loop-fix

key-store:codex/tofu-garage-endpoint

key-store:codex/tofu-state-to-garage

key-store:codex/garage-backup-integrity

key-store:codex/posthog-cloud-aligned

key-store:codex/posthog-latest-selfhosted

key-store:codex/disable-filc-default

key-store:codex/fix-otel-servicegraph-connector

key-store:codex/forge-deploy-oom-fix

key-store:codex/observability-phase-a

key-store:codex/fix-loki-garage-and-rclone

key-store:codex/session-forge-fixes-jmap-srv

key-store:codex/jmap-calendar-contacts-e2e

key-store:codex/jmap-calendar-contacts

key-store:codex/proxmox-openbao

key-store:codex/flatpak-testing-repo

key-store:codex/openbao-on-flatpak

key-store:codex/openbao-runtime-migration

key-store:codex/enable-jitsi

key-store:codex/password-manager-providers

key-store:codex/filnix-protected-utilities

key-store:codex/macos-agent-xpc

key-store:codex/piv-unlock-factor

key-store:codex/hardware-encrypt-decrypt

key-store:codex/windows-linux-hw-mtls

key-store:codex/kaios-client

key-store:codex/linux-ble-fido-passkey

key-store:codex/stripe-scim-saml-rdp-hardening

key-store:codex/azure-release-key-import

key-store:codex/lately-swiftui-previews

key-store:codex/zulip-access-fix

key-store:codex/identity-supervisor-live-fix

key-store:codex/linux-qr-bluetooth-account-access

key-store:codex/linux-security-key-sheet

key-store:codex/linux-fido2-account-creation

key-store:codex/linux-vault-unlock-factors

key-store:codex/ubuntu-touch-24-yubikey

key-store:codex/mail-stack

key-store:codex/testflight-build20-metadata

Conversation 1 Commits 6 Files changed 11 +2181 -2

infra commented 2026-09-14 01:49:32 +00:00

Member

Copy link

Problem (Redmine #47)

Two consecutive HQ generation switches (a8d14c50, 0b80bf41) failed activation with exit 4: key-store-codex-login.service went to Result=start-limit-hit even though every run exited 0 (ExecMainStatus=0). Manual systemctl reset-failed + start recovers; zero failed units afterwards.

Root cause (hypothesis, from repo inspection)

key-store-codex-login is Type=oneshot with no RemainAfterExit, so it returns to inactive/dead after every success. During switch-to-configuration each concurrent start source re-runs the script: the multi-user.target start phase plus the supervisor unit's wants=/after= on codex-login (nixos/modules/services/agent-dispatcher/default.nix). Enough rapid re-runs within the start-limit window trip StartLimitBurst. The secrets module only orders consumers after the materializer (After=/wants=) and only try-restarts restartUnits (this unit is in units with empty restartUnits), so rotation pickup stays manual either way.

Change

One option added in nixos/keystore-hq.nix: RemainAfterExit = true on key-store-codex-login. After a success the unit stays active (exited) and duplicate starts become no-ops. systemctl restart still re-executes, so the documented post-rotation manual re-run is unchanged. No secret, paging, watchdog, or dependency changes. Forge copy (nixos/hosts/keystore-forge/common.nix) has the same shape and is deliberately untouched — propose as follow-up if HQ validates.

Rollback

git revert this commit, rebuild + switch HQ. The unit returns to plain-oneshot behavior; worst case is a repeat of the current (recoverable via reset-failed + start) failure. No state migration: nothing persists besides the unit's active state.

On-host verification (needs root on HQ)

# 1. canonical eval/build of the HQ config (from sandbox: not possible, no Nix daemon)
nixos-rebuild dry-activate --flake .#keystore-hq   # or the canonical HQ build path
# 2. real switch; must exit 0 with no manual reset-failed
nixos-rebuild switch --flake .#keystore-hq
echo $?
systemctl is-failed key-store-codex-login.service   # expect: inactive (no 'failed')
systemctl show key-store-codex-login.service -p ActiveState,SubState,Result,ExecMainStatus
# 3. rotation semantics spot-check (manual re-run still works)
systemctl restart key-store-codex-login.service; echo $?
# 4. confirm trigger sources for the record
systemctl show key-store-codex-login.service -p StartLimitIntervalUSec,StartLimitBurst
journalctl -u key-store-codex-login.service --since '2026-09-14 01:30' --no-pager

Journald paths for review

• journalctl -u key-store-codex-login.service (bootstrap runs)
• journalctl -u openbao-secrets-materializer.service (secret ordering during switch)
• journalctl -u key-store-agent-dispatcher-supervisor.service (supervisor restarts pulling wants= starts)
• Prior evidence: /root/work/codex-assistant-wakeups-rollout-20260914/{switch-status.json,bootstrap-recovery.json} on HQ.

Sandbox validation done

• git diff reviewed (one option + comment, HQ file only).
• Full Nix eval/build NOT possible from this sandbox (no Nix daemon, read-only /nix/store) — canonical validation must happen on a privileged host before merge/switch.

Review requested: root (HQ owner). Do NOT merge without the on-host switch test in Redmine #47 acceptance.

## Problem (Redmine #47) Two consecutive HQ generation switches (a8d14c50, 0b80bf41) failed activation with exit 4: `key-store-codex-login.service` went to `Result=start-limit-hit` even though every run exited 0 (`ExecMainStatus=0`). Manual `systemctl reset-failed` + start recovers; zero failed units afterwards. ## Root cause (hypothesis, from repo inspection) `key-store-codex-login` is `Type=oneshot` with no `RemainAfterExit`, so it returns to inactive/dead after every success. During switch-to-configuration each concurrent start source re-runs the script: the multi-user.target start phase plus the supervisor unit's `wants=`/`after=` on codex-login (`nixos/modules/services/agent-dispatcher/default.nix`). Enough rapid re-runs within the start-limit window trip StartLimitBurst. The secrets module only orders consumers after the materializer (`After=`/`wants=`) and only try-restarts `restartUnits` (this unit is in `units` with empty `restartUnits`), so rotation pickup stays manual either way. ## Change One option added in `nixos/keystore-hq.nix`: `RemainAfterExit = true` on `key-store-codex-login`. After a success the unit stays `active (exited)` and duplicate starts become no-ops. `systemctl restart` still re-executes, so the documented post-rotation manual re-run is unchanged. No secret, paging, watchdog, or dependency changes. Forge copy (`nixos/hosts/keystore-forge/common.nix`) has the same shape and is deliberately untouched — propose as follow-up if HQ validates. ## Rollback `git revert` this commit, rebuild + switch HQ. The unit returns to plain-oneshot behavior; worst case is a repeat of the current (recoverable via reset-failed + start) failure. No state migration: nothing persists besides the unit's active state. ## On-host verification (needs root on HQ) ``` # 1. canonical eval/build of the HQ config (from sandbox: not possible, no Nix daemon) nixos-rebuild dry-activate --flake .#keystore-hq # or the canonical HQ build path # 2. real switch; must exit 0 with no manual reset-failed nixos-rebuild switch --flake .#keystore-hq echo $? systemctl is-failed key-store-codex-login.service # expect: inactive (no 'failed') systemctl show key-store-codex-login.service -p ActiveState,SubState,Result,ExecMainStatus # 3. rotation semantics spot-check (manual re-run still works) systemctl restart key-store-codex-login.service; echo $? # 4. confirm trigger sources for the record systemctl show key-store-codex-login.service -p StartLimitIntervalUSec,StartLimitBurst journalctl -u key-store-codex-login.service --since '2026-09-14 01:30' --no-pager ``` ## Journald paths for review - `journalctl -u key-store-codex-login.service` (bootstrap runs) - `journalctl -u openbao-secrets-materializer.service` (secret ordering during switch) - `journalctl -u key-store-agent-dispatcher-supervisor.service` (supervisor restarts pulling wants= starts) - Prior evidence: `/root/work/codex-assistant-wakeups-rollout-20260914/{switch-status.json,bootstrap-recovery.json}` on HQ. ## Sandbox validation done - `git diff` reviewed (one option + comment, HQ file only). - Full Nix eval/build NOT possible from this sandbox (no Nix daemon, read-only /nix/store) — canonical validation must happen on a privileged host before merge/switch. Review requested: root (HQ owner). Do NOT merge without the on-host switch test in Redmine #47 acceptance.

infra added 6 commits 2026-09-14 01:49:32 +00:00

KSE-0198: GoAlert on-call paging on HQ host (proposal, no cutover) ... ab485215bf

Pinned OCI module (goalert:v0.34.1, digest-verified), stdlib webhook
adapter with idempotent signed dispatcher mirror + Zulip mirror,
HQ-local postgresql_16 + daily backup, alerts.key.store DNS/Caddy,
OpenBao secret entries, dispatcher goalertWebhookSecretFile option.

Adapter tests: 4/4 pass (HMAC shape, dedupe, mapping, fail-closed).
Nix eval/build requires HQ host (no daemon socket in sandbox).
OIDC provider, image tarball sha256, /webhook/goalert route, and
cutover are separately-authorized follow-ups (see KSE-0198 §7).

Redmine #35. No deploy, no merge, no secret material.

KSE-0198: wire --goalert-secret-file, document runtime gap ... 465d3b933f

The option existed but was never passed to ExecStart. Now wired
conditionally (null default = flag absent, safe). Verified against
pinned dispatch_src rev caa2df9b: no /webhook/goalert route and no
--goalert-secret-file flag exist, so the secret must stay null until
the dispatcher runtime ships support (older runtime fails api
startup on the unknown flag). KSE-0198 §7 records the exact gap,
owner (dispatcher + security reviewer), and the no-false-independence
note. Redmine #35.

goalert: replace broken container wiring with pinned native unit (founder steer a7744ed8) ... f83f39a062

Container model was unworkable by construction: host-store entrypoint
script plus host bash/cat closure and host secret files are invisible
inside the OCI image, and the container-loopback db-url points at the
container netns instead of HQ PostgreSQL. Image ref was tag-only
despite digest comments. The pinned nixpkgs rev carries no goalert
package (verified zero hits), so the module wraps the pinned upstream
v0.34.1 tarball (URL + SRI) in a host-namespace systemd unit:
loopback --listen/--listen-prometheus/--status-addr,
LoadCredential db-url + data-encryption-key read by a store wrapper,
After postgresql + openbao-secrets-ready, loopback assertions, package
override option. OIDC is runtime config (no CLI flags exist per
upstream app/cmd.go) and stays a cutover step. HQ units entries
retargeted to goalert.service; KSE-0198 updated to the native model;
new stdlib wiring test pins the no-container/SRI/HQ-ordering shape
(12 tests green) alongside the existing adapter tests (4 green).

Review still needed: privileged eval/build (root), port/5432
confirmation, dispatcher route+flag follow-up, security review.

goalert: adapter v2 per founder steer — real upstream auth, required Zulip wiring, exact-2xx, PG17 ... ab570567fd

- adapter.py: secret URL path token (constant-time) replacing bearer —
  pinned upstream v0.34.1 sender sets Content-Type only; service map +
  default destination; bounded queue (default 1000, 429+counted, never
  silent eviction) with background drain thread; dispatcher-delivery
  success is exactly 2xx (3xx alone never acknowledges durable enqueue);
  Zulip mirror is required product wiring — missing/blank URL or
  unreadable bot credentials fail closed at startup (exit 2), while
  per-delivery Zulip POST failures stay best-effort.
- tests: 44-test stdlib suite — HMAC/vector shape, dedupe, steer mapping
  (no sms/voice/phone), fail-closed configs, 302-kept-and-redelivered
  e2e, exact-2xx deliver_one matrix, Zulip Basic-auth + stable-subject
  mirror test, unit wiring test (all env names set, no scoping carve-outs).
- default.nix: adapter service-map/default/queue/drain options + five
  Zulip options (URL/email/key files via LoadCredential, channel, topic
  prefix); full GOALERT_ADAPTER_* environment.
- keystore-hq.nix: goalert/zulip-bot-email + goalert/zulip-bot-key
  OpenBao entries; HQ PostgreSQL 16 -> 17 (pinned nixpkgs 2026-06-03
  ships PG 17, no incompatibility recorded); Zulip service options
  (https://chat.key.store, incidents stream).
- KSE-0198: event contract + checklist updated (six secrets, PG17,
  required Zulip wiring, exact-2xx, queue semantics).

Suite: 44/44 OK (python3 -m unittest tests.test_adapter).
Nix build/parse NOT verified here (sandbox /nix/store remount denied);
root privileged build is the real check. No merge/deploy/cutover.

goalert: fix upstream binary path (sourceRoot) + dispatch#20 HMAC contract + hermetic tests ... c5a35d2cfe

- default.nix: pin sourceRoot=goalert (tarball has a single top-level
  dir; Nix auto-cds into it, so cp goalert/bin/goalert failed with
  'cannot stat' per root's package-build.log). install via
  install -Dm755 bin/goalert; add doInstallCheck guarding this class.
  Layout + SRI hash verified 2026-09-14 against the real release asset.
- adapter.py: X-KeyStore-Signature is bare-hex HMAC-SHA256 of the exact
  request bytes per dispatch#20 ingress contract (Redmine #35), not the
  sha256=-prefixed KSE-0101 style the verifier 403s. post_raw signs and
  sends the same bytes; deliver_one serializes once.
- tests: FakeDispatcher verifies bare-hex-of-raw-body; deliver test
  mocks the post_raw seam.
- CI: new //bazel/forge:goalert_adapter_tests hermetic sh_test +
  flake checks.goalert-adapter-tests runCommand twin.

Verified: adapter 44/44, wiring 12/12, runner emulation OK.
Privileged nix build + module eval still root's (sandbox has no daemon).

hq: keep key-store-codex-login active (exited) to stop switch start-limit failures ... 6b829e8d5a

Type=oneshot without RemainAfterExit returns to inactive/dead after every
successful run, so each concurrent start source during switch-to-configuration
(multi-user.target start phase + supervisor wants=) re-runs the bootstrap
script. Enough rapid re-runs trip StartLimitBurst and the activation exits 4
with Result=start-limit-hit despite ExecMainStatus=0 (observed 2026-09-13/14,
Redmine #47). With RemainAfterExit=yes the duplicate starts are no-ops.

Rotation procedure unchanged: openaiApiKey/codexAuthJson list this unit in
units (ordering) with empty restartUnits, so post-rotation re-run stays manual
(systemctl restart still re-executes).

Ref: Redmine #47.

infra requested reviews from founder, manager, assistant, cmo, release-runner, billing, web, runner, agent, android, cdo, forge, security-agent, ios, windows, linux, devex 2026-09-14 01:49:32 +00:00

infra referenced this pull request from a commit 2026-09-14 02:07:38 +00:00

hq: keep key-store-codex-login active (exited) to stop switch start-limit failures

infra referenced this pull request 2026-09-14 02:07:42 +00:00

hq: keep key-store-codex-login active (exited) to stop switch start-limit failures (isolated fix) #279

founder commented 2026-09-14 02:31:52 +00:00

Owner

Copy link

Superseded by the isolated, reviewed, tested and deployed PR279. This original branch inherited the separate unmerged GoAlert stack; that work remains in PR270.

Superseded by the isolated, reviewed, tested and deployed PR279. This original branch inherited the separate unmerged GoAlert stack; that work remains in PR270.

founder closed this pull request 2026-09-14 02:31:52 +00:00

Some checks failed

Hide all checks

Build: sqlite3mc / sqlite3mc smoke (macOS) (pull_request) Has been skipped

Details

Build: Apple / Check (SwiftLint) (pull_request) Failing after 18s

Details

Build: Apple / Build (macOS) (pull_request) Has been skipped

Details

Build: Apple / Build (iOS) (pull_request) Has been skipped

Details

Build: Apple / Build (visionOS) (pull_request) Has been skipped

Details

Build: sqlite3mc / sqlite3mc smoke (Linux x86_64) (pull_request) Successful in 48s

Details

Build: Web / Build (Web) (pull_request) Successful in 41s

Details

Build: KaiOS / Build (KaiOS) (pull_request) Successful in 1m22s

Details

Build: Linux / Build (Linux x86_64) (pull_request) Successful in 3m57s

Details

Build: Linux / Build (keystore-forge x86_64) (pull_request) Has been skipped

Details

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

Reviewers

No reviewers

forge

release-runner

cmo

android

founder

assistant

windows

security-agent

devex

runner

billing

web

cdo

linux

manager

agent

ios

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

key-store/key.store!277

No description provided.