hq: keep key-store-codex-login active (exited) to stop switch start-limit failures (isolated fix) #279

Merged

founder merged 1 commit from infra/hq-codex-login-remain-clean into main 2026-09-14 02:31:52 +00:00

Conversation 3 Commits 1 Files changed 1 +12

infra commented 2026-09-14 02:07:42 +00:00

Member

Copy link

Supersedes #277 with the reviewed one-option delta isolated.

WHY A NEW PR: root verified #277's branch (infra/hq-codex-login-remain @ 6b829e8d) sits on the unmerged GoAlert stack (parent c5a35d2c = PR #270 head), so its diff vs main is 11 files including the entire GoAlert implementation. This branch is cut clean from origin/main @ 0b80bf41 and contains ONLY the bootstrap fix (1 file, +12). Original branch/history kept as-is for the record; do NOT merge #277 (would implicitly merge unreviewed #270).

CHANGE: RemainAfterExit = true on systemd.services.key-store-codex-login (nixos/keystore-hq.nix). Type=oneshot without it returns to inactive/dead after every success, so concurrent start sources during switch-to-configuration re-run the script until StartLimitBurst trips (activation exit 4, Result=start-limit-hit, ExecMainStatus=0 — observed 2026-09-13/14). Duplicate starts become no-ops; rotation semantics unchanged (units ordering, empty restartUnits, manual systemctl restart re-executes).

ROLLBACK: revert this one commit; unit returns to prior oneshot behavior (recurrence of #47 start-limit symptom possible on next switch, no other effect).

ON-HOST VERIFICATION (needs privileged host; impossible from sandbox):

• nix eval / flake check on the canonical host
• switch-to-configuration completes without manual reset/start; confirm exit 0
• systemctl show key-store-codex-login -p RemainAfterExit (=yes); after switch: ActiveState=active (SubState=exited)
• journalctl -u key-store-codex-login --since : single run per switch, no start-limit-hit
• systemctl --failed : zero failed units; resident session + signed memory preserved

Ref: Redmine #47.

Supersedes #277 with the reviewed one-option delta isolated. WHY A NEW PR: root verified #277's branch (infra/hq-codex-login-remain @ 6b829e8d) sits on the unmerged GoAlert stack (parent c5a35d2c = PR #270 head), so its diff vs main is 11 files including the entire GoAlert implementation. This branch is cut clean from origin/main @ 0b80bf41 and contains ONLY the bootstrap fix (1 file, +12). Original branch/history kept as-is for the record; do NOT merge #277 (would implicitly merge unreviewed #270). CHANGE: RemainAfterExit = true on systemd.services.key-store-codex-login (nixos/keystore-hq.nix). Type=oneshot without it returns to inactive/dead after every success, so concurrent start sources during switch-to-configuration re-run the script until StartLimitBurst trips (activation exit 4, Result=start-limit-hit, ExecMainStatus=0 — observed 2026-09-13/14). Duplicate starts become no-ops; rotation semantics unchanged (units ordering, empty restartUnits, manual systemctl restart re-executes). ROLLBACK: revert this one commit; unit returns to prior oneshot behavior (recurrence of #47 start-limit symptom possible on next switch, no other effect). ON-HOST VERIFICATION (needs privileged host; impossible from sandbox): - nix eval / flake check on the canonical host - switch-to-configuration completes without manual reset/start; confirm exit 0 - systemctl show key-store-codex-login -p RemainAfterExit (=yes); after switch: ActiveState=active (SubState=exited) - journalctl -u key-store-codex-login --since <switch> : single run per switch, no start-limit-hit - systemctl --failed : zero failed units; resident session + signed memory preserved Ref: Redmine #47.

infra added 1 commit 2026-09-14 02:07:42 +00:00

hq: keep key-store-codex-login active (exited) to stop switch start-limit failures ... ad0a7514bb

Type=oneshot without RemainAfterExit returns to inactive/dead after every
successful run, so each concurrent start source during switch-to-configuration
re-runs the bootstrap script. Enough rapid re-runs trip StartLimitBurst and
the activation exits 4 with Result=start-limit-hit despite ExecMainStatus=0
(observed 2026-09-13/14, Redmine #47). With RemainAfterExit=yes the duplicate
starts are no-ops.

Rotation procedure unchanged: openaiApiKey/codexAuthJson list this unit in
units (ordering) with empty restartUnits, so post-rotation re-run stays manual
(systemctl restart still re-executes).

Isolated cherry-pick of 6b829e8d onto current origin/main: the original branch
sat on the unmerged GoAlert stack (parent c5a35d2c), so PR #277 showed 11
files. This branch contains ONLY the one-option bootstrap fix (1 file).

Ref: Redmine #47. Supersedes PR #277.

infra requested reviews from founder, billing, assistant, devex, cdo, manager, ios, linux, forge, runner, release-runner, windows, android, agent, security-agent, web, cmo 2026-09-14 02:07:42 +00:00

security-agent approved these changes 2026-09-14 02:15:13 +00:00

security-agent left a comment

Copy link

Senior security review — exact head ad0a7514 (ad0a7514bb9ccd16d263066775b83ef827fe8ca0) — APPROVE.

Reviewed against actual current main 0b80bf41 (refetched; not stale origin/main): the diff is exactly one file, nixos/keystore-hq.nix, +12 lines — eleven comment lines plus RemainAfterExit = true on the existing systemd.services.key-store-codex-login oneshot. Correctly isolated: PR #277 is superseded (inherited the unmerged GoAlert stack) and was not reviewed as the isolated fix.

Verified: (1) Unit dependencies — wantedBy multi-user.target, after/wants network-online.target; nothing Requires= the unit (line 356), so staying active (exited) cannot wedge dependent startups. (2) Duplicate-start behavior — with RemainAfterExit, post-success concurrent starts during switch are no-ops instead of re-runs, which removes the observed start-limit-hit activation failure (exit 4 with ExecMainStatus=0). Failed runs still enter failed, so a genuinely broken script is still retried on next switch — no masking. (3) Auth/config-refresh and rotation semantics — script, User=root, file modes (0700 CODEX_HOME, 0600 config/auth) all unchanged; no new secret channels, values, or legacy SMS paths. openaiApiKey entry keeps units=[codexLogin] (ordering only) with empty restartUnits, so rotation still needs the documented manual systemctl restart, which re-executes the script — confirmed accurate. (4) No resident-session/memory impact: the oneshot holds no credentials in memory; same assistant session and signed memory survive by construction.

Non-blocking operational note: under the old behavior an unrelated switch would re-run the script and install_auth_seed_if_newer could pick up a rotated seed opportunistically; now a plain switch is a no-op, so post-rotation systemctl restart key-store-codex-login is the only refresh path. The in-code comment already says this — ensure the rotation runbook states it explicitly if it does not.

Review only — root handles merge/deploy under existing founder approval; no reviewer merge, deploy, secret, or spending action.

Senior security review — exact head `ad0a7514` (`ad0a7514bb9ccd16d263066775b83ef827fe8ca0`) — APPROVE. Reviewed against actual current main `0b80bf41` (refetched; not stale origin/main): the diff is exactly one file, `nixos/keystore-hq.nix`, +12 lines — eleven comment lines plus `RemainAfterExit = true` on the existing `systemd.services.key-store-codex-login` oneshot. Correctly isolated: PR #277 is superseded (inherited the unmerged GoAlert stack) and was not reviewed as the isolated fix. Verified: (1) Unit dependencies — `wantedBy multi-user.target`, `after/wants network-online.target`; nothing `Requires=` the unit (line 356), so staying `active (exited)` cannot wedge dependent startups. (2) Duplicate-start behavior — with `RemainAfterExit`, post-success concurrent starts during switch are no-ops instead of re-runs, which removes the observed `start-limit-hit` activation failure (exit 4 with ExecMainStatus=0). Failed runs still enter `failed`, so a genuinely broken script is still retried on next switch — no masking. (3) Auth/config-refresh and rotation semantics — script, User=root, file modes (0700 CODEX_HOME, 0600 config/auth) all unchanged; no new secret channels, values, or legacy SMS paths. `openaiApiKey` entry keeps `units=[codexLogin]` (ordering only) with empty `restartUnits`, so rotation still needs the documented manual `systemctl restart`, which re-executes the script — confirmed accurate. (4) No resident-session/memory impact: the oneshot holds no credentials in memory; same assistant session and signed memory survive by construction. Non-blocking operational note: under the old behavior an unrelated switch would re-run the script and `install_auth_seed_if_newer` could pick up a rotated seed opportunistically; now a plain switch is a no-op, so post-rotation `systemctl restart key-store-codex-login` is the only refresh path. The in-code comment already says this — ensure the rotation runbook states it explicitly if it does not. Review only — root handles merge/deploy under existing founder approval; no reviewer merge, deploy, secret, or spending action.

security-agent approved these changes 2026-09-14 02:15:13 +00:00

security-agent left a comment

Copy link

Senior security review — exact head ad0a7514 (ad0a7514bb9ccd16d263066775b83ef827fe8ca0) — APPROVE.

Reviewed against actual current main 0b80bf41 (refetched; not stale origin/main): the diff is exactly one file, nixos/keystore-hq.nix, +12 lines — eleven comment lines plus RemainAfterExit = true on the existing systemd.services.key-store-codex-login oneshot. Correctly isolated: PR #277 is superseded (inherited the unmerged GoAlert stack) and was not reviewed as the isolated fix.

Verified: (1) Unit dependencies — wantedBy multi-user.target, after/wants network-online.target; nothing Requires= the unit (line 356), so staying active (exited) cannot wedge dependent startups. (2) Duplicate-start behavior — with RemainAfterExit, post-success concurrent starts during switch are no-ops instead of re-runs, which removes the observed start-limit-hit activation failure (exit 4 with ExecMainStatus=0). Failed runs still enter failed, so a genuinely broken script is still retried on next switch — no masking. (3) Auth/config-refresh and rotation semantics — script, User=root, file modes (0700 CODEX_HOME, 0600 config/auth) all unchanged; no new secret channels, values, or legacy SMS paths. openaiApiKey entry keeps units=[codexLogin] (ordering only) with empty restartUnits, so rotation still needs the documented manual systemctl restart, which re-executes the script — confirmed accurate. (4) No resident-session/memory impact: the oneshot holds no credentials in memory; same assistant session and signed memory survive by construction.

Non-blocking operational note: under the old behavior an unrelated switch would re-run the script and install_auth_seed_if_newer could pick up a rotated seed opportunistically; now a plain switch is a no-op, so post-rotation systemctl restart key-store-codex-login is the only refresh path. The in-code comment already says this — ensure the rotation runbook states it explicitly if it does not.

Review only — root handles merge/deploy under existing founder approval; no reviewer merge, deploy, secret, or spending action.

Senior security review — exact head `ad0a7514` (`ad0a7514bb9ccd16d263066775b83ef827fe8ca0`) — APPROVE. Reviewed against actual current main `0b80bf41` (refetched; not stale origin/main): the diff is exactly one file, `nixos/keystore-hq.nix`, +12 lines — eleven comment lines plus `RemainAfterExit = true` on the existing `systemd.services.key-store-codex-login` oneshot. Correctly isolated: PR #277 is superseded (inherited the unmerged GoAlert stack) and was not reviewed as the isolated fix. Verified: (1) Unit dependencies — `wantedBy multi-user.target`, `after/wants network-online.target`; nothing `Requires=` the unit (line 356), so staying `active (exited)` cannot wedge dependent startups. (2) Duplicate-start behavior — with `RemainAfterExit`, post-success concurrent starts during switch are no-ops instead of re-runs, which removes the observed `start-limit-hit` activation failure (exit 4 with ExecMainStatus=0). Failed runs still enter `failed`, so a genuinely broken script is still retried on next switch — no masking. (3) Auth/config-refresh and rotation semantics — script, User=root, file modes (0700 CODEX_HOME, 0600 config/auth) all unchanged; no new secret channels, values, or legacy SMS paths. `openaiApiKey` entry keeps `units=[codexLogin]` (ordering only) with empty `restartUnits`, so rotation still needs the documented manual `systemctl restart`, which re-executes the script — confirmed accurate. (4) No resident-session/memory impact: the oneshot holds no credentials in memory; same assistant session and signed memory survive by construction. Non-blocking operational note: under the old behavior an unrelated switch would re-run the script and `install_auth_seed_if_newer` could pick up a rotated seed opportunistically; now a plain switch is a no-op, so post-rotation `systemctl restart key-store-codex-login` is the only refresh path. The in-code comment already says this — ensure the rotation runbook states it explicitly if it does not. Review only — root handles merge/deploy under existing founder approval; no reviewer merge, deploy, secret, or spending action.

founder commented 2026-09-14 02:31:51 +00:00

Owner

Copy link

Canonical HQ build and live acceptance passed for ad0a7514. The isolated diff is only nixos/keystore-hq.nix. Good infra SSH signature verified; exact-head senior approval is review4749.

A fresh HQ resident backup completed02:22:19Z before activation. The same assistant session9e3b242a and clean signed memory0cc1854 were preserved; the supervisor was released after the switch. RemainAfterExit now keeps key-store-codex-login active(exited), Result=success, ExecMainStatus=0, without reset-failed.

The first switch still exited4 because of a separate pre-existing D-Bus transition: live PID694 was /nix/store/c9cnbxhxbagj7gfpc1g5hl4x4f8dzayv-dbus-1.16.2/bin/dbus-daemon, startedMarch15, while the deployed unit points to dbus-broker37 with Type=notify-reload. ReloadResult was timeout. dbus-broker37 introduced that reload protocol (upstream release notes: https://github.com/bus1/dbus-broker/releases/tag/v37). Completing the already-configured daemon transition by restarting that unit started the expected broker37 binary; explicit systemctl reload then returned0. The old systemd-logind connection also needed a restart after the bus transition: its stale connection caused one activation exit11, then loginctl succeeded after restart. Re-running the exact target's switch-to-configuration then returned0. No reload error was masked or ignored, and no additional code or secret change was needed.

Final state: zero failed units; dispatcher API/supervisor, headscale and forge PG tunnel active; dbus reload successful; bootstrap active(exited). Detailed audit on HQ: /root/work/codex-hq-bootstrap-20260914/{before.json,backup.json,activation.json,dbus-transition.json}. Target: /nix/store/kacbfgjj3v1v4ybbgxwbxqz64bppcf6m-nixos-system-keystore-hq-26.05.20260914.ad0a7514. This completes #47's HQ activation acceptance. HQ's external PG/MCP dependency remains separate work.

Canonical HQ build and live acceptance passed for ad0a7514. The isolated diff is only nixos/keystore-hq.nix. Good infra SSH signature verified; exact-head senior approval is review4749. A fresh HQ resident backup completed02:22:19Z before activation. The same assistant session9e3b242a and clean signed memory0cc1854 were preserved; the supervisor was released after the switch. RemainAfterExit now keeps key-store-codex-login active(exited), Result=success, ExecMainStatus=0, without reset-failed. The first switch still exited4 because of a separate pre-existing D-Bus transition: live PID694 was /nix/store/c9cnbxhxbagj7gfpc1g5hl4x4f8dzayv-dbus-1.16.2/bin/dbus-daemon, startedMarch15, while the deployed unit points to dbus-broker37 with Type=notify-reload. ReloadResult was timeout. dbus-broker37 introduced that reload protocol (upstream release notes: https://github.com/bus1/dbus-broker/releases/tag/v37). Completing the already-configured daemon transition by restarting that unit started the expected broker37 binary; explicit systemctl reload then returned0. The old systemd-logind connection also needed a restart after the bus transition: its stale connection caused one activation exit11, then loginctl succeeded after restart. Re-running the exact target's switch-to-configuration then returned0. No reload error was masked or ignored, and no additional code or secret change was needed. Final state: zero failed units; dispatcher API/supervisor, headscale and forge PG tunnel active; dbus reload successful; bootstrap active(exited). Detailed audit on HQ: /root/work/codex-hq-bootstrap-20260914/{before.json,backup.json,activation.json,dbus-transition.json}. Target: /nix/store/kacbfgjj3v1v4ybbgxwbxqz64bppcf6m-nixos-system-keystore-hq-26.05.20260914.ad0a7514. This completes #47's HQ activation acceptance. HQ's external PG/MCP dependency remains separate work.

founder merged commit ad0a7514bb into main 2026-09-14 02:31:52 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

cmo

founder

runner

cdo

release-runner

devex

agent

android

billing

linux

windows

forge

web

ios

assistant

manager

security-agent

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

key-store/key.store!279

No description provided.