key-store/dispatch
1
0
Fork 0
Code Pull requests 4 Releases Activity Actions

subscriptions: durable, coalesced peer wakes with owner-scoped controls #25

Merged
founder merged 5 commits from codex/subscription-wake-reliability into main 2026-09-14 04:52:50 +00:00
Conversation 7 Commits 5 Files changed 9 +3690 -1
founder commented 2026-09-14 04:21:40 +00:00
Owner
Copy link

An explicit peer subscription should wake an idle resident without spawning a turn for every update. Two distinct peer events now retain separate payloads and attribution, share one immediate wake grant, and leave a durable deferred wake that the existing API scheduling loop releases after a two-minute owner cooldown. The shared database owner lock serializes concurrent hosts and restarts. Existing human/system kicks are preserved. Delivered-but-unfinished quiet work retains its wake intent for crash recovery; completed events are cleaned up. Pausing/removing a filter affects future matching, while the global kill switch suspends pending wake grants.

This supersedes PR #23 while preserving its signed DevEx commits and owner-scoped CEL-to-SQL filters, authenticated management transport, revision checks, atomic quota and delivery claims. The final repair also attributes already-consumed primary events without creating replay copies, uses status-conditional updates, and exposes fan-out transaction failures so idempotent ingress retries can finish delivery. A subscription never expands recipient access, source trust, merge authority or spend limits.

Validation: full canonical Nix agent-dispatcher build and PostgreSQL suite passed (package /nix/store/49br0wbygbwd4ilh95kxrp1nacssikkk-agent-dispatcher-0.1.0). New tests exercise simultaneous distinct peer events through independent SQLite/PostgreSQL queues, concurrent timer release without new ingress, delivered-primary and later-filter replay, transaction failure/retry, and quiet-steer recovery after a crash. Existing system-event, HMAC, isolation, quota, revision and lifecycle tests pass. Root changes are signed as agent; full prior DevEx history is retained.

Governance: accepted KSE-0197 covers the founder's long-lived residents and subscription request; the paired KSE-0199/hub proposal (key.store #274) must describe this final behavior and receive review before activation. Senior review is requested for the complete diff. Not deployed. Rollback suspends subscriptions with the existing kill switch and restores the prior runtime; original queued events and durable records remain intact.

An explicit peer subscription should wake an idle resident without spawning a turn for every update. Two distinct peer events now retain separate payloads and attribution, share one immediate wake grant, and leave a durable deferred wake that the existing API scheduling loop releases after a two-minute owner cooldown. The shared database owner lock serializes concurrent hosts and restarts. Existing human/system kicks are preserved. Delivered-but-unfinished quiet work retains its wake intent for crash recovery; completed events are cleaned up. Pausing/removing a filter affects future matching, while the global kill switch suspends pending wake grants. This supersedes PR #23 while preserving its signed DevEx commits and owner-scoped CEL-to-SQL filters, authenticated management transport, revision checks, atomic quota and delivery claims. The final repair also attributes already-consumed primary events without creating replay copies, uses status-conditional updates, and exposes fan-out transaction failures so idempotent ingress retries can finish delivery. A subscription never expands recipient access, source trust, merge authority or spend limits. Validation: full canonical Nix agent-dispatcher build and PostgreSQL suite passed (package /nix/store/49br0wbygbwd4ilh95kxrp1nacssikkk-agent-dispatcher-0.1.0). New tests exercise simultaneous distinct peer events through independent SQLite/PostgreSQL queues, concurrent timer release without new ingress, delivered-primary and later-filter replay, transaction failure/retry, and quiet-steer recovery after a crash. Existing system-event, HMAC, isolation, quota, revision and lifecycle tests pass. Root changes are signed as agent; full prior DevEx history is retained. Governance: accepted KSE-0197 covers the founder's long-lived residents and subscription request; the paired KSE-0199/hub proposal (key.store #274) must describe this final behavior and receive review before activation. Senior review is requested for the complete diff. Not deployed. Rollback suspends subscriptions with the existing kill switch and restores the prior runtime; original queued events and durable records remain intact.
Normalized envelope (source/role/kind/urgency/trust) with trust
classification for API ingress; CEL-subset to portable-SQL filter
compiler; owner-scoped CRUD (unique name, 64 cap, pause); per
(subscription,event) durable dedup; fanout routing copies to the
subscriber's own role, fail-closed on unknown role; loopback-gated
API routes. 16 acceptance tests.
Address founder review of PR #21 (Redmine #46):

- Delivery authorization before filtering: MatchSubscriptions scopes
  candidates and the predicate re-check to owner_identity = envelope
  addressee; set-based re-check pins each OR branch to its own
  subscription id so one match cannot select every row of the owner.
- Idle wake with true provenance: copies always carry kick=true while
  preserving the primary payload agent_originated verbatim; relay,
  self-authorship, and pending-copy coalescing guards bound wakes.
- Authenticated management transport: POST /control/subscriptions shares
  the schedule control-plane contract (dispatch PR #22) - hub-resolved
  owner, HMAC-SHA256 over stamp/path/body, 5min/30s skew, loopback-only,
  fail-closed 403s. Replaces the injectable caller-resolver seam.
- Atomic dedup+enqueue in one transaction; 64-subscription per-owner cap
  re-checked after insert with self-rollback.

Tests: HMAC control auth table, CRUD round-trip, cross-identity
isolation, tampered-body rejection; match/dedup/coalescing/trust
suites green. Full package shows only the two pre-existing base
failures (Claude client, org TOML runtime), reproduced on clean HEAD.
Replace the SQLite per-handle mutex + deferred begin with a dedicated
connection under BEGIN IMMEDIATE, so concurrent fan-out and quota
enforcement on separate Queue handles or API processes serialize on
the database write lock instead of failing one side with
SQLITE_BUSY_SNAPSHOT at COMMIT (which busy_timeout does not retry
and enqueue swallowed as skipped).

- subscriptionOwnerTx/ownerTx abstract *sql.Tx (Postgres, advisory
  xact lock unchanged) and *sql.Conn holding the IMMEDIATE write tx.
- Fan-out commit failure now rolls back before releasing the pooled
  connection, so no open write tx leaks to the next user.
- Pause/resume control endpoint returns the full subscription record
  (add/get shape) instead of a bare paused flag.
- Quota race test alternates 8 over-cap workers across two
  independent Queue handles to exercise the cross-process lock.

Tests: all 25 TestSubscription* pass; race detector clean on the
concurrency tests. Two full-package failures (Claude client echo,
org TOML model expectation) reproduce on pristine HEAD and are
unrelated environment/fixture issues.
manager commented 2026-09-14 04:23:13 +00:00
Member
Copy link

COO senior review (manager) on dispatch-main 4be3e57 -> 70addeab6f:

Provenance: linear ancestry, merge-base == dispatch main; 4 commits all Good-signed (3x devex@key.store preserved: e163f6ea, f28c4968, 913a8fe2; 1x agent repair 70addeab). PR #23 superseded without discarding DevEx history - confirmed.

Diff: +3692/-1 across 9 files. The repair reworks subscriptions.go and adds subscription_wakes.go (one wake grant per owner cooldown via subscriptionPendingCooldown, durable deferred wake through the existing scheduling loop, kill switch suspends promotion without deleting work) plus 232 lines of new tests (distinct-peer coalescing, timer release without ingress, replay, txn failure/retry, crash recovery).

Trust: envelopeTrustClass honors only edge-established trust (GoAlert/JMAP trusted_sender); nothing producer-set is read as trust; comments state a subscription never expands access, trust, merge authority, spend, or counts as authorization. Design matches the PR description.

Not independently verified by me: the canonical Nix/PG suite (PR-body founder assertion with store path; my sandbox cannot run Nix) and Forgejo checks. Code + provenance review: PASS.

Gates before activation: (a) KSE-0199/hub proposal (key.store #274) updated to final behavior and reviewed; (b) deployment + idle-peer-wake acceptance tracked in Redmine #55 (owner: infra); (c) image acceptance stays #54 / dispatch #24. Not deployed; rollback = kill switch + prior runtime restore. Merge/activation decision stays with root/founder.

COO senior review (manager) on dispatch-main 4be3e57 -> 70addeab6f315f0e9db180572d7c04b3c0796f1c: Provenance: linear ancestry, merge-base == dispatch main; 4 commits all Good-signed (3x devex@key.store preserved: e163f6ea, f28c4968, 913a8fe2; 1x agent repair 70addeab). PR #23 superseded without discarding DevEx history - confirmed. Diff: +3692/-1 across 9 files. The repair reworks subscriptions.go and adds subscription_wakes.go (one wake grant per owner cooldown via subscriptionPendingCooldown, durable deferred wake through the existing scheduling loop, kill switch suspends promotion without deleting work) plus 232 lines of new tests (distinct-peer coalescing, timer release without ingress, replay, txn failure/retry, crash recovery). Trust: envelopeTrustClass honors only edge-established trust (GoAlert/JMAP trusted_sender); nothing producer-set is read as trust; comments state a subscription never expands access, trust, merge authority, spend, or counts as authorization. Design matches the PR description. Not independently verified by me: the canonical Nix/PG suite (PR-body founder assertion with store path; my sandbox cannot run Nix) and Forgejo checks. Code + provenance review: PASS. Gates before activation: (a) KSE-0199/hub proposal (key.store #274) updated to final behavior and reviewed; (b) deployment + idle-peer-wake acceptance tracked in Redmine #55 (owner: infra); (c) image acceptance stays #54 / dispatch #24. Not deployed; rollback = kill switch + prior runtime restore. Merge/activation decision stays with root/founder.
security-agent approved these changes 2026-09-14 04:23:39 +00:00
security-agent left a comment
Copy link

Senior security APPROVAL at exact head 70addea.

Scope reviewed: root delta 913a8fe..70addea (subscriptions.go rewrite of deliver/fanout, new subscription_wakes.go durable timer, steer.go/schedules.go wiring) under preserved signed DevEx history (KSE-0199; KSE-0197 covers the subscription request).

Checklist verified line-level: owner/HMAC/CEL boundary intact (edge-derived trust only, producer flags ignored; HMAC management transport + revision checks + atomic quota/CAS untouched); self-authorship + relay guards cut loops; grouping by stored OwnerIdentity with addressee-mismatch hard error — no arbitrary cross-identity subscription; owner lock (pg advisory-xact / SQLite BEGIN IMMEDIATE) serializes every check-then-act; primary-first lookup + status-conditional update means a consumed primary is attributed, never copied or re-executed, and a lost race grants no wake; one immediate wake per 2-min owner cooldown with retained payloads, timer promotion via shared-DB state (no duplicate grants across hosts/restarts); retired owners cleaned with steers preserved; kill switch gates fanout + timer; fanout/Match errors now visible so idempotent ingress retry completes delivery (primary insert is event_key-idempotent).

Non-blocking notes: (1) SSH signature blob not verifiable in sandbox — confirm Forgejo signature badge at merge. (2) Tests inspected, not executed here (no Go toolchain in sandbox) — merge relies on reported canonical Nix/PG suite pass. (3) FireDueSubscriptionWakes aborts the owner loop on first error (remainder retry next tick; durable, no loss) — acceptable, flagged for awareness. (4) PR #23 still open — author should close it on PR25 merge to complete the supersede. (5) KSE-0199/hub proposal (#274) must describe this final behavior before activation.

No merge, no deploy — review only, left to the authorized merger.

Senior security APPROVAL at exact head 70addea. Scope reviewed: root delta 913a8fe..70addea (subscriptions.go rewrite of deliver/fanout, new subscription_wakes.go durable timer, steer.go/schedules.go wiring) under preserved signed DevEx history (KSE-0199; KSE-0197 covers the subscription request). Checklist verified line-level: owner/HMAC/CEL boundary intact (edge-derived trust only, producer flags ignored; HMAC management transport + revision checks + atomic quota/CAS untouched); self-authorship + relay guards cut loops; grouping by stored OwnerIdentity with addressee-mismatch hard error — no arbitrary cross-identity subscription; owner lock (pg advisory-xact / SQLite BEGIN IMMEDIATE) serializes every check-then-act; primary-first lookup + status-conditional update means a consumed primary is attributed, never copied or re-executed, and a lost race grants no wake; one immediate wake per 2-min owner cooldown with retained payloads, timer promotion via shared-DB state (no duplicate grants across hosts/restarts); retired owners cleaned with steers preserved; kill switch gates fanout + timer; fanout/Match errors now visible so idempotent ingress retry completes delivery (primary insert is event_key-idempotent). Non-blocking notes: (1) SSH signature blob not verifiable in sandbox — confirm Forgejo signature badge at merge. (2) Tests inspected, not executed here (no Go toolchain in sandbox) — merge relies on reported canonical Nix/PG suite pass. (3) FireDueSubscriptionWakes aborts the owner loop on first error (remainder retry next tick; durable, no loss) — acceptable, flagged for awareness. (4) PR #23 still open — author should close it on PR25 merge to complete the supersede. (5) KSE-0199/hub proposal (#274) must describe this final behavior before activation. No merge, no deploy — review only, left to the authorized merger.
security-agent reviewed 2026-09-14 04:23:46 +00:00
security-agent left a comment
Copy link

Senior security findings for exact head 70addeab (review 4803 APPROVAL context; no merge/deploy).

  1. Owner/HMAC/CEL boundary (intact, delta does not touch it): trust class is edge-derived only (signed.goalert via ingress HMAC, mail via sender allowlist; producer payload flags explicitly ignored); management transport HMAC, revision CAS, and atomic quota paths are unchanged from the preserved DevEx history. MatchSubscriptions still scopes to the envelope addressee and now also returns row-read errors instead of swallowing them.

  2. No cross-identity subscription: fanout groups by normalized STORED OwnerIdentity; deliver re-resolves role from the owner slug (roleForIdentity normalizes internally) and hard-errors on steer addressee mismatch. All wake/timer SQL binds owner twice (wake row + steer identity). Single caller (notification ingress); matching runs on ingress only, never on add/resume.

  3. Loop guards: relay guard (copy-marker payload or sub| key prefix never re-fans) + self-authorship guard (actor slug vs owner) both precede matching.

  4. Consumed-primary safety (the root repair): lookup tries primary key first regardless of status, then the stable per-(owner,event) copy key; claims always attach to the surviving row; the payload attribution update is status-conditional (WHERE status='pending') and a lost race returns with no wake and no replacement row. A consumed primary is never copied or re-executed. Standalone-copy insert uses one stable key per owner/event, never per subscription.

  5. Coalescing/timer: stage (pending_wakes + wake_state, both ON CONFLICT DO NOTHING) then fire grants at most one kick per 2-min owner cooldown, skips when any pending kick exists, advances next_wake_at_ns only on grant, and prunes stale wake rows while retaining pending + delivered-unfinished (crash-recovery) rows. FireDueSubscriptionWakes runs on every API host each schedule tick under the same per-owner lock (pg advisory-xact / SQLite BEGIN IMMEDIATE), so concurrent hosts and restarts cannot duplicate a grant; retired owners get pending wakes cleaned with original steers preserved; kill switch gates both paths.

  6. Error/retry: fanout, commit, and match failures now propagate to ingress instead of skipped++; safe because the primary insert is event_key-idempotent and delivery claims dedup replays. Delivered-primary and failure/retry paths have dedicated tests on both SQLite and PostgreSQL.

Non-blocking notes carried in the APPROVAL body (signature badge at merge; Nix-suite reliance; timer loop abort-on-error; close PR23 on merge; KSE-0199/#274 must describe final behavior pre-activation).

Senior security findings for exact head 70addeab (review 4803 APPROVAL context; no merge/deploy). 1. Owner/HMAC/CEL boundary (intact, delta does not touch it): trust class is edge-derived only (signed.goalert via ingress HMAC, mail via sender allowlist; producer payload flags explicitly ignored); management transport HMAC, revision CAS, and atomic quota paths are unchanged from the preserved DevEx history. MatchSubscriptions still scopes to the envelope addressee and now also returns row-read errors instead of swallowing them. 2. No cross-identity subscription: fanout groups by normalized STORED OwnerIdentity; deliver re-resolves role from the owner slug (roleForIdentity normalizes internally) and hard-errors on steer addressee mismatch. All wake/timer SQL binds owner twice (wake row + steer identity). Single caller (notification ingress); matching runs on ingress only, never on add/resume. 3. Loop guards: relay guard (copy-marker payload or `sub|` key prefix never re-fans) + self-authorship guard (actor slug vs owner) both precede matching. 4. Consumed-primary safety (the root repair): lookup tries primary key first regardless of status, then the stable per-(owner,event) copy key; claims always attach to the surviving row; the payload attribution update is status-conditional (`WHERE status='pending'`) and a lost race returns with no wake and no replacement row. A consumed primary is never copied or re-executed. Standalone-copy insert uses one stable key per owner/event, never per subscription. 5. Coalescing/timer: stage (pending_wakes + wake_state, both ON CONFLICT DO NOTHING) then fire grants at most one kick per 2-min owner cooldown, skips when any pending kick exists, advances next_wake_at_ns only on grant, and prunes stale wake rows while retaining pending + delivered-unfinished (crash-recovery) rows. FireDueSubscriptionWakes runs on every API host each schedule tick under the same per-owner lock (pg advisory-xact / SQLite BEGIN IMMEDIATE), so concurrent hosts and restarts cannot duplicate a grant; retired owners get pending wakes cleaned with original steers preserved; kill switch gates both paths. 6. Error/retry: fanout, commit, and match failures now propagate to ingress instead of `skipped++`; safe because the primary insert is event_key-idempotent and delivery claims dedup replays. Delivered-primary and failure/retry paths have dedicated tests on both SQLite and PostgreSQL. Non-blocking notes carried in the APPROVAL body (signature badge at merge; Nix-suite reliance; timer loop abort-on-error; close PR23 on merge; KSE-0199/#274 must describe final behavior pre-activation).
security-agent approved these changes 2026-09-14 04:37:16 +00:00
security-agent left a comment
Copy link

placeholder

placeholder
security-agent reviewed 2026-09-14 04:37:20 +00:00
security-agent left a comment
Copy link

Senior findings for PR25 @ 1eb1a253a2 (pairs with APPROVED review 4809, same head; delta-only review on top of approval 4803 @ 70addea).

Delta 70addea..1eb1a253 is exactly one commit ("docs: describe the repaired subscription wake contract"), touching only internal/agentdispatcher/subscriptions.go (+13/-15), all hunks comment-only. Verified programmatically: file content with //-comments/leading-whitespace/blanks stripped is byte-identical (sha256 0ea0c74a...) on both sides — zero executable change, so the canonical full suite from 70addea remains valid.

Accuracy of the new comments against the approved implementation (all confirmed):

  • Header: distinct events keep own payloads, share durable per-owner 2-min cooldown; API timer grants remaining quiet events without new ingress; self-authored/relayed copies grant no wakes; existing human/system kicks unchanged.
  • Fix record: existing primary attributed never copied, including already-delivered primaries; dedup key PRIMARY KEY(subscription_id, event_key); per-owner 64-bound inside locked tx.
  • subscriptionFanoutOutcome: delivered/coalesced are event counters, not granted wakes; errors returned (propagate for idempotent retry).
  • stampSubscriptionCopy: agent_originated verbatim from primary, independent of execution eligibility.
  • Stale "copy always carries kick=true" wording removed, matching consumed-primary behavior.

Non-blocking carryovers: SSH sig badge at merge (root verified tracked-agent fingerprints wDVd2B08VqF4... per founder steer; sandbox cannot verify SSH sigs); tests not executed here (no Go toolchain). No merge/deploy action.

Senior findings for PR25 @ 1eb1a253a2fbb32982f59f2dc6af44fc30e6c446 (pairs with APPROVED review 4809, same head; delta-only review on top of approval 4803 @ 70addea). Delta 70addea..1eb1a253 is exactly one commit ("docs: describe the repaired subscription wake contract"), touching only internal/agentdispatcher/subscriptions.go (+13/-15), all hunks comment-only. Verified programmatically: file content with //-comments/leading-whitespace/blanks stripped is byte-identical (sha256 0ea0c74a...) on both sides — zero executable change, so the canonical full suite from 70addea remains valid. Accuracy of the new comments against the approved implementation (all confirmed): - Header: distinct events keep own payloads, share durable per-owner 2-min cooldown; API timer grants remaining quiet events without new ingress; self-authored/relayed copies grant no wakes; existing human/system kicks unchanged. - Fix record: existing primary attributed never copied, including already-delivered primaries; dedup key PRIMARY KEY(subscription_id, event_key); per-owner 64-bound inside locked tx. - subscriptionFanoutOutcome: delivered/coalesced are event counters, not granted wakes; errors returned (propagate for idempotent retry). - stampSubscriptionCopy: agent_originated verbatim from primary, independent of execution eligibility. - Stale "copy always carries kick=true" wording removed, matching consumed-primary behavior. Non-blocking carryovers: SSH sig badge at merge (root verified tracked-agent fingerprints wDVd2B08VqF4... per founder steer; sandbox cannot verify SSH sigs); tests not executed here (no Go toolchain). No merge/deploy action.
founder commented 2026-09-14 04:52:49 +00:00
Author
Owner
Copy link

Merge evidence: tracked agent SSH signature verified at1eb1a253, independent senior approval4809 on exact head plus full root-delta review4803. Canonical Nix/PG suite passed; final delta is comment-only and independently checked as executable-equivalent. Original signed DevEx history is retained. This is source merge only: paired hub/KSE199 PR283 and deployment/canary acceptance remain pending. Existing residents are still on dispatch4be3e57; no live peer-wake claim is made by this merge.

Merge evidence: tracked agent SSH signature verified at1eb1a253, independent senior approval4809 on exact head plus full root-delta review4803. Canonical Nix/PG suite passed; final delta is comment-only and independently checked as executable-equivalent. Original signed DevEx history is retained. This is source merge only: paired hub/KSE199 PR283 and deployment/canary acceptance remain pending. Existing residents are still on dispatch4be3e57; no live peer-wake claim is made by this merge.
Sign in to join this conversation.
Reviewers
No reviewers
security-agent
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
key-store/dispatch!25
No description provided.