forge: bump forgejo-nsc to 82e3da07 (stop-monitor startup window fix + daemon TTL drain) #56

Merged

founder merged 1 commit from forge/forgejo-nsc-stop-monitor into main 2026-08-16 11:22:39 +00:00

Conversation 0 Commits 1 Files changed 2 +6 -6

founder commented 2026-08-16 11:20:51 +00:00

Owner

Copy link

Pulls in key-store/forgejo-nsc#1 (merged as 82e3da07).

Why: on the forge, 125 of 176 Namespace runner launches in the 24 h to 2026-08-16 10:00Z were destroyed 2–5 s after nsc run --wait returned. Reproduced as root: for the first ~3–6 s nsc describe --output json reports the resource with no container key, and instanceStopped() treated that as "all containers stopped". Also fixes the related hazard where daemon-mode runners accepted jobs right before their TTL (job 97045 / run 17183).

What changes on the forge after deploy (forgejo-nsc-dispatcher.service):

• Stop monitor: 10 s initial delay, then nsc describe every 10 s; only a tombstone / stopped container (or "not found" after a container was seen or 2 m grace) ends the wait; transient describe errors retry. New Info logs: runner first stop check (raw JSON), runner instance phase changed, runner instance reported gone.
• Daemon runners drain before TTL: FORGEJO_RUNNER_TTL_DEADLINE_EPOCH + FORGEJO_RUNNER_DRAIN_SECONDS (TTL/3 clamped 5m..30m → 30 m for the 90 m amd64/arm64-large targets, 5 m for 15 m small) → SIGTERM forgejo-runner daemon at deadline−drain with shutdown_timeout = drain−60 s. Override per target via env.FORGEJO_RUNNER_DRAIN_SECONDS.

go.mod/go.sum unchanged upstream → same vendorHash; verified locally with nix build .#forgejo-nsc-dispatcher.

Rollout: deploy the forge, then re-run the journalctl census (launched → destroyed within 15 s should drop to ~0 while jobs still get runners).

Pulls in [key-store/forgejo-nsc#1](https://git.key.store/key-store/forgejo-nsc/pulls/1) (merged as `82e3da07`). **Why:** on the forge, 125 of 176 Namespace runner launches in the 24 h to 2026-08-16 10:00Z were destroyed 2–5 s after `nsc run --wait` returned. Reproduced as root: for the first ~3–6 s `nsc describe --output json` reports the resource with **no `container` key**, and `instanceStopped()` treated that as "all containers stopped". Also fixes the related hazard where daemon-mode runners accepted jobs right before their TTL (job 97045 / run 17183). **What changes on the forge after deploy** (`forgejo-nsc-dispatcher.service`): - Stop monitor: 10 s initial delay, then `nsc describe` every 10 s; only a tombstone / stopped container (or "not found" after a container was seen or 2 m grace) ends the wait; transient describe errors retry. New Info logs: `runner first stop check` (raw JSON), `runner instance phase changed`, `runner instance reported gone`. - Daemon runners drain before TTL: `FORGEJO_RUNNER_TTL_DEADLINE_EPOCH` + `FORGEJO_RUNNER_DRAIN_SECONDS` (TTL/3 clamped 5m..30m → 30 m for the 90 m amd64/arm64-large targets, 5 m for 15 m small) → SIGTERM `forgejo-runner daemon` at deadline−drain with `shutdown_timeout = drain−60 s`. Override per target via `env.FORGEJO_RUNNER_DRAIN_SECONDS`. `go.mod`/`go.sum` unchanged upstream → same `vendorHash`; verified locally with `nix build .#forgejo-nsc-dispatcher`. **Rollout:** deploy the forge, then re-run the journalctl census (launched → destroyed within 15 s should drop to ~0 while jobs still get runners).

founder added 1 commit 2026-08-16 11:20:51 +00:00

forge: bump forgejo-nsc to 82e3da07 (stop-monitor startup window fix + daemon TTL drain) ... 93766a2f58

Pulls in key-store/forgejo-nsc#1: the dispatcher's stop monitor no longer
treats a fresh instance's missing container list as "stopped" (that was
destroying ~70% of Namespace launches 2-5 s after `nsc run --wait`
returned), and daemon-mode runners now stop accepting jobs ahead of their
`nsc run --duration` TTL (FORGEJO_RUNNER_TTL_DEADLINE_EPOCH /
FORGEJO_RUNNER_DRAIN_SECONDS, shutdown_timeout in runner.yaml).

go.mod/go.sum are unchanged upstream, so vendorHash stays; verified with
`nix build .#forgejo-nsc-dispatcher`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

founder requested reviews from billing, assistant, linux, web, infra, forge, manager, windows, android, cmo, security-agent, ios, runner, devex, cdo, release-runner, agent 2026-08-16 11:20:51 +00:00

founder force-pushed forge/forgejo-nsc-stop-monitor from 93766a2f58 to 9abe479091 2026-08-16 11:22:36 +00:00 Compare

founder merged commit 9abe479091 into main 2026-08-16 11:22:39 +00:00

founder deleted branch forge/forgejo-nsc-stop-monitor 2026-08-16 11:22:39 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

assistant

release-runner

web

forge

infra

security-agent

android

windows

ios

agent

billing

cmo

devex

linux

cdo

manager

runner

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

key-store/key.store!56

No description provided.