dispatcher: don't destroy fresh instances on the empty-container describe window; drain daemon runners before TTL #1
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "fix/stop-monitor-startup-window"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
Measured on the forge (
journalctl -u forgejo-nsc-dispatcher, 24h to 2026-08-16 10:00Z): 125 of 176runner launchedevents were followed byrunner destroyedwithin 2–5 s, across every machine type, while Forgejo showed those runners registered and online.Reproduced as root on the forge with
nsc run --wait --output json …; nsc describe --output json <id>:nsc run --waitreturnednsc describe --output jsonper_resource.<uid>has onlynamespace/name/uid— nocontainerkeycontainer: [{status: running, ready: true, started_at: …}](started_at ≈ 4 s after--waitreturned)tombstoneset, containerstatus: stopped+terminated_at… failed to start or was destroyed (FailedPrecondition)instanceStopped()treated the empty container list as "all containers stopped" (continue→ fall through →return true), so the first check inwaitForInstanceStopdestroyed the instance immediately.Fix
Stop monitor (
internal/nsc/dispatcher.go)describeInstancePhase()classifies describe output asunknown / starting / running / stopped. Only a tombstone or an explicitly stopped/terminated container isstopped; a missing/empty container list isstarting.waitForInstanceStop()skips the immediate check (10 s initial delay), keeps polling throughstarting, tracks whether a container was ever observed, and only trusts anot found/destroyeddescribe error once a container was seen or a 2 m startup grace has passed. Transient describe errors are now retried until the TTL+5m deadline instead of ending the wait (which made the caller destroy a possibly mid-job runner). First check logs the raw describe JSON (runner first stop check); phase transitions are logged.stopMonitorConfigso tests can shrink them.TTL drain (related hazard from the same investigation): daemon-mode runners accepted jobs right before the
nsc run --durationTTL expired (job 97045 in run 17183 was killed whennscloud-176cbcc386d5, launched 08:47 with a 90 m TTL, hit 10:17).LaunchRunnerpassesFORGEJO_RUNNER_TTL_DEADLINE_EPOCH(computed beforensc run, so never later than the real expiry) andFORGEJO_RUNNER_DRAIN_SECONDS(TTL/3 clamped to 5m..30m, ≤ TTL/2 → 30 m for 90 m TTL, 5 m for 15 m; overridable via targetenv).runner.shutdown_timeout = drain − 60 s(min 30 s) intorunner.yaml, runsforgejo-runner daemonin the background and SIGTERMs it atdeadline − drain. forgejo-runner 12.x then stops polling, finishes an in-flight job withinshutdown_timeout, and exits → container stops → monitor destroys the instance.Tests (
internal/nsc/dispatcher_linux_test.go)Phase-parser table using the JSON captured on the forge;
waitForInstanceStopwith a scripted fakensc(startup window, gone-during-grace, transient errors, timeout, ctx cancel); drain-margin table; launch env assertions; the drain shell fragment executed under bash with a fakeforgejo-runner(SIGTERM at the drain point, exit-status pass-through,shutdown_timeoutbuffer). The two pre-existingt.Setenv+t.Parallellaunch tests panicked on Linux and are fixed. Fullinternal/nscsuite run on the forge (Linux, cross-compiled): all pass, timing tests stable ×3.Rollout
key.store
forgejo_nsc_srcbump + forge redeploy follow; verification is the same journalctl census (instant destroys → ~0, jobs still get runners).