dispatcher: Windows guests via the Namespace ssh service (GetSSHConfig + direct-tcpip to WinRM), readiness probe, autoscaler circuit breaker #6
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "windows-readiness-probe"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
See the commit message. Root cause of every failed Namespace Windows launch: gate tunnels behind nsc instance port-forward/proxy answer websocket: bad handshake for the whole instance lifetime (5985/22/3389, nsc 0.0.516+0.0.557, with/without dial_host); the ssh service (Compute GetSSHConfig) forwards fine and WinRM answers ~20 s after create. TestWindowsWinRMScriptRoundTrip passed on the forge in 24 s. Full unit suite green.
Namespace Windows runners never bootstrapped: every gate tunnel opened by `nsc instance port-forward`/`proxy` (5985, 22, and even the READY rdp service on 3389) fails with 'websocket: bad handshake' for the instance's whole lifetime -- with nsc 0.0.516 and 0.0.557, with and without instance:dial_host in the token -- so the 2 min WinRM retry + 2 min password-ssh fallback only ever measured that, the instance was destroyed ~4.5 min after create and the autoscaler re-created an 8x16 every min_dispatch_interval until Forgejo's stale-job reaper cancelled the job (2026-08-17, key.store run 18412). DescribeInstance says RUNNING with ssh+rdp READY ~15 s after create and GetSSHConfig hands out a key for ssh.<region>.namespace.so (user = instance id). That server cannot exec on Windows ("/bin/bash": executable file not found in %PATH%) but forwards direct-tcpip, and through it the guest's WinRM answers on 127.0.0.1:5985 (HTTP 401 Negotiate, Microsoft-HTTPAPI/2.0) and runs PowerShell with the RDP credentials. - windows_ssh_tunnel.go: GetSSHConfig (retried) -> x/crypto/ssh client -> local listener whose connections become direct-tcpip channels to 127.0.0.1:5985; stop tears everything down; bound to ctx. - windows_winrm.go: transports [ssh-tunnel, port-forward]; each gets a real readiness probe (HTTP status line from /wsman -- a bare TCP connect says nothing) with a windows_ready_timeout budget (default 10m, KEYSTORE_NSC_WINDOWS_READY_TIMEOUT), 'windows guest ready ... after=' logging, then windows_bootstrap_retry (default 3m, KEYSTORE_NSC_WINDOWS_BOOTSTRAP_RETRY). The password-ssh exec fallback is gone (dead: Namespace's ssh service has no shell on Windows and the guest's 22 is unreachable). - autoscaler: per-target circuit breaker -- consecutive failed dispatches double the effective min_dispatch_interval up to max_dispatch_backoff (default 30m), reset on success. - tests: in-process ssh server (direct-tcpip) for the tunnel, readiness probe cases, breaker table, config knobs; TestWindowsWinRMScriptRoundTrip (NSC_WINDOWS_E2E=1) passed on the forge against a fresh windows/amd64:4x8 in 24 s (guest ready 'after=0s').