forgejo-nsc-dispatcher

This service exposes a simple HTTP API that tells Namespace Cloud to start ephemeral Forgejo Actions runners on demand. It glues together three pieces:

1. Forgejo Actions – the service requests a scoped registration token for the repository/organization/instance where you want to run jobs.
2. Namespace (nsc) – the dispatcher shells out to the nsc CLI to create a short-lived environment, runs the forgejo-runner container inside it, and defaults Linux runners to forgejo-runner daemon. The autoscaler launches one Namespace instance for one queued job, and the runner stays alive until it drains ahead of the Namespace TTL (see below) or the container exits. Set FORGEJO_RUNNER_MODE=one-job on a target only after that runner image and Forgejo version have live proof for one-job bootstrap.
3. Your automation – you call the service via HTTP (directly, through Caddy, via Forgejo webhooks, etc.) whenever a new runner is needed.

Directory layout

.
├── cmd/forgejo-nsc-dispatcher   # main entry point
├── internal/                    # service packages (config, forgejo client, nsc dispatcher, HTTP server)
├── config.example.yaml          # starter config referenced by README
├── flake.nix / flake.lock       # reproducible builds (Go binary + container image)
└── .forgejo/workflows           # CI on the key.store forge: go fmt/vet/test, nix builds, container image builds

Configuration

Copy config.example.yaml and update it for your Forgejo instance and Namespace profile. The important knobs are:

• forgejo.base_url – HTTPS endpoint of your Forgejo server. A PAT with actions:runner scope is required in forgejo.token.
• forgejo.instance_url – URL that spawned runners use to register back to Forgejo. This must be reachable from the runner (typically the public URL like https://git.key.store). On the forge host it commonly differs from base_url (which may be http://127.0.0.1:3000).
• forgejo.default_scope – where new runners register (instance, organization, or repository).
• forgejo.default_labels – labels applied to every spawned runner. GateForge workflows via runs-on: ["namespace-profile-linux-medium"] (or other namespace-profile-linux-* labels).
• namespace.nsc_binary – path to the nsc binary (the Nix container ships one compiled from namespacelabs/foundation so /app/bin/nsc works out of the box).
• namespace.image – OCI image used for Linux runners. key.store uses nscr.io/osibqsl33bahe/key-store/devbox-nix:runner, rebuilt by the Namespace image workflow when important flake/runner inputs change. Prefer a glibc Node image with Bash from Namespace's registry; startup rejects non-nscr.io refs so this path cannot drift to Docker Hub. The dispatcher bootstraps forgejo-runner when the image does not already contain the configured version, defaulting to 12.7.2.
• Per-target image overrides let heavyweight platform lanes use specialized images. key.store uses namespace-profile-android-large with the android-runner image so Android SDK, Gradle, and Bazel bootstrap work is present before the job starts.
• namespace.image_fallbacks – optional Namespace public-base fallback images tried only when Namespace cannot resolve the primary image tag. This is a positive allow-list for the same Namespace public runner-base path as the primary image, not a generic external-registry fallback. Do not use Docker Hub tags or arbitrary third-party nscr.io images here; key.store's fallback is the Namespace-hosted public runner base built by Scripts/namespace-build-runner-public-base.sh. The dispatcher derives the allowed fallback repository from namespace.image; for nscr.io/osibqsl33bahe/key-store/devbox-nix:runner, only nscr.io/osibqsl33bahe/key-store/runner-public-base:* is valid. It also validates fallbacks against the selected image override's Namespace account/path at dispatch time and rejects non-nscr.io Linux image overrides, so this cannot silently drift back to Docker Hub or cross to a different Namespace account.
• namespace.machine_type / namespace.duration – shape + TTL for the ephemeral Namespace environment. nsc run --wait returns when the container starts, so the dispatcher monitors the runner container and destroys the Namespace instance after the runner exits or reaches its TTL. This keeps TTL as the hard cap while committing cache volumes promptly and avoiding unbounded idle Linux runners.
  • The stop monitor polls nsc describe --output json every 10s, starting 10s after launch. Right after nsc run --wait returns, describe reports the resource with no container list for several seconds; that is treated as starting, never as stopped. Only a tombstone or an explicitly stopped/terminated container ends the wait, and a "not found"/"destroyed" describe error is trusted only once a container has been observed or the 2m startup grace has passed. Transient describe errors are retried until the TTL+5m deadline instead of destroying the instance. The first check logs the raw describe JSON (runner first stop check) and later phase changes are logged as runner instance phase changed.
  • Daemon-mode runners drain before the TTL instead of accepting jobs the instance cannot finish. The dispatcher passes FORGEJO_RUNNER_TTL_DEADLINE_EPOCH (Unix seconds, computed before nsc run) and FORGEJO_RUNNER_DRAIN_SECONDS (default: TTL/3 clamped to 5m..30m and at most TTL/2, e.g. 30m for a 90m TTL, 5m for 15m); the bootstrap runs forgejo-runner daemon in the background, writes runner.shutdown_timeout = drain - 60s (min 30s) into runner.yaml, and sends SIGTERM at deadline - drain. forgejo-runner then stops polling, lets an in-flight job finish within shutdown_timeout, and exits, which stops the container and lets the monitor destroy the instance. Set FORGEJO_RUNNER_DRAIN_SECONDS in a target's env to override the margin.
• FORGEJO_RUNNER_ONE_JOB_WAIT_TIMEOUT / FORGEJO_RUNNER_ONE_JOB_RETRY_INTERVAL tune the bounded retry used only when FORGEJO_RUNNER_MODE=one-job and Forgejo has not assigned a task immediately after registration. Defaults are 300 seconds and 5 seconds.
• namespace.macos_selectors – macOS support-disk selectors, for example macos.version=26.x,image.with=xcode-26. This is independent of a custom image ID so key.store can keep pinning the OS/Xcode lane while rolling a custom Namespace macOS image.
• namespace.macos_custom_image_id – optional explicit Namespace custom macOS image/base ID. When set, the dispatcher passes it to nsc create through the Compute experimental macos_base_image_id field. Leave it empty until the image exists in the Namespace workspace; selectors remain the safe default.
• macOS cache volumes use the same namespace.cache_volumes / NSC_CACHE_VOLUMES contract as Linux. The dispatcher attaches cache:* volume specs to macOS Compute and exports NSC_CACHE_PATH=/cache/nscloud plus KEYSTORE_BAZEL_STORAGE_CACHE=/cache/bazel when those mount points are present.
• namespace.macos_create_backend – how macOS instances are created: nsc (nsc create --bare ..., the path every production launch has used), compute (Namespace Compute API CreateInstanceRequest over Connect, authenticated with the same NSC_TOKEN_FILE bearer token the CLI uses; the dispatcher polls DescribeInstance until the instance is RUNNING and its ssh service, when listed, is READY, logging the phase and the application/container/service status, and destroys an instance that errors or never becomes ready), or auto/empty (the default: compute when macos_application_image is set, nsc otherwise). Both backends then run the same Compute-SSH bootstrap, TTL cut and nsc destroy. The two backends are not the same wire API: the CLI talks to the private InstanceService (selectors/volumes/base image in experimental, --bare = feature EXP_DISABLE_KUBERNETES), the compute backend to the public v1beta ComputeService (Shape.Selectors, Volumes, Experimental.MacosBaseImageId, no Kubernetes feature). The v1beta create has not been exercised for this tenant since 2026-05 (a 2026-02 note said Compute rejected the macOS support-disk selectors then), so flip one target to macos_create_backend: compute without an application first and watch the macos compute instance phase logs before enabling the application.
• namespace.macos_application_image – optional Namespace macOS Application: an nscr.io image built with nsc macos package <dir> --name <repo> whose command Namespace runs on the instance at VM start, forwarded as Applications: [{Name, ImageRef, Command, Args}] on the create request. nsc create (0.0.557) has no flag for applications, so setting this selects the compute backend (an explicit macos_create_backend: nsc is rejected). key.store uses it to hydrate /nix from a zstd snapshot of the aarch64-darwin closures, since macOS instances cannot put /nix on a cache volume; pin the image by digest and refresh it from the same pipeline as macos_custom_image_id. Companions: macos_application_name (default: the image's last path component), macos_application_command (default ./entrypoint -- Namespace unpacks the image at /opt/<application name> and resolves a bare word via $PATH, exit 127; runs as user runner with PACKAGE_ROOT set; image ENTRYPOINT/CMD are ignored by Namespace), macos_application_args, and macos_application_ready_file – a path the application creates when it is done. The dispatcher exports KEYSTORE_NSC_MACOS_APPLICATION_NAME / _IMAGE / _READY_FILE to the bootstrap, which waits for the ready file (bounded by KEYSTORE_NSC_MACOS_APPLICATION_READY_TIMEOUT, default 900 s; an optional KEYSTORE_NSC_MACOS_APPLICATION_FAILED_FILE ends the wait as soon as the application reports failure; both can be set per autoscaler target via env) before touching /nix, so a snapshot import and the Nix installer never race. On timeout/failure it logs, exports KEYSTORE_NSC_MACOS_APPLICATION_READY=0 and installs Nix itself as before; KEYSTORE_NSC_MACOS_APPLICATION_WAIT=job (autoscaler target env) instead registers the runner without waiting: while the application is still running the bootstrap exports KEYSTORE_NSC_MACOS_APPLICATION_PENDING=1, skips its own Nix install, and the job-side reader (key.store Scripts/nix-snapshot-state.sh via Scripts/ensure-nix.sh, driven by KEYSTORE_NSC_MACOS_APPLICATION_READY_FILE with _READY unset = "snapshot expected") waits for the outcome at the job's first Nix use -- including the window before the entrypoint's first marker (start grace) -- and only installs Nix itself when the application failed or never appeared; a Nix the application installed is put on PATH first so the installer is never run twice. Application contract: Namespace applications are JOB workloads and the instance terminates when the application's command exits (v1beta ApplicationRequest.workload_type), so the command must import, create the ready file, and then keep running (sleep loop, exit on SIGTERM) until the dispatcher destroys the instance; an entrypoint that exits after the import kills the runner. The command is resolved by Namespace as an absolute path, a path relative to the image root, or via the base image's default $PATH; the image is a plain tar of the packaged directory, so ship an executable file (nsc macos package keeps modes). Whoever can push the referenced nscr.io repository runs code as the instance's application user on every macOS runner: pin @sha256: digests (the dispatcher warns at startup for tag references).
• namespace.macos_create_wait_timeout / namespace.macos_create_timeout – macOS runner creation timeouts. The first value is passed to nsc create --wait_timeout; the second is the dispatcher-side command guard and must be longer. KEYSTORE_NSC_MACOS_CREATE_WAIT_TIMEOUT and KEYSTORE_NSC_MACOS_CREATE_TIMEOUT override generated configs, which is useful for host-specific Nix deployments when Namespace macOS capacity is slow.
• namespace.cache_volumes – optional default nsc run --volume specs attached before the runner bootstrap starts. Prefer leaving this empty and assigning cache volumes per autoscaler target with env.NSC_CACHE_VOLUMES; a shared default cache can be overwritten by unrelated concurrent Linux jobs. When a /cache/gradle volume is present, the dispatcher exports GRADLE_USER_HOME=/cache/gradle for the job. When a /cache/bazel volume is present, the cache helper appends Bazel disk/repository cache settings to the generated Namespace bazelrc and Scripts/ensure-nix.sh configures /cache/bazel/nix-binary-cache?trusted=true as the first Nix substituter via NIX_CONFIG. Android/F-Droid jobs hydrate that local file cache from nix.key.store dev-shell profiles, and Bazel Android actions inherit the cache environment so nested nix run .#android-gradle calls use the same local cache. When a previous profile snapshot exists, hydration imports that profile from the local binary cache before evaluating the dev shell. Direct /nix cache-volume reuse is only an opportunistic optimization. Android Bazel lanes also hydrate .#android-gradle as a package output because the nested genrule runs it outside the .#android dev-shell profile closure.

Running locally

# Ensure nsc is available (e.g. `go build ./foundation/cmd/nsc`)
cp config.example.yaml config.yaml
nix develop   # optional dev shell with Go toolchain
go run ./cmd/forgejo-nsc-dispatcher --config config.yaml

API example:

curl -X POST http://localhost:8080/api/v1/dispatch \
  -H 'Content-Type: application/json' \
  -d '{
    "count": 1,
    "ttl": "20m",
    "labels": ["namespace-profile-linux-medium"],
    "scope": {"level": "repository", "owner": "example", "name": "app"}
  }'

Deploying with Nix (and the container image)

• nix build .#packages.x86_64-linux.container-amd64 produces a deterministic tarball containing the service, the nsc binary, BusyBox, and forgejo-runner.
• The key.store forge does not run the container: it builds forgejo-nsc-dispatcher, forgejo-nsc-autoscaler and forgejo-nsc-macos-codex from this flake through its forgejo_nsc_src input and runs them as systemd services.
• CI (.forgejo/workflows/, runs on the forge's Namespace runners; the labels must be targets of the forge autoscaler's forgejo-nsc instance in key-store/key.store nixos/hosts/keystore-forge/common.nix): Build Go runs gofmt/vet/test plus a darwin/arm64 cross-build and builds the three Go packages with Nix; Build Container builds the amd64 and arm64 images with Nix as a build check. Nothing is pushed to a registry (the former ghcr.io publish needed a GitHub token the forge does not have).

How this fits behind Caddy (last-mile networking)

The dispatcher is just an HTTP server. You can:

1. Run it anywhere that can reach Forgejo and Namespace: bare metal, Namespace cluster, Kubernetes, Fly, etc.

2. Put Caddy (or any reverse proxy) in front to terminate TLS, do auth, or rewrite URLs. For example:

   forgejo-dispatcher.example.com {
     reverse_proxy 127.0.0.1:8080
     basicauth /api/* {
       user JDJhJDE...
     }
   }
   

The service doesn’t assume Caddy, nor does it manipulate HTTP clients directly – it simply waits for POST requests. As long as the dispatcher can reach Forgejo’s REST API and run the nsc binary, you can drop it anywhere.

Autoscaling (webhook + poller)

If you don’t want to call /api/v1/dispatch manually, there’s a companion autoscaler (cmd/forgejo-nsc-autoscaler) that watches Forgejo job queues and triggers the dispatcher for you. It operates in two modes simultaneously:

1. Polling – every instance can poll GET /api/v1/.../actions/runners to keep a minimum number of idle Namespace runners per label. Because Forgejo's runner-job API does not include timestamps, production should set ignore_existing_jobs_on_start: true; the autoscaler records a startup high-watermark per target and only dispatches for newer queued jobs.
2. Webhooks – once Forgejo reaches the autoscaler via the /webhook/{name} endpoint, the autoscaler reacts to workflow_job events in real time. Each payload is mapped to a target label set and results in a dispatch call.

You can manage multiple Forgejo instances by listing them under instances in autoscaler.example.yaml:

listen: ":8090"
dispatcher:
  url: "http://dispatcher:8080"

instances:
- name: key-store
  forgejo:
    base_url: "https://git.key.store"
    token: "PENDING-FORGEJO-PAT"
  scope:
    level: "repository"
    owner: "key-store"
    name: "key.store"
  disable_polling: false
  poll_after_webhook: true
  ignore_existing_jobs_on_start: true
  poll_interval: "2m"
  webhook_secret_file: "/run/secrets/forgejo-nsc-webhook-secret"
  webhook:
    url: "https://nsc-autoscaler.key.store/webhook/key-store"
    content_type: "json"
    events: ["workflow_job"]
    active: true
  targets:
    - labels: ["namespace-profile-linux-medium"]
      min_idle: 0  # set to 0 to scale-to-zero between jobs
      max_dispatch_per_poll: 2
      min_dispatch_interval: "2m"
      ttl: "20m"
    - labels: ["namespace-profile-windows-large"]
      min_idle: 0
      max_dispatch_per_poll: 1
      min_dispatch_interval: "5m"
      ttl: "45m"
      machine_type: "windows/amd64:8x16"

Set max_dispatch_per_poll on any polled targets that can accumulate stale queued jobs. A zero or missing value means "no cap". key.store production uses signed webhooks plus high-watermark polling so stale queued jobs that existed before autoscaler startup cannot spend the Namespace tenant quota. Set min_dispatch_interval when startup failures can briefly register and then drop a runner before it takes a job; this keeps polling from relaunching the same target every interval while the underlying image or capacity issue is being debugged. On top of that the autoscaler keeps a per-target circuit breaker: every consecutive failed dispatch (the dispatcher answered non-2xx, e.g. a Windows bootstrap that never came up) doubles the effective interval (2m -> 4m -> 8m ...) up to max_dispatch_backoff (default 30m); the next successful dispatch resets it. Without it a job whose instances never bootstrap re-created an 8x16 Windows instance every interval until Forgejo's stale-job reaper cancelled the job (2026-08-17).

Windows instances

Windows guests are reached through the Namespace ssh service, not through nsc instance port-forward/proxy: the dispatcher calls the Compute API GetSSHConfig for the instance (ssh.<region>.namespace.so, user = instance id, issued key), opens a direct-tcpip forward to 127.0.0.1:5985 on the guest and speaks WinRM/NTLM over it with the RDP credentials from nsc instance proxy -s rdp. Live evidence (2026-08-17, ord4): the gate tunnels behind port-forward/proxy fail with websocket: bad handshake for the whole instance lifetime (5985, 22, even the READY rdp service on 3389; with and without instance:dial_host in the token), while the ssh service authenticates (it cannot exec on Windows -- /bin/bash is not in %PATH% -- but forwards fine) and WinRM answers HTTP/1.1 401 ... Microsoft-HTTPAPI/2.0 about 20 s after nsc create returns. port-forward stays as a fallback transport. Before any bootstrap attempt the dispatcher probes the tunnel for a real HTTP status line from /wsman (a bare TCP connect says nothing: the forwarder accepts locally and resets later) with a windows_ready_timeout budget (default 10m, env KEYSTORE_NSC_WINDOWS_READY_TIMEOUT), logs windows guest ready ... after=<duration>, then retries the bootstrap for windows_bootstrap_retry (default 3m, KEYSTORE_NSC_WINDOWS_BOOTSTRAP_RETRY). NSC_WINDOWS_E2E=1 go test ./internal/nsc -run TestWindowsWinRMScriptRoundTrip -v (with nsc on PATH, NSC_TOKEN_FILE, NSC_COMPUTE_BASE_URL) proves the whole path against a fresh 4x8 instance in ~25 s.

Bootstrap failures are classified by what the pywinrm driver reports: a Python traceback / connection reset is a transport failure and is retried within windows_bootstrap_retry, then handed to the next transport; a winrm_stage_exit=<n> / winrm_exit=<n> marker means the PowerShell ran on the guest and exited non-zero, which ends the launch on that transport (the transport works, replaying the same script through another tunnel cannot help): a script exit (winrm_exit) ends it immediately, a staging exit (winrm_stage_exit: writing the script file, idempotent, nothing registered) is retried for 60 s first so a guest still settling after boot or Defender holding the half-written file gets a second chance. Before this rule "The command line is too long." and the runner-zip 404 were each replayed for the whole 3-minute window and then handed to port-forward, which probed a guest that was already up for the rest of the 10-minute ready budget, up to 6m50s / 40 resets per launch on 2026-08-17. Once one transport has reached the guest a fallback transport is probed for 45 s only. Every successful launch logs windows bootstrap finished (with the transcript size; the transcript tail itself is a debug line, since it is job output).

The guest runs forgejo-runner one-job in a bounded loop: forgejo-runner v12 fetches once and exits non-zero with "no task received" when nothing is assigned yet, so the bootstrap keeps asking for FORGEJO_RUNNER_ONE_JOB_WAIT_TIMEOUT seconds (default 300, the same knob as the Linux/macOS bootstraps; set it in the target's env) and then exits 75, which the dispatcher logs as windows runner received no job and reports as a failed launch (the instance is destroyed at once instead of the runner being re-registered for 3 minutes).

The bootstrap needs a forgejo-runner.exe: Forgejo publishes no Windows runner binaries (code.forgejo.org releases carry linux amd64/arm64 only; the old forgejo-runner-<v>-windows-amd64.zip URL answers 404 and Expand-Archive then fails with "End of Central Directory record could not be found"). Set namespace.windows_runner_url (env KEYSTORE_NSC_WINDOWS_RUNNER_URL) to a build you host -- key.store publishes its Nix build to https://blob.key.store/key-store-mirrors/forgejo-runner-windows/<version>/forgejo-runner.exe with a .sha256 sidecar, which the guest verifies. Runners register --ephemeral and run one-job.

Cache/persistent volumes are not supported on Windows compute instances: the Compute API rejects nsc create --volume on a Windows machine type with InvalidArgument ("persistent volumes are not supported on Windows instances"; cache volumes fail identically), any mount format, on nsc 0.0.516 and 0.0.557 (the same Linux create succeeds). The dispatcher therefore never attaches volumes on Windows, so a Windows job re-provisions its toolchain each run; persisting state needs Namespace enabling Windows compute volumes or a custom Windows base image. The nsc CLI is only half of it -- its --volume flag splits on : and cannot even express a drive-letter mount point -- so TestLiveWindowsVolumeProbe (KEYSTORE_LIVE_WINDOWS_VOLUME_PROBE=1, KEYSTORE_LIVE_PROBE_WINDOWS_VOLUME=cache:<mount>|pv:<mount>) sends the CreateInstanceRequest directly and inspects the guest over WinRM. Measured 2026-08-19 05:22-05:33Z (ord4, windows/amd64:4x8): a CACHE volume at C:\cache\probe, K:\ or D:\cache\probe is accepted (instance id returned) but the instance never leaves PENDING and Namespace destroys it itself after 40-52 s with no shutdown reason; /cache/probe, K: and an empty mount point are InvalidArgument; experimental.persistent_volumes with mount_point: K:\ is accepted and the instance boots (RUNNING in 22-24 s) but the guest has no such disk (Get-Disk/Get-Volume/mountvol show only C: 200 GB, D: 100 GB and Namespace's own read-only image disks mounted under C:\namespace\{nsc,github-runner,powertoys}). If a mounted volume ever appears, windowsCacheVolumeEnv already exports KEYSTORE_WINDOWS_CACHE_ROOT (key.store's git-mirror + swiftc build/module cache root) from the nscloud mount, next to NSC_CACHE_PATH and KEYSTORE_BAZEL_STORAGE_CACHE.

For key.store, the live forge should prefer managed runtime tokens under /var/lib/key-store/forgejo-runtime-tokens/ plus repo-generated config. The local Scripts/provision-forgejo-nsc.sh path only updates the older encrypted dispatcher/autoscaler YAML secrets and should be treated as a legacy recovery bridge, not the default operating model.

Run it next to the dispatcher:

go run ./cmd/forgejo-nsc-autoscaler --config autoscaler.yaml
# or build the binary/container via `nix build .#forgejo-nsc-autoscaler`

For production, prefer signed webhooks, keep min_idle: 0 for scale-to-zero, and enable ignore_existing_jobs_on_start whenever polling is enabled. Set poll_after_webhook: true when Forgejo can leave dependency-unlocked jobs waiting without sending a fresh webhook. The autoscaler auto-creates/updates the webhook using the PAT.

In Forgejo add a webhook pointing to https://nsc-autoscaler.key.store/webhook/key-store with the shared secret (or let the autoscaler create it by specifying webhook.url in config). If polling is enabled only for bootstrap, leave poll_after_webhook false so the autoscaler stops polling after the first valid webhook.