dispatcher: create macOS instances through the Compute API with a Namespace application #3

Merged
founder merged 3 commits from claude/macos-application-request into main 2026-08-17 02:02:07 +00:00
Owner

KSE-0186 D8 (key.store PR #125): macOS runners cannot put /nix on a cache volume, so key.store hydrates it at VM start from a Namespace macOS Application (nsc macos package → nscr.io). The nsc create CLI (0.0.557) has no applications flag, so the macOS create gains a Compute API backend.

Default behaviour unchanged: with no macos_application_image the backend resolves to nsc and the create path is byte-for-byte today's nsc create (verified by the existing tests). Everything after create (Compute SSH bootstrap bound to the TTL, drain, nsc destroy, volume mapping/retry) stays shared.

  • internal/nsc/macos_compute.go (new): CreateInstanceRequest{Shape (vcpu/mem from machine_type, os=macos, arch, support-disk selectors), Deadline=now+TTL, Labels, Volumes (NSC_CACHE_VOLUMES → CACHE), Applications=[{Name, ImageRef, Command, Args}], Experimental.MacosBaseImageId}; bearer from NSC_TOKEN_FILE; ResourceExhausted retry with backoff; polls DescribeInstance until RUNNING and the ssh service is READY (when listed), logging phase changes + application status (containers / services / shutdown reasons); destroys on ERROR/DESTROYED/timeout with the shutdown reasons in the error.
  • Config (namespace.*, mirrored in nsc.Options, logged at startup): macos_create_backend = auto (default: compute iff an application is configured) | nsc | compute; macos_application_image (nscr.io only; startup Warn unless @sha256:-pinned — whoever can push that repo runs code at VM start on every macOS runner), macos_application_name (default: image's last path component), macos_application_command (default entrypoint), macos_application_args, macos_application_ready_file.
  • Bootstrap: exports KEYSTORE_NSC_MACOS_APPLICATION_{NAME,IMAGE,READY_FILE}; when a ready file is named, waits for it (KEYSTORE_NSC_MACOS_APPLICATION_READY_TIMEOUT, default 900 s; optional KEYSTORE_NSC_MACOS_APPLICATION_FAILED_FILE ends the wait early) before the Nix installer decision, and puts /nix/var/nix/profiles/default/bin on PATH first so an application-installed Nix is seen instead of running the Determinate installer a second time; exports KEYSTORE_NSC_MACOS_APPLICATION_READY=1|0.
  • Documented contract (README, config.example.yaml, code): ApplicationRequest is a JOB workload — the instance terminates when the command exits — so the entrypoint must keep running after its ready marker (key.store's entrypoint holds until SIGTERM / <state>/release).
  • Tests: fake Compute API (Create/Describe/GetSSHConfig) + fake sshd + fake nsc — request shape, phase wait incl. RUNNING/ssh-PENDING hold, volume rejection + unparseable-spec retry, ResourceExhausted retry then give-up, ERROR/timeout destroy, wrapper env, backend resolution/validation, config parsing, bootstrap ordering (wait < PATH < installer) executed under bash 5 and bash 3.2; writeFakeExecutable retries ETXTBSY (pre-existing flake at main). go vet + go test ./... green (linux/amd64 go1.26.3, darwin/arm64 go1.25.5); go.mod/go.sum unchanged.

Unverified against the live API (no instance created): the v1beta macOS CreateInstance for tenant osibqsl33bahe (the 2026-02 CLI switch cited support-disk selector rejection) — key.store stays on the nsc path until a live probe with macos_create_backend: compute succeeds.

Workflow-orchestrated (implementer + adversarial reviewer); reviewer fixes folded in.

🤖 Generated with Claude Code

KSE-0186 D8 (key.store PR #125): macOS runners cannot put `/nix` on a cache volume, so key.store hydrates it at VM start from a **Namespace macOS Application** (`nsc macos package` → nscr.io). The `nsc create` CLI (0.0.557) has no applications flag, so the macOS create gains a Compute API backend. **Default behaviour unchanged**: with no `macos_application_image` the backend resolves to `nsc` and the create path is byte-for-byte today's `nsc create` (verified by the existing tests). Everything after create (Compute SSH bootstrap bound to the TTL, drain, `nsc destroy`, volume mapping/retry) stays shared. - `internal/nsc/macos_compute.go` (new): `CreateInstanceRequest{Shape (vcpu/mem from machine_type, os=macos, arch, support-disk selectors), Deadline=now+TTL, Labels, Volumes (NSC_CACHE_VOLUMES → CACHE), Applications=[{Name, ImageRef, Command, Args}], Experimental.MacosBaseImageId}`; bearer from `NSC_TOKEN_FILE`; ResourceExhausted retry with backoff; polls `DescribeInstance` until RUNNING **and** the `ssh` service is READY (when listed), logging phase changes + application status (containers / services / shutdown reasons); destroys on ERROR/DESTROYED/timeout with the shutdown reasons in the error. - Config (`namespace.*`, mirrored in `nsc.Options`, logged at startup): `macos_create_backend` = `auto` (default: compute iff an application is configured) | `nsc` | `compute`; `macos_application_image` (nscr.io only; startup Warn unless `@sha256:`-pinned — whoever can push that repo runs code at VM start on every macOS runner), `macos_application_name` (default: image's last path component), `macos_application_command` (default `entrypoint`), `macos_application_args`, `macos_application_ready_file`. - Bootstrap: exports `KEYSTORE_NSC_MACOS_APPLICATION_{NAME,IMAGE,READY_FILE}`; when a ready file is named, waits for it (`KEYSTORE_NSC_MACOS_APPLICATION_READY_TIMEOUT`, default 900 s; optional `KEYSTORE_NSC_MACOS_APPLICATION_FAILED_FILE` ends the wait early) **before** the Nix installer decision, and puts `/nix/var/nix/profiles/default/bin` on PATH first so an application-installed Nix is seen instead of running the Determinate installer a second time; exports `KEYSTORE_NSC_MACOS_APPLICATION_READY=1|0`. - Documented contract (README, config.example.yaml, code): `ApplicationRequest` is a JOB workload — *the instance terminates when the command exits* — so the entrypoint must keep running after its ready marker (key.store's entrypoint holds until SIGTERM / `<state>/release`). - Tests: fake Compute API (Create/Describe/GetSSHConfig) + fake sshd + fake nsc — request shape, phase wait incl. RUNNING/ssh-PENDING hold, volume rejection + unparseable-spec retry, ResourceExhausted retry then give-up, ERROR/timeout destroy, wrapper env, backend resolution/validation, config parsing, bootstrap ordering (wait < PATH < installer) executed under bash 5 and bash 3.2; `writeFakeExecutable` retries ETXTBSY (pre-existing flake at main). `go vet` + `go test ./...` green (linux/amd64 go1.26.3, darwin/arm64 go1.25.5); go.mod/go.sum unchanged. Unverified against the live API (no instance created): the v1beta macOS `CreateInstance` for tenant osibqsl33bahe (the 2026-02 CLI switch cited support-disk selector rejection) — key.store stays on the `nsc` path until a live probe with `macos_create_backend: compute` succeeds. Workflow-orchestrated (implementer + adversarial reviewer); reviewer fixes folded in. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The nsc package's tests keep forking in the background (Linux stop monitors
run `nsc describe` on millisecond intervals, macOS teardowns run `nsc
destroy`), so a fork that lands while a test is still writing its fake nsc
lends the child the write descriptor until it execs, and exec'ing the fresh
script in that window fails with ETXTBSY. On the forge (linux/amd64,
go1.26.3) TestLinuxLaunchPassesDaemonRunnerModeEnv failed this way about
once in eight runs at 92cd5bb0:

  fork/exec /tmp/.../nsc: text file busy

writeFakeExecutable writes the stand-in and probes it with BASH_ENV pointing
at `exit 0` (bash leaves before the script body runs, so nothing is logged or
counted) until the exec succeeds; every fake nsc / forgejo-runner writer uses
it. 10/10 green `go test ./...` runs on the forge afterwards.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Founder direction (2026-08-17): macOS runners cannot put /nix on a cache
volume, so /nix is hydrated at VM start from a Namespace macOS Application
(`nsc macos package` -> nscr.io image + zstd snapshot of the aarch64-darwin
closures, refreshed by the same pipeline as the custom base image). Only the
Compute API can attach an application to an instance: `nsc create` (0.0.557)
has no flag for it, and until now every macOS launch went through the CLI.

- internal/nsc/macos_compute.go: Compute-API create backend. Builds
  CreateInstanceRequest{Shape (vcpu/mem from machine_type, os=macos, arch,
  the same support-disk selectors as the CLI path), Deadline=now+TTL, labels,
  Volumes from NSC_CACHE_VOLUMES, Applications=[{Name, ImageRef, Command,
  Args}], Experimental.MacosBaseImageId for macos_custom_image_id; no
  Features = --bare}, authenticated with the NSC_TOKEN_FILE bearer token the
  CLI uses, retries CreateInstance with backoff on ResourceExhausted like the
  CLI, then polls DescribeInstance until RUNNING (MacosCreateWaitTimeout
  inside MacosCreateTimeout), logging the phase and what describe exposes
  about the application (container name/id, service readiness, shutdown
  reasons with container_failed_reason; there is no dedicated application
  status field in compute v1beta). ERROR/DESTROYED/timeout destroy the
  instance and fail the launch with the shutdown reasons.
- internal/nsc/macos_nsc.go: launchMacOSRunner (was launchMacOSRunnerViaNSC)
  picks the create backend and keeps everything else shared: private
  ExtraEnv copy, age identity, TTL drain env, cache-volume mapping,
  volume-less retry when Namespace rejects the volume request, tracked state,
  Compute SSH bootstrap bound to the TTL, `nsc destroy` teardown. The CLI
  create moved to createMacOSInstanceViaNSC unchanged.
- Options/config: macos_create_backend (nsc | compute | auto; default auto =
  compute when macos_application_image is set, nsc otherwise; explicit nsc +
  application is rejected, compute needs compute_base_url),
  macos_application_image (nscr.io only), macos_application_name (default:
  last path component of the image), macos_application_command (default
  "entrypoint"; image ENTRYPOINT/CMD are ignored by Namespace),
  macos_application_args, macos_application_ready_file. main.go wires them
  and logs the resolved backend at startup.
- Bootstrap: the wrapper exports KEYSTORE_NSC_MACOS_APPLICATION_NAME/_IMAGE/
  _READY_FILE; when a ready file is named, macosBootstrapScript waits for it
  (KEYSTORE_NSC_MACOS_APPLICATION_READY_TIMEOUT, default 900s) before the
  Nix installer runs, so a snapshot import and the installer never race; on
  timeout it logs, exports KEYSTORE_NSC_MACOS_APPLICATION_READY=0 and
  installs Nix itself as before.
- internal/nsc/macos.go: the dead 2026-02 Compute path (launchMacOSRunner
  running the bootstrap itself as an ApplicationRequest, plus its stop
  monitor) is removed; the new backend supersedes it.

Default behaviour is unchanged: without macos_application_image the
dispatcher still runs `nsc create --bare ...` with the same arguments.

Tests: go test ./... on darwin/arm64 (go1.25.5) and 10/10 on the forge
(linux/amd64, go1.26.3, CGO_ENABLED=0). New tests drive an in-process fake
Compute API (CreateInstance/DescribeInstance/GetSSHConfig) plus the fake sshd
and fake nsc: the request carries the application/shape/volumes/deadline,
PENDING->CREATING->RUNNING is waited for, volume rejections retry without
volumes, ResourceExhausted retries until the create timeout, ERROR and
never-RUNNING instances are destroyed, the wrapper env advertises the
application and ready file, config parsing/validation, and the bootstrap
ready-file wait. Not exercised against the live Compute API (no macOS
instance was created for this change).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dispatcher: macOS compute create waits for ssh READY; bootstrap sees an application-installed Nix and stops on a failed marker
Some checks failed
Build Container / Image (amd64) (pull_request) Waiting to run
Build Container / Image (arm64) (pull_request) Waiting to run
Build Container / Publish Manifest (pull_request) Blocked by required conditions
Build Go / Go Test (namespace-profile-linux-large) (pull_request) Waiting to run
Build Go / Go Test (namespace-profile-macos-large) (pull_request) Waiting to run
Build Go / Nix Build (pull_request) Waiting to run
Build Container / Image (amd64) (push) Has been cancelled
Build Container / Image (arm64) (push) Has been cancelled
Build Container / Publish Manifest (push) Has been cancelled
Build Go / Go Test (namespace-profile-linux-large) (push) Has been cancelled
Build Go / Go Test (namespace-profile-macos-large) (push) Has been cancelled
Build Go / Nix Build (push) Has been cancelled
f55ee11339
Review fixes on top of the Compute-API/application change:

- Readiness: DescribeInstance RUNNING alone is not enough for the next
  step (one Compute-SSH dial, no retry). A live macOS runner reports
  metadata.services ssh=READY (and vnc); when Namespace lists the ssh
  service the wait now also requires READY. A missing entry does not
  hold the wait (Namespace does not guarantee PENDING before READY).
- Bootstrap: the SSH session is a non-login `bash -s`, so a Nix that
  the application (or a custom base image) installed is not on PATH and
  `command -v nix` would run the Determinate installer a second time on
  a VM that already has /nix. Export the default profile bin first.
- Bootstrap: KEYSTORE_NSC_MACOS_APPLICATION_FAILED_FILE (optional, per
  autoscaler target env) ends the ready wait as soon as the application
  reports failure instead of burning the whole timeout; the wait
  fragment is a named constant and is executed under bash in tests
  (ready / failed / timeout / garbage timeout / inert).
- Docs: Namespace applications are JOB workloads and the instance
  terminates when the command exits (v1beta ApplicationRequest; a Linux
  instance whose job container stopped shows "job completed" and goes
  DESTROYING) - the entrypoint must keep running after creating the
  ready file. Note that the nsc CLI and the compute backend are
  different wire APIs and that the v1beta macOS create is unexercised
  for this tenant since 2026-05; startup warning for tag (non-digest)
  application images.
founder deleted branch claude/macos-application-request 2026-08-17 02:02:07 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
key-store/forgejo-nsc!3
No description provided.