Fix forge deploy OOM on cold Fil-C build (swap + CPU-bound deploys)

founder commented

2026-07-18 13:29:25 +00:00

Owner

Root cause of the 5-week-stale forge: a cache-miss Fil-C rebuild (full LLVM toolchain, ~14GB per ld.gold link) OOM-killed the deploy on a 124GB host with 65GB baseline usage and zero swap. Adds a 96GB swapfile + AllowedCPUs=0-7 on the deploy build (LLVM honors the cgroup CPU set, not NIX_BUILD_CORES). Verified: full nixos-rebuild build of .#keystore-forge from main now completes on the forge.

Deploying the fixed SHA next.

🤖 Generated with Claude Code

Root cause of the 5-week-stale forge: a cache-miss Fil-C rebuild (full LLVM toolchain, ~14GB per ld.gold link) OOM-killed the deploy on a 124GB host with 65GB baseline usage and zero swap. Adds a 96GB swapfile + AllowedCPUs=0-7 on the deploy build (LLVM honors the cgroup CPU set, not NIX_BUILD_CORES). Verified: full nixos-rebuild build of .#keystore-forge from main now completes on the forge. Deploying the fixed SHA next. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

founder added 1 commit

2026-07-18 13:29:25 +00:00

forge: fix deploy OOM on cold Fil-C build (swap + CPU-bound deploys)

Release: If Needed / Check (Release Needed) (push) Successful in 6s

Details

Deploy: API / Deploy (API) (push) Successful in 3m7s

Details

Deploy: Web / Deploy (Web) (push) Successful in 6m10s

Details

fae3af1bf1

The forge host stopped deploying because its config bakes Fil-C protected
utilities (KSE-0164), and Fil-C is a full LLVM/Clang toolchain that builds
from source on a cache miss. Its parallel ld.gold link steps peak at ~14 GB
each; on a 124 GB host whose own services already use ~65 GB, with zero swap
and unbounded nix parallelism, a few concurrent links triggered a global OOM
that killed the deploy (and risked killing forge services). This is why
deploy-forge-host has not landed since the host drifted five weeks behind main.

Fixes:
- deploy-forge-host.yml: run the deploy build under AllowedCPUs=0-7. LLVM/ninja
  ignore NIX_BUILD_CORES for link parallelism, but honor the cgroup CPU set via
  nproc, so this reliably caps concurrent links (~8, ~112 GB peak) without
  slowing Namespace CI (which does not use this path).
- keystore-forge.nix: add a persistent 96 GB swapfile so transient link spikes
  overflow to disk instead of OOM-killing, and set moderate nix cores/max-jobs.

Verified: a full nixos-rebuild build of .#keystore-forge from current main now
completes (TOPLEVEL_BUILD_EXIT=0) on the forge with a 96 GB swap + 8-CPU bound.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

founder requested reviews from manager, assistant, infra, cmo, security-agent, ios, web, cdo, forge, runner, agent, billing, windows, android, devex, release-runner, linux

2026-07-18 13:29:25 +00:00

founder merged commit fae3af1bf1 into main

2026-07-18 13:29:41 +00:00

Rows
Columns

Fix forge deploy OOM on cold Fil-C build (swap + CPU-bound deploys) #4