ci: Checkout never pages -- git log in the runner pty hung every job (runs 18819-18822) #13

Merged

founder merged 2 commits from claude/fnsc-ci-no-pager into main 2026-08-19 08:43:33 +00:00

Conversation 0 Commits 2 Files changed 4 +93 -21

founder commented 2026-08-19 08:26:32 +00:00

Owner

Copy link

Runs 18819-18822 (the first forge-native runs, PR #11) were picked up but every job hung right after Checkout printed the commit line, then died ~19 min later (this step has been cancelled: ctx canceled on the medium lane, stuck running forever on the arm64 one-job lane).

Root cause: the forgejo-runner host executor runs each run: step in a pty (forgejo/act host_environment.go: if true /* allocate Terminal */) and writes a 0x04 keepalive into it every second. git log -1 --oneline sees a tty, starts the default pager (pager -> more on the Debian base image), and the keepalive Ctrl-D pages it forever -- the blank lines after the commit line in every task log are more repainting. Reproduced byte-for-byte in debian:12-slim with a pty + 1 Hz 0x04 writer.

Fix: GIT_PAGER=cat / PAGER=cat / GIT_TERMINAL_PROMPT=0 + git --no-pager log, git fetch -q; timeout-minutes backstops under the lane TTLs; workflow_dispatch on both workflows; header comment documenting the pty/keepalive constraint.

🤖 Generated with Claude Code

Runs 18819-18822 (the first forge-native runs, PR #11) were picked up but every job hung right after Checkout printed the commit line, then died ~19 min later (`this step has been cancelled: ctx canceled` on the medium lane, stuck `running` forever on the arm64 one-job lane). **Root cause:** the forgejo-runner host executor runs each `run:` step in a pty (forgejo/act `host_environment.go`: `if true /* allocate Terminal */`) and writes a 0x04 keepalive into it every second. `git log -1 --oneline` sees a tty, starts the default pager (`pager` -> `more` on the Debian base image), and the keepalive Ctrl-D pages it forever -- the blank lines after the commit line in every task log are `more` repainting. Reproduced byte-for-byte in debian:12-slim with a pty + 1 Hz 0x04 writer. **Fix:** `GIT_PAGER=cat` / `PAGER=cat` / `GIT_TERMINAL_PROMPT=0` + `git --no-pager log`, `git fetch -q`; `timeout-minutes` backstops under the lane TTLs; `workflow_dispatch` on both workflows; header comment documenting the pty/keepalive constraint. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

founder added 1 commit 2026-08-19 08:26:32 +00:00

ci: Checkout never pages -- git log in the runner pty hung every job (runs 18819-18822) ... 7ef3937860

The first forge-native runs were picked up (medium + arm64 runners registered
and took their tasks) but every job stalled right after Checkout printed the
commit line and died ~19 min later with "this step has been cancelled: ctx
canceled" (medium lane TTL drain) or stayed `running` forever (arm64 one-job
lane, hard-killed at its TTL without reporting).

Root cause: the forgejo-runner host executor runs every `run:` step in a pty
(forgejo/act pkg/container/host_environment.go: `if true /* allocate Terminal
*/`) and writes a 0x04 keepalive byte into it once a second. `git log -1
--oneline` sees a tty, spawns the default pager (`pager` -> more on the
debian-12-slim arm64 base image, less/more on devbox-nix), and the keepalive
Ctrl-D pages it forever -- the ~20 blank lines after the commit line in
every task log are `more` repainting. Reproduced byte-for-byte in
debian:12-slim with a pty + 1 Hz 0x04 writer: "<commit>\r\n" + 22 x "\r\n"
+ "--More--(END)", then one more screen per second, never exits; with
GIT_PAGER=cat or --no-pager the step ends immediately.

- env GIT_PAGER=cat / PAGER=cat / GIT_TERMINAL_PROMPT=0 on both workflows,
  `git --no-pager log`, `git fetch -q` (progress meters are tty-driven too
  and only bloat the log); header comment documents the pty + keepalive so
  no future step pages or prompts.
- timeout-minutes as a backstop below the lanes' TTLs (18 m medium, 26 m
  container jobs) so a wedged step fails with a timeout instead of the
  opaque cancel / never-reported arm64 task.
- workflow_dispatch on both, so a run can be re-triggered without a push.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

founder referenced this pull request from a commit 2026-08-19 08:40:14 +00:00

ci: unsandboxed Nix builds on the Namespace runners (Go telemetry vs /homeless-shelter); scrub the guard test's env

founder added 1 commit 2026-08-19 08:40:14 +00:00

ci: unsandboxed Nix builds on the Namespace runners (Go telemetry vs /homeless-shelter); scrub the guard test's env ... 908a9db185

Second round on PR #13 (runs 18858/18859): Checkout now ends, Ensure Nix
installs Nix 2.33.0 into the fresh fnsc /nix volumes (3 s on arm64), and the
jobs got to their real work -- and failed twice more:

1. "Nix Build" / "Image (amd64|arm64)": `error: home directory
   '/homeless-shelter' exists; please remove it to assure purity of builds
   without sandboxing`. The runner containers cannot create mount
   namespaces, so Nix's sandbox-fallback runs every builder unsandboxed with
   HOME=/homeless-shelter as a real path. `go` creates
   $HOME/.config/go/telemetry on every invocation (reproduced with
   golang:1.25: `env -i HOME=/homeless-shelter go version` leaves the tree
   behind; GOTELEMETRY=off still creates .config/go), so the first
   buildGoModule build creates /homeless-shelter and Nix refuses to start
   the next derivation (Image (amd64): 20 s into the dispatcher build; the
   same runner's following Nix Build job failed on its first derivation).
   flake.nix: the three buildGoModule outputs share goModuleCommon with
   `preBuild = export HOME="$TMPDIR"`; buildGoModule forwards preBuild to
   the go-modules FOD too (`go mod download` has the same hazard), and the
   FOD's output path does not depend on it, so nix.key.store keeps serving
   it. The workflows also `rm -rf /homeless-shelter` before Ensure Nix, the
   key.store convention (images that pre-create it, warm daemon runners).

2. "Go Test": TestMacOSCacheVolumeGuardIsInertWithoutVolumes failed because
   runMacOSCacheVolumeGuard inherits os.Environ(), and on a cache-volume lane
   the dispatcher exports KEYSTORE_NSC_CACHE_VOLUME_ACTIVE=1 for the job
   itself -- the test saw ACTIVE=1 it never set. The helper now scrubs
   HOME/KEYSTORE_NSC_CACHE_VOLUME_*/KEYSTORE_BAZEL_STORAGE_CACHE/
   NSC_CACHE_PATH from the inherited environment (verified on the forge with
   those variables exported: ok).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

founder merged commit 908a9db185 into main 2026-08-19 08:43:33 +00:00

founder deleted branch claude/fnsc-ci-no-pager 2026-08-19 08:43:34 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

key-store/forgejo-nsc!13

No description provided.