Skip to content

[aw] Failure Investigator (6h) #518

[aw] Failure Investigator (6h)

[aw] Failure Investigator (6h) #518

Triggered via schedule August 23, 2026 18:54
Status Success
Total duration 19m 30s
Artifacts 10
Fit to window
Zoom out
Zoom in

Annotations

14 errors and 6 warnings
agent
`/exit code line anywhere in the 50-line tail.\n\nRun 32089118620 (same workflow, ~3.5h earlier) also failed but its step-level logs were unavailable during this investigation pass (transient API connectivity issue) — confirmed only via run metadata that it failed at the same workflow.\n</details>\n\n### Probable root cause\n\nTimeout/hang signature, not a clean failure: the CLI process appears to be killed while blocked on a large-diff `Read`, and nothing in the workflow surfaces that as a distinguishable error — the log just goes silent.\n\n### Proposed remediation\n\n1. Add an explicit step-level timeout with a clear `##[error]` message when the Claude Code CLI step exceeds its budget, instead of a bare kill.\n2. Cap or paginate the diff size handed to the agent — large `Read` calls on `pr.diff` are the apparent hang point in both failures.\n3. Watch the next few Design Decision Gate runs for size-correlation; add a diff-size guard before invoking the gate if confirmed.\n\n### Success criteria\n\nDesign Decision Gate runs complete (pass or fail) without silent mid-tool-use termination; timeouts show an explicit error in logs; similarly-sized PR diffs no longer trigger the hang.\n\n**Parent:** #53129\nRelated to #53129\n\n\n\n\n> Generated by [🔍 [aw] Failure Investigator (6h)](https://github.com/github/gh-aw/actions/runs/32109525808) · agent · 251.3 AIC · ⌖ 6.96 AIC · ⊞ 5.9K · [◷](https://github.com/search?q=repo:github%2Fgh-aw+is:issue+%22gh-aw-workflow-call-id:+github%2Fgh-aw%2Faw-failure-investigator%22&type=issues)\n> - [x] expires on Aug 24, 2026, 11:21 PM UTC-08:00\n\n\n\n\n\n\n---\n\n### Correction — this is not a silent hang, it's a 429 invocation-cap failure. Keep this open.\n\n**One-sentence rationale:** Design Decision Gate's run today fails fast and loud with `Maximum LLM invocations exceeded (30 / 30)`, a non-retryable rate-limit guard — not the silent hang this issue currently describes.\n\n**Fresh occurrence (2026-08-20):** Design Decision Gate — [§32394421229](https://github.com/github/gh-aw/actions/runs/32394421229). Claude Code CLI harness logs confirm `isRateLimitError=true` and treat the condition as non-retryable, terminating the run immediately once the 30-invocation cap is hit.\n\n**Why this stays open:** no commit since this issue opened has touched this workflow's `.md`/`.lock.yml` with a targeted fix — the one nearby commit doesn't address the invocation cap. Not closing per the tool-denial-limit rule.\n\n**Proposed remediation:** raise the per-run LLM invocation cap for this workflow, or split its task into smaller steps so it stays under 30 invocations. Update the issue title/description to reflect the real failure mode (429 cap, not hang) so future triage isn't misled.> Generated by [🔍 [aw] Failure Investigator (6h)](https://github.com/github/gh-aw/actions/runs/32406250112) · agent · 238.8 AIC · ⌖ 7.28 AIC · ⊞ 5.9K · [◷](https://github.com/search?q=repo:github%2Fgh-aw+is:issue+%22gh-aw-workflow-call-id:+github%2Fgh-aw%2Faw-failure-investigator%22&type=issues)\n\n\n\n---\n\n### Update (2026-08-22) — reproduced again today, keep this open\n\nDesign Decision Gate failed at the \"Execute Claude Code CLI\" step again: [§32548469031](https://github.com/github/gh-aw/actions/runs/32548469031) (2026-08-22 03:15 UTC). Given the prior correction on this issue (2026-08-20) established the real failure mode as the 30/30 LLM-invocation cap (`Maximum LLM invocations exceeded`), not a silent hang, today's recurrence is most likely the same cap being hit again rather than a new hang — but I could not confirm the exact error line for this run: `gh run view`/log download hit transient `api.github.com` connectivity errors during this investigation pass, same class of gap noted for run 32089118620 earlier in this issue's history.\n\nNot closing — no commit since this issue opened touches `design-decision-gate.md`/`.lock.yml` with a fix for either the invocation cap or the diff-size hang theory, so the underlying condition (whichever it is) is still unaddressed.
agent
Agent execution exited with code 1\n##[error]Process completed with exit code 1.\n```\n`audit` confirms both runs are healthy at the infra layer — firewall shows 73/73 requests allowed, 0 blocked, identical domain mix (`api.openai.com`, `chatgpt.com`, `github.com`, Sentry, Grafana) both times. This rules out egress blackout (#53935) and rules out the cross-engine SIGSEGV pattern in #54186 — that issue's signature is exit code **139**; both of these runs exited with code **1**, and neither shows the \"log stops abruptly right after MCP registration\" shape #54186 describes.\n\n### Affected workflows and run IDs\n\n- `AI Moderator` (`.github/workflows/ai-moderator.lock.yml`), Codex CLI:\n - [§32367352436](https://github.com/github/gh-aw/actions/runs/32367352436) — 2026-08-20 12:09 UTC, failed steps: `agent` and `evals` jobs, \"Execute Codex CLI\"\n - [§32367076703](https://github.com/github/gh-aw/actions/runs/32367076703) — 2026-08-20 12:05 UTC, same signature, 4 minutes earlier\n\n### Probable root cause\n\nUnknown and currently unlogged — the Codex CLI process exits 1 with no captured stdout/stderr reason in either the workflow logs or the `audit` tool's error extraction. Given no network anomaly and no segfault signature, this points at either an unhandled exception inside the Codex CLI wrapper itself or a prompt/tool-call condition specific to `AI Moderator`'s task (issue-moderation triage) that the CLI doesn't surface cleanly.\n\n### Proposed remediation\n\n1. Capture Codex CLI's raw stdout/stderr (not just the exit-code wrapper) in the step log so the actual exception/error text is visible — right now the failure is a black box.\n2. Re-run `AI Moderator` with verbose/debug logging enabled for the Codex engine to catch the next occurrence with full detail.\n3. Once real error text is captured, re-file with the specific exception — do not let this recur as another \"generic exit 1\" entry.\n\n### Success criteria / verification\n\n- Next `AI Moderator` failure (if any) surfaces a specific error message/stack trace instead of a bare exit code.\n- Zero repeat \"exit code 1, clean firewall, no error text\" occurrences for `AI Moderator` over the next 24h.\n\n**References:**\n- https://github.com/github/gh-aw/actions/runs/32367352436\n- https://github.com/github/gh-aw/actions/runs/32367076703\n\nRelated to #54114, distinct from #54186 (exit 139) and #53935 (egress blackout).\nRelated to #54114\n\n\n<!-- gh-aw-tracker-id: aw-failure-investigator -->\n\n> Generated by [🔍 [aw] Failure Investigator (6h)](https://github.com/github/gh-aw/actions/runs/32372296893) · agent · 137.8 AIC · ⌖ 8.41 AIC · ⊞ 5.9K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+is%3Aissue+%22gh-aw-workflow-call-id%3A+github%2Fgh-aw%2Faw-failure-investigator%22&type=issues)\n> - [x] expires <!-- gh-aw-expires: 2026-08-27T13:20:47.586Z --> on Aug 27, 2026, 5:20 AM UTC-08:00\n\n<!-- gh-aw-agentic-workflow: [aw] Failure Investigator (6h), gh-aw-tracker-id: aw-failure-investigator, engine: claude, model: agent, id: 32372296893, workflow_id: aw-failure-investigator, run: https://github.com/github/gh-aw/actions/runs/32372296893 -->\n\n<!-- gh-aw-workflow-id: aw-failure-investigator -->\n<!-- gh-aw-workflow-call-id: github/gh-aw/aw-failure-investigator -->\n\n---\n\n### Root cause now captured — it's a 401 invalid_project auth error, and it has spread to a new workflow\n\n**One-sentence rationale:** every Codex CLI call in the affected runs gets `401 Unauthorized ... auth error code: invalid_project` from the internal API proxy, and Daily Cache Strategy Analyzer is now hitting it too.\n\n**Fresh occurrences (2026-08-20):**\n- AI Moderator — [§32397326401](https://github.com/github/gh-aw/actions/runs/32397326401), [§32396393382](https://github.com/github/gh-aw/actions/runs/32396393382)\n- Daily Cache Strategy Analyzer — [§32403753754](https://github.com/github/gh-aw/actions/runs/32403753754) — **newly affected workflow, add to scope**\n\n**Concrete signature (previously missing from this issue):** Codex CLI's requ
agent
Agent execution exited with code 1\\n##[error]Process completed with exit code 1.\",\n \"impact\": \"Workflow did not complete successfully and may need intervention\"\n },\n {\n \"category\": \"performance\",\n \"severity\": \"high\",\n \"title\": \"Resource Heavy For Domain\",\n \"description\": \"This Code Fix run consumed a heavy execution profile for its task shape.\",\n \"impact\": \"Higher cost and latency than a comparable well-behaved run\"\n }\n ],\n \"recommendations\": [\n {\n \"priority\": \"high\",\n \"action\": \"Review error logs to identify root cause of failure\",\n \"reason\": \"Understanding failure causes helps prevent recurrence\",\n \"example\": \"Check the errors field for specific error messages, or inspect the log files in logs_path\"\n },\n {\n \"priority\": \"high\",\n \"action\": \"Compare this run to similar successful runs and trim unnecessary turns, tools, or write actions.\",\n \"reason\": \"This Code Fix run consumed a heavy execution profile for its task shape.\"\n }\n ],\n \"observability_insights\": [\n {\n \"category\": \"execution\",\n \"severity\": \"medium\",\n \"title\": \"Exploratory execution path\",\n \"summary\": \"The agent used 2 turns across 21 tool types, which indicates adaptive planning instead of a strictly linear path.\",\n \"evidence\": \"turns=2 tool_types=21\"\n },\n {\n \"category\": \"actuation\",\n \"severity\": \"info\",\n \"title\": \"Read-only posture observed\",\n \"summary\": \"The workflow stayed in an analysis posture and did not emit any GitHub write actions.\",\n \"evidence\": \"created_items=0 safe_items=0\"\n },\n {\n \"category\": \"network\",\n \"severity\": \"info\",\n \"title\": \"Network policy aligned\",\n \"summary\": \"The firewall observed 8 request(s) with 0 blocked, for a 0% block rate.\",\n \"evidence\": \"blocked=0 total=8\"\n },\n {\n \"category\": \"execution\",\n \"severity\": \"info\",\n \"title\": \"Log template patterns mined\",\n \"summary\": \"Analysis identified 3 distinct event templates across 6 pipeline stages from 23 events.\",\n \"evidence\": \"plan=1 tool_call=1 finish=1\"\n },\n {\n \"category\": \"reliability\",\n \"severity\": \"medium\",\n \"title\": \"2 anomalous event pattern(s) detected\",\n \"summary\": \"Anomaly detection flagged 2 event(s) as unusual based on template similarity and cluster rarity.\",\n \"evidence\": \"stage=plan score=0.65: new log template discovered; rare cluster (few observations); stage=tool_call score=0.65: new log template discovered; rare cluster (few observations)\"\n },\n {\n \"category\": \"execution\",\n \"severity\": \"info\",\n \"title\": \"Agent stage sequence\",\n \"summary\": \"The observed pipeline stage sequence for this run.\",\n \"evidence\": \"plan tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call finish\"\n }\n ],\n \"performance_metrics\": {\n \"most_used_tool\": \"create_discussion (3 calls)\",\n \"network_requests\": 8\n },\n \"prompt_analysis\": {\n \"prompt_size\": 16684,\n \"prompt_file\": \"aw-prompts/prompt.txt\"\n },\n \"session_analysis\": {\n \"wall_time\": \"5.1m\",\n \"turn_count\": 2,\n \"avg_turn_duration\": \"2.6m\",\n \"avg_time_between_turns\": \"5.1m (estimated)\",\n \"timeout_detected\": false,\n \"cache_warning\": \"Estimated avg TBT (5.1m) exceeds Anthropic 5-min cache TTL — prompt cache likely expiring between turns\"\n },\n \"jobs\": [\n {\n \"name\": \"activation\",\n \"status\": \"completed\",\n \"conclusion\": \"success\",\n \"duration\": \"35.0s\",\n \"steps\": [\n {\n \"name\": \"Set up job\",\n \"status\": \"completed\",\n
agent
Agent execution exited with code 1\n##[e...`
agent
Agent execution exited with code 1\n##[error]Process completed with
agent
`/exit code line anywhere in the 50-line tail.\n\nRun 32089118620 (same workflow, ~3.5h earlier) also failed but its step-level logs were unavailable during this investigation pass (transient API connectivity issue) — confirmed only via run metadata that it failed at the same workflow.\n</details>\n\n### Probable root cause\n\nTimeout/hang signature, not a clean failure: the CLI process appears to be killed while blocked on a large-diff `Read`, and nothing in the workflow surfaces that as a distinguishable error — the log just goes silent.\n\n### Proposed remediation\n\n1. Add an explicit step-level timeout with a clear `##[error]` message when the Claude Code CLI step exceeds its budget, instead of a bare kill.\n2. Cap or paginate the diff size handed to the agent — large `Read` calls on `pr.diff` are the apparent hang point in both failures.\n3. Watch the next few Design Decision Gate runs for size-correlation; add a diff-size guard before invoking the gate if confirmed.\n\n### Success criteria\n\nDesign Decision Gate runs complete (pass or fail) without silent mid-tool-use termination; timeouts show an explicit error in logs; similarly-sized PR diffs no longer trigger the hang.\n\n**Parent:** #53129\nRelated to #53129\n\n\n\n\n> Generated by [🔍 [aw] Failure Investigator (6h)](https://github.com/github/gh-aw/actions/runs/32109525808) · agent · 251.3 AIC · ⌖ 6.96 AIC · ⊞ 5.9K · [◷](https://github.com/search?q=repo:github%2Fgh-aw+is:issue+%22gh-aw-workflow-call-id:+github%2Fgh-aw%2Faw-failure-investigator%22&type=issues)\n> - [x] expires on Aug 24, 2026, 11:21 PM UTC-08:00\n\n\n\n\n\n\n---\n\n### Co","is_error":false}]},"parent_tool_use_id":null,"session_id":"bfd289ff-d550-4243-80ea-6f77cc824008","uuid":"0c898e77-cb78-4dbd-85e2-c222a52f9ccc","timestamp":"2026-08-23T19:06:47.851Z","tool_use_result":{"stdout":"TITLE: [aw-failures] [P1] Design Decision Gate hangs silently mid-tool-use in Claude Code CLI step\nSTATE: OPEN UPDATED: 2026-08-22T07:14:46Z\n5457\n### Fix: Design Decision Gate hangs silently mid-tool-use in the Claude Code CLI step\n\nFix the Design Decision Gate workflow's Claude Code CLI step — it hangs and dies mid-`Read` of the PR diff with zero error signal, hitting 2 runs in a single day.\n\n**Affected workflow:** `.github/workflows/design-decision-gate.lock.yml` (\"Design Decision Gate\")\n**Runs:** [§32101513747](https://github.com/github/gh-aw/actions/runs/32101513747) (2026-08-18 05:05 UTC), [§32089118620](https://github.com/github/gh-aw/actions/runs/32089118620) (2026-08-18 01:41 UTC)\n\n### Evidence\n\n<details>\n<summary>Captured log tail (run 32101513747)</summary>\n\nThe agent reads `/tmp/gh-aw/agent/adr-prefetch-summary.json` and `/tmp/gh-aw/agent/pr.json`, then issues a `Read` on `/tmp/gh-aw/agent/pr.diff` (24 files, 182 business-logic additions). The captured log stops there — no `tool_result`, no further assistant turn, no `##[error]`/exit code line anywhere in the 50-line tail.\n\nRun 32089118620 (same workflow, ~3.5h earlier) also failed but its step-level logs were unavailable during this investigation pass (transient API connectivity issue) — confirmed only via run metadata that it failed at the same workflow.\n</details>\n\n### Probable root cause\n\nTimeout/hang signature, not a clean failure: the CLI process appears to be killed while blocked on a large-diff `Read`, and nothing in the workflow surfaces that as a distinguishable error — the log just goes silent.\n\n### Proposed remediation\n\n1. Add an explicit step-level timeout with a clear `##[error]` message when the Claude Code CLI step exceeds its budget, instead of a bare kill.\n2. Cap or paginate the diff size handed to the agent — large `Read` calls on `pr.diff` are the apparent hang point in both failures.\n3. Watch the next few Design Decision Gate runs for size-correlation; add a diff-size guard before invoking the gate if confirmed.\n\n### Success criteria\n\nDesign Decision Gate runs complete (pass or fail) without silent mid-tool-use termination; timeouts show an explicit error in logs; simil
agent
Agent execution exited with code 1\n8\t##[error]Process completed with exit code 1.\n9\t```\n10\t`audit` confirms both runs are healthy at the infra layer — firewall shows 73/73 requests allowed, 0 blocked, identical domain mix (`api.openai.com`, `chatgpt.com`, `github.com`, Sentry, Grafana) both times. This rules out egress blackout (#53935) and rules out the cross-engine SIGSEGV pattern in #54186 — that issue's signature is exit code **139**; both of these runs exited with code **1**, and neither shows the \"log stops abruptly right after MCP registration\" shape #54186 describes.\n11\t\n12\t### Affected workflows and run IDs\n13\t\n14\t- `AI Moderator` (`.github/workflows/ai-moderator.lock.yml`), Codex CLI:\n15\t - [§32367352436](https://github.com/github/gh-aw/actions/runs/32367352436) — 2026-08-20 12:09 UTC, failed steps: `agent` and `evals` jobs, \"Execute Codex CLI\"\n16\t - [§32367076703](https://github.com/github/gh-aw/actions/runs/32367076703) — 2026-08-20 12:05 UTC, same signature, 4 minutes earlier\n17\t\n18\t### Probable root cause\n19\t\n20\tUnknown and currently unlogged — the Codex CLI process exits 1 with no captured stdout/stderr reason in either the workflow logs or the `audit` tool's error extraction. Given no network anomaly and no segfault signature, this points at either an unhandled exception inside the Codex CLI wrapper itself or a prompt/tool-call condition specific to `AI Moderator`'s task (issue-moderation triage) that the CLI doesn't surface cleanly.\n21\t\n22\t### Proposed remediation\n23\t\n24\t1. Capture Codex CLI's raw stdout/stderr (not just the exit-code wrapper) in the step log so the actual exception/error text is visible — right now the failure is a black box.\n25\t2. Re-run `AI Moderator` with verbose/debug logging enabled for the Codex engine to catch the next occurrence with full detail.\n26\t3. Once real error text is captured, re-file with the specific exception — do not let this recur as another \"generic exit 1\" entry.\n27\t\n28\t### Success criteria / verification\n29\t\n30\t- Next `AI Moderator` failure (if any) surfaces a specific error message/stack trace instead of a bare exit code.\n31\t- Zero repeat \"exit code 1, clean firewall, no error text\" occurrences for `AI Moderator` over the next 24h.\n32\t\n33\t**References:**\n34\t- https://github.com/github/gh-aw/actions/runs/32367352436\n35\t- https://github.com/github/gh-aw/actions/runs/32367076703\n36\t\n37\tRelated to #54114, distinct from #54186 (exit 139) and #53935 (egress blackout).\n38\tRelated to #54114\n39\t\n40\t\n41\t<!-- gh-aw-tracker-id: aw-failure-investigator -->\n42\t\n43\t> Generated by [🔍 [aw] Failure Investigator (6h)](https://github.com/github/gh-aw/actions/runs/32372296893) · agent · 137.8 AIC · ⌖ 8.41 AIC · ⊞ 5.9K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+is%3Aissue+%22gh-aw-workflow-call-id%3A+github%2Fgh-aw%2Faw-failure-investigator%22&type=issues)\n44\t> - [x] expires <!-- gh-aw-expires: 2026-08-27T13:20:47.586Z --> on Aug 27, 2026, 5:20 AM UTC-08:00\n45\t\n46\t<!-- gh-aw-agentic-workflow: [aw] Failure Investigator (6h), gh-aw-tracker-id: aw-failure-investigator, engine: claude, model: agent, id: 32372296893, workflow_id: aw-failure-investigator, run: https://github.com/github/gh-aw/actions/runs/32372296893 -->\n47\t\n48\t<!-- gh-aw-workflow-id: aw-failure-investigator -->\n49\t<!-- gh-aw-workflow-call-id: github/gh-aw/aw-failure-investigator -->\n50\t\n51\t---\n52\t\n53\t### Root cause now captured — it's a 401 invalid_project auth error, and it has spread to a new workflow\n54\t\n55\t**One-sentence rationale:** every Codex CLI call in the affected runs gets `401 Unauthorized ... auth error code: invalid_project` from the internal API proxy, and Daily Cache Strategy Analyzer is now hitting it too.\n56\t\n57\t**Fresh occurrences (2026-08-20):**\n58\t- AI Moderator — [§32397326401](https://github.com/github/gh-aw/actions/runs/32397326401), [§32396393382](https://github.com/github/gh-aw/actions/runs/32396393382)\n59\t- Daily Cache Strategy Analyze
agent
Agent execution exited with code 1\n##[error]Process completed with exit code 1.\n```\n`audit` confirms both runs are healthy at the infra layer — firewall shows 73/73 requests allowed, 0 blocked, identical domain mix (`api.openai.com`, `chatgpt.com`, `github.com`, Sentry, Grafana) both times. This rules out egress blackout (#53935) and rules out the cross-engine SIGSEGV pattern in #54186 — that issue's signature is exit code **139**; both of these runs exited with code **1**, and neither shows the \"log stops abruptly right after MCP registration\" shape #54186 describes.\n\n### Affected workflows and run IDs\n\n- `AI Moderator` (`.github/workflows/ai-moderator.lock.yml`), Codex CLI:\n - [§32367352436](https://github.com/github/gh-aw/actions/runs/32367352436) — 2026-08-20 12:09 UTC, failed steps: `agent` and `evals` jobs, \"Execute Codex CLI\"\n - [§32367076703](https://github.com/github/gh-aw/actions/runs/32367076703) — 2026-08-20 12:05 UTC, same signature, 4 minutes earlier\n\n### Probable root cause\n\nUnknown and currently unlogged — the Codex CLI process exits 1 with no captured stdout/stderr reason in either the workflow logs or the `audit` tool's error extraction. Given no network anomaly and no segfault signature, this points at either an unhandled exception inside the Codex CLI wrapper itself or a prompt/tool-call condition specific to `AI Moderator`'s task (issue-moderation triage) that the CLI doesn't surface cleanly.\n\n### Proposed remediation\n\n1. Capture Codex CLI's raw stdout/stderr (not just the exit-code wrapper) in the step log so the actual exception/error text is visible — right now the failure is a black box.\n2. Re-run `AI Moderator` with verbose/debug logging enabled for the Codex engine to catch the next occurrence with full detail.\n3. Once real error text is captured, re-file with the specific exception — do not let this recur as another \"generic exit 1\" entry.\n\n### Success criteria / verification\n\n- Next `AI Moderator` failure (if any) surfaces a specific error message/stack trace instead of a bare exit code.\n- Zero repeat \"exit code 1, clean firewall, no error text\" occurrences for `AI Moderator` over the next 24h.\n\n**References:**\n- https://github.com/github/gh-aw/actions/runs/32367352436\n- https://github.com/github/gh-aw/actions/runs/32367076703\n\nRelated to #54114, distinct from #54186 (exit 139) and #53935 (egress blackout).\nRelated to #54114\n\n\n<!-- gh-aw-tracker-id: aw-failure-investigator -->\n\n> Generated by [🔍 [aw] Failure Investigator (6h)](https://github.com/github/gh-aw/actions/runs/32372296893) · agent · 137.8 AIC · ⌖ 8.41 AIC · ⊞ 5.9K · [◷","is_error":false}]},"parent_tool_use_id":null,"session_id":"bfd289ff-d550-4243-80ea-6f77cc824008","uuid":"8e961ea3-3b9e-408f-9872-3bbefa5939dc","timestamp":"2026-08-23T19:06:03.064Z","tool_use_result":{"stdout":"TITLE: [aw-failures] [P1] AI Moderator: Codex CLI exits code 1 twice in 4 min with clean firewall — no error signature captured\n---BODY---\nFix `AI Moderator`'s Codex CLI generic exit-1 crash — it hit twice in 4 minutes with a clean firewall, so this is not egress-blackout or SIGSEGV, it's an unlogged Codex failure mode we can't yet name.\n\n### Problem statement\n\n`AI Moderator` (Codex engine, model `gpt-5.4`) failed twice back-to-back in this 6h window with no distinguishing error beyond the generic wrapper message:\n```\n##[error]Agent execution exited with code 1\n##[error]Process completed with exit code 1.\n```\n`audit` confirms both runs are healthy at the infra layer — firewall shows 73/73 requests allowed, 0 blocked, identical domain mix (`api.openai.com`, `chatgpt.com`, `github.com`, Sentry, Grafana) both times. This rules out egress blackout (#53935) and rules out the cross-engine SIGSEGV pattern in #54186 — that issue's signature is exit code **139**; both of these runs exited with code **1**, and neither shows the \"log stops abruptly right after MCP registration\" shape #54186 describes.\n\n### Affected workflows and run IDs\n\n- `AI Moderator` (`.github/workf
agent
Agent execution exited with code 1\\n##[error]Process completed with exit code 1.\",\n \"file\": \"agent\"\n }\n ],\n \"tool_usage\": [\n {\n \"name\": \"add_comment\",\n \"call_count\": 3\n },\n {\n \"name\": \"missing_data\",\n \"call_count\": 3\n },\n {\n \"name\": \"missing_tool\",\n \"call_count\": 3\n },\n {\n \"name\": \"noop\",\n \"call_count\": 3\n },\n {\n \"name\": \"push_to_pull_request_branch\",\n \"call_count\": 3\n },\n {\n \"name\": \"report_incomplete\",\n \"call_count\": 3","is_error":false}]},"parent_tool_use_id":null,"session_id":"bfd289ff-d550-4243-80ea-6f77cc824008","uuid":"5f30e8f7-d563-4721-aa9e-095083b12d1d","timestamp":"2026-08-23T19:00:43.341Z","tool_use_result":{"stdout":" \"errors\": [\n {\n \"type\": \"step_failure\",\n \"message\": \"##[error]Agent execution exited with code 1\\n##[error]Process completed with exit code 1.\",\n \"file\": \"agent\"\n }\n ],\n \"tool_usage\": [\n {\n \"name\": \"add_comment\",\n \"call_count\": 3\n },\n {\n \"name\": \"missing_data\",\n \"call_count\": 3\n },\n {\n \"name\": \"missing_tool\",\n \"call_count\": 3\n },\n {\n \"name\": \"noop\",\n \"call_count\": 3\n },\n {\n \"name\": \"push_to_pull_request_branch\",\n \"call_count\": 3\n },\n {\n \"name\": \"report_incomplete\",\n \"call_count\": 3","stderr":"","interrupted":false,"isImage":false,"noOutputExpected":false}}
agent
Agent execution exited with code 1\\n##[error]Process completed with exit code 1.\",\n \"file\": \"agent\"\n }\n ],\n \"tool_usage\": [\n {\n \"name\": \"create_discussion\",\n \"call_count\": 3\n },\n {\n \"name\": \"create_issue\",\n \"call_count\": 3\n },\n {\n \"name\": \"link_sub_issue\",\n \"call_count\": 3\n },\n {\n \"name\": \"missing_data\",\n \"call_count\": 3\n },\n {\n \"name\": \"missing_tool\",\n \"call_count\": 3\n },\n {\n \"name\": \"noop\",\n \"call_count\": 3\n },\n {\n \"name\": \"report_incomplete\",\n \"call_count\": 3\n },\n {\n \"name\": \"safeoutputs-create_discussion\",\n \"call_count\": 1\n },\n {\n \"name\": \"safeoutputs-create_issue\",\n \"call_count\": 1\n },\n {\n \"name\": \"safeoutputs-link_sub_issue\",\n \"call_count\": 1\n },\n {\n \"name\": \"safeoutputs-missing_data\",\n \"call_count\": 1\n },\n {\n \"name\": \"safeoutputs-missing_tool\",\n \"call_count\": 1\n },\n {\n \"name\": \"safeoutputs-noop\",\n \"call_count\": 1\n },\n {\n \"name\": \"safeoutputs-report_incomplete\",\n \"call_count\": 1\n },\n {\n \"name\": \"safeoutputs___create_discussion\",\n \"call_count\": 1\n },\n {\n \"name\": \"safeoutputs___create_issue\",\n \"call_count\": 1\n },\n {\n \"name\": \"safeoutputs___link_sub_issue\",\n \"call_count\": 1\n },\n {\n \"name\": \"safeoutputs___missing_data\",\n \"call_count\": 1\n },\n {\n \"name\": \"safeoutputs___missing_tool\",\n \"call_count\": 1\n },\n {\n \"name\": \"safeoutputs___noop\",\n \"call_count\": 1\n },\n {\n \"name\": \"safeoutputs___report_incomplete\",\n \"call_count\": 1\n }\n ]\n}\n\n[info] [agenticworkflows] Tool output: 40576 chars","is_error":false}]},"parent_tool_use_id":null,"session_id":"bfd289ff-d550-4243-80ea-6f77cc824008","uuid":"9323d4cb-8e6c-4a14-a82e-df21eed90618","timestamp":"2026-08-23T19:00:00.553Z","tool_use_result":{"stdout":"2648168009/sandbox/firewall/logs/audit.jsonl\",\n \"size\": 3917,\n \"description\": \"JSON Lines data file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32648168009/sandbox/firewall/logs/cache.log\",\n \"size\": 2762,\n \"description\": \"Log file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32648168009/sandbox/firewall/logs/cli-proxy-logs/access.jsonl\",\n \"size\": 213,\n \"description\": \"JSON Lines data file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32648168009/usage/activity/summary.json\",\n \"size\": 904,\n \"description\": \"JSON data file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32648168009/usage/agent/token_usage.jsonl\",\n \"size\": 0,\n \"description\": \"JSON Lines data file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32648168009/usage/detection/token_usage.jsonl\",\n \"size\": 0,\n \"description\": \"JSON Lines data file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32648168009/usage/github_rate_limits.jsonl\",\n \"size\": 1583,\n \"description\": \"JSON Lines data file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32648168009/workflow-logs/0_conclusion.txt\",\n \"size\": 76633,\n \"description\": \"Text file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32648168009/workflow-logs/3_safe_outputs.txt\",\n \"size\": 51513,\n \"description\": \"Text file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32648168009/workflow-logs/4_detection.txt\",\n \"size\": 137197,\n \"description\": \"Text file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32648168009/workflow-logs/5_agent.txt\",\n \"size\": 437571,\n \"description\": \"Text file\"\n },\n {\n \"path\": \"/tmp/gh-aw/a
evals
`/exit code line anywhere in the 50-line tail.\n\nRun 32089118620 (same workflow, ~3.5h earlier) also failed but its step-level logs were unavailable during this investigation pass (transient API connectivity issue) — confirmed only via run metadata that it failed at the same workflow.\n</details>\n\n### Probable root cause\n\nTimeout/hang signature, not a clean failure: the CLI process appears to be killed while blocked on a large-diff `Read`, and nothing in the workflow surfaces that as a distinguishable error — the log just goes silent.\n\n### Proposed remediation\n\n1. Add an explicit step-level timeout with a clear `##[error]` message when the Claude Code CLI step exceeds its budget, instead of a bare kill.\n2. Cap or paginate the diff size handed to the agent — large `Read` calls on `pr.diff` are the apparent hang point in both failures.\n3. Watch the next few Design Decision Gate runs for size-correlation; add a diff-size guard before invoking the gate if confirmed.\n\n### Success criteria\n\nDesign Decision Gate runs complete (pass or fail) without silent mid-tool-use termination; timeouts show an explicit error in logs; similarly-sized PR diffs no longer trigger the hang.\n\n**Parent:** #53129\nRelated to #53129\n\n\n\n\n> Generated by [🔍 [aw] Failure Investigator (6h)](https://github.com/github/gh-aw/actions/runs/32109525808) · agent · 251.3 AIC · ⌖ 6.96 AIC · ⊞ 5.9K · [◷](https://github.com/search?q=repo:github%2Fgh-aw+is:issue+%22gh-aw-workflow-call-id:+github%2Fgh-aw%2Faw-failure-investigator%22&type=issues)\n> - [x] expires on Aug 24, 2026, 11:21 PM UTC-08:00\n\n\n\n\n\n\n---\n\n### Correction — this is not a silent hang, it's a 429 invocation-cap failure. Keep this open.\n\n**One-sentence rationale:** Design Decision Gate's run today fails fast and loud with `Maximum LLM invocations exceeded (30 / 30)`, a non-retryable rate-limit guard — not the silent hang this issue currently describes.\n\n**Fresh occurrence (2026-08-20):** Design Decision Gate — [§32394421229](https://github.com/github/gh-aw/actions/runs/32394421229). Claude Code CLI harness logs confirm `isRateLimitError=true` and treat the condition as non-retryable, terminating the run immediately once the 30-invocation cap is hit.\n\n**Why this stays open:** no commit since this issue opened has touched this workflow's `.md`/`.lock.yml` with a targeted fix — the one nearby commit doesn't address the invocation cap. Not closing per the tool-denial-limit rule.\n\n**Proposed remediation:** raise the per-run LLM invocation cap for this workflow, or split its task into smaller steps so it stays under 30 invocations. Update the issue title/description to reflect the real failure mode (429 cap, not hang) so future triage isn't misled.> Generated by [🔍 [aw] Failure Investigator (6h)](https://github.com/github/gh-aw/actions/runs/32406250112) · agent · 238.8 AIC · ⌖ 7.28 AIC · ⊞ 5.9K · [◷](https://github.com/search?q=repo:github%2Fgh-aw+is:issue+%22gh-aw-workflow-call-id:+github%2Fgh-aw%2Faw-failure-investigator%22&type=issues)\n\n\n\n---\n\n### Update (2026-08-22) — reproduced again today, keep this open\n\nDesign Decision Gate failed at the \"Execute Claude Code CLI\" step again: [§32548469031](https://github.com/github/gh-aw/actions/runs/32548469031) (2026-08-22 03:15 UTC). Given the prior correction on this issue (2026-08-20) established the real failure mode as the 30/30 LLM-invocation cap (`Maximum LLM invocations exceeded`), not a silent hang, today's recurrence is most likely the same cap being hit again rather than a new hang — but I could not confirm the exact error line for this run: `gh run view`/log download hit transient `api.github.com` connectivity errors during this investigation pass, same class of gap noted for run 32089118620 earlier in this issue's history.\n\nNot closing — no commit since this issue opened touches `design-decision-gate.md`/`.lock.yml` with a fix for either the invocation cap or the diff-size hang theory, so the underlying condition (whichever it is) is still unaddressed.
evals
Agent execution exited with code 1\n##[error]Process completed with exit code 1.\n```\n`audit` confirms both runs are healthy at the infra layer — firewall shows 73/73 requests allowed, 0 blocked, identical domain mix (`api.openai.com`, `chatgpt.com`, `github.com`, Sentry, Grafana) both times. This rules out egress blackout (#53935) and rules out the cross-engine SIGSEGV pattern in #54186 — that issue's signature is exit code **139**; both of these runs exited with code **1**, and neither shows the \"log stops abruptly right after MCP registration\" shape #54186 describes.\n\n### Affected workflows and run IDs\n\n- `AI Moderator` (`.github/workflows/ai-moderator.lock.yml`), Codex CLI:\n - [§32367352436](https://github.com/github/gh-aw/actions/runs/32367352436) — 2026-08-20 12:09 UTC, failed steps: `agent` and `evals` jobs, \"Execute Codex CLI\"\n - [§32367076703](https://github.com/github/gh-aw/actions/runs/32367076703) — 2026-08-20 12:05 UTC, same signature, 4 minutes earlier\n\n### Probable root cause\n\nUnknown and currently unlogged — the Codex CLI process exits 1 with no captured stdout/stderr reason in either the workflow logs or the `audit` tool's error extraction. Given no network anomaly and no segfault signature, this points at either an unhandled exception inside the Codex CLI wrapper itself or a prompt/tool-call condition specific to `AI Moderator`'s task (issue-moderation triage) that the CLI doesn't surface cleanly.\n\n### Proposed remediation\n\n1. Capture Codex CLI's raw stdout/stderr (not just the exit-code wrapper) in the step log so the actual exception/error text is visible — right now the failure is a black box.\n2. Re-run `AI Moderator` with verbose/debug logging enabled for the Codex engine to catch the next occurrence with full detail.\n3. Once real error text is captured, re-file with the specific exception — do not let this recur as another \"generic exit 1\" entry.\n\n### Success criteria / verification\n\n- Next `AI Moderator` failure (if any) surfaces a specific error message/stack trace instead of a bare exit code.\n- Zero repeat \"exit code 1, clean firewall, no error text\" occurrences for `AI Moderator` over the next 24h.\n\n**References:**\n- https://github.com/github/gh-aw/actions/runs/32367352436\n- https://github.com/github/gh-aw/actions/runs/32367076703\n\nRelated to #54114, distinct from #54186 (exit 139) and #53935 (egress blackout).\nRelated to #54114\n\n\n<!-- gh-aw-tracker-id: aw-failure-investigator -->\n\n> Generated by [🔍 [aw] Failure Investigator (6h)](https://github.com/github/gh-aw/actions/runs/32372296893) · agent · 137.8 AIC · ⌖ 8.41 AIC · ⊞ 5.9K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+is%3Aissue+%22gh-aw-workflow-call-id%3A+github%2Fgh-aw%2Faw-failure-investigator%22&type=issues)\n> - [x] expires <!-- gh-aw-expires: 2026-08-27T13:20:47.586Z --> on Aug 27, 2026, 5:20 AM UTC-08:00\n\n<!-- gh-aw-agentic-workflow: [aw] Failure Investigator (6h), gh-aw-tracker-id: aw-failure-investigator, engine: claude, model: agent, id: 32372296893, workflow_id: aw-failure-investigator, run: https://github.com/github/gh-aw/actions/runs/32372296893 -->\n\n<!-- gh-aw-workflow-id: aw-failure-investigator -->\n<!-- gh-aw-workflow-call-id: github/gh-aw/aw-failure-investigator -->\n\n---\n\n### Root cause now captured — it's a 401 invalid_project auth error, and it has spread to a new workflow\n\n**One-sentence rationale:** every Codex CLI call in the affected runs gets `401 Unauthorized ... auth error code: invalid_project` from the internal API proxy, and Daily Cache Strategy Analyzer is now hitting it too.\n\n**Fresh occurrences (2026-08-20):**\n- AI Moderator — [§32397326401](https://github.com/github/gh-aw/actions/runs/32397326401), [§32396393382](https://github.com/github/gh-aw/actions/runs/32396393382)\n- Daily Cache Strategy Analyzer — [§32403753754](https://github.com/github/gh-aw/actions/runs/32403753754) — **newly affected workflow, add to scope**\n\n**Concrete signature (previously missing from this issue):** Codex CLI's requ
evals
Agent execution exited with code 1\\n##[error]Process completed with exit code 1.\",\n \"impact\": \"Workflow did not complete successfully and may need intervention\"\n },\n {\n \"category\": \"performance\",\n \"severity\": \"high\",\n \"title\": \"Resource Heavy For Domain\",\n \"description\": \"This Code Fix run consumed a heavy execution profile for its task shape.\",\n \"impact\": \"Higher cost and latency than a comparable well-behaved run\"\n }\n ],\n \"recommendations\": [\n {\n \"priority\": \"high\",\n \"action\": \"Review error logs to identify root cause of failure\",\n \"reason\": \"Understanding failure causes helps prevent recurrence\",\n \"example\": \"Check the errors field for specific error messages, or inspect the log files in logs_path\"\n },\n {\n \"priority\": \"high\",\n \"action\": \"Compare this run to similar successful runs and trim unnecessary turns, tools, or write actions.\",\n \"reason\": \"This Code Fix run consumed a heavy execution profile for its task shape.\"\n }\n ],\n \"observability_insights\": [\n {\n \"category\": \"execution\",\n \"severity\": \"medium\",\n \"title\": \"Exploratory execution path\",\n \"summary\": \"The agent used 2 turns across 21 tool types, which indicates adaptive planning instead of a strictly linear path.\",\n \"evidence\": \"turns=2 tool_types=21\"\n },\n {\n \"category\": \"actuation\",\n \"severity\": \"info\",\n \"title\": \"Read-only posture observed\",\n \"summary\": \"The workflow stayed in an analysis posture and did not emit any GitHub write actions.\",\n \"evidence\": \"created_items=0 safe_items=0\"\n },\n {\n \"category\": \"network\",\n \"severity\": \"info\",\n \"title\": \"Network policy aligned\",\n \"summary\": \"The firewall observed 8 request(s) with 0 blocked, for a 0% block rate.\",\n \"evidence\": \"blocked=0 total=8\"\n },\n {\n \"category\": \"execution\",\n \"severity\": \"info\",\n \"title\": \"Log template patterns mined\",\n \"summary\": \"Analysis identified 3 distinct event templates across 6 pipeline stages from 23 events.\",\n \"evidence\": \"plan=1 tool_call=1 finish=1\"\n },\n {\n \"category\": \"reliability\",\n \"severity\": \"medium\",\n \"title\": \"2 anomalous event pattern(s) detected\",\n \"summary\": \"Anomaly detection flagged 2 event(s) as unusual based on template similarity and cluster rarity.\",\n \"evidence\": \"stage=plan score=0.65: new log template discovered; rare cluster (few observations); stage=tool_call score=0.65: new log template discovered; rare cluster (few observations)\"\n },\n {\n \"category\": \"execution\",\n \"severity\": \"info\",\n \"title\": \"Agent stage sequence\",\n \"summary\": \"The observed pipeline stage sequence for this run.\",\n \"evidence\": \"plan tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call tool_call finish\"\n }\n ],\n \"performance_metrics\": {\n \"most_used_tool\": \"create_discussion (3 calls)\",\n \"network_requests\": 8\n },\n \"prompt_analysis\": {\n \"prompt_size\": 16684,\n \"prompt_file\": \"aw-prompts/prompt.txt\"\n },\n \"session_analysis\": {\n \"wall_time\": \"5.1m\",\n \"turn_count\": 2,\n \"avg_turn_duration\": \"2.6m\",\n \"avg_time_between_turns\": \"5.1m (estimated)\",\n \"timeout_detected\": false,\n \"cache_warning\": \"Estimated avg TBT (5.1m) exceeds Anthropic 5-min cache TTL — prompt cache likely expiring between turns\"\n },\n \"jobs\": [\n {\n \"name\": \"activation\",\n \"status\": \"completed\",\n \"conclusion\": \"success\",\n \"duration\": \"35.0s\",\n \"steps\": [\n {\n \"name\": \"Set up job\",\n \"status\": \"completed\",\n
evals
Agent execution exited with code 1\n##[e...`
agent
- Line 1: update_issue requires at least one of: 'status', 'title', 'body', 'labels', 'assignees', 'milestone' fields
agent
Validation errors found:
agent
Failed to scan directory /tmp/gh-aw/aw-mcp/logs/run-32658283193/usage: EACCES: permission denied, scandir '/tmp/gh-aw/aw-mcp/logs/run-32658283193/usage'
agent
Failed to scan directory /tmp/gh-aw/aw-mcp/logs/run-32654551225/usage: EACCES: permission denied, scandir '/tmp/gh-aw/aw-mcp/logs/run-32654551225/usage'
agent
Failed to scan directory /tmp/gh-aw/aw-mcp/logs/run-32648168009/usage: EACCES: permission denied, scandir '/tmp/gh-aw/aw-mcp/logs/run-32648168009/usage'
agent
Failed to scan directory /tmp/gh-aw/aw-mcp/logs/run-32171537745/usage: EACCES: permission denied, scandir '/tmp/gh-aw/aw-mcp/logs/run-32171537745/usage'

Artifacts

Produced during runtime
Name Size Digest
activation
9.43 MB
sha256:08d8f298879b4de78458ed48472429e602f2725a1284c97c1c1ae3fc854c4d54
agent
626 KB
sha256:9750f8a7bbb2cc3b4e7bc9c4aee5edac931a94eadb1d62782566541b7472780d
agent-output-fallback
15.7 KB
sha256:7cead3e58052292e7ae440ddb277629c58e257709baf7341e045f56ec04744a2
aic-usage-cache
425 Bytes
sha256:a140fb6944cd2f5f71141094c7558d85504907fc55893a4b8b4e2ace1f29f458
awfailureinvestigator-experiment
5.85 KB
sha256:247ed90e586776d58c256942dcda3d779a87256a3ec8bb0072b64f9b3c556f02
detection
19.1 KB
sha256:e5453cd2883f6bd93568a6f99bd4ecb299fc5703e08fb4f2c1616370c42a51bb
evals
412 Bytes
sha256:7ca0f642f0776725b542ddf0932906e5fdd38f300f6fbbb992c6ffab0133ce8c
github~gh-aw~3DCDON.dockerbuild
24.2 KB
sha256:555077b5dadaa9db8e680906ce4e510ad9135eea1c4523a817de4117ccc927aa
safe-outputs-items
1.09 KB
sha256:3eeb38fa5f469b05a0e4f303f1530252807d0f1f66993c80860b0094ba453bf1
usage
6.55 KB
sha256:7ae2864bb7f4ad0aa5ed262449e694b96e7d9f7f320f3fbc8f5f407918839479