[aw] Failure Investigator (6h) #513
aw-failure-investigator.lock.yml
on: schedule
Annotations
2 errors and 2 warnings
|
agent
Agent execution exited with code 139\n10\t##[error]Process completed with exit code 139.\n11\t```\n12\tExit 139 = SIGSEGV. Firewall analysis for this run shows **18/18 requests allowed, 0 blocked** to `api.githubcopilot.com`, Sentry, and Grafana — network/proxy is provably fine, ruling out egress-blackout causes like #53935. The crash is a genuine process segfault after MCP tool registration completes, before the agent produces further output.\n13\t\n14\tThree more runs in the same ~50-minute window show the identical *symptom* (step log ends abruptly right after MCP server/tool registration, zero further lines, job fails) but audit on the second could not complete in time (`context deadline exceeded`) and the remaining two were not separately audited to stay within this cycle's audit-call budget — **these are unconfirmed, log-pattern matches only**, not proven SIGSEGV:\n15\t- Daily VulnHunter Scan / Claude Code CLI — [§32337062080](https://github.com/github/gh-aw/actions/runs/32337062080)\n16\t- AI Moderator / Codex CLI — [§32336926049](https://github.com/github/gh-aw/actions/runs/32336926049)\n17\t- Ponytail Reviewer / Copilot CLI — [§32336023838](https://github.com/github/gh-aw/actions/runs/32336023838)\n18\t\n19\tCorroborating precedent (separate, auto-expiring issue, not a durable tracker): #54072 shows this exact repo's own \"[aw] Failure Investigator (6h)\" workflow crashed on 2026-08-19 with `panic (main thread): Segmentation fault at address 0x38` / `Bun has crashed`, on the `claude` engine. Same exit-signature class (native-runtime segfault), different run and engine — supports a shared-runtime or shared-wrapper root cause rather than a single CLI's bug.\n20\t\n21\t### Affected workflows and run IDs\n22\t\n23\tConfirmed:\n24\t- Auto-Triage Issues (Pi CLI) — [§32339225947](https://github.com/github/gh-aw/actions/runs/32339225947), exit 139, recurring (prior instance [§32317125168](https://github.com/github/gh-aw/actions/runs/32317125168) noted 2026-08-19 as unconfirmed \"generic exit 1\")\n25\t\n26\tSuspected, same log-pattern, unconfirmed:\n27\t- Daily VulnHunter Scan (Claude Code CLI) — [§32337062080](https://github.com/github/gh-aw/actions/runs/32337062080)\n28\t- AI Moderator (Codex CLI) — [§32336926049](https://github.com/github/gh-aw/actions/runs/32336926049)\n29\t- Ponytail Reviewer (Copilot CLI) — [§32336023838](https://github.com/github/gh-aw/actions/runs/32336023838)\n30\t\n31\t### Probable root cause\n32\t\n33\tA native-runtime segfault (Bun-based process, per #54072's crash report) in a component shared across agent CLI engines — possibly the MCP client bridge/wrapper rather than each CLI binary independently — triggered intermittently after MCP tool/server registration completes. Not network/proxy related (firewall clean on the confirmed instance).\n34\t\n35\t### Proposed remediation\n36\t\n37\t- Capture and file the `bun.report` crash link from #54072 with the Bun team, or pin the runner image to a Bun version known not to exhibit this segfault if a recent bump correlates.\n38\t- Confirm whether Pi CLI, Claude Code CLI, Codex CLI, and Copilot CLI share a common Bun-based bridge/wrapper process — if so, that shared component is the actual fix target, not four separate CLI bugs.\n39\t- Update the failure-classification logic so `exit code 139` / `Segmentation fault` is labeled \"engine crash (SIGSEGV)\" instead of generic \"exit code 1, no signature\" — this is why the pattern went untracked in the prior cycle.\n40\t\n41\t### Success criteria / verification\n42\t\n43\t- Zero exit-139/segfault-signature failures across these four engines over the next 24h of runs.\n44\t- If the underlying wrapper is confirmed and fixed, all four workflows above complete without an unexplained abrupt log stop after MCP registration.\n45\t\n46\t### Existing issue correlation\n47\t\n48\t- Not the same as #53935 (Cloud Hypervisor guest-network blackout — that signature is zero firewall requests pre-flight; ours has clean firewall traffic mid-run then a crash).\n49\t- Not the same as #
|
|
agent
Agent execution exited with code 139\n##[error]Process completed with exit code 139.\n```\nExit 139 = SIGSEGV. Firewall analysis for this run shows **18/18 requests allowed, 0 blocked** to `api.githubcopilot.com`, Sentry, and Grafana — network/proxy is provably fine, ruling out egress-blackout causes like #53935. The crash is a genuine process segfault after MCP tool registration completes, before the agent produces further output.\n\nThree more runs in the same ~50-minute window show the identical *symptom* (step log ends abruptly right after MCP server/tool registration, zero further lines, job fails) but audit on the second could not complete in time (`context deadline exceeded`) and the remaining two were not separately audited to stay within this cycle's audit-call budget — **these are unconfirmed, log-pattern matches only**, not proven SIGSEGV:\n- Daily VulnHunter Scan / Claude Code CLI — [§32337062080](https://github.com/github/gh-aw/actions/runs/32337062080)\n- AI Moderator / Codex CLI — [§32336926049](https://github.com/github/gh-aw/actions/runs/32336926049)\n- Ponytail Reviewer / Copilot CLI — [§32336023838](https://github.com/github/gh-aw/actions/runs/32336023838)\n\nCorroborating precedent (separate, auto-expiring issue, not a durable tracker): #54072 shows this exact repo's own \"[aw] Failure Investigator (6h)\" workflow crashed on 2026-08-19 with `panic (main thread): Segmentation fault at address 0x38` / `Bun has crashed`, on the `claude` engine. Same exit-signature class (native-runtime segfault), different run and engine — supports a shared-runtime or shared-wrapper root cause rather than a single CLI's bug.\n\n### Affected workflows and run IDs\n\nConfirmed:\n- Auto-Triage Issues (Pi CLI) — [§32339225947](https://github.com/github/gh-aw/actions/runs/32339225947), exit 139, recurring (prior instance [§32317125168](https://github.com/github/gh-aw/actions/runs/32317125168) noted 2026-08-19 as unconfirmed \"generic exit 1\")\n\nSuspected, same log-pattern, unconfirmed:\n- Daily VulnHunter Scan (Claude Code CLI) — [§32337062080](https://github.com/github/gh-aw/actions/runs/32337062080)\n- AI Moderator (Codex CLI) — [§32336926049](https://github.com/github/gh-aw/actions/runs/32336926049)\n- Ponytail Reviewer (Copilot CLI) — [§32336023838](https://github.com/github/gh-aw/actions/runs/32336023838)\n\n### Probable root cause\n\nA native-runtime segfault (Bun-based process, per #54072'\n=== 53263 ===\n### Problem\n\nStop letting one bad safe-output item sink an entire batch — two workflows this window had a `safe_outputs` job **fail outright** even though most of the batch's outputs had already succeeded.\n\n### Affected workflows and runs\n\n- `PR Sous Chef`: [§31980573442](https://github.com/github/gh-aw/actions/runs/31980573442) (2026-08-16 23:57 UTC) — `update_pull_request` on a stacked PR failed with `Updating a stacked PR's branch via this endpoint is not supported` (GitHub API, non-retryable). Other outputs in the same batch (issue create, comment post) succeeded before the job was marked failed.\n- `Smoke Copilot - AOAI (apikey)`: [§31974722434](https://github.com/github/gh-aw/actions/runs/31974722434) (2026-08-16 21:51 UTC) — `dispatch_workflow` targeting `haiku-printer` failed with `Required input 'message' not provided`. 11/13 other outputs in the same batch succeeded before the job was marked failed.\n\n### Probable root cause\n\nBoth are deterministic, non-retryable per-item errors (a GitHub API limitation on stacked/dependent PR branch updates, and a missing required workflow-dispatch input) that the safe_outputs processor correctly flags as non-retryable, but still propagates as an overall job failure rather than a partial-success outcome. This is the same shape as the `resolve_pull_request_review_thread` hard-fail already flagged for PR Sous Chef in the prior 6h window's report (#53129) — one bad item failing the whole job is a recurring pattern across output types, not isolated to one tool.\n\n### Proposed remediation\n\n1. For `update_pull_request`: detect s
|
|
agent
- Line 1: update_issue requires at least one of: 'status', 'title', 'body', 'labels', 'assignees', 'milestone' fields
|
|
agent
Validation errors found:
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
activation
Expired
|
9.29 MB |
sha256:cc0d3be4bdb012c9d401207a004afd9183fbe460ff691a54dc82059359c68db6
|
|
|
agent
|
372 KB |
sha256:bdf5b7973df79eeb07ed7ca1ad391f73422b3350073146f38a890a8db3e252c6
|
|
|
agent-output-fallback
|
3.11 KB |
sha256:ecec33682f5f30e80c3608b2d8db87acaef6172298218cf98a23b59b26540498
|
|
|
aic-usage-cache
|
318 Bytes |
sha256:f4f1c53bebff9bffda7426d5423e3c8620e6a4f87d5b59d51dbb064623853752
|
|
|
awfailureinvestigator-experiment
|
5.75 KB |
sha256:75d89952132327f5f98962e18b45f808aec5b0c959d6243683dcc5047dfbc8e9
|
|
|
detection
|
20 KB |
sha256:fcb3fa209faaba30780f56ad758ef4305bcc11f5a3f74830f9c4fe5f764bbb10
|
|
|
evals
|
388 Bytes |
sha256:a641a451348e4bce5880b4bf90ba3841a85ccd2f67d5a15c1e61aabbf550fa62
|
|
|
github~gh-aw~0AXEHL.dockerbuild
|
24.4 KB |
sha256:196d98079a532ec99b531beb4443544778fb96c31d0f993acfe9344f9b46ceaf
|
|
|
safe-outputs-items
|
904 Bytes |
sha256:04ffefec4dd85d3e0e755a6b806d1e4a759f3a247c898dff151c37232dc440a6
|
|
|
usage
|
5.74 KB |
sha256:91f7906e9bf1c0f98ab6aa90b6d5399b4b50364d073b0022c624e63e95495dfd
|
|