[aw] Failure Investigator (6h) #501
pelikhanaw-failure-investigator.lock.yml
on: schedule
Annotations
11 errors and 3 warnings
|
agent
` line, even though `capture_likely_missed_fault` is reported as `false`.\n\n### Affected workflows and runs\n\n| Workflow | Run | Time (UTC) | Failed step |\n|---|---|---|---|\n| Ponytail Reviewer | [§32249649808](https://github.com/github/gh-aw/actions/runs/32249649808) | 2026-08-19 11:51 | `Execute GitHub Copilot CLI` |\n| Ponytail Reviewer | [§32240222020](https://github.com/github/gh-aw/actions/runs/32240222020) | 2026-08-19 09:57 | `Execute GitHub Copilot CLI` |\n| Ponytail Reviewer | [§32239461549](https://github.com/github/gh-aw/actions/runs/32239461549) | 2026-08-19 09:48 | `Execute GitHub Copilot CLI` |\n| Daily Max Ai Credits Test | [§32246708586](https://github.com/github/gh-aw/actions/runs/32246708586) | 2026-08-19 11:15 | `Execute GitHub Copilot CLI` |\n\nDaily Max Ai Credits Test has failed at this exact step on **every scheduled run for the last 10 consecutive days** (2026-08-10 through 2026-08-19) — this is chronic, not a one-off blip. Ponytail Reviewer fails intermittently (3 of ~15 runs in the same window) at the same step.\n\n### Evidence\n\n`audit-diff` of failed run [§32249649808](https://github.com/github/gh-aw/actions/runs/32249649808) against the next successful Ponytail Reviewer run [§32250065978](https://github.com/github/gh-aw/actions/runs/32250065978) (5 minutes later):\n\n- Failed run made **0 allowed and 0 blocked** requests to `api.githubcopilot.com:443`. The successful run made 22 allowed requests to the same domain.\n- Failed run's token usage is 0.\n- No firewall block, no proxy-refusal signature — the process never attempted the network call at all.\n\nThis rules out proxy/firewall causes (unlike #52459's Anthropic proxy issue, which is a different engine and shows connection-refused *after* the attempt). The Copilot CLI is dying before it opens a connection — most consistent with a startup/config/auth failure inside the CLI binary itself.\n\n### Probable root cause\n\nTwo compounding bugs:\n1. **Tail-capture regression** (re-opening the #52501 defect class): the 50-line tail capture still isn't anchored to the failing step, so it grabs the MCP safeoutputs backend's `tools/list` handshake dump instead of the Copilot CLI's actual exit/error output.\n2. **Underlying Copilot CLI startup failure** (currently invisible due to #1): the CLI exits before any network call, on both a low-frequency workflow (Ponytail Reviewer) and a workflow that has failed at this exact point for 10+ straight days (Daily Max Ai Credits Test).\n\n### Proposed remediation\n\n1. Re-apply/verify the #52501 fix (step-anchored log capture, or backward-search for `##[error]`/non-zero exit before falling back to absolute tail) — it has clearly regressed since 2026-08-14.\n2. Once real error text is visible, pull the full raw log (not just the tail) for one fresh Ponytail Reviewer failure and one fresh Daily Max Ai Credits Test failure to pin the actual Copilot CLI startup error.\n3. Add a regression guard: flag prefetch entries whose `tail_lines` contain no `error`/`exception`/non-zero-exit marker as `capture_likely_missed_fault: true` even when the heuristic currently says `false` — the false-negative here is exactly what let this regress silently.\n\n### Success criteria / verification\n\n- A fresh Ponytail Reviewer or Daily Max Ai Credits Test failure's `truncated_error_logs.tail_lines` contains the actual Copilot CLI error, not backend startup logs.\n- Root cause of the pre-network Copilot CLI exit is identified and Daily Max Ai Credits Test passes 3 consecutive scheduled runs (or is confirmed as an intentionally-failing test workflow, in which case it should be excluded from failure tracking).\n\nParent report: #53933\nRelated: #52501 (closed, regressed)\n","labels":["bug","automation","agentic-workflows"],"parent":53933,"title":"[aw-failures] [P0] Log-capture blindness regressed — GH Copilot CLI failures (Ponytail Reviewer, Daily Max Ai Credits Test) invisible again (#52501 relapse)"}}}
|
|
agent
` line, even though `capture_likely_missed_fault` is reported as `false`.\n\n### Affected workflows and runs\n\n| Workflow | Run | Time (UTC) | Failed step |\n|---|---|---|---|\n| Ponytail Reviewer | [§32249649808](https://github.com/github/gh-aw/actions/runs/32249649808) | 2026-08-19 11:51 | `Execute GitHub Copilot CLI` |\n| Ponytail Reviewer | [§32240222020](https://github.com/github/gh-aw/actions/runs/32240222020) | 2026-08-19 09:57 | `Execute GitHub Copilot CLI` |\n| Ponytail Reviewer | [§32239461549](https://github.com/github/gh-aw/actions/runs/32239461549) | 2026-08-19 09:48 | `Execute GitHub Copilot CLI` |\n| Daily Max Ai Credits Test | [§32246708586](https://github.com/github/gh-aw/actions/runs/32246708586) | 2026-08-19 11:15 | `Execute GitHub Copilot CLI` |\n\nDaily Max Ai Credits Test has failed at this exact step on **every scheduled run for the last 10 consecutive days** (2026-08-10 through 2026-08-19) — this is chronic, not a one-off blip. Ponytail Reviewer fails intermittently (3 of ~15 runs in the same window) at the same step.\n\n### Evidence\n\n`audit-diff` of failed run [§32249649808](https://github.com/github/gh-aw/actions/runs/32249649808) against the next successful Ponytail Reviewer run [§32250065978](https://github.com/github/gh-aw/actions/runs/32250065978) (5 minutes later):\n\n- Failed run made **0 allowed and 0 blocked** requests to `api.githubcopilot.com:443`. The successful run made 22 allowed requests to the same domain.\n- Failed run's token usage is 0.\n- No firewall block, no proxy-refusal signature — the process never attempted the network call at all.\n\nThis rules out proxy/firewall causes (unlike #52459's Anthropic proxy issue, which is a different engine and shows connection-refused *after* the attempt). The Copilot CLI is dying before it opens a connection — most consistent with a startup/config/auth failure inside the CLI binary itself.\n\n### Probable root cause\n\nTwo compounding bugs:\n1. **Tail-capture regression** (re-opening the #52501 defect class): the 50-line tail capture still isn't anchored to the failing step, so it grabs the MCP safeoutputs backend's `tools/list` handshake dump instead of the Copilot CLI's actual exit/error output.\n2. **Underlying Copilot CLI startup failure** (currently invisible due to #1): the CLI exits before any network call, on both a low-frequency workflow (Ponytail Reviewer) and a workflow that has failed at this exact point for 10+ straight days (Daily Max Ai Credits Test).\n\n### Proposed remediation\n\n1. Re-apply/verify the #52501 fix (step-anchored log capture, or backward-search for `##[error]`/non-zero exit before falling back to absolute tail) — it has clearly regressed since 2026-08-14.\n2. Once real error text is visible, pull the full raw log (not just the tail) for one fresh Ponytail Reviewer failure and one fresh Daily Max Ai Credits Test failure to pin the actual Copilot CLI startup error.\n3. Add a regression guard: flag prefetch entries whose `tail_lines` contain no `error`/`exception`/non-zero-exit marker as `capture_likely_missed_fault: true` even when the heuristic currently says `false` — the false-negative here is exactly what let this regress silently.\n\n### Success criteria / verification\n\n- A fresh Ponytail Reviewer or Daily Max Ai Credits Test failure's `truncated_error_logs.tail_lines` contains the actual Copilot CLI error, not backend startup logs.\n- Root cause of the pre-network Copilot CLI exit is identified and Daily Max Ai Credits Test passes 3 consecutive scheduled runs (or is confirmed as an intentionally-failing test workflow, in which case it should be excluded from failure tracking).\n\nParent report: #53933\nRelated: #52501 (closed, regressed)\n","labels":["bug","automation","agentic-workflows"],"parent":53933,"title":"[aw-failures] [P0] Log-capture blindness regressed — GH Copilot CLI failures (Ponytail Reviewer, Daily Max Ai Credits Test) invisible again (#52501 relapse)"},"name":"create_issue"}}}
|
|
agent
ERR_API: Failed to execute search query: Connect Timeout Error\n```\nThe github-script step is configured with `retries: 0`. Any single transient 500/timeout from the Search API is guaranteed to hard-fail the entire workflow with no retry — this isn't a one-off, it's a design gap that will keep recurring under normal GitHub API flakiness. No existing issue covers `check_skip_if_match`/`pre_activation`.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eP2 — two more \"Execute Copilot CLI\" failures with unrelated root causes (do not merge with #52253)\u003c/summary\u003e\n\n- Daily GitHub Docs SEO Optimizer [§32217479194](https://github.com/github/gh-aw/actions/runs/32217479194): the outbound api-proxy on port 10002 accepted connections fine; the failure is the **local** `copilot-sdk` headless server never reporting ready — `unexpected error: copilot-sdk headless server did not become ready on 127.0.0.1:3002 within 60000ms (connect ETIMEDOUT 127.0.0.1:3002)`. This is internal-to-the-guest, unrelated to #52253's external ECONNREFUSED-on-10002 signature.\n- Metrics Collector [§32209589795](https://github.com/github/gh-aw/actions/runs/32209589795): `failureClass=model_not_supported` in the copilot-harness attempt log — a model-selection bug, not a connectivity issue.\n\nBoth are single occurrences this window — watching, not filing separately yet.\n\n\u003c/details\u003e\n\n### Existing issue correlation\n\n- #53263 (safe_outputs hard-fail) — **confirmed still broken**, updated with today's 2 runs.\n- #52459 (Anthropic proxy ConnectionRefused) — **does not cover today's AgentRx failure**; corrected with a note pointing to the new P0 sub-issue below.\n- #52253 (Copilot proxy ECONNREFUSED port 10002) — **does not cover any of today's 3 Copilot CLI failures**; corrected with a note — none of today's occurrences show ECONNREFUSED to the proxy.\n- #52726, #53262, #52502, #53191 — no new evidence this window; left open, no action.\n\n### Fix roadmap\n\n**P0** — Add a hard pre-flight gate that fails the job fast (with a clear `##[error]`) when the Cloud Hypervisor guest-connectivity probe reports `lo: DOWN`, instead of letting it cascade into an opaque engine-specific timeout/startup error. Root-cause the network bring-up race in the runner/hypervisor image itself. Tracked in the new sub-issue.\n\n**P1** — (a) File a fix for `check_skip_if_match.cjs`: add retries (e.g. 2 with backoff) around the Search API call before treating it as a hard gate failure. (b) Keep #53263 open until a landed commit actually retries/isolates a single bad safe-output instead of failing the whole batch.\n\n**P2** — No action needed; watch for recurrence of the copilot-sdk 3002 timeout and `model_not_supported` before filing.\n\n### Sub-issues created\n\n- New P0 sub-issue: Cloud Hypervisor guest network failure (linked below).\n\n**References:**\n- [§32220130823](https://github.com/github/gh-aw/actions/runs/32220130823)\n- [§32218752360](https://github.com/github/gh-aw/actions/runs/32218752360)\n- [§32217761657](https://github.com/github/gh-aw/actions/runs/32217761657)\n\n\n\u003c!-- gh-aw-tracker-id: aw-failure-investigator --\u003e\n\n\u003e Generated by [🔍 [aw] Failure Investigator (6h)](https://github.com/github/gh-aw/actions/runs/32225993349) · agent · 167.2 AIC · ⌖ 10.4 AIC · ⊞ 5.9K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+is%3Aissue+%22gh-aw-workflow-call-id%3A+github%2Fgh-aw%2Faw-failure-investigator%22\u0026type=issues)\n\u003e - [x] expires \u003c!-- gh-aw-expires: 2026-08-26T07:17:44.281Z --\u003e on Aug 25, 2026, 11:17 PM UTC-08:00\n\n\u003c!-- gh-aw-agentic-workflow: [aw] Failure Investigator (6h), gh-aw-tracker-id: aw-failure-investigator, engine: claude, model: agent, id: 32225993349, workflow_id: aw-failure-investigator, run: https://github.com/github/gh-aw/actions/runs/32225993349 --\u003e\n\n\u003c!-- gh-aw-workflow-id: aw-failure-investigator --\u003e\n\u003c!-- gh-aw-workflow-call-id: github/gh-aw/aw-failure-investigator --\u003e","createdAt":"2026-08-19T07:1
|
|
agent
` line, even though `ca...`
|
|
agent
` line, even though `capture_likely_missed_fault` is reported as `false`.\
|
|
agent
` line, even though `capture_likely_missed_fault` is reported as `false`.\n\n### Affected workflows and runs\n\n| Workflow | Run | Time (UTC) | Failed step |\n|---|---|---|---|\n| Ponytail Reviewer | [§32249649808](https://github.com/github/gh-aw/actions/runs/32249649808) | 2026-08-19 11:51 | `Execute GitHub Copilot CLI` |\n| Ponytail Reviewer | [§32240222020](https://github.com/github/gh-aw/actions/runs/32240222020) | 2026-08-19 09:57 | `Execute GitHub Copilot CLI` |\n| Ponytail Reviewer | [§32239461549](https://github.com/github/gh-aw/actions/runs/32239461549) | 2026-08-19 09:48 | `Execute GitHub Copilot CLI` |\n| Daily Max Ai Credits Test | [§32246708586](https://github.com/github/gh-aw/actions/runs/32246708586) | 2026-08-19 11:15 | `Execute GitHub Copilot CLI` |\n\nDaily Max Ai Credits Test has failed at this exact step on **every scheduled run for the last 10 consecutive days** (2026-08-10 through 2026-08-19) — this is chronic, not a one-off blip. Ponytail Reviewer fails intermittently (3 of ~15 runs in the same window) at the same step.\n\n### Evidence\n\n`audit-diff` of failed run [§32249649808](https://github.com/github/gh-aw/actions/runs/32249649808) against the next successful Ponytail Reviewer run [§32250065978](https://github.com/github/gh-aw/actions/runs/32250065978) (5 minutes later):\n\n- Failed run made **0 allowed and 0 blocked** requests to `api.githubcopilot.com:443`. The successful run made 22 allowed requests to the same domain.\n- Failed run's token usage is 0.\n- No firewall block, no proxy-refusal signature — the process never attempted the network call at all.\n\nThis rules out proxy/firewall causes (unlike #52459's Anthropic proxy issue, which is a different engine and shows connection-refused *after* the attempt). The Copilot CLI is dying before it opens a connection — most consistent with a startup/config/auth failure inside the CLI binary itself.\n\n### Probable root cause\n\nTwo compounding bugs:\n1. **Tail-capture regression** (re-opening the #52501 defect class): the 50-line tail capture still isn't anchored to the failing step, so it grabs the MCP safeoutputs backend's `tools/list` handshake dump instead of the Copilot CLI's actual exit/error output.\n2. **Underlying Copilot CLI startup failure** (currently invisible due to #1): the CLI exits before any network call, on both a low-frequency workflow (Ponytail Reviewer) and a workflow that has failed at this exact point for 10+ straight days (Daily Max Ai Credits Test).\n\n### Proposed remediation\n\n1. Re-apply/verify the #52501 fix (step-anchored log capture, or backward-search for `##[error]`/non-zero exit before falling back to absolute tail) — it has clearly regressed since 2026-08-14.\n2. Once real error text is visible, pull the full raw log (not just the tail) for one fresh Ponytail Reviewer failure and one fresh Daily Max Ai Credits Test failure to pin the actual Copilot CLI startup error.\n3. Add a regression guard: flag prefetch entries whose `tail_lines` contain no `error`/`exception`/non-zero-exit marker as `capture_likely_missed_fault: true` even when the heuristic currently says `false` — the false-negative here is exactly what let this regress silently.\n\n### Success criteria / verification\n\n- A fresh Ponytail Reviewer or Daily Max Ai Credits Test failure's `truncated_error_logs.tail_lines` contains the actual Copilot CLI error, not backend startup logs.\n- Root cause of the pre-network Copilot CLI exit is identified and Daily Max Ai Credits Test passes 3 consecutive scheduled runs (or is confirmed as an intentionally-failing test workflow, in which case it should be excluded from failure tracking).\n\nParent report: #53933\nRelated: #52501 (closed, regressed)\nEOF\nwc -l /tmp/gh-aw/agent/subissue-1.md"},"caller":{"type":"direct"}}],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":384,"cache_read_input_tokens":126016,"cache_creation":{"ephemeral_5m_input_tokens":384,"ephemeral_1h_input_tokens":0},"output_tokens":3,"service_tier":"standard"
|
|
agent
` line and surrounding context out of the captured tail before truncation happens.\\n\\n### Proposed remediation\\n\\n1. Anchor capture to the failing step: use its start/end log offsets (or the nearest `##[error]` marker) as the truncation window, not the job's absolute end.\\n2. If a step-anchored window isn't available, search backward from the tail for the first `##[error]` line and capture N lines centered on it, falling back to the absolute tail only if no `##[error]` is found.\\n3. Add a regression check: flag prefetch entries whose `tail_lines` contain no `error`/`panic`/`exception`/`##[error]`","is_error":false}]},"parent_tool_use_id":null,"session_id":"20d1a990-030c-4615-93c9-4bb0af51711d","uuid":"add6b430-48cd-4393-9812-9d2ec2b626dd","timestamp":"2026-08-19T13:13:19.303Z","tool_use_result":{"stdout":"{\"body\":\"### Fix the log-tail capture — it's blinding root-cause analysis for most failures this window\\n\\nAnchor error-log capture to the failing step, not the job's absolute tail. In this run's 6h window (2026-08-13 ~06:38–12:38 UTC), 4 of 5 detailed failures had `truncated_error_logs[].tail_lines` (last 50 lines) containing **only post-failure cleanup/teardown output** — never the actual error from the failing step.\\n\\n### Affected workflows and runs\\n\\n| Run | Workflow | Failing step | What the tail actually captured |\\n|---|---|---|---|\\n| [§31697879776](https://github.com/github/gh-aw/actions/runs/31697879776) | PR Sous Chef | `Start DIFC Proxy` | artifact upload, git config unset, job outputs |\\n| [§31695362450](https://github.com/github/gh-aw/actions/runs/31695362450) | Daily Max Ai Credits Test | `Execute GitHub Copilot CLI` | OTLP skip notice, git config unset, job outputs |\\n| [§31695218197](https://github.com/github/gh-aw/actions/runs/31695218197) | Daily Choice Type Test | `Execute Claude Code CLI` | git config unset, orphan-process termination |\\n| [§31684831832](https://github.com/github/gh-aw/actions/runs/31684831832) | Daily AW Cross-Repo Compile Check | `Execute Claude Code CLI` | git config unset, orphan-process termination |\\n\\nOnly 1 of 5 runs this window ([§31700716464](https://github.com/github/gh-aw/actions/runs/31700716464), the Bun segfault in #51984) had a usable error line — and only because the crash happened seconds before teardown started, so the panic text hadn't yet scrolled out of the last-50-lines window.\\n\\n### Probable root cause\\n\\nThe tail capture grabs the last N lines of the **entire job log**, not a window anchored to the failing step's own log range. Jobs that run substantial cleanup/teardown after a failing step (artifact upload, git config unset, OTLP export, orphan-process cleanup) push the real `##[error]` line and surrounding context out of the captured tail before truncation happens.\\n\\n### Proposed remediation\\n\\n1. Anchor capture to the failing step: use its start/end log offsets (or the nearest `##[error]` marker) as the truncation window, not the job's absolute end.\\n2. If a step-anchored window isn't available, search backward from the tail for the first `##[error]` line and capture N lines centered on it, falling back to the absolute tail only if no `##[error]` is found.\\n3. Add a regression check: flag prefetch entries whose `tail_lines` contain no `error`/`panic`/`exception`/`##[error]`","stderr":"","interrupted":false,"isImage":false,"noOutputExpected":false}}
|
|
evals
` line, even though `capture_likely_missed_fault` is reported as `false`.\n\n### Affected workflows and runs\n\n| Workflow | Run | Time (UTC) | Failed step |\n|---|---|---|---|\n| Ponytail Reviewer | [§32249649808](https://github.com/github/gh-aw/actions/runs/32249649808) | 2026-08-19 11:51 | `Execute GitHub Copilot CLI` |\n| Ponytail Reviewer | [§32240222020](https://github.com/github/gh-aw/actions/runs/32240222020) | 2026-08-19 09:57 | `Execute GitHub Copilot CLI` |\n| Ponytail Reviewer | [§32239461549](https://github.com/github/gh-aw/actions/runs/32239461549) | 2026-08-19 09:48 | `Execute GitHub Copilot CLI` |\n| Daily Max Ai Credits Test | [§32246708586](https://github.com/github/gh-aw/actions/runs/32246708586) | 2026-08-19 11:15 | `Execute GitHub Copilot CLI` |\n\nDaily Max Ai Credits Test has failed at this exact step on **every scheduled run for the last 10 consecutive days** (2026-08-10 through 2026-08-19) — this is chronic, not a one-off blip. Ponytail Reviewer fails intermittently (3 of ~15 runs in the same window) at the same step.\n\n### Evidence\n\n`audit-diff` of failed run [§32249649808](https://github.com/github/gh-aw/actions/runs/32249649808) against the next successful Ponytail Reviewer run [§32250065978](https://github.com/github/gh-aw/actions/runs/32250065978) (5 minutes later):\n\n- Failed run made **0 allowed and 0 blocked** requests to `api.githubcopilot.com:443`. The successful run made 22 allowed requests to the same domain.\n- Failed run's token usage is 0.\n- No firewall block, no proxy-refusal signature — the process never attempted the network call at all.\n\nThis rules out proxy/firewall causes (unlike #52459's Anthropic proxy issue, which is a different engine and shows connection-refused *after* the attempt). The Copilot CLI is dying before it opens a connection — most consistent with a startup/config/auth failure inside the CLI binary itself.\n\n### Probable root cause\n\nTwo compounding bugs:\n1. **Tail-capture regression** (re-opening the #52501 defect class): the 50-line tail capture still isn't anchored to the failing step, so it grabs the MCP safeoutputs backend's `tools/list` handshake dump instead of the Copilot CLI's actual exit/error output.\n2. **Underlying Copilot CLI startup failure** (currently invisible due to #1): the CLI exits before any network call, on both a low-frequency workflow (Ponytail Reviewer) and a workflow that has failed at this exact point for 10+ straight days (Daily Max Ai Credits Test).\n\n### Proposed remediation\n\n1. Re-apply/verify the #52501 fix (step-anchored log capture, or backward-search for `##[error]`/non-zero exit before falling back to absolute tail) — it has clearly regressed since 2026-08-14.\n2. Once real error text is visible, pull the full raw log (not just the tail) for one fresh Ponytail Reviewer failure and one fresh Daily Max Ai Credits Test failure to pin the actual Copilot CLI startup error.\n3. Add a regression guard: flag prefetch entries whose `tail_lines` contain no `error`/`exception`/non-zero-exit marker as `capture_likely_missed_fault: true` even when the heuristic currently says `false` — the false-negative here is exactly what let this regress silently.\n\n### Success criteria / verification\n\n- A fresh Ponytail Reviewer or Daily Max Ai Credits Test failure's `truncated_error_logs.tail_lines` contains the actual Copilot CLI error, not backend startup logs.\n- Root cause of the pre-network Copilot CLI exit is identified and Daily Max Ai Credits Test passes 3 consecutive scheduled runs (or is confirmed as an intentionally-failing test workflow, in which case it should be excluded from failure tracking).\n\nParent report: #53933\nRelated: #52501 (closed, regressed)\n","labels":["bug","automation","agentic-workflows"],"parent":53933,"title":"[aw-failures] [P0] Log-capture blindness regressed — GH Copilot CLI failures (Ponytail Reviewer, Daily Max Ai Credits Test) invisible again (#52501 relapse)"}}}
|
|
evals
` line, even though `capture_likely_missed_fault` is reported as `false`.\n\n### Affected workflows and runs\n\n| Workflow | Run | Time (UTC) | Failed step |\n|---|---|---|---|\n| Ponytail Reviewer | [§32249649808](https://github.com/github/gh-aw/actions/runs/32249649808) | 2026-08-19 11:51 | `Execute GitHub Copilot CLI` |\n| Ponytail Reviewer | [§32240222020](https://github.com/github/gh-aw/actions/runs/32240222020) | 2026-08-19 09:57 | `Execute GitHub Copilot CLI` |\n| Ponytail Reviewer | [§32239461549](https://github.com/github/gh-aw/actions/runs/32239461549) | 2026-08-19 09:48 | `Execute GitHub Copilot CLI` |\n| Daily Max Ai Credits Test | [§32246708586](https://github.com/github/gh-aw/actions/runs/32246708586) | 2026-08-19 11:15 | `Execute GitHub Copilot CLI` |\n\nDaily Max Ai Credits Test has failed at this exact step on **every scheduled run for the last 10 consecutive days** (2026-08-10 through 2026-08-19) — this is chronic, not a one-off blip. Ponytail Reviewer fails intermittently (3 of ~15 runs in the same window) at the same step.\n\n### Evidence\n\n`audit-diff` of failed run [§32249649808](https://github.com/github/gh-aw/actions/runs/32249649808) against the next successful Ponytail Reviewer run [§32250065978](https://github.com/github/gh-aw/actions/runs/32250065978) (5 minutes later):\n\n- Failed run made **0 allowed and 0 blocked** requests to `api.githubcopilot.com:443`. The successful run made 22 allowed requests to the same domain.\n- Failed run's token usage is 0.\n- No firewall block, no proxy-refusal signature — the process never attempted the network call at all.\n\nThis rules out proxy/firewall causes (unlike #52459's Anthropic proxy issue, which is a different engine and shows connection-refused *after* the attempt). The Copilot CLI is dying before it opens a connection — most consistent with a startup/config/auth failure inside the CLI binary itself.\n\n### Probable root cause\n\nTwo compounding bugs:\n1. **Tail-capture regression** (re-opening the #52501 defect class): the 50-line tail capture still isn't anchored to the failing step, so it grabs the MCP safeoutputs backend's `tools/list` handshake dump instead of the Copilot CLI's actual exit/error output.\n2. **Underlying Copilot CLI startup failure** (currently invisible due to #1): the CLI exits before any network call, on both a low-frequency workflow (Ponytail Reviewer) and a workflow that has failed at this exact point for 10+ straight days (Daily Max Ai Credits Test).\n\n### Proposed remediation\n\n1. Re-apply/verify the #52501 fix (step-anchored log capture, or backward-search for `##[error]`/non-zero exit before falling back to absolute tail) — it has clearly regressed since 2026-08-14.\n2. Once real error text is visible, pull the full raw log (not just the tail) for one fresh Ponytail Reviewer failure and one fresh Daily Max Ai Credits Test failure to pin the actual Copilot CLI startup error.\n3. Add a regression guard: flag prefetch entries whose `tail_lines` contain no `error`/`exception`/non-zero-exit marker as `capture_likely_missed_fault: true` even when the heuristic currently says `false` — the false-negative here is exactly what let this regress silently.\n\n### Success criteria / verification\n\n- A fresh Ponytail Reviewer or Daily Max Ai Credits Test failure's `truncated_error_logs.tail_lines` contains the actual Copilot CLI error, not backend startup logs.\n- Root cause of the pre-network Copilot CLI exit is identified and Daily Max Ai Credits Test passes 3 consecutive scheduled runs (or is confirmed as an intentionally-failing test workflow, in which case it should be excluded from failure tracking).\n\nParent report: #53933\nRelated: #52501 (closed, regressed)\n","labels":["bug","automation","agentic-workflows"],"parent":53933,"title":"[aw-failures] [P0] Log-capture blindness regressed — GH Copilot CLI failures (Ponytail Reviewer, Daily Max Ai Credits Test) invisible again (#52501 relapse)"},"name":"create_issue"}}}
|
|
evals
ERR_API: Failed to execute search query: Connect Timeout Error\n```\nThe github-script step is configured with `retries: 0`. Any single transient 500/timeout from the Search API is guaranteed to hard-fail the entire workflow with no retry — this isn't a one-off, it's a design gap that will keep recurring under normal GitHub API flakiness. No existing issue covers `check_skip_if_match`/`pre_activation`.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eP2 — two more \"Execute Copilot CLI\" failures with unrelated root causes (do not merge with #52253)\u003c/summary\u003e\n\n- Daily GitHub Docs SEO Optimizer [§32217479194](https://github.com/github/gh-aw/actions/runs/32217479194): the outbound api-proxy on port 10002 accepted connections fine; the failure is the **local** `copilot-sdk` headless server never reporting ready — `unexpected error: copilot-sdk headless server did not become ready on 127.0.0.1:3002 within 60000ms (connect ETIMEDOUT 127.0.0.1:3002)`. This is internal-to-the-guest, unrelated to #52253's external ECONNREFUSED-on-10002 signature.\n- Metrics Collector [§32209589795](https://github.com/github/gh-aw/actions/runs/32209589795): `failureClass=model_not_supported` in the copilot-harness attempt log — a model-selection bug, not a connectivity issue.\n\nBoth are single occurrences this window — watching, not filing separately yet.\n\n\u003c/details\u003e\n\n### Existing issue correlation\n\n- #53263 (safe_outputs hard-fail) — **confirmed still broken**, updated with today's 2 runs.\n- #52459 (Anthropic proxy ConnectionRefused) — **does not cover today's AgentRx failure**; corrected with a note pointing to the new P0 sub-issue below.\n- #52253 (Copilot proxy ECONNREFUSED port 10002) — **does not cover any of today's 3 Copilot CLI failures**; corrected with a note — none of today's occurrences show ECONNREFUSED to the proxy.\n- #52726, #53262, #52502, #53191 — no new evidence this window; left open, no action.\n\n### Fix roadmap\n\n**P0** — Add a hard pre-flight gate that fails the job fast (with a clear `##[error]`) when the Cloud Hypervisor guest-connectivity probe reports `lo: DOWN`, instead of letting it cascade into an opaque engine-specific timeout/startup error. Root-cause the network bring-up race in the runner/hypervisor image itself. Tracked in the new sub-issue.\n\n**P1** — (a) File a fix for `check_skip_if_match.cjs`: add retries (e.g. 2 with backoff) around the Search API call before treating it as a hard gate failure. (b) Keep #53263 open until a landed commit actually retries/isolates a single bad safe-output instead of failing the whole batch.\n\n**P2** — No action needed; watch for recurrence of the copilot-sdk 3002 timeout and `model_not_supported` before filing.\n\n### Sub-issues created\n\n- New P0 sub-issue: Cloud Hypervisor guest network failure (linked below).\n\n**References:**\n- [§32220130823](https://github.com/github/gh-aw/actions/runs/32220130823)\n- [§32218752360](https://github.com/github/gh-aw/actions/runs/32218752360)\n- [§32217761657](https://github.com/github/gh-aw/actions/runs/32217761657)\n\n\n\u003c!-- gh-aw-tracker-id: aw-failure-investigator --\u003e\n\n\u003e Generated by [🔍 [aw] Failure Investigator (6h)](https://github.com/github/gh-aw/actions/runs/32225993349) · agent · 167.2 AIC · ⌖ 10.4 AIC · ⊞ 5.9K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+is%3Aissue+%22gh-aw-workflow-call-id%3A+github%2Fgh-aw%2Faw-failure-investigator%22\u0026type=issues)\n\u003e - [x] expires \u003c!-- gh-aw-expires: 2026-08-26T07:17:44.281Z --\u003e on Aug 25, 2026, 11:17 PM UTC-08:00\n\n\u003c!-- gh-aw-agentic-workflow: [aw] Failure Investigator (6h), gh-aw-tracker-id: aw-failure-investigator, engine: claude, model: agent, id: 32225993349, workflow_id: aw-failure-investigator, run: https://github.com/github/gh-aw/actions/runs/32225993349 --\u003e\n\n\u003c!-- gh-aw-workflow-id: aw-failure-investigator --\u003e\n\u003c!-- gh-aw-workflow-call-id: github/gh-aw/aw-failure-investigator --\u003e","createdAt":"2026-08-19T07:1
|
|
evals
` line, even though `ca...`
|
|
agent
Failed to scan directory /tmp/gh-aw/aw-mcp/logs/run-32250065978/usage: EACCES: permission denied, scandir '/tmp/gh-aw/aw-mcp/logs/run-32250065978/usage'
|
|
agent
Failed to scan directory /tmp/gh-aw/aw-mcp/logs/run-32249649808/usage: EACCES: permission denied, scandir '/tmp/gh-aw/aw-mcp/logs/run-32249649808/usage'
|
|
agent
Patch diff size: 13 KB (12767 bytes) (configured limit: 10 KB (10240 bytes), effective with 20% overhead: 12 KB (12288 bytes))\\npush_repo_memory\\tPush repo-memory changes (default)\\t2026-08-19T12:34:13.7483543Z ##[warning]Patch content breakdown (git diff --stat):\\npush_repo_memory\\tPush repo-memory changes (default)\\t deep-report/extracted-tasks.json | 35 ++++++++++++++++++++++++++++++++++\\npush_repo_memory\\tPush repo-memory changes (default)\\t deep-report/flagged_items.md | 15 +++++++++++++++\\npush_repo_memory\\tPush repo-memory changes (default)\\t deep-report/known_patterns.md | 8 ++++++++\\npush_repo_memory\\tPush repo-memory changes (default)\\t deep-report/last_analysis_timestamp.md | 27 +++++++++++++++++++++++++-\\npush_repo_memory\\tPush repo-memory changes (default)\\t deep-report/processed-discussions.json | 4 ++++\\npush_repo_memory\\tPush repo-memory changes (default)\\t deep-report/trend_data.md | 15 +++++++++++++++\\npush_repo_memory\\tPush repo-memory changes (default)\\t 6 files changed, 103 insertions(+), 1 deletion(-)\\npush_repo_memory\\tPush repo-memory changes (default)\\t\\npush_repo_memory\\tPush repo-memory changes (default)\\t2026-08-19T12:34:13.7491193Z ##[error]Patch diff size (13 KB, 12767 bytes) exceeds maximum allowed size (12 KB, 12288 bytes, configured limit: 10 KB with 20% overhead allowance). Reduce the number or size of changes, or increase max-patch-size.\\npush_repo_memory\\tRestore actions folder\\t\\ufeff2026-08-19T12:34:13.7890048Z ##[group]Run actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7890866Z with:\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7891298Z repository: github/gh-aw\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7891798Z sparse-checkout: actions/setup\\npush_repo_memory\\tRestore actions folder\\t\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7892330Z sparse-checkout-cone-mode: true\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7892808Z clean: false\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7893225Z persist-credentials: false\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7896599Z token: ***\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7897008Z ssh-strict: true\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7897412Z ssh-user: git\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7897808Z fetch-depth: 1\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7898202Z fetch-tags: false\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7898964Z show-progress: true\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7899641Z lfs: false\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7900024Z submodules: false\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7900440Z set-safe-directory: true\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7900919Z allow-unsafe-pr-checkout: false\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7901408Z env:\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7902043Z OTEL_EXPORTER_OTLP_ENDPOINT: ***\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7902471Z OTEL_SERVICE_NAME: gh-aw.deep-report\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7903426Z OTEL_RESOURCE_ATTRIBUTES: gh-aw.workflow.name=Deep%20Report,gh-aw.repository=github/gh-aw,gh-aw.run.id=32252000817,github.run_id=32252000817,gh-aw.engine.id=claude\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7904564Z OTEL_EXPORTER_OTLP_HEADERS: x-sentry-auth=***\\npush_repo_memory\\tRestore actions folder\\t2026-08-19T12:34:13.7906611Z GH_AW_OTLP_ALL_HEADERS: x-sentry-auth=***,Authorization=***\",\n38\t \"capture_likely_missed_fault\": false\n39\t }\n40\t
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
activation
Expired
|
9.14 MB |
sha256:2d54cdd11fce02e9a508539f943e04c4c57f14c72c910b4139331cd90af4b237
|
|
|
agent
|
351 KB |
sha256:05a671ebae0a30ce0a8c1813387b62ee7f53b3ce01f1ab3eb79b2ea7bdb993a7
|
|
|
agent-output-fallback
|
7.41 KB |
sha256:8342b13fbaf53a09bb0a7bd753600592a344cd43c374aac834cc0150203989b0
|
|
|
aic-usage-cache
|
476 Bytes |
sha256:75e0e1552c8651e9d54c8cd97fa63b3a493a9f9e5dee5e6d5be2d157d4b9049a
|
|
|
awfailureinvestigator-experiment
|
5.43 KB |
sha256:2b494e3cb900f9a09dc694849eaa93387a64f00fae8562a4edfa9150ff48001b
|
|
|
detection
|
233 Bytes |
sha256:63d2fbfec15bad38fdb692a100276f45dad4fda9f62e2c0c458e01572a15938f
|
|
|
evals
|
383 Bytes |
sha256:f89d5552e36b2a15930c6f4aacb92e31950f5986cf6a72239ccea786affae8ce
|
|
|
github~gh-aw~MMUC2F.dockerbuild
|
24 KB |
sha256:70fb540dedf052a242869f3ccc8e749a43b3047db14e20dc537c7f91049bcc06
|
|
|
safe-outputs-items
|
827 Bytes |
sha256:29974890cec6cca5c36006792b3d720837d936b7fc3efb1b87909d288b3212f8
|
|
|
usage
|
4.29 KB |
sha256:404df2bf222d20718ee17b58699feed1b98a3fd537ee1c26ebcb9c5e636cf55a
|
|