[aw] Failure Investigator (6h) #495
aw-failure-investigator.lock.yml
on: schedule
Annotations
18 errors and 3 warnings
|
agent
Process completed with exit code 1.\n```\nThis cascades into the conclusion job:\n```\n##[error]❌ Failed to parse detection result: No THREAT_DETECTION_RESULT found in detection log. The detection model may have failed to follow the output format.\n```\n`audit-diff` between the two failed runs shows no meaningful drift (routine MCP tool-discovery noise only) — this is a stable, reproducing bug, not a regression tied to a specific commit.\n\n### Root cause\n\n`pkg/workflow/data/ecosystem_domains.json:259-269` defines the `\"threat-detection\"` ecosystem domain list used by `GetThreatDetectionAllowedDomains` (`pkg/workflow/domains.go:746`):\n```json\n\"threat-detection\": [\n \"api.business.githubcopilot.com\",\n \"api.enterprise.githubcopilot.com\",\n \"api.github.com\",\n \"api.githubcopilot.com\",\n \"api.individual.githubcopilot.com\",\n \"github.com\",\n \"host.docker.internal\",\n \"registry.npmjs.org\",\n \"telemetry.enterprise.githubcopilot.com\"\n]\n```\nThe comment above `GetThreatDetectionAllowedDomains` (`domains.go:738-744`) explains the list intentionally excludes `raw.githubusercontent.com` because \"the CLI binary is pre-installed\" — true for Copilot-based detection, but **not** true for the Goose engine: `goose_harness.cjs` downloads the Goose release binary at runtime via a `github.com/.../releases/download/...` URL, which GitHub redirects to `release-assets.githubusercontent.com`. That domain is missing from the list, so the redirect gets firewalled and the download 403s before Goose ever runs, so detection never emits `THREAT_DETECTION_RESULT`.\n\n### Proposed remediation\n\nAdd `\"release-assets.githubusercontent.com\"` to the `\"threat-detection\"` array in `pkg/workflow/data/ecosystem_domains.json`. Recompile all `.lock.yml` files that embed this ecosystem list. Consider also updating the comment at `domains.go:738-744` to note the Goose-engine exception.\n\n### Success criteria\n\n- Daily Team Evolution Insights, Daily Token Consumption Report, and Smoke Goose all complete their detection step with a valid `THREAT_DETECTION_RESULT` across the next 3 scheduled runs each, with no `Goose download failed` / `release-assets.githubusercontent.com` block in the logs.\n","labels":["bug","automation","agentic-workflows"],"parent":"aw_parent1","title":"[P0] Goose engine detection step blocked by missing release-assets.githubusercontent.com in firewall allowlist"}}}
|
|
agent
Process completed with exit code 1.\n##[error]❌ Failed to parse detection result: No THREAT_DETECTION_RESULT found in detection log.\n```\n`audit-diff` between the two failed runs shows no meaningful drift (only routine MCP tool-discovery noise) — this is a persistent bug, not a regression from a specific commit. Three workflows currently declare `engine: goose`: `daily-team-evolution-insights.md`, `daily-token-consumption-report.md`, `smoke-goose.md` — all three are structurally exposed since the detection step's firewall config is shared, not per-workflow.\n\n**Cluster B — Design Decision Gate invocation-cap exhaustion**\n\nThe main agent spawned a sub-agent with explicit instructions: *\"Use `push_to_pull_request_branch`... then use `add_comment`.\"* The sub-agent instead tried, in order: `git push` (denied, needs approval), `gh pr comment` (denied), `cat /tmp/adr-comment.md` (missing file), `ls /tmp` (blocked — outside allowed working dir), more file-path guesses, a GraphQL call (403 `access denied: unrecognized GraphQL operation`), and repeated `git push`/`gh` attempts — 21 tool calls total before hitting `API Error: Request rejected (429) · Maximum LLM invocations exceeded (30/30)`, which terminated the parent Claude Code CLI process with exit code 1. The workflow's `bash:` allowlist (`.github/workflows/design-decision-gate.md:70-80`) only permits read-only git/cat/grep/ls/wc/find/echo — it was never going to let raw `git push`/`gh pr comment` through, so the sub-agent's retry loop was guaranteed to fail before it started.\n\n**Cluster C — BYOK Ollama 503**\n\n4 consecutive Copilot-harness attempts (~33-38s each) all failed with `Failed to get response from the AI model; retried 5 times ... Last error: 503 Service Unavailable` against `api-proxy:10002/v1/models`, targeting Ollama at `host.docker.internal:11434`. No firewall blocks, no auth errors — looks like the Ollama service wasn't listening yet when the agent started.\n\n</details>\n\n### Sub-issues created\n\n- P0: Goose threat-detection firewall allowlist (this issue's #aw_p0goose sub-issue)\n- P1: Design Decision Gate invocation-cap exhaustion (this issue's #aw_p1ddgate sub-issue)\n\n**References:**\n- [§32064592237](https://github.com/github/gh-aw/actions/runs/32064592237)\n- [§32069307664](https://github.com/github/gh-aw/actions/runs/32069307664)\n- [§32076040028](https://github.com/github/gh-aw/actions/runs/32076040028)\n","labels":["bug","automation","agentic-workflows"],"temporary_id":"aw_parent1","title":"[aw] Failure Investigator Report — 2026-08-18 01:15 UTC (6h)"}}}
|
|
agent
Process completed with exit code 1.\",\n \"impact\": \"Workflow did not complete successfully and may need intervention\"\n },\n {\n \"category\": \"optimization\",\n \"severity\": \"low\",\n \"title\": \"Partially Reducible To Deterministic\",\n \"description\": \"About 50% of this run's turns appear to be data-gathering that could move to deterministic steps.\",\n \"impact\": \"Moving data-gathering turns to deterministic steps reduces inference cost\"\n }\n ],\n \"recommendations\": [\n {\n \"priority\": \"high\",\n \"action\": \"Review error logs to identify root cause of failure\",\n \"reason\": \"Understanding failure causes helps prevent recurrence\",\n \"example\": \"Check the errors field for specific error messages, or inspect the log files in logs_path\"\n },\n {\n \"priority\": \"medium\",\n \"action\": \"Move data-fetching work to frontmatter steps: (pre-agent) writing to /tmp/gh-aw/agent/ or post-steps: (post-agent) to reduce inference cost. See the DeterministicOps guide.\",\n \"reason\": \"About 50% of this run's turns appear to be data-gathering that could move to deterministic steps.\"\n }\n ],\n \"observability_insights\": [\n {\n \"category\": \"execution\",\n \"severity\": \"info\",\n \"title\": \"Directed execution path\",\n \"summary\": \"The run remained relatively linear with 4 turns and 0 tool types.\",\n \"evidence\": \"turns=4 tool_types=0\"\n },\n {\n \"category\": \"actuation\",\n \"severity\": \"info\",\n \"title\": \"Read-only posture observed\",\n \"summary\": \"The workflow stayed in an analysis posture and did not emit any GitHub write actions.\",\n \"evidence\": \"created_items=0 safe_items=0\"\n },\n {\n \"category\": \"network\",\n \"severity\": \"info\",\n \"title\": \"Network policy aligned\",\n \"summary\": \"The firewall observed 59 request(s) with 0 blocked, for a 0% block rate.\",\n \"evidence\": \"blocked=0 total=59\"\n },\n {\n \"category\": \"execution\",\n \"severity\": \"info\",\n \"title\": \"Log template patterns mined\",\n \"summary\": \"Analysis identified 2 distinct event templates across 6 pipeline stages from 2 events.\",\n \"evidence\": \"finish=1 plan=1\"\n },\n {\n \"category\": \"reliability\",\n \"severity\": \"low\",\n \"title\": \"1 anomalous event pattern(s) detected\",\n \"summary\": \"Anomaly detection flagged 1 event(s) as unusual based on template similarity and cluster rarity.\",\n \"evidence\": \"stage=plan score=0.65: new log template discovered; rare cluster (few observations)\"\n },\n {\n \"category\": \"execution\",\n \"severity\": \"info\",\n \"title\": \"Agent stage sequence\",\n \"summary\": \"The observed pipeline stage sequence for this run.\",\n \"evidence\": \"plan finish\"\n }\n ],\n \"performance_metrics\": {\n \"network_requests\": 59\n },\n \"engine_config\": {\n \"engine_id\": \"copilot\",\n \"engine_name\": \"GitHub Copilot CLI\",\n \"model\": \"auto\",\n \"version\": \"1.0.80\",\n \"firewall_version\": \"v0.28.1\",\n \"trigger_event\": \"schedule\",\n \"repository\": \"github/gh-aw\"\n },\n \"prompt_analysis\": {\n \"prompt_size\": 10742,\n \"prompt_file\": \"aw-prompts/prompt.txt\"\n },\n \"session_analysis\": {\n \"wall_time\": \"13.5m\",\n \"turn_count\": 4,\n \"avg_turn_duration\": \"3.4m\",\n \"avg_time_between_turns\": \"46.8s\",\n \"max_time_between_turns\": \"54.7s\",\n \"timeout_detected\": false\n },\n \"safe_output_summary\": {\n \"total_items\": 0,\n \"items_by_type\": {},\n \"summary\": \"No items\",\n \"temporary_id_map_status\": \"missing\"\n },\n \"jobs\": [\n {\n \"name\": \"activation\",\n \"status\": \"completed\",\n \"conclusion\": \"success\",\n \"duration\": \"36.0s\",\n \"steps\": [\n {\n \"name\": \"Set up job\",\n \"
|
|
agent
Process completed with exit code 35.\n...\n/home/runner/work/_temp/....sh: line 43: awf: command not found\n##[error]Process completed with exit code 127.\n```\n\nCascade: `Install AWF binary` fails → later evals steps are skipped, including `Upload evals results` → the downstream `conclusion` job then fails with `Unable to download artifact(s): Artifact not found for name: evals`, which is what the CI status surfaces as the \"real\" error even though the true cause is upstream.\n\u003c/details\u003e\n\n**Probable root cause:** `install_awf_binary.sh` has no retry/backoff around the `curl` download of `checksums.txt` from GitHub Releases, so a single transient TLS reset takes down the whole evals job and only surfaces downstream as a confusing artifact-not-found error.\n\n**Proposed remediation:** add `curl --retry 3 --retry-delay 2 --retry-connrefused` (or equivalent) to the checksum/binary download in `install_awf_binary.sh`. Also make the downstream artifact-download step tolerate a missing `evals` artifact with a clear \"evals skipped upstream\" message instead of a raw `Unable to download artifact(s)` error.\n\n**Success criteria:** the install step retries through a transient network blip instead of failing the whole job; if retries are exhausted, the top-level failure names the real cause (network) instead of only showing a missing-artifact error three jobs downstream.\n\n---\n\n### Fix the MCP Gateway config (schema or generator) to accept digest-pinned container images — `serena`'s `@sha256:...` reference fails validation and blocks the whole run.\n\nPackage Specification Librarian failed at `Start MCP Gateway` in [§31949448689](https://github.com/github/gh-aw/actions/runs/31949448689) (2026-08-16 13:23 UTC), before the agent ever ran.\n\n\u003cdetails\u003e\n\u003csummary\u003eRaw failure evidence\u003c/summary\u003e\n\n```\nConfiguration validation error (MCP Gateway version: v0.4.9):\nError: jsonschema validation failed with '.../mcp-gateway-config.schema.json#'\n Error: 'oneOf' failed, none matched\n Error: 'ghcr.io/oraios/serena:latest@sha256:[REDACTED]' does not match pattern '^[a-zA-Z0-9][a-zA-Z0-9./_-]*(:([a-zA-Z0-9._-]+|latest))?$'\n Error: missing property 'url'\n Error: value must be 'http'\n Error: additional properties 'entrypointArgs', 'mounts', 'args', 'container', 'entrypoint' not allowed\nERROR: Gateway failed to become ready\n```\n\u003c/details\u003e\n\n**Probable root cause:** the committed lock file's `\"serena\"` MCP server config uses a plain tag (`ghcr.io/oraios/serena:latest`), but at runtime the gateway config is resolved/re-pinned to include a `@sha256:...` digest suffix — and the schema's container-image regex pattern rejects any `@sha256:` suffix, only bare tags pass. This is a **shared-component bug**: both Linter Miner and Package Specification Librarian import the same `shared/mcp/serena.md`, so every workflow using this shared Serena MCP config is at risk of hitting the same schema rejection.\n\n**Proposed remediation:** either (a) loosen `mcp-gateway-config.schema.json`'s container-image pattern to accept `@sha256:\u003c64-hex\u003e` digest suffixes, or (b) stop injecting a resolved digest into the `container` field for stdio-type MCP servers at gateway-config-build time, keeping the plain tag the lock file already specifies.\n\n**Success criteria:** Package Specification Librarian (and any other workflow importing `shared/mcp/serena.md`) starts the MCP Gateway with `serena` registered successfully, with no `jsonschema validation failed` error, across the next 3 scheduled runs.\n\n**References:**\n- [§31957063541](https://github.com/github/gh-aw/actions/runs/31957063541)\n- [§31949448689](https://github.com/github/gh-aw/actions/runs/31949448689)\nRelated to #53129\n\n\n\u003c!-- gh-aw-tracker-id: aw-failure-investigator --\u003e\n\n\u003e Generated by [🔍 [aw] Failure Investigator (6h)](https://github.com/github/gh-aw/actions/runs/31965944876) · agent · 210.1 AIC · ⌖ 7 AIC · ⊞ 5.6K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+is%3A
|
|
agent
ERR_PARSE: ❌ No THREAT_DETECTION_RESULT found in detection log...`).\n - `run-31969869860/workflow-logs/2_detection.txt` — comparator/drift-check run (yesterday's failure of the same workflow), confirmed IDENTICAL root-cause signature to today's Cluster A failure.\n - **`.github/workflows/shared/reporting.md`** — read via Read tool in full. This is the authoritative formatting-guideline file (the `reporting` Skill tool itself was unavailable — calling it returned \"Unknown skill: reporting\"). Content:\n ```\n - Use `###` (or lower) headers only.\n - Keep summary and critical actions visible; move long detail into `<details>` blocks.\n - Structure reports as: overview → key metrics/issues → collapsible detail → next actions.\n - Format run IDs as links: `[§12345](https://github.com/owner/repo/actions/runs/12345)`.\n - Include up to 3 most relevant run URLs at end under `**References:**`.\n - Do NOT add footer attribution (system adds automatically).\n ```\n - **`/tmp/gh-aw/agent/parent_report.md`** — CREATED by me via Write tool (heredoc in Bash). Full 58-line body composed for the new parent report issue, structured per the reporting guideline: an assertive imperative lead sentence, `### Summary` (clusters table: Cluster/Severity/Workflow(s)/Run/Root cause — Cluster A=P0/Daily Team Evolution Insights/§32064592237/Goose firewall block; Cluster B=P1/Design Decision Gate/§32069307664/sub-agent invocation-cap exhaustion; Cluster C=P2/Daily BYOK Ollama Test/§32076040028/Ollama 503s), `### Existing issue correlation` (explains #52459 and #53263 checked and ruled out, no closures made), `### Fix roadmap` (P0/P1/P2 imperative one-liners), `<details><summary>Evidence detail` (full technical evidence per cluster including exact log excerpts and file:line citations), `### Sub-issues created` (references `#aw_p0goose` and `#aw_p1ddgate` as placeholders — **these are the temporary_id values that MUST be used when creating the two sub-issues next**), and `**References:**` (the 3 run URLs). This exact file content was piped into the `create_issue` call for the parent issue.\n\n4. Errors and fixes:\n - **Truncated error logs in prefetch.json were unhelpful**: the `truncated_error_logs` field only contained the last 50 lines of each failed job, which for all 3 runs showed only MCP tool-registration/schema JSON dumps rather than the actual failure point. Fix: used the `agenticworkflows audit` CLI tool (via `printf '{\"run_id\":\"...\"}' | agenticworkflows audit .`) to download full logs locally, then grepped the full log files directly with Bash for error patterns.\n - **`agenticworkflows audit` on 3 run_ids together produced pairwise diffs, not per-run root-cause detail**: initially ran `audit` with `run_ids_or_urls` containing all 3 IDs, which returned `firewall_diff`/`run_metrics_diff` comparisons between run pairs (1v2, 1v3) rather than individual failure diagnostics. Fix: switched to calling `audit` once per individual `run_id` to get per-run `errors[]`, `firewall_analysis`, and file listings, and separately grepped the downloaded log files for the real root cause since the `audit` summary's `errors[]` field was too generic (just \"Process completed with exit code 1\").\n - **Wrong log filename assumed for run 32069307664**: initially grepped for `3_agent.txt` (the filename pattern from run 32076040028), got \"No such file or directory\". Fix: ran `ls` on the run's `workflow-logs/` directory and found the actual filename was `5_agent.txt` (this run had extra steps: evals pipeline steps shifted the numbering).\n - **Grep output too large / binary file match warning**: a grep on `5_agent.txt` for run 32069307664 returned \"binary file matches\" and a 100.7KB output that got persisted to a side file rather than shown inline; also attempting to `Read` that persisted output file directly failed with \"File content (58068 tokens) exceeds maximum allowed tokens (25000)\". Fix: used the `Read` tool result summary that WAS shown inline as a preview, then ran a NEW, more
|
|
agent
Process completed with exit code 1.\n```\nThis cascades into the conclusion job:\n```\n##[error]❌ Failed to parse detection result: No THREAT_DETECTION_RESULT found in detection log. The detection model may have failed to follow the output format.\n```\n`audit-diff` between the two failed runs shows no meaningful drift (routine MCP tool-discovery noise only) — this is a stable, reproducing bug, not a regression tied to a specific commit.\n\n### Root cause\n\n`pkg/workflow/data/ecosystem_domains.json:259-269` defines the `\"threat-detection\"` ecosystem domain list used by `GetThreatDetectionAllowedDomains` (`pkg/workflow/domains.go:746`):\n```json\n\"threat-detection\": [\n \"api.business.githubcopilot.com\",\n \"api.enterprise.githubcopilot.com\",\n \"api.github.com\",\n \"api.githubcopilot.com\",\n \"api.individual.githubcopilot.com\",\n \"github.com\",\n \"host.docker.internal\",\n \"registry.npmjs.org\",\n \"telemetry.enterprise.githubcopilot.com\"\n]\n```\nThe comment above `GetThreatDetectionAllowedDomains` (`domains.go:738-744`) explains the list intentionally excludes `raw.githubusercontent.com` because \"the CLI binary is pre-installed\" — true for Copilot-based detection, but **not** true for the Goose engine: `goose_harness.cjs` downloads the Goose release binary at runtime via a `github.com/.../releases/download/...` URL, which GitHub redirects to `release-assets.githubusercontent.com`. That domain is missing from the list, so the redirect gets firewalled and the download 403s before Goose ever runs, so detection never emits `THREAT_DETECTION_RESULT`.\n\n### Proposed remediation\n\nAdd `\"release-assets.githubusercontent.com\"` to the `\"threat-detection\"` array in `pkg/workflow/data/ecosystem_domains.json`. Recompile all `.lock.yml` files that embed this ecosystem list. Consider also updating the comment at `domains.go:738-744` to note the Goose-engine exception.\n\n### Success criteria\n\n- Daily Team Evolution Insights, Daily Token Consumption Report, and Smoke Goose all complete their detection step with a valid `THREAT_DETECTION_RESULT` across the next 3 scheduled runs each, with no `Goose download failed` / `release-assets.githubusercontent.com` block in the logs.\nEOF\njq -Rs --arg title \"[P0] Goose engine detection step blocked by missing release-assets.githubusercontent.com in firewall allowlist\" \\\n '{title: $title, body: ., labels: [\"bug\",\"automation\",\"agentic-workflows\"], parent: \"aw_parent1\"}' \\\n /tmp/gh-aw/agent/p0_goose.md | safeoutputs create_issue .","description":"Create P0 sub-issue for Goose firewall allowlist bug"},"caller":{"type":"direct"}}],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":670,"cache_read_input_tokens":163218,"cache_creation":{"ephemeral_5m_input_tokens":670,"ephemeral_1h_input_tokens":0},"output_tokens":3,"service_tier":"standard","inference_geo":"global"},"context_management":null},"parent_tool_use_id":null,"session_id":"1afe11b5-10dc-4e1d-94e5-f841435d41d2","uuid":"70ee0705-29ef-4987-9abc-6f7a79c377f1","timestamp":"2026-08-18T01:18:47.058Z","request_id":"req_011Ce9LH9ChiEcZSJYec7uDm"}
|
|
agent
Process completed with exit code 1.\n##[error]❌ Failed to parse detection result: No THREAT_DETECTION_RESULT found in detection log.\n```\n`audit-diff` between the two failed runs shows no meaningful drift (only routine MCP tool-discovery noise) — this is a persistent bug, not a regression from a specific commit. Three workflows currently declare `engine: goose`: `daily-team-evolution-insights.md`, `daily-token-consumption-report.md`, `smoke-goose.md` — all three are structurally exposed since the detection step's firewall config is shared, not per-workflow.\n\n**Cluster B — Design Decision Gate invocation-cap exhaustion**\n\nThe main agent spawned a sub-agent with explicit instructions: *\"Use `push_to_pull_request_branch`... then use `add_comment`.\"* The sub-agent instead tried, in order: `git push` (denied, needs approval), `gh pr comment` (denied), `cat /tmp/adr-comment.md` (missing file), `ls /tmp` (blocked — outside allowed working dir), more file-path guesses, a GraphQL call (403 `access denied: unrecognized GraphQL operation`), and repeated `git push`/`gh` attempts — 21 tool calls total before hitting `API Error: Request rejected (429) · Maximum LLM invocations exceeded (30/30)`, which terminated the parent Claude Code CLI process with exit code 1. The workflow's `bash:` allowlist (`.github/workflows/design-decision-gate.md:70-80`) only permits read-only git/cat/grep/ls/wc/find/echo — it was never going to let raw `git push`/`gh pr comment` through, so the sub-agent's retry loop was guaranteed to fail before it started.\n\n**Cluster C — BYOK Ollama 503**\n\n4 consecutive Copilot-harness attempts (~33-38s each) all failed with `Failed to get response from the AI model; retried 5 times ... Last error: 503 Service Unavailable` against `api-proxy:10002/v1/models`, targeting Ollama at `host.docker.internal:11434`. No firewall blocks, no auth errors — looks like the Ollama service wasn't listening yet when the agent started.\n\n</details>\n\n### Sub-issues created\n\n- P0: Goose threat-detection firewall allowlist (this issue's #aw_p0goose sub-issue)\n- P1: Design Decision Gate invocation-cap exhaustion (this issue's #aw_p1ddgate sub-issue)\n\n**References:**\n- [§32064592237](https://github.com/github/gh-aw/actions/runs/32064592237)\n- [§32069307664](https://github.com/github/gh-aw/actions/runs/32069307664)\n- [§32076040028](https://github.com/github/gh-aw/actions/runs/32076040028)\nEOF\nwc -l /tmp/gh-aw/agent/parent_report.md","description":"Write parent report issue body to file"},"caller":{"type":"direct"}}],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":1108,"cache_read_input_tokens":159387,"cache_creation":{"ephemeral_5m_input_tokens":1108,"ephemeral_1h_input_tokens":0},"output_tokens":3,"service_tier":"standard","inference_geo":"global"},"context_management":null},"parent_tool_use_id":null,"session_id":"1afe11b5-10dc-4e1d-94e5-f841435d41d2","uuid":"7dc07e04-e087-4b4a-8e84-74c12510bc6f","timestamp":"2026-08-18T01:18:27.260Z","request_id":"req_011Ce9LEqnQDzEzWwMjuGxq1"}
|
|
agent
Process completed with exit code 1.\n1159:2026-08-16T20:18:32.7362307Z curl: (22) The requested URL returned error: 403\n1160:2026-08-16T20:18:32.7362693Z [goose-harness] Goose download failed\n1162:2026-08-16T20:18:32.7363565Z [WARN] - Blocked: release-assets.githubusercontent.com:443 (domain not in allowlist)\n1325:2026-08-16T20:18:34.1700895Z ##[error]ERR_PARSE: ❌ No THREAT_DETECTION_RESULT found in detection log. The detection model may have failed to follow the output format.","is_error":false}]},"parent_tool_use_id":null,"session_id":"1afe11b5-10dc-4e1d-94e5-f841435d41d2","uuid":"b33481e4-703e-4fb7-869b-172077ef240f","timestamp":"2026-08-18T01:17:34.511Z","tool_use_result":{"stdout":"627:2026-08-16T20:18:18.4966125Z \u001b[36;1m fail(spawnSync(\"curl\", [\"--fail\", \"--location\", \"--silent\", \"--show-error\", \"--output\", archive, releaseURL], { stdio: \"inherit\" }), \"Goose download\");\u001b[0m\n629:2026-08-16T20:18:18.4967893Z \u001b[36;1m throw new Error(\"Goose download checksum did not match\");\u001b[0m\n981:2026-08-16T20:18:30.3482428Z curl: (22) The requested URL returned error: 403\n982:2026-08-16T20:18:30.3528840Z [goose-harness] Goose download failed\n984:2026-08-16T20:18:30.6212207Z [WARN] - Blocked: release-assets.githubusercontent.com:443 (domain not in allowlist)\n1017:2026-08-16T20:18:32.6042872Z ##[error]Process completed with exit code 1.\n1159:2026-08-16T20:18:32.7362307Z curl: (22) The requested URL returned error: 403\n1160:2026-08-16T20:18:32.7362693Z [goose-harness] Goose download failed\n1162:2026-08-16T20:18:32.7363565Z [WARN] - Blocked: release-assets.githubusercontent.com:443 (domain not in allowlist)\n1325:2026-08-16T20:18:34.1700895Z ##[error]ERR_PARSE: ❌ No THREAT_DETECTION_RESULT found in detection log. The detection model may have failed to follow the output format.","stderr":"","interrupted":false,"isImage":false,"noOutputExpected":false}}
|
|
agent
Process completed with exit code 1.\\n##[error]❌ Failed to parse detection result: No THREAT_DETECTION_RESULT found in detection log. The detection model may have failed to follow the output format.\\n##[error]ERR_PARSE: ❌ No THREAT_DETECTION_RESULT found in detection log. The detection model may have failed to follow the output format.\",\n \"file\": \"detection\"\n },\n--\n \"errors\": 0,\n \"zero_touch\": 0,\n \"acceptance_rate\": 0,\n \"waste_rate\": 0,\n \"zero_touch_rate\": 0,\n \"domain_breakdowns\": [","is_error":false}]},"parent_tool_use_id":null,"session_id":"1afe11b5-10dc-4e1d-94e5-f841435d41d2","uuid":"03a31b36-d797-4e11-81f7-c2f00091b78c","timestamp":"2026-08-18T01:16:35.613Z","tool_use_result":{"stdout":" \"release-assets.githubusercontent.com:443\"\n ],\n \"total_requests\": 42,\n \"allowed_requests\": 42,\n \"blocked_requests\": 0,\n \"requests_by_domain\": {\n--\n \"release-assets.githubusercontent.com:443\": {\n \"allowed\": 1,\n \"blocked\": 0\n }\n }\n },\n--\n \"errors\": [\n {\n \"type\": \"step_failure\",\n \"message\": \"##[error]Process completed with exit code 1.\\n##[error]❌ Failed to parse detection result: No THREAT_DETECTION_RESULT found in detection log. The detection model may have failed to follow the output format.\\n##[error]ERR_PARSE: ❌ No THREAT_DETECTION_RESULT found in detection log. The detection model may have failed to follow the output format.\",\n \"file\": \"detection\"\n },\n--\n \"errors\": 0,\n \"zero_touch\": 0,\n \"acceptance_rate\": 0,\n \"waste_rate\": 0,\n \"zero_touch_rate\": 0,\n \"domain_breakdowns\": [","stderr":"","interrupted":false,"isImage":false,"noOutputExpected":false}}
|
|
agent
Process completed with exit code 35.\\n...\\n/home/runner/work/_temp/....sh: line 43: awf: command not found\\n##[error]Process completed with exit code 127.\\n```\\n\\nCascade: `Install AWF binary` fails → later evals steps are skipped, including `Upload evals results` → the downstream `conclusion` job then fails with `Unable to download artifact(s): Artifact not found for name: evals`, which is what the CI status surfaces as the \\\"real\\\" error even though the true cause is upstream.\\n</details>\\n\\n**Probable root cause:** `install_awf_binary.sh` has no retry/backoff around the `curl` download of `checksums.txt` from GitHub Releases, so a single transient TLS reset takes down the whole evals job and only surfaces downstream as a confusing artifact-not-found error.\\n\\n**Proposed remediation:** add `curl --retry 3 --retry-delay 2 --retry-connrefused` (or equivalent) to the checksum/binary download in `install_awf_binary.sh`. Also make the downstream artifact-download step tolerate a missing `evals` artifact with a clear \\\"evals skipped upstream\\\" message instead of a raw `Unable to download artifact(s)` error.\\n\\n**Success criteria:** the install step retries through a transient network blip instead of failing the whole job; if retries are exhausted, the top-level failure names the real cause (network) instead of only showing a missing-artifact error three jobs downstream.\\n\\n---\\n\\n### Fix the MCP Gateway config (schema or generator) to accept digest-pinned container images — `serena`'s `@sha256:...` reference fails validation and blocks the whole run.\\n\\nPackage Specification Librarian failed at `Start MCP Gateway` in [§31949448689](https://github.com/github/gh-aw/actions/runs/31949448689) (2026-08-16 13:23 UTC), before the agent ever ran.\\n\\n<details>\\n<summary>Raw failure evidence</summary>\\n\\n```\\nConfiguration validation error (MCP Gateway version: v0.4.9):\\nError: jsonschema validation failed with '.../mcp-gateway-config.schema.json#'\\n Error: 'oneOf' failed, none matched\\n Error: 'ghcr.io/oraios/serena:latest@sha256:[REDACTED]' does not match pattern '^[a-zA-Z0-9][a-zA-Z0-9./_-]*(:([a-zA-Z0-9._-]+|latest))?$'\\n Error: missing property 'url'\\n Error: value must be 'http'\\n Error: additional properties 'entrypointArgs', 'mounts', 'args', 'container', 'entrypoint' not allowed\\nERROR: Gateway failed to become ready\\n```\\n</details>\\n\\n**Probable root cause:** the committed lock file's `\\\"serena\\\"` MCP server config uses a plain tag (`ghcr.io/oraios/serena:latest`), but at runtime the gateway config is resolved/re-pinned to include a `@sha256:...` digest suffix — and the schema's container-image regex pattern rejects any `@sha256:` suffix, only bare tags pass. This is a **shared-component bug**: both Linter Miner and Package Specification Librarian import the same `shared/mcp/serena.md`, so every workflow using this shared Serena MCP config is at risk of hitting the same schema rejection.\\n\\n**Proposed remediation:** either (a) loosen `mcp-gateway-config.schema.json`'s container-image pattern to accept `@sha256:<64-hex>` digest suffixes, or (b) stop injecting a resolved digest into the `container` field for stdio-type MCP servers at gateway-config-build time, keeping the plain tag the lock file already specifies.\\n\\n**Success criteria:** Package Specification Librarian (and any other workflow importing `shared/mcp/serena.md`) starts the MCP Gateway with `serena` registered successfully, with no `jsonschema validation failed` error, across the next 3 scheduled runs.\\n\\n**References:**\\n- [§31957063541](https://github.com/github/gh-aw/actions/runs/31957063541)\\n- [§31949448689](https://github.com/github/gh-aw/actions/runs/31949448689)\\nRelated to #53129\\n\\n\\n<!-- gh-aw-tracker-id: aw-failure-investigator -->\\n\\n> Generated by [🔍 [aw] Failure Investigator (6h)](https://github.com/github/gh-aw/actions/runs/31965944876) · agent · 210.1 AIC · ⌖ 7 AIC · ⊞ 5.6K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+is%3Aissue+%22gh-a
|
|
agent
Process completed with exit code 1.\n2026-08-17T20:22:27.5398804Z ##[group]Run if [ -s /tmp/gh-aw/step-summary.md ]; then\n2026-08-17T20:22:27.5399482Z \u001b[36;1mif [ -s /tmp/gh-aw/step-summary.md ]; then\u001b[0m","is_error":false}]},"parent_tool_use_id":null,"session_id":"1afe11b5-10dc-4e1d-94e5-f841435d41d2","uuid":"1ecc9b17-2722-4aa9-ab46-33a0df6ee50c","timestamp":"2026-08-18T01:14:04.579Z","tool_use_result":{"stdout":"2026-08-17T20:22:21.3769773Z [entrypoint] Network information:\n2026-08-17T20:22:21.3825807Z [entrypoint] IP address: 172.30.0.20 \n2026-08-17T20:22:21.3838196Z [entrypoint] Hostname: a0097c355c0d\n2026-08-17T20:22:21.5941975Z [entrypoint] ==================================\n2026-08-17T20:22:21.5942817Z [entrypoint] Chroot mode enabled - dropping CAP_SYS_CHROOT and CAP_SYS_ADMIN\n2026-08-17T20:22:21.5978013Z [entrypoint] Switching to awfuser (UID: 1001, GID: 1001)\n2026-08-17T20:22:21.5987621Z [entrypoint] Executing command: /bin/bash -c /bin/bash -c 'set +o histexpand; export no_proxy=\"${NO_PROXY:-}\" && : \"${RUNNER_TOOL_CACHE:?RUNNER_TOOL_CACHE must be set}\"; GH_AW_TOOL_CACHE=\"$RUNNER_TOOL_CACHE\"; export PATH=\"$(find \"$GH_AW_TOOL_CACHE\" -maxdepth 5 -type d -name bin 2>/dev/null | tr '\\''\\n'\\'' '\\'':'\\'')$PATH\"; [ -n \"$GOROOT\" ] && export PATH=\"$GOROOT/bin:$PATH\" || true; [ -n \"$ERLANG_HOME\" ] && export PATH=\"$ERLANG_HOME/bin:$PATH\" || true && GH_AW_NODE_EXEC=\"${GH_AW_NODE_BIN:-}\"; if [ -z \"$GH_AW_NODE_EXEC\" ] || [ ! -x \"$GH_AW_NODE_EXEC\" ]; then GH_AW_NODE_EXEC=\"$(command -v node 2>/dev/null || true)\"; fi; if [ -z \"$GH_AW_NODE_EXEC\" ]; then echo \"node runtime missing on this runner — check runtimes.node in workflow YAML\" >&2; exit 127; fi; GH_AW_NPM_GLOBAL_ROOT=\"$(npm root -g 2>/dev/null || true)\"; if [ -n \"$GH_AW_NPM_GLOBAL_ROOT\" ]; then export NODE_PATH=\"${GH_AW_NPM_GLOBAL_ROOT}${NODE_PATH:+:${NODE_PATH}}\"; fi; \"$GH_AW_NODE_EXEC\" ${RUNNER_TEMP}/gh-aw/actions/goose_harness.cjs goose'\n2026-08-17T20:22:21.5990664Z \n2026-08-17T20:22:21.5990983Z [entrypoint] Chroot mode: running command inside host filesystem (/host)\n2026-08-17T20:22:21.6365719Z [entrypoint] Mounted procfs at /host/proc (nosuid,nodev,noexec,hidepid=2)\n2026-08-17T20:22:21.6464936Z [entrypoint] Bind-mounted container cgroup subtree at /host/sys/fs/cgroup (ro)\n2026-08-17T20:22:21.6608929Z [entrypoint] One-shot token library copied to chroot at /tmp/awf-lib/one-shot-token.so\n2026-08-17T20:22:21.6621865Z [entrypoint][WARN] /usr/local/bin/copilot is missing and 'copilot' could not be resolved from PATH/GITHUB_PATH\n2026-08-17T20:22:21.6627945Z [entrypoint] System CA bundle found at /etc/ssl/certs/ca-certificates.crt (already accessible in chroot)\n2026-08-17T20:22:21.6644564Z [entrypoint] DNS configuration created in chroot (/host/etc/resolv.conf)\n2026-08-17T20:22:21.6674994Z [entrypoint] Running as host user: runner (UID: 1001)\n2026-08-17T20:22:21.6676691Z [entrypoint] Chroot working directory: /home/runner/work/gh-aw/gh-aw\n2026-08-17T20:22:21.6679796Z [entrypoint] Using host PATH for chroot\n2026-08-17T20:22:21.6692065Z [entrypoint] Adding JAVA_HOME/bin to PATH: /usr/lib/jvm/temurin-17-jdk-amd64/bin\n2026-08-17T20:22:21.6775009Z [entrypoint] DNS configuration will be removed on exit\n2026-08-17T20:22:21.6792587Z [entrypoint] Transferred /host/tmp/gh-aw ownership to chroot user (1001:1001)\n2026-08-17T20:22:21.6977942Z [entrypoint] Node.js available on PATH: /opt/hostedtoolcache/node/24.19.0/x64/bin/node\n2026-08-17T20:22:22.7041534Z [entrypoint] Unsetting sensitive tokens from parent shell environment...\n2026-08-17T20:22:22.7042386Z [entrypoint] Unset COPILOT_GITHUB_TOKEN from /proc/1/environ\n2026-08-17T20:22:22.7043030Z [entrypoint] Unset OPENAI_API_KEY from /proc/1/environ\n2026-08-17T20:22:22.7043668Z [entrypoint] Unset CODEX_API_KEY from /proc/1/environ\n2026-08-17T20:22:22.7044317Z [entrypoint] Unset COPILOT_PROVIDER_API_KEY from /proc/1/environ\n2026-08-17T20:22:25.2907137Z curl: (22) The requested URL returned error: 403\n2026-08-17T20:22:25.2956546Z [goose-harness] Goose d
|
|
agent
Process completed with exit code 1.\n52\t2850:2026-08-17T22:38:32.8233194Z ##[group]Run node \"${RUNNER_TEMP}/gh-aw/actions/detect_agent_errors.cjs\"\n53\t2851:2026-08-17T22:38:32.8233713Z \u001b[36;1mnode \"${RUNNER_TEMP}/gh-aw/actions/detect_agent_errors.cjs\"\u001b[0m\n54\t2866:2026-08-17T22:38:32.8277196Z GH_AW_AGENTIC_EXECUTION_OUTCOME: failure\n55\t2867:2026-08-17T22:38:32.8277509Z GH_AW_ENGINE_STEP_TIMEOUT_MINUTES: 20\n56\t3252:2026-08-17T22:38:35.0185286Z [2026-08-17T22:34:32.117Z] [DEBUG] [config:validation_shared] Validating startup timeout\n57\t3253:2026-08-17T22:38:35.0186053Z [2026-08-17T22:34:32.117Z] [DEBUG] [config:validation_shared] Validating positive timeout: field=startupTimeout, value=120, jsonPath=gateway.startupTimeout\n58\t3260:2026-08-17T22:38:35.0191547Z [2026-08-17T22:34:32.120Z] [DEBUG] [config:config] MCP_GATEWAY_TOOL_TIMEOUT not set in environment\n59\t3261:2026-08-17T22:38:35.0192210Z [2026-08-17T22:34:32.120Z] [DEBUG] [config:config_stdin] Gateway configured: port=8080, toolTimeout=60, startupTimeout=120\n60\t3348:2026-08-17T22:38:35.0275594Z [2026-08-17T22:34:36.864Z] [DEBUG] [config:validation_env] Validation complete: valid=true, errors=0, warnings=0\n61\t3376:2026-08-17T22:38:35.0315798Z [2026-08-17T22:34:36.878Z] [DEBUG] [config:config] Resolved gateway config after defaulting: port=8080, startupTimeout=120, toolTimeout=60, keepaliveInterval=1500\n62\t3380:2026-08-17T22:38:35.0318767Z [2026-08-17T22:34:36.879Z] [DEBUG] [launcher:launcher] Using startup timeout: 2m0s\n63\t3381:2026-08-17T22:38:35.0319824Z [2026-08-17T22:34:36.879Z] [DEBUG] [launcher:pool] Creating new session connection pool: idleTimeout=6h0m0s, cleanupInterval=5m0s, maxErrors=10\n64\t3526:2026-08-17T22:38:35.0432973Z [2026-08-17T22:34:37.899Z] [DEBUG] [launcher:launcher] Starting server with timeout: serverID=safeoutputs, sessionID=, timeout=2m0s\n65\t3580:2026-08-17T22:38:35.0508212Z [2026-08-17T22:34:38.143Z] [INFO] [backend] [safeoutputs] [stderr] [2026-08-17T22:34:38.133Z] [safeoutputs] send: {\"jsonrpc\":\"2.0\",\"id\":1,\"error\":{\"code\":-32601,\"message\":\"Method not found: server/discover\"}}\n66\t3583:2026-08-17T22:38:35.0512245Z [2026-08-17T22:34:38.144Z] [DEBUG] [launcher:launcher] Starting server with timeout: serverID=github, sessionID=, timeout=2m0s\n67\t3604:2026-08-17T22:38:35.0532127Z [2026-08-17T22:34:38.155Z] [DEBUG] [mcp:connection_logging] Inbound RPC response: serverID=safeoutputs, payloadBytes=8948, hasError=false\n68\t3606:2026-08-17T22:38:35.0566056Z [2026-08-17T22:34:38.160Z] [DEBUG] [rpc] safeoutputs←resp 8948b {\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{\"ttlMs\":0,\"cacheScope\":\"\",\"tools\":[{\"description\":\"WRITE-ONCE: do NOT call this tool with empty or placeholder arguments to probe or discover its schema — required fields (title, body) are listed in this schema; if you are not ready to open the real issue, call `noop` instead. Creates a new GitHub issue for tracking bugs, feature requests, or tasks. Use this for actionable work items that need assignment, labeling, and status tracking. For reports, announcements, or status updates that don't require task tracking, use create_discussion instead. Compatibility: labels may be passed as either an array of strings or a comma-separated string; string input is split, trimmed, and normalized to an array. CONSTRAINTS: Maximum 1 issue(s) can be created. Labels [\\\"automation\\\" \\\"testing\\\"] will be automatically added. This tool records a real issue intent. Do not use it for placeholder titles/bodies, auth checks, or probing. Call it only when the final issue title/body are ready; otherwise use noop or report_incomplete.\",\"inputSchema\":{\"$defs\":{\"structured_data\":{\"additionalProperties\":true,\"description\":\"Optional structured data to carry machine-readable context through sanitization-safe channels. When provided, this object is preserved and appended to the body as fenced JSON.\",\"type\":\"object\"}},\"additionalProperties\":false,\"properties\":{\"body\":{\"description\":\"Detailed issue description in Markdown
|
|
agent
Process completed with exit code 1.\n2850:2026-08-17T22:38:32.8233194Z ##[group]Run node \"${RUNNER_TEMP}/gh-aw/actions/detect_agent_errors.cjs\"\n2851:2026-08-17T22:38:32.8233713Z \u001b[36;1mnode \"${RUNNER_TEMP}/gh-aw/actions/detect_agent_errors.cjs\"\u001b[0m\n2866:2026-08-17T22:38:32.8277196Z GH_AW_AGENTIC_EXECUTION_OUTCOME: failure\n2867:2026-08-17T22:38:32.8277509Z GH_AW_ENGINE_STEP_TIMEOUT_MINUTES: 20\n3252:2026-08-17T22:38:35.0185286Z [2026-08-17T22:34:32.117Z] [DEBUG] [config:validation_shared] Validating startup timeout\n3253:2026-08-17T22:38:35.0186053Z [2026-08-17T22:34:32.117Z] [DEBUG] [config:validation_shared] Validating positive timeout: field=startupTimeout, value=120, jsonPath=gateway.startupTimeout\n3260:2026-08-17T22:38:35.0191547Z [2026-08-17T22:34:32.120Z] [DEBUG] [config:config] MCP_GATEWAY_TOOL_TIMEOUT not set in environment\n3261:2026-08-17T22:38:35.0192210Z [2026-08-17T22:34:32.120Z] [DEBUG] [config:config_stdin] Gateway configured: port=8080, toolTimeout=60, startupTimeout=120\n3348:2026-08-17T22:38:35.0275594Z [2026-08-17T22:34:36.864Z] [DEBUG] [config:validation_env] Validation complete: valid=true, errors=0, warnings=0\n3376:2026-08-17T22:38:35.0315798Z [2026-08-17T22:34:36.878Z] [DEBUG] [config:config] Resolved gateway config after defaulting: port=8080, startupTimeout=120, toolTimeout=60, keepaliveInterval=1500\n3380:2026-08-17T22:38:35.0318767Z [2026-08-17T22:34:36.879Z] [DEBUG] [launcher:launcher] Using startup timeout: 2m0s\n3381:2026-08-17T22:38:35.0319824Z [2026-08-17T22:34:36.879Z] [DEBUG] [launcher:pool] Creating new session connection pool: idleTimeout=6h0m0s, cleanupInterval=5m0s, maxErrors=10\n3526:2026-08-17T22:38:35.0432973Z [2026-08-17T22:34:37.899Z] [DEBUG] [launcher:launcher] Starting server with timeout: serverID=safeoutputs, sessionID=, timeout=2m0s\n3580:2026-08-17T22:38:35.0508212Z [2026-08-17T22:34:38.143Z] [INFO] [backend] [safeoutputs] [stderr] [2026-08-17T22:34:38.133Z] [safeoutputs] send: {\"jsonrpc\":\"2.0\",\"id\":1,\"error\":{\"code\":-32601,\"message\":\"Method not found: server/discover\"}}\n3583:2026-08-17T22:38:35.0512245Z [2026-08-17T22:34:38.144Z] [DEBUG] [launcher:launcher] Starting server with timeout: serverID=github, sessionID=, timeout=2m0s\n3604:2026-08-17T22:38:35.0532127Z [2026-08-17T22:34:38.155Z] [DEBUG] [mcp:connection_logging] Inbound RPC response: serverID=safeoutputs, payloadBytes=8948, hasError=false\n3606:2026-08-17T22:38:35.0566056Z [2026-08-17T22:34:38.160Z] [DEBUG] [rpc] safeoutputs←resp 8948b {\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{\"ttlMs\":0,\"cacheScope\":\"\",\"tools\":[{\"description\":\"WRITE-ONCE: do NOT call this tool with empty or placeholder arguments to probe or discover its schema — required fields (title, body) are listed in this schema; if you are not ready to open the real issue, call `noop` instead. Creates a new GitHub issue for tracking bugs, feature requests, or tasks. Use this for actionable work items that need assignment, labeling, and status tracking. For reports, announcements, or status updates that don't require task tracking, use create_discussion instead. Compatibility: labels may be passed as either an array of strings or a comma-separated string; string input is split, trimmed, and normalized to an array. CONSTRAINTS: Maximum 1 issue(s) can be created. Labels [\\\"automation\\\" \\\"testing\\\"] will be automatically added. This tool records a real issue intent. Do not use it for placeholder titles/bodies, auth checks, or probing. Call it only when the final issue title/body are ready; otherwise use noop or report_incomplete.\",\"inputSchema\":{\"$defs\":{\"structured_data\":{\"additionalProperties\":true,\"description\":\"Optional structured data to carry machine-readable context through sanitization-safe channels. When provided, this object is preserved and appended to the body as fenced JSON.\",\"type\":\"object\"}},\"additionalProperties\":false,\"properties\":{\"body\":{\"description\":\"Detailed issue description in Markdown. Must be the final intended body — not a placeholder or test valu
|
|
agent
Process completed with exit code 1.\",\n \"file\": \"agent\"\n }\n ]\n}\n\n[info] [agenticworkflows] Tool output: 39368 chars","is_error":false}]},"parent_tool_use_id":null,"session_id":"1afe11b5-10dc-4e1d-94e5-f841435d41d2","uuid":"eb8e5d74-af88-49d2-a81b-a3324a11bc5c","timestamp":"2026-08-18T01:13:10.978Z","tool_use_result":{"stdout":"description\": \"JSON Lines data file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/sandbox/firewall/logs/audit.jsonl\",\n \"size\": 37292,\n \"description\": \"JSON Lines data file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/sandbox/firewall/logs/cache.log\",\n \"size\": 2763,\n \"description\": \"Log file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/usage/activity/summary.json\",\n \"size\": 698,\n \"description\": \"JSON data file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/usage/agent/token_usage.jsonl\",\n \"size\": 0,\n \"description\": \"JSON Lines data file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/usage/detection/token_usage.jsonl\",\n \"size\": 0,\n \"description\": \"JSON Lines data file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/usage/github_rate_limits.jsonl\",\n \"size\": 11912,\n \"description\": \"JSON Lines data file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/workflow-logs/0_conclusion.txt\",\n \"size\": 61572,\n \"description\": \"Text file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/workflow-logs/1_safe_outputs.txt\",\n \"size\": 48116,\n \"description\": \"Text file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/workflow-logs/2_detection.txt\",\n \"size\": 129421,\n \"description\": \"Text file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/workflow-logs/3_agent.txt\",\n \"size\": 528089,\n \"description\": \"Text file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/workflow-logs/4_activation.txt\",\n \"size\": 92037,\n \"description\": \"Text file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/workflow-logs/activation/system.txt\",\n \"size\": 589,\n \"description\": \"Text file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/workflow-logs/agent/system.txt\",\n \"size\": 828,\n \"description\": \"Text file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/workflow-logs/conclusion/system.txt\",\n \"size\": 1355,\n \"description\": \"Text file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/workflow-logs/detection/system.txt\",\n \"size\": 702,\n \"description\": \"Text file\"\n },\n {\n \"path\": \"/tmp/gh-aw/aw-mcp/logs/run-32076040028/workflow-logs/safe_outputs/system.txt\",\n \"size\": 780,\n \"description\": \"Text file\"\n }\n ],\n \"github_rate_limit_usage\": {\n \"total_requests_made\": 14,\n \"core_consumed\": 33,\n \"core_consumed_source\": \"response_headers_delta\",\n \"core_remaining\": 14954,\n \"core_limit\": 15000,\n \"resources\": [\n {\n \"resource\": \"core\",\n \"requests_made\": 14,\n \"quota_consumed\": 33,\n \"final_remaining\": 14954,\n \"limit\": 15000\n },\n {\n \"resource\": \"audit_log_streaming\",\n \"requests_made\": 0,\n \"quota_consumed\": 0,\n \"final_remaining\": 15,\n \"limit\": 15\n },\n {\n \"resource\": \"search\",\n \"requests_made\": 0,\n \"quota_consumed\": 0,\n \"final_remaining\": 30,\n \"limit\": 30\n },\n {\n \"resource\": \"enterprise_token_inventory\",\n \"requests_made\": 0,\n \"quota_consumed\": 0,\n \"final_remaining\": 1750,\
|
|
evals
Process completed with exit code 1.\n```\nThis cascades into the conclusion job:\n```\n##[error]❌ Failed to parse detection result: No THREAT_DETECTION_RESULT found in detection log. The detection model may have failed to follow the output format.\n```\n`audit-diff` between the two failed runs shows no meaningful drift (routine MCP tool-discovery noise only) — this is a stable, reproducing bug, not a regression tied to a specific commit.\n\n### Root cause\n\n`pkg/workflow/data/ecosystem_domains.json:259-269` defines the `\"threat-detection\"` ecosystem domain list used by `GetThreatDetectionAllowedDomains` (`pkg/workflow/domains.go:746`):\n```json\n\"threat-detection\": [\n \"api.business.githubcopilot.com\",\n \"api.enterprise.githubcopilot.com\",\n \"api.github.com\",\n \"api.githubcopilot.com\",\n \"api.individual.githubcopilot.com\",\n \"github.com\",\n \"host.docker.internal\",\n \"registry.npmjs.org\",\n \"telemetry.enterprise.githubcopilot.com\"\n]\n```\nThe comment above `GetThreatDetectionAllowedDomains` (`domains.go:738-744`) explains the list intentionally excludes `raw.githubusercontent.com` because \"the CLI binary is pre-installed\" — true for Copilot-based detection, but **not** true for the Goose engine: `goose_harness.cjs` downloads the Goose release binary at runtime via a `github.com/.../releases/download/...` URL, which GitHub redirects to `release-assets.githubusercontent.com`. That domain is missing from the list, so the redirect gets firewalled and the download 403s before Goose ever runs, so detection never emits `THREAT_DETECTION_RESULT`.\n\n### Proposed remediation\n\nAdd `\"release-assets.githubusercontent.com\"` to the `\"threat-detection\"` array in `pkg/workflow/data/ecosystem_domains.json`. Recompile all `.lock.yml` files that embed this ecosystem list. Consider also updating the comment at `domains.go:738-744` to note the Goose-engine exception.\n\n### Success criteria\n\n- Daily Team Evolution Insights, Daily Token Consumption Report, and Smoke Goose all complete their detection step with a valid `THREAT_DETECTION_RESULT` across the next 3 scheduled runs each, with no `Goose download failed` / `release-assets.githubusercontent.com` block in the logs.\n","labels":["bug","automation","agentic-workflows"],"parent":"aw_parent1","title":"[P0] Goose engine detection step blocked by missing release-assets.githubusercontent.com in firewall allowlist"}}}
|
|
evals
Process completed with exit code 1.\n##[error]❌ Failed to parse detection result: No THREAT_DETECTION_RESULT found in detection log.\n```\n`audit-diff` between the two failed runs shows no meaningful drift (only routine MCP tool-discovery noise) — this is a persistent bug, not a regression from a specific commit. Three workflows currently declare `engine: goose`: `daily-team-evolution-insights.md`, `daily-token-consumption-report.md`, `smoke-goose.md` — all three are structurally exposed since the detection step's firewall config is shared, not per-workflow.\n\n**Cluster B — Design Decision Gate invocation-cap exhaustion**\n\nThe main agent spawned a sub-agent with explicit instructions: *\"Use `push_to_pull_request_branch`... then use `add_comment`.\"* The sub-agent instead tried, in order: `git push` (denied, needs approval), `gh pr comment` (denied), `cat /tmp/adr-comment.md` (missing file), `ls /tmp` (blocked — outside allowed working dir), more file-path guesses, a GraphQL call (403 `access denied: unrecognized GraphQL operation`), and repeated `git push`/`gh` attempts — 21 tool calls total before hitting `API Error: Request rejected (429) · Maximum LLM invocations exceeded (30/30)`, which terminated the parent Claude Code CLI process with exit code 1. The workflow's `bash:` allowlist (`.github/workflows/design-decision-gate.md:70-80`) only permits read-only git/cat/grep/ls/wc/find/echo — it was never going to let raw `git push`/`gh pr comment` through, so the sub-agent's retry loop was guaranteed to fail before it started.\n\n**Cluster C — BYOK Ollama 503**\n\n4 consecutive Copilot-harness attempts (~33-38s each) all failed with `Failed to get response from the AI model; retried 5 times ... Last error: 503 Service Unavailable` against `api-proxy:10002/v1/models`, targeting Ollama at `host.docker.internal:11434`. No firewall blocks, no auth errors — looks like the Ollama service wasn't listening yet when the agent started.\n\n</details>\n\n### Sub-issues created\n\n- P0: Goose threat-detection firewall allowlist (this issue's #aw_p0goose sub-issue)\n- P1: Design Decision Gate invocation-cap exhaustion (this issue's #aw_p1ddgate sub-issue)\n\n**References:**\n- [§32064592237](https://github.com/github/gh-aw/actions/runs/32064592237)\n- [§32069307664](https://github.com/github/gh-aw/actions/runs/32069307664)\n- [§32076040028](https://github.com/github/gh-aw/actions/runs/32076040028)\n","labels":["bug","automation","agentic-workflows"],"temporary_id":"aw_parent1","title":"[aw] Failure Investigator Report — 2026-08-18 01:15 UTC (6h)"}}}
|
|
evals
Process completed with exit code 1.\",\n \"impact\": \"Workflow did not complete successfully and may need intervention\"\n },\n {\n \"category\": \"optimization\",\n \"severity\": \"low\",\n \"title\": \"Partially Reducible To Deterministic\",\n \"description\": \"About 50% of this run's turns appear to be data-gathering that could move to deterministic steps.\",\n \"impact\": \"Moving data-gathering turns to deterministic steps reduces inference cost\"\n }\n ],\n \"recommendations\": [\n {\n \"priority\": \"high\",\n \"action\": \"Review error logs to identify root cause of failure\",\n \"reason\": \"Understanding failure causes helps prevent recurrence\",\n \"example\": \"Check the errors field for specific error messages, or inspect the log files in logs_path\"\n },\n {\n \"priority\": \"medium\",\n \"action\": \"Move data-fetching work to frontmatter steps: (pre-agent) writing to /tmp/gh-aw/agent/ or post-steps: (post-agent) to reduce inference cost. See the DeterministicOps guide.\",\n \"reason\": \"About 50% of this run's turns appear to be data-gathering that could move to deterministic steps.\"\n }\n ],\n \"observability_insights\": [\n {\n \"category\": \"execution\",\n \"severity\": \"info\",\n \"title\": \"Directed execution path\",\n \"summary\": \"The run remained relatively linear with 4 turns and 0 tool types.\",\n \"evidence\": \"turns=4 tool_types=0\"\n },\n {\n \"category\": \"actuation\",\n \"severity\": \"info\",\n \"title\": \"Read-only posture observed\",\n \"summary\": \"The workflow stayed in an analysis posture and did not emit any GitHub write actions.\",\n \"evidence\": \"created_items=0 safe_items=0\"\n },\n {\n \"category\": \"network\",\n \"severity\": \"info\",\n \"title\": \"Network policy aligned\",\n \"summary\": \"The firewall observed 59 request(s) with 0 blocked, for a 0% block rate.\",\n \"evidence\": \"blocked=0 total=59\"\n },\n {\n \"category\": \"execution\",\n \"severity\": \"info\",\n \"title\": \"Log template patterns mined\",\n \"summary\": \"Analysis identified 2 distinct event templates across 6 pipeline stages from 2 events.\",\n \"evidence\": \"finish=1 plan=1\"\n },\n {\n \"category\": \"reliability\",\n \"severity\": \"low\",\n \"title\": \"1 anomalous event pattern(s) detected\",\n \"summary\": \"Anomaly detection flagged 1 event(s) as unusual based on template similarity and cluster rarity.\",\n \"evidence\": \"stage=plan score=0.65: new log template discovered; rare cluster (few observations)\"\n },\n {\n \"category\": \"execution\",\n \"severity\": \"info\",\n \"title\": \"Agent stage sequence\",\n \"summary\": \"The observed pipeline stage sequence for this run.\",\n \"evidence\": \"plan finish\"\n }\n ],\n \"performance_metrics\": {\n \"network_requests\": 59\n },\n \"engine_config\": {\n \"engine_id\": \"copilot\",\n \"engine_name\": \"GitHub Copilot CLI\",\n \"model\": \"auto\",\n \"version\": \"1.0.80\",\n \"firewall_version\": \"v0.28.1\",\n \"trigger_event\": \"schedule\",\n \"repository\": \"github/gh-aw\"\n },\n \"prompt_analysis\": {\n \"prompt_size\": 10742,\n \"prompt_file\": \"aw-prompts/prompt.txt\"\n },\n \"session_analysis\": {\n \"wall_time\": \"13.5m\",\n \"turn_count\": 4,\n \"avg_turn_duration\": \"3.4m\",\n \"avg_time_between_turns\": \"46.8s\",\n \"max_time_between_turns\": \"54.7s\",\n \"timeout_detected\": false\n },\n \"safe_output_summary\": {\n \"total_items\": 0,\n \"items_by_type\": {},\n \"summary\": \"No items\",\n \"temporary_id_map_status\": \"missing\"\n },\n \"jobs\": [\n {\n \"name\": \"activation\",\n \"status\": \"completed\",\n \"conclusion\": \"success\",\n \"duration\": \"36.0s\",\n \"steps\": [\n {\n \"name\": \"Set up job\",\n \"
|
|
evals
Process completed with exit code 35.\n...\n/home/runner/work/_temp/....sh: line 43: awf: command not found\n##[error]Process completed with exit code 127.\n```\n\nCascade: `Install AWF binary` fails → later evals steps are skipped, including `Upload evals results` → the downstream `conclusion` job then fails with `Unable to download artifact(s): Artifact not found for name: evals`, which is what the CI status surfaces as the \"real\" error even though the true cause is upstream.\n\u003c/details\u003e\n\n**Probable root cause:** `install_awf_binary.sh` has no retry/backoff around the `curl` download of `checksums.txt` from GitHub Releases, so a single transient TLS reset takes down the whole evals job and only surfaces downstream as a confusing artifact-not-found error.\n\n**Proposed remediation:** add `curl --retry 3 --retry-delay 2 --retry-connrefused` (or equivalent) to the checksum/binary download in `install_awf_binary.sh`. Also make the downstream artifact-download step tolerate a missing `evals` artifact with a clear \"evals skipped upstream\" message instead of a raw `Unable to download artifact(s)` error.\n\n**Success criteria:** the install step retries through a transient network blip instead of failing the whole job; if retries are exhausted, the top-level failure names the real cause (network) instead of only showing a missing-artifact error three jobs downstream.\n\n---\n\n### Fix the MCP Gateway config (schema or generator) to accept digest-pinned container images — `serena`'s `@sha256:...` reference fails validation and blocks the whole run.\n\nPackage Specification Librarian failed at `Start MCP Gateway` in [§31949448689](https://github.com/github/gh-aw/actions/runs/31949448689) (2026-08-16 13:23 UTC), before the agent ever ran.\n\n\u003cdetails\u003e\n\u003csummary\u003eRaw failure evidence\u003c/summary\u003e\n\n```\nConfiguration validation error (MCP Gateway version: v0.4.9):\nError: jsonschema validation failed with '.../mcp-gateway-config.schema.json#'\n Error: 'oneOf' failed, none matched\n Error: 'ghcr.io/oraios/serena:latest@sha256:[REDACTED]' does not match pattern '^[a-zA-Z0-9][a-zA-Z0-9./_-]*(:([a-zA-Z0-9._-]+|latest))?$'\n Error: missing property 'url'\n Error: value must be 'http'\n Error: additional properties 'entrypointArgs', 'mounts', 'args', 'container', 'entrypoint' not allowed\nERROR: Gateway failed to become ready\n```\n\u003c/details\u003e\n\n**Probable root cause:** the committed lock file's `\"serena\"` MCP server config uses a plain tag (`ghcr.io/oraios/serena:latest`), but at runtime the gateway config is resolved/re-pinned to include a `@sha256:...` digest suffix — and the schema's container-image regex pattern rejects any `@sha256:` suffix, only bare tags pass. This is a **shared-component bug**: both Linter Miner and Package Specification Librarian import the same `shared/mcp/serena.md`, so every workflow using this shared Serena MCP config is at risk of hitting the same schema rejection.\n\n**Proposed remediation:** either (a) loosen `mcp-gateway-config.schema.json`'s container-image pattern to accept `@sha256:\u003c64-hex\u003e` digest suffixes, or (b) stop injecting a resolved digest into the `container` field for stdio-type MCP servers at gateway-config-build time, keeping the plain tag the lock file already specifies.\n\n**Success criteria:** Package Specification Librarian (and any other workflow importing `shared/mcp/serena.md`) starts the MCP Gateway with `serena` registered successfully, with no `jsonschema validation failed` error, across the next 3 scheduled runs.\n\n**References:**\n- [§31957063541](https://github.com/github/gh-aw/actions/runs/31957063541)\n- [§31949448689](https://github.com/github/gh-aw/actions/runs/31949448689)\nRelated to #53129\n\n\n\u003c!-- gh-aw-tracker-id: aw-failure-investigator --\u003e\n\n\u003e Generated by [🔍 [aw] Failure Investigator (6h)](https://github.com/github/gh-aw/actions/runs/31965944876) · agent · 210.1 AIC · ⌖ 7 AIC · ⊞ 5.6K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+is%3A
|
|
agent
- Line 3: update_issue requires at least one of: 'status', 'title', 'body', 'labels', 'assignees', 'milestone' fields
|
|
agent
Validation errors found:
|
|
agent
Checkout failed (expected for closed PR): The process '/usr/bin/git' failed with exit code 128\n1037:2026-08-17T21:07:37.8385294Z ##[group]Run bash \"${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh\" ghcr.io/github/gh-aw-firewall/agent:0.28.1@sha256:5e3f6ee27eeae07195838b97ac4aa2f8aea42a7c55f1c0d3e17d8e88e294ad0d ghcr.io/github/gh-aw-firewall/api-proxy:0.28.1@sha256:288e7d2a12d5b430500d739f9c16e20bb1ed51b91f986f3f3eccde189f489f5c ghcr.io/github/gh-aw-firewall/cli-proxy:0.28.1@sha256:f931e5e1e13f765605d03ef9511fc755d779a51b76581ea14586e9871506a610 ghcr.io/github/gh-aw-firewall/squid:0.28.1@sha256:9d428af47899bf18ef2d5618075777d76ef344c91e76c1f44ec1aaa0ee347e5f ghcr.io/github/gh-aw-mcpg:v0.4.9@sha256:e5a1569aeaf41820fa7bdee3e94468cae448133cdbf00119ad24f5b74db1ab9f ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196 ghcr.io/github/github-mcp-server:v1.9.0@sha256:881b53d6f75f69bdbc1b5b10fc2f1361717c19054143b3a8529fb5c32061a50e\n1038:2026-08-17T21:07:37.8388281Z \u001b[36;1mbash \"${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh\" ghcr.io/github/gh-aw-firewall/agent:0.28.1@sha256:5e3f6ee27eeae07195838b97ac4aa2f8aea42a7c55f1c0d3e17d8e88e294ad0d ghcr.io/github/gh-aw-firewall/api-proxy:0.28.1@sha256:288e7d2a12d5b430500d739f9c16e20bb1ed51b91f986f3f3eccde189f489f5c ghcr.io/github/gh-aw-firewall/cli-proxy:0.28.1@sha256:f931e5e1e13f765605d03ef9511fc755d779a51b76581ea14586e9871506a610 ghcr.io/github/gh-aw-firewall/squid:0.28.1@sha256:9d428af47899bf18ef2d5618075777d76ef344c91e76c1f44ec1aaa0ee347e5f ghcr.io/github/gh-aw-mcpg:v0.4.9@sha256:e5a1569aeaf41820fa7bdee3e94468cae448133cdbf00119ad24f5b74db1ab9f ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196 ghcr.io/github/github-mcp-server:v1.9.0@sha256:881b53d6f75f69bdbc1b5b10fc2f1361717c19054143b3a8529fb5c32061a50e\u001b[0m\n1078:2026-08-17T21:08:05.9836738Z Attempt 1 of 3: Pulling ghcr.io/github/github-mcp-server:v1.9.0@sha256:881b53d6f75f69bdbc1b5b10fc2f1361717c19054143b3a8529fb5c32061a50e...\n1079:2026-08-17T21:08:07.0499041Z ghcr.io/github/github-mcp-server:v1.9.0@sha256:881b53d6f75f69bdbc1b5b10fc2f1361717c19054143b3a8529fb5c32061a50e\n1080:2026-08-17T21:08:07.0511359Z Successfully pulled ghcr.io/github/github-mcp-server:v1.9.0@sha256:881b53d6f75f69bdbc1b5b10fc2f1361717c19054143b3a8529fb5c32061a50e\n1519:2026-08-17T21:08:16.1998529Z [info] (Note: MCP_GATEWAY_DOMAIN is 'awmg-mcpg' for container access)\n1938:2026-08-17T21:08:59.8287421Z {\"type\":\"system\",\"subtype\":\"thinking_tokens\",\"estimated_tokens\":7,\"estimated_tokens_delta\":7,\"uuid\":\"91d4b24a-62e2-4716-8d3b-bec814e84529\",\"session_id\":\"f828c50a-d146-448a-9a7d-4ad200312ecd\"}\n1953:2026-08-17T21:09:05.1194519Z {\"type\":\"system\",\"subtype\":\"thinking_tokens\",\"estimated_tokens\":258,\"estimated_tokens_delta\":70,\"uuid\":\"5cc217d3-e6b7-4f1e-8cf6-44945295c01a\",\"session_id\":\"f828c50a-d146-448a-9a7d-4ad200312ecd\"}\n2027:2026-08-17T21:10:39.5399665Z {\"type\":\"user\",\"message\":{\"role\":\"user\",\"content\":[{\"type\":\"tool_result\",\"content\":\"This command requires approval\",\"is_error\":true,\"tool_use_id\":\"toolu_01SVbERNFJiesUN6WJEDXjoR\"}]},\"parent_tool_use_id\":\"toolu_014oTVmkWztnY4qBgq5WtxG2\",\"session_id\":\"f828c50a-d146-448a-9a7d-4ad200312ecd\",\"uuid\":\"d03701f7-297c-452c-b37d-8ec88a25ae6d\",\"timestamp\":\"2026-08-17T21:10:39.538Z\",\"tool_use_result\":\"Error: This command requires approval\",\"subagent_type\":\"general-purpose\",\"task_description\":\"Push ADR draft to PR branch and post blocking comment\"}\n2031:2026-08-17T21:10:53.9372778Z {\"type\":\"user\",\"message\":{\"role\":\"user\",\"content\":[{\"type\":\"tool_result\",\"content\":\"Parser did not consume trailing input\",\"is_error\":true,\"tool_use_id\":\"toolu_015BYVkheiArv8f3eEUes31G\"}]},\"parent_tool_use_id\":\"toolu_014oTVmkWztnY4qBgq5WtxG2\",\"session_id\":\"f828c50a-d146-448a-9a7d-4ad200312ecd\",\"uuid\":\"8a09ba41-d614-4810-8c9c-e804e24d90d1\",\"timestamp\":\"2026-08-17T21:10:53.936Z\",\"tool_use_result\":\"Erro
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
activation
Expired
|
9.07 MB |
sha256:7fa5caab282334fac52d9fffca38759d2f67b81ba85c9e017c30e7da688ee3d4
|
|
|
agent
|
540 KB |
sha256:fb5af3da4ef836c36399bd4d7fca6717b4a27a356def42f0d8e0bf0f2521fe93
|
|
|
agent-output-fallback
|
8.22 KB |
sha256:527a4602bc3d8a7c8fde2cb1eaa2e09926766f7b72c127d9877d270a1ae2406f
|
|
|
aic-usage-cache
|
473 Bytes |
sha256:4e071350e2be005df685d2b216ac847074e6139dd78f98a17e8645862fb6b875
|
|
|
awfailureinvestigator-experiment
|
5.33 KB |
sha256:bf5ef6796d72fe699a8000485065ac699ee6f9647376101d0401f6645b620494
|
|
|
detection
|
233 Bytes |
sha256:6ce60505b97485e389f75ae30ad147e315a6bc465be79a4eb377eb663263a6ae
|
|
|
evals
|
384 Bytes |
sha256:81e3731b0c777b908cc3a4b91b346e39fea185c94aa3651b97e6365ddd9f2282
|
|
|
github~gh-aw~L8K6B9.dockerbuild
|
24.8 KB |
sha256:c5dd28067dc89b25befb5519bf341c6e133c5bd552ba64bebccae28c1ce3a5b1
|
|
|
safe-outputs-items
|
815 Bytes |
sha256:67450386d78a2adfcab798b2ec497899c647d8cf28fb24f50a39f7236194315a
|
|
|
usage
|
6.21 KB |
sha256:48f1a88f628bd3d79042de9b93ae3540c513a3db25909ff0f7007e1bd0a1d06a
|
|