Harden research-architect report verification (#78)

## Summary

- fold the third live multi-model research run into `research-architect`
2.3.0 without promoting executor-specific quirks into universal rules
- distinguish citation support, domain applicability, and usable
evidence; classify exports as direct, resolvable, opaque, or absent
- resolve numbered bibliographies, reject blind zero-citation scores,
and preserve dated postmortem and executor-profile evidence

## Test plan

- [x] `.venv/bin/python -m pytest tests/ -q` (760 passed)
- [x] `python3
skills/skill-standardizer/scripts/test_skill_standardizer.py`
- [x] `python3 skills/skill-evals/scripts/validate_skill_contract.py
--skills-root skills --strict`
- [x] `python3 skills/skill-evals/scripts/check_skill_versions.py --base
origin/main`
- [x] generated docs, manifest, harness adapters, and catalog `--check`
commands
- [x] `python3 scripts/slop_scan.py`
- [x] effective-runtime probe against all three supplied research
reports

Author: davisbuilds

docs/catalog/index.html

@@ -204,19 +204,19 @@ to a trigger, or move completed decisions and work to the Roadmap or decision hi
 
 ### research-architect: remaining deferred tooling
 - **What**: `scripts/diff_runs.py` and `references/rubric-library.md` remain
-  deliberately deferred. (`scripts/score_report.py`, the third of the original
-  trio, shipped in 2.2.0 once two real runs justified it.)
-- **Why it matters**: Across two runs there is exactly one confirmed
-  discriminating rubric item (the per-tactic evidence floor, 2026-07-12).
-  Building a rubric library on one data point would encode guesses — the mistake
-  the deferral exists to avoid. `diff_runs.py` only pays off on multi-run plans
-  and depends on M1 section alignment holding in practice.
-- **Next**: Seed `rubric-library.md` once 2–3 more runs identify rubric items
-  that actually discriminate (items that always pass are dead weight and belong
-  in the postmortem, not the library). Build `diff_runs.py` on top of
-  `score_report.py`'s claim/citation extraction rather than duplicating it: align
-  sections by the M1 fixed order, then surface confident specifics appearing in
-  only one report as hallucination candidates.
+  deliberately deferred. (`scripts/score_report.py` shipped in 2.2.0 and gained
+  citation-coverage/applicability scoring in 2.3.0 after the third live run.)
+- **Why it matters**: Across three runs there are now two confirmed
+  discriminating rubric patterns: the per-tactic evidence floor (2026-07-12)
+  and complete benchmark metadata or an unusable verdict (2026-08-22). That is
+  still thin for a reusable library. The third run also proved manual cross-run
+  diffing valuable, but only one of three reports preserved M1's exact section
+  structure and two exports had opaque claim-to-URL linkage.
+- **Next**: Seed `rubric-library.md` after one more cross-domain run identifies
+  a reusable discriminating item. Build `diff_runs.py` only after another
+  multi-run exercise establishes a tolerant alignment strategy for missing,
+  added, and reordered sections; reuse `score_report.py`'s normalized citation
+  coverage instead of assuming every export carries direct URLs.
 
 ### skills-health: many canonical dojo skills aren't installed globally, so they're unmeasurable
 - **What**: As of 2026-07-15, 26 of 57 canonical `skills/` are installed in none

docs/project/BACKLOG.md

@@ -55,6 +55,7 @@ standing membership record._
 | Add research-architect skill | Deep-research prompt engineering from composable skeleton blocks, pluggable execution routing, executor-independent report verification, postmortem memory; stakes-based trigger split with deep-research (bumped to 2.0.0 as its execution backend) |
 | Harden research orchestration from first live run | `research-architect` 2.0.0 formalizes verified multi-run synthesis as stage 9, moves postmortem to stage 10, closes nine stage-seam findings, and hardens prompt linting; `deep-research` 2.1.0 replaces self-declared credibility with conservative URL-host registry scoring and explainable provenance fields |
 | Act on the second live research run | `research-architect` 2.2.0 splits source ranking into reliability vs. edge-relevance × recency (one ranking biased "what works now" questions toward "already arbitraged away"), defaults stages 3/5/8 to fresh subagents, verifies nodes as they land, lints statistics seeded without a retrievable source, and adds `score_report.py` for the stage-8 structural pass — sampling weighted toward quantitative and attribution claims, the dominant failure mode across every executor profiled. Skeleton stayed net-zero on shipped instruction count (3 additions, 3 deletions). `deep-research` 2.2.0 registers on-chain explorers and code hosts, which previously scored as unknown domains. `skill-evals` 1.3.0 exempts append-only run memory from release-version checks |
+| Act on the third live research run | `research-architect` 2.3.0 catches real-but-wrong-domain citations, forbids claims about un-ingested artifacts, makes merge structure checkable, and separates citation support from applicability. `score_report.py` now classifies direct/resolvable/opaque/absent coverage, resolves explicit `[n]` bibliography markers, refuses false hit rates when an export loses claim-to-URL linkage, and reports a usable-citation rate. Dated profiles capture the three external web DR executors without promoting one-run quirks into universal workflow rules |
 | Preserve verified first-party research sources | `deep-research` 2.3.0 adds registry-owned root/subdomain matching for first-party model, coding-harness, protocol, and agent-framework documentation. Relevant verified priority sources no longer disappear solely because an aggregate lexical score falls below threshold; retained exceptions carry an explicit reason and confidence gap, while off-topic, duplicate, lookalike, and over-budget controls remain intact. |
 | Harden spec/plan validation from cross-repo dogfooding | `write-spec` and `write-plan` 2.1.0 require meaningful acceptance magnitudes when trivial output could pass, define behavior-preserving reference oracles on result-deciding edge inputs, and map fixes across the whole defect/property class. Narrow validator advisories flag obvious weak checks without becoming schema gates. `write-plan` now resolves linked specs and modified files from the consumer plan's Git root, with an explicit override and documented no-Git fallback. |
 | Design systems layer | Added `design-md` (Google `@google/design.md` CLI wrapper) and `design-critique` (closed 37-pattern slop catalog with structured findings) plus 5 Refero exemplars; sibling-skills footer convention disambiguates the four-skill design pipeline |

docs/project/ROADMAP.md

@@ -201,7 +201,7 @@
       "name": "research-architect",
       "description": "Engineer high-quality deep-research prompts and orchestrate their execution and verification. Use when the user wants to draft, improve, or critique a research prompt or brief; commission or plan a multi-source or multi-model research run; run research through external deep-research products (Claude/OpenAI/Gemini DR); or verify and score a research report that something else produced. Triggers on \"research prompt\", \"research brief\", \"commission research\", \"plan a research run\", \"verify this report\", \"research architect\". For a direct low-stakes lookup where the user just wants the answer, use deep-research instead.",
       "path": "skills/research-architect",
-      "version": "2.2.2",
+      "version": "2.3.0",
       "triggers": [
         "research prompt",
         "research brief",

skills.json

@@ -1,3 +1,24 @@
+## 2.3.0 - 2026-08-23
+
+Acts on the 2026-08-22 coding-harnesses postmortem (three-run web DR merge).
+
+- **A4 checks domain and population fit** before a benchmark, dataset, or study
+  supports a major claim; real sources imported from adjacent domains no longer
+  pass merely because they exist.
+- **A6 forbids characterizing un-ingested artifacts**, including contents,
+  size, structure, and metadata. Two executors invented contradictory specifics
+  about the same ZIP after admitting they could not open it.
+- **A9 rejects costless rubric escape hatches**, and M1 merge runs put exact
+  section order plus summary/self-report presence into the shipped rubric.
+- **Stage 8 separates source support from applicability** and treats self-
+  reported confidence as candor evidence rather than correctness evidence.
+- **`score_report.py` classifies citation coverage** as direct, resolvable,
+  opaque, or absent; resolves explicit `[n]` bibliography markers; refuses
+  false hit rates on blind exports; and reports support, domain-fit, and
+  usable-citation rates.
+- Add dated profiles for Claude Opus 5, GPT-5.6-sol, and Gemini Flash 3.7 as
+  external web DR executors.
+
 ## 2.2.2 - 2026-08-14
 
 - Anchor runnable script commands to <skill-dir> so they resolve outside a dojo checkout

skills/research-architect/CHANGELOG.md

@@ -2,7 +2,7 @@
 name: research-architect
 description: Engineer high-quality deep-research prompts and orchestrate their execution and verification. Use when the user wants to draft, improve, or critique a research prompt or brief; commission or plan a multi-source or multi-model research run; run research through external deep-research products (Claude/OpenAI/Gemini DR); or verify and score a research report that something else produced. Triggers on "research prompt", "research brief", "commission research", "plan a research run", "verify this report", "research architect". For a direct low-stakes lookup where the user just wants the answer, use deep-research instead.
 skill-type: workflow
-version: 2.2.2
+version: 2.3.0
 triggers:
   - research prompt
   - research brief
@@ -198,28 +198,36 @@ nothing extra, pre-loads the cross-run diff, and can amend a still-pending
 node's prompt while amending is still cheap. Only step 3 genuinely needs all
 reports in hand.
 
-1. **Structural pass (deterministic):** required sections present; summary
-   block present; sample citations and fetch each — is the URL live, and does
-   the page actually support the claim it's attached to? Record hit rate.
+1. **Structural pass (deterministic):** required sections present in exact
+   order; parseable summary and complete A10 self-report (a–c) present; sample
+   citations and fetch each — is the URL live, does it support the exact claim,
+   and does its domain/population fit the report's use? Record support,
+   applicability, and usable-citation rates.
    `scripts/score_report.py` does the mechanical half:
 
    ```bash
    python3 <skill-dir>/scripts/score_report.py worksheet report.md > 08-worksheet.json
    # each sampled check is one claim/citation pair: fetch its url, then fill
-   # "verdicts" with supported / partial / unsupported / unreachable
+   # "verdicts" (supported/partial/unsupported/unreachable) and
+   # "applicability" (fit/adjacent/mismatch)
    python3 <skill-dir>/scripts/score_report.py score 08-worksheet.json
    ```
 
    Checks are per **claim/citation pair**, not per claim — a claim resting on
    three citations is three things to fetch, and one citation refuting it must
    not be hidden by another supporting it.
 
+   If `citation_coverage.status` is `opaque` or `absent`, do not report a hit
+   rate. Obtain an export with retrievable links or map the citations manually;
+   an empty sample from a blind instrument is not a clean result.
+
    The sample is **weighted toward quantitative and source-attribution
    claims**: across every executor profiled so far, mutated numbers and
    mischaracterized findings are the dominant failure mode, and a uniform
    sample under-tests exactly where reports break. The script never fetches —
    deciding whether a page supports its claim is the judgment this stage
    exists for, and a "URL resolves" hit rate would be worse than none.
+   Treat self-reported confidence as evidence of candor, never correctness.
 2. **Rubric pass (judgment):** spawn a fresh critique subagent — one that has
    not seen the drafting stages — to score the report against the shipped
    rubric, item by item, with evidence quotes. Pass/fail per item, not vibes.
@@ -244,8 +252,9 @@ the final synthesis and this stage is skipped.
 
 From the report's self-report (block A10 for external reports; the packet's
 `self_report` field for local `deep-research` runs) plus verification results,
-record: which instructions were followed, ignored, or misread; citation hit
-rate; which rubric items discriminated (items that always pass are dead
+record: which instructions were followed, ignored, or misread; citation support,
+applicability, and usable-citation rates; which rubric items
+discriminated (items that always pass are dead
 weight). Then append durable lessons to two shared files:
 
 - `references/postmortems.md` — dated lessons about the *skeleton and process*
@@ -290,8 +299,9 @@ the report.
   `docs/research/` in the repo the research serves).
 - The primary deliverables: one assembled, linted prompt per executor
   (`04-prompt-<executor>.md`), and after execution a verification verdict
-  (`08-verification.md`) with citation hit rate and per-rubric-item scores; a
-  multi-run plan also produces one decision-ready `09-synthesis.md`.
+  (`08-verification.md`) with citation support, applicability, usable-citation,
+  and per-rubric-item scores; a multi-run plan also produces one decision-ready
+  `09-synthesis.md`.
 - Every report, regardless of executor, ends in the same summary block the
   `deep-research` skill emits (`key_findings` / `citations` /
   `confidence_gaps` / `next_queries`) — the interchange shape stage 8 consumes.
@@ -302,7 +312,7 @@ the report.
   shipping — zero unfilled slots, zero drafting comments, budget respected.
 - Stage 5 ran and deleted (not just added) instructions, or the router
   explicitly waived it as a quick run.
-- Stage 8 verdicts cite evidence: citation hit rate from real fetches,
+- Stage 8 verdicts cite evidence: support/applicability rates from real fetches,
   pass/fail per rubric item with quotes — never vibes.
 - Multi-run stage 9 synthesis contains only stage-8-accepted claims and
   preserves unresolved disagreements.
@@ -323,9 +333,10 @@ the report.
   memory; read at stages 2–5, append at stage 10. Append-only run memory:
   exempt from the repo's skill release-version check, so recording a lesson
   never costs a version bump.
-- `scripts/score_report.py` — stage-8 structural pass: `worksheet` extracts
-  claims and citations and samples what to check; `score` computes the hit rate
-  from your verdicts. Never fetches.
+- `scripts/score_report.py` — stage-8 structural pass: `worksheet` classifies
+  citation coverage, extracts resolvable claim/citation pairs, and samples what
+  to check; `score` computes support, applicability, and usable-citation rates.
+  Never fetches.
 - `evals/golden-questions/` — frozen real-run drafting artifacts used as
   regression seeds. Still deferred until more runs justify them:
   `scripts/diff_runs.py` and `references/rubric-library.md`; until then, diff

skills/research-architect/SKILL.md

@@ -4,6 +4,15 @@ Observed behavior of specific executors, appended at stage 10. Read at stages
 2–4 to calibrate prompts and at stage 3 to seed scout expectations. Date every
 entry — access and behavior drift.
 
+## Index
+
+- [Claude Code terminal harness](#claude-code-terminal-harness-webfetchwebsearch--as-of-2026-07-12)
+- [Sonnet terminal executor](#sonnet-subagent-as-pipelineresearch-executor--as-of-2026-07-12)
+- [Gemini Deep Research](#gemini-deep-research-web--as-of-2026-07-12-one-full-run--verification)
+- [Claude Opus 5 web DR](#claude-opus-5-as-external-web-dr-executor--as-of-2026-08-22)
+- [GPT-5.6-sol web DR](#gpt-56-sol-as-external-web-dr-executor--as-of-2026-08-22)
+- [Gemini Flash 3.7 web DR](#gemini-flash-37-as-external-web-dr-executor--as-of-2026-08-22)
+
 ## Claude Code terminal harness (WebFetch/WebSearch) — as of 2026-07-12
 
 - X/Twitter: 402 on direct fetch. Reddit: hard-blocked on both `www` and
@@ -66,3 +75,42 @@ entry — access and behavior drift.
   base64 image data at the end of the export; strip before downstream
   processing. Citation markers are superscript numbers that survive export as
   bare digits mid-sentence.
+
+## Claude Opus 5 as external web DR executor — as of 2026-08-22
+
+- **Best primary sourcing and A10 candor of the three-run comparison.** It
+  admitted search-budget exhaustion and an unopenable dataset rather than
+  claiming full compliance.
+- **Strongest on operational specifics:** quota-exhaustion behavior, dated
+  incident history, and vendor-conceded negatives.
+- **Observed failures:** artifact-metadata date drift; selective enumeration
+  favoring its own vendor; and Recommendations/Caveats appended beyond a fixed
+  section contract. Restate structural compliance in the rubric for merge runs.
+- Export carried no retrievable claim-to-URL mapping, so the citation worksheet
+  must classify it as opaque and require manual mapping or a better export.
+
+## GPT-5.6-sol as external web DR executor — as of 2026-08-22
+
+- **Most methodologically skeptical of the three.** It downgraded the strongest
+  benchmark with six stated reasons, caught a model mismatch in the compaction
+  probe, and refused unsupported comparative rankings.
+- Parses an attached brief into an extracted-requirements table, providing a
+  useful compliance signal. Numeric confidence was calibrated, and it volunteered
+  the desk-research-only limitation.
+- **Observed failures:** artifact-metadata drift, verbosity, missing top-level
+  sections, and opaque `citeturn...` markers after export. Obtain a link-preserving
+  export or map citations manually before scoring.
+
+## Gemini Flash 3.7 as external web DR executor — as of 2026-08-22
+
+- **Highest surface polish, lowest reliability in this comparison.** All three
+  refuted claims were its: an inverted compaction result, doubled task count, and
+  a real benchmark imported from an unrelated domain with the magnitude changed.
+- Confidence was anti-correlated with accuracy: it rated the most-wrong section
+  High and omitted A10(c).
+- Structural/export failures extended the 2026-07-12 profile: invalid requested
+  summary shape, appended Works cited list, ambiguous bare-digit markers, and a
+  broken base64 image reference.
+- **Routing:** do not use this executor alone for verification-heavy questions.
+  It can serve as an adjudicative run only when Stage 8 independently verifies
+  every claim used downstream.