Skip to content

fix(build): gate full-route patches on demonstrated evidence, widen defer - #2729

Draft
alexeyv wants to merge 1 commit into
mainfrom
fix/full-route-triage-evidence
Draft

fix(build): gate full-route patches on demonstrated evidence, widen defer#2729
alexeyv wants to merge 1 commit into
mainfrom
fix/full-route-triage-evidence

Conversation

@alexeyv

@alexeyv alexeyv commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Problem

The full route's Classify block has the same gap PR #2697 closed for one-shot, in different clothes — and the full route amplifies it.

  • patch is gated on ease alone ("trivially fixable without human input"), so it filters nothing. Every cheap fix qualifies.
  • defer is scoped to provenance ("pre-existing issue not caused by this story"), so a true finding caused by this change but not worth fixing has no lawful exit except shipped code.
  • The edge-case-hunter reviewer ships a guard_snippet with every finding — a pre-packaged, trivially-applicable remedy — with trigger_condition as a description, not a demonstration. Combined with the ease-only gate, every speculative unhandled-path claim arrives dressed as a qualifying patch.

The full route makes this worse than one-shot, not merely equal:

  1. The loop. Up to 5 loopback iterations, each re-running reviewers against re-derived code, and patches are "the only findings that survive loopbacks" — accepted guards accumulate across iterations rather than being re-litigated.
  2. The verification gate can't see it. Over-engineering passes the spec's ## Verification section by construction — added guards and surface fail no check — so the loop's only automated gate never fires on this failure mode. The loop cap detects divergence (spec thrash, a loud failure); guard accretion is monotonic drift that reports convergence on every pass.
  3. The auto route. bmad-build-auto runs this step unattended, so speculative guards ship with no human eyes at all.

The compounding dynamic: a guard for an undemonstrated state is unreachable, so nothing can ever demonstrate its behavior right or wrong — but it is code, and code generates legitimate findings. The next pass finds the guard branch uncovered, or the fallback violating an invariant, and those second-generation findings are honestly about real code. Triage-time filtering only works at generation zero; the seed is the sole cheap intervention point.

Change

Both step-04-review.md files (bmad-build, bmad-build-auto), Classify block only. No new categories, no new mechanism — the disallowed-fix class drains into the escalation lanes the route already has.

  • patch now requires demonstrated substance — a defect that actually occurs, missing coverage for a specific case, or a broken gate or convention — and a smallest fix that is trivial, adds no public surface, and guards no undemonstrated state. A demonstrated finding whose smallest fix fails any condition is not a patch: it routes to intent_gap when the spec does not settle the fix (e.g., new public surface the spec never authorized), otherwise bad_spec. Unlike one-shot, no new HALT is needed — intent_gap is this route's HALT (human loopback), and bad_spec puts the contract change where it is structurally reviewed: in the spec, via the change log and re-derivation.
  • The parent carries the demonstration duty. Reviewers report claims, not demonstrations — by design they see the change without full-codebase context, and reachability is a whole-codebase property, so the filter cannot live in the reviewer prompt. Triage verifies each trigger itself (trace a reachable path, construct the input, run the failing case) and judges findings by their demonstrated trigger, never by the remedy they propose. Without this clause, a strict parent would conclude no reviewer finding ever clears the evidence bar — reviewers never demonstrate anything — and everything would drain to defer.
  • A loud failure on a state the parent cannot demonstrate is handling, not a defect. This is the head of the distribution of what edge-case reviewers emit, not a corner case: the crash is high-signal and branch-free; the fallback converts it into silent, unverifiable-by-construction wrong behavior far from the fault.
  • defer widens to every other real finding — pre-existing issues, improvement ideas, and claims believed but not demonstrated. reject is restricted to noise. The existing "prefer reject when unsure" tiebreaker stays: it is a confidence rule, not a worth rule.
  • Patch instructions handed to the step-03 subagent now specify the smallest fix.

The design rule is the same as #2697: the model checks substance, the human judges worth. Every predicate is mechanically checkable — is the trigger reachable, does the fix add public symbols, is the gate red. The cost-benefit judgment ("the solution must solve more problems than it creates") is compiled into the fix conditions at design time, not simulated per-finding at runtime; where the conditions can't settle it, the finding lands on a human channel — intent_gap loopback interactively, blocked status in auto.

Deliberate consequences

  • In bmad-build-auto, a demonstrated finding whose fix the spec doesn't settle now blocks the iteration (intent_gap → HALT blocked) instead of shipping silently. An unattended loop growing public API without eyes on it is exactly the failure this trades throughput against.
  • Re-deriving for a small contract change is disproportionate on its face, but that is the route's covenant: the contract lives in the spec and code is disposable. The 5-iteration cap bounds the cost.
  • The reviewer prompts are untouched. Their structural blindness is by design — they cannot check reachability, so asking them to self-censor would violate the checkable-predicate rule and lose findings the parent can judge with better context.

Validation

Draft until the one-shot A/B (#2697) reports. Same frozen-findings method applies here, with one addition: score second-generation findings — findings whose subject is code a previous pass's patch introduced — as their own metric, since this route's loop is where the avalanche compounds.

…efer

Port the one-shot triage redesign to both step-04 review files. The
patch category was gated on ease alone ("trivially fixable"), so every
cheap fix qualified, including guards for states nothing reaches; defer
was scoped to pre-existing issues, leaving real-but-unwanted findings no
exit except shipped code.

- patch now requires demonstrated substance (a defect that occurs,
  missing coverage for a specific case, or a broken gate or convention)
  and a smallest fix that is trivial, adds no public surface, and guards
  no undemonstrated state. Demonstrated findings whose smallest fix
  fails a condition drain to intent_gap when the spec does not settle
  the fix, otherwise bad_spec — the route's existing escalation lanes.
- The parent carries the demonstration duty: reviewers report claims
  without codebase context by design, so triage verifies each trigger
  itself and judges findings by trigger, never by the proposed remedy.
  A loud failure on a state the parent cannot demonstrate is handling,
  not a defect.
- defer widens to every other real finding, improvement ideas included;
  reject is restricted to noise. The confidence tiebreaker stays.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant