Add a fidelity evidence loop: hypothesis checks, best-candidate scoring, and edit-locality gates

[pzfreo]

Summary

A comparison of two complete 81-fixture CADGenBench runs shows that build123d-mcp already provides a strong execution/validity foundation, but it does not reliably stop an agent from confidently validating the wrong geometric hypothesis.

Metric	Claude Opus 5 xhigh / MCP 0.3.81	GPT-5.6 Sol xhigh / MCP 0.3.79
Overall	0.6391	0.5319
Generation	0.5956	0.4507
Editing	0.7058	0.6562
Valid outputs	80/81	80/81

The overall difference is +0.1072 (+20.2% relative), with about 82% of the gain coming from generation. Validity is identical, so this is overwhelmingly a fidelity and interpretation gap rather than a robustness gap.

Reports:

• Opus: https://huggingai4engineering-cadgenbench.hf.space/reports/pzfreo_build123d-mcp-v0381-claude-opus-5-xhigh-_20260807-101147.html
• Sol: https://huggingai4engineering-cadgenbench.hf.space/reports/pzfreo_gpt-5-6-sol-xhigh-floor-chain-v0379_20260717-045810.html

This is not a controlled model-only A/B: the MCP and harness revisions differ. The traces are nevertheless useful for identifying server-side interventions that should help both models.

What the traces show

Sol generally used more verification tooling, especially during generation:

Average generation calls	Opus	Sol
execute	14.3	13.7
validate	4.4	9.3
render_view	6.4	9.5
measurement	0.3	1.6
cross-sections	0.3	1.9

On fixtures where Opus gained at least 0.15, it averaged about 19.5 geometry executions versus Sol's 15.4, while Sol still performed substantially more checking. Tool-call volume is therefore not the explanation: Opus more often chose the right body family, dimension interpretation, or exact edit strategy; Sol often inspected and validated a plausible but wrong object.

Representative cases:

• Fixture 203 (edit): 0.974 vs 0.163. Opus separated the exact axisymmetric impeller core and repeated blade geometry, proved the decomposition, and repatterned an extracted source blade. Sol exhausted several defeaturing/boolean approaches and shipped an approximate parametric blade reconstruction. This provides new positive evidence for feat: rotational-pattern helper — detect N-fold symmetry, extract axisymmetric core + one instance, re-pattern M #319: the exact decomposition strategy works and is extremely valuable when discovered.
• Fixture 105 (generation): 0.915 vs 0.390. Opus modeled an approximately 100 × 87.1 × 50 detailed form; Sol settled on 100 × 100 × 30. Repeated rendering and validation did not expose the macro-form/orientation error.
• Fixture 130 (generation): 0.813 vs 0.290. Sol's reasoning initially identified a height near 39.2, but a later dimension-chain interpretation produced a 57.8-high part. No tool treated that contradiction as actionable.
• Fixture 245 (edit): 0.262 vs 0.890, reversed. Opus made a broad subassembly move and dismissed a comparison-spread warning as expected. Existing checks supplied evidence, but there was no enforced edit-locality contract or evidence threshold for overriding the warning.

Proposal: add a fidelity/evidence control loop

This issue proposes orchestration and contracts around existing/forthcoming primitives, not another generic validation tool.

1. Structured geometric hypothesis / evidence ledger

Allow the agent to register expected facts before or during construction:

• overall envelope and orientation;
• dimension-chain equations and tolerances;
• feature counts, axes, locations, symmetry and repetition;
• protected geometry and permitted change region for editing;
• evidence source for each claim: drawing dimension, imported B-rep measurement, inference, or agent assumption.

Re-check these facts after each candidate and return contradictions separately from topology/validity. A valid solid with unresolved or contradicted fidelity claims must not be described as fully verified.

2. Fidelity-aware candidate/snapshot scoring

Track a composite evidence record for every snapshot:

• validity/export gate;
• registered dimensional constraints;
• silhouette/profile agreement when available (Raise CADGenBench Shape+Interface fidelity: fix export/render friction + close the model↔drawing loop #304);
• feature count/topology expectations;
• edit locality and protected-region equality;
• unresolved warnings.

Expose best_candidate and default finalization guidance to the highest-evidence candidate rather than simply the most recent candidate. Detect repair cycles or repeated strategies that do not improve this score.

3. Enforced edit-locality report

For edit tasks, report and optionally gate:

• changed volume and symmetric-difference volume;
• affected bounding box and spatial spread;
• moved/added/removed connected components;
• protected-region equality checks;
• whether the change is localized to the declared target.

An unexpectedly broad change should be a failure or require a structured override backed by exact proof, rather than a warning the model can dismiss narratively.

4. Calibrated tool language

Separate these outcomes explicitly:

• structurally_valid
• exportable
• declared_constraints_verified
• drawing_fidelity_verified
• edit_locality_verified

Avoid a bare “all checks passed” when only structural checks ran. Return missing-evidence items alongside successes.

Relationship to existing issues

• Raise CADGenBench Shape+Interface fidelity: fix export/render friction + close the model↔drawing loop #304: drawing/model comparison and closing the visual fidelity loop. This proposal consumes those metrics in a persistent evidence/candidate loop.
• feat: rotational-pattern helper — detect N-fold symmetry, extract axisymmetric core + one instance, re-pattern M #319: exact rotational decomposition/repatterning. Fixture 203 now demonstrates the proposed strategy succeeding with near-perfect score.
• verify_spec: reproducible design bundle + verify→repair→re-verify loop + YAML specs #340 and verify_spec: broaden feature vocabulary (threads, fillets/chamfers, pockets/slots, ribs, symmetry, wall/tolerance, mass, material) #337: verify_spec loop and broader feature vocabulary. The hypothesis ledger could build on verify_spec rather than introduce a parallel format.
• Add a b123d-editing skill for safe imported-solid edits #404: safe imported-solid editing workflow. Edit-locality enforcement should become a machine-checkable part of that workflow.

Suggested implementation order

1. Calibrate validation responses to distinguish validity from unverified fidelity.
2. Add a lightweight hypothesis/evidence object backed by verify_spec concepts.
3. Add edit-locality reporting and protected-region gates.
4. Persist per-snapshot evidence and expose the best candidate / cycle detection.
5. Feed drawing comparison (Raise CADGenBench Shape+Interface fidelity: fix export/render friction + close the model↔drawing loop #304) and exact decomposition (feat: rotational-pattern helper — detect N-fold symmetry, extract axisymmetric core + one instance, re-pattern M #319) into the same loop as they mature.

Acceptance criteria

• A candidate whose measured envelope contradicts a registered drawing dimension receives a machine-readable contradiction even if validate() passes.
• An edit outside a declared target/protected-region contract cannot receive an unqualified success verdict.
• The session can identify and restore the best-evidenced snapshot after a later regression.
• Tool responses distinguish structural validity from drawing fidelity and list unverified claims.
• Replaying representative failures such as fixtures 105, 130 and 245 produces actionable feedback before export.

[pzfreo]

Same-model ablation adds evidence for candidate anchoring

The Opus 5 xhigh MCP/no-MCP comparison shows a useful generation-specific failure mode for the proposed best-candidate/evidence loop.

MCP achieved better overall score and validity, but no MCP led generation 0.631 vs 0.596. The gap is dominated by six complex-form fixtures (102, 106, 116, 126, 145, 150). In several MCP traces, the agent established a simplified valid checkpoint, attempted a risky dominant-form correction (large fillet, bend, curved profile, outline reinterpretation), encountered failure/friction, and finalized the structurally safe approximation. No-MCP retained a canonical script on disk and could perform larger global rewrites while keeping the prior STEP safe.

This suggests the evidence loop should distinguish:

• structurally_best_candidate
• drawing_fidelity_best_candidate
• unresolved dominant-form mismatch

and should not allow a valid checkpoint to become an implicit stop signal when drawing evidence still says the body family/profile is wrong. A useful acceptance test is fixture 145: both systems produced the correct 116 × 116 × 20 envelope and feature inventory, but no MCP's additional S-curve blade refinement scored 0.567 versus MCP's 0.267.

Related reports:

• https://huggingai4engineering-cadgenbench.hf.space/reports/pzfreo_opus5-xhigh-official-baseline-prompt-nom_20260811-034944.html
• https://huggingai4engineering-cadgenbench.hf.space/reports/pzfreo_build123d-mcp-v0381-claude-opus-5-xhigh-_20260807-101147.html

[pzfreo]

Programme link

The drawing-relative evidence source anticipated here and in #304 is now planned as a measured epic in #446.

#446 will first establish whether registered candidate-to-drawing projection discrepancies are reliable and improve the frozen Opus 5 generation sample. If that passes, this issue's evidence ledger can consume those observations, distinguish structurally-best from fidelity-best candidates, and preserve/restore the best-supported snapshot.

Keeping the phases separate avoids repeating the earlier verify_spec failure mode: no persistent best-candidate score should incorporate drawing fidelity until the underlying comparison signal has independently passed its registration and paired-score gates.