Epic: close the drawing-to-candidate loop for generation fidelity

[pzfreo]

Objective

Close the remaining generation-fidelity gap by giving CAD agents compact, calibrated drawing-to-candidate evidence and a bounded correction path, without weakening build123d-mcp's verified advantages in editing, validity, token cost, and runtime.

This epic is deliberately phased. Each phase must produce measured evidence and may be adopted, revised, or rejected independently. Do not proceed to a full benchmark sweep merely because an API or prompt is technically complete.

Why this is the next hypothesis

A controlled Opus 5 xhigh comparison found:

Metric	Official baseline / no MCP	build123d-mcp 0.3.81
Overall	0.627	0.639
Generation	0.631	0.596
Editing	0.621	0.706
Valid outputs	75/81	80/81

The generation deficit is concentrated. Six fixtures (102, 106, 116, 126, 145, 150) contribute +2.132 points to no-MCP while the net 49-fixture generation advantage is only +1.730; excluding those six, MCP is slightly ahead.

Trace review shows that the strongest no-MCP generations repeatedly:

• calibrated drawing pixels from printed dimensions;
• sampled complex silhouettes and undimensioned curves;
• rendered matching orthographic projections;
• overlaid candidate and reference views;
• made global source rewrites when the representation was wrong.

Current MCP tools inspect the candidate itself. They do not provide a machine-readable, registered comparison between that candidate and the input drawing.

Prior work and lessons

This epic builds on, but must not repeat uncritically:

• Raise CADGenBench Shape+Interface fidelity: fix export/render friction + close the model↔drawing loop #304 — identified the open-loop model↔drawing fidelity gap.
• Modeling skill: add a dominant-form correction loop for generation fidelity #412 and Expand modeling skill with canonical body-family recipes for dominant-form correction #418 / PR Add body-family recipes to modeling skill #419 — dominant-form guidance and body-family recipes. Helpful, but prose alone did not close the gap.
• Add a generic feature-inventory audit helper for generation checkpoints #417 / PR Add compact inspect_part checkpoint #420 — inspect_part supplies compact candidate-side evidence. Its generation A/B remains tracked in A/B test bounded inspect_part feedback during generation cadgenbench-build123d#57.
• verify_spec / suggest_spec — full-run evidence showed positive conformance verdicts could become premature stop signals. New comparison output must report evidence, not certify completion.
• design_audit — useful for parametric robustness, not for deciding whether the inferred design matches the drawing.
• Add a canonical source-backed generation workflow (edit script, execute, verify, checkpoint) #443 / PRs Add atomic source-backed execution #444–Use source-backed drawing-fidelity workflow for generation #445 — canonical source is a potentially useful global-rebuild actuator, but source access alone did not recover no-MCP generation quality.
• Source-backed sample mean: 0.475; original MCP on the same six: 0.446; no MCP: 0.640.
• Conditional-source sample mean: 0.413. Conditional source injection is rejected in its current form.
• Explore multi-candidate generation with generic proxy selection for high-variance fixtures cadgenbench-build123d#49 — multi-candidate selection remains untested and lacks a trustworthy drawing-relative selector.
• Consolidate the default MCP tool surface (decoupled from #417/#420) #421 — the large default tool surface remains a plausible source of generation interference and requires its own A/B gate.

Design principles

1. Evidence, not verdicts. Do not return PASS, conforms, or an authoritative completion signal for drawing fidelity.
2. Drawing-derived and candidate-derived facts stay distinct. Never present model-inferred expectations as observations from the image.
3. Printed dimensions remain authoritative. Pixel calibration may estimate undimensioned geometry but must not override explicit callouts.
4. Compact output. Return the few largest actionable discrepancies, not raw CV dumps or unbounded contours.
5. Preserve model agency. MCP supplies measurements and controlled execution; the agent owns the design hypothesis.
6. Selective escalation. Normal generation remains lightweight. Global source rewrite or a second candidate is reserved for measured structural mismatch.
7. No benchmark-specific geometry. Fixture IDs belong only in evaluation plans, never runtime rules, skills, examples, or detector code.
8. No full sweep without a passed gate. Small paired tests precede broader validation.

Reference evaluation set

Use the existing frozen Opus 5 runs as historical baselines.

MCP-loss cases:

• 102, 106, 116, 126, 145, 150

MCP-winning controls:

• 113, 119, 133, 138, 139

The controls are mandatory: an intervention that improves complex contours by distracting the model on ordinary generation is not a success.

Phase 0 — Freeze evidence and contracts

• Record exact baseline submissions, MCP version, model, effort, prompts, scores, validity, runtime, token usage, and per-fixture results.
• Classify the 11 fixtures by dominant geometry, view arrangement, explicit dimensions, undimensioned contours, and observed failure mechanism.
• Define separate schemas for drawing observations, candidate observations, registration, and discrepancies.
• Define confidence/ambiguity semantics and failure behaviour.
• Define an output token/size ceiling.
• Pre-register the phase-1 test and decision rule before scoring treatments.

Exit gate:

• Frozen comparison table and machine-readable provenance.
• No schema field implies ground truth or certification.
• Review demonstrates that printed dimensions cannot be silently overridden.

Phase 1 — Projection registration feasibility, offline first

Prototype and validate registration outside the public MCP surface:

• Locate the relevant orthographic drawing regions without using fixture-specific coordinates.
• Render the candidate in matching orthographic orientations.
• Calibrate image scale using explicit printed dimensions where available.
• Align candidate projections without normalizing away real scale or orientation errors.
• Produce silhouette overlap, contour-distance summaries, and a bounded overlay artifact.
• Surface ambiguity when view association or scale is not reliable.
• Test annotation, hatch, centreline, and dimension-line rejection.

Measure:

• Registration success/failure rate.
• False alignment rate.
• Sensitivity to deliberate scale, translation, rotation, missing-hole, filled-void, and contour perturbations.
• Runtime and memory.
• Stability across repeated renders.
• Whether discrepancy direction is correct, not merely whether a scalar changes.

Exit gate:

• Reliable registration on the development/selfbench set and the 11 frozen inputs.
• Synthetic perturbations are ranked in the correct direction.
• Uncertain cases fail explicitly rather than returning a confident misleading comparison.

If this gate fails, stop. Do not expose an MCP tool.

Phase 2 — Compact drawing-comparison API

Expose one experimental workflow-level entry point, not a cluster of low-level CV tools.

Candidate contract:

• drawing/view identity and registration confidence;

• scale/orientation evidence;

• a small number of ranked silhouette or interface discrepancies;

• optional bounded overlay paths;

• no completion verdict;

• no hidden automatic correction.

• Keep it experimental/off by default.

• Bound image sizes, contour counts, output size, and runtime.

• Isolate image processing so failure cannot destroy the CAD session.

• Add adversarial tests for annotation ink, section hatching, partial views, multiple components, symmetry, and similar nearby contours.

• Ensure tool descriptions distinguish evidence from instruction.

Exit gate:

• Detector and API tests pass.
• Output remains compact enough for an agent correction loop.
• Independent review finds no route by which a low-confidence alignment masquerades as a high-confidence discrepancy.

Phase 3 — Bounded Opus 5 comparison-only A/B

Intervention:

1. Build the first valid coarse candidate using the normal MCP workflow.
2. Run one drawing-aligned comparison.
3. Correct the single largest confirmed mismatch.
4. Permit one final comparison only after material geometry change.
5. Validate and export normally.

Do not add source injection, multiple candidates, semantic constructors, or inspect_part prompt changes in this phase.

• Control: frozen current MCP prompt/tool surface.
• Treatment: comparison-only addition.
• Same Opus 5 model, xhigh effort, MCP version, fixture set, timeout, and concurrency.
• Preserve logs, tool output, token usage, runtime, validity, and correction classification.
• Score through the authoritative CADGenBench service.

Primary measures:

• Paired CAD score delta.
• Median paired delta.
• Improved/tied/regressed counts.
• Recovery on the six loss cases.
• Regression on the five controls.

Secondary measures:

• Validity.
• Token/API-equivalent cost.
• Runtime.
• Comparison call success.
• Fraction of discrepancies causing material geometry changes.
• Fraction of those changes improving official score.

Phase-3 success gate:

• Recover at least half of the six-fixture 2.132-point deficit, or demonstrate a comparably convincing paired effect after accounting for scorer noise.
• Positive paired median.
• Gains are not explained by one fixture.
• No material validity regression.
• No meaningful aggregate regression on controls.
• Cost/runtime remain materially below no MCP.
• Log review confirms the tool caused correct geometric corrections rather than redundant narration.

Phase 4 — Comparison-informed global reconstruction

Only if phase 3 passes.

Test canonical source as an actuator after a measured structural mismatch:

• Define a narrow escalation condition: large registered contour mismatch, cross-view inconsistency, or confirmed wrong body family.
• Use atomic execute_file() for a complete candidate rewrite.
• Preserve the last valid model if rebuilding fails.
• Recompare after the rewrite.
• Do not inject build123d implementation source unless answering a focused API question.

A/B:

• Phase-3 comparison-only treatment versus comparison plus canonical reconstruction.
• Same 11-fixture set and measurement contract.

Exit gate:

• Additional paired improvement beyond phase 3.
• Global rewrites correlate with measured structural mismatches.
• No recurrence of the conditional-source regression pattern.
• Failure recovery preserves validity and prior state.

Phase 5 — Candidate-side semantic feedback

Only after drawing comparison is useful.

Factor inspect_part separately:

• Drawing comparison only.
• Drawing comparison plus one bounded inspect_part checkpoint.

Purpose:

• Test whether candidate feature inventory adds value after the image-derived plan is grounded.
• Determine whether model-authored expectations introduce errors or premature stopping.
• Keep inspect_part evidence separate from drawing observations.

Exit gate:

• Incremental paired improvement or a clear reduction in interface errors.
• Expectation construction is usually correct.
• No conformance-style stop-signal regression.

Phase 6 — Optional candidate branching

Only if comparison is a trustworthy selector and residual failures are early-hypothesis errors.

• Generate at most two coarse candidates.
• Compare both using the same registered drawing evidence.
• Record selection rationale and both artifacts.
• Elaborate only the selected candidate.
• Compare score gain against added cost and runtime.

Exit gate:

• Selection predicts the better official result often enough to justify cost.
• Gain is not obtainable more cheaply through one reconstruction pass.

Phase 7 — Tool-surface and prompt integration

After functionality is proven:

• Coordinate with Consolidate the default MCP tool surface (decoupled from #417/#420) #421 so comparison does not simply add more permanent schema burden.
• Decide whether generation receives a narrower surface than editing while preserving one supportable configuration.
• Keep orchestration in the harness and reusable CAD reasoning in the MCP skill.
• Remove or demote redundant guidance.
• Verify that source, skill, prompt, resource, and tool descriptions do not conflict.
• Run a final paired sample before considering a full sweep.

Phase 8 — Full validation

Only after all preceding adopted phases pass their gates:

• Freeze commits, dependencies, prompt, model, effort, and run metadata.
• Run the full benchmark.
• Package all fixtures correctly, including required empty directories.
• Publish filtered accreditation logs and safe cost/time aggregates.
• Compare overall, generation, editing, validity, tokens, cost, and runtime with both verified MCP 0.3.81 and official-baseline/no-MCP Opus 5.
• Record negative as well as positive results in this epic.

Programme-level success criteria

Adopt the complete workflow only if it:

• raises verified overall score beyond 0.639 by more than expected noise;
• raises generation beyond the current MCP 0.596, with a target of exceeding no-MCP 0.631;
• preserves the editing advantage near 0.706;
• preserves approximately 80/81 validity or improves it;
• retains a meaningful token/runtime advantage over no MCP;
• generalizes beyond the six diagnostic fixtures.

Non-goals

• Encoding benchmark fixture answers.
• Replacing Opus's spatial reasoning with a deterministic image-to-CAD solver.
• Returning a misleading fidelity certification.
• Adding a large public computer-vision tool family.
• Running all 81 fixtures before the sample gates pass.
• Treating source access as geometric evidence.
• Combining all interventions into one uninterpretable test.

Related

• Raise CADGenBench Shape+Interface fidelity: fix export/render friction + close the model↔drawing loop #304
• Consolidate the default MCP tool surface (decoupled from #417/#420) #421
• Add a canonical source-backed generation workflow (edit script, execute, verify, checkpoint) #443
• PR Add atomic source-backed execution #444
• PR Use source-backed drawing-fidelity workflow for generation #445
• Adopt Arko-T design-state levers: named-parameter prompt block + design_audit() self-check (pin bump, allowlist, prompt hooks) cadgenbench-build123d#34
• Thin harness prompts and delegate reusable CAD knowledge to MCP skills cadgenbench-build123d#41
• Tighten generation prompt: fillets are not dominant-form correction cadgenbench-build123d#48
• Explore multi-candidate generation with generic proxy selection for high-variance fixtures cadgenbench-build123d#49
• Add synthetic generation regression fixtures for common wrong-but-valid CAD failures cadgenbench-build123d#50
• A/B test bounded inspect_part feedback during generation cadgenbench-build123d#57

[pzfreo]

Relationship to #439

#439 is complementary and should be treated as an upstream/downstream integration point rather than duplicated here:

• Epic: close the drawing-to-candidate loop for generation fidelity #446 owns drawing-derived observations, projection registration, compact candidate-vs-drawing discrepancy evidence, and the phased generation A/B.
• Add a fidelity evidence loop: hypothesis checks, best-candidate scoring, and edit-locality gates #439 owns the persistent hypothesis/evidence ledger, structurally-best versus fidelity-best candidate anchoring, regression/cycle detection, and edit-locality contracts.

The phase-2 comparison API should emit evidence that #439 can later persist, but #446 must first prove that the drawing-relative signal is reliable and improves generation. Do not make the evidence-ledger work a prerequisite for the phase-1 registration feasibility test or phase-3 comparison-only A/B.

[pzfreo]

Phase 0 complete: frozen evidence, comparison contract, and acceptance dataset

The read-only Phase 0 analysis is complete locally. No MCP code was changed and no model/benchmark quota was used.

Main finding

Both Opus 5 configurations already performed substantial drawing crops, pixel measurement, and printed-dimension calibration. The missing capability is therefore not generic image access.

The separation is evidence-to-reconstruction:

• On the six MCP-loss fixtures, direct Opus more often converted projection evidence into a global representation change: true bend solids, an extracted transition outline, an S-curved blade, or a complete body rewrite.
• MCP often diagnosed the same broad mismatch but retained a valid checkpoint after a large correction or fillet failed.
• On the five controls, that structured/checkpointed MCP workflow was beneficial. Always rewriting or always chasing contours would regress ordinary parametric cases.

Frozen score totals:

• loss cases: direct advantage +2.132;
• controls: MCP advantage +0.996;
• all eleven: direct advantage +1.136;
• every candidate is valid in both arms, so this is a fidelity comparison.

Frozen outputs

The Phase 0 package contains:

1. fixture-by-fixture trace diagnosis and discrepancy taxonomy;
2. a JSON Schema separating drawing observations, candidate observations, registration, ranked discrepancies, and uncertainty;
3. an acceptance manifest resolving all 11 canonical drawings, both STEP candidates, both filtered logs, public scores, and 15 synthetic mutation families;
4. explicit Phase 1 stop gates.

Tightened Phase 1 gates

Do not expose an MCP tool unless:

• =90% of eligible view pairs register correctly;

• every uncertain/failed registration refuses a confident geometry conclusion;
• 100% of scale/rotation/reflection mutations are identified in the correct direction;

• =90% of eligible injected feature/contour defects are in the top three and localized correctly;

• output is <=4 KiB / five discrepancies;
• comparison is <=10 seconds per view and isolated from the CAD session.

The intended intervention remains evidence-only: no PASS, conforms, or completion verdict. Canonical source is a later actuator, not the missing signal.

Local files currently live under analysis/generation-phase0/; they should be published with the implementation/experiment branch rather than copied into runtime guidance.

[pzfreo]

Workflow contracts and 2D drawing boundary (decision, 2026-08-12)

The product/workflow discussion following the Phase 1 review reached the following decision for this epic.

Split the overloaded modelling guidance

Treat these as explicit workflow contracts, not stylistic personas and not separate server forks:

1. Collaborative design — iterative human-oriented parametric creation. Optimise for alternatives, previews, questions at material design decisions, editable source, and user-controlled stopping.
2. Drawing/spec → 3D reconstruction — evidence-driven reproduction. Optimise for a frozen dimension table, view/section consistency, autonomous correction of the largest confirmed mismatch, absolute scale, feature fidelity, and a stricter stopping rule.
3. Existing-model edit — preserve the current bounded baseline → edit → compare delta → validate contract.
4. Repair remains a cross-cutting recovery workflow rather than a user persona.

The current b123d-modeling skill mixes collaborative creation and reconstruction, whose interaction and stopping policies conflict. Phase 7 should therefore test a guidance-layer split (design versus reconstruct) while retaining one shared MCP server and compact core tool surface. Workflow state should be visible in provenance; transitions should be explicit, not inferred from one tool call.

Do not confound the Phase 3 comparison-only A/B with this skill split. First finish the currently running six-fixture fidelity-workflow sample; then A/B the reconstruction contract separately on the same balanced loss/control sample before considering a broad sweep.

3D → drawing is not a primary MCP workflow

Draftwright has materially superseded the built-in engineering-drawing workflow for active development. build123d-mcp should therefore:

• stop advertising 3D → engineering drawing as a primary capability;
• soft-deprecate the built-in drawing workflow as maintenance-only;
• remove drawing tools from the eventual default surface and retain explicit opt-in compatibility for existing users;
• keep the legacy b123d-drawing skill available during the compatibility window, clearly labelled;
• refer users seeking the actively developed drawing product to Draftwright.

Licence boundary: build123d-mcp remains Apache-2.0; Draftwright is a separate AGPL-3.0 project. Do not bundle, vendor, copy implementation, add it as a dependency, or present it as included functionality. File-level interoperability may be documented. Recommendations must identify Draftwright as separate and link to its licence.

This updates Phase 7's intended product surface: primary workflows are collaborative 3D design, drawing/spec-to-3D reconstruction, and parametric editing; built-in 2D drawing is legacy opt-in. The registration/tool-surface mechanics remain owned by #421.