Synthesis Reproducibility Benchmark

[oesteban]

Objective

Measure whether repeating the agent's synthesis of fMRIPrep produces the same pipeline every time. Synthesize fMRIPrep repeatedly for two targets — Claude /workflow (#3) and Nextflow (#7) — and quantify how often the emitted pipeline differs. This is the pipeline-repeatability benchmark; with the accuracy/recall benchmark (nipreps/fmriprep-equivalence#1) it forms the project's pipeline benchmark suite, complementing the KB/skills benchmark suite (nipreps/fmri-preprocessing-llm-wiki#6).

Rationale

If the KB grounds the agent well, synthesizing fMRIPrep twice should yield the identical pipeline — solid anchoring leaves the model no room to improvise. If the harnessing is insufficient, repeats diverge: the agent fills the gaps the KB doesn't pin down with its own, varying choices. So synthesis reproducibility is a direct, cheap measure of grounding quality, and divergences localize exactly which operations the KB fails to anchor. This is distinct from numerical reproducibility of pipeline outputs (the equivalence verifier's job, nipreps/fmriprep#3663); here we measure reproducibility of the pipeline the agent writes.

Methodological approach

1. Repeated synthesis. For each target — /workflow (fMRIPrep on Claude Code's /workflow (nipype ablation) #3) and Nextflow (fMRIPrep on Nextflow #7) — run the agent's synthesis of fMRIPrep N times under fixed inputs, capturing the emitted pipeline each time.
2. Compare. Diff the N pipelines per target: identical, equivalent up to harmless reordering, or genuinely different operations/methods? Report a synthesis-reproducibility rate per target and per operation.
3. Localize gaps. Trace every divergence to the KB facet that failed to anchor it, feeding the KB-hardening backlog.
4. Cross-target check. The pipeline the agent writes (operations and methods) should be the same whether it targets /workflow or Nextflow — engine-specific lowering aside; disagreement between targets is itself a grounding signal.

Outcomes/impact

1. A synthesis-reproducibility benchmark and its results: how often repeated synthesis of fMRIPrep yields the same pipeline, per target, with divergences localized to specific KB gaps.
2. The repeatability half of the project's pipeline benchmark suite (with accuracy/recall, Implementation Accuracy & Recall Benchmark fmriprep-equivalence#1); the other benchmark suite is the KB/skills benchmark (Wiki KB v1.0: Validation & KB/skills Benchmark fmri-preprocessing-llm-wiki#6).

Impact. Turns "is the KB actually anchoring the agent?" into a number, and points at exactly which operations need better grounding — the cheapest early-warning signal that the synthesis is trustworthy.

Direct upstream issues

(issues this one depends on / builds upon)

• fMRIPrep on Claude Code's /workflow (nipype ablation) #3
• fMRIPrep on Nextflow #7

Direct downstream issues

(issues that depend on this one)

None on the board — results feed the KB-hardening backlog.