The largest gap between this harness and a defensible result.
The bundled bundle evaluator re-hashes files and self-describes as development-only bundle integrity. It does not compile anything, run any test, or read the problem statement. The completed 19-attempt live campaign ran no evaluator at all, so the declared estimand end_to_end_within_budget_strict_success_rate evaluated to null on every row.
A live run also showed why this matters concretely: the differential oracle (solution passes, template fails) cannot distinguish a 4-test exercise whose statement makes five promises — two of them untested, so a student can violate them and pass — from an 18-test exercise where every promise has a test. Statement-promise-to-test traceability is the missing check.
Required, per docs/EVALUATION guidance now folded into docs/METHODOLOGY.md: a restricted suite manifest binding dataset and target digests, evaluator profile, case coverage, metric-card identities and toolchain digests; per case, stable executable requirement IDs, hidden oracles and boundary partitions, and reviewed non-equivalent mutants linked to the requirements they detect; separate evaluation layers with no weighted composite; and LLM judges kept exploratory until calibrated against blinded experts.
The process-evaluator protocol and CLI that a suite would plug into are in place (docs/PROCESS-EVALUATORS.md).
The largest gap between this harness and a defensible result.
The bundled
bundleevaluator re-hashes files and self-describes as development-only bundle integrity. It does not compile anything, run any test, or read the problem statement. The completed 19-attempt live campaign ran no evaluator at all, so the declared estimandend_to_end_within_budget_strict_success_rateevaluated tonullon every row.A live run also showed why this matters concretely: the differential oracle (solution passes, template fails) cannot distinguish a 4-test exercise whose statement makes five promises — two of them untested, so a student can violate them and pass — from an 18-test exercise where every promise has a test. Statement-promise-to-test traceability is the missing check.
Required, per
docs/EVALUATIONguidance now folded intodocs/METHODOLOGY.md: a restricted suite manifest binding dataset and target digests, evaluator profile, case coverage, metric-card identities and toolchain digests; per case, stable executable requirement IDs, hidden oracles and boundary partitions, and reviewed non-equivalent mutants linked to the requirements they detect; separate evaluation layers with no weighted composite; and LLM judges kept exploratory until calibrated against blinded experts.The process-evaluator protocol and CLI that a suite would plug into are in place (
docs/PROCESS-EVALUATORS.md).