exgen-bench

Benchmark systems that generate complete programming exercises.

Illustrative results site · Documentation · Contributing

Status: pre-alpha. The local smoke test and public result format work. An experimental Artemis adapter drives ordinary product APIs, but the Hyperion feature it targets is still on an unmerged Artemis branch. The included smoke evaluator checks file completeness, not whether an exercise is correct or suitable for teaching. One public development dataset exists; the restricted validation set, the sealed confirmatory set, and a validated evaluator do not. No empirical benchmark results have been released.

What exgen-bench does

A programming exercise usually needs a problem statement, starter code, a reference solution, tests, and platform settings. exgen-bench compares the complete systems that produce these artifacts—not model responses in isolation.

1. A benchmark plan fixes the exercise briefs, generation systems, repetitions, and resource limits before execution.
2. Each generation system receives the same planned work through a process-based adapter.
3. A separately versioned evaluator checks each candidate exercise without rerunning generation.
4. The release retains every planned outcome and provides checksummed files for analysis and the static results site.

The glossary defines these terms and their corresponding protocol fields.

Get started

The illustrative results site is the fastest way to inspect the release interface. It uses synthetic data and does not report benchmark findings.

To verify the complete local pipeline without credentials, install the Bun version listed in .bun-version, then run:

git clone https://github.com/ls1intum/exgen-bench.git
cd exgen-bench
bun install --frozen-lockfile
bun run smoke

The command runs generation, evaluation, release verification, and the static-site build. A successful run ends with Full pipeline passed. It uses a fixed mock generator and checks file integrity, not exercise quality.

After installation, preview the results site with the checked-in synthetic data:

bun run site:serve

The command prints the local URL.

Run the stages yourself

Use the small example to inspect planning, execution, saved state, and evaluation separately:

bun run cli validate examples/smoke/benchmark.yaml
bun run cli plan examples/smoke/benchmark.yaml
bun run cli run examples/smoke/benchmark.yaml --id quickstart
bun run cli status .exgen/runs/quickstart
bun run cli verify .exgen/runs/quickstart
bun run cli evaluate bundle .exgen/runs/quickstart

Use a new run ID when repeating the example. Run bun run cli --help for all commands. The bundle evaluator verifies the saved candidate files; it does not assess whether an exercise is correct or suitable for teaching.

A study replaces bundle with an evaluator that runs out of process under its own version:

bun run cli evaluate process .exgen/runs/quickstart \
  --config examples/process-evaluator/config.yaml

The process-evaluator guide defines the configuration, the stdin/stdout contract, secret handling, and resume behavior.

How results are counted

The primary rate is:

exercise successes / planned attempts

A planned attempt counts as a success only when it produces a complete candidate, passes the independent evaluation, and stays within its resource limits. Attempts that do not start, fail, abstain, exceed a limit, or lack an evaluation result remain in the denominator but do not count as successes. The methodology defines the outcome and missing-data rules.

Choose your next step

If you want to…	Read…
understand the components and saved data	System design
design or review a formal comparison	Methodology
run the 19-case Hyperion development pack	Hyperion development study
connect Artemis or Hyperion	Artemis integration
find out whether your system can be benchmarked at all	System-under-test requirements
connect another generation system	Protocol schemas
write an evaluator	Process evaluators
record what a system actually did	Telemetry profile
compare an unbilled model with public rates	Reference pricing
handle restricted evidence	Restricted archives
build or preview the results site	Results-site guide
find the meaning of a term	Glossary
contribute a change	Contributing guide

The documentation index groups all project guides and references by task.

Research safeguards

• Failures, abstentions, cancellations, and missing results stay separate.
• Every system in a formal comparison uses the same independent evaluator.
• The study configuration fixes the main analysis before the run.
• The Artemis adapter can reconcile what Artemis reports about itself against an independent OpenTelemetry trace and fail the attempt on disagreement. This is opt-in per call and specific to that adapter; the runner does not consult telemetry when it accepts an attempt.

Generated code is untrusted. Formal evaluation requires isolated Linux execution; the security policy defines the trust boundaries and minimum deployment baseline.

Support, citation, and license

Use GitHub Issues for reproducible bugs and methodology proposals. Report vulnerabilities privately through GitHub Security Advisories.

The TUM Applied Education Technologies group maintains exgen-bench. Citation metadata is in CITATION.cff. Code and documentation are available under the MIT License, and participation is governed by the Code of Conduct.