Commit 53f76f5
feat(graders): replace ReAct AgenticGrader with harness-based agent-as-judge
Replaces the in-process ReAct-based AgenticGrader with a harness-based
"agent as a judge" implementation that shells out to real coding-agent
CLIs (Claude Code, Codex, Cursor CLI) inside a lightweight,
physically-isolated sandbox, driven by a unified Rubric/Checkpoint
input schema.
- openjudge/graders/schema.py: add Checkpoint/Rubric/CheckpointResult/
RubricResult models.
- openjudge/harness/: new package with BaseHarness (file-based result
protocol + failure gates), ProcessSandbox (physical-copy sandbox with
symlink skipping), and ClaudeCodeHarness/CodexHarness/CursorAgentHarness.
- openjudge/graders/agentic_grader.py: full rewrite on the harness
protocol -- builds a prompt from rubrics, runs a single sandboxed
harness sample, and aggregates checkpoint results with a weighted
mean into GraderScore/GraderError.
- Deletes the legacy openjudge/agentic/ ReAct infra, cookbooks/agentic_grader/,
and SearchCorrectnessGrader outright (no migration path).
- Updates skills/openjudge/graders.md docs and adds cookbooks/agentic_judge/
runnable examples for all three CLIs.
This is a v1: each evaluation runs exactly one sandboxed harness sample
with a plain weighted-mean aggregation -- no must_have AND-gate, no
k-sample majority voting, no agreement reliability gate. Those are
deferred to a follow-up iteration.
Implemented via subagent-driven development against
docs/superpowers/plans/2026-07-07-agentic-judge-harness.md, with a
per-task implementer + reviewer loop and a final whole-branch review;
864 tests passing, 157 skipped.
Co-authored-by: Cursor <cursoragent@cursor.com>1 parent 344e45d commit 53f76f5
36 files changed
Lines changed: 1924 additions & 3162 deletions
File tree
- cookbooks
- agentic_grader
- adapters
- agentic_judge
- openjudge
- agentic
- adapters
- graders
- common
- harness
- skills/openjudge
- tests
- graders
- common
- harness
This file was deleted.
Lines changed: 0 additions & 79 deletions
This file was deleted.
This file was deleted.
This file was deleted.
0 commit comments