Skip to content

Commit 53f76f5

Browse files
chr6192cursoragent
andcommitted
feat(graders): replace ReAct AgenticGrader with harness-based agent-as-judge
Replaces the in-process ReAct-based AgenticGrader with a harness-based "agent as a judge" implementation that shells out to real coding-agent CLIs (Claude Code, Codex, Cursor CLI) inside a lightweight, physically-isolated sandbox, driven by a unified Rubric/Checkpoint input schema. - openjudge/graders/schema.py: add Checkpoint/Rubric/CheckpointResult/ RubricResult models. - openjudge/harness/: new package with BaseHarness (file-based result protocol + failure gates), ProcessSandbox (physical-copy sandbox with symlink skipping), and ClaudeCodeHarness/CodexHarness/CursorAgentHarness. - openjudge/graders/agentic_grader.py: full rewrite on the harness protocol -- builds a prompt from rubrics, runs a single sandboxed harness sample, and aggregates checkpoint results with a weighted mean into GraderScore/GraderError. - Deletes the legacy openjudge/agentic/ ReAct infra, cookbooks/agentic_grader/, and SearchCorrectnessGrader outright (no migration path). - Updates skills/openjudge/graders.md docs and adds cookbooks/agentic_judge/ runnable examples for all three CLIs. This is a v1: each evaluation runs exactly one sandboxed harness sample with a plain weighted-mean aggregation -- no must_have AND-gate, no k-sample majority voting, no agreement reliability gate. Those are deferred to a follow-up iteration. Implemented via subagent-driven development against docs/superpowers/plans/2026-07-07-agentic-judge-harness.md, with a per-task implementer + reviewer loop and a final whole-branch review; 864 tests passing, 157 skipped. Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 344e45d commit 53f76f5

36 files changed

Lines changed: 1924 additions & 3162 deletions

cookbooks/agentic_grader/01_native_react_native_tool.py

Lines changed: 0 additions & 109 deletions
This file was deleted.

cookbooks/agentic_grader/02_native_react_langchain_tool.py

Lines changed: 0 additions & 79 deletions
This file was deleted.

cookbooks/agentic_grader/03_langchain_agent.py

Lines changed: 0 additions & 87 deletions
This file was deleted.

cookbooks/agentic_grader/04_agentscope_agent.py

Lines changed: 0 additions & 108 deletions
This file was deleted.

0 commit comments

Comments
 (0)