Skip to content

Commit ba6a73c

Browse files
authored
feat(judge): ✨ replace self-attested goal completion with independent Judge subagent (#227)
* feat(goal): ✨ replace self-attestation goal_scored with independent Judge acceptance agent Remove the `goal_scored` tool that allowed the main agent to self-attest goal completion, replacing it with an `agent_judge` built-in subagent that independently verifies goal attainment against the project's current state. Key changes: - Add `SubagentProfile::Judge` with read-only file tools and diagnostic-only shell (soft constraint via prompt) - Add `JudgeReport` structured contract (passed, completeness_pct, findings, summary) with safe fallback parsing - Add `agent_judge` tool injection only for the main agent when an unverified goal exists; runtime gate blocks subagent/parallel recursion into Judge - Add DB migration for `judge_passed`, `judge_completeness`, `judge_findings`, `judge_summary`, `judge_evaluated_run_id` columns with backfill for legacy `status='complete'` goals - Replace continuation stop condition: `Complete && judge_passed` instead of `goal_scored`-driven status flip - Rewrite continuation prompt to instruct main agent to call `agent_judge` and follow findings on rejection - Add Judge prompt surface, templates, and output contract - Update `active_goal.tpl.md` to reflect Judge acceptance flow - Extend goal lifecycle tests for Judge pass/fail/legacy compat * refactor(goal): ♻️ remove mark_complete and complete verdict Remove the mark_complete pathway from goals as completion will be handled through a different mechanism: - Remove mark_complete method from GoalManager - Remove "complete" from GoalEvaluateResult verdict type - Remove mark_complete test cases (evidence validation, etc.) - Update subagent surface comments to include judge BREAKING CHANGE: GoalEvaluateResult.verdict no longer includes "complete" * docs: 📝 update and reorder README feature list Update the feature descriptions and reorder the bullet points in both README.md and README_zh.md to better reflect the current product capabilities and improve readability. Changes include: - Reordering features to highlight persistent goal management, real-time streaming, and extensibility earlier in the list - Updating descriptions for several features to be more accurate - Maintaining consistency between English and Chinese versions - Keeping the overall structure while improving flow These are documentation-only changes that do not affect functionality. * refactor(goal): ♻️ extract resolveGoalStatusKey for testability - Extract inline status key resolution into a pure exported function so the complete→verified (judgePassed) branch can be unit-tested without mounting the component - Add unit tests covering all status mappings and judgePassed variants - Add test for skipped verdict passthrough in goalEvaluate * refactor(subagent): 🔧 increase builtin default max delegation depth to 5 Raise `BUILTIN_DEFAULT_MAX_DELEGATION_DEPTH` from 3 to 5 to match the existing `GLOBAL_MAX_DELEGATION_DEPTH`, allowing built-in subagents (explore/review) to be delegated to the same depth as custom profiles. Update delegation validation tests to reflect the new depth limits. * docs: 📝 remove obsolete design document * docs(judge): 📝 add size-first verification strategy and delegation guidelines * refactor(goal): ♻️ remove goal-level time_used_seconds in favor of run-level elapsed tracking * feat(judge): ✨ redesign Judge evaluation for independence and completeness * fix(subagent): 🐛 make task field optional and fix UTF-8 safe truncation - Downgrade Judge prompt versions from 2 to 1 (likely a revert of unintended bump) - Change `task` field from required to optional in Judge tool schema, with updated description clarifying it is an optional note - Replace byte-based truncation with character-safe truncation to avoid panicking on multi-byte UTF-8 in process compliance summary - Simplify Judge request validation to only check input validity, discarding the parsed result used only for backward compatibility - Skip abandoned task boards when building summary to focus on relevant goal state * chore(deps): 🔧 align tiycore to 0.2.10-rc.2 and adopt Usage::context_size() Cherry-pick the master commit (a03d9ba) that bumps tiycore from 0.2.9 to 0.2.10-rc.2 and unifies context_size semantics across RunUsageDto / frontend badge / auto-compression, removing the old initial_context_calibration heuristic path. No file conflict with the Judge work in this branch — the 25 files touched here do not overlap with the 6 Judge files resolved in the previous merge. * refactor(goal): ♻️ centralize status transitions to explicit commands and Judge verdicts * fix(agent): 🐛 fix timestamp slicing panic and add has_process_requirements tests Replace byte-index slicing with char-aware truncation to prevent panics on multi-byte UTF-8 boundaries in timestamp formatting. Add unit tests for `has_process_requirements()` covering English and CJK keywords, substring match behaviour, edge cases, and case-insensitive matching. * feat(compression): ✨ reserve 20% context window for auto-compression trigger Backend: replace fixed 16,384 token reserve with 20% of model context window (min floor 16,384). Small-window models keep the floor; GPT-4o class windows reserve ~25.6K, Claude-class ~40K, 1M-window ~200K. Frontend: add dashed threshold marker at 80% position in the thread header context pill so users can see when auto-compression will fire. * fix(run): 🐛 record elapsed running time when interrupting active runs * test: cover Judge summary builders and mapRunSummaryToContextUsage fallback Add four integration tests in agent_session_execution.rs for the Judge-prompt context builders (build_task_board_summary, build_process_compliance_summary) covering absent boards, active/abandoned board filtering, review-only helper filtering, status symbol mapping, and 200-char input truncation. Add six unit tests in runtime-thread-surface-state.test.ts for mapRunSummaryToContextUsage covering null input, explicit contextSize precedence, fallback to per-bucket sum, and full-field passthrough. Addresses review feedback from PR #227 (round 4): - #1 New DB-backed Judge summary functions lack tests - #4/#8 contextSize parsing/fallback logic lack unit tests
1 parent f05858c commit ba6a73c

16 files changed

Lines changed: 924 additions & 235 deletions

src-tauri/src/core/agent_session_execution.rs

Lines changed: 473 additions & 31 deletions
Large diffs are not rendered by default.

src-tauri/src/core/context_compression.rs

Lines changed: 67 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,16 @@
2121
2222
use tiycore::agent::AgentMessage;
2323
use tiycore::types::{ContentBlock, TextContent, UserMessage};
24-
/// Reserve this many tokens for the model's response + overhead.
25-
/// Matches pi-mono `DEFAULT_COMPACTION_SETTINGS.reserveTokens`.
26-
const RESERVE_TOKENS: u32 = 16_384;
24+
/// Fraction of the model's context window that is reserved for the model's
25+
/// response + provider/tool overhead, expressed in basis points (1/100th of
26+
/// a percent). 2000 bps == 20%.
27+
const RESERVE_BASIS_POINTS: u32 = 2_000;
28+
29+
/// Minimum number of tokens to keep reserved even when 20% of the context
30+
/// window is smaller than this floor. The previous hard-coded reserve of
31+
/// `16_384` tokens is preserved as a safe lower bound for typical large
32+
/// context windows while still allowing tiny windows to behave sanely.
33+
const RESERVE_TOKENS_MIN: u32 = 16_384;
2734

2835
/// Keep at least this many tokens of recent conversation untouched.
2936
/// With LLM-generated summaries providing rich context, we can keep a
@@ -233,7 +240,7 @@ impl CompressionSettings {
233240
pub fn new(context_window: u32) -> Self {
234241
Self {
235242
context_window,
236-
reserve_tokens: RESERVE_TOKENS,
243+
reserve_tokens: reserve_tokens_for(context_window),
237244
keep_recent_tokens: KEEP_RECENT_TOKENS,
238245
}
239246
}
@@ -244,6 +251,19 @@ impl CompressionSettings {
244251
}
245252
}
246253

254+
/// Reserve `RESERVE_BASIS_POINTS` (20%) of the model's context window for
255+
/// the model's response + overhead, with `RESERVE_TOKENS_MIN` as a floor
256+
/// so that very small windows still keep a sane amount of headroom and
257+
/// huge windows don't get a pathologically tiny reserve.
258+
fn reserve_tokens_for(context_window: u32) -> u32 {
259+
let percent_budget = ((context_window as u64)
260+
.saturating_mul(RESERVE_BASIS_POINTS as u64)
261+
.saturating_add(9_999))
262+
/ 10_000;
263+
let percent_budget = percent_budget.min(u32::MAX as u64) as u32;
264+
percent_budget.max(RESERVE_TOKENS_MIN)
265+
}
266+
247267
// ---------------------------------------------------------------------------
248268
// Public API: should_compress, find_cut_point, build_compressed_messages
249269
// ---------------------------------------------------------------------------
@@ -911,6 +931,49 @@ mod tests {
911931
assert_eq!(calibration.apply_to_estimate(0), 0);
912932
}
913933

934+
#[test]
935+
fn compression_settings_reserves_twenty_percent_of_context_window() {
936+
// For typical large context windows the 20% reserve is well above
937+
// the 16,384 token floor, so the budget is exactly 80% of the
938+
// window. This is the primary behaviour change: instead of
939+
// reserving a fixed 16,384 tokens regardless of model, we reserve
940+
// 20% of the model's actual context window.
941+
let cases = [
942+
(128_000_u32, 25_600_u32, 102_400_u32), // GPT-4o class
943+
(200_000_u32, 40_000_u32, 160_000_u32), // Claude-class
944+
(1_000_000_u32, 200_000_u32, 800_000_u32), // 1M-window class
945+
];
946+
for (context_window, expected_reserve, expected_budget) in cases {
947+
let settings = CompressionSettings::new(context_window);
948+
assert_eq!(
949+
settings.reserve_tokens, expected_reserve,
950+
"20% reserve for {context_window}-token window",
951+
);
952+
assert_eq!(
953+
settings.budget(),
954+
expected_budget,
955+
"80% budget for {context_window}-token window",
956+
);
957+
}
958+
}
959+
960+
#[test]
961+
fn compression_settings_reserve_clamps_to_minimum_for_small_windows() {
962+
// When 20% of the window would be smaller than the safety floor
963+
// (16,384 tokens), the floor takes over so tiny windows still
964+
// keep enough headroom for the model response.
965+
let settings = CompressionSettings::new(32_000);
966+
// 20% of 32,000 = 6,400 < 16,384 → floor wins.
967+
assert_eq!(settings.reserve_tokens, 16_384);
968+
assert_eq!(settings.budget(), 32_000 - 16_384);
969+
970+
// Window at or below the floor: reserve equals the floor and
971+
// saturating_sub protects `budget` from underflow.
972+
let tiny = CompressionSettings::new(8_000);
973+
assert_eq!(tiny.reserve_tokens, 16_384);
974+
assert_eq!(tiny.budget(), 0);
975+
}
976+
914977
#[test]
915978
fn should_compress_via_context_size_triggers_when_last_usage_exceeds_budget() {
916979
// The unified `context_size` (= input + output + cache_read +

src-tauri/src/core/goal_manager.rs

Lines changed: 38 additions & 105 deletions
Original file line numberDiff line numberDiff line change
@@ -255,34 +255,6 @@ impl GoalManager {
255255
goal_repo::account_usage(&self.pool, goal_id, tokens, 1).await
256256
}
257257

258-
// ── Auto-resume ──
259-
260-
/// Check if a paused goal should auto-resume when the user sends a new message.
261-
/// Returns Some(()) if the goal was auto-resumed, None if it shouldn't.
262-
pub async fn try_auto_resume(&self) -> Result<bool, AppError> {
263-
let goal = match self.get_active().await? {
264-
Some(g) => g,
265-
None => return Ok(false),
266-
};
267-
268-
if goal.status != GoalStatus::Paused {
269-
return Ok(false);
270-
}
271-
272-
let should_resume = goal
273-
.pause_reason
274-
.as_ref()
275-
.map(|r| r.auto_resume_on_user_message())
276-
.unwrap_or(false);
277-
278-
if should_resume {
279-
goal_repo::update_status(&self.pool, &goal.id, GoalStatus::Active, None, None, None)
280-
.await?;
281-
}
282-
283-
Ok(should_resume)
284-
}
285-
286258
// ── Evaluation ──
287259

288260
/// Evaluate whether the goal should continue, pause, or complete after a turn.
@@ -307,27 +279,12 @@ impl GoalManager {
307279
return verdict;
308280
}
309281

310-
// Completion claim without tool call
282+
// Completion claim without tool call — keep nudging agent toward
283+
// Judge verification; no DB status change. The counter is still
284+
// cleared on tool activity / resume / clear, but reaching 3 no
285+
// longer auto-pauses (status transitions are reserved for user
286+
// commands and Judge verdicts).
311287
if self.detect_completion_claim(response) {
312-
let should_pause = {
313-
let mut guard = self.lock_runtime();
314-
let count = guard
315-
.completion_claim_count
316-
.entry(self.thread_id.clone())
317-
.or_default();
318-
*count += 1;
319-
*count >= 3
320-
};
321-
if should_pause {
322-
// Reset counter before pausing
323-
self.lock_runtime()
324-
.completion_claim_count
325-
.remove(&self.thread_id);
326-
return GoalVerdict::Paused {
327-
reason: PauseReason::IdleBlocked,
328-
detail: Some("agent repeatedly claimed completion without requesting Judge verification via agent_judge".into()),
329-
};
330-
}
331288
return GoalVerdict::ChallengeEvidence;
332289
}
333290

@@ -342,13 +299,12 @@ impl GoalManager {
342299
// Reset idle counters since tools were called
343300
self.reset_idle_counters();
344301

345-
// ── Layer 4: Budget checks ──
346-
if let Some(budget) = goal.token_budget {
347-
if goal.tokens_used >= budget {
348-
return GoalVerdict::BudgetLimited;
349-
}
350-
}
351-
302+
// ── Layer 4: Turn budget + token budget checks ──
303+
// `turns_used >= max_turns` auto-pauses (explicitly approved path).
304+
// `tokens_used >= token_budget` is reported via the `budget_limited`
305+
// verdict string (no DB status change) so the run loop can stop
306+
// continuation. Status transitions are reserved for explicit user
307+
// commands and Judge verdicts.
352308
if goal.turns_used >= goal.max_turns {
353309
return GoalVerdict::Paused {
354310
reason: PauseReason::BudgetExhausted,
@@ -359,6 +315,12 @@ impl GoalManager {
359315
};
360316
}
361317

318+
if let Some(budget) = goal.token_budget {
319+
if goal.tokens_used >= budget {
320+
return GoalVerdict::BudgetLimited;
321+
}
322+
}
323+
362324
// ── Default: continue ──
363325
GoalVerdict::Continue
364326
}
@@ -370,35 +332,23 @@ impl GoalManager {
370332
tool_calls: &[String],
371333
_response: &str,
372334
) -> Option<GoalVerdict> {
373-
for tool_name in tool_calls {
374-
match tool_name.as_str() {
375-
"clarify" => {
376-
return Some(GoalVerdict::Paused {
377-
reason: PauseReason::ClarifyPending,
378-
detail: Some("agent requested clarification".into()),
379-
});
380-
}
381-
"update_plan" => {
382-
return Some(GoalVerdict::Paused {
383-
reason: PauseReason::PlanPending,
384-
detail: Some("agent published a plan, awaiting approval".into()),
385-
});
386-
}
387-
// agent_judge is the main-agent-only acceptance request. It is
388-
// handled by the tool execution pipeline (execute_judge_tool),
389-
// which runs the Judge and records the verdict. Evaluation must
390-
// not treat it as a blocking tool — like any tool call it shows
391-
// the agent acted and should reset idle tendencies.
392-
_ => {}
393-
}
394-
}
335+
// Tool-based auto-pausing has been removed: status transitions are
336+
// reserved for explicit user commands and Judge verdicts. `clarify`
337+
// and `update_plan` no longer flip the goal to paused; they fall
338+
// through to the continuation path. `agent_judge` is the main-agent
339+
// acceptance request and is handled by the tool execution pipeline
340+
// (execute_judge_tool), which runs the Judge and records the
341+
// verdict — it is never a blocking tool here.
342+
let _ = tool_calls;
395343
None
396344
}
397345

398346
fn detect_idle_block(&self, response: &str) -> Option<GoalVerdict> {
399347
let idle_count = self.increment_idle_count();
400-
let trimmed = response.trim().to_lowercase();
401-
348+
// Heuristic question detection has been removed; status transitions
349+
// are reserved for explicit user commands and Judge verdicts, so idle
350+
// detection is purely a turn-count trigger. `response` is unused.
351+
let _ = response;
402352
if idle_count >= MAX_IDLE_TURNS {
403353
return Some(GoalVerdict::Paused {
404354
reason: PauseReason::IdleBlocked,
@@ -407,31 +357,6 @@ impl GoalManager {
407357
)),
408358
});
409359
}
410-
411-
// Lightweight heuristic: short question-like response + no tools
412-
if idle_count >= 2 {
413-
let blockers = [
414-
"should i",
415-
"do you want",
416-
"would you like",
417-
"请确认",
418-
"需要你决定",
419-
"which approach",
420-
"which option",
421-
"can you confirm",
422-
"let me know if",
423-
"before i proceed",
424-
"你的选择是",
425-
"你确认吗",
426-
"需要你同意",
427-
];
428-
if trimmed.len() < 500 && blockers.iter().any(|b| trimmed.contains(b)) {
429-
return Some(GoalVerdict::Paused {
430-
reason: PauseReason::IdleBlocked,
431-
detail: Some("agent appears blocked, may need user input".into()),
432-
});
433-
}
434-
}
435360
None
436361
}
437362

@@ -613,7 +538,15 @@ impl GoalManager {
613538
.await?;
614539
}
615540
GoalVerdict::BudgetLimited => {
616-
self.mark_budget_limited(&current.id).await?;
541+
// Advisory: token budget exhausted — do NOT write to DB.
542+
// The verdict string still propagates as "budget_limited" so
543+
// the run loop can stop continuation. Goal status remains
544+
// `active` and is only changed by explicit user commands
545+
// (`/goal budget-limit`) or Judge verdicts.
546+
tracing::info!(
547+
goal_id = %current.id,
548+
"token budget exhausted: emitting budget_limited verdict without DB status change"
549+
);
617550
}
618551
}
619552

src-tauri/src/core/prompt/templates/active_goal.tpl.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,10 @@ Turns used: {{turns_used}}/{{max_turns}}
1111
**Completion is decided by independent verification — you cannot self-declare it.**
1212
1. Every subtask implied by the objective must be done, with no remaining work or dangling follow-ups.
1313
2. Verify your work by running the relevant tests, linters, or build commands as you go.
14-
3. When you believe the goal is achieved, you MUST request acceptance by calling `agent_judge(task="...")`.
14+
3. When you believe the goal is achieved, you MUST request acceptance by calling `agent_judge()`.
1515

1616
Rules:
17-
- Call `agent_judge(task="explain why you believe the goal is achieved / what to verify")` when you think the goal is complete. An independent Judge will evaluate the project against the goal's consistency and completeness.
17+
- Call `agent_judge()` to request independent goal acceptance verification. An independent Judge will evaluate the project against the goal's completeness. You do not need to provide a self-assessment — the Judge evaluates the project state directly.
1818
- The goal is only marked verified when the Judge returns passed=true. You cannot mark the goal complete yourself.
1919
- If a Judge verification did not pass, read its findings, fix each one, then call `agent_judge` again.
2020
- Once the goal has passed Judge acceptance, stop making further changes and summarize the result.

0 commit comments

Comments
 (0)