feat(bench): Longitudinal Agent Evaluation — MemoryArena integration + coding agent scenarios

[novaspectrayan]

Summary

Adopt longitudinal agent evaluation frameworks (MemoryArena, STATE-Bench) and build Spector-native multi-session scenarios to prove that Spector's cognitive memory stack improves real agent behavior — not just retrieval quality.

Context & Motivation

Origin: Gork Critique Deep Analysis — P0 recommendation #2
Full Plan: Longitudinal Agent Evaluation Plan

The Gap

Spector's current evaluation infrastructure (CognitiveBenchmarkHarness, SubsystemAblationRunner) measures retrieval quality — nDCG@10, MRR, Recall@K. Results are impressive: cognitive nDCG@10 = 0.3183 vs baseline 0.1310 (+143%, Cohen's d = 0.59).

But retrieval quality ≠ agent intelligence. The unanswered question:

"Does an agent using Spector Memory make fewer mistakes, maintain consistency, and build on prior work better over 30 days of real sessions than an agent using a simple vector store?"

This is the single biggest gap in Spector's evidence base. Without it, the cognitive story remains "plausible but unproven."

Why Now

• MemoryArena (ICML 2026, Stanford/UCSD) — open-source framework for evaluating memory in multi-session interdependent tasks. Key finding: agents near-perfect on static recall benchmarks fail badly on agentic tasks.
• STATE-Bench (Microsoft) — 450 enterprise tasks measuring task completion, pass@1, UX scores. Has an "Agent Learning Track" for testing whether memory helps agents improve.
• The industry is moving from retrieval metrics → agent outcome metrics. Spector must participate or be measured by others on their terms.

Implementation Phases

Phase 1: MemoryArena Integration (Weeks 1–4)

Goal: Plug Spector as a memory backend in MemoryArena and run comparative evaluation.

• Clone MemoryArena repo, study memory backend interface
• Build SpectorMemoryArenaAdapter implementing MemoryArena's memory backend SPI:
  • remember() → Spector memory_remember (ingest)
  • recall() → Spector memory_recall (query)
  • reinforce() → Spector memory_reinforce (valence feedback)
  • forget() → Spector memory_forget (suppress)
• Configure 5 experimental conditions:
  • spector-full (CognitiveProfile = ALL)
  • spector-core (CognitiveProfile = CORE_ONLY)
  • spector-minimal (vector + basic importance, no graphs/consolidation)
  • baseline-vector (ChromaDB or equivalent)
  • baseline-context (raw context window, no external memory)
• Run all 4 MemoryArena domains (shopping, travel, search, reasoning)
• Statistical analysis (paired t-test, Wilcoxon, Cohen's d via existing StatisticalTests)
• Publish results in spectrayan/RnD/longitudinal-memoryarena-results.md

Success criteria: ≥1 Spector cognitive profile statistically significantly outperforms simple vector baseline on ≥2/4 domains on task completion (not retrieval).

Phase 2: Spector-Native Coding Agent Scenario (Weeks 3–6)

Goal: Build a multi-session coding agent evaluation testing Spector's MCP-native memory in realistic development workflows.

• Design 20-session scenario across 14 simulated days
  • Sessions: project setup, feature dev, debugging, code review, architecture decisions
  • Memory challenges: recall past decisions, remember preferences, avoid bug re-introduction
• Build CodingAgentEvalRunner in spector-bench
• Define ground-truth expected behaviors for each session
• Implement metrics: preference consistency, architecture recall accuracy, bug re-introduction rate, context coherence
• Run comparative experiments (same LLM + tools, different memory backends)
• Publish results

Success criteria: Measurable reduction in bug re-introduction and preference contradictions with Spector cognitive profiles vs baselines.

Phase 3: Cognitive Subsystem ROI Analysis (Weeks 5–8)

Goal: Use longitudinal data to determine which cognitive subsystems actually improve agent outcomes.

• Run longitudinal ablations: full stack vs no-Hebbian vs no-temporal vs no-consolidation vs no-valence
• Compute per-subsystem contribution to agent task completion (not just retrieval nDCG)
• Compare with existing retrieval ablation data:
  • Importance scoring: 35.2% retrieval contribution → ?% agent outcome contribution
  • Hebbian graphs: 4.1% retrieval → ?% agent outcome
  • Temporal chains: 1.5% retrieval → ?% agent outcome
• Decision: justify, keep, or demote subsystems based on evidence
• Update default CognitiveProfile recommendation

Success criteria: Clear per-subsystem ROI ranking. ≥1 subsystem confirmed high-value AND ≥1 identified as demotion candidate.

Primary Metrics

Metric	Description
Task Completion Rate	% of multi-session tasks successfully completed
Cross-Session Consistency	Contradiction rate across session boundaries
Preference Stability	% of stated preferences honored across sessions
Error Non-Repetition	Resolved bug/error re-introduction count
Context Coherence	LLM-as-judge coherence score over 10+ sessions
Memory Growth Rate	Storage growth per session (consolidation effectiveness)
Cost Per Task	LLM tokens + memory operations per completed task

New Files

spector/bench/spector-bench/src/main/java/
  com/spectrayan/spector/bench/longitudinal/
    ├── MemoryArenaAdapter.java          # MemoryArena memory backend implementation
    ├── MemoryArenaRunner.java           # Orchestrates MemoryArena domain runs
    ├── CodingAgentEvalRunner.java       # Multi-session coding agent harness
    ├── LongitudinalAblationRunner.java  # Ablation studies on agent outcomes
    ├── LongitudinalMetrics.java         # Agent outcome metrics computation
    └── ConsistencyDetector.java         # Cross-session contradiction detection

spector/bench/spector-bench/src/main/resources/
  datasets/
    └── coding-agent-longitudinal/
        ├── manifest.json                # Scenario definition
        ├── sessions/                    # 20 session scripts
        └── ground-truth/               # Expected behaviors per session

Effort Estimate

• Phase 1 (MemoryArena): 4 weeks
• Phase 2 (Coding Agent): 3 weeks (overlaps Phase 1)
• Phase 3 (Subsystem ROI): 3 weeks (depends on Phase 1+2 data)
• Total: ~8 weeks end-to-end

References

• MemoryArena (ICML 2026) — multi-session agent evaluation framework
• STATE-Bench (Microsoft) — enterprise task agent evaluation
• Longitudinal Evaluation Plan
• Gork Critique Analysis
• Existing ablation results

[titanspectrayan]

🏛️ Architectural Decision Record (ADR): Longitudinal Agent Evaluation Architecture (#482)

Status: APPROVED by @titan (Solutions Architect) & @jarvis (CTO)

1. Context & Architectural Goals

Current evaluation metrics (nDCG@10, MRR) measure static memory retrieval, but fail to measure downstream agent task completion, preference stability, and bug avoidance over multi-session horizons. To address this gap (identified in the Gork critique analysis), we are introducing a longitudinal agent evaluation framework in spector-bench.

2. Decision: Dual-Tier MemoryArena Architecture

We adopt a Dual-Tier Adapter Strategy to bridge MemoryArena (Python benchmark suite, ICML 2026) with Spector Memory Engine (Java):

1. Python Bridge (spector_memoryarena_adapter.py): Subclasses MemoryArena's Python BaseMemoryBackend and interacts with synapse/spector-mcp via stdio MCP tool calls (memory_remember, memory_recall, memory_reinforce, memory_forget).
2. Java-Native Harness (MemoryArenaAdapter.java & MemoryArenaRunner.java): In-process Java adapter inside spector-bench executing benchmark trajectories directly against SpectorMemoryEngine for fast, zero-IPC-overhead statistical ablation runs inside Maven test suites.

3. Module Boundaries & Class Structure

Target Package: com.spectrayan.spector.bench.longitudinal in spector-bench module.

• MemoryArenaAdapter.java: Maps MemoryArena lifecycle calls to SpectorMemoryEngine API & CognitiveProfile variants.
• LongitudinalSession.java: Immutable Java record representing a single multi-turn session.
• LongitudinalTrajectory.java: Immutable Java record representing a multi-day scenario trajectory.
• LongitudinalMetrics.java: Immutable Java record computing outcome metrics (task completion rate, cross-session consistency, preference stability, error non-repetition, context coherence).
• ConsistencyDetector.java: Contradiction & preference drift detection engine.
• MemoryArenaRunner.java: Benchmark orchestrator running baseline comparisons (spector-full, spector-core, spector-minimal, baseline-vector, baseline-context).

4. Quality & Governance Gates

• All model objects must use Java 21 records.
• Absolute thread safety using immutable data structures.
• Automated regression suite via JUnit 5 + AssertJ running on mini-longitudinal datasets.