Explore SGI (Semantic Grounding Index) for agent response quality scoring

[rstagi]

Context

Paper: Semantic Grounding Index: Geometric Bounds on Context Engagement in RAG Systems (Marín, 2025)

Key Concept

Semantic Grounding Index (SGI) measures whether an LLM response actually uses retrieved context or just paraphrases the question ("semantic laziness").

SGI = angular_distance(response, question) / angular_distance(response, context)

• SGI > 1: Response moved toward context (good grounding)
• SGI < 1: Response stayed near question (potential hallucination)

Empirical Results

• HaluEval (n=5000): Cohen's d = 0.92-1.28, AUC = 0.72-0.83
• Cross-model correlation: r = 0.85 (robust across embedding models)
• Calibration: ECE = 0.10 (scores usable as probabilities)
• Best performance: Long responses (d=2.05), short questions (d=1.22)

Critical Limitation

⚠️ TruthfulQA AUC = 0.478 — SGI measures topical engagement, NOT factual accuracy.

Relevance to Agentified

Metric	What it measures
Hit Rate	Did we retrieve the RIGHT context?
SGI	Did the agent actually USE the context?
Factual accuracy	Is the answer TRUE?

SGI fills a gap: we can detect when an agent ignores retrieved tool outputs and hallucinates instead.

Proposed Integration

1. Benchmark extension: Add SGI as secondary metric alongside Hit Rate
2. Re-ranker signal: SGI could inform V2 XGBoost model (contexts that lead to high SGI responses = better)
3. Production monitoring: Flag agent responses with low SGI for human review

Implementation Notes

• Computationally cheap: just embedding + cosine similarity operations
• Works with any embedding model (cross-model r=0.85)
• Integrates naturally with existing embedding infrastructure

Next Steps

• Prototype SGI calculation in benchmark harness
• Analyze correlation between Hit Rate and SGI on existing benchmark data
• Evaluate as feature in V2 re-ranker

/cc @rstagi