Harden semantic verification actions

Author: devjoe

docs/plans/claim-investigation-research.md

@@ -1068,20 +1068,32 @@ verbatim ordered atom parts, selects only one consequential proposition, and
 adds explicit low-risk abstention guidance. The local guard remains strict and
 now exposes precise structure reason codes for private development audits.
 
-The Adapter may now make one reason-specific semantic repair inside the same
-low-priority derived queue job, but only after a prepared output fails one of a
-small set of repairable local-guard reasons. The retry receives the rejected
-claim and reason, must rebuild from exact grounding text or abstain, and cannot
-bypass the unchanged guard. Non-consequential and unsupported outputs are not
-retried. The UI receives `prepared` only after the final claim forms the same
-validated action task; there is no extra foreground job or persisted content.
+An evaluation-only bounded semantic repair was then tested against the same old
+30-row development slice. Eight repair requests produced zero accepted actions:
+seven remained rejected by the unchanged guard and one ended in a format
+failure. The apparent four-to-seven increase between separate runs came from
+first-pass model variance, not from repair recovery. Product runtime therefore
+returns to one Adapter attempt. `semantic_once` remains available only as an
+explicit private diagnostic mode; the preregistered fresh audit requires
+`repairMode=none` so its execution path matches runtime.
+
+Independent review of the seven first-pass actions found five usable or
+usable-with-tightening results, one underspecified comparison, and one false
+action derived from a truncated related-link headline at the extraction tail.
+The candidate now rejects labeled navigation sections and incomplete
+tail-boundary fragments, rejects comparisons without a time, market or region,
+and metric, and requires a named evidence family rather than generic
+`evidence`. Regular Google search uses bounded claim/evidence anchors instead
+of copying the whole claim or publisher name. Self-contained follow-up
+questions no longer inherit an unrelated model summary; AI Mode alone may
+receive sanitized page metadata and URL.
 
 The private runner also verifies that each source hash is derived from the
 actual input text, refuses to overwrite an evaluation path, and records exact
 input and result digests. These changes improve evaluation integrity; they do
-not authorize the Agent action or establish release-level coverage. The same
-old development replay must improve before a preregistered fresh cohort is
-opened, and fresh rows cannot be used for further tuning.
+not authorize the Agent action or establish release-level coverage. The
+updated old-development replay is the final tuning check before opening a
+preregistered fresh cohort; fresh rows cannot be used for further tuning.
 
 ### C. Sufficiency and UX audit
 

docs/plans/general-page-reader.md

@@ -818,6 +818,14 @@ next candidate gate closed; no fresh holdout should be created yet.
   interactive skeleton elements, no scroll movement, and a complete ready
   first frame. Generated screenshots and measurements remain gitignored under
   `tmp/`.
+- An old-30 product-semantic replay showed that eight reason-specific Adapter
+  retries yielded no accepted actions. Runtime now performs one low-priority
+  Adapter attempt only; the private runner exposes repair solely through an
+  explicit diagnostic flag and records `repairMode` in run metadata. Local
+  action guards also reject incomplete navigation-tail quotes,
+  under-specified comparisons, and generic evidence requirements. Regular
+  Google search receives bounded claim/evidence anchors, while page metadata
+  and URL remain AI-Mode-only context.
 
 ## Verification Gates
 

scripts/lib/private-general-page-semantic-audit.mjs

@@ -3,6 +3,7 @@ import fs from "node:fs";
 import path from "node:path";
 
 export const PRIVATE_SEMANTIC_AUDIT_CORE_FILES = [
+  "src/background/general-page-investigation-background.ts",
   "src/lib/general-page-analysis.ts",
   "src/lib/general-page-investigation-adapter.ts",
   "src/lib/reading-question-policy.ts",
@@ -15,6 +16,15 @@ export function sha256Text(value) {
   return crypto.createHash("sha256").update(String(value)).digest("hex");
 }
 
+export function privateSemanticAuditRepairMode(argv) {
+  const index = argv.indexOf("--repair-mode");
+  const value = index >= 0 ? argv[index + 1] : "none";
+  if (value !== "none" && value !== "semantic_once") {
+    throw new Error("--repair-mode must be none or semantic_once");
+  }
+  return value;
+}
+
 export function semanticAuditCompletionsUrl(endpoint) {
   const url = new URL(endpoint);
   if (!/^https?:$/.test(url.protocol) || url.username || url.password || url.search || url.hash) {

scripts/private-general-page-semantic-audit-entry.ts

@@ -25,6 +25,7 @@ import {
   assertPrivateSemanticAuditFetchTarget,
   hashPrivateSemanticAuditCoreFiles,
   installPrivateSemanticAuditNetworkGuard,
+  privateSemanticAuditRepairMode,
   semanticAuditCompletionsUrl,
   sha256Text,
 } from "./lib/private-general-page-semantic-audit.mjs";
@@ -131,6 +132,7 @@ const declaredCategories = required("--data-categories");
 const expectedCount = Number(required("--sample-count"));
 const concurrency = Math.max(1, Math.min(4, Number(option("--concurrency", "2")) || 2));
 const timeoutMs = Math.max(1_000, Math.min(120_000, Number(option("--timeout-ms", "45_000")) || 45_000));
+const repairMode = privateSemanticAuditRepairMode(process.argv);
 if (split !== "dev" && split !== "holdout") throw new Error("--split must be dev or holdout");
 
 const allowedCompletionsUrl = semanticAuditCompletionsUrl(endpoint);
@@ -297,7 +299,7 @@ async function evaluateRow(row: InputRow): Promise<Record<string, unknown>> {
       source: row.sourceContext,
     }) : undefined;
     const firstReason = firstEligibility.ok ? (firstTask ? undefined : "invalid_question") : firstEligibility.reason;
-    if (firstReason && isRepairablePageClaimIneligibilityReason(firstReason)) {
+    if (repairMode === "semantic_once" && firstReason && isRepairablePageClaimIneligibilityReason(firstReason)) {
       repairReason = firstReason;
       const repairRequest: TierBGeneralPageInvestigationAdapterRequest = {
         ...adapterRequest,
@@ -445,6 +447,10 @@ const manifest = {
     timeoutMs,
     concurrency,
   },
+  adapter: {
+    repairMode,
+    runtimeParity: repairMode === "none",
+  },
   data: {
     sampleCount: rows.length,
     declaredCategories: declaredCategories.split(",").map((value) => value.trim()).filter(Boolean).sort(),
@@ -486,6 +492,7 @@ console.log(JSON.stringify({
   result: results.every((result) => result.ok) ? "pass" : "partial",
   runId,
   split,
+  repairMode,
   samples: rows.length,
   ...manifest.counts,
   publicSearchRequests: 0,

src/background/general-page-investigation-background.ts

@@ -15,8 +15,6 @@ import {
 } from "./model-work-scheduler";
 import {
   buildPageClaimInvestigationTask,
-  isRepairablePageClaimIneligibilityReason,
-  pageClaimInvestigationEligibility,
 } from "../sidepanel/page-claim-investigation";
 
 export interface ScheduleGeneralPageInvestigationPreparationOptions {
@@ -86,27 +84,11 @@ export function scheduleGeneralPageInvestigationPreparation(
     priority: "derived",
     dedupeKey: id,
     supersedeKey: `general-page-investigation:${request.tabId}:${request.scope}`,
-    run: async () => {
-      const first = await callAdapter(adapterRequest);
-      const firstClaim = first.ok && first.value?.decision === "prepared" ? first.value.claim : undefined;
-      if (!firstClaim) return first;
-      const eligibility = pageClaimInvestigationEligibility(firstClaim, request.context.mainText);
-      const task = eligibility.ok ? buildPageClaimInvestigationTask({
-        analysisKey: request.analysisKey,
-        scope: request.scope,
-        claimIndex: 0,
-        claim: firstClaim,
-        groundingText: request.context.mainText,
-        source,
-      }) : undefined;
-      const repairReason = eligibility.ok ? (task ? undefined : "invalid_question") : eligibility.reason;
-      if (!repairReason || !isRepairablePageClaimIneligibilityReason(repairReason)) return first;
-      return callAdapter({
-        ...adapterRequest,
-        candidateClaim: firstClaim,
-        repairReason: repairReason as TierBGeneralPageInvestigationAdapterRequest["repairReason"],
-      });
-    },
+    // Semantic repair remains available to the private evaluation harness, but
+    // runtime intentionally performs one adapter attempt only. The old-30
+    // review found no accepted repair, so retrying here added latency without
+    // producing a trustworthy user action.
+    run: () => callAdapter(adapterRequest),
   });
 
   void work.then((result) => {

src/lib/general-page-investigation-adapter.ts

@@ -15,7 +15,7 @@ export interface GeneralPageInvestigationAdapterInput {
   groundingText: string;
   source?: GeneralPageInvestigationSourceMetadata;
   outputLang?: Lang;
-  /** One bounded retry after the unchanged local guard rejects a prepared claim. */
+  /** Evaluation-only bounded retry. Product runtime never issues this request. */
   repairReason?: "atom_span_mismatch" | "compound_claim" | "vague_atom" | "generic_subject" |
     "ungrounded_atom" | "missing_attribution" | "invalid_attribution" | "invalid_question";
 }
@@ -187,12 +187,15 @@ export function buildGeneralPageInvestigationAdapterSystemPrompt(outputLang?: La
     "Use decision=prepared and reason=actionable only when the supplied page text supports one consequential, externally checkable atomic assertion.",
     "A prepared claim must contain c, why, need, q, atom:{s,p,o}, policy:{claimKind,consequence}, and sourceQuote; attribution:{source,relation,modality} is allowed only for a real outer source frame.",
     "sourceQuote must be one concise verbatim span copied from Exact grounding text that directly supports c. Preserve its source language and do not translate it.",
+    "Never prepare an action from a related or recommended link, navigation-tail headline, or incomplete fragment touching the Exact grounding text boundary; abstain instead.",
     "atom.s, atom.p, and atom.o must each be exact substrings of c, appearing once in that order. Never paraphrase, shorten, translate, or recombine an atom part.",
     "c must end with sentence punctuation and contain exactly one proposition. If the candidate is compound, select only one consequential proposition that the sourceQuote supports; otherwise abstain.",
     "If attribution is present, modality must be statement|report|estimate|allegation|forecast|analysis. Omit attribution when uncertain; never invent another modality.",
     "Source metadata alone is never claim attribution. Add attribution only when claim c itself contains a verbatim source and reporting relation outside atom s, p, and o; attribution source and relation must both be exact substrings of c.",
     "Do not use generic atom subjects such as death toll, number, report, officials, government, company, or agency. Include the event, place, organization, or other identifier already present in Exact grounding text, or abstain.",
     "Keep one proposition and preserve legal stage and attribution exactly. q must be one natural question containing the exact source-language s, p, and o.",
+    "For a comparative claim, require the grounding text to name the comparison scope (time plus region or market) and measurement metric; otherwise abstain.",
+    "need must name a named evidence family that could answer q, such as an official notice, registry record, court ruling, dataset, benchmark report, or result table. Never write only evidence, sources, data, or proof.",
     "policy.claimKind is fact|report|estimate|forecast|allegation|expert_analysis. policy.consequence is health|safety|money|rights|law|public_interest.",
     "Abstain for low-risk product availability or promotion, celebrity purchases or anecdotes, vague AI or marketing claims, pure opinion, generic controversy, or any assertion without a consequential externally checkable proposition.",
     "Otherwise output decision=abstain with reason=insufficient_context|unsafe_structure|non_consequential|unsupported_claim and omit claim.",
@@ -210,7 +213,7 @@ export function buildGeneralPageInvestigationAdapterPrompt(input: GeneralPageInv
   };
   return [
     ...(input.repairReason ? [
-      "This is the single allowed semantic repair attempt. The previous prepared claim failed the unchanged local guard.",
+      "This is the single allowed semantic repair attempt in an evaluation-only audit. Product runtime does not issue repair requests. The previous prepared claim failed the unchanged local guard.",
       `Local guard reason: ${input.repairReason}. Rebuild from Exact grounding text or abstain; never work around the guard.`,
     ] : []),
     "Prepare or abstain. URL is metadata only; it is not evidence.",