Outcome Is Not Verification: Auditing Hidden-State Verifiers with Counterfactual Local Validity

Ali Uyar Independent Researcher

Paper title: Outcome Is Not Verification: Auditing Hidden-State Verifiers with Counterfactual Local Validity

This repository accompanies a measurement-first audit of hidden-state verifiers in structured reasoning. It studies whether a model's internal representation can support strong outcome readout (predicting whether a trace ends correctly) while still failing the stricter test of process verification (recognising whether the current step is locally valid for the current world). The audit is instantiated in two executable domains — arithmetic DAG execution and grounded Horn-style logic — and centred on counterfactual local validity, where the same rendered step can be valid in one world and invalid in a matched counterfactual world.

Abstract

Hidden states can support strong outcome readout without thereby supporting reliable process verification. We make that distinction operational with an audit centered on counterfactual local validity: whether the same step remains valid when the underlying world changes. We instantiate the audit in two structured reasoning domains — arithmetic DAG execution and grounded Horn-style logic — using paired worlds, stored gold/recoverable/doomed traces, and strict same-step flips that hold the current-step text fixed. To isolate the target variable rather than the architecture, we keep the hidden-state extractor family and scorer class fixed and compare only supervision targets: final trace outcome, prefix outcome, local validity, and a flip-rank variant. Across domains, hidden states provide meaningful trace-level outcome readout, but outcome-supervised step scoring does not behave like process verification. Under the same canonical step pools, LocalValidity improves the strongest audit axes, especially strict same-step FlipAcc and PVS. The gains are not uniform: repaired arithmetic scale remains mixed on first-invalid-step localization, particularly on held-out renderer C, while logic scale remains strongly favorable to LocalValidity even on the same harder transfer slice. The resulting picture is not that local-validity supervision wins everything, but that counterfactual local validity exposes a real outcome/process gap while showing that process-verification evidence is metric-dependent and transfer-sensitive even in structured reasoning.

Main Finding

The central result is not that local-validity supervision wins uniformly on every metric. The stronger claim is: hidden states support strong outcome readout, but process-verification evidence is metric-dependent and transfer-sensitive even in structured reasoning.

Core pooled results at scale (Qwen3-4B-Instruct, teacher-forced, fixed scorer class):

Domain	TraceAUROC	FlipAcc (Prefix)	FlipAcc (Local)	PVS (Prefix)	PVS (Local)	Loc. (Prefix)	Loc. (Local)
Arithmetic	0.625	0.845	0.864	0.679	0.730	0.669	0.626
Logic	0.972	0.995	1.000	0.697	0.975	0.831	0.961

LocalValidity supervision improves the strongest audit axes — strict same-step FlipAcc and PVS — in both domains. First-invalid-step localization is the harder transfer test: it improves strongly in logic but remains mixed in repaired arithmetic, especially on held-out renderer C.

Contributions

1. We formalize the distinction between outcome readout and process verification for hidden-state verifiers.
2. We introduce counterfactual local validity and strict same-step flips as step-level audit objects that control current-step text while changing the world.
3. We provide paired arithmetic and grounded-Horn audit settings with gold, recoverable, doomed, and held-out-renderer slices.
4. Empirically, we show that hidden states support strong outcome readout, but the strongest evidence for process verification comes from FlipAcc and PVS; localization is a harder transfer test whose behaviour is domain-sensitive rather than guaranteed.

Scope

This audit is intentionally narrow and claim-safe.

• one fixed hidden-state extractor family: teacher-forced pooled last-four-layer states from Qwen3-4B-Instruct
• one fixed scorer class: linear scorers
• two structured reasoning domains: arithmetic DAG execution and grounded Horn-style logic
• one canonical step-instance universe across step-level scorers; one stored trace pool for the trace-level baseline
• structured labels computed from executor state, not from natural-language text parsing

The contribution is not breadth. It is a controlled audit object — counterfactual local validity — plus an honest report that the evidence for process verification is metric-dependent and transfer-sensitive.

Paper

• Compiled PDF: paper/tmlr_submission/OutcomeIsNotVerification_TMLR.pdf
• LaTeX source: paper/tmlr_submission/main.tex
• Tables and figures: paper/tmlr_submission/tables/, paper/tmlr_submission/figures/
• TMLR submission archive: paper/OutcomeIsNotVerification_TMLR_submission.zip

Repository Layout

• paper/ — TMLR manuscript, figures, tables, bibliography, and compiled PDF
• common/ — shared config loading, path handling, and manifest utilities
• data/ — dataset construction, schema libraries, record contracts, and QC
• models/ — step embedding extraction and linear scorers
• eval/ — metrics, main evaluation, and report generation
• configs/ — dataset, model, and report configurations
• tests/ — focused tests for invariants, extraction, metrics, and reporting
• docs/ — supplementary methodological notes

Generated artifacts land under artifacts/ and are intentionally excluded from version control.

Citation

@unpublished{uyar2026outcome,
  author = {Uyar, Ali},
  title  = {Outcome Is Not Verification: Auditing Hidden-State Verifiers with Counterfactual Local Validity},
  year   = {2026},
  note   = {Independent research}
}