Skip to content

Commit 8a5c044

Browse files
riccajasonclaude
andcommitted
ci: drop the hardcoded STATUS.md test count so it can't redden CI
STATUS.md pinned a unit-test total ("903 host unit tests", "Total: **903**") that only `cargo test --lib` can derive. Every time the suite grew, the prose went stale and check-status-freshness --full --strict failed CI until someone hand-edited the number — most recently 892 -> 903, the drift the preceding unblock commit had to repair. The number's own prose already said "numbers live in code, not prose"; this makes the file honor that instead of pinning a copy that drifts. What: - STATUS.md: remove the two hardcoded test totals. "At a glance" and the Test-coverage section now point at `make stats` / `cargo test --lib` for the live count and state explicitly that it is not pinned in prose. - tools/check-status-freshness.py: remove the unit-test number-drift check entirely — the TEST_COUNT_RES/MAKE_STATS_TESTS_RE regexes, the intra-prose consistency check, the `--full`/`make stats` live-compare path, and the `subprocess` import. The two checks that remain are the date heuristic (advisory) and the Kani-harness total (cheap, no compile). `--strict` now fails only on Kani-harness drift. Docstring + `--full` flag removed. - .github/workflows/ci.yml: the freshness gate no longer needs a built test binary, so it moves out of the `test` job (where it ran `--full --strict` after `cargo test`) into the pure-python `lints` job as a plain `--strict` step. The STATS_TEST_TARGET env it needed is gone with it. - Makefile: drop the now-dead `check-status-freshness-full` target. Why: a derived number does not belong hardcoded in prose. With nothing pinned, there is nothing to drift, and the compile-gated `--full` path that existed only to re-verify that number has no remaining purpose. The Kani-harness total stays pinned-and-checked because it is derivable without a compile. Out of scope: the per-area breakdown table in the Test-coverage section keeps its approximate counts (already labelled "breakdown drifts faster than the total"); only the authoritative total is dropped. STATUS.md's stale last_synced_to_code date is a separate advisory and is not touched here. Verification: - python3 tools/check-status-freshness.py: exit 0 (date advisory only). - python3 tools/check-status-freshness.py --strict: exit 0 (no Kani drift; this is the exact CI invocation). - python3 -m py_compile tools/check-status-freshness.py: OK; grep confirms no remaining TEST_COUNT_RES / MAKE_STATS_TESTS_RE / subprocess / args.full. - make check-status-freshness: exit 0 (Makefile wrapper still resolves). - make check-deferrals / check-assumptions / check-index-isolation: all exit 0; neither baseline tracks the edited files, so no line-drift resync. - grep: no orphan references to check-status-freshness-full remain. Staged files: .github/workflows/ci.yml Makefile STATUS.md tools/check-status-freshness.py Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 5599ea2 commit 8a5c044

4 files changed

Lines changed: 42 additions & 98 deletions

File tree

.github/workflows/ci.yml

Lines changed: 6 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -90,15 +90,6 @@ jobs:
9090
env:
9191
RUST_MIN_STACK: "8388608"
9292
run: cargo test --lib --target x86_64-unknown-linux-gnu
93-
# STATUS.md number-drift gate. The test binary is now built, so
94-
# `make stats` enumerates --lib tests cheaply (retargeted to the Linux
95-
# host via STATS_TEST_TARGET). --strict makes objective count drift
96-
# (prose vs build, Kani harnesses vs source) FAIL CI; the date heuristic
97-
# stays advisory and is not checked here.
98-
- name: check-status-freshness (counts vs build, --full --strict)
99-
env:
100-
STATS_TEST_TARGET: x86_64-unknown-linux-gnu
101-
run: python3 tools/check-status-freshness.py --full --strict
10293

10394
# ---------------------------------------------------------------------------
10495
# clippy — enforces missing_safety_doc = "deny" (Cargo.toml [lints.clippy],
@@ -130,9 +121,10 @@ jobs:
130121
# Commit-boundary / provenance gates are excluded on purpose: they read the
131122
# staged diff or session state and are no-ops or meaningless in CI —
132123
# check-index-isolation, check-lockfile (advisory), claude-preflight.
133-
# (check-status-freshness's number-drift check runs in the `test` job with
134-
# --full --strict, where a built test binary makes the counts verifiable; its
135-
# date heuristic stays local/advisory and is not run in CI.)
124+
# check-status-freshness runs here too (Kani-harness drift, --strict): it is
125+
# compile-free now that STATUS.md no longer pins a unit-test total, so it
126+
# belongs with the other pure-python gates. Its date heuristic stays
127+
# local/advisory and is not run in CI.
136128
#
137129
# Note: check-adrs regenerates docs/adr/INDEX.md as a side effect (exit 0 when
138130
# cross-refs are valid). CI does not commit, so a stale committed INDEX.md is
@@ -155,3 +147,5 @@ jobs:
155147
run: make check-banned-paths
156148
- name: check-unsafe-coverage (Convention 1 SAFETY comments)
157149
run: make check-unsafe-coverage
150+
- name: check-status-freshness (Kani-harness drift, --strict)
151+
run: python3 tools/check-status-freshness.py --strict

Makefile

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1310,13 +1310,6 @@ check-lockfile:
13101310
check-status-freshness:
13111311
@python3 tools/check-status-freshness.py
13121312

1313-
# Full variant: also re-derives the unit-test count via `make stats`
1314-
# (compiles). For CI / manual runs — NOT the pre-commit hook, which must
1315-
# stay compile-free. The cheap Kani-harness + prose-consistency checks
1316-
# run in both; only the live test-count compare is gated behind --full.
1317-
check-status-freshness-full:
1318-
@python3 tools/check-status-freshness.py --full
1319-
13201313
# Pre-edit audit for a file. Run before the first Edit on any file per
13211314
# session — especially under parallel-thread development. Surfaces
13221315
# uncommitted changes (HEAD commit, status, full diff) so the caller

STATUS.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ belongs in the linked ADR, not here.
1919
## At a glance
2020

2121
- **Tri-arch first-class**: clean release build on `x86_64`, `aarch64`, `riscv64gc`; all three boot in QEMU to an `cambios>` shell prompt. `make check-all` is the permanent regression gate.
22-
- **903 host unit tests passing** on `x86_64-apple-darwin`. Run `make stats` for current counts — numbers live in code, not prose.
22+
- **Host unit tests passing** on `x86_64-apple-darwin`. Run `make stats` for the current count — numbers live in code, not prose.
2323
- **Security model live end-to-end**: cryptographic identity, signed-ELF verification, capability-gated IPC, content-addressed ObjectStore, audit ring, kernel identity gate, userspace `recv_verified`.
2424
- **GUI stack live on x86_64 + aarch64**: scanout-virtio-gpu drives QEMU virtio-gpu-pci; compositor composites; virtio-input forwards HID keyboard/pointer events into the focused window; first-party app `pong` (continuous-motion 1-player vs AI) runs as the default GUI boot module on x86_64, and `worm` renders on aarch64 via `make run-aarch64-gui` now that the kernel has an ECAM-based PCI enumerator. `tree` (Minesweeper) stays buildable for regression.
2525
- **Persistent storage live**: virtio-blk + disk-backed ObjectStore + `arcobj` shell CLI; objects survive reboot.
@@ -204,7 +204,7 @@ Parity-target with x86_64 / AArch64. All phases landed as of 2026-04-19. Source:
204204

205205
## Test coverage
206206

207-
Total: **903** on `x86_64-apple-darwin`. Run `RUST_MIN_STACK=8388608 cargo test --lib --target x86_64-apple-darwin`, or `make stats` for the current number.
207+
Run `RUST_MIN_STACK=8388608 cargo test --lib --target x86_64-apple-darwin`, or `make stats`, for the current total on `x86_64-apple-darwin`. The number is not pinned in prose — it lives in code, and a hardcoded copy only drifts CI red as the suite grows.
208208

209209
Major categories (approximate; breakdown drifts faster than the total):
210210

tools/check-status-freshness.py

Lines changed: 34 additions & 77 deletions
Original file line numberDiff line numberDiff line change
@@ -4,43 +4,38 @@
44
"""
55
Pre-commit advisory: surface STATUS.md drift before it compounds.
66
7-
Two independent advisory checks, both exit-0 (warn-but-pass), in the
8-
same don't-grow-the-baseline family as the other tools/check-*.py gates.
7+
Two independent advisory checks, both exit-0 (warn-but-pass) by default, in
8+
the same don't-grow-the-baseline family as the other tools/check-*.py gates.
99
1010
1. DATE drift — `last_synced_to_code:` more than THRESHOLD_DAYS old.
11-
(Original behavior, unchanged. Rationale: the 2026-04-22 -> 2026-05-04
12-
12-day drift window that had to be recovered in one large catch-up sync.)
13-
14-
2. NUMBER drift — STATUS.md's prose counts no longer match the source.
15-
This is the gap that let the file sit at "561 host unit tests /
16-
47 harnesses" while the code was already at 892 / 48: a sync bumped
17-
the *date* but never re-derived the *numbers*, and a date-only check
18-
passed silently. The date moving is not evidence the counts moved.
19-
20-
- Kani harness count is derived CHEAPLY (count active `#[kani::proof]`
21-
under verification/, no compile) and compared to every *total*
22-
stated in STATUS.md prose. Safe to run on every commit.
23-
- The unit-test count cannot be derived without compiling, so by
24-
default this only checks that the stated test numbers agree with
25-
EACH OTHER (catches a partial update — one mention fixed, another
26-
missed). With `--full` it runs `make stats` and compares against the
27-
live count; that path compiles, so it is for CI / manual runs, NOT
28-
the pre-commit hook.
29-
30-
Posture unchanged: advisory, exit 0. Only config-level errors (STATUS.md
31-
missing, unparseable date) exit 1. Tighten to a hard block by flipping
32-
the relevant return once the warn cycle proves it changes behavior.
11+
Rationale: the 2026-04-22 -> 2026-05-04 12-day drift window that had to
12+
be recovered in one large catch-up sync. The date moving is not, on its
13+
own, evidence the content moved.
14+
15+
2. KANI-HARNESS number drift — STATUS.md's stated harness totals no longer
16+
match the source. Derived CHEAPLY (count active `#[kani::proof]` under
17+
verification/, no compile) and compared to every *total* stated in the
18+
prose. Safe to run on every commit.
19+
20+
The unit-test COUNT is deliberately NOT checked. STATUS.md no longer pins
21+
a test total in prose — it points at `make stats` instead — because that
22+
number can only be derived by compiling, and a hardcoded copy drifted CI
23+
red every time the suite grew (892 -> 903 was the last instance). With no
24+
pinned number there is nothing to compare and nothing to drift, so the
25+
`--full`/`make stats` path that used to verify it is gone entirely.
26+
27+
Posture: advisory, exit 0. `--strict` makes objective Kani-harness drift
28+
exit 1 (for CI); the date heuristic always stays advisory. Only config-level
29+
errors (STATUS.md missing, unparseable date) exit 1 unconditionally.
3330
3431
Usage:
35-
python3 tools/check-status-freshness.py # date + cheap number checks (pre-commit)
36-
python3 tools/check-status-freshness.py --full # also re-derive the test count via `make stats`
37-
make check-status-freshness # no-arg form
38-
make check-status-freshness-full # --full form
32+
python3 tools/check-status-freshness.py # date + Kani-harness checks
33+
python3 tools/check-status-freshness.py --strict # Kani-harness drift FAILs (CI)
34+
make check-status-freshness # no-arg form
3935
"""
4036
import argparse
4137
import datetime
4238
import re
43-
import subprocess
4439
import sys
4540
from pathlib import Path
4641

@@ -59,11 +54,6 @@
5954
re.compile(r"(\d+)\s+harnesses across \d+ proof crates"),
6055
re.compile(r"(\d+)\s+passing harnesses"),
6156
)
62-
TEST_COUNT_RES = (
63-
re.compile(r"(\d+)\s+host unit tests"),
64-
re.compile(r"Total:\s*\*\*(\d+)\*\*"),
65-
)
66-
MAKE_STATS_TESTS_RE = re.compile(r"Tests \(lib\):\s*(\d+)")
6757

6858
_WARNINGS = 0
6959

@@ -131,10 +121,10 @@ def stated_counts(text: str, patterns) -> list:
131121
return found
132122

133123

134-
def check_numbers(text: str, full: bool) -> int:
135-
"""Advisory number-drift checks. Returns the count of genuine drift
136-
mismatches (operational warnings missing dir, unparseable `make stats`
137-
do not count, so they never trip --strict / CI)."""
124+
def check_numbers(text: str) -> int:
125+
"""Advisory number-drift check (Kani harnesses). Returns the count of
126+
genuine drift mismatches; operational warnings (missing verification/ dir)
127+
do not count, so they never trip --strict / CI."""
138128
drift = 0
139129

140130
# --- Kani harnesses: cheap, every run ---
@@ -153,52 +143,19 @@ def check_numbers(text: str, full: bool) -> int:
153143
f"{', '.join(map(str, bad))}. Update the harness totals."
154144
)
155145

156-
# --- Unit tests: intra-prose consistency always; live compare on --full ---
157-
stated_t = stated_counts(text, TEST_COUNT_RES)
158-
if stated_t and len(set(stated_t)) > 1:
159-
drift += 1
160-
warn(
161-
"STATUS.md states inconsistent unit-test counts "
162-
f"({', '.join(map(str, sorted(set(stated_t))))}) — the prose "
163-
"disagrees with itself, a partial update."
164-
)
165-
if full and stated_t:
166-
try:
167-
out = subprocess.run(
168-
["make", "stats"], capture_output=True, text=True,
169-
timeout=900, check=False,
170-
).stdout
171-
except (OSError, subprocess.SubprocessError) as exc:
172-
warn(f"--full: could not run `make stats` ({exc}).")
173-
return drift
174-
match = MAKE_STATS_TESTS_RE.search(out)
175-
if not match:
176-
warn("--full: could not parse Tests(lib) from `make stats`.")
177-
return drift
178-
live = int(match.group(1))
179-
bad = sorted({n for n in stated_t if n != live})
180-
if bad:
181-
drift += 1
182-
warn(
183-
f"Unit-test drift: `make stats` reports {live} lib tests, "
184-
f"STATUS.md states {', '.join(map(str, bad))}. Update the "
185-
"test totals."
186-
)
146+
# The unit-test count is intentionally NOT checked. STATUS.md no longer
147+
# pins a total in prose (it points at `make stats`), so there is nothing
148+
# to drift against and nothing that would need a compile to verify.
187149
return drift
188150

189151

190152
def main() -> int:
191153
parser = argparse.ArgumentParser(
192-
description="STATUS.md freshness advisory (date + number drift)."
193-
)
194-
parser.add_argument(
195-
"--full", action="store_true",
196-
help="also re-derive the unit-test count via `make stats` "
197-
"(compiles; for CI / manual use, not the pre-commit hook)",
154+
description="STATUS.md freshness advisory (date + Kani-harness drift)."
198155
)
199156
parser.add_argument(
200157
"--strict", action="store_true",
201-
help="exit 1 on objective number drift (Kani/test count mismatch). "
158+
help="exit 1 on objective Kani-harness drift (source vs prose). "
202159
"The date heuristic stays advisory. For CI enforcement.",
203160
)
204161
args = parser.parse_args()
@@ -215,7 +172,7 @@ def main() -> int:
215172
rc = check_date(text)
216173
if rc != 0:
217174
return rc
218-
drift = check_numbers(text, full=args.full)
175+
drift = check_numbers(text)
219176

220177
if args.strict and drift:
221178
print(

0 commit comments

Comments
 (0)