Skip to content

Commit 0707f95

Browse files
noonioclaude
andcommitted
Add the Agda formalisation of the specification
Populate the literate sources with their Agda content and machine-check them as part of the spec build: - Re-add the Agda code blocks to the .lagda.typ tree (datum/redeemer types, transition relations, validity bundles, coverage/safety and security theorems and proofs), which now render in the formalisation appendix. - Add the standalone reference modules (Prelude, Reference, ReferenceBridge, OffChainReference, RefReflection). - build.sh now typechecks the tree with Agda before rendering, and runs the Agda<->Typst reference lint (check-refs.sh) and the trust-ledger drift check (check-trust-ledger.sh). - Wire the Agda toolchain (spec-agda) into the spec build and dev shell. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 611c1f0 commit 0707f95

20 files changed

Lines changed: 4304 additions & 122 deletions

nix/hydra/shell.nix

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,12 @@
1212
cleanPkgs = inputs'.nixpkgs.legacyPackages;
1313

1414
buildInputs = [
15-
# For working on the formal specification (typst render + the
16-
# annotate-notation.py tooltip postprocess, see spec/build.sh). The
17-
# wrapped typst carries the pinned @preview diagram packages via
18-
# TYPST_PACKAGE_CACHE_PATH, same as the nix build (see nix/hydra/spec.nix).
15+
# For working on the formal specification (agda typecheck + typst
16+
# render + the annotate-notation.py tooltip postprocess, see
17+
# spec/build.sh). The wrapped typst carries the pinned @preview diagram
18+
# packages via TYPST_PACKAGE_CACHE_PATH, same as the nix build (see
19+
# nix/hydra/spec.nix).
20+
self'.packages.spec-agda
1921
self'.packages.spec-typst
2022
(pkgs-2511.python3.withPackages (ps: [ ps.pymupdf ]))
2123
# To compile hydra scripts

nix/hydra/spec.nix

Lines changed: 27 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
{ self, ... }: {
1+
{ self, inputs, ... }: {
22

3-
perSystem = { config, pkgs, pkgs-2511, ... }:
3+
perSystem = { config, pkgs, pkgs-2411, pkgs-2511, ... }:
44
let
55
# Typst with the spec's diagram packages (@preview/cetz, fletcher, oxifmt)
66
# pinned from nixpkgs and supplied via the wrapper's
@@ -15,9 +15,21 @@
1515
p.oxifmt_0_2_1
1616
]);
1717

18-
# The Typst render, WITHOUT the notation-tooltip postprocess
19-
# (ANNOTATE_NOTATION=skip, see build.sh stage 3). Internal: consume
20-
# packages.spec, which adds the tooltips.
18+
agdaPackages = pkgs-2411.callPackage "${self}/spec/pkgs/initial-packages.nix" {
19+
inherit (pkgs-2411.haskellPackages) Agda;
20+
nixpkgs = inputs.nixpkgs-2411;
21+
};
22+
agdaLibraries = with agdaPackages; [
23+
abstract-set-theory
24+
formal-ledger
25+
standard-library
26+
standard-library-classes
27+
standard-library-meta
28+
];
29+
30+
# The Agda typecheck + lints + Typst render, WITHOUT the notation-tooltip
31+
# postprocess (ANNOTATE_NOTATION=skip, see build.sh stage 3). Internal:
32+
# consume packages.spec, which adds the tooltips.
2133
#
2234
# The postprocess runs as the separate seconds-long derivation below so
2335
# this minutes-long, disk-heavy build shares no build window with the
@@ -31,15 +43,16 @@
3143
pname = "hydra-spec-unannotated.pdf";
3244
version = "0.0.1";
3345
nativeBuildInputs = [
46+
config.packages.spec-agda
3447
config.packages.spec-typst
3548
];
3649
meta = { };
3750
src = "${self}/spec";
38-
# build.sh renders the literate-Typst sources with Typst (no
39-
# LaTeX/Inkscape toolchain needed). --ignore-system-fonts keeps Typst
40-
# reproducible: only the fonts bundled with Typst plus JuliaMono from
41-
# nixpkgs (code blocks, wired through JULIAMONO_FONT_DIR, see build.sh)
42-
# are used.
51+
# build.sh typechecks the literate-Typst sources with Agda and renders
52+
# the PDF with Typst (no LaTeX/Inkscape toolchain needed).
53+
# --ignore-system-fonts keeps Typst reproducible: only the fonts bundled
54+
# with Typst plus JuliaMono from nixpkgs (code blocks, wired through
55+
# JULIAMONO_FONT_DIR, see build.sh) are used.
4356
JULIAMONO_FONT_DIR = "${pkgs.julia-mono}/share/fonts/truetype";
4457
ANNOTATE_NOTATION = "skip";
4558
buildPhase = ''
@@ -57,6 +70,10 @@
5770
# exposed so the dev shell can offer the same `typst` for working on the spec.
5871
packages.spec-typst = spec-typst;
5972

73+
# Agda with the specification's libraries, reused by the spec build and
74+
# exposed so the dev shell can offer the same `agda` for working on the spec.
75+
packages.spec-agda = agdaPackages.withPackages agdaLibraries;
76+
6077
# The publishable spec PDF: the render above plus the notation hover
6178
# tooltips (build.sh stage 3, split out - see the spec-rendered comment).
6279
packages.spec = pkgs.stdenv.mkDerivation {

spec/build.sh

Lines changed: 16 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,31 @@
11
#!/usr/bin/env bash
2-
# Build the Hydra specification PDF via Typst.
2+
# Build the Hydra specification PDF via Agda (typecheck) + Typst (render).
33
#
4-
# `typst compile` renders the .lagda.typ tree in place (Typst reads the literate
5-
# files directly; code fences render as raw blocks, see template.typ for the
6-
# hidden/visible idiom) to _build/hydra-spec.pdf.
7-
#
8-
# NB the Agda typecheck of these same sources (and the Agda↔Typst/trust-ledger
9-
# lints) is added alongside the formalisation itself; until then the code fences
10-
# are prose-free placeholders and only the Typst render runs here.
4+
# Stage 1: `agda` typechecks the literate-Typst source tree (fails on type error,
5+
# preserving the machine-checked property).
6+
# Stage 2: `typst compile` renders the same .lagda.typ tree in place (Typst reads
7+
# the literate files directly; code fences render as raw blocks, see
8+
# template.typ for the hidden/visible idiom) to _build/hydra-spec.pdf.
119
set -euo pipefail
1210
cd "$(dirname "$0")"
1311

1412
SRC=src
1513
PDF=_build/hydra-spec.pdf
1614
ENTRY="$SRC/Hydra/Protocol/Main.lagda.typ"
1715

18-
# Render in place. --root=src so the root-relative imports (/template.typ,
19-
# /macros.typ, /short.bib, /agda.sublime-syntax) resolve; the .lagda.typ files
20-
# include each other by name, so no staging/extension-stripping is needed.
16+
# Stage 1: typecheck, then the Agda↔Typst reference consistency lint (W6) and the C3 trust-ledger drift check.
17+
agda "$ENTRY"
18+
bash check-refs.sh
19+
bash check-trust-ledger.sh
20+
21+
# Stage 2: render in place. --root=src so the root-relative imports (/template.typ,
22+
# /macros.typ, /short.bib, /agda.sublime-syntax) resolve; the .lagda.typ files include
23+
# each other by name, so no staging/extension-stripping is needed.
2124
#
2225
# JuliaMono (the code font, see template.typ) is not vendored: nix provides it
2326
# (nixpkgs `julia-mono`) via JULIAMONO_FONT_DIR, exported by both `nix build .#spec`
2427
# and the dev shell. Hard-error when unset: typst only warns on a missing font
2528
# family and would silently render code blocks with a fallback font.
26-
#
27-
# The @preview diagram packages (cetz, fletcher, oxifmt) are supplied at pinned
28-
# versions by the wrapped typst's TYPST_PACKAGE_CACHE_PATH (see nix/hydra/spec.nix),
29-
# so no --package-cache-path / in-repo vendoring is needed.
3029
mkdir -p "$(dirname "$PDF")"
3130
# Build via a temp file and rename into place ATOMICALLY: typst
3231
# truncate-and-writes its output, and a PDF viewer auto-reloading on change
@@ -43,14 +42,14 @@ typst compile --ignore-system-fonts --root "$SRC" \
4342
"$ENTRY" "$TMP"
4443

4544

46-
# Stage: stamp invisible hover-tooltips (definitions of the notation symbols)
45+
# Stage 3: stamp invisible hover-tooltips (definitions of the notation symbols)
4746
# over the prose math; see annotate-notation.py. The rendered pages are
4847
# pixel-identical; viewers with annotation popups (okular, pdf.js, Acrobat)
4948
# show the definition on hover.
5049
#
5150
# ANNOTATE_NOTATION=skip skips the stage: `nix build .#spec` runs it as a
5251
# SEPARATE seconds-long derivation (nix/hydra/spec.nix), so the minutes-long
53-
# Typst build does not share a build window with the python closure - a
52+
# Agda+Typst build does not share a build window with the python closure - a
5453
# busy builder's mid-build auto-GC (observed on the darwin CI builders) could
5554
# otherwise collect the late-used python environment out from under this step.
5655
if [ "${ANNOTATE_NOTATION:-}" = "skip" ]; then

spec/check-refs.sh

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
#!/usr/bin/env bash
2+
# Agda↔Typst consistency lint.
3+
#
4+
# Full type-checking of the Typst against Agda needs an Agda→Typst backend we
5+
# don't have, so this guards the two single-source links that matter:
6+
#
7+
# (1) every transition rule the prose cites as `<name>` rule is a real
8+
# constructor of the Agda relation `_⟶⟨_⟩_`; and
9+
# (2) the head state-machine DIAGRAM (data in diagrams.typ) has exactly the same
10+
# (source, rule, target) transitions as that Agda relation — so the picture
11+
# cannot drift from the formal state machine.
12+
set -euo pipefail
13+
cd "$(dirname "$0")"
14+
15+
ON=src/Hydra/Protocol/OnChain.lagda.typ
16+
DIAG=src/diagrams.typ
17+
fail=0
18+
19+
# --- Agda transition relation: emit "source|rule|target" per constructor ----
20+
agda_transitions() {
21+
awk '
22+
/data _⟶⟨_⟩_ :/ { inrel=1; next }
23+
inrel && /^```/ { inrel=0 }
24+
!inrel { next }
25+
/^ [a-z]/ { if (name != "") emit(); name=$1; buf=$0; next }
26+
{ buf = buf " " $0 }
27+
END { if (name != "") emit() }
28+
function emit( a, b, s, t) {
29+
s = (match(buf, /→[ ]*([A-Za-z]+)/, a) ? a[1] : "?")
30+
t = (match(buf, /⟩[ ]*([A-Za-z]+)/, b) ? b[1] : "?")
31+
print s "|" name "|" t
32+
}
33+
' "$ON" | sort -u
34+
}
35+
36+
# --- Diagram data: emit "from|rule|to" per transition --------------------------
37+
diagram_transitions() {
38+
grep -oE '\(from: "[A-Za-z]+", rule: "[A-Za-z]+", to: "[A-Za-z]+"' "$DIAG" \
39+
| sed -E 's/.*from: "([A-Za-z]+)", rule: "([A-Za-z]+)", to: "([A-Za-z]+)".*/\1|\2|\3/' \
40+
| sort -u
41+
}
42+
43+
# --- Check (1): cited rule names exist ----------------------------------------
44+
rules=$(agda_transitions | cut -d'|' -f2 | sort -u)
45+
cited=$(grep -oE '`[A-Za-z]+` rule' "$ON" | sed -E 's/`([A-Za-z]+)` rule/\1/' | sort -u)
46+
for c in $cited; do
47+
if ! grep -qx "$c" <<<"$rules"; then
48+
echo "ERROR: prose cites transition rule '$c' that is not a constructor of _⟶⟨_⟩_"
49+
fail=1
50+
fi
51+
done
52+
53+
# --- Check (2): diagram transitions == Agda transitions ------------------------
54+
only_agda=$(comm -23 <(agda_transitions) <(diagram_transitions))
55+
only_diag=$(comm -13 <(agda_transitions) <(diagram_transitions))
56+
if [ -n "$only_agda" ]; then
57+
echo "ERROR: transitions in the Agda relation but MISSING from the diagram (diagrams.typ):"
58+
echo "$only_agda" | sed 's/^/ /'
59+
fail=1
60+
fi
61+
if [ -n "$only_diag" ]; then
62+
echo "ERROR: transitions in the diagram but NOT in the Agda relation _⟶⟨_⟩_:"
63+
echo "$only_diag" | sed 's/^/ /'
64+
fail=1
65+
fi
66+
67+
# --- Check (3): diagrams.typ `state-fields` keys == HeadDatum constructors -----
68+
datum_ctors=$(awk '
69+
/data HeadDatum :/ { ind=1; next }
70+
ind && (/^```/ || /^$/ || /^data /) { ind=0 }
71+
ind && /^ [A-Z]/ { print $1 }
72+
' "$ON" | sort -u)
73+
state_keys=$(awk '
74+
/#let state-fields = \(/ { ins=1; next }
75+
ins && /^\)/ { ins=0 }
76+
ins && /^ "[A-Za-z]+":/ { gsub(/[",:]/,"",$1); print $1 }
77+
' "$DIAG" | sort -u)
78+
if [ -z "$datum_ctors" ] || [ -z "$state_keys" ]; then
79+
# A silent skip here would let real drift through: if either parse comes back empty the
80+
# anchors (`data HeadDatum :` / `#let state-fields = (`) no longer match the sources.
81+
echo "ERROR: check (3) parsed no HeadDatum constructors or no state-fields keys (anchor drift);"
82+
echo " update the awk anchors in check-refs.sh to match the current source layout."
83+
fail=1
84+
elif [ "$datum_ctors" != "$state_keys" ]; then
85+
echo "ERROR: diagrams.typ state-fields keys do not match HeadDatum constructors:"
86+
echo " HeadDatum: $(echo $datum_ctors)"
87+
echo " state-fields: $(echo $state_keys)"
88+
fail=1
89+
fi
90+
91+
# --- Check (4): every tx-rule diagram maps to a real transition rule -----------
92+
tx_rules=$(awk '
93+
/#let tx-rule = \(/ { intr=1; next }
94+
intr && /^\)/ { intr=0 }
95+
intr && /: "[A-Za-z]+"/ { match($0, /"([A-Za-z]+)"/, a); print a[1] }
96+
' "$DIAG" | sort -u)
97+
for r in $tx_rules; do
98+
if ! grep -qx "$r" <<<"$rules"; then
99+
echo "ERROR: diagrams.typ tx-rule maps to '$r', not a constructor of _⟶⟨_⟩_"
100+
fail=1
101+
fi
102+
done
103+
104+
# --- Check (5): every code fence opens as ``` (bare) or ```agda exactly --------
105+
# Agda's literate mode typechecks a fence only when its info string is empty or
106+
# exactly `agda`; a mistyped tag (```Agda, ```agda2, ```haskell) is silently NOT
107+
# typechecked AND renders as a literal raw block. Flag any non-`agda` tag.
108+
badfence=$(grep -nE '^```[^`]' src/Hydra/Protocol/*.lagda.typ | grep -vE ':```agda$' || true)
109+
if [ -n "$badfence" ]; then
110+
echo "ERROR: code fence with a tag other than 'agda' (skips Agda typecheck AND misrenders):"
111+
echo "$badfence" | sed 's/^/ /'
112+
fail=1
113+
fi
114+
115+
if [ "$fail" -eq 0 ]; then
116+
echo "check-refs: OK — cited rules exist, the head-state diagram matches the Agda"
117+
echo "relation, state-fields KEYS match the HeadDatum constructors (names only, not"
118+
echo "the per-state field tuples), tx diagrams map to real rules, and"
119+
echo "every code fence is bare or \`\`\`agda."
120+
fi
121+
exit "$fail"

spec/check-trust-ledger.sh

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
#!/usr/bin/env bash
2+
# Trust-ledger drift check (bridge layer).
3+
#
4+
# The machine-checked `spec ⇒ extracted-reference` bridge rests on a FIXED, enumerated trusted base:
5+
# (a) injected `Ops` mocks — the const-`true` boundaries the reference delegates (crypto / accumulator /
6+
# value-map conjuncts), and
7+
# (b) extraction-faithfulness / encoding postulates (hash/out-ref encodings and the
8+
# participant / no-mint faithfulness assumptions).
9+
#
10+
# This script extracts that set from the Agda sources and FAILS if it drifts from the ledger below.
11+
# So a NEW mock or postulate cannot enter the trusted base silently: adding one fails the build until
12+
# both the EXPECTED_* lists and this ledger table are updated.
13+
#
14+
# Bridge-layer trust ledger (what each trusted item assumes; the HeadValidatorAgreement test covers each
15+
# against the real validator/crypto where constructible). SCOPE: this gates ReferenceBridge/RefReflection
16+
# only; the abstract model's own axioms (Prelude value/crypto laws, accumulator laws, §7 assumptions) are
17+
# inventoried in the spec's "What the formalisation assumes" appendix section, not drift-checked here.
18+
# Ops mocks (const-true boundaries the reference delegates). The snapshot signature is the 6-tuple
19+
# message cid‖v‖s‖η#‖δ#‖κ# (accumulator + decommit-/commit-output-set hashes):
20+
# closeCryptoOK close snapshot signature + accumulator-commitment hash (real Ed25519 in the test,
21+
# incl. a tampered-δ#/κ# reject)
22+
# incCryptoOK increment/decrement snapshot signature incl. the recomputed commit-set hash
23+
# (increment, DepositDatumInvalid) / decommit-set hash (decrement) (real Ed25519)
24+
# contestCryptoOK contest snapshot signature, η binding, contest-once (real Ed25519)
25+
# fanoutCryptoOK fanout KZG membership + value conservation + the canonical-CRS datum binding
26+
# (crsBindOK; real BLS pairing, InvalidCRSDatum reject in the test)
27+
# recoverHashOK recover recovered-outputs serialisation hash (empty-deposit case)
28+
# initPlacementOK μHead seed-spent + token placement (real validateTokensMinting)
29+
# Postulates (typecheck-only):
30+
# cidToNat, refCodeOf head-id / out-ref → ℕ encodings (used with cong only)
31+
# signerCodes, ptCodes, participantSigned→ref signer / PT-name encodings + overlap faithfulness
32+
# mintEntryCount, noMint→ref mint-entry count encoding + noMint faithfulness
33+
set -euo pipefail
34+
cd "$(dirname "$0")"
35+
36+
BR=src/Hydra/Protocol/ReferenceBridge.agda
37+
RR=src/Hydra/Protocol/RefReflection.agda
38+
39+
# (a) Ops mocks: every `<field> = λ … → true` const-true binding (ANY field in a record, first or
40+
# not, single- OR multi-line; NOT record-pattern matches like `step =`, which don't bind `λ … → true`).
41+
# Newlines are flattened first so a multi-line record literal cannot hide a mock from the line-based grep.
42+
actual_mocks=$(tr '\n' ' ' < "$BR" | grep -oE '[a-zA-Z][a-zA-Z0-9]* = λ[^;{}]*→ true' | sed -E 's/ = λ.*//' | sort -u)
43+
44+
# (b) Postulated names, both single-line (`postulate name :`) and block (`postulate` then indented `name :`).
45+
actual_postulates=$(awk '
46+
/^[ ]*--/ {next}
47+
/^postulate$/ {inblock=1; next}
48+
/^postulate[ ]+[^ ]/ {print $2; inblock=0; next}
49+
inblock && /^[ ]+[^ ]+ +:/ {print $1; next}
50+
inblock && /^[^ ]/ {inblock=0}
51+
' "$BR" "$RR" | sort -u)
52+
53+
expected_mocks=$(printf '%s\n' \
54+
closeCryptoOK contestCryptoOK fanoutCryptoOK incCryptoOK initPlacementOK recoverHashOK \
55+
| sort -u)
56+
expected_postulates=$(printf '%s\n' \
57+
cidToNat mintEntryCount 'noMint→ref' 'participantSigned→ref' ptCodes refCodeOf signerCodes \
58+
| sort -u)
59+
60+
fail=0
61+
if [ "$actual_mocks" != "$expected_mocks" ]; then
62+
echo "check-trust-ledger: the injected Ops-mock set DRIFTED from the documented ledger (- expected, + actual):"
63+
diff <(echo "$expected_mocks") <(echo "$actual_mocks") || true
64+
fail=1
65+
fi
66+
if [ "$actual_postulates" != "$expected_postulates" ]; then
67+
echo "check-trust-ledger: the postulate set DRIFTED from the documented ledger (- expected, + actual):"
68+
diff <(echo "$expected_postulates") <(echo "$actual_postulates") || true
69+
fail=1
70+
fi
71+
if [ "$fail" -ne 0 ]; then
72+
echo "Update the trust-ledger table in this script's header comment and the EXPECTED_* lists."
73+
exit 1
74+
fi
75+
echo "check-trust-ledger: OK — the bridge-layer trusted base is the 6 documented Ops mocks + 7 documented postulates."

spec/src/Hydra/Protocol/Introduction.lagda.typ

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
```
2+
module Hydra.Protocol.Introduction where
3+
```
14

25
#import "/template.typ": *
36
#import "/macros.typ": *

spec/src/Hydra/Protocol/Main.lagda.typ

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,30 @@
1515
body,
1616
)
1717

18+
```
19+
module Hydra.Protocol.Main where
20+
21+
import Hydra.Protocol.Prelude
22+
import Hydra.Protocol.Introduction
23+
import Hydra.Protocol.Overview
24+
import Hydra.Protocol.Preliminaries
25+
import Hydra.Protocol.Setup
26+
import Hydra.Protocol.OnChain
27+
import Hydra.Protocol.OffChain
28+
import Hydra.Protocol.Security
29+
-- The machine-checked §7 proof terms (rendered: the statements appear under §7 "Machine-checked
30+
-- results", included below after Security; the proof bodies are typechecked but hidden).
31+
import Hydra.Protocol.SecurityProofs
32+
-- Extractable decidable reference checker + the bridge proving it reflects the on-chain
33+
-- validity bundles (Tier 2 differential-testing; not rendered in the document).
34+
import Hydra.Protocol.Reference
35+
import Hydra.Protocol.ReferenceBridge
36+
-- Extractable decidable reference for the OFF-CHAIN HeadLogic figure (Tier 2 differential, off-chain
37+
-- side; typecheck-only here, extracted via regenerate.sh, not rendered in the document).
38+
import Hydra.Protocol.OffChainReference
39+
-- On-chain coverage / non-stuckness + safety invariants (rendered: included below after OnChain).
40+
import Hydra.Protocol.OnChainCoverage
41+
```
1842

1943
#include "Introduction.lagda.typ"
2044
#include "Overview.lagda.typ"

0 commit comments

Comments
 (0)