-
Notifications
You must be signed in to change notification settings - Fork 105
leios fetch scheme
- Preliminaries — source docs and companion references.
- Summary of proposed approach — what Nick's proposal does.
- Claude's take: strengths and tensions — seven critical observations.
- Judging effectiveness on the testnet — four concrete measurements our machinery enables.
-
Tying to
$F_{\text{full}\mid C}$ — how the scheme feeds Yves Hauser's conditional-diffusion formula. - Proposed improvements to the scheme — eight incremental improvements (adaptive to observed conditions, including diversity-aware peer churn to counter the tendency for big-ledger peer sets to concentrate around block-production regions), multi-signal fusion for adversarial robustness, two more ambitious alternatives, and a suggested experimental design.
- What remains uncertain / Open questions — closing items.
- Deep dive: why chunking gains cap at high RTT (technical nuance) — the tcp-estimator math, the serial-vs-parallel distinction, and the system-level trade-off the memo makes.
- What the tcp-estimator work specifically illuminates (technical nuance) — supporting observations on the two-channel dynamics.
Working document analysing Nick Frisby's LeiosFetch proposal
(docs/targeted-design-investigations/mininet-LeiosFetch-test-bed/IDEA-NeighborhoodFarPeers.md
on branch nfrisby/april2026-checkpoint in ouroboros-leios), in
light of the tcp-estimator chunking analysis on branch
mw/tcp-estimator (analysis/tcp-estimator/README.md and the
notes/ subdirectory) and the branch's mux telemetry.
Source references:
-
Scheme proposal:
docs/targeted-design-investigations/mininet-LeiosFetch-test-bed/IDEA-NeighborhoodFarPeers.md(nfrisby/april2026-checkpoint). -
AUEB technical reports cited in Observation 1:
docs/TECHREP1.pdfanddocs/TECHREP3.pdfin ouroboros-leios. -
Chunking analysis:
analysis/tcp-estimator/notes/on branch mw/tcp-estimator, particularlyparallel-chunking.md,parallel-chunking-results.md,parallel-chunking-n2-puzzle.md,parallel-chunking-cdf-staircase.md,parallel-chunking-low-p.md,parallel-chunking-mc-confidence.md.
Companion documents:
-
leios-conditional-diffusion.md— Yves Hauser's §5.6$F_{\text{full}\mid C}$ formula; the scheme's ultimate output is a specific G(t) that plugs into this. -
leios-priority-measurement.md— the cross-node per-EB arrival log, which validates the scheme's effectiveness. -
leios-overlap.md— Frame 1 / Frame 2 framing; the scheme's adaptive-lever variants map onto Frame 2. -
leios-testnet.md— operational campaign shape; the scheme is a candidate for deployment on that campaign.
An asymmetric bandwidth-allocation strategy for the LeiosFetch decision logic, motivated by TCP fundamentals and the AUEB peer-selection recommendations:
- High-latency / big-ledger peers — one large request per peer, no hedging. Rationale: high-latency + wide-BDP paths need a single large request to reach steady-state throughput. Serial small requests hurt because they stall a wide-BDP path between chunks (request-response RTT gaps and application-limited growth) rather than filling it with one continuous transfer — not, on a persistent connection, a fresh slow-start per chunk.
- Low-latency / NBL peers — small chunks (32–64 kB), hedged across many peers, tight timeouts. Rationale: chunking a payload across many low-BDP paths shrinks the loss-induced right tail.
The appendix maps big-ledger → high-latency and NBL → low-latency as a proxy, simplifying the classifier.
Seed-node arithmetic (not stated in the written memo). The memo argues coverage qualitatively — "as long as at least one node in each neighborhood happens to utilize one of its high-latency peers promptly and fully, then the whole neighborhood benefits", and big-ledger peers are "likely to be honest". The quantitative seed-node model below is a development of that idea (Nick's, expanded in later discussion); the memo itself cites no coverage percentage and contains no seed terminology. We carry it here because the argument is sound and is the moral successor to the memo's neighborhood claim.
Take 25% of stake as honest committee-holding "seed nodes" that
possess (and have validated) the EB by the voter deadline
(
- Per-connection success rate:
$25/90 \approx 27.7%$ (25% of stake is honest committee, sampled from the top-90% stake pool). - Per-node probability of being connected to at least one seed:
$1 - (1 - 0.277)^5 \approx 80%$ . - A raw-25% version (not normalising for the top-90% pool) gives
$1 - (1 - 0.25)^5 \approx 76.2%$ . The 76.2% is slightly pessimistic; ~80% is slightly optimistic. Both are within the uncertainty of adjacent assumptions (peer independence, adversary influence on seed selection). - Either way, ~76–80% of nodes should get the whole EB in "one hop" from a well-connected honest big-ledger peer.
The remaining ~24% orphan nodes rely on multi-hop diffusion.
- The asymmetry matches TCP reality. High-latency + wide-BDP single-shot vs low-latency + narrow-BDP chunked exploits two qualitatively different regimes of the same congestion-control envelope.
-
Chunking benefit is real and quantified. At default params
(~12 MB payload, RTT=50 ms, p=1e-4), NBL chunked at n=32 drops
P99 from 6.72 s to 1.65 s — a 75% reduction (from
parallel-chunking-results.md; tcp-estimator sim used 12.5 MB — the ~4% delta from Leios's 12 MB doesn't move the numbers meaningfully). These figures carry model caveats — see Validity caveats on the tcp-estimator numbers. - The seed-node math is elegant. Under the simplifying assumption that the 5 big-ledger peer draws are independent, ~76% one-hop coverage is a strong baseline for the memo's core claim. Whether that independence assumption holds in practice under peer-selection churn is an empirical question — see tension 7 below and Improvement #8 for the analytical treatment.
- Big-ledger = high-latency is a simplification. Cardano peer selection doesn't strictly follow AUEB's high/low-RTT scheme. Big-ledger peers are randomly sampled from top-90% stake — they can be near or far in RTT terms. If a node's 5 big-ledger peers are all nearby, "one big request, wide-BDP" wastes their low-latency capacity. If all 5 are far, the assumption holds. Reality is a mix; static classification is wrong for the mix.
-
The 24% orphan problem is Yves's
$F_{\text{full}\mid C}$ question. The seed-node arithmetic covers 76% directly; the remaining 24% relies on multi-hop diffusion — which is not necessarily local. When an orphaned node's local cluster has at least one seed-connected neighbor, diffusion stays within the neighborhood; but when a whole remote cluster happens to have no direct big-ledger connection to the block-producing cluster (a plausible outcome under the churn tendency discussed in tension 7), the payload has to span cross-cluster hops before any node in the remote cluster receives it, and only then does neighborhood-local re-diffusion take over. Whether both cases complete within L_diff = 7 s is exactly what the empirical$\hat{F}_{\text{full}\mid C}(14)$ measures. -
Chunking's P99 gains collapse at low p, but its
resource-bounding benefits don't. The tcp-estimator work
(
parallel-chunking-low-p.md) shows that at$p \le 10^{-5}$ (well-engineered SPO paths, likely most inter-AWS routes), the marginal P99 becomes uninformative — most runs are loss-free and chunking's latency benefit shrinks to almost nothing. If empirical p on the SPO mesh is ~$10^{-5}$ rather than the assumed$10^{-4}$ , the NBL chunking machinery loses its headline P99-shrinkage argument. That doesn't make it pure complexity, though: chunking also bounds per-in-flight receive buffer footprint (32 × 391 kB chunk buffers vs a single 12 MB in-flight payload), gives cancellation granularity (waste one chunk on race resolution, not the remainder of a 12 MB transfer), and lets chunks distribute across multiple NBL peers naturally. Per-chunk request overhead is a small header plus a fixed one-RTT setup cost — negligible at low RTT and with small requests, so the incremental latency cost of taking these benefits is modest even when the P99 gain is zero. Whether they're worth the machinery complexity depends on how much concurrent EB / payload fetching a node does at scale; the memory-bounding story alone may justify chunking under the report's default assumptions. p directly measurable via TCPInfo; the resource story measurable via receive-buffer high-watermarks. -
Slow-start floor shrinks only weakly (logarithmically) with
chunking. See the deep-dive section below. Parallel chunking
lowers the per-chunk floor via the
$\log(N/n)$ term, but the RTT factor is irreducible, so at high RTT (250 ms+) the floor stays a large fraction of P99 and chunking's relative gain caps around 60–65%. - The two-channel race is under-specified. The appendix has TODOs on "send a cancel to the big-ledger peer" when NBL delivers first, and vice versa. Without concrete cancellation semantics, the scheme risks large wasted egress — the memo itself flags a worst-case ~6 MB of NBL redundancy at the tail.
- The "adversary picks the seed" concern. The 76.2% assumes the 5 big-ledger peers are uncorrelated draws from the top-90% stake. Both peer churning rules and the adversary's ability to influence seed selection push the effective probability lower.
- Peer-selection churn creates a correlation tendency (independent of any adversary). Cardano's churn scoring ("first to announce, first to deliver") tends to favour peers close to block-production regions, potentially concentrating each node's 5 big-ledger peers around the dense region rather than being independent draws from the top-90% stake pool. If this tendency dominates, the failure mode becomes bimodal per round — dense-origin blocks well-covered, sparse-origin blocks poorly covered — rather than a uniform 24% orphan rate. The actual strength of the tendency is an empirical question. See Improvement #8 for the analytical model and mitigations, and the appendix for the deep dive.
Four concrete measurements our machinery enables:
Both channels emit per-MiniProtocolNum DeltaQ traces on this o-n
branch. Measure the completion-time distribution for (a) big-ledger
single-shot and (b) NBL chunked/hedged fetches separately. Compare
against tcp-estimator predictions per channel. If reality matches,
the model is validated; if it diverges, that's diagnostic — which
caveat bit? Bandwidth-share model, p mis-estimate, RTT mis-estimate?
Cross-node arrival log + per-peer metadata (big-ledger flag) tells us, for each EB at each node, whether the winning path was big-ledger or NBL. Empirical validation of the 76.2% seed-hop claim. If the observed rate is 60%, the effective per-connection success is lower than assumed — investigate the churning/adversary caveats.
With per-mini-protocol per-connection estBurstS and the arrival
log, compute: bytes requested vs bytes delivered vs bytes actually
needed. The memo's worst-case 6 MB waste per round is a bound; the
empirical number tells us how tight it is in practice.
The persistently-slowest-node analysis from
leios-priority-measurement.md
also gives us the persistently-fastest node identity — likely the
seed nodes for many rounds. If seeds concentrate on a small set
(rotation between rounds is weak), the scheme's dependency on their
honesty is a real concentration risk.
The scheme is a specific instantiation of the G(t) that feeds
Yves's
- Under default report parameters, G(t) is the Mathis/CUBIC blended multi-hop diffusion CDF.
- Under the memo's scheme, G(t) is the parallel race between big-ledger single-shot and NBL chunked, which has fundamentally different tail behavior.
- We can compute
$F_{\text{full}\mid C}$ analytically under the scheme's G — plug the scheme-specific G into Yves's formula. Expected result: with 76% seed-hop coverage in 1–3 s and NBL chunked catching the remaining 24% in ~1.6 s,$F_{\text{full}\mid C}(14)$ should be ≈ 1.000 comfortably — but this is the body-only, honest-case figure. The scheme's real test is closure coverage and the adversarial$G_{\text{adv}}$ case (T22), where the two-channel race and seed honesty actually bind (seeleios-conditional-diffusion.md). - The empirical
$F_{\text{full}\mid C}$ from the testnet campaign directly validates or falsifies that. If the scheme delivers what it claims,$F_{\text{full}\mid C}$ should improve relative to the baseline (report's assumed G) or at least not regress.
The specific finding path: compute
The memo has several static-decision surfaces that could be replaced with observation-driven ones. Below are four incremental improvements that preserve the memo's structure, plus two more ambitious alternatives worth naming as research directions.
Before proposing changes, worth stating what the memo gets right:
- The asymmetric split (single-shot on high-BDP peers, chunked-hedge on low-BDP peers) matches TCP fundamentals. Any alternative needs to preserve this — treating high-BDP and low-BDP peers identically is worse than either extreme.
- The seed-node argument (~76% one-hop coverage via honest big-ledger seeds) is elegant and gives the mesh a strong probabilistic backbone. Alternatives should either preserve or replace with something quantitatively at least as strong.
- The one-big-request-per-high-latency-peer rule is right for sender-bandwidth reasons even if it costs receiver-P99. The receiver-cost is largely notional (bought back by seed-hop delivery); the sender-cost of the alternative is very real.
The refinements below don't touch these; they attack the memo's static-decision surface.
The memo's problem. The proxy "big-ledger peer = high-latency" is inexact. A big-ledger peer can be geographically nearby; an NBL peer can be far. Static classification is wrong for the actual mix.
The memo's underappreciated virtue. The static classification
is adversarially robust in one specific sense: big-ledger
membership is tied to real stake and cannot be forged by an
adversary just by manipulating their RTT signature. A pure
observation-driven classifier using PeerRTT.quantile alone can be
manipulated in both directions — trivially inflated (delay
responses) and, at higher cost, deflated (co-locate near victims).
The improvement, revised as a hybrid.
- Primary axis (adversarially robust): big-ledger membership. Peers in the big-ledger set → single-shot pool; peers outside → chunk-hedge pool. This is the memo's classification, preserved for its adversarial robustness.
-
Within-set refinement by estimated completion time (not RTT
alone): the single-shot big-ledger transfer is bulk, so completion
≈ slow-start ramp (RTT) + size ÷ throughput. Low RTT ≠ high
bandwidth — a nearby thin-pipe or heavily-loaded peer can lose to a
farther fat-pipe one — so order big-ledger peers by estimated
completion time using
PeerRTT(ramp) andestBurstS(throughput), the same per-hop transfer-time model that feeds$G(t)$ ; the NBL set uses the same estimate for chunk sizing (Improvement #2). Three caveats shape how to combine, not whether: consume the fusedestBurstS(raw is adversary-manipulable — pacing attacks — §fusion), fall back toward RTT whenestBurstSis stale (it needs recent bursts with the peer), and temper greedy min-completion selection with reliability history / a little diversity, since piling every node onto the fattest seeds worsens egress concentration (gap 19). - RTT cross-checked against alternative signals to detect adversarial manipulation (see "Multi-signal fusion for adversarial robustness" below).
Why hybrid rather than pure observation-driven. Pure RTT-based
classification is manipulable in either direction; an adversary can
move themselves into whichever pool gives them more leverage
(fake-high → single-shot critical-path slot; fake-low → many
chunk-hedge slots). Big-ledger membership requires real stake and
can't be forged via RTT games. Using it as the primary axis
preserves adversarial robustness; refining within the set on RTT and
(fused) estBurstS extracts the performance gain with the primary
axis — plus the hedge and Improvement #3's stall detection — capping
the exposure of the manipulable throughput signal.
Cost. A few kilobytes of per-peer state (already in PeerRTT).
Reclassify every ~minute. Threshold hysteresis needed to avoid
flapping. Additional bookkeeping for the cross-checks — see the
fusion section below.
The memo's problem. Static 32 kB (EB body) and 64 kB (closure) chunk sizes are one-size-fits-all across all NBL peers. A peer with 25 ms RTT and 100 Mbps effective throughput has BDP ≈ 313 kB; a 32 kB chunk fits in ~10% of one BDP window, wasting the peer's slow-start-cleared cwnd on unnecessary application-limit stalls.
The improvement. Size each chunk to a fraction of the peer's
bandwidth-delay product. estBurstS is a per-byte cost
(seconds/byte — throughput is
with
Cost. A single-line computation per peer per round. Trivial.
Adversarial note. Both inputs (estBurstS and PeerRTT) are
adversary-manipulable, letting them tune their assigned chunk size
to maximise leverage (bigger chunk → more damage per slow-loris
instance). The chunk-size formula should consume fused estimates
(see the fusion section below), not the raw PeerRTT / estBurstS.
Capping chunk size by a reliability-adjusted BDP is one concrete
mitigation.
The memo's problem. The proposed 1 s timeout is a coarse instrument. If a peer commits to delivering 12 MB at 25 ms RTT × 100 Mbps and should be completing in ~1 s, waiting 1 s to notice they've stalled at 0 bytes wastes the full L_diff budget.
The improvement. Monitor each active fetch's throughput
(1/estBurstS — recall estBurstS is seconds-per-byte). If it drops
below (say) 25% of its recent-history value after commitment —
equivalently, estBurstS rises above ~4× — treat as a slow-loris
candidate. Cancel and reissue to another peer at ~100 ms rather than
1000 ms — a 10× faster reaction. The memo has this as an implicit
TODO ("How to detect X"); the branch's estBurstS machinery is
exactly the sensor.
Cost. Small STM tracking per active fetch. False positives are
possible if a peer's throughput legitimately varies (congestion
event), so require the collapse to be sustained (throughput stays
low / estBurstS stays high) — reduce the sample period for active
fetches (say 100 ms), giving detection at ~300 ms.
Adversarial note (pacing attack). An adversary can defeat this
specific threshold by pacing delivery just above the trigger —
holding throughput at (say) 26% of its recent value indefinitely, so
it never crosses the 25% floor (estBurstS stays just under the ~4×
rise threshold). The coarse 1 s timeout eventually fires, but the
adversary has stretched delivery over much of L_diff. The fusion
section below discusses the cross-check: delivery-progress rate
(wire-level bytes/sec) is a complementary signal that catches pacing
attacks — a peer whose advertised estBurstS looks fine but whose
actual wire-delivery rate stays low is manipulating one signal.
The memo's problem. The memo picks one strategy statically for the whole round: single-shot on big-ledger + chunked on NBL. Calibrated for the "typical" round. Under stress (adversarial pressure, network trouble, unusual load), the same strategy might not deliver in time.
The improvement. Start with the memo's low-egress scheme.
Monitor delivery progress against the fetch's own deadline (voter:
- Add chunk-hedge to big-ledger peers as well (accept the elevated egress cost for this round).
- Increase parallelism on the NBL side.
Trade ecosystem cost for reliability, but only when reliability is actually threatened. Rounds where the memo's baseline delivers in ~2 s stay cheap; rounds trending toward missing the deadline get "spend more to make it" behaviour.
Cost. A simple state machine + observation. The trigger threshold ("past halfway to the deadline, >50% missing") is a design parameter to tune.
The memo's problem. The chunking-and-hedging machinery is
statically configured — same chunk sizes and hedging thresholds
regardless of actual link conditions. The tcp-estimator's
parallel-chunking-low-p.md shows that at
The improvement. Consume TCPInfo's tcpi_retrans / tcpi_lost
(Linux + tracetcpinfo cabal flag) to estimate per-connection
loss rate. Adapt hedging aggressiveness:
- Low
$p$ (~$10^{-5}$ or lower): drop the hedging — the redundant overlapping requests whose P99 payoff has collapsed — and enlarge chunks. Chunking itself may stay for its resource-bounding benefits (receive-buffer footprint, cancellation granularity; tension 3), so this trims hedging, not a wholesale return to single-shot. - Moderate
$p$ (~$10^{-4}$): apply the memo's chunk-and-hedge as written. - High
$p$ (~$10^{-3}$+): hedge aggressively; smaller chunks, more hedging overlap at the tail.
Cost. TCPInfo scraping (already in scope for Frame 1 measurement). Adaptive-threshold computation per peer per round.
Adversarial note. TCPInfo's tcpi_retrans reflects our
kernel's perspective on lost packets. The adversary could inject
losses on their side to bias our observation, but that's expensive
(they'd be losing themselves out). Cross-check with per-connection
estBurstS variance in the fusion section.
The memo's problem. The memo has TODOs on cancellation semantics: when NBL delivers a chunk first, "TODO also send a cancel to the big-ledger peer". Without a concrete rule, wasted egress grows unbounded.
The improvement. Once NBL has delivered the overlapping content,
the big-ledger transfer of it is redundant; cancelling stops the
wasted egress but costs a round-trip and gives up that stream as a
backup. Use estBurstS as the heuristic for whether it's worth it:
- Fast big-ledger peer (low
estBurstS): shipping redundant bytes at a high rate, so cancel eagerly — that is where wasted egress per second is largest, and a fast peer applies the cancel promptly. - Barely-progressing peer (high
estBurstS): wasting little per second, so cancel lazily — the saving is small and the near-idle stream is a cheap backup if NBL then stalls.
(A heuristic, not a derived optimum: on a wasted-bytes basis a slow peer has more redundant bytes still to come, which cuts the other way — so it's a rate-vs-total-bytes call to tune.)
Symmetrically: if NBL peers have been slow but the big-ledger peer is
speeding up (estBurstS falling), prefer the big-ledger stream and
cancel NBL requests for the remaining chunks.
Cost. Small per-connection state to track the last-observed
estBurstS. Cancellation itself is a mini-protocol message.
The memo's problem. The 76.2% seed-hop delivery depends on having an honest well-connected big-ledger peer, but the memo treats big-ledger peers uniformly — no way to prefer the ones that are genuinely seed-behaving.
The improvement. Track estBurstS and delivery-success history
per big-ledger peer across rounds. A big-ledger peer that has
consistently delivered EBs fast (low estBurstS) for many recent
rounds is likely a genuine seed (well-connected, honest, adequately
provisioned). When selecting the single-shot big-ledger peer for
this round, prefer the historical best-performers.
Cost. Per-peer historical counter (few hundred bytes per peer, tens of KB total on a public relay).
Adversarial note. An adversary could earn seed-node reputation by delivering honestly for many rounds, then slow-loris on the round they care about. Detection at the single-round level (via Improvement #3) still fires; history-based preference is a soft prior, not a hard commitment. Reputation gets slowly earned but rapidly lost — a single bad round should drop the peer's rank substantially.
Caveat under the per-block-origin view. If the churn tendency
described in Improvement #8 dominates, "seed status" varies by
block-origin: peer estBurstS averaged
across mixed round-types is then a noisier signal than under a
uniform-seed model. Improvement #7 remains valuable for the
dense-origin regime (most rounds), but the sparse-origin regime
needs Improvement #8's coverage rather than #7's reputation.
The memo's problem (extension). The seed-hop calculation
The tendency, then, is for every node's big-ledger peers to
concentrate around whichever region produces the most blocks —
the dense region, weighted by stake — regardless of where the
receiving node itself sits. This is a tendency, not a certainty:
the hourly random re-draw injects fresh stake-weighted diversity,
scoring variance across rounds resists perfect concentration, and
the tendency's strength depends on how concentrated block
production actually is. The actual concentration on real networks
is an empirical question (see
leios-testnet.md's measurement recipe).
This section assumes the tendency exists and quantifies its
implications if it dominates.
Analytical estimate (per-block-origin model). Since the
concentration tips peers toward block-production regions rather
than each receiver's region, the failure mode is best captured by
stratifying blocks by origin, not nodes by region. Let
-
Dense-origin blocks (fraction
$1 - \beta$ ): most nodes' big-ledger peers are close to the origin → high seed-hop coverage, illustratively 85–90% mesh-wide. -
Sparse-origin blocks (fraction
$\beta$ ): most nodes' big-ledger peers are far from the origin → low seed-hop coverage, illustratively 15–25% mesh-wide.
Weighted mesh-wide average, illustratively with
The important structural feature: asymmetry across block-origins, not across nodes. Per-node orphan-status is approximately uniform: any given node is orphaned some fraction of the time, mostly on sparse-origin rounds. Per-round orphan-fraction is bimodal: dense-origin rounds have most of the mesh covered; sparse-origin rounds have most of the mesh un-covered. This inverts the "which 24% is orphaned" question — under this mechanism, it's a rotating set determined by block origin, not a persistent set determined by node geography.
The numerical range for the sparse-origin coverage (15–25%) is illustrative; the actual value depends on the (empirical) strength of the churn tendency plus the (empirical) distribution of block-diffusion latencies from sparse origins to the mesh.
The improvement. Modify the churn to preserve diversity. Four variants, in increasing intervention:
-
Soft penalty (diversity term in scoring). Add a
clustering-penalty term to the churn score:
$\text{final-score}(p) = \text{performance-score}(p) - \lambda \cdot \text{clustering-penalty}(p, \text{other peers in set})$ . Theclustering-penaltygrows with peer$p$ 's ASN/region overlap with the current 5-peer set. Tunable$\lambda$ trades performance for diversity. -
Hard constraint (minimum diversity). Require the 5-peer set to
span at least
$k$ distinct ASNs (or countries, or continents). Stronger guarantee but potentially forces keeping low-performing peers. - Snapshot reset (periodic refresh). Every ~24 hours (or N rounds), reset the 5-peer set to a fresh stake-weighted random sample. Let churn re-optimize within the new set. Sacrifices some steady-state performance but bounds the drift from uniform.
- Bimodal split (3 performance + 2 diversity slots). Divide the 5 slots: 3 managed by performance-based churn, 2 managed by anti-clustering (guaranteed diverse). Balances performance with diversity coverage.
What each variant addresses. Under the per-block-origin mechanism, the value of these mitigations is to ensure every node has direct big-ledger connectivity to diverse block-origin regions — so that sparse-origin blocks get seed-hop delivery too, not just dense-origin ones. The variants differ in how strongly they guarantee this:
- Soft penalty: incentivises but doesn't guarantee diversity; strong churn signals can still concentrate the set.
- Hard constraint: guarantees at least
$k$ distinct regions, so blocks originating in at least one non-dense region are covered. - Snapshot reset: periodically returns the peer set to a random stake-weighted sample; concentration re-develops over the interval between resets, so the average concentration is bounded but not eliminated.
- Bimodal split: 2 slots definitely diverse (guaranteed connectivity to non-dense regions, so sparse-origin blocks reach some node peers), 3 slots follow performance-based churn (retains most performance).
Recommended variant. The bimodal split is probably the best practical option — modest complexity, quantifiable diversity guarantee (2/5 slots definitely diverse), retains most of the churn's performance benefit. Snapshot reset is a good complement (bounds long-term drift).
Cost. Peer ASN/country lookup (MaxMind or equivalent, already in scope for the multi-signal fusion below — see the fusion section's cost breakdown). Clustering-penalty computation per churn cycle (negligible). Snapshot reset introduces some churn overhead every reset window.
Adversarial note. An adversary aware of the diversity policy could distribute their controlled peers across ASNs/regions to avoid triggering the clustering penalty. Defense requires layered diversity — ASN + country + peer-graph structural diversity — such that manipulating any one axis doesn't defeat the policy. Concrete instance of the multi-signal-fusion principle applied to the peer-selection layer.
The four improvements above rely on runtime observations that are individually manipulable:
-
PeerRTT.quantile(app-layer round-trip via cookie echo). Trivially inflatable by delayed response; deflatable only with real co-location (expensive but possible for a resourced adversary). -
estBurstS(app-layer throughput). Adversary can pace delivery just above the detection threshold, or deliver fast early then stall late. - Delivery progress (bytes-received / bytes-total). Not manipulable — the adversary can't fake bytes not delivered — but only observable when data is actually flowing.
A robust adaptive scheme fuses these signals rather than relying on any single one. Available cross-checks:
PeerRTT.quantile vs tcpi_rtt. Cookie-echo measures
application-layer round-trip; tcpi_rtt measures TCP-layer
round-trip from the kernel's own ACK timing. Honest peers have
PeerRTT.quantile ≈ tcpi_rtt + (small processing latency). A
large gap (PeerRTT.quantile ≫ tcpi_rtt) is a signature of
application-layer delay — the adversary holds the echo while the
kernel ACKs promptly. (Linux + tracetcpinfo compile flag only;
usable on measurement nodes.)
Geographic minimum RTT. From a peer's IP, compute
PeerRTT.quantile below
Historical stability. Honest peers have stable RTT over
hours-to-days. TimedDigestWindow already captures the RTT
distribution; a coefficient-of-variation metric over a longer
window flags peers whose RTT is unusually variable — a signature
of intermittent adversarial delay.
Delivery reliability history. What fraction of past rounds
has this peer meaningfully contributed to? High advertised
capability but low actual contribution is suspicious regardless of
what their instantaneous PeerRTT or estBurstS claim.
Concretely, each improvement should consume a fused sensor value rather than a raw one:
-
#1 (classification): big-ledger membership is the primary
axis (non-manipulable). RTT-based within-set refinement uses
fused RTT (cross-checked against
tcpi_rtt, geographic minimum, historical stability). -
#2 (chunk sizing): use fused estimates of RTT and
estBurstS; cap chunk size by a reliability-adjusted BDP rather than raw BDP. -
#3 (slow-loris detection): an
estBurstSrise (throughput collapse) is one signal; cross-check with delivery-progress rate (wire-level bytes/sec) which the adversary cannot fabricate. Pacing attacks that keepestBurstSjust under the rise threshold still show a poor wire-delivery rate. - #4 (progressive escalation): robust as-is (progress observation is direct); could optionally weight progress by peer reliability history to discount honest-looking but poorly-performing peers.
-
#5 (hedging decision from p): TCPInfo's
tcpi_retransis one signal; cross-check withestBurstSvariance (high variance at claimed-low p is suspicious) and historical stability. -
#6 (cancellation logic):
estBurstSalone decides eager vs lazy cancel today; adding delivery-progress rate as a second signal catches peers whoseestBurstSlooks fast (low) but that aren't actually clearing bytes. -
#7 (seed-node preference): reputation history is a soft
prior; fuse with per-round
estBurstSobservation. Reputation earns slowly but is lost quickly on observed under-performance. - #8 (diversity-aware churn): ASN / country lookup is one diversity signal; layer with peer-graph structural diversity (from topology survey) and historical stability. Prevents the adversary from gaming a single-axis diversity metric.
Multi-signal fusion is the concrete mitigation for Proposal A of
threat-model-additions.md —
observation-manipulation attacks (T34, T35). The proposal argues
that adaptive protocol behaviour based on local observation opens
a new attack surface (adversary manipulates observations to
weaponise the target's own adaptive behaviour). Fusion is the
defensive layer: no single observation is authoritative, so
adversarial manipulation of any one signal is caught by the
others.
Bounded:
- TCPInfo scraping on Linux measurement nodes (already in scope for the Frame 1 validation campaign).
- Geographic-IP lookup — MaxMind or equivalent DB (few MB), microsecond lookups, refreshed daily.
- Historical-stability tracking — reuses the
TimedDigestWindowmachinery, adds a variance metric. - Delivery-reliability history — a small per-peer counter updated per round.
- Total storage: ~1–2 KB per peer for the composite signal, few MB total on a public relay.
Non-Linux nodes lose the tcpi_rtt cross-check and have weaker
adversarial robustness — a real bifurcation between Linux and
non-Linux nodes that CIP-0164 governance should address (see
gap 14 of leios-report-gaps.md on
fallback behaviour when adaptive machinery is unavailable).
Idea. Rather than reactive fetch, use proactive reservation. When an EB header arrives, the node immediately registers interest with N peers. Each interested peer commits to deliver a specific chunk (or the whole thing) and starts pushing. No pull requests — the delivery pattern is negotiated once, then executed.
Why it might be more robust.
- Cancellation semantics are baked in from the start (peers know who else is delivering what).
- Sender-side scheduling can optimise across multiple concurrent recipients.
- No "who offers first" race — the schedule is fixed at header-arrival time.
Why it might not.
- Requires a new mini-protocol (reservation channel).
- Adversarial modelling is harder (compromised reservations can silently block).
- Less nimble under changing conditions.
Idea. Split the EB into
Why it might be more robust.
- Slow-loris on individual chunks no longer critical-path.
- Redundancy factor
$M/K$ is tunable — small$M$ for cheap ecosystem operation, large$M$ under stress. - Naturally combines with escalation: adjust
$M/K$ based on observed conditions.
Why it might not.
- Significant complexity (Reed-Solomon or similar encoding at the sender, decoding at the receiver).
- Encoding CPU cost. At 12 MB EBs, RS coding is milliseconds of CPU per encode/decode; not prohibitive but not free.
- Requires reworking the block-fetch protocol.
- Changes the "seed" property — an honest big-ledger seed can no
longer deliver the full payload; they deliver
$K$ chunks.
Erasure coding is genuinely more robust to slow-loris but at a real complexity cost. Worth prototyping as a research direction if the incremental improvements aren't sufficient.
Improvements #1–#8 together preserve the memo's structural elegance and add adaptive behaviour that makes the parameters context-appropriate rather than statically-tuned. The combination attacks the memo's weakest points:
- Static classification → measured classification with adversarially robust primary axis (#1).
- Static chunk size → per-peer optimal (#2).
- Static 1 s timeout → observation-driven detection (10× faster reaction to slow-loris; #3).
- Static strategy → escalation under stress (#4).
- Static hedging assumption → adapted to measured loss rate (#5).
- Under-specified cancellation → observation-informed cancel policy (#6).
- Uniform big-ledger treatment → historical seed-node preference (#7).
- Uncorrelated 5-peer assumption → diversity-aware churn (#8).
Each is a small incremental change. Together they turn the memo's static design into a control-system design that responds to observed conditions, backed by the multi-signal fusion layer for adversarial robustness.
Alternative B (erasure coding) is genuinely more robust but architecturally larger — hold it as a research direction if the incremental improvements prove insufficient.
Alternative A (reservation-based) is intriguing but under-specified without a concrete protocol proposal.
Prerequisite status (important). Several improvements below cite
estBurstS as a live, per-peer, per-protocol, sub-second sensor. It
is not that today: estBurstS is emitted on the same ~10 s
TraceRecvDeltaQSample cadence, is per-connection and
protocol-mixed (the per-protocol split is the unimplemented Shape B
in track.md), and has no STM reader view — only PeerRTT is
exposed to running mini-protocols. So #2/#3/#6/#7 as described are
gated on three build items: (a) the per-protocol StatsA split
(Shape B), (b) an estBurstS reader view analogous to PeerRTT, and
(c) a tighter sampling cadence for active fetches. These are
straightforward extensions of existing machinery, but they are TODOs,
not shipped capabilities — treat the improvements as a design that the
branch enables, not one it currently supports out of the box.
With that caveat, all eight incremental improvements depend on the branch's mux telemetry (some also on Linux-only TCPInfo or peer-metadata lookups):
-
#1 needs
PeerRTT.quantile— direct output of the cookie-echo scheme. -
#2 needs
PeerRTT / estBurstS(the BDP) — both from the branch's traces. -
#3 needs
estBurstSmonitoring with sub-second cadence — the current sampling period could be tightened for active fetches. -
#4 needs progress monitoring on active fetches, cross-referenced
with round timing — traceable via existing block-fetch traces plus
per-connection
estBurstS. -
#5 needs
tcpi_retrans / tcpi_lostfromTraceTCPInfo(Linux +tracetcpinfocabal flag). -
#6 needs per-connection
estBurstSin real time to inform the eager-vs-lazy cancel decision. -
#7 needs per-peer historical
estBurstSand delivery-success aggregation across rounds. -
#8 needs peer ASN/country metadata (from IP database lookup —
orthogonal to mux telemetry), plus historical
estBurstSfrom #7 for the performance component of the split.
Without the branch's machinery (plus the per-protocol split, reader view, and cadence prerequisites noted above), none of the eight are implementable in the way described. This is a concrete case where the observability work unlocks specific protocol-design improvements that the memo's static scheme can't achieve — though the unlocking is partly prospective, gated on the prerequisites, not all present today.
The natural experiment: on the mininet lab or the testnet, compare four configurations:
- Memo's scheme as written (static, no adaptation) — baseline.
- Memo + improvement #1 (hybrid classification only) — the smallest deviation from baseline, tests whether within-set RTT refinement alone changes anything.
- Memo + improvements #1–#8 (full adaptive variant with fusion and diversity-aware churn) — the strongest incremental proposal.
- Erasure-coded variant (if we prototype Alternative B) — the architectural alternative.
Metrics per configuration:
- Empirical
$F_{\text{full}\mid C}(14)$ (perleios-conditional-diffusion.md). - P99 completion time.
- Redundant egress per round.
- Adversarial resistance under a controlled slow-loris injection (memo's key risk).
This gives direct data on which improvements pay off and by how much, and whether the incremental refinements are sufficient or Alternative B is warranted.
-
The scheme's real-world efficacy is unproven — it's a mininet-lab proposal, not deployed. Our observability judges it if deployed.
-
The two-channel independence assumption is not empirically vetted. If NBL peers share bottleneck failures with big-ledger peers (same ISP path, same regional bottleneck), the parallel-race gains are less than the model suggests.
-
Receiver-downlink self-contention is a distinct, unmodelled cost. The independence caveat above is about the sender side (shared upstream bottlenecks). Even with fully independent senders, the big-ledger 12 MB single-shot and the 20 NBL chunk streams all arrive at the same receiver's downlink. Below the receiver's aggregate line rate the hedge slows both channels — the parallel race the scheme relies on degrades into self-competition at the receiver. The memo's waste accounting is entirely sender-side (egress bytes) and never asks whether the racing channels contend at the receiver. Relatedly, the memo's ≥12 MB receive buffer for the big-ledger single-shot is a bufferbloat source for the thin protocols co-multiplexed on that connection. Measurable as receiver-side
estBurstSacross the concurrent channels during a fetch. -
The seed-node coverage is 76% marginal, not 76% joint per round. The 76% is a per-node probability (Marginal prop: any given node is covered by a seed on any given round). The 24% orphan set on a specific round may or may not be the same nodes every round (Joint prop: on any given round, exactly 76% of nodes are covered) — a distinction that matters for which of our proposed interventions actually pay off:
- Rotating (lottery-based) orphans. Committee sortition varies per round; a node's orphan-status varies with it. Over time every node experiences orphan-status ~24% of the time; no stable slow-node identity to target.
- Persistent (structural) orphans. Some nodes have big-ledger draws that systematically miss honest-committee coverage — e.g., a node whose initial peer draw was randomly poor and hasn't recovered. Same nodes orphaned every round; targetable.
There is a distinct third mechanism worth naming — one that looks like orphaning but has a very different structure:
-
Per-round bimodal coverage (churn-induced tendency). The peer-selection churn scores peers by "first to announce/deliver", which depends more on proximity to the block producer than on proximity to the receiving node. If the tendency does dominate, it produces a bimodal per-round pattern: dense-origin rounds have high mesh-wide seed-hop coverage; sparse-origin rounds have lower mesh-wide coverage. Per-node orphan-status is approximately uniform (any node can be orphaned in a sparse-origin round); per-round orphan-fraction is bimodal. See Improvement #8 below for the analytical model, and
leios-testnet.md's measurement recipe for direct empirical validation on the testnet or mainnet observation.- There are analytical reasons to expect this tips the mesh's big-ledger peer sets toward the block-production region (dense, stake-weighted) over time — not exclusively, but as a tendency. Countervailing forces exist within the big-ledger pool itself: the hourly churn's random re-draw (1 of 5 replaced each cycle) injects fresh stake-weighted diversity, scoring variance across rounds prevents perfect concentration, and if block production is genuinely distributed across regions the effect is diluted. Note that peer sharing does not counteract the tendency — it's an NBL-only mechanism and doesn't affect the big-ledger set, which must come from ledger sampling. The actual strength of this concentration is therefore an empirical question.
-
Mixed (likely reality). Mostly rotation with a small persistent tail.
Interventions are joint-distribution-sensitive. Fixing "the persistently orphaned 24%" only works if that set exists as a stable identity (persistent case). Round-adaptive throttling only helps if failures cluster within rounds (bimodal case). The 76% marginal number is silent about which case dominates, so the memo's seed-node argument is comfortable at 76% marginal but tells us nothing about whether the 7 s round budget is met on the hardest rounds. The campaign disambiguates by computing $\hat{F}{\text{full}\mid C}(14)$ separately for dense-origin and sparse-origin rounds (see
leios-testnet.md's churn-tendency measurement recipe, item 7): a materially lower $\hat{F}{\text{full}\mid C}(14)$ on sparse-origin rounds is the empirical signature of the bimodal case; a stable orphan set across all rounds is the persistent case; approximately equal$\hat{F}_{\text{full}\mid C}(14)$ across round types with the orphan identity varying is the rotating case.Which regime dominates determines whether "slowest-node targeting" (Frame 2 adaptive lever) and historical seed-node preference (Improvement #7) are high-leverage or wasted effort. Directly measurable from the per-EB arrival log (
leios-priority-measurement.md): compute how often specific nodes appear in the orphan set across many rounds, fit a rotation-vs-persistence mixture. This is the same cross-node covariance question raised inleios-conditional-diffusion.md(Yves's caveat 1 / gap 2), applied specifically to the seed-node lottery rather than to raw arrival times. -
The interaction with tx-submission's direction inversion is not addressed. The appendix treats the EB body/closure as flowing on the same channel semantics as blocks, but tx-submission (which delivers the closure content pre-diffusion) works in the opposite direction. Care needed if the memo's cancel semantics extend to pre-diffusion.
-
The scheme's response to observed p variation. The memo picks static chunk sizes for the default p; real-world p variation could change the optimal parameters significantly.
- Compute
$F_{\text{full}\mid C}(14)$ analytically under the scheme's G(t). Does the memo's approach actually give the safety property in the model? - Deploy the scheme (or its variants) in the mininet lab or on the testnet with the branch's tracing enabled. Measure the four metrics above.
- Explicit adversarial-analysis of the "slow loris picks the first big-ledger peer" attack. What fraction of nodes need to independently pick a different big-ledger peer for the neighborhood to be protected?
- Interaction with the tcp-estimator's conditional metric
(
P99 | ≥1 loss) at low p. If most SPO paths are effectively loss-free at p ~ 1e-5, is the whole chunking machinery justified? - Extension of the scheme to the pre-diffusion (tx-submission) path. The memo focuses on EB body/closure fetch; the same asymmetric logic could apply to tx-submission.
The claim "at high RTT (250 ms+), the floor dominates and chunking gains cap at ~65%" needs unpacking.
Every TCP file transfer's wall-clock time decomposes into two parts:
-
Slow-start floor — the unavoidable time to ramp cwnd from its
initial value (typically 10 MSS = 14 kB) up to whatever's needed
to fit the payload. This grows by doublings each RTT:
$$T_{\text{floor}} \approx \lceil \log_2(N/n / W_0) \rceil \cdot \text{RTT}$$ where$N/n$ is per-chunk bytes and$W_0$ is initial cwnd. This is linear in RTT; chunking shortens it only through the$\log(N/n)$ term — more (parallel) chunks mean a smaller per-chunk payload and thus fewer doublings, a weak logarithmic reduction — while the RTT factor itself is irreducible by chunking, redundancy, or any application-layer trick. -
Loss-induced tail — extra time from packet losses triggering
fast retransmit, congestion avoidance, or worse. At loss rate p,
the expected number of losses per transfer is
$p \cdot N/n$ ; the tail is dominated by rare rounds hitting multiple losses.
P99 = floor + loss tail. Chunking attacks the tail, not the floor.
From the RTT sweep in parallel-chunking-results.md (12 MB payload,
1 Gbps link, p=1e-4, n=32):
| RTT | Floor |
Baseline P99 | n=32 P99 | Relative gain |
|---|---|---|---|---|
| 50 ms | ~0.5 s | 6.6 s | 1.7 s | −75 % |
| 250 ms | ~2.5 s | 15.9 s | 6.0 s | −62 % |
| 1000 ms | ~10 s | 31.4 s | 11.4 s | −63 % |
- At 50 ms: floor is 7.6% of baseline P99; loss tail is 92%. Chunking kills most of the tail. n=32 P99 = 1.7 s.
- At 250 ms: floor is 16% of baseline; chunking hits it hard but P99 can't drop below 2.5 s. n=32 P99 = 6.0 s.
- At 1000 ms: floor is 32% of baseline; chunked P99 is 11.4 s — only 1.4 s above the floor. Chunking is very effective at killing the remaining tail, but there's just too much floor.
Perfect chunking (
At 50 ms:
At 1000 ms:
So the ~65% plateau is the asymptotic upper bound at high RTT — you can't do much better no matter how aggressive you get with chunking.
The "chunking gains cap at ~65% at high RTT" result above assumes parallel chunking across many distinct peers — each chunk fetched from a different peer, all racing simultaneously. This must not be confused with serial small-chunk requests to one peer.
Serial chunks on one peer — request 32 × 400 kB from a single high-latency peer, sequentially. The sender's TCP goes application-limited between chunks: each chunk incurs a request-response RTT gap that leaves a wide-BDP pipe underfilled, so the path never reaches steady state. (On one persistent connection cwnd persists across chunks — it is not a fresh slow-start each time — unless an inter-chunk gap exceeds the RFC 7661 idle threshold, which resets it.) This is what the memo forbids in "requests to a high-latency peer should request as much (useful) data as possible" — correctly, because serial small chunks on a high-BDP peer are wall-clock disaster.
Parallel chunks across many high-latency peers — request each of 32 × 400 kB from a different high-latency peer, all in parallel. This delivers the 62% P99 reduction from tcp-estimator's RTT-sweep at 250 ms — the max-of-32 statistical gain plus per-chunk-shorter tail. The memo also forbids this ("Never request the same EB body or parts of the same EB closure from more than one big ledger peer at a time"), but for ecosystem-level reasons, not receiver-P99 reasons.
The memo's rationale for forbidding parallel chunking across big-ledger peers:
- Sender bandwidth efficiency. A high-BDP peer serving 400 kB fills its cwnd once, then goes application-limited. Each big-ledger peer's egress capacity is wasted on 32 small transfers rather than one large one. Aggregated across the mesh, that's substantial wasted big-ledger egress.
- Egress concentration on the big-ledger set. Every node asking 32 big-ledger peers for 400 kB (rather than one big-ledger peer for 12 MB) concentrates and fragments the demand on big-ledger peers — worse for their scalability as a stake-weighted subset.
-
Adversarial resistance via the seed argument. The 76.2%
one-hop coverage depends on sequential big-ledger use — one
node in each neighborhood uses its big-ledger peer while others
don't. Under parallel chunk-hedge specifically, every peer's
chunk is required, so every peer is on the critical path — an
adversary controlling one big-ledger peer blocks completion for
every target that has them as a peer. Precisely: if
$\pi$ is the probability that the adversary is one of my 5 big-ledger peers (~1/5 if the adversary controls ~4% of the big-ledger pool), then critical-path probability is$\pi \times 1/5$ under sequential-use (both filters — "in my set" and "selected first") vs$\pi$ under parallel chunk-hedge (only the "in my set" filter — every peer is always on the critical path). The 5× ratio is the selection-order filter that the seed-node argument depends on; parallel chunk-hedge collapses it. (Note that under parallel redundant-hedge — each peer delivers the full payload — the adversary's slow-loris is defeated by any honest peer's completion; but the 5× egress waste is what rules that variant out on the sender-efficiency grounds above.) - Cancellation-cost containment. If 32 big-ledger peers each deliver a chunk in parallel with NBL peers chunking, the wasted egress bound explodes. The memo's worst-case ~6 MB waste assumes only NBL parallelizes.
The trade-off, made explicit. The memo trades ~10 s of absolute P99 reduction at 250 ms RTT (16 s → 6 s achievable via parallel chunking across big-ledger peers) for sender bandwidth efficiency, big-ledger egress preservation, and adversarial resistance via the seed-node argument. Whether that trade-off is worth it depends on:
- How much big-ledger egress the mesh can spare (empirical question).
- How much the seed-node one-shot delivery "buys back" for the 76% of nodes that have a seed peer (analytical + empirical).
- How large the residual 24%-orphan problem is under the memo's
scheme (Yves's
$F_{\text{full}\mid C}(14)$ under the scheme's G).
All three are measurable on the testnet — a controlled comparison of "memo's scheme" vs "naive parallel-hedge everyone" would quantify the trade-off directly.
- On truly intercontinental big-ledger paths (RTT ~250 ms+), single-shot has a ~2.5 s floor and ~16 s P99 for 12 MB. Parallel chunking across high-latency peers would reduce that P99 to ~6 s — a 10 s absolute win — but the memo forbids it for the ecosystem-level reasons above. The scheme depends on the seed-node one-shot delivery covering most nodes; the 24% orphan nodes still see the full single-shot P99 cost.
- On NBL / low-latency paths (RTT ~25–50 ms), parallel chunking is worth the complexity. 75% P99 reduction is the difference between "reliably within L_diff" and "occasionally blowing the deadline". This is where the memo's chunk-and-hedge proposal shines — and this is where the receiver-P99 gain the memo does embrace materialises.
- The seed-node one-shot delivery is what "buys back" the receiver-P99 gain the memo declines to take from parallelizing big-ledger peers. If 76% of nodes get a fast one-shot from an honest big-ledger seed (~1–3 s completion) anyway, the receiver-P99 loss on the big-ledger side is largely notional for most of the mesh. The remaining 24% is where the risk lives, and NBL chunking there is meant to catch them.
- The two paths have qualitatively different tail structures. Big-ledger single-shot has a fat tail (one packet loss late in a 12 MB transfer = full retransmission cost). NBL chunked has a tightly-managed tail (loss on one chunk doesn't stall the others).
-
The scheme's total P99 is dominated by
$\min$ of the two channel P99s, when both channels are racing. If big-ledger P99 is 6.7 s and NBL chunked P99 is 1.6 s, the parallel-race completion P99 is close to 1.6 s — whichever channel finishes first wins. -
But only if both channels are actually racing. If NBL has
failed (e.g., no chunks arrived) and big-ledger is stuck in a
loss retransmit, the combined P99 approaches
$\max$ , not$\min$ . The scheme's robustness depends on genuine channel independence. -
The CDF is a staircase (
parallel-chunking-cdf-staircase.md). Discrete regimes (no loss / one loss in slow-start round r / more losses) each concentrate runs at a tight time value. Chunking amplifies the step structure. Real-world measurements should show a matching staircase; if not, the model is wrong.
The 6.72 s → 1.65 s figures (and the RTT sweep) come from the
tcp-estimator Monte Carlo (io/estimator). Its model has limits that
should travel with the numbers:
- No loss-recovery time. A loss only reshapes cwnd for the next round — no RTO, no retransmit stall, no head-of-line delay. Real P99 tails are often RTO-dominated, so the modelled absolute times are best-case. (Chunking is a tail-shrinker, so its relative benefit is if anything understated — but don't read the absolute completion times as realistic.)
- i.i.d. Bernoulli loss only. Every published table uses per-packet independent loss; a Gilbert-Elliott bursty model exists in the code but is unused. Real loss is bursty and correlated.
-
Chunk independence is analytic, not simulated. The whole-file
CDF is
$F_{\text{chunk}}^{,n}$ (i.i.d. max), so shared-bottleneck correlation between chunks — the receiver-downlink case above, or common upstream loss — is not modelled; it would fatten the tail and erode the chunking gain. - Single flow, exogenous constant p, one RNG seed per table row. No competing flows, no self-induced congestion; row point-estimates are single-seed (a stability tool mitigates, but the tables are not multi-seed).
- Low-p conditional reduction is ~5×, not the ~10× first reported (retracted in-doc).
The right falsification hook is the staircase CDF above; the resolution for the correlated-loss and shared-bottleneck limits is to measure per-hop completion on the testnet rather than trust the sim.
Improvement #8 introduces the per-block-origin model with
illustrative numbers. This appendix expands on the analytical
treatment, including an alternative correlation-parameter model,
what determines the per-origin coverage ratio in practice, the
asymmetric-impact structural feature, and the mesh-wide effect on
All the analysis assumes the churn tendency exists and dominates.
Its actual strength on real networks is an empirical question —
see leios-testnet.md's measurement recipe.
An alternative angle: treat the 5 peer-draws as a mixture with an
effective correlation coefficient
-
$\rho = 0$ : independent draws.$P(\ge 1 \text{ seed}) \approx 80%$ (baseline). -
$\rho = 1$ : all 5 peers effectively drawn from the same sub-region — whichever region the churn signal points them toward, which is the block-producing region under the churn tendency.
At intermediate
The correlation-parameter model is more mathematically tractable but less intuitive than the two-population (dense-vs-sparse-origin) model. Both give qualitatively similar predictions; use whichever is easier to reason about for a specific analysis.
Let
Roughly:
For Cardano's real distribution — heavily weighted toward
Europe / N. America stake — dense-origin blocks vastly outnumber
sparse-origin blocks in stake-weighted expectation, and the churn
signal follows. Illustrative ranges for
-
$k \approx 0.5$ : modest concentration — sparse-origin blocks still find some churn-retained peers across the mesh. -
$k \approx 0.2$ : strong concentration — few mesh nodes have peers near sparse origins. -
$k \approx 0.05$ : near-total concentration — essentially only nodes physically near sparse origins have coverage.
Three factors push
- Stake-weighted block production. Sparse-region SPOs produce proportionally fewer blocks. The churn signal has more dense-region evidence to work with; sparse-region peers rarely win the "first to announce" race.
- Announcement-race dynamics. Block-diffusion latency from producer to peer typically dominates per-hop RTT to the receiver, so peers near block origins win regardless of their RTT to the receiver.
- Committee-eligibility concentration. Honest committee members concentrate by stake weight (Poisson sortition is stake-weighted). If stake concentrates in the dense region, committee members do too — even setting the churn dynamics aside.
Three factors push
- Random re-draw component of churn. Each hour, 1 of 5 big-ledger peers is replaced by a fresh stake-weighted random sample. This injects diversity roughly proportional to the sparse region's stake weight.
- Scoring variance across rounds. No peer wins every announcement race; variance in the score prevents perfect concentration.
- Distributed block production. If sparse-region SPOs produce a non-trivial fraction of blocks (say 15–20%), the churn signal itself has significant sparse-origin evidence, reducing the concentration pressure.
The observed
The failure mode is asymmetric across block-origins, not across nodes. Implications:
- Mesh-wide averages hide a bimodal per-round distribution. A 78% mesh-wide average might come from 85–90% coverage on dense-origin rounds (majority) and 15–25% coverage on sparse-origin rounds (minority). Both live inside the same "78%" number.
- Per-node orphan-status is approximately uniform — every node gets orphaned in sparse-origin rounds regardless of where the node sits. This resolves the rotating-vs-persistent question at the node level toward rotating: no persistent per-node slowness arises from this mechanism.
-
Per-round orphaning is highly correlated across nodes. On a
sparse-origin round, most of the mesh is simultaneously
orphaned. Exactly Yves's cross-node covariance concern (caveat
1 of
leios-conditional-diffusion.md) applied to the seed-node lottery: orphans co-occur within a round. - Interventions targeting per-node status (Improvement #7, slowest-node targeting from Frame 2) address a weaker signal under this mechanism. Interventions targeting round-type coverage (Improvement #8, block-origin-aware protocols) are the natural fit.
-
The safety-tail is dominated by sparse-origin rounds. Even
if they're a minority of rounds, they carry disproportionate
weight in the failure tail — those are the rounds most at risk
of
$F_{\text{full}\mid C}(14)$ dropping.
Under Yves's default report parameters
(
-
The blended-diffusion CDF's actual tail at 14 s. If sparse
regions have systematically worse multi-hop diffusion (fewer
hops available, longer paths, more congested links) than the
report's average, then sparse-origin rounds have both more
orphaning AND slower recovery. The tail is fatter than the
report models. Directly measurable via the empirical
$G(t)$ split by block-origin. - The correlation structure of orphans within a round. On a sparse-origin round, all orphaned nodes are simultaneously fetching via multi-hop diffusion from the same sparse origin. Their arrival times are highly correlated (shared bottlenecks, shared route). The "max over orphans" becomes "max over a correlated subset with joint-slow arrivals" — amplifying the tail beyond what independent-orphan modelling would predict.
Both effects compound on sparse-origin rounds — the same rounds where the tendency's failure is concentrated.
Empirical threshold estimate. Under Yves's parameters,
per-round
The "problematic" threshold depends less on absolute mesh-wide
seed-hop rate and more on how the sparse-origin rounds behave
specifically. This makes Improvement #8 (diversity-aware churn)
valuable not primarily for raising the raw mesh-wide average, but
for raising the coverage floor on sparse-origin rounds — the
worst-case rounds where the safety property is most at risk. Even
a modest improvement to sparse-origin coverage (say 20% → 40%
mesh-wide on those rounds) can disproportionately improve the
per-round
Improvement #7 (historical seed-node preference) has a weaker
rationale under the per-block-origin mechanism. "Seed status" is
per-block-origin: a peer that wins on dense-origin rounds may lose
on sparse-origin ones. Historical estBurstS averaged across
mixed round-types is a noisier signal than under a uniform-seed
model. Improvement #7 remains useful for the dense-origin regime
(most rounds) but doesn't address the sparse-origin problem.
Frame 2's "slowest-node targeting" is weaker still under this mechanism: no persistently-slow nodes exist, only persistently-slow round-types. The right operational lever is round-type detection (is this a sparse-origin round?) with stronger protocols applied during those rounds — essentially a version of Improvement #4 (progressive escalation) triggered by block-origin rather than by observed progress.
All these interactions are measurable — see the linked recipe.