Skip to content

improved peer ranking

Marcin Wójtowicz edited this page Jul 21, 2026 · 32 revisions

Robust peer ranking by delivery quality

This document is an attempt at a detailed investigation of the problem of how to rank our peers for the purpose of issuing various requests in the hope that they are fulfilled in the most timely fashion to meet various protocol deadlines. A substantial development is laid out, based on reasonable heuristics, efficiency and trust concerns, and principled analytical devices are leveraged in the pursuit of that goal. Discussions of various hazards, tradeoffs and caveats, which are inherent in a trustless and permission-less environment of the Cardano network, are provided and which also inform and supplement this analysis, and leave some issues deserving future enhancements identified. A novel addition admitted by the proposed scheme relates to hedging requests, and is outlined in the section When to hedge.

The departure point is given a node with some connected peers and a request which needs fulfillment, and several peers which could be burdened with the task, which one(s) should we select. What is needed to answer this question is, especially in the light of data-demanding protocols such as Leios, a sophisticated ranking system which exhibits a predictive power and which is adversarially robust. The metric we are most immediately concerned with is to rank those peers by how quickly we expect them to actually deliver what we want to fetch — EB bodies, closures — so fetch logic can route to the best and demote the worst (feeding the improvements proposed in leios-fetch-scheme.md — Improvement #1's peer classification, adaptive chunk sizing, and the like). The posterior ground truth is directly-measured completion timeT_wait, the observed request-to-full-reply timing — and where a fetch has completed we rank on it: a paced-slow peer measures slow — its delivery is exactly what T_wait records — so the signal can't be gamed. Its one limitation is that it is retrospective: you cannot measure a fetch you have not made, so to choose a peer before the fetch, and for chunk-sizing and diagnostics, we need a forward prediction — a time-to-deliver estimate, ramp(RTT) + estBurstS × size (hereafter the estimate), built from network-mux's DeltaQ/RTT instrumentation (estBurstS, seconds/byte, from its burst-SDU serialisation estimator; RTT from its cookie-echo window; a branch feature today, not yet on main).

Leveraging that instrumentation has a catch: the throughput input estBurstS is adversarially gameable — a peer can pace its SDUs so estBurstS reads fast (high apparent bandwidth) while it dribbles our data, earning a rank it doesn't deserve and winning traffic it then serves slowly. (RTT is manipulable too, but asymmetrically: a peer can't fake the cookie-echo RTT low — it can't echo faster than the real round-trip — so cheap manipulation only inflates it, which self-penalises by raising its time-to-deliver, especially for small transfers, as described later. Lowering its actual RTT means co-locating near the victim: real, but expensive, where gaming estBurstS is free. A co-located peer that looks genuinely fast while dribbling the payload is caught by the actual-vs-predicted per-fetch check, not by RTT.)

One prerequisite deserves flagging up front: the estimate needs the datum's size before the fetch starts, so the protocol must advertise the size of what it offers (an EB, a closure). In Cardano, payload sizes aren't carried in network-layer messages by default, but a protocol's message schema can mandate one — a CBOR-encoded field in its CDDL spec. The catch is trust — unless the size is bound to signed metadata (a header; see the size-manipulation section), an advertised size is just the peer's unverified word, since nodes rely on independent verification of the downloaded datum rather than each other's claims. Such an unverified size is a third gameable input to the estimate, and the most direct one, since the peer states it outright — a peer can win the fetch not by being fast but by understating the size (or merely undercutting what competitors advertise), steering us its way. The lie surfaces only after the fact — the delivered size overshoots the advertised one, so actual ≫ predicted — the same delivered-vs-advertised deviation tx-submission already penalises.

So the prediction has to be robust: a peer must not be able to look good in the estimate while delivering badly. This document lays out how — a promptness-gated burst window that yields a trustworthy estBurstS, why ranking on time-to-deliver estimate makes a gamer self-penalise, and an actual-vs-predicted backstop that audits the estimate against measured T_wait and churns consistently-off peers. The hierarchy throughout is that direct measured completion is the preferred basis: the peer ranking — the very decision a pacer sets out to game — rests on measured completion wherever a fetch has completed, and there pacing gains nothing. The estBurstS hardening exists to make the prediction trustworthy for the roles measurement can't fill: choosing a peer before a fetch, chunk-sizing, slow-loris, and diagnostics.


The attacks we consider

All four deny us a timely fetch — an EB body or closure that doesn't arrive in time for the round's diffusion/certification deadline — cheaply for the adversary. Three of them win by looking good enough to be chosen and then not delivering; slow-loris simply drip-feeds a fetch it already holds. They differ in how they're caught.

  • RTT manipulation. Make apparent RTT low to score well on the ramp term. A peer can't fake the cookie-echo RTT low — it can't echo faster than the round-trip — but it can buy a low real RTT by co-locating near us (expensive), or inflate its RTT by delaying echoes, which only self-penalises. Caught by: the ranking (inflation self-penalises); a co-located peer that then dribbles falls to the per-fetch check. → Using the RTT signals.
  • Pacing (idle-gap). Time the delivery so the estimate reads fast — a flatteringly low time-to-deliver — while the closure actually arrives slowly; win the fetch on that estimate, then dribble the payload. Caught by: the promptness-gated burst window and the adversary bind (the timing trick, and why it backfires, are spelled out there). → The window-sizing tension; The adversary bind.
  • Byte-padding. Make the estimate look good by serving other traffic fast, while dribbling the closure we actually asked for — targeted withholding (the data-withholding threats T20/T21). The estimate stays flattering because it reflects the peer's aggregate traffic, not our specific fetch. Caught by: per-fetch completion timing (T_wait) and the actual-vs-predicted check, on persistence. → The adversary bind (byte-padding).
  • Slow-loris. Accept the fetch and drip-feed it — no attempt to hide; it ties up our fetch slot and, at scale, our concurrent-fetch capacity, denying us throughput cheaply (we wait on it instead of a good peer). Unlike pacing it doesn't fake fast — it looks slow. Caught by: the ranking (it scores low) and throughput-collapse detection (estBurstS rising) — though recourse is limited, since no per-fetch cancel exists yet (only wait / reset / hedge, each a trade-off). → The bigger picture.

Those four game a single delivery measurement, and the machinery in the rest of this document is what catches them. A distributed blockchain setting adds three that game the ranking system rather than any one measurement — across time, items, and identities — and these lean on redundancy and peer-selection layers beyond the delivery score:

  • Reputation farming, then a deadline-timed defection. Behave well to earn a top rank cheaply over low-stakes rounds, then withhold the one fetch that matters — an EB near its certification deadline — and revert. Score-based churn reacts only to a sustained low score, so a single withheld body — one non-first-delivery against hours of farmed good score — barely moves the peer's standing: the betrayal is close to free. The defence is therefore fetch redundancy for consensus-critical items — hedge across peers (Improvement #5) so no single defector is decisive — backed by weighting a consensus-critical miss far above an ordinary one in the score. Redundancy, not punishment, is what protects the fetch.
  • Surgical withholding of the critical item. Byte-padding targets us across a whole connection; the per-item version serves everything fast except the single block/EB an adversary wants suppressed (censorship, fork support). A per-peer average rank barely moves, so the ranking alone won't flag it. The defence is protocol-level: multi-source diffusion fetches the item from someone else, and a missed consensus-critical fetch should weigh far above its share of the peer's history. The ranking's only job is to not rate the withholder top on the strength of the 99% it does serve.
  • Baseline poisoning via Sybils. The actual-vs-predicted threshold is a p99 over a population baseline of honest peers; in a permissionless network an adversary runs many peers and inflates that baseline until a byte-padder fits under the loosened band. The impact is bounded — it desensitises detection, it doesn't falsely accuse honest peers — and it's mitigated by robust, per-subnet-capped aggregation — weight the baseline by IP-prefix / ASN rather than by peer, so a Sybil cluster in one subnet can't dominate it (identities are cheap, distinct network locations are not) — and by capping the band with the estimate itself — a physical model of delivery time — rather than the empirical p99 alone: the plausible residual (serve time, loss-recovery, stalls) has a physical ceiling a poisoned baseline can't inflate past.

A note on the response rather than a distinct attack. Currently in cardano-node, churn is score-based: peers first to offer a header or first to serve a requested body score well, and the worst-scoring ~20% are evicted each hour and replaced with a random sample. (Transient failure is a separate, faster path — a peer that blows response timeouts under a long GC or under-provisioning is dropped immediately, not slowly churned.) This design is largely eclipse-resistant rather than a ratchet: the resample is random, so a Sybil fleet gets no better-than-proportional odds of backfilling a freed slot, and the worst-20% eviction is absolute, so it also culls the weakest of the adversary's own peers — even a brief local eclipse leaks back out as those marginal peers churn and random (mostly honest) draws replace them. What survives is only the patient top-scorer of the reputation-farming attack: an adversary can farm score to hold a hot slot, but it cannot use the churn machinery to displace honest peers or take over the set. Cheaply forcing an honest peer out isn't available — only expensive, on-path degradation (nation-state / ISP-tier, and not ranking-specific) can. Diversity-/anti-eclipse-aware selection (Improvement #8) hardens the residual, but the base mechanism already does most of the work.

Handling advertised-size manipulation

The estimate's size input is advertised, hence gameable. The organising fact is that for a hash-named datum the content fixes the true size — the byte count we will actually transfer — so any disagreement is a provable lie once the datum is in hand; the difficulty is only the choice before the download.

  • All advertisers agree. No steering — the common size scales every estimate identically, so ranking still turns on speed; run it normally. The twist is what it means if that agreed size proves a lie on download: no reachable peer told the truth, so either the whole neighbourhood is dishonest, faulty, or we are eclipsed. That is worth acting on beyond demoting the advertisers — but the response (defensively resampling for fresh peers to break a captured view) belongs to the peer-selection / anti-eclipse layer (Improvement #8), not to the delivery score. (A uniform honest size is entirely benign, misinforming only absolute deadline-feasibility, not the peer choice.)

  • Genuinely distinct datums (different hashes — several VRF election winners). Not equivocation; a capacity / protocol-storm question rather than a size dispute. Fetch what you can, prioritising by the consensus signal — which candidates the votes favour — and estimate each independently.

  • Same hash, conflicting sizes — the weaponised case, and the one to decide up front rather than defer to punishment. Because the hash fixes the content there is one true size; the advertisements are noisy, possibly adversarial observations of it, and a few cheap heuristics narrow it down before we commit:

    • Corroborate, and reject outliers. The size a plurality of advertisers agree on is the likely truth; a lone dissenter — above all one below the rest, the steering direction — is discounted, not averaged in.
    • Bound by a class prior. A size outside the plausible range for the datum class (a closure "advertised" at kilobytes where closures run to megabytes) is rejected on sight — a cheap filter for the crudest lies.
    • Weight by credibility when advertisers genuinely conflict. With no majority to corroborate, believe the higher-ranked, longer-honest advertiser over an unknown; reputation is a prior on truthfulness, not only speed.
    • Lean on the signal we measure, not the one we're told. We measure RTT ourselves; size we are merely told. So under residual size uncertainty, tilt toward the peer whose advantage holds across the plausible size range: a low-RTT peer wins a small datum outright and is no disaster on a large one (its ramp term stays small either way), whereas a far, high-throughput peer pays off only if the datum really is large. This both hedges the uncertainty and defangs the lie — a far peer that understates its size gains nothing, since we weight the RTT we measured over the size it claims.

    Rank on the estimate these produce and pick the best, random tiebreak among genuine equals. The after-the-fact check is then only a backstop, not the plan: on delivery the true size is known, so a peer whose advertisement was inconsistent is provably lying and is demoted — a stronger, cleaner tx-submission deviation penalty (inconsistency with canonical content, not measurement noise) — which also feeds the reputation the credibility heuristic leans on.

  • Conflict arrives mid-fetch (a second peer offers the same datum at a different size while the first is in flight). Same recourse as a slow peer — wait, reset, or hedge — but the switch must be driven by the first peer's observed progress, never the newcomer's claimed smaller size; hedging on the claim alone rewards the lie in dynamic form.

Two structural notes bound the problem. The dispute only bites on the expensive, pull-based body fetch — and headers are the opposite: small enough to be pushed (Praos and EB headers diffuse proactively), so we generally hold and have already validated the datum's header before choosing whom to fetch the body from. If that header carries the body's size, we can trust it outright: having verified the header — its signature and our usual checks — we have verified everything it commits to, the size included, no matter which peer pushed it. The offer is then checked against a value we already trust, not taken on the offering peer's word. That closes the hole for EBs (and Praos blocks) whose headers carry a size, leaving only unattested aggregates such as closures. And the damage is bounded regardless: the worst outcome of a size lie is a single recoverable mis-selection — caught by actual-vs-predicted, remedied by hedge/reset — never a failed fetch.

The window-sizing tension, and the gated extension that resolves it

Requests are small — often a single SDU — but responses arrive as a burst of many SDUs, and the cookie on our request is echoed back on that burst. The first response SDU's echo gives the RTT sample; the follow-up SDUs of the same burst feed estBurstS, the serialisation rate. A blocking request is the exception — we subscribe and the peer pushes its first SDU only once it has a block, so that echo's round-trip times the idle wait, not the network. Such a sample lands in the far tail, which the robust RTT read already discounts: we take RTT_hi at the p80–85 quantile — not p99 — precisely so a tail poisoned by GC pauses and hiccups does not move it, and a blocking-wait round-trip is just one more such tail value. The wait itself is no stall either, since the window opens only on that first SDU (see the stall signal). That poses the question the burst window exists to answer: for how long do we keep attributing incoming SDUs to this response's burst — counting them toward its throughput — before a new SDU counts as a fresh exchange instead?

Its job is accurate measurement, not anti-pacing. It must extend far enough to let a genuine multi-second closure accrue the byte-coverage that earns an estBurstS its confidence weight (the appendix) — closing early yields not just a shorter estimate but a less trustworthy one — while stopping short of folding a severe stall (a deep RTO backoff) into the rate, which should register as a stall event instead (below). A fixed value serves neither end, because it is wrong across the RTT spectrum rather than scaled to the peer (see Not tuning-free). And the window is not what denies a pacer — a pacer is caught regardless of the threshold (see the appendix), so the anti-gaming rests on coverage, the bind and the actual-vs-predicted backstop.

There is no fixed timeout to reach, then — the window extends while the peer keeps serving and ends when it goes quiet. That is the gate. For a fetch, where the peer streams a one-way response, "still serving" is read directly off the incoming SDU spacing: attribute SDUs to the burst while successive SDUs arrive within the RTT-relative spacing threshold G; a longer gap means the peer has stopped, and the burst ends there. A silent pacer's burst ends here too — but ending it is measurement, not the catch: the short burst carries low coverage and shrinks to the prior (appendix), and that shrinkage — at any threshold — is what costs the pacer its rank.

The spacing threshold has one honest hazard — a loss/RTO briefly stalls the in-order stream and could trip it — but this is a measurement hazard, not a hole in anti-gaming (which rests on coverage, the bind and the backstop, not the gate; see the appendix). RTT-relative sizing handles the common case: a fast-retransmit gap (~1 RTT) stays under the threshold and folds into estBurstS rather than tripping it — only a severe RTO-scale stall ends the burst, and that is a genuine event, registered by the stall signal. Any residual one-off trip is absorbed by the actual-vs-predicted backstop (below), which churns only a peer whose delivery is consistently worse than its advertised rate.

Using the RTT signals — and a circularity to avoid

There is a way to game the burst from the echo side. Coverage — the trust an estBurstS earns (the appendix) — accrues for as long as the burst window stays open, but nothing so far ties that to our RTT staying fresh. A peer that keeps SDUs arriving, holding the window open across what may be several distinct exchanges, while never echoing the cookies we send, banks coverage over a rotting RTT probe: a trustworthy-looking rate resting on a round-trip we last measured long ago, or never.

The fix binds the two together — coverage must be paid for in RTT freshness. To keep the burst, and the trust its coverage buys, the peer must echo something within G; every matched echo is a fresh RTT sample, so a burst that goes on accruing is one whose RTT we go on re-measuring, and a peer cannot bank coverage while starving us of RTT updates.

The gate that enforces this is the echo-side twin of the SDU-spacing one: while a cookie is outstanding, end the burst if none is matched within G. The clock is the time since the last matched echo, reset by each echo and restarted when a fresh wait begins (a cookie entering an empty queue), so it never carries a stale gap into a new wait. It needs no queue traversal — only whether a cookie is outstanding, plus those two timestamps the cookie machinery produces anyway; the queue's one structural job stays RTT-matching, and the gate rides on it for free. Cumulative eviction keeps it honest: a match drops every older cookie with it, so a peer echoing a fresher cookie still resets the clock and leaves nothing stale behind — we never ask which cookie is oldest.

The "while a cookie is outstanding" clause is what leaves a plain one-way fetch alone: its single request cookie is matched on the first response SDU, the queue empties, and with nothing outstanding the gate is dormant — the SDU-spacing gate carries the stream. The echo gate bites only where cookies keep flowing (another protocol exchange, or mux-injected cookies), which is exactly where the starvation it guards against lives.

Two things keep this sound. First, it is a wall-clock receive-side gap, not a comparison of the oldest cookie's age against the self-measured RTT quantile — which would be circular: a peer delaying every echo uniformly inflates the front-age and the RTT together (a cookie echoed at t0 + RTT + D gives both a front-age peak and an RTT sample of RTT + D), and G = k·RTT_hi rises with it, so that test never trips. The gap sidesteps the loop because G is fed only by matched echoes, while the gap is measured independently of them. Second, resetting the gap requires an echo, and both of the peer's options cost it: a recent cookie (a low, honest sample → RTT fresh) or a stale one (a high-lag sample → high RTT_hi → ranked slow) — refresh RTT, or be ranked slow. It is also robust where a span would over-fire: a peer echoing recent cookies while abandoning a pile of old ones has fresh RTT and rightly does not trip; the gap grows only when echoes stop. The same k in G absorbs a one-RTT loss-recovery, and only an RTO-scale gap trips and registers as a stall event, as on the data channel.

An independent RTT reference buys cleaner, faster attribution, not the core defence. A kernel tcpi_rtt or handshake baseline gives a single-observation tell of adversarial app-layer echo-holding (kernel RTT low, app-RTT high ⇒ the peer is sitting on cookies — literally the existing "app-RTT vs kernel-RTT divergence" cross-check), distinguishing it from an honestly far peer (both high). That matters only for escalating from deprioritise — which self-measured signals already do — to punish/demote. It has caveats: it needs Linux + the tracetcpinfo build flag (not portable), and the handshake baseline is drift-prone.

Attribution isn't contingent on it, though: the actual-vs-predicted comparison (below) recovers a form of it portably — a persistent actual ≫ predicted identifies targeted withholding using only app-layer signals (direct T_wait, estBurstS, RTT), no kernel/handshake reference. It convicts more slowly (it needs persistence to clear the serve_time/cold-cache confound) and it catches the attack by a different tell.

The adversary bind — and why aggregate estBurstS is a strength

What makes the gate sound is a genuine dilemma the adversary faces on a single measured stream:

  • To make delivery slow, the stream must be idle a lot (low byte-rate).
  • To game a gap-based estBurstS, that idle must fall outside bursts (the burst timer expires during silence, so the stall isn't counted against it, ie. does not increase estBurstS when it should have).
  • But idle = no SDUs = no echoes → the cookie queue ages / RTT samples inflate.
  • To keep echoes prompt, it must fill the idle with SDUs — but then the burst stays alive, the fill is folded into estBurstS, and the estimate reads the true (slow) rate.

So it cannot simultaneously game a gap-based throughput estimate and keep the liveness/echo signal flowing: whichever activity carries the prompt echoes is measured, and going silent to hide the stall shows up as echo lag. Pick one, caught by the other.

Cardano's mini-protocols are mostly pull-based, which tightens this further: the peer sends SDUs only in response to our requests, so it cannot inject filler at will to keep echoes flowing. The fill would have to be responses to some other request we happen to have outstanding to it — an opportunity it cannot manufacture — so busy-fill is opportunistic and hard to exploit reliably. Absent such traffic, the only way to withhold our closure is to fall silent, which the echo-lag / SDU-spacing gate catches directly.

Aggregate (protocol-mixed) estBurstS is what closes the loophole. The bind holds only if the echo-carrying SDUs are inside the measurement — and today estBurstS is connection-level, so every received SDU, whatever mini-protocol carries the echo, feeds the aggregate burst. There is no unmeasured side-channel to park prompt echoes on. The protocol-mixed aggregate we usually treat as a limitation is, for this attack, the strength.

Caveat for a future per-protocol split. The loophole reopens if per-protocol estBurstS (the proposed "Shape B" split — per-mini-protocol DeltaQ buckets, not implemented today) is added without also binding the cookie/RTT per-protocol — because then the throughput signal and the echo signal live at different granularities. Concretely: we fetch a 12 MB closure over LeiosFetch (protocol B) and are also driving a second mini-protocol A (keep-alives, say, or another fetch) on the same connection.

  • It delivers the closure in tight 50 kB micro-bursts, then goes silent on B for ~500 ms, repeat. Per-protocol estBurstS_B counts only the fast micro-bursts — the 500 ms silences exceed B's burst gap and fall outside its bursts — so it reads fast. Gamed.
  • During each B-silence the peer keeps answering our A requests promptly, each response echoing our latest cookie. The cookie PSQ is connection-level, so those A echoes keep the queue drained and RTT low — the gate stays quiet.

Result: estBurstS_B says LeiosFetch is fast and the gate says nothing is held back → we rank the peer a great closure-deliverer, route closures to it, and it dribbles them. A is a side-channel that estBurstS_B doesn't see, yet it still satisfies the connection-level gate — exactly the gap the bind depends on being closed. The loophole is gated on A traffic frequent enough to keep draining the connection-level queue through the B-silences, though — consistent with the pull-based limit above, the peer can't fabricate that; it can only ride A traffic we ourselves drive.

The aggregate estimate doesn't have this gap: those same A responses are inside the aggregate burst, so they either keep it alive (⇒ estBurstS reads the slow closure rate) or, if the peer goes fully silent to avoid that, the echoes lag (⇒ caught). The fix under a per-protocol split is to keep the two signals at the same granularity — judge B's delivery against echoes that ride B, so a silence on B costs echo-lag on B. Rule: go per-protocol on estBurstS ⇒ go per-protocol on the echo/RTT too.

Where the payoff hides — and how each strategy leaks. The adversary's payoff is always the same: pace the bursts so estBurstS reads fast, win the fetch, then dribble the payload. Stalling echoes or inflating RTT is never the goal — it is only a leak of the pacing, and a different signal catches each leak.

  • Busy-fill — fill the stalls with SDUs so the echoes stay prompt and nothing looks held back. But those filler SDUs keep the burst alive, so they fold into estBurstS, which then reads the honest, slow rate. estBurstS itself catches it.
  • Idle-gap — go silent between micro-bursts so the stalls fall outside the burst and the raw estBurstS reads fast. That backfires two ways: the short bursts starve coverage, so the confidence-weighted estBurstS shrinks back to the average; and where cookies keep flowing, the silence delays their echoes and inflates the RTT quantile. Either self-penalises in the ranking — and the RTT inflation, against an independent reference (kernel tcpi_rtt / handshake), is attributable as app-layer echo-holding.

(A queue-vs-self-RTT comparison that might otherwise flag the stall is circular — Using the RTT signals — so we rely on the bind, which catches the idle-gap gamer without it.) The third leak, byte-padding, escapes both of these and is treated next.

The one escape the bind does not close: byte-padding. The peer sends other bytes fast, serving another protocol request, so the aggregate byte-rate looks high while it dribbles the closure you actually asked for. estBurstS reads fast, the padding carries prompt echoes (the gate stays quiet), yet your closure is slow. A measurement bound to that specific fetch — direct completion timing (T_wait on the closure; next section) — catches it, because the padding isn't the bytes you're waiting on. And comparing that T_wait to the peer's own estimate upgrades detection to attribution: under padding the prediction is fast but the actual is slow, and actual ≫ predicted means the peer has the capacity (it's spending it elsewhere) and is withholding from us (the data-withholding threats T20/T21), not merely slow. Lose-lose for the padder: advertise fast and dribble ours ⇒ discrepancy; advertise slow to match ⇒ ranked down anyway; deliver ours fast ⇒ not the attack. Caveat: honest confounds fire the same discrepancy — server-side serve time for our closure, HOL/contention on the shared bearer, stale estBurstS/RTT — so deprioritise on a one-off and punish (demote) only on persistence with those ruled out.

The serve-time confound is the sharpest, because it produces the same observable, not just a noisier one. estBurstS/RTT capture the byte-flow rate and the network round-trip, but not the peer's cost to produce the bytes — reading our closure's txs from its store, reassembling it:

actual    = RTT + serve_time + size × per-fetch-rate + loss/RTO
predicted = RTT +              size × estBurstS       (aggregate signals)

The prediction has no serve_time term, so actual − predicted ≈ serve_time: a large serve time alone reads as actual ≫ predicted. So a peer whose TxCache holds others' closures (hot, served fast) but not ours (cold, disk-read) shows a fast aggregate and a slow closure — capacity that exists but isn't reaching our request, exactly like byte-padding, yet benign (which closures are hot/cold is diffusion history, not the peer choosing to starve us). Decomposing our fetch into time-to-first-byte (which absorbs serve time) vs the whole-fetch byte-rate separates the two if the peer batch-reads (long TTFB, then a normal rate ≠ dribbling) — but a slow streaming read shows a slow rate throughout and is indistinguishable from withholding by rate alone. So the discrepancy stays a deprioritise signal (correct either way — the peer is slow for us now); promoting it to malice needs the slowness to be systematic across our requests to that peer (targeting), not per-request random (cache-luck).

The queue is the structure — the extension gate

The cookie/promptness queue is the key feature: the gate that lets the burst window extend to measure an honest large closure, which a quiet peer forfeits (the window-sizing tension above). The consistency and actual-vs-predicted checks sit on top of it as the backstop, not as a substitute for it. It reuses machinery already present — the t-digest and the cookie PSQ; what is new is gating the extension on them.

Setting the threshold for the actual-vs-predicted check. The band — how far actual may exceed predicted before the byte-padding check acts — is an empirical high quantile (say p99) of the residual actual − predicted, drawn from a population baseline across honest peers rather than the suspect's own samples, which it could inflate to widen its band and hide under. The tracked estDeltaQVVar is unsuitable here: as per-SDU micro-jitter it is the wrong granularity — blind to the correlated per-fetch terms (ramp-round count, loss-recovery, serve_time), it under-estimates the real spread, so the band would come out far too tight — and the wrong shape, since the residual is heavy-tailed and a σ-band mis-fits it. This band is only the threshold, though: the check's second gate is a per-peer hold-back (below) — the estBurstS standard error SE, variance-derived but distinct from the rejected estDeltaQVVar, vetoing a conviction that falls within the prediction's own error bar.

Stratify that baseline by regime, because the residual's spread is not one distribution. It scales with message size (more bytes ⇒ more loss-recovery exposure and a longer transfer) and with RTT (both the ramp-round count and the RTO magnitude grow with it — and that dependence survives even though predicted already subtracts a ramp(RTT) term). Loss rate is the natural third axis, but it is non-portable — only via kernel tcpi_retrans/tcpi_lost (Linux + tracetcpinfo, design-phase) — so a first pass stratifies on size × RTT only and pools clean and lossy paths in the same bin. The cost is a deliberately conservative band: the lossy fetches in a bin inflate its p99, widening the threshold, so first-pass detection is less sensitive (a padder can hide under the loss-inflated band) but rarely false-positives. Add the loss stratum to tighten it once kernel telemetry is available on the measuring node.

The bigger picture: rank on measured completion, not estBurstS

The pacing attack only matters to the extent we use estBurstS as a proxy for delivery quality (peer ranking, chunk sizing). But those decisions are being offloaded onto direct completion timing (T_wait / observed arrival). Against direct completion timing, pacing is self-defeating: the adversary's paced-slow delivery is the measured completion time, so it ranks low no matter how it games the per-SDU gaps. So the highest-leverage defence against estBurstS-pacing is not to trust estBurstS for the decision that pacing targets — rank on observed completion, and pacing buys the adversary nothing.

Even short of direct timing, ranking on the estimate (Improvement #1's fused metric) already absorbs the idle-gap gamer, as shown above. This needs no independent cross-check — you never have to distinguish the gamer from an honestly slow peer; the estimate ranks both down regardless. The gamer is ranked down on a coverage-shrunk estBurstS — its short bursts starve coverage, so the fast rate it engineered decays to the average — and, where cookies keep flowing, on an RTT its silence inflates; an honest slow peer, on its genuine rate and RTT. So we deprioritise both without telling them apart, and the tcpi_rtt/handshake anchor is only for attributing and punishing the gamer specifically. Two caveats: for the RTT route to bite, the ranking quantile must be high enough to capture the stall inflation (occasional long stalls hide above p80); and byte-padding — a different attack — evades the estimate outright, with a genuinely fast estBurstS and low RTT (see the bind). So the ordering is: measured completion timing closes both; the fused estimate is a good fallback that handles idle-gap for free.

That leaves estBurstS needing hardening only for its residual uses — diagnostic decomposition, and slow-loris detection. Slow-loris is the mirror image of pacing: the peer makes no attempt to hide its slowness, it just accepts the fetch and trickles it, tying up our slot. Because it looks slow, the time-to-deliver ranking deprioritises it for free; and a peer whose estBurstS rises mid-transfer — throughput collapsing below a floor — is the slow-loris signature (Improvement #3). Answering it is harder than spotting it, though: there is no per-fetch cancellation today, so the current stuck fetch has only three imperfect moves — wait (risking the deadline), reset the connection (degrading our peer topology and, done reflexively, opening an eclipse surface — and a slow peer can beat the even-slower one we might draw to replace it), or hedge to another peer (Improvement #5), the most resource-intensive, since our socket buffers and ingress must then service both the slow-loris and the hedge. (When to hedge develops the triggers for it — a loss signal, and a progress comparison against the remaining work.) The deprioritisation above is the only lever needing no cancellation — but it is preventive (we stop choosing the peer), not curative (it can't rescue a fetch already in flight). So estBurstS earns its keep here in the opposite direction from pacing — flagging a peer that looks slow, not one faking fast — where pacing-to-look-fast can't help the adversary. The estBurstS-vs-RTT consistency check, meanwhile, is worth keeping as a sanity gate for when you do lean on estBurstS, but it shouldn't be load-bearing.

Why leaning on estBurstS is sound

estBurstS is low-level and gameable, so building the prediction on it looks risky — but it's safe because of the role it plays:

  • Center-line, not tolerance. It supplies the peer's advertised rate in the estimate. The tolerance — how far actual may exceed that — is a separate quantity: a high quantile (p99) of the residual actual − predicted measured across honest peers (the threshold discussion above). Call that honest-peer residual distribution the baseline. estBurstS's missing macro terms (serve_time, stalls, loss-recovery) don't corrupt the check: because predicted omits them, they surface in honest peers' residuals too, so they're already folded into the baseline — a suspect peer is flagged only by deviating above it.
  • Audited, not trusted. actual (direct T_wait) is the auditor; predicted (from estBurstS) is the claim under audit — its unreliability is fine because it's the thing being checked, not the ground truth.
  • Inaccuracy cancels; gaming self-defeats. Because that baseline is computed with the same estBurstS, any systematic bias in estBurstS (say it reads 10% fast for everyone) shifts every peer's predicted alike, shifts the baseline with it, and cancels in the residual-vs-baseline comparison — so the check needs estBurstS consistent across peers, not accurate. And the only useful lie is to make estBurstS look low, which shrinks predicted and makes the check more sensitive; lying it high advertises slow and ranks the peer out.

The one role estBurstS is not given — trusted ground truth for ranking — is the one it couldn't survive; that's why ranking ultimately prefers measured time-to-deliver. Correspondingly, the tolerance band is the empirical residual, not the per-SDU estDeltaQVVar (the threshold discussion above).

Relative signals, absolute anchors

Several mechanisms here and in the companion documents share a shape: a judgement made relative to a population — an estBurstS shrunk toward μ_p, a residual measured against the honest-peer baseline, a mesh dip scored against each peer's own norm. Relativity is not the hazard; it is often a strength, since a bias common to a signal and its reference cancels (the inaccuracy cancels argument above: a suspect is flagged only by deviating from a baseline built with the same estBurstS). The hazard is a reference frame with nothing outside it — a loop that audits the population only by the population, so a slow shared drift, or an attack that moves the whole frame, is never seen. Three parts keep these mechanisms clear of that trap:

  • Relative signals carry attention and confidence, never conviction — the population decides where to look and how much a reading is worth (shrinkage, precision-weighting), not guilt.
  • An absolute, non-adaptable measurement pins the frame. Conviction rests on a quantity not derived from the population and unable to drift with it: T_wait against the real deadline, an absolute event (a missed deadline, an RTO-scale stall), or a fixed external envelope. It is the same reason a threshold must never be fed by the signal it gates (Using the RTT signals: G co-moving with cookie front-age would let a uniform-delay attack inflate both and never trip) — the anchor is chosen precisely because it cannot co-move.
  • Act only safe-directionally, so the loop is negative feedback. Every belief-driven action adds caution — deprioritise, hedge, back off, churn, down-weight — and each counteracts the condition it reacts to, so a false alarm erases its own evidence instead of compounding it: a mistaken "stressed" reading backs production off, load lightens, delivery recovers, and fewer peers read as degraded, so the belief subsides; a mistaken "concentrated" reading churns in fresh peers and dilutes the correlation that raised it. A positive-feedback loop would let the action feed its own evidence and run away; this one regulates toward a fixed external target instead — the CIP-0164 envelope, or the honest-majority pack size the diversity belief needs. That fixed target is the absolute setpoint: the loop hunts it the way TCP's AIMD hunts a link's true capacity — back off on the signal, ease back up once it clears — settling around reality rather than a figure of its own making.

Ranking is the first instance: the estimate is a population-relative prediction audited by absolute T_wait (audited, not trusted), and the echo gate is anchored by a gap measured independently of G. The same invariant governs the two derived beliefs that pool this machinery — the mesh-stress filter, whose relative dip-count is backed by the fixed CIP-0164 envelope that cannot adapt away (leios-testnet.md), and the effective-diversity belief, whose correlation is taken only after the common mode is removed and over absolute degradation events (leios-fetch-scheme.md). Both are relative for speed and absolute for the frame, and neither acts except to add caution.

What this does not buy. The guarantee is only as strong as the anchors staying trustworthy and outside adversary reach. That holds today — T_wait is our own measurement, the envelope a spec constant, a missed deadline absolute — but it is the load-bearing assumption; an anchor that became gameable would void the rest. Three costs come with it. A fixed anchor and its relative counterpart fail in opposite directions, and the anchor pays for not adapting: a relative channel drifts with a legitimate slide until it falls silent (the boiling frog), so the fixed one is set not to — which means a healthy-but-slower network, legitimately past the design envelope yet otherwise fine, trips it instead (crying wolf). Safe-directional, but a real tuning cost, and the reason mesh-stress runs both channels rather than either alone. A relative estimate over a small pool (the five-peer correlation) is noisy and will sometimes see a cluster or dip that isn't there; persistence and shrinkage damp it and safe-directionality caps the fallout at throughput rather than safety, but the false alarms are real. And the discipline holds by construction, not by anything that enforces it — the first time a purely relative belief drives a consequential action with positive feedback and no absolute cross-check, this protection is gone, so extending the machinery means re-checking all three parts rather than assuming they carry over.

The safe-directional boundary. The third part has a boundary worth naming, because a live attack sits exactly on it. "Safe-directional ⇒ negative feedback" holds only while the load an action sheds is load the same loop controls. If a continuous adversary outside the loop reoccupies the capacity a safe-directional backoff frees — the expensive tier of the sustained-overload livelock in leios-testnet.md and threat-model-additions.md's T42, where a saturator floods the voter quorum regardless of anyone's body size — then backing off is a capacity donation, and the loop drives honest load toward zero while the adversary sustains the overload. Safe-directionality is self-defeating on that continuous channel. (A merely stake-gated attacker, one who over-produces only on the slots it wins, does not trigger this: it cannot reoccupy honest rounds, so honest backoff there heals — the boundary is specifically about a continuous, out-of-loop load.) The repair keeps the principle and guards the action: back off only while the shed load actually reduces the total, and where it does not — a continuous adversary is refilling — switch from self-shedding to attribution and denial, deprioritising the party that will not share the loop's restraint. Conviction still rests on the absolute anchor; what changes is the recognition that the safe direction is only safe when those bearing the restraint are the ones generating the load.

Smaller points

  • Your own GC is a confound for the gate. A GC pause makes us timestamp echoes (and incoming SDUs) late, stretching the echo gap and the SDU-spacing reading — looking like the peer stalled, a false extension-revocation that blames the peer for our own pause. Same class of false trip as a loss/RTO (the window-sizing section). Two things absorb it: the backstop's persistence requirement over time, and — more immediately — the cross-peer common-mode reject (appendix), since a self-pause stalls every peer at once, so a synchronised stall is charged to us, not any peer, and suppressed (no gate trip, no stall count). Both are portable; kernel tcpi_rtt, where available, is immune to our GC anyway — a reason to prefer it for the RTT input.
  • Lazy sampling. The RTT quantile feeding ranking/prediction is read from the t-digest every ~1 s in processIngress, not per-SDU — preserving the finger-tree's deferred <>. (The extension gate itself is a cheap echo-gap / SDU-spacing check, separate from the t-digest.)
  • Not tuning-free — and the promptness threshold is RTT-relative. The knobs are the promptness threshold G (max back-to-back SDU spacing / echo gap that keeps the burst extending), the backstop quantile (p99), and the churn-persistence count. G must scale with RTT, not be a fixed millisecond value: G = max(floor, k · RTT_hi), where RTT_hi is the high RTT quantile we already track (say p80–85) and the multiple k (distinct from the appendix's coverage-scale k) is small. The quantile carries the jitter, so k covers only the fast-retransmit recovery (~1 round-trip, plus a hair for loss coinciding with an elevated RTT) — not variability twice over: k ≈ 1.5 is reasonable, whereas k ≈ 2 would double-count and risk pushing G up into RTO-scale, where a severe stall should instead trip and register as a stall event. Floor a few tens of ms (scheduling/GC jitter). A fixed absolute value strands far peers — their ~1-RTT loss-recovery gaps exceed it, so honest transfers fragment into low-coverage bursts that never earn a trusted estBurstS; scaling to RTT folds a fast-retransmit into the burst as a slower-but-trusted rate, while a severe RTO-scale stall still exceeds G, ends the burst, and registers as a stall event. Its risk is false trips (a genuine loss/RTO or GC), not gaming — a steady-dribbler just under it keeps extending and is measured slow, which is correct — and those false trips are what the backstop's persistence tolerates. The knobs are physically grounded (an RTT multiple, a percentile, a count) but still need calibration.
  • Download liveness needs no cookies. In a one-directional download the incoming-SDU-spacing gate supplies liveness directly; the cookie signal (for RTT) can go stale there — cookie-carrying outbound is sparse — so inject a cookie periodically (a keep-alive echo on a dedicated mini-protocol) to keep a fresh RTT and to keep the sent-cookie queue aging as a live gauge through a long download. This is not supported today and would take some implementation effort — work this analysis motivates.
  • Response on detection is cheap and low-blast-radius: the ranking already deprioritises a gamer; on a persistent actual-vs-predicted discrepancy, churn/demote it. It's a peer trying to look good, so a false demotion costs little.

Bottom line

The burst window's job is accurate measurement, not anti-pacing: gating the extension on peer promptness (cookie-echo gap, or incoming SDU spacing on a download) keeps an honest large transfer's window open so it accrues coverage, while a genuinely quiet peer closes it. On the echo side that coverage is paid for in RTT freshness — a burst keeps accruing only while the peer keeps echoing, so it cannot bank coverage while starving our RTT probe. The threshold G is RTT-relative (max(floor, k · RTT_hi)), so a fast-retransmit folds into estBurstS and only a severe stall ends the burst — far peers aren't stranded by an absolute cap. Anti-gaming does not rest on the gate's tightness — a pacer is caught by coverage, the bind, and the actual-vs-predicted backstop whatever the threshold: churn a peer whose delivery is consistently worse than its advertised rate, which also absorbs the gate's false trips (loss/RTO, GC) via the consistency requirement. An independent RTT reference (kernel tcpi_rtt / handshake) is a nice-to-have for fast attribution, not a prerequisite. And all of this hardens the prediction: the ranking itself prefers directly-measured completion wherever a fetch has finished, where a pacer gains nothing. More broadly, it is one instance of the discipline the whole design follows — relative signals, absolute anchors: population-referenced quantities (the estimate, coverage, the baseline) allocate attention and weight confidence, while an absolute measurement that cannot drift with them (T_wait, a missed deadline) carries conviction, and actions only ever add caution. The mesh-stress and effective-diversity beliefs built on this machinery obey the same rule.


Small downloads and TCP slow start

For a small download the estimate is ramp(RTT)-dominated: estBurstS × size is negligible, so the ranking should be RTT-led and the noisy serialisation term given little regard — the same "lean on the signal we measure" logic the size-manipulation section applies to round-trip- dominated transfers. The complication is that ramp is not a fixed function of RTT; it depends on the connection's congestion state. Once a connection sits idle beyond about one RTO (Linux's tcp_slow_start_after_idle), cwnd is reset to the restart window, and the first ~log₂(size/IW) round-trips are window-limited rather than path-limited. The same peer therefore has a one-RTT ramp warm and a several-RTT ramp cold — which a fixed ramp(RTT) cannot express, and which for a small fetch dominates the whole estimate.

Two things follow. First, model the state rather than ignore it: we already keep the peer's last-use timestamp (the one driving the staleness decay), so idle beyond an RTO flags a likely slow-start restart, and the ramp is inflated by the slow-start rounds for that size — or, where tcp_info's tcpi_snd_cwnd is available (the non-portable path), cwnd is read directly. This is partly a real effect, not just noise — a warm connection genuinely delivers a small fetch faster — so modelling it makes the ranking more correct, not merely less punitive; keep it a per-fetch ramp adjustment, though, not a rewrite of the peer's long-run μ_p. (Truly tiny transfers, size < IW, fit in the first window and pay no slow-start penalty at all; the bite is the medium-small range that spans several windows yet is still ramp-dominated.)

Second, do not convict on small fetches. The actual-vs-predicted backstop has little discriminating power when the residual is dominated by connection state we cannot fully observe: a small-fetch actual ≫ predicted is far more likely slow-start-after-idle than withholding. Reserve conviction for larger fetches, where the serialisation term — the one that reflects sustained withholding — dominates and estBurstS is meaningful. The size×RTT stratification already widens the band in the small-size bins (their honest residuals swing with connection warmth, inflating the per-bin p99); the rule is simply "rank small fetches, but don't punish on them."

The confound can also be removed at the source: keeping connections warm (periodic keep-alives, or disabling tcp_slow_start_after_idle on these sockets) makes the ramp predictable. A node's hot upstreams may see enough traffic to stay warm anyway; it is the rarely-used peers — exactly the ones the staleness decay is already cooling — where both effects bite together, a peer cold in the estimate likely sitting on a cold connection.

When to hedge

Two complementary triggers — both portable — say when to hedge or switch away from a peer mid-fetch: a loss signal that the current peer is failing, and a progress comparison that a candidate would finish the remaining work sooner. They interact — a severe stall is an alarm in its own right and also feeds the comparison, flipping the stalled peer onto cold footing (below). Hedging itself is Improvement #5's portable, Linux-free fallback (leios-fetch-scheme.md).

The loss signal

The loss signal warns that the current peer is degrading, from two app-layer readings that need no kernel telemetry: the estBurstS variance for minor loss, and a stall counter for the severe tail.

The first is the estBurstS variance. It rises under retransmission, so it proxies minor loss — the one legitimate use of a signal the confidence appendix otherwise rejects (there it is perverse: a metronomic pacer fakes a low variance to look high-confidence). It is safe here precisely because hedging acts only on high variance, never on the low reading a pacer can manufacture; and doubly so because a pacer able to fake that low reading is unlikely to be selected anyway — its short bursts starve coverage, shrinking its estBurstS to the average, so only a weak or eclipsed pack lets it through.

That loss-proxy use also argues for tracking something the burst window currently drops: the stalls that expire it. G sets the division of labour. A sub-threshold loss — a fast-retransmit, ~1 RTT, within G (see Not tuning-free) — folds into estBurstS as a slower-but-trusted rate and shows in its variance; a severe event — a deep RTO backoff exceeding G — ends the burst and escapes both, exactly where a loss-driven hedge matters most. So estBurstS absorbs the routine recovery, and a separate signal must catch the severe tail.

Track that as a separate stall signal: the rate and duration of gaps long enough to expire the burst window — separate because the scales don't mix (millisecond per-SDU jitter would swamp 100 ms–second stalls if folded into the variance). A candidate gap must expire an open window — one that opens on the first response SDU and closes when the transfer completes — so the wait before a burst never counts. That is what keeps a blocking request honest: the arbitrary idle before its pushed first response lies outside any window, and the echo-channel gate is self-resetting, so that pre-response gap never enters the burst's accounting.

One exclusion is essential: don't count slow-start gaps. A connection ramping from a cold congestion window — a fresh peer, or one idle long enough for tcp_slow_start_after_idle to reset cwnd — delivers in cwnd-limited windows with ~1-RTT gaps between them: normal operation, not loss. Counting them would make us hedge on essentially every cold transfer, worst on the rarely-used peers the staleness decay is already cooling. Gate the signal on connection warmth (the same slow-start-state inference the section above uses — idle time, or cwnd where tcp_info is available); the tell is that slow-start gaps coincide with rising throughput as cwnd grows, the opposite of a loss stall's collapse.

The signal is cause-agnostic: link congestion, peer-side GC, and adversarial pacing all mean the same for the decision — "this peer is stalling: hedge away, rank down" — and disentangling them is the backstop/bind/independent-RTT's job, not the counter's. Use it under the same asymmetric rule as the variance: high → hedge more / rank down, but never let stall absence raise confidence or drop a hedge (a metronomic pacer stalls-free on purpose).

Two side-benefits: it makes an idle-gap pacer's between-burst silence visible — today that silence escapes both estBurstS and the in-burst variance — reinforcing the gate and the bind; and it completes the portable loss proxy (variance for minor loss, stalls for severe), keeping the fetch scheme's hedging fallback Linux-free. The cost is an O(1) counter on the same last_update timestamp; its one confound is our own GC — it delays our timestamp of the resumption and mis-blames the peer — absorbed, as for the gate, by the persistence requirement.

The stall signal aggregated across peers is also a common-mode reject for that self-pause confound — and better than persistence, because it acts in the moment. A self-pause (our GC, scheduler, or uplink) stalls every peer's stream at once, so a stall synchronised across a high fraction of active peers is common-mode — our side, or a network-wide event — not any peer's fault. Suppress attribution across that window (no gate trip, no stall count, no actual-vs-predicted sample for anyone) and attribute only isolated stalls, which are peer-specific. It is a portable self-pause detector (no kernel tcpi_rtt needed), sharpens with peer count (fall back to persistence when peers are few), and is adversarially sound: an adversary can synchronise only its own peers, so faking a common-mode window takes near-total control (eclipse) — and even then blunts only the stall signal, not the after-the-fact actual ≫ predicted.

The progress comparison

Even with no stall, a peer can simply be slower than the alternatives for what remains. The decision is then a direct comparison of projected finish times for the remaining bytes, driven by observed progress (the principle from Handling advertised-size manipulation, never a newcomer's claim). Mid-fetch on peer A, with downloaded of an advertised size, the work left is size_left = size − downloaded.

  • A is warm, so it pays no ramp — only serialisation of what is left, at the live rate we are measuring on this very fetch (better than its historical estBurstS): t_A ≈ estBurstS_A_live × size_left.
  • A candidate B is cold — a fresh request starts from a cold cwnd — so t_B ≈ ramp(RTT_B) + estBurstS_B × size_left, the ramp carrying the slow-start rounds for size_left (Small downloads and TCP slow start). We assume cold because B's cwnd is not portably observable (Linux-only, and unreliable), and the assumption is conservative — it inflates t_B, biasing against a switch.

Switch the remaining chunks to B only when t_B beats t_A by a margin that covers the switch's own cost. The size-left dependence falls straight out and unifies with the small-download logic: when size_left is small, t_B ≈ ramp(RTT_B) — cold-start/RTT-dominated — so B rarely beats a warm A on the tail and we hold; when size_left is still significant, the full estimate decides.

The two triggers meet here — but a severe stall does not zero A forever. It collapses A's cwnd, so A must re-pay the ramp: its side of the comparison flips from the warm estBurstS_A_live × size_left to the full cold estimate ramp(RTT_A) + estBurstS_A × size_left, with estBurstS_A now taken from A's prior — the shrunk value into which recent degradation has folded (a slow-loris reads slow there), not the pre-stall live rate that may flatter a peer which has since failed. The comparison then runs as usual: a cold, degraded A usually loses to B, though not automatically. So the loss signal is the fast alarm that flips A onto cold footing; the comparison settles whether to abandon it.

One mechanical dependency: resuming only size_left needs a chunk boundary to resume from — chunked LeiosFetch has one, whereas an atomic datum degrades to comparing B's whole-datum estimate (or a true parallel hedge, B re-fetching everything).

Counter-offers

Often B is not a peer we picked but one that offers the same datum mid-fetch at a different — usually smaller — size (Handling advertised-size manipulation). Its size then feeds t_B, so it must be trusted with care. Where a signed header carries the size (EBs, Praos blocks), trust it outright — having verified the header we have verified everything it commits to, whoever pushed it — and B's identity does not enter. Only unattested aggregates (closures) leave the size as B's word; there, drive the switch on A's observed progress, never B's claim alone (which would reward the size-lie in dynamic form), and weight B's offer by its rank: a long-credible peer earns a bounded bonus — a smaller switch margin, or more benefit of the doubt on its claimed size — while an unproven one must clear the bar on progress alone. The bonus stays bounded because leaning on a high rank is exactly what reputation-farming (behave, then defect) sets out to exploit; and the exposure is small regardless — the worst case is one recoverable mis-selection, caught by actual-vs-predicted and remedied by a further hedge or reset, never a failed fetch.

Appendix: attributing confidence to estBurstS

An estBurstS built from a long, sustained transfer — one where the burst window kept extending across many SDUs — deserves more weight than one scraped from a short or paced burst. To lean on the good estimates and discount the thin ones, estBurstS needs a confidence attached to it. Of two candidate measures — the variance of the estBurstS regression, and the number of samples it was built from — the count is the better basis, for a reason that is structural rather than statistical.

Variance and count are not rivals. If estBurstS is a fitted slope (or a mean rate over near-constant SDU sizes), the statistically correct confidence is not the raw residual variance — that is estDeltaQVVar — but the standard error of the estimate, SE ≈ residual_sd / √N, which folds the residual variance and the sample count into a single number — their principled combination.

But variance — and SE with it — is the wrong signal for telling an honest estimate from a paced one, and is in fact perverse. A metronomic pacer emitting perfectly regular micro-bursts produces near-zero residual variance, hence a tiny SE, and would therefore score as high confidence. Variance measures precision and consistency, not honesty or representativeness; a precise measurement of a gamed rate is still gamed. Treating low variance as a positive trust signal rewards exactly the regular emission a pacer can manufacture. (This is the same reason estDeltaQVVar is rejected for the tolerance band above — it measures the wrong thing.) Variance is not useless everywhere, though — it has one legitimate, safe-direction use, as a loss proxy for hedging (When to hedge).

Byte-coverage is the trustworthy quality signal, for a structural reason. The confidence weight is the payload delivered under the gate — Σsize, plain accumulated bytes — rather than a raw SDU count: a flood of tiny SDUs runs a counter up cheaply, whereas accumulated bytes cannot be inflated without actually delivering them. Its trustworthiness is structural, not statistical: coverage is downstream of the extension gate. Under the gate, coverage can only accumulate by serving promptly for the whole burst, and serving promptly throughout is precisely what forces estBurstS to the true sustained rate (the bind). A pacer that stalls forfeits the extension, so its burst stays short and its coverage low. High byte-coverage therefore certifies a gate-vetted, sustained, prompt-served transfer — and hence an honest estBurstS, whether that honest rate is fast or slow. The coverage is less a confidence-in-a-number than a receipt that the gate let the estimate run long, which it does only for honest service; it costs one accumulator.

The two measures then do different jobs. Byte-coverage is the trust weight: a thin, low-coverage estimate is shrunk toward a prior — the peer's long-run estBurstS, or a population rate — in the manner of empirical-Bayes shrinkage, while a high-coverage estimate is trusted as measured. This is how a long extended burst earns its keep — it accrues weight — and it handles the cold-start case gracefully. SE, meanwhile, is the error bar on the estimate itself — the uncertainty in estBurstS, and so in the predicted time-to-deliver — kept distinct from the tolerance band, which stays the empirical residual across honest peers (the threshold discussion above). Its role is to hold the actual-vs-predicted check back when a discrepancy lies within the prediction's own error bars — a thin estimate shouldn't convict — not to set the detection threshold.

From trust weight to peer ranking. Leaning on estBurstS means letting it move a peer's rank in proportion to the coverage behind it, and no more. For a candidate peer p and a datum of (robust-aggregate) size S: shrink the peer's raw estBurstS toward a prior μ_p,

estBurstS_p^ = w·estBurstS_raw + (1−w)·μ_p
with w = cov / (cov + k)

k a coverage scale — see Setting the coverage scale k; form the estimate

ramp(RTT_p) + estBurstS_p^ · S

rank ascending and take the lowest. Coverage enters only through wSE stays the audit band, not a ranking term. Pin μ_p as a separate, slower track — a long-horizon, coverage-gated running estimate of the peer's rate (a population rate with no history), not the previous decision's shrunk output fed back (see Maintaining the prior μ_p below).

Two properties make this the right lean. It degrades to the trusted signal: a thin estBurstS shrinks to the prior, so among under-measured peers the throughput terms coincide and the ranking separates them on their measured RTT — estBurstS casts a differentiating vote only once its coverage earns one. And it closes the thin-sample gambit: thin means low w and no advantage, while the only route to a trusted-and-fast estBurstS is sustained service, which the bind measures honestly.

Resist bandit-style optimism. The tempting move — a UCB-type (upper-confidence-bound) exploration bonus for high-uncertainty peers — is backwards here, since an adversary can arrange to be under-measured, so the bonus rewards the attack. Shrinkage does the opposite: unproven is treated as average (the prior), not optimistically fast. Exploration still happens — average peers are chosen sometimes and then measured by T_wait — it is simply not handed out as a bonus.

Confidence-weighting is not the conviction step; it and direct T_wait are two halves of one box, guarding different regimes. Where no fetch has completed — choosing a peer, chunk-sizing, diagnostics — coverage-shrinkage rules, so a cheap fast reading cannot buy a top rank (a thin paced estimate shrinks to the prior). Where a fetch has completed, T_wait rules, and a peer is ranked on how slowly it actually delivered, pacing and all. So a pacer is capped at average while unproven and ranked slow once measured — never a trusted top rank at either stage, and warming up its estBurstS over a long paced session does not help, since that same session stamps a slow T_wait on every fetch and T_wait outranks the estimate. The actual-vs-predicted backstop sits on top of both as the attribution that escalates a persistent gap from deprioritise to demote/churn. The subsections below make this precise, and the last shows it holds against a rational pacer whatever the window threshold.

Setting the coverage scale k

k fixes the crossover between prior and sample: at cov = k, w = ½, so estBurstS_p^ is an even blend of the fresh sample and the prior — below k the prior dominates, above it the sample does. In coverage units (bytes), k is therefore "how much a peer must deliver before its own measurement outweighs the baseline."

The low-coverage extreme is the cold-start case, handled with no special rule. A brand-new peer has cov = 0, so w = 0 and the estimate is purely the prior — and with no history of its own, that prior is the population rate. A fresh peer is thus estimated at the typical rate: average until proven, neither optimistically fast (which would let it win traffic it hasn't earned, and reward anyone spinning up fresh identities) nor pessimistically slow (which would starve a genuine newcomer). As it delivers, cov grows, w rises, and the estimate slides continuously from that default toward the peer's own measured rate — the same mechanism by which a long burst earns its keep, now warming a cold peer up — while μ_p personalises from the population rate toward the peer's own long-run rate.

Four considerations fix it:

  • The principled value. Empirical-Bayes shrinkage gives w = τ² / (τ² + σ²/cov), which is cov / (cov + k) with k = σ²/τ²σ² the within-peer measurement noise (per unit coverage), τ² the between-peer spread of true rates, both estimable from data. When peers' rates differ widely (τ² large), a little coverage already places a peer against the pack, so k is small; when peers are alike (τ² small), the population prior is nearly as good as any sample, so k is large. At cov = k the measurement variance σ²/cov has fallen to the prior variance τ², so sample and prior are equally certain — which is why that coverage is the half-trust point.
  • A pragmatic anchor. Absent estimates of σ² and τ², anchor k to the workload: a fraction of a typical transfer's bytes, so one honest transfer reaches roughly half-trust and a few push w toward 1 — the behaviour a long extended burst should earn.
  • An adversarial floor. A pacer emits thin samples, so k must sit above the coverage one cheaply produces in a burst; anchoring it near a real transfer does this for free, since reaching trust then requires delivering roughly a transfer's worth, and delivering that sustained is what the bind measures honestly. The ceiling is that k must not be so large that honest peers never earn trust. Between the two, lean high — in an adversarial setting, better slow to trust than easy to game.
  • Calibration. With T_wait as ground truth, backtest k on honest historical fetches to minimise prediction error (estimate vs actual T_wait), subject to the floor — turning "about a transfer's worth" into a measured number.

Maintaining the prior μ_p

μ_p is a coverage-weighted average of the peer's past estBurstS observations, maintainable two equivalent ways.

The online form folds each new burst in one step, in the familiar "new = old + step × error" shape:

μ_p ← μ_p + η·cov·(raw − μ_p)

raw − μ_p is how far this burst's fresh estimate sits from the baseline; η is a small base rate setting the horizon (small = slow, long memory); and cov is the burst's byte-coverage, scaling the nudge so a fat honest burst moves the anchor while a thin sample barely does. The step is η·cov; writing α = η·cov, this is exactly μ_p ← (1−α)·μ_p + α·raw, a coverage-weighted EWMA. Keep α ≤ 1 for stability — clamp it, or feed cov normalised to a typical transfer so a huge burst can't overshoot past raw.

The batch form is the same coverage-weighting kept as running sums:

μ_p = Σ cov_i · raw_i  /  Σ cov_i

each observation weighted by its coverage, so a 10 MB burst counts 100× a 100 kB one.

Both forms are constant-space — one scalar for the EWMA, two accumulators for the batch mean — and retain no sample window, unlike the finger-tree / t-digest the RTT quantile needs: a mean updates in O(1), an order statistic cannot without the windowed distribution. Constant space is not constant value, though: the EWMA's lone scalar stays within the range of raw and cannot overflow, whereas the batch's running sums grow with lifetime traffic — a reason to prefer the EWMA, or to bound the sums.

The two differ only in memory. The batch sum has infinite memory — an old burst counts as much per byte as a recent one, an unbiased lifetime average that will not follow a peer whose true rate drifts. The EWMA is the same coverage-weighting with exponential forgetting (what η buys), so it tracks drift; the batch sum is just its no-forgetting (η → 0) limit. Use the batch form for a stable lifetime baseline, the EWMA (or a windowed weighted mean) to follow genuine rate changes. Either way, coverage is what keeps a thin sample from moving the anchor — the same guard the per-decision shrinkage applies, now on the prior itself.

Both forms so far forget over observations — a new burst decays older ones — which does nothing for a peer that has gone unused: with no new bursts arriving, its estimate sits frozen and grows stale as network conditions drift. Staleness needs decay in wall-clock time, not observation count. Two decays, keyed on the elapsed Δt since the peer was last measured, walk a stale peer back toward cold:

  • Decay the trust weight. Scale coverage by elapsed time, cov_eff = cov · exp(−Δt / T), so as Δt grows w → 0 and the estimate slides continuously back to the prior μ_p; T tracks how fast conditions change. (A hard "reset to cold after X" is the step-function special case; the exponential avoids the cliff where a peer used just under X ago is fully trusted and just over it fully cold.) Applied on every update — decay cov before adding the new burst's coverage — this same rule keeps cov a leaky accumulator, not a lifetime sum: it settles at a bounded steady state (≈ delivery-rate × T) rather than growing without limit, so it cannot overflow, and it reflects recent coverage. Since w saturates for cov ≫ k, cov can be clamped to a few k in any case.
  • Decay the personalisation. μ_p is itself stale for a long-unused peer, so fade it toward the population rate, μ_p_eff = d·μ_p + (1−d)·population with d = exp(−Δt / T_person). The population rate is the right target because the peers you are using keep it current, whereas μ_p froze when you last touched this one. T_person is longer than T: a peer's characteristic rate (its provisioning, its distance) is steadier than a momentary reading.

With both, a long-unused peer lands back on the population estimate — it starts cold and, unused, decays back to cold. This stays O(1) and lazy: keep one last_update timestamp per peer and apply the decay when the peer is next read or updated, no timer or sweep. And it is adversarially clean — reverting to the population average is the neutral "average until proven" default, so a peer gains nothing by going quiet and must re-earn a fast estimate by delivering (which the bind measures honestly); equally, a peer is not condemned forever on stale bad data once conditions may have moved. This closes the persistent pacer's last refuge: having warmed its estBurstS over a long paced session, it cannot bank that warmth by falling silent — the decay walks it back to average, and re-warming means delivering again, which re-exposes it to T_wait.

A tuning judgment, and a possible extension. T_person need not carry the personalisation all the way to the population rate. A peer's rate has a durable part (its provisioning, its distance) and a transient part (momentary congestion); full decay forgets both, resetting a well-provisioned nearby peer to the average over an idle hour and forcing it to re-learn what never changed. Retain a fraction instead — decay μ_p toward a blend, not fully to the population:

μ_p_eff = d·μ_p + (1−d)·[ρ·μ_p + (1−ρ)·population],   d = exp(−Δt / T_person)

so it rests at ρ·μ_p + (1−ρ)·population at long idle. ρ ∈ [0,1] is how far to decay: ρ = 0 is the full reset above, ρ = 1 disables it.

A fixed ρ > 0 carries an adversarial cost, though: it reopens the pacer's refuge in proportion. A warmed μ_p now rests at ρ·warm + (1−ρ)·population, so a pacer banks fraction ρ of its gamed warmth across an idle gap — once its own T_wait record has gone stale — and re-enters pre-fetch ranking above average; bounded, since T_wait re-catches it on the next fetch, but real. The fix is to make ρ earned per peer rather than a global constant: nudge it up a small step on every fetch that completes on time (additive, to a cap ρ_max < 1) and cut it hard on any miss — a late or failed delivery, regardless of cause, with no attempt to tell an accidental slip from a betrayal (multiplicative, e.g. ρ → 0). This AIMD shape — slow to earn, fast to revoke — makes retention a reliability privilege: only a peer with a sustained on-time record keeps its warm prior across a gap, while a pacer or defector never accumulates ρ (a single miss zeroes it), so its warmth still decays fully to average and the refuge stays shut for exactly the peers it should. It inherits the usual reputation-farming shape — build ρ, then spend it on one defection — but ρ adds only a bounded across-gap nudge to the record farming already buys, and the hard cut denies a second. Calibrate the step, cap, and cut like k, leaning conservative: a wrongly-slashed ρ costs only a fall back to the safe full-decay default, to be re-earned.

Idle-gap and busy-fill, re-derived from shrinkage

The two ways to pace — idle-gap and busy-fill from the bind section — are the two horns of a single dilemma, and reading them through the shrinkage weight (now that k, the prior, and what "shrunk to the prior" means are pinned down above) shows why the window's promptness threshold G — the max SDU spacing that keeps a burst extending — can be set purely for honest measurement and fairness (RTT-relative, generous enough to absorb a fast-retransmit; the window-sizing section), because it does not govern whether a pacer is caught. A pacer wants a fast estBurstS (to win selection) while delivering slow (to withhold); delivering slow needs idle time, and the only lever G gives is where that idle falls relative to the burst:

  • Busy-fill — idle ≤ G. The gaps stay inside the burst and are counted, so estBurstS = Σgap/Σsize includes them and reads the true, slow rate. Coverage is high, but the peer ranks slow — caught by estBurstS.
  • Idle-gap — idle > G. Each idle terminates the burst, so it is not counted and estBurstS, measured over the fast micro-bursts, reads fast; but the burst is short, its byte-coverage is low, the estimate shrinks to the prior μ_p, and the peer ranks at ~average, not top. Caught by coverage — and, per the bind section, doubly, since the same silence starves the echoes.

The rational pacer takes the second branch (average beats slow) and lands at ~average — the prior of Setting the coverage scale kfor any G. Tightening G only forces smaller idles, loosening it forces bigger ones; the destination, a fast-but-thin estimate shrunk to the prior, is unchanged. So G decides not whether the pacer is caught but only which horn catches it, estBurstS-reads-slow or coverage-shrinks-to-average — the adversary is factored out of the threshold choice entirely. This is the rigorous footing under the bind section's two attacks: they are not separately-patched holes but the two branches of one dilemma that shrinkage closes at once.

One qualification keeps this honest. "Shrinks to the prior" is the thin-history reading: coverage accumulates (Setting the coverage scale k), so a persistent idle-gap pacer — one that has delivered enough fast micro-bursts to build up cov — warms its estimate back toward that fast within-burst rate and no longer sits at the average. This is not a second byte-padding, and not a hole the bind misses. Going silent is exactly what the bind catches: the idle-gaps starve the echoes, and the delivery is actually slow (the idle time is real), so measured T_wait and the actual-vs-predicted backstop rank the peer down regardless of what its estBurstS has warmed to. The staging is the point — coverage- shrinkage denies a thin pacer the top rank up front, and the bind and measured completion catch one that has delivered enough volume to escape the shrinkage. Byte-padding stays the one escape from the bind proper precisely because only it keeps echoes flowing, on the padding, while withholding the closure we asked for — a persistent idle-gap pacer that filled its silences to do the same would simply be byte-padding.

Two consequences close the loop. First, because G buys no anti-pacing, a tight G is pure downside: an honest far peer's ~1-RTT loss-recovery gaps exceed it, fragmenting its transfer into exactly the short, low-coverage, shrunk-to-average bursts a pacer produces — a tight gate makes an honest far peer indistinguishable from a pacer. Sizing G RTT-relative removes that collision while conceding nothing to the adversary. Second, ~average is not zero risk — a pacer shrunk to the prior can still win in a weak or eclipsed pack — but that residual is the eclipse condition that weakens every defence, is itself G-independent, and is closed after the fact by the backstop convicting a selected pacer on actual ≫ predicted.

Clone this wiki locally