-
Notifications
You must be signed in to change notification settings - Fork 105
improved peer ranking
This document is an attempt at a detailed investigation of the problem of how to rank our peers for the purpose of issuing various requests in the hope that they are fulfilled in the most timely fashion to meet various protocol deadlines. A substantial development is laid out, based on reasonable heuristics, efficiency and trust concerns, and principled analytical devices are leveraged in the pursuit of that goal. Discussions of various hazards, tradeoffs and caveats, which are inherent in a trustless and permission-less environment of the Cardano network, are provided and which also inform and supplement this analysis, and leave some issues deserving future enhancements identified. A novel addition admitted by the proposed scheme relates to hedging requests, and is outlined in the section When to hedge.
The departure point is given a node with some connected peers and a request which
needs fulfillment, and several peers which could be burdened with the task,
which one(s) should we select. What is needed to answer this question is, especially
in the light of data-demanding protocols such as Leios, a sophisticated ranking
system which exhibits a predictive power and which is adversarially robust.
The metric we are most immediately concerned with is to rank those peers
by how quickly we expect them to actually
deliver what we want to fetch — EB bodies, closures — so fetch logic can route to
the best and demote the worst (feeding the improvements proposed in
leios-fetch-scheme.md — Improvement #1's peer classification,
adaptive chunk sizing, and the like). The posterior ground truth is
directly-measured completion time — T_wait, the observed
request-to-full-reply timing — and where a fetch has completed we rank on
it: a paced-slow peer measures slow — its delivery is exactly what
T_wait records — so the signal can't be gamed. Its one limitation is
that it is retrospective: you cannot measure a fetch you have not made,
so to choose a peer before the fetch, and for chunk-sizing and
diagnostics, we need a forward prediction — a time-to-deliver estimate,
ramp(RTT) + estBurstS × size (hereafter the estimate), built from
network-mux's DeltaQ/RTT instrumentation
(estBurstS, seconds/byte, from its burst-SDU serialisation estimator;
RTT from its cookie-echo window; a branch feature today, not yet on
main).
Leveraging that instrumentation has a catch: the throughput input estBurstS is adversarially
gameable — a peer can pace its SDUs so estBurstS reads fast (high
apparent bandwidth) while it dribbles our data, earning a rank it doesn't
deserve and winning traffic it then serves slowly. (RTT is manipulable
too, but asymmetrically: a peer can't fake the cookie-echo RTT low — it
can't echo faster than the real round-trip — so cheap manipulation only
inflates it, which self-penalises by raising its time-to-deliver, especially
for small transfers, as described later. Lowering its actual RTT
means co-locating near the victim: real, but expensive, where gaming
estBurstS is free. A co-located peer that looks genuinely fast while
dribbling the payload is caught by the actual-vs-predicted per-fetch check,
not by RTT.)
One prerequisite deserves flagging up front: the estimate needs the
datum's size before the fetch starts, so the protocol must advertise
the size of what it offers (an EB, a closure). In Cardano, payload sizes aren't carried in network-layer messages by
default, but a protocol's message schema can mandate one — a CBOR-encoded
field in its CDDL spec. The catch is trust — unless the size is bound
to signed metadata (a header; see the size-manipulation section),
an advertised size is just the peer's unverified word, since nodes rely
on independent verification of the downloaded datum rather than each
other's claims. Such an unverified size is a third gameable input to the estimate,
and the most direct one, since the peer states it outright — a peer can win
the fetch not by being fast but by understating the size (or merely
undercutting what competitors advertise), steering us its way. The lie
surfaces only after the fact — the delivered size overshoots the
advertised one, so actual ≫ predicted — the same delivered-vs-advertised
deviation tx-submission already penalises.
So the prediction has to be robust: a peer must not be able to look
good in the estimate while delivering badly. This document lays out how —
a promptness-gated burst window that yields a trustworthy estBurstS,
why ranking on time-to-deliver estimate makes a gamer self-penalise, and an
actual-vs-predicted backstop that audits the estimate against measured
T_wait and churns consistently-off peers. The hierarchy throughout is
that direct measured completion is the preferred basis: the peer ranking
— the very decision a pacer sets out to game — rests on measured
completion wherever a fetch has completed, and there pacing gains
nothing. The estBurstS hardening exists to make the prediction
trustworthy for the roles measurement can't fill: choosing a peer before
a fetch, chunk-sizing, slow-loris, and diagnostics.
All four deny us a timely fetch — an EB body or closure that doesn't arrive in time for the round's diffusion/certification deadline — cheaply for the adversary. Three of them win by looking good enough to be chosen and then not delivering; slow-loris simply drip-feeds a fetch it already holds. They differ in how they're caught.
- RTT manipulation. Make apparent RTT low to score well on the ramp term. A peer can't fake the cookie-echo RTT low — it can't echo faster than the round-trip — but it can buy a low real RTT by co-locating near us (expensive), or inflate its RTT by delaying echoes, which only self-penalises. Caught by: the ranking (inflation self-penalises); a co-located peer that then dribbles falls to the per-fetch check. → Using the RTT signals.
- Pacing (idle-gap). Time the delivery so the estimate reads fast — a flatteringly low time-to-deliver — while the closure actually arrives slowly; win the fetch on that estimate, then dribble the payload. Caught by: the promptness-gated burst window and the adversary bind (the timing trick, and why it backfires, are spelled out there). → The window-sizing tension; The adversary bind.
-
Byte-padding. Make the estimate look good by serving other
traffic fast, while dribbling the closure we actually asked for —
targeted withholding (the data-withholding threats T20/T21). The
estimate stays flattering because it reflects the peer's aggregate
traffic, not our specific fetch. Caught by: per-fetch completion
timing (
T_wait) and the actual-vs-predicted check, on persistence. → The adversary bind (byte-padding). -
Slow-loris. Accept the fetch and drip-feed it — no attempt to
hide; it ties up our fetch slot and, at scale, our concurrent-fetch
capacity, denying us throughput cheaply (we wait on it instead of a
good peer). Unlike pacing it doesn't fake fast — it looks slow.
Caught by: the ranking (it scores low) and throughput-collapse
detection (
estBurstSrising) — though recourse is limited, since no per-fetch cancel exists yet (only wait / reset / hedge, each a trade-off). → The bigger picture.
Those four game a single delivery measurement, and the machinery in the rest of this document is what catches them. A distributed blockchain setting adds three that game the ranking system rather than any one measurement — across time, items, and identities — and these lean on redundancy and peer-selection layers beyond the delivery score:
- Reputation farming, then a deadline-timed defection. Behave well to earn a top rank cheaply over low-stakes rounds, then withhold the one fetch that matters — an EB near its certification deadline — and revert. Score-based churn reacts only to a sustained low score, so a single withheld body — one non-first-delivery against hours of farmed good score — barely moves the peer's standing: the betrayal is close to free. The defence is therefore fetch redundancy for consensus-critical items — hedge across peers (Improvement #5) so no single defector is decisive — backed by weighting a consensus-critical miss far above an ordinary one in the score. Redundancy, not punishment, is what protects the fetch.
- Surgical withholding of the critical item. Byte-padding targets us across a whole connection; the per-item version serves everything fast except the single block/EB an adversary wants suppressed (censorship, fork support). A per-peer average rank barely moves, so the ranking alone won't flag it. The defence is protocol-level: multi-source diffusion fetches the item from someone else, and a missed consensus-critical fetch should weigh far above its share of the peer's history. The ranking's only job is to not rate the withholder top on the strength of the 99% it does serve.
- Baseline poisoning via Sybils. The actual-vs-predicted threshold is a p99 over a population baseline of honest peers; in a permissionless network an adversary runs many peers and inflates that baseline until a byte-padder fits under the loosened band. The impact is bounded — it desensitises detection, it doesn't falsely accuse honest peers — and it's mitigated by robust, per-subnet-capped aggregation — weight the baseline by IP-prefix / ASN rather than by peer, so a Sybil cluster in one subnet can't dominate it (identities are cheap, distinct network locations are not) — and by capping the band with the estimate itself — a physical model of delivery time — rather than the empirical p99 alone: the plausible residual (serve time, loss-recovery, stalls) has a physical ceiling a poisoned baseline can't inflate past.
A note on the response rather than a distinct attack. Currently in cardano-node, churn
is score-based: peers first to offer a header or first to serve a
requested body score well, and the worst-scoring ~20% are evicted each
hour and replaced with a random sample. (Transient failure is a
separate, faster path — a peer that blows response timeouts under a long
GC or under-provisioning is dropped immediately, not slowly churned.) This
design is largely eclipse-resistant rather than a ratchet: the resample
is random, so a Sybil fleet gets no better-than-proportional odds of
backfilling a freed slot, and the worst-20% eviction is absolute, so it
also culls the weakest of the adversary's own peers — even a brief local
eclipse leaks back out as those marginal peers churn and random (mostly
honest) draws replace them. What survives is only the patient top-scorer
of the reputation-farming attack: an adversary can farm score to hold a
hot slot, but it cannot use the churn machinery to displace honest peers
or take over the set. Cheaply forcing an honest peer out isn't available —
only expensive, on-path degradation (nation-state / ISP-tier, and not
ranking-specific) can. Diversity-/anti-eclipse-aware selection
(Improvement #8) hardens the residual, but the base mechanism already does
most of the work.
The estimate's size input is advertised, hence gameable. The
organising fact is that for a hash-named datum the content fixes the
true size — the byte count we will actually transfer — so any
disagreement is a provable lie once the datum is in hand; the difficulty
is only the choice before the download.
-
All advertisers agree. No steering — the common size scales every estimate identically, so ranking still turns on speed; run it normally. The twist is what it means if that agreed size proves a lie on download: no reachable peer told the truth, so either the whole neighbourhood is dishonest, faulty, or we are eclipsed. That is worth acting on beyond demoting the advertisers — but the response (defensively resampling for fresh peers to break a captured view) belongs to the peer-selection / anti-eclipse layer (Improvement #8), not to the delivery score. (A uniform honest size is entirely benign, misinforming only absolute deadline-feasibility, not the peer choice.)
-
Genuinely distinct datums (different hashes — several VRF election winners). Not equivocation; a capacity / protocol-storm question rather than a size dispute. Fetch what you can, prioritising by the consensus signal — which candidates the votes favour — and estimate each independently.
-
Same hash, conflicting sizes — the weaponised case, and the one to decide up front rather than defer to punishment. Because the hash fixes the content there is one true size; the advertisements are noisy, possibly adversarial observations of it, and a few cheap heuristics narrow it down before we commit:
- Corroborate, and reject outliers. The size a plurality of advertisers agree on is the likely truth; a lone dissenter — above all one below the rest, the steering direction — is discounted, not averaged in.
- Bound by a class prior. A size outside the plausible range for the datum class (a closure "advertised" at kilobytes where closures run to megabytes) is rejected on sight — a cheap filter for the crudest lies.
- Weight by credibility when advertisers genuinely conflict. With no majority to corroborate, believe the higher-ranked, longer-honest advertiser over an unknown; reputation is a prior on truthfulness, not only speed.
- Lean on the signal we measure, not the one we're told. We measure RTT ourselves; size we are merely told. So under residual size uncertainty, tilt toward the peer whose advantage holds across the plausible size range: a low-RTT peer wins a small datum outright and is no disaster on a large one (its ramp term stays small either way), whereas a far, high-throughput peer pays off only if the datum really is large. This both hedges the uncertainty and defangs the lie — a far peer that understates its size gains nothing, since we weight the RTT we measured over the size it claims.
Rank on the estimate these produce and pick the best, random tiebreak among genuine equals. The after-the-fact check is then only a backstop, not the plan: on delivery the true size is known, so a peer whose advertisement was inconsistent is provably lying and is demoted — a stronger, cleaner tx-submission deviation penalty (inconsistency with canonical content, not measurement noise) — which also feeds the reputation the credibility heuristic leans on.
-
Conflict arrives mid-fetch (a second peer offers the same datum at a different size while the first is in flight). Same recourse as a slow peer — wait, reset, or hedge — but the switch must be driven by the first peer's observed progress, never the newcomer's claimed smaller size; hedging on the claim alone rewards the lie in dynamic form.
Two structural notes bound the problem. The dispute only bites on the expensive, pull-based body fetch — and headers are the opposite: small enough to be pushed (Praos and EB headers diffuse proactively), so we generally hold and have already validated the datum's header before choosing whom to fetch the body from. If that header carries the body's size, we can trust it outright: having verified the header — its signature and our usual checks — we have verified everything it commits to, the size included, no matter which peer pushed it. The offer is then checked against a value we already trust, not taken on the offering peer's word. That closes the hole for EBs (and Praos blocks) whose headers carry a size, leaving only unattested aggregates such as closures. And the damage is bounded regardless: the worst outcome of a size lie is a single recoverable mis-selection — caught by actual-vs-predicted, remedied by hedge/reset — never a failed fetch.
Requests are small — often a single SDU — but responses arrive as a
burst of many SDUs, and the cookie on our request is echoed back on
that burst. The first response SDU's echo gives the RTT sample; the
follow-up SDUs of the same burst feed estBurstS, the serialisation
rate. A blocking request is the exception — we subscribe and the peer
pushes its first SDU only once it has a block, so that echo's round-trip
times the idle wait, not the network. Such a sample lands in the far tail,
which the robust RTT read already discounts: we take RTT_hi at the
p80–85 quantile — not p99 — precisely so a tail poisoned by GC pauses and
hiccups does not move it, and a blocking-wait round-trip is just one more
such tail value. The wait itself is no stall either, since the window
opens only on that first SDU (see the stall signal). That poses the
question the burst window exists to answer: for how
long do we keep attributing incoming SDUs to this response's burst —
counting them toward its throughput — before a new SDU counts as a fresh
exchange instead?
Its job is accurate measurement, not anti-pacing. It must extend far
enough to let a genuine multi-second closure accrue the byte-coverage
that earns an estBurstS its confidence weight (the appendix) — closing
early yields not just a shorter estimate but a less trustworthy one —
while stopping short of folding a severe stall (a deep RTO backoff) into
the rate, which should register as a stall event instead (below). A fixed
value serves neither end, because it is wrong across the RTT
spectrum rather than scaled to the peer (see Not tuning-free). And the
window is not what denies a pacer — a pacer is caught regardless of the
threshold (see the appendix), so the anti-gaming rests on coverage,
the bind and the actual-vs-predicted backstop.
There is no fixed timeout to reach, then — the window extends while the
peer keeps serving and ends when it goes quiet. That is the gate. For a
fetch, where the peer streams a one-way response, "still serving" is read
directly off the incoming SDU spacing: attribute SDUs to the burst
while successive SDUs arrive within the RTT-relative spacing threshold G;
a longer gap means the peer has stopped, and the burst ends there. A silent
pacer's burst ends here too — but ending it is measurement, not the
catch: the short burst carries low coverage and shrinks to the prior
(appendix), and that shrinkage — at any threshold — is what costs the
pacer its rank.
The spacing threshold has one honest hazard — a loss/RTO briefly stalls
the in-order stream and could trip it — but this is a measurement
hazard, not a hole in anti-gaming (which rests on coverage, the bind and
the backstop, not the gate; see the appendix). RTT-relative sizing handles
the common case: a fast-retransmit gap (~1 RTT) stays under the threshold
and folds into estBurstS rather than tripping it — only a severe
RTO-scale stall ends the burst, and that is a genuine event, registered by
the stall signal. Any residual one-off trip is absorbed by the
actual-vs-predicted backstop (below), which churns only a peer whose
delivery is consistently worse than its advertised rate.
There is a way to game the burst from the echo side. Coverage — the
trust an estBurstS earns (the appendix) — accrues for as long as the
burst window stays open, but nothing so far ties that to our RTT staying
fresh. A peer that keeps SDUs arriving, holding the window open across what
may be several distinct exchanges, while never echoing the cookies we send,
banks coverage over a rotting RTT probe: a trustworthy-looking rate resting
on a round-trip we last measured long ago, or never.
The fix binds the two together — coverage must be paid for in RTT
freshness. To keep the burst, and the trust its coverage buys, the peer
must echo something within G; every matched echo is a fresh RTT sample,
so a burst that goes on accruing is one whose RTT we go on re-measuring,
and a peer cannot bank coverage while starving us of RTT updates.
The gate that enforces this is the echo-side twin of the SDU-spacing one:
while a cookie is outstanding, end the burst if none is matched within
G. The clock is the time since the last matched echo, reset by each
echo and restarted when a fresh wait begins (a cookie entering an empty
queue), so it never carries a stale gap into a new wait. It needs no queue
traversal — only whether a cookie is outstanding, plus those two timestamps
the cookie machinery produces anyway; the queue's one structural job stays
RTT-matching, and the gate rides on it for free. Cumulative eviction keeps
it honest: a match drops every older cookie with it, so a peer echoing a
fresher cookie still resets the clock and leaves nothing stale behind —
we never ask which cookie is oldest.
The "while a cookie is outstanding" clause is what leaves a plain one-way fetch alone: its single request cookie is matched on the first response SDU, the queue empties, and with nothing outstanding the gate is dormant — the SDU-spacing gate carries the stream. The echo gate bites only where cookies keep flowing (another protocol exchange, or mux-injected cookies), which is exactly where the starvation it guards against lives.
Two things keep this sound. First, it is a wall-clock receive-side gap,
not a comparison of the oldest cookie's age against the self-measured
RTT quantile — which would be circular: a peer delaying every echo
uniformly inflates the front-age and the RTT together (a cookie echoed
at t0 + RTT + D gives both a front-age peak and an RTT sample of
RTT + D), and G = k·RTT_hi rises with it, so that test never trips.
The gap sidesteps the loop because G is fed only by matched echoes,
while the gap is measured independently of them. Second, resetting the gap
requires an echo, and both of the peer's options cost it: a recent
cookie (a low, honest sample → RTT fresh) or a stale one (a high-lag
sample → high RTT_hi → ranked slow) — refresh RTT, or be ranked slow. It
is also robust where a span would over-fire: a peer echoing recent cookies
while abandoning a pile of old ones has fresh RTT and rightly does not
trip; the gap grows only when echoes stop. The same k in G
absorbs a one-RTT loss-recovery, and only an RTO-scale gap trips and
registers as a stall event, as on the data channel.
An independent RTT reference buys cleaner, faster attribution, not the core
defence. A kernel tcpi_rtt or handshake baseline gives a
single-observation tell of adversarial app-layer echo-holding
(kernel RTT low, app-RTT high ⇒ the peer is sitting on cookies —
literally the existing "app-RTT vs kernel-RTT divergence" cross-check),
distinguishing it from an honestly far peer (both high). That matters
only for escalating from deprioritise — which self-measured signals
already do — to punish/demote. It has caveats: it needs Linux + the
tracetcpinfo build flag (not portable), and the handshake baseline is
drift-prone.
Attribution isn't contingent on it, though: the actual-vs-predicted
comparison (below) recovers a form of it portably — a persistent
actual ≫ predicted identifies targeted withholding using only
app-layer signals (direct T_wait, estBurstS, RTT), no kernel/handshake
reference. It convicts more slowly (it needs persistence to clear the
serve_time/cold-cache confound) and it catches the attack by a different
tell.
What makes the gate sound is a genuine dilemma the adversary faces on a single measured stream:
- To make delivery slow, the stream must be idle a lot (low byte-rate).
- To game a gap-based
estBurstS, that idle must fall outside bursts (the burst timer expires during silence, so the stall isn't counted against it, ie. does not increaseestBurstSwhen it should have). - But idle = no SDUs = no echoes → the cookie queue ages / RTT samples inflate.
- To keep echoes prompt, it must fill the idle with SDUs — but then
the burst stays alive, the fill is folded into
estBurstS, and the estimate reads the true (slow) rate.
So it cannot simultaneously game a gap-based throughput estimate and keep the liveness/echo signal flowing: whichever activity carries the prompt echoes is measured, and going silent to hide the stall shows up as echo lag. Pick one, caught by the other.
Cardano's mini-protocols are mostly pull-based, which tightens this further: the peer sends SDUs only in response to our requests, so it cannot inject filler at will to keep echoes flowing. The fill would have to be responses to some other request we happen to have outstanding to it — an opportunity it cannot manufacture — so busy-fill is opportunistic and hard to exploit reliably. Absent such traffic, the only way to withhold our closure is to fall silent, which the echo-lag / SDU-spacing gate catches directly.
Aggregate (protocol-mixed) estBurstS is what closes the loophole.
The bind holds only if the echo-carrying SDUs are inside the
measurement — and today estBurstS is connection-level, so every
received SDU, whatever mini-protocol carries the echo, feeds the
aggregate burst. There is no unmeasured side-channel to park prompt
echoes on. The protocol-mixed aggregate we usually treat as a limitation
is, for this attack, the strength.
Caveat for a future per-protocol split. The loophole reopens if
per-protocol estBurstS (the proposed "Shape B" split — per-mini-protocol
DeltaQ buckets, not implemented today) is added without also binding the
cookie/RTT per-protocol — because then the throughput signal and the
echo signal live at different granularities. Concretely: we fetch a
12 MB closure over LeiosFetch (protocol B) and are also driving a second
mini-protocol A (keep-alives, say, or another fetch) on the same
connection.
- It delivers the closure in tight 50 kB micro-bursts, then goes silent
on B for ~500 ms, repeat. Per-protocol
estBurstS_Bcounts only the fast micro-bursts — the 500 ms silences exceed B's burst gap and fall outside its bursts — so it reads fast. Gamed. - During each B-silence the peer keeps answering our A requests promptly, each response echoing our latest cookie. The cookie PSQ is connection-level, so those A echoes keep the queue drained and RTT low — the gate stays quiet.
Result: estBurstS_B says LeiosFetch is fast and the gate says nothing
is held back → we rank the peer a great closure-deliverer, route
closures to it, and it dribbles them. A is a side-channel that
estBurstS_B doesn't see, yet it still satisfies the connection-level
gate — exactly the gap the bind depends on being closed. The loophole
is gated on A traffic frequent enough to keep draining the
connection-level queue through the B-silences, though — consistent with
the pull-based limit above, the peer can't fabricate that; it can only
ride A traffic we ourselves drive.
The aggregate estimate doesn't have this gap: those same A responses
are inside the aggregate burst, so they either keep it alive (⇒
estBurstS reads the slow closure rate) or, if the peer goes fully
silent to avoid that, the echoes lag (⇒ caught). The fix under a
per-protocol split is to keep
the two signals at the same granularity — judge B's delivery against
echoes that ride B, so a silence on B costs echo-lag on B. Rule: go
per-protocol on estBurstS ⇒ go per-protocol on the echo/RTT too.
Where the payoff hides — and how each strategy leaks. The
adversary's payoff is always the same: pace the bursts so estBurstS
reads fast, win the fetch, then dribble the payload. Stalling echoes or
inflating RTT is never the goal — it is only a leak of the pacing, and
a different signal catches each leak.
-
Busy-fill — fill the stalls with SDUs so the echoes stay prompt
and nothing looks held back. But those filler SDUs keep the burst
alive, so they fold into
estBurstS, which then reads the honest, slow rate.estBurstSitself catches it. -
Idle-gap — go silent between micro-bursts so the stalls fall
outside the burst and the raw
estBurstSreads fast. That backfires two ways: the short bursts starve coverage, so the confidence-weightedestBurstSshrinks back to the average; and where cookies keep flowing, the silence delays their echoes and inflates the RTT quantile. Either self-penalises in the ranking — and the RTT inflation, against an independent reference (kerneltcpi_rtt/ handshake), is attributable as app-layer echo-holding.
(A queue-vs-self-RTT comparison that might otherwise flag the stall is circular — Using the RTT signals — so we rely on the bind, which catches the idle-gap gamer without it.) The third leak, byte-padding, escapes both of these and is treated next.
The one escape the bind does not close: byte-padding. The peer
sends other bytes fast, serving another protocol request,
so the aggregate byte-rate looks high while it dribbles
the closure you actually asked for. estBurstS reads fast,
the padding carries prompt echoes (the gate stays quiet), yet your closure
is slow. A measurement bound to that specific fetch — direct
completion timing (T_wait on the closure; next section) — catches it,
because the padding isn't the bytes you're waiting on. And comparing
that T_wait to the peer's own estimate upgrades detection to
attribution: under padding the prediction is fast but the actual is
slow, and actual ≫ predicted
means the peer has the capacity (it's spending it elsewhere) and is
withholding from us (the data-withholding threats T20/T21), not merely
slow. Lose-lose for the padder: advertise fast and dribble ours ⇒
discrepancy; advertise slow to
match ⇒ ranked down anyway; deliver ours fast ⇒ not the attack. Caveat:
honest confounds fire the same discrepancy — server-side serve time for
our closure, HOL/contention on the shared bearer, stale estBurstS/RTT
— so deprioritise on a one-off and punish (demote) only on
persistence with those ruled out.
The serve-time confound is the sharpest, because it produces the same
observable, not just a noisier one. estBurstS/RTT capture the
byte-flow rate and the network round-trip, but not the peer's cost to
produce the bytes — reading our closure's txs from its store,
reassembling it:
actual = RTT + serve_time + size × per-fetch-rate + loss/RTO
predicted = RTT + size × estBurstS (aggregate signals)
The prediction has no serve_time term, so actual − predicted ≈ serve_time: a large serve time alone reads as actual ≫ predicted. So
a peer whose TxCache holds others' closures (hot,
served fast) but not ours (cold, disk-read) shows a fast aggregate and a
slow closure — capacity that exists but isn't reaching our request,
exactly like byte-padding, yet benign (which closures are hot/cold is
diffusion history, not the peer choosing to starve us). Decomposing our
fetch into time-to-first-byte (which absorbs serve time) vs the
whole-fetch byte-rate separates the two if the peer batch-reads (long
TTFB, then a normal rate ≠ dribbling) — but a slow streaming read
shows a slow rate throughout and is indistinguishable from withholding
by rate alone. So the discrepancy stays a deprioritise signal (correct
either way — the peer is slow for us now); promoting it to malice
needs the slowness to be systematic across our requests to that peer
(targeting), not per-request random (cache-luck).
The cookie/promptness queue is the key feature: the gate that lets the burst window extend to measure an honest large closure, which a quiet peer forfeits (the window-sizing tension above). The consistency and actual-vs-predicted checks sit on top of it as the backstop, not as a substitute for it. It reuses machinery already present — the t-digest and the cookie PSQ; what is new is gating the extension on them.
Setting the threshold for the actual-vs-predicted check. The band —
how far actual may exceed predicted before the byte-padding check acts
— is an empirical high quantile (say p99) of the residual
actual − predicted, drawn from a population baseline across honest
peers rather than the suspect's own samples, which it could inflate to
widen its band and hide under. The tracked estDeltaQVVar is unsuitable
here: as per-SDU micro-jitter it is the wrong granularity — blind to the
correlated per-fetch terms (ramp-round count, loss-recovery, serve_time),
it under-estimates the real spread, so the band would come out far too
tight — and the wrong shape, since the residual is heavy-tailed and a
σ-band mis-fits it. This band is only the threshold, though: the check's
second gate is a per-peer hold-back (below) — the estBurstS standard
error SE, variance-derived but distinct from the rejected
estDeltaQVVar, vetoing a conviction that falls within the prediction's
own error bar.
Stratify that baseline by regime, because the residual's spread is not
one distribution. It scales with message size (more bytes ⇒ more
loss-recovery exposure and a longer transfer) and with RTT (both the
ramp-round count and the RTO magnitude grow with it — and that
dependence survives even though predicted already subtracts a
ramp(RTT) term). Loss rate is the natural third axis, but it is
non-portable — only via kernel tcpi_retrans/tcpi_lost (Linux +
tracetcpinfo, design-phase) — so a first pass stratifies on
size × RTT only and pools clean and lossy paths in the same bin. The
cost is a deliberately conservative band: the lossy fetches in a bin
inflate its p99, widening the threshold, so first-pass detection is less
sensitive (a padder can hide under the loss-inflated band) but rarely
false-positives. Add the loss stratum to tighten it once kernel
telemetry is available on the measuring node.
The pacing attack only matters to the extent we use estBurstS as a
proxy for delivery quality (peer ranking, chunk sizing). But those
decisions are being offloaded onto direct completion timing (T_wait /
observed arrival). Against direct completion timing, pacing is
self-defeating: the adversary's paced-slow delivery is the
measured completion time, so it ranks low no matter how it games the
per-SDU gaps. So the highest-leverage defence against estBurstS-pacing
is not to trust estBurstS for the decision that pacing targets —
rank on observed completion, and pacing buys the adversary nothing.
Even short of direct timing, ranking on the estimate (Improvement #1's
fused metric) already absorbs the idle-gap gamer, as shown above. This
needs no independent cross-check — you never have to distinguish the
gamer from an honestly slow peer; the estimate ranks both down regardless.
The gamer is ranked down on a coverage-shrunk estBurstS — its short
bursts starve coverage, so the fast rate it engineered decays to the
average — and, where cookies keep flowing, on an RTT its silence inflates;
an honest slow peer, on its genuine rate and RTT. So we deprioritise both
without telling them apart, and the tcpi_rtt/handshake anchor is only for
attributing and punishing the gamer specifically. Two caveats: for the
RTT route to bite, the ranking quantile must be high enough to capture
the stall inflation (occasional long stalls hide above p80); and
byte-padding — a different attack — evades the estimate outright, with a
genuinely fast estBurstS and low RTT (see the bind). So the ordering
is: measured completion timing closes both; the fused estimate is a good
fallback that handles idle-gap for free.
That leaves estBurstS needing hardening only for its residual
uses — diagnostic decomposition, and slow-loris detection. Slow-loris is
the mirror image of pacing: the peer makes no attempt to hide its
slowness, it just accepts the fetch and trickles it, tying up our slot.
Because it looks slow, the time-to-deliver ranking deprioritises it for
free; and a peer whose estBurstS rises mid-transfer — throughput
collapsing below a floor — is the slow-loris signature (Improvement #3).
Answering it is harder than spotting it, though: there is no per-fetch
cancellation today, so the current stuck fetch has only three imperfect
moves — wait (risking the deadline), reset the connection
(degrading our peer topology and, done reflexively, opening an eclipse
surface — and a slow peer can beat the even-slower one we might draw to
replace it), or hedge to another peer (Improvement #5), the most
resource-intensive, since our socket buffers and ingress must then
service both the slow-loris and the hedge. (When to hedge develops the
triggers for it — a loss signal, and a progress comparison against the
remaining work.) The deprioritisation above is
the only lever needing no cancellation — but it is preventive (we stop
choosing the peer), not curative (it can't rescue a fetch already in
flight). So estBurstS earns its keep here in the opposite direction
from pacing
— flagging a peer that looks slow, not one faking fast — where
pacing-to-look-fast can't help the adversary. The estBurstS-vs-RTT
consistency check, meanwhile, is worth keeping as a sanity gate for when
you do lean on estBurstS, but it shouldn't be load-bearing.
estBurstS is low-level and gameable, so building the prediction on it
looks risky — but it's safe because of the role it plays:
-
Center-line, not tolerance. It supplies the peer's advertised
rate in the estimate. The tolerance — how far
actualmay exceed that — is a separate quantity: a high quantile (p99) of the residualactual − predictedmeasured across honest peers (the threshold discussion above). Call that honest-peer residual distribution the baseline.estBurstS's missing macro terms (serve_time, stalls, loss-recovery) don't corrupt the check: becausepredictedomits them, they surface in honest peers' residuals too, so they're already folded into the baseline — a suspect peer is flagged only by deviating above it. -
Audited, not trusted.
actual(directT_wait) is the auditor;predicted(fromestBurstS) is the claim under audit — its unreliability is fine because it's the thing being checked, not the ground truth. -
Inaccuracy cancels; gaming self-defeats. Because that baseline is
computed with the same
estBurstS, any systematic bias inestBurstS(say it reads 10% fast for everyone) shifts every peer'spredictedalike, shifts the baseline with it, and cancels in the residual-vs-baseline comparison — so the check needsestBurstSconsistent across peers, not accurate. And the only useful lie is to makeestBurstSlook low, which shrinkspredictedand makes the check more sensitive; lying it high advertises slow and ranks the peer out.
The one role estBurstS is not given — trusted ground truth for
ranking — is the one it couldn't survive; that's why ranking ultimately
prefers measured time-to-deliver. Correspondingly, the tolerance band is
the empirical residual, not the per-SDU estDeltaQVVar (the threshold
discussion above).
Several mechanisms here and in the companion documents share a shape: a
judgement made relative to a population — an estBurstS shrunk toward
μ_p, a residual measured against the honest-peer baseline, a mesh dip
scored against each peer's own norm. Relativity is not the hazard; it is
often a strength, since a bias common to a signal and its reference cancels
(the inaccuracy cancels argument above: a suspect is flagged only by
deviating from a baseline built with the same estBurstS). The hazard is a
reference frame with nothing outside it — a loop that audits the
population only by the population, so a slow shared drift, or an attack that
moves the whole frame, is never seen. Three parts keep these mechanisms
clear of that trap:
- Relative signals carry attention and confidence, never conviction — the population decides where to look and how much a reading is worth (shrinkage, precision-weighting), not guilt.
-
An absolute, non-adaptable measurement pins the frame. Conviction
rests on a quantity not derived from the population and unable to drift
with it:
T_waitagainst the real deadline, an absolute event (a missed deadline, an RTO-scale stall), or a fixed external envelope. It is the same reason a threshold must never be fed by the signal it gates (Using the RTT signals:Gco-moving with cookie front-age would let a uniform-delay attack inflate both and never trip) — the anchor is chosen precisely because it cannot co-move. - Act only safe-directionally, so the loop is negative feedback. Every belief-driven action adds caution — deprioritise, hedge, back off, churn, down-weight — and each counteracts the condition it reacts to, so a false alarm erases its own evidence instead of compounding it: a mistaken "stressed" reading backs production off, load lightens, delivery recovers, and fewer peers read as degraded, so the belief subsides; a mistaken "concentrated" reading churns in fresh peers and dilutes the correlation that raised it. A positive-feedback loop would let the action feed its own evidence and run away; this one regulates toward a fixed external target instead — the CIP-0164 envelope, or the honest-majority pack size the diversity belief needs. That fixed target is the absolute setpoint: the loop hunts it the way TCP's AIMD hunts a link's true capacity — back off on the signal, ease back up once it clears — settling around reality rather than a figure of its own making.
Ranking is the first instance: the estimate is a population-relative
prediction audited by absolute T_wait (audited, not trusted), and the
echo gate is anchored by a gap measured independently of G. The same
invariant governs the two derived beliefs that pool this machinery — the
mesh-stress filter, whose relative dip-count is backed by the fixed
CIP-0164 envelope that cannot adapt away
(leios-testnet.md), and the
effective-diversity belief, whose correlation is taken only after the
common mode is removed and over absolute degradation events
(leios-fetch-scheme.md).
Both are relative for speed and absolute for the frame, and neither acts
except to add caution.
What this does not buy. The guarantee is only as strong as the anchors
staying trustworthy and outside adversary reach. That holds today — T_wait
is our own measurement, the envelope a spec constant, a missed deadline
absolute — but it is the load-bearing assumption; an anchor that became
gameable would void the rest. Three costs come with it. A fixed anchor and its relative
counterpart fail in opposite directions, and the anchor pays for not
adapting: a relative channel drifts with a legitimate slide until it falls
silent (the boiling frog), so the fixed one is set not to — which means a
healthy-but-slower network, legitimately past the design envelope yet
otherwise fine, trips it instead (crying wolf). Safe-directional, but a real
tuning cost, and the reason mesh-stress runs both channels rather than
either alone. A relative estimate over a small pool (the five-peer
correlation) is noisy and will sometimes see a cluster or dip that isn't
there; persistence and shrinkage damp it and safe-directionality caps the
fallout at throughput rather than safety, but the false alarms are real. And
the discipline holds by construction, not by anything that enforces it —
the first time a purely relative belief drives a consequential action with
positive feedback and no absolute cross-check, this protection is gone, so
extending the machinery means re-checking all three parts rather than
assuming they carry over.
The safe-directional boundary. The third part has a boundary worth
naming, because a live attack sits exactly on it. "Safe-directional ⇒
negative feedback" holds only while the load an action sheds is load the
same loop controls. If a continuous adversary outside the loop reoccupies
the capacity a safe-directional backoff frees — the expensive tier of the
sustained-overload livelock in
leios-testnet.md
and threat-model-additions.md's T42, where a
saturator floods the voter quorum regardless of anyone's body size — then
backing off is a capacity donation, and the loop drives honest load toward
zero while the adversary sustains the overload. Safe-directionality is
self-defeating on that continuous channel. (A merely stake-gated
attacker, one who over-produces only on the slots it wins, does not trigger
this: it cannot reoccupy honest rounds, so honest backoff there heals — the
boundary is specifically about a continuous, out-of-loop load.) The repair
keeps the principle and guards the action: back off only while the shed load
actually reduces the total, and where it does not — a continuous adversary
is refilling — switch from self-shedding to attribution and denial,
deprioritising the party that will not share the loop's restraint. Conviction
still rests on the absolute anchor; what changes is the recognition that the
safe direction is only safe when those bearing the restraint are the ones
generating the load.
-
Your own GC is a confound for the gate. A GC pause makes us
timestamp echoes (and incoming SDUs) late, stretching the echo gap and
the SDU-spacing reading — looking like the peer stalled, a false
extension-revocation that blames the peer for our own pause. Same class
of false trip as a loss/RTO (the window-sizing section). Two things
absorb it: the backstop's persistence requirement over time, and —
more immediately — the cross-peer common-mode reject (appendix),
since a self-pause stalls every peer at once, so a synchronised stall
is charged to us, not any peer, and suppressed (no gate trip, no stall
count). Both are portable; kernel
tcpi_rtt, where available, is immune to our GC anyway — a reason to prefer it for the RTT input. -
Lazy sampling. The RTT quantile feeding
ranking/prediction is read from the t-digest every ~1 s in
processIngress, not per-SDU — preserving the finger-tree's deferred<>. (The extension gate itself is a cheap echo-gap / SDU-spacing check, separate from the t-digest.) -
Not tuning-free — and the promptness threshold is RTT-relative. The
knobs are the promptness threshold
G(max back-to-back SDU spacing / echo gap that keeps the burst extending), the backstop quantile (p99), and the churn-persistence count.Gmust scale with RTT, not be a fixed millisecond value:G = max(floor, k · RTT_hi), whereRTT_hiis the high RTT quantile we already track (say p80–85) and the multiplek(distinct from the appendix's coverage-scalek) is small. The quantile carries the jitter, sokcovers only the fast-retransmit recovery (~1 round-trip, plus a hair for loss coinciding with an elevated RTT) — not variability twice over:k ≈ 1.5is reasonable, whereask ≈ 2would double-count and risk pushingGup into RTO-scale, where a severe stall should instead trip and register as a stall event. Floor a few tens of ms (scheduling/GC jitter). A fixed absolute value strands far peers — their ~1-RTT loss-recovery gaps exceed it, so honest transfers fragment into low-coverage bursts that never earn a trustedestBurstS; scaling to RTT folds a fast-retransmit into the burst as a slower-but-trusted rate, while a severe RTO-scale stall still exceedsG, ends the burst, and registers as a stall event. Its risk is false trips (a genuine loss/RTO or GC), not gaming — a steady-dribbler just under it keeps extending and is measured slow, which is correct — and those false trips are what the backstop's persistence tolerates. The knobs are physically grounded (an RTT multiple, a percentile, a count) but still need calibration. - Download liveness needs no cookies. In a one-directional download the incoming-SDU-spacing gate supplies liveness directly; the cookie signal (for RTT) can go stale there — cookie-carrying outbound is sparse — so inject a cookie periodically (a keep-alive echo on a dedicated mini-protocol) to keep a fresh RTT and to keep the sent-cookie queue aging as a live gauge through a long download. This is not supported today and would take some implementation effort — work this analysis motivates.
- Response on detection is cheap and low-blast-radius: the ranking already deprioritises a gamer; on a persistent actual-vs-predicted discrepancy, churn/demote it. It's a peer trying to look good, so a false demotion costs little.
The burst window's job is accurate measurement, not anti-pacing:
gating the extension on peer promptness (cookie-echo gap, or
incoming SDU spacing on a download) keeps an honest large transfer's
window open so it accrues coverage, while a genuinely quiet peer closes
it. On the echo side that coverage is paid for in RTT freshness — a
burst keeps accruing only while the peer keeps echoing, so it cannot bank
coverage while starving our RTT probe. The threshold G is RTT-relative
(max(floor, k · RTT_hi)), so a
fast-retransmit folds into estBurstS and only a severe stall ends the
burst — far peers aren't stranded by an absolute cap. Anti-gaming does
not rest on the gate's tightness — a pacer is caught by coverage, the
bind, and the actual-vs-predicted backstop whatever the threshold: churn
a peer whose delivery is consistently worse than its advertised rate,
which also absorbs the gate's false trips (loss/RTO, GC) via the
consistency requirement. An independent RTT reference (kernel tcpi_rtt / handshake)
is a nice-to-have for fast attribution, not a prerequisite. And all of
this hardens the prediction: the ranking itself prefers
directly-measured completion wherever a fetch has finished, where a pacer
gains nothing. More broadly, it is one instance of the discipline the whole
design follows — relative signals, absolute anchors: population-referenced
quantities (the estimate, coverage, the baseline) allocate attention and
weight confidence, while an absolute measurement that cannot drift with them
(T_wait, a missed deadline) carries conviction, and actions only ever add
caution. The mesh-stress and effective-diversity beliefs built on this
machinery obey the same rule.
For a small download the estimate is ramp(RTT)-dominated: estBurstS × size is negligible, so the ranking should be RTT-led and the noisy
serialisation term given little regard — the same "lean on the signal we
measure" logic the size-manipulation section applies to round-trip-
dominated transfers. The complication is that ramp is not a fixed
function of RTT; it depends on the connection's congestion state. Once a
connection sits idle beyond about one RTO (Linux's
tcp_slow_start_after_idle), cwnd is reset to the restart window, and the
first ~log₂(size/IW) round-trips are window-limited rather than
path-limited. The same peer therefore has a one-RTT ramp warm and a
several-RTT ramp cold — which a fixed ramp(RTT) cannot express, and
which for a small fetch dominates the whole estimate.
Two things follow. First, model the state rather than ignore it: we
already keep the peer's last-use timestamp (the one driving the staleness
decay), so idle beyond an RTO flags a likely slow-start restart, and the
ramp is inflated by the slow-start rounds for that size — or, where
tcp_info's tcpi_snd_cwnd is available (the non-portable path), cwnd is
read directly. This is partly a real effect, not just noise — a warm
connection genuinely delivers a small fetch faster — so modelling it makes
the ranking more correct, not merely less punitive; keep it a per-fetch
ramp adjustment, though, not a rewrite of the peer's long-run μ_p.
(Truly tiny transfers, size < IW, fit in the first window and pay no
slow-start penalty at all; the bite is the medium-small range that spans
several windows yet is still ramp-dominated.)
Second, do not convict on small fetches. The actual-vs-predicted
backstop has little discriminating power when the residual is dominated by
connection state we cannot fully observe: a small-fetch actual ≫ predicted is far more likely slow-start-after-idle than withholding.
Reserve conviction for larger fetches, where the serialisation term — the
one that reflects sustained withholding — dominates and estBurstS is
meaningful. The size×RTT stratification already widens the band in the
small-size bins (their honest residuals swing with connection warmth,
inflating the per-bin p99); the rule is simply "rank small fetches, but
don't punish on them."
The confound can also be removed at the source: keeping connections warm
(periodic keep-alives, or disabling tcp_slow_start_after_idle on these
sockets) makes the ramp predictable. A node's hot upstreams may see enough
traffic to stay warm anyway; it is the rarely-used peers — exactly the
ones the staleness decay is already cooling — where both effects bite
together, a peer cold in the estimate likely sitting on a cold connection.
Two complementary triggers — both portable — say when to hedge or switch
away from a peer mid-fetch: a loss signal that the current peer is
failing, and a progress comparison that a candidate would finish the
remaining work sooner. They interact — a severe stall is an alarm in its
own right and also feeds the comparison, flipping the stalled peer onto
cold footing (below). Hedging itself is Improvement #5's portable,
Linux-free fallback (leios-fetch-scheme.md).
The loss signal warns that the current peer is degrading, from two
app-layer readings that need no kernel telemetry: the estBurstS variance
for minor loss, and a stall counter for the severe tail.
The first is the estBurstS variance. It rises under retransmission,
so it proxies minor loss — the one legitimate use of a signal the
confidence appendix otherwise rejects (there it is perverse: a metronomic
pacer fakes a low variance to look high-confidence). It is safe here
precisely because hedging acts only on high variance, never on the low
reading a pacer can manufacture; and doubly so because a pacer able to fake
that low reading is unlikely to be selected anyway — its short bursts
starve coverage, shrinking its estBurstS to the average, so only a weak
or eclipsed pack lets it through.
That loss-proxy use also argues for tracking something the burst window
currently drops: the stalls that expire it. G sets the division of
labour. A sub-threshold loss — a fast-retransmit, ~1 RTT, within G (see
Not tuning-free) — folds into estBurstS as a slower-but-trusted rate
and shows in its variance; a severe event — a deep RTO backoff exceeding
G — ends the burst and escapes both, exactly where a loss-driven hedge
matters most. So estBurstS absorbs the routine recovery, and a separate
signal must catch the severe tail.
Track that as a separate stall signal: the rate and duration of gaps long enough to expire the burst window — separate because the scales don't mix (millisecond per-SDU jitter would swamp 100 ms–second stalls if folded into the variance). A candidate gap must expire an open window — one that opens on the first response SDU and closes when the transfer completes — so the wait before a burst never counts. That is what keeps a blocking request honest: the arbitrary idle before its pushed first response lies outside any window, and the echo-channel gate is self-resetting, so that pre-response gap never enters the burst's accounting.
One exclusion is essential: don't count slow-start gaps. A connection
ramping from a cold congestion window — a fresh peer, or one idle long
enough for tcp_slow_start_after_idle to reset cwnd — delivers in
cwnd-limited windows with ~1-RTT gaps between them: normal operation, not
loss. Counting them would make us hedge on essentially every cold
transfer, worst on the rarely-used peers the staleness decay is already
cooling. Gate the signal on connection warmth (the same slow-start-state
inference the section above uses — idle time, or cwnd where
tcp_info is available); the tell is that slow-start gaps coincide with
rising throughput as cwnd grows, the opposite of a loss stall's
collapse.
The signal is cause-agnostic: link congestion, peer-side GC, and adversarial pacing all mean the same for the decision — "this peer is stalling: hedge away, rank down" — and disentangling them is the backstop/bind/independent-RTT's job, not the counter's. Use it under the same asymmetric rule as the variance: high → hedge more / rank down, but never let stall absence raise confidence or drop a hedge (a metronomic pacer stalls-free on purpose).
Two side-benefits: it makes an idle-gap pacer's between-burst silence
visible — today that silence escapes both estBurstS and the in-burst
variance — reinforcing the gate and the bind; and it completes the portable
loss proxy (variance for minor loss, stalls for severe), keeping the fetch
scheme's hedging fallback Linux-free. The cost is an O(1) counter on the
same last_update timestamp; its one confound is our own GC — it delays
our timestamp of the resumption and mis-blames the peer — absorbed, as for
the gate, by the persistence requirement.
The stall signal aggregated across peers is also a common-mode reject
for that self-pause confound — and better than persistence, because it
acts in the moment. A self-pause (our GC, scheduler, or uplink) stalls
every peer's stream at once, so a stall synchronised across a high
fraction of active peers is common-mode — our side, or a network-wide
event — not any peer's fault. Suppress attribution across that window (no
gate trip, no stall count, no actual-vs-predicted sample for anyone) and
attribute only isolated stalls, which are peer-specific. It is a
portable self-pause detector (no kernel tcpi_rtt needed), sharpens with
peer count (fall back to persistence when peers are few), and is
adversarially sound: an adversary can synchronise only its own peers, so
faking a common-mode window takes near-total control (eclipse) — and even
then blunts only the stall signal, not the after-the-fact
actual ≫ predicted.
Even with no stall, a peer can simply be slower than the alternatives
for what remains. The decision is then a direct comparison of projected
finish times for the remaining bytes, driven by observed progress (the
principle from Handling advertised-size manipulation, never a newcomer's
claim). Mid-fetch on peer A, with downloaded of an advertised size, the
work left is size_left = size − downloaded.
-
A is warm, so it pays no ramp — only serialisation of what is left,
at the live rate we are measuring on this very fetch (better than its
historical
estBurstS):t_A ≈ estBurstS_A_live × size_left. -
A candidate B is cold — a fresh request starts from a cold
cwnd— sot_B ≈ ramp(RTT_B) + estBurstS_B × size_left, therampcarrying the slow-start rounds forsize_left(Small downloads and TCP slow start). We assume cold because B'scwndis not portably observable (Linux-only, and unreliable), and the assumption is conservative — it inflatest_B, biasing against a switch.
Switch the remaining chunks to B only when t_B beats t_A by a margin
that covers the switch's own cost. The size-left dependence falls straight
out and unifies with the small-download logic: when size_left is small,
t_B ≈ ramp(RTT_B) — cold-start/RTT-dominated — so B rarely beats a warm A
on the tail and we hold; when size_left is still significant, the full
estimate decides.
The two triggers meet here — but a severe stall does not zero A forever.
It collapses A's cwnd, so A must re-pay the ramp: its side of the
comparison flips from the warm estBurstS_A_live × size_left to the full
cold estimate ramp(RTT_A) + estBurstS_A × size_left, with estBurstS_A
now taken from A's prior — the shrunk value into which recent degradation
has folded (a slow-loris reads slow there), not the pre-stall live rate
that may flatter a peer which has since failed. The comparison then runs as
usual: a cold, degraded A usually loses to B, though not automatically. So
the loss signal is the fast alarm that flips A onto cold footing; the
comparison settles whether to abandon it.
One mechanical dependency: resuming only size_left needs a chunk
boundary to resume from — chunked LeiosFetch has one, whereas an atomic
datum degrades to comparing B's whole-datum estimate (or a true parallel
hedge, B re-fetching everything).
Often B is not a peer we picked but one that offers the same datum
mid-fetch at a different — usually smaller — size (Handling
advertised-size manipulation). Its size then feeds t_B, so it must be
trusted with care. Where a signed header carries the size (EBs, Praos
blocks), trust it outright — having verified the header we have verified
everything it commits to, whoever pushed it — and B's identity does not
enter. Only unattested aggregates (closures) leave the size as B's word;
there, drive the switch on A's observed progress, never B's claim alone
(which would reward the size-lie in dynamic form), and weight B's offer
by its rank: a long-credible peer earns a bounded bonus — a smaller
switch margin, or more benefit of the doubt on its claimed size — while an
unproven one must clear the bar on progress alone. The bonus stays bounded
because leaning on a high rank is exactly what reputation-farming (behave,
then defect) sets out to exploit; and the exposure is small regardless —
the worst case is one recoverable mis-selection, caught by
actual-vs-predicted and remedied by a further hedge or reset, never a
failed fetch.
An estBurstS built from a long, sustained transfer — one where the
burst window kept extending across many SDUs — deserves more weight than
one scraped from a short or paced burst. To lean on the good estimates
and discount the thin ones, estBurstS needs a confidence attached to
it. Of two candidate measures — the variance of the estBurstS
regression, and the number of samples it was built from — the count is the
better basis, for a reason that is structural rather than statistical.
Variance and count are not rivals. If estBurstS is a fitted slope
(or a mean rate over near-constant SDU sizes), the statistically correct
confidence is not the raw residual variance — that is estDeltaQVVar —
but the standard error of the estimate, SE ≈ residual_sd / √N, which
folds the residual variance and the sample count into a single number —
their principled combination.
But variance — and SE with it — is the wrong signal for telling an
honest estimate from a paced one, and is in fact perverse. A metronomic
pacer emitting perfectly regular micro-bursts produces near-zero residual
variance, hence a tiny SE, and would therefore score as high
confidence. Variance measures precision and consistency, not honesty or
representativeness; a precise measurement of a gamed rate is still gamed.
Treating low variance as a positive trust signal rewards exactly the
regular emission a pacer can manufacture. (This is the same reason
estDeltaQVVar is rejected for the tolerance band above — it measures
the wrong thing.) Variance is not useless everywhere, though — it has one legitimate,
safe-direction use, as a loss proxy for hedging (When to hedge).
Byte-coverage is the trustworthy quality signal, for a structural
reason. The confidence weight is the payload delivered under the gate —
Σsize, plain accumulated bytes — rather than a raw SDU count: a flood of
tiny SDUs runs a counter up cheaply, whereas accumulated bytes cannot be
inflated without actually delivering them. Its
trustworthiness is structural, not statistical: coverage is downstream
of the extension gate. Under the gate, coverage can only accumulate by
serving promptly for the whole burst, and serving promptly throughout is
precisely what forces estBurstS to the true sustained rate (the bind). A pacer that stalls forfeits the extension, so its burst
stays short and its coverage low. High byte-coverage therefore certifies
a gate-vetted, sustained, prompt-served transfer — and hence an honest
estBurstS, whether that honest rate is fast or slow. The coverage is
less a confidence-in-a-number than a receipt that the gate let the
estimate run long, which it does only for honest service; it costs one
accumulator.
The two measures then do different jobs. Byte-coverage is the trust
weight: a thin, low-coverage estimate is shrunk toward a prior — the
peer's long-run estBurstS, or a population rate — in the manner of
empirical-Bayes shrinkage, while a high-coverage estimate is trusted as
measured. This is how a long extended burst earns its keep — it accrues
weight — and it handles the cold-start case gracefully. SE, meanwhile,
is the error bar on the estimate itself — the uncertainty in
estBurstS, and so in the predicted time-to-deliver — kept distinct from
the tolerance band, which stays the empirical residual across honest
peers (the threshold discussion above). Its role is to hold the actual-vs-predicted check back
when a discrepancy lies within the prediction's own error bars — a thin
estimate shouldn't convict — not to set the detection threshold.
From trust weight to peer ranking. Leaning on estBurstS means
letting it move a peer's rank in proportion to the coverage behind it, and
no more. For a candidate peer p and a datum of (robust-aggregate) size
S: shrink the peer's raw estBurstS toward a prior μ_p,
estBurstS_p^ = w·estBurstS_raw + (1−w)·μ_p
with w = cov / (cov + k)
k a coverage scale — see Setting the coverage scale k; form the
estimate
ramp(RTT_p) + estBurstS_p^ · S
rank ascending and take the lowest. Coverage enters only through w — SE stays the audit band, not
a ranking term. Pin μ_p as a separate, slower track — a long-horizon,
coverage-gated running estimate of the peer's rate (a population rate with
no history), not the previous decision's shrunk output fed back (see
Maintaining the prior μ_p below).
Two properties make this the right lean. It degrades to the trusted
signal: a thin estBurstS shrinks to the prior, so among under-measured
peers the throughput terms coincide and the ranking separates them on
their measured RTT — estBurstS casts a differentiating vote only once
its coverage earns one. And it closes the thin-sample gambit: thin means
low w and no advantage, while the only route to a trusted-and-fast
estBurstS is sustained service, which the bind measures honestly.
Resist bandit-style optimism. The tempting move — a UCB-type
(upper-confidence-bound) exploration bonus for high-uncertainty peers —
is backwards here, since
an adversary can arrange to be under-measured, so the bonus rewards the
attack. Shrinkage does the opposite: unproven is treated as average (the
prior), not optimistically fast. Exploration still happens — average peers
are chosen sometimes and then measured by T_wait — it is simply not
handed out as a bonus.
Confidence-weighting is not the conviction step; it and direct T_wait
are two halves of one box, guarding different regimes. Where no fetch has
completed — choosing a peer, chunk-sizing, diagnostics — coverage-shrinkage
rules, so a cheap fast reading cannot buy a top rank (a thin paced estimate
shrinks to the prior). Where a fetch has completed, T_wait rules, and a
peer is ranked on how slowly it actually delivered, pacing and all. So a
pacer is capped at average while unproven and ranked slow once measured —
never a trusted top rank at either stage, and warming up its estBurstS
over a long paced session does not help, since that same session stamps a
slow T_wait on every fetch and T_wait outranks the estimate. The
actual-vs-predicted backstop sits on top of both as the attribution that
escalates a persistent gap from deprioritise to demote/churn. The
subsections below make this precise, and the last shows it holds against a
rational pacer whatever the window threshold.
k fixes the crossover between prior and sample: at cov = k, w = ½,
so estBurstS_p^ is an even blend of the fresh sample and the prior —
below k the prior dominates, above it the sample does. In coverage
units (bytes), k is therefore "how much a peer must deliver before its
own measurement outweighs the baseline."
The low-coverage extreme is the cold-start case, handled with no special
rule. A brand-new peer has cov = 0, so w = 0 and the estimate is
purely the prior — and with no history of its own, that prior is the
population rate. A fresh peer is thus estimated at the typical rate:
average until proven, neither optimistically fast (which would let it
win traffic it hasn't earned, and reward anyone spinning up fresh
identities) nor pessimistically slow (which would starve a genuine
newcomer). As it delivers, cov grows, w rises, and the estimate
slides continuously from that default toward the peer's own measured rate
— the same mechanism by which a long burst earns its keep, now warming a
cold peer up — while μ_p personalises from the population rate toward
the peer's own long-run rate.
Four considerations fix it:
-
The principled value. Empirical-Bayes shrinkage gives
w = τ² / (τ² + σ²/cov), which iscov / (cov + k)withk = σ²/τ²—σ²the within-peer measurement noise (per unit coverage),τ²the between-peer spread of true rates, both estimable from data. When peers' rates differ widely (τ²large), a little coverage already places a peer against the pack, sokis small; when peers are alike (τ²small), the population prior is nearly as good as any sample, sokis large. Atcov = kthe measurement varianceσ²/covhas fallen to the prior varianceτ², so sample and prior are equally certain — which is why that coverage is the half-trust point. -
A pragmatic anchor. Absent estimates of
σ²andτ², anchorkto the workload: a fraction of a typical transfer's bytes, so one honest transfer reaches roughly half-trust and a few pushwtoward 1 — the behaviour a long extended burst should earn. -
An adversarial floor. A pacer emits thin samples, so
kmust sit above the coverage one cheaply produces in a burst; anchoring it near a real transfer does this for free, since reaching trust then requires delivering roughly a transfer's worth, and delivering that sustained is what the bind measures honestly. The ceiling is thatkmust not be so large that honest peers never earn trust. Between the two, lean high — in an adversarial setting, better slow to trust than easy to game. -
Calibration. With
T_waitas ground truth, backtestkon honest historical fetches to minimise prediction error (estimate vs actualT_wait), subject to the floor — turning "about a transfer's worth" into a measured number.
μ_p is a coverage-weighted average of the peer's past estBurstS
observations, maintainable two equivalent ways.
The online form folds each new burst in one step, in the familiar "new = old + step × error" shape:
μ_p ← μ_p + η·cov·(raw − μ_p)
raw − μ_p is how far this burst's fresh estimate sits from the
baseline; η is a small base rate setting the horizon (small = slow,
long memory); and cov is the burst's byte-coverage, scaling the nudge
so a fat honest burst moves the anchor while a thin sample barely does.
The step is η·cov; writing α = η·cov, this is exactly
μ_p ← (1−α)·μ_p + α·raw, a coverage-weighted EWMA. Keep α ≤ 1 for
stability — clamp it, or feed cov normalised to a typical transfer so a
huge burst can't overshoot past raw.
The batch form is the same coverage-weighting kept as running sums:
μ_p = Σ cov_i · raw_i / Σ cov_i
each observation weighted by its coverage, so a 10 MB burst counts 100× a 100 kB one.
Both forms are constant-space — one scalar for the EWMA, two accumulators
for the batch mean — and retain no sample window, unlike the finger-tree
/ t-digest the RTT quantile needs: a mean updates in O(1), an order
statistic cannot without the windowed distribution. Constant space is
not constant value, though: the EWMA's lone scalar stays within the range
of raw and cannot overflow, whereas the batch's running sums grow with
lifetime traffic — a reason to prefer the EWMA, or to bound the sums.
The two differ only in memory. The batch sum has infinite memory — an
old burst counts as much per byte as a recent one, an unbiased lifetime
average that will not follow a peer whose true rate drifts. The EWMA is
the same coverage-weighting with exponential forgetting (what η buys),
so it tracks drift; the batch sum is just its no-forgetting (η → 0)
limit. Use the batch form for a stable lifetime baseline, the EWMA (or a
windowed weighted mean) to follow genuine rate changes. Either way,
coverage is what keeps a thin sample from moving the anchor — the same
guard the per-decision shrinkage applies, now on the prior itself.
Both forms so far forget over observations — a new burst decays older
ones — which does nothing for a peer that has gone unused: with no new
bursts arriving, its estimate sits frozen and grows stale as network
conditions drift. Staleness needs decay in wall-clock time, not
observation count. Two decays, keyed on the elapsed Δt since the peer
was last measured, walk a stale peer back toward cold:
-
Decay the trust weight. Scale coverage by elapsed time,
cov_eff = cov · exp(−Δt / T), so asΔtgrowsw → 0and the estimate slides continuously back to the priorμ_p;Ttracks how fast conditions change. (A hard "reset to cold afterX" is the step-function special case; the exponential avoids the cliff where a peer used just underXago is fully trusted and just over it fully cold.) Applied on every update — decaycovbefore adding the new burst's coverage — this same rule keepscova leaky accumulator, not a lifetime sum: it settles at a bounded steady state (≈ delivery-rate ×T) rather than growing without limit, so it cannot overflow, and it reflects recent coverage. Sincewsaturates forcov ≫ k,covcan be clamped to a fewkin any case. -
Decay the personalisation.
μ_pis itself stale for a long-unused peer, so fade it toward the population rate,μ_p_eff = d·μ_p + (1−d)·populationwithd = exp(−Δt / T_person). The population rate is the right target because the peers you are using keep it current, whereasμ_pfroze when you last touched this one.T_personis longer thanT: a peer's characteristic rate (its provisioning, its distance) is steadier than a momentary reading.
With both, a long-unused peer lands back on the population estimate — it
starts cold and, unused, decays back to cold. This stays O(1) and
lazy: keep one last_update timestamp per peer and apply the decay when
the peer is next read or updated, no timer or sweep. And it is
adversarially clean — reverting to the population average is the neutral
"average until proven" default, so a peer gains nothing by going quiet and
must re-earn a fast estimate by delivering (which the bind measures
honestly); equally, a peer is not condemned forever on stale bad data once
conditions may have moved. This closes the persistent pacer's last refuge:
having warmed its estBurstS over a long paced session, it cannot bank
that warmth by falling silent — the decay walks it back to average, and
re-warming means delivering again, which re-exposes it to T_wait.
A tuning judgment, and a possible extension. T_person need not carry
the personalisation all the way to the population rate. A peer's rate has a
durable part (its provisioning, its distance) and a transient part
(momentary congestion); full decay forgets both, resetting a
well-provisioned nearby peer to the average over an idle hour and forcing
it to re-learn what never changed. Retain a fraction instead — decay μ_p
toward a blend, not fully to the population:
μ_p_eff = d·μ_p + (1−d)·[ρ·μ_p + (1−ρ)·population], d = exp(−Δt / T_person)
so it rests at ρ·μ_p + (1−ρ)·population at long idle. ρ ∈ [0,1] is how
far to decay: ρ = 0 is the full reset above, ρ = 1 disables it.
A fixed ρ > 0 carries an adversarial cost, though: it reopens the
pacer's refuge in proportion. A warmed μ_p now rests at ρ·warm + (1−ρ)·population, so a pacer banks fraction ρ of its gamed warmth across
an idle gap — once its own T_wait record has gone stale — and re-enters
pre-fetch ranking above average; bounded, since T_wait re-catches it on
the next fetch, but real. The fix is to make ρ earned per peer rather
than a global constant: nudge it up a small step on every fetch that
completes on time (additive, to a cap ρ_max < 1) and cut it hard on any
miss — a late or failed delivery, regardless of cause, with no attempt
to tell an accidental slip from a betrayal (multiplicative, e.g. ρ → 0).
This AIMD shape — slow to earn, fast to revoke — makes retention a
reliability privilege: only a peer with a sustained on-time record keeps
its warm prior across a gap, while a pacer or defector never accumulates
ρ (a single miss zeroes it), so its warmth still decays fully to average
and the refuge stays shut for exactly the peers it should. It inherits the
usual reputation-farming shape — build ρ, then spend it on one defection
— but ρ adds only a bounded across-gap nudge to the record farming
already buys, and the hard cut denies a second. Calibrate the step, cap,
and cut like k, leaning conservative: a wrongly-slashed ρ costs only a
fall back to the safe full-decay default, to be re-earned.
The two ways to pace — idle-gap and busy-fill from the bind section —
are the two horns of a single dilemma, and reading them through the
shrinkage weight (now that k, the prior, and what "shrunk to the prior"
means are pinned down above) shows why the window's promptness threshold
G — the max SDU spacing that keeps a burst extending — can be set purely
for honest measurement and fairness (RTT-relative, generous enough to
absorb a fast-retransmit; the window-sizing section), because it does
not govern whether a pacer is caught. A pacer wants a fast estBurstS
(to win selection) while delivering slow (to withhold); delivering slow
needs idle time, and the only lever G gives is where that idle falls
relative to the burst:
-
Busy-fill — idle ≤
G. The gaps stay inside the burst and are counted, soestBurstS= Σgap/Σsize includes them and reads the true, slow rate. Coverage is high, but the peer ranks slow — caught byestBurstS. -
Idle-gap — idle >
G. Each idle terminates the burst, so it is not counted andestBurstS, measured over the fast micro-bursts, reads fast; but the burst is short, its byte-coverage is low, the estimate shrinks to the priorμ_p, and the peer ranks at ~average, not top. Caught by coverage — and, per the bind section, doubly, since the same silence starves the echoes.
The rational pacer takes the second branch (average beats slow) and lands
at ~average — the prior of Setting the coverage scale k — for any
G. Tightening G only forces smaller idles, loosening it forces
bigger ones; the destination, a fast-but-thin estimate shrunk to the
prior, is unchanged. So G decides not whether the pacer is caught but
only which horn catches it, estBurstS-reads-slow or
coverage-shrinks-to-average — the adversary is factored out of the
threshold choice entirely. This is the rigorous footing under the
bind section's two attacks: they are not separately-patched holes but the
two branches of one dilemma that shrinkage closes at once.
One qualification keeps this honest. "Shrinks to the prior" is the
thin-history reading: coverage accumulates (Setting the coverage scale
k), so a persistent idle-gap pacer — one that has delivered enough
fast micro-bursts to build up cov — warms its estimate back toward that
fast within-burst rate and no longer sits at the average. This is not a
second byte-padding, and not a hole the bind misses. Going silent is
exactly what the bind catches: the idle-gaps starve the echoes, and the
delivery is actually slow (the idle time is real), so measured T_wait
and the actual-vs-predicted backstop rank the peer down regardless of what
its estBurstS has warmed to. The staging is the point — coverage-
shrinkage denies a thin pacer the top rank up front, and the bind and
measured completion catch one that has delivered enough volume to escape
the shrinkage. Byte-padding stays the one escape from the bind proper
precisely because only it keeps echoes flowing, on the padding, while
withholding the closure we asked for — a persistent idle-gap pacer that
filled its silences to do the same would simply be byte-padding.
Two consequences close the loop. First, because G buys no anti-pacing, a
tight G is pure downside: an honest far peer's ~1-RTT loss-recovery
gaps exceed it, fragmenting its transfer into exactly the short,
low-coverage, shrunk-to-average bursts a pacer produces — a tight gate
makes an honest far peer indistinguishable from a pacer. Sizing G
RTT-relative removes that collision while conceding nothing to the
adversary. Second, ~average is not zero risk — a pacer shrunk to the prior
can still win in a weak or eclipsed pack — but that residual is the
eclipse condition that weakens every defence, is itself G-independent,
and is closed after the fact by the backstop convicting a selected pacer
on actual ≫ predicted.