fix(refiller): don't fail the USDG sweep when Paxos rejects it as below minimum - #3729
Open
droplet-rl wants to merge 1 commit into
Open
fix(refiller): don't fail the USDG sweep when Paxos rejects it as below minimum#3729droplet-rl wants to merge 1 commit into
droplet-rl wants to merge 1 commit into
Conversation
…ow minimum
The mainnet USDG sweep started failing every run at 2026-08-15T09:55Z, when the
refiller's balance (137.911345 USDG) first crossed MIN_USDG_SWEEP_AMOUNT. Paxos
rejected the order quote with HTTP 400 "Order amount is below the minimum of
$160.93 (160924200 base units)": their per-order minimum floats with execution
costs and currently sits well above both MIN_USDG_SWEEP_AMOUNT and the $5 floor
in PAXOS_TRANSIT_MINIMUMS. That threw out of runRefiller, so the serverless hub
reported the spoke as rejected and paged on a condition that resolves itself
once the balance grows.
- Treat a below-minimum rejection in sweepMainnetUsdgToRobinhood as a skip, with
the live minimum logged; anything else still throws.
- Throw a typed PaxosTransitApiError from PaxosTransitClient so Paxos's
{ error: { message, status } } body survives. The SDK's fetch helpers only
lift a *string* `error` key off the body, so the message was reaching the logs
as "HttpError: [object Object]" and the status wasn't logged at all.
- Only retry transient failures. A rejected request was being replayed twice for
the same answer.
droplet-rl
requested review from
bmzig,
dijanin-brat,
mrice32,
nicholaspai and
pxrl
as code owners
August 15, 2026 10:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
zion-across-usdg-refillerhas been failing every run since 2026-08-15T09:55Z, paging through the serverless hub (Some spoke calls returned errors,errorOutputs: zion-across-usdg-refiller).The trigger was benign: the refiller's mainnet USDG balance (137.911345) crossed its
MIN_USDG_SWEEP_AMOUNTof 100 for the first time, sosweepMainnetUsdgToRobinhoodstarted actually calling Paxos. Paxos rejects the order quote with HTTP 400:Their per-order minimum floats with execution costs and currently sits well above both
MIN_USDG_SWEEP_AMOUNTand the $5 floor hardcoded inPAXOS_TRANSIT_MINIMUMS. The error propagated out ofrunRefiller, so a "wait for the balance to grow" condition was reported as a bot failure — and repeated every 5 minutes. Verified against the live API: the same request at 200 USDG returns200with a valid quote, so the route, key and endpoint are healthy.Two things also made it hard to diagnose: the message never reached the logs (
HttpError: [object Object]) because the SDK'sbaseFetchonly lifts a stringerrorkey off the body, and Paxos returns{ error: { code, message, status } }; and the client retried the rejected request twice for the same answer.What
Refiller.sweepMainnetUsdgToRobinhoodtreats a below-minimum rejection as a skip and logs the live minimum. Every other error still throws.PaxosTransitClientthrows a typedPaxosTransitApiError(extends the SDK'sHttpError, addsapiStatus) so the message and status survive into the logs. Modelled onAcrossApiHttpErrorinAcrossApiBaseClient.ts, which exists for the same reason.getWithRetryonly retries transient failures (transport errors, 408/425/429, 5xx).src/refiller/README.md: the "$5 minimum per order" line was misleading — that's the documented floor, not the live value.There is no machine-readable discriminator for the below-minimum case in Paxos's response, so classification matches on the message; if that wording changes the failure mode is the current behavior (a throw), not a silent skip.
Testing
test/PaxosTransitClient.tscovers the error shape, the retry policy, and below-minimum classification against the verbatim body Paxos returned.yarn typecheck,yarn lint,test/PaxosTransitClient.ts,test/generic-adapters/PaxosTransitBridge.tsandtest/Refiller.tspass (21 passing).Note for operators
This stops the paging but does not itself move the funds: the 137.91 USDG sweeps on the first run after the balance clears Paxos's minimum. Raising
MIN_USDG_SWEEP_AMOUNTabove it inbot-configswould avoid the doomed API calls in the meantime, but is not required.🤖 Generated with Claude Code