Skip to content

fix(refiller): don't fail the USDG sweep when Paxos rejects it as below minimum - #3729

Open
droplet-rl wants to merge 1 commit into
masterfrom
droplet/T90K0AL22-C098EUCH7UJ-1786788196-251479
Open

fix(refiller): don't fail the USDG sweep when Paxos rejects it as below minimum#3729
droplet-rl wants to merge 1 commit into
masterfrom
droplet/T90K0AL22-C098EUCH7UJ-1786788196-251479

Conversation

@droplet-rl

Copy link
Copy Markdown
Contributor

Why

zion-across-usdg-refiller has been failing every run since 2026-08-15T09:55Z, paging through the serverless hub (Some spoke calls returned errors, errorOutputs: zion-across-usdg-refiller).

The trigger was benign: the refiller's mainnet USDG balance (137.911345) crossed its MIN_USDG_SWEEP_AMOUNT of 100 for the first time, so sweepMainnetUsdgToRobinhood started actually calling Paxos. Paxos rejects the order quote with HTTP 400:

Order amount is below the minimum of $160.93 (160924200 base units). Please increase your offer amount.

Their per-order minimum floats with execution costs and currently sits well above both MIN_USDG_SWEEP_AMOUNT and the $5 floor hardcoded in PAXOS_TRANSIT_MINIMUMS. The error propagated out of runRefiller, so a "wait for the balance to grow" condition was reported as a bot failure — and repeated every 5 minutes. Verified against the live API: the same request at 200 USDG returns 200 with a valid quote, so the route, key and endpoint are healthy.

Two things also made it hard to diagnose: the message never reached the logs (HttpError: [object Object]) because the SDK's baseFetch only lifts a string error key off the body, and Paxos returns { error: { code, message, status } }; and the client retried the rejected request twice for the same answer.

What

  • Refiller.sweepMainnetUsdgToRobinhood treats a below-minimum rejection as a skip and logs the live minimum. Every other error still throws.
  • PaxosTransitClient throws a typed PaxosTransitApiError (extends the SDK's HttpError, adds apiStatus) so the message and status survive into the logs. Modelled on AcrossApiHttpError in AcrossApiBaseClient.ts, which exists for the same reason.
  • getWithRetry only retries transient failures (transport errors, 408/425/429, 5xx).
  • src/refiller/README.md: the "$5 minimum per order" line was misleading — that's the documented floor, not the live value.

There is no machine-readable discriminator for the below-minimum case in Paxos's response, so classification matches on the message; if that wording changes the failure mode is the current behavior (a throw), not a silent skip.

Testing

test/PaxosTransitClient.ts covers the error shape, the retry policy, and below-minimum classification against the verbatim body Paxos returned. yarn typecheck, yarn lint, test/PaxosTransitClient.ts, test/generic-adapters/PaxosTransitBridge.ts and test/Refiller.ts pass (21 passing).

Note for operators

This stops the paging but does not itself move the funds: the 137.91 USDG sweeps on the first run after the balance clears Paxos's minimum. Raising MIN_USDG_SWEEP_AMOUNT above it in bot-configs would avoid the doomed API calls in the meantime, but is not required.

🤖 Generated with Claude Code

…ow minimum

The mainnet USDG sweep started failing every run at 2026-08-15T09:55Z, when the
refiller's balance (137.911345 USDG) first crossed MIN_USDG_SWEEP_AMOUNT. Paxos
rejected the order quote with HTTP 400 "Order amount is below the minimum of
$160.93 (160924200 base units)": their per-order minimum floats with execution
costs and currently sits well above both MIN_USDG_SWEEP_AMOUNT and the $5 floor
in PAXOS_TRANSIT_MINIMUMS. That threw out of runRefiller, so the serverless hub
reported the spoke as rejected and paged on a condition that resolves itself
once the balance grows.

- Treat a below-minimum rejection in sweepMainnetUsdgToRobinhood as a skip, with
  the live minimum logged; anything else still throws.
- Throw a typed PaxosTransitApiError from PaxosTransitClient so Paxos's
  { error: { message, status } } body survives. The SDK's fetch helpers only
  lift a *string* `error` key off the body, so the message was reaching the logs
  as "HttpError: [object Object]" and the status wasn't logged at all.
- Only retry transient failures. A rejected request was being replayed twice for
  the same answer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant