feat(airbender): draft from-scratch prover autoscaler - #4919
Draft
deniallugo-ml wants to merge 1 commit into
Draft
feat(airbender): draft from-scratch prover autoscaler#4919deniallugo-ml wants to merge 1 commit into
deniallugo-ml wants to merge 1 commit into
Conversation
Standalone crate (own workspace, no zksync deps) that scales Airbender prover deployments: polls per-chain /airbender/queue_report endpoints, plans replica counts with a pure drain-horizon planner (confirmation streaks, scale-down stabilization, pending-pods-as-requested-capacity, starvation hold, snark-only pre-warm, group GPU budget), and patches Deployment scale directly via kube-rs — no per-cluster agents, single GPU card class, manual combined/split topology in config. Ships with 10 planner unit tests, dry-run and --once modes, an example config, and a README listing the external prerequisites (queue-report endpoint, prover-server SNARK lease / idle fallback / SIGTERM drain). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
deniallugo-ml
marked this pull request as draft
July 23, 2026 16:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What ❔
A new standalone crate
airbender_autoscaler/— a from-scratch autoscaler for Airbender prover fleets. Own cargo workspace at the repo root (likeairbender_prover_server), nozksync_*dependencies: it talks to chain proof data handlers over HTTP and to Kubernetes API servers directly via kube-rs. The legacyprover_autoscaleris untouched and keeps serving the old proof system.Design
planner.rs): all policy is a side-effect-free function, unit-tested (10 tests); dry-run runs identical code and skips only the actuator. Drain-horizon control law:desired = clamp(ceil(work × duration_p95 / drain_window), min, max), then a hard per-groupmax_gpusbudget (snark-only funded first, fri-only cut first).spec.replicascounts as capacity already requested (Pending pods are never re-requested); a pod Pending beyondstartup_p95marks the cluster starved and the planner holds and says so — no pool racing, no aggressive mode. Scale-up needs consecutive confirmations (bypassed on latency-SLO breach); scale-down is one replica at a time after a stabilization window, only on a settled fleet.fri-snark) or split (fri-only+snark-only) per compatibility group; multichain workers (feat(airbender): round-robin prover fetching across multiple chain job servers #4905) poll all the group's chains round-robin. Config validation rejectsfri-onlywithout SNARK-capable capacity.snark-onlypre-warms from FRI jobs expected to finish within machine startup time.Prerequisites tracked in the README (not in this PR)
GET /airbender/queue_reporton the proof data handler, with predicates mirroring the DAL locking queries (lock_batch_for_proving/lock_batch_for_snark).submit_fri+will_wrap_locally→picked_for_snark, closing the duplicate-wrap race withsnark-onlyclaimers), idle SNARK fallback infri-snarkmode, SIGTERM drain that also submits the local SNARK follow-up.Why ❔
The existing
prover_autoscalertargets the previous proof system (agents, GPU-type ranking, aggressive pool racing, prover-job-monitor). The Airbender setup is different: few machines of a single GPU card class, FRI ~10–15 min + SNARK wrap, shared multi-chain fleets, and a hard cost objective. A small purpose-built controller is simpler than retrofitting the legacy one.Checklist
zk_supervisor fmtandzk_supervisor lint.🤖 Generated with Claude Code