Add DNS-seeder prober with its own deploy pipeline - #18
Draft
andres-pcg wants to merge 4 commits into
Draft
Conversation
The 2026-05-28 near-stall came from seeders handing out dead peer IPs while DNS resolved fine -- invisible to a recursive DNS check. The prober enumerates each seeder's authoritative nameservers, queries each one directly, and handshakes the returned IPs (Zcash version/verack) to confirm they are live nodes, flagging a nameserver that diverges from its siblings. It hardcodes no IPs (it checks the live seeder output, so it can't drift), runs as a small always-on Cloud Run service exposing /status for Uptime Kuma, and ships a dedicated pipeline kept separate from Kuma (no DB/secrets/VPC). Includes the root-cause + hardening doc.
Prod counterpart of cd-deploy-prober-dev.yml: builds and deploys the always-on prober to Cloud Run on a published release, using the prod environment vars.
- Dockerfile: declare USER nonroot:nonroot explicitly (checkov CKV_DOCKER_3) and skip CKV_DOCKER_2 — Cloud Run ignores Docker HEALTHCHECK and the service is health-checked via /livez + Kuma polling /status (distroless has no shell). - ci-lint-codebase: disable super-linter's Go validators (it runs golangci-lint from the repo root, but the module is in prober/, so it can't resolve imports and reports false positives) and add a dedicated Go job that runs gofmt, vet and tests with the module context. - README: 'repo' -> 'repository' (terminology lint).
andres-pcg
marked this pull request as draft
June 25, 2026 03:36
…ndshakes Pinned to v0.4.0 the prober advertised 170140 (NU6.1) while the fleet serves at 170150 (NU6.2), so every peer refused it: 0 live on all six nameservers with TCP to :8233 fine. It reported that as INCONCLUSIVE and /healthz stayed 200, which is the blind spot the tool exists to remove. Adds DOWN(probe) for records resolving run-wide with no peer handshaking anywhere. Judged run-wide because probing one target faster than its cooldown can zero that target on its own, which is expected and stays soft.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The DNS-seeder prober — the detection layer for the 2026-05-28 incident class
(seeders handing out dead peer IPs while DNS resolves fine) — plus its own
dedicated deploy pipeline.
Why
A recursive DNS check only sees "did it resolve?", which is exactly why the old
monitoring stayed green during the near-stall. The prober instead:
ns1..nsN) by iterativeresolution from the root,
nodes, and
the node-side rate-limiting that depresses everything uniformly).
It hardcodes no IPs — it checks the live seeder output, so it can't drift the
way the Corefile bootstrap list does.
What's here
prober/— Go tool. One-shot CLI and--servemode exposing/status(200 / 503),
/results(JSON),/(table). Reuses dnsseeder's own handshake(pinned, with the
btcdZcash-fork replace). Unit-tested. Bruno collection forlocal poking under
prober/bruno/.prober/Dockerfile— distroless, non-root..github/workflows/cd-deploy-prober-dev.yml— dedicated pipeline (build onthe runner via buildx → deploy to Cloud Run). Kept separate from the Kuma
deploy because the prober has no DB / secrets / Cloud SQL / VPC.
docs/dns-seeder-resilience.md— root-cause analysis + upstream hardening recs.Deployment shape
Always-on Cloud Run:
--min-instances=1 --no-cpu-throttling(background probeloop),
--execution-environment=gen2(UDP egress for the iterative DNS),outbound-only. Uptime Kuma watches
/status(/healthzis reserved by GoogleFront End on
*.run.app).Prerequisites for the pipeline (TF, separate)
The pipeline reuses the existing WIF + SAs: build impersonates
artifact-publisher(has
artifactregistry.writer), deploy impersonatesinstance-deployer(hasrun.admin+serviceAccountUser). No Cloud Build needed. Nothing new requiredbeyond what's already granted.
Status
Already deployed to dev manually and verified (Kuma polls
/statusevery 60s,200). This PR adds the source of truth + automation so it deploys on change.