Skip to content

Versionless devshard stats endpoints#1453

Open
a-kuprin wants to merge 2 commits into
ak/pixelplex-refactoring-into-r2from
ak/unversioned-stats-endpoints
Open

Versionless devshard stats endpoints#1453
a-kuprin wants to merge 2 commits into
ak/pixelplex-refactoring-into-r2from
ak/unversioned-stats-endpoints

Conversation

@a-kuprin

Copy link
Copy Markdown
Collaborator

Summary

  • Stop accidental protocol-version bind from public observability GETs: only the escrow owner can bind, via signed POST …/chat/completions.
  • Make observability versionless end-to-end (canonical /devshard/sessions|stats|metrics), with join-proxy internal rewrites so legacy /devshard/{version}/… scrapers keep working unchanged.
  • Route versionless session obs in versiond by Postgres sessions.version when available (fan-out fallback for SQLite / PG errors).
  • Harden polling / ops: O(1) stats detail + negative cache, nginx devshard_obs rate limits, numeric primary-version sort, visible PG-lookup fan-out errors, clarified /healthz vs /metrics.

Motivation

Escrow create is version-agnostic. Version was bound on first CreateSession, and observability GETs (diffs / mempool / signatures / stats detail) went through getOrCreate. Dashboards that hardcode /devshard/v2/sessions/{id}/diffs could lock an escrow to v2 before the creator’s first v3 chat, after which the creator hits ErrSessionVersionConflict.

Design notes

Stop the bleed (devshardd). Split resolve paths:

Path Resolver
Obs GETs, gossip, challenge, verify-timeout, payloads, stats detail SessionServerExisting — recover only, never CreateSession
Owner chat BindOwnerChat — verify owner signature before bind; reuse fetched escrow for create (single GetEscrow)

Unbound obs / host protocol → 404. Chat-only bind.

Join proxy. Internal rewrite … last (not public 308) from versioned obs URLs onto versionless paths. Protocol POSTs and payloads stay versioned. Public obs uses devshard_obs rate zone (DEVSHARD_OBS_RATE_LIMIT_RPS default 10, burst 20); chat/gossip/payloads stay exempt.

versiond + paths. Versionless obs without a version prefix. Session-scoped / stats-detail: PG lookup of sessions.version → correct child; unbound → 404; no PG → fan-out. Process-level /metrics and /stats/shards list pin to newest child by numeric/dotted compare (v10 > v2, v0.2.11 > v0.2.9). PG lookup errors fan out with a rate-limited warn + LookupFanoutErrors counter (not silent).

Health vs metrics. /devshard/healthz is versiond supervisor (mux). /devshard/{version}/healthz reaches that child (not rewritten). /devshard/metrics pins newest child.

Stats detail. GET /stats/shards/{id} uses O(1) GetSessionMeta + short negative cache (~10s, cap 4096); warn only on cache insert.

Dashboard  GET /devshard/v2/sessions/42/diffs
        → join proxy rewrite → /devshard/sessions/42/diffs
        → versiond (PG lookup or fan-out) → bound child
        → SessionServerExisting (no CreateSession)

Creator   POST /devshard/v3/sessions/42/chat/completions (owner sig)
        → BindOwnerChat → CreateSession(Version=v3)

Test plan

  • go test ./server/ ./cmd/devshardd/session/ ./transport/ — unbound obs/gossip no bind; owner chat binds; non-owner does not; first bind single GetEscrow
  • go test ./internal/proxy/ (versioned) — versionless routing, numeric primary, lookup, fan-out + lookup-error counter
  • go test . -run Versionless (devshard paths helpers)
  • Stats detail O(1) + negative cache (no list scan; repeat miss skips store)
  • Manual / join stack: legacy versioned obs URL returns 200 with no Location; unbound → 404; owner chat then binds chosen version
  • HA + PG: versionless diffs routes to the bound version child
  • Obs GETs hit devshard_obs rate limit; chat stays exempt
  • /devshard/healthz = versiond; /devshard/{v}/healthz = child

Config / ops

Env Where Role
PGHOST / DATABASE_URL versiond Enable bound-version lookup for versionless session obs
VERSIOND_DISABLE_SESSION_LOOKUP versiond Force fan-out even when PG is configured
DEVSHARD_OBS_RATE_LIMIT_RPS join proxy Per-IP obs GET limit (default 10/s)
DEVSHARD_OBS_BURST join proxy Obs burst (default 20)

Preferred monitor URLs: /devshard/sessions/{id}/diffs, /devshard/stats/shards/{id} (legacy versioned diffs/stats/metrics still work via rewrite). Child health: /devshard/{version}/healthz.

akup added 2 commits July 14, 2026 21:33
Public GET diffs/stats no longer CreateSession; only owner-signed chat
binds. Join proxy rewrites legacy versioned obs URLs internally;
versiond serves versionless paths via PG sessions.version (fan-out
fallback).
O(1) stats detail + negative cache, nginx obs rate limits, numeric
primary version sort, single GetEscrow on owner bind, healthz not
rewritten to versiond, and visible PG lookup fan-out errors.
@a-kuprin a-kuprin added this to the v0.2.14-devshard4 milestone Jul 14, 2026

@x0152 x0152 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non-blocking: lookupBoundVersionActiveSession appears to be intended to return only active sessions, but it currently does not

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants