gateway+devshard Always-stream upstream + shared scratch spool + citest

[a-kuprin]

Merge order

Merge after:

1. #1574 — same-nonce reconnect + host ML drain + LiveStream spool
2. #1575 — streamed hop timestamps (: devshard-ts)

Those PRs land through e708fb3. This branch continues from that tip with always-stream forcing, shared devshard/spool, and force-upstream citest.

---

Motivation

Non-streaming clients today wait for a single opaque JSON body. That hides the signals we need for smarter routing and host quality:

• TTFT / first-token — no progressive chunks → no first-token timer that matches reality
• Inter-chunk liveness — a stuck host looks the same as a slow one until the whole response arrives
• Sustained stream speed (bytes/sec) — soft signal for Decide / quarantine later; useless if the wire is one blocking read

Always-stream flips the upstream shape to SSE for every chat request (when the admin flag is on), while the client still gets the shape it asked for: SSE for stream:true, one aggregated chat.completion for stream:false. Routing and soft signals can observe every request the same way; silent non-stream waits stop being a blind spot.

Same-nonce reconnect (#1574) and hop timestamps (#1575) already assume a streamed host path. This PR finishes the gateway half: force upstream stream, fold for JSON clients, keep client intent honest across double-normalization, and bound aggregate memory with a shared scratch spool.

Default remains off (ForceUpstreamStreaming false) until Step 14 soak; this PR lands the machinery and e2e that gate flipping it on.

---

Always-stream flow

sequenceDiagram
  participant C as Client
  participant G as Gateway
  participant H as Host
  participant M as ML

  C->>G: chat.completions stream true or false
  Note over G: Snapshot client stream usage and logprobs intent
  Note over G: ForceUpstreamStreaming wires stream true include_usage and forced logprobs
  G->>H: HostRequest plus nonce
  H->>M: POST streamed completion
  M-->>H: SSE chunks
  H-->>G: SSE plus receipt plus meta
  alt Client wants SSE
    Note over G: Forward chunks and strip forced fields client did not ask for
    G-->>C: text event-stream until DONE
  else Client wants JSON
    Note over G: Accumulate SSE in RAM then spool and fold to chat.completion
    G-->>C: application json chat.completion
  end

Loading

Client-visible contract (flag on or off for shape; flag on for upstream force):

Client ask	Upstream (flag on)	Client response
stream: false	Always SSE	One application/json chat.completion (aggregator)
stream: true	Always SSE	text/event-stream; no trailing usage unless include_usage
logprobs omitted	Forced on wire for validation	Stripped at client boundary
logprobs + any top_logprobs > 0	Forced top_logprobs: 5	Keep forced width 5 (not truncated to client N)
logprobs without top_logprobs	Forced tops upstream	top_logprobs emptied in client response

Intent handoff: gateway normalizes once for auth/limits, then the runtime proxy normalizes again. Client stream / usage / logprobs intent is pinned in request context so the second normalize cannot treat forced wire fields as the client’s ask (R1). Mid-flight admin flips of ForceUpstreamStreaming do not change an in-flight request’s snapshot (F3).

Escalation (flag on): every attempt is a streamed attempt — first-token / attempt_failed apply even for stream:false clients; reduced-max_tokens timers stay for the legacy path only.

Docs: gateway-always-stream-upstream-plan.md, gateway-streaming-ha-overview.md, citest streaming-ha-scenarios.md.

---

Shared scratch spool (devshard/spool)

Gateway aggregation (fold a full SSE body for JSON clients) and host LiveStream resume (append-only log for reconnect) both need scratch disk: RAM up to a threshold, then a temp file, then read-back, then delete. Access patterns are opposites (write-once / read-once vs concurrent ReadAt while appending), so they do not share one buffer type — they share a substrate.

Design: spool-shared-library.md.

Shared (devshard/spool)	Stays local
Dir — open / probe / prefix sweep / 0o700 / unlink-at-create	SSE fold, NDJSON logprobs framing (devshardctl)
File — buffered or unbuffered write, ReadableLen, ReadAt	LiveStream ring, trim, cursor, hop stamps (host)
Buffer — mem-first spill, DegradeToRAM / FailRequest	Durable payloads (common/storage/payloads)
Index — event → byte offset sidecar
Budget / Slots — byte ceilings and concurrency caps

Call sites after migration:

• Gateway body: aggregateResponseBuffer → spool.Buffer + process-wide Slots
• Gateway logprobs: Dir.Create + request foldBudget
• Host: streamSpool → spool.File + Dir.CreateIndex, env caps (DEVSHARDD_LIVESTREAM_*)

Scratch is anonymous by default (no plaintext ls), never RemoveAll of a configurable tree, and a CI guard fails if gateway/host recreate scratch outside spool. Optional later: promote to common/spool when a second module needs it.

---

Commits after e708fb3

1. 944106adb — Force upstream streaming and aggregate non-stream clients safely

Lands the always-stream gateway path behind ForceUpstreamStreaming (default false):

• PostLimits rules force stream / stream_options.include_usage (and existing forced logprobs) when the flag is on
• streamClientIntent + context handoff so proxy branching / cache keys / usage strip use the client ask
• aggregateSSEStream + handleAggregated: fold winner SSE into one chat.completion; RAM/disk aggregate buffer with degrade / typed oversize errors
• Suppress forced usage chunks for streaming clients that did not ask include_usage
• Escalation unification for force-on non-stream clients (first-token, not reduced-max_tokens)
• Unit coverage across intent handoff, aggregate, usage suppress, force-upstream stream/escalation

2. d7ae13a43 — Extract shared scratch spool for gateway aggregate and host LiveStream

Introduces devshard/spool and migrates both consumers onto it:

• New package: Dir / File / Buffer / Index / Budget / Slots + CI guard
• Gateway aggregate + logprobs spill use the shared Dir/Buffer APIs
• Host LiveStream spool uses File + Index with prefix sweep (no RemoveAll) and optional env caps
• Design doc: spool-shared-library.md; overview cross-links

3. 4c041ca20 — Add force-upstream streaming citest and document streaming HA scenarios

Closes the unit-only gap for gateway → proxy handoff and documents how to run suites:

• Citest: client shape, usage suppression, logprob strip, differential aggregate, cache isolation, mid-flight flag flip, aggregate spill, oversize abort
• make citest-force-upstream-streaming; mock-openai logprobs + max_tokens padding for spill
• Scenario docs: testenv/docs/streaming-ha-scenarios.md, attempt-reconnect-scenarios.md
• Plans/overview updated; top_logprobs contract documented as keep forced width 5

---

Test plan

• go test under devshard: ./cmd/devshardctl/ ./host/ ./spool/ ./transport/ ./user/ (and common/completionapi usage-strip)
• make -C devshard/testenv citest-force-upstream-streaming
• make -C devshard/testenv citest-attempt-reconnect (still green on top of this stack)
• Manual: admin force_upstream_streaming: true; stream:false → JSON; stream:true → SSE without forced usage; flip flag mid-flight keeps in-flight shape
• Confirm default-off: without the admin flag, client shapes and legacy non-stream escalation unchanged

---

Deferred / out of scope

• Step 14 default-on soak and dashboards
• Cross-instance ML reattach (#1466 §4)
• Soft-signal persist / full Prometheus reconnect suite (gated on postgres + observability e2e)
• common/spool promotion (optional Phase 4)

[a-kuprin]

Always stream feature is cleaner and is more strictly separated at #1581

spool and local fold will be cherry-picked from this branch after #1581 lands