|
| 1 | +# ADR-0004: Defer hibernation + sandbox-router; PR #1 keeps path-based routing and `operatingMode: Running` |
| 2 | + |
| 3 | +- **Status:** Proposed |
| 4 | +- **Date:** 2026-06-18 |
| 5 | +- **Deciders:** HC Lo (hclo) |
| 6 | +- **Depends on:** [ADR-0001](0001-adopt-agent-sandbox-model.md), [ADR-0003](0003-per-tenant-sandboxtemplate.md) |
| 7 | + |
| 8 | +## Context |
| 9 | + |
| 10 | +Today scale-to-zero is provided by the **KEDA HTTP add-on**: a per-tenant |
| 11 | +`HTTPScaledObject` targets the tenant `Deployment`, scales it 0↔1 on request |
| 12 | +rate (idle 900s), and its interceptor holds an inbound request while a |
| 13 | +scaled-to-zero tenant starts. Routing is **path-based**: |
| 14 | +`claw.example.com/t/<tenant>/` via a Gateway API `HTTPRoute`. |
| 15 | + |
| 16 | +Adopting the Sandbox model (ADR-0001) removes the `Deployment`. A `Sandbox` is a |
| 17 | +singleton with no `/scale` subresource, so KEDA HTTP can no longer target it. |
| 18 | +The agent-sandbox model offers its own scale-to-zero via **hibernation** |
| 19 | +(pause on idle, "automatic resume on incoming network connections") and a |
| 20 | +**sandbox-router** for scalable access. We researched both before committing. |
| 21 | + |
| 22 | +Verified findings that change the picture: |
| 23 | + |
| 24 | +1. **sandbox-router is header-based, not path-based.** It routes by an |
| 25 | + `X-Sandbox-ID` (+ namespace/port) HTTP header to |
| 26 | + `<id>.<ns>.svc.cluster.local`. Our contract is path-based (`/t/<tenant>/`), |
| 27 | + sent by browsers/auth-ui. Stock Gateway API cannot easily translate a |
| 28 | + dynamic path segment into a header, so adopting the router changes the |
| 29 | + routing/UX contract — it is not a drop-in addition. |
| 30 | +2. **The router does not itself wake a hibernated sandbox.** Its own tests |
| 31 | + assert that an unreachable sandbox returns **502 Bad Gateway**. So the router |
| 32 | + is a reverse proxy, not an activator. |
| 33 | +3. **The wake/hibernation mechanism is unverified.** We could not find a doc |
| 34 | + that states whether hibernation *pauses* the pod (endpoint stays, connection |
| 35 | + can trigger resume) or *deletes* it (then who triggers resume?). The |
| 36 | + controller flags we found are concurrency-only; there is **no idle-timeout |
| 37 | + knob**, and `shutdownTime` is a delete-TTL, not idle-hibernate. The cost |
| 38 | + story (scale-to-zero) hinges entirely on this unknown. |
| 39 | + |
| 40 | +## Decision |
| 41 | + |
| 42 | +Split the scale-to-zero redesign out of PR #1. |
| 43 | + |
| 44 | +**PR #1** runs each tenant Sandbox with **`operatingMode: Running`** (always-on, |
| 45 | +no hibernation). Routing stays **path-based**: the per-tenant `HTTPRoute` |
| 46 | +backend is repointed from the old Deployment Service to the **Sandbox's |
| 47 | +controller-created headless Service**. The KEDA `HTTPScaledObject` is **removed** |
| 48 | +in PR #1 (it targeted a Deployment that no longer exists and cannot target a |
| 49 | +Sandbox); we accept an always-on interim cost posture. |
| 50 | + |
| 51 | +**The sandbox-router, hibernation, and the scale-to-zero replacement are |
| 52 | +deferred to PR #1.5**, gated on first resolving (a) the wake mechanism |
| 53 | +(pause-vs-delete and what triggers resume — to be answered by reading the |
| 54 | +controller source or by empirical test on the verification cluster) and (b) the |
| 55 | +path→header routing strategy. Each will be captured in its own ADR. |
| 56 | + |
| 57 | +## Options considered |
| 58 | + |
| 59 | +- **A. PR #1 keeps path routing + `Running`; defer router/hibernation (chosen):** |
| 60 | + isolates the verified, low-risk lifecycle adoption from the unresolved |
| 61 | + scale-to-zero redesign; PR #1 is deploy-verifiable now. |
| 62 | +- **B. Adopt sandbox-router + hibernation in PR #1:** rejected — wake mechanism |
| 63 | + unverified, router is header-based vs our path-based, router returns 502 not |
| 64 | + wake; high risk of shipping a broken scale-to-zero. |
| 65 | +- **C. Drop scale-to-zero permanently:** rejected — the cost story is a stated |
| 66 | + value of the sample. We defer, we do not abandon it. |
| 67 | + |
| 68 | +## Consequences |
| 69 | + |
| 70 | +**Positive** |
| 71 | +- PR #1 is shippable and deploy-verifiable without the unresolved hibernation |
| 72 | + question. |
| 73 | +- Routing and UX contract are unchanged for PR #1. |
| 74 | + |
| 75 | +**Negative / interim** |
| 76 | +- Tenants run **always-on** in PR #1 → higher idle cost until PR #1.5. If useful, |
| 77 | + a `shutdownTime`/TTL can still bound truly-abandoned workspaces. |
| 78 | +- The original framing "hibernation replaces KEDA scale-to-zero" is **not yet |
| 79 | + proven**; PR #1.5 must validate it end-to-end before we claim the cost benefit. |
| 80 | + |
| 81 | +## Resolution (2026-06-18, verified from controller source @ tag v0.4.5) |
| 82 | + |
| 83 | +The pause-vs-delete question is settled by reading |
| 84 | +`controllers/sandbox_controller.go` at the v0.4.5 tag: |
| 85 | + |
| 86 | +- v0.4.5 (`v1alpha1`) uses `Sandbox.Spec.Replicas` (0/1); there is **no** |
| 87 | + `operatingMode: Suspended` (that is a later `v1beta1` field). |
| 88 | +- When `Replicas == 0` the controller **deletes the pod** |
| 89 | + (`"Deleting Pod because .Spec.Replicas is 0"` → `r.Delete(ctx, pod)`), so the |
| 90 | + Service endpoint disappears. **This is Case B (delete), not pause.** |
| 91 | +- There is **no connection-triggered resume** anywhere in the v0.4.5 core or |
| 92 | + SandboxClaim controllers. The "automatic resume on incoming network |
| 93 | + connections" described in the current docs is a later (`v1beta1`) capability, |
| 94 | + not present in v0.4.5. |
| 95 | + |
| 96 | +**Consequence for KEDA:** on v0.4.5, the Sandbox model does **not** replace |
| 97 | +KEDA's scale-to-zero at all — it provides isolation + lifecycle + identity only. |
| 98 | +Scale-to-zero with wake still requires both an idle scaler and an activator |
| 99 | +(KEDA's two roles). The sandbox-router does not wake (returns 502). |
| 100 | + |
| 101 | +**Therefore PR #1 ships always-on (`Replicas=1`, KEDA removed), and the |
| 102 | +scale-to-zero path forks into three options for a later decision:** |
| 103 | + |
| 104 | +1. **No scale-to-zero** — accept always-on (simplest). |
| 105 | +2. **Custom activator on v0.4.5** — a path-native component that patches |
| 106 | + `Sandbox.Spec.Replicas` 0↔1 on idle/request (we own and maintain it). |
| 107 | +3. **Migrate to v1beta1** — gain native hibernation + wake-on-connection, but |
| 108 | + the claim schema changes to `warmPoolRef` (re-do ADR-0002/0003); larger |
| 109 | + migration. |
| 110 | + |
| 111 | +This decision is deferred to its own ADR; it does not block PR #1. |
| 112 | + |
| 113 | +## References |
| 114 | + |
| 115 | +- Sandbox Router README: `https://github.com/kubernetes-sigs/agent-sandbox/blob/main/clients/python/agentic-sandbox-client/sandbox-router/README.md` |
| 116 | +- Controller configuration flags: `https://github.com/kubernetes-sigs/agent-sandbox/blob/main/docs/configuration.md` |
| 117 | +- [docs/agent-sandbox.md](../agent-sandbox.md) — original KEDA→hibernation framing (revised by this ADR) |
0 commit comments