What happens
Any redeploy of api-rs (a values change, an image bump) kills every running agent turn at once with sandbox stdout closed before terminal output; stdout reattach attempts exhausted. Observed on a single-node production deployment at 46bebdf: a one-line config merge destroyed three live turns at 36, 20, and 19 minutes elapsed; two more earlier the same day at 73 and 34 minutes; and a review turn killed mid-flight could not be retried at all afterward, because the idempotency key is stable per commit and re-requests return the dead execution with created=false.
Why it happens
The architecture already carries the right idea and the default deployment shape defeats it. session_executions has stdout_owner_id and stdout_owner_lease_expires_at, renewing on a ~60s TTL — so a replacement instance is clearly expected to claim the lease and reattach to a still-running sandbox. It cannot, because:
replicas: 1 strategy: RollingUpdate maxSurge 25% maxUnavailable 25% grace 35s
With one replica there is no live instance to hand the lease to. The old pod's exec streams close on SIGTERM; the sandbox keeps producing output with no reader; a Kubernetes exec stream does not buffer for an absent reader, so those bytes are gone. The new pod's reattach finds a stream missing its terminal output, retries, and exhausts. The sandbox itself is healthy throughout — only the reader and the un-replayable bytes are lost.
Open questions that decide the fix
- Is api-rs safe multi-instance (workflow runtime, sandbox reaper, warm-pool management), or do those assume a singleton? The lease schema implies HA was designed for; the default chart shape never exercises it.
- Can sandbox output survive a reader gap at all?
session_events already persists a complete turn transcript, so the bytes exist somewhere durable — if that path is independent of the stdout pipe, reattach could reconcile from it instead of requiring an unbroken stream.
Proposed direction
Any of: HA replicas with verified lease handoff; durable/replayable sandbox output (spool in harness-server, or reconcile reattach from the persisted event stream); or drain-then-handoff on SIGTERM. Failing all three, an interim mitigation: let a turn killed by a roll be retried (terminal failure should release the idempotency key) — losing the work is bad, losing it irrecoverably is worse.
Related: #1432 (api-rs restart also loses in-process idle timers — same restart, different loss).
What happens
Any redeploy of api-rs (a values change, an image bump) kills every running agent turn at once with
sandbox stdout closed before terminal output; stdout reattach attempts exhausted. Observed on a single-node production deployment at46bebdf: a one-line config merge destroyed three live turns at 36, 20, and 19 minutes elapsed; two more earlier the same day at 73 and 34 minutes; and a review turn killed mid-flight could not be retried at all afterward, because the idempotency key is stable per commit and re-requests return the dead execution withcreated=false.Why it happens
The architecture already carries the right idea and the default deployment shape defeats it.
session_executionshasstdout_owner_idandstdout_owner_lease_expires_at, renewing on a ~60s TTL — so a replacement instance is clearly expected to claim the lease and reattach to a still-running sandbox. It cannot, because:With one replica there is no live instance to hand the lease to. The old pod's exec streams close on SIGTERM; the sandbox keeps producing output with no reader; a Kubernetes exec stream does not buffer for an absent reader, so those bytes are gone. The new pod's reattach finds a stream missing its terminal output, retries, and exhausts. The sandbox itself is healthy throughout — only the reader and the un-replayable bytes are lost.
Open questions that decide the fix
session_eventsalready persists a complete turn transcript, so the bytes exist somewhere durable — if that path is independent of the stdout pipe, reattach could reconcile from it instead of requiring an unbroken stream.Proposed direction
Any of: HA replicas with verified lease handoff; durable/replayable sandbox output (spool in harness-server, or reconcile reattach from the persisted event stream); or drain-then-handoff on SIGTERM. Failing all three, an interim mitigation: let a turn killed by a roll be retried (terminal failure should release the idempotency key) — losing the work is bad, losing it irrecoverably is worse.
Related: #1432 (api-rs restart also loses in-process idle timers — same restart, different loss).