viewer: add worker and cold process timeouts and stream large assets

[warun7]

Summary

Prevents permanent server lockups on hung worker/subprocess calls and prevents high memory usage on large file transfers:

1. Warm Worker Request Timeout & Deadlock Prevention:

   • Implemented _read_line with bounded timeout (VIEWER_CADGEN_TIMEOUT, default 300s; VIEWER_CAD_WORKER_PING_TIMEOUT, default 10s) in worker_client.py.
   • When a worker request times out or faults, _WorkerTransportError is raised, triggering process termination (_reap()) and a single transparent respawn. Subsequent requests no longer block forever on self._lock.

2. Cold Subprocess Execution Timeout:

   • Added timeout handling (subprocess.TimeoutExpired) to cadgen_bridge.py:run_cadgen_cold, returning {ok: false, error: "cadgen <module> timed out after <N>s"} rather than hanging indefinitely.

3. Chunked Asset & Static File Streaming:

   • Updated _serve_static_file and _serve_asset in server.py to stream files in 64 KiB chunks using shutil.copyfileobj instead of buffering whole multi-GB files in memory via handle.read().

Testing

• Added viewer/server_py/tests/test_worker_timeouts.py:
  • test_worker_read_line_timeout_raises_transport_error
  • test_worker_request_timeout_reaps_and_recovers
  • test_cadgen_bridge_cold_subprocess_timeout
  • test_stream_file_serves_chunks
• All 180 Python tests in viewer/server_py/tests/ passed.
• All 364 JavaScript tests in viewer/ passed.
• Verified viewer self-containment check.

[earthtojake]

Sorry for the delayed response, and thanks for the contributions so far.

The streaming work is good and stands alone. 64 KiB chunks with an explicit
content-length beats loading whole GLBs into memory, and nothing about it depends on
the timeout question. Split it into its own PR and it can land right away.

The timeouts have a problem no constant can fix. Real builds run long: a mid-size
robot regenerates for around 30 seconds, and the generation runner's own comments
call out multi-minute gen_step() runs. The first open in the viewer is exactly
when that full cost lands. There is also no cross-run cache, so a kill at 300s does
worse than delay the build: the retry restarts from zero and dies at the same wall,
and a slow model becomes one the viewer can never build. Killing a warm worker also
throws away its paid OCP import. Anyone who raises VIEWER_CADGEN_TIMEOUT to cover
their largest model has switched the guard off, which says wall-clock time is the
wrong signal.

The signal that separates hung from slow is progress. Builds already write it
(.generation.progress.json feeds the viewer's progress UI, and the cold path
streams progress lines), so an idle watchdog fits here: kill after N seconds of
silence rather than N seconds of work. A hung process goes quiet and gets caught in
seconds, while a slow build keeps reporting and lives. The 10s ping timeout is fine
as-is, since a ping is bounded work.

One architecture note: worker_client.py is deleted on release/0.5.0 when the
viewer joins the shared warm daemon pool, so timeout work built there gets discarded
at the rebase. The cold-path bridge survives the move. That's another reason to land
streaming separately and keep the timeout redesign small.

If you want to chat further, message me on Discord: https://discord.gg/5FGB9DwJYU

[warun7]

Reworked per review (dbeea4f), and the streaming half is split out into #333 so it can land on its own.

Timeouts are now an idle watchdog, not a wall clock:

• worker.py already routes all build narration to stderr; the client now always drains that pipe (an undrained stderr pipe was also a latent hang: a chatty build fills the 64 KiB buffer and blocks the worker mid-write) and every line stamps a liveness time.
• The response wait slides its deadline while narration arrives and declares the worker dead only after VIEWER_CADGEN_IDLE_TIMEOUT (default 300s, <=0 disables) of total silence. A hung readline is caught at one quiet budget; a build narrating phases waits as long as it needs. Ping keeps its short bounded budget.
• The cold subprocess got the same rule by swapping capture_output for pumped pipes, killing on the same silence budget (run_cadgen_cold survives the 0.5.0 daemon move, as you noted).
• Transport faults still reap + respawn once before failing the request, so the module-global lock can't wedge forever.

Tests: silent worker → dead at the budget; narrating worker that outlives a wall-clock cap → survives; drainer freshness pinned; same pair for the cold path. Mutation-checked both directions.

[earthtojake]

The idle-watchdog redesign is right, and I verified the hard part against a real
build: a 9.8s OCP artifact build (planetary gear, forced) ran under the watchdog,
narrated, and returned ok=True. The sliding deadline, the pipe drain, and the
kill-on-silence all behave as the tests describe.

One bug blocks the merge: the rewrite of run_cadgen_cold dropped the function's
final return. The last line computes message and then the function ends, so every
subprocess that exits without printing a JSON line returns None instead of an
error dict. develop's version ends with:

return {"ok": False, "exitCode": proc.returncode, "error": message}

That path is routine, and the caller crashes on it. Reproduction:

from server_py import cadgen_bridge
result = cadgen_bridge.run_cadgen_cold(
    "cadgen.step_artifact_cli",
    ["--repo-root", root, "--source-path", target],   # missing required --step
    root,
)
# result is None; develop returns {"ok": False, "error": "...required: --step..."}

On develop the argparse message reaches the client as a build error. On this branch
result.get(...) raises AttributeError in backend._run_artifact_build and the
route 500s.

The tests miss it because each one either times out (early return) or prints valid
JSON. Two asks:

1. Restore the terminal return.
2. Add a test where the subprocess exits without a JSON line and assert the error
   dict comes back, so the path stays covered.

Everything else is ready; with those two this merges.

[warun7]

Both asks done in 24e9d28:

1. Terminal return {"ok": False, "exitCode": ..., "error": message} restored — good catch, that was a straight-up drop in the rewrite.
2. New test: a subprocess that exits with a usage message and no JSON line must return the error dict (asserts not-None, ok false, exit code, and the message content). Mutation-verified: removing the return fails the test.