feat(docs): update devshard docs

devshard/docs/devshard-update/.gitignore

@@ -0,0 +1 @@
+sandbox/

devshard/docs/devshard-update/README.md

@@ -0,0 +1,161 @@
+# Devshard Gateway Update
+
+Update a devshard gateway to a new image version behind nginx without dropping in-flight requests. Assumes multi-devshard **gateway-proxy** mode (routing prefix `/devshard-gateway/`) behind the nginx `proxy` container. Single-instance host, two ways — **Manual** (steps by hand) and **Automated** (one script). For a 2+ instance pool, see the Pool section below.
+
+An **escrow** and a **devshard** are the same object. The examples assume:
+
+```bash
+KEY="$DEVSHARD_ADMIN_API_KEY"
+MAIN="http://127.0.0.1:18080"
+TEMP="http://127.0.0.1:18081"
+REPO="ghcr.io/gonka-ai/devshard-gateway"
+# <OLD> / <NEW> = current / target image tag; <MODEL> = a model id you serve
+```
+
+## Manual update (single instance)
+
+Run from the gateway host's deploy dir (e.g. `deploy/join`). A temp gateway holds traffic while main updates.
+
+**1. Disable escrow rotation on main** (so nothing settles on-chain mid-update):
+
+```bash
+curl -fsS "$MAIN/v1/admin/settings" -H "Authorization: Bearer $KEY" \
+  | jq '.escrow_rotation.enabled = false' \
+  | curl -fsS -X POST "$MAIN/v1/admin/settings" -H "Authorization: Bearer $KEY" \
+      -H 'Content-Type: application/json' -d @-
+```
+
+**2. Start a temp gateway** on the new image, with its own storage and no escrows; reuse main's env file (keeps secrets out of `/tmp`):
+
+```bash
+docker run -d --name devshard-gateway-temp --restart unless-stopped \
+  --network join_default --network-alias devshard-gateway-temp \
+  --env-file ./config.devshard.env \
+  -e DEVSHARDS_JSON='[]' -e DEVSHARD_STORAGE_DIR=/root/.devshardctl/temp -e DEVSHARD_PORT=8080 \
+  -p 127.0.0.1:18081:8080 -v "$PWD/.devshardctl:/root/.devshardctl" \
+  "$REPO:<NEW>"
+curl -fsS "$MAIN/v1/admin/settings" -H "Authorization: Bearer $KEY" \
+  | jq '.escrow_rotation.enabled = false | .escrow_rotation.models = []' \
+  | curl -fsS -X POST "$TEMP/v1/admin/settings" -H "Authorization: Bearer $KEY" \
+      -H 'Content-Type: application/json' -d @-
+```
+
+**3. Mint temp escrows — for every model main serves** (a model with no temp escrows is unavailable during the update; each mint spends on-chain funds):
+
+```bash
+curl -fsS -X POST "$TEMP/v1/admin/escrows" -H "Authorization: Bearer $KEY" \
+  -H 'Content-Type: application/json' \
+  -d '{"amount":5000000000,"model_id":"<MODEL>","private_key_env":"DEVSHARD_PRIVATE_KEY","protocol_version":"1"}'
+# repeat per escrow and per model; then record the created ids:
+curl -fsS "$TEMP/v1/admin/devshards" -H "Authorization: Bearer $KEY" | jq '.devshards[].id'
+```
+
+**4. Verify temp** — status responds and a chat smoke test passes (test each model):
+
+```bash
+curl -fsS "$TEMP/v1/status" -H "Authorization: Bearer $KEY" | jq '[.devshards[].active_requests]|length'
+curl -fsS -X POST "$TEMP/v1/chat/completions" -H 'Content-Type: application/json' \
+  -d '{"model":"<MODEL>","max_tokens":1,"messages":[{"role":"user","content":"ok"}]}' >/dev/null && echo ok
+```
+
+**5. Switch nginx to temp** (graceful reload; in-flight streams finish). This matches both a direct `proxy_pass http://host:8080` and a named-upstream `server host:8080;`:
+
+```bash
+docker exec proxy sh -lc "cp /etc/nginx/nginx.conf /tmp/nginx.conf.bak && \
+  sed -E -i 's#http://devshard-gateway:8080#http://devshard-gateway-temp:8080#g; \
+             s#(server[[:space:]]+)devshard-gateway:8080#\1devshard-gateway-temp:8080#g' /etc/nginx/nginx.conf && \
+  nginx -t && nginx -s reload"
+```
+
+**6. Drain main** — repeat until this prints `0`:
+
+```bash
+curl -fsS "$MAIN/v1/status" -H "Authorization: Bearer $KEY" | jq '[.devshards[].active_requests]|max'
+```
+
+**7. Bump the image tag in compose:**
+
+```bash
+sed -i "s#$REPO:<OLD>#$REPO:<NEW>#g" docker-compose.devshard-gateway.yml
+```
+
+**8. Recreate main on the new image** (compose is the image source — not an env var):
+
+```bash
+docker compose -f docker-compose.devshard-gateway.yml up -d --no-deps --force-recreate devshard-gateway
+```
+
+**9. Verify main** — status responds and a chat smoke test passes (same as step 4, against `$MAIN`).
+
+**10. Switch nginx back to main** — same as step 5 with `devshard-gateway-temp` → `devshard-gateway`.
+
+**11. Drain temp, then stop AND remove it** (`--restart unless-stopped` means a leftover container would resurrect and corrupt state):
+
+```bash
+curl -fsS "$TEMP/v1/status" -H "Authorization: Bearer $KEY" | jq '[.devshards[].active_requests]|max'   # until 0
+docker stop devshard-gateway-temp && docker rm devshard-gateway-temp
+```
+
+**12. Import + activate the temp escrows into main** (so their funds aren't stranded) — for each temp escrow `<ID>`:
+
+```bash
+BODY='{"id":"<ID>","model":"<MODEL>","storage_path":"/root/.devshardctl/temp/escrow-<ID>/state.db","protocol_version":"1","private_key_env":"DEVSHARD_PRIVATE_KEY"}'
+curl -fsS -X POST "$MAIN/v1/admin/devshards/import" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "$(jq '.active=false' <<<"$BODY")"
+curl -fsS -X POST "$MAIN/v1/admin/devshards"        -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "$BODY"
+```
+
+**Rollback:** if a step fails, do not switch back to a bad main — keep the alias on temp, restore `/tmp/nginx.conf.bak` and reload, and re-pull the old image tag. If you stopped after minting temp escrows, finish steps 11–12 (or settle each via `POST /v1/admin/devshards/{id}/settle`) so their funds are recovered.
+
+## Automated update (single instance)
+
+The script runs steps 1–12 from a JSON config (models and everything else are config-driven — nothing hardcoded).
+
+```bash
+cp scripts/update.config.sample.json update.config.json     # edit: image tags, models[], nginx, compose
+./scripts/update.sh --config update.config.json run          # dry run — prints the full plan
+./scripts/update.sh --config update.config.json run --run    # execute
+```
+
+- `--run` executes (default is dry-run); `--yes` skips confirmations (unattended).
+- Run a single step: `./scripts/update.sh --config update.config.json <step>`.
+- Recover stranded temp escrows after an aborted run: `./scripts/update.sh --config update.config.json recover` (or `recover --settle`).
+
+The config (see [scripts/update.config.sample.json](scripts/update.config.sample.json)) holds the image (`repository`, `from_tag`, `to_tag`), the `models[]` array (`model`, `escrow_count`, `escrow_amount`), `allow_unavailable_models`, and the nginx / compose / timeout blocks. Env vars override individual fields.
+
+## Restore the nginx backup
+
+The switch backs up the nginx config **before every change**, inside the `proxy` container next to the config: `<config_path>.blue-green-backup` (e.g. `/etc/nginx/nginx.conf.blue-green-backup`). The manual step 5 above instead writes `/tmp/nginx.conf.bak`. To revert routing, restore whichever you used and reload:
+
+```bash
+docker exec proxy sh -lc "cp /etc/nginx/nginx.conf.blue-green-backup /etc/nginx/nginx.conf && nginx -t && nginx -s reload"
+# manual path: cp /tmp/nginx.conf.bak /etc/nginx/nginx.conf && nginx -t && nginx -s reload
+```
+
+Confirm the upstream it now points at:
+
+```bash
+docker exec proxy sh -lc "grep -nE 'devshard.*:8080' /etc/nginx/nginx.conf"
+```
+
+**Caveat — the backup is overwritten on each switch.** It holds the routing as of just *before the most recent switch*: after `switch-to-main` it points back at **temp**, not the pristine original. For a guaranteed clean revert, copy the original aside before you start:
+
+```bash
+docker exec proxy sh -lc "cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.orig"
+```
+
+## Pool (multi-instance)
+
+If 2+ gateway instances sit behind nginx, update one at a time — no temp gateway:
+
+1. Remove one instance from the nginx `upstream` (comment its `server` line or mark it `down`); `nginx -t && nginx -s reload`.
+2. Drain it: `GET /v1/status` until `max(.devshards[].active_requests) == 0`.
+3. Bump its image tag in compose; `docker compose -f <file> up -d --no-deps --force-recreate <service>`.
+4. Verify: `/v1/status` responds and a `POST /v1/chat/completions` returns.
+5. Return it to the pool; reload. Repeat for the next instance.
+
+## References
+
+- [references/admin-api.md](references/admin-api.md) — gateway admin endpoints used here.
+- [references/nginx-alias-switching.md](references/nginx-alias-switching.md) — how the upstream switch keeps in-flight streams alive.
+- [references/troubleshooting.md](references/troubleshooting.md) — drain stalls, unroutable escrows, rollback.
+- [SKILL.md](SKILL.md) — operator/agent skill wrapping this process.

devshard/docs/devshard-update/SKILL.md

@@ -0,0 +1,52 @@
+---
+name: devshard-update
+description: Update a Gonka devshard gateway to a new image version without dropping in-flight /v1/chat/completions requests, driven by a JSON config. Single-instance blue/green (temp gateway + nginx switch + drain + config-driven temp escrows + import); a recover command reclaims stranded temp escrows; multi-instance pools use a rolling update.
+---
+
+# Devshard Gateway Update
+
+Companion to [README.md](README.md) (manual steps + the command) and [scripts/update.sh](scripts/update.sh).
+
+## One command (single instance)
+
+```bash
+cp scripts/update.config.sample.json update.config.json     # edit: image tags, models[], nginx, compose
+./scripts/update.sh --config update.config.json run          # dry run — prints the plan
+./scripts/update.sh --config update.config.json run --run    # execute
+```
+
+Add `--yes` for unattended. Everything is config-driven (models included) — nothing is hardcoded.
+
+## Safety rules
+
+- Treat any live gateway host as production. Show the exact command and get approval before running it.
+- Dry-run first (`run` without `--run`) and read the plan.
+- MAIN's image comes from the compose file; the `bump-main-image` step rewrites the tag from `image.from_tag` to `image.to_tag`.
+- Temp escrows must cover every model MAIN serves — `preflight` fails closed if a served model has no temp coverage and isn't in `allow_unavailable_models`.
+- Public verification runs only if `nginx.public_base_url` is set — never trust loopback.
+- Do not assume the nginx config path — inspect with `docker exec <proxy> sh -lc 'nginx -T'`.
+
+## Config
+
+`scripts/update.config.sample.json` — blocks: `image {repository, from_tag, to_tag}`, `models[] {model, escrow_count, escrow_amount}`, `allow_unavailable_models[]`, `escrow`, `main`, `temp`, `nginx`, `compose`, `timeouts`, `rotation`. Any field is overridable by the matching env var.
+
+## Steps & actions
+
+Full-flow order (aborts on the first failed gate and names the step):
+`init` → `preflight` → `disable-main-rotation` → `create-temp-gateway` → `create-temp-escrows` → `check-temp` → `switch-to-temp` → `check-alias-temp` → `drain-main` → `bump-main-image` → `update-main` → `check-main-direct` → `switch-to-main` → `check-alias-main` → `drain-temp` → `stop-temp` → `import-temp` → `activate-temp` → `restore-main-rotation` → `status`.
+
+- Single step: `./scripts/update.sh --config <file> <step> --run`.
+- Resume: `... run --run --from-step drain-main`.
+- Recover stranded temp escrows after an aborted run: `./scripts/update.sh --config <file> recover --run` (add `--settle` to settle them instead of folding into main).
+- `plan` / `validate` / `list-steps` for inspection (no side effects).
+
+## Remote execution
+
+```bash
+ssh -p <port> <user>@<host> "$(cat <<'REMOTE'
+set -euo pipefail
+cd <deploy-dir>
+./scripts/update.sh --config update.config.json run --run --yes
+REMOTE
+)"
+```

devshard/docs/devshard-update/references/admin-api.md

@@ -0,0 +1,62 @@
+# Reference — Gateway Admin API (migration subset)
+
+The endpoints a broker touches during a zero-downtime update. All admin routes require `Authorization: Bearer $DEVSHARD_ADMIN_API_KEY`; `/v1/status` is public. Handlers live in `devshard/cmd/devshardctl/gateway.go`.
+
+## Status & draining
+
+| Method / path | Purpose |
+|---|---|
+| `GET /v1/status` | Public health + per-devshard runtime snapshot. **The drain gate.** Each devshard reports `active_requests`. |
+| `GET /v1/admin/devshards` | Full admin list of persisted devshards (active + inactive) with model, protocol_version, storage, balance. |
+| `GET /v1/admin/state` | Broader admin snapshot (devshards + effective settings). |
+
+**Drain check** — poll until zero:
+
+```bash
+curl -fsS http://127.0.0.1:18080/v1/status \
+  -H "Authorization: Bearer $DEVSHARD_ADMIN_API_KEY" \
+  | jq '[.devshards[]?.active_requests // 0] | max // 0'
+```
+
+`active_requests` is an atomic counter per runtime, incremented on request reserve and decremented on release, then serialized under `devshards[].active_requests`. It is also exported as the Prometheus gauge `devshard_runtime_active_requests`.
+
+## Settings
+
+| Method / path | Purpose |
+|---|---|
+| `GET /v1/admin/settings` | Effective persisted gateway settings. |
+| `POST /v1/admin/settings` | Update settings live (e.g. `escrow_rotation`, `max_concurrent_requests`). |
+
+During a migration, copy main's settings to the temp gateway but force rotation off so temp never auto-settles:
+
+```bash
+curl -fsS http://127.0.0.1:18080/v1/admin/settings -H "Authorization: Bearer $KEY" \
+  | jq '.escrow_rotation.enabled = false | .escrow_rotation.models = []' \
+  | curl -fsS -X POST http://127.0.0.1:18081/v1/admin/settings -H "Authorization: Bearer $KEY" \
+      -H 'Content-Type: application/json' -d @-
+```
+
+`escrow_rotation.enabled = false` is the master off switch — it stops the rotator and thus all auto-settlement. Disable it on **MAIN** directly (a running gateway keeps whatever is in `gateway.db`, not the first-boot default); `settlement_enabled` and `models` are finer controls you don't need to touch for a swap.
+
+## Escrow lifecycle
+
+| Method / path | Purpose |
+|---|---|
+| `POST /v1/admin/escrows` | Create a **new** escrow on-chain (`amount`, `model_id`, `private_key_env`, `protocol_version`) and optionally register it. Used to seed **temp** escrows. |
+| `POST /v1/admin/devshards/import` | Import an **existing** escrow into this gateway by `id` + `storage_path` + `private_key_env` (+ optional `perf_path`), with `active:false`. Carries an escrow's sqlite state into a fresh gateway process — the storage-portability tool. |
+| `POST /v1/admin/devshards` | Register / **activate** a devshard (`id`, `model`, `storage_path`, `protocol_version`, `private_key_env`). Adds it to the routing pool. |
+| `POST /v1/admin/devshards/{id}/deactivate` | Set `active=false` — removes from the routing pool, keeps the runtime loaded, **does not settle**. In-flight requests finish. |
+| `POST /v1/admin/devshards/{id}/settle` | Deliberately settle an escrow on-chain (drain-aware). |
+| `DELETE /v1/admin/devshards/{id}` | Remove a devshard cleanly. |
+
+## protocol_version
+
+Only `"1"` (aka `""` / `"v1"`) parses; anything else is rejected. It is a stored field but effectively pinned to v1 — a gateway image "vX → vY" update is **not** a protocol change. Pass `"1"` (or leave default) when creating/importing/activating.
+
+## Notes for migration
+
+- **Import needs `storage_path`.** It is how an existing escrow's `state.db` is reattached to a new process. Point it at the temp escrow's container path (e.g. `/root/.devshardctl/temp-<run-id>/escrow-<id>/state.db`).
+- **Create vs import:** `create` mints a *new* on-chain escrow; `import` re-attaches an *existing* one. Temp escrows are `create`d, then later `import`ed into main.
+- **Deactivate ≠ settle.** Deactivating during a swap is safe and reversible; settlement is a separate, config-gated, drain-aware action.
+- **`escrow` and `devshard` are the same object.** `create` mints it (`/v1/admin/escrows`); everything else lists/imports/activates/deactivates it (`/v1/admin/devshards`, and it appears under `/v1/status .devshards[]`).
+- **Field name differs by call:** `create` takes `model_id`; `import` and `activate` take `model`.

devshard/docs/devshard-update/references/nginx-alias-switching.md

@@ -0,0 +1,54 @@
+# Reference — nginx upstream switch
+
+The `proxy` container forwards `/v1/chat/completions` to the gateway. The zero-downtime switch repoints that target from the old gateway to the new one and reloads, without cutting in-flight streams.
+
+## Why in-flight streams survive
+
+`nginx -s reload` is graceful:
+
+1. `nginx -t` validates the new config.
+2. `nginx -s reload` starts **new** workers with the new config and tells the **old** workers to stop accepting connections.
+3. Old workers keep running until their in-flight requests finish (including long SSE streams), then exit.
+
+New requests go to the new target; existing requests drain on the old workers. Nothing is cut mid-stream — which is why the switch uses reload, never a proxy-container restart.
+
+## What the switch changes
+
+Both the manual `sed` (README step 5) and the script replace the upstream host `<old>:PORT` → `<new>:PORT` in the nginx config (inside the `proxy` container), then run `nginx -t && nginx -s reload`. One `sed` handles **both** config styles:
+
+```nginx
+location /devshard-gateway/ { proxy_pass http://devshard-gateway:8080/; }   # direct pass — host swapped
+upstream pool { server devshard-gateway:8080; }                              # named upstream — host swapped
+# proxy_pass http://pool;  — references the block by name, left untouched (correct)
+```
+
+In the config these are `nginx.old_upstream`, `nginx.new_upstream`, `nginx.upstream_port`, `nginx.config_path`, `nginx.proxy_container`.
+
+## Pool (2+ gateways)
+
+A named upstream with multiple `server` lines. Rolling update: mark one `down` (or comment it), reload, drain+update that instance, restore it, reload; repeat. nginx load-balances across whoever is up, so capacity never drops to zero.
+
+```nginx
+upstream devshard_gateway {
+    server gw-a:8080;
+    server gw-b:8080;   # comment out (or `down`) while updating gw-b, then restore
+}
+```
+
+## Finding the live config
+
+Do not assume the path. The effective config may be `/etc/nginx/nginx.conf`, a file under `conf.d/`, or an included fragment:
+
+```bash
+docker exec proxy sh -lc 'nginx -T 2>&1 | grep -n "devshard\|v1/chat\|proxy_pass\|upstream"'
+```
+
+Set `nginx.config_path`, `nginx.proxy_container`, `nginx.old_upstream`, `nginx.new_upstream`, and `nginx.upstream_port` in the config to match what you find.
+
+The gateway's compose (`deploy/join/docker-compose.devshard-gateway.yml`) registers it under two network aliases — `${DEVSHARD_INSTANCE_NAME:-devshard-gateway}` and `devshard-pool` — so the proxy could reach it by either. The proxy's own config is not in this repo, so `nginx.old_upstream` (default `devshard-gateway`) is a guess: confirm the real upstream host with `nginx -T` before you switch.
+
+## Gotchas
+
+- **Verify through the public URL, not `127.0.0.1`.** Public verification runs only if `nginx.public_base_url` is set; a loopback check can pass while the public route is broken.
+- **Host+port replacement.** The switch rewrites `http://<host>:PORT` and `server <host>:PORT;`. A `proxy_pass` that points at an upstream *name*, a variable, a map, or a non-configured port won't match — set `nginx.old_upstream` / `nginx.new_upstream` / `nginx.upstream_port`, or switch by hand.
+- **Always `nginx -t` before reload.** A reload with a broken config keeps the old workers serving but blocks the switch. The switch gates the reload on `nginx -t` and backs up the config first — restore the backup to revert routing instantly.