QVAC-22704 feat[api]: multi-job continuous batching in the SDK

.cursor/rules/sdk/docs/kv-cache-system.mdc

@@ -99,7 +99,9 @@ Anonymous caches created by `kvCache: true` are tracked with internal marker met
 - Least-recently-used auto caches are removed when their total size exceeds 512 MiB on React Native or 4 GiB on other runtimes.
 - Cache paths used by active turns are excluded from eviction.
 
-Named string-key caches are not inferred from their directory names, are not subject to automatic retention, and remain caller-managed through `deleteCache()`. Generated caches from older SDK versions become managed when they are next reused; operators can use `deleteCache({ all: true })` for a one-time removal of unreachable legacy caches.
+Named string-key caches are not inferred from their directory names and remain caller-managed through `deleteCache()`. Retention acts only on cache directories that carry an auto-cache marker, and a marker is written only on an auto (`kvCache: true`) turn — a purely named cache never gets one. The one exception: a named key chosen to exactly equal a generated auto key (16 lowercase hex characters) resolves to the *same* directory as that auto cache, so it shares the auto cache's lock and its retention. Avoid that shape for named keys you want to manage independently. Generated caches from older SDK versions become managed when they are next reused; operators can use `deleteCache({ all: true })` for a one-time removal of unreachable legacy caches.
+
+**Case-only key variants are unsupported.** Two named keys that differ only in case (e.g. `"Session"` and `"session"`) are the same file on case-insensitive filesystems (default macOS/Windows). The per-cache-path write lock case-folds its key so such variants still serialise and can't corrupt one file, but the other bookkeeping (active-path refs, saved-message counts, retention's directory matching) stays case-sensitive — so retention may not recognise a case-variant alias as active. Pick a single canonical case per named key.
 
 ## Key Implementation Files
 
@@ -139,6 +141,8 @@ if (shouldCommit) await session.commitTurn(turn, ...);  // suppresses the deferr
 
 `commitTurn` flips an internal flag on the turn handle; `rollback` reads the flag and short-circuits. The happy path commits and the deferred rollback becomes a no-op. Every other exit path (cancel, zero-token, addon error, rename failure, tool-call turn) lets `scope[Symbol.asyncDispose]` run the deferred rollback, which atomically unlinks the `.bin` file, deletes the `initializedCaches` entry, and forgets the `cachedMessageCounts` entry.
 
+**Late-commit vs cancel is intentional, not a race.** A commit only proceeds past its abort re-check for a turn that fully generated and whose file passed `verifySaveAndRecord`. If a cancel lands during the async commit (after that check, during rename/verify), the commit finishes: a **complete, verified** turn is persisted while the operation reports cancelled. This is by design — the persisted cache matches the tokens the client already received, and committing fully-generated verified work beats discarding it. A strict "cancel wins" rollback here would need target-aware cleanup and risks deleting valid completed state, so it is not done.
+
 The maps that backed `clearCacheRegistry` and `cachedMessageCounts` are **private to `kv-cache-session.ts`**. No other module reads or writes them.
 
 ### Cache Initialization (primeIfMissing) and Addon Non-Transactional Save

.cursor/rules/sdk/docs/request-lifecycle-system.mdc

@@ -123,7 +123,7 @@ Extend the `CancelTarget` discriminated union in `request-registry.ts`. Today su
 
 ### What's the relationship to the addon's `cancel`?
 
-The addon (`@qvac/llm-llamacpp` and friends) exposes a synchronous `cancel(jobId?)` that signals the C++ side to stop decoding ASAP. The SDK wires `signal.addEventListener("abort", onAbort, { once: true })` so a registry cancel fires `addon.cancel()` at the binding leaf — but the SDK's source of truth for "was this cancelled" is still `signal.aborted`, not addon state. This keeps the SDK decoupled from per-addon cancel semantics; an addon without a `cancel()` method still gets best-effort cancellation through the SDK's signal-propagated termination of the streaming loop.
+Addon cancellation differs by handler capability. Request-scoped llama.cpp completion, batch completion, and LLM translate route an aborted registry context to that run's `response.cancel()`. Model-scoped handlers wire the signal to the addon's model-wide `cancel()`, while handlers with no cancel surface stop observing the result and let native work finish. In every case the SDK's source of truth for "was this cancelled" remains `signal.aborted`, not addon state.
 
 ### Why declare cancel capability on the plugin manifest?
 
@@ -141,19 +141,17 @@ The truth-table cell for each built-in plugin is pinned by `test/unit/plugin-can
 
 ### Why a concurrency policy on the registry?
 
-The llama.cpp addon owns one KV-cache + one decode loop per model. Two concurrent `completionStream` requests against the same model interleave their token streams on the same logical session — observably "the wrong tokens for my prompt" from the client's point of view, with the cache layer also drifting because two `KvCacheSession.beginTurn(...)` calls would race for the same `cachePath`.
+`@qvac/llm-llamacpp` now admits multiple concurrent `run()` calls per model — each its own native job with its own tokens — so completions no longer interleave on one session. The registry caps admission at the model's `parallel` and the KV-cache layer serializes only genuine same-file turns via a per-cache-path lock in `KvCacheSession`; turns on different cache keys decode concurrently.
 
 The registry exposes an admission-control hook so callers don't have to externally enforce "one completion per model":
 
 ```typescript
 registry.policy({ kind: "completion", maxConcurrentPerModel: 1, onOverflow: "queue" });
 ```
 
-`maxConcurrentPerModel` is the per-`(lane, modelId)` slot count (`1` today — the single-context reality; raise it to the addon's batching width once continuous batching lands). `onOverflow` decides what an over-capacity `begin(...)` does: `"queue"` (the default) **serializes** — the second `begin({ kind: "completion", modelId: "m1" })` **waits FIFO** behind the in-flight one; `"reject"` throws `RequestRejectedByPolicyError` (52420) immediately (the legacy `oneAtATimePerModel` behavior). On the queue path, `begin` rejects only when the per-model wait queue is already at its `maxQueueDepthPerModel` cap (default 64), or when a waiter exceeds an optional `queueTimeoutMs`. The error is structured (carries `requestId`, `kind`, `modelId`, `reason`) rather than a generic 500, so clients can `instanceof` narrow and surface "model saturated" to the user.
+`maxConcurrentPerModel` is the per-`(lane, modelId)` **top-level request-permit** count. The completion/batch handlers pass the loaded model's own `parallel` as that request cap; the registered `1` is only the fallback when a caller supplies none. A multi-prompt batch still consumes one permit rather than one permit per native sequence, so the addon's scheduler remains authoritative for actual sequence occupancy. `onOverflow` decides what an over-capacity `begin(...)` does: `"queue"` (the default) **serializes** — the second `begin({ kind: "completion", modelId: "m1" })` **waits FIFO** behind the in-flight one; `"reject"` throws `RequestRejectedByPolicyError` (52420) immediately (the legacy `oneAtATimePerModel` behavior). On the queue path, `begin` rejects only when the per-model wait queue is already at its `maxQueueDepthPerModel` cap (default 64), or when a waiter exceeds an optional `queueTimeoutMs`. The error is structured (carries `requestId`, `kind`, `modelId`, `reason`) rather than a generic 500, so clients can `instanceof` narrow and surface "model saturated" to the user.
 
-The worker singleton wires `completion` **and** `batchCompletion` with `maxConcurrentPerModel: 1, onOverflow: "queue"` and a shared `llamacppCompletion` lane (`sharedSlotGroup`), so the two kinds contend for one slot per model — see "Shared slot groups" in `request-lifecycle-primitives.mdc`. It also gives `audiogen` its own one-slot FIFO lane because ACE-Step replaces the active response when a second same-model run starts.
-
-Other kinds (`embeddings`, `transcribe`, `translate`, `finetune`) intentionally do **not** register a policy: those addons tolerate concurrent requests against the same model (embeddings is genuinely concurrent on the addon side; the streaming kinds share the addon's job-state but the registry's per-request lifecycle is sufficient for cancel routing without admission control). Per-kind policies can be registered if and when an addon's concurrency invariants change.
+The worker singleton registers the admission policies: `completion`, `batchCompletion`, and LLM `translate` share the `llamacppCompletion` lane as readers capped at the model's `parallel`, `finetune` joins it as an exclusive writer, and `embeddings` / `transcribe` / NMT `translate` stay ungated. See "Shared slot groups" in `request-lifecycle-primitives.mdc` for the full lane rules (this doc doesn't restate them). `audiogen` gets its own one-slot FIFO lane because ACE-Step replaces the active response when a second same-model run starts.
 
 The policy runs **before** controller / scope allocation in `begin(...)`, so a rejected `begin` leaves no registry entry behind. Disposing the holder of the in-flight slot releases admission for the next request — `await using` makes this automatic.
 
@@ -197,13 +195,13 @@ Test coverage: `same-tick cancel-before-begin retroactively aborts the later beg
 | `server/bare/ops/cancel.ts`                                                | Broad-cancel op: pure registry walk, legacy addon-cancel fallback removed in 0.11.0                      |
 | `server/rpc/handlers/cancelHandler.ts`                                     | RPC entry point: 2-arm `request` / `broad` dispatch (5-arm union collapsed in 0.11.0). Targeted `request` goes through `RequestRegistry.cancel({ requestId })` plus an optional `markClearCacheForRequest(...)` for downloads; `broad` delegates to `server/bare/ops/cancel.ts` |
 | `server/rpc/handlers/delete-cache.ts`                                      | Delegates to `deleteKvCacheState(...)` — zero direct references to the three KV-cache layers             |
-| `server/bare/plugins/llamacpp-completion/plugin.ts`                        | Reference plugin manifest; declares `cancel: { scope: "model", hard: true }`; builds `withRequestContext(...)` once per request and threads it into `completion(...)`; `finetune` declares `{ scope: "model", hard: true }`; `translate` handler threads `requestId` into the shared bare op |
+| `server/bare/plugins/llamacpp-completion/plugin.ts`                        | Reference plugin manifest; `completion`/`batchCompletion`/`translate` declare `cancel: { scope: "request", hard: true }` (per-job `response.cancel()`); `finetune` declares `{ scope: "model", hard: true }` (global cancel, made safe by exclusive admission); builds `withRequestContext(...)` once per request |
 | `server/bare/plugins/llamacpp-completion/ops/completion-stream.ts`         | Reference implementation of the canonical handler shape; uses `KvCacheSession`; accepts a request-scoped `logger` |
 | `server/bare/plugins/llamacpp-completion/ops/finetune.ts`                  | `startFinetune` opens a registry context (kind `"finetune"`), wires `signal → model.cancel()`, registers `scope.defer` cleanups for `clearFinetuneRuntimeState` + `handle.removeListener`; `cancelFinetune` is a thin wrapper over `registry.cancel({ modelId, kind: "finetune" })` |
 | `server/bare/plugins/llamacpp-completion/ops/kv-cache-session.ts`          | `KvCacheSession` factory + `deleteKvCacheState`: single owner of the three KV-cache layers; accepts a per-instance logger |
 | `server/bare/ops/embed.ts`                                                 | Registry-routed (kind `"embeddings"`, `scope: "model", hard: true`); accepts an optional `requestId`, falls back to a server-generated UUID |
 | `server/bare/ops/transcribe.ts`                                            | `transcribe` + `transcribeStream` registry-routed (kind `"transcribe"`); `restorePrompt` runs via `scope.defer`; per-iteration `if (ctx.signal.aborted) break;` in the response loop |
-| `server/bare/ops/translate.ts`                                             | Registry-routed (kind `"translate"`); LLM-translate branch wires `signal → addon.cancel()` (hard); NMT-translate branch is soft-cancel only (the `nmtcpp-translation.translate` handler declares `scope: "none"`); covers the `runBatch(...)` early-return path |
+| `server/bare/ops/translate.ts`                                             | Registry-routed (kind `"translate"`); LLM translate routes per-request cancellation through `response.cancel()`; NMT translate is soft-cancel only (the `nmtcpp-translation.translate` handler declares `scope: "none"`); covers the `runBatch(...)` early-return path |
 | `server/bare/plugins/{llamacpp-embedding,whispercpp-transcription,parakeet-transcription,nmtcpp-translation,onnx-tts,ggml-ocr,sdcpp-generation}/plugin.ts` | Built-in plugin manifests — each handler declares its `cancel: { scope, hard? }` truth-table row; inference handlers thread `requestId` from the request envelope into the shared bare ops |
 | `schemas/plugin.ts`                                                        | `PluginHandlerCancel` / `PluginHandlerCancelScope` types + runtime schema validation on `pluginHandlerDefinitionRuntimeSchema` |
 | `client/api/completion-stream.ts`                                          | Client-side construction of `InferenceCancelledError` on `stopReason: "cancelled"`                       |