QVAC-23810 doc: cover multi-job continuous batching and per-request cancel (#3980)

* doc: cover multi-job continuous batching and per-request cancel

- ai-capabilities/text-generation.mdx: new "Concurrent completions"
  subsection covering `modelConfig.parallel >= 2` to fan out several
  `completion()` calls on one loaded model, the burst-finishes-in-time-
  of-slowest throughput promise, and per-request `cancel({ requestId })`
  isolation from concurrent peers. Embeds the SDK's
  `multi-job-completion.ts` example (JS + TS tabs). Adds a matching
  bullet in "Features".
- ai-capabilities/batch-processing.mdx: cross-link callout in the
  Overview pointing at the new subsection, flagging that mixing
  `completion()` and `batchCompletion()` on one model does not multiply
  concurrency past `parallel`.
- runtime/cancellation.mdx: sharpened the "peer isolation" sentence in
  the targeted-cancel section to name it as the right form for a
  per-request stop button that leaves the rest of the workload alone;
  rewrote the `RequestRejectedByPolicyError` bullet — the old
  `oneAtATimePerModel` framing is stale — to describe the shared
  per-model queue (up to `parallel` concurrent, 64-deep FCFS wait,
  65th rejected up-front), the exclusive-writer finetune, and a
  short "raise `parallel`, back-pressure, or retry" recovery hint.
- cli/http-server/index.mdx: one extra paragraph in "Request
  cancellation" naming `serve.models.<alias>.config.parallel` for how
  many HTTP requests to the same alias run in parallel, and confirming
  a client disconnect cancels only its own request.

Reflects the SDK behavior shipped in #3682 (merged
2026-08-19) and its published `multi-job-completion.ts` example.

* doc: address review — clarify parallel/ctx_size and cancel scopes

- text-generation.mdx: state `modelConfig.parallel` default (1);
  explain that `ctx_size` is split evenly across `parallel` slots
  with the `parallel × per-request context` sizing rule; link the
  addon-side `packages/llm-llamacpp/docs/continuous-batching.md`.
- cancellation.mdx: scope the targeted-cancel isolation promise to
  `completion()`, `batchCompletion()` and LLM `translate()`; add a
  "watch out" paragraph and matching Coverage-callout bucket for
  `embed()` / `transcribe()` / `audioGen()`, which route to model-wide
  cancel and abort peers on the same model; disambiguate NMT
  `translate` from LLM `translate`; on the `RequestRejectedByPolicyError`
  bullet, cross-link `modelConfig.parallel` and note that raising
  `parallel` requires raising `ctx_size` proportionally.
- packages/sdk/examples/multi-job-completion.ts: drop the "Non-cached"
  qualifier from the header comment — the disk-KV-cache-in-private-lane
  design was reverted before #3682 merged, so cached completions ride
  the same admission lane. Docs page renders the fix on next site build
  via `prebuild:examples`.

* doc: drop external addon-side link from concurrent completions

- text-generation.mdx: remove the trailing sentence pointing at
  `packages/llm-llamacpp/docs/continuous-batching.md` in the
  Concurrent completions section. The section already explains the
  `ctx_size`-shared-across-slots interaction inline, and linking
  to an addon repo doc from the SDK-facing page mixes audiences.

* doc: bound throughput claim and name HTTP parallel default

- text-generation.mdx: the "burst finishes in roughly the time of
  the slowest prompt" promise now carries "up to `parallel`" on both
  the Features bullet and the Concurrent completions paragraph, with
  "extras queue" / "beyond that they queue" to keep the selling point
  intact without over-claiming for `N > parallel`.
- cli/http-server/index.mdx: the Request-cancellation paragraph on
  same-alias parallelism now states that `serve.models.<alias>.config.parallel`
  defaults to `1` and instructs the operator to set it explicitly to
  serve an alias concurrently — the CLI ships no default, so the prior
  wording read as a description of behavior that only kicks in after
  explicit opt-in.

* doc: sharpen parallel and cancel accuracy per review

- cancellation.mdx: move audioGen() out of the "targeted cancel also aborts
  peers" bucket in the Coverage callout and the Watch out callout. Its
  admission is capped at 1 and audio-gen-stream.ts drops queued waiters
  without calling model-wide cancel, so peers survive — the collateral
  warning only applies to embed() and transcribe().
- cancellation.mdx: rewrite the RequestRejectedByPolicyError bullet so
  finetune no longer sits with the readers. completion / batchCompletion /
  LLM translate share the "parallel + 64 waiters" queue; a second finetune
  while one runs is rejected earlier with CompletionFailedError (52406)
  before it reaches admission, so branch on that code to distinguish
  "already training" from "queue full".
- text-generation.mdx: rewrite the Concurrent completions paragraph so it
  captures the two-layer behaviour of parallel without jargon. Each call is
  one request (a big batchCompletion never blocks more than one peer from
  starting), but each prompt in a batch takes one decode slot — so a large
  batch fills every parallel slot itself and a concurrent completion() only
  starts decoding once the batch has room.
- cli/http-server/index.mdx: repoint the parallel link from
  /reference/api#loadmodel (auto-generated, does not document parallel) to
  /ai-capabilities/text-generation#concurrent-completions, matching the
  anchor already used from cancellation.mdx.

Author: BrunoCampana

docs/website/content/docs/ai-capabilities/batch-processing.mdx

@@ -20,6 +20,10 @@ To use it, load the model with `modelConfig.parallel >= 2` (this is what lets it
 Batch processing is specific to **LLM text generation**. It does not apply to other capabilities such as embeddings or transcription.
 </Callout>
 
+<Callout type="info">
+**Related:** if instead of grouping many prompts into one `batchCompletion()` call you want to fire several independent `completion()` calls in parallel on the same model — one stream per call, cancellable independently — see [Concurrent completions](/ai-capabilities/text-generation#concurrent-completions). Both features use the same `modelConfig.parallel` slots, so mixing them on one model does not add extra concurrency.
+</Callout>
+
 ## Functions
 
 Use the following sequence of function calls:

docs/website/content/docs/ai-capabilities/text-generation.mdx

@@ -40,6 +40,7 @@ You can load any [`llama.cpp`](https://github.com/ggml-org/llama.cpp)-compatible
 * Raw output: with `emitRawDeltas: true`, every raw model token is also emitted as a `rawDelta` event in parallel to the structured events — useful for debugging or full-fidelity logging.
 * KV cache: cache and reuse the model's key/value attention state to speed up follow-up turns in long conversations. The `kvCache` parameter sets the cache key — pass a string to manage a session manually, or `true` to let the SDK auto-generate one.
 * Multimodal: attach images (and other media) to prompts so the model reasons over text and images together in the same conversation — see [Multimodal](/ai-capabilities/multimodal).
+* Multi-job continuous batching: load with `modelConfig.parallel >= 2` (default `1`) to run several `completion()` calls in parallel on one loaded model. Up to `parallel` prompts finish in roughly the time of the slowest one (extras queue), and each call has its own `requestId` so you can cancel one without touching the others — see [Concurrent completions](#concurrent-completions).
 * Batch processing: run multiple prompts through a single loaded model in one call with `batchCompletion()` — optimizes resources and reduces total run time. See [Batch processing](/ai-capabilities/batch-processing).
 
 ## Examples
@@ -78,6 +79,36 @@ The canonical way to consume `completion()` is the `events` async iterable plus
 The examples below (`Tool call`, `MCP`, `KV cache`) still consume `result.tokenStream` and `result.toolCallStream`, which are convenience wrappers around the canonical `events` / `final` stream shown above. Both APIs are supported; new code should prefer `events` / `final`.
 </Callout>
 
+### Concurrent completions
+
+Use this when you have more than one prompt in flight against the same loaded model at the same time — for example, serving several users, or fanning out related sub-tasks — and want them to finish together instead of one after another.
+
+Load the model with `modelConfig.parallel >= 2`, then call `completion()` several times without awaiting between calls. The SDK runs up to `parallel` of them together on the model, so **up to `parallel` prompts finish in roughly the time of the slowest one** instead of the sum of all of them; beyond that they queue. Each call has its own `requestId`; `cancel({ requestId })` stops just that call and its peers keep going — useful for a per-request stop button, rather than killing the whole model.
+
+The default is `1` (one request at a time), so you have to set `parallel` explicitly to enable batching. Also raise `ctx_size` with it: the model's context window is split evenly across the `parallel` slots, so `parallel: 4` on a `ctx_size: 4096` model gives each request only ~1024 tokens. Size `ctx_size` for `parallel × per-request context`.
+
+`completion()` and `batchCompletion()` share the model's `parallel` slots. Each call is one request — a big `batchCompletion({ prompts: N })` never blocks more than one concurrent `completion()` from starting — but every prompt in the batch takes one decode slot, so a large batch fills all `parallel` slots itself and any concurrent `completion()` you launch will only start decoding once the batch has room. If you already have every prompt ready up front, [`batchCompletion()`](/ai-capabilities/batch-processing) is often simpler; use concurrent `completion()` when prompts arrive over time or you want independent streams and cancellation per call.
+
+The following script loads a `parallel: 4` model, fires four completions at once, and then cancels one of two long runs to show its peer keeps decoding:
+
+<Tabs>
+<Tab value="js" label="JavaScript" default>
+<WrapCode>
+
+```js file=<rootDir>/packages/sdk/dist/examples/multi-job-completion.js title="multi-job-completion.js" lineNumbers
+```
+</WrapCode>
+</Tab>
+
+<Tab value="ts" label="TypeScript">
+<WrapCode>
+
+```ts file=<rootDir>/packages/sdk/examples/multi-job-completion.ts title="multi-job-completion.ts" lineNumbers
+```
+</WrapCode>
+</Tab>
+</Tabs>
+
 ### Tool call
 
 The following script shows how to provide tool definitions to `completion()`, consume the streaming output, and read the parsed tool calls.

docs/website/content/docs/cli/http-server/index.mdx

@@ -1341,6 +1341,8 @@ Abort the job (if still `queued` / `in_progress`) and drop its rendered assets.
 
 When an HTTP client disconnects before a response finishes (closes the connection or aborts the request), the server cancels the in-flight inference for that request instead of letting it run to completion — freeing the model to serve the next call. This applies to both blocking and streaming requests across the inference routes (`/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/embeddings`, `/v1/audio/*`).
 
+Multiple HTTP requests to the same LLM alias run in parallel up to that model's `serve.models.<alias>.config.parallel` (see [`modelConfig.parallel`](/ai-capabilities/text-generation#concurrent-completions)), which defaults to `1` — set it explicitly to serve an alias concurrently. Additional requests wait in a first-come-first-serve queue. A client disconnect cancels only that one request — other requests still running on the same model keep going.
+
 Video jobs are asynchronous and are not tied to the creating connection; cancel them explicitly with `DELETE /v1/videos/{id}` (see [Videos](#videos)).
 
 ### Authentication

docs/website/content/docs/runtime/cancellation.mdx

@@ -11,9 +11,11 @@ Every long-running SDK operation that goes through the request registry can be c
 The mental model is: **the primary path is `requestId`** — pass the run's `requestId` to `cancel()` to stop that exact call. **The `modelId` path is an escape hatch** — use it for model unload, app shutdown, admin sweeps, or for ops whose addons cannot interrupt mid-decode (`translate`, `textToSpeech`, `ocr`, `diffusion`, `upscale`).
 
 <Callout title="Coverage" type="info">
-**Targeted cancel by `requestId`** works for: `completion()`, `batchCompletion()`, `audioGen()`, `loadModel()`, `embed()`, `transcribe()`, `downloadAsset()`, and `rag*()` (`ragIngest`, `ragSaveEmbeddings`, `ragReindex`).
+**Targeted cancel by `requestId` — only the cancelled call stops**: `completion()`, `batchCompletion()`, LLM `translate()`, `audioGen()`, `loadModel()`, `downloadAsset()`, and `rag*()` (`ragIngest`, `ragSaveEmbeddings`, `ragReindex`).
 
-**Broad cancel by `modelId`** additionally covers `translate()`, `textToSpeech()`, `ocr()`, `diffusion()`, and `upscale()`. These accept `cancel({ modelId })` but their addons cannot interrupt mid-decode — the in-flight call stops yielding when `signal.aborted` flips on the next yield point, and the C++ work runs to completion in the background.
+**Accepted by `requestId` but also aborts other in-flight calls on the same model**: `embed()`, `transcribe()`.
+
+**Broad cancel by `modelId`** additionally covers NMT `translate()`, `textToSpeech()`, `ocr()`, `diffusion()`, and `upscale()`. These accept `cancel({ modelId })` but their addons cannot interrupt mid-decode — the in-flight call stops yielding when `signal.aborted` flips on the next yield point, and the C++ work runs to completion in the background.
 
 **Duplex sessions** — `transcribeStream(...)` and `textToSpeechStream(...)` use `.destroy()` on the returned session.
 
@@ -75,7 +77,11 @@ Outcome on the consumer side:
 
 Other operations that go through `cancel({ requestId })` (`loadModel`, `downloadAsset`, `embed`, `transcribe`, `rag*`) all reject their returned promise with the same `InferenceCancelledError` (code `52419`) — the error class is reused across non-inference handlers, no new code was added.
 
-Only the targeted call is affected — other in-flight calls on the same `modelId` keep running. To cancel `translate`, `textToSpeech`, `ocr`, `diffusion`, or `upscale` — or to sweep every in-flight call on a model in one shot — use the broad-cancel form below.
+Only the cancelled call stops — other in-flight `completion()`, `batchCompletion()` and LLM `translate()` calls on the same `modelId` keep running. Use this form when you want a per-request stop button that doesn't disturb the rest of your workload.
+
+**Watch out** for `embed()` and `transcribe()`: a targeted cancel by `requestId` **also stops every other in-flight call on the same model**. If several requests share a model, expect them all to be aborted together.
+
+To cancel NMT `translate`, `textToSpeech`, `ocr`, `diffusion`, or `upscale` — or to sweep every in-flight call on a model in one shot — use the broad-cancel form below.
 
 ## Broad cancel by `modelId` (escape hatch)
 
@@ -182,6 +188,6 @@ The following script loads a model, starts a streaming `completion()`, cancels i
 - `InferenceCancelledError` (code `52419`) — expected on the `final` promise (and any aggregate promise) after a consumer-initiated cancel. Treat it as a normal outcome, not a failure. Carries `requestId` plus a `partial: { text?, toolCalls?, stats? }` payload with whatever was accumulated before the cancel point.
 - `RequestNotFoundError` (code `52418`) — registry lookup miss for the given `requestId`. Rare in practice because `cancel({ requestId })` against an already-terminated id is a no-op on the handler (returns `success: true, cancelled: 0`); consumer code that narrows by class will see this for other call sites that look up a request by id.
 - `RequestIdConflictError` (code `52417`) — two requests landed with the same `requestId`. Astronomically unlikely with UUIDv4; if you see it, report.
-- `RequestRejectedByPolicyError` (code `52420`) — the registry's concurrency policy rejected the request before it began (e.g. `oneAtATimePerModel` for `completion` — the second concurrent completion against the same model is admissibility-rejected). Carries `requestId`, `kind`, `modelId`, and a human-readable `reason`.
+- `RequestRejectedByPolicyError` (code `52420`) — the model's queue is full and your request was rejected before it started. On an LLM, `completion`, `batchCompletion`, and LLM `translate` share one queue: [`modelConfig.parallel`](/ai-capabilities/text-generation#concurrent-completions) running slots plus 64 waiters. A running `finetune` locks that queue, so readers pile up behind it and can hit the same cap. Launching a second `finetune` while one runs gives you `CompletionFailedError` (code `52406`) instead — branch on it if you need to tell "already training" from "queue full". To serve bigger bursts, raise `parallel` at load (and raise [`ctx_size`](/ai-capabilities/text-generation#concurrent-completions) proportionally), back-pressure at the app level, or catch and retry. Carries `requestId`, `kind`, `modelId`, and a `reason`.
 - `AsyncDisposeUnavailableError` (code `53503`) — the runtime is missing `Symbol.asyncDispose` (older Bare builds). Upgrade Bare.
 

packages/sdk/examples/multi-job-completion.ts

@@ -17,9 +17,8 @@
  *  - `requestId`  — cancel just this call with `cancel({ requestId })`; peers
  *                   keep decoding.
  *
- * Non-cached completions and batches share the model's `parallel` admission
- * cap: with `parallel` in flight, admitting one more waits FIFO until a slot
- * frees.
+ * Completions and batches share the model's `parallel` admission cap: with
+ * `parallel` in flight, admitting one more waits FIFO until a slot frees.
  *
  * Run from packages/sdk:
  *   bun run examples/multi-job-completion.ts