QVAC-22704 feat[api]: multi-job continuous batching in the SDK

[donriddo]

🎯 What problem does this PR solve?

• The SDK admitted only one inference request per loaded model at a time, so multiple completion() / batchCompletion() calls serialized even on models loaded with parallel > 1, leaving the addon's continuous-batching throughput unreachable from the SDK.
• Cancelling one completion aborted the whole model, so a completion could not be cancelled while other completions were in flight on the same model.

📝 How does it solve it?

• Completion and batchCompletion share one admission lane per model, capped at that model's own parallel value. Single and batch requests compete first-come-first-serve for admission permits in that lane — one permit per request — and the addon then schedules the actual native sequence slots. Admission bounds concurrent requests, not native sequences: a multi-prompt batch takes one permit but can occupy several native slots, and a same-key cache-lock waiter holds a permit while doing no native work. (Tightening admission to weighted, slot-accurate accounting is a tracked follow-up.)
• The request registry gained per-request cap and lane overrides, so each model's concurrency comes from its own parallel config rather than a single global number.
• Admission always queues the surplus FCFS. A single-slot model (parallel = 1 or unset) still admits one request at a time and queues the rest — unchanged from before. An N-way model admits up to parallel concurrently and queues the surplus.
• Cancelling a completion or a batch routes to that request's own native job/group (addon cancelJob), leaving concurrent peers on the same model decoding. Both completionStream and batchCompletionStream advertise cancel: { scope: 'request' }.
• Completions that persist a disk KV cache share the same admission lane as every other completion, so they count toward parallel and keep their first-come-first-serve place. Same-file write safety comes from a per-cache-path lock inside the KV-cache session, held across the turn: two turns writing the same cache file serialize, while turns on different cache keys still decode concurrently. At parallel = 1 everything is already serial.

🧪 How was it tested?

CI run: https://github.com/tetherto/qvac/actions/runs/32268049857?pr=3682

• Registry unit tests: per-model cap, shared-lane first-come-first-serve for completion and batch, N-way concurrency, per-model scoping, and a cached completion sharing the lane without bypassing the cap or jumping the queue.
• Bare cancel-capability truth-table and queued-cancel tests against the published addon.
• Typecheck and eslint clean against the published @qvac/llm-llamacpp@0.44.0 types.
• Desktop E2E (completion + kv-cache + cancellation) via the MQTT harness against the published addon. completion-concurrent-overlap fires four independent completions on a parallel:4 model, requires engine avgConcurrentSeq > 1, and reports client decode-interval overlap as supporting diagnostics. kv-cache-concurrent-same-key proves same-file serialization. kv-cache-auto-concurrency separately requires cached-vs-cached overlap and mixed cached/plain overlap. cancel-isolates-concurrent-batches requires the survivor batch to make observable progress after the cancelled batch's acknowledgement.
• Runnable proof: examples/multi-job-completion.ts fires concurrent completions on a parallel > 1 model and cancels one by its request id while its peers decode to a clean finish.

🔌 API Changes

Additive API surface with no schema removals or signature breaks. Runtime admission and cache-administration behavior changes as described below.

const runs = prompts.map((p) => completion({ modelId, history: p, stream: true }))
const outputs = await Promise.all(runs.map((r) => r.final))

// Cancel one without disturbing its peers:
await cancel({ requestId: runs[0].requestId })

A model loaded with parallel = 1 (or unset) behaves exactly as before: same-model requests serialize FIFO.

Cache deletion now rejects key/model combinations that resolve to the cache root or broaden a targeted model deletion to the whole key directory. Omit modelId intentionally to delete every model under a key, or use { all: true } for a full cache wipe.

⚠️ Compatibility note — shared admission-lane queue

New behavior for LLM translate and finetune: this PR routes them into the shared llama.cpp completion lane, so they are now subject to that lane's per-model bounded wait queue. The 64-deep FIFO queue is shared across completion, batch completion, LLM translate, and finetune waiters per model; once 64 are waiting, the 65th queued request is rejected with RequestRejectedByPolicyError. NMT translate passes no concurrency cap, never enters the lane, and is unaffected.

Availability cost of the exclusive-writer finetune: while a finetune runs it holds the shared lane alone, so every completion, batchCompletion, and LLM translate on that model queues behind it for the finetune full duration. The per-model queue is bounded (64) but the wait is not - a queued reader can be blocked for the whole training run with no timeout. This is intentional; a timeout would turn ordinary inference into a new typed failure and needs an owner/QIP-approved duration and policy across every shared-lane kind. Do not share a model between finetune and interactive requests if you need inference responsiveness during training.

🏛️ QIP disposition

QIP triage is positive because this changes a stable public SDK scheduling/cancellation contract and the llama.cpp addon interaction.

One proposal is appropriate: Per-model multi-job admission and cancellation contract, covering request-count versus native-slot accounting, the shared lane, the fixed 64-entry queue, exclusive finetune behavior, and per-request cancellation. KV-cache lock implementation and the localized cancellation/shutdown race fixes do not need separate proposals.

🔗 Dependencies

• Requires @qvac/llm-llamacpp@^0.44.0 — the multi-job continuous-batching addon line (from QVAC-18397: Multi-job queue at addon-cpp and LLM (Needed for LLM Continuous Batching Optimizations) #3445), matching main's current pin. Published to npm.

---

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1216426887490068

[Copilot]

Pull request overview

This PR updates the SDK runtime to support multi-job continuous batching for llama.cpp models by admitting multiple concurrent completion() / batchCompletion() requests per model (up to that model’s configured parallel), and by switching completionStream cancellation to a per-request (job-scoped) cancel path so cancelling one run doesn’t abort its peers.

Changes:

• Add per-request overrides in the request registry (maxConcurrentPerModel, slotGroup, onOverflow) to enforce per-model concurrency and specialized serialization lanes.
• Update llama.cpp completion/batch handlers to (a) admit requests up to the model’s parallel, (b) reject overflow on single-slot models, (c) serialize disk KV-cache writes on a dedicated lane, and (d) route cancellation through response-level cancel for per-request completion cancels.
• Update unit/bare/e2e tests and docs, bump @qvac/llm-llamacpp to ^0.40.0, and add an example demonstrating concurrent completions + isolated cancellation.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
packages/sdk/test/unit/runtime/request-registry.test.ts	Adds/updates registry tests for per-request caps, lanes, and overflow behavior.
packages/sdk/test/bare/plugin-cancel-capability.test.ts	Updates cancel capability truth table for request-scoped completion cancels.
packages/sdk/server/bare/runtime/request-registry.ts	Implements per-request overrides for concurrency cap/lane/overflow in admission gating.
packages/sdk/server/bare/runtime/request-registry-singleton.ts	Updates default policies for shared completion/batch lane and adds cached KV serialization lane constant.
packages/sdk/server/bare/runtime/index.ts	Re-exports the cached-lane constant from the runtime entrypoint.
packages/sdk/server/bare/registry/model-registry.ts	Extends the run response interface to include a cancel() method.
packages/sdk/server/bare/plugins/llamacpp-completion/plugin.ts	Uses per-model parallel for admission, configures addon overflow behavior, and switches completionStream cancel scope to request.
packages/sdk/server/bare/plugins/llamacpp-completion/ops/completion-stream.ts	Routes abort to response.cancel() (job-scoped) and registers admitted responses for targeted cancellation.
packages/sdk/server/bare/plugins/llamacpp-completion/ops/batch-completion-stream.ts	Routes abort through batch response cancel (still model-scoped today, ready for future per-batch cancel handle).
packages/sdk/package.json	Bumps @qvac/llm-llamacpp dependency to ^0.40.0.
packages/sdk/examples/multi-job-completion.ts	Adds a runnable example showing concurrent completions and cancelling one by requestId.
packages/sdk/e2e/tests/shared/executors/completion-executor.ts	Updates E2E executor expectations for single-slot rejection contract.
packages/sdk/e2e/tests/shared/executors/cancellation-executor.ts	Updates cancellation E2E to reflect single-slot rejection + admitted run success.
.cursor/rules/sdk/request-lifecycle-primitives.mdc	Updates internal SDK lifecycle docs for request-scoped cancels and per-request registry overrides.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

License compliance — clean

No new dependency license findings in this PR.

Warn-only (shadow) mode — this check does not block merges yet.

Updated automatically by the canonical license compliance workflow.

NOTICE presence (advisory)

Missing NOTICE (advisory, does not block):

• ./.github/actions/release-merge-guard
• ./docs/website
• ./packages/ggml-coload-smoke
• ./packages/fabric/test/integration
• ./packages/inference-addon-cpp/mobile
• ./packages/sdk/e2e
• ./packages/llm-llamacpp/benchmarks/performance
• ./packages/llm-llamacpp/benchmarks/server
• ./packages/vla-ggml/sim/server
• ./packages/embed-llamacpp/benchmarks/performance
• ./packages/embed-llamacpp/benchmarks/server
• ./packages/asr-ggml/benchmarks/server

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (4)

packages/sdk/server/bare/runtime/request-registry.ts:482

• maxConcurrentPerModel is treated as an arbitrary finite number, so non-integers can accidentally over-admit. For example, if a handler passes parallel: 1.5, the gate will admit 2 requests because active is an integer and the check is active < maxConcurrent. Since model parallel is effectively a slot count, it should be clamped to an integer >= 1 here.

    // The per-request override (the model's own `parallel`) wins over the
    // per-kind default. A non-finite effective cap disables gating entirely; a
    // finite value is floored at 1 (matching `normalizePolicy`) so a stray 0
    // can't wedge the lane into queuing every request forever.
    const requestedMax = opts.maxConcurrentPerModel ?? policy.maxConcurrent
    if (!Number.isFinite(requestedMax)) {
      return { slotKey: undefined }
    }
    const maxConcurrent = requestedMax < 1 ? 1 : requestedMax

packages/sdk/server/bare/plugins/llamacpp-completion/plugin.ts:59

• getModelParallel() currently returns Number(config.parallel) || 1, which allows negative numbers and non-integers (e.g. 1.5). Because the request registry treats the cap as a numeric threshold, a non-integer can result in admitting more jobs than intended. Clamping to an integer >= 1 also aligns with the comment (“slot count”).

// The model's concurrent sequence slots. Missing / 0 / NaN all mean single-slot.
function getModelParallel(config: { parallel?: number | undefined }) {
  return Number(config.parallel) || 1
}

.cursor/rules/sdk/request-lifecycle-primitives.mdc:220

• The truth-table row for llamacpp-completion.batchCompletionStream still lists { scope: "model" }, but the handler now declares per-request cancel ({ scope: "request", hard: true }). Update the table to match the code and the pinned test.

| Plugin                       | Handler(s)                            | `cancel`                          |
|------------------------------|---------------------------------------|-----------------------------------|
| `llamacpp-completion`        | `completionStream`                    | `{ scope: "request", hard: true }`|
| `llamacpp-completion`        | `batchCompletionStream`               | `{ scope: "model", hard: true }`  |
| `llamacpp-completion`        | `translate`                           | `{ scope: "model", hard: true }`  |
| `llamacpp-completion`        | `finetune`                            | `{ scope: "model", hard: true }`  |

.cursor/rules/sdk/request-lifecycle-primitives.mdc:203

• This doc still says batchCompletionStream uses model-scoped cancel “pending a per-batch addon handle”, but the plugin now declares batchCompletionStream: { scope: "request", hard: true } and the truth-table test was updated accordingly. The bullet should be generic and not single out batchCompletionStream anymore.

This issue also appears on line 215 of the same file.

`cancel.scope`:

- `"request"` — addon targets a specific in-flight request by id. Framework can route `cancel({ requestId })` straight to the addon without colliding with siblings on the same model.
- `"model"` — addon cancels "whatever is currently running on this model" (`addon.cancel()` semantics). Under continuous batching this also stops concurrent peers on the same model, so it is a coarse fallback, not a per-request cancel. `batchCompletionStream` still uses it, pending a per-batch addon handle.
- `"none"` — addon does not expose a cancel surface. SDK falls back to soft-cancel (stop yielding, drop result, skip post-processing; C++ work runs to completion).

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (3)

packages/sdk/server/bare/runtime/request-registry.ts:482

• maxConcurrentPerModel can be a non-integer (e.g. config parallel: 1.5 or 2.3). With the current comparison (st.active < maxConcurrent) this effectively rounds up and can admit more concurrent requests than intended (e.g. 1.5 admits 2). Coerce the effective cap to an integer (and still clamp to >=1) so admission matches the configured slot count deterministically.

    const requestedMax = opts.maxConcurrentPerModel ?? policy.maxConcurrent
    if (!Number.isFinite(requestedMax)) {
      return { slotKey: undefined }
    }
    const maxConcurrent = requestedMax < 1 ? 1 : requestedMax

packages/sdk/server/bare/plugins/llamacpp-completion/plugin.ts:59

• getModelParallel() currently returns the raw Number(parallel), which can be negative or non-integer. That value feeds both rejectWhenBusy and the registry cap, so a config like parallel: 2.9 can accidentally behave like 3 slots and parallel: -1 can behave inconsistently. Normalize to a finite integer >= 1 (and treat missing/0/NaN/<1 as 1) to match the semantics described in the comment.

// The model's concurrent sequence slots. Missing / 0 / NaN all mean single-slot.
function getModelParallel(config: { parallel?: number | undefined }) {
  return Number(config.parallel) || 1
}

packages/sdk/server/bare/plugins/llamacpp-completion/plugin.ts:393

• When useCachedLane is true, the request moves to a different slotGroup with its own cap (1), so it no longer contends for the shared completion lane capped at parallel. That makes it possible to admit parallel plain completions plus one cached completion concurrently, pushing queueing back into the addon and breaking the PR description’s "one admission pool per model capped at parallel" and FCFS behavior across all completion requests.

          requestId: request.requestId ?? generateServerRequestId(),
          kind: 'completion',
          modelId: request.modelId,
          maxConcurrentPerModel: useCachedLane ? 1 : parallel,
          ...(useCachedLane && { slotGroup: LLAMACPP_COMPLETION_CACHED_SLOT_GROUP })

[github-actions]

Review Status

Current Status: ❌ PENDING
Approvals so far: none

Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member.

[github-actions]

QVAC E2E — android — ⚠️ no results

Config: suite=(none) · filter=(none) · exclude=(none)
Device pool: Private - Pixel 9 Pro - Android 15 — ⛅ available
View run · Artifacts: reports · Device Farm logs

The test job did not produce a results artifact (e.g. no device started within the start-timeout, or a job-level failure). Check the run and the device pool status above.

[github-actions]

QVAC E2E — windows — ✅ all tests passed (502/505, 2094s)

Config: suite=(none) · filter=(none) · exclude=(none)
View run · Artifacts: reports

[github-actions]

QVAC E2E — ios — ✅ all tests passed (368/505, 2028s)

Config: suite=(none) · filter=(none) · exclude=(none)
Device pool: Private - iPhone 16 - iOS 26 — ⛅ available
View run · Artifacts: reports · Device Farm logs

[github-actions]

QVAC E2E — linux — ✅ all tests passed (502/505, 1571s)

Config: suite=(none) · filter=(none) · exclude=(none)
View run · Artifacts: reports

[github-actions]

QVAC E2E — macos — ✅ all tests passed (502/505, 1069s)

Config: suite=(none) · filter=(none) · exclude=(none)
View run · Artifacts: reports