Skip to content

CHANGELOG.md

Latest commit

 

History

History
535 lines (352 loc) · 54.2 KB

CHANGELOG.md

File metadata and controls

535 lines (352 loc) · 54.2 KB
Raw

Qwen3.6-27B — Changelog

Dated history for Qwen3.6-27B configs in this repo. Combines the single-card and dual-card timelines (both were previously separate repos; consolidated here 2026-04-28).

2026-08-02 — dual-NVFP4: first native-FP4 validation, and an envelope correction

vllm/qwen-27b-dual-nvfp4 was authored blind and had never had a clean boot on the hardware it targets. #849 (@paulp83, 2× RTX 5090 sm_120, no power cap) is that boot, and it is a full gate.

Measured: verify-full 8/8 · verify-stress 7/7 (NIAH 240,636 = 91% of 262K, free margin 1,545 MB, Δ −18 MB across the entire ladder) · soak-continuous PASS (VRAM flat) · decode 168.0 narr / 215.7 code TPS (CV ~1%) · TTFT 69/77 ms · prefill 4,545 @10K / 3,919 @90K · MTP accept-len 4.0 at 100% per-position · peak 30,576 / 32,607 MiB per card (93.8%) · KV pool 14.68 GiB / 841,541 tok.

Quality — the arc's longest-standing open question is closed. 8-pack 117/150 think-off · 116/150 think-on (benchlocal v0.9.8; pass@3 117 / 125). ⚠️ This is not "+7 over" the 110/150 measured on the Ampere Marlin-fallback path in 2026-07-11 — those were scored on materially different benchlocal versions. The like-for-like modern control is @henrykrinkle01's v0.9.9 FP8 run, which also lands 117 think-off. Read together: native-FP4 activations cost nothing measurable on this model, which is exactly what the compose header had been carrying as unmeasured since the slug was authored.

Envelope correction (supersedes the 2026-08-01 derate). #847 responded to #838's OOM by setting util 0.92 → 0.85 and batched-tokens 8192 → 2048, both desk-derived from a crash log. #849 ran util 0.85 with batched left at 8192 and passed the full gate, which separates the two:

card 32,607 MiB
util 0.85 → vLLM budget 27,716 MiB physical headroom 4,891 MiB
util 0.92 → vLLM budget 29,998 MiB physical headroom 2,609 MiB
actually held 30,576 MiB = 2,860 MiB above its own budget

That ~2.9 GB overshoot is the un-profiled GDN activation peak (vllm#44209). It fits in 0.85's headroom and does not fit in 0.92's — so util was the binding knob and the batched-token cut was not. The GDN prefill scratch is still real and still linear in batched tokens (8192 → exactly the 96.00 MiB that appeared in #838's traceback), but at 0.85 that 96 MiB is noise: it was the straw, not the load.

Changed: MAX_NUM_BATCHED_TOKENS restored 2048 → 8192 (the derate was costing prefill — 3,919 tok/s @90K is measured at 8192); GPU_MEMORY_UTILIZATION stays 0.85; compose header, registry status_note and BENCHMARKS updated with the measured envelope. The "no clean community boot on target hardware" caveat is retired.

Unchanged — deliberately. Status stays ⚠️ Production w/ caveats and the vllm#50021 MTP exposure caveat stays. A passing gate does not retire a probabilistic crash: the same reporter hit #50021 at n=3 in #838, and #758's gradient shows n=3 surviving "many sessions in a day" rather than being safe. SPEC=off remains the reliability setting until the fix is inherited via a pin bump (v0.26.0 predates it).

Method note worth keeping: the derate that #849 corrected was explicitly labelled desk-derived in the compose header, which named a 2× 5090 owner as the validator and asked for a confirm-or-correct. That is the labelling paying for itself — the correction took hours, not a release cycle.

2026-07-06 — dual-max KV switch: int8_per_token_head → fp8

Switches vllm/qwen-27b-dual-max (compose: dual/fp8/mtp.yml) from int8_per_token_head KV cache to fp8.

2026-05-30 — beellama.cpp as a first-class compose engine (DFlash, single-card)

Onboards beellama.cpp (Anbeeld's llama.cpp fork — DFlash cross-attention spec-dec + SWA windowed KV) as a registry engine, with one single-card DFlash compose per model:

  • Qwen3.6-27B → beellama/dflash (port 8060): Unsloth Q5_K_S target + Anbeeld DFlash-IQ4_XS draft (--spec-type dflash), q5_0(K)/q4_1(V) KV, 102K ctx, -np 1. Anbeeld's "Precision combo". DFlash is tool-grammar-neutral here — the external drafter does not amplify the CodeAct attractor the built-in MTP head does (#237).
  • Gemma-4-31B → beellama/gemma-dflash (port 8061): Unsloth Q4_K_S target + Anbeeld DFlash-IQ4_XS draft, same q5_0/q4_1 KV + 102K ctx. The only single-card engine that does Gemma-4 windowed KV (big ctx) and spec-dec in one GGUF (ik-llama walls ~24K; mainline ~10 TPS).

Status: 🧪 Experimental — both reference a locally-built image ${BEELLAMA_IMAGE:-beellama-cpp:local} (upstream ships no pullable image; build via the fork's .devops/cuda.Dockerfile with CUDA_DOCKER_ARCH=86 + -DGGML_CUDA_FA_ALL_QUANTS=ON). Surfaced as (NA: experimental) in --list; launch is --force-gated.

Catalog wiring: new engine profile engines/beellama-local.yml; two DFlash GGUF drafters (anbeeld-qwen-dflash, anbeeld-gemma-dflash); compose_registry.py entries (kvcalc_key="SKIP" — llama.cpp-family); weights-map + hardware (q4_1) updates; INFERENCE_ENGINES + UPSTREAM notes.

Default-resolver invariant preserved: beellama is #1 in ENGINE_PREFERENCE[single], but its (NA) status + the absence of a DEFAULTS row means the resolver skips it → qwen3.6-27b single still resolves to ik-llama/iq4ks-mtp (asserted by test-model-default-resolver.sh). Auto-promotes to the single default only when a published image lands and the composes flip to production. Full suite green (only the pre-existing test-submit-bench.sh fixture failure remains).

2026-05-30 — Model-default resolver + user-pinnable defaults (<model>/default, --set-default)

Adds a two-layer default scheme on top of the existing <engine>/default map. <engine>/default stays the maintainer's recommendation (read-only to users); the new <model>/default is the user's preference — their .env pin if set, else a curated pick for the detected topology.

What changed:

  • compose_registry.py: two maintainer knobs next to DEFAULTS — RECOMMENDED_DEFAULT_MODELS (a short opt-in shortlist, not an exhaustive ranking; new models are not auto-added) and ENGINE_PREFERENCE (per-topology engine order: single = [beellama, ik-llama, llamacpp, vllm], dual/multi = [vllm, ik-llama, llamacpp, beellama]). Plus resolver helpers (curated_default_target, community_default_target stub, model_default_pin_key, engine_set/model_set, etc.).
  • registry-emit.sh: shared resolver model_default_target(root, model, topology) — the single injection point for both launchers. Precedence ladder: --variant (caller) → .env pin → community seam (returns None today) → curated ENGINE_PREFERENCE walk (skips non-functional (NA) slugs) → degradation (notice + nearest-lower topology, else a clear "pick explicitly" message — never crashes). Plus x_default_dispatch: X/default with X ∈ engine-set → engine rec; X ∈ model-set → model default; else error (engines and model-ids are disjoint).
  • switch.sh: <model>/default token; --set-default <slug> / --clear-default <model> (round-trip the .env pin CLUB3090_DEFAULT_<MODELID>); a "Defaults" view appended to --list (also standalone via --defaults) showing each model's resolved default + whether it's a user pin or curated.
  • launch.sh: bare invocation → first installed shortlist model → its <model>/default (no full wizard); a pinned fast-path ("Launch your default <slug>? [Y/n]"); a post-boot offer ("Make <slug> your default for <model>? [y/N]"). Any narrowing flag keeps the explicit wizard path.
  • Pin validation is warn + fall back, never blocking: unknown slug / wrong model / topology-mismatch / (NA) status → notice + curated default.
  • Docs: README single-card realign (ik-llama = fastest blessed single default, llama.cpp = cliff-immune alternative) + "pin your default" in Quick start; FAQ (extended switch-model entry + new "set my own default" Q); SINGLE_CARD / DUAL_CARD / GETTING_STARTED / ADDING_MODELS resolver notes; UPSTREAM beellama Docker-image row (gates beellama onboarding; auto-promotes to single default on catalog). New test scripts/tests/test-model-default-resolver.sh.

Builds on the slug health flag below (the resolver skips non-functional defaults). Verified: 45 entries unchanged, kv-calc calibration 17/17, full suite green (only the pre-existing test-submit-bench.sh fixture failure remains).

2026-05-30 — Slug health/availability flag (registry status + --list markers + launch gate)

Added a lifecycle/health flag to every registry slug so switch.sh --list and the launch/switch path are no longer blind to a compose's lifecycle stage. Previously status lived only in compose-header comments, which drifted — dual/autoround-int4/tq3-mtp-genesis.yml declared ✅ Working (with Genesis) even though the Genesis pin is parked/drifted and the compose won't boot clean.

What changed:

  • compose_registry.py: _entry() gains a keyword-only status (default production) + status_note, validated against the canonical enum (production · caveats · experimental · preview · upstream-gated · deprecated). Added a compose_header_status() helper that maps a compose's profile-schema Status: emoji to the enum.
  • Swept every compose Status: header to a canonical enum value (decision §12.5/§13): non-conforming strings normalized — e.g. genesis ✅ Working (with Genesis) → ⏸️ Upstream-gated, ⛔ TOMBSTONED → 🗑️ Deprecated, 🔵 v0.7.3 ONBOARDING → 🧪 Experimental, 🔵 PREVIEW → 👁️ Preview, ⭐ Code-optimized → ✅ Production.
  • Re-flagged the non-functional slugs: all *genesis* + gemma-4-31b single fp8 → upstream-gated; carnice → caveats; qwopus + qwen-a3b-preview → preview; the bf16/int8 A/B composes, llamacpp bounded-thinking, all PRISM/APEX eval lanes, and the gemma-4-26b-a4b onboarding composes → experimental; tq3-mtp → deprecated.
  • registry-emit.sh now emits status + status_note on the VARIANT row (last two fields); both loaders + the parity tests read the extended field list.
  • switch.sh --list shows a health marker: production unmarked · caveats → (caveats) · the (NA) set → (NA: <word>). Model·topology grouping (#264) preserved.
  • Launch/switch gate: production launches silently, caveats launches with a one-line notice, the (NA) set warns + requires --force. launch.sh surfaces the same flag before delegating to switch.sh.
  • New drift-guard test scripts/tests/test-compose-status-drift.sh: asserts every registry status ∈ enum, every compose header maps to the enum, and the two agree.

Foundational for the upcoming model-default resolver (it must skip non-functional slugs when picking a curated default) and a safety fix on its own — users could previously boot the pin-drifted Genesis composes blind. Verified: 45 entries unchanged, kv-calc calibration 17/17, full test suite green (only the pre-existing test-submit-bench.sh fixture failure remains).

2026-05-29 — vLLM nvlink-* dual composes removed (NVLink auto-detected)

The four dual/autoround-int4/nvlink-*.yml composes (nvlink-fp8-mtp, nvlink-turbo, nvlink-dflash, nvlink-dflash-noviz) are removed, along with their vllm/dual-nvlink* launch slugs. They were thin extends: stubs whose only override was NVLINK_MODE=force_on — redundant since every dual compose now auto-detects NVLink at boot via detect_nvlink.sh (NVLINK_MODE=auto → flips on NCCL_P2P_LEVEL=NVL + custom-all-reduce when a bridge is present, else NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce). NVLink rigs now get the same path from the base dual compose with no separate slug; force it explicitly with NVLINK_MODE=force_on scripts/switch.sh vllm/dual if auto-detect misses. The historical NVLink bench rows (JusefPol PR #31, danbedford #74/#92/#96) are preserved in BENCHMARKS.md. The earlier dated entries below that describe adding these composes are kept as append-only history.

2026-05-07 — llamacpp/default adds --reasoning-format none (opencode unblock)

@syangsao reported opencode hangs indefinitely against llamacpp/default despite the server returning 200 with content tokens generated successfully (#97). Diagnosis via curl SSE capture: every delta was in the reasoning_content field, never content — Qwen3.6's thinking mode emits <think> blocks that llama.cpp's peg-native parser routes to reasoning_content by default. opencode (and most simple OpenAI-compat clients) ignore reasoning_content and wait indefinitely for content deltas that never arrive.

Fix: added --reasoning-format ${REASONING_FORMAT:-none} to models/qwen3.6-27b/llama-cpp/compose/docker-compose.yml and single/concurrent.yml. Default none collapses thinking into the content stream — opencode and other simple clients work out-of-box. Power users wanting reasoning_content separation set REASONING_FORMAT=auto in .env or shell.

Cross-rig validated by @syangsao (1× 3090 water, 330W cap, b9014 image): Fix 2 path (chat_template_kwargs.enable_thinking: false in opencode config) confirmed unblocked. Fix 1 (server-side flag) is the same root-cause solution applied at the compose layer so every contributor doesn't hit this. Bench numbers from his unblocked session: 28.88 TPS decode / 741 TPS prompt at 45K accumulated context — within Q3_K_XL Qwen3.6 + DeltaNet hybrid expectations.

Companion observation: DeltaNet hybrid prevents prefix-cache reuse across turns ("forcing full prompt re-processing due to lack of cache data ... SWA or hybrid/recurrent memory"). Each multi-turn opencode interaction does full prefill — known characteristic, not a regression.

2026-05-07 — NVLink + DFlash compose variants added (#92, #96)

Two new community-contributed composes from @danbedford for 2× 3090 with NVLink bridge:

  • dual/nvlink-dflash.yml (port 8018) — 185K ctx + DFlash N=5 + vision. NCCL P2P over NVLink + custom_all_reduce ENABLED. Drops expandable_segments=True (NVLink startup-crash fix from JusefPol/PR #31). Bench (2× 3090 NVLink, 230W cap): 101.55 / 163.33 narr/code wall TPS (CV 1.8%/1.9%), +17% narr / +16% code over his PCIe dual-dflash baseline of 86.62 / 141.02. PASSES verify-full 8/8 + verify-stress 7/7 + continuous soak (0 err, 100% retention).

  • dual/nvlink-dflash-noviz.yml (port 8019) — text-only variant of the above. Drops MoonViT to free ~0.78 GB/card → max_model_len pushed from 185K to 188K. Empirically determined: 189K had 1/3 success rate (flaky on fresh reboot), 188K is the stable ceiling. Bench: 103.24 / 167.45 narr/code wall TPS (CV 2.2%/3.6%), +17% narr / +17% code over PCIe dual-dflash-noviz baseline (88.31 / 142.79). PASSES same validation chain.

Both variants registered in scripts/launch.sh and scripts/switch.sh. Sibling-list headers updated across dual.yml, dual-nvlink.yml, dual-dflash.yml, dual-dflash-noviz.yml for cross-reference. Marked community-contributed, experimental in headers.

Note: both composes use --tool-call-parser qwen3_coder but are direct-cmd (no entrypoint script), so they don't currently receive the qwen3coder_tool_parser_deferred_commit.py sidecar shipped 2026-05-07 for issue #72. Consistent with the existing direct-cmd pattern (dual.yml, dual-dflash.yml also lack the sidecar). If the SSE-silence bug fires on these variants, follow-up PR can add an entrypoint script.

2026-05-06 — PYTORCH_CUDA_ALLOC_CONF override knob added to 14 composes

Follow-up to the v7.72.2-uplift pin bump: a single-card RTX 3090 Ti rig on WSL2 (driver 596.36) hit gptq_marlin_repack boot crashes (CUDA driver error: device not ready) on the new nightly. The minimal compose (no Genesis, no spec-decode, no TQ3 KV) reproduced cleanly with just --quantization auto_round, and CUDA_LAUNCH_BLOCKING=1 did not move the failure site (rules out async-residual error from a prior kernel).

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False resolves the crash. This is the same workaround that already addresses JusefPol's NVLink boot-crash report (PR #31, hardcoded in the dual-nvlink*.yml composes). The exact failing call hasn't been isolated.

All 14 single-card and PCIe dual-card composes now expose PYTORCH_CUDA_ALLOC_CONF as a ${...}:- override knob (defaults preserved); affected users drop PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False into a .env. The two dual-nvlink*.yml composes are unchanged.

Single-observation note: weight-load on a fresh boot (caches cleared) was 32 sec with expandable_segments:True and 13 sec with expandable_segments:False on this rig. Not a controlled benchmark — cache state and other factors weren't held constant. Suggestive only.

See docs/HARDWARE.md for the full failure signature and override recipe.

2026-05-05 — v7.72.2-uplift: Genesis pin bump + sidecar consolidation ⭐

Aligns Qwen3.6-27B configs with Genesis v7.72.2 (pin SHA 7b9fd319) and Sander's PROD-validated vLLM pin (nightly-01d4d1ad375..., allowlist entry #2).

Pin bumps:

  • scripts/setup.sh GENESIS_PIN: 2db18df (v7.69) → 7b9fd319 (v7.72.2)
  • All 16 composes' vLLM image: nightly-7a1eb8ac2ec... → nightly-01d4d1ad375...

Sidecars retired — 6 local .py patches deleted from vllm/patches/, all confirmed redundant on v7.72.2: patch_inputs_embeds_optional.py (PN35 supersedes), patch_pn30_dst_shaped_temp_fix.py (PN30 v7.68), patch_pn25_genesis_register_fix.py (PN25), patch_tolist_cudagraph.py (P78), patch_workspace_lock_disable.py (PN34), patch_pr40798_workspace.py (research artifact).

PN59 added to 7 Genesis-loaded composes (docker-compose.yml, dual-turbo.yml, long-text.yml, long-text-no-mtp.yml, long-vision.yml, bounded-thinking.yml, tools-text.yml) for consistency.

dual/docker-compose.yml left intentionally Genesis-free as a debugging fallback for cross-engine bisect.

dual-turbo bench (2× 3090, single-stream): 81.21 narr / 108.20 code wall TPS (5 measured runs each, CV 2.3%/0.9%), AL 3.46. VRAM dropped from 22.1 GB/card → 20.0 GB/card (PN35 native fold + Sander's audit-pass cleanups).

Cross-rig PN59 finding: single-card 24 GB Cliff 2b unchanged on long-text.yml despite v7.72.2's PN59 streaming-GDN orchestrator. Filed Sandermage/genesis-vllm-patches#22 with reproducer + 4 fix proposals.

v7.72.1 closes #57 (lex's xgrammar-patternProperties fire on long-prompt agentic IDE traffic).

See cross-cutting CHANGELOG.md entry for the full narrative + bench delta table; vllm/patches/README.md for what's load-bearing now.

2026-05-04 — Carnice-V2-27B + BF16 MTP overlay — new compose variant ⭐

Adds dual/carnice-bf16mtp.yml: kai-os/Carnice-V2-27b (Hermes-style agentic fine-tune of Qwen3.6-27B) quantized to INT4 via delta-merge of Lorbus's AutoRound grid, with a BF16 MTP overlay for clean spec-decode acceptance.

Key findings from the diagnostic push:

  • Hypothesis B (MTP quant-grid mismatch) accounted for ~70% of the AL gap. Un-quantizing 7 mtp.layers.0.* projections (BF16 overlay) recovered AL from 2.0 → 3.0.
  • Tool-call format: Carnice's Hermes-style template used XML, but vLLM's --tool-call-parser hermes expects JSON. Patched chat template instructs JSON output inside <tool_call> tags. Vendored at patches/carnice-chat-template.jinja.
  • Full 262K context confirmed (22,246 MiB/card), 2 streams, same fp8 KV + MTP n=3 as dual.yml.

Validation:

  • verify-full.sh: 7/8 PASS (thinking test lenient — Carnice is concise, not verbose)
  • verify-stress.sh: 6/7 PASS (needle recall at ≥60K — model-level GDN attention ceiling)
  • bench.sh (n=5): 71.75 narr / 80.35 code wall TPS, MTP AL 3.02-3.14, TTFT 141ms
  • soak-test.sh (8×3 turns): PASS — 0 MiB growth, 0 errors, 101.6% TPS retention

Compose: dual/carnice-bf16mtp.yml

2026-05-03 late PM — multi4-dflash.yml TP=4 DFlash validated on 4× RTX 3090 PCIe ⭐

Adds multi4/dflash.yml, a 4-card full-context DFlash variant validated on Whamp's 4× RTX 3090 PCIe rig for club-3090 discussion #26. This is a capacity / 262K-code variant, not a replacement for the faster 2-card DFlash short-prompt path.

Config accepted by vLLM pre-check:

  • tensor_parallel_size=4
  • max_model_len=262144
  • max_num_seqs=2
  • max_num_batched_tokens=8192
  • dtype=bfloat16, FP16/default KV (required by DFlash on Ampere)
  • speculative_config={"method":"dflash","num_speculative_tokens":5}
  • reported GPU KV cache size: 207,264 tokens
  • reported max concurrency at 262K/request: 2.27×

Validation:

  • Boot: clean, ready after 375s on a warm image/model cache.
  • verify-full.sh: PASS.
  • verify-stress.sh: PASS 7/7. Canonical Cliff 2 probe 7 recalled both large needles: 58,570 tokens and 91,070 tokens.
  • bench.sh: 64.00 narrative / 104.40 code wall TPS (CV 2.8% / 3.0%), TTFT 143ms / 164ms.
  • DFlash AL during code bench: last three log samples 4.43 / 4.37 / 4.35.
  • Peak VRAM during bench: 21,960 MiB/card.

Interpretation: TP=4 DFlash gives a useful code-speed uplift over multi4.yml (104 vs 76 code TPS) while retaining full 262K admission, but PCIe TP=4 allreduce keeps it below the 2-card DFlash variants' raw single-stream TPS. Use it for 4-card, full-context, code-heavy work with two admitted streams.

2026-05-03 PM — multi4.yml TP=4 baseline validated on 4× RTX 3090 PCIe ⭐

Adds multi4/docker-compose.yml, a measured 4-card fp8/MTP baseline derived from dual.yml by scaling tensor parallelism and streams from 2 → 4. Validation came from Whamp's 4× RTX 3090 PCIe rig in club-3090 discussion #26.

Config accepted by vLLM pre-check:

  • tensor_parallel_size=4
  • max_model_len=262144
  • max_num_seqs=4
  • max_num_batched_tokens=8192
  • kv_cache_dtype=fp8_e5m2
  • reported GPU KV cache size: 483,200 tokens
  • reported max concurrency at 262K/request: 6.77×

Validation:

  • Boot: clean, ready after 355s on a warm image/model cache.
  • verify-full.sh: PASS after warm retry (first Paris request hit a cold-path 30s script timeout; direct retry returned HTTP 200 in 0.2s and full rerun passed).
  • verify-stress.sh: PASS 7/7. Canonical Cliff 2 probe 7 recalled both large needles: 58,569 tokens and 91,070 tokens.
  • bench.sh: 63.01 narrative / 76.25 code wall TPS (CV 2.1% / 4.0%), TTFT 111ms / 132ms.
  • MTP AL during code bench: last three log samples 3.42 / 3.53 / 3.62.
  • Peak VRAM during bench: 23,494 MiB/card.

Interpretation: TP=4 gives the first published 4×3090 Cliff 2 boundary data and higher full-context concurrency headroom, but single-stream TPS is lower than TP=2 on PCIe-only allreduce (published TP=2 fp8/MTP baseline is ~69 / 89 TPS). Use multi4.yml for 4-card capacity / Cliff 2 margin, not for fastest single-user short-prompt decode.

2026-05-02 PM — Genesis v7.69 + vllm#35975 backport — Cliff 2 60K CLOSED ⭐⭐

Genesis pin bump fc89395 (v7.66) → 2db18df (v7.69 dev tip). All three v7.66/v7.68 regressions we surfaced upstream landed in v7.69, plus a local backport of vllm#35975 brings the Cliff 2 single-prompt envelope to 60K cleanly on TQ3 + MTP K=3 at 24 GB. Two shippable single-card recipes ship at this pin.

v7.69 closes upstream:

Patch What Status before v7.69 Status in v7.69
PN30 v7.68 part3 DS conv state row-stride fix (replaces our dst-shaped sidecar) drift-markers too generic — silent re-fail landed clean, our sidecar drops
P103 worker self-install FLA Cliff 2 chunked fwd_h+fwd_o orchestrator survives exec vllm serve setattr lost on worker spawn → "rebound at 0 caller sites" self-install hook in chunk.py, fires on TP=1
PN32 v1 GDN chunked-prefill threshold + chunk size env-vars not yet shipped landed (PN32_GDN_CHUNKED_PREFILL=1, PN32_GDN_CHUNK_SIZE=8192, PN32_GDN_CHUNK_THRESHOLD=16384)

Codex P103 cu_seqlens gate fix queued for v7.70: filed Genesis #18 — _single_seq_cu detection lets cu_seqlens.shape[0] == 2 enter the chunked path. Diagnostic logging on dev205 showed q.shape[1] always 4128 (capped by vLLM max_num_batched_tokens for spec-decode K=3) so MAX_T=16384 never engages on real serving — gate-redirect works but doesn't fire under our admission cap. Cliff 2 in practice is residency, not gate logic: 50 MiB OOM after 394 successful T=4128 chunks → cumulative state filling 22.96 GiB of 23.56 GiB total. Lower mem-util buys activation headroom by spending KV capacity. Three distinct ceilings clarified per Codex round 2: declared max_model_len (admission), safe single-prompt prefill length (Cliff 2), concurrency capacity.

Local backport added: vllm#35975 (skip inputs_embeds GPU buffer for text-only models). Two-site regex text-patch via patch_inputs_embeds_optional.py:

  • gpu_model_runner.py: wraps self.inputs_embeds = self._make_buffer(...) in if self.supports_mm_inputs or self.enable_prompt_embeds:
  • llm_base_proposer.py: wraps self.inputs_embeds = torch.zeros(...) in if self.supports_mm_inputs:

Measured ~444 MiB freed on text-only paths (claimed ~64 MiB upstream — claim assumes a smaller config; our 180K + MTP K=3 path benefits more).

Mem-util matrix on long-text 180K + MTP K=3 (60K stress):

mem-util #35975 60K result
0.95 no OOM (Cliff 2)
0.95 yes OOM (Cliff 2)
0.92 yes PASS — 643s wall
0.93 yes PASS — 623s wall ⭐ shipped

Two shippable Cliff-2-closed variants:

Variant max-model-len mem-util MTP K TPS regime Single-prompt envelope Use when
long-text.yml (Balanced MTP) ⭐ 180000 0.93 3 50 narr / 67 code (cold short prompt); decode wins from MTP at high accept 60K PASS @ 623s; 90K indeterminate (curl >25 min) default for steady-state agent + chat
long-text-no-mtp.yml (Max-context) 200000 0.95 off ~33 narr / ~40 code (no spec-decode) 60K PASS @ 537s one-shot >50K input where you can wait + don't need MTP

Both top out at the 60K hardware-physical wall on 24 GB single-card. 90K MTP-off attempt didn't complete in 25 min (HTTP 000 at 1500s curl); not shipped, not failed — indeterminate.

Sidecars dropped (3, all closed natively in v7.69):

  • patch_pn25_genesis_register_fix.py → covered by v7.66 PN25 + v7.69 worker-spawn registration
  • patch_pn30_dst_shaped_temp_fix.py → covered by v7.69 PN30 part3
  • patch_workspace_lock_disable.py → covered by v7.69 PN34_WORKSPACE_LOCK_RELAX env-gate

Sidecars retained on master (2):

  • patch_inputs_embeds_optional.py (NEW) — backport of vllm#35975. Drop when upstream merges.
  • patch_tolist_cudagraph.py — unchanged.

Genesis env bundle (long-text.yml):

GENESIS_ENABLE_P103=1
GENESIS_ENABLE_PN32_GDN_CHUNKED_PREFILL=1
GENESIS_PN32_GDN_CHUNK_SIZE=8192
GENESIS_PN32_GDN_CHUNK_THRESHOLD=16384
GENESIS_FLA_FWD_H_MAX_T=16384
GENESIS_ENABLE_PN34_WORKSPACE_LOCK_RELAX=1
GENESIS_VLLM_SSM_CONV_STATE_LAYOUT=DS

Branch v7.69-cliff2-test merged to master at commit 15b84df. Per-round bisect log + mem-util sweep at results/v0.20-migration/v769-codex-r1-test.summary (404 lines, six bisect rounds).

Cross-rig contributions filed:

2026-05-02 — Genesis v7.66 + Cliff 1 mech B closed ⭐

Genesis pin bump 753344b → fc89395 (v7.66 dev tip). Cliff 1 mech B closed across all 4 TQ3 composes via two local backports:

PN25 v3 import-time backport (patch_pn25_genesis_register_fix.py): Sander's PN25 mechanisms — both v7.65 @torch.library.custom_op (which crashes on infer_schema inside dynamo trace) AND v7.66 direct_register_custom_op + Library("genesis", "FRAGMENT") (which crashes on instantiate_user_defined_class_object inside dynamo trace) — fail on TP=1 spawn. Our v3 text-patches vllm/model_executor/layers/activation.py to register the op at module-import time, BEFORE any trace context exists. Survives both Sander mechanisms because it sidesteps registration-during-trace entirely.

PN30 dst-shaped temp fix (patch_pn30_dst_shaped_temp_fix.py): Sander's PN30 a9977d8 corrupts DS conv state row strides by raw-memcpying a compact .contiguous() source-tail into a strided destination. Our fix builds a destination-shaped temp inside collect_mamba_copy_meta (where both source AND destination block IDs are known) and does the strided copy correctly. Diagnosis credit: ChatGPT/Codex CLI cross-check.

Validation matrix (v7.66 + local sidecars, verify-stress.sh 7-probe ladder):

Compose Ctx mem-util Probes Failure
long-text 180K 0.95 6/7 Cliff 2 architectural
long-vision 145K 0.95 6/7 Cliff 2 architectural
bounded-thinking 180K 0.95 6/7 Cliff 2 architectural
dual-turbo (TP=2) 262K 0.85 6/7 Cliff 2 architectural

Backoff from 214K + 0.985 → 180K + 0.95 (long-text/bounded-thinking) and 198K + 0.98 → 145K + 0.95 (long-vision) was needed to give activation headroom for the PN12+PN25 FFN pool residence + PN30 dst-shaped temp lifecycle. Vision tower's persistent ~1 GB tightens long-vision further.

Sander's v7.66 PN33 partial: PN33 (default ON) closes BOOT-time profile_run workspace_lock issue, but the runtime decode workspace_lock at turboquant_attn.py:1350:_decode_attention still fires on TP=1. Cross-rig data sent to Sander via discussion #19 reply.

Sander's v7.66 PN31: still doesn't fit on 24 GB. Per-shape persistent buffers + PN12+PN25 pool residence outpace activation budget at chunk_fwd_o. Lower mem-util (0.95) is sufficient to close the 25K tool-RETURN path PN31 was designed to fix without needing PN31 itself.

Sidecars retained on master (4):

  • patch_pn25_genesis_register_fix.py (PN25 v3 import-time, TP=1 only)
  • patch_pn30_dst_shaped_temp_fix.py (replaces Sander's compact .contiguous())
  • patch_workspace_lock_disable.py (PN33 narrowed but didn't close runtime decode path)
  • patch_tolist_cudagraph.py (cudagraph capture fix, unchanged)

Per-config + cross-rig results in results/v0.20-migration/v766-pin-results.summary and the per-compose *-pn30.summary files.

2026-05-01 PM — vLLM v0.20 + Genesis v7.65 dev tip migration ⭐

Master pin migration from vllm-openai:nightly-07351e088... (0.19.2rc1.dev205) + Genesis v7.64 (64dd18b) to vllm-openai:nightly-7a1eb8ac2ec... (0.20.1rc1.dev16+g7a1eb8ac2) + Genesis v7.65 dev tip (commit d89a089). v0.20's revised TQ FA prefill paths (vllm#40092) and Genesis v7.65's PN26b sparse-V kernel + PN17 FA2 lse-clamp + P38B/P15B in-source hooks together close Cliff 1 mech B sub-mechanisms that forced the dev205 backoffs. Three of our local sidecars (patch_pn12_ffn_pool_anchor.py, patch_pn12_compile_safe_custom_op.py, patch_fa_max_seqlen_clamp.py) replaced by Genesis-native equivalents.

Sidecars retained:

  • patch_workspace_lock_disable.py (NEW) — relaxes vllm#39226 strict assertion to one-shot WARNING. Sandermage's P98 covers the same surface but auto-skips on v0.20 (drift-marker false-positive). Drop when Sandermage ships marker fix.
  • patch_tolist_cudagraph.py — unchanged.

Sidecars dropped:

  • patch_pn12_ffn_pool_anchor.py → covered natively by PN12 on v0.20
  • patch_pn12_compile_safe_custom_op.py → covered by Genesis PN25
  • patch_fa_max_seqlen_clamp.py → covered by PN17 + P15B

Mamba block_size cap fix: v0.20 enforces long_prefill_token_threshold >= block_size; on hybrid Mamba+TQ3 the engine forces block_size=4128. Bumped GENESIS_PROFILE_RUN_CAP_M and GENESIS_PREALLOC_TOKEN_BUDGET from 4096 → 4128 across all 5 main composes.

Restored ceilings (vs dev205 backoff):

Variant Before (dev205+v7.64) After (v0.20+v7.65 dev) Δ
long-text.yml 185K + 0.975 214K + 0.985 +29K (+16%)
long-vision.yml 140K + 0.95 198K + 0.98 +58K (+41%)
bounded-thinking.yml 185K + 0.975 214K + 0.985 +29K (+16%)
tools-text.yml 75K + 0.97 (fp8) 75K + 0.97 (unchanged) flat
dual-turbo.yml 262K + 0.85 262K + 0.85 (full v7.65 PROD env-vars) flat ctx, +9% TPS

Bench results (n=5, 3 warmups + 5 measured, canonical narr+code prompts):

Variant Narr wall_TPS (CV) Code wall_TPS (CV) TTFT AL Avg accept VRAM KV pool tokens
long-text.yml 214K 49.74 (2.6%) 67.39 (2.7%) 154/155 ms 3.34-3.51 78-84% 23.4 GB 284,832 (1.03×)
long-vision.yml 198K 50.32 (2.3%) 66.12 (4.1%) 159/158 ms 3.40-3.56 79-85% 22.3 GB 264,192 (1.02×)
bounded-thinking.yml 214K 49.77 (1.4%) 65.80 (2.3%) 155/154 ms 3.25-3.61 75-87% 21.7 GB 284,832 (1.03×)
tools-text.yml 75K (fp8) 53.32 (2.3%) 69.66 (1.4%) 150/153 ms 3.53-3.59 84-87% 22.2 GB 104,000 (1.05×)
dual-turbo.yml 262K (TP=2) 58.33 (2.9%) 76.01 (4.5%) 112/110 ms 3.39-3.51 79-84% 19.8 GB/card 1,523,232 (4.67×)

Concurrent throughput on dual-turbo (canonical code prompt, 2 runs per stream):

Streams Total TPS Per-stream mean Per-stream CV Speedup
1 74.03 73.99 3.7% 1.00×
2 128.74 65.57 14.1% 1.74×
3 126.52 55.41 31.9% 1.71×
4 269.03 74.05 3.1% 3.63×

n=4 lands at near-single-stream per-stream TPS — true parallel decoding of full-context streams, not interleaved. The n=2/n=3 dips are scheduler artifacts on small bench sizes (high CV at n=3 confirms interleave behavior). Practically: dual-turbo serves either 1 stream at 76 TPS or 4 streams at 269 TPS aggregate.

Validation: verify-full ✅ 8/8 on every variant. verify-stress 33K AND 50K tool-prefill ✅ PASS on every variant (the 50K cliff that fired on EVERY dev205 config no longer reproduces). Branch v0.20-migration; bench captures at results/v0.20-migration/.

2026-04-30 PM — Cliff 1 closes; long-text 218K + long-vision 198K

PN12 was silently no-op'd on dev205+ via anchor drift (same bug class as P101). Genesis apply_all reported "PN12 applied" while live vllm/model_executor/layers/activation.py retained the vanilla SiluAndMul.forward_cuda. Local sidecar patch_pn12_ffn_pool_anchor.py repairs it; bundled Genesis tree carries the fix via PR #13. Combined with local patch_fa_max_seqlen_clamp.py (P104 FA softmax_lse clamp), Cliff 1 closes on TQ3 paths.

New shipped ceilings:

  • long-text.yml: 205K → 218K at 0.985 mem-util (no vision, no override). Engine ceiling vLLM-reported 218K. Verify-stress + verify-full pass; MTP AL 2.66; VRAM 23.7/24 GB.
  • long-vision.yml: 192K → 198K at 0.98 mem-util (vision on). Engine ceiling vLLM-reported 198K. 0.985 + vision reopens Cliff 1 (more goes to KV at the cost of activation budget; vision tower's persistent ~1 GB makes 0.98 the right balance).
  • --num-gpu-blocks-override 50 no longer needed at 0.985 — anchor-fixed PN12 cuts allocator churn enough that natural activation budget at higher mem-util is sufficient on text-only path.
  • 0.99 mem-util ruled out — driver/system reserves ~440 MiB; vLLM startup check fails at 0.99.

Cliff 2 unchanged. Single-prompt >50–60K still OOMs in DeltaNet GDN. Both long-* variants stay "steady-state accumulation across many turns, not single-shot big prompts."

Variants stay distinct: docker-compose.yml (48K, below both cliffs, fast boot) and tools-text.yml (FP8 path for IDE agents) remain valuable for their respective use cases. Four-variant menu kept; the long-* options now ship at higher ceilings.

Branch cliff1-fa-clamp carries the changeset; commits 41eabac (PN12 sidecar) → f3e5b52 (218K bisection) → 26e5f65 (docs).

2026-04-29 — Genesis v7.62.x + PN8 enabled on FP8 paths

Sandermage shipped Genesis v7.62.x (commit 917519b) on 2026-04-29 with PN8 (MTP draft online-quant propagation — backport of vllm#40849) targeting the FP8+MTP memory-headroom problem. We benched the patch across all 5 single-card composes that use Genesis:

Compose KV mem-util PN8 effect TPS Δ Verdict
tools-text.yml (75K, fp8) fp8 0.97 −900 MiB at boot · Cliff 1 closes ⭐ −7% code PN8 enabled
fast-chat.yml (20K, fp8) fp8 0.95 −800 MiB at boot −4.7% code PN8 enabled
docker-compose.yml (48K, TQ3) TQ3 0.92 no-op (already plenty of headroom) −3% / −5% PN8 not enabled
long-vision.yml (192K, TQ3) TQ3 0.98 KV pool +230 MiB, engine ceiling 192K → 198K, but Cliff 1 still fires −5% PN8 not enabled (commented in env, opt-in)
long-text.yml (205K, TQ3) TQ3 0.98 no effect (engine ceiling capped by block-size divisor at 206K) not benched PN8 not enabled

Why split-decision: the Cliff 1 OOM that ampersandru hit on long-vision.yml is an FFN intermediate-buffer activation peak (138 MiB allocate at intermediate_size=17408 × max-num-batched-tokens=4128), not a draft-model footprint. PN8's quant-config propagation doesn't reach that buffer on TQ3 paths. On FP8 paths the draft head's own footprint shrinks meaningfully — that's where the win is.

Cross-rig data + analysis posted to Sandermage: single-3090 #1 comment 4343317153.

Other v7.62.x items relevant to us (not yet benched here):

  • PN11 (Quentin-M, vllm#41142 streaming tool-call IndexError fix) — applies cleanly via the auto-detected REC; planned to enable in tools-text + fast-chat next pass.
  • TurboQuant k8v4 unlocked on hybrid GDN via P4 + P98 — Sandermage's A5000 measurement +1.9%; we'll bench on dual.
  • Per-GPU recommendation system (vllm/_genesis/gpu_profile.py) — boot log now lists [REC]/[OFF] per patch on this card. Nice ergonomics.

2026-04-28 (post-launch) — llama.cpp Q3_K_XL + Docker compose + stress-test findings + VRAM diagram

  • First measured TPS for UD-Q3_K_XL on this stack: 21.22 narr / 20.79 code @ 262K context + vision (single 3090, q4_0 KV). VRAM 20.17 GB / 24 GB at boot. Lower than memory's 28.5 baseline (Q4_K_M, 2026-04-23 on llama.cpp commit 9ab47e7d8) — investigating mainline regression vs current 0d0764dfd. ngram-mod path measured at 22.04 / 26.11 (+25% on code, draftless via --spec-type ngram-mod).
  • llama.cpp Docker compose at models/qwen3.6-27b/llama-cpp/compose/:
    • docker-compose.yml — single slot, 262K ctx, q4_0 KV, vision via mmproj. Uses ghcr.io/ggml-org/llama.cpp:server-cuda.
    • single/concurrent.yml — 4 parallel slots, 192K ctx pool, vision. Multi-tenant variant.
  • All three llama.cpp configs pass verify-full + verify-stress on this stack. Crucial finding: llama.cpp R1 (Q4_K_M @ 262K + q4_0 KV), Q3_K_XL @ 262K + vision, and Q4_K_M + ngram-mod @ 32K all clear the 90K needle ladder + 25K tool-prefill checks. No Cliff 1, no Cliff 2 — the prefill OOMs that bite vLLM single-card 192K configs don't fire in llama.cpp on this model. Trade is the ~2-3× lower TPS (21 vs 51-55 vLLM). Reframes our launch positioning around "vLLM dual = max throughput, llama.cpp single = max robustness." Single feature gap: llama.cpp doesn't peel <think> into reasoning_content (parser issue, not model). Tool calling, streaming, vision, output quality all clean on --jinja.
  • models/qwen3.6-27b/README.md — added "VRAM allocation across configs" section with embedded docs/img/vram-budget-dual.svg. Per-card stacked bars across 7 configs (3 single, 4 dual) showing weights / KV / vision / DFlash draft / activations / free headroom on the 24 GB budget. Visualizes the TP=2 unlock concretely.
  • models/qwen3.6-27b/llama-cpp/README.md — quant table updated. UD-Q3_K_XL marked ⭐ as our default with citation to Benjamin Marie's Kaitchup Q3.6-27B GGUF eval — independent H100-validated pick of Q3_K_XL as the optimal accuracy/efficiency/footprint balance, complementary to our 3090 speed measurements.

2026-04-28 — Split verify-full.sh → verify-full.sh (fast) + verify-stress.sh (boundary)

Recent additions to verify-full.sh (#8 tool-prefill OOM, #9 cascade detection, #10 MTP AL) made the script slow — the longctx needle ladder (#7) alone could run 5+ min, and the full 10-check suite was approaching 10 min. Awkward for "is the stack functional" iteration during dev work.

Split into two scripts:

  • verify-full.sh — fast functional smoke, 8 checks, ~1-2 min. Contains: server reachability, Genesis patches applied, basic completion (Paris), tool calling, streaming, thinking mode, output quality / cascade detection, MTP acceptance length. Run after every config change to confirm the stack still serves cleanly.

  • verify-stress.sh — boundary-case stress test, 2 checks, ~5-10 min. Contains: long-context needle ladder (4 depths up to 90K tokens) + tool-response prefill OOM (~25K-token mock tool message). Run before publishing or when investigating prefill-OOM regressions specifically.

Same env-var conventions (URL, MODEL, CONTAINER, SKIP_LONGCTX, SKIP_TOOL_PREFILL, PREFILL_TARGET_CHARS). Both pass on the new club-3090 default + dual.yml + dual-turbo.

2026-04-28 — Dual-card re-bench on club-3090 substrate (revised TPS numbers)

The published dual-card TPS numbers were measured pre-v714 formalization (April 24-25 timeframe), on a different vLLM nightly + Genesis tree. Re-benched all 4 dual composes on the club-3090 unified substrate (dev205 + Genesis v7.51-stable + Marlin pad fork mounted) to reconcile.

Also: caught a stale mount path in dual-turbo.yml — predecessor mounted patch_tolist_cudagraph.py from ../patches/genesis/ (where it lived in the old qwen36-dual-3090 layout); club-3090 has it at ../patches/ (top-level). Fixed before measurement; container booted clean. All other composes already had correct paths.

Measured numbers (3 warmup + 5 measured per prompt arm, narr 1000 tok + code 800 tok):

Compose Narr TPS (CV) Code TPS (CV) TTFT MTP/DFlash AL VRAM/card Was claimed Δ%
dual.yml 69.05 (2.3%) 88.58 (3.4%) 145ms 3.38-3.48 23.6 GB 71/89 -3% / -1%
dual-turbo.yml (now TQ3) 53.65 (2.7%) 72.93 (2.7%) 113ms 3.41-3.42 24.1 GB 58/69 (k8v4) -8% / +6%
dual-dflash.yml 81.94 (4.3%) 124.93 (5.8%) 138ms 4.10-4.35 23.6 GB 78/128 +5% / -2%
dual-dflash-noviz.yml 78.19 (2.5%) 126.99 (2.2%) 143ms 4.24-4.37 23.8 GB 77/124 +2% / +2%

Net: most numbers within run-to-run variance. The dual.yml fp8 path is essentially unchanged. dual-turbo's TQ3 swap (from k8v4) cost ~8% narrative but recovered ~6% code — net trade for ~9× the KV pool capacity.

All 4 composes pass verify-full.sh functional checks (skipped longctx ladder on the DFlash variants for time; fp8 + MTP variants pass full 10/10 including the 90K-token needle). Updated all docs (README compose table, USE_CASES.md, dual.yml header, dual-turbo.yml header) with the measured numbers.

2026-04-28 — Add long-vision + long-text composes (R3' / R3''' from formal v714 round)

Previously the 192K and 205K opt-in tiers were documented as "edit max-model-len + mem-util in docker-compose.yml" — fragile for reproducibility against published bench numbers. Promoted both to dedicated compose files:

  • single/long-vision.yml — TQ3 + Genesis P65 + MTP n=3 + 192K + 0.98 mem-util + vision tower active. Matches R3' bench row (50.93 narr / 67.69 code TPS, AL 3.40-3.58 80-86% accept). Container name: vllm-qwen36-27b-long-vision. Same prefill caveats as edit-the-default did.
  • single/long-text.yml — Same config + --language-model-only + max-model-len 205K. Matches R3''' (50.11 narr / 65.84 code TPS). Container name: vllm-qwen36-27b-long-text.

Trade-off: 2 more compose files (now 11 vs 9). Net: every published bench row from the v714 formalization round (R2, R3, R3', R3''', R4, R6, R7) now boots cleanly with one -f flag — no error-prone editing for users who want to reproduce. R1 (eager) and R5 (longctx) stay deleted (obsolete, not niche).

Header references updated: model README compose table, USE_CASES.md frontier-context section, default's variant matrix, vllm/README.md "Pick a compose" code block.

2026-04-28 — Repo migration to club-3090

Configs migrated from the predecessor repos (qwen36-27b-single-3090, qwen36-dual-3090) into this repo's models/qwen3.6-27b/vllm/compose/ directory. File renames:

Old path New path
qwen36-27b-single-3090/compose/docker-compose.yml models/qwen3.6-27b/vllm/compose/docker-compose.yml
qwen36-27b-single-3090/compose/docker-compose.fast-chat.yml models/qwen3.6-27b/vllm/compose/docker-compose.fast-chat.yml
qwen36-27b-single-3090/compose/single/tools-text.yml models/qwen3.6-27b/vllm/compose/single/autoround-int4/tools-text.yml
qwen36-27b-single-3090/compose/docker-compose.no-genesis-mtp.yml models/qwen3.6-27b/vllm/compose/docker-compose.no-genesis-mtp.yml
qwen36-27b-single-3090/compose/single/minimal.yml models/qwen3.6-27b/vllm/compose/single/autoround-int4/minimal.yml
qwen36-dual-3090/compose/docker-compose.yml models/qwen3.6-27b/vllm/compose/dual/autoround-int4/fp8-mtp.yml
qwen36-dual-3090/compose/docker-compose.turbo.yml models/qwen3.6-27b/vllm/compose/dual/autoround-int4/turbo.yml
qwen36-dual-3090/compose/docker-compose.dflash.yml models/qwen3.6-27b/vllm/compose/dual/autoround-int4/dflash.yml
qwen36-dual-3090/compose/docker-compose.dflash-noviz.yml models/qwen3.6-27b/vllm/compose/dual/autoround-int4/dflash-noviz.yml
qwen36-27b-single-3090/patches/patch_tolist_cudagraph.py models/qwen3.6-27b/vllm/patches/patch_tolist_cudagraph.py

Functional content identical — only paths changed. Anyone with scripts referencing the old paths needs to update; the old repos still serve the old paths read-only.

2026-04-28 — Compose rename: v7.14 is the zero-arg vLLM default (single-card)

Breaking change at the time (mitigated by being on a small-audience repo).

  • docker-compose.v714.yml → docker-compose.yml. Running docker compose up -d (with no -f flag) now boots the production-safe TQ3 + Genesis v7.14 + MTP n=3 + 48K + 0.92 config.
  • The previous zero-arg default (fp8 + MTP n=3 + 20K) → docker-compose.fast-chat.yml. Pick this one when you only need ≤20K context and want the maximum-TPS chat path (~5-7% faster than the new default).
  • docker-compose.longctx-experimental.yml → deleted. Superseded by the default's opt-in 128K + 0.95 tier.

2026-04-28 — Prefill-OOM tests + safe v714 default

Triggered by ampersandru's production OOM report (noonghunna/qwen36-27b-single-3090#1) — a Hermes-class agent fetching ~25K tokens of news as a tool reply at 192K context crashed the engine.

Discovered two distinct activation-memory cliffs on this hardware:

  • Cliff 1 — TurboQuant attention scratch + tool-response prefill, fires on ≥25K-token tool messages at high --gpu-memory-utilization. OOM site: TurboQuant forward (dequant scratch + mid_o/output buffers), ~138 MiB allocate.
  • Cliff 2 — DeltaNet/GLA recurrent state buffer, fires on any single prompt above ~50-60K tokens regardless of mem-util. OOM site: fla.ops.chunk.chunk_gated_delta_rule_fwd_h.h.new_empty(...). NT grows linearly with prompt length; chunked-prefill doesn't help.

Shipped:

  • verify-full.sh extended from 7 → 10 checks: #8 tool-response prefill OOM, #9 output quality / cascade detection, #10 MTP acceptance length threshold.
  • verify-full.sh #7 long-context needle ladder treats engine HTTP 400 (oversize ctx rejection) as a clean "skipped at this depth" rather than a failure.
  • vLLM single-card default lowered to 48K + 0.92 — below both cliffs. All 10 checks pass.
  • README/docs document the full opt-in matrix (64K → 205K) with safe single-prompt + tool-prefill envelopes per tier.
  • Three-layer defense documented: vLLM --max-model-len HTTP 400 rejection + agent-framework truncation + system-prompt limits.

TPS unchanged at the new default: 51 narr / 68 code TPS (CV ~2.3%). Hardware-bound.

2026-04-28 — Inherited prefill-OOM tests on dual-card

The dual-card stack adopted the new verify-full.sh checks for safety even though TP=2 + fp8 KV (the dual-card default) gives much wider safety margins than single-card TQ3 KV — the cliffs are not active failure modes on dual hardware.

All compose variants pinned to vllm/vllm-openai:nightly-07351e0883470724dd5a7e9730ed10e01fc99d08 (= vLLM dev205+g07351e088). Previously some tracked :nightly and drifted with upstream.

2026-04-27 — Full-matrix re-bench + substrate unification (single-card)

Discovered and fixed four real compose drift bugs during a complete re-bench cycle:

  • Image split: composes had drifted across two different vLLM image pins. All six unified to vllm/vllm-openai:nightly-07351e0883470724dd5a7e9730ed10e01fc99d08.
  • eager.yml config drift: shipped with gpu-memory-utilization=0.92 and max-model-len=131072 while @ampersandru's actual measurement was 0.97 + 125000. As-shipped failed to boot. Compose deleted entirely.
  • v714.yml mount path: patch_tolist_cudagraph.py was mounted from a wrong path. Fixed.
  • Bench harness regression: scripts/bench.sh had silently dropped the code-prompt arm. Restored.
  • Genesis exoneration: A/B between default + Genesis vs no-Genesis confirmed Genesis is performance-neutral on the fp8+MTP path. Cross-rig confirmed by u/sudeposutemizligi on TP=2 + dev45 + no Genesis (55 narrative / 68 code, same hardware class).

2026-04-27 — Removed: docker-compose.eager.yml (single-card)

Originally proposed by @ampersandru as a 125K path that bypasses the cudagraph bug class via --enforce-eager. Re-bench cycle measured 25.5 narr / 32.3 code — strictly dominated by longctx-experimental.yml at the same 125K context (38/50 TPS). Compose removed.

2026-04-27 — Patch hardening (single-card)

  • patches/patch_tolist_cudagraph.py was silently failing on (a) any non-docker setup (hardcoded dist-packages path) and (b) any vLLM nightly past the one we initially tested against. Fixed: patcher auto-discovers vLLM via import vllm and uses single-line regex anchors. Bug reported by @3dluvr in single-3090 #1.

2026-04-25 — Genesis v7.14 (Sandermage upstream)

Genesis v7.14 shipped with the P65 patch root-causing vllm#40880 — the silent tool-call cascade bug under MTP × TurboQuant × cudagraph. P65 forces cudagraph_mode=PIECEWISE for spec-decode → eager continuation runs the correct branch.

This shipped as a workaround. The proper fix is a custom multi-query Triton kernel (P67) that handles K+1 query against compressed cached KV under cudagraph capture — designed-but-not-implemented as of v7.14.

The dual-card Turbo variant (dual/turbo.yml) loads Genesis v7.14 with P64/P65/P66/P68/P69 enabled via env vars. ~25% per-stream TPS regression vs fp8 default but 4.59× concurrency at full 262K vs fp8's 2.36× — aggregate throughput exceeds fp8 above ~3 concurrent streams.

We adjusted two consumer-Ampere knobs vs Sandermage's A5000-class defaults: gpu-memory-utilization 0.92 → 0.85 and max-num-batched-tokens 8192 → 4128. Without these, deep-prefill (60K+) requests OOM on 24 GB cards.

2026-04-22 — DFlash N=5 + Qwen3.6-27B (Luce z-lab fork)

Luce z-lab's DFlash spec-decode draft model for Qwen3.6-27B clears verify-full.sh on dual-3090. Single-stream 78 / 128 TPS narr/code — substantially faster than MTP n=3's 71 / 89.

Two DFlash variants ship in the dual-card path:

  • dual/dflash.yml — vision + DFlash N=5 + 185K context
  • dual/dflash-noviz.yml — text-only + DFlash N=5 + 200K context

Required workaround: vllm#40334 (DFlash combine_hidden_states dtype mismatch) is open. Compose sets --dtype bfloat16 to match the draft's training dtype.

2026-04-15 — Marlin pad-sub-tile-n (PR #40361 — our patch)

Filed vllm#40361 — fixes a crash in vLLM's Marlin INT4 kernel where output features < 64 cause GPTQ_MARLIN_MIN_THREAD_N (64) > out_features on TP=2.

Status: OPEN, MERGEABLE, awaiting maintainer review. Until it lands, dual-card composes volume-mount our patched fork at /opt/ai/engines/vllm/primary/.

2026-04-08 — Initial dual-card release

vLLM-based dual-3090 recipe shipping at TP=2 with fp8 KV + MTP n=3, full feature parity with the single-card project plus the Marlin pad workaround. Was its own repo at the time; now lives here.

Earlier — Initial single-card release

Initial single-card release shipped a docker-compose.longctx-experimental.yml at 125K with cudagraph_mode=NONE as the long-context option. v7.14 superseded this; deprecated and removed during 2026-04-27 cleanup.