Skip to content

QVAC-23799 feat[asr-ggml]: add opt-in CUDA GPU backend for whisper and parakeet - #4059

Closed
Zbig9000 wants to merge 1 commit into
tetherto:mainfrom
Zbig9000:QVAC-23799-asr-cuda
Closed

QVAC-23799 feat[asr-ggml]: add opt-in CUDA GPU backend for whisper and parakeet#4059
Zbig9000 wants to merge 1 commit into
tetherto:mainfrom
Zbig9000:QVAC-23799-asr-cuda

Conversation

@Zbig9000

@Zbig9000 Zbig9000 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

🎯 What problem does this PR solve?

  • asr-ggml could never run on CUDA. Both engines already resolved and reported CUDA at runtime — WhisperModel.cpp and ParakeetModel.cpp map ggml's CUDA* backend name to BackendId.CUDA (2), and gpu.test.js / parakeet-gpu-smoke.test.js already had cuda branches — but no build ever compiled the backend in.
  • The cause was purely dependency wiring: vcpkg.json requested speech-cpp with vulkan (desktop/Android), metal (Apple) and opencl (Android), never cuda. On NVIDIA hosts a use_gpu / useGPU request therefore always landed on Vulkan.

📝 How does it solve it?

  • New cuda feature in vcpkg.json forwarding to speech-cpp[cuda]ggml-speech[cuda] (GGML_CUDA=ON, nvcc resolved by the port).
  • New ASR_CUDA CMake option appending "cuda" to VCPKG_MANIFEST_FEATURES, following the SD_CUDA precedent in diffusion-cpp.
  • New build:cuda / build:native:cuda scripts (bare-make generate -D ASR_CUDA=ON).
  • Opt-in, not default, because it needs nvcc on the build host, which the prebuild runners do not carry. Only the NVIDIA driver is needed at runtime.
  • CUDA is compiled alongside Vulkan rather than replacing it. ggml registers CUDA ahead of Vulkan, so a GPU request prefers CUDA and falls back to Vulkan when no supported device is present.
  • Apple and Android are excluded via "supports": "!(osx | ios | android)".
  • The cuda feature carries its own speech-cpp floor, 2026-08-24#1 — the version that introduced the feature. Feature dependencies only apply when the feature is selected, so default builds keep resolving at 2026-08-18 and are unaffected.
  • No C++ changes.

No companion PRs needed

  • Registry: speech-cpp[cuda]ggml-speech[cuda] already shipped in tetherto/qvac-registry-vcpkg@cc90801 (QVAC-23802, speech-cpp 2026-08-24#1). Nothing to add there.
  • vcpkg-configuration.json baseline: unchanged. The pinned baseline dca20a94 carries only speech-cpp 2026-08-10 / 2026-08-07, yet main already requires >= 2026-08-18 and builds green — version constraints resolve against fetched registry HEAD, not the baseline commit. 2026-08-24#1 resolves the same way.
  • Version bump: none, matching the established flow — QVAC-23758 fix: drain queued streaming output before JobEnded in parakeet endStreaming #3950 added an Unreleased entry without touching version, and bumps land in dedicated release PRs (e.g. chore[mod]: release asr-ggml 0.3.1, ...). This PR adds to ## [Unreleased].

🧪 How was it tested?

  • New scripts/__tests__/build-backends.test.js (6 tests) pins the wiring: the option defaults to OFF, build:native does not enable it, ASR_CUDA maps to the cuda manifest feature, the feature forwards to speech-cpp[cuda] with default-features: false, the platform guards match, and the CUDA floor is not below the base floor.
  • node --test "scripts/__tests__/*.test.js"46/46 pass.
  • Mutation-checked the two CMake regex assertions (option renamed, feature renamed, default flipped to ON) — each correctly fails, so the assertions are not vacuous.
  • vcpkg format-manifest accepts the manifest. It also wants to canonically reorder the pre-existing dependency blocks, which the committed file does not follow, so that churn was left out to keep the diff reviewable.
  • Prettier clean on the new test file.

Not covered: an actual -D ASR_CUDA=ON compile and a CUDA-vs-Vulkan runtime check. Both need a host with the CUDA toolkit; by design this PR adds no CI job, so the first real CUDA build is manual. The claim that ggml prefers CUDA over Vulkan is documented but unmeasured.

🔌 API Changes

No API changes. BackendId.CUDA (2) was already exported and already returned by getBackendInfo(); this PR only makes it reachable.

# opt into CUDA at build time
npm run build:cuda      # or: bare-make generate -D ASR_CUDA=ON

…d parakeet

Both engines already resolved and reported CUDA at runtime (BackendId.CUDA,
2) and the GPU integration tests already had cuda branches, but no build ever
compiled the backend in: the speech-cpp dependencies only ever requested
vulkan, metal and opencl.

Add a cuda manifest feature forwarding to speech-cpp[cuda] ->
ggml-speech[cuda] (GGML_CUDA=ON), selected by a new ASR_CUDA CMake option and
the build:cuda / build:native:cuda scripts.

CUDA is opt-in rather than default because it needs nvcc on the build host,
which the prebuild runners do not carry; only the NVIDIA driver is needed at
runtime. It is compiled alongside Vulkan and ggml registers CUDA first, so a
use_gpu / useGPU request prefers CUDA and falls back to Vulkan when no
supported device is present. Apple and Android are excluded.

The cuda feature carries its own speech-cpp floor (2026-08-24#1, the version
that introduced the feature). Feature dependencies only apply when the
feature is selected, so default builds keep resolving at 2026-08-18.
@github-actions

Copy link
Copy Markdown
Contributor

Review Status

Current Status: ❌ PENDING
Approvals so far: none

Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member.

@Zbig9000 Zbig9000 added verify tier1 run-desktop-addon-tests CI: run desktop integration tests (requires verified) labels Aug 25, 2026

@GustavoA1604 GustavoA1604 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not open from fork

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-desktop-addon-tests CI: run desktop integration tests (requires verified) tier1 verify

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants