Skip to content

gfx: scene render pipeline rework + GPU video I/O interop layer#2109

Draft
jcelerier wants to merge 24 commits into
masterfrom
clean/gpu-interop
Draft

gfx: scene render pipeline rework + GPU video I/O interop layer#2109
jcelerier wants to merge 24 commits into
masterfrom
clean/gpu-interop

Conversation

@jcelerier

@jcelerier jcelerier commented Jul 12, 2026

Copy link
Copy Markdown
Member

Overview

This branch bundles two stacked features (merge as one), 22 commits on top of master:

1. Scene-aware GPU render pipeline (6 commits)

Reworks the gfx render graph around a flat, incrementally-rebuilt scene model:

  • OffsetAllocator-backed GPU resource arenas, a shared decoded-asset cache, scene math / pipeline-state helpers and a flat scene state + packer
  • ISF parser and nodes gain 3D samplers, uniform inputs and geometry/vertex stages; the compute and raster pipelines become scene-aware (MRT, auxiliary outputs); the render graph updates edges / render targets / passes in place instead of rebuilding wholesale
  • scene preprocessor and scene-filter / geometry-merge nodes
  • avnd, js and threedim node lifecycles adapted to the new pipeline

2. GPU video I/O interop layer (16 commits)

Adds a vendor-neutral layer for capture-card and PipeWire video I/O without a CPU round-trip:

  • high-bit-depth / planar / packed GPU video encoders and wire-format (v210, r210) + hardware decoders
  • Vulkan external-memory + texture sharing, a CUDA layer (functions / VMM allocator / capability probing), Vulkan↔CUDA semaphores and fences, host-pinned / imported / RDMA GPU-buffer rings, dma-buf / DRM-PRIME import-export, and CPU-staged / RDMA output strategies with a runtime selector
  • direct video output and DMA capture graph nodes, an external-genlock render clock, and a NVIDIA GPUDirect-for-Video bridge (dlopen'd at runtime, driver-only)
  • native PipeWire video input/output (shm + zero-copy dma-buf)
  • a few core robustness fixes (panel-delegate null tolerance, minimal-app teardown order, Vulkan-instance guard) surfaced by the harnesses

Validation

Every commit builds independently on Linux (clang/ninja; score_plugin_gfx at each commit, consumer plugins from their adaptation commit, full tree at the tip). The interop paths were hardware-validated end-to-end (AJA Kona 5 SDI/HDMI, DeckLink, Magewell, PipeWire) and are ASan/UBSan-clean.

🤖 Generated with Claude Code

@jcelerier
jcelerier force-pushed the clean/gpu-interop branch from 562c054 to c78e765 Compare July 12, 2026 18:32
This was referenced Jul 12, 2026
Closed
Closed
Closed
jcelerier and others added 21 commits July 12, 2026 22:50
@jcelerier @claude
gfx: add the OffsetAllocator submodule
5b7c7fb 
Bring in OffsetAllocator to back the slab-allocated GPU arenas.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: rework the render pipeline to be scene-aware and incrementally r…
b824245 
…ebuilt

The core scene-GPU rework. Adds the supporting infrastructure — OffsetAllocator-backed
GPU arenas (GpuResourceRegistry), a shared decoded-asset cache (AssetTable +
TextureLoader), scene math and pipeline-state helpers, the flat scene state and packer
(SceneGPUState), an offscreen RHI device and preview widget — then reworks the pipeline
on top of it: the ISF parser and nodes gain 3D samplers, uniform inputs and
geometry/vertex stages; the compute and raster pipelines become scene-aware with multiple
render targets and auxiliary outputs; the render graph updates edges, render targets and
passes in place instead of rebuilding wholesale; and the filter, texture-port, window,
capture and video-output paths follow the new node and renderer interfaces.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: add the scene preprocessor and scene filter nodes
f7bdb1d 
ScenePreprocessorNode turns a scene spec into flat draw commands and arena uploads;
SceneFilterNode, FlattenedSceneFilterNode and MergeGeometriesNode operate on flattened
scenes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
avnd: make the CPU and GPU node lifecycles scene-aware
5ebd4e7 
Split the CPU filter/analysis and GPU node lifecycles into init / initState / release
and add the scene-port concept with its storage helpers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
js: rework the GPU node lifecycle with deterministic teardown
8b2c463 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
threedim: adapt ModelDisplay to the new pipeline API
e9f311a 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: prefer EGL on xcb and force a desktop OpenGL context
38cd38d 
Request the xcb EGL integration on X11 so that dma-buf import can stay
zero-copy, and pin a desktop-GL (not GLES) core-profile context with a
stencil buffer on the desktop code path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
core: tolerate a panel delegate factory that declines to create a widget
998c6d0 
A PanelDelegateFactory may now legitimately return no widget. Don't store
a null panel, guard the plugin-load instantiation path against it, and make
the device-explorer panel lookup null-tolerant instead of dereferencing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
core: fix object teardown order in the minimal applications
e4dafbc 
Drain the event queue before deleting the presenter: deferred slots queued
during plugin load read the presenter-owned application context, so running
them after the presenter is gone is a use-after-free. Also take the settings
models down while the QApplication is still alive, which Qt >= 6.11 requires.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: do not return a Vulkan instance when creation failed
6b0b345 
staticVulkanInstance() could hand back an instance whose create() had failed;
return null instead so callers fall back cleanly rather than crashing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
packagemanager: allow skipping the first-run library download in sani…
3e79986 
…tizer builds

Honor SCORE_SANITIZE_SKIP_CHECKS in firstTimeLibraryDownload so that ASan and
UBSan runs don't reach out to the network at startup.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: add the NVIDIA GPUDirect-for-Video bridge
4a144f2 
A small static bridge that dlopens libdvp at runtime (the driver ships it; no
CUDA toolkit install is required) and exposes its buffer/texture transfer API
to the gfx plugin.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: add a video pixel format vocabulary and libav bridge
c835069 
A backend-neutral pixel-format description plus conversions to and from
ffmpeg's AVPixelFormat, shared by the video encoders, decoders and the
GPU interop strategies.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: add high-bit-depth, planar and packed GPU video encoders
5675251 
P010, packed high-bit-depth RGB, and planar/packed YCbCr (YUY2, yuv422p10)
GPU encoders; parameterize the packed-RGBA encoder for the RGBA, ABGR and
ARGB byte orders; and add a factory that selects the encoder for a given
wire format.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: add the GPU texture-interop layer
aefe2d2 
Vendor-neutral primitives for moving frames between the GPU and capture cards
without a CPU round-trip:

  - Vulkan external memory and cross-API texture sharing
  - a CUDA layer (function loader, VMM allocator, capability probing)
  - Vulkan<->CUDA binary semaphores and an interop fence
  - host-pinned, imported and RDMA GPU-buffer rings (NVIDIA and AMD)
  - dma-buf / DRM-PRIME import and export helpers
  - a paced frame pump and a stage profiler
  - CPU-staged and RDMA output strategies with a runtime selector

Everything is compiled unconditionally but only activates when the matching
driver and hardware are present at runtime.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: add wire-format and hardware video decoders
a00b969 
v210 and r210 wire decoders, with correct padded strides at widths that are
not a multiple of six, plus a decoder factory and the DRM-PRIME, VAAPI, CUDA
and Vulkan hardware-decode paths. Video inputs can now report a dynamic
format change and carry the hardware surface's underlying software pixel
format.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: add direct video output and capture graph nodes
2499013 
DirectVideoOutputNode drives a capture card's SDI/HDMI output through a shader
encode plus the selected interop strategy, and DMACaptureInputNode ingests a
card's input straight into GPU textures.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: drive rendering from an external genlock clock
26c1f5e 
Extract a RenderClock abstraction behind which the display-vsync and timer
render clocks sit, and add an opt-in external clock so the render loop can be
paced by a capture card's vertical interval. Off by default.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: add PipeWire video input and output devices
c9e44d2 
Native PipeWire video input and output, with shared-memory and zero-copy
dma-buf transport negotiated per pixel format.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: use the new encoders and dma-buf paths in the existing video bac…
72b9b76 
…kends

Wire the high-bit-depth encoders into the Libav and GStreamer video outputs
and the dma-buf import into the window-capture, shmdata and sh4lt inputs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: add GPU encoder and PipeWire round-trip test harnesses
ef992e4 
An offscreen self-test for the GPU video encoders and a PipeWire
producer/consumer round-trip harness covering every pixel format and
transport.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier
jcelerier force-pushed the clean/gpu-interop branch from dae2a5b to ef992e4 Compare July 13, 2026 02:53
This was referenced Jul 14, 2026
Merged
Open
Open
Open
jcelerier and others added 3 commits July 22, 2026 01:24
@jcelerier @claude
gfx: reset the DMA capture node's cached mesh buffer on release
a544a6f 
The quad's vertex buffer is owned by RenderList::m_vertexBuffers and is
raw-deleted by RenderList::release(), but the renderer kept a private copy
of the handle. The `if(m_meshBuffer.buffers.empty())` guard in init() then
skipped re-acquisition, so after a release()+init() cycle -- which
maybeRebuild() runs on every window resize/expose -- the node drew with a
freed buffer. init() allocates the process/material UBOs immediately after,
so the allocator hands that block straight back and the stale handle points
at a live UniformBuffer: a deterministic
`buf->usage().testFlag(QRhiBuffer::VertexBuffer)` abort as soon as a real
on-screen Window renders a Direct Video Input.

Every other renderer using this caching idiom already resets it
(VideoNodeRenderer, DirectVideoNodeRenderer, SimpleRenderedISFNode).

Also give the renderer a defensive destructor mirroring release()'s
documented teardown ordering, for deletion paths that never reach release().

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: drive rendering off a timer on Wayland instead of the vsync loop
635b6ac 
In VSync mode the on-screen render loop re-drives itself solely through
QWindow::requestUpdate(). On xcb that is an unconditional timer, so the loop
self-heals. On Wayland requestUpdate() is frame-callback gated: if the
compositor withholds a frame callback past QT_WAYLAND_FRAME_CALLBACK_TIMEOUT
(default 100ms) -- occluded window, blanked/DPMS output, another surface on
top, or heavy per-frame work -- Qt marks the window inexposed and the loop,
having no other driver, produces no frames. The render itself is fine: an
offscreen readback of the same frame returns correct pixels, so the symptom
is a black on-screen window despite correct rendering.

Confirmed by flipping only the VSync setting on a KDE-Wayland session, and by
a minimal QRhi repro whose requestUpdate loop dies under a forced 1ms
frame-callback timeout while a timer-driven loop is unaffected.

Gate the vsync loop off when the Qt platform is Wayland so it takes the timer
path. The compositor supplies frame pacing there, so no tearing results from
dropping swap-chain vsync. xcb, eglfs and the embedded backends are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
@jcelerier @claude
gfx: rebuild DMACaptureInputNode on live input-resolution change
045f5e7 
VideoCaptureSlotRing gains a Spout-style seqlock format channel
(formatGeneration + detected{Width,Height,PixelFormat,Rate}): publishFormat()
on the capture thread signals a wire change, loadFormat() reads it on the render
thread with a parity retry. DMACaptureInputNode::update() polls it beside
latestFrameId and, on a new generation, runs release()+init() to re-create its
size-dependent GPU resources (input texture, slot pool, pipelines,
VideoMaterialUBO.textureSize) at the new geometry. m_lastFormatGen is baselined
at the end of init() so the first frame doesn't spuriously rebuild.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jcelerier