Python: headless viewer, vsync control, pipelined frame capture#11
Open
haixuanTao wants to merge 8 commits into
Open
Python: headless viewer, vsync control, pipelined frame capture#11haixuanTao wants to merge 8 commits into
haixuanTao wants to merge 8 commits into
Conversation
The viewer could only present to its window; there was no way to get rendered pixels back to Python for headless frame/video export (as mujoco.Renderer.render() or a Genesis camera.render() would give). kiss3d's Window already exposes snap_image(), so this just wires it through: - nexus_viewer3d::NexusViewer::snap_rgb() -> (w, h, RGB bytes) - Python NexusViewer.render() -> (H, W, 3) uint8 NumPy array Call render_frame() to draw, then render() to read the frame back.
Adds NexusViewer::new_with_size and exposes width/height (default 1200x900) on the Python constructor, so headless capture and path tracing can run at arbitrary resolutions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
render_frame always drew the egui main panel, which then appeared in snap_rgb captures. Expose a toggle (default on) so headless recording gets clean frames. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
haixuanTao
force-pushed
the
feat/python-headless-capture
branch
from
e000dc5 to
5bd7f21
Compare
haixuanTao
changed the title
Adds NexusViewer.raytrace_frame() (kiss3d 0.45 GPU path tracer with cross-frame sample accumulation), set_raytracer_samples_per_frame/ max_bounces/denoise, and snap_rgb() returning the framebuffer as an (H, W, 3) uint8 numpy array. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The zero-readback sync kernel writes body poses straight into kiss3d's GPU instance buffers, but the ray tracer rebuilds its acceleration structure from the CPU-side instance data — so path-traced frames froze at the initial scene. Route sync through the readback path whenever a RayTracer exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
NexusViewer.raytracer_backend() returns "hardware" (RT-core ray queries) or "software" (compute-shader BVH fallback) so users can verify which backend kiss3d selected on their GPU. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…thon) The windowed capture loop was vsync-locked: kiss3d's swapchain defaults to AutoVsync and the blocking per-frame snap_rgb() readback defeats frame pipelining, so each render_frame+render iteration ate ~2 vblanks (~30 fps at 60 Hz) no matter how fast the GPU was. Fixes, exposed through nexus3d: - NexusViewer(width, height, headless=True): no OS window, no swapchain — frames render into an off-screen texture (kiss3d headless Window), never throttled by the display and usable without a display server. - NexusViewer.set_vsync(enabled)/vsync(): uncap a windowed viewer instead. - NexusViewer.render_async()/render_flush(): pipelined capture — returns the previous frame while this frame's GPU->CPU copy runs in the background (one frame of latency; flush collects the last frame). Cube-drop capture at 640x480: 33.1 -> 92.1 gen-fps (GPU physics), 26.5 -> 37.5 (Rapier CPU backend). Requires kiss3d feat/headless-capture-pipeline (new_headless_with_setup + snap_begin/snap_finish).
haixuanTao
force-pushed
the
feat/python-headless-capture
branch
from
5bd7f21 to
6a3c7e1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #8 (which stacks on #7) — merge order: #7 → #8 → this. After those merge, this PR reduces to its single commit (
feat: headless viewer, vsync control, and pipelined frame capture (Python)); until then the diff below includes their changes.New capture API in the
nexus3dPython bindings:NexusViewer(..., headless=True)— no OS window, no swapchain: frames render into an off-screen texture, never throttled by display vsync, and it works without a display server.set_vsync(enabled)/vsync()— uncap a windowed viewer instead.snap_rgb_async()/snap_rgb_flush()— pipelined capture: returns the previous frame while the current frame's GPU→CPU copy runs in the background (one frame of latency; flush collects the last frame).Why: the windowed capture loop was vsync-locked at ~30 fps — the swapchain defaults to Fifo and the blocking per-frame
snap_rgb()readback defeats frame pipelining, so each iteration ate ~2 vblanks regardless of GPU load. Headless + pipelined readback reaches 92 gen-fps at 640×480 (GPU physics; 37.5 with the Rapier CPU backend) on an RTX 5080, up from 33.🤖 Generated with Claude Code