Configurable served/IPC camera framerate to reduce CPU for on-device vision (Reachy Mini Lite locked to 1080p@60)

[rafaheringer]

Summary

On the Reachy Mini Lite, the camera only supports 1920x1080@60fps (plus higher 4K modes) — there is no lower-resolution or lower-framerate mode. For apps that need the video stream but not at 60fps (e.g. on-device face/hand tracking with MediaPipe), the daemon serving 1080p@60 over the IPC branch saturates the Raspberry Pi: the daemon (~180% CPU) plus the local client receiving/converting the stream (~130% CPU) starve the tracking loop, so head/hand tracking becomes too laggy to be usable.

It would help a lot to be able to decouple the served/IPC framerate from the camera capture rate, so a local vision app can request e.g. ~10 fps and keep CPU low while still getting frames.

Environment

• Hardware: Reachy Mini Lite (camera VID 0x38FB, PID 0x1002, reported name: lite)
• Daemon reachy_mini tested on both 1.7.0 and 1.8.x
• Raspberry Pi 4 (4 cores), daemon running natively, backend LOCAL (GStreamer IPC camera)

What I observed

• media.camera.set_resolution(...) rejects every mode below 1080p@60 with:
  ValueError: Resolution not supported by the camera. Available resolutions are: [R1920x1080at60fps, R3840x2592at30fps, R3840x2160at30fps, R3264x2448at30fps]
• Forcing a lower mode at the pipeline level (changing default_resolution, or inserting a videorate before the tee in media_server.py) breaks capture — get_frame() returns None ("Internal data stream error"). The unixfdsink FD-buffer path is sensitive to extra elements. This is consistent with set_resolution() + camera.open() fails with GStreamer pipeline error on Reachy Mini Lite (daemon ≥ 1.8.0) #1234.
• The CPU cost is dominated by serving/converting the 1080p@60 stream (YUV2→BGR), consistent with 679 wireless apps always have a cpu at 100 #680. But the no_video workaround from 679 wireless apps always have a cpu at 100 #680 does not apply here: the app needs the video for tracking — it just doesn't need 60 fps.

Proposed solutions (any one would help)

1. A configurable framerate for the IPC/served branch (e.g. media.set_served_framerate(fps) or a daemon env var), implemented so it respects the unixfdsink FD-buffer requirements (a plain videorate before the tee breaks it).
2. Expose lower camera capture modes on the Lite if the sensor supports them (e.g. 1280x720), so set_resolution can actually reduce the load.
3. Document the recommended pattern for low-CPU on-device vision/tracking on the Lite.

Why it matters

The Lite is the entry-level / most CPU-constrained variant, and on-device head/hand tracking is a flagship use case. Today it is effectively unusable on a Pi 4 because the only camera mode is 1080p@60.

Related: #680, #1234, #964, #133. Happy to test patches.

[alozowski]

Hello @rafaheringer,
The latest release (v1.9.0) includes the head-tracking PR #1177, and it ships the mechanism from this issue: the daemon now serves the local IPC feed through a single videorate capped at 10 fps, while capture and WebRTC stay at 30. Local clients, including the new face tracker, only pay for the frames they actually receive, we measured it on Wireless. You were right that pacing the feed once, daemon-side, is the way to do it, thank you! The cap is hardcoded for now, but I opened the following issue #1263