Skip to content

Add vision subpackage with MediaPipe HeadTracker - #1177

Merged
alozowski merged 62 commits into
mainfrom
feat/1175-move-mediapipe
Jul 8, 2026
Merged

Add vision subpackage with MediaPipe HeadTracker#1177
alozowski merged 62 commits into
mainfrom
feat/1175-move-mediapipe

Conversation

@alozowski

@alozowski alozowski commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator

Adds daemon-side head tracking, that can run out-of-process so it never stalls the 50 Hz control loop, and to coexist with the conversation app on the CPU-bound CM4 / Pi 4 (supersedes the original in-process MediaPipe migration this branch started as)

What changed

  • new vision/subpackage: YuNet detector (cv2.FaceDetectorYN) + a single-target tracker, run in a spawned multiprocessing.Process (own GIL/core). Replaces the in-process MediaPipe tracker, which serialised on the GIL and starved the loop.
  • daemon eases the tracking aim per 50 Hz tick -> motion stays smooth regardless of detection rate, so the detector can run slow (10 fps) and cheap
  • dedicated valve-gated, downscaled + rate-limited camera branch feeds the worker over its own unixfdsink socket – converts frames only while tracking, nothing when idle.
  • full-frame detection at 320 px / 10 fps (no ROI crop), YuNet model fetched from the HF hub
  • multi-person: gaze follows the closest / most-prominent face, with hysteresis so it doesn't ping-pong
  • enable_head_tracking(weight=0) pauses (frees the head + detector CPU) without tearing down the process -> cheap per-turn on/off for the conv-app speak/listen handoff
  • decouples the local IPC framerate from capture (helps Configurable served/IPC camera framerate to reduce CPU for on-device vision (Reachy Mini Lite locked to 1080p@60) #1241): serves the IPC feed at a capped fps so on-device vision apps don't pay the full capture rate
  • drops the vision / mediapipe==0.10.14 extra, vision installs via the existing opencv extra. Missing OpenCV raises ImportError instead of exit(), so it can't kill the robot process

Validated on-robot (Wireless/CM4):

  • loop holds ~50 Hz while tracking; coexists with the conversation app (~1.5 of 4 cores free).
  • collision-safe with speaking + emotions/dances; crash isolation – a worker crash doesn't take the daemon down and self-heals on the next turn

DoA-driven search isn't yet in this PR, could be a next PR, as @FabienDanieau suggested.

Closes #1175
Conv app PR: pollen-robotics/reachy_mini_conversation_app#381

@alozowski alozowski moved this from Backlog to In progress in Reachy Mini Jun 1, 2026
@FabienDanieau

Copy link
Copy Markdown
Contributor

Is it an issue to have reachy_mini_toolbox as a dep in the conv app?

@pierre-rouanet

Copy link
Copy Markdown
Member

LGTM, indeed as @FabienDanieau points out. Maybe we can use toolbox as a dependency. Not strong opinion on this. Just if we do, we have to make sure that everything included runs on a RM wireless.

Maybe next step is ton add start/stop/getface API entry points to trigger the head tracking in background?

@FabienDanieau

Copy link
Copy Markdown
Contributor

I made some tests. rf-detr-nano is too heavy for the rpi. Mediapipe seems to be the most suitable solution for edge devices. There is a lighter model that we can use instead of the facemesh of the tool box: see face_track.py.
We need to mind the numpy 1.x dependency though.

Maybe next step is ton add start/stop/getface API entry points to trigger the head tracking in background?

I agree with @pierre-rouanet, the right call would be to enable head tracking on the daemon side (in a way similar we did for the wobbling see #1001). Basically we need to add a appsink to the video pipeline to feed the head tracker.

Also the tracking could be extended with the DoA. But let's keep that for another issue :)

@alozowski
alozowski marked this pull request as draft June 25, 2026 18:16
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@alozowski

Copy link
Copy Markdown
Collaborator Author

next checkpoint, almost the final: 5b42110

  • we now resample instead of crop: detect on the whole frame downscaled to 320 px (gstreamer videoscale in the daemon tracking branch) every tick, without ROI window, so full FOV and robust to fast motion
  • detection rate is 10 fps, but the head stays smooth because the daemon eases the aim at 50 Hz independent of the detection rate
  • YuNet score threshold is 0.6 so it holds the face through head turns and at ~1 m
  • YuNet model fetched from HF hub instead of the vendored ONNX (first enable needs network once, then it's cached)
  • weight 0 now pauses tracking without tearing down the detector; weight > 0 resumes instantly
  • local IPC camera feed capped at 15 fps (also touches the client camera_gstreamer.py) so on-device apps don't pay the full 30 fps convert

@alozowski

Copy link
Copy Markdown
Collaborator Author

another checkpoint, f710992

we don't yet have DoA tracking but I added a simple multi-person one:

  • gaze follows the most prominent face (largest ≈ closest, weighted toward center) instead of sticking to the first one acquired
  • switches only after a challenger leads by a margin for a dwell window, so no ping-pong between comparable faces as it used to be

@alozowski
alozowski marked this pull request as ready for review July 6, 2026 22:19
@alozowski

Copy link
Copy Markdown
Collaborator Author

and another big checkpoint (5db721e) that dropped opencv from the tracking runtime, YuNet now runs on onnxruntime with a numpy decode:

  • same detector and same weights: the model is the official dynamic-input YuNet export (face_detection_yunet_2026may, byte-identical weights to 2023mar), hosted on our HF and pinned by revision
  • the decode reproduces cv2.FaceDetectorYN bit-for-bit, there's a parity test running both on the same model and frame
  • frames are padded up to /32 (320x180 -> 320x192) because YuNet's feature pyramid needs stride-aligned dims, opencv did this silently, raw ORT crashes without it
  • ORT session is pinned to 1 thread so the detector can't spread across cores and fight the control loop
  • the robot installs [vision] (onnxruntime, ~16 MB) instead of [opencv] (~50 MB wheel / ~119 MB unpacked), so opencv is now dev/calibration-only
  • onnxruntime bumped to 1.27.0 for both vision and nn_kinematics (verified FK/IK bit-identical), which needs python >=3.11

Comment thread src/reachy_mini/vision/face_tracking.py Outdated
Comment thread src/reachy_mini/media/media_server.py Outdated
Comment thread src/reachy_mini/media/media_server.py Outdated
Comment thread src/reachy_mini/media/media_server.py Outdated
Comment thread src/reachy_mini/vision/face_detector.py
Comment thread src/reachy_mini/vision/face_tracking.py Outdated
@alozowski

Copy link
Copy Markdown
Collaborator Author

another checkpoint, a5018b0: the tracker is now just a camera client

  • dedicated tracking branch/socket/valve deleted: the tracker reads the same camera IPC as any local app, pause = the worker just disconnects
  • one videorate total: IPC_FPS=10 on the daemon paces everyone, detector included (measured with a fakesink client: 123 frames / 12 s)
  • aim at the nose now (eye midpoint made the robot look slightly above), single-face only
  • explicit gst elements, direct try_pull_sample(), v4l2convert on RPi, no more ctrl+c traceback
  • example + SDK docs added

next to try: drop the worker process and use a plain thread, and it would save ~200 MB of duplicated python/ORT/GStreamer stack

@FabienDanieau FabienDanieau left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah thanks!

@alozowski
alozowski merged commit 2f4615f into main Jul 8, 2026
10 checks passed
@alozowski
alozowski deleted the feat/1175-move-mediapipe branch July 8, 2026 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move MediaPipe head tracker from reachy_mini_toolbox to reachy_mini

4 participants