Skip to content

Add vision subpackage with MediaPipe HeadTracker#1177

Draft
alozowski wants to merge 39 commits into
mainfrom
feat/1175-move-mediapipe
Draft

Add vision subpackage with MediaPipe HeadTracker#1177
alozowski wants to merge 39 commits into
mainfrom
feat/1175-move-mediapipe

Conversation

@alozowski

@alozowski alozowski commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator

Adds a vision subpackage with the MediaPipe HeadTracker, migrated from reachy_mini_toolbox. Also, mediapipe==0.10.14 is added as a new optional extra. The original exit() on missing mediapipe is replaced with a proper ImportError to avoid killing the robot process on import

Closes #1175
The conv app PR: pollen-robotics/reachy_mini_conversation_app#381

@alozowski alozowski moved this from Backlog to In progress in Reachy Mini Jun 1, 2026
@FabienDanieau

Copy link
Copy Markdown
Contributor

Is it an issue to have reachy_mini_toolbox as a dep in the conv app?

@pierre-rouanet

Copy link
Copy Markdown
Member

LGTM, indeed as @FabienDanieau points out. Maybe we can use toolbox as a dependency. Not strong opinion on this. Just if we do, we have to make sure that everything included runs on a RM wireless.

Maybe next step is ton add start/stop/getface API entry points to trigger the head tracking in background?

@FabienDanieau

Copy link
Copy Markdown
Contributor

I made some tests. rf-detr-nano is too heavy for the rpi. Mediapipe seems to be the most suitable solution for edge devices. There is a lighter model that we can use instead of the facemesh of the tool box: see face_track.py.
We need to mind the numpy 1.x dependency though.

Maybe next step is ton add start/stop/getface API entry points to trigger the head tracking in background?

I agree with @pierre-rouanet, the right call would be to enable head tracking on the daemon side (in a way similar we did for the wobbling see #1001). Basically we need to add a appsink to the video pipeline to feed the head tracker.

Also the tracking could be extended with the DoA. But let's keep that for another issue :)

@alozowski alozowski marked this pull request as draft June 25, 2026 18:16
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@alozowski

Copy link
Copy Markdown
Collaborator Author

let's say that mediapipe works here though it's not ideal:
6c7a911

I'm continuing my experiment with completely another approach in this branch

@alozowski

alozowski commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator Author

next checkpoint, commit 9a5b52f

  • opencv works
  • start the daemon from this branch
  • enable headtracking with curl -s -XPOST localhost:8000/api/media/tracking/enable -H Content-Type:application/json -d '{}'
  • disable it with curl -s -XPOST localhost:8000/api/media/tracking/disable -H Conte{} (truncated command, I know, but it works)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

Move MediaPipe head tracker from reachy_mini_toolbox to reachy_mini

4 participants