Add vision subpackage with MediaPipe HeadTracker - #1177
Conversation
|
Is it an issue to have reachy_mini_toolbox as a dep in the conv app? |
|
LGTM, indeed as @FabienDanieau points out. Maybe we can use toolbox as a dependency. Not strong opinion on this. Just if we do, we have to make sure that everything included runs on a RM wireless. Maybe next step is ton add start/stop/getface API entry points to trigger the head tracking in background? |
|
I made some tests. rf-detr-nano is too heavy for the rpi. Mediapipe seems to be the most suitable solution for edge devices. There is a lighter model that we can use instead of the facemesh of the tool box: see face_track.py.
I agree with @pierre-rouanet, the right call would be to enable head tracking on the daemon side (in a way similar we did for the wobbling see #1001). Basically we need to add a appsink to the video pipeline to feed the head tracker. Also the tracking could be extended with the DoA. But let's keep that for another issue :) |
# Conflicts: # uv.lock
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
next checkpoint, almost the final: 5b42110
|
|
another checkpoint, f710992 we don't yet have DoA tracking but I added a simple multi-person one:
|
…0 (Python >=3.11)
|
and another big checkpoint (5db721e) that dropped opencv from the tracking runtime, YuNet now runs on onnxruntime with a numpy decode:
|
|
another checkpoint, a5018b0: the tracker is now just a camera client
next to try: drop the worker process and use a plain thread, and it would save ~200 MB of duplicated python/ORT/GStreamer stack |
Adds daemon-side head tracking, that can run out-of-process so it never stalls the 50 Hz control loop, and to coexist with the conversation app on the CPU-bound CM4 / Pi 4 (supersedes the original in-process MediaPipe migration this branch started as)
What changed
cv2.FaceDetectorYN) + a single-target tracker, run in a spawnedmultiprocessing.Process(own GIL/core). Replaces the in-process MediaPipe tracker, which serialised on the GIL and starved the loop.enable_head_tracking(weight=0)pauses (frees the head + detector CPU) without tearing down the process -> cheap per-turn on/off for the conv-app speak/listen handoffvision/mediapipe==0.10.14extra, vision installs via the existing opencv extra. Missing OpenCV raisesImportErrorinstead ofexit(), so it can't kill the robot processValidated on-robot (Wireless/CM4):
DoA-driven search isn't yet in this PR, could be a next PR, as @FabienDanieau suggested.
Closes #1175
Conv app PR: pollen-robotics/reachy_mini_conversation_app#381