Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
3e5f9d1
feat: add vision subpackage with MediaPipe HeadTracker
alozowski Jun 1, 2026
c89faea
fix: add type annotations and make landmark helpers private
alozowski Jun 1, 2026
cc3a6c8
Merge remote-tracking branch 'origin/main' into feat/1175-move-mediapipe
alozowski Jun 25, 2026
02a7287
feat: add BlazeFace head tracker
alozowski Jun 25, 2026
c7c540a
feat: add daemon head tracking controls
alozowski Jun 25, 2026
0b46d50
feat: expose head tracking in SDKs
alozowski Jun 25, 2026
a50cdb1
test: cover daemon head tracking
alozowski Jun 25, 2026
767860f
Merge remote-tracking branch 'origin/main' into feat/1175-move-mediapipe
alozowski Jun 25, 2026
c86a701
fix: keep disabled tracking branch from blocking media startup
alozowski Jun 26, 2026
458e03c
fix: start head tracking without blocking sdk websocket
alozowski Jun 26, 2026
06e4e0e
fix: report daemon head tracking availability
alozowski Jun 26, 2026
4c774ff
fix: smooth daemon head tracking aim
alozowski Jun 26, 2026
e20e07a
fix: reduce head tracking daemon logs
alozowski Jun 26, 2026
69a78f4
test: cover SDK head tracking availability
alozowski Jun 26, 2026
a49229c
docs: update OpenAPI spec for head tracking
alozowski Jun 26, 2026
ff8a0bd
fix: reduce head tracking movement churn
alozowski Jun 26, 2026
a2b578f
fix: reduce IK churn during head tracking
alozowski Jun 26, 2026
c2b33f6
fix: stabilize daemon head tracking with aim gain and aspect-correct …
alozowski Jun 26, 2026
2833c67
fix: remove head-tracking upward bias and hold aim on transient face …
alozowski Jun 26, 2026
9865a95
feat: raise head-tracking rate via mediapipe VIDEO mode and smaller f…
alozowski Jun 26, 2026
6c7a911
feat: cap head-tracking at 15 fps to ease CM4 control-loop contention
alozowski Jun 27, 2026
f880042
refactor: drop in-process head-tracking detector and mediapipe extra,…
alozowski Jun 27, 2026
3aee278
feat: add YuNet face detector and bundle its ONNX model
alozowski Jun 27, 2026
ce98bde
feat: run head-tracking detector out-of-process with per-tick aim easing
alozowski Jun 27, 2026
b7235e1
fix: keep head-tracking detector cheap and low-priority, and track th…
alozowski Jun 28, 2026
5db905d
feat: feed the face tracker a dedicated downscaled camera branch, not…
alozowski Jun 28, 2026
a5217bd
fix: read tracker frames with the try-pull-sample action signal
alozowski Jun 28, 2026
26f7c6c
fix: forward sticky events on the tracking valve so unixfdsink negoti…
alozowski Jun 28, 2026
f1516d4
fix: disable tracking-branch unixfdsink preroll so the valve gate del…
alozowski Jun 28, 2026
9a5b52f
feat: steady head tracking, deadband jitter, hold then recenter on lo…
alozowski Jun 28, 2026
252ac94
fix: lower YuNet score threshold to 0.6 so tracking holds the face th…
alozowski Jun 29, 2026
55b2aa6
feat: ROI-crop face detection to cut worker CPU
alozowski Jun 29, 2026
8c8e3cb
feat: cache YuNet input size + fixed-size ROI crop to skip per-frame …
alozowski Jun 29, 2026
126104c
feat: lower tracking fps to 10 and ROI crop to 192 for CPU headroom u…
alozowski Jun 29, 2026
a3ade9a
feat: pause head tracking at weight 0 to free the head and CPU withou…
alozowski Jun 29, 2026
15cd4cb
Merge remote-tracking branch 'origin/main' into feat/1175-move-mediapipe
alozowski Jun 30, 2026
cb9d5bf
feat: shrink head-tracking ROI crop to 128 (half detection CPU, equal…
alozowski Jun 30, 2026
4e277fa
debug: log tracker fps/hit/locked every 2s (temp)
alozowski Jun 30, 2026
20960b1
chore: remove temp tracker debug logging
alozowski Jun 30, 2026
c3063c2
Merge remote-tracking branch 'origin/main' into feat/1175-move-mediapipe
alozowski Jul 1, 2026
47abfa3
Merge remote-tracking branch 'origin/main' into feat/1175-move-mediapipe
alozowski Jul 6, 2026
a1b9fe2
refactor: fetch YuNet model from HF hub instead of vendoring the onnx
alozowski Jul 6, 2026
8035c94
refactor: drop the ROI crop, detect on the full downscaled frame
alozowski Jul 6, 2026
0c6e9be
perf: lower tracking fps to 15 to halve the detector CPU
alozowski Jul 6, 2026
1109702
feat: serve the local IPC camera feed at a capped framerate below cap…
alozowski Jul 6, 2026
5b42110
perf: lower head-tracking detection rate 15→10 fps to cut worker CPU
alozowski Jul 6, 2026
f710992
feat: switch head-tracking gaze to a more prominent face with hysteresis
alozowski Jul 6, 2026
09c0789
fix: log head-tracking weight at debug
alozowski Jul 7, 2026
c0b1d46
chore: point face detector at the renamed face_detection_yunet_2023ma…
alozowski Jul 7, 2026
130fa85
chore: use the recent _MODEL_REVISION for face_detection_yunet_2023mar
alozowski Jul 7, 2026
ea4ed62
feat: run YuNet on onnxruntime with a numpy decode
alozowski Jul 7, 2026
5db721e
chore: add the onnxruntime vision extra and bump onnxruntime to 1.27.…
alozowski Jul 7, 2026
aa5f72b
Merge remote-tracking branch 'origin/main' into feat/1175-move-mediapipe
alozowski Jul 7, 2026
7ab6ef9
fix: ignore SIGINT in the face-tracking worker so daemon Ctrl+C doesn…
alozowski Jul 7, 2026
0b94275
refactor: make the face tracker a camera-IPC client, aim at the nose,…
alozowski Jul 8, 2026
9869c60
refactor: single fps cap on the IPC feed paces the tracker
alozowski Jul 8, 2026
a5018b0
docs: add head-tracking example and SDK documentation
alozowski Jul 8, 2026
ee267fa
refactor: run the face tracker in a daemon thread instead of a spawne…
alozowski Jul 8, 2026
fe2a77b
docs: resolve the look-at coordinate-frame TODOs and record the aimin…
alozowski Jul 8, 2026
a996d7d
chore: move onnxruntime to main deps and sort the list for clarity
alozowski Jul 8, 2026
a5b4546
refactor: import vision and kinematics deps directly now that onnxrun…
alozowski Jul 8, 2026
84aa23b
docs: drop the vision extra install notes
alozowski Jul 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
181 changes: 181 additions & 0 deletions docs/source/API/openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -1456,6 +1456,114 @@
}
}
},
"/api/media/tracking/enable": {
"post": {
"summary": "Enable Tracking",
"description": "Enable daemon-side visual head tracking.",
"operationId": "enable_tracking_api_media_tracking_enable_post",
"requestBody": {
"content": {
"application/json": {
"schema": {
"anyOf": [
{
"$ref": "#/components/schemas/EnableTrackingRequest"
},
{
"type": "null"
}
],
"title": "Body"
}
}
}
},
"responses": {
"200": {
"description": "Successful Response",
"content": {
"application/json": {
"schema": {
"additionalProperties": {
"anyOf": [
{
"type": "string"
},
{
"type": "boolean"
}
]
},
"type": "object",
"title": "Response Enable Tracking Api Media Tracking Enable Post"
}
}
}
},
"422": {
"description": "Validation Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
}
}
}
}
},
"/api/media/tracking/disable": {
"post": {
"summary": "Disable Tracking",
"description": "Disable daemon-side visual head tracking.",
"operationId": "disable_tracking_api_media_tracking_disable_post",
"responses": {
"200": {
"description": "Successful Response",
"content": {
"application/json": {
"schema": {
"additionalProperties": {
"anyOf": [
{
"type": "string"
},
{
"type": "boolean"
}
]
},
"type": "object",
"title": "Response Disable Tracking Api Media Tracking Disable Post"
}
}
}
}
}
}
},
"/api/media/tracking/face": {
"get": {
"summary": "Get Tracked Face",
"description": "Return the latest face observed by daemon-side head tracking.",
"operationId": "get_tracked_face_api_media_tracking_face_get",
"responses": {
"200": {
"description": "Successful Response",
"content": {
"application/json": {
"schema": {
"additionalProperties": true,
"type": "object",
"title": "Response Get Tracked Face Api Media Tracking Face Get"
}
}
}
}
}
}
},
"/api/media/sounds/upload": {
"post": {
"summary": "Upload Sound",
Expand Down Expand Up @@ -2742,6 +2850,9 @@
}
],
"title": "Hardware Id"
},
"face_target": {
"$ref": "#/components/schemas/FaceTarget"
}
},
"type": "object",
Expand Down Expand Up @@ -2776,6 +2887,76 @@
"title": "DoAInfo",
"description": "Direction of Arrival info from the microphone array."
},
"EnableTrackingRequest": {
"properties": {
"weight": {
"type": "number",
"maximum": 1.0,
"minimum": 0.0,
"title": "Weight",
"default": 1.0
}
},
"type": "object",
"title": "EnableTrackingRequest",
"description": "Request body for enabling daemon-side head tracking."
},
"FaceTarget": {
"properties": {
"detected": {
"type": "boolean",
"title": "Detected",
"default": false
},
"x": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"title": "X"
},
"y": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"title": "Y"
},
"roll": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"title": "Roll"
},
"ts": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"title": "Ts"
}
},
"type": "object",
"title": "FaceTarget",
"description": "Latest daemon-side face target used for head tracking."
},
"FullBodyTarget": {
"properties": {
"target_head_pose": {
Expand Down
2 changes: 1 addition & 1 deletion docs/source/SDK/media-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ The daemon starts its media pipeline automatically unless the `--no-media` flag
1. Opens the camera (platform-aware: v4l2, libcamera, DirectShow, AVFoundation, or UDP for simulation).
2. Opens the audio device (platform-aware: PulseAudio, ALSA, WASAPI, CoreAudio).
3. Feeds both into a WebRTC server (`webrtcsink`) for remote streaming.
4. Exposes raw camera frames via a local IPC endpoint (`unixfdsink` on Linux/macOS, `win32ipcvideosink` on Windows).
4. Exposes raw camera frames via a local IPC endpoint (`unixfdsink` on Linux/macOS, `win32ipcvideosink` on Windows). The IPC feed is served at a capped framerate (10 fps) below the capture rate: local vision clients convert every frame they are served, so the cap keeps on-device CPU low regardless of the camera mode.

[![Reachy Mini Media Daemon](https://github.com/pollen-robotics/reachy_mini/raw/main/docs/assets/reachymini_media_daemon.png)]()

Expand Down
15 changes: 15 additions & 0 deletions docs/source/SDK/python-sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,21 @@ with ReachyMini(media_backend="default") as mini:
```
The returned frame is a numpy array with shape `(height, width, 3)` and data type `uint8`.

### Head Tracking 👀

The daemon can track the closest face and turn the head to follow it (aiming at the nose). Detection runs inside the daemon.

```python
from reachy_mini import ReachyMini

with ReachyMini() as mini:
mini.start_head_tracking()
face = mini.get_tracked_face() # detected, x, y in [-1, 1], roll
mini.stop_head_tracking()
```

`start_head_tracking(weight=...)` blends tracking with application motion: `1.0` lets tracking own the head, `0.0` pauses detection (freeing the head and CPU) without stopping the tracker, so applications can toggle it cheaply per turn. See the [Head Tracking example](../examples/head_tracking.md).

### IMU 🧭

> ⚠️ The IMU is only available with the wireless version of Reachy Mini
Expand Down
2 changes: 2 additions & 0 deletions docs/source/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,8 @@
title: Minimal demo
- local: examples/look_at
title: Look at
- local: examples/head_tracking
title: Head Tracking
- local: examples/sequence
title: Sequence
- local: examples/goto_interpolation_playground
Expand Down
18 changes: 18 additions & 0 deletions docs/source/examples/head_tracking.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Head Tracking

This example enables daemon-side head tracking: Reachy Mini turns its head to follow the closest face, aiming at the nose. Detection runs inside the daemon (YuNet on ONNX Runtime), so the script only toggles tracking and polls the latest face target.

Run with:
```bash
python head_tracking.py
```

`start_head_tracking(weight=...)` accepts a blend factor: `1.0` lets tracking own the head, `0.0` pauses detection (freeing the head and CPU) without stopping the tracker — useful to hand the head back to an application between conversation turns.

<literalinclude>
{"path": "../../../examples/head_tracking.py",
"language": "python",
"start-after": "START doc_example",
"end-before": "END doc_example"
}
</literalinclude>
30 changes: 30 additions & 0 deletions examples/head_tracking.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
"""Enable daemon-side head tracking and print the tracked face target.

Reachy Mini turns its head to follow the closest face. Detection runs inside
the daemon, so this script only toggles tracking and reads the result.

Note:
The daemon must be running before executing this script.

"""

# START doc_example

from reachy_mini import ReachyMini

with ReachyMini() as mini:
mini.start_head_tracking()
try:
while True:
# Waits for the next daemon status broadcast, so the loop runs at ~1 Hz.
face = mini.get_tracked_face()
if face.detected:
print(f"Face at x={face.x:+.2f}, y={face.y:+.2f}")
else:
print("No face detected")
except KeyboardInterrupt:
pass
finally:
mini.stop_head_tracking()

# END doc_example
51 changes: 26 additions & 25 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,38 +8,41 @@ version = "1.9.0rc1"
authors = [{ name = "Pollen Robotics", email = "contact@pollen-robotics.com" }]
description = ""
readme = "README.md"
requires-python = ">=3.10"
requires-python = ">=3.11"
dependencies = [
"numpy>=2.2.5",
"scipy>=1.15.3, <2.0.0",
"reachy_mini_motor_controller>=1.5.5",
"psutil",
"jinja2",
"uvicorn[standard]",
"aiohttp>=3.14.0",
"asgiref",
"fastapi",
"pydantic>=2.12.5,<3.0.0",
"python-multipart>=0.0.27",
"starlette<1.0.0", # TODO: update TemplateResponse(request, name, context) before allowing Starlette 1.x
"pyserial",
"huggingface-hub>=1.17.0,<2.0.0",
"reachy-mini-rust-kinematics>=1.0.3",
"asgiref",
"aiohttp>=3.14.0",
"log-throttling==0.0.3",
"pyusb>=1.2.1",
"jinja2",
"libusb_package>=1.0.26.3",
"log-throttling==0.0.3",
"numpy>=2.2.5",
"onnxruntime==1.27.0",
"pip>=26.1",
"platformdirs",
"rich",
"questionary",
"websockets>=12,<16",
"toml",
"rustypot>=1.4.2",
"psutil",
"pulsectl>=24.1.0; sys_platform == 'linux'",
"pycaw>=20240210; sys_platform == 'win32'",
"pydantic>=2.12.5,<3.0.0",
"pyserial",
"python-multipart>=0.0.27",
"pyusb>=1.2.1",
"pyyaml>=6.0",
"questionary",
"requests>=2.33.0",
"rich",
"rustypot>=1.4.2",
"scipy>=1.15.3, <2.0.0",
"starlette<1.0.0", # TODO: update TemplateResponse(request, name, context) before allowing Starlette 1.x
"toml",
"uvicorn[standard]",
"websockets>=12,<16",
"zeroconf>=0.149.7,<1",
"pycaw>=20240210; sys_platform == 'win32'",
"pulsectl>=24.1.0; sys_platform == 'linux'",

# Reachy Mini specifics
"reachy-mini-rust-kinematics>=1.0.3",
"reachy_mini_motor_controller>=1.5.5",

"gstreamer-bundle==1.28.3; sys_platform != 'linux'",
# OS-specific: use PyGObject for Linux
Expand All @@ -55,7 +58,6 @@ examples = [
"gradio-client",
]
mujoco = ["mujoco==3.3.0"]
nn_kinematics = ["onnxruntime==1.22.1"]
placo_kinematics = [
"placo==0.9.14; sys_platform != 'win32'",
"tornado>=6.5.5; sys_platform != 'win32'", # security floor for meshcat transitive dependency
Expand All @@ -76,7 +78,6 @@ opencv = ["opencv-python<=5.0", "cv2_enumerate_cameras>=1.2.1"]
all = [
"reachy_mini[examples]",
"reachy_mini[mujoco]",
"reachy_mini[nn_kinematics]",
"reachy_mini[placo_kinematics]",
"reachy_mini[rerun]",
"reachy_mini[wireless-version]",
Expand Down
Loading
Loading