335 head wobbler is now handled on the core side - #336
Merged
alozowski merged 15 commits intoJun 4, 2026
Conversation
The reachy_mini SDK now ships its own audio-reactive head wobbler that runs inside the daemon's GStreamer pipeline. Audio pushed by the conversation app via push_audio_sample already flows through that same pipeline, so the daemon can drive head motion directly with no need for the app to compute speech offsets in-process. Replace the local HeadWobbler instance with calls to mini.enable_wobbling() / mini.disable_wobbling() at startup and shutdown. Drop the head_wobbler argument from ToolDependencies() construction; consumers that still reference deps.head_wobbler will see None and no-op via their existing guards (those guards are removed in follow-up commits). Behaviour change: in --gradio mode, audio is rendered in the browser and never reaches the daemon's pipeline, so the head no longer wobbles during gradio playback. The non-gradio (production) path is unchanged in spirit and gains the new wobbler's stricter silence handling.
The local HeadWobbler is gone; deps.head_wobbler is always None. Remove the now-dead "if head_wobbler is not None" blocks that called .reset(), .request_reset_after_current_audio(), and .feed() during turn-handling and audio-delta dispatch. The daemon's wobbler reacts to the audio stream itself, so explicit reset hooks are no longer needed; silence in the audio stream produces silence in motion automatically. Also drops two tests that were exercising those exact call sites: - test_tool_completion_does_not_reset_head_wobbler - test_output_audio_done_schedules_head_wobbler_reset Both asserted on MagicMock head_wobbler interactions that no longer happen.
LocalStream's audio path no longer needs to feed a local wobbler with a playback-delay-aware start offset; the daemon's wobbler analyses the same audio that we hand off via push_audio_sample. Remove the "if head_wobbler is not None" block and the now-unused _estimate_pending_playback_seconds helper that only existed to compute that start_delay_s. Drops the matching test test_play_loop_feeds_head_wobbler_with_local_playback_delay in tests/test_console.py.
Removes everything that exists only to support the in-process speech
tapper that the daemon now replaces:
- Delete src/reachy_mini_conversation_app/audio/{head_wobbler.py,
speech_tapper.py,__init__.py} and the now-empty audio/ package.
- Delete tests/audio/test_head_wobbler.py and the audio/ test dir.
- Drop head_wobbler: Any | None = None from ToolDependencies.
- In MovementManager, drop set_speech_offsets() and the supporting
state (state.speech_offsets, _pending_speech_offsets,
_speech_offsets_lock, _speech_offsets_dirty, the speech-side of
_apply_pending_offsets, the speech term in _get_secondary_pose).
Speech-induced offsets are now produced by the daemon's wobbler
and composed server-side via SetSpeechOffsetsCmd, so the local
composition path is no longer needed.
Updates the only remaining test that mocked head_wobbler:
test_gemini_turn_buffers_transcripts (was
test_gemini_turn_buffers_transcripts_and_schedules_motion_reset).
Drops the wobbler mock and assertions, keeps the transcript and
listening-state assertions which still describe real behaviour, and
swaps the wait-for predicate to use output_queue size and listening
call count instead of wobbler interaction.
|
🗑️ Preview space removed — PR was merged. |
Drop three imports that became unused once the head_wobbler-only test cases were removed (numpy and pytest in tests/test_console.py, base64 in tests/test_openai_realtime.py). Re-collapse a wrapped lambda in tests/test_gemini_live.py back onto one line per ruff format. Pure tooling fix; no behaviour change. CI ruff check now passes.
RemiFabre
force-pushed
the
335-head-wobbler-is-now-handled-on-the-core-side
branch
from
April 29, 2026 13:59
9be0334 to
5f56e1a
Compare
RemiFabre
marked this pull request as ready for review
April 29, 2026 14:00
The merge of main brought in base_realtime.py and huggingface_realtime tests authored against the previous ToolDependencies.head_wobbler field, which this branch had already deleted. Remove the three lifted blocks in base_realtime.py, drop the now-meaningless wobbler test in test_huggingface_realtime.py, fix the dangling head_wobbler symbol in test_gemini_live.py, and clear unused imports the merge resolution left in test_console.py. Assisted-by: Claude:claude-opus-4-7
Assisted-by: Claude:claude-opus-4-7
alozowski
requested changes
May 21, 2026
alozowski
left a comment
alozowski
left a comment
Collaborator
There was a problem hiding this comment.
It's nice to see how cleaner the project is with this work! The main thing before merging: Gradio wobbling is silently broken – let's fix it minimally, even despite we are going to remove it very soon
| @@ -810,8 +806,6 @@ async def _run_realtime_session(self) -> None: | |||
| if event.type == "response.output_audio.delta": | |||
Collaborator
There was a problem hiding this comment.
In Gradio mode, we used to feed each audio chunk to the head wobbler here, but with the current changes, the head won't wobble during the robot's speech in Gradio mode. Let's restore the Gradio mode functionality, please (though it will be removed soon)
Contributor
There was a problem hiding this comment.
There is now a _tap_audio_for_daemon_wobbler function that foward audio samples to reachy mini so the wobbling works (and the audio output of the robot is muted)
Comment on lines
-1
to
-7
| """Moves head given audio samples.""" | ||
|
|
||
| import time | ||
| import queue | ||
| import base64 | ||
| import logging | ||
| import threading |
In Gradio mode the assistant audio plays in the browser and never reaches the daemon's media pipeline, so the daemon-side wobbler had nothing to react to and the head sat still. Push each decoded delta through robot.media.push_audio_sample so the wobbler can tap it, and start the playback pipeline (LocalStream did this in headless mode; nothing was doing it in Gradio mode, leaving the gstreamer appsrc in NULL state). Silence the robot speaker for the session via SetVolumeCmd to avoid double playback, and restore the previous volume on shutdown. Also addresses two review comments on PR #336: narrows the enable_wobbling() comment to describe both modes, and trims the past-change sentence from MovementManager._get_secondary_pose's docstring. Assisted-by: Claude:claude-opus-4-7
Contributor
My solution is to also push audio sample to a mute reachy mini in the case of gradio. So we still use the core wobbler |
The merge of main reintroduced test code wiring a head_wobbler into ToolDependencies and asserting the OpenAI handler drives it. This branch moved head wobbling to the core side: ToolDependencies no longer has that field and the handler never touches a wobbler. Drop the head_wobbler param, the ToolDependencies kwarg, the obsolete reset test, and the now unused base64 import. Assisted-by: Claude:claude-opus-4-8
requests is used directly in main.py but ships no type stubs, so mypy fails with import-untyped in the project environment. Add types-requests to the dev group. Assisted-by: Claude:claude-opus-4-8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Drop the in-app head wobbler, use the core daemon's instead.
The
reachy_miniSDK now ships its own audio-reactive head wobbler that runs inside the daemon's media pipeline. Since we already usedpush_audio_sample, this PR is very simple. The local DSP, threading, scheduling, and offset-application code we maintained in the app are no longer needed.Roughly 970 deletions vs 11 insertions.
What changes
main.py: replace the localHeadWobblerlifecycle withrobot.enable_wobbling()/robot.disable_wobbling().gemini_live.py,openai_realtime.py): drop theif deps.head_wobbler is not None:blocks that called.reset(),.request_reset_after_current_audio(), and.feed(). The daemon's wobbler reacts to the audio stream itself, so explicit reset hooks are unnecessary.console.py: drop thehead_wobbler.feed_pcmcall and the_estimate_pending_playback_secondshelper that only existed to compute itsstart_delay_s.src/reachy_mini_conversation_app/audio/:head_wobbler.py,speech_tapper.py, and the package itself.MovementManager: dropset_speech_offsetsand the supporting state (_pending_speech_offsets,_speech_offsets_lock,_speech_offsets_dirty,state.speech_offsets, the speech term in_apply_pending_offsetsand_get_secondary_pose). Speech offsets are now composed server-side viaSetSpeechOffsetsCmd; the local composition path is gone.ToolDependencies: drop thehead_wobblerfield.tests/audio/test_head_wobbler.py, the two head-wobbler-only tests intests/test_openai_realtime.py, the playback-delay test intests/test_console.py, and updatetests/test_gemini_live.py's remaining wobbler-aware test to drop the wobbler mock while keeping its transcript and listening-state coverage.Mathematical equivalence
Side note: the implementation on the core side is almost identical to the one we had here. The only change is an improvement, basically:
final_new = compose_world_offset( compose_world_offset(primary, face), speech )
final_old = compose_world_offset( primary, single_matrix(speech + face) )
-> The impact should not be noticeable but it's still an improvement.
Behaviour change to be aware of
In
--gradiomode, audio is rendered in the browser and never reaches the daemon's media pipeline, so the head no longer wobbles in that path. Production (non-gradio) paths usepush_audio_sample, which routes audio through the daemon's pipeline and the wobbler works.Test plan
pytest tests/test_console.py tests/test_openai_realtime.pypasses (24 / 24).Notes
linked PR from the SDK side: pollen-robotics/reachy_mini#1001