This document explains, in complete detail, how video playback was added to a Fyne application by embedding libmpv — and crucially, how this was made to work on Wayland, where the traditional "embed a video window into another window" trick does not exist.
It is written to be read top to bottom by someone who has never seen this code. Every file, every function, and every non-obvious decision is described.
- The problem, and why it is hard on Wayland
- The solution in one paragraph
- The two Fyne 2.8 features that make it possible
- Architecture overview
- The data/control flow of a single frame
- File-by-file reference
- The tricky details
- Build & run instructions
- What was verified, and what is still open
- Glossary
Goal: show a playing video inside a Fyne window/container/widget, and have it work on Wayland, ideally by embedding mpv.
Why the "obvious" approach fails on Wayland. The classic way to embed a
video player is window embedding: you create a child OS window and tell mpv to
render into it by passing its window ID (X11's --wid option). The host GUI
reserves a rectangle, and the video player draws into that rectangle as a
separate native window.
This model does not exist on Wayland. Wayland deliberately has no concept of "here is a numeric ID of an arbitrary window belonging to another process; draw into it." A Wayland client can only manage its own surfaces. The only sanctioned way to composite another process's output is an explicit subsurface protocol plus a nested compositor — heavy, fragile, and not something a toolkit like Fyne exposes. This is the crux of the long-standing requests:
- Fyne feature request: fyne-io/fyne#449
- libmpv on Wayland: mpv-player/mpv#1242
So --wid-style embedding is a dead end here.
Do not embed a window at all. Embed pixels.
libmpv offers a second integration path called the Render API
(mpv/render.h). Its software flavour (MPV_RENDER_API_TYPE_SW) renders each
video frame into a plain memory buffer that we own, in a packed RGB format
of our choosing. That buffer is, byte for byte, a Go image.RGBA.
Fyne 2.8 added canvas.Shader, a canvas object drawn with a custom GLSL
fragment shader that can be handed Go images as sampler2D textures. So the
whole integration is:
- mpv renders a frame into our buffer (CPU, no GL involved),
- we wrap the buffer in an
image.RGBAand set it as the shader's texture, - Fyne's painter uploads it to the GPU and runs our fragment shader, which samples the texture with aspect-correct letterboxing.
Because this never touches the windowing system, it behaves identically on X11,
Wayland, macOS and Windows. The Wayland --wid problem simply never arises:
there is no second window to embed. And because canvas.Shader is a built-in
Fyne 2.8 primitive, no Fyne fork, patch or replace directive is needed.
Two facts, one from each project, make this possible.
From mpv/render.h: a render context created with
MPV_RENDER_PARAM_API_TYPE = MPV_RENDER_API_TYPE_SW renders frames to memory
surfaces. Each mpv_render_context_render() call takes four
MPV_RENDER_PARAM_SW_* parameters describing the target:
SW_SIZE— target width/height in pixels,SW_FORMAT— pixel format by name; we use"rgb0"(bytes r, g, b, then one unused byte per pixel),SW_STRIDE— bytes per line (we usewidth * 4),SW_POINTER— the destination buffer.
"rgb0" rows top-to-bottom is exactly Go's image.RGBA memory layout
(Pix []uint8 in r,g,b,a order, top row first), so the buffer needs zero
conversion before Fyne sees it. Two caveats, both handled later:
- The 4th byte is documented as uninitialized garbage, so the sampling shader must force alpha to opaque rather than trust it.
- Everything — colour conversion, scaling, OSD — happens on the CPU. For a demo this is fine; it is the documented price of avoiding GL interop.
The API is thread-flexible: there is no GL context to bind, so the render
context can be created eagerly at startup and rendered from any single thread
(we use Fyne's main thread, via fyne.Do).
Like the GL flavour, it supports mpv_render_context_set_update_callback —
libmpv calls us (from any thread) whenever a new frame should be rendered — and
mpv_render_context_update(), which reports whether a new frame is actually
available (MPV_RENDER_UPDATE_FRAME).
canvas.Shader (in fyne.io/fyne/v2/canvas) is a first-class canvas object:
type Shader struct {
Name string
Source []byte // GLSL fragment shader, desktop OpenGL
SourceES []byte // GLSL fragment shader, OpenGL ES / mobile / web
Textures map[string]image.Image
Uniforms map[string]float32
}Source/SourceESare compiled once per uniqueNameand cached by the painter, so per-frame use costs no recompilation.Texturesentries are uploaded to the GPU and exposed asuniform sampler2D <name>. Upload happens once per distinct image: replacing an entry with a differentimage.Imagevalue re-uploads on the next paint, while re-setting the same pointer costs nothing. (This drives the "fresh image header per frame" pattern in §6.1.)Uniformsentries are exposed asuniform float <name>and applied every paint — we use one (aspect) to drive letterboxing.- The shader is given Fyne's standard vector-shader uniforms:
uniform vec2 frame(output size in pixels) anduniform vec4 bounds(this object's bounds x1,y1,x2,y2 in pixels, y-down Fyne coordinates), and is expected to compute colour fromgl_FragCoord.
The painter has a fast path for *image.RGBA textures — it passes Pix
straight to glTexImage2D with no per-pixel conversion — which is what makes
the mpv rgb0 buffer essentially free to upload.
Everything lives in one module — there is no fork and no submodule:
fyne-mpv-video/
├── go.mod plain fyne.io/fyne/v2 v2.8.0 requirement, no replace
├── mpv.go libmpv cgo bindings + software Render API → image.RGBA
├── video_widget.go reusable Video widget: canvas.Shader + playback controls
├── main.go tiny entry point
└── FyneApp.toml app metadata (declares the fyneDo migration)
The boundary:
┌─────────────────────── fyne v2.8.0 (upstream) ───────────────┐
│ canvas.Shader (a CanvasObject) │
│ • Textures["tex"] ← image.RGBA wrapping mpv's buffer │
│ • Uniforms["aspect"] ← display aspect ratio │
│ GL painter: │
│ • compiles our fragment shader once (keyed by Name) │
│ • uploads each new texture, applies uniforms, draws │
└───────────────────────────┬───────────────────────────────────┘
│ implements videoController
┌───────────────────────────┴──────────── demo app ────────────┐
│ mpvPlayer (cgo → libmpv) │
│ • Frame() → mpv_render_context_render(SW buffer) │
│ • Aspect / Play / Pause / Seek / Position / Duration │
│ Video widget: Shader + play button + seek bar + clock │
└───────────────────────────────────────────────────────────────┘
The key architectural rule: Fyne itself is untouched. The app depends only
on public, upstream Fyne 2.8 API (canvas.Shader), and only on public libmpv
API (the software Render API). Either side can be upgraded independently.
Here is the complete life of one video frame, end to end:
-
mpv decodes a frame on its own internal threads. When a new frame is ready, libmpv invokes the update callback we registered (
goRenderUpdate). This can happen on any thread. -
We mark "needs paint" and ask the widget.
goRenderUpdatesets an atomic flag and calls the app-suppliedonUpdateclosure, which pushes into a buffered (size 1) channel — coalescing bursts of signals. -
The widget's
refreshLoopgoroutine receives the signal and marshalsupdateFrameonto Fyne's main thread withfyne.Do. -
We render mpv into our buffer.
updateFramecallsplayer.Frame(), which checks the atomic flag andmpv_render_context_update(), then callsmpv_render_context_renderwith theSW_*params pointing at a reusable byte slice sized to the video's display dimensions (dwidth×dheight— after aspect/rotation correction, so no bars are baked in). mpv writes the frame as packedrgb0. -
We hand the frame to the shader.
Frame()wraps the buffer in a freshimage.RGBAheader (same backing array) andupdateFrameassigns it toshader.Textures["tex"], updatesshader.Uniforms["aspect"], and callsshader.Refresh(), marking the object dirty. -
Fyne repaints. On the render pass, the painter sees a new image value in
Textures, uploadsPixviaglTexImage2D(RGBA fast path), setsframe,boundsandaspectuniforms, and draws the object's rectangle with our fragment shader. The shader convertsgl_FragCoordto texture UVs, outsets the UVs to letterbox to the video's aspect, discards out-of-range fragments (bars = window background), and samples the texture, forcing alpha to 1. -
Repeat. Step 1 fires for every new mpv frame, so playback is smooth.
All GL work is done by Fyne's painter on the correct thread; all mpv render
calls happen on Fyne's main thread (step 4 runs inside fyne.Do); the only
cross-thread signals are the atomic flag and the buffered channel.
The only file containing cgo and libmpv. It implements the widget's
videoController interface.
#cgo pkg-config: mpv pulls in compiler/linker flags. It includes client.h
and render.h — notably not render_gl.h, which the software renderer
does not need. Three small C helpers exist because cgo has awkward edges
around tagged-union arrays and function pointers:
make_sw_init_params()— builds the parameter array formpv_render_context_create:API_TYPE = MPV_RENDER_API_TYPE_SW,ADVANCED_CONTROL = 1(enables the update-driven model andmpv_render_context_update), terminator.render_to_buffer(ctx, buf, w, h)— assembles the fourMPV_RENDER_PARAM_SW_*params (size,"rgb0"format, stridew*4, pointer) and callsmpv_render_context_render. The target size is the video's display size, so mpv does no scaling and adds no bars of its own.set_update_callback— registers the Go-exportedgoRenderUpdatewith an opaque context pointer.
type mpvPlayer struct {
mpv *C.mpv_handle
render *C.mpv_render_context
needsPaint atomic.Bool
onUpdate func()
self cgo.Handle
buf []byte // frame buffer in rgb0 layout
bufW, bufH int
stop chan struct{}
stopOnce sync.Once
done chan struct{}
}mpv— the core libmpv handle;render— the software render context.needsPaint— set by the update callback, cleared byFrame().self— aruntime/cgo.Handle(see §7.2) so C can call back into this Go object safely.buf/bufW/bufH— the reusable frame buffer and its current dimensions; reallocated only when the video size changes.
mpv_create+mpv_initialize(fatal path on error).- Options:
hwdec=auto-safe(GPU decode when available; frames are still copied back to RAM for software rendering),vo=libmpv(select Render API output instead of opening a window). - Creates the render context eagerly — unlike the GL renderer, software rendering has no context-current thread rule, so there is no reason to wait for a paint callback. Registers the update callback.
- Starts the event-drain goroutine (see §7.5), then issues
loadfile.
Called only on Fyne's main thread (from the widget's updateFrame):
- Bail out if
needsPaintis clear ormpv_render_context_update()lacksMPV_RENDER_UPDATE_FRAME— returningnilmeans "nothing new", and the widget skips the refresh. - Read
dwidth/dheight(display size, i.e. after aspect and rotation correction). Bail if not yet known. - Reallocate
bufif the size changed. render_to_buffer— mpv writes the frame asrgb0.- Return
&image.RGBA{Pix: p.buf, Stride: w*4, Rect: ...}— a new header each call. This matters: the painter re-uploads a shader texture only when the image value differs from last time, so reusing one header would freeze the first frame on screen (§7.3).
displaySize reads the dwidth/dheight properties (zeros while unknown).
Aspect returns their ratio, or 0 — which the widget turns into "fill the
object" until the real ratio arrives.
Thin wrappers over mpv properties/commands, unchanged in spirit from any mpv client:
Play/Pause/TogglePause/IsPaused— thepauseflag property.Position—time-pos;Duration—duration.SeekTo(s)—seek <s> absolute.getPropertyDouble/getPropertyFlag/setPropertyFlag/command/setOption— cgo plumbing formpv_get_property/mpv_set_property/mpv_command/mpv_set_option_string, guarding against a closed handle.checkMPV— negative status → Go error viampv_error_string.
Stops the event loop first and waits for it (done) so it never calls
mpv_wait_event on a freed handle; then frees the render context, destroys the
core, deletes the cgo.Handle, and drops the buffer. Guarded so it is safe to
call once.
goRenderUpdate(ctx) (//export) — libmpv's "new frame" signal. Recovers
the *mpvPlayer from the cgo.Handle, sets needsPaint, and calls
onUpdate. May run on any thread, so it does no rendering — it only flags and
signals. (The GL-era goGetProcAddress export is gone; software rendering
needs no GL entry points.)
Wraps a canvas.Shader with playback controls into a proper Fyne widget.
videoController interface — what the widget needs from its backend:
type videoController interface {
SetOnUpdate(func())
Frame() *image.RGBA // nil when no new frame
Aspect() float32
Play(); Pause(); TogglePause() bool; IsPaused() bool
Position() float64; Duration() float64; SeekTo(float64)
Close()
}*mpvPlayer satisfies this, but the widget has no libmpv/cgo import — any
backend that can produce image.RGBA frames will do (a test source, a
different player library, …).
videoShaderSource (desktop, #version 110) and videoShaderSourceES
(#version 100, ES) implement the same logic:
uniform vec2 frame;
uniform vec4 bounds;
uniform float aspect;
uniform sampler2D tex;
void main() {
vec2 uv = vec2(
(gl_FragCoord.x - bounds[0]) / (bounds[2] - bounds[0]),
(frame.y - gl_FragCoord.y - bounds[1]) / (bounds[3] - bounds[1]));
if (aspect > 0.0) {
float objAspect = (bounds[2] - bounds[0]) / (bounds[3] - bounds[1]);
uv = (uv - 0.5) * vec2(max(objAspect / aspect, 1.0),
max(aspect / objAspect, 1.0)) + 0.5;
}
if (uv.x < 0.0 || uv.x > 1.0 || uv.y < 0.0 || uv.y > 1.0) {
gl_FragColor = vec4(0.0);
} else {
gl_FragColor = vec4(texture2D(tex, uv).rgb, 1.0);
}
}Line by line:
- UV mapping.
boundsare in Fyne's y-down pixel space;gl_FragCoordis y-up. So u is measured from the left edge, and v from the top edge viaframe.y - gl_FragCoord.y. Fyne uploads images with the first row at texture v=0, so top-based v samples the frame right side up (§7.4). - Letterboxing. Stretching the UV window past [0,1] on the axis where the
object is proportionally too long, then rejecting out-of-range fragments,
shrinks the sampled image to the video's aspect and centres it — pillarbox
or letterbox as needed.
aspect == 0(unknown) skips this and fills. - Bars. Out-of-range fragments get
vec4(0.0)— fully transparent, so the window background shows through, matching how the old GLVideo painter left bars undrawn. - Forced alpha. mpv's
rgb0leaves the 4th byte garbage; the sample's RGB is used with alpha forced to 1 (§7.6).
The ES variant only adds the mandatory precision declaration.
NewVideo(player):
- Creates the shader (
canvas.NewShader("mpv-video", src, srcES)) with a 320×180 minimum size, an empty texture map, andaspect = 0. - Builds the play/pause button, clock label, and seek slider — including the
settingSeekguard that stops the ticker's programmatic slider updates from being mistaken for user drags (which used to cause a periodic seek-stall). - Registers
SetOnUpdateto push into the bufferedframeReadychannel. - Starts two goroutines:
refreshLoop(frames) andtick(UI clock).
refreshLoop → fyne.Do(v.updateFrame) per signal. updateFrame pulls
player.Frame(); on a non-nil image it assigns Textures["tex"], updates
Uniforms["aspect"], and calls shader.Refresh().
tick (500 ms) updates the clock label and — unless the user is dragging —
moves the slider to pos/duration.
Close() closes stop (guarded, idempotent) and closes the player.
CreateRenderer lays out the shader in the centre of a Border with the
controls along the bottom.
- Requires one CLI argument: the video file or URL.
newMPVPlayer(file)(fatal on error).app.NewWithID("io.fyne.mpvdemo"), creates the window and theVideowidget,SetCloseInterceptsovideo.Close()runs before window close.- Sizes the window 800×520 and
ShowAndRun().
require fyne.io/fyne/v2 v2.8.0
That is the whole story: upstream Fyne 2.8, no replace directive, and no
direct GLFW dependency (the old GL backend needed glfw.GetProcAddress and
native display accessors; software rendering needs neither). GLFW is still
present as an indirect dependency — Fyne's own desktop driver uses it.
App metadata plus [Migrations] fyneDo = true, declaring the code is written
against the fyne.Do threading model.
The update callback may fire on any thread, but mpv_render_context_render is
only ever called inside fyne.Do(v.updateFrame) — Fyne's main thread, the
same place painting happens. Two birds with one stone: mpv render calls are
serialized on a single thread (the software API's requirement), and the frame
buffer is never touched concurrently (the painter uploads the texture on that
same thread). No locks needed around buf.
C needs to call back into a specific *mpvPlayer (the update callback's
context). You must not hand a raw Go pointer to C to store — the GC may
move or collect it. runtime/cgo.Handle returns a stable integer that keeps
the object alive; we pass it as an opaque void* and recover the object in the
callback. Close deletes the handle.
bindShaderTextures compares the new Textures entry with the cached one and
re-uploads only if the image value differs. Frame() therefore returns a
new image.RGBA struct (new pointer, same Pix backing array) every call:
cheap, no copy, and the painter sees "a different image" and uploads the new
pixels. Returning the same pointer each time would upload once and never again.
Three parties disagree about "up": mpv's software renderer writes rows
top-to-bottom; Go images are top-row-first; OpenGL texture v=0 is
conventionally "bottom" — but Fyne uploads image rows in order, putting the
first row at v=0, and its built-in image drawing treats v=0 as the top. Our
shader follows Fyne's convention: v measured from the top of bounds (via
frame.y - gl_FragCoord.y) samples row 0 of the upload, which is the top of
both the Go image and mpv's output. Net effect: the video is right side up.
(The SW renderer ignores FLIP_Y entirely, per the mpv docs, so there is no
mpv-side switch to set — orientation is purely a shader concern.)
libmpv buffers events in a fixed-size queue; never consuming them makes mpv
throttle and stall playback ("plays a few seconds, pauses, resumes"). A
dedicated goroutine loops on mpv_wait_event with a 100 ms timeout (so it can
observe stop), and Close waits for that goroutine to exit before destroying
the handle.
mpv documents the 4th byte of rgb0 as uninitialized. Two places could trip
on it: the texture upload (it doesn't — RGBA upload just copies bytes) and
blending (it would — the painter blends with standard src-alpha). The shader
sidesteps both by outputting vec4(sample.rgb, 1.0).
We render at dwidth×dheight — the display dimensions after aspect-ratio
and rotation correction — rather than the coded width×height. Two wins:
anamorphic/rotated videos come out pre-corrected, and mpv does no scaling or
bar-baking of its own (the SW docs note mpv letterboxes into mismatched
targets), so the shader receives exactly the image it should letterbox itself.
Programs are cached by Shader.Name for the life of the GL context, including
cached failures. Using one stable name ("mpv-video") means the shader is
compiled once, at first paint, and every frame after that is just a texture
upload plus uniform sets.
An earlier iteration rendered GPU-side: libmpv's OpenGL Render API drew into
an FBO owned by a custom canvas.GLVideo type carried in a Fyne fork
(submodule + replace directive), with per-backend painter files
(glvideo_desktop.go, glvideo_gles.go, glvideo_other.go) and native
display plumbing (nativedisplay_*.go) for zero-copy hwdec. The software
renderer + canvas.Shader replaces all of that with upstream API at the cost
of a per-frame GPU→CPU→GPU round trip (§7.10). Deleted pieces: the fork
submodule, the four nativedisplay_*.go files, the get_proc_address bridge,
and the goGetProcAddress export.
Software rendering means colour conversion on the CPU and a full-frame texture
upload every frame (memcpy out of mpv, glTexImage2D back up). For typical
desktop video sizes this is comfortably realtime — the demo plays 30 fps
content smoothly — but it is not what you want for 4K/HDR or battery-critical
playback, and hwdec only saves decode effort, not the copy. The upside is
simplicity and portability: no fork, no GL context sharing, no per-platform GL
interop, and it works identically under every Fyne backend including ones
where sharing GL contexts with mpv is awkward.
For a quantified version of this analysis — and a concrete proposal for the
smallest Fyne change that would restore zero-copy — see
LIMITATIONS.md, and the glvideo-fork branch for the
zero-copy implementation it refers to.
Prerequisites:
- Go (built/tested with 1.26; module declares 1.22).
- A C toolchain (gcc) — cgo is required.
libmpv-dev(provideslibmpv.soand headers;pkg-config mpvmust work):sudo apt install libmpv-dev- The usual Fyne/GLFW system deps (X11/Wayland/OpenGL dev packages).
Build and run:
go build .
./fyne-mpv-video /path/to/video.mp4
You can also pass a URL that mpv understands. To generate a quick test clip:
ffmpeg -f lavfi -i testsrc=duration=30:size=640x360:rate=30 -pix_fmt yuv420p test.mp4
No special tags are needed by this app — it contains no GL code of its own.
Fyne's own tags behave as usual (e.g. -tags wayland to link only Wayland in
GLFW, -tags gles for the GLES painter; the ES shader variant is selected
automatically on those targets).
Verified working (on this machine, a Wayland/sway session):
- Video renders live inside the Fyne window via the libmpv software Render
API +
canvas.Shader, on a native Wayland client (no XWayland). - Colours match an independently decoded reference frame bar-for-bar; the video is right side up; letterbox/pillarbox bars are symmetric and correct.
- Playback advances (a headless Go test drove
Frame()directly: first frame 640×360, aspect 1.7778, position advancing, ~30 frames/s for 30 fps content). - Duration is read correctly and the position/seek bar tracks playback; the play/pause button toggles state and icon.
- The build is clean against upstream
fyne.io/fyne/v2 v2.8.0with noreplacedirective.
Not yet done / open items:
- CPU/GPU cost — software rendering trades away zero-copy playback
(§7.10). A GL-interop path would need something like the old fork's
canvas.GLVideo(see fyne-io/fyne#449). - The ES shader variant is untested — written by direct translation of
the desktop shader, but not yet run on a
gles/mobile/web target. - The
Videowidget lives in the demo. Promoting it to a reusable package would just mean exporting it; the backend interface is already mpv-free. - No audio-specific controls (volume/mute) or track selection yet — mpv supports them; they would be more property wrappers like the existing ones.
- Error surfacing is minimal —
Frame()silently returns nil on render errors. A production widget would expose load/playback errors to the UI.
- libmpv Render API — mpv's headless rendering interface (
render.h): mpv draws frames into a target you provide instead of managing its own window. The software flavour (MPV_RENDER_API_TYPE_SW) targets memory buffers; the OpenGL flavour targets FBOs in your GL context. rgb0— a packed 4-bytes-per-pixel format (r, g, b, unused) offered by the software renderer; byte-identical to Go'simage.RGBAlayout.canvas.Shader— Fyne 2.8's built-in canvas object drawn with a custom GLSL fragment shader, withTextures(images →sampler2Duniforms) andUniforms(floats) maps.--wid/ window embedding — the old X11 technique of telling a player to draw into another program's window by numeric ID. Not supported on Wayland; the reason a different approach was required.- VO (video output) — mpv's output driver.
vo=libmpvselects the embedding mode used by the Render API. dwidth/dheight— mpv properties: the video's display dimensions after aspect-ratio and rotation correction.- cgo.Handle — Go's mechanism for safely giving C a stable reference to a Go object across the cgo boundary.
- Letterbox / pillarbox — bars added top/bottom or left/right to fit a video of one aspect ratio into a box of another without stretching. Here the bars are the window background showing through transparent fragments.