Commit ca1bda5
feat(vision-mixer): per-source crop/zoom (punch-in) for PiP compositions (#622)
* feat(vision-mixer): per-source crop/zoom (punch-in) for PiP compositions
Add vMix-style source cropping to the vision mixer: each input can carry
a normalized crop window per PiP, scaling the cropped region to fill its
zone box (e.g. a portrait box framing a person's head from a landscape
webcam source).
Data model & API:
- New SourceCrop (normalized edge fractions) + PipTransforms map in
strom-types, threaded through UpdatePipConfigRequest/Response and
PipState with #[serde(default)] for backward compatibility
- VisionMixerState + page config expose negotiated per-input resolutions
(inputs are NOT normalized to the PGM resolution/aspect)
- Backend prunes transforms for inputs that leave the PiP
Rendering (GPU backend):
- Crop maps to glvideomixer pad crop-* properties (source pixels from
negotiated caps); the CPU compositor has no crop pads - warn + ignore
- sizing-policy follows the crop: upstream keep-aspect-ratio fits by the
UNCROPPED input DAR (verified in gstglvideomixer.c), which letterboxes
and distorts cropped content, so cropped pads use "none" (the UI locks
the crop aspect to the box aspect, making the fill geometrically exact)
- resolve_zone_pads: a single cropped source fills its zone rect exactly
instead of auto-tiling into an aspect-preserving cell
- Crop animates with the existing zone morph and with takes (shared pads
ease punch-in/out, fresh pads snap, departing pads keep their crop)
Control UI (vision-mixer.html, rev 6):
- Crop/Zoom editor beside the zone canvas: draggable/resizable crop
frame on a source-aspect canvas, zoom slider, aspect lock, reset,
X/Y/W/H inputs in source pixels, shared snap/grid (quarters + thirds)
- Aspect-locked crops re-fit automatically when their zone box reshapes
- Drag-start guards + NaN sanitization: a panel rebuild on the same
mousedown left a detached 0-size canvas whose division poisoned zone
rects with NaN, serializing to null and freezing all PiP updates (422)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(vision-mixer): explicit pad geometry, drop sizing-policy reliance
Toggling between a cropped PiP source and the same raw input snapped
visibly for non-16:9 sources (e.g. 2.39:1 scope): sizing-policy is an
enum that cannot animate, so the punch-out stretched the frame into the
16:9 rect under "none" and then jumped to the letterboxed size when the
deferred flip to keep-aspect-ratio landed.
Own the geometry instead: input video pads run sizing-policy=none
permanently and the layout code aspect-fits every rect itself.
- strom-types: aspect_fit_rect + effective_source_aspect helpers;
resolve_zone_pads takes per-source aspects and fits each source inside
its cell (single-source zones use the full zone rect as the cell), so
a locked crop fills its box, an unlocked crop letterboxes correctly,
and odd-aspect uncropped sources letterbox inside their cells — this
also retires the inputs-share-PGM-aspect assumption for zone cells
- backend: all layout paths (fullscreen PGM, PVW big, classic-take
reset, PiP regions, bg) fit with the negotiated per-source aspect;
morph endpoints are exact so transitions no longer snap; the
sizing-policy flip and its deferred-restore timeout are removed
(DSK pads keep keep-aspect-ratio — they never crop)
- reactive geometry: explicit rects are static at apply time, so a CAPS
event probe on every input video sink pad (WeakRef captures) re-applies
aspect-correct geometry + crop pixels when caps arrive at preroll or
change mid-stream; FTB-driven alphas are left untouched during refresh
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(vision-mixer): consolidate UI version markers into one
The page had two version indicators: the original bottom-bar string
(v2.17-bus-cleanup) and a console.log marker added during crop-editor
iteration. Keep only the bottom-bar string, bumped to v2.18-source-crop.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(vision-mixer): restructure PiP editor into two labeled columns
Replace the tall middle toolbar + per-column legend paragraphs with a
cleaner two-column layout now that the editor hosts both zones and crop:
- Zones column: header row (title, BG selector, +Zone, shared Snap/Grid
toggles) above the canvas; one compact control row beneath with the
active zone's X/Y/W/H pixel rect and capacity
- Crop/Zoom column: same structure — header with the source selector,
source canvas, zoom slider, single-row X/Y/W/H, lock + reset
- One dim one-line legend spans the editor bottom instead of the two
hint paragraphs; detailed guidance lives in control tooltips
- Drop the now-unused CSS (canvas-wrap, toolbar, hint, px-row,
toggle-row, crop-head); narrow px inputs so control rows stay single-line
Bump UI version to v2.19-editor-layout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(vision-mixer): dark-theme native number-input spinners
Chrome painted the number-input spin buttons on a white box inside the
purple inputs. Set color-scheme: dark on the page so native widget
chrome (spinners, scrollbars) renders in dark theme. Bump to v2.20.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(vision-mixer): defer caps-probe geometry refresh to the main loop
Refreshing inline from the CAPS event probe raced caps negotiation:
sticky caps are stored on the pad only after the probes return, and
sibling branches negotiate concurrently on their own streaming threads.
When two probes of the same input fired simultaneously, each missed the
other pad's caps, the input's aspect never entered the refresh, and its
multiview thumbnail stayed stretched at the fallback aspect forever
(timing-dependent per source — PGM/PVW/PiP self-healed on the next user
action, thumbnails had no later trigger).
Defer each refresh via glib::idle_add_once: serialized on the main
context, the triggering caps are stored by the time the refresh reads
them, and the last refresh always sees every negotiated pad.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(vision-mixer): animated crop on the CPU compositor backend
The CPU `compositor` element has no crop pad properties, so crop was
GL-only (warn + ignore). Full parity, including animation:
- A videocrop element sits upstream of every croppable compositor sink
pad (dist, PVW, PiP branches — not thumbnails) in the CPU pipeline
- Shared crop module (gst/crop.rs): set_pad_crop targets GL pad crop-*
or the upstream videocrop (found via pad.peer()); source_dims_for_pad
always returns UNCROPPED dimensions (videocrop's sink-side caps on
CPU — the compositor pad's own caps are post-crop there, and using
them would apply the crop window twice in the aspect math)
- Crop ANIMATES on CPU too: videocrop's left/right/top/bottom are
controllable and it calls gst_object_sync_values per buffer (verified
upstream in gst_video_crop_before_transform), so the same
InterpolationControlSource morphs drive both backends;
setup_int_animation now binds to any controllable GstObject
- Each animated videocrop step renegotiates caps toward the compositor
(it does not advertise GstVideoCropMeta) — a short burst during the
morph window. The geometry caps probe now tracks last-known uncropped
source dims in the overlay state and skips refreshes where nothing
changed, so the burst cannot clear control bindings mid-animation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(vision-mixer): retain per-source crops when a source leaves the PiP
Swap-zone workflows (capacity 1, pushing back and forth between two
sources) expect each source's punch-in framing to survive the round
trip. Stop pruning transforms for inputs outside the PiP: retained
entries are inert (layout only applies crop to sources that render) and
re-apply when the source returns — with the UI's aspect lock re-fitting
the window if it lands in a differently-shaped box. Removing a crop is
now an explicit act (Reset in the UI / deleting the entry while the
source is present), not a side effect of leaving the composition.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(vision-mixer): operator guide for crop/zoom (punch-in)
New §4.4 covering the crop editor (frame, zoom slider, source-pixel
fields, aspect lock, snap/grid), the behavior contract (animated, crops
retained when a source leaves the PiP, Reset as the explicit clear,
per-PiP per-source, single-source zones fill their rect), plus updated
editor walkthrough, controls reference and glossary. Auto-tile row now
reflects per-source aspect fitting.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(vision-mixer): fix three stale comments found in review
- crop.rs claimed CPU videocrop values are rounded to even — that
rounding was dropped when crop animation landed (videocrop handles
arbitrary offsets; animations interpolate odd values per buffer)
- vision-mixer.html said the backend prunes transforms — it retains
them since the swap-zone change; clarify that only present sources
are listed and a retained crop is edited by bringing the source back
- apply_pip_layout_to_region said hidden pads reset to no crop — they
apply their retained transform; non-PiP reveal paths do the wiping
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent 5e156be commit ca1bda5
23 files changed
Lines changed: 2617 additions & 311 deletions
File tree
- backend
- src
- api
- blocks/builtin/vision_mixer
- builder
- gst
- pipeline
- transitions
- static
- docs
- types
- src
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1747 | 1747 | | |
1748 | 1748 | | |
1749 | 1749 | | |
1750 | | - | |
1751 | | - | |
| 1750 | + | |
| 1751 | + | |
1752 | 1752 | | |
1753 | 1753 | | |
1754 | | - | |
| 1754 | + | |
| 1755 | + | |
| 1756 | + | |
| 1757 | + | |
| 1758 | + | |
| 1759 | + | |
| 1760 | + | |
| 1761 | + | |
1755 | 1762 | | |
1756 | 1763 | | |
1757 | 1764 | | |
| |||
1766 | 1773 | | |
1767 | 1774 | | |
1768 | 1775 | | |
1769 | | - | |
1770 | | - | |
| 1776 | + | |
| 1777 | + | |
| 1778 | + | |
1771 | 1779 | | |
1772 | 1780 | | |
1773 | | - | |
| 1781 | + | |
| 1782 | + | |
| 1783 | + | |
| 1784 | + | |
| 1785 | + | |
1774 | 1786 | | |
1775 | | - | |
| 1787 | + | |
1776 | 1788 | | |
1777 | 1789 | | |
1778 | 1790 | | |
1779 | 1791 | | |
1780 | 1792 | | |
1781 | 1793 | | |
1782 | 1794 | | |
| 1795 | + | |
1783 | 1796 | | |
1784 | 1797 | | |
1785 | 1798 | | |
| |||
1803 | 1816 | | |
1804 | 1817 | | |
1805 | 1818 | | |
1806 | | - | |
| 1819 | + | |
1807 | 1820 | | |
1808 | 1821 | | |
1809 | 1822 | | |
| |||
1824 | 1837 | | |
1825 | 1838 | | |
1826 | 1839 | | |
| 1840 | + | |
1827 | 1841 | | |
1828 | 1842 | | |
1829 | 1843 | | |
| |||
1833 | 1847 | | |
1834 | 1848 | | |
1835 | 1849 | | |
| 1850 | + | |
| 1851 | + | |
| 1852 | + | |
| 1853 | + | |
1836 | 1854 | | |
1837 | 1855 | | |
1838 | 1856 | | |
| |||
1844 | 1862 | | |
1845 | 1863 | | |
1846 | 1864 | | |
| 1865 | + | |
1847 | 1866 | | |
1848 | 1867 | | |
1849 | 1868 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
153 | 153 | | |
154 | 154 | | |
155 | 155 | | |
156 | | - | |
157 | | - | |
| 156 | + | |
| 157 | + | |
158 | 158 | | |
159 | 159 | | |
160 | 160 | | |
161 | 161 | | |
| 162 | + | |
162 | 163 | | |
163 | 164 | | |
164 | 165 | | |
165 | 166 | | |
166 | 167 | | |
| 168 | + | |
167 | 169 | | |
168 | 170 | | |
169 | 171 | | |
170 | 172 | | |
171 | 173 | | |
172 | 174 | | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
173 | 182 | | |
174 | 183 | | |
175 | 184 | | |
| |||
188 | 197 | | |
189 | 198 | | |
190 | 199 | | |
| 200 | + | |
191 | 201 | | |
192 | 202 | | |
193 | 203 | | |
| |||
Lines changed: 30 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
34 | | - | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
35 | 47 | | |
36 | 48 | | |
37 | 49 | | |
| |||
68 | 80 | | |
69 | 81 | | |
70 | 82 | | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
75 | 87 | | |
76 | 88 | | |
77 | 89 | | |
| |||
112 | 124 | | |
113 | 125 | | |
114 | 126 | | |
| 127 | + | |
| 128 | + | |
115 | 129 | | |
116 | 130 | | |
117 | | - | |
| 131 | + | |
118 | 132 | | |
119 | 133 | | |
120 | 134 | | |
| |||
152 | 166 | | |
153 | 167 | | |
154 | 168 | | |
| 169 | + | |
| 170 | + | |
155 | 171 | | |
156 | 172 | | |
157 | | - | |
| 173 | + | |
158 | 174 | | |
159 | 175 | | |
160 | 176 | | |
| |||
172 | 188 | | |
173 | 189 | | |
174 | 190 | | |
| 191 | + | |
| 192 | + | |
175 | 193 | | |
176 | 194 | | |
177 | | - | |
| 195 | + | |
178 | 196 | | |
179 | 197 | | |
180 | 198 | | |
| |||
220 | 238 | | |
221 | 239 | | |
222 | 240 | | |
| 241 | + | |
| 242 | + | |
223 | 243 | | |
224 | 244 | | |
225 | | - | |
| 245 | + | |
226 | 246 | | |
227 | 247 | | |
228 | 248 | | |
| |||
256 | 276 | | |
257 | 277 | | |
258 | 278 | | |
259 | | - | |
| 279 | + | |
260 | 280 | | |
261 | 281 | | |
262 | 282 | | |
| |||
0 commit comments