Skip to content

Commit ca1bda5

Browse files
srperensclaude
andauthored
feat(vision-mixer): per-source crop/zoom (punch-in) for PiP compositions (#622)
* feat(vision-mixer): per-source crop/zoom (punch-in) for PiP compositions Add vMix-style source cropping to the vision mixer: each input can carry a normalized crop window per PiP, scaling the cropped region to fill its zone box (e.g. a portrait box framing a person's head from a landscape webcam source). Data model & API: - New SourceCrop (normalized edge fractions) + PipTransforms map in strom-types, threaded through UpdatePipConfigRequest/Response and PipState with #[serde(default)] for backward compatibility - VisionMixerState + page config expose negotiated per-input resolutions (inputs are NOT normalized to the PGM resolution/aspect) - Backend prunes transforms for inputs that leave the PiP Rendering (GPU backend): - Crop maps to glvideomixer pad crop-* properties (source pixels from negotiated caps); the CPU compositor has no crop pads - warn + ignore - sizing-policy follows the crop: upstream keep-aspect-ratio fits by the UNCROPPED input DAR (verified in gstglvideomixer.c), which letterboxes and distorts cropped content, so cropped pads use "none" (the UI locks the crop aspect to the box aspect, making the fill geometrically exact) - resolve_zone_pads: a single cropped source fills its zone rect exactly instead of auto-tiling into an aspect-preserving cell - Crop animates with the existing zone morph and with takes (shared pads ease punch-in/out, fresh pads snap, departing pads keep their crop) Control UI (vision-mixer.html, rev 6): - Crop/Zoom editor beside the zone canvas: draggable/resizable crop frame on a source-aspect canvas, zoom slider, aspect lock, reset, X/Y/W/H inputs in source pixels, shared snap/grid (quarters + thirds) - Aspect-locked crops re-fit automatically when their zone box reshapes - Drag-start guards + NaN sanitization: a panel rebuild on the same mousedown left a detached 0-size canvas whose division poisoned zone rects with NaN, serializing to null and freezing all PiP updates (422) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(vision-mixer): explicit pad geometry, drop sizing-policy reliance Toggling between a cropped PiP source and the same raw input snapped visibly for non-16:9 sources (e.g. 2.39:1 scope): sizing-policy is an enum that cannot animate, so the punch-out stretched the frame into the 16:9 rect under "none" and then jumped to the letterboxed size when the deferred flip to keep-aspect-ratio landed. Own the geometry instead: input video pads run sizing-policy=none permanently and the layout code aspect-fits every rect itself. - strom-types: aspect_fit_rect + effective_source_aspect helpers; resolve_zone_pads takes per-source aspects and fits each source inside its cell (single-source zones use the full zone rect as the cell), so a locked crop fills its box, an unlocked crop letterboxes correctly, and odd-aspect uncropped sources letterbox inside their cells — this also retires the inputs-share-PGM-aspect assumption for zone cells - backend: all layout paths (fullscreen PGM, PVW big, classic-take reset, PiP regions, bg) fit with the negotiated per-source aspect; morph endpoints are exact so transitions no longer snap; the sizing-policy flip and its deferred-restore timeout are removed (DSK pads keep keep-aspect-ratio — they never crop) - reactive geometry: explicit rects are static at apply time, so a CAPS event probe on every input video sink pad (WeakRef captures) re-applies aspect-correct geometry + crop pixels when caps arrive at preroll or change mid-stream; FTB-driven alphas are left untouched during refresh Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(vision-mixer): consolidate UI version markers into one The page had two version indicators: the original bottom-bar string (v2.17-bus-cleanup) and a console.log marker added during crop-editor iteration. Keep only the bottom-bar string, bumped to v2.18-source-crop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(vision-mixer): restructure PiP editor into two labeled columns Replace the tall middle toolbar + per-column legend paragraphs with a cleaner two-column layout now that the editor hosts both zones and crop: - Zones column: header row (title, BG selector, +Zone, shared Snap/Grid toggles) above the canvas; one compact control row beneath with the active zone's X/Y/W/H pixel rect and capacity - Crop/Zoom column: same structure — header with the source selector, source canvas, zoom slider, single-row X/Y/W/H, lock + reset - One dim one-line legend spans the editor bottom instead of the two hint paragraphs; detailed guidance lives in control tooltips - Drop the now-unused CSS (canvas-wrap, toolbar, hint, px-row, toggle-row, crop-head); narrow px inputs so control rows stay single-line Bump UI version to v2.19-editor-layout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(vision-mixer): dark-theme native number-input spinners Chrome painted the number-input spin buttons on a white box inside the purple inputs. Set color-scheme: dark on the page so native widget chrome (spinners, scrollbars) renders in dark theme. Bump to v2.20. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(vision-mixer): defer caps-probe geometry refresh to the main loop Refreshing inline from the CAPS event probe raced caps negotiation: sticky caps are stored on the pad only after the probes return, and sibling branches negotiate concurrently on their own streaming threads. When two probes of the same input fired simultaneously, each missed the other pad's caps, the input's aspect never entered the refresh, and its multiview thumbnail stayed stretched at the fallback aspect forever (timing-dependent per source — PGM/PVW/PiP self-healed on the next user action, thumbnails had no later trigger). Defer each refresh via glib::idle_add_once: serialized on the main context, the triggering caps are stored by the time the refresh reads them, and the last refresh always sees every negotiated pad. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(vision-mixer): animated crop on the CPU compositor backend The CPU `compositor` element has no crop pad properties, so crop was GL-only (warn + ignore). Full parity, including animation: - A videocrop element sits upstream of every croppable compositor sink pad (dist, PVW, PiP branches — not thumbnails) in the CPU pipeline - Shared crop module (gst/crop.rs): set_pad_crop targets GL pad crop-* or the upstream videocrop (found via pad.peer()); source_dims_for_pad always returns UNCROPPED dimensions (videocrop's sink-side caps on CPU — the compositor pad's own caps are post-crop there, and using them would apply the crop window twice in the aspect math) - Crop ANIMATES on CPU too: videocrop's left/right/top/bottom are controllable and it calls gst_object_sync_values per buffer (verified upstream in gst_video_crop_before_transform), so the same InterpolationControlSource morphs drive both backends; setup_int_animation now binds to any controllable GstObject - Each animated videocrop step renegotiates caps toward the compositor (it does not advertise GstVideoCropMeta) — a short burst during the morph window. The geometry caps probe now tracks last-known uncropped source dims in the overlay state and skips refreshes where nothing changed, so the burst cannot clear control bindings mid-animation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(vision-mixer): retain per-source crops when a source leaves the PiP Swap-zone workflows (capacity 1, pushing back and forth between two sources) expect each source's punch-in framing to survive the round trip. Stop pruning transforms for inputs outside the PiP: retained entries are inert (layout only applies crop to sources that render) and re-apply when the source returns — with the UI's aspect lock re-fitting the window if it lands in a differently-shaped box. Removing a crop is now an explicit act (Reset in the UI / deleting the entry while the source is present), not a side effect of leaving the composition. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(vision-mixer): operator guide for crop/zoom (punch-in) New §4.4 covering the crop editor (frame, zoom slider, source-pixel fields, aspect lock, snap/grid), the behavior contract (animated, crops retained when a source leaves the PiP, Reset as the explicit clear, per-PiP per-source, single-source zones fill their rect), plus updated editor walkthrough, controls reference and glossary. Auto-tile row now reflects per-source aspect fitting. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(vision-mixer): fix three stale comments found in review - crop.rs claimed CPU videocrop values are rounded to even — that rounding was dropped when crop animation landed (videocrop handles arbitrary offsets; animations interpolate odd values per buffer) - vision-mixer.html said the backend prunes transforms — it retains them since the swap-zone change; clarify that only present sources are listed and a retained crop is edited by bringing the source back - apply_pip_layout_to_region said hidden pads reset to no crop — they apply their retained transform; non-PiP reveal paths do the wiping Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 5e156be commit ca1bda5

23 files changed

Lines changed: 2617 additions & 311 deletions

Cargo.lock

Lines changed: 5 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ members = ["types", "backend", "frontend", "mcp-server"]
44
default-members = ["backend"]
55

66
[workspace.package]
7-
version = "0.6.1"
7+
version = "0.6.2-dev"
88
edition = "2021"
99
license = "MIT OR Apache-2.0"
1010
authors = ["Per Enstedt <per.enstedt@eyevinn.se>"]

backend/src/api/flows.rs

Lines changed: 27 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1747,11 +1747,18 @@ pub async fn update_pip_config(
17471747
}
17481748

17491749
info!(
1750-
"Updating PiP {} on vision mixer {} in flow {}: bg={:?}, zones={:?}",
1751-
pip_idx, block_id, flow_id, req.bg, req.zones
1750+
"Updating PiP {} on vision mixer {} in flow {}: bg={:?}, zones={:?}, transforms={:?}",
1751+
pip_idx, block_id, flow_id, req.bg, req.zones, req.transforms
17521752
);
17531753
state
1754-
.apply_vision_mixer_pip_config(&flow_id, &block_id, pip_idx, req.bg, req.zones.clone())
1754+
.apply_vision_mixer_pip_config(
1755+
&flow_id,
1756+
&block_id,
1757+
pip_idx,
1758+
req.bg,
1759+
req.zones.clone(),
1760+
req.transforms.clone(),
1761+
)
17551762
.await
17561763
.map_err(|e| {
17571764
error!("Failed to update PiP config: {}", e);
@@ -1766,20 +1773,26 @@ pub async fn update_pip_config(
17661773

17671774
// Read back authoritative state. Validation runs in
17681775
// `apply_vision_mixer_pip_config`, so the only mutation vs. the request
1769-
// is rect clamping (NormRect → [0,1]).
1770-
let (bg, zones) = if let Some(s) =
1776+
// is rect/crop clamping (NormRect → [0,1], SourceCrop clamped + zero
1777+
// entries dropped).
1778+
let (bg, zones, transforms) = if let Some(s) =
17711779
crate::blocks::builtin::vision_mixer::overlay::get_overlay_state(&block_id)
17721780
{
1773-
(s.pip_bg_input(pip_idx), s.pip_zones(pip_idx))
1781+
(
1782+
s.pip_bg_input(pip_idx),
1783+
s.pip_zones(pip_idx),
1784+
s.pip_transforms(pip_idx),
1785+
)
17741786
} else {
1775-
(req.bg, req.zones)
1787+
(req.bg, req.zones, req.transforms)
17761788
};
17771789

17781790
Ok(Json(strom_types::api::UpdatePipConfigResponse {
17791791
message: format!("PiP {} updated", pip_idx),
17801792
pip_idx,
17811793
bg,
17821794
zones,
1795+
transforms,
17831796
}))
17841797
}
17851798

@@ -1803,7 +1816,7 @@ pub async fn update_pip_config(
18031816
)
18041817
)]
18051818
pub async fn get_vision_mixer_state(
1806-
State(_state): State<AppState>,
1819+
State(state): State<AppState>,
18071820
Path((flow_id, block_id)): Path<(FlowId, String)>,
18081821
) -> Result<Json<strom_types::api::VisionMixerState>, (StatusCode, Json<ErrorResponse>)> {
18091822
let overlay = crate::blocks::builtin::vision_mixer::overlay::get_overlay_state(&block_id)
@@ -1824,6 +1837,7 @@ pub async fn get_vision_mixer_state(
18241837
.map(|i| strom_types::api::PipState {
18251838
bg: overlay.pip_bg_input(i),
18261839
zones: overlay.pip_zones(i),
1840+
transforms: overlay.pip_transforms(i),
18271841
})
18281842
.collect();
18291843

@@ -1833,6 +1847,10 @@ pub async fn get_vision_mixer_state(
18331847
.map(|a| a.load(std::sync::atomic::Ordering::Relaxed))
18341848
.collect();
18351849

1850+
let input_resolutions = state
1851+
.vision_mixer_input_resolutions(&flow_id, &block_id, overlay.num_inputs)
1852+
.await;
1853+
18361854
Ok(Json(strom_types::api::VisionMixerState {
18371855
program_input: overlay.pgm_input(),
18381856
preview_input: overlay.pvw_input(),
@@ -1844,6 +1862,7 @@ pub async fn get_vision_mixer_state(
18441862
dsk_enabled,
18451863
overlay_alpha: overlay.overlay_alpha(),
18461864
pips,
1865+
input_resolutions,
18471866
}))
18481867
}
18491868

backend/src/api/vision_mixer_page.rs

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -153,23 +153,32 @@ async fn render_vision_mixer_page(
153153
// therefore empty until the operator configures them.
154154
let pips: Vec<serde_json::Value> = (0..num_pips)
155155
.map(|i| {
156-
let (bg, zones) = if let Some(s) = overlay.as_ref() {
157-
(s.pip_bg_input(i), s.pip_zones(i))
156+
let (bg, zones, transforms) = if let Some(s) = overlay.as_ref() {
157+
(s.pip_bg_input(i), s.pip_zones(i), s.pip_transforms(i))
158158
} else {
159159
(
160160
vm_props::parse_pip_bg(&vm_block.properties, i, num_inputs),
161161
Vec::<strom_types::vision_mixer::Zone>::new(),
162+
strom_types::vision_mixer::PipTransforms::new(),
162163
)
163164
};
164165
serde_json::json!({
165166
"bg": bg,
166167
"zones": zones,
168+
"transforms": transforms,
167169
})
168170
})
169171
.collect();
170172
let pvw_pip: Option<usize> = overlay.as_ref().and_then(|s| s.pvw_pip());
171173
let pgm_pip: Option<usize> = overlay.as_ref().and_then(|s| s.pgm_pip());
172174

175+
// Negotiated per-input resolutions (None until caps arrive). The crop
176+
// editor needs the real source aspect — inputs are NOT normalized to the
177+
// PGM resolution/aspect.
178+
let input_resolutions = state
179+
.vision_mixer_input_resolutions(flow_id, block_id, num_inputs)
180+
.await;
181+
173182
// Build a single JSON config object (safe injection via <script type="application/json">)
174183
let config = serde_json::json!({
175184
"flow_id": flow_id.to_string(),
@@ -188,6 +197,7 @@ async fn render_vision_mixer_page(
188197
"pgm_pip": pgm_pip,
189198
"pgm_w": pgm_w,
190199
"pgm_h": pgm_h,
200+
"input_resolutions": input_resolutions,
191201
});
192202

193203
let html = VISION_MIXER_HTML.replace("{{VM_CONFIG_JSON}}", &config.to_string());

backend/src/blocks/builtin/vision_mixer/builder/pad_layout.rs

Lines changed: 30 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,19 @@ fn initial_pad_geom_for_input(
3131
return (rx, ry, rw, rh, 1.0, bg_zorder as u64);
3232
}
3333
let zones = p.pip_zones.get(pip_idx).map(Vec::as_slice).unwrap_or(&[]);
34-
let layouts = strom_types::vision_mixer::resolve_zone_pads(rx, ry, rw, rh, zones, src_aspect);
34+
// Transforms are runtime-only (like zones) and caps are not negotiated
35+
// yet — both maps are empty at build time. The caps probe re-applies
36+
// aspect-correct geometry once each input's caps arrive.
37+
let layouts = strom_types::vision_mixer::resolve_zone_pads(
38+
rx,
39+
ry,
40+
rw,
41+
rh,
42+
zones,
43+
src_aspect,
44+
&strom_types::vision_mixer::PipTransforms::new(),
45+
&strom_types::vision_mixer::SourceAspects::new(),
46+
);
3547
if let Some(l) = layouts.iter().find(|l| l.input == input) {
3648
(
3749
l.x,
@@ -68,10 +80,10 @@ pub(super) fn build_pad_properties(
6880
use strom_types::vision_mixer::Source;
6981
let canvas_w = p.pgm_w as i32;
7082
let canvas_h = p.pgm_h as i32;
71-
// Source aspect for PiP-tile cell math (assumes inputs share the PGM aspect,
72-
// which is typical for broadcast workflows). resolve_zone_pads sizes each
73-
// tile to this aspect so `keep-aspect-ratio` pads fill cleanly without
74-
// transparent letterbox bands letting the bg peek through.
83+
// Fallback aspect for PiP-tile cell math at build time: caps are not
84+
// negotiated yet, so the slot grid and fits use the PGM canvas aspect.
85+
// The caps probe re-applies per-source aspect-correct geometry once each
86+
// input's caps arrive.
7587
let pgm_aspect = if canvas_h > 0 {
7688
canvas_w as f64 / canvas_h as f64
7789
} else {
@@ -112,9 +124,11 @@ pub(super) fn build_pad_properties(
112124
props.insert("width".to_string(), PropertyValue::Int(w as i64));
113125
props.insert("height".to_string(), PropertyValue::Int(h as i64));
114126
props.insert("zorder".to_string(), PropertyValue::UInt(zorder));
127+
// Explicit geometry: layout code aspect-fits every rect itself,
128+
// so the pad must render exactly (width, height). See aspect_fit_rect.
115129
props.insert(
116130
"sizing-policy".to_string(),
117-
PropertyValue::String("keep-aspect-ratio".to_string()),
131+
PropertyValue::String("none".to_string()),
118132
);
119133
}
120134

@@ -152,9 +166,11 @@ pub(super) fn build_pad_properties(
152166
"zorder".to_string(),
153167
PropertyValue::UInt(vision_mixer::MV_THUMBNAIL_ZORDER as u64),
154168
);
169+
// Explicit geometry: layout code aspect-fits every rect itself,
170+
// so the pad must render exactly (width, height). See aspect_fit_rect.
155171
props.insert(
156172
"sizing-policy".to_string(),
157-
PropertyValue::String("keep-aspect-ratio".to_string()),
173+
PropertyValue::String("none".to_string()),
158174
);
159175
}
160176

@@ -172,9 +188,11 @@ pub(super) fn build_pad_properties(
172188
"zorder".to_string(),
173189
PropertyValue::UInt(vision_mixer::MV_BIG_DISPLAY_ZORDER as u64),
174190
);
191+
// Explicit geometry: layout code aspect-fits every rect itself,
192+
// so the pad must render exactly (width, height). See aspect_fit_rect.
175193
props.insert(
176194
"sizing-policy".to_string(),
177-
PropertyValue::String("keep-aspect-ratio".to_string()),
195+
PropertyValue::String("none".to_string()),
178196
);
179197
}
180198

@@ -220,9 +238,11 @@ pub(super) fn build_pad_properties(
220238
props.insert("height".to_string(), PropertyValue::Int(h as i64));
221239
props.insert("alpha".to_string(), PropertyValue::Float(alpha));
222240
props.insert("zorder".to_string(), PropertyValue::UInt(zorder));
241+
// Explicit geometry: layout code aspect-fits every rect itself,
242+
// so the pad must render exactly (width, height). See aspect_fit_rect.
223243
props.insert(
224244
"sizing-policy".to_string(),
225-
PropertyValue::String("keep-aspect-ratio".to_string()),
245+
PropertyValue::String("none".to_string()),
226246
);
227247
}
228248

@@ -256,7 +276,7 @@ pub(super) fn build_pad_properties(
256276
props.insert("zorder".to_string(), PropertyValue::UInt(zorder));
257277
props.insert(
258278
"sizing-policy".to_string(),
259-
PropertyValue::String("keep-aspect-ratio".to_string()),
279+
PropertyValue::String("none".to_string()),
260280
);
261281
}
262282
}

0 commit comments

Comments
 (0)