fix(videoenc): cap VBR excursions and bound frame sizes via VBV - #631
Merged
Conversation
The encoder block set only target bitrate, leaving peak behavior unconstrained. On NVENC in VBR mode (the block default), max-bitrate is unset and vbv-buffer-size=0, so single frames can spike to ~10x the average frame size. Such frames leave the NIC as line-rate packet bursts (hundreds of packets back-to-back) that overflow shallow buffers on constrained viewer paths — observed as bursty packet loss on WebRTC outputs while average bitrate looked fine. - NVENC: max-bitrate = 1.2x target (VBR ceiling, ignored in CBR), vbv-buffer-size = 0.5 s of target bitrate - x264/x265: vbv-buf-capacity = 500 ms, set explicitly instead of relying on the upstream 600 ms default Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
srperens
added a commit
that referenced
this pull request
Jun 8, 2026
* fix(videoenc): cap VideoToolbox bursts via data-rate-limits The vtenc branch set only target bitrate, so in ABR mode (the block default maps VBR/CQP to abr) peak behavior was unconstrained: data-rate-limits defaults to disabled and single frames can spike far above the target. Same failure mode as the NVENC VBR fix (#631) - line-rate packet bursts overflowing shallow buffers on constrained viewer paths. Set data-rate-limits to 1.2x target averaged over a 0.5 s window (kVTCompressionPropertyKey_DataRateLimits) - the VideoToolbox equivalent of NVENC's max-bitrate + vbv-buffer-size pair. vtenc itself skips the property in CBR mode, matching NVENC's "ignored in CBR" semantics. Verified against VideoToolbox on macOS (GST_DEBUG=vtenc:6): ABR applies the limit (DataRateLimits 300000 bytes / 0.5 s at 4000 kbps target, status noErr), CBR logs "Ignoring data-rate-limits property". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(videoenc): floor force-keyunit interval to break PLI/IDR storms Each WebRTC viewer PLI becomes a force-keyunit event at the encoder, and min-force-key-unit-interval defaults to 0 (honor every request). Under burst-induced loss this spirals: a big frame causes packet loss, the viewer PLIs, the encoder emits an outsized IDR, which causes more loss and more PLIs - observed as output bursting to ~10x the target bitrate during high-motion transitions, far above what the encoder's own rate limits allow for normal frames. Set a 1 s floor (GstVideoEncoder base-class property, applies to all encoder implementations). New viewers and genuine recoveries still get a keyframe quickly; a PLI storm can add at most one extra IDR per second. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(videoenc): map VBR to CBR on VideoToolbox - ABR ignores bitrate Measured on Apple Silicon (GStreamer 1.28, 720p30, 5 Mbit/s target, demanding content): vtenc in ABR mode encodes at ~95 Mbit/s - it does not rate-control at all. Root cause: vtenc unconditionally sets kVTCompressionPropertyKey_Quality (property default 0.5) after AverageBitRate, and Apple Silicon hardware then runs constant-quality and ignores the bitrate target. A direct VTCompressionSession with the same settings minus Quality tracks the target correctly (~6.4 Mbit/s), and adding Quality=0.5 reproduces the runaway (~98 Mbit/s) - upstream GStreamer bug, to be reported. CBR uses a different VT key (ConstantBitRate) that is honored in steady state, so map VBR to CBR: a user asking for VBR means "roughly the target bitrate", never "unbounded constant quality". CQP keeps mapping to ABR, whose constant-quality behavior is exactly what CQP requests. Also correct the data-rate-limits comment: on Apple Silicon hardware the cap is accepted (noErr) but has no measurable effect even in ABR, and scene changes overshoot by ~2-4 MB across ~10 frames in every mode (intrinsic to Apple's rate control, also bypasses DataRateLimits in a direct VT session). Strictly bounded bursts require a software encoder (x264 VBV) via encoder_preference=software. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The video encoder block sets only target bitrate, leaving peak behavior unconstrained. This PR bounds how far the encoder can spike above target:
nv*):max-bitrate= 1.2× target (VBR ceiling, ignored in CBR mode),vbv-buffer-size= 0.5 s worth of target bitrate. Previously both were left at defaults (max-bitrateunset,vbv-buffer-size=0= "NVENC default"), allowing single frames ~10× the average frame size.vbv-buf-capacity= 500 ms, set explicitly instead of relying on the upstream 600 ms default.Both are guarded with
has_propertyso encoder variants lacking the knobs are unaffected.Background
Diagnosed on a live WHEP viewer session (sender-side packet capture + client media stats): a stream averaging ~27 Mbit/s emitted 200–450 KB frames, each leaving the NIC as 160–360 RTP packets back-to-back at line rate (~14 µs spacing, 65–79 Mbit/s per 100 ms window). Those bursts overflow shallow buffers on constrained viewer paths and the tail gets dropped — observed as bursty packet loss (+200–400 packets at a time) while average bitrate, jitter (13–19 ms) and RTT (~26 ms) all looked healthy.
Companion to #629 (RTX repairs residual loss; this PR shrinks the bursts that cause it). The remaining lever is per-flow configuration: lower target bitrate and CBR rate control for outputs facing constrained networks.
🤖 Generated with Claude Code