lance_video: GOP decode index + planned prefetch for video blobs - #298
Draft
AyushExel wants to merge 4 commits into
Draft
lance_video: GOP decode index + planned prefetch for video blobs#298AyushExel wants to merge 4 commits into
AyushExel wants to merge 4 commits into
Conversation
AyushExel
marked this pull request as draft
July 21, 2026 16:00
AyushExel
force-pushed
the
lance-video-gop-index
branch
2 times, most recently
from
July 22, 2026 13:27
e75ac05 to
38ab56f
Compare
Adds three optional columns to the videos table (moov_range, gop_frame_idx, gop_byte_offset; tagged swm:video_index_version=1) and teaches the format to use them end to end. No new tables, no change to existing columns or blob bytes; datasets without the columns fall back to the streaming ranged reader, and old readers ignore the new columns. Writer: parses the MP4 sample tables at write time (bytes already in memory, no extra IO; ~130-570 bytes of index per episode) via the new mp4_index module — a pure-Python MP4 box parser that needs only ranged reads, so backfilling existing datasets costs ~2 MB per episode, not a re-download. Appending to pre-index videos tables keeps their schema. Reader: loads the index up front (KBs), and per DataLoader batch maps every window to its covering GOP byte ranges, prefetches all episodes' ranges concurrently (_SparseBlobIO + thread pool), then decodes from memory. Reads outside the plan fall through to ranged fetches, so correctness never depends on the plan. Decoding a mid-file GOP from planned ranges only (head + moov + GOP, ~3.7 MB of a 173 MB VPT episode) is bit-identical to full-file decode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AyushExel
force-pushed
the
lance-video-gop-index
branch
from
July 22, 2026 13:29
38ab56f to
cb11c53
Compare
A materialized Python list of (episode, start) tuples costs ~100 bytes per window. On corpora with tens of millions of windows (VPT Minecraft: 21M) that is 2.4 GB PER DATALOADER WORKER, replicated at every worker spawn — measured: 5 GB worker RSS (OOM on shared nodes) and two minutes of pickling at every epoch/val boundary. The (N, 2) int64 array is 336 MB, pickles as one buffer copy in <0.5 s, and rows unpack exactly like the old tuples. GoalDataset's future-frame filter becomes a vectorized mask over the same array. Measured on the VPT training pipeline (batch 32, 12 workers, S3): worker RSS 5.0 -> ~2 GB (12-worker run OOM'd a 62 GB box before, now completes), dataset pickle 4.4 GB/minutes -> 338 MB/0.4 s, throughput 0.16 -> 0.32 it/s from the recovered workers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…el decode Three round-trip/latency optimizations for the planned (GOP-index) path, motivated by a latency-bound training box (25 ms to S3, workers at 5% CPU on an idle 200 Gbps link — ~100 serialized ranged reads per batch): - ONE read_blob_ranges call per batch (pylance >= 9) covering every planned range of every episode the batch touches — header ranges for cold episodes plus window GOP ranges, span-merged (64 KiB gap) and gap-filtered against each source's buffered chunks. Runtime-probed: without the API (or without an index) sources fall back to their own ranged fetches on a thread pool, results identical (tested). - Decoder-open prefetch now includes the moov-following box header and the first-packet region: ffmpeg touches both during open, which cost hidden fallback round trips on every cold episode (32 per batch under random-window training). - Per-episode decodes run on a persistent per-worker pool (torchcodec releases the GIL); decoder creation rides in the task so cold moov parses overlap too. Same-region S3 benchmark (6 workers, batch 16, vpt_9x): 1.13 -> 0.35 s/batch (14.1 -> 45.2 windows/s, 3.2x) at unchanged bytes/window; the gain grows with link RTT since the wave replaces ~ranges x RTT of serialized waiting with ~1 x RTT. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
take_blobs is a network round trip per CALL, not per row — but the batch path opened each cold episode's blob with its own single-row call, ~32 serialized requests per batch that the fetch wave did not cover. Found on an HF-bucket training run: the gateway's request quota was exhausted by blob opens alone (44x 429 crash) after the wave had already collapsed the range reads. Cold keys now collect first and open in one multi-row take_blobs, so a batch costs exactly two requests: one blob-open call + one range wave. Returned entries are pinned by the batch's dict, so mid-batch cache eviction (batches touching more episodes than decoder_cache_size) cannot invalidate in-flight work. Same-region S3 benchmark (6 workers, batch 16, vpt_9x): 0.35 -> 0.25 s/batch (45 -> 65 windows/s); cumulative over the pre-wave reader: 1.13 -> 0.25 s/batch (4.5x). On request-quota'd or high-RTT stores the change is qualitative (429 crash -> ~7x under quota). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased on main now that #297 is merged — single commit, builds on #297's ranged streaming reader (which remains the fallback path).
What
Adds three optional columns to the videos table —
moov_range,gop_frame_idx,gop_byte_offset(taggedswm:video_index_version=1) — and teaches the existinglance_videoformat to use them end to end. Not a new format: no new tables, no changes to existing columns or blob bytes. Datasets without the columns fall back to #297's streaming path; old readers ignore the new columns.Why
#297 stops downloading whole episode MP4s, but the decoder still discovers byte ranges by seeking, one round-trip at a time. With a GOP index the reader knows every byte range a DataLoader batch needs before decoding: it prefetches all episodes' ranges concurrently and decodes from memory. On high-latency links (cross-region object storage) this collapses the per-window round-trip chain into one parallel fetch wave. It is also the prerequisite for
Dataset.read_blob_ranges(lance#7864) — which needs the ranges as arguments — where the thread-pool fetch can later be swapped for lance-side coalesced IO.How
mp4_index.py(new): pure-Python MP4 box parser → GOP-level index (stss/stsz/stsc/stco/co64). Needs only ranged reads: building an index from object storage costs ~2 MB per episode, not a re-download. Index weight is ~130–570 bytes per episode (measured on VPT Minecraft: 128-frame GOPs).__getitems__maps windows → covering GOP byte ranges, prefetches per-episode ranges concurrently (_SparseBlobIO), then decodes. Unplanned reads fall through to ranged fetches, so correctness never depends on plan completeness.Validation
ffprobe -count_packetsexactly.🤖 Generated with Claude Code
Update: a second commit (
3945b44) rides along —clip_indicesas an int64 array instead of a Python list of tuples. On 21M-window corpora the list costs 2.4 GB and ~minutes of pickling per DataLoader worker spawn; measured on the VPT training pipeline this was the difference between 12 workers OOM-killing a 62 GB box and completing at ~2 GB/worker with 2× throughput. Happy to split it into its own PR if preferred — it is logically independent of the GOP index.