Skip to content

lance_video: GOP decode index + planned prefetch for video blobs - #298

Draft
AyushExel wants to merge 4 commits into
galilai-group:mainfrom
AyushExel:lance-video-gop-index
Draft

lance_video: GOP decode index + planned prefetch for video blobs#298
AyushExel wants to merge 4 commits into
galilai-group:mainfrom
AyushExel:lance-video-gop-index

Conversation

@AyushExel

@AyushExel AyushExel commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Rebased on main now that #297 is merged — single commit, builds on #297's ranged streaming reader (which remains the fallback path).

What

Adds three optional columns to the videos table — moov_range, gop_frame_idx, gop_byte_offset (tagged swm:video_index_version=1) — and teaches the existing lance_video format to use them end to end. Not a new format: no new tables, no changes to existing columns or blob bytes. Datasets without the columns fall back to #297's streaming path; old readers ignore the new columns.

Why

#297 stops downloading whole episode MP4s, but the decoder still discovers byte ranges by seeking, one round-trip at a time. With a GOP index the reader knows every byte range a DataLoader batch needs before decoding: it prefetches all episodes' ranges concurrently and decodes from memory. On high-latency links (cross-region object storage) this collapses the per-window round-trip chain into one parallel fetch wave. It is also the prerequisite for Dataset.read_blob_ranges (lance#7864) — which needs the ranges as arguments — where the thread-pool fetch can later be swapped for lance-side coalesced IO.

How

  • mp4_index.py (new): pure-Python MP4 box parser → GOP-level index (stss/stsz/stsc/stco/co64). Needs only ranged reads: building an index from object storage costs ~2 MB per episode, not a re-download. Index weight is ~130–570 bytes per episode (measured on VPT Minecraft: 128-frame GOPs).
  • Writer: parses the index at write time — the MP4 bytes are already in memory, zero extra IO. Appending to a pre-index videos table keeps that table's schema (no mismatch).
  • Reader: loads index columns up front (KBs for tens of thousands of episodes); __getitems__ maps windows → covering GOP byte ranges, prefetches per-episode ranges concurrently (_SparseBlobIO), then decodes. Unplanned reads fall through to ranged fetches, so correctness never depends on plan completeness.

Validation

  • Sparse decode from planned ranges only (64 KB head + moov + one GOP ≈ 3.7 MB of a 173 MB VPT episode) is bit-identical to full-file decode, on both 6.x and 10.x recorder generations.
  • Index frame counts match ffprobe -count_packets exactly.
  • 17/17 tests: all existing decode/parity tests now run through the planned path, plus new tests for writer emission + reader planning parity, partial-blob fetch, no-index fallback, and append-to-legacy-table.

🤖 Generated with Claude Code


Update: a second commit (3945b44) rides along — clip_indices as an int64 array instead of a Python list of tuples. On 21M-window corpora the list costs 2.4 GB and ~minutes of pickling per DataLoader worker spawn; measured on the VPT training pipeline this was the difference between 12 workers OOM-killing a 62 GB box and completing at ~2 GB/worker with 2× throughput. Happy to split it into its own PR if preferred — it is logically independent of the GOP index.

@AyushExel
AyushExel marked this pull request as draft July 21, 2026 16:00
@AyushExel
AyushExel force-pushed the lance-video-gop-index branch 2 times, most recently from e75ac05 to 38ab56f Compare July 22, 2026 13:27
Adds three optional columns to the videos table (moov_range,
gop_frame_idx, gop_byte_offset; tagged swm:video_index_version=1) and
teaches the format to use them end to end. No new tables, no change to
existing columns or blob bytes; datasets without the columns fall back
to the streaming ranged reader, and old readers ignore the new columns.

Writer: parses the MP4 sample tables at write time (bytes already in
memory, no extra IO; ~130-570 bytes of index per episode) via the new
mp4_index module — a pure-Python MP4 box parser that needs only ranged
reads, so backfilling existing datasets costs ~2 MB per episode, not a
re-download. Appending to pre-index videos tables keeps their schema.

Reader: loads the index up front (KBs), and per DataLoader batch maps
every window to its covering GOP byte ranges, prefetches all episodes'
ranges concurrently (_SparseBlobIO + thread pool), then decodes from
memory. Reads outside the plan fall through to ranged fetches, so
correctness never depends on the plan. Decoding a mid-file GOP from
planned ranges only (head + moov + GOP, ~3.7 MB of a 173 MB VPT
episode) is bit-identical to full-file decode.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@AyushExel
AyushExel force-pushed the lance-video-gop-index branch from 38ab56f to cb11c53 Compare July 22, 2026 13:29
AyushExel and others added 3 commits July 24, 2026 20:21
A materialized Python list of (episode, start) tuples costs ~100 bytes
per window. On corpora with tens of millions of windows (VPT Minecraft:
21M) that is 2.4 GB PER DATALOADER WORKER, replicated at every worker
spawn — measured: 5 GB worker RSS (OOM on shared nodes) and two minutes
of pickling at every epoch/val boundary. The (N, 2) int64 array is
336 MB, pickles as one buffer copy in <0.5 s, and rows unpack exactly
like the old tuples. GoalDataset's future-frame filter becomes a
vectorized mask over the same array.

Measured on the VPT training pipeline (batch 32, 12 workers, S3):
worker RSS 5.0 -> ~2 GB (12-worker run OOM'd a 62 GB box before, now
completes), dataset pickle 4.4 GB/minutes -> 338 MB/0.4 s, throughput
0.16 -> 0.32 it/s from the recovered workers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…el decode

Three round-trip/latency optimizations for the planned (GOP-index) path,
motivated by a latency-bound training box (25 ms to S3, workers at 5%
CPU on an idle 200 Gbps link — ~100 serialized ranged reads per batch):

- ONE read_blob_ranges call per batch (pylance >= 9) covering every
  planned range of every episode the batch touches — header ranges for
  cold episodes plus window GOP ranges, span-merged (64 KiB gap) and
  gap-filtered against each source's buffered chunks. Runtime-probed:
  without the API (or without an index) sources fall back to their own
  ranged fetches on a thread pool, results identical (tested).
- Decoder-open prefetch now includes the moov-following box header and
  the first-packet region: ffmpeg touches both during open, which cost
  hidden fallback round trips on every cold episode (32 per batch under
  random-window training).
- Per-episode decodes run on a persistent per-worker pool (torchcodec
  releases the GIL); decoder creation rides in the task so cold moov
  parses overlap too.

Same-region S3 benchmark (6 workers, batch 16, vpt_9x): 1.13 -> 0.35
s/batch (14.1 -> 45.2 windows/s, 3.2x) at unchanged bytes/window; the
gain grows with link RTT since the wave replaces ~ranges x RTT of
serialized waiting with ~1 x RTT.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
take_blobs is a network round trip per CALL, not per row — but the batch
path opened each cold episode's blob with its own single-row call, ~32
serialized requests per batch that the fetch wave did not cover. Found
on an HF-bucket training run: the gateway's request quota was exhausted
by blob opens alone (44x 429 crash) after the wave had already collapsed
the range reads.

Cold keys now collect first and open in one multi-row take_blobs, so a
batch costs exactly two requests: one blob-open call + one range wave.
Returned entries are pinned by the batch's dict, so mid-batch cache
eviction (batches touching more episodes than decoder_cache_size) cannot
invalidate in-flight work.

Same-region S3 benchmark (6 workers, batch 16, vpt_9x): 0.35 -> 0.25
s/batch (45 -> 65 windows/s); cumulative over the pre-wave reader:
1.13 -> 0.25 s/batch (4.5x). On request-quota'd or high-RTT stores the
change is qualitative (429 crash -> ~7x under quota).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant