Add DeltaLakeService to resolve a Delta table's parquet files - #911
Closed
SeanDuHare wants to merge 1 commit into
Closed
Add DeltaLakeService to resolve a Delta table's parquet files#911SeanDuHare wants to merge 1 commit into
SeanDuHare wants to merge 1 commit into
Conversation
Delta Lake does not modify the parquet format: a table is a directory of ordinary parquet data files plus a "_delta_log" transaction log. DuckDB has a delta extension but it is not built for wasm, so to read one we have to work out the file list ourselves. Scanning every parquet under the table root is not enough. Files a later UPDATE, DELETE or OPTIMIZE superseded are still on disk, and only the log knows they are no longer part of the table -- scanning them resurrects deleted rows and duplicates updated ones. So this replays the log: seed from the checkpoint, then apply each commit's add/remove actions. Every request is a GET at a path the protocol makes deterministic, and the log is never listed. That is what lets a table on a plain web server (nginx with autoindex off, Azure blob, a CDN) work the same as one on S3 -- object listing is an S3 API, not an HTTP one. Reading the checkpoint needs a parquet reader, which this class has no business owning, so it takes a CheckpointReader callback instead. Two things worth knowing about the error handling. 403 is treated as "absent" alongside 404, because an S3 bucket granting s3:GetObject but not s3:ListBucket answers 403 for a key that does not exist -- the usual setup for a public read-only bucket, and without this the walk cannot find its end. And table features that change how data files are read (deletion vectors, column mapping, reader v3+, multi-part and v2 checkpoints) are rejected outright rather than silently misread. The tests serve a log from memory over a stubbed HTTP client. Most cases hand-write it, so one replays a verbatim delta-rs log kept as a fixture -- otherwise the suite would only prove we parse our own idea of the format, not what a real writer emits.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second PR in the Delta Lake stack. Base branch is
prep/delta-lake-groundwork,not
main— review only this branch's own diff (two new files, nothing else touched).Adds the service that works out which parquet files make up a Delta Lake table.
Nothing calls it yet; wiring it into
DatabaseServiceis the next PR.Why replay the log instead of scanning the directory
Files that a later UPDATE, DELETE, MERGE or OPTIMIZE superseded are still on
disk. Only the log knows they are no longer part of the table.
Measured on a real delta-rs table with one delete in its history: scanning every
parquet returned 3 rows (2 of them deleted); log replay returned 1, matching the
deltalakereference reader.Why this also unblocks non-S3 hosting
Every request is a GET at a path the Delta protocol makes deterministic, and the
log is never listed. Object listing is an S3 API, not an HTTP one — so a table on
nginx with
autoindex off, Azure blob, or a CDN works exactly like one on S3.Two error-handling decisions worth a look
403 counts as "absent" alongside 404. An S3 bucket granting
s3:GetObjectbutnot
s3:ListBucketanswers 403 for a key that does not exist — the usual setup fora public read-only bucket, including this repo's own datasets bucket (verified).
Without this the commit walk cannot find its end. The trade: a genuinely
unreadable table reports "no data files found" rather than "access denied".
Unsupported table features are rejected, not approximated. Deletion vectors,
column mapping, reader v3+, and multi-part or v2 checkpoints each raise a specific
error rather than silently returning wrong rows.
Design note
Reading a checkpoint needs a parquet reader, which this class has no business
owning — so it takes a
CheckpointReadercallback, supplied byDatabaseServicein the next PR. That keeps this service purely about transport, and free of DOM
APIs so it can run inside the DuckDB web worker.
Verification
s3:→ https conversion,aged-out logs, 403-as-absent, each rejected feature. One asserts no request ever
contains
list-type.— file lists match the reference reader exactly.
typeCheck,lintand the full suite pass on this commit alone.