feat: push down nested struct field projections#706
Open
yyzhao2025 wants to merge 1 commit into
Open
Conversation
Bridge Spark nested schema pruning to Lance scanner column projection so non-nullable struct subfields can be read as leaf columns instead of forcing Lance to read the full top-level struct. This adds a nested projection planning layer that derives Lance leaf column paths from Spark's required schema, carries those projected columns through LanceScan and LanceInputPartition, and uses them when constructing ScanOptions on executors. It also reconstructs projected struct columns for Spark's columnar path by wrapping projected child vectors and filling unprojected siblings with null constant vectors. Nullable parent structs, arrays, and maps remain on the conservative top-level projection path to preserve existing correctness semantics. Include projected columns in scan equality/hashCode so Spark ReusedExchange does not incorrectly reuse scans with different nested projections. Closes lance-format#704.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements nested struct subfield projection pushdown for Lance Spark.
Spark's
nestedSchemaPruningcan pass a pruned nested schema to data sources, but Lance Spark previously still projected the top-level struct column into the Lance scanner. While this remained correct throughReadSchemaNestedStructWidening, it meant Lance still had to read the full struct from disk, so the IO benefit of nested schema pruning was lost.This change bridges Spark's required nested schema to Lance's leaf column projection support, allowing non-nullable nested struct subfields to be read as leaf projections during Lance scans.
Main Changes
ReadSchemaNestedColumnProjectionto derive Lance projection columns from Spark's required read schema.projectedColumnsthroughLanceScanandLanceInputPartition.projectedColumnsinLanceScan.equals()andhashCode()so SparkReusedExchangedoes not incorrectly reuse scans with different nested projections.ScanOptions.columns()in the fragment scanner.ProjectedStructColumnVectorto reconstruct Spark struct vectors from projected child vectors in the columnar read path.Correctness Boundaries
This implementation is intentionally conservative.
Only non-nullable parent structs are expanded into leaf projections. Nullable parent structs continue to use top-level projection so that parent validity bitmap semantics are preserved. Arrays and maps also stop projection recursion for now.
This PR does not change existing
_rowaddr, blob, zonemap, or SPJ-related read behavior.Testing
Ran full module-level validation locally with an arm64 JDK sandbox:
JAVA_HOME=/tmp/lance-spark-jdk17-arm64 ./mvnw test \ -pl lance-spark-3.5_2.12 \ -DrecompileMode=allResult:
Also validated the affected test matrix across Spark/Scala modules:
JAVA_HOME=/tmp/lance-spark-jdk17-arm64 ./mvnw test \ -pl lance-spark-3.4_2.12,lance-spark-3.4_2.13,lance-spark-3.5_2.12,lance-spark-3.5_2.13,lance-spark-4.0_2.13,lance-spark-4.1_2.13 \ -DrecompileMode=all \ -Dtest=ReadSchemaNestedColumnProjectionTest,ProjectedStructColumnVectorTest,LanceFragmentScannerTest,LanceScanTest,LanceColumnarPartitionReaderTest,LanceCountStarPartitionReaderTest,LanceDatasetReadTest,SparkConnectorReadTestResult:
Closes #704.