[MINOR][CH] Correct misleading Spark 3.2 comment in ReadBufferBuilder#12583
Open
LuciferYang wants to merge 2 commits into
Open
[MINOR][CH] Correct misleading Spark 3.2 comment in ReadBufferBuilder#12583LuciferYang wants to merge 2 commits into
LuciferYang wants to merge 2 commits into
Conversation
The fallback path at `if (!file_size.has_value()) { ... }` was commented
as "only for spark3.2 file partition not contained file size", but the
fallback is still triggered on Spark 3.3+ whenever the Substrait
`LocalFilesNode` was built without a `fileSizes` list — e.g. by
`IcebergLocalFilesBuilder`. Reword the comment to describe the actual
condition. No code change.
|
Run Gluten Clickhouse CI on x86 |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates a misleading Spark 3.2-specific comment in the ClickHouse backend’s ReadBufferBuilder.cpp to better reflect when the file-size fallback path is taken (i.e., when Substrait LocalFiles inputs omit file size metadata), without changing runtime behavior.
Changes:
- Reworded the HDFS file-size fallback comment to be version-agnostic and describe the real triggering condition.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Address Copilot review feedback on apache#12583: distinguish the producer-side identifier (`fileSizes` list in `LocalFilesNode`) from the per-file protobuf property (`fileSize`) so the comment matches the code.
|
Run Gluten Clickhouse CI on x86 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Final follow-up in the Spark 3.2 cleanup series (#12525, #12541, #12550; open #12543 / #12548 / #12574).
The fallback path in
ReadBufferBuilder.cppatis still triggered on Spark 3.3+ whenever the Substrait
LocalFilesNodewas built without afileSizeslist — e.g. byIcebergLocalFilesBuilder— so the comment "only for spark3.2" is factually wrong on the currently supported version matrix.This PR rewords the comment to describe the actual condition. The fallback code itself is unchanged.
How was this patch tested?
gluten-substrait/.../LocalFilesNode.java:194-203that thefileProperties.setFileSize(...)block only fires when the caller passes a non-emptyfileSizeslist; producers such asIcebergLocalFilesBuilderdo not pass one, so the fallback is unavoidable on Spark 3.3+.grep -rn 'spark3.2\|Spark 3.2' cpp-ch: 0 remaining occurrences after this change.clang-format 12.0.1and the project requiresclang-format-15; running the mismatched version would produce a diff CI rejects (see AGENTS.md).Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-4-7