[MINOR][TEST] Drop 'min Spark 3.2' guards in test suites#12574
Open
LuciferYang wants to merge 2 commits into
Open
[MINOR][TEST] Drop 'min Spark 3.2' guards in test suites#12574LuciferYang wants to merge 2 commits into
LuciferYang wants to merge 2 commits into
Conversation
Spark 3.2 is no longer supported by Gluten (3.3 is the minimum). The
following 12 test callsites used `testWithMinSparkVersion("...", "3.2")`
as a version guard; the check is always true on any supported profile,
so replace them with a plain `test("...")`:
* `gluten-ut/.../GlutenExpressionMappingSuite.scala` (2 sites)
* `gluten-hudi/.../HudiSuite.scala` (3 sites)
* `backends-velox/.../VeloxColumnarCacheSuite.scala` (1 site)
* `gluten-delta/.../DeltaSuite.scala` (6 sites); also drop the adjacent
"NameMapping is supported in Delta 2.0 (related to Spark3.2.0)" comment,
which was pinned to the 3.2 guard.
The `testWithMinSparkVersion` helper itself is retained — it is still
used at 60+ other sites for 3.4/3.5/4.0 minVersion guards, which remain
legitimate.
No behavior change on any supported Spark version.
|
Run Gluten Clickhouse CI on x86 |
LuciferYang
marked this pull request as draft
July 20, 2026 08:08
Contributor
There was a problem hiding this comment.
Pull request overview
This PR continues the Spark 3.2 deprecation cleanup by removing testWithMinSparkVersion(..., "3.2") wrappers from selected test suites, replacing them with plain test("...") since Gluten only supports Spark 3.3+.
Changes:
- Replaced 3.2-gated
testWithMinSparkVersionwrappers with unconditionaltest(...)in several suites. - Removed a Spark-3.2/Delta-2.0-specific comment tied to the dropped guard.
- Simplified test definitions in Velox cache, Hudi, Delta, and expression-mapping suites.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| gluten-ut/test/src/test/scala/org/apache/gluten/expressions/GlutenExpressionMappingSuite.scala | Converts two Spark-3.2-min-guarded tests to plain test(...). |
| gluten-hudi/src/test/scala/org/apache/gluten/execution/HudiSuite.scala | Converts three Spark-3.2-min-guarded Hudi tests to plain test(...). |
| gluten-delta/src/test/scala/org/apache/gluten/execution/DeltaSuite.scala | Converts several Spark-3.2-min-guarded Delta tests to plain test(...) and drops a related comment. |
| backends-velox/src/test/scala/org/apache/gluten/execution/VeloxColumnarCacheSuite.scala | Converts one Spark-3.2-min-guarded test to plain test(...). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Address Copilot review feedback on apache#12574: the original single-line grep missed 6 additional `testWithMinSparkVersion(..., "3.2")` sites in DeltaSuite that span three lines each (inside `Seq("name", "id").foreach` column-mapping loops at L112, L155, L187, L222, L249, L274). Convert those to plain `test(...)` for consistency. After this commit a multi-line `awk` scan finds zero remaining `testWithMinSparkVersion(..., "3.[012]")` sites in the repo.
|
Run Gluten Clickhouse CI on x86 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Follow-up to the Spark 3.2 cleanup series (#12525, #12541, #12550, and the open #12543 / #12548). This drops the
testWithMinSparkVersion("...", "3.2")guard from 18 test callsites: the guard is always true on any currently supported Spark profile (3.3+), so the wrapper is replaced with a plaintest("...").Callsites converted (18 total across 4 test suites)
gluten-ut/test/.../GlutenExpressionMappingSuite.scalagluten-hudi/.../HudiSuite.scalabackends-velox/.../VeloxColumnarCacheSuite.scalagluten-delta/.../DeltaSuite.scala— single-linegluten-delta/.../DeltaSuite.scala— multi-line insideSeq("name", "id").foreach { … }The
DeltaSuite.scala:69comment "NameMapping is supported in Delta 2.0 (related to Spark3.2.0)" is also dropped — it was pinned to the same 3.2 guard.Not touched
The
testWithMinSparkVersionhelper itself (gluten-substrait/.../GlutenQueryTest.scala:132) is kept — it is still used at 60+ other sites with3.4/3.5/4.0minVersionarguments, which remain legitimate on the currently supported Spark matrix.No behavior change on any supported Spark version.
How was this patch tested?
./build/mvn -Pbackends-velox -Pspark-3.5 -Pspark-ut -pl :gluten-ut-test -am test-compile -DskipTests: SUCCESS./build/mvn -Pbackends-velox -Pspark-3.5 -Pdelta -pl :gluten-delta -am test-compile -DskipTests: SUCCESS./build/mvn -Pbackends-velox -Pspark-3.5 -Phudi -pl :gluten-hudi -am test-compile -DskipTests: SUCCESS./build/mvn -Pbackends-velox -Pspark-3.5 -pl backends-velox -am test-compile -DskipTests: SUCCESS./dev/format-scala-code.sh: no additional diffawkscan fortestWithMinSparkVersion(..., "3.[012]")across all*/test/*.scalafiles: 0 remaining occurrencesWas this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-4-7