Skip to content

[MINOR] Clean up Spark 3.2 build/config/docs leftovers #12550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jackylee-ch merged 3 commits into from
Jul 20, 2026
Merged

[MINOR] Clean up Spark 3.2 build/config/docs leftovers #12550

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7cc7bb8
[MINOR] Clean up Spark 3.2 build/config/docs leftovers
LuciferYang Jul 17, 2026
87c3513
[MINOR] Drop volatile line number from NewToGluten troubleshooting ex…
LuciferYang Jul 17, 2026
891736e
Merge branch 'apache:main' into gluten-minor-build-spark32-leftovers
LuciferYang Jul 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 0 additions & 15 deletions LICENSE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,28 +212,13 @@
./backends-clickhouse/src/main/scala/org/apache/spark/sql/execution/SparkWriteFilesCommitProtocol.scala
./cpp-ch/local-engine/Parser/aggregate_function_parser/BloomFilterAggParser.cpp
./gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenExplainUtils.scala
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/FileSourceScanExecShim.scala
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatDataWriter.scala
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/WriteFiles.scala
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala.deprecated
./shims/spark32/src/main/scala/org/apache/spark/sql/execution/stat/StatFunctions.scala
./shims/spark32/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
./shims/spark33/src/main/scala/org/apache/spark/sql/execution/FileSourceScanExecShim.scala
./shims/spark33/src/main/scala/org/apache/spark/sql/execution/datasources/WriteFiles.scala
./shims/spark33/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
./shims/spark33/src/main/scala/org/apache/spark/sql/execution/stat/StatFunctions.scala
./tools/gluten-it/common/src/main/scala/org/apache/spark/sql/TestUtils.scala

Delta Lake(https://github.com/delta-io/delta)
./backends-clickhouse/src-delta-20/main/scala/org/apache/spark/sql/delta/DeltaLog.scala
./backends-clickhouse/src-delta-20/main/scala/org/apache/spark/sql/delta/Snapshot.scala
./backends-clickhouse/src-delta-20/main/scala/org/apache/spark/sql/delta/commands/DeleteCommand.scala
./backends-clickhouse/src-delta-20/main/scala/org/apache/spark/sql/delta/commands/MergeIntoCommand.scala
./backends-clickhouse/src-delta-20/main/scala/org/apache/spark/sql/delta/commands/OptimizeTableCommand.scala
./backends-clickhouse/src-delta-20/main/scala/org/apache/spark/sql/delta/commands/UpdateCommand.scala
./backends-clickhouse/src-delta-23/main/scala/org/apache/spark/sql/delta/DeltaLog.scala
./backends-clickhouse/src-delta-23/main/scala/org/apache/spark/sql/delta/Snapshot.scala
./backends-clickhouse/src-delta-23/main/scala/org/apache/spark/sql/delta/commands/DeleteCommand.scala
Expand Down
2 changes: 0 additions & 2 deletions dev/bloop-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,6 @@ declare -A MODULE_MAP=(

# Shims modules
["shims/common"]="spark-sql-columnar-shims-common"
["shims/spark32"]="spark-sql-columnar-shims-spark32"
["shims/spark33"]="spark-sql-columnar-shims-spark33"
["shims/spark34"]="spark-sql-columnar-shims-spark34"
["shims/spark35"]="spark-sql-columnar-shims-spark35"
Expand All @@ -87,7 +86,6 @@ declare -A MODULE_MAP=(
# Unit test modules
["gluten-ut/common"]="gluten-ut-common"
["gluten-ut/test"]="gluten-ut-test"
["gluten-ut/spark32"]="gluten-ut-spark32"
["gluten-ut/spark33"]="gluten-ut-spark33"
["gluten-ut/spark34"]="gluten-ut-spark34"
["gluten-ut/spark35"]="gluten-ut-spark35"
Expand Down
2 changes: 0 additions & 2 deletions dev/run-scala-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,6 @@ declare -A MODULE_MAP=(

# Shims modules (Java only, no scala subdirectory)
["spark-sql-columnar-shims-common"]="shims/common:java"
["spark-sql-columnar-shims-spark32"]="shims/spark32:java"
["spark-sql-columnar-shims-spark33"]="shims/spark33:java"
["spark-sql-columnar-shims-spark34"]="shims/spark34:java"
["spark-sql-columnar-shims-spark35"]="shims/spark35:java"
Expand All @@ -174,7 +173,6 @@ declare -A MODULE_MAP=(
# Unit test modules
["gluten-ut-common"]="gluten-ut/common:scala"
["gluten-ut-test"]="gluten-ut/test:scala"
["gluten-ut-spark32"]="gluten-ut/spark32:scala"
["gluten-ut-spark33"]="gluten-ut/spark33:scala"
["gluten-ut-spark34"]="gluten-ut/spark34:scala"
["gluten-ut-spark35"]="gluten-ut/spark35:scala"
Expand Down
4 changes: 2 additions & 2 deletions docs/developers/NewToGluten.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -428,7 +428,7 @@ The following failure may occur in GitHub CI or local tests:
```log
- TPC-H q5 *** FAILED ***
Mismatch for query 5
Actual Plan path: /tmp/tpch-approved-plan/v2-bhj/spark322/5.txt
Golden Plan path: /opt/gluten/backends-velox/target/scala-2.12/test-classes/tpch-approved-plan/v2-bhj/spark322/5.txt (VeloxTPCHSuite.scala:101)
Actual Plan path: /tmp/tpch-approved-plan/v1-bhj/spark35/5.txt
Golden Plan path: /opt/gluten/backends-velox/target/scala-2.12/test-classes/tpch-approved-plan/v1-bhj/spark35/5.txt (VeloxTPCHSuite.scala:101)

@LuciferYang LuciferYang Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Dropped the :101 line number in 87c3513 — the source has already drifted to line 107, and keeping a specific line number in a troubleshooting example just guarantees future drift. The filename alone is enough for navigation.

```
To update the golden files, find the actual Gluten plans in GitHub CI Artifacts or the local `/tmp/` directory, then update the corresponding golden files in the `tpch-approved-plan/` directory.
4 changes: 1 addition & 3 deletions ep/build-clickhouse/src/resources/bin/gluten.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,7 @@ function start() {
DRIVER_OPTIONS="${DRIVER_OPTIONS} $(cat ${GLUTEN_HOME}/conf/gluten.properties | grep "^spark.driver.extraJavaOptions" | cut -d "=" -f 2)"

GLUTEN_JARS=
if [ "${SPARK_MAJOR_MINOR_VERSION}" == "3.2" ]; then
GLUTEN_JARS=${GLUTEN_HOME}/jars/spark32/*
elif [ "${SPARK_MAJOR_MINOR_VERSION}" == "3.3" ]; then
if [ "${SPARK_MAJOR_MINOR_VERSION}" == "3.3" ]; then
GLUTEN_JARS=${GLUTEN_HOME}/jars/spark33/*
else
echo "Unsupported spark version: ${SPARK_MAJOR_MINOR_VERSION}"
Expand Down
1 change: 0 additions & 1 deletion package/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,6 @@
<fileset>
<directory>target</directory>
<excludes>
<exclude>*spark3.2*</exclude>
<exclude>*spark3.3*</exclude>
<exclude>*spark3.4*</exclude>
<exclude>*spark3.5*</exclude>
Expand Down
Loading