[GLUTEN-12474][CORE] Preserve V1 write ordering for dynamic partition writes#12514

Draft

wForget wants to merge 6 commits into

apache:mainfrom

wForget:GLUTEN-12474

wForget commented Jul 15, 2026 •

edited

Loading

Member

What changes are proposed in this pull request?

Fixes #12474.

Spark V1 writes prepare an ordering for partition and bucket columns before physical planning. WriteFilesExec assumes this ordering is preserved and therefore does not expose it through requiredChildOrdering.

Gluten may replace SortAggregateExec with a hash aggregate, which invalidates the ordering prepared by Spark. For dynamic partition writes using a single output writer, rows from the same partition may then become non-contiguous. This can cause the writer to reopen an existing output file and fail with FileAlreadyExistsException.

This PR:

derives the required V1 write ordering for WriteFilesExec in EnsureLocalSortRequirements;
inserts a local sort when the transformed child no longer satisfies that ordering;
handles static partition columns for both Hadoop filesystem and Hive writes;
adds Spark-version shims around V1WritesUtils.getSortOrder for Spark 3.4 through 4.1, while retaining empty ordering for Spark 3.3;
avoids unnecessarily applying the columnar heuristic transform when both the parent and child use the row path.

How was this patch tested?

Added a regression test to GlutenV1WriteCommandSuite for Spark 3.4, 3.5, 4.0, and 4.1.

The test performs a dynamic partition overwrite after an aggregation with:

spark.sql.maxConcurrentOutputFileWriters=0
spark.sql.sources.partitionOverwriteMode=DYNAMIC

It verifies that the write completes successfully and that the target table contains the expected rows. The test was also confirmed to reproduce the original write failure before applying the fix.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)


          test GLUTEN-12474

5f5ecb9

github-actions Bot commented Jul 15, 2026

Run Gluten Clickhouse CI on x86

github-actions Bot added the CORE label

wForget changed the title ~~Test~~ Test GLUTEN-12474


          Preserve V1 write ordering when inserting local sorts

91bf3e2

wForget commented Jul 15, 2026

Member Author

Successfully reproduced this issue, https://github.com/apache/gluten/actions/runs/29396675906/job/87292215408?pr=12514

2026-07-15T08:05:01.9825974Z - Gluten - GLUTEN-12474: preserve ordering for dynamic partition writes *** FAILED ***
2026-07-15T08:05:01.9827943Z   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 64.0 failed 1 times, most recent failure: Lost task 0.0 in stage 64.0 (TID 82) (705c9412ef57 executor driver): org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while writing rows to file:/__w/gluten/gluten/gluten-ut/spark35/target/scala-2.13/test-classes/unit-tests-working-home/spark-warehouse/gluten_12474_tgt/.spark-staging-2f3d085e-dac8-4ee5-92b7-b06bc0b10194.
2026-07-15T08:05:01.9830247Z 	at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:775)
2026-07-15T08:05:01.9831086Z 	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:420)
2026-07-15T08:05:01.9831899Z 	at org.apache.spark.sql.execution.datasources.WriteFilesExec.$anonfun$doExecuteWrite$1(WriteFiles.scala:100)
2026-07-15T08:05:01.9832558Z 	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:893)
2026-07-15T08:05:01.9833090Z 	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:893)
2026-07-15T08:05:01.9833638Z 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
2026-07-15T08:05:01.9834137Z 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
2026-07-15T08:05:01.9834561Z 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
2026-07-15T08:05:01.9834982Z 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
2026-07-15T08:05:01.9835494Z 	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
2026-07-15T08:05:01.9835965Z 	at org.apache.spark.scheduler.Task.run(Task.scala:141)
2026-07-15T08:05:01.9836424Z 	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
2026-07-15T08:05:01.9837231Z 	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
2026-07-15T08:05:01.9837858Z 	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
2026-07-15T08:05:01.9838472Z 	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
2026-07-15T08:05:01.9838952Z 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
2026-07-15T08:05:01.9839993Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2026-07-15T08:05:01.9840686Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2026-07-15T08:05:01.9841178Z 	at java.base/java.lang.Thread.run(Thread.java:833)
2026-07-15T08:05:01.9842856Z Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: File already exists: file:/__w/gluten/gluten/gluten-ut/spark35/target/scala-2.13/test-classes/unit-tests-working-home/spark-warehouse/gluten_12474_tgt/.spark-staging-2f3d085e-dac8-4ee5-92b7-b06bc0b10194/_temporary/0/_temporary/attempt_202607150105011536507608415885554_0064_m_000000_82/day=2026-06-01/part-00000-2f3d085e-dac8-4ee5-92b7-b06bc0b10194.c000.snappy.orc
2026-07-15T08:05:01.9844817Z 	at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:421)
2026-07-15T08:05:01.9845372Z 	at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:459)
2026-07-15T08:05:01.9846018Z 	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:433)
2026-07-15T08:05:01.9846650Z 	at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:521)
2026-07-15T08:05:01.9847179Z 	at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:500)
2026-07-15T08:05:01.9847670Z 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195)
2026-07-15T08:05:01.9848100Z 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)
2026-07-15T08:05:01.9848567Z 	at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:94)
2026-07-15T08:05:01.9849202Z 	at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:230)
2026-07-15T08:05:01.9849796Z 	at org.apache.orc.OrcFile.createWriter(OrcFile.java:1040)
2026-07-15T08:05:01.9850358Z 	at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:49)
2026-07-15T08:05:01.9851115Z 	at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:89)
2026-07-15T08:05:01.9852012Z 	at org.apache.spark.sql.execution.datasources.BaseDynamicPartitionDataWriter.renewCurrentWriter(FileFormatDataWriter.scala:298)
2026-07-15T08:05:01.9852980Z 	at org.apache.spark.sql.execution.datasources.DynamicPartitionDataSingleWriter.write(FileFormatDataWriter.scala:365)
2026-07-15T08:05:01.9853873Z 	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithMetrics(FileFormatDataWriter.scala:85)
2026-07-15T08:05:01.9855256Z 	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:92)
2026-07-15T08:05:01.9856136Z 	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:403)
2026-07-15T08:05:01.9856886Z 	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1397)
2026-07-15T08:05:01.9857571Z 	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:410)
2026-07-15T08:05:01.9858070Z 	... 17 more


          add unit tests for other spark version

1817d26

wForget mentioned this pull request

[VL] Gluten fails to satisfy the ordering required by dynamic partition column #12474

Open

github-actions Bot commented Jul 15, 2026

Run Gluten Clickhouse CI on x86

1 similar comment

github-actions Bot commented Jul 15, 2026

Run Gluten Clickhouse CI on x86

wForget added 2 commits

July 15, 2026 17:46

fix

50ccf39


          fix spotless check

9d9c4bc

github-actions Bot commented Jul 15, 2026

Run Gluten Clickhouse CI on x86

1 similar comment

github-actions Bot commented Jul 15, 2026

Run Gluten Clickhouse CI on x86

wForget changed the title ~~Test GLUTEN-12474~~ [GLUTEN-12474][CORE] Preserve V1 write ordering for dynamic partition writes

wForget commented Jul 16, 2026

Member Author

ClickHouse CI failures:

20:24:15  12:24:15.295 ERROR org.apache.spark.sql.execution.datasources.FileFormatWriter: Aborting job 1a7f0bae-125a-451d-a16d-04978ef8c357.
20:24:15  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 245.0 failed 1 times, most recent failure: Lost task 0.0 in stage 245.0 (TID 245) (gluten-gluten-ci-20575-cmcll-fz22k-wcm3r executor driver): org.apache.spark.SparkRuntimeException: [EXCEED_LIMIT_LENGTH] Exceeds char/varchar type length limitation: 5.
20:24:15  	at org.apache.spark.sql.errors.QueryExecutionErrors$.exceedMaxLimit(QueryExecutionErrors.scala:2577)
20:24:15  	at org.apache.spark.sql.errors.QueryExecutionErrors.exceedMaxLimit(QueryExecutionErrors.scala)
20:24:15  	at org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils.trimTrailingSpaces(CharVarcharCodegenUtils.java:31)
20:24:15  	at org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils.charTypeWriteSideCheck(CharVarcharCodegenUtils.java:44)
20:24:15  	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
20:24:15  	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
20:24:15  	at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
20:24:15  - length check for input string values: partitioned columns *** FAILED ***
20:24:15    java.lang.ClassCastException: class org.apache.spark.SparkRuntimeException cannot be cast to class org.apache.spark.SparkException (org.apache.spark.SparkRuntimeException and org.apache.spark.SparkException are in unnamed module of loader 'app')
20:24:15    at org.apache.spark.sql.CharVarcharTestSuite.$anonfun$$init$$84(CharVarcharTestSuite.scala:342)
20:24:15    at org.apache.spark.sql.CharVarcharTestSuite.$anonfun$$init$$84$adapted(CharVarcharTestSuite.scala:336)
20:24:15    at org.apache.spark.sql.CharVarcharTestSuite.$anonfun$testTableWrite$1(CharVarcharTestSuite.scala:309)
20:24:15    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
20:24:15    at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
20:24:15    at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
20:24:15    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
20:24:15    at org.apache.spark.sql.test.SQLTestUtilsBase.withTable(SQLTestUtils.scala:306)
20:24:15    at org.apache.spark.sql.test.SQLTestUtilsBase.withTable$(SQLTestUtils.scala:304)
20:24:15    at org.apache.spark.sql.FileSourceCharVarcharTestSuite.withTable(CharVarcharTestSuite.scala:953)
20:24:15    ...
......
20:24:24  12:24:24.247 ERROR org.apache.spark.sql.execution.datasources.FileFormatWriter: Aborting job 12d3d584-5c04-43b9-9661-c250ea02e220.
20:24:24  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 579.0 failed 1 times, most recent failure: Lost task 0.0 in stage 579.0 (TID 551) (gluten-gluten-ci-20575-cmcll-fz22k-wcm3r executor driver): org.apache.spark.SparkRuntimeException: [EXCEED_LIMIT_LENGTH] Exceeds char/varchar type length limitation: 5.
20:24:24  	at org.apache.spark.sql.errors.QueryExecutionErrors$.exceedMaxLimit(QueryExecutionErrors.scala:2577)
20:24:24  	at org.apache.spark.sql.errors.QueryExecutionErrors.exceedMaxLimit(QueryExecutionErrors.scala)
20:24:24  	at org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils.trimTrailingSpaces(CharVarcharCodegenUtils.java:31)
20:24:24  	at org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils.charTypeWriteSideCheck(CharVarcharCodegenUtils.java:44)
20:24:24  	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
20:24:24  	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
20:24:24  	at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
20:24:24  - char/varchar type values length check: partitioned columns of other types *** FAILED ***
20:24:24    java.lang.ClassCastException: class org.apache.spark.SparkRuntimeException cannot be cast to class org.apache.spark.SparkException (org.apache.spark.SparkRuntimeException and org.apache.spark.SparkException are in unnamed module of loader 'app')
20:24:24    at org.apache.spark.sql.FileSourceCharVarcharTestSuite.$anonfun$new$76(CharVarcharTestSuite.scala:1051)
20:24:24    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
20:24:24    at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
20:24:24    at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
20:24:24    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
20:24:24    at org.apache.spark.sql.test.SQLTestUtilsBase.withTable(SQLTestUtils.scala:306)
20:24:24    at org.apache.spark.sql.test.SQLTestUtilsBase.withTable$(SQLTestUtils.scala:304)
20:24:24    at org.apache.spark.sql.FileSourceCharVarcharTestSuite.withTable(CharVarcharTestSuite.scala:953)
20:24:24    at org.apache.spark.sql.FileSourceCharVarcharTestSuite.$anonfun$new$75(CharVarcharTestSuite.scala:1040)
20:24:24    at org.apache.spark.sql.FileSourceCharVarcharTestSuite.$anonfun$new$75$adapted(CharVarcharTestSuite.scala:1039)
20:24:24    ...

github-actions Bot commented Jul 17, 2026

Run Gluten Clickhouse CI on x86


          Enhance local sort handling for GlutenPlan support

71747fb

wForget force-pushed the GLUTEN-12474 branch from 2f0194c to 71747fb Compare

July 17, 2026 10:06

github-actions Bot commented Jul 17, 2026

Run Gluten Clickhouse CI on x86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE