Skip to content

Commit 3f1019f

Browse files
hamersawhamersaw
authored
fix: exclude netty from bundle jars to prevent split-package conflicts (#458)
## Summary The `lance-spark-bundle-*` jars currently ship a partial copy of Netty (specifically `io/netty/buffer/*` and `io/netty/util/*` from `io.netty:netty-common` and `io.netty:netty-buffer`, pulled transitively via Arrow's `arrow-memory-netty` at `compile` scope). Netty's `channel/transport` classes are **not** included and are expected from the Spark runtime. This split-package layout is fragile. Classpath ordering decides which jar wins per package, and a version skew between the bundled `util`/`buffer` packages and the runtime's `channel`/`transport` packages blows up at super-call time. ### Reproducer Using the Spark 4.1 bundle against a Spark runtime with a mismatched Netty in `$SPARK_HOME/jars/`: ``` java.lang.NoSuchMethodError: 'void io.netty.util.concurrent.SingleThreadEventExecutor.<init>( io.netty.util.concurrent.EventExecutorGroup, java.util.concurrent.Executor, boolean, boolean, int, io.netty.util.concurrent.RejectedExecutionHandler)' at io.netty.channel.SingleThreadEventLoop.<init>(SingleThreadEventLoop.java:83) at io.netty.channel.SingleThreadIoEventLoop.<init>(SingleThreadIoEventLoop.java:102) at io.netty.channel.MultiThreadIoEventLoopGroup.newChild(MultiThreadIoEventLoopGroup.java:212) ... at org.apache.spark.network.util.NettyUtils.createEventLoop(NettyUtils.java:86) at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:499) at org.apache.spark.SparkContext.<init>(SparkContext.scala:501) ``` The 4.2 `SingleThreadIoEventLoop` (from Spark's netty-transport) calls the 6-arg super-ctor on `SingleThreadEventExecutor`, but the resolved `SingleThreadEventExecutor` is the 4.1 version which only has a 5-arg ctor. ### Fix Exclude `io.netty:*` from the shade plugin `artifactSet` in each bundle pom. Spark owns Netty at runtime and hard-references `io.netty.*` directly — shading/relocation isn't viable, so the bundle must defer entirely to the runtime. Arrow's `arrow-memory-netty-buffer-patch` classes (e.g. `NettyArrowBuf`, `PooledByteBufAllocatorL`) also live in the `io.netty.buffer` package to reach package-private APIs. They ship under a different groupId and remain in the bundle — they're Arrow's code, not Netty's, and don't participate in Netty version conflicts. Applied to all 6 bundle modules: 3.4 / 3.5 / 4.0 / 4.1 across Scala 2.12 and 2.13. ### Verification Rebuilt `lance-spark-bundle-4.1_2.13` and confirmed: - Before: bundle contained `io/netty/util/concurrent/SingleThreadEventExecutor.class` + full Netty util/buffer packages from `io.netty:netty-common` 4.2.7 and `io.netty:netty-buffer` 4.2.7. - After: only 9 `io/netty/buffer/*` entries remain, all from `arrow-memory-netty-buffer-patch` — no real Netty classes. ## Test plan - [x] `./mvnw -pl lance-spark-bundle-4.1_2.13 -am package -DskipTests` succeeds - [x] `unzip -l` on the rebuilt bundle confirms no Netty jar contents remain (only Arrow's netty-buffer-patch classes) - [ ] Run the failing PySpark integration against the rebuilt 4.1 bundle to confirm the `NoSuchMethodError` is gone - [ ] Spot-check one 3.x bundle build to confirm identical behavior across Spark versions 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent 0146c3b commit 3f1019f

6 files changed

Lines changed: 30 additions & 0 deletions

File tree

‎lance-spark-bundle-3.4_2.12/pom.xml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,11 @@
8585
<phase>package</phase>
8686
<configuration>
8787
<finalName>${project.artifactId}-${project.version}</finalName>
88+
<artifactSet>
89+
<excludes>
90+
<exclude>io.netty:*</exclude>
91+
</excludes>
92+
</artifactSet>
8893
<transformers>
8994
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
9095
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer" />

‎lance-spark-bundle-3.4_2.13/pom.xml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,11 @@
8787
<phase>package</phase>
8888
<configuration>
8989
<finalName>${project.artifactId}-${project.version}</finalName>
90+
<artifactSet>
91+
<excludes>
92+
<exclude>io.netty:*</exclude>
93+
</excludes>
94+
</artifactSet>
9095
<transformers>
9196
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
9297
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer" />

‎lance-spark-bundle-3.5_2.12/pom.xml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,11 @@
8282
<phase>package</phase>
8383
<configuration>
8484
<finalName>${project.artifactId}-${project.version}</finalName>
85+
<artifactSet>
86+
<excludes>
87+
<exclude>io.netty:*</exclude>
88+
</excludes>
89+
</artifactSet>
8590
<transformers>
8691
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
8792
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer" />

‎lance-spark-bundle-3.5_2.13/pom.xml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,11 @@
8686
<phase>package</phase>
8787
<configuration>
8888
<finalName>${project.artifactId}-${project.version}</finalName>
89+
<artifactSet>
90+
<excludes>
91+
<exclude>io.netty:*</exclude>
92+
</excludes>
93+
</artifactSet>
8994
<transformers>
9095
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
9196
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer" />

‎lance-spark-bundle-4.0_2.13/pom.xml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,11 @@
8888
<phase>package</phase>
8989
<configuration>
9090
<finalName>${project.artifactId}-${project.version}</finalName>
91+
<artifactSet>
92+
<excludes>
93+
<exclude>io.netty:*</exclude>
94+
</excludes>
95+
</artifactSet>
9196
<transformers>
9297
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
9398
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer" />

‎lance-spark-bundle-4.1_2.13/pom.xml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,11 @@
8888
<phase>package</phase>
8989
<configuration>
9090
<finalName>${project.artifactId}-${project.version}</finalName>
91+
<artifactSet>
92+
<excludes>
93+
<exclude>io.netty:*</exclude>
94+
</excludes>
95+
</artifactSet>
9196
<transformers>
9297
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
9398
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer" />

0 commit comments

Comments
 (0)