You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+14Lines changed: 14 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -73,6 +73,20 @@ Apache Wayang provides a flexible architecture which enables easy addition of ne
73
73
74
74
For a quick guide on how to run WordCount see [here](guides/tutorial.md).
75
75
76
+
### Spark Dataset / DataFrame pipelines
77
+
78
+
Wayang’s Spark platform can now execute end-to-end pipelines on Spark `Dataset[Row]` (aka DataFrames). This is particularly useful when working with lakehouse-style storage (Parquet/Delta) or when you want to plug Spark ML stages into a Wayang plan without repeatedly falling back to RDDs.
79
+
80
+
To build a Dataset-backed pipeline:
81
+
82
+
1.**Use the Dataset-aware plan builder APIs.**
83
+
-`PlanBuilder.readParquet(..., preferDataset = true)` (or `JavaPlanBuilder.readParquet(..., ..., true)`) reads Parquet files directly into a Dataset channel.
84
+
-`DataQuanta.writeParquet(..., preferDataset = true)` writes a Dataset channel without converting it back to an RDD.
85
+
2.**Keep operators dataset-compatible.** Most operators continue to work unchanged; if an operator explicitly prefers RDDs, Wayang will insert the necessary conversions automatically (at an additional cost). Custom operators can expose `DatasetChannel` descriptors to stay in the dataframe world.
86
+
3.**Let the optimizer do the rest.** The optimizer now assigns a higher cost to Dataset↔RDD conversions, so once you opt into Dataset sources/sinks the plan will stay in Dataset form by default.
87
+
88
+
No extra flags are required—just opt into the Dataset-based APIs where you want dataframe semantics. If you see unexpected conversions in your execution plan, check that the upstream/downstream operators you use can consume `DatasetChannel`s; otherwise Wayang will insert a conversion operator for you.
89
+
76
90
## Quick Guide for Developing with Wayang
77
91
78
92
For a quick guide on how to use Wayang in your Java/Scala project see [here](guides/develop-with-Wayang.md).
Licensed to the Apache Software Foundation (ASF) under one or more
4
+
contributor license agreements. See the NOTICE file distributed with
5
+
this work for additional information regarding copyright ownership.
6
+
The ASF licenses this file to You under the Apache License, Version 2.0
7
+
(the "License"); you may not use this file except in compliance with
8
+
the License. You may obtain a copy of the License at
9
+
10
+
http://www.apache.org/licenses/LICENSE-2.0
11
+
12
+
Unless required by applicable law or agreed to in writing, software
13
+
distributed under the License is distributed on an "AS IS" BASIS,
14
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15
+
See the License for the specific language governing permissions and
16
+
limitations under the License.
17
+
18
+
-->
19
+
20
+
---
21
+
title: Spark Dataset pipelines
22
+
description: How to build Wayang jobs that stay on Spark Datasets/DataFrames from source to sink.
23
+
---
24
+
25
+
Wayang’s Spark backend can now run entire pipelines on Spark `Dataset[Row]` (a.k.a. DataFrames). Use this mode when you ingest from lakehouse formats (Parquet/Delta), interoperate with Spark ML stages, or simply prefer schema-aware processing. This guide explains how to opt in.
26
+
27
+
## When to use Dataset channels
28
+
29
+
-**Lakehouse storage:** Reading Parquet/Delta directly into datasets avoids repeated schema inference and keeps Spark’s optimized Parquet reader in play.
30
+
-**Spark ML:** Our ML operators already convert RDDs into DataFrames internally. Feeding them a dataset channel skips that conversion and preserves column names.
31
+
-**Federated pipelines:** You can mix dataset-backed stages on Spark with other platforms; Wayang will insert conversions only when strictly necessary.
32
+
33
+
## Enable Dataset sources and sinks
34
+
35
+
1.**Plan builder APIs:**
36
+
-`PlanBuilder.readParquet(..., preferDataset = true)` (Scala) or `JavaPlanBuilder.readParquet(..., ..., true)` loads Parquet files into a `DatasetChannel` instead of an `RddChannel`.
37
+
-`DataQuanta.writeParquet(..., preferDataset = true)` writes a dataset back to Parquet without converting to RDD first.
38
+
2.**Prefer dataset-friendly operators:** Most unary/binary operators accept either channel type, but custom operators can advertise dataset descriptors explicitly. See `DatasetChannel` in `wayang-platforms/wayang-spark` for details.
39
+
3.**Let the optimizer keep it:** The optimizer now assigns costs to Dataset↔RDD conversions, so once your plan starts with a dataset channel it will stay in dataset form unless an operator demands an RDD.
40
+
41
+
## Mixing with RDD operators
42
+
43
+
If a stage only supports RDDs, Wayang inserts conversion operators automatically:
44
+
45
+
-`SparkRddToDatasetOperator` converts an RDD of `org.apache.wayang.basic.data.Record` into a Spark `Dataset[Row]` (using sampled schema inference or `RecordType`).
46
+
-`SparkDatasetToRddOperator` turns a `Dataset[Row]` back into a JavaRDD.`Record`.
47
+
48
+
Both conversions carry non-trivial load profiles. You’ll see them in plan explanations if you mix dataset- and RDD-only operators.
49
+
50
+
## Developer checklist
51
+
52
+
-**Use `RecordType` when possible.** Providing field names in your logical operators helps the converter derive a precise schema.
53
+
-**Re-use `sparkExecutor.ss`.** When writing custom Spark operators that build DataFrames, use the provided `SparkExecutor` instead of `SparkSession.builder()` to avoid extra contexts.
54
+
-**Watch plan explanations.** Run `PlanBuilder.buildAndExplain(true)` to verify whether conversions are inserted. If they are, consider adding dataset descriptors to your operators.
55
+
56
+
## Current limitations
57
+
58
+
- Only Parquet sources/sinks expose dataset-specific APIs today. Text/Object sources still produce RDD channels.
59
+
- ML4All pipelines currently emit plain `double[]`/`Double` RDDs. They still benefit from the internal DataFrame conversions but do not expose dataset channels yet.
60
+
61
+
Contributions to widen dataset support (e.g., dataset-aware `map`/`filter` or ML4All stages) are welcome.
0 commit comments