Skip to content

feat: vendor the specification's example extensions into substrait-extensions - #60

Merged
nielspardon merged 1 commit into
mainfrom
vendor-spec-examples
Aug 14, 2026
Merged

feat: vendor the specification's example extensions into substrait-extensions#60
nielspardon merged 1 commit into
mainfrom
vendor-spec-examples

Conversation

@nielspardon

Copy link
Copy Markdown
Member

Problem

The specification keeps its example extension and type YAML under site/examples, and those files are the only worked examples of large parts of the simple-extension schema: parameterized user-defined types, NSTRUCT structures, variadic and optional type parameters, lambda signatures, metadata at every level, and deprecated markers.

An SDK implementing an extension parser wants exactly that as a fixture set, but none of it has been reachable from a published artifact — so a consumer had to vendor it out of the documentation tree itself. This is the packaging half of substrait-io/substrait#1136, which settled the question of where the examples live by concluding that shipping them is a packaging decision rather than a reason to restructure the specification layout.

Change

Vendor site/examples/extensions and site/examples/types into all four substrait-extensions artifacts, alongside the extensions/, text/ and function-test-case trees the same scripts already assemble from the spec subtree.

Artifact How Where it lands
python cp -r + a package-data entry substrait_extensions.examples.extensions / .types
rust include_dir!, matching how the test cases are exposed, plus the crate include list substrait_extensions::examples::EXAMPLES
cpp added to the existing install(DIRECTORY ...) <datadir>/examples/{extensions,types}/
java two from(...) { into(...) } lines substrait/examples/{extensions,types}/ on the classpath

The python entry is declared on the top-level package as examples/**/*.yaml rather than per subpackage, because package-data globs are matched per package and are not recursive — a ["*.yaml"] on substrait_extensions.examples would ship nothing.

Examples are kept out of the catalog

They go in a separate examples/ tree rather than being merged into extensions/, because they are documentation illustrations, not entries in the Substrait extension catalog. Anything enumerating the catalog must not see them — the Rust crate builds its EXTENSIONS map and SIMPLE_EXTENSIONS URN table by walking extensions/, so an example landing there would be registered as a real extension.

Each artifact therefore asserts both directions: the examples are present, and absent from the catalog surface. Those guards are load-bearing rather than decorative — pointing Java's copy at substrait/extensions instead of substrait/examples/extensions fails examplesAreNotBundledAsExtensions.

The guards key on file stem and resource path, which is what this repository controls. Whether an example's URN collides with an official extension's is the specification's invariant to keep, and re-asserting it here would tie these tests to whichever spec release they are built against: on releases before substrait-io/substrait#1136, lambda_function_example.yaml declares extension:io.substrait:functions_list — the official functions_list URN — so a URN-collision assertion in this repo would fail against the current release even though nothing here is wrong. Worth adding on the spec side; there is no equivalent check there yet.

Not included

The plan examples in site/examples/proto-textformat are protobuf text format, not simple-extension YAML, so they fit the protobuf artifacts rather than this one. Easy to add later if there is appetite for plan round-trip fixtures.

Notes for review

  • No change to attach_subtree.sh — it attaches the whole specification repository, so site/examples is already present in CI and in the publish workflows.
  • Verified against two specification states: the current release layout and main after refactor(extensions)!: move example YAML files out of the extensions folder substrait#1136 (which relocates two files into site/examples/extensions and rewrites the example URNs). All four artifacts build and test clean against both, which is the property that lets this merge independently of the next spec release.
  • Touching all four language trees means all four ci_* workflows run.

…tensions

The specification keeps example extension and type YAML under `site/examples`, and those files are the only worked examples of large parts of the simple-extension schema — parameterized user-defined types, NSTRUCT structures, variadic and optional parameters, lambda signatures, metadata and deprecation markers. An SDK implementing an extension parser wants them as fixtures, but they have not been reachable from any published artifact, so a consumer had to vendor them out of the docs tree itself.

Vendor them into the four `substrait-extensions` artifacts alongside `extensions/`, `text/` and the function test cases, which the same scripts already assemble from the spec subtree.

They are deliberately kept in a separate `examples/` tree rather than merged into `extensions/`, because they are documentation illustrations and not entries in the Substrait extension catalog. Code that enumerates the catalog must not see them: the Rust crate builds its `EXTENSIONS` map and `SIMPLE_EXTENSIONS` URN table by walking `extensions/` only. Each artifact carries a test asserting the examples are both present and absent from the catalog surface, keyed on file stem and resource path, so a change that flattens the copy fails rather than silently registering examples as real extensions. Whether an example's *URN* collides with an official one is the specification's invariant to keep, not this repository's, so it is not re-asserted here — that would tie these tests to the spec release they happen to be built against.

Per artifact: Python gains an `examples` package, plus a `package-data` entry — the globs are per-package and not recursive, so the nested `examples/extensions` and `examples/types` need the pattern declared on the top-level package. Rust embeds the tree with `include_dir`, matching how the test cases are exposed, and adds it to the crate's `include` list. C++ installs the directory alongside the other data dirs. Java bundles the two trees as classpath resources under `substrait/examples/`.

The plan examples in `site/examples/proto-textformat` are not vendored: they are protobuf text format rather than simple-extension YAML, so they fit the protobuf artifacts rather than this one. That can be revisited separately.

This needs no change to `attach_subtree.sh`, which already attaches the whole specification repository, so `site/examples` is present in CI and in the publish workflows.
@nielspardon
nielspardon merged commit 4a845cd into main Aug 14, 2026
5 checks passed
@nielspardon
nielspardon deleted the vendor-spec-examples branch August 14, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants