feat: vendor the specification's example extensions into substrait-extensions - #60
Merged
Conversation
…tensions The specification keeps example extension and type YAML under `site/examples`, and those files are the only worked examples of large parts of the simple-extension schema — parameterized user-defined types, NSTRUCT structures, variadic and optional parameters, lambda signatures, metadata and deprecation markers. An SDK implementing an extension parser wants them as fixtures, but they have not been reachable from any published artifact, so a consumer had to vendor them out of the docs tree itself. Vendor them into the four `substrait-extensions` artifacts alongside `extensions/`, `text/` and the function test cases, which the same scripts already assemble from the spec subtree. They are deliberately kept in a separate `examples/` tree rather than merged into `extensions/`, because they are documentation illustrations and not entries in the Substrait extension catalog. Code that enumerates the catalog must not see them: the Rust crate builds its `EXTENSIONS` map and `SIMPLE_EXTENSIONS` URN table by walking `extensions/` only. Each artifact carries a test asserting the examples are both present and absent from the catalog surface, keyed on file stem and resource path, so a change that flattens the copy fails rather than silently registering examples as real extensions. Whether an example's *URN* collides with an official one is the specification's invariant to keep, not this repository's, so it is not re-asserted here — that would tie these tests to the spec release they happen to be built against. Per artifact: Python gains an `examples` package, plus a `package-data` entry — the globs are per-package and not recursive, so the nested `examples/extensions` and `examples/types` need the pattern declared on the top-level package. Rust embeds the tree with `include_dir`, matching how the test cases are exposed, and adds it to the crate's `include` list. C++ installs the directory alongside the other data dirs. Java bundles the two trees as classpath resources under `substrait/examples/`. The plan examples in `site/examples/proto-textformat` are not vendored: they are protobuf text format rather than simple-extension YAML, so they fit the protobuf artifacts rather than this one. That can be revisited separately. This needs no change to `attach_subtree.sh`, which already attaches the whole specification repository, so `site/examples` is present in CI and in the publish workflows.
andrew-coleman
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The specification keeps its example extension and type YAML under
site/examples, and those files are the only worked examples of large parts of the simple-extension schema: parameterized user-defined types,NSTRUCTstructures, variadic and optional type parameters, lambda signatures,metadataat every level, anddeprecatedmarkers.An SDK implementing an extension parser wants exactly that as a fixture set, but none of it has been reachable from a published artifact — so a consumer had to vendor it out of the documentation tree itself. This is the packaging half of substrait-io/substrait#1136, which settled the question of where the examples live by concluding that shipping them is a packaging decision rather than a reason to restructure the specification layout.
Change
Vendor
site/examples/extensionsandsite/examples/typesinto all foursubstrait-extensionsartifacts, alongside theextensions/,text/and function-test-case trees the same scripts already assemble from the spec subtree.cp -r+ apackage-dataentrysubstrait_extensions.examples.extensions/.typesinclude_dir!, matching how the test cases are exposed, plus the crateincludelistsubstrait_extensions::examples::EXAMPLESinstall(DIRECTORY ...)<datadir>/examples/{extensions,types}/from(...) { into(...) }linessubstrait/examples/{extensions,types}/on the classpathThe python entry is declared on the top-level package as
examples/**/*.yamlrather than per subpackage, becausepackage-dataglobs are matched per package and are not recursive — a["*.yaml"]onsubstrait_extensions.exampleswould ship nothing.Examples are kept out of the catalog
They go in a separate
examples/tree rather than being merged intoextensions/, because they are documentation illustrations, not entries in the Substrait extension catalog. Anything enumerating the catalog must not see them — the Rust crate builds itsEXTENSIONSmap andSIMPLE_EXTENSIONSURN table by walkingextensions/, so an example landing there would be registered as a real extension.Each artifact therefore asserts both directions: the examples are present, and absent from the catalog surface. Those guards are load-bearing rather than decorative — pointing Java's copy at
substrait/extensionsinstead ofsubstrait/examples/extensionsfailsexamplesAreNotBundledAsExtensions.The guards key on file stem and resource path, which is what this repository controls. Whether an example's URN collides with an official extension's is the specification's invariant to keep, and re-asserting it here would tie these tests to whichever spec release they are built against: on releases before substrait-io/substrait#1136,
lambda_function_example.yamldeclaresextension:io.substrait:functions_list— the officialfunctions_listURN — so a URN-collision assertion in this repo would fail against the current release even though nothing here is wrong. Worth adding on the spec side; there is no equivalent check there yet.Not included
The plan examples in
site/examples/proto-textformatare protobuf text format, not simple-extension YAML, so they fit the protobuf artifacts rather than this one. Easy to add later if there is appetite for plan round-trip fixtures.Notes for review
attach_subtree.sh— it attaches the whole specification repository, sosite/examplesis already present in CI and in the publish workflows.mainafter refactor(extensions)!: move example YAML files out of the extensions folder substrait#1136 (which relocates two files intosite/examples/extensionsand rewrites the example URNs). All four artifacts build and test clean against both, which is the property that lets this merge independently of the next spec release.ci_*workflows run.