Skip to content

feat: upgrade to substrait 0.63 and migrate extension URIs to URNs - #16

Open
EpsilonPrime wants to merge 1 commit into
mainfrom
substrait-063-urn-migration
Open

feat: upgrade to substrait 0.63 and migrate extension URIs to URNs#16
EpsilonPrime wants to merge 1 commit into
mainfrom
substrait-063-urn-migration

Conversation

@EpsilonPrime

Copy link
Copy Markdown
Member

PR 1 of a stack upgrading substrait-textplan to the latest substrait protos, then switching to substrait-packaging.

What

Bump the substrait crate 0.61 → 0.63 and the proto submodule to the matching spec commit (spec 0.85.0 / 2aaae7c0). The 0.63 protos remove the deprecated SimpleExtensionURI message and the extension_uris / extension_uri_reference fields in favor of SimpleExtensionURN / extension_urns / extension_urn_reference, so this migrates all URI handling to URNs.

Changes

  • Converter/printer/parser switched from extension_urisextension_urns and SimpleExtensionUriSimpleExtensionUrn; FunctionData.extension_uri_referenceextension_urn_reference.
  • Grammar: the extension_space token now accepts URN strings (extension:<owner>:<id>) which contain : — the old fragment-based URI token couldn't tokenize them.
  • Visitor generator: disambiguate leaf names that now collide because NamedTable, PrecisionTimestamp, PrecisionTime and IntervalCompound each exist in two message scopes (via NAME_OVERRIDES).
  • Printer: add not-yet-implemented arms for the new Lambda / LambdaInvocation expressions and Func type — full support lands later in the stack.
  • Fixtures: migrate all TPC-H roundtrip fixtures and the q6 converter fixture from yaml URIs (/functions_X.yaml) to URNs (extension:io.substrait:functions_X).

Tests

All 48 tests pass (cargo test); cargo fmt --check clean.

Bump the substrait crate 0.61 -> 0.63 and the proto submodule to the
matching spec commit (spec 0.85.0 / 2aaae7c0). The 0.63 protos remove the
deprecated SimpleExtensionURI message and the extension_uris /
extension_uri_reference fields in favor of SimpleExtensionURN /
extension_urns / extension_urn_reference, so migrate all URI handling to
URNs:

- Converter/printer/parser switched from extension_uris to extension_urns
  and from SimpleExtensionUri to SimpleExtensionUrn; FunctionData's
  extension_uri_reference renamed to extension_urn_reference.
- Grammar: the extension_space token now accepts URN strings
  (extension:<owner>:<id>) which contain ':' characters.
- Visitor generator: disambiguate leaf names that now collide because
  NamedTable, PrecisionTimestamp, PrecisionTime and IntervalCompound each
  exist in two message scopes, via NAME_OVERRIDES.
- Printer: add not-yet-implemented arms for the new Lambda /
  LambdaInvocation expressions and Func type (full support lands later in
  the stack).
- Migrate all TPC-H roundtrip fixtures and the q6 converter fixture from
  yaml URIs (/functions_X.yaml) to URNs (extension:io.substrait:functions_X).

All 48 tests pass.
@EpsilonPrime

Copy link
Copy Markdown
Member Author

Missing / stubbed features introduced in this PR (for later decisions)

The URN bump pulled in new 0.63 proto types that this PR only stubs (they compile and round-trip, but have no textplan syntax and emit *_NOT_YET_IMPLEMENTED when printed):

  • Lambda / LambdaInvocation expressions — new higher-order-function exprs; printer returns LAMBDA_NOT_YET_IMPLEMENTED / LAMBDA_INVOCATION_NOT_YET_IMPLEMENTED.
  • Func type — the higher-order function type; print_type returns FUNC_TYPE_NOT_YET_IMPLEMENTED.
  • Type aliasestype_alias_reference is not surfaced; the type-level Alias prints ALIAS_TYPE_NOT_YET_IMPLEMENTED.

These need a text-syntax design + fixtures to implement fully. Deferred per the stack plan; PR #17 lists the full consolidated gap list.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant