fix(engine/rocky-compiler): import-dbt correctness — SELECT* microbatch, wrong profile, seed-backed project - #984
Merged
Conversation
…T*, profile selection, seeds)
B1: a microbatch model imported with --microbatch-as=time_interval whose
compiled body is `SELECT * FROM up` failed `rocky compile` with E020 — the
emitted wrapper `SELECT * FROM (SELECT * FROM up) AS _rocky_microbatch`
resolved to an empty output schema because the SELECT* expansion didn't
recurse through a derived table whose own body is itself a SELECT*. Carry the
inner subquery's source-table names on the lineage TableReference
(derived_sources) and resolve them transitively against the model/source graph
in the semantic-graph builder, so the event_time column is present in the
output schema. Only fires when the inner columns can't be enumerated, so the
explicit-projection case is untouched and E020 still fires when the upstream
genuinely omits the time_column.
B2: the importer ignored dbt_project.yml's `profile:` key and selected the
alphabetically-first profile in profiles.yml, emitting the wrong adapter type
and catalog/schema. Read the named profile and select it; a named profile
absent from profiles.yml is a loud fallback, never a silent alphabetical pick.
B8: an emitted seed-backed project failed `rocky seed` ("Catalog main does not
exist") and bare-name `rocky run`, because the importer emitted no seed
sidecar and `rocky seed` defaults a transformation pipeline to main.seeds.
Emit a per-seed sidecar pinning the target to the project default
catalog/schema so seeds load where bare-name model refs resolve; existing
sidecars are left untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rocky import-dbthad three correctness defects when converting a dbt project into a runnable Rocky repo. This fixes all three.Bugs fixed
1. Microbatch
SELECT *failedrocky compilewith E020A microbatch model imported with
--microbatch-as=time_intervalwhose compiled body isSELECT * FROM <up>got wrapped asSELECT * FROM (SELECT * FROM up) AS _rocky_microbatchand failed compilation with E020 — the outer star resolved to an empty output schema becauseSELECT *expansion didn't recurse through a derived table whose own body is itself aSELECT *. The inner subquery's source-table names are now carried on the lineageTableReference(derived_sources) and resolved transitively against the model/source graph in the semantic-graph builder, so the event_time column is present in the output schema. This only fires when the inner columns can't be enumerated, so the explicit-projection case is untouched and E020 still fires when the upstream genuinely omits the time column.2. Wrong profile / adapter selected
The importer ignored
dbt_project.yml'sprofile:key and picked the alphabetically-first profile inprofiles.yml, emitting the wrong adapter type and catalog/schema. It now reads the named profile and selects it; a named profile that is absent fromprofiles.ymlis a loud fallback, never a silent alphabetical pick.3. Seed-backed project failed
rocky seed/ bare-namerocky runAn emitted seed-backed project failed
rocky seed("Catalog main does not exist") and bare-namerocky run, because the importer wrote no seed sidecar androcky seeddefaults a transformation pipeline tomain.seeds. The importer now writes a per-seed<stem>.tomlsidecar pinning the seed target to the project's default catalog/schema, so seeds land where bare-name model refs resolve. Existing sidecars are left untouched so hand-tuned overrides survive a re-import.How verified
cargo test -p rocky-compiler(370 tests, green) — transitive inner-star resolution plus E020 regression guards (must-not-fire when upstream projects the column / must-still-fire when it omits it); named-profile selection (named wins over alphabetical / named-but-absent surfaces a loud fallback / noprofile:key uses the sole profile); seed sidecar written + does-not-clobber-existing.cargo test -p rocky-sqllineage (green) —derived_sourcesrecorded for an inner-star derived table.cargo test -p rocky-cli --test import_dbt_emit(green) — end-to-end emit from the rich dbt fixture, honoring theprofile:key.