Skip to content

fix(engine/rocky-compiler): import-dbt correctness — SELECT* microbatch, wrong profile, seed-backed project - #984

Merged
hugocorreia90 merged 1 commit into
mainfrom
fix/import-dbt-correctness
Jun 27, 2026
Merged

fix(engine/rocky-compiler): import-dbt correctness — SELECT* microbatch, wrong profile, seed-backed project#984
hugocorreia90 merged 1 commit into
mainfrom
fix/import-dbt-correctness

Conversation

@hugocorreia90

Copy link
Copy Markdown
Contributor

rocky import-dbt had three correctness defects when converting a dbt project into a runnable Rocky repo. This fixes all three.

Bugs fixed

1. Microbatch SELECT * failed rocky compile with E020

A microbatch model imported with --microbatch-as=time_interval whose compiled body is SELECT * FROM <up> got wrapped as SELECT * FROM (SELECT * FROM up) AS _rocky_microbatch and failed compilation with E020 — the outer star resolved to an empty output schema because SELECT * expansion didn't recurse through a derived table whose own body is itself a SELECT *. The inner subquery's source-table names are now carried on the lineage TableReference (derived_sources) and resolved transitively against the model/source graph in the semantic-graph builder, so the event_time column is present in the output schema. This only fires when the inner columns can't be enumerated, so the explicit-projection case is untouched and E020 still fires when the upstream genuinely omits the time column.

2. Wrong profile / adapter selected

The importer ignored dbt_project.yml's profile: key and picked the alphabetically-first profile in profiles.yml, emitting the wrong adapter type and catalog/schema. It now reads the named profile and selects it; a named profile that is absent from profiles.yml is a loud fallback, never a silent alphabetical pick.

3. Seed-backed project failed rocky seed / bare-name rocky run

An emitted seed-backed project failed rocky seed ("Catalog main does not exist") and bare-name rocky run, because the importer wrote no seed sidecar and rocky seed defaults a transformation pipeline to main.seeds. The importer now writes a per-seed <stem>.toml sidecar pinning the seed target to the project's default catalog/schema, so seeds land where bare-name model refs resolve. Existing sidecars are left untouched so hand-tuned overrides survive a re-import.

How verified

  • cargo test -p rocky-compiler (370 tests, green) — transitive inner-star resolution plus E020 regression guards (must-not-fire when upstream projects the column / must-still-fire when it omits it); named-profile selection (named wins over alphabetical / named-but-absent surfaces a loud fallback / no profile: key uses the sole profile); seed sidecar written + does-not-clobber-existing.
  • cargo test -p rocky-sql lineage (green) — derived_sources recorded for an inner-star derived table.
  • cargo test -p rocky-cli --test import_dbt_emit (green) — end-to-end emit from the rich dbt fixture, honoring the profile: key.

…T*, profile selection, seeds)

B1: a microbatch model imported with --microbatch-as=time_interval whose
compiled body is `SELECT * FROM up` failed `rocky compile` with E020 — the
emitted wrapper `SELECT * FROM (SELECT * FROM up) AS _rocky_microbatch`
resolved to an empty output schema because the SELECT* expansion didn't
recurse through a derived table whose own body is itself a SELECT*. Carry the
inner subquery's source-table names on the lineage TableReference
(derived_sources) and resolve them transitively against the model/source graph
in the semantic-graph builder, so the event_time column is present in the
output schema. Only fires when the inner columns can't be enumerated, so the
explicit-projection case is untouched and E020 still fires when the upstream
genuinely omits the time_column.

B2: the importer ignored dbt_project.yml's `profile:` key and selected the
alphabetically-first profile in profiles.yml, emitting the wrong adapter type
and catalog/schema. Read the named profile and select it; a named profile
absent from profiles.yml is a loud fallback, never a silent alphabetical pick.

B8: an emitted seed-backed project failed `rocky seed` ("Catalog main does not
exist") and bare-name `rocky run`, because the importer emitted no seed
sidecar and `rocky seed` defaults a transformation pipeline to main.seeds.
Emit a per-seed sidecar pinning the target to the project default
catalog/schema so seeds load where bare-name model refs resolve; existing
sidecars are left untouched.
@hugocorreia90
hugocorreia90 merged commit 56d33ca into main Jun 27, 2026
13 checks passed
@hugocorreia90
hugocorreia90 deleted the fix/import-dbt-correctness branch June 27, 2026 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant