You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: tighten READMEs and make the dbt comparison accurate and fair (#737)
Reduce density and AI-speech tells across the four READMEs, and rework the
dbt comparison so it holds up to scrutiny.
- Move the root README's comparison table below the demos into a
"How it compares to dbt Core" section so it no longer leads.
- Fix debunktable cells: dbt Core has model contracts (1.5+) and dbt retry
(1.6+); replace snark with neutral architectural distinctions.
- Header is now "What dbt Core does"; split the Fusion OSS engine from
dbt's paid platform; note that dbt Core defined this category.
- Drop the year-stale Fusion beta date (still in public beta as of 2026-04).
- Collapse repeating intro paragraphs; tighten the seven trust dimensions.
- Sweep em-dash overuse (39->1 in the root README), triple-rhythm fragments,
and balanced-antithesis flourishes.
Copy file name to clipboardExpand all lines: README.md
+47-49Lines changed: 47 additions & 49 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,33 +12,14 @@
12
12
13
13
**Rocky is the typed graph between your code and whichever warehouse, table format, or query engine you've chosen.**
14
14
15
-
Not a warehouse. Not a table format. Not a query engine. The trust plane for your data — a typed compiler, named branches, deterministic replay, column-level lineage, compile-time contracts, and per-model cost — running over your existing Databricks / Snowflake / BigQuery / DuckDB. Apache 2.0.
15
+
It is a typed compiler that runs over your existing Databricks, Snowflake, BigQuery, or DuckDB and owns the graph between your code and your data: named branches, deterministic replay, column-level lineage, compile-time contracts, and per-model cost. Storage and compute stay where they are, and Rocky works on the SQL you already have. The `.rocky` DSL is there when you want it. Apache 2.0.
16
16
17
-
Rocky exists because the disasters that cost data teams real money — silent schema drift wrecking a revenue dashboard, a column rename quietly poisoning 47 downstream models, an auditor asking who touched `fct_revenue.amount` and when, a cost spike that no one can attribute to a model — all share a common shape. They're problems the warehouse can't see and the templating engine on top of it was never asked to. Rocky owns the graph between your code and the warehouse so those problems become compile errors, blocked PRs, and verifiable artifacts instead of pages and post-mortems.
18
-
19
-
It's built for the team running production-critical multi-tenant pipelines on Databricks today, on Snowflake or BigQuery tomorrow, who can't tolerate another silent failure. Storage and compute stay where they are. **Rocky works on the SQL you already have** — the `.rocky` DSL is an acceleration when you want it, not a gate.
17
+
The failures that cost data teams the most are invisible to the warehouse and out of scope for the templating layer above it: schema drift, column-rename blast radius, dialect divergence, cost spikes nobody can attribute. Rocky turns them into compile errors and blocked PRs.
20
18
21
19
<palign="center">
22
20
<imgsrc="docs/public/demo-quickstart.gif"alt="Rocky quickstart — create a project, compile, and run 3 models in under 15s"width="900" />
23
21
</p>
24
22
25
-
## The disasters Rocky prevents
26
-
27
-
| Disaster | What dbt does | What Rocky does |
28
-
|---|---|---|
29
-
| Upstream changes a column type | Silent — fails downstream, hours later |`E013` at compile, blocks the PR |
30
-
| Required column dropped from a contract | No contract concept |`E010` at compile, blocks the PR |
31
-
| Column rename with unknown blast radius |`dbt docs` post-hoc, table-level (dbt Cloud Enterprise has column lineage in their UI, also post-hoc, not PR-blocking) |`rocky lineage-diff` at PR time, column-level, downstream consumers listed, blocks the merge |
32
-
|`SELECT *` pulls a new column you didn't expect | Silent |`P002` warning, downstream consumers named |
33
-
| Snowflake-only function written for a Databricks project | Runs in dev, fails in prod |`P001` dialect-portability lint at compile |
34
-
| Run cost doubles, no one knows which model | Manual warehouse spelunking |`RunOutput.cost_summary` per model, every run |
35
-
| Auditor asks: who changed `fct_revenue.amount`, when, and why? | Git blame + screenshots |`rocky replay <run_id>` — content-addressed record of the exact code and the output it produced |
36
-
| Sev-2 at 3 AM, half the pipeline already ran | Re-run everything |`rocky run --resume-latest` — checkpoint, three-state circuit breaker, skip what succeeded |
37
-
38
-
Each row is a real failure mode, with a Rocky command that turns it into a non-event. The same primitives — typed compiler, content-addressed state, column-level lineage, per-model cost — back every row.
39
-
40
-
**Already on dbt?**`rocky import-dbt` converts a vanilla dbt project to Rocky in one command. The "What dbt does" column above is dbt Core's behavior. **dbt Fusion** — dbt Labs' Rust rewrite of dbt Core (public beta) — catches some compile-time issues that dbt Core misses, but doesn't ship named branches, content-addressed deterministic replay, per-model cost attribution as a first-class column, dialect-portability lint across warehouses, or declarative governance + masking outside dbt platform's paid tiers. Those stay Rocky's surface, Apache 2.0.
41
-
42
23
## Try it in 60 seconds
43
24
44
25
```bash
@@ -55,27 +36,27 @@ cd my-first-project
55
36
rocky compile && rocky test&& rocky run
56
37
```
57
38
58
-
No credentials needed — the playground runs end-to-end on local DuckDB.
39
+
No credentials needed; the playground runs end-to-end on local DuckDB.
59
40
60
41
## Who Rocky is for
61
42
62
-
Rocky is built first for **data platform engineers running production-critical, multi-tenant pipelines on Databricks** — the team that's hit dbt's ceiling, where silent failures cost real money, and where Dagster is already the orchestrator. That's the launch wedge, and that's where Rocky is most battle-tested.
43
+
Rocky is built first for **data platform engineers running production-critical, multi-tenant pipelines on Databricks**, where silent failures cost real money and Dagster is already the orchestrator. That is the launch wedge, and where Rocky is most battle-tested.
63
44
64
-
The next ring out: **Snowflake and BigQuery shops**currently evaluating SQLMesh, who want correctness moved to the compiler (not the planner) and prefer SQL by default over Python-first ergonomics. Adapters are Beta today; see [Where Rocky is today](#where-rocky-is-today) below.
45
+
The next ring out: **Snowflake and BigQuery shops** evaluating SQLMesh, who want correctness in the compiler rather than the planner and prefer SQL by default. Adapters are Beta today; see [Where Rocky is today](#where-rocky-is-today) below.
65
46
66
47
## See it in action
67
48
68
-
Each demo below is a self-contained POC in [`examples/playground/pocs/`](examples/playground/) — `cd` in, run `./run.sh`, reproduce locally.
49
+
Each demo below is a self-contained POC in [`examples/playground/pocs/`](examples/playground/): `cd` in, run `./run.sh`, and reproduce it locally.
69
50
70
51
### Detects schema drift the moment it happens
71
52
72
-
A source column type changes upstream. On the next run, Rocky diffs source vs. target, drops the target, and recreates it. No silent data corruption, no dbt-style quiet divergence.
53
+
A source column type changes upstream. On the next run, Rocky diffs source against target, drops the target, and recreates it. No silent data corruption.
73
54
74
55
<palign="center">
75
56
<imgsrc="docs/public/demo-drift-recover.gif"alt="rocky run detects source type change and recreates the target"width="900" />
### AI model generation with a compile-validate loop
111
92
112
-
Describe what you want in plain English. Rocky generates a Rocky DSL model, compiles it, and retries on parse failure — the `Attempts: 2` line shows the loop catching a first-pass error invisibly.
93
+
Describe what you want in plain English. Rocky generates a Rocky DSL model, compiles it, and retries on parse failure. The `Attempts: 2` line shows the loop catching a first-pass error.
113
94
114
95
<palign="center">
115
96
<imgsrc="docs/public/demo-ai-model-generation.gif"alt="rocky ai generates a .rocky model from natural language intent, Attempts: 2"width="900" />
### PR-time blast-radius with `rocky lineage-diff`
121
102
122
-
Compare two git refs and get a per-changed-column readout of downstream consumers — pre-rendered Markdown drops straight into a GitHub PR comment. CODEOWNERS-style review tooling can't reach this granularity without a compiled engine.
103
+
Compare two git refs and get a per-changed-column readout of downstream consumers; the pre-rendered Markdown drops straight into a GitHub PR comment. CODEOWNERS-style review tooling can't reach this granularity without a compiled engine.
123
104
124
105
<palign="center">
125
106
<imgsrc="docs/public/demo-lineage-diff.gif"alt="rocky lineage-diff main lists added and removed columns across two models with downstream consumers per change"width="900" />
### Classify columns, mask by environment, gate CI
131
112
132
-
Tag PII columns in the model sidecar; bind tags to mask strategies in `[mask]` / `[mask.<env>]`. `rocky compliance --env prod --fail-on exception` exits 1 the moment a classified column has no resolved strategy — a one-line CI gate against accidentally-unmasked data.
113
+
Tag PII columns in the model sidecar, and bind tags to mask strategies in `[mask]` / `[mask.<env>]`. `rocky compliance --env prod --fail-on exception` exits 1 the moment a classified column has no resolved strategy: a one-line CI gate against accidentallyunmasked data.
133
114
134
115
<palign="center">
135
116
<imgsrc="docs/public/demo-classification-masking.gif"alt="rocky compliance rolls up classification tags to mask strategies; --fail-on exception exits 1, gating CI on unmasked PII"width="900" />
### Incremental loads with persistent watermark state
141
122
142
-
`strategy = "incremental"` plus a `timestamp_column` is all it takes. Rocky writes the high-water mark to the embedded state store; subsequent runs only `INSERT … WHERE timestamp > watermark`. Append 25 rows after a 500-row load — run 2 still finishes in 0.2s.
123
+
`strategy = "incremental"` plus a `timestamp_column` is all it takes. Rocky writes the high-water mark to the embedded state store, and subsequent runs only `INSERT … WHERE timestamp > watermark`. Append 25 rows after a 500-row load, and run 2 still finishes in 0.2s.
143
124
144
125
<palign="center">
145
126
<imgsrc="docs/public/demo-incremental-watermark.gif"alt="rocky run with incremental strategy: run 1 copies 500 rows; appended 25 rows; run 2 only copies the delta in 0.2s"width="900" />
The trust primitives — compiler, branches, replay, lineage, contracts, cost attribution — are production-grade on Databricks. We're explicit about the rest:
133
+
The trust primitives (compiler, branches, replay, lineage, contracts, cost attribution) are production-grade on Databricks. The rest is in progress:
153
134
154
-
-**Databricks is the production target for 2026.** Snowflake, BigQuery, and Trino adapters are Beta — connection, execution, and the core run loop work, but conformance coverage is still growing. If your enterprise warehouse is Snowflake or BigQuery and you need it production-grade today, talk to us.
155
-
-**AI is a growing surface, not a finished product.** The compile-validate loop (generate → type-check → auto-fix → land) is real and shipped; the broader story (mass refactor across the DAG, auto-migration from a columntype change, schema-aware assertion generation) is on the roadmap, not the changelog.
156
-
-**Iceberg.** REST-catalog source discovery is Beta. Content-addressed writes round-trip as Iceberg through Delta UniForm — shipped end-to-end (Wave 2). First-class Iceberg-native writes without the Delta intermediate are on the 2026 roadmap.
135
+
-**Databricks is the production target for 2026.** Snowflake, BigQuery, and Trino adapters are Beta: connection, execution, and the core run loop work, but conformance coverage is still growing. If your enterprise warehouse is Snowflake or BigQuery and you need it production-grade today, talk to us.
136
+
-**AI is a growing surface, not a finished product.** The compile-validate loop (generate, type-check, auto-fix, then land) is shipped. The broader story (mass refactor across the DAG, auto-migration from a column-type change, schema-aware assertion generation) is on the roadmap.
137
+
-**Iceberg.** REST-catalog source discovery is Beta. Content-addressed writes round-trip as Iceberg through Delta UniForm, shipped end-to-end. First-class Iceberg-native writes without the Delta intermediate are on the 2026 roadmap.
157
138
-**No built-in semantic layer.** Rocky's typed IR is the right home for one. Today, integrate with Cube, the dbt Semantic Layer, or your existing metric store.
158
-
-**Orchestration: Dagster is first-class.** A `rocky serve` standalone path exists; native Airflow / Prefect integrations are not yet shipped — they're called from the CLI like any other binary.
139
+
-**Orchestration: Dagster is first-class.** A `rocky serve` standalone path exists; native Airflow and Prefect integrations are not yet shipped, so they're called from the CLI like any other binary.
140
+
141
+
If those gaps are blockers for your team, [open a discussion](https://github.com/rocky-data/rocky/discussions). The roadmap is shaped by where production pipelines are actually getting hurt.
142
+
143
+
## How it compares to dbt Core
144
+
145
+
| Disaster | What dbt Core does | What Rocky does |
146
+
|---|---|---|
147
+
| Upstream changes a column type | Silent; surfaces as a downstream failure later |`E013` at compile, blocks the PR |
148
+
| Required column dropped from a contract | Caught at build time via `contract: enforced`|`E010` at compile, blocks the PR |
149
+
| Column rename with unknown blast radius |`dbt docs` is post-hoc and table-level; dbt Cloud Enterprise adds column lineage in the UI, also post-hoc and not PR-blocking |`rocky lineage-diff` at PR time, column-level, downstream consumers listed, blocks the merge |
150
+
|`SELECT *` pulls a new column you didn't expect | Silent |`P002` warning, downstream consumers named |
151
+
| Snowflake-only function written for a Databricks project | No dialect-portability lint; runs in dev, fails in prod |`P001` dialect-portability lint at compile |
152
+
| Run cost doubles, no one knows which model | No per-model cost attribution; reconstruct it from warehouse query history |`RunOutput.cost_summary` per model, every run |
153
+
| Auditor asks: who changed `fct_revenue.amount`, when, and why? | Run history in dbt Cloud, but no content-addressed record of code and output |`rocky replay <run_id>`: a content-addressed record of the exact code and the output it produced |
154
+
| Sev-2 at 3 AM, half the pipeline already ran |`dbt retry` resumes from the failed model; no within-run checkpoint or circuit breaker |`rocky run --resume-latest`: checkpoint, three-state circuit breaker, skip what succeeded |
155
+
156
+
Each row is a real failure mode and a Rocky command that turns it into a non-event. The same primitives back every row: typed compiler, content-addressed state, column-level lineage, per-model cost.
159
157
160
-
If those gaps are blockers for your team, [open a discussion](https://github.com/rocky-data/rocky/discussions) — the roadmap is shaped by where production pipelines are actually getting hurt.
158
+
dbt Core defined this category, and `rocky import-dbt` converts a vanilla dbt project to Rocky in one command. dbt Fusion (dbt Labs' Rust rewrite of dbt Core, in public beta) closes part of the compile-time gap by type-checking SQL, though it still templates with Jinja. Neither open-source engine ships named branches, content-addressed replay, per-model cost as a first-class column, cross-warehouse dialect lint, or declarative RBAC and masking. Some governance features live in dbt's paid platform; Rocky's are open source under Apache 2.0.
|[`integrations/dagster/`](integrations/dagster/)|`dagster-rocky` PyPI wheel | Python | Dagster resource and component wrapping the Rocky CLI |
168
-
|[`editors/vscode/`](editors/vscode/)| Rocky VSIX | TypeScript | VS Code extension — LSP client + commands for AI features |
166
+
|[`editors/vscode/`](editors/vscode/)| Rocky VSIX | TypeScript | VS Code extension; LSP client + commands for AI features |
169
167
|[`examples/playground/`](examples/playground/)| (config only) | TOML / SQL | Self-contained DuckDB sample pipeline used for smoke tests and benchmarks |
170
168
171
169
Each subproject has its own README with detailed usage. The [`engine/README.md`](engine/README.md) is the canonical product reference for the Rocky CLI.
@@ -196,7 +194,7 @@ just test # runs all test suites
196
194
just lint # cargo clippy/fmt + ruff + eslint
197
195
```
198
196
199
-
`just` is optional — you can also build each subproject directly. See [`CONTRIBUTING.md`](CONTRIBUTING.md) for per-subproject build commands.
197
+
`just` is optional; you can also build each subproject directly. See [`CONTRIBUTING.md`](CONTRIBUTING.md) for per-subproject build commands.
200
198
201
199
## Releases
202
200
@@ -210,11 +208,11 @@ See [`CONTRIBUTING.md`](CONTRIBUTING.md#releases) for the full release flow.
Full documentation lives at **[rocky-data.dev](https://rocky-data.dev)**: concepts, guides, CLI reference, Dagster integration, and the adapter SDK.
214
212
215
213
## Contributing
216
214
217
-
See [`CONTRIBUTING.md`](CONTRIBUTING.md). Before opening a PR, please read the cross-project change guidance — schema and DSL changes must update consumers atomically.
215
+
See [`CONTRIBUTING.md`](CONTRIBUTING.md). Before opening a PR, please read the cross-project change guidance: schema and DSL changes must update consumers atomically.
0 commit comments