wip schema query batching - #60
Conversation
880fd84 to
be1668e
Compare
| @@ -8,9 +8,30 @@ module SchemaStatements # :nodoc: | |||
| def indexes(table_name) | |||
| indexes = [] | |||
| current_index = nil | |||
There was a problem hiding this comment.
TABLE_NAME INDEX_NAME SEQ_IN_INDEX COLUMN_NAME
books index_books_on_author_id_and_name 1 author_id
books index_books_on_author_id_and_name 2 name
adjustments index_adjustments_on_adjustable 1 adjustable_type
adjustments index_adjustments_on_adjustable 2 adjustable_id
The current_index iteration is kind of hard to grok, so documenting my understanding while I have it: each row is not an index but a column within an index. So the looping is to build an IndexDefinition per index, which requires looping all the rows for an index to collect all the columns.
c1a7679 to
ca0d6e1
Compare
|
Fable review: DetailsStack Review: Batched schema-cache dump queries (11 commits,
|
| Suite | MariaDB 12.3 | MySQL 8.4 | PostgreSQL |
|---|---|---|---|
abstract_mysql_adapter/schema_test + case_insensitive_table_names_test |
15 runs, 0 F/E (1 skip: lctn=0) | ✅ (no skips, lctn=1 exercised) | n/a |
connection_adapters/schema_cache_test |
27 runs, 0 F/E | ✅ | ✅ |
migration_test + schema_dumper_test + active_schema_test (+ table_options, charset_collation on MariaDB) |
177 runs, 0 F/E | 158 runs, 0 F/E | 169 runs, 0 F/E |
postgresql/schema_test |
n/a | n/a | 112 runs, 0 F/E (2 skips) |
Probe results
# Literal 'NULL' string default (commit 4 regression)
base (fbd03fc7fa), MySQL 8.4: default="NULL" ✓ old behavior
HEAD, MySQL 8.4: default=nil ✗ regression
HEAD, MariaDB 12: default="NULL" ✓
# PostgreSQL, HEAD
CREATE TABLE zero_col () → columns raises StatementInvalid ✗ (was [])
primary_keys('missing') → [] (was: raise) ⚠ documented change
primary_key('missing') → nil (was: raise) ⚠ documented change
columns/primary_keys/indexes('MixedCase') → correct ✓
# PostgreSQL, at commit 9876eaba6e (mid-stack)
primary_keys('MixedCase') → [] (should be ["id"]) ✗ fixed in commit 10
ca0d6e1 to
3272f42
Compare
On its own this change may appear worse than the previous solution since we have to manually construct the information schema query. However, using information schema enables a future refactor which will query the indexes for multiple tables at once (which SHOW KEYS FROM) cannot. This will be particularly useful for dumping the Schema Cache, which currently queries the indexes (among other things) for every table sequentially. One consequence of this change is that indexes will now be returned in alphabetical order instead of definition order for MySQL. This now aligns the MySQL behavior with PostgreSQL, where we are already sorting indexes by their name.
Add internal indexes_for_tables interface which iterates through tables and calls indexes, but can be optimized by adapter subclasses to be batched. This new method is used in the Schema Cache dumping code to replace N queries to retrieve index metadata with a single query (when supported). Also implement this new method for MySQL. In my _very_ large application, it reduces schema dump time by ~25%.
More verbose, but this will enable extending the API to support querying by multiple table_names at once. Since information_schema returns an empty set instead of raising for a missing table, `column_definitions` now raises `StatementInvalid` explicitly to preserve the old behavior. information_schema.COLUMNS.COLUMN_DEFAULT differs from the old `SHOW FULL FIELDS` Default in two ways, both handled here: - MariaDB reports a nullable no-default column's default as the string "NULL" rather than SQL NULL; normalize that to nil so the column is treated as having no default. This is gated on `mariadb?` — MySQL already returns SQL NULL for a no-default column and leaves a literal "NULL" string default unquoted, so normalizing unconditionally would wipe MySQL's literal "NULL" defaults. - MariaDB quotes string-literal defaults (e.g. `'abc'`) and escapes embedded single quotes — by doubling them in varchar columns (`''`) but with a backslash in text columns (`\'`). MySQL leaves string defaults unquoted, so the strip-and-unescape step is gated on `mariadb?`; it strips the wrapping quotes and undoes both escape forms so the default matches what `SHOW FULL FIELDS` reported.
update_fields_for_mariadb now uses information_schema.COLUMNS.COLUMN_DEFAULT to distinguish function defaults from literal strings that look like function calls. MariaDB quotes string literals there (e.g. 'uuid()') but leaves function defaults unquoted (e.g. uuid()), so the disambiguation no longer requires parsing SHOW CREATE TABLE output. Because column_definitions already selects COLUMN_DEFAULT AS 'Default' from information_schema.columns, the field's Default value already carries the quoted/unquoted distinction - so no extra query is needed (unlike the SHOW FULL FIELDS based path), and update_fields_for_mariadb no longer needs the table name.
Introduce a `columns_for_tables` method on the abstract adapter whose default implementation just loops over each table and calls `columns`, leaving room for adapters to override it with a single batched query. `SchemaCache#add_all` now goes through it instead of calling `columns` once per table, mirroring the `indexes_for_tables` batch added previously. This sets up the next commit, which overrides `columns_for_tables` on the MySQL adapter to fetch every table's columns in one query.
Override `columns_for_tables` on the MySQL adapter to pull every requested
table's columns from `information_schema.columns` in one go, ordering by
(`table_name`, `ORDINAL_POSITION`) so the rows come back grouped by table.
`column_definitions_for_tables` builds its result from the query rows (keyed
by the database's own `TABLE_NAME`) rather than pre-seeding by the requested
name, and `column_definitions` recovers its single table via `each_value.first`.
This keeps single-table introspection correct under `lower_case_table_names = 1/2`,
where the database's table name can differ in case from the requested one.
`SchemaCache#add` keeps using the single-table `columns` getter rather than
the batched `columns_for_tables`, because the batch keys results by the
database's `TABLE_NAME` — so a miscased `add("MixedCase")` would be attributed
to the database's lowercased name. `add_all` is the only batched caller, and
it passes the real data-source names from `tables_to_cache`, which already
match the database's `TABLE_NAME`.
`SchemaCache#add_all` (used by `db:schema:cache:dump`) now issues one columns
query instead of one per table.
`SchemaCache#add_all` was still calling `primary_keys` once per table after the indexes and columns batches. Introduce a `primary_keys_for_tables` interface on the abstract adapter — defaulting to a per-table loop, like `indexes_for_tables` and `columns_for_tables` — and route `add_all` through it. On MySQL, override `primary_keys_for_tables` to read every requested table's primary key columns from `information_schema.statistics` in one query (the same table the indexes batch reads), ordered by `table_name, seq_in_index` so composite keys come back in order. The single-table `primary_keys` now delegates to it, so both paths share one query. `add_all` stores the key the way `primary_key` does — the column name for single-column keys, the array for composite ones — so `SchemaCache#primary_keys` keeps returning what callers expect.
`SchemaCache#add_all` was still calling `primary_keys` once per table on PostgreSQL, after the indexes and columns batches. Override `primary_keys_for_tables` to fetch every requested table's primary key columns from `pg_index` / `pg_attribute` in one query, and route `add_all` through it (mirroring the MySQL batch). The query scopes `pg_index.indrelid` with `to_regclass` rather than casting the names to `regclass`: `to_regclass` resolves each name the same way (an unqualified name picks the first schema on the search path, a qualified name pins its schema) but returns NULL for a missing relation instead of raising, so one bad name can't fail the whole batch. Composite keys are ordered per table with `array_position(i.indkey, a.attnum)`. Rows are keyed by the requested table-name string (not the relname), so schema-qualified requests round-trip through `add_all`' `[table.to_s]`. `primary_keys` now delegates to `primary_keys_for_tables` for a single table; a missing relation returns `[]` instead of raising, which the schema cache already guards against with `data_source_exists?`.
`SchemaCache#add_all` was still calling `columns` once per table on PostgreSQL. Override `columns_for_tables` to fetch every requested table's columns in one query, and route `add_all` through it (mirroring the MySQL batch). The batch lives in `column_definitions_for_tables`, which reads `pg_attribute` for all the tables at once, scoping `a.attrelid` with `to_regclass` (as `primary_keys_for_tables` does) and ordering by `attrelid, attnum` so each table's columns come back in order. Rows are keyed by the requested table-name string, so schema-qualified requests round-trip through `add_all`'s `.fetch(table.to_s)`. `columns_for_tables` maps each table's fields to columns with the existing `new_column_from_field`. `column_definitions` now delegates to `column_definitions_for_tables` for a single table. Because `to_regclass` returns NULL for a missing relation instead of raising, and the batch pre-seeds an empty array for every requested name, an empty result no longer distinguishes a missing table from a zero-column one (PostgreSQL permits `CREATE TABLE t ()`). `column_definitions` therefore checks `to_regclass` resolution and raises `StatementInvalid` only when the relation is actually missing, preserving the previous behavior for both missing and zero-column tables.
Building a schema cache dump issues one query per table for indexes, columns, and primary keys. The columns and primary keys queries were already batched for PostgreSQL; indexes were not, so dumping a cache for a database with many tables paid a round-trip per table just to read indexes. Add indexes_for_tables, which fetches indexes for several tables in one query by scoping pg_class/pg_namespace to the requested names. It reuses the same WHERE shape as indexes (relname + search-path namespace for unqualified names, relname + pinned namespace for qualified names), so the existing multi-schema search-path behavior is preserved exactly. The single-table indexes method now delegates to indexes_for_tables, and the row-to-IndexDefinition logic is extracted into a private helper so both paths share it. Indexes are keyed by the requested table-name string, matching how SchemaCache#add_all looks them up, so schema-qualified names round-trip correctly.
3272f42 to
6f84b26
Compare
2 participants
opening for easier diff review