feat(CE): add mysql source and destination connectors

[dataengineergaurav]

Description

Adds MySQL source and destination connectors for the Multiwoven Reverse ETL platform. Both connectors use the Sequel mysql2 adapter (same as the existing MariaDB connector).

• Source connector: check_connection, discover, read (with limit/offset)
• Destination connector: check_connection, discover, write (QueryBuilder-based SQL + db.run)

Related Issue

Closes #10

Type of Change

• New Connector (Destination or Source Connector)

How Has This Been Tested?

All 15 tests pass (9 source + 6 destination). Run with:

bundle exec rspec spec/multiwoven/integrations/source/mysql/client_spec.rb
bundle exec rspec spec/multiwoven/integrations/destination/mysql/client_spec.rb

Checklist:

• Branch name prefixed with feature/
• Added unit tests for the changes made
• Commit messages follow guidelines
• Tested on local/staging (all 15 tests pass)
• Added the new connector in rollout.rb
• Updated the version number of the gem (0.35.0 → 0.35.1)
• Changes for new connector documented in docs repo
• No new runtime dependencies needed (mysql2 already transitive via MariaDB)
• Code follows best practices

Summary by CodeRabbit

• New Features
  • Added a MySQL source connector with connection checking, schema discovery, and batch raw-SQL reads.
  • Added a MySQL destination connector with connection checking, schema discovery, and per-record writes.
  • Published MySQL source/destination configuration metadata and JSON specs (including required connection fields).
• Chores
  • Bumped the Rollout integration version to 0.38.0 and enabled MySQL for both sources and destinations.
• Tests
  • Added RSpec coverage for MySQL source and destination connection, discover, and read/write behavior.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds MySQL source and destination connectors (clients), configuration specs and metadata, registers the connectors in the integration loader and rollout, and updates the integrations gem version to 0.38.0 with comprehensive RSpec test suites.

Changes

MySQL Connectors

Layer / File(s)	Summary
MySQL Destination Connector Implementation and Tests integrations/lib/multiwoven/integrations/destination/mysql/client.rb, integrations/lib/multiwoven/integrations/destination/mysql/config/meta.json, integrations/lib/multiwoven/integrations/destination/mysql/config/spec.json, integrations/spec/multiwoven/integrations/destination/mysql/client_spec.rb	Implements destination Client with check_connection, discover (information_schema → Catalog/Streams), and write (per-record SQL via QueryBuilder.perform, db.run, success/failure tracking). Adds meta/spec JSON files and RSpec tests validating connection status, discovery, write behavior, logging, and metadata naming.
MySQL Source Connector Implementation and Tests integrations/lib/multiwoven/integrations/source/mysql/client.rb, integrations/lib/multiwoven/integrations/source/mysql/config/meta.json, integrations/lib/multiwoven/integrations/source/mysql/config/spec.json, integrations/spec/multiwoven/integrations/source/mysql/client_spec.rb	Implements source Client with check_connection, discover (information_schema → Catalog/Streams), and read (executes configured SQL, optional batch limit/offset → RecordMessage with emitted_at). Adds meta/spec JSON files and RSpec tests covering connection, read (including batching), discovery, exception routing, and internal method presence.
Integration Registration and Rollout integrations/lib/multiwoven/integrations.rb, integrations/lib/multiwoven/integrations/rollout.rb, server/Gemfile	Registers MySQL source and destination clients via require_relative in the integration loader, adds "Mysql" to ENABLED_SOURCES and ENABLED_DESTINATIONS, bumps rollout version from 0.35.0 to 0.38.0, and updates multiwoven-integrations gem dependency from ~> 0.34.6 to ~> 0.38.0.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

• Multiwoven/multiwoven#785: Both PRs modify server/Gemfile to update the multiwoven-integrations gem version constraint.
• Multiwoven/multiwoven#797: Both PRs update the integration rollout configuration to enable new connectors and bump the rollout VERSION.
• Multiwoven/multiwoven#806: Both PRs modify server/Gemfile to bump the multiwoven-integrations gem version.

Suggested reviewers

• TivonB-AI2
• Shibayan95

Poem

🐰 In rows and schemas I hop and play,
I spy information_schema along the way,
I fetch and write with a carrot grin,
Tests tucked in my burrow, ready to spin,
Two connectors launched—let the data dance begin!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change—adding MySQL source and destination connectors to the CE version.
Description check	✅ Passed	The PR description covers the main objectives, includes related issue, type of change, testing details, and addresses most checklist items comprehensively.
Linked Issues check	✅ Passed	The PR successfully implements all objectives from issue #10: adds a MySQL connector for source use in open-source, provides built-in MySQL connectivity, and enables MySQL as a source option.
Out of Scope Changes check	✅ Passed	All changes are directly aligned with the PR objectives. The version bump, rollout.rb updates, and Gemfile dependency changes are necessary supporting changes for the connector implementation.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 7

🧹 Nitpick comments (1)

integrations/spec/multiwoven/integrations/source/mysql/client_spec.rb (1)

92-109: ⚡ Quick win

Assert that the SQL was actually batched in this example.

This test only toggles limit and offset; because fetch is stubbed without inspecting its argument, it still passes if batched_query is never applied. Please assert the query sent to Sequel contains the batch clause, or stub batched_query directly.

One way to make the expectation meaningful

- allow(sequel_client).to receive(:fetch).and_return(dataset)
+ expect(sequel_client)
+   .to receive(:fetch)
+   .with(a_string_matching(/LIMIT\s+100\s+OFFSET\s+1/i))
+   .and_return(dataset)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@integrations/spec/multiwoven/integrations/source/mysql/client_spec.rb` around
lines 92 - 109, The test currently stubs sequel_client.fetch without verifying
the SQL, so update it to assert the batched query is used: either set an
expectation on sequel_client to receive(:fetch).with(a_string_including("LIMIT
100").and(a_string_including("OFFSET 1"))) when client.read(s_config) is called,
or instead stub client.batched_query to return a known SQL string and assert
fetch is called with that string; reference the test's s_config, client.read,
client.batched_query, sequel_client.fetch and the limit/offset values (100/1)
when making the assertion.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@integrations/lib/multiwoven/integrations/destination/mysql/client.rb`:
- Around line 7-13: The Sequel Database opened by create_connection is never
closed causing pooled connections to accumulate; for check_connection (and
likewise discover and write) assign the result of create_connection to a local
variable (e.g. db = create_connection(connection_config)) and wrap the work in a
begin/rescue/ensure block where the ensure calls db.disconnect (or db.disconnect
if method name differs in your Sequel version) if db is present, preserving
existing rescue behavior for errors; update discover and write the same way so
every created DB is always disconnected in an ensure.
- Around line 17-20: The SQL in discover builds query by interpolating
connection_config[:database] into query (query variable in discover), which
allows injection; change this to use a parameterized/bound query or properly
escape the schema value instead of direct interpolation: remove
"#{connection_config[:database]}" from the query string and pass
connection_config[:database] as a bound parameter when executing the query (or
use the client/driver escape method if prepared statements aren’t available),
updating the call that executes query to supply the parameter so discovery only
queries the intended schema.
- Around line 48-62: The code is leaking full generated SQL via logger.debug and
log_request_response (see Multiwoven::Integrations::Core::QueryBuilder.perform
output and its use in logger.debug, log_request_response, and the exception
handling). Remove the raw query text from all logs and messages: stop
interpolating or passing the `query` string to logger.debug,
`log_request_response`, and `handle_exception` payloads; instead log
non-sensitive metadata such as `action`, `table_name`, `primary_key`, and
`sync_config.sync_id/sync_run_id`, or a deterministic fingerprint/hash of the
query if you need correlation, and ensure errors only include the safe metadata
and error message (no PII-containing SQL).

In `@integrations/lib/multiwoven/integrations/source/mysql/client.rb`:
- Around line 7-13: The Sequel::Database handles created (via create_connection)
in methods like check_connection (and the similar discover/read paths) are never
disconnected; wrap the connection allocation so the returned Sequel::Database is
disconnected after use (use an ensure block or a block-yielding helper) and call
db.disconnect (or db.disconnect_all) on the Sequel::Database instance; locate
create_connection and callers such as check_connection and ensure any
early-return/rescue paths also call disconnect to avoid leaking sockets/pools.
- Around line 75-88: The group_by_table method is incorrectly keying result by
index causing one entry per column; change it to key by table_name (use
table_name as the unique key), initialize result[table_name] when missing, set
result[table_name][:tablename] = table_name, and push each column_data into
result[table_name][:columns] (rather than replacing with a single-element array)
so all columns for the same table are aggregated under one entry; update
references to result[index] -> result[table_name] and ensure is_nullable
handling and column_data remain unchanged.
- Around line 17-19: The discovery SQL currently interpolates
connection_config[:database] into query (variable query) which allows SQL
injection; change to a parameterized query instead: build the SQL with a
placeholder and pass connection_config[:database] as a bound parameter to
query_execution (or use Sequel's bound variable API) after creating the
connection with create_connection, so query_execution receives the SQL and the
database name as a parameter rather than embedding connection_config[:database]
directly; update any callers of query_execution if necessary to accept bound
parameters.

In `@integrations/spec/multiwoven/integrations/destination/mysql/client_spec.rb`:
- Around line 82-85: The spec currently stubs Sequel#fetch with
and_return(dataset) so the block passed by client.discover is never executed;
change the stub for sequel_client.fetch to yield rows to the block (e.g., yield
dataset or an array/row object) so the block path inside discover is exercised
and the mapping of result rows is validated — update the stub on
sequel_client.fetch used before calling
client.discover(sync_config_json[:destination][:connection_specification]) to
use a block-yielding stub (reference: Sequel#fetch, discover, sequel_client,
dataset).

---

Nitpick comments:
In `@integrations/spec/multiwoven/integrations/source/mysql/client_spec.rb`:
- Around line 92-109: The test currently stubs sequel_client.fetch without
verifying the SQL, so update it to assert the batched query is used: either set
an expectation on sequel_client to
receive(:fetch).with(a_string_including("LIMIT
100").and(a_string_including("OFFSET 1"))) when client.read(s_config) is called,
or instead stub client.batched_query to return a known SQL string and assert
fetch is called with that string; reference the test's s_config, client.read,
client.batched_query, sequel_client.fetch and the limit/offset values (100/1)
when making the assertion.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5c4563f7-1b6b-405c-b976-9ebf402a792a

📥 Commits

Reviewing files that changed from the base of the PR and between b8798d5 and 617fdfb.

⛔ Files ignored due to path filters (2)

• integrations/lib/multiwoven/integrations/destination/mysql/icon.svg is excluded by !**/*.svg
• integrations/lib/multiwoven/integrations/source/mysql/icon.svg is excluded by !**/*.svg

📒 Files selected for processing (10)

• integrations/lib/multiwoven/integrations.rb
• integrations/lib/multiwoven/integrations/destination/mysql/client.rb
• integrations/lib/multiwoven/integrations/destination/mysql/config/meta.json
• integrations/lib/multiwoven/integrations/destination/mysql/config/spec.json
• integrations/lib/multiwoven/integrations/rollout.rb
• integrations/lib/multiwoven/integrations/source/mysql/client.rb
• integrations/lib/multiwoven/integrations/source/mysql/config/meta.json
• integrations/lib/multiwoven/integrations/source/mysql/config/spec.json
• integrations/spec/multiwoven/integrations/destination/mysql/client_spec.rb
• integrations/spec/multiwoven/integrations/source/mysql/client_spec.rb

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@integrations/spec/multiwoven/integrations/source/mysql/client_spec.rb`:
- Around line 129-149: The test for client.discover only asserts one column and
not that multiple columns from the same table are aggregated; update the spec to
validate all mocked columns/streams: call
client.discover(sync_config[:source][:connection_specification]) and then assert
message.catalog.streams length and that the stream for "test_table" includes
properties for "col1", "col2", and "col3" (or alternatively change the
integration contract to expect one stream per column and assert accordingly).
Focus assertions around the produced catalog/streams from client.discover and
ensure behavior aligns with the implementation in group_by_table (which sets
result[index][:columns] = [column_data]) so the test verifies whether columns
are aggregated into a single stream or emitted as multiple streams.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 95aad96a-6720-48a7-b523-9e069ac86496

📥 Commits

Reviewing files that changed from the base of the PR and between 617fdfb and e9bdeb3.

⛔ Files ignored due to path filters (2)

• integrations/lib/multiwoven/integrations/destination/mysql/icon.svg is excluded by !**/*.svg
• integrations/lib/multiwoven/integrations/source/mysql/icon.svg is excluded by !**/*.svg

📒 Files selected for processing (10)

• integrations/lib/multiwoven/integrations.rb
• integrations/lib/multiwoven/integrations/destination/mysql/client.rb
• integrations/lib/multiwoven/integrations/destination/mysql/config/meta.json
• integrations/lib/multiwoven/integrations/destination/mysql/config/spec.json
• integrations/lib/multiwoven/integrations/rollout.rb
• integrations/lib/multiwoven/integrations/source/mysql/client.rb
• integrations/lib/multiwoven/integrations/source/mysql/config/meta.json
• integrations/lib/multiwoven/integrations/source/mysql/config/spec.json
• integrations/spec/multiwoven/integrations/destination/mysql/client_spec.rb
• integrations/spec/multiwoven/integrations/source/mysql/client_spec.rb

✅ Files skipped from review due to trivial changes (2)

• integrations/lib/multiwoven/integrations/destination/mysql/config/spec.json
• integrations/lib/multiwoven/integrations/rollout.rb

🚧 Files skipped from review as they are similar to previous changes (7)

• integrations/lib/multiwoven/integrations/source/mysql/config/spec.json
• integrations/lib/multiwoven/integrations/source/mysql/config/meta.json
• integrations/lib/multiwoven/integrations/destination/mysql/config/meta.json
• integrations/lib/multiwoven/integrations.rb
• integrations/spec/multiwoven/integrations/destination/mysql/client_spec.rb
• integrations/lib/multiwoven/integrations/destination/mysql/client.rb
• integrations/lib/multiwoven/integrations/source/mysql/client.rb

[TivonB-AI2]

@dataengineergaurav Raise to 0.38.0. We have some other connector PRs that need to be merged before this one

[dataengineergaurav]

covered.

[TivonB-AI2]

@dataengineergaurav Can you post a video of a demo sync for both Source and Destination. Show the creation of the connector and model for source then do a sync. Preferably a sync where the MySQL Source Connector is upserting data to a different Destination from the MySQL. Same thing when doing the MySQL Destination Connector. Show the Destination has received the data as verification of success.

[dataengineergaurav]

Addressed all CodeRabbit review items in the latest commit (dc56d0f8). Here's what was changed:

🔌 Connection leak (source + destination)
Added db&.disconnect in ensure blocks on check_connection, discover, read, and write. Every Sequel connection is now properly closed after use.

🛡️ SQL injection (source + destination)
Replaced '#{connection_config[:database]}' string interpolation with a ? bound parameter in both discover queries.

🔒 PII logging (destination)
Removed raw generated SQL from logger.debug and log_request_response. Logs now contain only safe metadata: action, table_name, primary_key, sync_id, sync_run_id.

♻️ Destination group_by_table simplified
Refactored to match the source pattern — uses table_name as the hash key directly, removing the unnecessary extra .group_by / .transform_values pass.

🧪 Specs updated
Added allow(sequel_client).to receive(:disconnect) stubs to both spec files to match the new ensure block behavior. Updated the destination discover test to use a Sequel::Dataset double with .all.

All 15 RSpec unit tests and 17 end-to-end integration tests pass.