fix(migrations): serialize datetime as ISO string for new drivers - #8170
Closed
mjq wants to merge 6 commits into
Closed
fix(migrations): serialize datetime as ISO string for new drivers#8170mjq wants to merge 6 commits into
mjq wants to merge 6 commits into
Conversation
… drivers The migration runner passes datetime.now() as a parameter in INSERT FORMAT JSONEachRow queries to record migration status. The native ClickHouse driver handles datetime serialization via its binary protocol, but the clickhouse-connect HTTP driver passes parameters through json.dumps which cannot serialize Python datetime objects. This causes all migrations to fail with 'TypeError: Object of type datetime is not JSON serializable' in regions where the use_clickhouse_connect_driver flag is enabled. Convert to .isoformat() which is JSON-serializable and parsed correctly by ClickHouse's DateTime type. Fixes SENTRY-5RCC
mjq
marked this pull request as ready for review
July 9, 2026 19:10
mjq
marked this pull request as draft
July 9, 2026 19:14
…ization The clickhouse-connect HTTP driver serializes query parameters via json.dumps, which raises on bare Python datetime objects. The native driver handles them transparently via its binary protocol, so callers (e.g. the migration runner) pass datetime values directly. Add a _make_json_safe shim in ClickhouseConnectPool that converts datetime objects to ISO format strings before they reach the clickhouse-connect binding layer. This keeps the existing contract (callers can pass datetime objects) working on both driver paths. Reverts the runner.py change from the previous commit — the fix belongs in the driver layer, not in every caller. Fixes SENTRY-5RCC
…driver Reproduces the production failure from SENTRY-5RCC: the migration runner passes datetime.now() in an INSERT FORMAT JSONEachRow query, which the clickhouse-connect HTTP driver routes through json.dumps. Without the _make_json_safe fix, this raises TypeError.
The native clickhouse-driver treats INSERT … FORMAT JSONEachRow with a data list specially, serializing the rows as JSON lines in the request body. clickhouse-connect's query() instead interprets the parameters as %-style bind params, which fails with 'not all arguments converted during string formatting'. Detect the JSONEachRow INSERT pattern in _execute_once and inline the data rows (after datetime sanitization) directly into the query string. Regular queries with bind params are unaffected.
Iterate params directly (already narrowed to Sequence by isinstance) and apply _make_json_safe per-row, avoiding an intermediate variable whose Params union type mypy cannot narrow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The migration runner passes datetime.now() as a parameter in INSERT FORMAT JSONEachRow queries to record migration status. The native ClickHouse driver handles datetime serialization via its binary protocol, but the clickhouse-connect HTTP driver passes parameters through json.dumps which cannot serialize Python datetime objects.
This causes all migrations to fail with 'TypeError: Object of type datetime is not JSON serializable' in regions where the use_clickhouse_connect_driver flag is enabled.
Convert to .isoformat() which is JSON-serializable and parsed correctly by ClickHouse's DateTime type.
Fixes SENTRY-5RCC
🤖: Claude diagnosed and fixed this.