fix: eliminate MySQL readiness race in mysql-influxdb e2e - #61
Merged
Conversation
The Tier-1 mysql-influxdb e2e was flaky/failing due to a MySQL TCP-readiness
race. On CI, MySQL always first-initializes from empty (the data volume is
commented out and e2e teardown runs `down -v`), so for ~12s it listens only on
its unix socket, not TCP 3306. The distributor eagerly opens a MySQL connection
during host startup (ServerVersion.AutoDetect via AddHostedService<StorageExporter>),
throws "Unable to connect to any of the specified MySQL hosts", exits, and
`restart: always` crash-loops it. run-e2e.sh's wait_for_port then gets a false
positive because docker-proxy binds the published port immediately, so the sender
fires into the crash-looping distributor and hits Connection refused.
Add healthchecks to mysql and influxdb and switch distributor/query to
`depends_on: { condition: service_healthy }`. The MySQL healthcheck pings over
`-h 127.0.0.1` to force TCP, so it only passes once the real server binds 3306
(not the socket-only first-init server). This holds distributor/query — and thus
the docker-proxy binding of 4317/5775 — until both backends truly accept
connections, eliminating the race.
Preserves the commented-out data volume, restart policy, ports, env and seed
wiring. Verified locally: `docker compose up -d` blocks until mysql/influxdb are
healthy, all containers stay at RestartCount=0 with no MySQL-connect error, and
both e2e cases pass (2/2).
Defense-in-depth for the e2e mysql-influxdb readiness race, on top of the compose healthcheck gating. Even once the distributor container is up, its gRPC endpoint can briefly report Unavailable (connection refused / not yet listening) while it finishes wiring up storage. Retry the initial trace/metric export on transient gRPC status (Unavailable / DeadlineExceeded) for a bounded 30s window before failing the case, so a healthy-but-slow-starting distributor does not flake the run. Non-transient failures and window exhaustion still surface the original RpcException. Scoped to the e2e sender only; no src/ production code changed.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #61 +/- ##
==========================================
- Coverage 71.26% 71.24% -0.03%
==========================================
Files 156 156
Lines 4023 4023
Branches 528 528
==========================================
- Hits 2867 2866 -1
Misses 991 991
- Partials 165 166 +1 🚀 New features to boost your workflow:
|
latte-gh
approved these changes
Jul 28, 2026
latte-gh
left a comment
latte-gh
left a comment
There was a problem hiding this comment.
LGTM. Two files, both e2e-only: (1) docker-compose-mysql-influxdb.yml adds proper healthchecks with service_healthy conditions for MySQL/InfluxDB, correctly using TCP ping (-h 127.0.0.1) to avoid false-positive readiness during MySQL socket-only init; (2) OtlpSender.cs adds bounded 30s startup retry for Unavailable/DeadlineExceeded as defense-in-depth. No src/ changes, both commits GPG-verified, CI already proves Tier1 mysql-influxdb passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Tier-1 full-stack
mysql-influxdbe2e leg fails on CI (flaky; sometimes passed). Root cause: on CI, MySQL first-initializes from an empty data dir and listens only on its unix socket (~12s) before binding TCP 3306. Thedistributoreagerly connects to MySQL during host startup (ServerVersion.AutoDetectvia theStorageExporterhosted service), throws, andrestart: alwayscrash-loops it.run-e2e.sh'swait_for_portgets a false-positive from docker-proxy (host port bound immediately), so the OTLP sender fires into the crash-looping distributor and getsConnection refused. The litedb leg has no external DB, so it never races; Tier-2 (Testcontainers) also passes — confirming the storage adapters are fine and the issue is full-stack compose readiness.Fix
docker/docker-compose-mysql-influxdb.yml: add healthchecks formysql(mysqladmin ping -h 127.0.0.1— forces TCP so it only passes once the real server binds 3306, not the socket-only first-init server) andinfluxdb(curl /health), and changedistributor/querytodepends_on: { condition: service_healthy }. App containers (and thus docker-proxy binding:4317/:5775) no longer start until the databases truly accept connections.e2e/Mocha.E2E.Abstractions/OtlpSender.cs: bounded 30s retry on the initial OTLP export for gRPCUnavailable/DeadlineExceeded(defense-in-depth for the startup window).No
src/production code changed.Verification
Locally (real docker stack, mysql-influxdb): databases reach healthy before app containers start; all 4 containers
RestartCount=0; distributor log clean (noUnable to connect to any of the specified MySQL hosts, noRestarting); e2e2/2 case(s) passed. This PR's CI run should show the E2Emysql-influxdbleg green.