Redis-Synchronized Time for Distributed Circuit Breaker Coordination

[bolshakov]

Implement Redis as the authoritative time source for all circuit breaker operations, eliminating time skew issues across multi-instance deployments. This allows Stoplight to reliably coordinate state transitions and metrics aggregation in distributed environments.

Problem

In multi-instance deployments, clock skew between instances can cause:

• Incorrect state transitions -- out-of-order state changes due to instance clock differences
• Metrics corruption -- last_error_at / last_success_at timestamps can go backwards
• Recovery timing issues -- cool-off periods and recovery probes misfire with skewed clocks

The circuit breaker depends on reliable timestamps for its entire state machine and metrics, making time synchronization critical.

Solution

Make Redis the authoritative time source via a now() Lua function that calls redis.call("TIME"):

• State transitions use now() -- no Ruby timestamps passed to Lua
• Metrics use now() -- unbounded_metrics and window_metrics synchronized
• Remove out-of-order protections -- impossible with monotonic Redis time
• Simplified Lua logic -- cleaner, more maintainable scripts

Architecture

Time flow in distributed deployment:

Instance 1 ──┐
Instance 2 ──┤─→ [Redis Instance] ─→ now() = 1669333004321 (authoritative)
Instance 3 ──┘

All instances call the same now() function, eliminating clock skew.

[github-actions]

Review price tag

🟢 195 effective lines — about 23–59 min (down from 29–72 min) of focused review (based on 200–500 lines/hour). This is within the range where reviewers find the most issues per line, and small changes usually receive feedback the fastest.

Spread across many files. This PR changes 16 files — more than about 9 in 10 PRs touch. Each extra file is another piece of context a reviewer has to load and hold at once.

Why these numbers?

These minutes are what careful defect-finding costs at 200–500 lines/hour — the rate review studies report, not how long a skim takes. "Effective lines" already exclude generated files and lockfiles. Treat the rates and the 200/400 thresholds as guardrails, not laws.

[bolshakov]

no time traveling in feature tests anymore