Commit a9fc64f
feat: implement Kubernetes health endpoints with Quarkus format compatibility
Add comprehensive health check endpoints for Kubernetes readiness and liveness
probes, achieving full parity with Java Quarkus SmallRye Health format.
This fixes critical pod restart issues where the TrustyAI Operator expects
Quarkus-compatible health endpoints that were not previously implemented.
Features:
- /q/health/ready: Readiness probe with storage backend checks
- /q/health/live: Liveness probe for application health
- /q/health: Combined health endpoint for debugging
- Quarkus SmallRye Health format: {"status": "UP|DOWN", "checks": [...]}
- HTTP 200 for UP, 503 for DOWN status codes
Health Checks:
- PVC storage: Verifies mount exists and is writable (cached)
- MariaDB storage: Tests database connectivity (cached)
- HTTP server: Confirms server is responding
- Application: Basic liveness check
Performance Optimizations:
- TTL-based caching (configurable via HEALTH_CACHE_TTL, default 5s)
- Thread-safe cache with statistics tracking (hits/misses)
- 33% reduction in I/O operations (4 ops/min vs 6 ops/min)
- Fast database timeout (2s) prevents probe blocking
Code Quality:
- Specific exception handling (mariadb.Error, OSError, TimeoutError)
- Proper logging levels (warning for expected failures, exception for bugs)
- Storage format standardization (canonical "MARIA" value)
- Comprehensive test coverage (27 tests, all passing)
Security:
- Production mode (ENVIRONMENT=production) redacts paths from errors
- Generic error messages in production, detailed in development
- No sensitive data in health responses
Bug Fixes:
- Fixed Hypercorn port binding when TLS unavailable (critical)
Previously: config.bind defaulted to port 8000, ignoring HTTP_PORT
Now: Correctly uses HTTP_PORT for HTTP-only mode
Environment Variables:
- HEALTH_CACHE_TTL: Cache TTL in seconds (default: 5)
- ENVIRONMENT: Set to "production" for security features
Tests:
- 27 comprehensive tests covering all scenarios
- Cache behavior (store, expire, multiple keys, statistics)
- PVC storage (success, missing path, not writable, production mode)
- MariaDB (success, failure, network error, library missing)
- HTTP endpoints (200 OK, 503 Service Unavailable)
- All linting, formatting, and type checking passed
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Sudip Sinha <Sudip.Sinha@RedHat.com>1 parent c1733a3 commit a9fc64f
5 files changed
Lines changed: 869 additions & 14 deletions
File tree
- src
- service
- tests
- service
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
109 | 109 | | |
110 | 110 | | |
111 | 111 | | |
112 | | - | |
| 112 | + | |
113 | 113 | | |
114 | 114 | | |
115 | 115 | | |
116 | 116 | | |
117 | 117 | | |
118 | 118 | | |
| 119 | + | |
119 | 120 | | |
120 | 121 | | |
121 | 122 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
39 | 44 | | |
40 | 45 | | |
41 | 46 | | |
| |||
184 | 189 | | |
185 | 190 | | |
186 | 191 | | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
187 | 220 | | |
188 | 221 | | |
189 | 222 | | |
| |||
199 | 232 | | |
200 | 233 | | |
201 | 234 | | |
202 | | - | |
| 235 | + | |
| 236 | + | |
203 | 237 | | |
204 | | - | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
205 | 245 | | |
206 | 246 | | |
207 | 247 | | |
208 | 248 | | |
209 | 249 | | |
210 | 250 | | |
211 | 251 | | |
212 | | - | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
213 | 256 | | |
214 | | - | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
215 | 264 | | |
216 | 265 | | |
217 | 266 | | |
| |||
253 | 302 | | |
254 | 303 | | |
255 | 304 | | |
256 | | - | |
257 | | - | |
258 | | - | |
259 | | - | |
260 | 305 | | |
261 | 306 | | |
262 | 307 | | |
263 | 308 | | |
264 | 309 | | |
265 | 310 | | |
| 311 | + | |
266 | 312 | | |
267 | 313 | | |
268 | 314 | | |
| 315 | + | |
| 316 | + | |
269 | 317 | | |
| 318 | + | |
270 | 319 | | |
271 | 320 | | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
272 | 324 | | |
273 | 325 | | |
274 | 326 | | |
| |||
0 commit comments