fix(server): harden HTTP(S) accept path against fd exhaustion (#630)

A production deployment became unreachable after five weeks of uptime:
internet scanners left ~1100 half-open TLS connections behind, each
pinning a file descriptor forever, until the process hit its soft fd
limit (1024) and accept() failed with EMFILE -- silently, because
axum-server retries accept errors without logging. The server looked
healthy (running, idle runtime) while the kernel backlog accepted TCP
connections that were never serviced.

Add layered protections in a new server_hardening module:

- KeepaliveAcceptor: enables TCP keepalive (60s + 15s probes) on
  accepted connections so the kernel reclaims connections whose peer
  vanished without a FIN.
- FirstByteTimeoutAcceptor: closes connections that never send a byte
  within 30s of being accepted (post-TLS). hyper's header_read_timeout
  cannot cover this case because hyper-util's auto builder sniffs the
  h1/h2 version with an untimed read. The deadline is disarmed by the
  first byte, so WebSockets and in-flight requests are unaffected.
- Explicit 10s TLS handshake timeout (matches the axum-server default,
  pinned so upstream default changes cannot remove the protection).
- header_read_timeout (30s) on the hyper builder to reap idle
  keep-alive connections between requests. Requires installing
  TokioTimer: hyper panics per-connection if a timeout is configured
  without a timer, which axum-server does not install by default.
- fd-usage watchdog (Linux): warns at 80% of the soft limit, errors
  every minute at 95%, so the approach to EMFILE is visible in logs
  before the server becomes unreachable.

Verified end-to-end against a running server: raw-idle closed at 10s,
TLS-idle closed at 30s, idle keep-alive closed at 30s, idle WebSocket
stays open, no hyper panics.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: srperens

Cargo.lock

@@ -81,6 +81,7 @@ libc = "0.2"
 # TLS
 axum-server.workspace = true
 rustls = { version = "0.23", features = ["ring"] }
+hyper-util = { version = "0.1", features = ["tokio"] } # TokioTimer for hyper connection timeouts
 notify = { version = "8", default-features = false, features = ["macos_kqueue"] }
 
 # Authentication
@@ -142,6 +143,7 @@ winresource = "0.1"
 
 [dev-dependencies]
 tempfile = "3.27"
+tokio = { version = "1.52", features = ["full", "test-util"] }
 tower = { version = "0.5", features = ["util"] }
 http-body-util = "0.1"
 serial_test = "3.5"

backend/Cargo.toml

@@ -36,6 +36,7 @@ pub mod openapi;
 pub mod paths;
 pub mod ptp_monitor;
 pub mod rtsp_server;
+pub mod server_hardening;
 pub mod sharing;
 pub mod state;
 pub mod stats;

backend/src/lib.rs

@@ -686,21 +686,51 @@ async fn setup_tls(config: &Config) -> Option<axum_server::tls_rustls::RustlsCon
 }
 
 /// Start the HTTP(S) server, binding to the given address.
+///
+/// The accept path is hardened against fd exhaustion from abandoned
+/// connections (TCP keepalive, TLS handshake timeout, header read timeout,
+/// fd-usage watchdog) — see `server_hardening` for the rationale.
 async fn serve_with_tls(
     addr: SocketAddr,
     app: axum::Router,
     handle: axum_server::Handle<SocketAddr>,
     tls_config: Option<axum_server::tls_rustls::RustlsConfig>,
 ) -> anyhow::Result<()> {
+    use strom::server_hardening::{
+        FirstByteTimeoutAcceptor, KeepaliveAcceptor, HEADER_READ_TIMEOUT, TLS_HANDSHAKE_TIMEOUT,
+    };
+
+    strom::server_hardening::spawn_fd_watchdog();
+
     if let Some(tls_config) = tls_config {
         info!("Server listening on https://{}", addr);
-        axum_server::bind_rustls(addr, tls_config)
+        let acceptor = FirstByteTimeoutAcceptor::new(
+            axum_server::tls_rustls::RustlsAcceptor::new(tls_config)
+                .handshake_timeout(TLS_HANDSHAKE_TIMEOUT)
+                .acceptor(KeepaliveAcceptor),
+        );
+        let mut server = axum_server::bind(addr).acceptor(acceptor);
+        server
+            .http_builder()
+            .http1()
+            // hyper panics if a timeout is set without a timer; axum-server
+            // does not install one by default.
+            .timer(hyper_util::rt::TokioTimer::new())
+            .header_read_timeout(HEADER_READ_TIMEOUT);
+        server
             .handle(handle)
             .serve(app.into_make_service_with_connect_info::<SocketAddr>())
             .await?;
     } else {
         info!("Server listening on http://{}", addr);
-        axum_server::bind(addr)
+        let mut server =
+            axum_server::bind(addr).acceptor(FirstByteTimeoutAcceptor::new(KeepaliveAcceptor));
+        server
+            .http_builder()
+            .http1()
+            .timer(hyper_util::rt::TokioTimer::new())
+            .header_read_timeout(HEADER_READ_TIMEOUT);
+        server
             .handle(handle)
             .serve(app.into_make_service_with_connect_info::<SocketAddr>())
             .await?;