[Feature]: Metrics

[shimib]

Problem Statement

Currently no comprehensive metrics provided

Proposed Solution

Implementation status legend: ✅ implemented · 🔄 in review · ⬜ not yet implemented. All implemented metrics live under the llm_d_async subsystem and carry queue_id, queue_name, and pool_name labels.

Traffic Layer - Queue Health & Backlog

These metrics tell users how much work is pending and if they are meeting their SLOs (minutes/hours).

• ✅ Backlog Size (Gauge): Total items in each Redis SortedSet. (not easy with PubSub, requires external guide how to integrate) — implemented as async_broker_backlog (feat: add queue depth, in-flight, and broker backlog metrics #244).
  Why: Essential for auto-scaling the processor and understanding system pressure.
  Note: For GCP PubSub this requires some setup to get this metric from external endpoint as PubSub does not provide this through client API.
• ⬜ Deadline Proximity (Histogram): The distribution of deadline - now() for items in the queue (requires pre-processing).
  Why: Helps identify if a large batch of work is about to "expire" simultaneously.
  Note: Will require a pre-processing stage as we don’t look at the messages before consumption currently. Feasible for redis-sortedset only (via the periodic backlog poller + ZRANGEBYSCORE); not feasible for PubSub (no peek).
• ✅ Queue Residence Time (Histogram): Time from message ingestion to being pulled by a worker. — implemented as async_queue_residence_time_millis (feat: add queue residence time metric #254).
  Why: Measures the actual "async delay" introduced by the system.

Intelligence Layer - Dispatch Gates & Throttling

Since llm-d-async uses gates (Prometheus, Redis, Quota) to protect the inference pool, users need to know why processing is slowing down.

• 🔄 Effective Dispatch Budget (Gauge): The value [0.0 - 1.0] currently returned by the gate for each queue. — async_dispatch_budget (PR feat(metrics): add dispatch budget and pool worker-limit gauges (#217) #290, in review).
• 🔄 Gate Decision Breakdown (Counter): Why was a message not pulled?
  Labels: reason="gate_closed", reason="quota_exhausted", reason="no_budget". — async_gate_decisions_total (PR feat(metrics): add gate-decision breakdown counter (#217) #291, in review).
• ⬜ Saturation Metric Value (Gauge): If using prometheus-saturation, the raw value being compared against the threshold.
  Why: Helps debug why a gate is closed (e.g., "The pool is at 85% saturation, threshold is 80%").

Execution Layer - Worker Performance & Utilization

• 🔄 Worker Concurrency Utilization (Gauge): Number of active workers vs. the concurrency limit. In-flight work is exposed via async_inflight_requests / async_queue_depth (feat: add queue depth, in-flight, and broker backlog metrics #244); the per-pool concurrency limit async_pool_worker_limit is added in PR feat(metrics): add dispatch budget and pool worker-limit gauges (#217) #290 (in review), so active-vs-limit utilization is now derivable (async_inflight_requests / async_pool_worker_limit).
• ✅ Inference Request Latency (Histogram): Time spent calling the Inference Gateway (IGW). — implemented as async_inference_latency_time_millis.
  Why: Separates "model time" from "queue time."
• ✅ Shedding vs. Failure (Counter): — implemented as async_shedded_requests_total and async_failed_requests_total.
  Shedded: 429s/Busy signals that trigger a retry.
  Failed: 4xx/5xx or validation errors that end the request.

LLM-Specific Throughput

• ⬜ Token Throughput (Counter): Total tokens (input + output) processed successfully.
  Why: The ultimate unit of value for LLM workloads. Even if the processor doesn't parse every payload, it can track this from the IGW response metadata.

Alternatives Considered

No response

Willingness to Contribute

Yes, I can submit a PR

Additional Context

No response