Merge branch 'main' into fix/fc-saturation-staleness

Author: LukeAVanDrie

.github/workflows/pr-hold-gate.yml

@@ -10,13 +10,24 @@ on:
 
 permissions:
   contents: read
+  pull-requests: read
 
 jobs:
   hold:
     runs-on: ubuntu-latest
     steps:
       - name: Block merge while held
-        if: contains(github.event.pull_request.labels.*.name, 'hold') || contains(github.event.pull_request.labels.*.name, 'do-not-merge/hold')
+        env:
+          GH_TOKEN: ${{ github.token }}
+          PR_URL: ${{ github.event.pull_request.html_url }}
         run: |
-          echo "::error::A hold label is present (hold or do-not-merge/hold). Remove it (/hold cancel) to allow merge."
-          exit 1
+          blocking_labels=(hold do-not-merge/hold)
+
+          # fetch current labels
+          labels=$(gh pr view "$PR_URL" --json labels -q '.labels[].name')
+          for label in "${blocking_labels[@]}"; do
+            if grep -qxF "$label" <<< "$labels"; then
+              echo "::error::Blocking label '$label' is present. Remove it to allow merge (e.g., /hold cancel for hold)."
+              exit 1
+            fi
+          done

.github/workflows/prow-github.yml

@@ -2,62 +2,21 @@
 
 A Go service that orchestrates multi-phase LLM inference pipelines (Encode/Prefill/Decode) across specialized worker pools. It exposes OpenAI-compatible APIs and routes requests through an Inference Gateway to disaggregated vLLM workers.
 
+For the architecture, request lifecycle, EPP integration, and plugin API, see [docs/coordinator_architecture.md](docs/coordinator_architecture.md). For the exact per-stage wire formats, see [docs/communication.md](docs/communication.md).
+
 ## Table of Contents
 
 - [LLM-D Coordinator](#llm-d-coordinator)
   - [Table of Contents](#table-of-contents)
-  - [Architecture](#architecture)
   - [Quick Start](#quick-start)
   - [Configuration](#configuration)
-    - [Server](#server)
-    - [Gateway](#gateway)
-    - [Pipeline](#pipeline)
-    - [Built-in Step Parameters](#built-in-step-parameters)
-    - [Gateway Routing](#gateway-routing)
-  - [Plugin API](#plugin-api)
-    - [Step Interface](#step-interface)
-    - [Writing a Custom Step](#writing-a-custom-step)
-    - [Registering the Step](#registering-the-step)
-    - [Dependency Injection](#dependency-injection)
-    - [RequestContext](#requestcontext)
   - [API Endpoints](#api-endpoints)
   - [Docker](#docker)
   - [Development](#development)
     - [Running Tests](#running-tests)
       - [Unit Tests](#unit-tests)
       - [End-to-End Tests](#end-to-end-tests)
 
-## Architecture
-
-```
-+--------+
-| Client |
-+--------+
-    |
-    v
-+-------------------+       +-------------+
-| Inference Gateway | <---> | Coordinator |
-+-------------------+       +-------------+
-    |
-    v
-+-----+
-| EPP |
-+-----+
-    |
-    v
-+--------------+
-| vLLM Workers |
-+--------------+
-```
-
-The Coordinator processes each request through a configurable pipeline of steps:
-
-1. **replace-media-urls** - Downloads image URLs and inlines them as base64
-2. **render** - Sends request to an external rendering/tokenization service
-3. **encode** - Parallel fan-out: one encode request per multimodal entry
-4. **prefill** - Combines encode results, sends to prefill worker
-5. **decode** - Forwards the final request to decode worker, streams response back
-
 ## Quick Start
 
 The coordinator targets live in `Makefile.coord.mk`, which the root `Makefile`
@@ -79,188 +38,7 @@ make -f Makefile.coord.mk test
 
 ## Configuration
 
-Configuration is a YAML file passed via the `--config` flag. See `config/coordinator/coordinator.yaml` for the default.
-
-### Server
-
-```yaml
-server:
-  listen_addr: ":8080"        # Address to listen on
-  read_timeout: 30s           # HTTP read timeout
-  write_timeout: 120s         # HTTP write timeout (long for streaming)
-```
-
-### Gateway
-
-Connection settings for the Inference Gateway that routes to vLLM worker pools:
-
-```yaml
-gateway:
-  address: "http://inference-gateway:80"
-  max_idle_conns_per_host: 100   # Connection pool size
-  idle_conn_timeout: 90s
-  timeout: 60s                   # Per-request timeout
-```
-
-The rendering service address is not a top-level setting; it is the `address` parameter of the `render` pipeline step (see below).
-
-### Pipeline
-
-The pipeline is an ordered list of steps. Each step has a `type` (registered name) and optional `params`:
-
-```yaml
-pipeline:
-  steps:
-    - type: replace-media-urls
-      params:
-        download_timeout: 10s
-        max_concurrent_downloads: 10
-    - type: render
-      params:
-        address: "http://rendering-service:8080"
-    - type: encode
-      params:
-        max_parallel: 8
-    - type: prefill
-    - type: decode
-```
-
-To remove a step, delete it from the list. To reorder, move entries up or down. Steps execute sequentially in the order listed.
-
-### Built-in Step Parameters
-
-| Step | Parameter | Default | Description |
-|------|-----------|---------|-------------|
-| replace-media-urls | `download_timeout` | `10s` | Timeout for each image download |
-| replace-media-urls | `max_concurrent_downloads` | `10` | Max parallel downloads |
-| render | `address` | (required) | Base URL of the rendering service |
-| render | `timeout` | `30s` | Timeout for a single render call |
-| render | `max_total_tokens` | `0` (unlimited) | Reject requests whose tokenized prompt exceeds this |
-| render | `max_total_placeholder_tokens` | `0` (unlimited) | Reject requests whose summed image-placeholder length exceeds this |
-| encode | `max_parallel` | `8` | Max parallel encode requests |
-
-### Gateway Routing
-
-The coordinator sends every sub-request to the same gateway address. It does not use phase-specific URL prefixes; instead it stamps an `EPP-Phase` header (`encode`, `prefill`, or `decode`) so the Endpoint Picker can route to the correct worker pool. The request path is chosen by the request format:
-
-| Phase | Header | Path |
-|-------|--------|------|
-| Encode | `EPP-Phase: encode` | `/v1/completions` for completions requests; otherwise `/inference/v1/generate`, or `/v1/chat/completions` when `use_openai_format` is set |
-| Prefill | `EPP-Phase: prefill` | same as encode |
-| Decode | `EPP-Phase: decode` | original client request path (`/v1/chat/completions` or `/v1/completions`) |
-
-The decode step preserves the original client request path so the gateway can route it to the correct OpenAI-compatible endpoint on the decode worker.
-
-## Plugin API
-
-Custom pipeline steps can be added by implementing the `Step` interface and registering a factory function.
-
-### Step Interface
-
-```go
-package pipeline
-
-type Step interface {
-    Name() string
-    Execute(ctx context.Context, reqCtx *RequestContext) error
-}
-```
-
-### Writing a Custom Step
-
-```go
-package mystep
-
-import (
-    "context"
-    "github.com/llm-d/llm-d-router/pkg/coordinator/pipeline"
-)
-
-func init() {
-    pipeline.Register("my-step", NewMyStep)
-}
-
-type MyStep struct {
-    someParam string
-}
-
-func NewMyStep(params map[string]any) (pipeline.Step, error) {
-    s := &MyStep{someParam: "default"}
-    if v, ok := params["some_param"].(string); ok {
-        s.someParam = v
-    }
-    return s, nil
-}
-
-func (s *MyStep) Name() string { return "my-step" }
-
-func (s *MyStep) Execute(ctx context.Context, reqCtx *pipeline.RequestContext) error {
-    // Access and modify the request context:
-    // - reqCtx.Body              (parsed JSON body, mutable)
-    // - reqCtx.TokenIDs          (token IDs from the render step)
-    // - reqCtx.MultimodalEntries (multimodal content)
-    // - reqCtx.ECTransferParams  (encoder cache transfer params, per encode response)
-    // - reqCtx.KVTransferParams  (KV cache transfer params)
-    // - reqCtx.Model             (model name)
-    // - reqCtx.Stream            (whether client requested streaming)
-    //
-    // Return nil to continue, or an error to abort the pipeline.
-    return nil
-}
-```
-
-### Registering the Step
-
-Import your step package in `cmd/coordinator/main.go`:
-
-```go
-import _ "github.com/llm-d/llm-d-router/pkg/coordinator/steps/mystep"
-```
-
-Then add it to the pipeline config:
-
-```yaml
-pipeline:
-  steps:
-    - type: my-step
-      params:
-        some_param: "value"
-    - type: decode
-```
-
-### Dependency Injection
-
-A step that needs the shared gateway HTTP client implements `gateway.ClientAware`. After building each step, the coordinator type-asserts it against this interface and calls `SetGatewayClient` when it matches:
-
-```go
-// gateway.ClientAware receives the shared gateway HTTP client.
-type ClientAware interface {
-    SetGatewayClient(*Client)
-}
-```
-
-Step parameters from the YAML `params` map are the mechanism for everything else. For example, the render step reads its service address from `params.address` in its factory rather than through an injected interface. The render step does expose a `SetServiceAddress` method, but it is used only by tests to point the step at a local server and is not called in production.
-
-### RequestContext
-
-The `RequestContext` is the shared state passed between steps:
-
-```go
-type RequestContext struct {
-    RequestID         string              // Unique request ID
-    OriginalPath      string              // Client request path (e.g., /v1/chat/completions)
-    OriginalHeaders   http.Header         // Inbound request headers (forwarded upstream, minus hop-by-hop)
-    OriginalBody      []byte              // Raw request body
-    Body              map[string]any      // Parsed/mutable JSON body
-    Model             string              // Model name
-    Stream            bool                // SSE streaming requested
-    TokenIDs          []int               // Token IDs from the render step
-    MultimodalEntries []MultimodalEntry   // Downloaded multimodal content
-    ECTransferParams  []map[string]any    // Encode results, one entry per encode response (mm_hash -> descriptor)
-    KVTransferParams  map[string]any      // Prefill KV-cache transfer hints, consumed by the KV connector at decode
-    ResponseWriter    http.ResponseWriter // Client response writer; decode steps stream the final response to it
-}
-```
+Configuration is a YAML file passed via the `--config` flag. See `config/coordinator/coordinator.yaml` for the annotated default, and [Configuring the pipeline](docs/coordinator_architecture.md#configuring-the-pipeline) for the full reference (top-level structure, environment overrides, connector selection, and the built-in steps).
 
 ## API Endpoints
 

.prowlabels.yaml

@@ -0,0 +1,2 @@
+# Security
+Please see the [llm-d Security Policy](https://github.com/llm-d/llm-d/blob/main/SECURITY.md) in the main repository for vulnerability reporting and disclosure information.

README.coord.md

@@ -111,6 +111,7 @@ import (
 	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/scheduling/filter/prefixcacheaffinity"
 	sessionaffinityfilter "github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/scheduling/filter/sessionaffinity"
 	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/scheduling/filter/sloheadroomtier"
+	utilizationfilter "github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/scheduling/filter/utilization"
 	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/scheduling/picker/maxscore"
 	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/scheduling/picker/random"
 	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/scheduling/picker/weightedrandom"
@@ -558,6 +559,7 @@ func (r *Runner) registerInTreePlugins() {
 	fwkplugin.Register(bylabel.PrefillRoleType, bylabel.PrefillRoleFactory)
 	fwkplugin.Register(endpointattributefilter.EndpointAttributeFilterType, endpointattributefilter.EndpointAttributeFilterFactory)
 	fwkplugin.Register(sessionaffinityfilter.SessionAffinityType, sessionaffinityfilter.Factory)
+	fwkplugin.Register(utilizationfilter.UtilizationFilterType, utilizationfilter.Factory)
 
 	// dataparallel profile handler
 	fwkplugin.Register(dataparallel.DataParallelProfileHandlerType, dataparallel.ProfileHandlerFactory)
@@ -911,7 +913,7 @@ func (r *Runner) initAdmissionControl(
 			UsageLimitPolicy:   eppConfig.FlowControlConfig.UsageLimitPolicy,
 		},
 	)
-	return endpointCandidates, requestcontrol.NewFlowControlAdmissionController(fc, opts.PoolName), registry
+	return endpointCandidates, requestcontrol.NewFlowControlAdmissionController(fc, opts.PoolName, endpointCandidates), registry
 }
 
 // runWithFileDiscovery handles the execution path when a discovery plugin is configured.

SECURITY.md

@@ -1,9 +1,14 @@
 # EPP configuration for the probabilistic-admitter plugin.
 #
-# The probabilistic-admitter handles all saturation-based admission decisions,
-# so the system-level utilization-detector is configured with effectively
-# infinite thresholds to prevent it from gating requests before the admitter
-# can evaluate them.
+# The probabilistic-admitter handles all saturation-based admission decisions
+# at the admission extension point. To let it own admission exclusively, the
+# system-level utilization-detector is configured with effectively infinite
+# thresholds (queueDepthThreshold: 999999999, kvCacheUtilThreshold: 1.0).
+# This prevents the flow control dispatch gate from queueing requests before
+# the admitter can evaluate them.
+#
+# Note: If these utilization-detector thresholds are raised, requests will
+# undergo flow control queueing before reaching the probabilistic-admitter.
 apiVersion: inference.networking.x-k8s.io/v1alpha1
 kind: EndpointPickerConfig
 plugins: