Skip to content

[KVEvents] add dp rank to KV events - #2379

Open
sagearc wants to merge 1 commit into
llm-d:mainfrom
sagearc:add-dp-rank-to-kvevents
Open

[KVEvents] add dp rank to KV events#2379
sagearc wants to merge 1 commit into
llm-d:mainfrom
sagearc:add-dp-rank-to-kvevents

Conversation

@sagearc

@sagearc sagearc commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

atm llm-d-router reads dp rank from emitted kv event batches but drops it during adapter conversion.
kv events are instead attributed to the source ip:port attached to each subscriber. this is only sufficient for external load balancing where each dp rank exposes standalone frontend with a distinct port.

with internal load balancing, multiple dp ranks can share one http endpoint, so the endpoint alone cannot identify which rank owns the cached blocks.

refs:

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
Copilot AI lite review requested due to automatic review settings August 13, 2026 16:35
@github-actions github-actions Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Aug 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR preserves the data-parallel (DP) rank from incoming KV event batches by threading it through the engine adapters into kvevents.EventBatch, enabling downstream components to correctly attribute KV events when multiple ranks share a single HTTP endpoint.

Changes:

  • Add optional DataParallelRank to kvevents.EventBatch.
  • Propagate DP rank during msgpack batch decoding in both vLLM and SGLang engine adapters.
  • Consolidate the shared msgpack batch struct into engineadapter/common.go and extend adapter tests to assert DP rank parsing.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file
File Description
pkg/kvevents/events.go Adds DataParallelRank to the public EventBatch type.
pkg/kvevents/engineadapter/common.go Introduces shared msgpackEventBatch with optional DP rank field.
pkg/kvevents/engineadapter/vllm_adapter.go Decodes DP rank from batch and sets it on kvevents.EventBatch.
pkg/kvevents/engineadapter/vllm_adapter_test.go Updates test payload and asserts DP rank is preserved.
pkg/kvevents/engineadapter/sglang_adapter.go Decodes DP rank from batch and sets it on kvevents.EventBatch.
pkg/kvevents/engineadapter/sglang_adapter_test.go Updates test payload and asserts DP rank is preserved.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants