Skip to content

feat: add configurable routing profiles to llm-d generator - #281

Open
amito wants to merge 1 commit into
llm-d-incubation:mainfrom
amito:feat/routing-profiles
Open

feat: add configurable routing profiles to llm-d generator#281
amito wants to merge 1 commit into
llm-d-incubation:mainfrom
amito:feat/routing-profiles

Conversation

@amito

@amito amito commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Description

PR #274 introduced minimal llm-d manifest generation with hardcoded default EPP routing plugins. This PR makes the routing configuration selectable via named profiles, end-to-end from UI to generated Helm values.

Three routing profiles are available:

  • default - Current behavior (prefix-cache-scorer, decode-filter, max-score-picker, single-profile-handler). Backward-compatible: existing API calls produce identical output.
  • session-affinity - Adds session-affinity-scorer (weight 3) for conversational/chat workloads where requests from the same session benefit from being routed to the same replica.
  • throughput-optimized - Replaces prefix-cache-scorer with load-aware-scorer (weight 3) for batch/throughput-oriented workloads where even load distribution matters more than cache locality.

Changes

  • src/planner/configuration/llmd_generator.py - Added ROUTING_PROFILES dict with TypedDict annotations defining plugin configurations for each profile. generate_all() now accepts an optional routing_profile parameter (default: "default"), validates it, and injects the profile's plugins into the template context.
  • src/planner/configuration/templates/llmd/values.yaml.j2 - Replaced hardcoded plugin list with Jinja2 loops over plugins and scheduling_plugins template variables. Fixed weight check to use is not none (handles weight: 0 correctly).
  • src/planner/api/routes/configuration.py - Added routing_profile field to DeploymentRequest with Literal type validation. Passed through to llmd_generator.generate_all() when stack=llm-d.
  • ui/components/deployment.py - Added a "Routing Profile" selectbox that appears when the llm-d stack is selected. Changing the profile resets generated YAML. The selected profile is passed through to the backend API call.
  • tests/unit/test_llmd_generator.py - Added TestRoutingProfiles class (4 tests), 2 API endpoint tests, and extracted _extract_epp_config() helper to reduce duplication.

How Has This Been Tested?

6 new unit tests added, full suite passes:

cd src && uv run pytest ../tests/unit/test_llmd_generator.py -v

Tests cover:

  • test_default_routing_profile_unchanged - Verifies backward compatibility: default profile produces identical output to pre-profile behavior (same 4 plugins, same scheduling weights).
  • test_session_affinity_profile - Verifies session-affinity-scorer appears in both plugins and scheduling config with weight 3.
  • test_throughput_optimized_profile - Verifies load-aware-scorer replaces prefix-cache-scorer with weight 3.
  • test_invalid_routing_profile_raises - Verifies ValueError for unknown profile names.
  • test_deploy_with_stack_llmd_with_routing_profile - API integration test: POST to /api/v1/deploy with routing_profile: "session-affinity" returns 200.
  • test_deploy_with_invalid_routing_profile_returns_422 - API returns 422 for an invalid routing profile (Pydantic Literal validation).

UI manually tested: selecting llm-d stack shows the routing profile dropdown; toggling between profiles regenerates deployment files with the correct EPP plugin configuration.

Merge criteria:

  • The commits are squashed in a cohesive manner and have meaningful messages.
  • Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
  • The developer has manually tested the changes and verified that the changes work

@amito
amito force-pushed the feat/routing-profiles branch from 46d4e5f to 7751dfa Compare July 10, 2026 06:25
@amito
amito force-pushed the feat/routing-profiles branch 4 times, most recently from c065d5d to 3b88cbe Compare July 22, 2026 05:41
@amito
amito marked this pull request as ready for review July 29, 2026 13:02
Comment thread src/planner/api/routes/configuration.py Outdated
from planner.shared.schemas import DeploymentMode, DeploymentRecommendation

StackType = Literal["vllm", "llm-d"]
RoutingProfileType = Literal["default", "session-affinity", "throughput-optimized"]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a duplicate of the ROUTING_PROFILES dict keys in llmd_generator.py#L47. I think we should use the keys from that file as the source of truth.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, addressing this in the follow-up commit.

Comment thread ui/components/deployment.py Outdated
prev_profile = st.session_state.get("routing_profile", "default")
profile = st.selectbox(
"Routing Profile",
options=["default", "session-affinity", "throughput-optimized"],

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This list is a duplicate of ROUTING_PROFILES.keys() from llmd_generator.py. Can we import the dict?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, addressing this in the follow-up commit.

@amito
amito force-pushed the feat/routing-profiles branch from 3b88cbe to 019348e Compare August 20, 2026 06:21
Introduce named routing profiles (default, session-affinity,
throughput-optimized) that control EPP plugin configuration in
the generated Helm values.

- Add ROUTING_PROFILES dict with TypedDict annotations
- Add routing_profile parameter to generate_all() and /deploy API
- Use Literal type for API-level validation of routing_profile
- Fix Jinja2 weight check to handle weight: 0 correctly
- Add API test for invalid routing profile (422)
- Extract _extract_epp_config() test helper to reduce duplication

Signed-off-by: Amit Oren <amoren@redhat.com>
@amito
amito force-pushed the feat/routing-profiles branch from 019348e to e4ebd31 Compare August 20, 2026 06:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants