feat: add configurable routing profiles to llm-d generator - #281
Open
amito wants to merge 1 commit into
Open
Conversation
amito
force-pushed
the
feat/routing-profiles
branch
from
July 10, 2026 06:25
46d4e5f to
7751dfa
Compare
amito
force-pushed
the
feat/routing-profiles
branch
4 times, most recently
from
July 22, 2026 05:41
c065d5d to
3b88cbe
Compare
amito
marked this pull request as ready for review
July 29, 2026 13:02
jgchn
reviewed
Aug 3, 2026
| from planner.shared.schemas import DeploymentMode, DeploymentRecommendation | ||
|
|
||
| StackType = Literal["vllm", "llm-d"] | ||
| RoutingProfileType = Literal["default", "session-affinity", "throughput-optimized"] |
Collaborator
There was a problem hiding this comment.
This is a duplicate of the ROUTING_PROFILES dict keys in llmd_generator.py#L47. I think we should use the keys from that file as the source of truth.
Collaborator
Author
There was a problem hiding this comment.
Thanks, addressing this in the follow-up commit.
| prev_profile = st.session_state.get("routing_profile", "default") | ||
| profile = st.selectbox( | ||
| "Routing Profile", | ||
| options=["default", "session-affinity", "throughput-optimized"], |
Collaborator
There was a problem hiding this comment.
This list is a duplicate of ROUTING_PROFILES.keys() from llmd_generator.py. Can we import the dict?
Collaborator
Author
There was a problem hiding this comment.
Thanks, addressing this in the follow-up commit.
amito
force-pushed
the
feat/routing-profiles
branch
from
August 20, 2026 06:21
3b88cbe to
019348e
Compare
Introduce named routing profiles (default, session-affinity, throughput-optimized) that control EPP plugin configuration in the generated Helm values. - Add ROUTING_PROFILES dict with TypedDict annotations - Add routing_profile parameter to generate_all() and /deploy API - Use Literal type for API-level validation of routing_profile - Fix Jinja2 weight check to handle weight: 0 correctly - Add API test for invalid routing profile (422) - Extract _extract_epp_config() test helper to reduce duplication Signed-off-by: Amit Oren <amoren@redhat.com>
amito
force-pushed
the
feat/routing-profiles
branch
from
August 20, 2026 06:35
019348e to
e4ebd31
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
PR #274 introduced minimal llm-d manifest generation with hardcoded default EPP routing plugins. This PR makes the routing configuration selectable via named profiles, end-to-end from UI to generated Helm values.
Three routing profiles are available:
default- Current behavior (prefix-cache-scorer, decode-filter, max-score-picker, single-profile-handler). Backward-compatible: existing API calls produce identical output.session-affinity- Adds session-affinity-scorer (weight 3) for conversational/chat workloads where requests from the same session benefit from being routed to the same replica.throughput-optimized- Replaces prefix-cache-scorer with load-aware-scorer (weight 3) for batch/throughput-oriented workloads where even load distribution matters more than cache locality.Changes
src/planner/configuration/llmd_generator.py- AddedROUTING_PROFILESdict withTypedDictannotations defining plugin configurations for each profile.generate_all()now accepts an optionalrouting_profileparameter (default:"default"), validates it, and injects the profile's plugins into the template context.src/planner/configuration/templates/llmd/values.yaml.j2- Replaced hardcoded plugin list with Jinja2 loops overpluginsandscheduling_pluginstemplate variables. Fixed weight check to useis not none(handlesweight: 0correctly).src/planner/api/routes/configuration.py- Addedrouting_profilefield toDeploymentRequestwithLiteraltype validation. Passed through tollmd_generator.generate_all()whenstack=llm-d.ui/components/deployment.py- Added a "Routing Profile" selectbox that appears when the llm-d stack is selected. Changing the profile resets generated YAML. The selected profile is passed through to the backend API call.tests/unit/test_llmd_generator.py- AddedTestRoutingProfilesclass (4 tests), 2 API endpoint tests, and extracted_extract_epp_config()helper to reduce duplication.How Has This Been Tested?
6 new unit tests added, full suite passes:
Tests cover:
test_default_routing_profile_unchanged- Verifies backward compatibility: default profile produces identical output to pre-profile behavior (same 4 plugins, same scheduling weights).test_session_affinity_profile- Verifies session-affinity-scorer appears in both plugins and scheduling config with weight 3.test_throughput_optimized_profile- Verifies load-aware-scorer replaces prefix-cache-scorer with weight 3.test_invalid_routing_profile_raises- VerifiesValueErrorfor unknown profile names.test_deploy_with_stack_llmd_with_routing_profile- API integration test: POST to/api/v1/deploywithrouting_profile: "session-affinity"returns 200.test_deploy_with_invalid_routing_profile_returns_422- API returns 422 for an invalid routing profile (PydanticLiteralvalidation).UI manually tested: selecting llm-d stack shows the routing profile dropdown; toggling between profiles regenerates deployment files with the correct EPP plugin configuration.
Merge criteria: