feat: add support for new field prefill.servicePort - #200
Conversation
- to support prefill use a different port than decode's port - if prefill.servicePort is not set, fall back to use routing.servicePort - if both prefill.servicePort and prefill.container.ports are set, servicePort take precedence to be containerPorts Signed-off-by: Wen Zhou <wenzhou@redhat.com>
| prefill: | ||
| create: true | ||
| replicas: 1 | ||
| servicePort: 8300 # External service port (prefill pod recieves requests here) |
There was a problem hiding this comment.
I'm actually wondering about this setup. @kalantar also had the same question. This would produce these args in the sidecar: --port 8000 --vllm-port 8200, right? But how does the routing sidecar know that prefill port is at 8300? Is the sidecar communicating between prefill and decode, or is the EPP doing so?
There was a problem hiding this comment.
I asked around and now realize my original idea to create 2 infpool which supports different port respectively wont work with only 1 EPP. 😢
so in this case, yes, seems this is the limitation for the current design that we must have the same port for prefill and decode ,thus to add this support in modelservice is not really useful.
i am going to close this PR , thanks for the review and all the good discussions 🙏
2 participants
changes