What would you like to be added:
A lightweight estimate of output sequence length, fit for common agentic patterns and done inline with EPP code. This provides another tradeoff point of accuracy/effort (e.g., simplest and least accurate using TokenLoadScorer estimates load of 1.5 × ISL, vs a likely more accurate prediction via an external sidecar/service).
Produce an estimate as a request attribute that can be consumed by various plugins.
Why is this needed:
The EPP has limited awareness of output sequence length (OSL). Scheduling decisions treat a short 50-token tool-call the same as an 8,000-token reasoning chain, causing potential scheduling issues:
- Wrong load balancing.
TokenLoadScorer estimates load as 1.5 × ISL. In agentic workloads input and output are nearly independent, the GLM-5.2 production trace shows median input 195K tokens, median output 317 tokens. Thus, the EPP can overload pods running short requests and underloads pods running reasoning chains.
- Head-of-line blocking. A single reasoning chain (p99 = 7,551 tokens) at the front of the Flow Control queue stalls an entire burst of short tool-calls (p50 = 317 tokens) behind it. Sub-agent bursts arrive 51-deep at p90.
- No KV-pressure awareness by request type. Under memory pressure, LONG and SHORT requests are throttled equally, even though admitting a SHORT request is far less risky.
This is a research spike at this stage and depends on finding reasonably accurate and lightweight estimator.
What would you like to be added:
A lightweight estimate of output sequence length, fit for common agentic patterns and done inline with EPP code. This provides another tradeoff point of accuracy/effort (e.g., simplest and least accurate using
TokenLoadScorerestimates load of 1.5 × ISL, vs a likely more accurate prediction via an external sidecar/service).Produce an estimate as a request attribute that can be consumed by various plugins.
Why is this needed:
The EPP has limited awareness of output sequence length (OSL). Scheduling decisions treat a short 50-token tool-call the same as an 8,000-token reasoning chain, causing potential scheduling issues:
TokenLoadScorerestimates load as 1.5 × ISL. In agentic workloads input and output are nearly independent, the GLM-5.2 production trace shows median input 195K tokens, median output 317 tokens. Thus, the EPP can overload pods running short requests and underloads pods running reasoning chains.This is a research spike at this stage and depends on finding reasonably accurate and lightweight estimator.