Release v0.10.0 · llm-d/llm-d-inference-sim

What's New

/v1/completions prompt can be array of text prompts, array of tokens, and array of arrays of tokens
Add /v1/completions/render and /v1/chat/completions/render simulation
Add /admin/config endpoint for configuration updates
Add --force-dummy-tokenizer flag to force the use of the dummy tokenizer regardless of model name
Add Mooncake bootstrap /query endpoint

Deprecated: The /fake-metrics endpoint is no longer supported. Please migrate your fake metrics configuration to the /admin/config endpoint.

fix: ensure all metrics are written before signaling request completion by @irar2 in #544
Add logprobs to /responses response by @mayabar in #483
feat: add n parameter for multiple completion choices by @umakantv in #537
fix: send finish_reason in last tokens chunk for generate streaming, fix missing finish_reason for non-stop completions by @irar2 in #545
fix: suppress expected shutdown noise in WriteToChannel and ZMQ dialer by @irar2 in #547
feat: add kwargs_data to multimodal render features and bump vLLM render image to v0.21.0 by @irar2 in #549
fix: correct spelling typos across codebase and docs, suppress PN typo false positive in _typos.toml by @irar2 in #553
Detailed documentation of latency calculations and proposed values by @irar2 in #501
feat: add Anthropic /v1/messages endpoint by @irar2 in #551
Fix typos by @mayabar in #555
docs: document /v1/messages Anthropic endpoint and kwargs_data render field by @irar2 in #556
refactor: merge vllm-api and openai-server-api into single pkg/api package by @irar2 in #558
feat: add support for image and audio content types in Responses API by @umakantv in #548
Fix name of usage.prompt_tokens_details by @mayabar in #559

Full Changelog: v0.9.2...v0.10.0