feat(telemetry): capture prefix-cache effectiveness (cache_n, cached_tokens) and route-switch counts (#2968)

* feat(telemetry): capture prefix-cache effectiveness and route-switch counts

Closes #2955.

- Record llama-server timings.cache_n and OpenAI-wire
  usage.prompt_tokens_details.cached_tokens as cache_tokens
  (latest per model + cumulative totals), on streaming and
  non-streaming paths, local and cloud.
- Count collection.router decisions and per-conversation route
  switches, keyed by a stable conversation fingerprint (hash of
  system prompt + first user message; metrics key only).
- Expose everything in GET /v1/stats and Prometheus /metrics.
- TelemetryCallback now passes StreamingProxy::TelemetryData instead
  of five scalars so backend-reported fields extend without churn.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(telemetry): address review — stale cache gauge, cloud usage injection, Responses field

- cache_tokens latest gauge resets to unreported (JSON null) on every
  recorded request, so a request that reports nothing can no longer
  inherit the previous request's value.
- Cloud streaming injects stream_options.include_usage when the client
  did not request it and swallows the resulting usage-only frame, so
  telemetry gets provider usage while the client-visible stream is
  unchanged; client-requested usage frames pass through untouched.
- Streaming parser also reads Responses-API
  usage.input_tokens_details.cached_tokens.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(telemetry): atomic per-request recording and non-streaming Responses capture

- Collapse the three per-request telemetry recorders into one
  record_request_telemetry_for_model() under a single lock hold, so
  concurrent requests can no longer interleave fields (one request's
  cache reset landing on another's value).
- TelemetryData carries prompt_tokens; StreamingProxy::extract_telemetry()
  parses complete response bodies (usage chat/Responses field names,
  cached-token details, llama.cpp timings), replacing the duplicated
  handler blocks.
- Non-streaming /v1/responses now records telemetry, including
  usage.input_tokens_details.cached_tokens.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: ramkrishna2910

docs/api/lemonade.md

@@ -1250,7 +1250,15 @@ curl http://localhost:13305/v1/stats
   "tokens_per_second": 33.33,
   "input_tokens": 128,
   "output_tokens": 5,
-  "prompt_tokens": 9
+  "prompt_tokens": 9,
+  "cache_tokens": 96,
+  "request_count_total": 12,
+  "input_tokens_total": 1536,
+  "output_tokens_total": 60,
+  "prompt_tokens_total": 108,
+  "cache_tokens_total": 1152,
+  "routing_decisions_total": 4,
+  "routing_switches_total": 1
 }
 ```
 
@@ -1261,6 +1269,10 @@ curl http://localhost:13305/v1/stats
 - `input_tokens` - Number of tokens processed
 - `output_tokens` - Number of tokens generated
 - `prompt_tokens` - Total prompt tokens including cached tokens
+- `cache_tokens` - Prompt tokens served from the backend's prefix cache on the last request (llama.cpp `timings.cache_n`, or `usage.prompt_tokens_details.cached_tokens` / Responses-API `input_tokens_details.cached_tokens` from OpenAI-compatible cloud providers). `null` when the last request did not report cache usage
+- `*_total` - Cumulative counters since server start
+- `routing_decisions_total` - Routing decisions made by `collection.router` dispatch
+- `routing_switches_total` - Routing decisions that changed a conversation's routed model (a proxy for route ping-pong; conversations are identified by a hash of the system prompt and first user message)
 
 ## `GET /v1/system-stats`
 <sub>![Status](https://img.shields.io/badge/status-fully_available-green)</sub>

modelscope-blog/assets/modelscope-chat.png

@@ -0,0 +1,49 @@
+# Search ModelScope from Lemonade
+
+**Date:** July 27, 2026 · **Author:** Lemonade Team
+
+> Lemonade v11.5 adds ModelScope as a second model registry, right next to Hugging Face. Same search box, twice the catalog.
+
+## Twice the catalog, same search box
+
+Starting with v11.5, Lemonade supports [ModelScope](https://modelscope.cn) as a model registry alongside Hugging Face. Search it the same way you search Hugging Face — right from the Model Manager — and download models from whichever registry has what you need. This matters most for users in regions where ModelScope mirrors are significantly faster, and for models that publish to ModelScope first.
+
+There's nothing to set up — no accounts, no extra configuration. Type a name, browse the GGUF variants, and hit download.
+
+And you won't waste a download on a repo that can't run. Before a result earns a download button, Lemonade inspects the repository's actual file tree and only offers models with GGUF files it can serve. What you see is what will run.
+
+## Take it for a spin
+
+Open the Lemonade app (or the web app at `http://localhost:13305/app`) and head to the **Model Manager**. Start typing — `Qwen 2.5`, say. After three characters the search goes live, and two new sections appear below your local models: **FROM HUGGING FACE** and **FROM MODELSCOPE**.
+
+![Model Manager showing search results from Hugging Face and ModelScope](assets/modelscope-search-results.png)
+
+Every result tells you what you're getting before you commit: the repository name, a source badge (**MS** or **HF**), the download size, and how many times the community has pulled it. Want a different quantization? Pick one from the dropdown — Q4_K_M is preselected as a sensible default. Then click download and watch it go.
+
+![Download Manager pulling a model from ModelScope](assets/modelscope-download.png)
+
+When the download finishes, the model registers itself and is ready to use — select it in the chat panel, point your favorite OpenAI-compatible client at it, whatever you'd do with any other Lemonade model. Where it came from stops mattering the moment it lands on your disk.
+
+![Chatting with a model downloaded from ModelScope](assets/modelscope-chat.png)
+
+## Prefer a terminal?
+
+The CLI speaks ModelScope too. Give `lemonade pull` a checkpoint and tell it where to look — or just paste a `modelscope.cn` model URL and it figures out the rest.
+
+```bash
+lemonade pull Qwen/Qwen2.5-3B-Instruct-GGUF --source modelscope
+```
+
+If you're building on top of the server, the same search that powers the Model Manager is one GET away:
+
+```bash
+curl "http://localhost:13305/v1/registry/search?source=modelscope&query=qwen"
+```
+
+Add `format=gguf` to bias results toward GGUF repositories, or `limit` to control how many come back (1–50, default 12). Responses include tags, task, download counts, and a GGUF hint per model — see the [API docs](https://lemonade-server.ai/docs/api/lemonade/) for the full shape.
+
+## Go find something new
+
+Update to Lemonade v11.5 and search both registries from one box.
+
+[Install Lemonade](https://lemonade-server.ai/#getting-started) · [Join the Discord](https://discord.gg/5xXzkMu8Zk)

modelscope-blog/assets/modelscope-download.png

@@ -5,11 +5,13 @@
 #include <memory>
 #include <deque>
 #include <functional>
+#include <list>
 #include <map>
 #include <mutex>
 #include <set>
 #include <condition_variable>
 #include <thread>
+#include <unordered_map>
 #include <vector>
 #include <optional>
 #include <nlohmann/json.hpp>
@@ -241,11 +243,14 @@ class Router {
     // Get loaded backend metadata and per-model telemetry for metrics rendering.
     json get_metrics_snapshot() const;
 
-    void update_telemetry(const std::string& model_name,
-                         int input_tokens, int output_tokens,
-                         double time_to_first_token, double tokens_per_second);
+    // Record one completed request's telemetry as a single atomic update.
+    void update_request_telemetry(const std::string& model_name,
+                                  const StreamingProxy::TelemetryData& telemetry);
 
-    void update_prompt_tokens(const std::string& model_name, int prompt_tokens);
+    // Route-stability accounting for collection.router dispatch. The
+    // fingerprint is a metrics key only (hash of the conversation's stable
+    // prefix) — it never influences routing and stores no message content.
+    void note_route_decision(uint64_t conversation_fingerprint, const std::string& route_to);
 
     bool begin_exclusive(std::atomic<bool>* cancel = nullptr);
     void end_exclusive();
@@ -277,6 +282,13 @@ class Router {
     Telemetry aggregate_telemetry_;
     std::map<std::string, ModelTelemetryRecord> telemetry_by_model_;
 
+    uint64_t routing_decisions_total_ = 0;
+    uint64_t routing_switches_total_ = 0;
+    std::list<uint64_t> route_fingerprint_lru_;
+    std::unordered_map<uint64_t,
+                       std::pair<std::string, std::list<uint64_t>::iterator>>
+        route_last_target_;
+
     // Concurrency control for load operations
     mutable std::mutex load_mutex_;              // Protects loading state and loaded_servers_
     bool is_loading_ = false;                    // True when a load operation is in progress
@@ -378,12 +390,8 @@ class Router {
     std::unique_ptr<WrappedServer> create_backend_server(const ModelInfo& model_info);
     std::string resolve_model_name(const std::string& model_name) const;
     ModelTelemetryIdentity get_telemetry_identity(WrappedServer* server) const;
-    void record_telemetry_for_model(const ModelTelemetryIdentity& identity,
-                                    int input_tokens,
-                                    int output_tokens,
-                                    double time_to_first_token,
-                                    double tokens_per_second);
-    void record_prompt_tokens_for_model(const ModelTelemetryIdentity& identity, int prompt_tokens);
+    void record_request_telemetry_for_model(const ModelTelemetryIdentity& identity,
+                                            const StreamingProxy::TelemetryData& telemetry);
 
     template<typename Func>
     auto execute_inference(const json& request, Func&& inference_func) -> decltype(inference_func(nullptr));

modelscope-blog/assets/modelscope-search-results.png

@@ -126,6 +126,10 @@ class Server {
     void handle_model_update_check(const httplib::Request& req, httplib::Response& res);
     void handle_model_files(const httplib::Request& req, httplib::Response& res);
     void handle_chat_completions(const httplib::Request& req, httplib::Response& res);
+    // Log and atomically record one non-streaming response's telemetry
+    // (usage/timings, cached tokens) against the serving model.
+    void record_response_telemetry(const nlohmann::json& response,
+                                   const nlohmann::json& request_json);
     // Server-side tool-calling orchestration for Omni "collection" models.
     void handle_collection_chat_completions(const nlohmann::json& request_json,
                                             const ModelInfo& collection_info,

modelscope-blog/modelscope-search.md

@@ -17,6 +17,13 @@ class StreamingProxy {
     struct TelemetryData {
         int input_tokens = 0;
         int output_tokens = 0;
+        // usage.prompt_tokens / usage.input_tokens as reported (includes cached
+        // tokens). -1 = not reported; llama.cpp timings then leave input_tokens
+        // holding only the processed count.
+        int prompt_tokens = -1;
+        // Prompt tokens served from the backend's prefix cache. -1 = the
+        // backend did not report cache usage (0 is a real cold-prefill value).
+        int cache_tokens = -1;
         double time_to_first_token = 0.0;
         double tokens_per_second = 0.0;
         std::string error_message = "";
@@ -62,6 +69,11 @@ class StreamingProxy {
 
     static TelemetryData parse_telemetry(const std::string& buffer);
 
+    // Extract telemetry from a complete (non-streaming) response body or a
+    // single SSE chunk payload: OpenAI usage (chat and Responses field names,
+    // cached-token details) and llama.cpp timings.
+    static TelemetryData extract_telemetry(const nlohmann::json& payload);
+
     static void accumulate_responses_delta(const nlohmann::json& parsed, std::string& accumulated_text);
 
 private:

src/cpp/include/lemon/router.h

@@ -0,0 +1,77 @@
+#pragma once
+
+#include <cstdint>
+#include <nlohmann/json.hpp>
+#include <string>
+
+namespace lemon {
+namespace utils {
+
+// 64-bit FNV-1a fingerprint of a conversation's stable prefix (system text
+// plus the first user message text). Used as a metrics key for route-stability
+// counters only — it never influences routing and no message content is
+// retained alongside it. Later turns append to the conversation, so the
+// fingerprint stays constant for the lifetime of one conversation.
+
+inline void fingerprint_append(uint64_t& hash, const std::string& text) {
+    for (unsigned char c : text) {
+        hash ^= static_cast<uint64_t>(c);
+        hash *= 1099511628211ULL;
+    }
+}
+
+inline void fingerprint_append_content(uint64_t& hash, const nlohmann::json& content) {
+    if (content.is_string()) {
+        fingerprint_append(hash, content.get<std::string>());
+    } else if (content.is_array()) {
+        for (const auto& part : content) {
+            if (part.is_string()) {
+                fingerprint_append(hash, part.get<std::string>());
+            } else if (part.is_object() && part.contains("text") && part["text"].is_string()) {
+                fingerprint_append(hash, part["text"].get<std::string>());
+            }
+        }
+    }
+}
+
+inline uint64_t conversation_fingerprint(const nlohmann::json& request_json) {
+    uint64_t hash = 14695981039346656037ULL;
+
+    const nlohmann::json* messages = nullptr;
+    if (request_json.contains("messages") && request_json["messages"].is_array()) {
+        messages = &request_json["messages"];
+    } else if (request_json.contains("input") && request_json["input"].is_array()) {
+        messages = &request_json["input"];
+    }
+
+    if (messages) {
+        for (const auto& message : *messages) {
+            if (message.is_object() && message.value("role", "") == "system" &&
+                message.contains("content")) {
+                fingerprint_append_content(hash, message["content"]);
+                break;
+            }
+        }
+        fingerprint_append(hash, "\x1f");
+        for (const auto& message : *messages) {
+            if (message.is_object() && message.value("role", "") == "user" &&
+                message.contains("content")) {
+                fingerprint_append_content(hash, message["content"]);
+                break;
+            }
+        }
+        return hash;
+    }
+
+    if (request_json.contains("prompt")) {
+        fingerprint_append(hash, "\x1f");
+        fingerprint_append_content(hash, request_json["prompt"]);
+    } else if (request_json.contains("input") && request_json["input"].is_string()) {
+        fingerprint_append(hash, "\x1f");
+        fingerprint_append(hash, request_json["input"].get<std::string>());
+    }
+    return hash;
+}
+
+} // namespace utils
+} // namespace lemon

src/cpp/include/lemon/server.h

@@ -18,6 +18,7 @@
 #include "model_residency.h"
 #include "backend_manager.h"
 #include "recipe_options.h"
+#include "streaming_proxy.h"
 #include "backends/backend_descriptor.h"
 
 namespace lemon {
@@ -38,21 +39,28 @@ struct Telemetry {
     double time_to_first_token = 0.0;
     double tokens_per_second = 0.0;
     int prompt_tokens = 0;  // From usage.prompt_tokens (includes cached tokens)
+    // Prompt tokens served from the backend's prefix cache on the latest
+    // request. -1 = the latest request did not report cache usage; rendered as
+    // JSON null so a stale numeric value is never attributed to it.
+    int cache_tokens = -1;
     uint64_t request_count_total = 0;
     uint64_t input_tokens_total = 0;
     uint64_t output_tokens_total = 0;
     uint64_t prompt_tokens_total = 0;
+    uint64_t cache_tokens_total = 0;
 
     void reset() {
         input_tokens = 0;
         output_tokens = 0;
         time_to_first_token = 0.0;
         tokens_per_second = 0.0;
         prompt_tokens = 0;
+        cache_tokens = -1;
         request_count_total = 0;
         input_tokens_total = 0;
         output_tokens_total = 0;
         prompt_tokens_total = 0;
+        cache_tokens_total = 0;
     }
 
     json to_json() const {
@@ -62,10 +70,12 @@ struct Telemetry {
             {"time_to_first_token", time_to_first_token},
             {"tokens_per_second", tokens_per_second},
             {"prompt_tokens", prompt_tokens},
+            {"cache_tokens", cache_tokens >= 0 ? json(cache_tokens) : json(nullptr)},
             {"request_count_total", request_count_total},
             {"input_tokens_total", input_tokens_total},
             {"output_tokens_total", output_tokens_total},
-            {"prompt_tokens_total", prompt_tokens_total}
+            {"prompt_tokens_total", prompt_tokens_total},
+            {"cache_tokens_total", cache_tokens_total}
         };
     }
 };
@@ -467,11 +477,7 @@ class WrappedServer : public ICompletionServer {
 
     // Forward streaming requests to the wrapped server (public for Router access)
     // Virtual so backends can transform request (e.g., FLM needs checkpoint in model field)
-    using TelemetryCallback = std::function<void(int input_tokens,
-                                                 int output_tokens,
-                                                 double time_to_first_token,
-                                                 double tokens_per_second,
-                                                 const std::string& error_message)>;
+    using TelemetryCallback = std::function<void(const StreamingProxy::TelemetryData& telemetry)>;
 
     virtual void forward_streaming_request(const std::string& endpoint,
                                            const std::string& request_body,