|
| 1 | +--- |
| 2 | +name: ratel-observability |
| 3 | +description: Wire Ratel's lean observability SDK (ratel-ai) into an existing Python agent so every model interaction ships a usage rollup to the dashboard. Attributes token counts to the five context sources (skills/tools/history/memory/user_input), optionally computes Ratel tool savings, and is best-effort — never breaks the host app, no-ops without a key. Invoke when instrumenting a customer's agent for Ratel cloud analytics. |
| 4 | +--- |
| 5 | + |
| 6 | +# /ratel-observability |
| 7 | + |
| 8 | +Instruments an **existing** Python agent — one that already has its own LLM loop — with Ratel's lean cloud-analytics layer. One `track()` call per model interaction ships a usage rollup to `POST {RATEL_HOST}/api/v1/events` (default host `https://cloud.ratel.sh`), best-effort in a background thread, so the dashboard fills with real per-interaction token data. You are **not** asking the customer to adopt Ratel's tool catalog or restructure their loop; instrumentation works with whatever they already have. |
| 9 | + |
| 10 | +Two invariants hold throughout, and you should reassure the customer of both: |
| 11 | + |
| 12 | +- **It never breaks their app.** Export is background, batched, and best-effort — it never blocks or raises into their code. |
| 13 | +- **No key → no-op.** `get_client()` returns a singleton that silently does nothing until `RATEL_API_KEY` is set; you can add the calls now and turn ingestion on later. |
| 14 | + |
| 15 | +## Procedure |
| 16 | + |
| 17 | +### 1. Install and set the key |
| 18 | + |
| 19 | +```bash |
| 20 | +pip install 'ratel-ai' |
| 21 | +export RATEL_API_KEY=rtl_... # the project ingest key from the dashboard |
| 22 | +export RATEL_HOST=https://cloud.ratel.sh # optional; this is the default |
| 23 | +``` |
| 24 | + |
| 25 | +`RATEL_API_KEY` is the only required env var. Absent it, every call below is a no-op — safe to land in the customer's codebase before they have a key. |
| 26 | + |
| 27 | +### 2. Find the per-interaction boundary |
| 28 | + |
| 29 | +Locate the one place the customer calls the model per agent turn — the function that builds the prompt and invokes the LLM (e.g. `client.chat.completions.create(...)`, `anthropic.messages.create(...)`, an `agent.run(...)` step). That call site is where one `track()` goes. **One `track()` per agent interaction** — not per token, not per retry. |
| 30 | + |
| 31 | +If their loop calls the model several times for one logical interaction (tool-use round-trips), pick the granularity that matches a dashboard "run": usually one `track()` per user turn, summing the tokens across the inner calls. |
| 32 | + |
| 33 | +### 3. Add one `track()` per interaction |
| 34 | + |
| 35 | +After the model call, attribute the prompt's tokens to the five context sources Ratel reports — **`skills`, `tools`, `history`, `memory`, `user_input`** — from whatever the customer already has, and ship the rollup: |
| 36 | + |
| 37 | +```python |
| 38 | +from ratel_ai import get_client |
| 39 | + |
| 40 | +client = get_client() # env-configured singleton; no-op without RATEL_API_KEY |
| 41 | + |
| 42 | +# ... the customer's existing model call ... |
| 43 | +response = chat.completions.create(model="claude-sonnet-4-6", messages=messages, ...) |
| 44 | + |
| 45 | +client.track( |
| 46 | + tokens_by_category={ |
| 47 | + "skills": skills_tokens, # system/skill instructions, playbooks |
| 48 | + "tools": tools_tokens, # tool / function definitions in the prompt |
| 49 | + "history": history_tokens, # prior turns carried forward |
| 50 | + "memory": memory_tokens, # retrieved memory / RAG context |
| 51 | + "user_input": user_input_tokens, # this turn's user message |
| 52 | + }, |
| 53 | + model="claude-sonnet-4-6", |
| 54 | + output_tokens=response.usage.completion_tokens, |
| 55 | + latency_ms=elapsed_ms, # optional |
| 56 | + cost_usd=None, # optional; auto-estimated from model + tokens if omitted |
| 57 | +) |
| 58 | +``` |
| 59 | + |
| 60 | +Mapping the customer's reality to the five sources: |
| 61 | + |
| 62 | +- Map whatever they have to the closest source; **omit or zero a source they don't use** (a bare prompt-completion agent might only have `history` + `user_input`). |
| 63 | +- Prefer **real token counts** when the provider already returns a per-segment breakdown or the customer tokenizes the prompt themselves. |
| 64 | +- **No token counts? Approximate with `len(text) // 4`** — a serviceable chars-per-token estimate. The dashboard cares about proportions and trends, so a consistent estimate is fine: |
| 65 | + |
| 66 | + ```python |
| 67 | + def toks(text: str) -> int: |
| 68 | + return len(text) // 4 |
| 69 | + |
| 70 | + tokens_by_category = { |
| 71 | + "skills": toks(system_prompt), |
| 72 | + "tools": toks(tools_json), |
| 73 | + "history": sum(toks(m["content"]) for m in prior_messages), |
| 74 | + "memory": toks(retrieved_context), |
| 75 | + "user_input": toks(user_message), |
| 76 | + } |
| 77 | + ``` |
| 78 | + |
| 79 | +Optional fields: `latency_ms`, `cost_usd` (auto-estimated from `model` + tokens when omitted), and `occurred_at` (a `datetime`; defaults to now server-side — pass it only when backfilling). |
| 80 | + |
| 81 | +### 4. (Optional) Compute tool savings if they have a tool list |
| 82 | + |
| 83 | +If the customer passes a list of tool/function definitions to the model, Ratel can measure what selection would keep **out** of the prompt. Build a `ToolCatalog(observe=True)`, register their tools once, and after each `search` read `cat.last_savings`: |
| 84 | + |
| 85 | +```python |
| 86 | +from ratel_ai import ToolCatalog, ExecutableTool |
| 87 | + |
| 88 | +cat = ToolCatalog(observe=True) |
| 89 | +for t in customer_tools: # register each of their tools once, at startup |
| 90 | + cat.register(ExecutableTool( |
| 91 | + id=t["name"], name=t["name"], description=t["description"], |
| 92 | + input_schema=t["parameters"], execute=lambda args: {}, # metadata-only is fine for sizing |
| 93 | + )) |
| 94 | + |
| 95 | +# per interaction, before the model call: |
| 96 | +cat.search(user_message, top_k=5) |
| 97 | +# cat.last_savings → {"full_catalog_tokens", "selected_tokens", "tokens_saved", "top_k"} |
| 98 | + |
| 99 | +client.track( |
| 100 | + tokens_by_category={...}, |
| 101 | + saved_by_category={"tools": cat.last_savings["tokens_saved"]}, # if they act on selection |
| 102 | + model="claude-sonnet-4-6", |
| 103 | + output_tokens=response.usage.completion_tokens, |
| 104 | +) |
| 105 | +``` |
| 106 | + |
| 107 | +Two modes, pick the one that matches what the customer's loop actually does: |
| 108 | + |
| 109 | +- **They send the selected top-K** to the model → feed the saving into `saved_by_category={"tools": cat.last_savings["tokens_saved"]}` (what Ratel kept out of the prompt this run). |
| 110 | +- **Observe-only** — they still send the full catalog but want to know the upside → use `saveable_by_category={"tools": cat.last_savings["tokens_saved"]}` instead (what it *could* save). |
| 111 | + |
| 112 | +Skip this step entirely if the customer has no tool list — `track()` with just `tokens_by_category` is a complete rollup. |
| 113 | + |
| 114 | +### 5. Flush on shutdown |
| 115 | + |
| 116 | +Rollups ship from a background thread. Drain it before the process exits so nothing is lost: |
| 117 | + |
| 118 | +```python |
| 119 | +client.flush() # also auto-flushed at exit |
| 120 | +``` |
| 121 | + |
| 122 | +`flush()` is auto-registered at interpreter exit, so an explicit call is belt-and-suspenders — add it to the customer's shutdown path (signal handler, `atexit`, web-server lifespan teardown, or the `finally` of a script) when the process may be killed before atexit runs. |
| 123 | + |
| 124 | +## Conventions |
| 125 | + |
| 126 | +- **Sources are exactly five**: `skills`, `tools`, `history`, `memory`, `user_input`. Don't invent new keys — unmapped context folds into the nearest of these. |
| 127 | +- **One `track()` per interaction.** A "usage rollup" is per agent turn, not per provider call or per token. |
| 128 | +- **Best-effort, never load-bearing.** Don't wrap `track()` in error handling that changes app behavior, and don't block the request path on it — the SDK already swallows failures and runs off-thread. |
| 129 | +- **No key is a feature.** Landing the instrumentation before the customer has provisioned a key is fine and expected; it stays a no-op until `RATEL_API_KEY` is set. |
| 130 | + |
| 131 | +## Why this exists |
| 132 | + |
| 133 | +The lean observability layer and its `POST /api/v1/events` wire contract are recorded in [ADR 0013](../../../docs/adr/0013-observability-and-analytics.md). A runnable end-to-end demo — live skill/tool suggestions plus an SDK-driven "Ratel off → on" adoption story — lives at [`src/sdk/python/examples/observability_demo.py`](../../../src/sdk/python/examples/README.md). |
0 commit comments