How to Use the Claude LLM Gateway

1. Installation

# Clone / enter the project directory
cd claude-llm-gateway

# Create and activate a virtual environment (recommended)
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate

# Install all dependencies (including optional ones for full features)
pip install -r requirements.txt

Minimal install (no semantic cache, no Redis):
pip install anthropic fastapi "uvicorn[standard]" pyyaml

2. Configuration

# Set your Anthropic API key
export ANTHROPIC_API_KEY=sk-ant-...

Or create a .env file:

cp .env.example .env
# Edit .env and fill in ANTHROPIC_API_KEY

Optional: edit config.yaml to tune thresholds, enable Redis, or disable the semantic cache.

3. Start the server

uvicorn gateway.main:app --reload --port 8000

You should see:

INFO  gateway.main — Gateway ready
INFO  uvicorn — Application startup complete

4. Send your first request

The gateway accepts the same JSON format as the Anthropic Messages API.

curl (macOS / Linux / Git Bash)

curl -s http://localhost:8000/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "max_tokens": 256
  }' | python -m json.tool

curl (Windows PowerShell)

# Use curl.exe (not the PowerShell alias) to avoid quoting issues
curl.exe -s http://localhost:8000/v1/messages `
  -H "Content-Type: application/json" `
  -d "{\"messages\":[{\"role\":\"user\",\"content\":\"What is the capital of France?\"}],\"max_tokens\":256}" `
  | python -m json.tool

# Or the cleaner PowerShell-native way:
$body = @{
    messages  = @(@{ role = "user"; content = "What is the capital of France?" })
    max_tokens = 256
} | ConvertTo-Json

Invoke-RestMethod -Uri http://localhost:8000/v1/messages `
  -Method Post -ContentType "application/json" -Body $body

Python (direct HTTP)

import requests

resp = requests.post("http://localhost:8000/v1/messages", json={
    "messages": [{"role": "user", "content": "Explain async/await in Python"}],
    "max_tokens": 512,
})
print(resp.json()["content"][0]["text"])

# Check which model was used and how much was saved
print("Model used:", resp.headers["X-Gateway-Model"])
print("Saved:     ", resp.headers["X-Gateway-Saved-USD"], "USD")
print("Cache:     ", resp.headers["X-Gateway-Cache"])

Python (Anthropic SDK — drop-in replacement)

import anthropic

client = anthropic.Anthropic(
    api_key="any-string",           # the gateway handles the real key
    base_url="http://localhost:8000",
)

message = client.messages.create(
    model="claude-opus-4-7",        # ignored — gateway decides the model
    max_tokens=1024,
    messages=[{"role": "user", "content": "Design a distributed caching system"}],
)
print(message.content[0].text)

Setting base_url is all you need to switch an existing codebase to use the gateway.

5. Using a system prompt

resp = requests.post("http://localhost:8000/v1/messages", json={
    "system": "You are a concise technical assistant.",
    "messages": [{"role": "user", "content": "What are SOLID principles?"}],
    "max_tokens": 512,
})

6. How escalation works

When the gateway gets a response back from a model it checks the quality before returning it. If the response looks uncertain, it automatically retries with the next stronger tier — all transparently to the caller.

What triggers escalation:

Trigger	Example
Response was cut off	`stop_reason == "max_tokens"`
Uncertainty phrase found	`"I'm not sure"`, `"I cannot"`, `"I don't have"`, `"I'm unable"`
Too many hedge words	>5 of: `"perhaps"`, `"possibly"`, `"might"`, `"allegedly"`

Flow:

Haiku answers → validator detects trigger
             → escalate to Sonnet → check again
             → escalate to Opus   → stop (max 2 escalations)

Force escalation in a test — two reliable ways:

import requests

# Method 1: max_tokens too small → response truncated → escalates
resp = requests.post("http://localhost:8000/v1/messages", json={
    "messages": [{"role": "user", "content": "List the differences between REST and GraphQL"}],
    "max_tokens": 8,   # guaranteed truncation
})
print(resp.headers["X-Gateway-Model"])   # will be Sonnet or Opus, not Haiku

# Method 2: ask for real-time data → model says "I don't have" → escalates
resp = requests.post("http://localhost:8000/v1/messages", json={
    "messages": [{"role": "user", "content": "What is the live stock price of NVIDIA right now?"}],
    "max_tokens": 128,
})
print(resp.headers["X-Gateway-Model"])   # escalated up to Opus

Check /v1/metrics/recent after — the escalated and escalation_count fields confirm it happened.

Tune escalation in config.yaml:

max_escalations: 2   # set to 0 to disable escalation entirely

7. Dashboard

Open the live HTML dashboard in your browser:

http://localhost:8000/dashboard

It auto-refreshes every 10 seconds and shows:

KPI cards — total requests, cache hit rate, total saved, actual cost, escalation rate
Savings bar — % of baseline (all-Opus) cost avoided
Model distribution — donut chart (Haiku / Sonnet / Opus)
Cache breakdown — exact hits vs semantic hits vs misses
Classifier paths — how many queries each layer handled (heuristic / embedding / haiku)
Recent requests table — every request with model, cache type, classifier, escalation flag, tokens, cost, savings

For the raw JSON instead:

# Aggregated stats
curl -s http://localhost:8000/v1/metrics | python -m json.tool

# Last 10 individual decisions
curl -s "http://localhost:8000/v1/metrics/recent?n=10" | python -m json.tool

8. Run the test suite

The included test_gateway.py exercises all features end-to-end with coloured output:

python test_gateway.py

# Or against a non-default URL:
python test_gateway.py --base http://localhost:8000

What it tests:

Test	What it verifies
0 — Health	Gateway is reachable
1 — Haiku routing	Factual / trivial queries → `claude-haiku-4-5`
2 — Sonnet routing	Reasoning / code queries → `claude-sonnet-4-6`
3 — Opus routing	Design / complex queries → `claude-opus-4-7`
4 — Exact cache	Repeat query returns `X-Gateway-Cache: exact`
5 — Escalation (truncation)	`max_tokens=8` → truncated → escalates automatically
6 — Escalation (uncertainty)	Real-time data query → Haiku says "I don't have" → escalates to Opus
7 — System prompt	System prompt is passed through correctly
8 — Metrics	Prints full routing summary with model distribution bar chart

10. Tune the classifier

Edit config.yaml to change routing behavior:

# Disable the Haiku API fallback (use heuristics + embedding only, no extra cost)
use_haiku_classifier: false

# Tighten the semantic cache (only very close matches)
cache:
  semantic_threshold: 0.97

# Loosen the semantic cache (catch more paraphrases, some risk of wrong hits)
cache:
  semantic_threshold: 0.90

Restart the server after editing config.yaml.

11. Enable Redis for persistent caching

# Start Redis (Docker)
docker run -d -p 6379:6379 redis:alpine

# Update config.yaml
cache:
  redis_url: redis://localhost:6379

Or set the environment variable:

export REDIS_URL=redis://localhost:6379

12. Understanding routing decisions

Each response from /v1/messages includes these headers:

Header	Meaning
`X-Gateway-Model`	The model actually used
`X-Gateway-Cache`	`none`, `exact`, or `semantic`
`X-Gateway-Saved-USD`	Dollars saved vs. always using Opus
`X-Gateway-Classifier`	How the tier was decided: `heuristic`, `embedding`, `haiku`, or `n/a` (cache hit)
`X-Gateway-Query-ID`	Short ID for correlating with `/v1/metrics/recent`

13. Production checklist

Set ANTHROPIC_API_KEY via a secret manager, not a .env file
Run with a production ASGI server: uvicorn gateway.main:app --workers 4 --port 8000
Point Redis at a persistent instance with AOF or RDB enabled
Add authentication middleware to the gateway (the current code has none)
Monitor X-Gateway-Saved-USD over time to measure ROI
Periodically review /v1/metrics/recent for escalations — each one is a classifier training signal

Common questions

Q: Will the gateway change the response content?
No. It proxies the exact response from the Anthropic API. Only the model that answered may differ from what the client specified.

Q: Can I force a specific model?
Not via the API field (it's ignored). To pin a model, set use_haiku_classifier: false in config.yaml and remove the heuristics for your use case — or add a direct call that bypasses the gateway.

Q: What happens if the Anthropic API is down?
The gateway propagates the error as a 500 response with the original error message.

Q: Does the semantic cache ever return wrong answers?
It can if the similarity threshold is too low. The default of 0.95 is conservative. If you see wrong cache hits, raise it to 0.97 or 0.99.

Q: How do I clear the cache?
In-memory: restart the server. Redis: redis-cli FLUSHDB.

FilesExpand file tree

HOW_TO_USE.md

Latest commit

History