|
| 1 | +# Evals |
| 2 | + |
| 3 | +LLM eval suite for tool-use quality. Six suites run the same 16 cases and |
| 4 | +the same three models (Claude Sonnet 4.6, GPT 5.4, Gemini 3 Flash) but with |
| 5 | +different tool sets and system prompts: |
| 6 | + |
| 7 | +| Suite name | Tools available | System prompt | |
| 8 | +| ------------------------- | ------------------------------------------- | --------------------------------------------- | |
| 9 | +| `search_replace` | `search_replace` only | Minimal custom "precise code editor" prompt | |
| 10 | +| `search_replace_few` | `search_replace` only | Variant prompt encouraging fewer tool calls | |
| 11 | +| `edit_file` | `edit_file` only | Minimal custom `edit_file` prompt | |
| 12 | +| `basic_agent` | `search_replace`, `write_file` | Production `LOCAL_AGENT_BASIC_SYSTEM_PROMPT` | |
| 13 | +| `pro_agent` | `search_replace`, `edit_file`, `write_file` | Production `LOCAL_AGENT_SYSTEM_PROMPT` (Pro) | |
| 14 | +| `pro_agent_experimental` | `search_replace`, `edit_file`, `write_file` | Editable copy of the Pro prompt for tweaking | |
| 15 | + |
| 16 | +Each case gives the model a real source file plus an editing instruction, |
| 17 | +runs the model with the suite's tools wired up, applies the produced edits, |
| 18 | +and then asks an LLM judge (GPT 5.4) whether the result satisfies the |
| 19 | +instruction. |
| 20 | + |
| 21 | +## Prerequisites |
| 22 | + |
| 23 | +All models are routed through the Dyad Engine gateway, so you only need one |
| 24 | +credential: a Dyad Pro API key, exposed as `DYAD_PRO_API_KEY`. The |
| 25 | +`edit_file` tool additionally calls the engine's `/tools/turbo-file-edit` |
| 26 | +endpoint to apply sketched edits — that uses the same key. |
| 27 | + |
| 28 | +The suite is skipped entirely when `DYAD_PRO_API_KEY` is unset — no tests will |
| 29 | +fail, they just won't run. This keeps regular `vitest run` safe for contributors |
| 30 | +without a key. |
| 31 | + |
| 32 | +Export the key for the session (plus the two required filter vars — see |
| 33 | +[Running the suite](#running-the-suite)): |
| 34 | + |
| 35 | +```bash |
| 36 | +export DYAD_PRO_API_KEY="..." |
| 37 | +EVAL_SUITE=all EVAL_MODEL=all npm run eval |
| 38 | +``` |
| 39 | + |
| 40 | +Or set everything inline for a single command: |
| 41 | + |
| 42 | +```bash |
| 43 | +DYAD_PRO_API_KEY="..." EVAL_SUITE=all EVAL_MODEL=all npm run eval |
| 44 | +``` |
| 45 | + |
| 46 | +Optional: override the gateway URL with `DYAD_ENGINE_URL` (defaults to |
| 47 | +`https://engine.dyad.sh/v1`). |
| 48 | + |
| 49 | +## Running the suite |
| 50 | + |
| 51 | +**Both `EVAL_SUITE` and `EVAL_MODEL` are required.** A full run of every |
| 52 | +suite against every model is expensive, so the suite will not run unless |
| 53 | +the caller opts in explicitly. If either variable is unset, the eval prints |
| 54 | +a warning describing how to configure it and registers a single skipped |
| 55 | +placeholder — it does not fail CI, but it also does not run any cases. |
| 56 | + |
| 57 | +Use the special value `all` to mean "run everything": |
| 58 | + |
| 59 | +```bash |
| 60 | +# Run every suite against every model against every case. |
| 61 | +EVAL_SUITE=all EVAL_MODEL=all DYAD_PRO_API_KEY="..." npm run eval |
| 62 | +``` |
| 63 | + |
| 64 | +**Heads up — this is expensive.** A full `all`/`all` run issues one |
| 65 | +generation per (suite × model × case) triple plus one judge call per case, |
| 66 | +across 6 suites, 3 models, and 16 cases. The `edit_file`, `pro_agent`, and |
| 67 | +`pro_agent_experimental` suites also make additional engine calls for each |
| 68 | +sketched edit the model produces through `edit_file`. Expect dozens of LLM requests, some of which run reasoning |
| 69 | +models on 300+ line fixtures. Use sparingly; prefer narrow filters during |
| 70 | +development. |
| 71 | + |
| 72 | +### Running a single suite |
| 73 | + |
| 74 | +Set `EVAL_SUITE` to the exact `name` (case-insensitive) of the suite — the |
| 75 | +same name that appears as a folder under `eval-results/`. A comma-separated |
| 76 | +list runs multiple suites: |
| 77 | + |
| 78 | +```bash |
| 79 | +# Just the original search_replace-only suite |
| 80 | +EVAL_SUITE=search_replace EVAL_MODEL=all DYAD_PRO_API_KEY="..." npm run eval |
| 81 | + |
| 82 | +# The basic_agent suite (Basic agent prompt, search_replace + write_file) |
| 83 | +EVAL_SUITE=basic_agent EVAL_MODEL=all DYAD_PRO_API_KEY="..." npm run eval |
| 84 | + |
| 85 | +# The pro_agent suite (Pro agent prompt, search_replace + edit_file + write_file) |
| 86 | +EVAL_SUITE=pro_agent EVAL_MODEL=all DYAD_PRO_API_KEY="..." npm run eval |
| 87 | +``` |
| 88 | + |
| 89 | +Note: `EVAL_SUITE` matches suite `name`s exactly (case-insensitive), and |
| 90 | +accepts a comma-separated list for multiple suites (e.g. |
| 91 | +`EVAL_SUITE=search_replace,edit_file`). Unknown names error out with the |
| 92 | +available list. |
| 93 | + |
| 94 | +### Running a single case |
| 95 | + |
| 96 | +Vitest's `-t` flag filters by test name. Case names are the `name` field in |
| 97 | +the `CASES` array of [tool_use.eval.ts](tool_use.eval.ts). |
| 98 | + |
| 99 | +```bash |
| 100 | +EVAL_SUITE=all EVAL_MODEL=all DYAD_PRO_API_KEY="..." \ |
| 101 | + npm run eval -- -t "Extract a helper function" |
| 102 | +``` |
| 103 | + |
| 104 | +`-t` matches as a substring, so a short unique fragment works too: |
| 105 | + |
| 106 | +```bash |
| 107 | +EVAL_SUITE=all EVAL_MODEL=all DYAD_PRO_API_KEY="..." npm run eval -- -t "zod" |
| 108 | +``` |
| 109 | + |
| 110 | +### Running against one model |
| 111 | + |
| 112 | +Set `EVAL_MODEL` to a case-insensitive substring of the model's label or |
| 113 | +model name. It matches against both, so short fragments like `sonnet`, `gpt`, |
| 114 | +or `gemini` work: |
| 115 | + |
| 116 | +```bash |
| 117 | +EVAL_SUITE=all EVAL_MODEL=sonnet DYAD_PRO_API_KEY="..." npm run eval |
| 118 | +``` |
| 119 | + |
| 120 | +### Combining filters |
| 121 | + |
| 122 | +`EVAL_SUITE`, `EVAL_MODEL`, and `-t` compose. A tight development loop: |
| 123 | + |
| 124 | +```bash |
| 125 | +EVAL_SUITE=search_replace EVAL_MODEL=sonnet \ |
| 126 | + DYAD_PRO_API_KEY="..." npm run eval -- -t "Extract a helper function" |
| 127 | +``` |
| 128 | + |
| 129 | +Note: vitest's `-t` pattern is applied across the full describe/test |
| 130 | +hierarchy as a regex, which makes "model label > case name" style patterns |
| 131 | +brittle across vitest versions. Prefer `EVAL_SUITE` / `EVAL_MODEL` for |
| 132 | +suite and model filtering and reserve `-t` for case-name filtering. |
| 133 | + |
| 134 | +## Where results are stored |
| 135 | + |
| 136 | +Every run writes structured output to `eval-results/` at the repo root. The |
| 137 | +directory is gitignored and never cleaned automatically — delete old runs by |
| 138 | +hand when you want to. |
| 139 | + |
| 140 | +Layout: |
| 141 | + |
| 142 | +``` |
| 143 | +eval-results/ |
| 144 | + <suite-name>/ ← one top-level folder per suite |
| 145 | + <run-start-ts>__<model-label>/ ← one folder per (run, model) |
| 146 | + <case-name>/ ← one folder per case |
| 147 | + record.json ← full structured record |
| 148 | + record.txt ← human-readable render of the same |
| 149 | + details/ ← per-record split views |
| 150 | + file_before.<ext> ← file at the start of the run |
| 151 | + file_after.<ext> ← file at the end of the run |
| 152 | + diff.patch ← cumulative unified diff |
| 153 | + system_prompt.txt ← system prompt sent to the model |
| 154 | + instructions.txt ← case instructions (no file content) |
| 155 | + user_prompt.txt ← full user message (file + instructions) |
| 156 | + metadata.json ← run metadata without big blobs |
| 157 | + metadata.txt ← same info, human-readable |
| 158 | + tool_calls/ |
| 159 | + 01.txt ← combined view of tool call #1 |
| 160 | + 01/ ← split view, one piece per file |
| 161 | + file_before.<ext> |
| 162 | + file_after.<ext> |
| 163 | + diff.patch |
| 164 | + meta.txt |
| 165 | + <arg_name>.<ext> ← one file per tool arg (see below) |
| 166 | + 02.txt |
| 167 | + 02/ |
| 168 | + ... |
| 169 | +``` |
| 170 | + |
| 171 | +The top-level folder is the suite `name`, so each suite lands in its own |
| 172 | +directory: |
| 173 | + |
| 174 | +- `eval-results/search_replace/` |
| 175 | +- `eval-results/search_replace_few/` |
| 176 | +- `eval-results/edit_file/` |
| 177 | +- `eval-results/basic_agent/` |
| 178 | +- `eval-results/pro_agent/` |
| 179 | +- `eval-results/pro_agent_experimental/` |
| 180 | + |
| 181 | +`<run-start-ts>` is captured once at process start, so every case from the |
| 182 | +same `npm run eval` invocation for a given (suite, model) pair clusters into |
| 183 | +one folder. Folder names sort chronologically under `ls`. |
| 184 | + |
| 185 | +### Record format |
| 186 | + |
| 187 | +`record.json` contains the complete machine-readable record. Key fields: |
| 188 | + |
| 189 | +- `timestamp`, `suite`, `caseName` — identifying metadata. |
| 190 | +- `model` — `{label, provider, modelName, responseModelId}`. `responseModelId` |
| 191 | + is the exact model string the gateway echoed back, which can differ from |
| 192 | + `modelName` (e.g. dated snapshots). |
| 193 | +- `prompt` — `{system, instructions, user}`. `system` is the full system |
| 194 | + prompt sent to the model (including the production agent prompts when the |
| 195 | + suite uses one). `instructions` is the bare case instruction — useful for |
| 196 | + scanning what was asked without the fixture file inlined. `user` is the |
| 197 | + full user message actually sent (file content + instructions). |
| 198 | +- `file` — `{name, before, after}`. The fixture file name plus its content |
| 199 | + at the start and end of the run. `before` / `after` are also written to |
| 200 | + `details/file_before.<ext>` / `details/file_after.<ext>` for easy editor |
| 201 | + opening with matching syntax highlighting. |
| 202 | +- `llm.totalDurationMs`, `llm.totalUsage` — wall-clock time and token totals |
| 203 | + for the model under test (not the judge). |
| 204 | +- `llm.requests` — per-step breakdown: each entry is one HTTP round-trip with |
| 205 | + its own duration, usage, and `finishReason`. |
| 206 | +- `toolCalls` — every tool call the model made. Each entry records |
| 207 | + `toolName`, `filePath`, an `args` map (keyed by the tool's parameter names, |
| 208 | + so `old_string`/`new_string` for `search_replace`, `content` for |
| 209 | + `write_file`, `content`/`instructions` for `edit_file`), the file before |
| 210 | + and after the call, and a unified diff of just that call. |
| 211 | +- `diff` — unified diff from the original fixture to the final file |
| 212 | + (i.e. the cumulative effect of all tool calls). |
| 213 | +- `judge` — the judge's verdict: `label`, `modelName`, `durationMs`, |
| 214 | + `usage`, `pass` (boolean), and `explanation` (the judge's written |
| 215 | + reasoning, with the trailing `PASS`/`FAIL` verdict line stripped). |
| 216 | +- `passed` — the overall test outcome. Requires the judge to say `PASS` *and* |
| 217 | + all structural checks to pass *and* no exceptions to be thrown. |
| 218 | +- `errorMessage` — set when the test threw (tool-call failure, structural |
| 219 | + check failure, judge FAIL, etc.); `null` otherwise. |
| 220 | + |
| 221 | +`record.txt` is a readable render of the same information — headers, the |
| 222 | +system prompt and instructions, inline tool-call bodies, usage totals, the |
| 223 | +final diff, and the judge's explanation. Open it when you want a quick |
| 224 | +human-readable summary instead of parsing JSON. |
| 225 | + |
| 226 | +### The `details/` folder |
| 227 | + |
| 228 | +`details/` is a split view of the record, intended for quick inspection and |
| 229 | +diffing without having to parse JSON or scroll through `record.txt`: |
| 230 | + |
| 231 | +- `file_before.<ext>` / `file_after.<ext>` — raw file content before and |
| 232 | + after the run, with the fixture's extension preserved so editors apply |
| 233 | + the right syntax highlighting. |
| 234 | +- `diff.patch` — the same unified diff as `record.diff`. |
| 235 | +- `system_prompt.txt`, `instructions.txt`, `user_prompt.txt` — the three |
| 236 | + views of the prompt input. |
| 237 | +- `metadata.json` / `metadata.txt` — everything from `record.json` minus the |
| 238 | + large content blobs that already have their own files (no inline file |
| 239 | + contents and no per-tool-call entries). Useful for skimming token counts, |
| 240 | + judge verdict, and model identity across many runs. |
| 241 | + |
| 242 | +### The `tool_calls/` folder |
| 243 | + |
| 244 | +One `NN.txt` (combined view) and one `NN/` folder (split view) per tool |
| 245 | +call. The split view contains the raw pieces as standalone files: |
| 246 | + |
| 247 | +- `file_before.<ext>`, `file_after.<ext>`, `diff.patch` — file state around |
| 248 | + the single call. |
| 249 | +- `meta.txt` — timestamp, tool name, target path, and per-arg length summary. |
| 250 | +- One file per tool argument, named after the arg's key. String args use the |
| 251 | + target file's extension (for syntax highlighting); non-string args become |
| 252 | + JSON blobs. So a `search_replace` call produces `old_string.ts` and |
| 253 | + `new_string.ts`; a `write_file` call produces `content.ts` and |
| 254 | + `description.ts`; an `edit_file` call produces `content.ts` and |
| 255 | + `instructions.ts`. |
0 commit comments