Skip to content

Commit f22a014

Browse files
authored
test: add reasoning output fixtures
Add public reasoning-output fixture corpus and publish json-from-llm 0.1.3.
1 parent cb027a1 commit f22a014

16 files changed

Lines changed: 169 additions & 3 deletions

CHANGELOG.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,18 @@ All notable changes to this project are documented here. The format follows
44
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project adheres
55
to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
66

7+
## [0.1.3] - 2026-06-05
8+
9+
### Added
10+
11+
- Added a public reasoning-output fixture corpus covering `<think>` /
12+
`<reasoning>` blocks, fenced JSON, prose wrappers, trailing commas, type
13+
expectations and no-JSON failures.
14+
- Added tests that read the published fixture corpus and verify `tryExtractJson`
15+
results.
16+
- Published the `fixtures/` directory in the npm package for downstream parser
17+
and agent-loop tests.
18+
719
## [0.1.2] - 2026-06-04
820

921
### Changed

README.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ const data = extractJson<{ score: number }>(modelOutput);
3939
- **Handles the real wrappers.** Markdown fences (`json` and bare ```), conversational prose before/after, and the JSON sitting bare in the text.
4040
- **String-aware, never corrupts.** The scanner and the trailing-comma repair both respect string contents — a `}` or `,` inside `"a string value"` is left alone.
4141
- **Conservative repair.** Removes trailing commas (the most common malformation); it will never rewrite your data.
42+
- **Fixture-backed edge cases.** Public fixtures cover reasoning tags, fenced JSON, prose wrappers, trailing commas, top-level type expectations and no-JSON failures.
4243
- **Two entry points.** `extractJson` throws on failure; `tryExtractJson` returns `{ found }`.
4344
- **Zero dependencies**, ESM + CJS, fully typed.
4445

@@ -82,6 +83,21 @@ extractJson('[1,2] then the answer {"a":1}', { expect: 'object' }); // { a: 1 }
8283

8384
The low-level pieces (`stripReasoning`, `fencedBlocks`, `balancedSpans`, `removeTrailingCommas`) are exported too.
8485

86+
## Fixture corpus
87+
88+
The package includes a small public corpus under [`fixtures/`](./fixtures):
89+
90+
- `deepseek-thinking-object.txt`
91+
- `gemini-reasoning-array.txt`
92+
- `prose-trailing-commas.txt`
93+
- `expect-object-skips-array.txt`
94+
- `no-json.txt`
95+
- expected `tryExtractJson` outputs under `fixtures/expected/`
96+
97+
The tests read these files directly, so parser changes are checked against
98+
stable, reusable examples. The fixtures are synthetic and safe for public CI:
99+
they contain no prompts, secrets, user data or live provider responses.
100+
85101
## Related
86102

87103
- [`tool-schema`](https://www.npmjs.com/package/tool-schema) — turn a JSON Schema into a provider tool/function schema (define the shape you then extract).

fixtures/README.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Reasoning Output Fixture Corpus
2+
3+
This corpus contains deterministic model-output examples that commonly break
4+
plain `JSON.parse`:
5+
6+
- reasoning/thinking tags with brace-laden prose
7+
- fenced JSON blocks inside conversational text
8+
- trailing commas in otherwise valid JSON
9+
- competing arrays/objects when callers expect a specific top-level type
10+
- negative text with no recoverable JSON
11+
12+
Each `.txt` file has a matching expected JSON file under `fixtures/expected/`.
13+
The fixtures are synthetic and safe for public CI: they contain no prompts,
14+
secrets, user data or live provider responses.
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
<think>
2+
The user asked for a risk score. I might draft {score: maybe 6}, but that is
3+
not JSON and should not be selected.
4+
</think>
5+
6+
Final answer:
7+
```json
8+
{
9+
"score": 8,
10+
"reason": "clear evidence",
11+
"tags": ["portable", "safe"]
12+
}
13+
```
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
First, a quick list: ["draft", "ignore"].
2+
3+
The object you asked for:
4+
{
5+
"city": "Santiago",
6+
"units": "metric"
7+
}
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"found": true,
3+
"expect": "object",
4+
"value": {
5+
"score": 8,
6+
"reason": "clear evidence",
7+
"tags": ["portable", "safe"]
8+
}
9+
}
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"found": true,
3+
"expect": "object",
4+
"value": {
5+
"city": "Santiago",
6+
"units": "metric"
7+
}
8+
}
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"found": true,
3+
"expect": "array",
4+
"value": [
5+
{ "id": "alpha", "score": 0.91 },
6+
{ "id": "beta", "score": 0.72 }
7+
]
8+
}

fixtures/expected/no-json.json

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"found": false,
3+
"expect": "any"
4+
}
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"found": true,
3+
"expect": "object",
4+
"value": {
5+
"providers": ["openai", "anthropic", "gemini"],
6+
"ok": true
7+
}
8+
}

0 commit comments

Comments
 (0)