Skip to content

Commit 3e44840

Browse files
fix(agent-forge): update reviewer guide and final proof pack with latest VM proof details
- Enhanced `AGENTFORGE-REVIEWER-GUIDE.md` to include the latest VM proof results from 2026-05-03, detailing the status of local and Tier 2/Tier 4 evaluations. - Revised `FINAL-PROOF-PACK.md` to document the latest VM artifacts and clarify the distinction between checked-in and VM-only results, emphasizing the need to attach these artifacts to the final submission. - Added instructions for running commands within the OpenEMR container to ensure proper environment configuration. These updates improve the clarity and completeness of the documentation, ensuring accurate representation of the latest evaluation results and submission requirements.
1 parent b2d03a7 commit 3e44840

2 files changed

Lines changed: 31 additions & 2 deletions

File tree

AGENTFORGE-REVIEWER-GUIDE.md

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,15 @@ Current checked-in proof snapshot:
7777
| Deployed latency trace | VM artifact `/root/repos/openemr/agent-forge/eval-results/deployed-latency-trace-20260503-190443.json`; `agent-forge/docs/operations/LATENCY-RESULTS.md` | A1c 20/20, p95 `3212 ms`; visit briefing 20/20, p95 `8309 ms`; both under `10000 ms` demo budget |
7878
| Deployed browser proof pack | `agent-forge/docs/submission/browser-proof/` | Four attached browser proof screenshots for A1c trend, visit briefing, missing microalbumin, and clinical-advice refusal |
7979

80-
Additional later VM proof was captured after the checked-in Tier 2/Tier 4 summaries, but the corresponding JSON files are not present in this checkout. The checked-in summaries above are the local artifacts a reviewer can inspect directly from the repository.
80+
Latest VM proof supplied on 2026-05-03 after the checked-in summaries:
81+
82+
| Check | Latest VM artifact or result | Status |
83+
| --- | --- | --- |
84+
| Local AgentForge gate from a normal VM shell | `agent-forge/scripts/check-local.sh`; deterministic eval artifact `/root/repos/openemr/agent-forge/eval-results/eval-results-20260503-201548.json` | PASS; PHP syntax, shell syntax, isolated PHPUnit `298 tests / 1547 assertions`, deterministic evals `32 passed, 0 failed`, PHPStan `161/161`, PHPCS no changed AgentForge PHP files |
85+
| Tier 2 live-provider proof from the OpenEMR container | `/var/www/localhost/htdocs/openemr/agent-forge/eval-results/tier2-live-20260503-202550.json` | 14 passed, 0 failed; tokens in/out `5943/2476`; estimated cost `$0.015599`; provider `openai/gpt-5.4-mini` |
86+
| Tier 4 deployed HTTP/session/audit smoke from the VM | `/root/repos/openemr/agent-forge/eval-results/deployed-smoke-20260503-201547.json` | 5 passed, 0 failed; aggregate latency `14734 ms`; audit assertions enabled; includes `tier4_visit_briefing_live_verified`; code version `6769aa908887` |
87+
88+
The checked-in summaries above are the local artifacts a reviewer can inspect directly from the repository. The later VM artifacts should be attached to the final submission packet if possible.
8189

8290
To reproduce the deployed smoke proof from the VM host:
8391

@@ -90,6 +98,8 @@ export AGENTFORGE_DEPLOYED_URL='https://openemr.titleredacted.cc/'
9098
php agent-forge/scripts/run-deployed-smoke.php
9199
```
92100

101+
If the current shell is already inside `agent-forge/`, run `php scripts/run-deployed-smoke.php` instead.
102+
93103
Browser proof request ids from the later deployed UI proof pass: A1c `7cf183f7-5607-403e-9559-e2689a0769aa`, visit briefing `bbbddd92-df71-4835-951b-f14279abe18c`, dosing refusal `ee2fe6c2-56cc-47ac-8731-a3fd885ad9e3`, missing microalbumin `e4ca6da4-9cd9-4222-a9c3-06651098fb49`, and cross-patient refusal `7489b25d-2af1-42d8-9c04-ec7ee3166dbc`.
94104

95105
Browser proof files are stored under `agent-forge/docs/submission/browser-proof/`, with request ids listed in `agent-forge/docs/submission/FINAL-PROOF-PACK.md` and `agent-forge/docs/submission/browser-proof/MANIFEST.md`. The stale/cross-patient conversation boundary is covered by the deployed smoke runner and should return HTTP 403 with `tools_called=[]` and `verifier_result=not_run`.

agent-forge/docs/submission/FINAL-PROOF-PACK.md

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,12 +27,24 @@ The proof rows below now distinguish checked-in repository artifacts from later
2727
2828
Additional later VM-only Tier 2 and Tier 4 runs were captured after the checked-in summaries, but their JSON artifacts are not present in this checkout. The table above is the synchronized repository-local proof set.
2929

30+
Latest VM proof supplied on 2026-05-03:
31+
32+
| Check | Artifact | Result |
33+
| --- | --- | --- |
34+
| Local AgentForge gate from a normal VM shell | `agent-forge/scripts/check-local.sh`; deterministic eval artifact `/root/repos/openemr/agent-forge/eval-results/eval-results-20260503-201548.json` | PASS; PHP syntax, shell syntax, isolated PHPUnit `298 tests / 1547 assertions`, deterministic evals `32 passed, 0 failed`, PHPStan `161/161`, PHPCS no changed AgentForge PHP files |
35+
| Tier 2 live model from OpenEMR container | `/var/www/localhost/htdocs/openemr/agent-forge/eval-results/tier2-live-20260503-202550.json` | 14 passed, 0 failed; tokens in/out `5943/2476`; estimated cost `$0.015599`; provider `openai/gpt-5.4-mini` |
36+
| Tier 4 deployed smoke from VM | `/root/repos/openemr/agent-forge/eval-results/deployed-smoke-20260503-201547.json` | 5 passed, 0 failed; aggregate latency `14734 ms`; audit assertions enabled; includes `tier4_visit_briefing_live_verified`; code version `6769aa908887` |
37+
38+
Attach these VM/container JSON artifacts to the final submission packet if possible; they are referenced here because they are not present in this local checkout.
39+
3040
The checked-in Tier 4 smoke command was run against the deployed app on 2026-05-03:
3141

3242
```sh
3343
php agent-forge/scripts/run-deployed-smoke.php
3444
```
3545

46+
If the current shell is already inside `agent-forge/`, run `php scripts/run-deployed-smoke.php` instead.
47+
3648
## Deployed Browser Proof
3749

3850
Manual browser screenshots are attached under `agent-forge/docs/submission/browser-proof/` for the four deployed UI scenarios below. They were captured on 2026-05-03 against fake patient `900001 / AF-DEMO-900001` only.
@@ -70,7 +82,12 @@ Live model tier:
7082
php agent-forge/scripts/run-tier2-evals.php
7183
```
7284

73-
This requires provider credentials configured out of band.
85+
This requires provider credentials configured out of band. If host PHP reports that Tier 2 is using the fixture provider, run the command inside the OpenEMR container so Docker Compose supplies the configured provider environment:
86+
87+
```sh
88+
cd docker/development-easy
89+
docker compose exec openemr php /var/www/localhost/htdocs/openemr/agent-forge/scripts/run-tier2-evals.php
90+
```
7491

7592
Deployed smoke tier from the VM host:
7693

@@ -83,6 +100,8 @@ export AGENTFORGE_DEPLOYED_URL='https://openemr.titleredacted.cc/'
83100
php agent-forge/scripts/run-deployed-smoke.php
84101
```
85102

103+
If the current shell is already inside `agent-forge/`, use `php scripts/run-deployed-smoke.php`.
104+
86105
Deployed latency trace from the VM host:
87106

88107
```sh

0 commit comments

Comments
 (0)