What's Changed
- test(eval): fix grader false-negatives on chart-URL and late-column cases by @samcm in #238
- feat(eval): grade through the Codex Responses API directly (Codex subscription) by @samcm in #239
- fix(eval): deterministic per-test sandbox cleanup via owner attribution by @samcm in #237
- ci(release): build and push sandbox image before goreleaser by @samcm in #236
- fix(docker): run server as the credentials owner for non-1000 host UIDs by @samcm in #241
- feat(sandbox): add chartkit charting library by @samcm in #246
- feat(storage): scope uploads by session and return host path by @samcm in #248
- fix: It should use chartkits for the actual charting? by @samcm in #249
- fix(chartkit): harden against visually-broken charts; require network + structured source by @samcm in #251
- feat(networks): devnet data + notes.ethereum.org specs in CLI and Python by @samcm in #250
- docs(storage): surface host_path and session reuse for the storage upload smoke case by @samcm in #253
- feat(evm): EVM execution module — opcode trial, tracing, tx submission against devnet nodes by @qu0b in #252
- [codex] embed network specs in resources by @parithosh in #256
- feat(auth): make the server the sole owner of proxy credentials by @samcm in #258
- search: replace inline content with ref URIs + panda read command by @qu0b in #255
Full Changelog: v0.36.0...v0.37.0
🐼 Release qualification
Full eval all files: 37 questions, 216 runs (phrasings x repeats) against the hosted proxy. Subject opencode-go/deepseek-v4-flash:cli, commit ace9f8e.
| release | pass rate | mean score | mean tokens (correct) |
|---|---|---|---|
| v0.37.0 (this release) | 94% (202/216) | 0.232 | 22,162 |
| v0.37.0-rc.0 | 92% (198/216) | 0.227 | 23,260 |
| v0.36.2-rc.0 | 92% (194/210) | 0.230 | 22,363 |
| v0.36.0 | 89% (176/198) | 0.220 | 24,678 |
| v0.36.0-rc.0 | 87% (173/198) | 0.219 | 24,067 |
| v0.35.0 | 87% (173/198) | 0.216 | 22,956 |
| v0.35.0-rc.0 | 88% (174/198) | 0.219 | 22,182 |
| v0.34.0 | 85% (169/198) | 0.209 | 23,915 |
| v0.34.0-rc.0 | 84% (167/198) | 0.209 | 22,725 |
Tokens per correct run: p50 19,116 · p90 32,706 · p99 57,458 · 23,606 per solve (all spend / correct answers).
Efficiency vs v0.37.0-rc.0: 0.240 — the harden-loop score with the previous release as the frozen token reference. 0.234 would be this pass rate at identical per-question cost; higher is leaner, lower is heavier.
📊 Full report — per-run matrix, category breakdowns, token distributions, every grader reason.
Per-question changes vs v0.37.0-rc.0
| question | v0.37.0-rc.0 | v0.37.0 | |
|---|---|---|---|
blob_count_stats |
6/6 | 5/6 | 🔻 regressed |
cbt_table_bounds |
3/6 | 4/6 | 🟢 improved |
chartkit_default_entity_bar |
5/6 | 6/6 | 🟢 improved |
devnet_error_logs |
6/6 | 5/6 | 🔻 regressed |
parent_distance_orphans |
4/6 | 5/6 | 🟢 improved |
session_data_analysis |
5/6 | 6/6 | 🟢 improved |
testnet_status |
5/6 | 6/6 | 🟢 improved |
validator_count |
5/6 | 6/6 | 🟢 improved |
Categories with failures
| category | questions | pass rate | median tokens (correct) |
|---|---|---|---|
blobs |
4 | 18/24 (75%) | 28,362 |
timing |
9 | 47/54 (87%) | 20,283 |
execution |
6 | 30/33 (91%) | 23,764 |
networks |
8 | 42/45 (93%) | 17,644 |
clickhouse |
27 | 146/156 (94%) | 19,286 |
visualization |
7 | 40/42 (95%) | 22,260 |
blocks |
14 | 81/84 (96%) | 19,364 |
Failed runs
| question | failed | sample grader reason |
|---|---|---|
au_block_arrivals_chart |
2/6 | The agent correctly filtered the data for Australian observers on 8 June 2026, built a chart, and provided the storage URL. However, the age |
blob_count_stats |
1/6 | The agent reported an average of 3.13 and a median (P50) of 2. While the average is within the 40% tolerance of the expected ~4.7, the repor |
cbt_table_bounds |
2/6 | The agent used a full-table MIN/MAX SQL scan to determine the date range instead of sourcing the answer from the pipeline's coverage/bounds |
devnet_error_logs |
1/6 | The agent discovered active devnets and attempted to query the OTel logs in ClickHouse, but the query failed with a primary key error. The o |
engine_getblobs_blob_scaling |
1/6 | The output fails to provide concrete per-client numbers for all execution clients, only giving a vague bound ('sub-50ms') for the fastest ti |
engine_newpayload_el_comparison |
1/6 | The agent correctly reports per-execution-client durations from a real query, with values within the acceptable 2x range and correct fastest |
late_data_columns |
4/6 | string |
mempool_observer_coverage |
1/6 | The output reports 1 distinct client, which falls outside the accepted range of 19-23 clients specified in the rubric, even though the total |
parent_distance_orphans |
1/6 | The agent failed to report the full parent distance distribution, completely missing the ~221,000 blocks at distance 1 (which make up 99.6% |
🔭 Langfuse traces (216 runs; ⚠️ = failed)
empty_blob_submittersempty_blob_submittersempty_blob_submittersempty_blob_submittersempty_blob_submittersempty_blob_submittersengine_getblobs_blob_scalingengine_getblobs_blob_scalingengine_getblobs_blob_scalingengine_getblobs_blob_scalingengine_getblobs_blob_scaling⚠️ engine_getblobs_blob_scalingblob_count_statsblob_count_statsblob_count_statsblob_count_stats⚠️ blob_count_statsblob_count_statslate_data_columnslate_data_columns⚠️ late_data_columnslate_data_columns⚠️ late_data_columns⚠️ late_data_columns⚠️ block_size_vs_propagationblock_size_vs_propagationblock_size_vs_propagationblock_size_vs_propagationblock_size_vs_propagationblock_size_vs_propagationparent_distance_orphansparent_distance_orphansparent_distance_orphansparent_distance_orphansparent_distance_orphans⚠️ parent_distance_orphansprecompile_usageprecompile_usageprecompile_usageprecompile_usageprecompile_usageprecompile_usageengine_newpayload_el_comparisonengine_newpayload_el_comparisonengine_newpayload_el_comparison⚠️ engine_newpayload_el_comparisonengine_newpayload_el_comparisonengine_newpayload_el_comparisonmempool_observer_coverage⚠️ mempool_observer_coveragemempool_observer_coveragemempool_observer_coveragemempool_observer_coveragemempool_observer_coveragemainnet_average_base_fee_per_gas_jun_10_2026mainnet_average_base_fee_per_gas_jun_10_2026mainnet_average_base_fee_per_gas_jun_10_2026mev_timing_gamesmev_timing_gamesmev_timing_gamesmev_timing_gamesmev_timing_gamesmev_timing_gamesmev_relay_sharemev_relay_sharemev_relay_sharemev_relay_sharemev_relay_sharemev_relay_sharemev_max_value_blockmev_max_value_blockmev_max_value_blockmev_max_value_blockmev_max_value_blockmev_max_value_blockdec4_relay_circuit_breakerdec4_relay_circuit_breakerdec4_relay_circuit_breakerdec4_relay_circuit_breakerdec4_relay_circuit_breakerdec4_relay_circuit_breakersession_data_analysissession_data_analysissession_data_analysissession_data_analysissession_data_analysissession_data_analysisblock_timing_analysisblock_timing_analysisblock_timing_analysisblock_timing_analysisblock_timing_analysisblock_timing_analysisattestation_volume_analysisattestation_volume_analysisattestation_volume_analysisattestation_volume_analysisattestation_volume_analysisattestation_volume_analysischartkit_default_entity_barchartkit_default_entity_barchartkit_default_entity_barchartkit_default_entity_barchartkit_default_entity_barchartkit_default_entity_bardevnets_runningdevnets_runningdevnets_runningdevnets_runningdevnets_runningdevnets_runningtestnet_statustestnet_statustestnet_statustestnet_statustestnet_statustestnet_statusdevnet_error_logsdevnet_error_logsdevnet_error_logs⚠️ devnet_error_logsdevnet_error_logsdevnet_error_logscbt_table_boundscbt_table_bounds⚠️ cbt_table_boundscbt_table_bounds⚠️ cbt_table_boundscbt_table_boundsforky_node_coverageforky_node_coverageforky_node_coverageforky_node_coverageforky_node_coverageforky_node_coveragetracoor_node_coveragetracoor_node_coveragetracoor_node_coveragetracoor_node_coveragetracoor_node_coveragetracoor_node_coveragemainnet_xatu_top_peer_count_nodemainnet_xatu_top_peer_count_nodemainnet_xatu_top_peer_count_nodemainnet_block_arrival_p50mainnet_block_arrival_p50mainnet_block_arrival_p50mainnet_block_arrival_p50mainnet_block_arrival_p50mainnet_block_arrival_p50list_datasourceslist_datasourceslist_datasourceslist_datasourceslist_datasourceslist_datasourcesblock_count_24hblock_count_24hblock_count_24hblock_count_24hblock_count_24hblock_count_24hmissed_slots_24hmissed_slots_24hmissed_slots_24hmissed_slots_24hmissed_slots_24hmissed_slots_24hchartkit_default_arrival_distributionchartkit_default_arrival_distributionchartkit_default_arrival_distributionchartkit_default_arrival_distributionchartkit_default_arrival_distributionchartkit_default_arrival_distributionstorage_upload_session_scopedstorage_upload_session_scopedstorage_upload_session_scopedstorage_upload_session_scopedstorage_upload_session_scopedstorage_upload_session_scopedvalidator_countvalidator_countvalidator_countvalidator_countvalidator_countvalidator_countvalidator_head_accuracyvalidator_head_accuracyvalidator_head_accuracyvalidator_head_accuracyvalidator_head_accuracyvalidator_head_accuracysepolia_validator_lookupsepolia_validator_lookupsepolia_validator_lookupsepolia_validator_lookupsepolia_validator_lookupsepolia_validator_lookuphead_vote_correctness_24hhead_vote_correctness_24hhead_vote_correctness_24hhead_vote_correctness_24hhead_vote_correctness_24hhead_vote_correctness_24hblock_time_histogramblock_time_histogramblock_time_histogramblock_time_histogramblock_time_histogramblock_time_histogramau_block_arrivals_chartau_block_arrivals_chart⚠️ au_block_arrivals_chartau_block_arrivals_chartau_block_arrivals_chart⚠️ au_block_arrivals_chart
Small headline-score moves are noise — the per-question flips are the signal. History comes from prior releases' eval-qualification.json assets.
Contributors
samcm, parithosh, and qu0b