@@ -45,16 +45,12 @@ serving, rate limits, and stochastic generation can affect reruns.
4545
4646## Evaluation and reproduction
4747
48- Evaluation is conducted with
49- [ AgentZero] ( https://github.com/agentscope-ai/AgentZero ) , our open-source
50- Harbor-based evaluation framework, so benchmark-specific adapters, task
51- definitions, verifiers, and generated runs remain separate from the Scroll
52- implementation.
53-
54- AgentZero pins this repository as a Git submodule and records the exact QwenPaw
55- commit used by an evaluation. It provides the Harbor workflows for
56- LongMemEval, BEAM, RULER, and LOCA, including environment construction,
57- parallel trials, traces, and scoring.
48+ Implementations for reproducing the reported results are available in
49+ [ AgentZero] ( https://github.com/agentscope-ai/AgentZero ) . AgentZero uses Harbor
50+ and pins this repository as a Git submodule, recording the exact QwenPaw commit
51+ used by each evaluation. It provides reproduction workflows for LongMemEval,
52+ BEAM, RULER, and LOCA, including environment construction, parallel trials,
53+ traces, and scoring.
5854
5955``` bash
6056git clone --recurse-submodules https://github.com/agentscope-ai/AgentZero.git
@@ -66,42 +62,48 @@ source .venv/bin/activate
6662uv pip install " harbor==0.18.0" " ijson>=3.3.0"
6763```
6864
69- Build the pinned Scroll/QwenPaw wheel, then generate a small LongMemEval task
70- set:
65+ Build the pinned Scroll/QwenPaw wheel:
7166
7267``` bash
7368uv build --project qwenpaw --wheel --out-dir dist
7469export QWENPAW_WHEEL=" $( find " $PWD /dist" -name ' qwenpaw-*.whl' -print -quit) "
7570export PYTHONPATH=" $PWD ${PYTHONPATH: +: $PYTHONPATH } "
76-
77- python scripts/download_longmemeval_data.py --dataset oracle
78- python benchmarks/longmemeval/generate.py \
79- benchmarks/longmemeval/data/longmemeval_oracle.json \
80- --split smoke \
81- --output local-tasks/longmemeval \
82- --limit 3
8371```
8472
85- Supply your own evaluated-model and judge API keys, base URLs, and model
86- identifiers, then launch Harbor:
73+ Choose a benchmark (` longmemeval ` , ` beam ` , ` ruler ` , or ` loca ` ) and prepare its
74+ task packages using the corresponding AgentZero instructions. Then select its
75+ task path and adapter:
76+
77+ | Benchmark | Task path | Adapter |
78+ | --- | --- | --- |
79+ | LongMemEval | ` local-tasks/longmemeval/{split} ` | ` adapters.qwenpaw.longmemeval:QwenPawLongMemEvalAgent ` |
80+ | BEAM | ` local-tasks/beam/{task} ` | ` adapters.qwenpaw.beam:QwenPawBeamAgent ` |
81+ | RULER | ` local-tasks/ruler ` | ` adapters.qwenpaw.ruler:QwenPawRulerAgent ` |
82+ | LOCA | ` local-tasks/loca/{task} ` | ` adapters.qwenpaw.loca:QwenPawLOCAAgent ` |
83+
84+ Supply your own model API key, base URL, and model identifier, replace the
85+ placeholders below, and launch Harbor:
8786
8887``` bash
88+ export BENCHMARK=YOUR_BENCHMARK
89+ export TASK_PATH=YOUR_TASK_PATH
90+ export ADAPTER=YOUR_ADAPTER
91+
8992harbor run \
90- --job-name longmemeval- scroll-smoke \
91- -p local-tasks/longmemeval/smoke \
92- -a adapters.qwenpaw.longmemeval:QwenPawLongMemEvalAgent \
93+ --job-name " scroll-${BENCHMARK} " \
94+ -p " $TASK_PATH " \
95+ -a " $ADAPTER " \
9396 -m YOUR_PROVIDER_ID/YOUR_MODEL_ID \
9497 --ae QWENPAW_WHEEL=" $QWENPAW_WHEEL " \
9598 --ae QWENPAW_MODEL_API_KEY=" $QWENPAW_MODEL_API_KEY " \
9699 --ae QWENPAW_MODEL_BASE_URL=" $QWENPAW_MODEL_BASE_URL " \
97- --ve LONGMEMEVAL_JUDGE_API_KEY=" $LONGMEMEVAL_JUDGE_API_KEY " \
98- --ve LONGMEMEVAL_JUDGE_BASE_URL=" $LONGMEMEVAL_JUDGE_BASE_URL " \
99- --ve LONGMEMEVAL_JUDGE_MODEL=" $LONGMEMEVAL_JUDGE_MODEL " \
100- -n 3
100+ -n 8
101101```
102102
103- See the AgentZero README for Oracle validation, full S/M runs, other
104- benchmarks, concurrency guidance, and result inspection.
103+ For judged benchmarks, also pass the benchmark-specific judge API key, base
104+ URL, and model identifier documented in AgentZero. See the AgentZero README
105+ for task generation, Oracle validation, concurrency guidance, and result
106+ inspection.
105107
106108## License
107109
0 commit comments