Skip to content

fix(backtest): keep excess_return consistent with corrected benchmark_return - #1058

Merged
warren618 merged 1 commit into
HKUDS:mainfrom
Shizoqua:fix/excess-return-benchmark-consistency
Aug 11, 2026
Merged

fix(backtest): keep excess_return consistent with corrected benchmark_return#1058
warren618 merged 1 commit into
HKUDS:mainfrom
Shizoqua:fix/excess-return-benchmark-consistency

Conversation

@Shizoqua

Copy link
Copy Markdown
Contributor

Summary

  • BaseEngine.run_backtest() corrects the displayed benchmark_return once an external benchmark's price series contains a non-positive-prior-price bar, but never recomputes excess_return from that same corrected value
  • Recomputes excess_return right where benchmark_return is already corrected, so the two fields remain consistent for this corrected benchmark path

Why

Issue #872 replaced the fragile pct_change-based return calculation with a safe price relative (buy_and_hold_return) for the displayed benchmark_return, after finding it could report numbers like +39,560% where the true return was -42.7%. calc_metrics() has its own internal re-derivation of the same total, still using the exact (1 + bench_ret).prod() - 1 pattern #872 found unsafe. base.py corrects the displayed field but never re-derives excess_return from the corrected value, so the metrics dict can become self-contradictory: benchmark_return reads correctly while excess_return implies a different number, both in the same JSON output the backtest tool returns to the LLM.

I audited every benchmark-derived metric in calc_metrics() (benchmark_return, excess_return, information_ratio, tracking_error, benchmark_beta). Only excess_return is affected. The other three are computed from the per-bar bench_ret series directly, which is already the corrected, #872-safe series by the time it reaches calc_metrics().

Changes

  • agent/backtest/engines/base.py: after the existing benchmark_metadata override, excess_return is recomputed as total_return minus the corrected benchmark_return, but only when an external benchmark was actually resolved. The internal-basket-benchmark path (no config["benchmark"]) is untouched.
  • agent/tests/test_benchmark_excess_return_consistency.py: two new regression tests. One runs the real production path (ChinaAEngine.run_backtest, real calc_metrics(), only the network-dependent resolve_benchmark() fetch is stubbed) with a benchmark price series crossing a non-positive-prior-price bar, and asserts total_return minus excess_return equals benchmark_return. The other is a control confirming the fix does not touch excess_return when no external benchmark is configured.

Test Plan

  • Confirmed the load-bearing regression test fails on the pre-fix implementation with a real numeric assertion failure (excess_return read 1.0 instead of 0.5), and passes with the fix
  • pytest tests/ -k "engine or backtest or benchmark or metrics" -q --ignore=tests/e2e_backtest (813 passed, 6 skipped, pre-existing)
  • black --check on both changed files: pre-existing, unrelated formatting issues in base.py confirmed via diff-isolation against the unpatched file (identical content at shifted line numbers); the new test file is clean
  • ruff check on both changed files: all checks passed

Checklist

  • No changes to protected areas (src/agent/, src/session/, src/providers/): this only touches agent/backtest/
  • No hardcoded values
  • Code follows CONTRIBUTING.md guidelines
  • Documentation updated (n/a: internal correctness fix, no public interface change)
  • DCO signed-off

…_return

BaseEngine.run_backtest() overwrites the displayed benchmark_return with
bench_result.total_ret, a price relative computed the HKUDS#872-safe way once
an external benchmark's price series contains a non-positive-prior-price
bar (suspended stock, vendor glitch). excess_return is never recomputed:
it stays derived from calc_metrics()'s internal bench_return, which is
still (1 + bench_ret).prod() - 1, the exact compounded-product pattern
HKUDS#872 already found unsafe and fixed for the displayed field.

The result is a self-contradictory metrics dict: benchmark_return reads
the correct value while excess_return implies a different one, both in
the same JSON output the backtest tool returns.

Recompute excess_return from the same corrected benchmark_return right
where benchmark_metadata already overwrites the other field, so the two
can never disagree again. Only applies when an external benchmark was
actually resolved; the internal-basket-benchmark path (no
config["benchmark"]) is untouched.

Signed-off-by: Lanre Shittu <136805224+Shizoqua@users.noreply.github.com>
@warren618
warren618 merged commit 0578419 into HKUDS:main Aug 11, 2026
5 checks passed
warren618 added a commit that referenced this pull request Aug 11, 2026
#1058 re-derived excess_return from the corrected benchmark total but
left it unrounded, and m.update(benchmark_metadata) had already replaced
the rounded benchmark_return with a raw float. calc_metrics() rounds both
fields to 6 decimals, so the external-benchmark path was emitting a
different precision style for the same keys into JSON and CSV output.

Both fields now round exactly the way calc_metrics() does -- rounding the
result, not the inputs -- so the corrected path and the default path are
indistinguishable in shape.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants