Skip to content

fix(speed_benchmark): align benchmarks with the Reactive_E2E contract (#92)#96

Merged
riita10069 merged 1 commit into
autowarefoundation:mainfrom
gcordova10:fix/benchmarks-post-refactor
Jun 30, 2026
Merged

fix(speed_benchmark): align benchmarks with the Reactive_E2E contract (#92)#96
riita10069 merged 1 commit into
autowarefoundation:mainfrom
gcordova10:fix/benchmarks-post-refactor

Conversation

@gcordova10

Copy link
Copy Markdown
Contributor

Problem

Model/speed_benchmark/speed_benchmark.py and planner_benchmark.py still assume
the pre-refactor API and crash on the current main (post Reactive_E2E refactor).

Fix

  • speed_benchmark.py
    • AutoE2E no longer accepts fusion_mode (concat / cross_attn and the knob were
      removed; BEV is the only multi-view fusion) → construct with
      view_fusion_kwargs={"bev_h": 8, "bev_w": 8} and drop the fusion_mode sweep.
    • AutoE2E.forward now takes map_input and returns the trajectory tensor
      directly (the (loss/traj, ego_hidden, future) 3-tuple was removed) → feed a
      map image and stop unpacking a tuple.
  • planner_benchmark.py
    • gru was removed from PLANNER_REGISTRY → benchmark flow_matching vs bezier.
    • planners now return the trajectory directly, so read out (not out[0]) for the
      smoothness metric.

Verification

  • planner_benchmark.py runs end-to-end: bezier jerk ~5e-6 vs flow_matching ~3.4
    (the Bernstein-basis smoothness guarantee holds, as expected).
  • speed_benchmark.py forward path validated on CPU (is_pretrained=False).
  • ruff + mypy clean.

Closes #92.

…autowarefoundation#92)

The benchmark harnesses still assumed the pre-refactor API and crash on
the current main:

- AutoE2E no longer accepts `fusion_mode` (concat/cross_attn and the knob
  were removed; BEV is the only multi-view fusion) -> construct with
  view_fusion_kwargs={bev_h,bev_w} and drop the fusion_mode sweep.
- AutoE2E.forward now takes `map_input` and returns the trajectory tensor
  directly (the (loss/traj, ego_hidden, future) 3-tuple was removed) ->
  feed a map image and stop unpacking a tuple.
- planner_benchmark: gru was removed from PLANNER_REGISTRY -> benchmark
  flow_matching vs bezier; planners now return the trajectory directly,
  so read `out` (not out[0]) for the smoothness metric.

planner_benchmark verified end-to-end (bezier jerk ~5e-6 vs flow_matching
~3.4); speed_benchmark forward path validated on CPU. ruff + mypy clean.

Signed-off-by: GABRIELA CORDOVA <100548769@alumnos.uc3m.es>
@gcordova10

Copy link
Copy Markdown
Contributor Author

CI green. The two harnesses under Model/speed_benchmark/ were broken against current main after the Reactive_E2E refactor; this aligns them with the live contract:

  • speed_benchmark.py: AutoE2E no longer accepts fusion_mode (concat/cross_attn and the knob were removed; BEV is the only multi-view fusion) → construct with view_fusion_kwargs={bev_h, bev_w}. forward now takes map_input and returns the trajectory tensor directly (the (loss/traj, ego_hidden, future) 3-tuple was removed) → feed a map image, stop unpacking.
  • planner_benchmark.py: gru was removed from PLANNER_REGISTRY → benchmark flow_matching vs bezier; planners return the trajectory directly, so read out (not out[0]) for the smoothness metric.

Verified end-to-end (planner_benchmark: bezier jerk ~5e-6 vs flow_matching ~3.4 — the Bernstein-basis smoothness guarantee holds); speed_benchmark forward path validated on CPU. ruff + mypy clean. Closes #92.

@riita10069 riita10069 merged commit a47d0bc into autowarefoundation:main Jun 30, 2026
2 checks passed
@gcordova10 gcordova10 deleted the fix/benchmarks-post-refactor branch June 30, 2026 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Speed benchmarking still tries to benchmark against modules that are not present

2 participants