feat(checks): add Trace.steps() helper for step-based grouping#2481
feat(checks): add Trace.steps() helper for step-based grouping#2481harsh21234i wants to merge 6 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces logical step grouping for interactions within a trace. It adds a steps() method to the Trace class to group interactions by a step_index metadata field and updates the ScenarioRunner to automatically tag interactions with these indices during scenario execution. Comprehensive tests were added to verify both the grouping logic and the tagging process. Review feedback suggests simplifying the type signature of the new tagging helper by removing unused generic parameters and optimizing list operations to avoid redundant allocations.
There was a problem hiding this comment.
Code Review
This pull request introduces logical step grouping for interactions within a trace by adding a steps() method to the Trace class and updating the ScenarioRunner to tag interactions with a step_index. Review feedback suggests using a more idiomatic type check for the step_index to strictly ensure it is an integer and raises concerns regarding the O(S^2) performance complexity of the tagging implementation due to repeated list copying in long scenarios.
|
Thanks for your contribution! I will take a look afterward. Quick note: I think it could be nice to have a |
|
I addressed the remaining comments in two parts:
Instead of post-processing I also added coverage for multi-yield interactions within a single step to make sure all generated interactions keep the same step index. |
|
hey @kevinmessiaen can you look into this?? |
Closes #2419
Summary
Trace.steps()to group interactions by logical scenario stepstep_indexmetadata onto interactions at the scenario runner boundarystep_indexas step0Design
Traceitself does not knowScenario.steps, so the grouping signal is recorded at runtime where step boundaries areactually known:
ScenarioRunner.This keeps the architecture aligned with the existing design:
Scenarioowns step structureScenarioRunnermaterializes interactionsTraceremains an immutable history object with helper accessors over recorded dataTesting
uv run -m pytest -q libs/giskard-checks/tests/trace/test_trace.pyuv run ruff check libs/giskard-checks/src/giskard/checks/core/interaction/trace.py libs/giskard-checks/src/giskard/ checks/scenarios/runner.py libs/giskard-checks/tests/trace/test_trace.py