Skip to content

feat(checks): aggregate LLM token usage on SuiteResult - #2677

Open
davidberenstein1957 wants to merge 3 commits into
mainfrom
feat/suite-token-usage-rollup
Open

feat(checks): aggregate LLM token usage on SuiteResult#2677
davidberenstein1957 wants to merge 3 commits into
mainfrom
feat/suite-token-usage-rollup

Conversation

@davidberenstein1957

@davidberenstein1957 davidberenstein1957 commented Jul 29, 2026

Copy link
Copy Markdown
Member

Summary

  • Add SuiteUsage and a context-scoped collector that wraps LLM generators during Suite.run()
  • Expose aggregated token counts on SuiteResult.usage (defaults to zeros)
  • Print a one-line tokens: in=… out=… total=… footer in print_report() when usage is non-zero

Closes #2671

Test plan

  • make test-unit PACKAGE=giskard-checks (761 passed, 4 skipped)
  • LLM judge suite aggregates known mock usage onto SuiteResult.usage
  • Non-LLM suites keep zero usage and omit the footer line
  • Report footer renders when total_tokens > 0

@davidberenstein1957
feat(scan): add generation progress and verbose quiet mode
97e1217 
Show Rich or INFO progress during generate_suite, add verbose flag to
scan helpers to suppress UI and print_report for CI, and cover quiet
mode plus generation progress in unit tests.

Closes #2670
@davidberenstein1957
chore: apply ruff 0.16 markdown code block formatting
e73c87b 
Reformat README snippets so make check passes with ruff format on CI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

safe for build Scope: Agents Scope: Checks Scope: LLM Scope: Scan

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(checks): aggregate LLM token usage on SuiteResult

1 participant

@davidberenstein1957