Skip to content

feat(checks): opt-in progress logging for scenario/check execution (#… - #2607

Open
demoiverrakada wants to merge 3 commits into
Giskard-AI:mainfrom
demoiverrakada:feature/2421-verbose-suite
Open

feat(checks): opt-in progress logging for scenario/check execution (#…#2607
demoiverrakada wants to merge 3 commits into
Giskard-AI:mainfrom
demoiverrakada:feature/2421-verbose-suite

Conversation

@demoiverrakada

Copy link
Copy Markdown

…2421)

Description

Adds opt-in, per-step progress logging to the scenario runner so long Suite.run()
executions surface which scenario and checks are running — useful in CI, scripts, and
notebooks where the rich progress bar doesn't render.

Suite.run() already exposes a verbose flag + _SuiteProgress bar for scenario-level
progress; this closes the check-level gap, where ScenarioRunner._run_once previously
emitted nothing.

Changes

  • scenarios/runner.py: module logger; INFO logs at step start (scenario name + step
    index + check identities) and completion (pass/fail). check.name falls back to
    check.kind when unset.
  • tests/scenarios/test_runner_logging.py: covers logs-at-INFO and quiet-by-default.

Design

  • Opt-in via standard logging levels — silent by default; enable with
    logging.getLogger("giskard").setLevel(logging.INFO). No new API, backward-compatible
    (uses the Python-logging option suggested in the issue).
  • Async-safe — every log line includes the scenario name, so interleaved output under
    parallel=True stays readable.
  • Scenario-level rich bar left untouched.

Testing — full giskard-checks suite (754 passed), whole monorepo (1,268 passed,
0 failed), serial + parallel=True verified manually, make check clean.

Developed with AI assistance (human-in-the-loop; I reviewed and verified all changes).

Related Issue

Type of Change

  • 📚 Examples / docs / tutorials / dependencies update
  • 🔧 Bug fix (non-breaking change which fixes an issue)
  • [x ] 🥂 Improvement (non-breaking change which improves an existing feature)
  • 🚀 New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to change)
  • 🔐 Security fix

Coding agents

Autonomous agents with no human in the loop must read AUTONOMOUS.md before opening a PR.

PR title: agent-opened PRs must end the title with 🤖🤖🤖🤖 (exactly four robot emojis). Do not omit — that suffix is how the expedited agent PR workflow picks up the PR.

Checklist

  • [ x] I've read the CODE_OF_CONDUCT.md document.
  • [ x] I've read the CONTRIBUTING.md guide.
  • [ x] I've written tests for all new methods and classes that I created.
  • I've written the docstring in NumPy format for all the methods and classes that I created or modified.
  • I've updated the uv.lock running uv lock (only applicable when pyproject.toml has been
    modified)

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces opt-in progress logging at the INFO level for the scenario runner, allowing callers to track step-by-step execution and check status. It also adds a comprehensive test suite to verify this behavior and ensure backward compatibility. The feedback suggests wrapping the logging calls and string formatting inside if logger.isEnabledFor(logging.INFO): checks to prevent unnecessary overhead when logging is disabled.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread libs/giskard-checks/src/giskard/checks/scenarios/runner.py Outdated
Comment thread libs/giskard-checks/src/giskard/checks/scenarios/runner.py Outdated
davidberenstein1957 and others added 2 commits July 29, 2026 16:55
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@davidberenstein1957

Copy link
Copy Markdown
Member

@demoiverrakada, thank for the contribution! Would you be able to resolve the conflict?

@davidberenstein1957 davidberenstein1957 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Ready to merge pending CI.

@davidberenstein1957
davidberenstein1957 enabled auto-merge (squash) August 5, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

Feature request: optional verbosity during Suite.run (per check / per generation)

2 participants