Skip to content

feat(scan): add PostHog telemetry for scan entry points#2600

Draft
kevinmessiaen wants to merge 2 commits into
feat/garak-scan-integrationfrom
cursor/scan-posthog-telemetry-91d2
Draft

feat(scan): add PostHog telemetry for scan entry points#2600
kevinmessiaen wants to merge 2 commits into
feat/garak-scan-integrationfrom
cursor/scan-posthog-telemetry-91d2

Conversation

@kevinmessiaen

Copy link
Copy Markdown
Member

Summary

Adds PostHog telemetry to the three scan entry points in giskard-scan, following the same started/finished event pattern used by suite, scenario, and test-case runners in giskard-checks.

Events

Entry point Started Finished
quality_scan scan_quality_run_started scan_quality_run_finished
vulnerability_scan scan_vulnerability_run_started scan_vulnerability_run_finished
third_party_scan scan_third_party_run_started scan_third_party_run_finished

Each event carries aggregate, non-PII shape properties (generator counts/types, language count, target mode, parallel settings, pass/fail/error counts, duration).

Implementation

  • New _telemetry_props.py mirrors giskard-checks/_telemetry_props.py for scan-specific dimensions.
  • Each scan wraps its run in telemetry_run_context() with giskard_component / giskard_operation tags.
  • Suite-level telemetry from suite.run() continues to fire inside the scan context (nested scopes are supported).

Verification

  • make check — passed
  • make test-unit PACKAGE=giskard-scan — 162 passed, 25 skipped
Open in Web Open in Cursor 

Mirror the suite/scenario telemetry pattern for quality_scan,
vulnerability_scan, and third_party_scan with started/finished
events and aggregate non-PII shape properties.

Co-authored-by: Kevin Messiaen <kevinmessiaen@users.noreply.github.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces telemetry tracking for quality, vulnerability, and third-party scans within giskard-scan, capturing non-identifying execution properties (such as scan kind, generator types, and duration) at the start and end of each run. It also disables telemetry during testing and adds comprehensive unit tests. One issue was identified in generator_type_counts where generator elements can be either classes or instances; calling type(generator).__name__ on a class returns "type", which would corrupt telemetry data. A check should be added to handle both classes and instances correctly.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +11 to +12
def generator_type_counts(generators: list[ScenarioGenerator]) -> dict[str, int]:
return dict(Counter(type(generator).__name__ for generator in generators))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The generators list returned by the registries can contain both generator classes (e.g., HallucinationScenarioGenerator) and generator instances (e.g., GCGInjectionScenarioGenerator()). Calling type(generator).__name__ on a class returns "type" instead of the actual class name, which will corrupt the telemetry data.

We should check if the generator is a class (using isinstance(generator, type)) and use generator.__name__ in that case, falling back to type(generator).__name__ for instances.

def generator_type_counts(generators: list[ScenarioGenerator | type[ScenarioGenerator]]) -> dict[str, int]:
    return dict(
        Counter(
            generator.__name__ if isinstance(generator, type) else type(generator).__name__
            for generator in generators
        )
    )

Use shared scan_run_started/finished events with a scan_type tag
across quality, vulnerability, and third-party scans. Include
languages, group_by, seed, probe ids, and document counts instead
of only aggregate booleans.

Co-authored-by: Kevin Messiaen <kevinmessiaen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

2 participants