feat(scan): add PostHog telemetry for scan entry points#2600
feat(scan): add PostHog telemetry for scan entry points#2600kevinmessiaen wants to merge 2 commits into
Conversation
Mirror the suite/scenario telemetry pattern for quality_scan, vulnerability_scan, and third_party_scan with started/finished events and aggregate non-PII shape properties. Co-authored-by: Kevin Messiaen <kevinmessiaen@users.noreply.github.com>
There was a problem hiding this comment.
Code Review
This pull request introduces telemetry tracking for quality, vulnerability, and third-party scans within giskard-scan, capturing non-identifying execution properties (such as scan kind, generator types, and duration) at the start and end of each run. It also disables telemetry during testing and adds comprehensive unit tests. One issue was identified in generator_type_counts where generator elements can be either classes or instances; calling type(generator).__name__ on a class returns "type", which would corrupt telemetry data. A check should be added to handle both classes and instances correctly.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| def generator_type_counts(generators: list[ScenarioGenerator]) -> dict[str, int]: | ||
| return dict(Counter(type(generator).__name__ for generator in generators)) |
There was a problem hiding this comment.
The generators list returned by the registries can contain both generator classes (e.g., HallucinationScenarioGenerator) and generator instances (e.g., GCGInjectionScenarioGenerator()). Calling type(generator).__name__ on a class returns "type" instead of the actual class name, which will corrupt the telemetry data.
We should check if the generator is a class (using isinstance(generator, type)) and use generator.__name__ in that case, falling back to type(generator).__name__ for instances.
def generator_type_counts(generators: list[ScenarioGenerator | type[ScenarioGenerator]]) -> dict[str, int]:
return dict(
Counter(
generator.__name__ if isinstance(generator, type) else type(generator).__name__
for generator in generators
)
)Use shared scan_run_started/finished events with a scan_type tag across quality, vulnerability, and third-party scans. Include languages, group_by, seed, probe ids, and document counts instead of only aggregate booleans. Co-authored-by: Kevin Messiaen <kevinmessiaen@users.noreply.github.com>
Summary
Adds PostHog telemetry to the three scan entry points in
giskard-scan, following the same started/finished event pattern used by suite, scenario, and test-case runners ingiskard-checks.Events
quality_scanscan_quality_run_startedscan_quality_run_finishedvulnerability_scanscan_vulnerability_run_startedscan_vulnerability_run_finishedthird_party_scanscan_third_party_run_startedscan_third_party_run_finishedEach event carries aggregate, non-PII shape properties (generator counts/types, language count, target mode, parallel settings, pass/fail/error counts, duration).
Implementation
_telemetry_props.pymirrorsgiskard-checks/_telemetry_props.pyfor scan-specific dimensions.telemetry_run_context()withgiskard_component/giskard_operationtags.suite.run()continues to fire inside the scan context (nested scopes are supported).Verification
make check— passedmake test-unit PACKAGE=giskard-scan— 162 passed, 25 skipped