feat(checks): add contains_any and contains_all checks#2597
feat(checks): add contains_any and contains_all checks#2597TheManishCode wants to merge 2 commits into
Conversation
Add ContainsAny/ContainsAll built-in checks for validating that a response contains at least one or all of a list of expected strings, so users no longer need custom FnCheck lambdas for this common case. Closes Giskard-AI#2361
There was a problem hiding this comment.
Code Review
This pull request introduces two new text-matching checks, ContainsAny and ContainsAll, along with comprehensive unit tests and a refactored _format_str helper function. The review feedback highlights an optimization opportunity in both checks, pointing out that the current implementation for computing matched and missing values has
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| formatted_text = _format_str(text, self.normalization_form, self.case_sensitive) | ||
| matched = [ | ||
| v | ||
| for v in values | ||
| if _format_str(v, self.normalization_form, self.case_sensitive) | ||
| in formatted_text | ||
| ] | ||
| missing = [v for v in values if v not in matched] |
There was a problem hiding this comment.
The current implementation of ContainsAny has an missing because it performs a linear lookup (v not in matched) for each element in values. Additionally, it calls _format_str twice per matching element. We can optimize this to a single values once and partitioning them into matched and missing lists directly.
| formatted_text = _format_str(text, self.normalization_form, self.case_sensitive) | |
| matched = [ | |
| v | |
| for v in values | |
| if _format_str(v, self.normalization_form, self.case_sensitive) | |
| in formatted_text | |
| ] | |
| missing = [v for v in values if v not in matched] | |
| formatted_text = _format_str(text, self.normalization_form, self.case_sensitive) | |
| matched = [] | |
| missing = [] | |
| for v in values: | |
| if _format_str(v, self.normalization_form, self.case_sensitive) in formatted_text: | |
| matched.append(v) | |
| else: | |
| missing.append(v) |
| formatted_text = _format_str(text, self.normalization_form, self.case_sensitive) | ||
| matched = [ | ||
| v | ||
| for v in values | ||
| if _format_str(v, self.normalization_form, self.case_sensitive) | ||
| in formatted_text | ||
| ] | ||
| missing = [v for v in values if v not in matched] |
There was a problem hiding this comment.
The current implementation of ContainsAll has an missing because it performs a linear lookup (v not in matched) for each element in values. Additionally, it calls _format_str twice per matching element. We can optimize this to a single values once and partitioning them into matched and missing lists directly.
| formatted_text = _format_str(text, self.normalization_form, self.case_sensitive) | |
| matched = [ | |
| v | |
| for v in values | |
| if _format_str(v, self.normalization_form, self.case_sensitive) | |
| in formatted_text | |
| ] | |
| missing = [v for v in values if v not in matched] | |
| formatted_text = _format_str(text, self.normalization_form, self.case_sensitive) | |
| matched = [] | |
| missing = [] | |
| for v in values: | |
| if _format_str(v, self.normalization_form, self.case_sensitive) in formatted_text: | |
| matched.append(v) | |
| else: | |
| missing.append(v) |
Addresses Gemini Code Assist review feedback: computing matched/missing via list membership was O(N^2) and reformatted each value twice.
|
Hi maintainers 👋 — this PR is ready for review but blocked on two org-level gates that only a maintainer can clear:
In the meantime I've addressed the Gemini Code Assist review feedback (single-pass O(N) partitioning in Could someone approve the workflow run and/or add the |
Summary
ContainsAny/ContainsAllbuilt-in checks for validating that a response contains at least one, or all, of a list of expected strings (e.g. topic coverage), so users no longer need customFnChecklambdas for this.StringMatching/RegexMatchingpattern intext_matching.py: JSONPath extraction (text_key/values_key), Unicode normalization, and configurable case sensitivity._format_str) out ofStringMatchinginto a module-level function so it can be reused without changingRegexMatching's API.Test plan
tests/builtin/test_contains_matching.pycovering pass/fail cases, case sensitivity, JSONPath extraction, Unicode normalization, empty/missing/invalidvalues, and constructor validation errors.ruff format --check/ruff checkclean.basedpyright --level error libs/giskard-checks— 0 errors.pytest libs/giskard-checks -m "not functional"— 751 passed (pre-existingRegoPolicy/celine-regorusfailures unrelated, no Windows wheel for that dependency).Closes #2361