Add faithfulness check

[josediegorobles]

Adds a Faithfulness check for evaluating whether an answer is grounded in the provided source material.

The check follows the existing LLM judge pattern, uses a structured judge response with a 0-1 faithfulness score, and supports a configurable passing threshold. Source material can be passed directly or resolved from the trace with source_key; answers can also be passed directly or resolved with answer_key.

Includes unit tests for passing and failing cases, threshold handling, trace-based inputs, prompt contents, serialization, and exports.

Validation:

• make test-unit PACKAGE=giskard-checks
• make check

Closes #2368

[gemini-code-assist]

Code Review

This pull request introduces a new LLM-based check, Faithfulness, along with its structured output model FaithfulnessCheckResult, a Jinja2 prompt template, and corresponding unit tests. The check evaluates whether an AI agent's answer faithfully represents the provided source material. The feedback recommends validating that both the answer and source are successfully resolved before running the check to prevent sending unresolved inputs (like None or NoMatch) to the LLM, and suggests adding a unit test to verify this error handling.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

If answer or source cannot be resolved (resulting in None or NoMatch), the check currently proceeds and passes the stringified representation (e.g., "None" or "NoMatch(...)") to the LLM. This leads to incorrect evaluations and wasted API calls.

We should validate that both answer and source are successfully resolved and are not None before proceeding.

    @override
    async def get_inputs(self, trace: Trace[InputType, OutputType]) -> dict[str, Any]:
        answer = provided_or_resolve(
            trace,
            key=self.answer_key,
            value=provide_not_none(self.answer),
        )
        if answer is None or isinstance(answer, NoMatch):
            raise ValueError(
                f"Could not resolve answer from trace using key '{self.answer_key}' "
                "and no direct answer was provided."
            )

        source = self._resolve_source(trace)
        if source is None or isinstance(source, NoMatch):
            raise ValueError(
                f"Could not resolve source from trace using key '{self.source_key}' "
                "and no direct source was provided."
            )

        return {
            "answer": answer,
            "source": self._format_source(source),
            "threshold": self.threshold,
        }

[josediegorobles]

Closing to consolidate efforts. The feedback here is solid and actionable — will revisit in the next iteration if we return to faithfulness evals. For now, focusing on #2555 which has a strategic duplicate scenario to resolve first. Thanks for the thorough review.—

[gemini-code-assist]

It would be beneficial to add a unit test to verify that a ValueError is raised when the answer or source cannot be resolved from the trace.

Suggested change

	assert data["kind"] == "faithfulness"
	assert isinstance(restored, Faithfulness)
	assert restored.answer_key == "trace.last.outputs.answer"
	assert restored.source_key == "trace.last.metadata.source"
	assert restored.threshold == 0.9
	assert data["kind"] == "faithfulness"
	assert isinstance(restored, Faithfulness)
	assert restored.answer_key == "trace.last.outputs.answer"
	assert restored.source_key == "trace.last.metadata.source"
	assert restored.threshold == 0.9
	async def test_unresolved_inputs_raise_value_error() -> None:
	check = Faithfulness(
	answer_key="trace.last.outputs.non_existent",
	source_key="trace.last.metadata.non_existent",
	)
	with pytest.raises(ValueError, match="Could not resolve answer"):
	await check.run(Trace())

[HarperZ9]

This looks directionally aligned with #2368, and the follow-up commit covers the unresolved-input case from the earlier automated review.

One thing I would consider before this lands: preserve source-fragment boundaries when source is a list. Right now _format_source() joins list items with blank lines, which is readable but loses stable source ids. For a faithfulness check this can make later diagnostics harder: a judge may say the answer is faithful, but the result cannot point back to which source fragment supported, contradicted, or failed to support the claim.

A small-compatible change would be to format list sources as labeled blocks, for example:

<SOURCE id="source_0">
...
</SOURCE>

<SOURCE id="source_1">
...
</SOURCE>

Then the prompt can ask the judge to mention source ids in reason when a distortion or unsupported claim is found. You would not need to change the public API, but it gives users a path from scalar score -> actionable evidence location.

The regression test I would add is a two-source case where one fragment supports the answer and another contradicts it, and assert the rendered prompt keeps the two fragments distinguishable. We have hit the same boundary in Telos/Gather-style provenance receipts: a retrieved source is only useful as evidence if the claim-to-source relation remains inspectable after compression.

[davidberenstein1957]

Closing: this overlaps the existing Groundedness check already on main — https://github.com/Giskard-AI/giskard-oss/blob/main/libs/giskard-checks/src/giskard/checks/judges/groundedness.py — which judges whether an answer is supported by context (same job as “faithfulness” here). See also #2368. If you need a distinct claim-level API, please open a new issue describing how it differs from Groundedness.