feat(checks): parallel-safe eval checkpoint and resume

[davidberenstein1957]

Summary

Mid-run eval checkpoints so a killed Suite.run or generate_suite can pick up where it left off. Storage is an append-only JSONL RunStore (manifest.json + events.jsonl), safe under parallel scenario/generator workers.

By default, runs write under .giskard/checkpoints/<fingerprint_hash>/ and resume when that fingerprint already exists. Change the suite identity (or generation knobs) and you get a new hash / fresh store. Closes #2657.

Usage

Default (auto checkpoint + resume)

# Writes to .giskard/checkpoints/<hash>/ ; second call skips finished scenarios
result = await suite.run()

Same for scan entrypoints — kwargs pass through:

from giskard.scan import vulnerability_scan, quality_scan

report = await vulnerability_scan(agent, ..., checkpoint_dir=None)  # auto

Explicit root, or turn off

# Custom root; store is still <root>/<fingerprint_hash>/
await suite.run(checkpoint_dir="./my-ckpts")

# Disable for this call
await suite.run(checkpoint_dir=False)

# Start over in the same fingerprint dir (ignore prior events)
await suite.run(checkpoint_dir="./my-ckpts", resume=False)

# Resume even if the stored fingerprint disagrees (escape hatch)
await suite.run(checkpoint_dir="./my-ckpts", resume="force")

Env knobs

Env	Effect
GISKARD_CHECKPOINT=0	Disable auto checkpointing only. Explicit checkpoint_dir=... still works.
GISKARD_CHECKPOINT_DIR	Override the auto root (default .giskard/checkpoints).
GISKARD_CHECKPOINT_RESUME	0 / false → no resume; force → force mode; otherwise resume on.

# CI / one-off: no shared .giskard resume pollution
GISKARD_CHECKPOINT=0 pytest ...

Generation

from giskard.scan import generate_suite

suite = await generate_suite(
    description="...",
    ...,
    checkpoint_dir=None,  # auto; finished generators skipped on resume
)

Behavior notes

• Granularity (v1): one event per finished scenario (Suite.run) or finished generator (generate_suite). Not step/item-level yet; schema is meant to grow.
• Fingerprint: run phase keys on suite name + stable scenario checkpoint_ids; generate phase keys on description, languages, generators, seed, target mode, max_scenarios. Tags/steps are not in the hash yet — rearranging scenarios that keep the same ids can still resume.
• Parallel: writers share an asyncio lock and flush after each append; result order stays suite order even when execution is concurrent.
• Privacy: events can hold prompts, traces, and model outputs. .giskard/ is gitignored — keep checkpoints local.
• Tests: checks/scan conftest.py set GISKARD_CHECKPOINT=0 so unit tests do not share or resume the default tree.
• Serialization: resumed generate dumps scenarios with exclude={"target"} so callables do not break JSON.

Test plan

• Unit tests for RunStore (append, concurrent writers, fingerprint mismatch / force)
• Suite.run checkpoint + default resume (serial/parallel order preserved)
• generate_suite generator-level resume
• Main CI (lint + unit matrix) green
• Integration jobs need safe for build label if you want those secrets-backed checks too