feat(cli): eval → curate → SFT loop + unified SFT trainer (tinker/fireworks)

[jeffreysijuntan]

Summary

Adds the eval → curate → SFT loop to the rLLM CLI: run an eval with k samples per task, curate the trajectories by aggregate metrics into an SFT dataset, and fine-tune on it — all from the CLI. Also refactors SFT around a backend-agnostic dispatcher with pluggable backends (tinker + fireworks; verl deferred).

rllm eval math500 --model <m> --attempts 8
rllm dataset from-eval <run_id> --name math500-rft --filter "0 < avg < 1" --select correct
rllm sft math500-rft --backend fireworks --epochs 3

Design doc: design/sft-distillation.md.

What's included

Curation (rllm dataset from-eval)

• rllm/eval/curation.py — load eval run dirs, pool attempts by stable task_id across runs, filter tasks, select trajectories (correct/best/best-n/shortest/all), lazy-load only chosen episodes, emit {"messages": ...} rows with provenance, register via DatasetRegistry.
• rllm/eval/filter_dsl.py — safe per-task boolean DSL over aggregates (avg, best, worst, solved, n, n_correct, and budget-aware pass@k via a name@k rewrite + AST node-whitelist). avg@k is treated as k-invariant.
• --dry-run, task-level train/val holdout.

Unified SFT trainer

• rllm/trainer/sft/ — SFTSpec (backend-agnostic input), SFTBackend ABC (each backend owns its fit()), and AgentSFTTrainer as the dispatcher (mirrors the RL stack's AgentTrainer/launcher seam).
• TinkerSFTBackend (migrated loop) and FireworksSFTBackend(TinkerSFTBackend) — Fireworks provisions through the training-shape path (init_fireworks_infra("sft", …) with a fireworks_infra doc), shares the tinker-cookbook data pipeline, and runs a synchronous pipelined loop.
• rllm sft CLI (registered in main.py); panel shows the backend-resolved model.
• Clean break: removed the old ad-hoc AgentSFTTrainer._train_verl/_train_tinker and the deprecated Tinker SFT trainer/dataset; kept the AgentSFTTrainer name.

Verified

• Fireworks SFT confirmed end-to-end against the live service (Qwen3.5-9B LoRA): provisions via the shape path, trains real steps (forward_backward/optim_step, loss from result.metrics), saves DCP checkpoints, tears down the trainer on exit.
• Curation engine, filter DSL, dispatcher, and CLI: 74 unit tests (GPU-free; the Fireworks provision doc is parsed offline as a regression guard). ruff clean.

Deferred (follow-ups)

• VerlSFTBackend + torchrun launcher (--backend verl currently returns a clear "not wired yet").
• Fireworks trainer/checkpoint reuse (skip the ~7 min re-provision via --keep-trainer/--fireworks-job-id; cross-run resume).
• Default Fireworks model is Qwen3.5-9B because Fireworks ships no 3.5-4B shape; tinker default stays Qwen3.5-4B.

🤖 Generated with Claude Code