0001-eval-methodology.md

Two-mode evaluation methodology

Voice-assistant evaluates the model in two modes: Baseline mode (unmodified LFM2.5-Audio-1.5B running as plain ASR) and Fine-tuned mode (fine-tuned model, same ASR-mode behavioral switch). Each mode reports three layered metrics (format compliance, function-name accuracy, argument accuracy under strict equality) on an Eval subset of up to 10 samples per function (41 functions, ~410 samples total; capped lower for rare functions whose test partition has fewer than 10 examples), drawn from a deterministic 5% Test set that is published once (by scripts/prepare_raw_data.py) as the test split of Paulescu/OHF-Voice-audio-20260504, a 95/5 fork of LiquidAI/OHF-Voice-audio-20260504. finetuning/ only ever loads split="train"; evaluation/ only ever loads split="test", so disjointness is enforced by the HF split API and cannot drift over time. The two-mode setup makes the case for fine-tuning empirical (the gap between modes is the headline result); the published train/test split is a direct response to the contamination in liquid-audio-staging's eval_ohf_voice.py, which shuffles the upstream train split with the same seed family used by the trainer's val split.

The original design called for an in-context function-specs prompt in Baseline mode. That turned out to be infeasible: llama-liquid-audio-server (from llama.cpp PR #18641) only accepts a closed set of approved system prompts (the strings in liquid_audio_chat.py's SYSTEM_PROMPTS), and user-message text alongside the audio is ignored once the model is locked into ASR mode. Both modes therefore use system_prompt: "Perform ASR.", the only viable audio-to-text behavioral switch on this runtime. The headline shifts from "fine-tuning improves on prompting" to "fine-tuning unlocks function-call output that prompting alone cannot elicit on this runtime", which is a stronger story given the constraint.

Because the inference runtime mandates "Perform ASR." and that string is a strong pretrained behavioral trigger (the unmodified model uses it to enter verbatim-transcription mode), the fine-tune must train under the same chat shape it will be served under, otherwise the served prompt overrides the fine-tune at inference. We therefore prepend a {"role": "system", "content": "Perform ASR."} ChatMessage to every training sample inside scripts/preprocess_ohf_voice.py's OHFVoiceIterator. Empirically verified 2026-05-12: a 500-step run that omitted the system message during training produced correct function calls in PyTorch (where the chat shape can match the omission) but produced plain transcriptions through the GGUF server. After re-preprocessing with the system message and a 1000-step retrain, the same Q8_0 GGUF served from the same runtime hits 100% / 99.2% / 90.4% on format / function-name / argument accuracy on the 397-sample test subset.

Considered options

• Single-mode eval (fine-tuned only). Rejected: without a baseline, the reader can't distinguish improvement from fine-tuning vs. improvement from prompting, even if the baseline floor is ~0%.
• Single accuracy score. Rejected: format compliance, function-name accuracy, and argument accuracy fail in different ways across the two modes (baseline fails on all three; fine-tuned mostly fails on argument values). Collapsing them hides the story.
• Proportional sampling for the eval subset. Rejected: timer functions (HassDecreaseTimer, HassStartTimer alone) cover ~28% of OHF-Voice and the top-3 functions cover ~37%; proportional sampling would make rare functions like HassRespond (94 total samples) statistically unstable. Per-function stratification at up to 10/function gives stable per-bucket numbers where data exists.
• Reuse staging's eval script as-is. Rejected: that script samples from the train split with the same seed family used by the train/val split, so the headline 95% number from staging is not on a clean held-out set.
• In-context function-specs prompt for baseline. Rejected after empirical test: llama-liquid-audio-server rejects long custom system prompts with "Unsupported system prompt" and silently ignores user-message text when an approved system prompt is in force. Specs injection cannot reach the model through this runtime.