Perfect short clips into ideal XTTS references (refine.py)

[sinhaankur]

The idea

XTTS-v2 is zero-shot — it clones from ~6 seconds of clean speech. So when you only have a little audio of a loved one, the win isn't gathering more audio; it's making those few seconds immaculate and handing XTTS exactly the reference it wants.

What's new

refine.py — turns a short, imperfect clip into the best possible clone reference:

• Scores every segment — SNR, voiced-ratio, clipping, length.
• Picks the single cleanest contiguous window (clean beats long; never blends two speakers).
• Renders 24 kHz mono, gently normalized — XTTS's sweet spot, not broadcast loudnorm (which dulls timbre).
• Quality report + warnings — you know the sample is good before cloning (only 5.6s…, multiple speakers…).
• Returns JSON for automation; degrades gracefully without Demucs/Whisper.

analyze.py — speaker separation now uses voice timbre (log-mel band energies + pitch, k-means), not 3 fixed pitch bands. A mother and a daughter at the same pitch no longer get merged (the classic failure that poisons a clone). Pure-Python FFT with a numpy fast path; pitch-band fallback retained.

test_refine.py — 7 offline synthetic tests: scoring ranks clean > noisy, clipping penalized, best-window selection, no cross-speaker blending, same-pitch timbre separation, single-speaker stability, FFT correctness.

Verified end-to-end

An 11s three-speaker mix → a 24 kHz mono reference of just the dominant speaker, SNR 12.8 dB, with an honest report. All 7 tests pass.

{ "ok": true, "duration": 5.6, "snr_db": 12.8, "speaker": "A",
  "report": ["reference: 5.6s @ 24 kHz mono", "est. SNR: 12.8 dB (good)",
             "speakers detected: 3 — used 'A'"],
  "warnings": ["only 5.6s — add another clip for a truer voice",
               "multiple speakers — verify the chosen voice"] }

Companion change in cognitive-twin-agent: scripts/clone-voice-from-video.sh now uses refine.py and surfaces the quality report.