Skip to content

AGENTS.md

Latest commit

 

History

History
88 lines (74 loc) · 5.11 KB

AGENTS.md

File metadata and controls

88 lines (74 loc) · 5.11 KB
Raw

Experiments Agent Notes

A workbench for self-contained ML/systems experiments. Each experiment is its own top-level directory with its own containers, harness, and results. Remote: github.com/svankina/experiments.

Read CLAUDE.md first — it has the authoritative host constraints and the full asr-latency command list. NOTES.md is the durable per-experiment findings log. agent.json sets fastMode: true.

Host environment (the constraints that bite)

  • GPU: single RTX 3090 Ti, 24 GB (Ampere sm_86).
  • Docker runs without sudo, but the interactive shell aliases docker → sudo docker. In scripts/non-interactive shells use command docker ... to bypass the alias.
  • GPU-in-Docker needs nvidia-container-toolkit, which is NOT installed by default and requires sudo to set up. You cannot run sudo non-interactively here — ask the user to run the install steps in CLAUDE.md / each README. Verify with command docker run --rm --gpus all nvidia/cuda:12.6.2-base-ubuntu22.04 nvidia-smi.
  • Model weights cache to the host ~/.cache/huggingface (mounted into containers); first runs download multi-GB.

Experiments

Both follow the same shape: per-engine Dockerfile/service in compose.yaml, a shared bench/ harness mounted into every container (so timing code is byte-identical across engines), analyze.py to aggregate, run_all.sh to do the full sweep, and gitignored results/ (JSONL) and audio/. Results files are often root-owned because Docker writes them.

  • asr-latency/ — relative latency + WER of nemotron-speech-streaming-en-0.6b vs parakeet-tdt-0.6b-v2 (both share the nemotron image) vs faster-whisper (small/medium/large-v3). Offline RTF is the primary apples-to-apples metric; only Nemotron streams natively (check the native flag in the streaming table). See asr-latency/README.md, SUMMARY.md (one-table digest), RESULTS.md (full writeup).
  • tts-latency/ — relative latency of MisoTTS (own GPU container) vs the existing local Kokoro server. Kokoro must be running first: /home/svankina/src/voice/kokoro-server.sh → http://127.0.0.1:8880 (the bench container uses host networking to reach the loopback-bound server). See tts-latency/README.md.

Run pattern

cd asr-latency        # or tts-latency
command docker compose build      # build images (no GPU needed)
./run_all.sh                      # fetch audio + full sweep + analysis
# step-by-step examples live in each README

New experiment → add a sibling top-level directory with its own README, Dockerfile(s), compose service, and bench/ harness.

Gotchas (from CLAUDE.md / READMEs / NOTES.md)

  • faster-whisper.transcribe() is lazy (returns a generator) — the harness must consume segments inside the timed region or the measurement is meaningless.
  • NeMo timing needs torch.cuda.synchronize() around the measured region.
  • Nemotron native streaming is version-sensitive; the adapter falls back to a chunked re-decode flagged native=False. Trust the offline numbers; validate streaming before trusting.
  • TTS: upstream MisoTTS hard-codes the gated meta-llama/Llama-3.2-1B tokenizer; the adapter defaults MISOTTS_TOKENIZER=unsloth/Llama-3.2-1B (public mirror). MisoTTS measured RTF was very slow (~4.6) but confirmed running on GPU, not CPU.

Hard-won notes (from past sessions)

  • Inside the engine containers python is not on PATH — use python3 (both the faster-whisper and nemotron images; bit twice).
  • command docker can't be passed to timeout (command is a shell builtin) — use the binary directly, e.g. timeout 600 /usr/bin/docker run ….
  • WER is computed first-party from stored transcripts: offline runs persist each transcript, so WER comes from word-level Levenshtein vs audio/manifest.json references (normalize: lowercase, strip punctuation) — no re-run needed.
  • faster-whisper medium repetition-hallucination: it can transcribe a clip correctly then loop and repeat half the utterance, spiking that clip's WER. A real Whisper failure mode, not a measurement bug.
  • Parakeet (parakeet-tdt-0.6b-v2) cannot stream natively — it's full-context (att_context_style: regular, only valid att_context_size [[-1,-1]]); its conformer_stream_step/streaming_cfg exist only by inheritance. For true streaming pull a streaming checkpoint instead. Consequence: streaming TTFT clusters ~1s for every engine (pseudo window / first-word arrival floored) and is not a model signal — use first-token compute (~45–184 ms) and final-lag as the real metrics.
  • Table conventions (user pref): human-facing tables say "Nemotron 3.5 ASR" / "Parakeet TDT 0.6b-v2" with params as 600M (not 0.6B); the code/CLI engine keys stay nemotron / parakeet.
  • Versions: faster-whisper 1.1.0 (CTranslate2, needs cuDNN 9 on CUDA 12) · NeMo 2.7.3 / torch 2.6+cu124. Parakeet reuses the nemotron image (same NeMo toolkit) — no new image needed.
  • Durable findings also live in Claude memory: ~/.claude/projects/-home-svankina-src-experiments/memory/MEMORY.md.