iPhoneme Experiment — v4

Three-condition gaze+phoneme benchmark that actually uses the original working components from your Gaze_trigger_ex project rather than reimplementing them from scratch:

• lib/head_tracker.js — the 753-line tested HeadTracker with One-Euro filtering and 5-point calibration
• lib/head_calibration.js — 5-point calibration wizard
• lib/mouth_calibration.js — mouth open/close threshold calibration
• lib/human/human.esm.js — Human.js / MediaPipe (2.0 MB)
• server/phoneme_server.py — your Whisper.cpp + g2p_en ARPAbet phoneme streaming server

---

What changed in v4

v3	v4
4 conditions (dwell/mouth/blink/iphoneme)	3 conditions (blink removed per feedback)
Dwell had "hover fires click" bug	Dwell has real 800ms timer + visible progress ring
Mouth used v3-invented provider	Mouth uses HeadTracker.on('mouthRatio') — real calibrated threshold
Phoneme detection was Web Speech only	Whisper.cpp via phoneme_server.py (fallback to Web Speech if server down)
Click condition then click task	Click condition → wizard runs → tasks auto-start
Manual questionnaire button	Questionnaire auto-opens after final task
Face tracker had custom built-in	Uses original HeadTracker from Gaze_trigger_ex

---

Running it (full production flow)

Step 1 — Start the phoneme server (for iPhoneme condition only)

cd server
pip install websockets g2p_en
# you also need whisper.cpp built; adjust the --whisper-path as needed
python3 phoneme_server.py --port 8765

Output: [Server] listening on ws://localhost:8765

If you skip this step, iPhoneme automatically falls back to Web Speech API (Chrome only, needs internet).

Step 2 — Start the web app

python3 -m http.server 8080   # Windows: python -m http.server 8080

Open http://localhost:8080/ in Chrome or Edge. Firefox won't work for Web Speech fallback.

Step 3 — Run a session

1. Welcome screen appears → click "Begin Session"
2. Side panel (Alt+E to toggle): enter Participant ID, pick Latin-square order
3. Click a condition button (Dwell, Mouth, or iPhoneme)
4. Wizard runs automatically:
   • Camera permission prompt
   • 5-point head calibration (fixate each dot)
   • Mouth calibration (if mouth condition): keep closed, then wide open
   • Microphone permission (if iphoneme condition)
   • Smoke test: you have 20s to fire one confirmed command to prove the condition works. For dwell, look at the test target for 1s. For mouth, open & close. For iphoneme, say "the".
5. Condition Ready screen → click "Begin Tasks"
6. All 14 tasks run in order with auto-advance and block summaries between each
7. Questionnaire opens automatically after task 14
8. Submit → "Condition Complete" screen → dismiss → repeat for next condition

After all 3 conditions: click "📥 Export JSONL" in the side panel.

---

Development shortcuts (URL flags)

?mouse=1           drive gaze with mouse (skips HeadTracker entirely)
?iphoneme_kbd=1    keys 1-5 as phonemes (skips phoneme server)
?skip_wizard=1     skip the setup wizards
?phoneme_server=ws://other:8765   point iPhoneme at non-default server

Full dev combo (no webcam, no mic, no server needed):

http://localhost:8080/?mouse=1&iphoneme_kbd=1&skip_wizard=1

Keys 1–5 map to the top-5 trigger phonemes from our paper:

• 1 → DH → select
• 2 → HH → back
• 3 → TH → swipe
• 4 (hold) → Y → drag
• 5 → W → modifier

During any trial: X flags as excluded, N skips to next trial.

---

The three conditions

Dwell

• Uses HeadTracker gaze points
• 800ms continuous on-target = select (Tobii default)
• Visible progress ring fills around cursor as dwell approaches fire
• Target deactivates after firing — must exit and re-enter to fire again
• Single command vocabulary (select only) — fundamental limit per Jacob 1990

Mouth

• Uses HeadTracker mouthRatio stream
• Calibrated threshold from HeadTracker.getCalibration().mouth.threshold
• Live mouth ratio indicator bar at bottom-left
• Three commands: quick open-close = select, double = back, hold = drag

iPhoneme

• Connects to ws://localhost:8765 (phoneme_server.py runs Whisper.cpp)
• Falls back to Web Speech API with proxy words if server unreachable
• Five commands: DH=select, HH=back, TH=swipe, Y=drag, W=modifier
• Live "last phoneme" log at bottom showing DH › HH › TH...

---

Latin square

With 3 conditions, the full counterbalanced Williams design has 6 unique orders — your N must be a multiple of 6.

P01: D → M → I
P02: M → I → D
P03: I → D → M
P04: D → I → M
P05: M → D → I
P06: I → M → D

Recommended sample size: N = 12 or N = 18 for CHI-tier power.

---

Task list (14 benchmarks, ~260 trials per condition)

Competitive (10) — all conditions can perform:

1. ISO Tapping (MacKenzie 1992) — 55 trials
2. ISO Dragging (Soukoreff-MacKenzie 2004) — 48 trials
3. Target Crossing (Accot-Zhai 1997) — 72 trials
4. Midas Touch (Stampe-Reingold 1995) — 10 trials with comprehension
5. Hierarchical Menu (Landauer-Nachbar 1985) — 15 trials
6. Text Entry (Soukoreff-MacKenzie 2003) — 15 phrases
7. Continuous Scrolling (Kumar EyePoint 2007) — 9 trials
8. Multi-Step Workflow (Findlater-McGrenere 2004) — 5 scenarios
9. Web Browsing (Kumar 2007) — 10 trials
10. Dual-Task (Salvucci 2005) — 20 trials + continuous secondary

Capability (4) — iPhoneme's richer vocabulary required: 11. Concurrent Multi-Select (Buxton 1986) 12. Time-Critical Reactive (Majaranta 2002) 13. Path Tracing (Buxton 1990) 14. Semantic Disambiguation

---

Analysis

pip3 install numpy scipy
python3 scripts/analyze_logs.py --input data/logs --output data/results

Produces per-condition × per-task stats, Fitts effective-throughput regressions, text-entry WPM/CER/NCER/KSPC, comprehension rates, Friedman + Wilcoxon pairwise with Bonferroni, SUS and NASA-TLX composites.

Verify pipeline with synthetic data:

python3 scripts/smoke_test.py --out data/logs --n 6
python3 scripts/analyze_logs.py --input data/logs --output data/results

---

Directory layout

iphoneme_v4/
├── index.html
├── README.md
│
├── lib/                        # WORKING ORIGINAL COMPONENTS
│   ├── human/human.esm.js      # Human.js / MediaPipe 2.0 MB
│   ├── head_tracker.js         # 753-line HeadTracker module
│   ├── head_calibration.js     # 5-point calibration wizard
│   └── mouth_calibration.js    # mouth open/close threshold
│
├── server/
│   └── phoneme_server.py       # Whisper.cpp + g2p_en WebSocket server
│
├── css/
│   ├── experiment.css
│   ├── tasks.css
│   ├── questionnaire.css
│   ├── v3_additions.css
│   └── v4_additions.css        # wizard, dwell ring, indicators
│
├── js/
│   ├── experiment_app.js       # v4 controller with condition wizard flow
│   ├── condition_engine.js     # 3-condition manager
│   ├── condition_wizard.js     # permissions + calibration + smoke test
│   ├── gaze_cursor.js          # consumes HeadTracker 'point' events
│   │
│   ├── infrastructure/
│   │   ├── timing.js
│   │   ├── logger.js
│   │   └── live_metrics.js
│   │
│   ├── providers/
│   │   ├── base_provider.js
│   │   ├── dwell_provider.js       # HeadTracker + 800ms timer + ring
│   │   ├── mouth_provider.js       # HeadTracker mouthRatio + calibration
│   │   └── iphoneme_provider.js    # WebSocket → phoneme server / Web Speech
│   │
│   ├── tasks/
│   │   ├── base_task.js            # auto-advance + summary + questionnaire launch
│   │   ├── competitive/ (10 tasks)
│   │   └── capability/ (4 tasks)
│   │
│   └── ui/
│       ├── experimenter_panel.js
│       └── questionnaire.js
│
├── config/experiment.default.json
├── scripts/
│   ├── analyze_logs.py
│   └── smoke_test.py
└── data/
    ├── logs/
    └── results/