Tauri 2.x desktop dictation app. Multi-cloud transcription, AI-powered text enhancement, and native clipboard paste — including terminal support.
┌─────────────────────────────────────────────────────────────┐
│ Frontend (React 19) │
│ │
│ ┌──────────────────┐ ┌───────────────────────────┐ │
│ │ DictationOverlay │ │ SettingsPanel │ │
│ │ 120×120 always │ │ 900×680 hidden │ │
│ │ on-top, transp. │ │ by default │ │
│ └────────┬─────────┘ └────────────┬──────────────┘ │
│ │ │ │
│ ┌────────┴──────────────────────────────┴──────────────┐ │
│ │ Hooks & Services Layer │ │
│ │ useAudioRecording useSettings useHotkey │ │
│ │ tauriApi.ts │ │
│ └──────────────────────┬───────────────────────────────┘ │
└──────────────────────────┼──────────────────────────────────┘
│ Tauri IPC (invoke / emit)
┌──────────────────────────┼──────────────────────────────────┐
│ Backend (Rust) │
│ │
│ ┌──────────┐ ┌──────────────┐ ┌───────────┐ ┌─────────┐ │
│ │ Audio │ │ Transcription│ │ Reasoning │ │Clipboard│ │
│ │ (cpal) │ │ cloud │ │ (AI post) │ │ (Win32) │ │
│ └──────────┘ └──────────────┘ └───────────┘ └─────────┘ │
│ ┌──────────┐ ┌──────────────┐ │
│ │ Database │ │ Settings │ │
│ │ (SQLite) │ │ (plugin-store│ │
│ └──────────┘ └──────────────┘ │
│ │
│ System Tray · Plugins │
└─────────────────────────────────────────────────────────────┘
Whisperi is split into two processes connected by Tauri's IPC bridge:
- Frontend — React 19 + TypeScript (strict) + Tailwind CSS v4 + shadcn/ui + i18next. Runs in a Webview. Handles all user interaction, audio-level visualization, settings forms, transcription history, and multi-language UI (9 locales).
- Backend — Rust. Owns audio capture, transcription orchestration, AI enhancement, native clipboard access, database persistence, and the system tray. Exposes ~20 Tauri commands that the frontend invokes.
Most dictation tools paste via the OS clipboard + Ctrl+V. This fails in terminal emulators that expect Ctrl+Shift+V or have custom paste semantics. Whisperi detects the foreground window class at paste time and selects the correct keystroke sequence via Win32 SendInput. Nine terminal families are recognized (Windows Terminal, mintty, ConEmu, Alacritty, WezTerm, PuTTY, Hyper, MobaXterm, cmd.exe).
cpal's audio Stream is !Send — it cannot cross thread boundaries. Whisperi solves this by spawning a dedicated recording thread that owns the Stream for its entire lifetime. Shared state (samples buffer, peak level, error slot) is accessed through Arc<Mutex<T>> and Arc<AtomicBool>. The main thread flips the atomic flag to signal stop; the recording thread exits its loop and is joined. Panics inside the recording thread are caught with catch_unwind and surfaced to the UI.
Every dictation flows through a linear pipeline:
Hotkey → Record → WAV Encode → Transcribe → [Enhance] → Save → Paste
Each stage is independently configurable: transcription uses OpenAI / Groq / Mistral / Qwen / OpenRouter; AI enhancement is optional (OpenAI / Anthropic / Gemini / Groq / Qwen / OpenRouter); paste can be toggled off. The pipeline lives in the useAudioRecording hook on the frontend side, calling into Rust commands for each stage. On-device models and executable sidecars are intentionally unsupported.
Two Tauri windows render the same React bundle but show different views based on their window label:
| Window | Size | Traits | Purpose |
|---|---|---|---|
main |
100×100 | always-on-top, transparent, no taskbar, no decorations, position persisted | Floating mic button |
settings |
760×800 | hidden by default, resizable, no decorations, position/size persisted | Full settings panel + history |
Both windows persist their position via tauri-plugin-window-state. The fixed-size overlay skips the plugin's automatic full-state restore and restores only POSITION during setup, preventing saved physical dimensions from compounding when the window crosses mixed-DPI monitors. The resizable settings window restores its full state. The system tray toggles visibility of the settings window. This keeps the overlay minimal and unobtrusive while still providing a full configuration surface.
- Transient state (recording phase, audio level, current transcript) lives in React hooks and resets naturally on component unmount.
- User preferences persist via
tauri-plugin-store(a JSON file), loaded on mount with defaults back-filled for any missing keys. - Transcription history is stored in SQLite (
{app_data}/whisperi.db), queryable with pagination.
There is no global state manager (no Redux, Zustand, etc.). Each concern owns its state through a dedicated hook.
Whisperi is Windows-first. Clipboard read/write, terminal detection, and keystroke simulation all use the Win32 API directly (via the windows crate). This trades cross-platform portability for reliable, low-level control over system interactions that abstraction layers tend to get wrong.
| Module | File(s) | Responsibility |
|---|---|---|
| audio | audio/recorder.rs |
Device enumeration, recording lifecycle, sample-rate negotiation (16k → 44.1k → 48k → default), WAV encoding (16-bit PCM mono), audio-level events |
| transcription | transcription/cloud.rs |
Cloud providers (OpenAI, Groq, Mistral, Qwen, OpenRouter) — multipart HTTP or multimodal chat completions |
| transcription/streaming | transcription/streaming/{mod.rs, audio_pump.rs, providers.rs, realtime_openai_compatible.rs} |
Live mode: WebSocket streaming ASR over the OpenAI Realtime API wire protocol. Online resampler + PCM16 encoder feeds 100ms audio chunks; .completed utterance events emit Tauri events for the frontend to type into the focused window. |
| reasoning | reasoning/openai.rs, anthropic.rs, gemini.rs |
AI text enhancement. OpenAI-compatible (OpenAI, Groq, Qwen, OpenRouter) via Chat Completions; Anthropic via Messages API; Gemini via Generative API |
| clipboard | clipboard/mod.rs |
Win32 clipboard get/set, foreground-window terminal detection, paste via SendInput with terminal-aware key combos |
| database | database/mod.rs, migrations.rs |
SQLite via rusqlite. Single transcriptions table. Auto-migrates on startup. Mutex<Connection> for thread safety |
| settings | commands/settings.rs |
Thin wrapper over tauri-plugin-store — get/set/get-all |
| models | models/mod.rs |
Streaming HTTP download with progress events, atomic file rename, .part temp files |
| commands | commands/audio.rs, app.rs, changelog.rs, clipboard.rs, database.rs, models.rs, reasoning.rs, settings.rs, transcription.rs |
Tauri #[command] handlers — thin wrappers that delegate to domain modules |
| main.rs | main.rs |
Binary entry point, calls whisperi_lib::run() |
| lib.rs | lib.rs |
App entry point: plugin registration, state injection, tray menu, command handler registration |
| Layer | File(s) | Responsibility |
|---|---|---|
| Views | App.tsx |
Window-label router: overlay vs settings |
| Overlay | components/DictationOverlay.tsx |
Mic button, audio-level ring, status text, drag handle, hotkey response, startup checks (version change → What's New, first launch → open settings) |
| Settings | components/settings/* |
Tabbed settings shell (SettingsPanel.tsx) + 7 section components: general (UI language, output language, hotkey, mic, behavior), transcription, enhancement, dictionary, agent, developer, about. Shared ProviderModelSelector for provider/model dropdowns |
| Hooks | hooks/useAudioRecording.ts |
Full dictation pipeline state machine (idle → recording → processing → idle) |
hooks/useSettings.ts |
Load/save all settings from plugin-store with defaults | |
hooks/useHotkey.ts |
Global shortcut registration, tap vs push-to-talk modes | |
| Services | services/tauriApi.ts |
Typed invoke() wrappers for every Rust command, event listeners |
| Config | config/constants.ts, prompts.ts, promptData.json, languageRegistry.json |
Default values, centralized prompt templates, coherent cleanup profiles (internal/language/dictionary policy), agent-name interpolation, and enhancement intensity levels (Light/Standard/Full) with temperature mapping |
| Models | models/modelRegistryData.json, models/dictionary.ts |
Static provider/model registry; custom-dictionary normalization, prompt hints, and deterministic alias correction |
| Utils | utils/sounds.ts, languageSupport.ts |
Web Audio API tone generation (no static assets); language support validation, auto-detect and per-language instruction assembly |
| i18n | i18n/index.ts, i18n/i18next.d.ts, i18n/locales/*.json |
i18next initialization, typed translation keys, 9 locale files (en, zh, ja, ko, de, fr, es, pt, ru) |
| UI Kit | components/ui/* |
shadcn/ui primitives + custom components: StyledSelect, LanguageSelector, ProviderTabs, HotkeyInput, ApiKeyInput, ProviderIcon, Toast, SettingsSection, WhatsNewModal |
Border radii use two semantic CSS custom properties defined in @theme in src/index.css:
| Token | Tailwind Class | Default | Usage |
|---|---|---|---|
--radius-control |
rounded-control |
0.375rem (6px) | Buttons, inputs, dropdowns, toasts, tab bars, nav items |
--radius-inner |
rounded-inner |
0.25rem (4px) | Option rows, tab buttons, kbd keys, search inputs inside dropdowns |
| (built-in) | rounded-full |
50% | Toggles, indicator dots, progress bars, circular buttons |
To adjust corner radii app-wide, change the two --radius-* values in index.css. No component files need editing.
Language codes from languageRegistry.json use locale format (en-US, en-GB). The Rust command layer normalizes these to ISO 639-1 (en) before passing to transcription APIs.
1. User presses hotkey (or clicks overlay)
↓
2. useHotkey fires → useAudioRecording.start()
↓
3. invoke("start_recording") → Rust spawns recording thread
← "audio-level" events emitted at 50 ms intervals
↓
4. User releases hotkey / clicks stop
↓
5. useAudioRecording.stop()
invoke("stop_recording") → Rust joins thread, returns WAV bytes
↓
6. Cloud transcription:
invoke("transcribe_cloud", { audio, provider, api_key, model, ... })
→ Provider receives canonical vocabulary as recognition context where supported
→ Prompt-echo cleanup preserves an exact canonical term the user may have spoken
↓
7. Apply local "Always replace" dictionary aliases using whole-word boundaries
↓
8. Enhancement (optional):
invoke("process_reasoning", { text, provider, model, system_prompt, api_key })
→ Context-aware dictionary aliases are supplied as explicit correction mappings
→ Standard/Full restore sentence and clause punctuation from meaning even when
the raw transcript contains no punctuation
→ English punctuation is normalized to half-width ASCII; Chinese punctuation
is normalized to full-width marks
→ Strip <think>...</think> tags from output (reasoning model artifacts)
↓
9. invoke("save_transcription", { original, processed, method, agent })
↓
10. invoke("paste_text", { text })
→ Rust: clipboard write + terminal detection + SendInput
Live mode streams audio over WebSocket to a cloud ASR provider, typing utterances into the focused window as they complete:
1. User presses hotkey with mode=live
↓
2. useLiveDictation.start() snapshots foreground HWND
↓
3. invoke("start_live_session", { provider, api_key, model, language, dictionary })
→ Rust opens WebSocket to provider
→ OpenAI receives canonical vocabulary in its transcription prompt
→ Providers without prompt support (currently Qwen) omit the field
→ Spawns audio pump tokio task; returns the new `session_id` (u64)
↓
4. cpal feeds samples at 100ms ticks → pump online-resamples to provider rate, encodes PCM16, sends base64 over WS
↓
5. WS receives `.completed` utterance events
→ A capture-order ReorderBuffer (keyed by the provider's `item_id`, ranked on
`input_audio_buffer.committed`/`.speech_started`) holds out-of-order
completions so a short utterance whose transcription finishes before a
longer earlier one is still emitted in spoken order
→ Tauri emits "live-utterance" {{ text, utterance_seq }} payload (in spoken order)
→ Frontend preserves exact canonical terms, applies local "Always replace"
dictionary aliases, accumulates the corrected transcript, and invokes
invoke("type_text_chunk", { text })
→ Rust simulates SendInput keystrokes (with sanitization) into the focused window,
returning the UTF-16 unit count AND the focus target (window + focused control via
GetGUIThreadInfo) it typed into; the frontend records each chunk under its target so
the post-stop swap can scope itself to a single box
On error: Tauri emits "live-error" {{ message, kind }}; on natural close: "live-session-closed" (session_id)
↓
6. User releases hotkey / clicks stop
↓
7. invoke("stop_live_session", { session_id }) → returns ()
→ Rust signals cancel, runs ~800ms soft-flush (commit_utterance + drain), closes WS
→ Final "live-utterance" events may arrive during the soft-flush; the frontend chains them onto its accumulator
↓
8. Enhancement (optional):
Raw transcript → invoke("process_reasoning", ...) → enhanced version
↓
9. If enhanced text differs from raw, replace the live-typed text with the polished
version, SCOPED to the box currently focused (Live types "where you look", so the
transcript may be spread across boxes/windows):
→ Group typed chunks by focus target; for the box focused now, invoke
"swap_typed_text_cmd" { backspace_count, new_text, expected_hwnd, expected_control }
to backspace ONLY that box's characters and retype its polished slice (re-polishing
the slice when the session spanned multiple boxes). swap_typed_text refuses to act
if the focused window OR control has drifted.
→ If the box can't be uniquely identified — web/Electron fields where many boxes share
one render HWND, or focus on a box never typed into — skip the destructive swap and
copy the polished text to the clipboard (set_clipboard_text) with a toast instead
↓
10. invoke("save_transcription", { original: raw, processed: enhanced, method: "live", agent })
App mount → useSettings loads all keys from plugin-store
→ back-fills defaults for any missing keys
→ returns { settings, update(key, value) }
→ App.tsx reads uiLanguage from store, calls i18n.changeLanguage()
→ listens for "settings-changed" events to sync language across windows
User changes a setting → update() writes to store immediately
→ React state updated, component re-renders
→ emits "settings-changed" event (cross-window sync)
customDictionary is stored as DictionaryEntry[], where every entry has a
canonical term, zero or more likely ASR aliases, and a policy of
contextual or always. The loader also accepts the legacy string[] format
and normalizes it in memory, so existing settings remain valid.
- Canonical terms bias buffered providers and OpenAI Live transcription.
alwaysaliases are replaced locally before optional enhancement in both Standard and Live modes.contextualaliases are expressed as explicit mappings in the AI enhancement prompt and are changed only when surrounding text supports the correction.- Exact canonical terms are protected from echo removal. This protection is exact-phrase based, rather than any-token based, so multi-term silence echoes can still be discarded.
Single table in {app_data}/whisperi.db:
CREATE TABLE transcriptions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
original_text TEXT NOT NULL,
processed_text TEXT,
is_processed BOOLEAN DEFAULT 0,
processing_method TEXT DEFAULT 'none',
agent_name TEXT,
error TEXT
);Queried with ORDER BY id DESC LIMIT ? OFFSET ? for paginated history display.
Main Thread (Tauri runtime)
├── Webview (frontend React)
├── Tauri command handlers (async Tokio)
│ ├── HTTP requests (reqwest)
│ ├── Database ops (rusqlite behind Mutex)
│
└── Recording Thread (spawned per session)
├── Owns cpal Stream (!Send)
├── Writes samples to Arc<Mutex<Vec<f32>>>
├── Updates peak level Arc<Mutex<f32>>
└── Exits when AtomicBool flipped to false
Audio Level Emitter Thread (spawned per session)
├── Polls peak level every 50 ms
└── Calls app.emit("audio-level", level)
bun install # install frontend deps
bun run tauri dev # Vite dev server + Tauri (hot reload)
bun run typecheck # TypeScript strict check
cd src-tauri && cargo test # 8 Rust unit tests (audio + database)
cd src-tauri && cargo clippySingle check-and-build job: TypeScript check → Vite build → cargo test → cargo clippy → tauri build → upload NSIS installer artifact
Triggered on version tags (v*). Its Windows release job builds and signs the NSIS/MSI installers via tauri-apps/tauri-action@v0, then publishes them with updater metadata as a GitHub Release.
WinGet submission is deliberately separate from GitHub Actions. Microsoft's open-source enterprise limits classic PATs to eight days, while WinGetCreate does not support fine-grained PATs for cross-owner public-repository contributions. The release workstation uses WinGetCreate's cached OAuth login instead:
wingetcreate token -s
powershell -ExecutionPolicy Bypass -File scripts/submit-winget.ps1 vX.Y.Z -Preview
powershell -ExecutionPolicy Bypass -File scripts/submit-winget.ps1 vX.Y.ZThe submission script resolves the published release through GitHub's public API, verifies there is exactly one x64 NSIS installer, generates the three manifests in a temporary directory, checks the inherited metadata and GitHub asset digest, then invokes wingetcreate submit without putting a token on the command line. The temporary directory is removed on success or failure.
| Crate / Package | Purpose |
|---|---|
tauri 2.x |
App framework, IPC, windows, tray, plugins (autostart, store, updater, etc.) |
cpal 0.15 |
Cross-platform audio capture |
hound 3.5 |
WAV encoding |
reqwest 0.12 |
HTTP client (multipart uploads, streaming downloads) |
rusqlite 0.32 |
SQLite (bundled) |
windows 0.58 |
Win32 API (clipboard, SendInput, window class queries) |
react 19 |
Frontend UI |
tailwindcss 4 |
Styling |
@radix-ui/* |
Accessible UI primitives (via shadcn/ui) |
i18next + react-i18next |
Internationalization (9 locales) |
whisperi/
├── src/ # Frontend
│ ├── App.tsx # Dual-view router
│ ├── main.tsx # React entry point
│ ├── components/
│ │ ├── DictationOverlay.tsx # Floating mic overlay
│ │ ├── SettingsPanel.tsx # Tabbed settings shell
│ │ ├── settings/ # Individual settings sections
│ │ │ ├── GeneralSection.tsx # Language, hotkey, mic, behavior
│ │ │ ├── TranscriptionSection.tsx
│ │ │ ├── AIModelsSection.tsx # Enhancement provider/model
│ │ │ ├── DictionarySection.tsx
│ │ │ ├── AgentSection.tsx
│ │ │ ├── DeveloperSection.tsx
│ │ │ └── AboutSection.tsx
│ │ └── ui/ # Shared UI components + shadcn/ui primitives
│ │ ├── StyledSelect.tsx # Generic styled dropdown (no search)
│ │ ├── LanguageSelector.tsx # Language picker with search + flags
│ │ ├── ApiKeyInput.tsx # Masked API key input
│ │ ├── HotkeyInput.tsx # Key binding capture
│ │ ├── ProviderTabs.tsx # Provider tab bar with sliding indicator
│ │ ├── ProviderIcon.tsx # Provider letter/icon badges
│ │ ├── WhatsNewModal.tsx # Version changelog popup
│ │ └── ... # button, input, toggle, badge, toast
│ ├── i18n/
│ │ ├── index.ts # i18next init, SUPPORTED_LANGUAGES
│ │ ├── i18next.d.ts # Typed translation keys
│ │ └── locales/ # 9 locale JSON files (en, zh, ja, ko, de, fr, es, pt, ru)
│ ├── hooks/
│ │ ├── useAudioRecording.ts # Recording state machine
│ │ ├── useSettings.ts # Persistent settings
│ │ └── useHotkey.ts # Global shortcut binding
│ ├── services/
│ │ └── tauriApi.ts # Typed Tauri command wrappers
│ ├── config/
│ │ ├── constants.ts # App defaults
│ │ ├── prompts.ts # AI prompt assembly (system + language + dictionary)
│ │ ├── promptData.json # System prompt templates (internal, user-visible, chat)
│ │ └── languageRegistry.json # Per-language instructions and punctuation rules
│ ├── models/
│ │ └── modelRegistryData.json # Provider/model catalog
│ └── utils/
│ ├── sounds.ts # Web Audio tone generation
│ └── languageSupport.ts # Language support validation and instruction assembly
│
├── src-tauri/ # Backend
│ ├── src/
│ │ ├── lib.rs # App setup, tray, plugins
│ │ ├── audio/recorder.rs # cpal recording + WAV
│ │ ├── transcription/
│ │ │ └── cloud.rs # Cloud providers
│ │ ├── reasoning/
│ │ │ ├── mod.rs # Dispatch
│ │ │ ├── openai.rs # OpenAI / compatible
│ │ │ ├── anthropic.rs # Anthropic Messages
│ │ │ └── gemini.rs # Google Generative
│ │ ├── clipboard/mod.rs # Win32 clipboard + paste
│ │ ├── database/
│ │ │ ├── mod.rs # CRUD operations
│ │ │ └── migrations.rs # Schema setup
│ │ ├── commands/ # Tauri command handlers
│ │ │ ├── mod.rs # Module exports
│ │ │ ├── audio.rs # Recording commands
│ │ │ ├── app.rs # App lifecycle (quit, show settings)
│ │ │ ├── changelog.rs # Read bundled CHANGELOG.md
│ │ │ ├── clipboard.rs # Paste/read clipboard
│ │ │ ├── database.rs # Transcription CRUD
│ │ │ ├── reasoning.rs # AI reasoning dispatch
│ │ │ ├── settings.rs # Store get/set
│ │ │ └── transcription.rs # Cloud transcription
│ ├── capabilities/default.json # Permission scopes
│ ├── tauri.conf.json # Window + plugin config
│ ├── build.rs # Tauri build setup
│ └── Cargo.toml # Rust dependencies
│
├── docs/
│ ├── ARCHITECTURE.md # This file
│ ├── CHANGELOG.md # Version history
│ ├── PROGRESS.md # Operational/release notes
│ └── TODO.md # Follow-up work
├── scripts/
│ └── submit-winget.ps1 # Local OAuth WinGet submission
├── .github/
│ ├── README.md # GitHub repo readme
│ └── workflows/
│ ├── ci.yml # CI pipeline (push/PR)
│ └── release.yml # Release pipeline (version tags)
├── package.json # Frontend deps + scripts
└── CLAUDE.md # Claude Code project instructions