Whisper STT: re-architect for real-time or defer to post-processing

[BraedenBDev]

Status

Whisper on-device STT is functional but too slow for production. Current architecture:

• 3-second audio chunks buffered via ScriptProcessorNode
• Single-threaded WASM inference (~3-5s per chunk)
• Chrome MV3 CSP blocks multi-threaded ONNX (blob: URLs in script-src)
• Result: ~6-8s latency per utterance, missing real-time intent matching

Web Speech API (Google) works well for real-time but requires internet.

Options

A: Streaming Whisper — 500ms chunks, lower accuracy, still WASM-limited
B: WebGPU backend — significant speedup, not all extension contexts support it
C: Post-processing — Web Speech for real-time, Whisper refines after capture ends
D: Accept Web Speech as default — Whisper is opt-in private mode with known latency

Recommendation: D for now, C as next evolution.

Technical debt

• Migrate ScriptProcessorNode to AudioWorklet
• numThreads=1 forced by CSP — revisit if Chrome relaxes MV3

🤖 Generated with Claude Code

[m13v]

option D (Web Speech default, Whisper opt-in) is the right call for now. we went through the same evaluation for an AI assistant and landed in a similar place. Web Speech API's latency is unbeatable when internet is available, and the accuracy is genuinely good for conversational English. for the opt-in private mode, one thing that helped us: use a smaller Whisper model (tiny or base) for the real-time path. the quality drop vs large is noticeable for edge cases but the latency improvement is massive (sub-1s on decent hardware). option C (Web Speech real-time, Whisper post-refinement) is worth pursuing next because it gives you the best of both: instant feedback during conversation plus accurate final transcript for search and reference.

[m13v]

our on-device transcription service with model size selection and the hybrid online/offline fallback approach: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift

[BraedenBDev]

Solid take on D and C.

The tiny/base model suggestion for the real-time path is one we've already landed on. src/sidepanel/whisper-worker.ts defaults to Xenova/whisper-tiny.en. The bottleneck we hit isn't the model size, it's that Chrome MV3 CSP blocks blob: URLs in script-src, which forces numThreads=1 for ONNX wasm. Even tiny takes ~6-8s for a 3s chunk under those conditions. I've been benchmarking against the ceiling I can actually hit in the extension, not on desktop.

C has been the endgame in my head for a while. Web Speech streams during capture, Whisper runs over the full audio buffer once the user stops and produces the canonical transcript. Real-time feel, accurate final output, no commitment to sub-second on-device inference. Also fits our zero-opCost rule, everything stays client-side.

Reading fazm/TranscriptionService.swift now. Question on the hybrid path: when the Whisper post-refinement disagrees with the real-time transcript, how do you reconcile? Does Whisper always win, or is there a confidence merge or diff you surface? Trying to figure out what makes sense for a user who's already seen the live Web Speech output in the UI and then gets a corrected version at capture-end.