Pause-aware speech completion
I built an on-device C++ prompter that detects a hesitation, transcribes recent speech, and suggests what you might say next. The microphone keeps capturing audio while the models run.
Why I built it
When I speak French, I can usually keep a conversation going. But sometimes I know exactly what I want to say and get stuck on a single word. That pause can make me sound less fluent than I am. I wanted to see whether a tool could use what I’d already said to suggest a word or phrase and help me keep going.
How it works
- Capture audio
- Detect a pause
- Suggest a continuation
Challenges and how I resolved them
Capture audio while inference runs
The microphone callback needs to accept new samples while transcription and prediction are still running. I used a lock-free ring buffer to hand samples to a worker and kept both models off the capture thread.
Keep suggestions relevant when speech resumes
A speaker can resume before a suggestion arrives. I attached a revision number to each inference job so the display can discard outdated results even if the model is still working on them.
Architecture
- Capture. The CoreAudio callback writes microphone samples to a single-producer, single-consumer ring buffer.
- Detect. A worker runs Silero through ONNX Runtime, tracks pauses, and maintains a bounded audio history.
- Infer. At a hesitation, the worker hands a snapshot to the inference thread. whisper.cpp transcribes it, then llama.cpp generates candidate continuations.
- Display. The terminal shows the candidates for the current speech revision in a simulated AR display.
Buffer ownership and inference caching
The voice activity worker is the sole reader of the capture buffer. Inference receives a copy of recent audio, which keeps model work from sharing the live buffer. The predictor can reuse the model’s KV cache when prompt tokens match.
What I measured
- Median predictor latency
- 41% lower
- p95 predictor latency
- 54% lower
- Peak memory use
- 30% lower
SmolLM2-360M vs. Qwen2.5-0.5B at 3 generated tokens, across 80 English and French prompts on a dual-core Intel i5. The benchmark covers the predictor stage.
How I tested it
I used an injected clock to exercise pause timing and recorded WAVs to replay inputs consistently.
Test setup and repository notes
Debug builds run AddressSanitizer and UndefinedBehaviorSanitizer on project code. Model-dependent tests report a skip when weights are missing.
Read the repository handoffWhat I learned and what I’d measure next
Keeping audio capture responsive and keeping the suggestions relevant needed separate solutions. Next I’d measure the time from hesitation to display and check whether the suggestions actually help speakers continue in English and French.