Users who say it feels fast are really reacting to the slowest of six chained stages: wake word, segmentation, recognition, generation, synthesis, and playback.
Users don't look at your architecture diagram; they just say "it's so slow." And "slow" is usually not one especially slow stage — it's the sum of six stages chained together.
From the moment you speak to the moment you hear a response, the chain looks like this:
Wake → Segmentation (VAD) → Speech Recognition (ASR) → LLM Generation → Speech Synthesis (TTS) → Playback
Each stage has fixed processing time and network round-trip time. Executed serially, their sum is the latency the user perceives; the essence of optimization is to parallelize what can be parallelized and to do as early as possible what can be done earlier.
The easiest pitfall is "collect the whole segment, then upload." Streaming recognition sends audio out while the user is still speaking, so the recognition result is essentially ready the moment speech ends, eliminating a whole stretch of upload wait.
Wake-word detection and basic commands that can be handled on-device don't need the network and still work offline. Keeping this step local drives the latency of the chain's first stage down to nearly zero.
The generation side hands off to synthesis as soon as it has the first sentence, and the synthesis side hands off to playback as soon as it has the first segment. When the user hears the first sentence depends on when the first token appears, not when the whole passage is finished.
"Interrupt anytime" is the watershed of experience. The device must capture audio while playing its own sound, which demands clean echo cancellation (AEC); otherwise it mistakes the speaker's output for the user speaking.
Suggested optimization order: Do streaming and first-token-first first (small changes, big gains), then localize the wake word and basic commands, and only then consider swapping models or adding machines.
Voice interaction tolerates the long tail the least. With a 1-second average latency but one 3-second stall every ten interactions, the user's experience is "it always lags." That's why we focus on P95 rather than the average.
The Xiaozhi voice conversation platform turns this chain into a single service, including streaming recognition, multi-model dialogue, and streaming synthesis, with default optimizations based on the lessons above. If latency is holding you back, feel free to contact us to talk through your scenario.