Ask someone why a voice agent feels wrong and they will say it is "slow". Ask how slow and the answer is a shrug. Latency in a voice pipeline is not one number, it is six, and until you write them down you cannot fix anything.
Here is the budget we hold ourselves to for a warm connection, from the moment the caller stops speaking to the moment they hear the first syllable of the reply.
Endpoint detection — 90 ms. Deciding that the caller has actually finished, rather than pausing for breath. Too eager and you interrupt people; too patient and every exchange has a dead beat in it. This is the single most underrated number in the pipeline.
Final transcription — 40 ms. Streaming recognition means most of the transcript already exists by the time the caller stops. Only the tail needs finalising.
Retrieval — 60 ms. Fetching the passages that ground the answer. This is a hard ceiling, not an average: if retrieval is slow we answer without it and say we are unsure, rather than making the caller wait.
First token — 120 ms. Time to the model's first output token. Not the full response — we start speaking before generation is finished.
Speech synthesis first chunk — 70 ms. Enough audio to begin playback.
Network and jitter buffer — 20 ms. On a decent connection.
That totals 400 ms, and it only works because the stages overlap rather than queue. Synthesis begins on the first clause while the model is still writing the second. Retrieval starts on the partial transcript rather than the final one.
The interesting part is what we gave up. We do not run a separate reranking pass, because it cost 80 ms and improved answer quality by an amount we could not measure in call outcomes. We do not run a safety classifier in-line on output; it runs in parallel and can cut the audio, which costs nothing in the happy path.
If you are building this yourself, the advice is simple: write your budget down before you optimise anything, measure at the boundary the caller experiences rather than at your service boundary, and be willing to delete a stage that only improves a metric nobody hears.