We have shipped voice agents with a mediocre model and tight latency, and agents with an excellent model and a 1.4-second turn gap. The first one is judged as competent. The second one gets “hello? are you there?” and a hang-up.
The budget
A natural conversational gap is 200–500ms. By about 800ms the caller assumes the line dropped. Everything in the path has to fit inside that:
- Telephony ingress and jitter buffer
- Streaming transcription up to a confident endpoint
- Model time to first token
- Text-to-speech time to first audio
- Egress back to the caller
Nothing here is generous, and the first line is doing more than moving bytes: telephony audio arrives at 8kHz, which is not what the transcription model was trained on. Time-to-first-token matters far more than total generation speed, because the caller only needs to hear the start of the reply to know they are being answered.
What actually buys time
Start before the caller stops. Endpoint detection that waits for confident silence adds hundreds of milliseconds. Begin generating on a likely endpoint and cancel if the caller continues.
Stream the speech. Do not wait for the full response text before synthesising. Send the first clause as soon as it exists.
Handle barge-in properly. When the caller talks over the agent, stop the audio immediately and discard the rest of the turn. An agent that keeps talking over someone reads as broken far faster than one that is slow.
Keep the first sentence short. Long openers stack the whole reply behind the synthesis of a paragraph the caller mostly won’t listen to.
What does not buy time
Reaching for a smaller model, usually. The model is rarely the bottleneck once you are streaming — transcription endpointing and TTS startup tend to dominate. Measure the whole path before trading away quality; on voice AI builds the time is almost always sitting in the plumbing rather than in the model.
Measure the right thing
Median turn latency hides the problem. The calls that fail are the tail: the turn where transcription hesitated, or the tool call took 600ms longer than usual. Track p95 and p99 of caller-perceived gap — from the end of their speech to the start of the agent’s audio — and treat anything above a second as an outage rather than a slow response.
The one gap a caller will forgive is the one they were told about, which is why a transfer to a human has to carry the context with it rather than dropping the caller into a queue. Silence they cannot account for is what ends the call.