Every voice agent has to answer one question hundreds of times per call: has the person stopped talking, or are they thinking? Get it wrong in either direction and the conversation stops feeling like one.
Silence duration is a blunt instrument
The default approach waits for N milliseconds of silence. Set it to 500ms and the agent talks over anyone who pauses mid-sentence — which is everyone reciting a number, an address, or an email. Set it to 1200ms and every exchange gains a full second of dead air.
There is no value that is right for both “yes” and “my reference is four four seven one, slash, B”.
Use what the words tell you
The transcript is a signal and most implementations ignore it. A turn ending in “my number is” is grammatically incomplete — the caller is still going, whatever the silence says. A turn ending in “that’s correct” is complete.
Combining a short silence threshold with a completeness check gets you both: fast when the utterance is clearly finished, patient when it obviously is not. Streaming ASR gives partial transcripts continuously, so this costs nothing extra in latency.
Context should set the expectation
The agent knows what it just asked. A yes/no question should endpoint aggressively — 400ms is plenty. A question that invites a card number or a spelled-out surname should be patient, because digits and letters come in groups with natural gaps between them.
Carry an expected-response type through the turn and let it choose the threshold. It is one field and it removes most of the complaints.
Fill the gap rather than lengthening it
Even well-tuned, there is a period between the caller stopping and the agent starting. Silence in that window reads as a dropped call, and callers say “hello?” — which the agent then treats as a new turn.
A short acknowledgement token played immediately on endpoint detection covers it. It buys several hundred milliseconds of perceived responsiveness for no actual speed-up, which is the cheapest latency work available.
Measure it separately
Log endpoint decisions with the silence observed, the threshold used, and whether the caller resumed within a second afterwards. Resuming within a second means you cut them off. That rate — not average latency — is the number that predicts whether people finish the call.