The first voice agent build usually plays audio to completion. The caller starts speaking three seconds into a fifteen-second menu explanation and is ignored until it finishes. Everyone has been on that call and everyone hates it. A caller who gives up mid-sentence and hangs up can still be counted as handled, which is one of the reasons deflection flatters a bad agent.
What barge-in actually requires
Interrupting is not one feature. It is four things that must all work within a couple of hundred milliseconds, and all four are pipeline decisions rather than settings, which is why they get made early in a voice AI build:
Detect speech while playing. Voice activity detection running on the inbound stream at the same time as outbound audio is playing. That means full duplex — a half-duplex pipeline cannot do this at all.
Stop the audio immediately. Not at the end of the current sentence or buffer. Flush the playback queue on detection. Anything still buffered downstream will keep playing after you have stopped sending.
Cancel the generation. If the model is still producing tokens and the TTS is still synthesising, both need aborting. Otherwise you pay for output nobody hears, and the next turn arrives late because the pipeline is still draining.
Reconcile what was actually heard. The caller heard part of a sentence. The conversation state must record what was played, not what was generated, or the agent will assume it communicated something it did not. That matters most at a read-back, when the agent is a turn away from changing something in a real system.
That last point is the one most often missed and it produces the strangest failures — an agent referring back to information it never actually delivered.
Echo is what makes it hard
The microphone hears the speaker. Without acoustic echo cancellation your VAD triggers on the agent’s own voice, the agent interrupts itself, and the call degrades into stuttering.
On a telephony path AEC usually exists in the carrier leg. On WebRTC it is the browser’s job and generally works. Mixing the two — a SIP trunk bridged into a browser session — is where it silently stops working, and it is worth testing that specific path deliberately.
Tune the threshold against real audio
VAD sensitivity is a trade. Too eager and a cough, a door, or background conversation stops the agent mid-sentence. Too relaxed and the caller has to raise their voice to be heard, which they experience as being ignored.
There is no correct default. Tune it against recordings from the actual deployment — a hotel lobby, a call centre, a car — because the noise floor is what determines the right number, and it is different in every one of those.