How Voice Agents Detect When a Caller Stops Talking

The caller says “my order number is”, then pauses to dig the receipt out of a drawer. Half a second of silence, and the agent jumps in: “Sorry, are you still there?” The caller starts over, annoyed.

Everyone has been on the other end of that call. The agent is not slow or unhelpful; it guessed wrong about when you were done talking. That single decision, when the caller has finished their turn, is the hardest part of building a voice agent, and the difference between a call that feels natural and one that feels like arguing with a machine.

Get it right and the conversation flows. Get it wrong in either direction, cutting in too early or sitting in silence too long, and callers notice within the first exchange.

What voice agent endpointing is and why turn-taking is hard

Voice agent endpointing is the step where the agent decides your turn is over and it is time to respond. It sounds trivial, the way it is for a person, but a person brings a lifetime of conversational instinct to it. A voice agent makes that call from an audio stream, in real time, with no idea what you are about to say next.

The bar is high because human turn-taking is fast and precise. In natural conversation the gap between one person finishing and the next starting is very short, typically around 200 milliseconds, roughly a fifth of a second, and often close to zero. We hit that by predicting when a turn is about to end, not by waiting for silence and then reacting to it.

A voice agent cannot predict like that yet, so it waits and measures, and both directions hurt. End the turn too early and you cut the caller off mid-sentence, which feels rude. Wait too long and the pause before a reply feels broken, like the line dropped. There is no single turn-taking latency that is right for every moment.

It helps to separate two terms that get used interchangeably. Voice activity detection asks whether there is speech in this instant of audio. Endpointing asks the bigger question of whether the caller’s whole turn is finished. The first is a signal; the second is a decision built on top of it, plus silence timing and, increasingly, the words themselves.

Endpointing is a stack of decisions, then: is there speech right now, has the speech stopped, and has the turn actually ended. The rest of this post walks that stack from the raw audio up to the meaning of the words, because getting turn-taking right means getting all three layers right.

How voice activity detection decides speech from silence

The bottom layer is voice activity detection, or VAD: the job of labeling each small slice of audio as speech or silence. Everything above it depends on getting this right, because an endpointer reasoning about pauses is only as good as its view of where speech actually is.

Classic VAD works on simple signal features. Chop the audio into short frames, around 20 to 30 milliseconds each, and for each frame compute how much energy it carries and how often the signal crosses zero. Speech has a characteristic energy and zero-crossing profile that differs from silence or steady background noise, so a threshold on those features labels each frame.

Modern VAD replaces the hand-tuned threshold with a learned model. Silero VAD is a small neural detector; the classic WebRTC VAD is a lightweight Gaussian mixture model. Both classify each frame more robustly than raw energy, holding up against background noise and varied voices. The output is the same shape either way: a speech-or-silence label per frame, many times a second.

The piece that makes VAD usable for turn-taking is the hangover timer. Speech is not continuous; there are tiny gaps between and within words. Declare the turn over the instant energy drops and you cut on every micro-pause. So after speech stops, the detector keeps labeling frames as speech for a short window, commonly around 200 milliseconds, before accepting that silence has really begun.

The hangover length is itself a tradeoff. Too short and the detector chops off soft word endings and trailing consonants; too long and every real pause drags. Many implementations make it adaptive rather than a fixed constant, lengthening it when the speaker has been talking in long runs and shortening it for quick back-and-forth.

Here is the core of it as code: frame the signal, measure energy, apply a threshold with a hangover, and declare the turn over after a run of trailing silence.

import math

def endpoint(signal, sr=16000, frame_ms=20, energy_thresh=0.02, hangover=10, end_silence_ms=500):
    """Classic endpointing: frame -> RMS energy -> threshold + hangover -> end after trailing silence.

    hangover=10 frames is ~200 ms at 20 ms/frame; energy_thresh assumes float audio in [-1, 1].
    """
    fl = int(sr * frame_ms / 1000)
    frames = [signal[i:i+fl] for i in range(0, len(signal) - fl + 1, fl)]
    speech_seen, silence_run, hang = False, 0, hangover
    need = max(1, int(end_silence_ms / frame_ms) - hangover)  # trailing silence excludes the hangover window
    for idx, fr in enumerate(frames):
        energy = math.sqrt(sum(x * x for x in fr) / len(fr))
        if energy >= energy_thresh:
            speech_seen, silence_run, hang = True, 0, hangover
        else:
            if hang > 0: hang -= 1
            else: silence_run += 1
            if speech_seen and silence_run >= need:
                return idx * frame_ms  # ms at which the turn is declared over
    return None

def _demo():
    fl = int(16000 * 0.02)
    speech_then_silence = [0.1] * (fl * 10) + [0.0] * (fl * 40)  # 10 speech frames, then a long pause
    assert endpoint(speech_then_silence) is not None

if __name__ == "__main__":
    _demo()

Notice the knob that matters: end_silence_ms, how long a pause has to last before the turn is called over. The whole problem of turn-taking lives in that one number, and the next section is why no single value for it is ever right.

The fixed silence timer, and why one number never fits

The simplest endpointer is a fixed silence timer: wait for a set number of milliseconds of continuous silence, then assume the caller is done. It is the default in most voice stacks because it is trivial to implement. It is also wrong for every real conversation, because that number is never right twice.

Set the timer short, say 300 milliseconds, and quick exchanges feel snappy, but the agent guillotines anyone who pauses to think. “My account number is” and a beat to remember it becomes a cut-off. “Let me check, it’s” and a glance at a card becomes an interruption. The caller is mid-thought and the agent has already started talking over them.

Set it long, say 1.5 seconds, and you protect the thinkers but punish everyone else. Every turn now ends with an awkward pause while the agent waits out a timer the caller stopped needing a second ago. The conversation develops a stutter, statement, silence, reply, silence, and it feels slow even though nothing is actually slow.

The deeper problem is that the right wait depends on what was just said, and a timer cannot know that. “Yes.” is a complete turn that needs almost no wait. “I was calling because” is obviously unfinished and needs a long one. The same silence duration means opposite things after those two phrases, and a fixed timer treats them identically.

It also depends on the person and the channel. Fast talkers and slow talkers need different waits. A noisy line, a hands-free car, a caller reading a number off a screen, all shift where the natural pauses fall. A constant tuned for your demo in a quiet room falls apart on a real phone network with real people.

This is why a fixed timer is a floor, not a solution. You can tune it to be least-bad on average, but least-bad on average still means wrong on most individual turns. The fix is to stop treating silence duration as the only signal, which is where the next two layers come in.

Why streaming transcripts don’t mark the end of a turn

Most voice agents already run streaming speech-to-text to get a transcript, so it is tempting to lean on it for turn-end too. The model is listening anyway, and it emits text as the caller speaks, so surely it knows when a sentence finished. In practice, streaming STT gets turn-end wrong in specific, important ways.

The root of it is interim versus final transcripts. A streaming recognizer emits interim results continuously, its best guess so far, marked as not final, and these change as more audio arrives. “I want to” might become “I want two” a moment later. Only when the model is confident does it emit a final result for that chunk. The interims are fast but unstable; the finals are stable but lag.

Teams reach for the final transcript as a turn-end signal, and it half works. A final often lands near the end of an utterance, but the recognizer finalizes on its own internal logic, not on whether the caller’s turn is over.

It can finalize mid-turn at a clause boundary, or hold one back while it waits for more audio, so the timing drifts from the actual turn.

Guessed finals make it worse. Under time pressure a recognizer will commit to a final it is not sure about, and if that guess is wrong, the agent acts on a transcript that is about to change. You get the agent replying to “I want two” when the caller said “I want to cancel”, because it finalized early and started formulating an answer.

The honest takeaway is that STT timing and turn-end are correlated but not the same event. The recognizer answers what did they say; endpointing answers are they done. Bolting the second question onto the first gives you a signal that is close enough to be tempting and wrong often enough to hurt. STT is an input to endpointing, not a replacement for it.

Semantic turn detection uses the words, not just the silence

The newest layer stops treating silence as the main signal and starts reading the words. Semantic turn detection asks a different question: given everything the caller has said so far, does this sound like a complete thought. A pause after “my order number is” means keep waiting; the same pause after “that’s all, thanks” means the turn is clearly over.

It works by reading cues beyond silence. Some models fuse the acoustic signal, falling pitch and the way people trail off, with the linguistic signal, whether the sentence sounds complete. Others, trained on audio alone, predict the turn-end from prosody without a transcript at all. Silence is still an input, but it is no longer the only one.

This is an active area in 2025 and 2026, with real implementations. LiveKit ships a turn-detector model, Pipecat has an open smart-turn model, and Deepgram and Krisp offer end-of-turn detection that goes beyond silence. They differ in the details, some read the words, some the audio alone, but all replace the stopwatch with a model.

The payoff is being fast and patient at the same time. Because the model can tell a finished thought from an unfinished one, it can respond almost immediately after a clear ending while still waiting through a mid-thought pause. The latency becomes a tunable budget, traded against how often it cuts someone off, rather than a fixed delay you set once. Table A compares the approaches.

Semantic detection is not free. It needs a model running in the loop and more compute than a threshold, and it can still be wrong on genuinely ambiguous turns, the ones a human would also hesitate on. But it moves the hard decision from a number you guess to a judgment the system can make, which is why turn detection in voice AI is converging on it.

What a broken turn sounds like to the caller

All of this theory shows up for the caller as a small number of very recognizable bad experiences. If you have tested a voice agent, you have hit every one of these, and each maps back to a specific failure in the stack above.

Getting cut off mid-sentence is the endpointer firing too early: a short silence threshold, or a VAD that mistook a breath for the end of the turn. The false “are you still there?” is the mirror image inside a single turn, the agent deciding a thinking pause was the end and prompting a caller who never stopped.

Long dead air before a reply is usually a timer set too long, or an STT final that lagged, so the caller finishes and then waits, wondering if the line dropped. Double-talk, where agent and caller speak at once, is a barge-in handling failure: the agent does not stop talking when the caller starts, or does not detect the overlap at all.

The subtlest one is the reply that never arrives. The caller finishes, the agent’s endpointer never fires because the audio never cleanly crossed its silence threshold, perhaps on a noisy line, and the turn just hangs. These are the failures that drive callers to zero out to a human, and evaluating voice agents means measuring how often each one happens.

Mapping symptoms to causes this way is what makes turn-taking debuggable. “The agent feels awkward” is not actionable; “the agent cuts off 8% of turns where the caller pauses past 400 milliseconds” is. Every row in that table is a metric you can put a number on, which is the first step to fixing it.

How do you test turn-taking before it reaches real callers?

You cannot tune turn-taking by trying it yourself a few times, because the failures depend on conditions you will not reliably reproduce by hand: specific pause lengths, background noise, people talking over the agent, slow readers. The only way to get ahead of them is to test against those conditions on purpose, before real callers hit them.

That means simulation. Instead of calling the agent and hoping to trigger a pause at the wrong moment, you script calls that deliberately include the hard cases: a caller who pauses mid-sentence to find a number, one who barges in while the agent is talking, one on a noisy line, one who speaks slowly with long gaps. You run those scenarios repeatedly and measure what the agent does.

The metrics that matter are concrete. Interrupt rate: how often the agent cuts the caller off. Endpoint latency: how long after the caller actually finishes the agent starts replying. False-endpoint rate: how often it jumps in on a mid-turn pause. Barge-in handling: whether it stops when talked over. Each is a number you can track across versions, so a change that fixes one and breaks another shows up immediately.

Testing this way turns turn-taking from a vibe into a regression suite. Layered voice testing runs these scenarios at every change, so when you adjust a silence threshold or swap in a semantic detector, you see the effect on interrupt rate and latency across hundreds of scripted calls, not just the one you happened to try. That is the difference between tuning and guessing.

It also catches the interactions. Lowering the silence timer to reduce dead air might spike your interrupt rate on slow talkers, a tradeoff you would never see from a single test call but that a scenario suite surfaces instantly. Simulation across many conditions is how you find those before production does.

The point is to move the discovery of turn-taking failures off your live callers and into a test you control. Real conditions, scripted and repeatable, scored on metrics you chose, run on every change. That is how turn-taking stops being the thing you apologize for and becomes the thing you measured and fixed.

Testing turn-taking under real conditions with Future AGI

This is the job Future AGI does in the voice stack, and it is worth being precise about what it does and does not do. Future AGI does not do endpointing, VAD, or turn detection; those live in your voice pipeline. What it does is test the agent that does them, under the conditions that break turn-taking.

You can simulate voice agents built on Vapi, Retell, or Bland.ai, or any agent you can reach by phone, and drive them through real-condition scenarios: mid-sentence pauses, barge-in, background noise, and slow speech. The simulation runs the call, the agent’s own endpointer makes its decisions, and you score how the turn-taking held up. The voice providers reference lists the stacks it connects to.

From there it is ordinary evaluation: run a voice simulation across your scenarios, score interrupt rate and endpoint behavior, and compare versions, as part of end-to-end voice evaluation. The evaluation and tracing SDK is Apache-2.0 and self-hostable; the simulation runs on the platform. It will not pick your silence threshold for you, but it will tell you, before a caller does, which threshold actually holds up.

Building voice agents that wait for the caller to finish

Go back to the caller digging for their order number while the agent cuts in. That failure was not bad luck; it was a silence timer doing exactly what it was set to do, with no idea the caller was mid-thought. The fix is the whole stack working together, not a better guess at one number.

Build it in layers: VAD to find speech, adaptive timing instead of a fixed wait, and semantic signals to tell a finished thought from a pause. Then test it under real conditions, because the only way to know your turn-taking holds is to try it against the pauses, noise, and interruptions real calls bring. Start by simulating your agent against the hard cases.