01 · Introduction

Same user. Same sentence. Different result.

The difference between a voice agent that feels broken and one that feels natural comes down to how well it handles interrupts and turn-taking.

Broken

User: "I want to fly to—"

Agent keeps talking — ignores interruption for 1.8 s

User: "no wait, a HOTEL—" (ignored)

1.8 s delay

Working

User: "I want to fly to—"

✖ STOP — interruption detected

User: "actually, a hotel in Sydney"

Agent: "Hotel in Sydney — got it!"

180 ms

03 / 23

02 · The constraint

The 200 ms
constraint

04 / 23

The 200 ms target

Human turn-gap: measured across 10 languages

200 ms is the time from when the user stops speaking to when the agent starts responding. Below that, conversation feels natural. Above it, users pause, repeat themselves, or assume the call dropped. Chat agents get seconds — voice agents don't get a second chance.

0 ms

200 ms

400 ms

800 ms

1500 ms+

Natural Acceptable Noticeable Broken Call dropped?

200 ms

Best measured human TTFA
Stivers et al., PNAS 2009 — 10 languages, 10K+ conversations

755 ms

Best measured TTFA — cascaded pipeline
Qiu et al., Salesforce 2026

05 / 23

03 · Architecture

The voice
agent pipeline

06 / 23

Frame-based · Async · Backpressure-aware

Voice agents need a pipeline

Speech-to-Text (STT), a Language Model (LLM), and Text-to-Speech (TTS) are all critical components of a voice agent. But VAD — voice activity detection — is an often-overlooked layer. It decides when the user has finished speaking. Get it wrong and everything downstream breaks.

WebRTCDaily
20 ms frames

→

VADSilero v5
32 ms frames

→

STTCartesia Ink
Streaming

→

LLMClaude / GPT
Token stream

→

TTSCartesia
PCM 24 kHz

→

WebRTCDaily
out

Smart Turn Detection

Listens to VAD + audio features. Decides: respond or wait.

Interruption Handler

Propagates InterruptionFrame — flushes TTS + LLM buffers.

What is Pipecat?

Open-source framework for real-time voice agents — wires STT, LLM, TTS, and VAD into a frame-based pipeline with backpressure, interruption handling, and streaming built in.

10k+

GitHub Stars

07 / 23

04 · Level 1

Silero VAD:
take control

The simplest component you fully own. This is where most production systems start.

08 / 23

04 · Level 1

Silero VAD — 309K parameters, <1 ms

STFT16 kHz PCM
spectral

→

4× Conv1d+ ReLU
features

→

LSTM (128)Temporal
context

→

Linear+ Sigmoid
P(speech)

→

0 → 1Speech prob

309K

parameters

2 MB

model size

<1 ms

per 32 ms frame

MCC 0.72

vs WebRTC 0.41

vad = SileroVADAnalyzer(params=VADParams(stop_secs=0.3)) # confirmed: Qiu et al. (2026) same config

09 / 23

04 · Level 1

Tuning stop_secs — the critical parameter

Sales / 0.2 s

Snappy. May cut off thinking.

General / 0.5 s

Balanced. Most users happy.

Health / 0.8 s

Patient. Respects pauses.

Data entry / 1.2 s

Never interrupts. Dead-air risk.

# The critical parameter class VADParams: threshold = 0.5 min_speech_ms = 250 min_silence_ms = 300 # THIS ONE speech_pad_ms = 100

10 / 23

Three levels of solving this — increasing control, increasing complexity

What VAD alone can't solve

01 Silence detection

When did they stop talking?

Is the user done, or just thinking? 300 ms and 1200 ms silence look identical to raw VAD.

02 Barge-in

They started talking. Stop, fade, or finish?

The decision depends on context — correction, affirmation, or ambient noise. Most agents don't distinguish.

03 Turn detection

Same silence means four things

Complete sentence. Incomplete thought. Thinking pause. Backchannel "yeah". VAD fires on all of them.

11 / 23

05 · Level 2

Built-in turn
detection

Smarter than raw VAD. State-of-the-art STT models are now starting to incorporate turn detection.

12 / 23

05 · Level 2

Cartesia Ink-2 and Deepgram Flux

Cartesia Ink-2

—

Turn detection built into the STT WebSocket protocol — emits turn.start / turn.end events

—

No local VAD or turn analyzer needed — the server drives boundaries

—

P50 STT latency: 299 ms · P95: 328 ms

—

Pipecat: CartesiaTurnsSTTService

Deepgram Flux

—

Endpointing built into the Deepgram streaming API — single-service turn detection

—

Configurable endpointing parameter controls silence threshold

—

P50 STT latency: 247 ms · P95: 298 ms (nova-3)

—

Trade-off: no visibility into model decisions, vendor-specific behaviour

Both are Level 2 — smarter than raw VAD, no local model required. Best for prototypes and latency-sensitive deployments where control is less critical.

13 / 23

06 · Level 3

Silero VAD +
Smart Turn

Full control. Transparent. Tunable. Portable.

14 / 23

06 · Level 3

Smart Turn v3.2 — open source and deployable today

System	Recall	Precision	Size	License	Status
Smart Turn v3.2 ✓ Open source	58.9%	68.4%	8–32 MB	BSD-2	pip install today
Helwani (Meta, 2026)	87.7%	57.2%	1.14M params	No code released	Research only
Cartesia Ink-2	—	—	API	Proprietary	Built into STT
Deepgram Flux	—	—	API	Proprietary	Built into STT
LiveKit Turn Detector	—	—	v1 / v1-mini	Proprietary	v1: cloud (LiveKit); v1-mini: local CPU

Smart Turn: 12 ms inference, 200–400 ms effective detection. Runs only during silence. CPU (8 MB quantized) and GPU (32 MB unquantized) versions available.

pipecat-ai/smart-turn

15 / 23

06 · Level 3

The full stack — interruption in 50 ms

t = 0 ms UserStartedSpeakingFrame

t = 32 ms VAD confirms speech (> threshold)

t = 33 ms InterruptionFrame propagates downstream

t = 35 ms TTS buffer flushed — playback stops

t = 36 ms LLM generation cancelled

t = 50 ms Pipeline ready for new input

Decision logic

STOP

Correction detected. Cut audio immediately.

FADE

Affirmation ("yeah"). Taper over 200 ms.

FINISH

Noise / backchannel. Complete current sentence.

16 / 23

03 · Architecture

Three levels — swap one line

Level 1 · Silero VAD

# 01-silero-vad.py from pipecat.audio.vad.silero import SileroVADAnalyzer user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( vad_analyzer=SileroVADAnalyzer() ), ) pipeline = Pipeline([ transport.input(), stt, user_aggregator, llm, tts, transport.output(), assistant_aggregator, ])

Level 2 · Built-in VAD in STT (Cartesia)

# 02-cartesia-turns.py from pipecat.services.cartesia.turns.stt \ import CartesiaTurnsSTTService # Server drives turn boundaries — # no local VAD needed stt = CartesiaTurnsSTTService( api_key=os.environ["CARTESIA_API_KEY"] ) pipeline = Pipeline([ transport.input(), stt, user_aggregator, llm, tts, transport.output(), assistant_aggregator, ])

Level 3 · Smart Turn

# 03-smart-turn.py
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 \
    import LocalSmartTurnAnalyzerV3

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        vad_analyzer=SileroVADAnalyzer(),
        user_turn_strategies=UserTurnStrategies(
            stop=[TurnAnalyzerUserTurnStopStrategy(
                turn_analyzer=LocalSmartTurnAnalyzerV3()
            )]
        ),
    ),
)

pipeline = Pipeline([
    transport.input(),
    ...
])

17 / 23

03 · Solution overview

Three levels.
Pick your tradeoff.

Level 1

Silero VAD
Own your silence detection

Level 2

Built-in turn detection
Smarter, but vendor-managed

Level 3

Silero VAD + Smart Turn
Full control, fully portable

18 / 23

07 · Latency

Where time
goes

19 / 23

07 · Latency

The latency waterfall — voice-to-voice breakdown

0 200 ms 400 ms 600 ms 800 ms 1000 ms 1200 ms 1400 ms

Mic + encoding

~40 ms (mic input + Opus encode)

Network in + jitter

~52 ms (transit + jitter buffer)

Transcription + endpointing

300 ms

STT streaming + turn detection

LLM TTFB

500–650 ms

dominant bottleneck

Sentence aggregation

~20 ms

TTS TTFB

120 ms

~120 ms (TTS streaming)

Network out + playback

~93 ms (jitter + decode + speaker)

Total: ~1,100–1,300 ms (API-based) · ~500 ms achievable with co-located models

Source: voiceaiandvoiceagents.com — learn more about voice agent latency breakdown

20 / 23

07 · Latency

LLM TTFT — the bottleneck. Target: < 700 ms.

P50 P95 Multi-turn

< 700 ms target

nemotron-3-ultra

529 ms

529 ms 655 ms 98.3%

gpt-4.1

536 ms

536 ms 1771 ms 96.3%

gpt-5.4 (low)

782 ms

782 ms 1706 ms 97.0%

Claude Haiku 4.5

637 ms

637 ms 1615 ms 98.0%

gemini-3.5-flash

960 ms

960 ms 1588 ms 99.0%

Claude Sonnet 4.6

850 ms

850 ms 4126 ms 100%

Note

TTFT alone isn't enough. Instruction following must not degrade across multiple turns — if it does, you'll need workarounds like context pruning or context trashing to keep the model on-task. The aiewf-eval benchmark tests 30-turn conversations specifically to surface this.

Source: aiewf-eval multi-turn benchmark (kwindla, 2026) — 30-turn conversations, 10 runs each. TTFT = time to first token. · wirjo.github.io/voice-ai-benchmarks · github.com/kwindla/aiewf-eval

21 / 23

08 · Production and future

Production lessons and open problems

3×

escalation rate when agents incorrectly interrupt
Hamming — 4M calls

755 ms

best measured TTFA (cascaded pipeline)
Qiu et al. 2026

58.9%

Smart Turn v3 recall — best deployable today
Helwani et al. 2026

Infrastructure is hard

Multiple components — WebRTC transport, VAD, STT, LLM, TTS — each with its own scaling, latency, and failure modes. Coordinating them in production is non-trivial.

Pipecat Cloud

Managed hosting for Pipecat pipelines. Handles scaling, session management, and WebRTC — so you focus on the agent logic, not the plumbing.

AWS Guidance

For enterprise deployments: secure, governed, within your AWS environment. Reference architecture covering VPC, IAM, Bedrock integration, and SageMaker bidirectional streaming.

22 / 23

Try it now

Now go build your voice agent.

Repositories

Demo & code
github.com/wirjo/
pipecat-turn-detection-demo

AWS Guidance
github.com/aws-solutions-library-samples/
sample-voice-agent

LLM Benchmarks
wirjo.github.io/voice-ai-benchmarks

Stack

Pipecat · pipecat.ai

Cartesia STT & TTS · cartesia.ai

Daily managed WebRTC · daily.co

Smart Turn · github.com/pipecat-ai/smart-turn

Silero VAD · github.com/snakers4/silero-vad

Key papers

Stivers (PNAS 2009)

Qiu et al. (arXiv:2603.05413)

Helwani et al. (arXiv:2603.13379)

Zhang et al. (arXiv:2502.14145)

Speakers

Chintan Agrawal

achintan@amazon.com

Daniel Wirjo

wirjo@amazon.com

Solution Architect
Amazon Web Services (AWS)

Open source voice agent framework by Daily. Thanks to the Pipecat community for building the tools that make this possible.

23 / 23

Voice agents
that handle interrupts

Eight stops

Same user. Same sentence. Different result.

The 200 ms
constraint

Human turn-gap: measured across 10 languages

The voice
agent pipeline

Voice agents need a pipeline

Silero VAD:
take control

Silero VAD — 309K parameters, <1 ms

Tuning stop_secs — the critical parameter

What VAD alone can't solve

When did they stop talking?

They started talking. Stop, fade, or finish?

Same silence means four things

Built-in turn
detection

Cartesia Ink-2 and Deepgram Flux

Silero VAD +
Smart Turn

Smart Turn v3.2 — open source and deployable today

The full stack — interruption in 50 ms

Three levels — swap one line

Three levels.
Pick your tradeoff.

Where time
goes

The latency waterfall — voice-to-voice breakdown

LLM TTFT — the bottleneck. Target: < 700 ms.

Production lessons and open problems

Now go build your voice agent.

Voice agentsthat handle interrupts

Eight stops

Same user. Same sentence. Different result.

The 200 msconstraint

Human turn-gap: measured across 10 languages

The voiceagent pipeline

Voice agents need a pipeline

Silero VAD:take control

Silero VAD — 309K parameters, <1 ms

Tuning stop_secs — the critical parameter

What VAD alone can't solve

When did they stop talking?

They started talking. Stop, fade, or finish?

Same silence means four things

Built-in turndetection

Cartesia Ink-2 and Deepgram Flux

Silero VAD +Smart Turn

Smart Turn v3.2 — open source and deployable today

The full stack — interruption in 50 ms

Three levels — swap one line

Three levels.Pick your tradeoff.

Where timegoes

The latency waterfall — voice-to-voice breakdown

LLM TTFT — the bottleneck. Target: < 700 ms.

Production lessons and open problems

Now go build your voice agent.

Voice agents
that handle interrupts

The 200 ms
constraint

The voice
agent pipeline

Silero VAD:
take control

Built-in turn
detection

Silero VAD +
Smart Turn

Three levels.
Pick your tradeoff.

Where time
goes