How our Pipecat voice pipeline is wired
A short write-up of the Bridge's voice pipeline (self-hosted Pipecat, Oct 2026): what's in it, measured numbers from real calls, and the choices that keep the gap between "you stop talking" and "the agent starts talking" short.
The short version. Nothing waits for anything else to finish. The transcript is warm before you stop talking, the end of your turn is detected by a model (not a fixed silence timer), the agent's reply is streamed out the moment it starts, and speech synthesis starts on the first sentence while the rest is still being written. Everything runs on one machine on the same LAN as the client.
The pipeline
Everything in the blue box is one Pipecat pipeline: a single Python process (a macOS LaunchAgent on an M1 Max). Audio frames flow left to right through each Pipecat service, and every stage passes partial results on as soon as it has them.
Browser
🎙️ Micyou talk
🔊 Speakeragent's voice
⇄
Pipecat bot (Python, on the M1)
SmallWebRTCTransportpeer-to-peer audio in/out, LAN
Silero VADspeech started / stopped
→
Smart Turn v3model: are you done?
WhisperSTTServiceMLX (our interim-transcript subclass; large-v3-turbo on Metal)interim transcript every 0.7 s while you talk · final ≈1.4 s after
OpenAILLMService → the brainsends your turn, streams reply text back (SSE)
ElevenLabs TTS service (websocket)first sentence → audio in ≈0.5 s, rest streams behind
Barge-in + echo guard2 real words interrupt it; the agent's own voice is ignored
⇄
Elsewhere
🧠 The Bridge
/v1/chat/completions → the live agent (memory + tools)☁️ ElevenLabsvoice synthesis
One turn, on a timeline (a typical quick reply; seconds)
The stages overlap: STT runs while you talk, and audio starts while the agent is still writing. The orange bar (the agent thinking) is the only one that grows with the question.
0246810
| Stage | What we use | Why it's fast |
|---|---|---|
| Transport | SmallWebRTCTransport, peer-to-peer from the browser to the bot | No media server or cloud hop. The bot runs on an M1 Max on the same LAN (reachable over VPN when away). |
| Speech detection | Silero VAD (start ≥ 400 ms, min volume 0.75) | Cheap, local, frame-by-frame. |
| End of turn | Smart Turn v3 (local model) on top of VAD; VAD stop 1.0 s, hard cap 3.0 s | A model decides whether you're done, rather than waiting out a long silence window. Natural pauses mid-thought don't end the turn, and a finished sentence doesn't have to wait for a long timeout. |
| Speech-to-text | Whisper large-v3-turbo on MLX (Apple Metal), in-process. Interim transcripts every 0.7 s while you talk. Model is warmed at boot. | No network round-trip for STT. The transcript is mostly built before you stop. The first call doesn't pay a ~6 s model load. |
| Brain | The live agent behind an OpenAI-compatible /v1/chat/completions endpoint that streams SSE | Pipecat's stock OpenAILLMService talks to it. The endpoint posts the user turn to the agent and streams reply text back as each block is written, not when the whole reply is done. |
| Text-to-speech | ElevenLabs over a websocket (v3/v4 via the Text-to-Dialogue socket, otherwise multi-stream) | Audio starts on the first sentence. Measured TTS time-to-first-audio is usually ~0.4–0.6 s. |
| Barge-in | 2 recognized (non-echo) words interrupt the agent; an echo guard drops the agent's own voice from the speakers | You can cut in without false triggers from room noise or the agent hearing itself. |
Measured on real calls (Oct 13–14)
From the bot's Pipecat metrics. "End to end" is from the moment the turn was judged over to the first audio out.
| Measure | Typical | Notes |
|---|---|---|
| STT final after you stop | ≈ 1.4 s | Whisper on the M1, largely overlapped by the turn-detection wait. |
| Brain endpoint first byte | 0.04–0.13 s | The stream opens immediately. Real text follows when the agent writes it. |
| TTS first audio | 0.4–0.6 s | Occasional spikes to 1–2 s on longer first sentences. |
| End to end, conversational reply | 0.7–5.5 s | Best seen 0.68 s; most quick replies land at about 3–5 s. |
| End to end, agent doing real work | 10–35 s | When the agent runs tools before answering (checks logs, reads code). That time is the work itself, not the pipeline. |
Honest caveat. The pipeline overhead is small, about 1.5–2 s, and most of it is turn detection plus STT. The rest of any delay is the brain. Our brain is a full agent with memory and tools, not a bare LLM call, so its thinking time is the big variable. What makes it feel fast is the next section.
What actually makes it feel fast
- Stream at every boundary. STT interims while speaking, SSE from the brain, sentence-level TTS start, WebRTC playback. If any stage waits for a complete result before passing it on (a full transcript, a full reply, a full audio file), you add seconds.
- Say something first, then do the work. The agent is instructed to lead with a short spoken line ("Let me look at that…") before it runs tools. That line streams straight to TTS, so you hear a voice in about 1 s even when the full answer takes 20.
Next step we're weighing: right now this is an instruction the agent can occasionally skip. Agent harnesses like Claude Code support hooks, scripts that run automatically at fixed points such as "before a tool call." APreToolUsehook could notice that the agent hasn't said anything yet this turn and either remind it to speak first or block the tool call until it does. That turns "say something first" from a habit into a guarantee, at least for voice turns. - Model-based end-of-turn, not a long silence timer. Many stacks wait 1.5–2 s of silence and still cut people off. Smart Turn lets us use a short VAD stop with a 3 s ceiling.
- Keep STT local and warm. Cloud STT adds a round-trip plus queueing. MLX Whisper on Apple Silicon is fast enough for interims, and a warm-up at boot removes the cold first turn.
- Set Pipecat's STT latency budget correctly.
ttfs_p99_latencyunset defaults to 1.0 s, which equals our VAD stop and collapses the STT wait to zero. Slow transcripts then land after the turn ends, producing an empty turn and a wasted LLM call. We set it to 2.5 s. - Keep the media path short. P2P WebRTC to a box on the LAN. No TURN relay or media server when you don't need one.
If you're at 4–5 seconds, check these first
- Is the LLM response streamed into TTS, or does TTS get the whole reply?
- Is TTS on a streaming websocket, or REST returning a full file?
- How long is the end-of-speech silence window, and is there a turn model?
- Is STT a cloud round-trip after the user stops, with nothing done while they talk?
- Is there an agent framework doing planning or tool calls before the first token? If so, have it speak a short acknowledgment first.
- Any cold starts: model loads, container spin-up, first-call TLS?
Stack: Pipecat 1.x (Python, uv), Whisper large-v3-turbo via mlx-whisper, Silero VAD, Smart Turn v3, ElevenLabs websocket TTS, SmallWebRTC, running as a macOS LaunchAgent on an M1 Max (64 GB). The brain is a custom OpenAI-compatible endpoint in front of our agent.