Design a real-time voice assistant

If each stage waits for the previous one, you get ASR 1 s + LLM 2 s + TTS 1 s = 4 s of silence, which feels broken. The fix is streaming at every boundary. ASR streams partial results as the user speaks, so the final transcript is ready ~50-100 ms after they stop. The LLM streams tokens, and a sentence splitter hands the first clause to TTS as soon as it ends (at a comma or full stop, with a minimum of ~5 words so speech sounds natural). TTS streams audio chunks so the first sound plays in ~100-150 ms. Spoken replies are prompted to be short, which also reduces time to the first sentence. The biggest remaining item is endpointing: waiting 800 ms of silence to be sure the user finished is safe but slow, and 200 ms cuts people off. A turn-detection model that combines silence with the meaning of the words lets you end turns at ~250 ms when the sentence is complete and wait longer after 'my number is 98...'. Other wins: keep the LLM prompt prefix stable so provider prompt caching cuts time to first token; colocate orchestrator, ASR and TTS in one region to keep internal hops under 5 ms; speculative LLM calls on a stable partial transcript, discarded if the final differs. Measure each stage per turn, because the p95 is usually dominated by one stage.

Natural conversation means the user talks over the assistant. Three things must work. First, detection: the device microphone also hears the assistant's own voice, so echo cancellation on the device (WebRTC's built-in AEC) is essential, and the server VAD needs a minimum speech duration (~150-200 ms) so a cough or a backchannel 'haan' does not stop the reply. Second, stopping fast: the client holds a small playout buffer and must flush it on a stop event; the server cancels the TTS stream and the LLM stream so you stop paying for tokens no one will hear. Target under ~300 ms from the user starting to talk to silence. Third, state correctness: the LLM generated a full paragraph, but the user heard only the first sentence. If you store the full paragraph as the assistant's turn, the next reply may refer to things the user never heard. The client reports a playback cursor (audio milliseconds played), the orchestrator maps it back to text via TTS word timestamps, and stores only the heard part. This detail is what separates a demo from a production voice agent.

A cascaded pipeline (ASR → LLM → TTS) is modular: you can pick the best Hindi ASR, the cheapest capable LLM and a natural Indian-English TTS voice, inspect text at every step, apply text-based safety and PII redaction, and log a readable transcript. Its costs are the extra latency at each boundary and loss of tone: the LLM never hears that the user sounds angry. Speech-to-speech realtime models take audio in and produce audio out in one model, often reaching lower latency and more natural turn-taking, and they understand prosody. Their downsides today: less control over each step, harder evaluation and safety filtering, fewer choices of voice and language quality for Indian languages, vendor lock-in, and usually a higher price per minute. A strong answer designs the orchestrator so the 'brain' is pluggable: the same session, tool, confirmation and audit layers work whether the middle is a cascade or one realtime model. Many teams use a cascade for regulated flows (banking, where every word must be logged and checked) and realtime models for open-ended assistants.

A voice session is stateful and latency-sensitive, so it should be owned by one orchestrator process for its whole life, close to the user. The session service picks a region by measured latency, and the media edge and orchestrator are pinned for the session; load balancing happens only at session start, by current concurrent sessions per node, not by request. Use WebRTC over UDP: Opus with forward error correction handles 2-5% packet loss, and the jitter buffer smooths uneven arrival, whereas a WebSocket over TCP stalls on every lost packet (head-of-line blocking). Keep a TCP/443 fallback for networks that block UDP. Phone calls arrive as 8 kHz narrowband audio, so the ASR needs a telephony-tuned model. For Indian users, ASR must handle code-switching within a sentence ('mera order kab aayega, it's been 3 days'); choose a model trained on Hinglish rather than running language detection per utterance. Numbers, names and addresses are the main error source, so confirm them back, and let users type or tap when ASR confidence is low.

WebRTC gives UDP transport, Opus with packet-loss concealment, jitter buffers, echo cancellation and congestion control, built into browsers and mobile SDKs. It needs TURN servers and more operational work. WebSockets are simple but run over TCP, so one lost packet delays all following audio. Use WebRTC by default with a WebSocket fallback.

A 200 ms silence timeout feels snappy but cuts users off mid-thought, especially when reading out numbers. A 900 ms timeout is safe but feels slow. A semantic turn detector plus a context-dependent timeout (longer after a question that asks for an order number) gives both.

Bigger models reason better but have higher time to first token. Voice turns are mostly short and tool-driven, so a fast mid-size model with good tool calling usually wins; escalate to a larger model only for complex turns, accepting a filler phrase ('let me check that') to cover the delay.

Concurrent streams, not requests per second, size the fleet. At 10× sessions, pre-scale before announced sales, use server-side VAD so silent audio never reaches ASR, cap concurrent sessions per region with a polite 'please try in a minute' or a text chat fallback, and spill over to a second region if latency allows.

3,600 calls/s can hit provider limits. Use the AI gateway with multiple providers and per-feature token budgets, keep prompts short and prefix-cached, and route simple intents (order status) to a deterministic flow that skips the LLM entirely.

The media edge detects the lost connection and asks the session service for a new owner; the new orchestrator loads the Redis snapshot from the last completed turn and says 'Sorry, I missed that, could you repeat?'. At most one turn is lost; tool calls are safe to retry because they carry idempotency keys.

Health checks on time-to-first-audio trigger failover to a secondary engine (possibly a less natural voice). If ASR fails entirely, the app switches to a text chat with the same session state; on a phone call, the call is transferred to a human agent queue.

After ~1 s with no first sentence, the orchestrator plays a short filler ('One moment, checking your order') from pre-recorded audio so the line is never silent, and the gateway hedges the request to a backup model after a deadline.