Design an AI mock interviewer

People expect a reply within about a second in conversation; at two seconds the AI feels slow, and if it interrupts the candidate it feels rude. Every stage must stream. Audio travels over WebRTC (UDP, jitter buffers, echo cancellation in the browser) to a media server in an Indian region, because a round trip to a US region alone adds ~250 ms. Streaming STT returns partials continuously, so final text is ready ~100 ms after speech ends. The hardest part is end-of-turn detection: fixed silence thresholds either cut candidates off mid-thought or add a long pause. Combine voice activity detection with a small model that judges whether the utterance is complete, use a longer threshold during problem solving ('hmm, so...') and allow the candidate to press a 'done' key in coding rounds. The LLM must start fast: use a fast model for live turns, keep the prompt prefix cacheable, ask for short answers, and send text to TTS sentence by sentence. Support barge-in: stop playback immediately when the candidate speaks and tell the conductor what was actually spoken, so the transcript matches what the candidate heard. Measure the full voice-to-voice latency per turn, not per component.

If you give an LLM 'you are an interviewer, run a 45-minute system design interview' it will drift: spend 30 minutes on requirements, forget to ask about scaling, give away answers, or end early. So the server owns the structure. The interview plan comes from the question bank and the level: sections with time budgets (requirements 5 min, high-level design 15, deep dive 15, wrap-up 5), the topics each section must cover, and a follow-up tree of probes. The conductor tracks time used, topics covered and hints given in a state object, and each turn tells the LLM what to do next ('candidate has not mentioned caching; ask about read latency'; 'time for this section is up; transition to deep dive'). A cheap classifier call can update 'topics covered' from each answer. The LLM's job is wording, natural follow-ups and judging whether an answer needs a probe. This gives consistency between candidates, keeps interviews on time, and produces grader notes (which hints were given) that make scores fairer. State is snapshotted after every turn, so if a server dies another instance reloads it and continues, telling the candidate 'sorry, I lost you for a moment'.

The report is the product's credibility. Grade after the interview, not live, with a stronger model that sees the whole transcript, final code and plan notes. Use an explicit rubric per interview type and level, with descriptions of what a 2, 3 and 4 look like for each dimension, and require evidence: every score must cite quotes with timestamps, which the UI links to the replay. This makes scores explainable and reduces made-up praise. Run at temperature 0, and check consistency: grade twice (or with two models) and if dimension scores differ by more than one point, re-grade or show lower confidence. Use facts where possible: code test results from the runner, time taken, hints used. Calibrate against humans: a panel of experienced interviewers grades a few hundred interviews; track agreement (for example the correlation or the share within one point) and tune rubric wording until the model agrees with humans about as well as humans agree with each other. Check fairness: compare score distributions across accents and languages for transcripts of similar quality, because STT errors on Indian English accents can make an answer look worse than it was. Version the rubric and prompt, and never compare scores across versions without saying so.

A voice session is stateful and long, unlike a web request. Media servers and voice pipeline workers are assigned per session (sticky), scaled on concurrent sessions rather than CPU, and drained gracefully on deploy: a server stops taking new sessions and waits for live ones to finish (up to an hour), so deploys need long drain windows or connection migration. Placement season gives sharp evening peaks; scale ahead by schedule, and offer booking slots when capacity is short instead of failing mid-interview. Plan limits are checked when an interview is created, and the counter is consumed only when the interview actually starts, so a failed connection does not cost the user a credit. Cost per interview is dominated by STT and TTS minutes, not the LLM: transcribe only the candidate's audio, pause STT when muted, and use a cheaper TTS voice for free tiers. Degrade in steps: if TTS fails, show the interviewer's text; if STT fails or the network is weak (common on Tier-2/3 mobile data), switch to text mode with the same conductor and state; if the LLM provider fails, the gateway fails over to a second fast model. The interview continues in every case.

Real-time speech-to-speech models give lower latency and more natural turn-taking. The cascaded pipeline gives a text transcript at every step, control over each component (choose STT that handles Indian accents, any LLM, any voice), easier grading and logging, and cheaper fallbacks to text. For an assessment product where the transcript is the evidence, the cascade is the safer default; a speech-to-speech model can be tried later for the live part only.

Grading during the interview could adapt difficulty and give instant results but adds latency and cost to every turn and grades without full context. Lightweight live signals (topics covered) plus full grading afterwards gives better scores at lower live latency.

WebSocket audio is simpler to build but runs over TCP, so packet loss on mobile networks causes stalls. WebRTC handles loss, jitter and echo cancellation and is the standard for voice. Use WebRTC for audio and a WebSocket for control and code sync.

Providers limit concurrent streams. At 7,000 sessions you may need several accounts or providers, routed by a voice gateway with failover, or a self-hosted STT model on GPUs for the bulk of traffic.

Late in an interview the full transcript is long, making each turn slower and costlier. Keep recent turns verbatim, replace earlier sections with summaries generated at section ends, and rely on prompt caching for the stable prefix.

The client's WebRTC connection drops; it reconnects through signalling to a new worker, which loads plan state and transcript from Redis/Postgres. The interviewer says it lost the connection briefly and repeats the last question. Time lost is credited back.

Packet loss and high jitter are detected from WebRTC stats; the app suggests switching to text mode, keeping the same session and state.

The consistency check catches large disagreements and re-grades; if still inconsistent, the report shows 'low confidence' for that dimension and the case goes into a human review sample used for calibration.