InterviewSam
Log inStart free
Interview questions
AI-era systemsMediumSDE-3

Your chat API streams LLM answers over SSE. At 500 new chats per second, each streaming for about 20 seconds, servers run out of memory and DB connections even though CPU is at 30%. How do you design the serving tier?

Asked at
AtlassianJioHotstarOpenAI

Think first

Answer out loud as you would in the room — clarify, give numbers, name the trade-off — then compare with the model answer.

Go deeper

Design a high-concurrency streaming API server Lesson: Streaming LLM responses at scale

Saved on this device. Create a free account to keep it on every device.

Practise this with Sam

A real mock interview, out loud, with an honest scorecard. Free to start.

Start free