Hands-onIntermediate 35 min· Working with LLM APIs · 3 of 19

Streaming responses to a UI

Stream a model's answer token by token: first in the terminal, then through a FastAPI endpoint to a plain JavaScript page with a Stop button. Learn why it feels faster and how to handle errors mid-stream.

What you will build: a tiny chat page where the answer appears word by word, like in ChatGPT. You will first stream to the terminal, then build a FastAPI backend that streams text, and a plain HTML and JavaScript page that reads the stream and has a Stop button.

Prerequisites: the lesson *Your first LLM API call (free Gemini API)* (an API key in .env, the google-genai package), and basic Python and JavaScript. FastAPI is introduced in the Backend Development track; here you only need the few lines shown.

Why stream?

An LLM writes one token at a time. Without streaming, your user stares at a spinner until the entire answer is finished, which can take many seconds. With streaming, the server forwards each piece as it is produced, so the first words appear quickly. Total time is the same, but it feels much faster, and users can start reading or stop a wrong answer early.

Normal vs streaming

Aspect	Normal call	Streaming call
First text appears	After the full answer is ready	After the first chunks
Code	One response object	A loop over chunks
Good for	Background jobs, JSON output	Chat and anything a person watches
Tricky parts	Few	Errors mid-way, cancelling, parsing partial data

Step 1: Stream to the terminal

stream_terminal.py

from dotenv import load_dotenv
from google import genai

load_dotenv()
MODEL = "gemini-flash-latest"  # use a current model ID
client = genai.Client()

stream = client.models.generate_content_stream(
    model=MODEL,
    contents="Explain how a hash map works, in about 120 words.",
)

for chunk in stream:
    if chunk.text:  # some chunks carry no text (for example the last one)
        print(chunk.text, end="", flush=True)
print()

generate_content_stream returns an iterator. Each chunk holds the next piece of text.

end="" stops print adding a newline after each piece, and flush=True pushes the text to the screen immediately. Run it and you will see the answer type itself out. A chunk is not always one word: it can be a few words or part of one.

Step 2: Stream from a backend

A browser must never hold your API key, so the browser calls your server, and your server calls Gemini and forwards the pieces. FastAPI can do this with StreamingResponse, which accepts a generator that yields text.

Terminal

pip install -U fastapi "uvicorn[standard]" google-genai python-dotenv

main.py

from dotenv import load_dotenv
from fastapi import FastAPI
from fastapi.responses import FileResponse, StreamingResponse
from google import genai
from google.genai import types
from pydantic import BaseModel, Field

load_dotenv()
MODEL = "gemini-flash-latest"  # use a current model ID
client = genai.Client()
app = FastAPI()

SYSTEM = (
    "You are a friendly assistant for Brew & Bean, a coffee shop in Indiranagar, "
    "Bengaluru. Keep answers short and helpful."
)


class ChatRequest(BaseModel):
    message: str = Field(min_length=1, max_length=2000)


def generate(message: str):
    try:
        stream = client.models.generate_content_stream(
            model=MODEL,
            contents=message,
            config=types.GenerateContentConfig(
                system_instruction=SYSTEM, max_output_tokens=800
            ),
        )
        for chunk in stream:
            if chunk.text:
                yield chunk.text
    except Exception as e:
        # The HTTP status is already 200 once streaming starts,
        # so report the problem inside the stream itself.
        print("Stream error:", e)
        yield "\n\n[Sorry, something went wrong. Please try again.]"


@app.post("/chat")
def chat(req: ChatRequest):
    return StreamingResponse(generate(req.message), media_type="text/plain; charset=utf-8")


@app.get("/")
def home():
    return FileResponse("index.html")

/chat streams plain text; / serves the page we write next. The max_length guard stops huge inputs from costing you money.

Errors in the middle of a stream

Once a streaming response has started, the status code is already sent as 200. You cannot change it to 500 later. That is why the code catches errors inside the generator and writes a message into the stream. Also log the real error on the server and show the user a friendly one, never the raw exception.

Step 3: The page that reads the stream

The browser's fetch gives you response.body, a stream of bytes. You read it in a loop with a reader, decode the bytes to text, and append the text to the page. This works with POST requests, which the simpler EventSource API does not.

index.html

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Brew & Bean assistant</title>
  <style>
    body { font-family: system-ui, sans-serif; max-width: 640px; margin: 2rem auto; padding: 0 1rem; }
    #answer { white-space: pre-wrap; min-height: 6rem; padding: 1rem; border: 1px solid #ccc; border-radius: 8px; margin-top: 1rem; }
    input { width: 100%; padding: .6rem; box-sizing: border-box; }
    button { margin-top: .5rem; padding: .5rem 1rem; }
  </style>
</head>
<body>
  <h1>Ask Brew & Bean</h1>
  <input id="q" placeholder="e.g. What goes well with a cold brew?">
  <button id="send">Ask</button>
  <button id="stop" disabled>Stop</button>
  <div id="answer"></div>

  <script>
    const q = document.getElementById("q");
    const sendBtn = document.getElementById("send");
    const stopBtn = document.getElementById("stop");
    const answer = document.getElementById("answer");
    let controller = null;

    async function ask() {
      const message = q.value.trim();
      if (!message) return;

      answer.textContent = "";
      sendBtn.disabled = true;
      stopBtn.disabled = false;
      controller = new AbortController();

      try {
        const res = await fetch("/chat", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({ message }),
          signal: controller.signal,
        });
        if (!res.ok) throw new Error("Server error " + res.status);

        const reader = res.body.getReader();
        const decoder = new TextDecoder();
        while (true) {
          const { done, value } = await reader.read();
          if (done) break;
          // stream:true keeps multi-byte characters (like Hindi) intact across chunks
          answer.textContent += decoder.decode(value, { stream: true });
        }
      } catch (err) {
        if (err.name !== "AbortError") {
          answer.textContent += "\n[Could not get an answer. Try again.]";
        }
      } finally {
        sendBtn.disabled = false;
        stopBtn.disabled = true;
      }
    }

    sendBtn.addEventListener("click", ask);
    q.addEventListener("keydown", (e) => { if (e.key === "Enter") ask(); });
    stopBtn.addEventListener("click", () => controller && controller.abort());
  </script>
</body>
</html>

Save next to main.py. It uses textContent, not innerHTML, so model output can never inject HTML into your page.

Terminal

uvicorn main:app --reload

Open http://127.0.0.1:8000 in your browser.

What you should see

You type: What goes well with a cold brew?
The answer appears gradually: "A Butter Croissant or Banana Bread is a lovely match..."
Pressing Stop freezes the text where it is and re-enables the Ask button.

What each piece does

The data path

1The browser sends the question with fetch and an AbortController so it can cancel later.
2FastAPI runs generate, which asks Gemini to stream and yields each text chunk.
3StreamingResponse sends every yielded piece to the browser straight away instead of waiting.
4The browser's reader gets the bytes, TextDecoder turns them into text and the page appends it.
5Stop calls controller.abort(); the browser closes the connection and the loop ends.

Everything appears at once?

If the text arrives in one big lump instead of smoothly, something between you and the browser is buffering the response. This can happen behind some proxies and when you add compression. Test locally first. Behind nginx you may need to turn proxy buffering off for this route.

Going further

Chat history with streaming. The SDK's chat helper has a streaming version. It also stores the finished reply in its history, so follow-up questions work.

chat_stream.py

chat = client.chats.create(model=MODEL)

for chunk in chat.send_message_stream("What is a REST API?"):
    if chunk.text:
        print(chunk.text, end="", flush=True)
print()

for chunk in chat.send_message_stream("Give me a Brew & Bean ordering example."):
    if chunk.text:
        print(chunk.text, end="", flush=True)
print()

The second question works because the chat object remembers the first exchange.

Server-Sent Events (SSE). Plain text streaming is the simplest option. When you also need to send other kinds of events (for example "sources found" or "done"), the usual choice is SSE: each message is a line data: ... followed by a blank line, with the media type text/event-stream. Many chat UIs use it. The idea is identical: the server flushes small pieces, the client reads them as they come.

Frameworks. If you build with React, libraries exist that wrap this fetch-and-read loop. Learn the raw version first so you understand what they do.

Things to watch. Streaming does not reduce cost. Structured JSON output is hard to use while streaming because a half-finished JSON object cannot be parsed; for JSON, a normal call is usually better (see the next lesson). Rate limits still apply, and a user who closes the tab may leave your server generating text nobody reads, so enforce limits per user.

Prompt · Add streaming to my app

I have a [FastAPI / Express] backend that calls the Gemini API with the google-genai SDK and a [React / plain JS] frontend. Show me how to change the endpoint to stream the answer and how to read the stream in the frontend with fetch. Include a Stop button using AbortController and an error message if the stream fails midway.

Backend code:
[paste]

Frontend code:
[paste]

What to expect: Check that the API key stays on the server, that errors are handled inside the stream, and that output is inserted as text and not as HTML.

Try it: Add typing feedback and a copy button

Extend index.html: show the text "Thinking..." until the first chunk arrives, then replace it with the answer. After the stream ends, show a Copy button that copies the answer with navigator.clipboard.writeText.

Key takeaways

Streaming sends tokens as they are generated, so the answer feels faster even though total time is similar.
generate_content_stream returns chunks; check chunk.text before using it.
Keep the API key on the server; stream to the browser with StreamingResponse and read it with fetch and a reader.
After streaming starts the status code is fixed, so report errors inside the stream.
Use an AbortController for a Stop button, and textContent to avoid HTML injection.
Streaming does not cut cost, and partial JSON cannot be parsed, so use it for human-facing text.

Check yourself

Finishing the quiz marks the lesson as done.

1. What is the main benefit of streaming an LLM response?

2. Why does the browser call your server instead of calling Gemini directly?

3. An exception happens halfway through a streamed answer. Why can't you just return HTTP 500?

4. Why use textContent instead of innerHTML when showing model output?

5. Why is streaming a poor fit for structured JSON output you want to parse?

Saved on this device. Create a free account to keep it on every device.

Tokens, cost, latency and choosing a model Structured output with JSON schemas

Practise this with Sam

A real mock interview, out loud, with an honest scorecard. Free to start.

Start free