Tokens, cost, latency and choosing a model

Every LLM API bills, limits and slows down by one unit: the token. If you understand tokens, you can predict what a feature will cost, why it feels slow, and which model to pick. This lesson gives you the mental model and a few lines of code to measure it. For a gentler introduction to tokens and context windows, see *Tokens, context windows and why chats forget* in AI Foundations.

A token is a chunk of text, usually a short word, part of a longer word, a number or a punctuation mark. The model never sees letters or words, only token IDs. As a rough rule for English, one token is about four characters, or three-quarters of a word. This is only an estimate. Other languages, including Hindi, Tamil and other Indian languages, usually need more tokens for the same meaning, so the same sentence can cost more and fill the context window faster. Code and JSON are also token-hungry because of symbols and indentation.

Do not guess. The API can count for you, and every response tells you exactly what was used.

from dotenv import load_dotenv
from google import genai

load_dotenv()
MODEL = "gemini-flash-latest"  # use a current model ID
client = genai.Client()

text = "Brew & Bean opens at 8 am and closes at 10 pm every day."

# 1) Count BEFORE sending (cheap, does not generate anything)
counted = client.models.count_tokens(model=MODEL, contents=text)
print("Input tokens:", counted.total_tokens)

# 2) Read the real numbers AFTER a call
response = client.models.generate_content(model=MODEL, contents="Summarise: " + text)
u = response.usage_metadata
print("Prompt tokens:", u.prompt_token_count)
print("Output tokens:", u.candidates_token_count)
print("Total tokens:", u.total_token_count)
print("Thinking tokens:", u.thoughts_token_count)  # None if the model did not think

Counts input tokens before sending, then prints the usage numbers from the response.

A request has up to three kinds of tokens, and providers usually price them differently.

Two consequences follow. First, the whole history is input every turn in a chat, so a 20-turn conversation costs far more per message than a fresh question. Second, long outputs cost more and also take longer, because the model writes one token at a time.

The formula is simple: cost = (input tokens x input price) + (output tokens x output price), where prices are quoted per million tokens. Real prices change and differ by model, so look them up on the provider's pricing page rather than trusting a blog post or this lesson. Below, the prices are made-up numbers just to show the arithmetic.

# Hypothetical prices, in rupees per 1 million tokens. NOT real prices.
INPUT_PRICE = 25.0
OUTPUT_PRICE = 200.0

def estimate_cost(input_tokens, output_tokens):
    return (input_tokens * INPUT_PRICE + output_tokens * OUTPUT_PRICE) / 1_000_000

# A support bot: 1,500 input tokens (instructions + retrieved docs + question)
# and 300 output tokens per answer, 10,000 answers a month.
per_call = estimate_cost(1500, 300)
print(f"Per call:  Rs {per_call:.4f}")
print(f"Per month: Rs {per_call * 10_000:.2f}")

Run it and swap in the real prices from the pricing page for your model.

Latency has two parts. Time to first token is how long before anything appears. It grows with input size and with any thinking the model does. Generation speed is how many tokens per second arrive after that. Total time is roughly the first-token wait plus output length divided by speed.

Providers offer a family of models: small and fast, a balanced middle, and large and slow. Names change often, so learn the pattern rather than memorising IDs. For Gemini the pattern has been Flash-Lite (cheapest and fastest), Flash (balanced) and Pro (strongest). Check the models page in the Gemini API docs for what exists today.

The method that works: start with the cheapest model that might work, test it on 20 to 50 real examples, and move up only if quality is not good enough. Most beginners do the reverse and overpay. The evaluation lesson later in this track shows how to measure quality properly.

The context window is the maximum number of tokens (input plus output) a model handles in one request. Modern models have large windows, but a bigger prompt costs more, runs slower and can make the model miss details in the middle. Do not stuff a whole book into every call: retrieve only what is relevant (the RAG lessons cover this).

Rate limits cap how many requests and tokens you can use per minute and per day. Free tiers have tight limits. When you exceed them you get HTTP 429, and the fix is to slow down, retry with a delay, batch work, or move to a paid tier.

Interviewers for AI engineer roles often ask: "How would you reduce the cost of this feature?" or "Why is the response slow?" Strong answers name concrete levers: measure tokens first, shorten prompts, retrieve less, cache repeated prompts, use a smaller model for easy cases (routing), cap output, and stream. Weak answers say "use a cheaper model" and stop there.

Here is my system prompt and a typical request. Identify wasted tokens: repeated instructions, unneeded examples, verbose wording, data the model does not need. Rewrite the prompt to be at least 30 percent shorter while keeping the same behaviour, and list what you removed and why.

System prompt:
[paste]

Typical user request:
[paste]

What to expect: Check the shortened prompt on your test cases. Shorter is only better if answers stay as good.