Tokens, context windows and why chats forget
AI models read and write tokens, not words, and can only look at a limited amount of text at once. Learn what tokens and context windows are, why long chats drift or forget, and how to work around it.
Have you ever had a long chat with an AI where, an hour in, it forgot something you told it at the start? Or pasted a big document and got an answer that ignored half of it? This is not carelessness. It comes from two ideas: tokens and the context window.
Think of the model as a student sitting an open-book exam with a desk of fixed size. Everything on the desk (your messages, its replies, pasted files) is visible. Anything that does not fit on the desk simply is not seen.
What is a token?
A token is the small chunk of text a model works with. It can be a whole short word (" coffee"), part of a longer word ("cappu" + "ccino"), a number, a punctuation mark or a space. The model never sees raw letters; it sees a list of token numbers.
For ordinary English, a token is roughly three-quarters of a word, so 100 words is about 130 tokens. This is only a rule of thumb. The exact split depends on the model's tokeniser, and text in Hindi, Tamil, Telugu or other Indian languages often needs more tokens per word than English, so the same message uses up more of your limit.
Illustration: how a sentence might be split (real tokenisers differ)
| Text | Possible tokens | Count |
|---|---|---|
| Hello | Hello | 1 |
| Cappuccino | Capp / uccino | 2 |
| I love filter coffee | I / love / filter / coffee | 4 |
| https://brewbean.example/menu | https / :// / brew / bean / .example / /menu | 6 or more |
Why does it matter? Three reasons. First, models have a limit on how many tokens they can handle. Second, paid APIs charge by tokens (input and output), so developers care about it. Third, it explains odd behaviour: a model sees "strawberry" as a couple of chunks, not ten letters, which is why counting letters or reversing words can go wrong.
The context window
The context window is the maximum number of tokens the model can consider in one go. It includes everything: the app's hidden instructions, your messages, the model's earlier replies, any files you attached and the answer it is currently writing.
Chat apps create the feeling of a conversation by re-sending the whole conversation so far with each new message. The model has no memory between calls. If the chat gets longer than the window, the app has to drop or shorten the oldest parts, and the model genuinely cannot see them any more.
What fills the window
| Part | Example |
|---|---|
| Hidden system instructions | The app's rules for how the assistant should behave |
| Your earlier messages | "I am a 2nd-year B.Tech student..." |
| The model's earlier replies | Its long explanations count too |
| Attached files and pasted text | A 40-page PDF, an error log |
| The answer being written | Output tokens share the same window |
Window sizes have grown a lot, from a few thousand tokens to hundreds of thousands, and in some models over a million. They differ by model and change often, so check the tool's own documentation instead of trusting a number from an article. Bigger is not automatically better, as the next section shows.
Why chats seem to forget
Common symptoms and their causes
| Symptom | Likely cause | What to do |
|---|---|---|
| It ignores a rule you gave at the start | Early text was cut off, or buried among much newer text | Repeat the rule, or start a new chat with a summary |
| Answers get vaguer in a very long chat | Lots of old, mixed topics compete for attention | New chat per topic |
| It misses a detail in the middle of a long document | Models can pay less attention to the middle of very long input | Ask about one section at a time; quote the part you mean |
| It stops mid-answer | Hit the output limit | Say "continue" or ask for it in parts |
| It "remembers" you in a new chat | The app has a memory feature that saves notes and feeds them back as text | Check memory settings; turn it off if you prefer |
The summary-and-restart trick
When a chat gets long and messy, ask: "Summarise our conversation so far in under 200 words: my goal, decisions made, and open questions." Copy the summary, open a new chat and paste it as your first message. You keep the useful context and drop the clutter.
Summarise this conversation so I can continue it in a new chat. Include:
1. My goal and who I am (level, situation).
2. Decisions and facts we have agreed.
3. What is still unresolved.
4. Any rules or style preferences I gave you.
Write it as a message I can paste straight into a new chat, under [200] words.What to expect: Read the summary before reusing it. If it dropped something important, add it by hand.
Working with the limit
Habits that save your context window
- One topic per chat; start a new chat for a new task.
- Put the most important instructions at the start and repeat them at the end of long prompts.
- Paste only the relevant part of a document, not everything.
- Ask for short answers when you do not need long ones, because replies use up the window too.
- For long documents, work section by section and ask the model to quote the lines it used.
- Keep important facts (names, numbers, deadlines) in your own notes; do not rely on the chat to hold them.
A big window does not guarantee perfect recall
Even when your text fits, the model may overlook details in very long inputs. For anything important, ask it to point to the exact passage that supports its answer, then check that passage yourself.
How this comes up in interviews
For developer and data roles, expect questions like "What is a context window?" or "How would you handle a document longer than the context window?". A solid short answer: the context window is the token limit for input plus output; for long documents you split into chunks, summarise or retrieve only the relevant chunks (retrieval-augmented generation, RAG), instead of pasting everything.
Try it: Test the forgetting yourself
1. Open a new chat in any free AI tool. Say: "For this whole chat, end every reply with the word BREW." 2. Ask five or six unrelated questions (a recipe, a maths problem, a history fact and so on) with long answers. 3. Does it still add BREW? When does it stop, if at all? 4. Now paste the same rule again and see whether it returns.
Then ask the same chat: "Roughly how many tokens is this conversation, and what is your context limit?" and note how sure or unsure it sounds.
Key takeaways
- Models read and write tokens: chunks of text, roughly three-quarters of an English word on average.
- The context window is the token limit for everything the model can see at once, including its own reply.
- Chat apps re-send the conversation each time; what falls outside the window is invisible to the model.
- Long chats drift; use one topic per chat and the summary-and-restart trick.
- Bigger windows help but do not guarantee the model uses every detail; ask for quotes and verify.
Check yourself
Finishing the quiz marks the lesson as done.
1. What is a token?
2. What does the context window include?
3. Your 90-minute chat now ignores a rule you gave at the start. What is the best fix?
4. Why might a Hindi message use up more of the limit than the same English message?
5. A document is longer than a model's context window. Which approach is sensible?
Saved on this device. Create a free account to keep it on every device.
Practise this with Sam
A real mock interview, out loud, with an honest scorecard. Free to start.