Models don't remember anything. Memory is an illusion — and you're the one building it.
It don't remember nothing — not your name, not your face, Every call's a blank page, every turn's a clean slate. Memory's an illusion that you build from the ground — What you keep, what you cut, what you carry around.
I The Stateless Illusion
When you chat with ChatGPT, it feels like it remembers your conversation. It references things from three messages ago. It builds on previous answers. But every single API call is completely independent.The model has zero memory between calls. Your application re-sends the entire conversation with every new message. The model isn't "remembering" — it's re-reading from scratch. The world's most diligent goldfish.
Analogy
Think of it like a doctor who reads your entire medical file before every appointment, then develops complete amnesia the moment you walk out the door. The file creates the continuity, not the doctor's brain.
This reframe is critical. You're not configuring the model's memory — you're designing what gets re-sent.Every "memory" feature is really a decision about what context to include in the next API call. And since context windows are finite, those decisions involve real tradeoffs.
II Conversation History Strategies
Three fundamental approaches, and every production AI app uses some variation.
Full History
Send the entire conversation every time. Implementation is trivial, fidelity is perfect. But a 20-turn conversation hits 10,000+ tokens easily. A 100-turn support session blows through most context windows. Works for short conversations, doesn't scale.
Sliding Window
Keep only the last N turns. Simple, predictable token usage. The downside is brutal: if the user mentioned their name in turn 1 and you're at turn 15 with a 10-turn window, the model doesn't know their name anymore. It will ask again — or worse, confidently make one up.
Summarization
Periodically compress old messages into a summary. Preserves key facts in fewer tokens. The tradeoff: summaries lose nuance — tone, phrasing, the exact back-and-forth. And summarization itself costs money (another LLM call each time).
Builder Tip
No strategy is universally best. Use full history for short, high-stakes conversations. Sliding window for casual chat where recency beats history. Summarization for long sessions where key facts must persist. Most production apps use a hybrid.
Simulate
Memory Strategy Simulator
Step through a 20-turn conversation and see what each memory strategy retains.
Turn:1 / 20
Hi, I need help with my account.
Tokens used:30 / 8,192
III Smart Summarization
Progressive summarization is the first key technique: summarize the previous summary plus new messages rather than re-summarizing from scratch. Cheaper and faster, but introduces information decay. Each round can lose fidelity — a photocopy of a photocopy of a photocopy. After five rounds, important details may have quietly vanished.
Entity extractioncombats this decay. Always preserve names, numbers, dates, and commitments. "Sarah" and "$49.99 refund" and "March 15th" should survive every summarization round. Some teams maintain a structured entity list alongside the narrative summary — a small JSON of key facts that never gets compressed away.
Key-fact trackingtakes this further: a structured list of critical facts (customer name, account details, commitments, preferences) that's cheap in tokens and ensures the model never loses the most important context.
Key Insight
Every summarization call is another LLM invocation — more latency, more money. A 100-message conversation might need 10 summarization calls. You need to decide: how often to trigger summarization and what fidelity to maintain. These are product decisions that directly impact cost and experience.
IV External Memory
Sometimes the answer isn't fitting more into the context window — it's storing information outsideand retrieving only what's relevant.
Approach
What it does
Token cost
Best for
Vector databases
Store info as embeddings, retrieve semantically similar content per query
Only retrieved fragments
Large knowledge bases, RAG
User profiles
Persist preferences and history across sessions
~200-500 tokens
Personalization
Knowledge bases
Domain expertise retrieved on demand (help center, docs)
2-3 articles per query
Support agents, RAG
This is the bridge to RAG — Retrieval-Augmented Generation — one of the most important patterns in production AI. External memory turns a fixed-size context window into an effectively unlimited knowledge base.The challenge shifts from "how do we fit everything in?" to "how do we retrieve the right things?" And retrieval quality — not model intelligence — often determines whether your product feels smart or frustrating.
Design
Design Your Memory System
For each app scenario, pick the best memory strategy. Consider the tradeoffs.
1 of 3
Casual AI Chatbot — "A fun conversational chatbot for a social app. Conversations average 8–12 turns. Users expect it to remember what they just said but don't expect long-term memory. Budget is tight."
V The Memory Stack
Production AI applications don't use a single memory strategy — they layer their context window like a stack. Understanding this architecture is essential for anyone designing conversational AI features.
Here's the pattern, from top to bottom:
System prompt(fixed) — The model's personality, rules, and capabilities. Always present at the top of the context. This is your product's DNA.
User profile (retrieved) — Preferences, history, account details. Loaded per user at the start of each session. Cheap in tokens, high in personalization value.
Summarized history (compressed) — Key facts from older conversation turns. The distilled essence of what happened before the recent window.
Recent messages (full fidelity) — The last N turns at full detail. This is where nuance, tone, and exact wording are preserved.
Retrieved context (dynamic) — Relevant documents, code snippets, knowledge base articles, or data pulled in per query. Changes with every message.
The order matters— and if you read Part 1, you know why. Research on the "lost in the middle" problem shows that models pay most attention to content at the beginning and end of their context. So: system prompt at the start, the user's latest message at the end, and everything else layered in between, ordered by importance. The retrieved context sits close to the user's message because it's most relevant to the current query.
Builder Tip
Order matters in the memory stack. Models pay most attention to the beginning and end of context — the "lost in the middle" problem from Part 1. Getting this wrong is like burying the lead on page 7 of a memo nobody finishes. Put system instructions first, recent messages last, and retrieval results near the end.
Every layer competes for the same finite context window. Spending more tokens on your system prompt means fewer tokens for retrieved context. A longer conversation history means less room for knowledge base articles. The widget below lets you feel these tradeoffs directly.
Build
The Memory Stack Builder
Allocate your context budget across the memory stack layers. Click a preset to see recommended configurations.
System Prompt500
User Profile200
Summarized History500
Recent Messages2,000
Retrieved Context1,000
Free
Total:4,200 / 200,000 tokens (2%)
VI Wrapping Up
Takeaway
Memory is application logic, not model magic. The model is stateless. What it sees is what you send.
Match strategy to use case. Full history, sliding window, summarization, retrieval — each has tradeoffs. No universal best.
The memory stack is the architecture. System prompt, user profile, summarized history, recent messages, retrieved context — layered by importance, constrained by the window.
External memory extends context beyond any window. Vector databases and retrieval turn a finite window into an effectively unlimited knowledge base.
These concepts are foundational. They shape how you design every conversational AI feature — from a simple chatbot to a multi-session assistant. Every time a user says "why did it forget?" or "how does it know that?", the answer traces back here.
Test your understanding
Article Recap
5 questions covering the key concepts from this article.
1 of 5
A stakeholder reviews your AI chatbot prototype and says: "The model clearly remembers users between sessions — it greeted a returning user by name." How should you explain what's actually happening?