Context Engineering

I Beyond Prompting

Parts 1 and 2 taught you the constraints: context windows are finite, position matters, and memory is an illusion you build. This article is about the craft of building that illusion well.

The shift is bigger than it sounds. Context engineering is the new prompt engineering. Prompt engineering is crafting a single string of instructions. Context engineering is designing the entire system that assembles what the model sees — instructions, history, retrieved documents, tool definitions, user data, output constraints — dynamically, per request, at runtime.

Tobi Lütke, CEO of Shopify, defines it precisely: "Context Engineering is the art of providing all the context for the task to be plausibly solvable by the LLM." Not just a good prompt. All the context. Assembled correctly. At the right time.

This matters because most agent failures are not model failures anymore — they're context failures. Two products using the same model on the same task can produce dramatically different results. One feels magical; the other feels like a cheap demo. The model is identical. The context isn't.

II System Prompt Design

The system prompt is the most expensive real estate in your context window. It's sent on every request, sits at the position of highest attention (the top), and defines every behavior the model exhibits. And most teams write it once and never think about it again.

A well-designed system prompt has four layers, in order:

1. Identity and role.Who is the model? Not "You are a helpful assistant" — that's the default. "You are a senior tax advisor specializing in US corporate tax for companies with $10M-100M revenue. You are cautious about edge cases and always cite specific IRS code sections." Specificity constrains the trajectory. A model told it's a tax advisor will refuse to write poetry; a model told it's "helpful" will try anything and do most of it badly.

2. Capabilities and constraints.What can this model do and what can't it? "You have access to the company's financial database via the query_financialstool. You do NOT have access to employee personal data. If asked about personnel matters, say so and redirect to HR." Explicit boundaries prevent the model from hallucinating capabilities it doesn't have.

3. Output format and style.How should responses look? "Respond in JSON with keys: answer, confidence, sources. Keep answers under 200 words. Use a professional but approachable tone." Format constraints are the cheapest way to improve consistency — they cost a few tokens and save hours of post-processing.

4. Error handling and edge cases.What should the model do when it doesn't know? "If you are uncertain about a tax ruling, say 'I'm not confident about this — please verify with a CPA' rather than guessing. Never fabricate an IRS code section." This is where most system prompts fail. They define the happy path but not the failure path.

III Few-Shot Examples

Parts 1 and 2 never mentioned few-shot examples. They're one of the most powerful context engineering tools you have, and they're chronically underused.

A few-shot example shows the model a completed input-output pair: "Given this input, here's what a perfect response looks like." The model pattern-matches against the examples and produces output that follows the same structure, tone, and level of detail. It's teaching by demonstration rather than instruction.

How many? 2–3 examples hit the sweet spot. One example is often misread as a rigid template. Four or more consume too many tokens for diminishing returns. Exception: classification tasks, where 5–10 examples covering each category dramatically improve accuracy.

Which ones?Select examples that cover your most common case, your hardest edge case, and the output format you want. Don't pick three examples that all look the same — pick three that show the range of acceptable responses. If your model writes customer support replies, include: a straightforward answer, a response that politely declines, and a response that escalates to a human.

Where to place them? After the system prompt, before the conversation history. Examples in the system message get high positional attention and consistent treatment across turns. Placing them in the user message works for single-turn tasks but gets pushed further from the top as conversation grows.

Format matters. Use a clear delimiter between examples and the actual task. A simple pattern works:

## Example 1
User:What's the return policy for damaged items?
Response: {"answer": "Damaged items can be returned within 30 days with photos of the damage. We'll cover return shipping.", "confidence": "high", "sources": ["returns-policy-v3.md"]}
 
## Example 2
User: Can I get a refund in Bitcoin?
Response: {"answer": "We only process refunds to the original payment method. We don't support cryptocurrency refunds.", "confidence": "high", "sources": ["payment-methods.md"]}
 
## Now handle this:

IV Retrieval Quality Control

Part 1 covered where to put retrieved documents in the context window (edges, not middle). This section covers what to retrieve and when to retrieve nothing.

Relevance thresholds.Every retrieval system returns a similarity score. Most teams use the top-K results regardless of score. This is wrong. A query about "parental leave policy" might return "parking policy" as the third-best match if your corpus is small. Set a minimum similarity threshold and return fewer results when nothing is truly relevant. Three good chunks beat five chunks where two are noise.

Chunk size selection.Too small (100 tokens) and you lose surrounding context — a sentence about a return policy without the conditions that apply. Too large (1,000 tokens) and you waste budget on irrelevant content surrounding the relevant sentence. 200–500 tokens per chunk is the pragmatic range for most use cases. Overlap chunks by 10–20% so you don't accidentally split a key sentence at the boundary.

Retrieval isn't just search.A basic RAG pipeline embeds the user's query and finds similar chunks. Better pipelines add: query expansion (rephrase the question to catch different formulations), reranking (a second model scores the top-20 results and returns the top-5), and metadata filtering (only retrieve chunks from the relevant department, date range, or document type).

When to suppress retrieval.If no chunk exceeds your relevance threshold, don't inject anything. An empty retrieval result is better than a misleading one. The model is more likely to say "I don't know" with no retrieved context than with bad retrieved context — because bad context gives it something plausible to confabulate from.

V Token Compression

Every token costs money, latency, and attention. When your context window is tight, compression is the difference between a working product and an API error.

Structured data beats prose.A table of user preferences as JSON costs ~60% fewer tokens than the same information in natural language sentences. "The user prefers formal tone, works in finance, timezone EST, language English" is 15 tokens. The same as four bullet points is ~25 tokens. The same as a paragraph is ~40 tokens. Use the most compact format the model can still parse.

Summaries beat transcripts.Don't inject a 3,000-token meeting transcript when a 300-token summary captures the key decisions. But remember Part 2's warning: summaries lose nuance. Preserve named entities (names, numbers, dates, commitments) explicitly — they're the first casualties of compression and the hardest to recover.

What to cut first.When budget is tight, cut in this order: (1) Old conversation history beyond the recent 5 turns. (2) Low-relevance retrieved chunks. (3) Verbose few-shot examples — shorten them, don't remove them. (4) System prompt redundancies. Never cut: output token reservation, the current user message, or high-relevance retrieved context.

Graceful degradation.Build your context assembly to degrade gracefully under pressure. If the window is 90% full after system prompt + history + retrieval, your assembler should automatically switch from full history to summarized history, drop the weakest retrieved chunk, and warn in logs that context is running hot. Don't wait for an API error to discover you're over budget.

VI Debugging Context

When your AI product produces a bad output, the instinct is to blame the model. The first thing to actually check: what did the model see?

Log the full assembled context.Not just the user message — the entire payload: system prompt, few-shot examples, conversation history, retrieved chunks, tool definitions. When you can see exactly what the model saw, most "model failures" reveal themselves as context failures. The wrong document was retrieved. The summary lost a critical number. The system prompt contradicted itself.

Detect silent truncation.When your context exceeds the window, some APIs silently truncate from the beginning. Your system prompt — the most important part — vanishes first. The model suddenly "forgets" its role, its constraints, its format requirements. It's not a model regression. It's truncation. Count tokens before sending and warn if you're within 10% of the limit.

Test with the context, not the prompt.When debugging, don't paste your prompt into a playground and check if the model "can" produce a good answer. Paste the full assembled context — system prompt, history, retrieval results, everything. The model might produce a perfect answer from a clean prompt and a terrible answer from the same prompt buried under 6,000 tokens of conversation history and irrelevant retrieved documents.

Common patterns that break at scale:

• History bloat. Conversations that started crisp at turn 3 degrade at turn 30 because the history consumes the entire budget.
• Retrieval poisoning. One bad document in your vector store that scores high on similarity but contains outdated or contradictory information. Every query that matches it gets wrong answers.
• Instruction drift.System prompt says "be concise." Few-shot examples are 200 words each. The model learns from the examples, not the instruction.
• Position-dependent failures.The answer to the user's question is in the retrieved context, but it landed in the "lost in the middle" zone and the model missed it.