Everybody love the demo, standing ovation,
Ship it to prod and it's pure frustration.
Wrong tool, fake data, loops running wild —
Know where it breaks before you ship, child.
I The Reality Check
After Parts 1 and 2, agents sound incredible. Loop through problems, call tools, adapt on the fly. So why doesn't every product just use them for everything?
Because agents fail. Not occasionally — regularly. Unpredictably, expensively, and in ways that make debugging feel like archaeology. Andrej Karpathy calls this the "loopy era" — agent capabilities are being discovered in real-time, and the field lacks stable intuitions about what works. The gap between a demo that wows the room and a product that survives production is almost entirely about understanding failure.
Analogy
Think of agents like self-driving cars. Impressive in a controlled demo, terrifying when they encounter a scenario nobody anticipated. You wouldn't ship a self-driving feature without exhaustive failure analysis. Agents deserve the same scrutiny — they just fail with words instead of fenders.
II How Agents Fail
Agent failures fall into five predictable categories. Learn them once, and you'll spot them everywhere — in demos, competitor products, and your own prototypes.
1. Wrong tool selection. The agent calls search_contacts when it should call search_companies. More tools in the toolbox means more opportunities to grab the wrong one.
2. Hallucinated parameters. The agent invents values wholesale. It calls get_user(id: "usr_847293") with a user ID it fabricated from thin air. The function call is structurally perfect and factually fiction.
3. Infinite loops. The agent retries a failed action with minor variations, forever. "No results... let me try a different query... still nothing... one more time..." Your bill, meanwhile, grows with each lap. Garry Tan has flagged a related pattern: agents making the same mistakes repeatedly across sessions. Tools can detect the repetition, but fixing it requires process changes — the agent has no memory of its past failures.
4. Premature stopping. The opposite: the agent quits early. "Here are the top 2 bugs" when you asked for 5. Task declared complete; task not actually complete.
5. Misinterpreting results. The tool returns correct data; the agent misreads it. A flight API returns prices in cents, and the agent reports a $45,000 ticket instead of $450. Correct plumbing, wrong interpretation.
Explore these failure modes in the interactive demo below:
Key Insight
Every failure shares a root cause: the LLM predicts plausible next tokens, not ground truth. It doesn't "know" the right user ID — it generates a convincing one. It doesn't "understand" it's looping — it keeps producing the next likely action. These aren't bugs to fix with better prompts. They're fundamental characteristics you design around.
III The Cost Equation
Every loop costs money. The LLM re-processes the full conversation history — all previous tool calls and results — on every iteration. Costs grow super-linearly with step count.
- Input tokens accumulate each loop (the context snowballs)
- Output tokens are generated each loop (reasoning + tool call)
- A 3-step task can cost 5-10x a single LLM call
- A pricier model that finishes in 3 steps often costs less than a cheap model that stumbles through 8
Play with the calculator below to see this in action:
Builder Tip
Model the cost before committing to an agent architecture. $0.01 per action sounds trivial until you multiply by 100,000 DAU. Conversely, an expensive model might be the right call if it halves the loop count and doubles reliability.
IV Building Guardrails
Understanding failure modes is only useful if you can contain them. Production agent systems rely on five guardrails:
| Guardrail | What it does | Failure it prevents |
|---|
| Max iterations | Hard cap on loop count (typically 5-10). Returns partial results with a note if hit. | Infinite loops, runaway costs |
| Parameter validation | Validates tool inputs before execution. Does that user ID exist? Is the date valid? | Hallucinated parameters |
| Confirmation steps | Agent proposes high-stakes actions (send email, delete data); human approves. | Irreversible mistakes |
| Human escalation | Routes to a person when the agent fails, loops out, or hits ambiguity. | Unrecoverable edge cases |
| Observability | Logs every iteration: what it thought, called, received, and decided. | Undebuggable failures |
These aren't optional polish. They're as fundamental to an agent product as error handling is to a web app. There's also a deeper structural issue: LLMs live in a perpetual present. They have no native ability to learn from past interactions or carry forward lessons without architectural workarounds. Every team building agents is trying to patch memory differently, and nobody has a bulletproof solution yet. Ship without guardrails and your first production week will feel like an incident response marathon — except nobody knows what the incidents are, because you forgot to log anything.
Takeaway
The best AI products aren't fully autonomous. They're human-in-the-loop systems: the agent handles the routine, humans handle the exceptions. Designing that boundary is your job.
V The Decision Framework
Part 1 introduced a simple prompt-vs-workflow-vs-agent framework. Now that you understand tools and failure modes, sharpen it with four questions:
- How predictable is the task? Same steps every time = workflow. Steps vary = agent.
- What's the cost of failure? Drafting text tolerates errors. Financial transactions do not.
- How much does latency matter? Agents are slow — multiple LLM calls per task. Users expecting instant feedback need a simpler approach.
- What's the budget per task? Thousands of daily executions turn small per-task costs into real line items. Model the economics before you model the architecture.
Run your scenario through the decision tool below:
Key Insight
The four questions aren't a checklist — they're a negotiation between capability and risk. Most real-world products land somewhere between "workflow" and "agent with guardrails." Your job is finding that line.
VI Wrapping Up
Three parts, one mental model:
- Part 1: The Agent Loop — Think, act, observe, decide. The cycle that lets agents handle ambiguity.
- Part 2: Tools & Function Calling — Agents interact with the world through structured function calls. Tool design directly impacts reliability.
- Part 3: Planning, Failure & Tradeoffs — Agents fail predictably, cost money per loop, and need guardrails. The right architecture depends on predictability, stakes, latency, and budget.
Builder Tip
Whether you're building agents or making decisions about them, you need to speak the language. When someone says "the agent is selecting the wrong tool," think tool descriptions and toolbox size. When someone asks "why can't it just do everything automatically," think failure modes, costs, and human-in-the-loop design.
The most successful AI products aren't the most autonomous. They're the ones where someone made smart decisions about what to automate, what to guard, and where to keep a human in the loop. The proof is in the numbers: on SWE-bench Verified, coding agent performance rose from around 60% to roughly 80% in a single year — among the most dramatic capability jumps in any domain. But that success came precisely because coding has machine-checkable outputs, well-defined tools, and tight feedback loops. The domains that lack those properties are still waiting.
That someone is you.