Fast enough beats perfect, that's the game,
Stream the first word early, the UX ain't the same.
Cache it, trim it, batch it — three moves deep,
Cost, latency, quality — pick two to keep.
I Why LLMs Feel Slow
Google returns results in 200 milliseconds. Your AI feature takes 3 seconds. It feels broken — even if the answer is brilliant.
Two numbers explain why:
Time to First Token (TTFT) — how long before the model starts generating. Includes network round-trip plus processing your entire input. Longer prompts and larger models both push it up. Range: 200ms (small models) to 2+ seconds (large ones).
Time Per Output Token (TPOT) — how fast each subsequent token appears. LLMs generate sequentially, one token at a time. A 200-token response at 50 tokens/second takes 4 full seconds. Smaller models are faster: 100+ tokens/second vs. 30-50 for large models.
Analogy
Think of TTFT as the waiter walking to your table, and TPOT as how fast they read the specials. A slow walk plus a slow read means you're sitting there wondering if the restaurant is even open.
Total latency = TTFT + (output tokens x TPOT). For a large model generating 300 tokens: 1.5s + (300 x 0.025s) = ~9 seconds. An eternity in UX terms. But users don't need to wait for the entire response.
II Streaming Changes Everything
Without streaming: a blank screen for 5, 10, 15 seconds, then the full response appears at once. With streaming: tokens appear as they're generated, word by word. Total time is identical. The perceived experience is not even close.
Streaming transforms a wait into a conversation. The user reads the first sentence while the model generates the third. By the time the response finishes, they've absorbed half of it. The loading spinner is the content itself.
Key Insight
The speed users perceive matters more than the speed you measure. A streamed 8-second response feels faster than a non-streamed 4-second response, because the first byte arrives in 300ms instead of 4,000ms.
See the difference for yourself:
Same content. Same total time. Completely different experience. Every major AI product streams for exactly this reason. If your feature shows responses to users and you're not streaming, you're volunteering for bad NPS scores.
III The Optimization Playbook
Your AI feature works. Now: "How do we make it faster and cheaper?" Five levers, roughly ordered by impact:
| Lever | How it works | Impact |
|---|
| Model routing | Small model for easy tasks, large model for hard ones. A lightweight classifier routes each request. | 50-80% cost reduction |
| Prompt caching | Cache the system prompt so the provider skips reprocessing it. | 30-60% input cost & TTFT reduction |
| Shorter prompts | Cut redundant instructions, compress examples. A 2,000-token prompt trimmed to 800 is 60% cheaper. | Proportional to tokens cut |
| Response caching | Cache answers to common queries. Semantic caching is harder but more powerful. | Eliminates 20-50% of LLM calls |
| Fewer output tokens | Ask the model to be concise, use max_tokens, or request structured output. | Proportional to tokens saved |
Builder Tip
Start with model routing. Most production workloads are 70% simple tasks that a small model handles perfectly. Sending every request to frontier-class models is the API-cost equivalent of hiring a surgeon to apply band-aids. This isn't hypothetical — Anthropic's data shows that ~50% of all agentic tool calls are software engineering, where teams have already learned to route simple completions to fast, cheap models and reserve heavyweight reasoning for complex tasks.
Explore how these levers interact:
Notice how optimizations stack. Each saves 30-50% alone, but combined they cut costs 80-90%. This isn't a "nice to have" — it's the difference between a viable product and one that burns through its API budget like a venture-funded bonfire.
IV The Cost-Latency-Quality Triangle
Every AI product decision is a three-way tradeoff. Pick two:
- Cost — price per request and how it scales.
- Latency — response speed and perceived snappiness.
- Quality — output accuracy and user satisfaction.
A large model gives quality but costs more. A small model is cheap and fast but less capable. Aggressive caching is cheap and fast but risks stale output. You can't max all three — the physics won't allow it. The industry is already voting with its wallets: a16z reported that the 2025 coding tools ecosystem alone generated over $1 billion in new revenue, largely because those tools got the triangle right — fast enough for developer flow, capable enough to be useful, and priced to sustain daily use.
Builder Tip
Your job is to find the right position in this triangle for each feature. Autocomplete needs speed above all else — small model, aggressive caching. Legal document review needs quality — large model, accept the cost. A support bot needs balance — medium model, streaming, smart caching. One product, three different positions.
Where does your feature sit? Explore below:
V Wrapping Up
Across these two parts, you've built a complete mental model for LLM economics:
- Tokens are the unit of cost — not the same as words.
- Output tokens cost 3-5x more than input tokens.
- Model tiers span a 5x cost range — match the model to the task, not to the hype.
- Streaming transforms perceived latency without changing actual latency.
- Five optimization levers (routing, caching, shorter prompts, response caching, fewer output tokens) can stack to cut costs 80-90%.
- Every feature lives in a cost-latency-quality triangle — and the right position varies by use case.
Takeaway
These aren't details you can ignore. They're the foundation of every pricing decision, architecture choice, and "can we afford to build this?" conversation. Understanding the tradeoffs is what separates shipping a feature from shipping a cost center.