Fast enough beats perfect. The levers that make AI features feel instant — and cost 80% less.
Fast enough beats perfect, that's the game, Stream the first word early, the UX ain't the same. Cache it, trim it, batch it — three moves deep, Cost, latency, quality — pick two to keep.
I Why LLMs Feel Slow
Google returns results in 200 milliseconds. Your AI feature takes 3 seconds. It feels broken — even if the answer is brilliant.
Two numbers explain why:
Time to First Token (TTFT) — how long before the model starts generating. Includes network round-trip plus processing your entire input. Longer prompts and larger models both push it up. Range: 200ms (small models) to 2+ seconds (large ones).
Time Per Output Token (TPOT) — how fast each subsequent token appears. LLMs generate sequentially, one token at a time. A 200-token response at 50 tokens/second takes 4 full seconds. Smaller models are faster: 100+ tokens/second vs. 30-50 for large models.
Analogy
Think of TTFT as the waiter walking to your table, and TPOT as how fast they read the specials. A slow walk plus a slow read means you're sitting there wondering if the restaurant is even open.
Total latency = TTFT + (output tokens x TPOT). For a large model generating 300 tokens: 1.5s + (300 x 0.025s) = ~9 seconds. An eternity in UX terms. But users don't need to wait for the entire response.
II Streaming Changes Everything
Without streaming: a blank screen for 5, 10, 15 seconds, then the full response appears at once. With streaming: tokens appear as they're generated, word by word. Total time is identical. The perceived experience is not even close.
Streaming transforms a wait into a conversation.The user reads the first sentence while the model generates the third. By the time the response finishes, they've absorbed half of it. The loading spinner is the content itself.
Key Insight
The speed users perceive matters more than the speed you measure. A streamed 8-second response feels faster than a non-streamed 4-second response, because the first byte arrives in 300ms instead of 4,000ms.
See the difference for yourself:
Experience it
Streaming vs. Non-Streaming
Both responses are identical and take the same total time. Notice how different they feel.
Without streaming
—
With streaming
—
Same content. Same total time. Completely different experience. Every major AI product streams for exactly this reason. If your feature shows responses to users and you're not streaming, you're volunteering for bad NPS scores.
III The Optimization Playbook
Your AI feature works. Now: "How do we make it faster and cheaper?" Five levers, roughly ordered by impact:
Lever
How it works
Impact
Model routing
Small model for easy tasks, large model for hard ones. A lightweight classifier routes each request.
50-80% cost reduction
Prompt caching
Cache the system prompt so the provider skips reprocessing it.
30-60% input cost & TTFT reduction
Shorter prompts
Cut redundant instructions, compress examples. A 2,000-token prompt trimmed to 800 is 60% cheaper.
Proportional to tokens cut
Response caching
Cache answers to common queries. Semantic caching is harder but more powerful.
Eliminates 20-50% of LLM calls
Fewer output tokens
Ask the model to be concise, use max_tokens, or request structured output.
Proportional to tokens saved
Builder Tip
Start with model routing. Most production workloads are 70% simple tasks that a small model handles perfectly. Sending every request to frontier-class models is the API-cost equivalent of hiring a surgeon to apply band-aids. This isn't hypothetical — Anthropic's data shows that ~50% of all agentic tool calls are software engineering, where teams have already learned to route simple completions to fast, cheap models and reserve heavyweight reasoning for complex tasks.
Explore how these levers interact:
Experiment
Optimization Playbook
Toggle optimization strategies on/off and see the cumulative impact on cost and latency.
Baseline: Medium model, 1,000 input tokens, 300 output tokens, no optimizations
Cost per 1K requests
$7.50$7.50
Avg latency
3.5s3.5s
Toggle optimizations above to see their impact.
Notice how optimizations stack. Each saves 30-50% alone, but combined they cut costs 80-90%. This isn't a "nice to have" — it's the difference between a viable product and one that burns through its API budget like a venture-funded bonfire.
IV The Cost-Latency-Quality Triangle
Every AI product decision is a three-way tradeoff. Pick two:
Cost — price per request and how it scales.
Latency — response speed and perceived snappiness.
Quality — output accuracy and user satisfaction.
A large model gives quality but costs more. A small model is cheap and fast but less capable. Aggressive caching is cheap and fast but risks stale output. You can't max all three — the physics won't allow it. The industry is already voting with its wallets: a16z reported that the 2025 coding tools ecosystem alone generated over $1 billion in new revenue, largely because those tools got the triangle right — fast enough for developer flow, capable enough to be useful, and priced to sustain daily use.
Builder Tip
Your job is to find the right position in this triangle for each feature. Autocomplete needs speed above all else — small model, aggressive caching. Legal document review needs quality — large model, accept the cost. A support bot needs balance — medium model, streaming, smart caching. One product, three different positions.
Where does your feature sit? Explore below:
Explore
The Cost-Latency-Quality Triangle
Select a product feature to see where it falls on the triangle and the recommended approach.
Cost Priority
Latency Priority
Quality Priority
Select a feature above
Test your understanding
Article Recap
5 questions covering the key concepts from this article.
1 of 5
A stakeholder complains: "Our AI feature takes 6 seconds to respond. Google Search takes 200ms. Why is our product so slow?" How do you best explain the discrepancy?
V Wrapping Up
Across these two parts, you've built a complete mental model for LLM economics:
Tokens are the unit of cost — not the same as words.
Output tokens cost 3-5x more than input tokens.
Model tiers span a 5x cost range — match the model to the task, not to the hype.
Streaming transforms perceived latency without changing actual latency.
Five optimization levers (routing, caching, shorter prompts, response caching, fewer output tokens) can stack to cut costs 80-90%.
Every feature lives in a cost-latency-quality triangle — and the right position varies by use case.
Takeaway
These aren't details you can ignore. They're the foundation of every pricing decision, architecture choice, and "can we afford to build this?" conversation. Understanding the tradeoffs is what separates shipping a feature from shipping a cost center.