How LLMs Actually Work

I Before the Magic

You've used ChatGPT, Claude, or Gemini. You've typed a prompt and gotten a surprisingly good response. You've wondered: how does it actually know this?

You don't need a CS degree for what follows. You do need a working mental model of what these systems are, how they produce text, and where they hit walls. That model will change how you write product specs, evaluate vendors, and make build-vs-buy decisions.

II Training vs. Inference

Training is the months-long process where the model learns from data. Billions of pages of text go into a neural network that adjusts billions of internal numbers (parameters) until it predicts language patterns well. Thousands of GPUs. Tens of millions of dollars. Once per model version.

Inference is what happens when you type a prompt. The trained model generates a response, token by token. Fractions of a cent per request. Billions of times per day.

You can't alter what the model knows (baked in during training). But you control how it responds — through prompts, fine-tuning, and context at inference time. Every product decision lives in the inference layer.

III Next-Token Prediction

Every LLM does exactly one thing: predict the most likely next token given everything before it.

A token is a chunk of text, roughly three-quarters of a word. The model sees your prompt as a token sequence, calculates probabilities over its vocabulary (~100K tokens), and picks one. Appends it. Predicts again. One token at a time, the response grows.

This is why models "hallucinate" — they produce text that soundsplausible because it's statistically plausible, even when it's wrong. The model isn't looking up facts. It's completing patterns. Try it yourself:

IV Parameters, Models & Capability

If all LLMs predict the next token, why are some dramatically better than others? Because prediction quality depends on three things — and none of them are magic.

Parameter countis the number everyone fixates on, but it's the wrong headline. Parameters are the model's internal numbers — its memory capacity. More generally means more nuance, but a well-trained 70B model routinely embarrasses a sloppy 200B one.

Training data quality matters more than quantity. Curated textbooks and expert writing beat ten times as much scraped web spam. What goes in determines what comes out — always.

Fine-tuning and alignmentturn a raw model into something you'd actually ship. RLHF (reinforcement learning from human feedback) is what turned GPT from "interesting research paper" into "product that answers your question instead of rambling like a Wikipedia article."

V The Capability Landscape

Proprietary frontier models (GPT-5, Claude Sonnet, Gemini Pro) are the most capable. Complex reasoning, broad knowledge, best instruction-following. The price: your data goes to a third party, you pay per token, and the model can change under you without warning.

Open-source models (Llama, Mistral, Command R+) run on your infrastructure. You own the data, the cost structure, and the version. The price: you need ML talent to deploy and maintain them, and they trail proprietary models on the hardest tasks — usually by a few months, sometimes by more.

Explore the landscape:

VI Limitations That Matter

Capabilities are obvious in demos. Limitations reveal themselves in production, usually at the worst possible time.

Hallucinations.LLMs generate text that sounds authoritative and is completely wrong. They predict plausible tokens, not facts. Ask for a citation and you might get a perfectly formatted reference to a paper that doesn't exist. This isn't a bug waiting to be fixed — it's structural. Every product needs a verification layer.

Knowledge cutoff. A model only knows its training data. Your product launched last month? The model has no idea it exists. RAG (retrieval-augmented generation) is the workaround: feed current documents into the prompt at inference time.

Reasoning limits. LLMs approximate reasoning through pattern completion — and approximation has limits. They struggle with novel multi-step logic, reliable arithmetic, and maintaining state across long chains of thought.

See how training and inference costs compare at scale:

VII Where to Go Next

With this foundation, the rest of the AI product toolkit makes sense:

• Context Windows & Memory— Why models "forget" and patterns for persistent memory.
• Cost & Latency Tradeoffs — Tokens, pricing, and the optimization levers that make AI products viable at scale.
• Grounding & Guardrails — How RAG grounds outputs in real data and how to build safety guardrails.
• Evaluation & Testing — How to measure whether your AI feature is actually good.
• Agents & Tool Use — How the agent loop turns a token predictor into a system that takes action.
• AI UX & Human-in-the-Loop — UX patterns for uncertainty and human oversight that catches what the model misses.