How a Machine Learns to Write.
Type a few words and watch a real language model finish them, one character at a time, one weighted dice-roll after another. No math, no jargon. Just you, and a machine that has read nothing but songs.
Text is just numbers.
Before a machine can write a word, your words have to become something it can do arithmetic on.
This is all it gets.
Your seed phrase, nothing else. No meanings, no dictionary. Just a short string of characters.
It knows 91 symbols.
Letters, digits, punctuation, the space, the line break. That's the machine's entire world — every song it ever read was spelled with these.
Each character becomes a number.
Hover any character to see its ID. The machine never sees ‘love’ — it sees a short list of integers, in order.
It writes one letter at a time.
The machine never plans a sentence. It asks one question, over and over: what single character comes next?
Every step is a bet.
Given everything written so far, the model assigns a probability to each of its 91 characters. Watch the bars — that's it thinking.
Then it rolls the dice.
It doesn't always pick the favourite. It samples — usually the likely thing, sometimes a surprise. That's where the life comes from.
And feeds the answer back in.
The chosen character joins the context, and the question is asked again. Predict, roll, append, repeat — that loop is all that ‘writing’ is.
Sometimes there's no contest.
After a q, watch the distribution collapse — nearly all the probability lands on u. The machine has learned spelling without ever being taught it.
Where is it looking?
Before each guess, the model looks back over everything written so far — but not evenly. This is attention, and you can watch it.
Every character looks back.
Each position gets to study all the ones before it. Brighter means more weight in the decision.
Early layers stay close.
The first layer mostly watches the last few characters — enough to finish a word it has started.
Deeper layers reach back.
Later layers attend across whole lines — to where a verse began, to sounds worth rhyming with. Nobody programmed that. It emerged.
The dice can be loaded.
The model proposes probabilities; the sampler decides how to roll. Three little knobs change everything about the voice.
Cold and careful, hot and wild.
Temperature reshapes the whole distribution. Low: the favourite always wins. High: every character gets a chance — including the wrong ones.
Too cold, and it gets stuck.
Freeze the dice completely and the model loops — the same phrase, forever. Certainty without variety is a broken record.
Top-p: trim the nonsense, keep the spark.
Only sample from the smallest set of characters that covers, say, 90% of the probability. Confident moments stay sharp; open moments stay playful.
How did it learn?
Nobody taught it spelling, rhyme, or heartbreak. It read the same songs ten thousand times, guessing the next character and being told how wrong it was.
First, noise learns its letters.
Early in training the model produces almost-words — right letters, wrong order, no shape.
Then words find their lines.
Halfway through, real words appear, and something like verse structure — short lines, breaks in the right places.
Finally, a voice.
The finished model writes lines with rhythm and repetition — the statistical ghost of every song it read.
Human or machine?
You've seen how it writes. Now, can you tell? One line at a time: did a person write this, or the machine?
Break it.
It has 91 characters and no idea what any word means. It will finish any sentence you start, beautifully, confidently, and sometimes completely wrong.
Confidence is not correctness. The machine continues “Photosynthesis requires” in lovesick song-speak because song-speak is all it knows. It cannot say “I don’t know”, nobody built that option.
Watch it learn.
Everything above used the finished model. Here is the whole journey that made it, replayed. As the error falls, the same word grows from noise into something that almost rhymes.
Then, if you'd like to write with it: step into The Studio →