How do large language models work? At the bottom of every chatbot reply sits one repeated operation: guess the next token. A token is a chunk of text — usually a word piece, sometimes a whole short word, sometimes a single character. The model scores every token in its vocabulary, picks one, attaches it to the input, and runs again. A 700-word answer is that loop running roughly 900 times.
Almost everything odd about these systems follows from that sentence, including the parts that sound like a person and the parts that invent a citation to a paper that does not exist.
The Only Objective
Training has one target: given the text so far, predict what comes next. The objective never mentions truth, logic, or helpfulness. Those qualities show up because text written by careful people is full of them, and because later training stages push the model toward answers that human raters prefer.
The number the model minimizes is cross-entropy, which measures how surprised it was by the actual next token. A model that assigns 90 percent probability to the right answer scores better than one that assigns 10 percent. That is the entire scoreboard.
Tokens, Not Words
Text is split into tokens before the model sees anything. English averages about 1.3 tokens per word, so this article lands near 950 tokens. Frequent words get a single token; rare ones break into fragments, which is why a model can handle a misspelling it has never seen before.
Numbers are the weak spot. “1987” may arrive as two tokens, so the model never sees the digits as one quantity. That is part of why arithmetic fails in ways that look careless rather than incapable.
The Transformer Is the Engine
Nearly every current model is a transformer. Each layer runs attention, where every token looks at every other token in the window and decides what information to pull in, followed by a small neural network applied to each position on its own. Stack 30 to 120 of those layers and you have the model.
Attention solved a problem that had stalled the field for years. Older recurrent networks read left to right and lost the beginning of a long passage. Attention compares all positions directly, so a pronoun on page three can be tied back to a name on page one.
Those comparisons are matrix multiplications, which is what graphics processors do fastest. The hardware match, more than any single clever idea, is why these models became cheap enough to train at scale.
Training in Three Stages
- Pretraining. Predict the next token across trillions of tokens of web text, books, and code. Grammar, facts, style, and translation all get absorbed here, and nearly all of the compute is spent here.
- Instruction tuning. Train on curated question-and-answer pairs so the model learns the shape of a reply instead of continuing a document.
- Preference tuning. Humans rank pairs of answers and the model is adjusted toward the winners. This stage produces the agreeable, cautious tone that a raw pretrained model does not have.
The Scale, in Numbers
| Part | Typical size | What it controls |
|---|---|---|
| Parameters | 1 billion to over 1 trillion | Capacity to store patterns |
| Training tokens | 1 to 15 trillion | Breadth of language and facts |
| Context window | 8,000 to 1,000,000 tokens | How much it can hold in view |
| Vocabulary | 50,000 to 200,000 tokens | The set of chunks it can emit |
Parameter count gets the headlines, but training tokens matter at least as much. A model with too few parameters for its data wastes compute; a model with far more parameters than data hits a ceiling. Researchers at DeepMind described this balance in 2022, and it changed how labs budget their training runs.
Why Fluent and Wrong at the Same Time
The objective rewards plausible continuations, not correct ones. A model that has read millions of scientific claims learns the rhythm of a citation — author, year, journal, volume — and can produce one that fits the pattern perfectly while referring to nothing at all.
So how do large language models work when the output is false? Exactly the same way they work when it is true. There is no separate truth-checking step that fails; the same next-token guess simply landed on a sequence that reads well.
Fluent prose is therefore weak evidence of accuracy. The confidence of the writing is a property of the training data, not a signal about the claim.
What They Still Cannot Do
- Remember you. Every conversation starts from the same weights; memory has to be bolted on externally.
- Count reliably. Tokenization and attention both work against exact enumeration.
- Report their own limits. A model has no sensor for its uncertainty unless it was specifically trained to produce one.
- Stay current. Knowledge stops at the training cutoff unless a search tool is attached.
Keep Exploring
- What is a neural network? — the layer math underneath every model
- How does machine learning work? — the wider training loop
- What is an algorithm? — the older idea this descends from
