Type "see you" on your phone and the keyboard offers a few ways to finish the thought. That is the smallest everyday version of a language model: a program that looks at the text so far, estimates how likely each possible next word is, and picks one. A chatbot does the same thing over and over. It picks a word, adds it to the text, predicts again, and keeps going until the reply is finished.
Models don't read in whole words, exactly. They split text into tokens: chunks that are often a short word, a piece of a longer word, or a punctuation mark. In your dice models, every word and every full stop is one token. The text a model learns from is its training data. Everything a language model "knows" comes from patterns in that text.
To keep every number checkable by hand, this page uses a tiny practice text of our own. In class you'll use the text printed on your gameboards, but the ideas are identical.
The simplest possible model gives every token the same chance on every roll. With 17 different tokens in the practice text, each one has a 1-in-17 chance each time, no matter what came before. It uses the training data for exactly one thing: the list of tokens it's allowed to say. The result is word salad, which is the point. A model with no sense of which words are common or which words go together can't write anything that sounds like language.
Equal chance sounds easy: just roll some dice. But if you add dice together, the totals are not equally likely. There are many ways to make a total in the middle and only one way to make the smallest or largest. Try the dice below.
Pick a dice setup. Each bar is one possible total, and its height is how likely that total is. If a board gives each word its own total, a word is only as likely as its total.
Model 2 starts using the training data properly: it counts. If a token makes up 24% of the text, the model picks it about 24% of the time. That is a probability learned from data, and it's the first real step from random noise toward language. To roll on it with dice, you turn each count into a share of the die's faces.
How often each token appears in the practice text. Model 2 rolls on exactly this.
Using the practice text, how many results on a 100-result roll (a D10 and a D00 together) should belong to the token "the"?
Real sentences aren't piles of common words. Which word comes next depends on the word before it. After "the" you might expect "robot" or "door", almost never a full stop. Model 3 builds that in: instead of one table for the whole text, it has one follower table for every token, listing only the tokens that came right after it in the training data.
The idea is named for Andrey Markov, a Russian mathematician who, in the early 1900s, counted how often letters followed other letters in a famous Russian poem and showed that the next letter depends on the one before it. A Markov chain is a system that moves from one state to the next, with probabilities that depend only on where it is right now. In your game, the "state" is the last word you wrote.
Using the practice text, build the follower table for the token “the” and turn it into faces on a D20.
Roll a few sentences with each model. Then peek at the tables underneath and find the table that produced your last word.
Pick a model, then roll a sentence. Each word comes from one roll of a 100-faced die. Roll several sentences with each model and compare what you get. (Model 3 starts with a word from Model 2, then chains. Every sentence stops at a full stop or after 14 tokens.)
| Token | Count | Chance | Faces on a 100-roll |
|---|---|---|---|
| robot | 4 | 40.0% | 1–40 |
| cat | 3 | 30.0% | 41–70 |
| door | 3 | 30.0% | 71–100 |
Model 3 looks back exactly one token. So it can chain neat-looking pairs into a sentence that forgets how it started. For example, every neighboring pair in the cat likes the robot beeps at the door . appears in the practice text, yet the sentence falls apart halfway through. By the time it writes "beeps", the model has forgotten that "the cat likes" came first.
A deterministic algorithm gives the same output every time you give it the same input. A calculator always says 2 + 2 = 4. A probabilistic system uses chance, so the same input can lead to different outputs. Your dice models are probabilistic: start from "the" three times and you might get "robot", then "door", then "cat".
A chatbot works the same way. At each step it has a probability for every possible next token and picks from them, so the same question can come back worded differently. Many tools have a setting, often called temperature, that controls how adventurous the picks are. At very low settings the chatbot almost always takes the most likely token and becomes close to deterministic.
Your dice models and the model behind a chatbot are in the same family: both estimate a probability for each possible next token and pick one. The differences are enormous.
| Your dice models | A real LLM | |
|---|---|---|
| Training data | A few sentences | A huge share of books, websites, code and more — trillions of tokens for the largest models |
| Tokens | Whole words and full stops | Chunks of text: short words, pieces of longer words, punctuation |
| What it looks at | Nothing (Models 1–2) or just the one word before (Model 3) | Thousands of earlier tokens at once, and far more in recent models |
| Where the knowledge lives | A table someone counted by hand | Billions of numbers (parameters, also called weights) adjusted during training |
| How it learns | Count once | Guess the next token, compare with the real text, nudge the numbers — repeated an enormous number of times |
| Changes while you use it? | No — the table never changes | No — the weights stay fixed during your chat |
A model with a vocabulary of about 50,000 tokens needs 50,000 rows for a one-token context. For two tokens of context there are 50,000 × 50,000 = 2.5 billion possible pairs, and for a context of a thousand tokens the number of possible histories is far beyond anything a table could hold. Real models skip the table. They compress patterns into their weights, which lets them make a sensible guess even for text they have never seen.
Every neighboring pair in the robot beeps at the door . appears in the practice text, so Model 3 can write it. But the practice text only ever said the robot beeps at the cat. The model recombined likely pairs into a new sentence that says something the text never said, and it has no way to know the difference.
A real LLM does the same thing with far more skill. Its recombinations are usually right, because patterns in huge amounts of text usually reflect the real world. When they aren't, the output sounds exactly as confident. A fluent output that isn't supported by facts is called a hallucination.
Your dice models could only say things built from their training text. Real LLMs learn from enormous amounts of writing, much of it by people who were never asked. Courts in the United States are still working out whether that is allowed. As of late 2026, no federal appeals court has settled whether training an AI on copyrighted work counts as fair use. Some lower-court decisions have treated training on lawfully obtained books differently from copying pirated ones, and at least one lawsuit has ended in a settlement worth more than a billion dollars.
The model doesn't store or copy the text. It learns patterns, the way a person learns to write by reading. Using work to create something new and different is the kind of use fair use was designed to protect.
Writers and artists never agreed and weren't paid, and the model can produce work that competes with theirs, sometimes in their own style. A person who reads a book isn't turning it into a product that can replace the author.