Principles of AI · Unit 1: Inside the Machine ·  Activity 1.1.1

Next-Word Dice — Build a Tiny Language Model

Three dice models, each a little smarter than the last, and what they reveal about every chatbot  ·  Approx. 2 class days

StarringModel 1 · Equal ChanceModel 2 · Word FrequencyModel 3 · Context Chains
Unit 1: Inside the Machine

Quick reference for the three dice models and how they connect to real chatbots.

Next-Word Dice infographic: predicting the next word, three dice models, counts to dice faces, dice versus LLM, deterministic versus probabilistic, and fluent versus true

🧭 Plot Summary

Every chatbot you have ever used does one thing at heart: it predicts the next word, over and over, until a reply appears. In this activity you build that machine out of dice. Your team gets a tiny text and three gameboards. Model 1 picks words with equal chance, and the results are noise. Model 2 picks words by how often they appear, so the common words start to dominate. Model 3 adds one word of context: it looks at the word it just wrote and rolls on a table of the words that followed it in the text. That is a Markov chain, and its sentences start to sound like the original.

Then comes the honest part. Your dice model never understood a single word. It counted. Real LLMs use the same basic move, a probability for each possible next token, but they look back over thousands of tokens and store what they learned as billions of numbers instead of a table you can hold in your hand.

What you'll do in this activity

  • Roll three dice-powered models of the same tiny text, each a little smarter than the last, and write down what each one "says."
  • Watch pure chance produce nonsense, then watch word counts and one word of context turn dice rolls into sentences that almost sound right.
  • Turn counts into percentages and dice faces. This is the arithmetic every language model is built on.
  • Learn the words deterministic and probabilistic, and see why the same prompt can get a different answer each time.
  • Bridge to real chatbots: what stays the same (probabilities, numbers fixed after training) and what is very different (context, learned weights).
  • Ask whether a sentence that sounds right is true, and who owns the text a model learned from.

Why it matters

This is the idea the rest of Unit 1 stands on. Once you know a language model is choosing likely words, not looking up facts, it makes sense why a chatbot can sound completely sure and be completely wrong, which is the whole point of Activity 1.2.2 and the Red-Team Report in Project 1.2.4.

✅ Self-Check Before You Roll On

Check off each item as you get there. These aren't grades. They're your own signal.