A prediction machine

Start typing a message on your phone and the keyboard offers the next word. Type “The cat sat on the” and it may offer “mat”. A language model is a much bigger version of that idea.

A language model is a function. You give it a piece of text, and it returns one number for every entry in its vocabulary: the probability that this entry comes next. The probabilities are all between 0 and 1 and add up to 1.

Here is a made-up table for the text “The cat sat on the”. These numbers are invented to show the shape of the output; they do not come from any real product.

• mat: 0.40
• floor: 0.25
• sofa: 0.15
• roof: 0.09
• moon: 0.07
• banana: 0.04

Check the total: 0.40 + 0.25 + 0.15 + 0.09 + 0.07 + 0.04 = 1.00. A real model has a vocabulary of tens of thousands of entries, so almost every probability is tiny.

One step makes one choice. A reply is many steps. After the model returns its probabilities, a separate rule picks one entry (World 3 shows the rules). That entry is added to the text, and the model runs again on the longer text. The loop is: compute, pick, append, repeat.

A chat works the same way. The conversation so far is the input text. The reply is built one piece at a time, and every new piece becomes part of the input for the next step.

A note on words. This lab says a model computes and predicts, not that it “thinks” or “understands”. The words are not a style choice. They keep the picture honest: what happens inside is arithmetic on lists of numbers, and every claim in this lab can be checked against that arithmetic.