Chopping text into tokens

A model cannot take raw letters or whole words directly. Text is first cut into tokens: pieces taken from a fixed list called the vocabulary, which is built before training. Each entry has an integer ID. What the model receives is a list of integers.

A token can be a common whole word, part of a longer word, a punctuation mark, or a digit group. The space before a word is usually stuck to the word, so “cat” and “ cat” (with a space) are normally different tokens with different IDs. Capital letters matter too: “Cat” is another token again.

Here are illustrative splits, with a | between tokens. They are made up to show the idea; real tokenizers split differently.

• cat is one token, because it is very common.
• un|believ|ably is three tokens: the word is not in the vocabulary whole, but its pieces are.
• A random string of letters breaks into many tiny tokens, nearly one per letter.

How is a vocabulary built? A popular method is byte-pair encoding (BPE). The idea comes from data compression; Sennrich and colleagues (2016) adapted it for neural translation. The recipe:

• Start with single characters as the vocabulary.
• Count every pair of neighbouring pieces in a large text.
• Merge the most frequent pair into one new piece and add it to the vocabulary.
• Repeat until the vocabulary reaches the wanted size.

Try it on a tiny corpus: low ×3, slow ×2, newer ×4, wider ×2.

• Most frequent pair: e + r, 6 times (4 in “newer”, 2 in “wider”). Merge it into er.
• Next, l + o and o + w both occur 5 times. A fixed rule breaks ties; take lo.
• Then lo + w occurs 5 times. Merge it into low.

After three merges the words are low, s|low, n|e|w|er and w|i|d|er. Frequent chunks became single tokens, and rarer words stay in pieces.

A rule of thumb for ordinary English text: one token is about four characters, or roughly three quarters of a word, so 100 tokens is about 75 words. It is only a rough guide. It changes with the tokenizer, the topic, and the language.

Text in languages that were rare in the tokenizer's training text, plus code, unusual names and long numbers, often needs more tokens for the same content. More tokens means more computation and more of the context window used up. The context window is the maximum number of tokens the model can take in at once, and it is counted in tokens, not words.