A map of meaning

Token IDs are only labels. ID 4821 is not “close in meaning” to ID 4822. To compute with meaning-like patterns, the model needs a representation in which similarity is a number you can calculate.

That representation is an embedding: for every token, a list of numbers called a vector. The vectors sit in a big table. Their values start random and are adjusted during training. Real models use hundreds to thousands of numbers per token.

What makes a vector useful? Tokens that occur in similar contexts end up with similar vectors. This idea was demonstrated at scale by Mikolov and colleagues (2013, “word2vec”), and it echoes a linguist's slogan: a word is characterised by the company it keeps. “Cat” and “dog” appear near words like “pet”, “fed” and “vet”, so their vectors end up close. “Cat” and “carburettor” do not.

Measuring similarity: cosine similarity. For two vectors a and b:

cos = (a · b) / (|a| × |b|)

where a · b is the sum of products of matching entries, and |a| is the square root of the sum of squares. The result is always between −1 and 1: 1 means the vectors point the same way, 0 means they are at right angles, −1 means opposite directions.

Toy example with two numbers each (invented): cat = (3, 4) and dog = (4, 3). The dot product is 3×4 + 4×3 = 24. Both lengths are 5. So cos = 24 / 25 = 0.96. Now take car = (4, −3): the dot product with cat is 12 − 12 = 0, so cos = 0.

You cannot draw 300 dimensions. To show a map, the vectors are squeezed down to 3 (or 2) dimensions with a method such as PCA or t-SNE; the AI Lab, in its World “From six features to a point cloud”, shows how these work and where they bend distances. Squeezing always distorts something: two words that look neighbours on the map may be farther apart in the full space.

Static embeddings have a limit. If each token has one fixed vector, the word “bank” has the same vector in “river bank” and “bank account”. One vector has to serve both meanings. Later in this lab (World 2) you will see how attention lets the numbers for a token depend on the words around it.