Meaning as arithmetic

If vectors capture how words are used, then differences between vectors might capture relations. The classic example from Mikolov and colleagues (2013): in some word-vector sets, king − man + woman lands near queen.

The recipe: compute the vector a − b + c entry by entry, then search the vocabulary for the nearest vector, leaving out the three query words. Without that exclusion the nearest vector is often one of the inputs, such as “king” itself.

A toy case with two numbers each (chosen to be clean; real vectors are noisy):

• France = (2, 1), Paris = (2, 3), Italy = (5, 1).
• Paris − France + Italy = (2 − 2 + 5, 3 − 1 + 1) = (5, 3).
• The vocabulary holds Rome = (5, 3), Madrid = (8, 3) and Berlin = (2, 5).

The nearest vector is Rome. It works here because the step from a country to its capital is the same arrow, (0, 2), for every country in the toy table. In real data that arrow is only roughly shared.

Closeness means similar contexts, not truth or sameness. “Hot” and “cold” often sit close together in word vectors. They are opposites, but they turn up in the same kinds of sentences (“the water is ...”). A vector set can place two words together because people write about them in the same places, whatever the words mean.

Vectors copy the patterns of the text they were trained on. Bolukbasi and colleagues (2016) reported that analogy queries on word vectors trained on a large news collection returned stereotyped answers about occupations and gender. The vectors had not been told anything about stereotypes; they reflected patterns in the writing. World 7 returns to this: biased data gives biased results.