Chapter 6Vector Space Models

Meaning you can measure

So far this book has treated a word as a symbol, an atom with an identity and nothing else. Everything has handled words as opaque strings, from the n-gram counts ahead to the tokeniser of Chapter 5 and even the term weighting of Chapter 4, where two strings are either equal or unequal.

Symbols have a fatal poverty, which is that they cannot be similar. To a symbol processor, cat and dog are exactly as different as cat and democracy.

We want a machine to know something obvious, though, which is that a review calling a film “superb” means much what one calling it “excellent” means. So we need a representation where nearness carries meaning, with cat and dog sitting close together and democracy far away. We need to turn meaning into geometry, which is the programme of this part of the book and the foundation this chapter lays.

Make the poverty of symbols concrete first. The obvious way to feed a word to a numerical model is a one-hot vector, where you fix the vocabulary, give each word its own dimension, and represent a word by a vector that is $1$ on its own axis and $0$ everywhere else.

The sizes are alarming. Jane Austen’s Emma has a vocabulary of about $7{,}000$ types, so each word becomes a vector of some seven thousand dimensions, while the Google News corpus would demand three million.

Worse, the representation encodes nothing, because any two distinct one-hot vectors are orthogonal and so the cosine similarity of Chapter 4 between any two different words is exactly $0$ .

One-hot vectors make every word equidistant from every other, so cat is no nearer dog than it is to democracy. The geometry is there, in the sense that we have vectors and a space, and it carries no meaning at all. The rest of this chapter fills that empty space with structure.

The distributional hypothesis

Where could the structure come from? Not from a word’s spelling, which is arbitrary, but from its use.

This is one of the oldest and most productive ideas in linguistics, and it has two famous statements. The linguist J. R. Firth put it as a slogan, “you shall know a word by the company it keeps”, while Zellig Harris (Harris 1954) gave it a form a computer can act on: words that occur in the same contexts tend to have similar meanings.

That is the distributional hypothesis, and its promise is enormous because a word’s contexts are something we can simply count. No dictionary is needed, no grammar and no human annotator, only a corpus and the patience to tally, for each word, what it appears near. Meaning is distilled from distribution.

The operational move follows at once. Represent each word not by its own lonely axis but by a vector of the contexts it keeps, so the dimensions become other words and the entry for word $w$ on dimension $c$ records how often $w$ and $c$ co-occur. This is a word context co-occurrence matrix $M$ , and each row of it is a word’s distributional fingerprint.

Two words with similar rows are used similarly and, by the hypothesis, therefore mean similarly. Similarity of meaning has become similarity of vectors, and similarity of vectors is just the cosine of Chapter 4, so the empty one-hot space is replaced by one where position is earned by usage.

First order and second order

There are two quite different things one can mean by “words that go together”, and keeping them apart is the most important idea in this chapter.

First order association is direct co-occurrence, so two words are first order associates if they appear near each other, as with coffee and cup, doctor and hospital, or San and Francisco. These are the entries in a row of $M$ , the words a target actually keeps company with, and first order structure captures collocations and thematic links. They are the syntagmatic relations, words that combine in running text.

Second order association is subtler and, for meaning, more powerful. Two words are second order associates if they appear in the same kinds of context, so their rows of $M$ are similar even when the two words never occur together at all.

Cat and dog are the standard example, since both co-occur with pet, vet, feed and fur, which makes their rows of $M$ point in nearly the same direction even though a sentence rarely contains both.

Their similarity is that they appear like each other rather than with each other. This is the paradigmatic relation, and it is what lets distributional models discover synonyms and, as Chapter 9 will show, analogies.

The rule of thumb is short. First order association is read along a row, and second order association is read by comparing rows.

A matrix you can check by hand

A three word example makes the distinction unmistakable. Suppose that after counting co-occurrences over some corpus, three words come out with these rows over three context dimensions:

	pet	feed	drive
dog	2	1	0
cat	2	1	0
car	0	0	3

The rows for dog and cat are identical, so the cosine between them is exactly $1$ , which is maximal similarity, and yet nothing in the table says dog and cat ever occur together. That is second order association at full strength. The row for car shares no nonzero dimension with either animal, so its cosine with each is $0$ .

Now contrast this with one-hot vectors for the same three words, where all three pairwise cosines are $0$ . One-hot cannot tell that dog and cat are more alike than dog and car, and the co-occurrence representation can.

That single change is what turns a symbol into a meaning, and all it took was going from an axis per word to a vector of contexts.

What counts as context

The definition of “context” is a real design choice rather than a detail, because it decides what kind of similarity the vectors capture.

A narrow window of one or two words on each side picks up syntactic, substitutable similarity, meaning words that could grammatically replace one another such as cat and dog, or run and walk. A wide window of a sentence or a paragraph picks up topical similarity instead, meaning words from the same subject matter such as doctor and hospital, which are related without being substitutable.

The unit matters too. Using the document a word appears in gives the term document matrix of information retrieval from Chapter 4, while using neighbouring words gives the term term matrix that this part of the book pursues.

You can go further and let grammatical relations define context instead of mere proximity (Padó and Lapata 2007), so that a word’s company becomes its syntactic dependents rather than its linear neighbours. Each choice tunes the meaning the geometry will express, and none of them is uniquely correct.

From counting to learning

The vector space idea is now in place. Fix a notion of context, count co-occurrences into a matrix, then read similarity as the cosine between rows.

This is a distributional semantic model, and even in the rough form sketched here it works, recovering synonyms and thematic clusters from raw text with no supervision at all.

Two problems stand between this clean idea and a usable representation, though. Raw counts are a crude reflection of association and must be reweighted, so that they measure genuine association rather than mere frequency. The matrix itself is unwieldy, being as wide as the vocabulary, mostly zeros, and rebuilt from scratch whenever the corpus grows, so it must be compressed until thousands of sparse dimensions become a few hundred dense ones.

Those two repairs are pointwise mutual information and the singular value decomposition, and they are the business of Chapter 7.

Later, in Chapter 9, even that classical pipeline is set aside in favour of learning the vectors directly, while the distributional hypothesis of this chapter comes along completely unchanged. It is the permanent idea, and only the arithmetic that cashes it out keeps improving.