Chapter 6Vector Space Models

Meaning you can measure

So far this book has treated a word as a symbol. An atom with an identity and nothing else.

Everything has handled words as opaque strings. The n-gram counts ahead. The tokeniser of Chapter 5. Even the term weighting of Chapter 4. Two strings are either equal or unequal.

Symbols have a fatal poverty. They cannot be similar. To a symbol processor, cat and dog are exactly as different as cat and democracy.

But we want a machine to know something obvious. A review calling a film “superb” means much what one calling it “excellent” means. So we need a representation where nearness carries meaning. Cat and dog should sit close together, and democracy far away.

We need to turn meaning into geometry. That is the programme of this part of the book, and this chapter lays its foundation.

Make the poverty of symbols concrete first. The obvious way to feed a word to a numerical model is a one-hot vector. Fix the vocabulary. Give each word its own dimension. Represent a word by a vector that is $1$ on its own axis and $0$ everywhere else.

The sizes are alarming. Jane Austen’s Emma has a vocabulary of about $7{,}079$ types, so each word becomes a $7{,}079$-dimensional vector. The Google News corpus would demand three million dimensions.

Worse, the representation encodes nothing. Any two distinct one-hot vectors are orthogonal. So the cosine similarity of Chapter 4 between any two different words is exactly $0$.

One-hot vectors make every word equidistant from every other. Cat is no nearer dog than it is to democracy. The geometry is there, in that we have vectors and a space, but it carries no meaning. The rest of this chapter fills that empty space with structure.

The distributional hypothesis

Where could the structure come from? Not from a word’s spelling, which is arbitrary. From its use.

This is one of the oldest and most productive ideas in linguistics, and it has two famous statements. The linguist J. R. Firth put it as a slogan: “you shall know a word by the company it keeps”. Zellig Harris gave it a form a computer can act on. Words that occur in the same contexts tend to have similar meanings.

That is the distributional hypothesis. Its promise is enormous, because a word’s contexts are something we can simply count. No dictionary is needed. No grammar, and no human annotator. We need a corpus and the patience to tally, for each word, what it appears near. Meaning is distilled from distribution.

The operational move follows at once. Represent each word not by its own lonely axis, but by a vector of the contexts it keeps.

The dimensions become other words. The entry for word $w$ on dimension $c$ records how often $w$ and $c$ co-occur. This is a word context co-occurrence matrix $M$, and each row of it is a word’s distributional fingerprint.

Two words with similar rows are used similarly. By the hypothesis, they therefore mean similarly. Similarity of meaning has become similarity of vectors, and similarity of vectors is just the cosine of Chapter 4. The empty one-hot space is replaced by one where position is earned by usage.

First order and second order

There are two quite different things one can mean by “words that go together”. Keeping them apart is the most important idea in this chapter.

First order association is direct co-occurrence. Two words are first order associates if they appear near each other. Coffee and cup. Doctor and hospital. San and Francisco.

These are the entries in a row of $M$, the words a target actually keeps company with. First order structure captures collocations and thematic links. These are the syntagmatic relations, words that combine in running text.

Second order association is subtler, and for meaning it is more powerful. Two words are second order associates if they appear in the same kinds of context. Their rows of $M$ are similar, even when the two words never occur together at all.

Cat and dog are the standard example. Both co-occur with pet, vet, feed and fur. Their rows of $M$ therefore point in nearly the same direction. Yet a sentence rarely contains both.

Their similarity is not that they appear with each other. It is that they appear like each other. This is the paradigmatic relation. It is what lets distributional models discover synonyms, and, as Chapter 9 will show, analogies.

Here is the rule of thumb. First order association is read along a row. Second order association is read by comparing rows.

A matrix you can check by hand

A three word example makes the distinction unmistakable. Suppose that after counting co-occurrences over some corpus, three words have these rows over three context dimensions:

The rows for dog and cat are identical. So the cosine between them is exactly $1$, which is maximal similarity. Nothing in the table says dog and cat ever occur together. That is second order association at full strength.

The row for car shares no nonzero dimension with either animal, so its cosine with each is $0$.

Now contrast this with one-hot vectors for the same three words. There, all three pairwise cosines are $0$. One-hot cannot tell that dog and cat are more alike than dog and car. The co-occurrence representation can.

That single change is what turns a symbol into a meaning. We went from an axis per word to a vector of contexts.

What counts as context

The definition of “context” is a real design choice, not a detail. It decides what kind of similarity the vectors capture.

A narrow window of one or two words on each side picks up syntactic, substitutable similarity. Words that could grammatically replace one another, like cat and dog, or run and walk.

A wide window of a sentence or a paragraph picks up topical similarity. Words from the same subject matter, like doctor and hospital, which are related but not substitutable.

The unit matters too. Using the document a word appears in gives the term document matrix of information retrieval, from Chapter 4. Using neighbouring words gives the term term matrix that this part of the book pursues.

You can go further and let grammatical relations define context instead of mere proximity . Then a word’s company is its syntactic dependents, not its linear neighbours.

Each choice tunes the meaning the geometry will express. None is uniquely correct.

From counting to learning

The vector space idea is now in place. Fix a notion of context. Count co-occurrences into a matrix. Read similarity as the cosine between rows.

This is a distributional semantic model. Even in the rough form sketched here it works. It recovers synonyms and thematic clusters from raw text, with no supervision at all.

But two problems stand between this clean idea and a usable representation.

Raw counts are a crude reflection of association. They must be reweighted, so they measure genuine association rather than mere frequency.

The matrix itself is unwieldy. It is as wide as the vocabulary, mostly zeros, and rebuilt from scratch whenever the corpus grows. It must be compressed, so thousands of sparse dimensions become a few hundred dense ones.

Those two repairs are pointwise mutual information and the singular value decomposition. They are the business of Chapter 7.

Later, in Chapter 9, even that classical pipeline is set aside in favour of learning the vectors directly. The distributional hypothesis of this chapter comes along completely unchanged. It is the permanent idea. Only the arithmetic that cashes it out keeps improving.

Further reading.

is the classic statement of distributional structure. Firth’s “company it keeps” slogan comes from his 1957 Papers in Linguistics. is an early and influential construction of a word space from co-occurrence. develops the dependency based notion of context. Manning, Raghavan and Schütze treat the vector space model and cosine similarity in the retrieval setting.