Natural language processing is the job of getting computers to work with human language. To read it, write it, translate it, answer questions about it, and now to hold a conversation in it.
Put that way it sounds like any other programming problem. It is not, and why it is not is the subject of this book.
A programming language such as Python was designed to be unambiguous. Every well formed program has exactly one meaning. A grammar of a few pages fixes that meaning, and a compiler works it out mechanically.
Human language was designed by nobody. It grew. And at every level it is full of the one thing a compiler cannot tolerate: ambiguity.
So here is the first idea to absorb. Understanding language is not a lookup. It is an act of disambiguation. And the information you need to disambiguate is often not in the sentence at all.
Take the word bank. It is a place that keeps money. It is the side of a river. It is what a plane does when it turns. It is also a verb, meaning to rely on something.
You picked the right sense just now without noticing, from context. But the word on its own really is ambiguous four ways. A machine has to settle that question before it can do anything else. This is lexical ambiguity, and almost every common word has some.
Ambiguity gets worse when words combine. Consider “I saw the man with the telescope”. It has two grammatical structures. Either I had the telescope, or the man did. English gives you no rule for choosing. Only knowledge of the situation does that.
Newspaper headlines show the effect in its purest form, because editors squeeze the words so hard:
“Juvenile court to try shooting defendant”
“Local high school dropouts cut in half”
“Safety experts say school bus passengers should be belted”
Each one has a sensible reading and an absurd one. Both are equally grammatical. What rules out the absurd reading is not grammar. It is what we know about the world. This is syntactic and semantic ambiguity, and it is why parsing a sentence cannot be separated from understanding it.
Past the sentence lies pragmatics, which is meaning that depends on context, intent and convention. “Can you pass the salt?” is a question about your abilities only if you read it literally. At a dinner table it is a request. Sarcasm flips meaning outright. “Well, that went great” after a disaster means the opposite of what it says, and even people find this hard. At least they have tone to go on.
The same lesson keeps coming back. The words on the page do not pin down the meaning. Closing that gap takes knowledge the text assumes but never states.
That is why NLP is hard. Language is an efficient code between minds that already share a world. The computer starts out sharing none of it.
Faced with a problem that large, the field breaks it into tasks. A task is a specific, measurable job that carves off one piece of the whole. You will meet many of them in this book, so it is worth naming a few now.
Sentiment analysis labels a text by the attitude it expresses. Is this review positive or negative? Question answering returns the answer to a question, either from a passage you supply or from general knowledge. Give it “The cat sat on the mat”. Ask “Where did the cat sit?” It should answer the mat. Textual entailment decides whether one sentence follows from another. Machine translation turns text in one language into another. Named entity recognition finds the people, places and organisations in a text. Summarisation shortens a text. Text generation continues it. The list runs on through speech recognition, spelling correction and dialogue.
What holds the list together is the shape of each item. Every task has defined inputs, expected outputs, and a way to score how well a system did. That last part is what makes progress measurable.
The tasks themselves have barely changed in decades. How they are solved has changed completely. For most of the field’s history, each task had its own specialised machinery. Now a single kind of model can do most of them, often just by being asked. That model is a large language model, and it is trained to do one thing: predict the next word. ChatGPT, Claude and Gemini are the visible surface of that change.
This book tells the story of how the field got from counting words to models that converse. It is one idea, deepening, rather than a catalogue of techniques.
The idea is this. Language resists rules, but it yields to numbers and to learning.
The story has four movements, and they are the four parts that follow.
First, language as data. We gather text into corpora and clean it. Then we measure the statistical regularities that refuse to go away. That is the rest of Part I.
Second, meaning as geometry. We turn words into vectors, so that similar meaning becomes nearness in space. That is Part II.
Third, language models. We learn to assign probabilities to word sequences. We start with plain counting and build through neural and recurrent models. That is Part III.
Fourth, the transformer era. The self attention architecture, the large language models it made possible, and the craft of working with them. That is Parts IV and V.
Each movement keeps what the previous one built and adds one new ability. By the end you will see what the model of Chapter 17 really is. It is a direct descendant of the word counting in Chapter 10. The same question, what comes next?, answered with far more powerful machinery.
One word on how to read this. The text explains how each idea works. The problems at the end of every chapter make you prove it, usually by computing a small case by hand. In this subject a worked example is almost always the fastest route to understanding.
We start, in the next chapter, with the raw material for everything that follows. The corpus.
Jurafsky and Martin’s Speech and Language Processing is the standard textbook and a good companion to this one. Manning and Schütze’s Foundations of Statistical Natural Language Processing is the classic treatment of the statistical foundations. The Handbook of Natural Language Processing surveys the full breadth of the field’s tasks.
Find the ambiguity. For each sentence, describe the two readings and say what knowledge a reader uses to prefer one: (a) “The chicken is ready to eat.” (b) “Flying planes can be dangerous.” (c) “She told her mother that she had won.” Which of the three is settled by world knowledge rather than by grammar?
Programming versus natural language. List three concrete properties of a programming language that make it mechanically interpretable. For each one, give the property of English that makes it not. Then say why this difference means that parsing a sentence cannot be cleanly separated from understanding it.
Turn a goal into a task. Pick something you would like a computer to do, say flag toxic comments, or answer questions about a manual. Specify it as a concrete NLP task. What are the inputs? What are the outputs? And, hardest of all, how would you score whether a system did it well? Why is the scoring definition the part that makes progress possible?
Lexical ambiguity census. Take the word run and list as many distinct senses as you can, both noun and verb. For three of them, write a short sentence and underline the context words that settle the meaning. What does the length of your list suggest about representing a word by one fixed meaning? Chapter 15 returns to that difficulty.