{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# What one vector per word costs\n", "\n", "Averaging two senses into one vector, and the layer weights ELMo learns instead.\n", "\n", "From chapter 15, [Contextual Embeddings](https://nlp.jcrlabz.com/book/contextual/), of the course notes.\n", "\n", "Source: `book/code/worked_examples/contextual.py`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What you need\n", "\n", "Nothing. This example uses only the Python standard library." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Nothing to install: this example uses only the Python standard library.\n", "import sys; print(sys.version.split()[0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The script\n", "\n", "Everything the example defines, in one cell. Run it and the definitions\n", "are live in the notebook." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "\"\"\"Contextual embeddings, worked as Chapter 15 works them.\n", "\n", "Three things, all reproducing the chapter's tables:\n", "\n", " the averaging cost what one vector per word does to 'bank'\n", " ELMo layer mixing the softmax weights, and what each layer contributes\n", " masked prediction why BERT masks 15 per cent and not 100\n", "\n", " python3 contextual.py # all three\n", " python3 contextual.py --bank # static against contextual\n", " python3 contextual.py --elmo # layer weights, three task profiles\n", " python3 contextual.py --mask # the masking budget\n", "\n", "Install: nothing, the Python standard library is enough\n", "\"\"\"\n", "\n", "import argparse\n", "import math\n", "\n", "# A toy space on three axes: (money, river, generic). The two senses of 'bank'\n", "# are given separately, as a contextual model would produce them.\n", "SPACE = {\n", " \"bank/money\": (0.90, 0.05, 0.20),\n", " \"bank/river\": (0.05, 0.90, 0.20),\n", " \"deposit\": (0.95, 0.02, 0.15),\n", " \"loan\": (0.92, 0.03, 0.18),\n", " \"vault\": (0.88, 0.01, 0.10),\n", " \"water\": (0.02, 0.93, 0.15),\n", " \"erosion\": (0.01, 0.90, 0.08),\n", " \"flood\": (0.03, 0.95, 0.12),\n", "}\n", "\n", "\n", "def cosine(u, v):\n", " d = sum(a * b for a, b in zip(u, v))\n", " n = math.sqrt(sum(a * a for a in u)) * math.sqrt(sum(b * b for b in v))\n", " return d / n if n else 0.0\n", "\n", "\n", "def neighbours(vec, exclude=(), top=4):\n", " scored = [(w, cosine(vec, v)) for w, v in SPACE.items() if w not in exclude]\n", " return sorted(scored, key=lambda p: -p[1])[:top]\n", "\n", "\n", "# ------------------------------------------------------------- the averaging\n", "\n", "def show_bank():\n", " money = SPACE[\"bank/money\"]\n", " river = SPACE[\"bank/river\"]\n", " static = tuple((a + b) / 2 for a, b in zip(money, river))\n", "\n", " print(\"\\nSTEP 1 what one vector per word costs\\n\")\n", " print(\" Two senses of 'bank', as a contextual model would give them:\\n\")\n", " print(f\" bank (money) {money}\")\n", " print(f\" bank (river) {river}\")\n", " print(f\"\\n A static embedding has to pick one vector. Trained on a corpus\")\n", " print(f\" with both senses in equal measure, it lands on their average:\\n\")\n", " print(f\" bank (static) ({static[0]:.3f}, {static[1]:.3f},\"\n", " f\" {static[2]:.3f})\")\n", "\n", " print(f\"\\n Now ask each of the three for its nearest neighbours.\\n\")\n", " for label, vec, ex in [(\"bank, money sense\", money, (\"bank/money\",)),\n", " (\"bank, river sense\", river, (\"bank/river\",)),\n", " (\"bank, static average\", static, ())]:\n", " top = neighbours(vec, exclude=ex + (\"bank/money\", \"bank/river\"))\n", " cells = \" \".join(f\"{w} {s:+.3f}\" for w, s in top)\n", " print(f\" {label:<22}{cells}\")\n", "\n", " print(f\"\\n The two sense vectors give clean, single-topic lists. The\")\n", " print(f\" average gives a list that mixes both and commits to neither.\")\n", "\n", " print(f\"\\n Measure it. Cosine of the static average against each sense:\\n\")\n", " print(f\" to the money sense {cosine(static, money):.4f}\")\n", " print(f\" to the river sense {cosine(static, river):.4f}\")\n", " print(f\" the two senses to each other {cosine(money, river):.4f}\")\n", "\n", " print(f\"\\n The average sits {cosine(static, money):.2f} from each sense,\")\n", " print(f\" which sounds close until you notice the senses are only\")\n", " print(f\" {cosine(money, river):.2f} from each other. The static vector is\")\n", " print(f\" equally wrong in two directions rather than right in one.\")\n", "\n", " print(f\"\\n And the damage scales with how balanced the corpus is. A\")\n", " print(f\" corpus that is 90 per cent money sense gives:\\n\")\n", " for share in (0.5, 0.7, 0.9, 0.99):\n", " v = tuple(share * a + (1 - share) * b for a, b in zip(money, river))\n", " print(f\" {share:>5.0%} money to money {cosine(v, money):.4f}\"\n", " f\" to river {cosine(v, river):.4f}\")\n", " print(f\"\\n So a static vector does not represent a word. It represents\")\n", " print(f\" the word's frequency-weighted mixture in whatever corpus you\")\n", " print(f\" happened to train on.\")\n", "\n", "\n", "# --------------------------------------------------------------- ELMo mixing\n", "\n", "def show_elmo():\n", " print(\"\\n\\nSTEP 2 ELMo mixes the layers, and the task picks the mix\\n\")\n", " print(\" A biLM of L layers gives L+1 representations per token. ELMo\")\n", " print(\" combines them with softmax-normalised task weights:\\n\")\n", " print(\" ELMo_k = gamma * sum_j s_j * R_kj\\n\")\n", " print(\" Only s_0..s_L and gamma are learned downstream. The biLM is\")\n", " print(\" frozen, which is why ELMo was cheap to adopt.\\n\")\n", "\n", " layers = [\"layer 0, characters\", \"layer 1, lower biLSTM\",\n", " \"layer 2, upper biLSTM\"]\n", " profiles = {\n", " \"part-of-speech tagging\": [0.2, 1.8, 0.5],\n", " \"word sense disambiguation\": [0.1, 0.6, 2.0],\n", " \"no preference\": [1.0, 1.0, 1.0],\n", " }\n", "\n", " for task, raw in profiles.items():\n", " m = max(raw)\n", " e = [math.exp(r - m) for r in raw]\n", " s = [x / sum(e) for x in e]\n", " print(f\" {task}\")\n", " print(f\" {'layer':<24}{'raw weight':>12}{'s_j softmax':>14}\")\n", " print(\" \" + \"-\" * 50)\n", " for name, r, w in zip(layers, raw, s):\n", " print(f\" {name:<24}{r:>12.2f}{w:>14.4f}\")\n", " print(f\" {'':<24}{'':>12}{sum(s):>14.4f}\\n\")\n", "\n", " print(\" The three profiles are the finding of the ELMo paper, stated as\")\n", " print(\" weights. Lower layers carry more syntax, upper layers more\")\n", " print(\" meaning, and which one a task wants is a fact you can read off\")\n", " print(\" the learned s_j rather than a thing you decide in advance.\")\n", " print(\"\\n That is also the strongest evidence that depth is doing\")\n", " print(\" something structured rather than just adding capacity.\")\n", "\n", "\n", "# ------------------------------------------------------------- masking budget\n", "\n", "def show_mask():\n", " print(\"\\n\\nSTEP 3 why mask 15 per cent\\n\")\n", " print(\" BERT trains by hiding tokens and predicting them. The masking\")\n", " print(\" rate is a trade, and both ends of it are easy to see.\\n\")\n", "\n", " n = 512\n", " print(f\" A sequence of {n} tokens.\\n\")\n", " print(f\" {'rate':>7}{'masked':>9}{'visible':>9}\"\n", " f\"{'training signal':>18}{'context left':>15}\")\n", " print(\" \" + \"-\" * 60)\n", " for rate in (0.01, 0.15, 0.50, 0.90):\n", " masked = int(n * rate)\n", " print(f\" {rate:>6.0%}{masked:>9}{n - masked:>9}\"\n", " f\"{masked:>18}{1 - rate:>14.0%}\")\n", "\n", " print(\"\\n At 1 per cent you get 5 predictions from a forward pass that\")\n", " print(\" cost you the whole sequence. Training is correct and slow.\")\n", " print(\"\\n At 90 per cent you get plenty of predictions and almost no\")\n", " print(\" context to make them from. The task becomes guessing.\")\n", " print(\"\\n BERT chose 15 per cent, which gives 76 gradients per sequence\")\n", " print(\" with 85 per cent of the sentence still readable. Later work has\")\n", " print(\" found higher rates workable on larger models, so the number is\")\n", " print(\" a balance point rather than a constant of nature.\")\n", "\n", " print(\"\\n\\n THE PRETRAIN AND FINE-TUNE MISMATCH\\n\")\n", " print(\" The token [MASK] appears in every pretraining batch and never\")\n", " print(\" once at fine-tuning time. A model that keyed on it would break\")\n", " print(\" the moment it was used for real.\")\n", " print(\"\\n So of the 15 per cent chosen, BERT does this:\\n\")\n", " chosen = 100\n", " split = [(\"replaced with [MASK]\", 0.80),\n", " (\"replaced with a random token\", 0.10),\n", " (\"left unchanged\", 0.10)]\n", " print(f\" {'treatment':<32}{'share':>8}{'per 100 chosen':>17}\")\n", " print(\" \" + \"-\" * 58)\n", " for name, share in split:\n", " print(f\" {name:<32}{share:>7.0%}{share * chosen:>17.0f}\")\n", "\n", " print(\"\\n The last two rows are the interesting ones. Because a chosen\")\n", " print(\" position might be unchanged, the model cannot tell which\")\n", " print(\" positions were chosen. So it has to build a usable\")\n", " print(\" representation of every token, not just the masked ones.\")\n", " print(\"\\n The random-token row does something else. It forces the model\")\n", " print(\" to notice when a token does not fit its context, which is a\")\n", " print(\" skill no amount of [MASK] prediction would teach.\")\n", "\n", "\n", "def main():\n", " ap = argparse.ArgumentParser(description=__doc__)\n", " for f in (\"bank\", \"elmo\", \"mask\"):\n", " ap.add_argument(f\"--{f}\", action=\"store_true\")\n", " a = ap.parse_args()\n", " picked = a.bank or a.elmo or a.mask\n", " if a.bank or not picked:\n", " show_bank()\n", " if a.elmo or not picked:\n", " show_elmo()\n", " if a.mask or not picked:\n", " show_mask()\n", " print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Running it\n", "\n", "The script is a command line tool, so each run below sets the arguments\n", "and calls `main()`. The heading shows the equivalent shell command." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import sys\n", "\n", "def run(*args):\n", " sys.argv = [\"contextual.py\", *args]\n", " main()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `python3 contextual.py`\n", "\n", "The cost of averaging, then the layer weights." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "run()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `python3 contextual.py --bank`\n", "\n", "One vector for two senses sits between both and matches neither." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "run(\"--bank\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `python3 contextual.py --mask`\n", "\n", "The masking budget, and what it buys." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "run(\"--mask\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "Read the chapter this comes from: [Contextual Embeddings](https://nlp.jcrlabz.com/book/contextual/)." ] } ], "metadata": { "colab": { "name": "contextual.ipynb", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }