{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Unrolling, and the vanishing gradient\n", "\n", "What unrolling does to the parameter count, and what it does to the gradient.\n", "\n", "From chapter 12, [Recurrent Networks](https://nlp.jcrlabz.com/book/recurrent/), of the course notes.\n", "\n", "Source: `book/code/worked_examples/rnn.py`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What you need\n", "\n", "`pip install numpy`. The next cell does it. Colab usually has numpy already, but nothing here assumes that." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "%pip install -q numpy" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The script\n", "\n", "Everything the example defines, in one cell. Run it and the definitions\n", "are live in the notebook." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "\"\"\"The recurrent network, worked as Chapter 12 works it.\n", "\n", "Four things, all reproducing the chapter's tables:\n", "\n", " unrolling three time steps by hand, h_0 to h_3\n", " the parameter count why an RNN does not grow with the sequence\n", " vanishing gradients the product of Jacobians, and where it dies\n", " gradient clipping what the rescale actually does\n", "\n", " python3 rnn.py # all four\n", " python3 rnn.py --unroll # the three-step trace\n", " python3 rnn.py --params # RNN against a feed-forward window\n", " python3 rnn.py --vanish # the decay table, and the 47-word example\n", " python3 rnn.py --clip # clipping, before and after\n", "\n", "Install: pip install numpy\n", "\"\"\"\n", "\n", "import argparse\n", "import math\n", "\n", "try:\n", " import numpy as np\n", "except ImportError:\n", " raise SystemExit(\"this one needs numpy: pip install numpy\")\n", "\n", "\n", "# ------------------------------------------------------------- the recurrence\n", "\n", "# A model small enough to print in full. Two hidden units, three vocabulary\n", "# words, embeddings of size two.\n", "WORDS = [\"the\", \"dog\", \"ran\"]\n", "E = np.array([[0.50, -0.20], # the\n", " [0.10, 0.80], # dog\n", " [-0.60, 0.30]]) # ran\n", "W = np.array([[0.60, -0.30], # input to hidden, 2 x 2\n", " [0.20, 0.50]])\n", "U = np.array([[0.40, 0.10], # hidden to hidden, 2 x 2\n", " [-0.20, 0.70]])\n", "bh = np.array([0.05, -0.05])\n", "V = np.array([[0.70, -0.30], # hidden to vocabulary, 3 x 2\n", " [-0.40, 0.60],\n", " [0.10, 0.20]])\n", "by = np.zeros(3)\n", "\n", "\n", "def softmax(z):\n", " e = np.exp(z - z.max())\n", " return e / e.sum()\n", "\n", "\n", "def show_unroll():\n", " print(\"\\nSTEP 1 unrolling three time steps by hand\\n\")\n", " print(\" h_t = tanh(W x_t + U h_{t-1} + b_h), y_t = softmax(V h_t + b_y)\")\n", " print(f\"\\n The same W, U and V are used at every step. That is what the\")\n", " print(f\" word 'recurrent' means, and it is why the parameter count does\")\n", " print(f\" not grow with the sentence.\\n\")\n", "\n", " seq = [\"the\", \"dog\", \"ran\"]\n", " h = np.zeros(2)\n", " print(f\" h_0 = {h} the state before any word is read\\n\")\n", " def vec(a):\n", " return \"(\" + \", \".join(f\"{v:+.4f}\" for v in a) + \")\"\n", "\n", " print(f\" {'t':>2}{'word':>6}{'x_t':>21}{'W x + U h + b':>21}{'h_t':>21}\")\n", " print(\" \" + \"-\" * 71)\n", " for t, w in enumerate(seq, start=1):\n", " x = E[WORDS.index(w)]\n", " pre = W @ x + U @ h + bh\n", " h = np.tanh(pre)\n", " print(f\" {t:>2}{w:>6}{vec(x):>21}{vec(pre):>21}{vec(h):>21}\")\n", "\n", " print(f\"\\n Every h_t carries the whole prefix, not a fixed window.\")\n", " print(f\" h_3 was computed from h_2, which was computed from h_1.\")\n", "\n", " print(f\"\\n The output at each step is a distribution over the vocabulary:\\n\")\n", " h = np.zeros(2)\n", " print(f\" {'t':>2}{'read':>6} \" + \"\".join(f\"{w:>9}\" for w in WORDS)\n", " + f\"{' most likely next':>20}\")\n", " print(\" \" + \"-\" * 62)\n", " for t, w in enumerate(seq, start=1):\n", " h = np.tanh(W @ E[WORDS.index(w)] + U @ h + bh)\n", " p = softmax(V @ h + by)\n", " print(f\" {t:>2}{w:>6} \" + \"\".join(f\"{v:>9.4f}\" for v in p)\n", " + f\"{WORDS[int(p.argmax())]:>20}\")\n", " print(\"\\n The weights are random here, so the predictions mean nothing.\")\n", " print(\" The shape is the point. One distribution per time step, and a\")\n", " print(\" language model needs exactly one distribution per time step.\")\n", "\n", "\n", "# ------------------------------------------------------------ the parameters\n", "\n", "def show_params(V_size=10000, d=100, H=500):\n", " print(f\"\\n\\nSTEP 2 the parameter count stops depending on the sequence\\n\")\n", " print(f\" |V| = {V_size}, embedding d = {d}, hidden H = {H}\\n\")\n", " print(f\" {'matrix':<10}{'shape':<18}{'parameters':>14}\")\n", " print(\" \" + \"-\" * 44)\n", " rows = [(\"E\", f\"{V_size} x {d}\", V_size * d),\n", " (\"W\", f\"{H} x {d}\", H * d),\n", " (\"U\", f\"{H} x {H}\", H * H),\n", " (\"V\", f\"{H} x {V_size}\", H * V_size),\n", " (\"biases\", f\"{H} + {V_size}\", H + V_size)]\n", " total = sum(n for _, _, n in rows)\n", " for name, shape, n in rows:\n", " print(f\" {name:<10}{shape:<18}{n:>14,}\")\n", " print(\" \" + \"-\" * 44)\n", " print(f\" {'total':<28}{total:>14,}\")\n", "\n", " print(f\"\\n Not one of those shapes mentions the sentence length.\")\n", " print(f\" U is H x H whether the sentence has 5 words or 500.\\n\")\n", "\n", " print(f\" Compare the feed-forward model of Chapter 11, whose W_h is\")\n", " print(f\" (context x d) x H and therefore grows with the window:\\n\")\n", " print(f\" {'context words':>15}{'feed-forward W_h':>20}{'RNN U':>12}\")\n", " print(\" \" + \"-\" * 48)\n", " for ctx in (3, 5, 10, 50, 500):\n", " print(f\" {ctx:>15}{ctx * d * H:>20,}{H * H:>12,}\")\n", "\n", " print(f\"\\n The left column grows without limit. The right column is a\")\n", " print(f\" constant. That single fact is why the recurrence was worth\")\n", " print(f\" inventing, and it is what lets an RNN read a whole paragraph.\")\n", "\n", "\n", "# ------------------------------------------------------- vanishing gradients\n", "\n", "def show_vanish():\n", " print(\"\\n\\nSTEP 3 why the memory fades\\n\")\n", " print(\" Backpropagation through time multiplies one Jacobian per step:\\n\")\n", " print(\" dE/dh_0 = dE/dh_tau * product over t of dh_t / dh_{t-1}\\n\")\n", " print(\" Each factor is diag(tanh'(...)) U. So the whole product behaves\")\n", " print(\" like a number raised to the power of the distance.\\n\")\n", "\n", " print(f\" {'distance':>9}\" + \"\".join(f\"{f'x {g}':>13}\" for g in\n", " (0.5, 0.9, 1.0, 1.1, 1.2)))\n", " print(\" \" + \"-\" * 74)\n", " for tau in (1, 5, 10, 20, 47, 100):\n", " cells = \"\".join(f\"{g ** tau:>13.3e}\" for g in (0.5, 0.9, 1.0, 1.1, 1.2))\n", " print(f\" {tau:>9}{cells}\")\n", "\n", " print(f\"\\n Read the columns. Below 1 the gradient dies. Above 1 it\")\n", " print(f\" explodes. Exactly at 1 it survives, and nothing keeps it there.\")\n", " print(f\"\\n The threshold is the largest eigenvalue of U. If every\")\n", " print(f\" eigenvalue satisfies |lambda| < 1 the gradients vanish, and if\")\n", " print(f\" any satisfies |lambda| > 1 they explode.\")\n", "\n", " print(f\"\\n\\n THE 47 WORD EXAMPLE\\n\")\n", " print(\" 'Raj entered CoffeeDay to meet his partner Dru. ... After a long\")\n", " print(\" and fruitful discussion, Raj said goodbye to his ______'\")\n", " print(\"\\n The answer is 'partner', 47 words back. What reaches it:\\n\")\n", " for g in (0.01, 0.5, 1.2):\n", " v = g ** 47\n", " verdict = (\"gone entirely\" if v < 1e-12\n", " else \"exploded\" if v > 1 else \"too small to matter\")\n", " print(f\" {g} ^ 47 = {v:.6e} {verdict}\")\n", "\n", " print(\"\\n A gradient of 1e-94 is not a small update. It is no update.\")\n", " print(\" The network cannot learn the dependency, however true it is.\")\n", " print(\"\\n Note what is not broken. The forward pass carries information\")\n", " print(\" fine. It is the backward pass that cannot deliver the blame,\")\n", " print(\" so the weights that would capture the link never move.\")\n", "\n", "\n", "# ---------------------------------------------------------- gradient clipping\n", "\n", "def show_clip(threshold=5.0):\n", " print(f\"\\n\\nSTEP 4 gradient clipping (threshold = {threshold})\\n\")\n", " print(\" Vanishing needs a new architecture. Exploding has a cheap fix.\")\n", " print(\" If the gradient is longer than the threshold, rescale it to the\")\n", " print(\" threshold and keep its direction:\\n\")\n", " print(\" if ||g|| > threshold: g <- (threshold / ||g||) g\\n\")\n", " rng = np.random.default_rng(0)\n", " print(f\" {'step':>5}{'||g|| before':>15}{'||g|| after':>14}\"\n", " f\"{'direction kept':>17}\")\n", " print(\" \" + \"-\" * 52)\n", " for step, scale in enumerate([0.4, 1.2, 3.0, 40.0, 900.0], start=1):\n", " g = rng.normal(0, 1, 6) * scale\n", " n = np.linalg.norm(g)\n", " g2 = g * (threshold / n) if n > threshold else g\n", " cos = float(g @ g2 / (np.linalg.norm(g) * np.linalg.norm(g2)))\n", " print(f\" {step:>5}{n:>15.4f}{np.linalg.norm(g2):>14.4f}\"\n", " f\"{cos:>17.4f}\")\n", "\n", " print(\"\\n The last column is 1.0000 every time. Clipping changes how far\")\n", " print(\" the step goes and never which way it points.\")\n", " print(\"\\n That is the whole trick, and it is why one line of code stops\")\n", " print(\" a training run from diverging. It does nothing at all for the\")\n", " print(\" vanishing case, where the gradient is already too small.\")\n", "\n", "\n", "def main():\n", " ap = argparse.ArgumentParser(description=__doc__)\n", " for f in (\"unroll\", \"params\", \"vanish\", \"clip\"):\n", " ap.add_argument(f\"--{f}\", action=\"store_true\")\n", " a = ap.parse_args()\n", " picked = a.unroll or a.params or a.vanish or a.clip\n", " if a.unroll or not picked:\n", " show_unroll()\n", " if a.params or not picked:\n", " show_params()\n", " if a.vanish or not picked:\n", " show_vanish()\n", " if a.clip or not picked:\n", " show_clip()\n", " print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Running it\n", "\n", "The script is a command line tool, so each run below sets the arguments\n", "and calls `main()`. The heading shows the equivalent shell command." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import sys\n", "\n", "def run(*args):\n", " sys.argv = [\"rnn.py\", *args]\n", " main()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `python3 rnn.py`\n", "\n", "Unrolling, the flat parameter count, the decay and the fix." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "run()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `python3 rnn.py --vanish`\n", "\n", "Backpropagation through time multiplies one Jacobian per step. At a per-step factor of 0.5, what reaches 47 words back is 7.1e-15." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "run(\"--vanish\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `python3 rnn.py --clip`\n", "\n", "Clipping, before and after." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "run(\"--clip\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "Read the chapter this comes from: [Recurrent Networks](https://nlp.jcrlabz.com/book/recurrent/)." ] } ], "metadata": { "colab": { "name": "rnn.ipynb", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }