From a251aae293a47c17d8caf6a7902d5025d04face9 Mon Sep 17 00:00:00 2001 From: Hamel Husain Date: Sat, 5 Mar 2022 18:32:44 -0800 Subject: [PATCH] update explanation of negative log loss (cross entropy loss) (#501) * update explanation of nll * spelling * clean * clean * add back stuff * fix lr syntax --- 04_mnist_basics.ipynb | 6 +- 05_pet_breeds.ipynb | 365 +++++++++++++++++++++++++++--------- clean/04_mnist_basics.ipynb | 6 +- clean/05_pet_breeds.ipynb | 46 ++++- 4 files changed, 318 insertions(+), 105 deletions(-) diff --git a/04_mnist_basics.ipynb b/04_mnist_basics.ipynb index e6368e4..ae4d5a8 100644 --- a/04_mnist_basics.ipynb +++ b/04_mnist_basics.ipynb @@ -2870,7 +2870,7 @@ "w -= gradient(w) * lr\n", "```\n", "\n", - "This is known as *stepping* your parameters, using an *optimizer step*.\n", + "This is known as *stepping* your parameters, using an *optimizer step*. Notice how we _subtract_ the `gradient * lr` from the parameter to update it. This allows us to adjust the parameter in the direction of the slope by increasing the parameter when the slope is negative and decreasing the parameter when the slope is positive. We want to adjust our parameters in the direction of the slope because our goal in deep learning is to _minimize_ the loss.\n", "\n", "If you pick a learning rate that's too low, it can mean having to do a lot of steps. <> illustrates that." ] @@ -3004,7 +3004,7 @@ "\n", "If we can solve this problem for the three parameters of a quadratic function, we'll be able to apply the same approach for other, more complex functions with more parameters—such as a neural net. Let's find the parameters for `f` first, and then we'll come back and do the same thing for the MNIST dataset with a neural net.\n", "\n", - "We need to define first what we mean by \"best.\" We define this precisely by choosing a *loss function*, which will return a value based on a prediction and a target, where lower values of the function correspond to \"better\" predictions. For continuous data, it's common to use *mean squared error*:" + "We need to define first what we mean by \"best.\" We define this precisely by choosing a *loss function*, which will return a value based on a prediction and a target, where lower values of the function correspond to \"better\" predictions. It is important for loss functions to return _lower_ values when predictions are more accurate, as the SGD procedure we defined earlier will try to _minimize_ this loss. For continuous data, it's common to use *mean squared error*:" ] }, { @@ -5853,7 +5853,7 @@ "split_at_heading": true }, "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" } diff --git a/05_pet_breeds.ipynb b/05_pet_breeds.ipynb index 2c53e2d..be6dfb1 100644 --- a/05_pet_breeds.ipynb +++ b/05_pet_breeds.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "code", - "execution_count": null, + "execution_count": 134, "metadata": {}, "outputs": [], "source": [ @@ -14,7 +14,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "metadata": {}, "outputs": [], "source": [ @@ -816,7 +816,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 113, "metadata": {}, "outputs": [], "source": [ @@ -826,7 +826,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 114, "metadata": {}, "outputs": [ { @@ -840,7 +840,7 @@ " [ 1.0698, 1.6187]])" ] }, - "execution_count": null, + "execution_count": 114, "metadata": {}, "output_type": "execute_result" } @@ -859,7 +859,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 115, "metadata": {}, "outputs": [ { @@ -873,7 +873,7 @@ " [0.7446, 0.8346]])" ] }, - "execution_count": null, + "execution_count": 115, "metadata": {}, "output_type": "execute_result" } @@ -895,7 +895,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 116, "metadata": {}, "outputs": [ { @@ -904,7 +904,7 @@ "tensor([0.6025, 0.5021, 0.1332, 0.9966, 0.5959, 0.3661])" ] }, - "execution_count": null, + "execution_count": 116, "metadata": {}, "output_type": "execute_result" } @@ -940,7 +940,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 117, "metadata": {}, "outputs": [ { @@ -954,7 +954,7 @@ " [0.3661, 0.6339]])" ] }, - "execution_count": null, + "execution_count": 117, "metadata": {}, "output_type": "execute_result" } @@ -1010,14 +1010,14 @@ " return torch.where(targets==1, 1-inputs, inputs).mean()\n", "```\n", "\n", - "Just as we moved from sigmoid to softmax, we need to extend the loss function to work with more than just binary classification—it needs to be able to classify any number of categories (in this case, we have 37 categories). Our activations, after softmax, are between 0 and 1, and sum to 1 for each row in the batch of predictions. Our targets are integers between 0 and 36.\n", + "Just as we moved from sigmoid to softmax, we need to extend the loss function to work with more than just binary classification—it needs to be able to classify any number of categories (in this case, we have 37 categories). Our activations, after softmax, are between 0 and 1, and sum to 1 for each row in the batch of predictions. Our targets are integers between 0 and 36. Furthermore, cross-entropy loss generalizes our binary classification loss and allows for more than one correct label per example (which is called multi-label classificaiton, which we will discuss in Chapter 6).\n", "\n", - "In the binary case, we used `torch.where` to select between `inputs` and `1-inputs`. When we treat a binary classification as a general classification problem with two categories, it actually becomes even easier, because (as we saw in the previous section) we now have two columns, containing the equivalent of `inputs` and `1-inputs`. So, all we need to do is select from the appropriate column. Let's try to implement this in PyTorch. For our synthetic 3s and 7s example, let's say these are our labels:" + "In the binary case, we used `torch.where` to select between `inputs` and `1-inputs`. When we treat a binary classification as a general classification problem with two categories, it actually becomes even easier, because (as we saw in the previous section) we now have two columns, containing the equivalent of `inputs` and `1-inputs`. Since there is only one correct label per example, all we need to do is select the appropriate column (as opposed to multiplying multiple probabilities). Let's try to implement this in PyTorch. For our synthetic 3s and 7s example, let's say these are our labels:" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 118, "metadata": {}, "outputs": [], "source": [ @@ -1033,7 +1033,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 119, "metadata": {}, "outputs": [ { @@ -1047,7 +1047,7 @@ " [0.3661, 0.6339]])" ] }, - "execution_count": null, + "execution_count": 119, "metadata": {}, "output_type": "execute_result" } @@ -1065,7 +1065,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 120, "metadata": {}, "outputs": [ { @@ -1074,7 +1074,7 @@ "tensor([0.6025, 0.4979, 0.1332, 0.0034, 0.4041, 0.3661])" ] }, - "execution_count": null, + "execution_count": 120, "metadata": {}, "output_type": "execute_result" } @@ -1088,61 +1088,73 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "To see exactly what's happening here, let's put all the columns together in a table. Here, the first two columns are our activations, then we have the targets, the row index, and finally the result shown immediately above:" + "To see exactly what's happening here, let's put all the columns together in a table. Here, the first two columns are our activations, then we have the targets and the row index. We explain the last column, `result` below:" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 136, "metadata": {}, "outputs": [ { "data": { "text/html": [ - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
3 7 targ idx loss
0.6024690.39753100-0.602469
0.5020650.49793511-0.497935
0.1331880.86681102-0.133188
0.996640.0033601713-0.00336017
0.5959490.40405114-0.404051
0.3661180.63388205-0.366118
" + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
37targidxresult
0.6024690.397531000.602469
0.5020650.497935110.497935
0.1331880.866811020.133188
0.9966400.003360130.003360
0.5959490.404051140.404051
0.3661180.633882050.366118
\n" ], "text/plain": [ "" @@ -1158,7 +1170,7 @@ "df = pd.DataFrame(sm_acts, columns=[\"3\",\"7\"])\n", "df['targ'] = targ\n", "df['idx'] = idx\n", - "df['loss'] = -sm_acts[range(6), targ]\n", + "df['result'] = sm_acts[range(6), targ]\n", "t = df.style.hide_index()\n", "#To have html code compatible with our script\n", "html = t._repr_html_().split('')[1]\n", @@ -1170,18 +1182,19 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Looking at this table, you can see that the final column can be calculated by taking the `targ` and `idx` columns as indices into the two-column matrix containing the `3` and `7` columns. That's what `sm_acts[idx, targ]` is actually doing.\n", - "\n", - "The really interesting thing here is that this actually works just as well with more than two columns. To see this, consider what would happen if we added an activation column for every digit (0 through 9), and then `targ` contained a number from 0 to 9. As long as the activation columns sum to 1 (as they will, if we use softmax), then we'll have a loss function that shows how well we're predicting each digit.\n", - "\n", - "We're only picking the loss from the column containing the correct label. We don't need to consider the other columns, because by the definition of softmax, they add up to 1 minus the activation corresponding to the correct label. Therefore, making the activation for the correct label as high as possible must mean we're also decreasing the activations of the remaining columns.\n", - "\n", + "Looking at this table, you can see that the `result` column can be calculated by taking the `targ` and `idx` columns as indices into the two-column matrix containing the `3` and `7` columns. That's what `sm_acts[idx, targ]` is actually doing. The really interesting thing here is that this actually works just as well with more than two columns. To see this, consider what would happen if we added an activation column for every digit (0 through 9), and then `targ` contained a number from 0 to 9." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ "PyTorch provides a function that does exactly the same thing as `sm_acts[range(n), targ]` (except it takes the negative, because when applying the log afterward, we will have negative numbers), called `nll_loss` (*NLL* stands for *negative log likelihood*):" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 13, "metadata": {}, "outputs": [ { @@ -1190,7 +1203,7 @@ "tensor([-0.6025, -0.4979, -0.1332, -0.0034, -0.4041, -0.3661])" ] }, - "execution_count": null, + "execution_count": 13, "metadata": {}, "output_type": "execute_result" } @@ -1201,7 +1214,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 14, "metadata": {}, "outputs": [ { @@ -1210,7 +1223,7 @@ "tensor([-0.6025, -0.4979, -0.1332, -0.0034, -0.4041, -0.3661])" ] }, - "execution_count": null, + "execution_count": 14, "metadata": {}, "output_type": "execute_result" } @@ -1230,24 +1243,26 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Taking the Log" + "> warning: Confusing Name, Beware: The nll in `nll_loss` stands for \"negative log likelihood,\" but it doesn't actually take the log at all! It assumes you have _already_ taken the log. PyTorch has a function called `log_softmax` that combines `log` and `softmax` in a fast and accurate way. `nll_loss` is designed to be used after `log_softmax`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "The function we saw in the previous section works quite well as a loss function, but we can make it a bit better. The problem is that we are using probabilities, and probabilities cannot be smaller than 0 or greater than 1. That means that our model will not care whether it predicts 0.99 or 0.999. Indeed, those numbers are so close together—but in another sense, 0.999 is 10 times more confident than 0.99. So, we want to transform our numbers between 0 and 1 to instead be between negative infinity and 0. There is a mathematical function that does exactly this: the *logarithm* (available as `torch.log`). It is not defined for numbers less than 0, and looks like this:" + "#### Taking the Log\n", + "\n", + "Recall that cross entropy loss may involve the multiplication of many numbers. Multiplying lots of negative numbers together can cause problems like [numerical underflow](https://en.wikipedia.org/wiki/Arithmetic_underflow) in computers. Therefore, we want to transform these probabilities to larger values so we can perform mathematical operations on them. There is a mathematical function that does exactly this: the *logarithm* (available as `torch.log`). It is not defined for numbers less than 0, and looks like this between 0 and 1:" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 124, "metadata": {}, "outputs": [ { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD7CAYAAACVMATUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAegElEQVR4nO3de3ycVYH/8c/JrWnuzb1Jm6Rpmt5paUNb7iCgwHJTdAFBwRVRLu7PxXVdV1kr6Etx1133ta8FZBdWVy5eAREFlFuhIpRS2tJLkrZJ2zTNPU0yuScz5/fHDKXUtEmbZ+aZPPN9v17zMp0cZ749Gb59cp4zzxhrLSIi4g1xbgcQERHnqNRFRDxEpS4i4iEqdRERD1Gpi4h4SILbAXJzc21ZWZnbMUREppS333673Vqbd/T9rpd6WVkZGzdudDuGiMiUYozZN9b9Wn4REfEQlbqIiIeo1EVEPESlLiLiISp1EREPUamLiHiISl1ExENc36cuIhILAgFLq2+IvR197OvoY29HP7edN5f05ERHn0elLiLikEDA0uIbpL69j30d/ext7zv89b7OPgZHAofHJsYbrlxexIJClbqIiGustXT2DVPf3kdde9/h4n6vvAdG/IfHJsXHUZKTQllOKmfPy6U0N5U5OamU5qRQlDWd+DjjeD6VuojIGPqHR4PF3fZ+ade191Hf1kvP4OjhcQlxhpLsFMpyUzmzIpeyUHGX5aYwMzM8xX08KnURiVmBgKWpZ5C6tl72tPZS197HnrZe6tr6aOoe/MDYosxk5uSlcuXyYubkph6+zZoxnYT46NlzolIXEc8bHPGzt6OPPa197G7tZU9b7+HyPnK5JH1aAuV5qawpz6E8N5XyvLTD5T09Kd7Fv8HEqdRFxDN8gyPsbu1lV2vwyHt3ay+723pp6OwnYINjjIHirOnMzUtj1Zxs5ualhW6p5KVPw5jILpc4TaUuIlNOd/8Ita0+drX0sqvVFyzyll6ae95fMkmKj2NObipLijK5cnkxFfnB4i7PTZsyR90nQ6UuIlGrZ3CEXS0+alt6qWn2sas1+HWbb+jwmJSkeCry0zijIoeK/DTm5adTkZ/G7Chb644UlbqIuG5wxM+ull5qWnzUtvioaQ7+75EnK1OS4pmXn8Y58/KoLEijsiBY3sVZ04mL8A6TaKZSF5GICQQs+zv7qW7uobrZR3WTj5oWH/s6+g6veSclxFGRl8bqOdnML8w4XOAq74lRqYtIWHQPjFDd1MPOpmCB72z2UdvsO7zbxBgozU5hfmE6ly8rYkFhOpUF6ZTlpMTksolTVOoiMimBgKXhUD87DvawI1TiO5t8NHYNHB4zIyWRBYUZXLtqNgsLM5hfmM68gjRSklRBTtOMisiEDY0G1763H+w+osR99A4F32EZZ6A8L40VpTO4fk0JC2dmsGhmBvke2Co4VajURWRMfUOj7GzqYVtjN9sO9rD9YA+7WnyMhha/U5PiWVSUwcdWFLO4KIOFMzOoLEgnOdG72wWnAkdL3RhzB3ATsBR43Fp7k5OPLyLh4RscYfvBYIG/29jNtsZu6tr7sKGTl7lpSSwqyuT8+XksLspkUVEGpdkpOnEZhZw+Uj8IfBv4CDDd4ccWEQf0D4+yrbGHrQe62NbYzdbGbuqPKPDCjGSWFGdyxbLgEfjSWZlaPplCHC11a+0TAMaYKmCWk48tIidueDRAdXMPWxq62HKgm60Hutjd2nt4++DMzGSWFmfy0eXFLJmVydLiTHLTprkbWibFlTV1Y8wtwC0AJSUlbkQQ8RxrLXs7+tnccIgtDd1sbuhix8Eehv3BD2bISU3ilFmZXLJkJqfMygwdgSe7nFqc5kqpW2sfBB4EqKqqsm5kEJnquvqH2dzQxeaGLt7Z38WWA1109Y8AMD0xnqWzMvnMmWWcMiuLZbMzKc6ariWUGKDdLyJTQCBg2dXay6b9h3h73yE27T9EXVsfEHwTT2V+OhcvLmT57CyWzc5iXn6a3sATo1TqIlGod2iUzfu7eHvfITbu62RzQxe+0KftZKcmsaIki6tXzOLUkixOmZVF2jT9pyxBTm9pTAg9ZjwQb4xJBkattaPH/3+KxLbm7kHe2tvJxr2dbNx3iJ1NPQRs8Ch8fkE6VywrYmXpDFaUzKA0J0XLKHJMTv/z/g3gm0f8+QbgW8Bah59HZMqy1lLf3seG+k421Hfy1r5OGjqDb6mfnhjPitIs7vjQPKpKZ7C8JIuMZGc/bV68zektjWtRgYt8QCBgqW318WZdJ2/Wd7ChvpP23mEg+KaeqtJsbjpjDqeVzWDhzAwStRYuk6CFOBGHBQKW6mYfb9R18EZdBxv2dh7elVKcNZ2z5+Wxak42q+ZkU56bqqUUcZRKXWSSrLXsaevl9T0dvL67gzfrOzgUKvGS7BQ+vKiA1XNyWF2ezawZKS6nFa9TqYuchMauAf60u53Xd7fz+p4OWkMfr1acNZ0LFhZwenkOa+bmUJylq2VIZKnURSage2CEP+/pYP3uNv60u4P69uAe8dy0aZwxN4czK3I4vTyXkhwdiYu7VOoiYxj1B9hyoItXa9t5bVcbmxu6CNjg52SuKc/hhjWlnFWRS2VBmtbEJaqo1EVCmroHWFfTxqu72nhtVzu+wVHiDJwyK4s7zq/grHl5LJ+dRVKCdqdI9FKpS8wa8QfYuPcQr9S08kpNGzUtPiB46dlLl8zknMo8zqzIISslyeWkIhOnUpeY0t47xMvVrbxc08prte34hkZJjDecVpbNP61cwLmV+VpSkSlNpS6eZq2lpsXHCztaeLG6lc0NXVgL+enTuHTpTM5fkM9Z83J17RTxDL2SxXNG/QE27O3kjztaeGFny+G34J8yK5MvXVDJBQvzWVyUoaNx8SSVunjCwLCfdbVt/GFHMy9Vt9LVP0JSQhxnV+Ry23kVXLAgn/wMfSCEeJ9KXaasnsERXtrZyrPbmlhX28bgSIDM6YlcsDCfDy8q4Ox5eaRqWUVijF7xMqV0D4zwxx0tPPtuE6/tamfYHyA/fRqfWDmbi5cUsmpOti6IJTFNpS5RzzcYLPJntjbx2q42RvyW4qzpfPr0Ui5ZWsips2cQF6f1cRFQqUuUGhj282J1C7/dcpCXa9oYHg1QnDWdz5w5h0uXzmTZrEyd6BQZg0pdosaoP8D63e08vfkgz29vpm/YT376NK5fXcLly4o4dXaWilxkHCp1cZW1lm2NPTz5TiNPbzlIe+8QGckJXL6siCuWFbG6PId4La2ITJhKXVzR3D3IU5sb+fXbB9jV2ktSfBwXLMznqlOLOW9+HtMS4t2OKDIlqdQlYoZG/bywo5Vfvt3Aq7VtBCysLJ3Bdz66hMuWFpGZos/iFJkslbqEXXVzDz9/q4Gn3mnkUP8IMzOTue28Cq5eOYs5ualuxxPxFJW6hEX/8CjPbGnisQ372dzQRWK84cOLC/nrqtmcVZGrdXKRMFGpi6Nqmn08+uY+ntzUiG9olIr8NO66bBEfPbWY7FRdwlYk3FTqMmnDowGe397MT9/Yx4b6TpIS4virpTP55OoSqkpnaBuiSASp1OWktfoGefzNBh59cx+tviFKslP42iUL+ETVbB2Vi7hEpS4nbFtjNw+vr+e3Ww8y4recW5nHvVeXcW5lnt6uL+IylbpMSCBgeWFnC/+zvp4N9Z2kJsVz/epSPn16KeV5aW7HE5EQlboc1+CIn19vOsBDr9VT195HcdZ0vn7pQv76tNlkTte+cpFoo1KXMXUPjPDIG/v43z/V0947zNLiTP7zulO5ZEkhCbq0rUjUUqnLB7T3DvHQ+noe+fM+fEOjnFOZxxfOLef08hztYhGZAlTqAkBLzyA/WlfHYxv2MTQa4NKlM7n13LksKc50O5qInACVeoxr6Rnk/lf28NiG/fgDlo+eWsyt581lrk5+ikxJKvUY1d47xH0v7+HRN/cxGrB8fMUsbj+/gpKcFLejicgkqNRjTPfACP/9ah0P/6mewRE/H1sxiy9+qILSHF1YS8QLHC11Y0w28BDwYaAd+Jq19jEnn0NOzuCIn//7817+6+U9dA+McNkpM/m7iyq1zCLiMU4fqf8XMAwUAMuB3xljtlhrtzv8PDJBgYDlyXca+bc/1tLYNcC5lXn8w8XzWVykE6AiXuRYqRtjUoGrgSXW2l5gvTHmaeBTwD869TwycW/WdXDP73awrbGHpcWZ/MvHT+GMily3Y4lIGDl5pF4J+K21tUfctwU49+iBxphbgFsASkpKHIwgAA2d/Xzndzt5bnszRZnJ/PCa5VyxrEjXZRGJAU6WehrQfdR93UD60QOttQ8CDwJUVVVZBzPEtIFhP/e/spsHXq0jIc7w5YsqufnscqYn6fM+RWKFk6XeC2QcdV8G4HPwOWQM1lqe397CPc/soLFrgCuWFfFPly6kMDPZ7WgiEmFOlnotkGCMmWet3RW6bxmgk6Rh1NDZz9qnt/NidSsLCtP5+S1rWF2e43YsEXGJY6Vure0zxjwB3G2MuZng7pcrgTOceg5536g/wEPr6/n3F2qJM4avX7qQm84sI1EX2xKJaU5vabwNeBhoBTqAW7Wd0XnbD3bz1V9vZVtjDxcuLODuKxdTlDXd7VgiEgUcLXVrbSdwlZOPKe8bGvXzny/u5v51e5iRksh916/gkiWFunqiiBymywRMEdsPdvPlX2yhutnH1StmcddlC8lK0eeAisgHqdSjnD9gue/l3fzHi7uYkZrEQzdWccHCArdjiUiUUqlHsYbOfu78xWbe2nuIy06ZyT1XLmFGqo7OReTYVOpR6jebG/nGk9uwwA+vWc5Vpxa7HUlEpgCVepQZHPGz9unt/OytBlaWzuCH1yxndraucS4iE6NSjyK7W3u547FNVDf7uO28udx5UaU+5FlETohKPUr8/t0m/v6XW0hOjOcnf7OKcyvz3I4kIlOQSt1l/oDlX56v4YF1ezi1JIv7r1+pa7aIyElTqbuou3+EOx7fxGu72vnk6hK+efkipiXoiooicvJU6i6pb+/jsz95i4bOfr73saVcu0rXlReRyVOpu+D1Pe3c+sgm4gw8evMaVs3JdjuSiHiESj3CnnznAF/55VbKclN5+MbTKMnRdkURcY5KPUKstfzo1Tq+92w1p5fn8KNPryQjOdHtWCLiMSr1CAgELHc/s4Mfv76Xy5cV8a+fOEUnREUkLFTqYTbqD/APv9rKE+808tmz5vD1SxfqA6BFJGxU6mE0PBrg//3sHZ7d1sxXPjKf28+vcDuSiHicSj1MBkf83PrI27xc08Zdly3is2fNcTuSiMQAlXoYDI36+fxP3+bVXW1892NLuU570EUkQlTqDhseDXD7o5tYV9vGvVcv5ZrTVOgiEjm6BKCDRvwB/vbxd3hhZyv3XLVEhS4iEadSd4i1lq/+eivPbW/mny9bxKfWlLodSURikErdIfc+V8MTmxq586JK/kYnRUXEJSp1Bzy8vp4H1u3hhjUlfPFD2rYoIu5RqU/SM1sPcvczO7h4cSHfumIJxuiNRSLiHpX6JGxp6OLLv9jCaWUz+OG1y4nXO0VFxGUq9ZPU3D3I5/5vI3np03jghpUkJ+paLiLiPpX6SRgY9nPLTzfSNzTK/9xYRU7aNLcjiYgAevPRCbPW8rUntvJuYzf//akqFhRmuB1JROQwHamfoMc27OepzQe588JKLlxU4HYcEZEPUKmfgG2N3Xzrtzs4pzJPV1wUkaikUp+gnsERbn9sE9kpSfzwmuW6JrqIRCWtqU9AcB39XQ4cGuDnt6whOzXJ7UgiImPSkfoEPL3lIL/b2sSdF1VSVZbtdhwRkWNypNSNMXcYYzYaY4aMMT924jGjRXP3IHc9tY2VpTP4wrlz3Y4jInJcTi2/HAS+DXwEmO7QY7rOWstXfrWFEb/lB59YpneMikjUc6TUrbVPABhjqoBZTjxmNHjkzf28tqude65cTFluqttxRETG5cqaujHmltByzca2tjY3IoyrqXuA7/1+J2fPy+UGXRtdRKYIV0rdWvugtbbKWluVl5fnRoRx3f3bHYwGLN+5aqmuvCgiU8a4pW6MecUYY49xWx+JkJH2ck0rz25r5osfqqAkJ8XtOCIiEzbumrq19rwI5IgagyN+vvmb7ZTnpfK5c8rdjiMickIcOVFqjEkIPVY8EG+MSQZGrbWjTjx+JN338m72d/bz2M2rmZagy+mKyNTi1Jr6N4AB4B+BG0Jff8Ohx46YA4f6eWBdHVctL+KMily344iInDCntjSuBdY68Vhu+rc/1GIMfPWSBW5HERE5KbpMQMiOgz08ubmRz5w5h5mZnnn/lIjEGJV6yL3PVZORnMituhSAiExhKnXg9d3trKtt4/bz55KZkuh2HBGRkxbzpW6t5d7nqinKTObTp5e5HUdEZFJivtRfqWljy4FuvnRhJcmJ2sIoIlNbzJf6/a/soSgzmY+uKHY7iojIpMV0qW/c28mGvZ187pxyEuNjeipExCNiuskeWLeHGSmJXHPabLejiIg4ImZLvabZxws7W7nxjDJSkvRRrSLiDTFb6j96dQ/TE+O5UTteRMRDYrLUD3YN8PTmg1y3qoQZqUluxxERcUxMlvrP32rAby2fObPM7SgiIo6KuVL3Byy/3NjA2fPymJ2tD8AQEW+JuVJ/tbaNg92DXKcdLyLiQTFX6o9t2E9uWhIXLCxwO4qIiONiqtRbewZ5qbqVj6+cTVJCTP3VRSRGxFSz/fLtA/gDlmu19CIiHhUzpR4IWH721n7OmJtDWW6q23FERMIiZkr9jboOGjoHuHZVidtRRETCJmZK/XfvNpGSFM+HF+kEqYh4V0yUuj9geX57C+fPz9c100XE02Ki1DftP0R77xAXLyl0O4qISFjFRKk/+24zSQlxnL8g3+0oIiJh5flSt9by/PZmzpmXS9o0XWJXRLzN86W+9UA3jV0DXLxkpttRRETCzvOl/uy2ZhLiDBfpsgAiEgM8XerWWp7b1sTpc3PITEl0O46ISNh5utRrWnzs7ejXrhcRiRmeLvWXqlsBuEhvOBKRGOHpUn+jrpPKgjTy05PdjiIiEhGeLfURf4CNezs5vTzH7SgiIhHj2VLfeqCb/mE/a1TqIhJDPFvqb9R1ALBqTrbLSUREImfSpW6MmWaMecgYs88Y4zPGvGOMucSJcJPxRl0H8wvSyUmb5nYUEZGIceJIPQFoAM4FMoG7gF8YY8oceOyTMjwaYOPeQ5w+V0svIhJbJn0xFGttH7D2iLueMcbUAyuBvZN9/JPxbmMXAyN+1pRr6UVEYovja+rGmAKgEth+nDG3GGM2GmM2trW1OR2BN+o6AVg1R0fqIhJbHC11Y0wi8CjwE2tt9bHGWWsftNZWWWur8vLynIwAwJ/3dLCgMJ3s1CTHH1tEJJqNW+rGmFeMMfYYt/VHjIsDfgoMA3eEMfNxDY8G2LivU1sZRSQmjbumbq09b7wxxhgDPAQUAJdaa0cmH+3kbD3QxeBIQKUuIjHJqU+NuB9YCFxorR1w6DFPypv1wfX01dqfLiIxyIl96qXA54HlQLMxpjd0u37S6U7CjqYeSrJTmKH1dBGJQU5sadwHGAeyOKK6qYf5heluxxARcYWnLhMwOOKnvr2PhSp1EYlRnir13a29BCzML8xwO4qIiCs8Veo7m3oAWDBTR+oiEps8Veo1zT6mJcRRlpPqdhQREVd4qtSrm31UFqQTHxc1521FRCLKc6W+QCdJRSSGeabU23uHaO8d0nZGEYlpnin1mmYfAAtnaueLiMQuz5T64Z0vOlIXkRjmmVKvafaRmzZNH18nIjHNM6Ve3exjofani0iM80Sp+wOW2hYf8wtU6iIS2zxR6ns7+hgaDbBAJ0lFJMZ5otSrm4I7X3SSVERinSdKvaa5h/g4Q0V+mttRRERc5YlS39vRT3HWdJIT492OIiLiKk+UenPPIIWZyW7HEBFxnSdKvaVnkIIMlbqIyJQvdWstLT2DFGboTUciIlO+1HsGRhkcCehIXUQED5R6c88ggEpdRAQPlHpLqNR1olRExAOl/t6ReqGO1EVEpn6pt3QHSz0vXSdKRUSmfqn7BpmRkqg3HomI4IFSb+4e0klSEZGQKV/qeuORiMj7PFHqOkkqIhI0pUt91B+gvXeIAm1nFBEBpnipt/UOEbBQoEsEiIgAU7zUW3qGAO1RFxF5z5Qu9eZuXSJARORIU7rUW3TdFxGRD3Ck1I0xjxhjmowxPcaYWmPMzU487nhaegZJjDfkpCZF4ulERKKeU0fq3wXKrLUZwBXAt40xKx167GNq7hkkPz2ZuDgT7qcSEZkSHCl1a+12a+3Qe38M3eY68djH09IzSL52voiIHObYmrox5j5jTD9QDTQBvz/O2FuMMRuNMRvb2tpO+jlbeoa080VE5AiOlbq19jYgHTgbeAIYOs7YB621Vdbaqry8vJN+zpZuXSJARORI45a6MeYVY4w9xm39kWOttX5r7XpgFnBruEID9A2N4hsaVamLiBwhYbwB1trzTvJxw7qmfvjDMTK1pi4i8p5JL78YY/KNMdcaY9KMMfHGmI8A1wEvTT7esWmPuojIXxr3SH0CLMGllgcI/iOxD/iStfY3Djz2ManURUT+0qRL3VrbBpzrQJYT0tyt676IiBxtyl4moKVnkPRpCaROc+KXDRERb5jSpa43HomIfNCUPcxdUpxJWW6q2zFERKLKlC3128+vcDuCiEjUmbLLLyIi8pdU6iIiHqJSFxHxEJW6iIiHqNRFRDxEpS4i4iEqdRERD1Gpi4h4iLHWuhvAmDaCV3acqFygPUxxJiNac0H0ZovWXBC92aI1F0RvNq/mKrXW/sVHx7le6ifKGLPRWlvldo6jRWsuiN5s0ZoLojdbtOaC6M0Wa7m0/CIi4iEqdRERD5mKpf6g2wGOIVpzQfRmi9ZcEL3ZojUXRG+2mMo15dbURUTk2KbikbqIiByDSl1ExENU6iIiHhJ1pW6MyTbGPGmM6TPG7DPGfPIY44wx5l5jTEfo9n1jjImSbGuNMSPGmN4jbuVhzHWHMWajMWbIGPPjccb+nTGm2RjTbYx52BgTtg96nWguY8xNxhj/UfN1XhhzTTPGPBT6GfqMMe8YYy45zvhIztmEs7kwb48YY5qMMT3GmFpjzM3HGRvJOZtQrkjP11HPPc8YM2iMeeQY33euz6y1UXUDHgd+DqQBZwHdwOIxxn0eqAFmAcXADuALUZJtLfBIBOfsY8BVwP3Aj48z7iNAC7AYmAG8AnwvCnLdBKyP4Hylhn5GZQQPbC4DfEBZFMzZiWSL9LwtBqaFvl4ANAMro2DOJporovN11HP/AXjtWL3gZJ9F1ZG6MSYVuBq4y1rba61dDzwNfGqM4TcCP7DWHrDWNgI/IPhDi4ZsEWWtfcJa+xTQMc7QG4GHrLXbrbWHgHsI45ydQK6Istb2WWvXWmv3WmsD1tpngHpg5RjDIz1nJ5ItokJzMPTeH0O3uWMMjfScTTSXK4wx1wJdwIvHGeZYn0VVqQOVgN9aW3vEfVsI/kt8tMWh7403zo1sAJcbYzqNMduNMbeGMdeJGGvOCowxOS7lOdKpxpj20K/PdxljIvah6MaYAoI/3+1jfNvVORsnG0R43owx9xlj+oFqoAn4/RjDIj5nE8wFkZ+vDOBu4MvjDHWsz6Kt1NMILmkcqRtIn8DYbiAtjOvqJ5LtF8BCIA/4HPDPxpjrwpTrRIw1ZzD23yGSXgWWAPkEfxu6DvhKJJ7YGJMIPAr8xFpbPcYQ1+ZsAtkiPm/W2tsI/t3PBp4AhsYYFvE5m2AuN15n9xD8raVhnHGO9Vm0lXovkHHUfRkE1xTHG5sB9NrQApWb2ay1O6y1B621fmvt68B/AB8PU64TMdacwdjzGzHW2jprbX1oueFdgkc2YZ8vY0wc8FNgGLjjGMNcmbOJZHNr3kKv6/UE13/H+i3UlTkbL1ek58sYsxy4EPj3CQx3rM+irdRrgQRjzLwj7lvG2L96bg99b7xxbmQ7mgXCujNngsaasxZrbVSteROB+QodAT0EFABXW2tHjjE04nN2AtmOFunXWQJjr127/To7Vq6jhXu+ziN4wnu/MaYZ+HvgamPMpjHGOtdnbpwJHucs8c8I7jJJBc7k2DtMvgDsJHimuCg0AeHe/TLRbFcSPOtvgFVAI3BjGHMlAMnAdwke3SUDCWOMu5jgzoBFoXwvEd5dCRPNdQlQEPp6AbAN+GaYf5YPAG8AaeOMi+icnWC2iM0bwSWLawkuE8QT3OHSB1zp5pydYK6Ivs6AFKDwiNu/Ar8C8sYY61ifhe2FOYmJyAaeCv1g9gOfDN1/NsFfR94bZ4DvA52h2/cJXcsmCrI9TnDHRy/BEzd/G+Zca3n/rP97t7VASShDyRFj7yS43awH+F9CW8HczBV6sbeE5rWO4K/FiWHMVRrKMhjK8d7t+iiYswlni+S8ETw/tI7gLo4e4F3gc6HvuTZnJ5Ir0q+zY/z38Ejo67D1mS7oJSLiIdG2pi4iIpOgUhcR8RCVuoiIh6jURUQ8RKUuIuIhKnUREQ9RqYuIeIhKXUTEQ/4/JB4qbDe3fLAAAAAASUVORK5CYII=\n", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEMCAYAAADAqxFbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAhJUlEQVR4nO3deXhc1X3/8fdXm2Xt1m7JlmVblndsbGGb3QRCgLIlKQkJWduEBkLbhCRNt/wCIWmTtL82bRogtNC0EEJCAoSQQhICBhwWY4xtvEjeZFuWtcvad83pHzN2JNeLxtbMneXzep55rLlz5fkea3w/uuece6455xARETkqwesCREQksigYRERkHAWDiIiMo2AQEZFxFAwiIjJOktcFTIb8/HxXXl7udRkiIlHlrbfeanXOFRy/PSaCoby8nI0bN3pdhohIVDGzAyfarq4kEREZR8EgIiLjKBhERGQcBYOIiIyjYBARkXEiLhjMLNfMnjSzXjM7YGYf9romEZF4EonTVb8HDAFFwHLgl2a2xTm33dOqRETiREQFg5mlA+8HljjneoD1ZvY08FHgLz0tTkTEYz6fo7l7kP1tvRxo62V/Wx+3r51LZmrypL5PRAUDUAmMOud2jdm2Bbj0+B3N7FbgVoCysrLwVCciEmI+n6Ope4Da1l4OtPWxv7X32NcH2nsZGPYd2zc50bhheQkLimM7GDKAzuO2dQKZx+/onHsAeACgqqpKdxsSkajhnKO9d4ja1l72tfYeO/gfDYD+4dFj+6YkJlCWl0Z5XjoXz8tnVn46s/PSmZWXRknOVBITbNLri7Rg6AGyjtuWBXR7UIuIyFnpGxrxH/xbfn/g39faS21LD10DI8f2S0owynLTKM9P58KKfMoDB//y/DSmZ4fm4H8qkRYMu4AkM5vnnNsd2LYM0MCziEQkn8/R0DXAvpYe9jb3sK+1l70tPexr6aWhc2DcviXZqcwuSOeG5aXMzk8/9pgxbSpJiZEzSTSigsE512tmTwBfM7NP4Z+VdANwgaeFiUjcGxgeZX9bL3ube9nT3MPelp5jATC26ydzShJzCtJZMyePOfnpzCnIOBYAU1MSPWzBxEVUMATcDjwENANtwG2aqioi4dI9MMye5h52N/vPAPY097CnpYe69j58gdFMMyjNmcrcggxWzc5lbkFG4JFOQeYUzMLb9TPZIi4YnHPtwI1e1yEisa2zb5hdzd3sbuphd3O3Pwyaemjs+n33T0piArPz01lSks0Ny0upKPQf/OfkZ0TNb/9nIuKCQURkMnUNDLO7qZtdTT3UNHazu9n/dUv34LF90lISqSjM4IKKPCoKM5hXmElFYQYzI6zvP1wUDCISEwaGR9nd1ENNUze7mrqpafT/OXYAOC0lkXmFGVwyr4DKogwqi/wBUJozlYQwz/yJZAoGEYkqPp/jYHsf1Y1dVDd2U93QTU1TNwfaeo+NAaQkJVBRkMHq2bnML846FgIKgIlRMIhIxOrsH6a6oYudDf4Q2NnYza7G7mOzgMxgVm4a84szuW5ZCQuKM6ksyqQ8Ly0uu4Ami4JBRDzn8znqjvSx43AXOwJBsLOhm/qO/mP7TEtLZkFxFjevmsnC4izmF2cyryiDtBQdxiab/kVFJKwGR/xjAdsPd44Jgm56Bv1XAicYzCnIYMWsadyypoyF07NYND2LwhiYBhotFAwiEjK9gyPsbOhiW30n2w53sf1wF7ubuhkJDAakpySyqCSL960oZXFJFgunZ1FZlElqcuxOBY0GCgYRmRTdA8NsP+wPgXfqO9lW38m+1l5cYEA4PyOFRSXZXDa/gMUl2SwqyWJWbpoGgyOQgkFEgtY3NMK2+i62HupgW30nW+s7qR0TAsVZqSwpzeb6Zf4zgaUzstUVFEUUDCJySkMjPqobu9hS18GWQ51sPdTBnuaeY1NDp2ensrQ0m/cuL2XJjGyWlmaTnzHF26LlrCgYROQY5xz72/rYXHeELXWdbK7rYMfhLoZG/TeHyUtP4ZwZ2Vy9ZDrnzMgOnAmkely1TDYFg0gc6+gbYnNdB5vrOnj7YAdbDnXQ0TcMwNTkRJbOyOaTF5Zzzowcls3MpjRnqrqD4oCCQSRO+HyO3c09bDp4hLcOHGHTwSPsa+kF/BeKVRZmctXiYpbPzGHZzBzmFWboIrE4pWAQiVE9gyNsPtjBWweOsPFAO5vrOugO3DUsNz2FFWU5vH/FDM4ty+GcGTlkTNHhQPz0SRCJEY2dA7y5v52N+9vZeOAIOxu68Dn/2cD8okyuX1bCylnTWFE2jVl5aeoSkpNSMIhEIeccta29bKhtZ0NtO28eaKeu3b98xNTkRFbMyuGOd82jatY0lpflkJWa7HHFEk0UDCJRwOdz7Gru5o197bxR28aG2nZae4YA/4VjVbNy+cQFszmvfBoLp2eRrLEBOQsKBpEI5PM5qhu7eX1fG6/va2PD/vZjs4VKc6Zy8bwCVs3OZdXsXObkp6tbSCaVgkEkAjjn2NvSw6t723h1Txtv1LZxJBAEZblpXLmoiNWz81g9J5cZ09I8rlZinYJBxCP1Hf38bk8rr+5p5dW9bTQHbjVZmjOVyxcWcf6cPNbMzaM0Z6rHlUq8UTCIhEln/zCv7W1j/Z4WfrenjdpW/zUE+RlTuGBuHhdW5HH+nHzK8nRGIN5SMIiEyMiojy2HOnh5Vyuv7G5hc10HPue/7/CaOXl8ZM0sLqrIp7IoQ2MEElEUDCKTqKGzn5dqWnh5dwuv7G6le2CEBINzZuRwx2UVXDSvgOUzc0hJ0qwhiVwKBpGzMDzqY+P+I6yraWZdTQs1Td2Af9npa5ZM55LKAi6syCMnLcXjSkUmTsEgEqTWnkFerG7mxZpmXtnVSvfgCMmJxnnlufz1ygVcWlmo7iGJagoGkdNwzlHT1M3zO5r4bXUzm+s6cA4KM6dwzdLpXLagkIvm5WutIYkZ+iSLnMDIqI8N+9v5zY4mnt/ZdGy5iXNmZPO5yyu5fGEhi0uydFYgMUnBIBLQPzTKS7ta+PWORl6obqajb5iUpAQursjn9rUVXL6gkMIs3ZRGYp+CQeJa18AwL+xs5tltDby0q4WBYR/ZU5O5fGEhVy4q4uJ5BaSri0jijD7xEnc6+4f5zY4mnn2ngVd2tzI06qMwcwo3rZzJVUuKWTU7V4vQSVxTMEhc6B7wh8EzWxt4ZXcLw6OO0pypfOz8WVy9tJhzZ04jIUHjBSKgYJAY1j80ym+rm/jFlsO8WNPC0IiP0pypfPLC2VyzdDrLZmRr8FjkBBQMElNGRn2s39PK05sP86vtjfQOjVKYOYVbVpdx3bISzp2ZozAQOY2ICQYzuwP4BLAU+JFz7hOeFiRRwznHtvounny7nqe3HKa1Z5Cs1CSuW1bC9ctKWD0nj0R1E4lMWMQEA3AY+DrwHkDrDMtpNXYO8NTmen721iF2N/eQkpjA5QsLufHcUtbOL2BKUqLXJYpEpYgJBufcEwBmVgXM8LgciVCDI6M8v6OZx9+q4+VdLfgcrJw1jW+8dwnXLi0hO033NhY5WxETDMEys1uBWwHKyso8rkZCrbqxix+/WcdTb9dzpG+Y6dmp3L62gvevnMHs/HSvyxOJKVEbDM65B4AHAKqqqpzH5UgI9A2N8MyWBh7dcJDNdR0kJxpXLi7mA1UzuagiX+MGIiESlmAws3XApSd5+XfOuYvCUYdEh5rGbn74xgGe3FRP9+AIFYUZfOXaRbz33FJy07V8tUiohSUYnHNrw/E+Er2GRnz8ansjD79+gA217aQkJfAHS6fz4dVlVM2apimmImEUMV1JZpaEv55EINHMUoER59yIt5VJKDV3D/CjN+r44RsHaO4epCw3jb+6egE3Vc3U2YGIRyImGIC/Bb465vlHgLuBuzypRkJqW30nD62v5RdbDzM86ri0soBvvb+cSysLtDSFiMciJhicc3ehEIhpPp/j+Z1N/Mf6WjbUtpOeksgtq2fxsfNnMacgw+vyRCQgYoJBYtfA8Cg/23SIB1+pZV9rL6U5U/mbaxbygfNmkj1V1x2IRBoFg4RMZ/8wj7x+gP/8XS2tPUMsLc3mux86l6uXFJOkZa1FIpaCQSZda88gD66v5ZHXDtA9OMIllQV85tI5nD8nT7OLRKKAgkEmTVPXAN9/aR+PbjjA4IiPa5ZO57ZL57KkNNvr0kQkCAoGOWtNXQPct24vj244yKjP8d5zS7lt7VzmakBZJCopGOSMtfYMcu+Le/nhGwcY8Tn+cMUMPntZBWV5aV6XJiJnQcEgQevsH+bfX97HQ7+rZWB4lPetmMGfvquCWXlazE4kFigYZMIGhkf579f2870X99LZP8y150zn8++uVJeRSIxRMMhp+XyOJ9+u559+s4v6jn4urSzgL66az+ISDSqLxCIFg5zSG/vauOeXO9hW38XS0mz+4Q/P4YKKfK/LEpEQUjDICdW19/GNX+7kue2NlGSn8p0PLuf6ZSVax0gkDigYZJz+oVHuW7eH+1/eR1KC8YV3V/Kpi+cwNUX3TxaJFwoGAcA5x6+2N3HPMzuo7+jn+mUl/PU1CynOTvW6NBEJMwWDUNfex11Pb+e31c0sKM7kx7euYfWcPK/LEhGPKBji2MiojwfX1/LPz+8iwYy/uWYhn7iwnGQtcCcS1xQMcWr74U6+/LOtbKvv4oqFRXzthsWU5Ez1uiwRiQAKhjgzODLKd3+7h/te2su0tGTuvWUFVy8p1qqnInKMgiGObD/cyRd+soXqxm7ev2IGX7l2ITlpuq+yiIynYIgDoz7HvS/u4V9+u5tp6Sk8+PEqLl9Y5HVZIhKhFAwxrq69jzt/spk39x/h2nOmc88NS5iWrrMEETk5BUMM+/nmev72yW044DsfXM6N55Z6XZKIRAEFQwwaGB7lrqe389ibdaycNY3vfHA5M3N1jwQRmRgFQ4zZ09zDHY9uorqxm9vXzuXOd1eSpOsSRCQICoYY8j/vNPDFx7eQmpzIf/3RKi6tLPC6JBGJQgqGGDDqc/zDr2q4/6W9nFuWw323rNQaRyJyxhQMUa6zb5g7frSJV3a38uHVZXz1ukVMSdJKqCJy5hQMUay2tZc//q83qWvv45vvW8rNq8q8LklEYoCCIUq9ureV2x7ZRILBDz+1hlWzc70uSURihIIhCj359iG+9PhWyvPTeejj51GWp6moIjJ5FAxRxDnH91/exzefreb8OXl8/2MryUpN9rosEYkxCoYo4fM5vvbMDn7w6n6uW1bCP950jgaZRSQkFAxRYGTUx1/8dCtPvF3PH180m7+5ZiEJCVomW0RCQ8EQ4YZGfPz5Y2/z7LZGvvSe+Xz2sgqvSxKRGBcRayWY2RQze9DMDphZt5m9bWZXe12X1waGR/mThzfy7LZGvnLtIoWCiIRFpJwxJAF1wKXAQeAa4CdmttQ5t9/LwrwyODLKnzz8Fi/vbuHv37eUD+kaBREJk4gIBudcL3DXmE3PmFktsBLY70VNXhoa8fHZH27ipV0tfOv9S/ngeQoFEQmfCQWDmRUBVwLLgBygA9gC/MY51zjZRQXerxLYfop9bgVuBSgri50D5/Cojz/70ds8v7OZe25colAQkbA75RiDmS00s58CO4CPAslAY+DPjwLbzeynZrZosgoys2Tgh8B/OeeqT7afc+4B51yVc66qoCA2VhF1zvHln23lue2N/L9rF/HRNbO8LklE4tDpzhh+APwDcItzbvD4F80sBbgBeBA4/2R/iZmtwz9+cCK/c85dFNgvAXgYGALuOE1tMedbz9XwxKZ67nx3JX900WyvyxGROHXKYHDOrT7N60PA44HHqfZbe7pCzMzwB0wRcI1zbvh03xNLHlpfy/0v7eUja8r403dp9pGIeGfC01XNrPIk2y+cpFruAxYC1znn+ifp74wKz2w9zNee2cFVi4u5+/ol+DNSRMQbwVzH8IaZ3Xb0iZklm9m3gCfOtggzmwX8CbAcaDSznsDjlrP9uyPdlroOvvCTLZxXPo3v3LycRF3RLCIeC2a66lrgv83sWuCfAo/D+A/mZ8U5dwCIuyNiY+cAn/7vjRRkTuH+j6wkNVlrH4mI9yZ8xuCc2wKsBmYCvwbedM5d7ZxrCFVxsax/aJRbH95I7+AI//HxKvIypnhdkogIENwYQynwDP4ZQ38O3GBmf2dmEXGRXDRxzvFXT2zlnfpO/uXmc1lQnOV1SSIixwQzxrAZeA1Y45z7N/xdSFXAxskvK7Y9uuEgT20+zJ1XVHLFoiKvyxERGSeY3/avd869dvSJc64euNLM/mzyy4pd2+o7ufsXO7ikskCL4olIRApmjOG1k2z/18krJ7Z1DQzz2Uc3kZuWwnc+uFz3VBCRiHS6JTGeMLPzTrPPeWZ21lNWY51/XOEdDh3p598+fC656SlelyQickKn60q6H7jXzLKAl4AaoBvIxL/I3Vr8C+r9behKjA1PbznML7c28KX3zKeqPNfrckRETup0S2L8Gvi1mVUBV+OfrpoDHAG2Ajc7594OdZHRrrFzgK88tY2Vs6bxmUvnel2OiMgpTWjw2Tm3Ec0+OiPOOb700y0Mjzr+/03LdGWziES8Cc9KMrM5J3lpEGhwzvkmp6TY8sgbB3lldyv33LCY8vx0r8sRETmtYKar7gEcv1+6wo15zWdmTwO3O+eaJqu4aNfQ2c83/2cnF8/L5yO6t4KIRIlgLnD7NP4b6MwDUoH5wCPA7cBS/CHzvckuMJp97Rc7GPE5vnHjUq2YKiJRI5gzhruBCufcQOD5nsBqq7ucc983s08Auye7wGj1Yk0zz25r5ItXVlKWl+Z1OSIiExbMGUMCUH7ctjLg6JKgPQQXNDFrYHiUr/58O3MK0vn0JScbmhERiUzBHMi/A7xgZv8J1AEzgE8GtgP8Af61lOLevS/u4WB7H49+ajVTkrSUtohElwkHg3Pu22a2FbgJWAE0AH/snHsu8PpTwFMhqDGqHDrSx/0v7ePG5SVcUJHvdTkiIkELqusnEALPhaiWmPBPv96FGXz56gVelyIickaCuR9DspndbWb7zGwg8OfdZqZFfwJ2HO7iyc31fPLC2UzPnup1OSIiZySYM4ZvA6uAzwAHgFnAV4As4POTX1r0+dZz1WSlJnOblr0QkSgWTDDcBCxzzrUFnteY2SZgCwoGXt3Tyku7WvjraxaQnZbsdTkiImcsmOmqJ7tCK+6v3HLO8a3nqinJTuVj55d7XY6IyFkJJhgeB35hZu8xs4VmdhX+WUiPh6SyKLKupoUthzr53BWVpCZreqqIRLdgupL+Av99F74HlAD1wGPAPSGoK6rct24vJdmpvHdFqdeliIictVMGg5m967hN6wIP4/eL6F0EvDDZhUWLjfvb2bC/na9et4jkxGBOwEREItPpzhgePMn2o6FwNCDidt2H+1/ay7S0ZD543kyvSxERmRSnu4Pb7HAVEo1qGrt5fmczn7tiHmkpWiZKRGKD+j7Owvdf3svU5EQ+rplIIhJDFAxn6HBHP09vPsyHVpUxLV0Xf4tI7FAwnKEfv1nHqHN88sJyr0sREZlUCoYzMOpzPL6xjovnFTAzVzfhEZHYomA4Ay/vauFw5wAf0kwkEYlBCoYz8OiGg+RnpHD5wiKvSxERmXQKhiA1dw3wQnUzf7hyJilJ+ucTkdgTMUc2M3vEzBrMrMvMdpnZp7yu6UQef+sQoz7HzepGEpEYFTHBAPw9UO6cywKuB75uZis9rmkcn8/x2JsHuWBuHuX56V6XIyISEhETDM657c65waNPA4+IuuPN6/vaqGvv5+ZVZV6XIiISMhETDABmdq+Z9QHVQAPwP6fY91Yz22hmG1taWsJS3y/faSAtJZErF2nQWURiV0QFg3PudiATuBh4Ahg8xb4POOeqnHNVBQUFIa9t1Of41fYmLptfqHsuiEhMC0swmNk6M3Mneawfu69zbtQ5tx6YAdwWjvomYtPBI7T2DHLVkmKvSxERCamwLAnqnFt7Bt+WRASNMTz7TiMpSQlctqDQ61JEREIqIrqSzKzQzG42swwzSzSz9wAfIkJuAOSc41fbG7lkXj4ZU7S8tojEtogIBvwzkG4DDgFHgH8EPuec+7mnVQVsPdRJfUc/Vy2Z7nUpIiIhFxG//jrnWoBLva7jZJ7d1khSgvFuLYEhInEgUs4YIpZzjue2NXD+3Dyy05K9LkdEJOQUDKdR09TN/rY+zUYSkbihYDiNF6qbAXi3LmoTkTihYDiN1/e1U1mUQWFmqteliIiEhYLhFIZHfWzc3875c/K8LkVEJGwUDKew9VAnfUOjrFEwiEgcUTCcwuv72gBYNTvX40pERMJHwXAKr+9rY35RJnkZU7wuRUQkbBQMJzE04mPj/iOcP1fdSCISXxQMJ/FOfQf9w6OsmaNuJBGJLwqGk3h9XzsAq2brjEFE4ouC4SRe29vGguJMctNTvC5FRCSsFAwnMDTiY+OBdk1TFZG4pGA4ga2HOhgY9ikYRCQuKRhO4I1a//jCal2/ICJxSMFwAjsauijLTWOaxhdEJA4pGE6guqGL+cWZXpchIuIJBcNxBoZHqW3tZaGCQUTilILhOHuae/A5mF+c5XUpIiKeUDAcZ2dDFwALpuuMQUTik4LhODWN3UxJSqA8L93rUkREPKFgOE51YzeVRZkkJpjXpYiIeELBcJzqxm4WaOBZROKYgmGM1p5BWnsGNVVVROKagmGMmsZuABZO14wkEYlfCoYxjs1I0hmDiMQxBcMYNY3d5GdM0a08RSSuKRjGqG7sZqGuXxCROKdgCBj1OXY1dTO/SMEgIvFNwRCwv62XwREfCzTwLCJxTsEQUN3gn5GkgWcRiXcKhoCaxi4SE4yKwgyvSxER8ZSCIWB/Wx+lOVNJTU70uhQREU9FXDCY2TwzGzCzR8L5vo1dAxRnp4bzLUVEIlLEBQPwPeDNcL9pU9cARVkKBhGRiAoGM7sZ6AB+G873dc7R1DVAcZYubBMRiZhgMLMs4GvAFya4/61mttHMNra0tJzVe3f1jzAw7NMZg4gIERQMwD3Ag865uons7Jx7wDlX5ZyrKigoOKs3buwaAFAwiIgQpmAws3Vm5k7yWG9my4ErgH8ORz3HawoEgwafRUQgKRxv4pxbe6rXzexzQDlw0MwAMoBEM1vknFsR6vqOnjEU64xBRCQ8wTABDwCPjXn+RfxBcVs43ryp0x8MBZkafBYRiYhgcM71AX1Hn5tZDzDgnDu7UeUJauoeYFpasi5uExEhQoLheM65u8L5fo2dgxp4FhEJiKRZSZ7RxW0iIr+nYIDAxW0KBhERUDAwMuqjtWeQIk1VFREBFAy09Azic1Ck5TBERAAFA01dg4CuYRAROSrug6GxU8thiIiMFffB0KR1kkRExlEwdA2QnGjkpad4XYqISESI+2Bo7BqgMDOVhATzuhQRkYgQ98HQ1DVAoWYkiYgco2DoGtSMJBGRMRQMnVoOQ0RkrLgOht7BEboHRxQMIiJjxHUwHLtBT7bGGEREjorrYNA1DCIi/5eCAQWDiMhYcR0MjZ1aJ0lE5HhxHQxNXQNkTkkifUpE3shORMQTcR8MurhNRGS8uP5VeUlpNuX56V6XISISUeI6GD57WYXXJYiIRJy47koSEZH/S8EgIiLjKBhERGQcBYOIiIyjYBARkXEUDCIiMo6CQURExlEwiIjIOOac87qGs2ZmLcCBIL4lH2gNUTmRKh7bDPHZ7nhsM8Rnu8+2zbOccwXHb4yJYAiWmW10zlV5XUc4xWObIT7bHY9thvhsd6jarK4kEREZR8EgIiLjxGswPOB1AR6IxzZDfLY7HtsM8dnukLQ5LscYRETk5OL1jEFERE5CwSAiIuMoGEREZJyYDAYzyzWzJ82s18wOmNmHT7Hv582s0cw6zewhM4vam0BPtN1m9nEze8vMuszskJl928yi8m5+wfysx3zPC2bmorXNEPRnfI6ZPWNm3WbWambfDmetkyWIz7eZ2dfNrD7w/3qdmS0Od72TwczuMLONZjZoZj84zb6TdiyLyWAAvgcMAUXALcB9J/pgmNl7gL8ELgfKgTnA3eErc9JNqN1AGvA5/FdNrsbf/i+GqcbJNtE2A2BmtxAbt7Sd6Gc8BfgN8AJQDMwAHgljnZNpoj/rm4A/Ai4GcoHXgIfDVeQkOwx8HXjoVDtN+rHMORdTDyAd/4encsy2h4FvnmDfR4G/G/P8cqDR6zaEut0n+N47gV943YZQtxnIBnYBawAHJHndhlC3G7gVeMXrmsPc5i8DPxnzfDEw4HUbzrL9Xwd+cIrXJ/VYFotnDJXAqHNu15htW/B/OI63OPDa2P2KzCwvhPWFSjDtPt4lwPaQVBVawbb574D7gMZQFxZiwbR7DbDfzJ4NdCOtM7OlYalycgXT5seACjOrNLNk4OPAc2Go0UuTeiyLxWDIADqP29YJZE5g36Nfn2jfSBdMu48xs08CVcA/hqiuUJpwm82sCrgQ+G4Y6gq1YH7WM4CbgX8FSoBfAj8PdDFFk2Da3AC8AtQA/fi7lj4f0uq8N6nHslgMhh4g67htWUD3BPY9+vWJ9o10wbQbADO7EfgmcLVzLhpXpZxQm80sAbgX+HPn3EiYagulYH7W/cB659yzzrkh/L8A5AELQ1vipAumzV8FzgNmAqn4+9pfMLO0kFborUk9lsViMOwCksxs3phtyzhxV8n2wGtj92tyzrWFsL5QCabdmNlVwL8D1znn3glDfaEw0TZn4T8r+rGZNQJvBrYfMrOLQ1/mpAvmZ70V/3hKtAumzcuAHzvnDjnnRpxzPwCmAYtCX6ZnJvdY5vWgSogGah4DfoR/wOpC/KdVi0+w31X4+5sX4f/gvMAEBmsj9RFEu98FtAGXeF1zONoMGP4ZOUcf5+E/WJYCKV63IcQ/6/lAH3AFkIi/S2VvNLY7iDZ/FViPf/ZSAvBRoBfI8boNZ9DmJPxnPX+Pf7A9lRNMmpjsY5nnDQ/RP2Yu8FTgw3AQ+HBgexn+U66yMfveCTQBXcB/AlO8rj/U7QZeBEYC244+nvW6/lD/rMd8TzlRPCsp2HYD7wP2BD7j6050MI2GRxCf71T8U1sbAm3eBFzldf1n2Oa7Ap/VsY+7Qn0s0yJ6IiIyTiyOMYiIyFlQMIiIyDgKBhERGUfBICIi4ygYRERkHAWDiIiMo2AQEZFxFAwiIjKOgkFERMZRMIhMMjOba2btZrYi8LwkcC+Etd5WJjIxWhJDJATM7NP4165ZCTwJvOOci9bbp0qcUTCIhIiZPQ3Mxr/w2XnOuUGPSxKZEHUliYTOvwNLgO8qFCSa6IxBJATMLAP/fXdfBK4Gljrn2r2tSmRiFAwiIWBmDwKZzrkPmNkD+G8S8wGv6xKZCHUliUwyM7sB/x21PhPYdCewwsxu8a4qkYnTGYOIiIyjMwYRERlHwSAiIuMoGEREZBwFg4iIjKNgEBGRcRQMIiIyjoJBRETGUTCIiMg4/wsk29CRKJuN7wAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] @@ -1259,7 +1274,14 @@ } ], "source": [ - "plot_function(torch.log, min=0,max=4)" + "plot_function(torch.log, min=0,max=1, ty='log(x)', tx='x')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Additionally, we want to ensure our model is able to detect differences between small numbers. For example, consider the probabilities of .01 and .001. Indeed, those numbers are very close together—but in another sense, 0.01 is 10 times more confident than 0.001. By taking the log of our probabilities, we prevent these important differences from being ignored." ] }, { @@ -1279,7 +1301,31 @@ "\n", " log(a*b) = log(a)+log(b)\n", "\n", - "When we see it in that format, it looks a bit boring; but think about what this really means. It means that logarithms increase linearly when the underlying signal increases exponentially or multiplicatively. This is used, for instance, in the Richter scale of earthquake severity, and the dB scale of noise levels. It's also often used on financial charts, where we want to show compound growth rates more clearly. Computer scientists love using logarithms, because it means that multiplication, which can create really really large and really really small numbers, can be replaced by addition, which is much less likely to result in scales that are difficult for our computers to handle." + "When we see it in that format, it looks a bit boring; but think about what this really means. It means that logarithms increase linearly when the underlying signal increases exponentially or multiplicatively. This is used, for instance, in the Richter scale of earthquake severity, and the dB scale of noise levels. It's also often used on financial charts, where we want to show compound growth rates more clearly. Computer scientists love using logarithms, because it means that multiplication, which can create really really large and really really small numbers, can be replaced by addition, which is much less likely to result in scales that are difficult for our computers to handle.\n", + "\n", + "Observe that the log of a number approaches negative infinity as the number approaches zero. In our case, since the result relfects the predicted probability of the correct label, we want our loss function to return a small value when the prediction is \"good\" (closer to 1) and a large value when the prediction is \"bad\" (closer to 0). We can achieve this by taking the negative of the log:" + ] + }, + { + "cell_type": "code", + "execution_count": 132, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEdCAYAAAAPT9w1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAApwUlEQVR4nO3deXxcZb3H8c8ve7M2e9Kmabq3dKF0oywtmwioXBQFURBRERB3cLl471VUrle9XNQriBfBi7LrFWTfN6FsbYFutE33Nm2TJl2ytUmzPPePM6lDTNtJm5mTmfN9v17zaubMmXl+TzL9nmeec+Ycc84hIiKJL8nvAkREJDYU+CIiAaHAFxEJCAW+iEhAKPBFRAJCgS8iEhAKfEk4ZnanmT3ndx1+O5Lfg5ltNLN/HYC2B+R1ZGAp8Acxv4LLzK43s7WxbjdRmNlaM7ve7zoSnZlNNrM/m9kaM+s2s9v9rmmwU+CL+MTM0vyuIc5lApuBHwFLfK4lLijw45iZTTCzx82sJXR71MzG9lrnU2a2zszazOw1M/uImTkzO/ko2i03s/vNbI+Z7TOzl8xsVtjjqWZ2k5nVmFm7mW03s/vDHp9sZk+Hnt9qZivN7DOHaG+LmV0edv8PoT6MDVu2ycyu7vW8K0LLm8zsYTMr7vX4mWa2INSHrWb2v2ZWGPb4nWb23OFep9drvgSMAX4QqtGZWZWZnRr6+cNm9qqZtQFXmNllZtbZ6zUqQuueGrZsrJn9JfQ7221mz5jZ1IPVcZDaZpjZk2a2I/R+WWhmZ/ex6hAzuz3U3wYz+5mZJYW9TkroU+CG0PtqhZld2Z9aBoJzbqFz7lrn3F1AY6zbj0cK/DhlZkOAZ4AM4JTQLRt4qmfkaGYzgXuA+4BjgZ8DvzzKdg34KzAR+AgwB6gDnjWzotBqXwUuBC4BxgH/BLwR9jL3ATuBE4GpwDXA7kM0+yJwRtj904D6nmVmNgaoBF4IW2d2aL0PA2cD04Ebw/pxOvAwcD8wDfgoUAU8FOpjRK/Th/OBjcB/AeWh25awx/8L7+8wCe/3eFhmVgq8CuwA5gFzgdXAS4fa+PQhF6+/pwIzgKeBR8xsfK/1vgpsw+v7N4GvAN8Ie/x2vH5eGerHj4CfmdkX+lELYQOVg97683oSAeecboP0BtwJPHeQx74A7AWKwpaVAvuAS0P37wFe6fW8qwAHnHyIdq8H1h7ksTNCzz8mbFk6sB34fuj+r/DC1w7yGo3AZf34PVwG1IV+Hhfq978Bfwot+yKwrdfvrR5ID1v2z8D2sPsvAT/t1U5lqG/TI32dg9S7Fri+17JTQ6/9mT761tlrWUVo3VPD/h5v9FrHgHXAN47k/RO2zhLgX8Lub+zjPfMToCb08yigG5jYa53vA+/2ep1/PUzbYw9368d75CXg9iP5fxakWwoSryYD7znnGnoWOOfqzGx16DGAY4DeO31fH4B2dzrn3gtrt93M3gxr93+BZ4G1ZvZs6OdHnXP7Q4/fCNxuZpfh/Ud9xDn39iHafB4oMbMpwEl4o92ngK+FRuOn8/7RPcBK51x72P2teBvEHrOBuWb2lT7aGwe8G+Hr9NdbR/Cc2cDMPka8Q/BqjUjo08AP8X5fZUAK3ifEkb1W7f0eWQBcZ2a5wCy8jc2i938QIgXoirQWAOecDgyIMQV+fOvrVKfWa3k0Tod6yHadc++a2SjgTLzpkF8BPzazuc65Jufcj83sHrwpktOB75nZz51zfR7G55zbYmbr8D5dnIgX7ovx3r/TQm18r9fT9ve670I19kgCfgbc1UeTtf14nf5q7XW/u491UnvdT8Lb6PW1cerP3PWdeJ9ivgNswPs0eD9wuJ3HvX9v4P0d9vZar1/vtUimbJxz2f15TTk0BX78WgFcZWZFPaP80FzveP4+x/wecEKv580dgHaLzOyYnlG+maXjzeX/pmcl51wL8BDenPhP8KZ8TgEeDT2+PrT+b8zsn4FvA4c6bvsFvMA/HrjROddtZn/Dm28u5R9H+IezCJgcpVHmfiA5wnV3AMlmVuqcqwstm9FrnUV4Uz9bnXP7jqKu+cB3nHOPAJhZFjAaWN5rvd7vkRPwpsyazGxxaFmlc+6xo6gFvP0hEkMK/MEv28ym91rWBtyLN2/6gJl9G28UdiPelMMDofVuAhaa2Y+Au/F2tF4beuxwo7G0PtrtxgvWt4B7zezLeCPMf8ObGrgVIFTPNrxpkb3Ap/A+7lebWTbeyPoveKPMoXgj/fc4tBfwRuOtwNthy24ENjjnNh7m+b19H3jGzH4B/AFoxpseuQD4ylEG6wbgJDOrxOv/rkOs+1ao7Z+GNoxjQrWFuxlvn81fzewGvJ3AFcA5wOPOudcirGs1cLGZvYq3QfoRfW+Yppv3PYJ78aZwvo63HwHn3Foz+z3wOzP7Dt70TxYwEyh2zv0swlqOekondHDCMaG72UBB6D27P3zKUcL4vRNBt4Pf8D6Cuz5uq0KPTwCeAFpCt8fotaMLL2zXAe14/zkvDL3GzEO0e/1B2m0LPV6ONxWwB29a4GVgVtjzr8SbcmkK1bUQOC/0WAZekGzA23DtwNtAjTjM76IEb4PzcNiyqaG6bu/j9/Zcr2WXeG/39y2bh7ePoxlvQ7IS7yimlP68Th+1zgr1f1+ovir+vtO2oo/1Pxxqex/efPlZhO20Da0zEm8nfH3ob7kJbyM+6jDvn+fC7k8FXgu1sxG4OtT/O8PW2Qj8O95+mCa8jdV/Aslh6yTjTQutwvs00xB6D1zQ63UOudN2AP5/VB3kfbrR7/+7g/VmoV+cBISZXYr3n7nQObfH53JEJIY0pZPgzOxbeMex78I72uNnwJ8V9iLBo8BPfNPw5u0L8OZ+7wZ+4GtFIuILTemIiASETq0gIhIQg3pKp6ioyFVVVfldhohIXFm8eHGDc+4fzrM0qAO/qqqKRYsW+V2GiEhcMbNNfS3XlI6ISEAo8EVEAkKBLyISEAp8EZGAUOCLiASEAl9EJCAU+CIiAZGQgf/QOzXc/Uafh6GKiARWQgb+E8tquet1Bb6ISLiEDPyy3Azqmtv8LkNEZFBJyMAvzU1nz94O2jq6/C5FRGTQSNDAzwCgrkmjfBGRHgkZ+GV5PYHf7nMlIiKDR0IGfs8Iv1YjfBGRAxI68OsaFfgiIj0SMvBzM1IYkpqsOXwRkTAJGfhmRmluuqZ0RETCJGTggzets0M7bUVEDkjYwC/Ly9AIX0QkTMIGfmmuF/jOOb9LEREZFBI68Pd3dtO4r8PvUkREBoWEDfwyHYsvIvI+CRv4pbnpANTqWHwRESChA98b4etIHRERT8IGfknPCF9TOiIiQAIHfnpKMgVZaQp8EZGQhA186PnylQJfRAQSPPDLdHoFEZEDEjrwS3MzqG3UTlsREQhA4O9sbaejq9vvUkREfJfQgV+Wl4FzUN+sUb6ISMwD38zGmVmbmd0d7bZKdWimiMgBfozwbwEWxqKhv3/5SoEvIhLTwDezi4A9wPOxaO/A+XR0egURkdgFvpnlAj8Crj3MeleY2SIzW1RfX39UbeZnppGabNTq9AoiIjEd4f8YuMM5t+VQKznnbnPOzXLOzSouLj6qBpOSjJIcfflKRAQgJRaNmNl04APAcbFoL5yufCUi4olJ4AOnAlXAZjMDyAaSzewY59yMaDZcmpvOqtrmaDYhIhIXYjWlcxswBpgeuv0WeBw4K9oNj8jPpGbXPjr15SsRCbiYBL5zbq9zrrbnBrQAbc65o9srG4GJ5Tns7+pmQ0NrtJsSERnUYjWl8z7Ouetj1dbEslwAVtY2M640J1bNiogMOgl9agWAMcXZpCQZq2ub/C5FRMRXCR/4aSlJjCnOZtV27bgVkWBL+MAHmFCWoyN1RCTwAhH4E8tz2LpnH01tHX6XIiLim2AEfpm3s3a1RvkiEmABCXzvSB1N64hIkAUi8MvzMsjJSGHVdh2pIyLBFYjANzMmleVqSkdEAi0QgQ/ejttVtc045/wuRUTEF4EJ/AllObS0d1Kze5/fpYiI+CIwgd+z41bTOiISVIEJ/AmhQzNX6RQLIhJQgQn87PQURhQM0aGZIhJYgQl8gAmluQp8EQmsQAX+MeU5rK9vobW90+9SRERiLlCBP6uqgG4Hizbt9rsUEZGYC1jg55OSZLyxfqffpYiIxFygAj8zLYVjRwxV4ItIIAUq8AHmji5gaU2j5vFFJHACF/gnjC6iq9uxcOMuv0sREYmpwAX+jJFDSU023livwBeRYAlc4GempXBshebxRSR4Ahf4AHNHF7JsayMtmscXkQAJZOCfMKZQ8/giEjiBDPwZlfmheXxN64hIcAQy8IekJTN9xFDtuBWRQAlk4AOcMLqQZTV7aGrr8LsUEZGYCGzgnzKhmG4HL67a4XcpIiIxEdjAP25EPiU56Ty1vNbvUkREYiKwgZ+UZJw1uYyXVtezb3+X3+WIiERdYAMf4JwpZezr6OLlak3riEjiC3TgzxlVQH5mKk9qWkdEAiDQgZ+SnMQHjynjhZU7aO/UtI6IJLaYBb6Z3W1m282sycyqzezyWLV9KGdPKaO5vZPX1upLWCKS2GI5wv8PoMo5lwv8E3CDmc2MYft9OnFsITnpKTy5fLvfpYiIRFXMAt85t8I5195zN3QbE6v2DyY9JZkzJpXw7Ht1dHZ1+12OiEjUxHQO38x+Y2Z7gVXAduCJPta5wswWmdmi+vr6mNR1ztRydu/t4JW1DTFpT0TEDzENfOfc1UAOMA94EGjvY53bnHOznHOziouLY1LXaRNKKMxK44G3tsSkPRERP8T8KB3nXJdz7lWgAvhSrNvvS1pKEh+fWcFzK+vY0dzmdzkiIlHh52GZKQyCOfweF80eQWe34/8W1/hdiohIVMQk8M2sxMwuMrNsM0s2s7OATwEvxKL9SIwuzub4UQU8sHAL3d3O73JERAZcrEb4Dm/6pgbYDdwIfMM593CM2o/Ip+ZUsmnnXl7XhVFEJAGlxKIR51w9cEos2joaZ08pI++RVO57azMnjS3yuxwRkQEV6FMr9JaRmsz5M4bz9Ipadrb8wwFEIiJxTYHfy6fnVNLR5bjnzc1+lyIiMqAU+L2MK83htAnF3PnaRp0nX0QSigK/D186dSy7Wvfz58X6IpaIJI5DBr6ZpZjZ+WZ2R+h0B2tD/95hZp8ws5js9I212VX5zByZz/+8vJ4OnV9HRBLEQQPfzK4E1gNXAuuAfweuCv27DvgisN7MropBnTFlZnzplDFs3bOPx5fqLJoikhgONUIfD8xxzvV1OaiHgJ+YWTlwbVQq89npE0sYX5rNrS+t47zpwzAzv0sSETkqBx3hO+euPUjYh6+z3Tn3rYEvy39JScZVp4xhdV0zz6/UNW9FJP5FtNPWzC7pY5mZ2XUDX9Lgce6xwxhZmMmNz6ymS6dbEJE4F+lROj8wswfMLB/AzEYDrwIfilplg0BqchLf+uAEVtU289d3tvpdjojIUYk08KcDTcAyM/sx8BbwGHFwuoSj9eGp5UwdnsdNz1bT1qHj8kUkfkUU+M65VuB7wC7gX4BHgZ865xL+mMWkJOO6cyaydc8+7n5jk9/liIgcsUjn8D8MLAFeBKbhHcHzipmNimJtg8aJY4uYP76Ym19cS+O+Dr/LERE5IpFO6fwW+Kxz7uvOueV4lyh8GlgUtcoGme+ePYHGfR3c/MIav0sRETkikQb+NOfcsz13nHPdzrkfA2dGp6zBZ/KwPD45awS/X7CRVbVNfpcjItJvh/qmbVrPz8653X2t45x728zSo1HYYPTdsyeSm5HCvz60XFfFEpG4c6gR/lIz+46ZDevrQTMrN7PvAO9Ep7TBJz8rjevOmcSiTbt17VsRiTuHCvyTgRJgiZlVm9mjZnZv6N/VwLtAITA/BnUOGp+YWcGskfn8x5Mr2d263+9yREQidqhTKzSETpswHPgC8CSwHHgC+DxQ4Zz7rnOuISaVDhJJScYNH5tCU1snNzy+0u9yREQidtjTGzvn9gOvhG4CTCzL5apTRnPLi+s4e0oZZx5T6ndJIiKHFdH57M3s8wd5qB2oAd5wzgXqIrBfP2M8L6yq57oHlzKjcj6F2YHZdy0icSrSC5hcCpwA1OEFfAVQinccfhWAmZ3nnAvMcflpKUncdOGxnHfzAv7loeXceskMnUJZRAa1SI/DXwF82zlX6Zw70TlXiXce/Hfwwv9W4NdRqnHQmlSeyzUfHM9TK2p5SCdXE5FBLtLA/zRwc69ltwIXO+cc8J/AMQNZWLz44rzRzK7K5/sPr2B9fYvf5YiIHFSkgV8HnNtr2YeBniuDZACBPMlMcpLxq4uOIzXZuPqet3VGTREZtCIN/K8BfzSzBWZ2v5ktAO4Cvhp6/HgCOKXTY9jQIdz0yemsqm3mh4+u8LscEZE+RbTT1jn3jJmNAc4BhuEdi/+4c25nz+PAM1GrMg6cNqGEq08dw29eWsfsqgLOn1Hhd0kiIu8T6VE6OOcazOxlvC9ibe0Je/m7a84cz+JNu/neQ8sYV5LD1Io8v0sSETkg0vPhl4fCfg3wILDWzP52sPPsBFVKchK3XDyDwqx0Lv/jQuqa2vwuSUTkgEjn8G/FuwBKgXOuHMjHOyTzt9EqLF4VZadz+2dn0dzWyRV/XKSduCIyaEQa+CcD14YuddhzycPvACdGq7B4Nqk8l19+cjpLtzby7f9bqlMpi8igEGng7+Yfj7OfAOwZ0GoSyAcnl/Gdsyby6JJt/PSpVX6XIyIS8U7bnwPPmdkdwCZgJPA54N+iVVgiuOqU0Wxv3Mdtf1tPSU46l88b7XdJIhJgEY3wnXO/Az4JFOF9AasI+JRz7rZInm9m6WZ2h5ltMrNmM3vHzM454qrjhJnxg3Mn86GpZdzw+EoeflenXxAR//TnsMwXgBeOop0twCnAZuBDwJ/MbKpzbuMRvmZcSE4ybrpwOrta3+LaPy0hKy2FD+h0yiLiA/NOhdPHA2Y/iuQFnHPfP6KGzZYCP3TO/eVg68yaNcstWpQYJ+BsbuvgkjveYuW2Jm67dCanTijxuyQRSVBmttg5N6v38kNN6YyI4HZEXyc1s1JgPN5ZOHs/doWZLTKzRfX19Ufy8oNSTkYqf/zcHMaVZnPFXYtZsDZQFwoTkUHgoCP8qDVolop3ucR1zrkrD7VuIo3we+xq3c+nf/cGG3e2cvulszl5XJHfJYlIgjmSEX40ikjCO+nafuArsWx7sCjISuPuy4+nqjCLz/9hIc+vrPO7JBEJiJgFvnmXg7oD70pZH3fOBfJ0yuB9G/f+K+YyqSyHK+9azONLt/tdkogEQCxH+LcCk4BznXP7YtjuoDQ00xvpH1c5lK/e9zb3vbXZ75JEJMHFJPDNbCRwJTAdqDWzltDt4li0P1jlZKTyh8/P4ZTxxVz34DJ+8Ww1sd6nIiLB0e/AN7PH+/sc59wm55w55zKcc9lht3v6+1qJJjMthdsuncUFMyv41fNruO7BZXR2dftdlogkoIi/eBVm3oBXEXCpyUn8/BPTKM3N4OYX17KtsY2bP30cuRmpfpcmIgnkSKZ0bMCrEMyMb501gZ99fCqvrW3g/N+8xqadrX6XJSIJ5EgC/5DHzsvR+eTsSu76wvE0tLTz0VsW8Po6XVhMRAZGvwPfOXdvNAqRvzthTCF/vfokCrLSuOSON7nj1Q3amSsiRy2mX7ySyFUVZfHXL5/EGRNL+PFj7/GNB95l335dPUtEjpwCfxDLyUjlt5fM5NtnTeCRJdv46C0LWLuj2e+yRCROKfAHuaQk48unjeUPn5tDQ0s75/56AX9ZXON3WSIShxT4cWL++GKe+Po8plXkce2fl3Dtn5bQ0t7pd1kiEkcU+HGkNDeDey4/nq+dMY6H3qnhQ796hbc37/a7LBGJEwr8OJOSnMQ1Z47ngStPoKvbccFvX+eXz1XToW/nishhKPDj1OyqAp78xjzOnVbOL59bw/m/eY3qOu3QFZGDU+DHsdyMVH550XHcevEMtu7Zx0f++1VufWmdzsUjIn1S4CeAc6aW88w353P6xBJ+9tQqzrtlAcu3NvpdlogMMgr8BFGUnc6tl8zg1otnUNfUznm3LOCnT67Sl7VE5AAFfgIxM86ZWs7z15zCx2cM57cvr+PMX7zMC6t0GUURUeAnpLzMVH7+iWO5/4q5ZKQm8/k7F3HlXYuo2b3X79JExEcK/AQ2d3QhT3xtHt8+awIvV9fzgZte5lfPraGtQ9M8IkGkwE9waSlJfPm0sTx/7amcMbGUXzxXzQduepknlm3XGThFAkaBHxDDhw7hlotncO8Xjyc7PYWr73mbT/7PGyyt2eN3aSISIwr8gDlxTBGPf20eP/nYVNY3tPBPNy/g6/e/w5Zdmt8XSXQ2mD/Wz5o1yy1atMjvMhJWc1sHt760jt8v2EB3N1wydyRfOX0sBVlpfpcmIkfBzBY752b9w3IFvtQ2tvGLZ6v58+ItZKal8IWTR3H5vFHk6CLqInFJgS+HtaaumZuerebJ5bUMzUzlyvljuPSEkWSlp/hdmoj0gwJfIrasppEbn1nNy9X1FGalceUpo7lk7kgy0xT8IvFAgS/9tnjTbn75XDWvrGmgMCuNL8wbxWfmjtRUj8ggp8CXI7Z40y7++/m1vFxdT96QVC47sYrLTqwiXzt3RQYlBb4ctaU1e/j1C2t59r06hqQmc9GcEVw+bzTDhw7xuzQRCaPAlwGzpq6ZW19exyPvbgPgI9PK+eL80UweludzZSICCnyJgprde/n9qxu5f+Fm9u7v4qSxhXzh5FGcOr6EpCTzuzyRwFLgS9Q07u3gnrc28YfXNlLX1M7o4iw+d9Iozj9uuA7pFPGBAl+irqOrmyeWbeeOVzewtKaRnIwULpg5gktPGElVUZbf5YkEhgJfYsY5x9ub9/CH1zbyxLLtdDnH/HHFfGbuSE6bWEKypntEokqBL76oa2rj3jc3c//CzdQ1tTN86BAumj2CC2ePoDQ3w+/yRBKS74FvZl8BLgOmAvc55y473HMU+Imjo6ub596r4643NvHaup0kJxkfmFTCRXMqmT+uWKN+kQF0sMCP5R61bcANwFmADtwOmNTkJM6ZWs45U8vZ0NDK/W9t5s+La3h6RR3leRlcMGsEF8ysYERBpt+liiSsmE/pmNkNQIVG+LK/s5vnVtZx/8ItvLKmHufghNGFXDi7grMnlzMkLdnvEkXiku9TOmGFHDLwzewK4AqAysrKmZs2bYphdeKXrXv28ZfFNfzf4ho279pLdnoKH5paxvkzKphTVaDj+kX6IW4CP5xG+MHT3e14c8MuHny7hieWbad1fxfDhw7ho8cN42PHVTC2JNvvEkUGPQW+xJ29+zt5ZkUdD76zlVfX1NPtYMrwXM47djgfObac8jztChLpiwJf4tqO5jYeeXcbjy7ZxpKaRsxgTlUB5x47jHOmlFGYne53iSKDhu+Bb2YpeEcF/QCoAL4IdDrnOg/2HAW+9GVDQyuPvLuNR5ZsZV19K8lJxoljCvnw1HLOmlym0zZL4A2GwL8eL+zD/dA5d/3BnqPAl0NxzrGqtpnHlm7jsaXb2bRz74HwP2dKOR+cXEqRRv4SQL4H/pFQ4EuknHOs2NbE48u288QyL/yTDOaMKuDsyWWcNaVMc/4SGAp8CQznHCu3N/Pk8u08ubyWtTtaADi2Io8PTi7jrMmljCnOxkyHekpiUuBLYK2rb+HpFbU8vbyWJTWNAIwqyuLMY0r5wKRSZlQOJSU5yecqRQaOAl8EqG1s49mVdTyzopY31u+ko8uRn5nKaRNKOH1SCfPHF5Ori7RLnFPgi/TS3NbB36obeG5lHS+u3sGevR2kJBmzqvI5fWIJp08s0dSPxCUFvsghdHZ1886WPTy/cgcvrtrB6rpmACryh3DahBJOnVDMCWMKyUzTFbxk8FPgi/TD1j37eHHVDl5avYMFa3eyr6OLtOQk5owqYP74IuaPL2ZCaY5G/zIoKfBFjlB7ZxcLN+zm5eodvFxdT3Wdd9RPSU4688YVM398ESeNLdIx/zJoKPBFBsj2xn38rbqeV9Y08OraBvbs7QBgUnkuJ48t5KSxRcwZVaDpH/GNAl8kCrq6HSu2NXrhv6aBxZt2s7+rm9RkY0ZlPieOKeLEsYUcWzGUtBQd+imxocAXiYF9+7tYuHEXC9Y1sGBtAyu2NeEcDElNZlZVPieMKeTEMUVMGZarY/8lahT4Ij7Ys3c/b6zfxWvrGnh93U7WhL71m52ewuyqfOaOLuT40YXaAMiAGgzXtBUJnKGZaZw9pYyzp5QBUN/czhvrdx64vbi6HoCstGRmjPQ2AHNGFTCtIo/0FF3iUQaWRvgiPtrR3MbCDbt5c8NO3ly/68Dx/2kpSUyvGMrsUfnMripgxsh8fQNYIqYpHZE4sLt1P4s27ebN9TtZuHEXy7c10dXtSDKYUJbL7Kp8Zo7MZ1ZVAcPyMvQ9AOmTAl8kDrW2d/Lulj28tWEXizft5p3Nu2nd3wVAWW4GM6vymVHpbQSOKc/VkUACaA5fJC5lpadw0ljvi13gnQJiVW0zizftPnB7fOl2ANJTkpg6PI8ZI/OZUTmUGZX5lORm+Fm+DDIa4YvEudrGNt7e7IX/25t3s2JrE/u7ugEYlpfBcZX5TB8xlOmVQ5kyLI8hadoZnOg0pSMSEO2dXSzf2sS7W/bw7pY9vL1pN1v37AMgOcmYUJrDsSOGctyIoUwbkce4khySk7QvIJFoSkckINJTkpk50pvX71Hf3M6S0AZgSc0eHlu6jfve2gxAZloyU4blMa0ij2kjhjJteB4jCzO1QzgBaYQvEkDd3Y4NO1tZWrOHJVsaWVKzhxXbmtjf6U0F5Q1JZerwPKZW5Hn/Ds+jIn+INgJxQiN8ETkgKckYU5zNmOJsPnZcBQAdXd1U1zWztKaRpTWNLN/ayO2vrKejyxsUDs1MZcqwPCYPz2Xq8DymDMujsiCTJE0HxQ2N8EXkoNo7u1hd28yyrd4GYGlNI9V1zQc2AjnpKUwaluttCIblMnl4LmOKs0nVaSJ8pRG+iPRbekoy0yqGMq1i6IFl+zu9TwLLtzayYlsTy7c1cu9bm2jr8KaD0lKSmFCaw+RhuRwzLJdjynOZWJ5Ldrrixm/6C4hIv6SlJDFleB5ThucdWNbV7djQ0MKKbU2s2NbEe9uaeHpFLfcv3HJgncqCTCaV5zCpPJdJ5d6GQPsFYkuBLyJHLTnJGFuSw9iSHM6bPhwA5xy1TW2s3O5tAN7b3sTK7c08814dPTPJOekpTCjLYWJ5DhPLcplUnsP40hxydN6gqFDgi0hUmBnleUMozxvC6RNLDyxvbe9kdV0zq7Y3s3J7Eyu3N/HwO9u4u33zgXWGDx3CxNCGYHyptzEYXZylfQNHSYEvIjGVlZ7CjErvHEA9nHNs3bOP1bXNrArdVtc28XJ1PZ3d3seB1GRjdFE248tymFCazbjSHCaU5jCiIFNfHIuQAl9EfGdmVORnUpGfyRmT/v5poL2zi/X1rVTXeRuB6tpm3tm8m0eXbDuwTnpKEmNLshlfmsO40mzGl3ifCiryh+iQ0V4U+CIyaKWnJB/YyXte2PKW9k7W1DWzpq6F1XXNVNc18/q6nTz0ztYD62SkehuCcSU5jC3JDv2cTWVBZmCvLqbAF5G4k52ewnGV+RwXNi0E0Livg7U7WlhT10x1XQtr61t4c/37NwRpyUlUFWV6G4HibMaENgaji7IT/sRyCnwRSRh5Q1L/4TxCAM1tHayrb/U2BjuaWbejhfe2NfHU8lpCuwgw83YW93wDeUxJFqOLvH+Ls9MT4vBRBb6IJLycjFTvFNEjhr5veVtHFxt3trJuRyvr6ltYu6OFdfUtvLVhF/s6usKen8Lo4mzGFGUxujiL0cXZjC7Ooqowi4zU+PlUoMAXkcDKSE1mYlkuE8ty37e8u9uxrXEf6+tbWV/fwrr6VtY3tPD6+p08GDY9ZAbD8oZ4G4GiLEYVZTGqOJvRRVkMGzpk0B09FLPAN7MC4A7gg0ADcJ1z7t5YtS8iEqmkpL8fNTR/fPH7Hmtt72RDQyvrG7yNwYaGVjY0tPKXt7fS0t55YL205CQqCzOpKsxiVFEmo4qyqSrKZFRRFqU5Gb4cQRTLEf4twH6gFJgOPG5mS5xzK2JYg4jIUclKT/mHU0uA912C+pZ2Njbs9TYEO1vZGNoY/G1N/YFTT4N3BFFVYRYjCzOpKvKmhqoKs6gqyozqxiAmgW9mWcDHgSnOuRbgVTN7BPgM8M+xqEFEJJrMjJKcDEpyMpgzquB9j/VMEW1s2MuGna1samhl405vJ/KLq+oPXJISvO8VVBZk8tvPzGRMcfaA1hirEf54oMs5Vx22bAlwSu8VzewK4AqAysrK2FQnIhJF4VNEJ48ret9jXd2O7aGNwaZdrWzauZeNDa3kZ6YNeB2xCvxsoLHXskYgp/eKzrnbgNvAOx9+9EsTEfFPcvjGgKLDP+EoxOrrZi1Abq9luUBzjNoXEQm8WAV+NZBiZuPClh0LaIetiEiMxCTwnXOtwIPAj8wsy8xOAs4D7opF+yIiErsRPsDVwBBgB3Af8CUdkikiEjsxOw7fObcL+Gis2hMRkfcL5jlCRUQCSIEvIhIQCnwRkYAw5wbvd5vMrB7Y1I+nFOGdmC1IgthnCGa/g9hnCGa/j7bPI51zxb0XDurA7y8zW+Scm+V3HbEUxD5DMPsdxD5DMPsdrT5rSkdEJCAU+CIiAZFogX+b3wX4IIh9hmD2O4h9hmD2Oyp9Tqg5fBERObhEG+GLiMhBKPBFRAJCgS8iEhBxFfhmVmBmD5lZq5ltMrNPH2Ldb5pZrZk1mtnvzSw9lrUOpEj7bWafNbPFZtZkZjVm9nMzi+WF6gdMf/7WYc95wcxcvPYZ+v0eH21mj5lZs5k1mNnPY1nrQOnH+9vM7AYz2xr6f/2SmU2Odb0Dwcy+YmaLzKzdzO48zLoDlmVxFfjALcB+oBS4GLi1rz+4mZ2Fd3H0M4AqYDTww9iVOeAi6jeQCXwD71t6x+P1/1sxqnGgRdpnAMzsYmJ49tcoivQ9ngY8C7wAlAEVwN0xrHMgRfq3vgD4PDAPKABeJ36vqbENuAH4/aFWGvAsc87FxQ3IwntTjA9bdhfw0z7WvRf4Sdj9M4Bav/sQ7X738dxrgEf97kO0+wzk4V1VbS7ggBS/+xDtfgNXAK/4XXOM+/xd4E9h9ycDbX734Sj7fwNw5yEeH9Asi6cR/nigyzlXHbZsCd4fvbfJocfC1ys1s8Io1hct/el3b/OJz8tI9rfPPwFuBWqjXViU9affc4GNZvZkaDrnJTObGpMqB1Z/+nw/MNbMxptZKvBZ4KkY1OinAc2yeAr8bKCx17JGICeCdXt+7mvdwa4//T7AzD4HzAJujFJd0RRxn81sFnAS8OsY1BVt/flbVwAXAf8NDAMeBx4OTfXEk/70eTvwCrAa2Ic3xfPNqFbnvwHNsngK/BYgt9eyXKA5gnV7fu5r3cGuP/0GwMw+CvwUOMc5F49nGYyoz2aWBPwG+LpzrjNGtUVTf/7W+4BXnXNPOuf2423YC4FJ0S1xwPWnzz8AZgMjgAy8uewXzCwzqhX6a0CzLJ4CvxpIMbNxYcuOpe8pixWhx8LXq3PO7YxifdHSn35jZmcDvwPOdc4ti0F90RBpn3PxPsU8YGa1wMLQ8hozmxf9Mgdcf/7WS/H2V8S7/vT5WOAB51yNc67TOXcnkA8cE/0yfTOwWeb3Tot+7uC4H+8C6Fl4H+Mbgcl9rHc23nzuMXhviBeIYCfnYL31o9+nAzuB+X7XHIs+A4Z3hErPbTZeCA4H0vzuQ5T/1hOAvcAHgGS8qY118djvfvT5B8CreEfzJAGfAVqBoX734Qj6nIL3KeU/8HZSZ9DHwQYDnWW+d7yfv6QC4K+hP/Jm4NOh5ZV4H30qw9a9BqgDmoD/BdL9rj/a/QZeBDpDy3puT/pdf7T/1mHPqSKOj9Lpb7+B84G1off4S32FZDzc+vH+zsA7hHN7qM9vA2f7Xf8R9vn60Hs1/HZ9tLNMJ08TEQmIeJrDFxGRo6DAFxEJCAW+iEhAKPBFRAJCgS8iEhAKfBGRgFDgi4gEhAJfRCQgFPgiIgGhwBeJkJmNMbNdZjYjdH9Y6Fz0p/pbmUhkdGoFkX4wsy/indtkJvAQsMw5F6+XkZSAUeCL9JOZPQKMwjvh1WznXLvPJYlERFM6Iv33O2AK8GuFvcQTjfBF+sHMsvGuK/oicA4w1Tm3y9+qRCKjwBfpBzO7A8hxzl1oZrfhXXzjQr/rEomEpnREImRm5+Fdgeiq0KJrgBlmdrF/VYlETiN8EZGA0AhfRCQgFPgiIgGhwBcRCQgFvohIQCjwRUQCQoEvIhIQCnwRkYBQ4IuIBMT/A9Il7rAp/3hbAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "plot_function(lambda x: -1*torch.log(x), min=0,max=1, tx='x', ty='- log(x)', title = 'Log Loss when true label = 1')" ] }, { @@ -1293,14 +1339,128 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Taking the mean of the positive or negative log of our probabilities (depending on whether it's the correct or incorrect class) gives us the *negative log likelihood* loss. In PyTorch, `nll_loss` assumes that you already took the log of the softmax, so it doesn't actually do the logarithm for you." + "Let's go ahead and update our previous table with an additional column, `loss` to reflect this loss function:" + ] + }, + { + "cell_type": "code", + "execution_count": 137, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
37targidxresultloss
0.6024690.397531000.6024690.506720
0.5020650.497935110.4979350.697285
0.1331880.866811020.1331882.015990
0.9966400.003360130.0033605.695763
0.5959490.404051140.4040510.906213
0.3661180.633882050.3661181.004798
\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "#hide_input\n", + "from IPython.display import HTML\n", + "df['loss'] = -torch.log(tensor(df['result']))\n", + "t = df.style.hide_index()\n", + "#To have html code compatible with our script\n", + "html = t._repr_html_().split('')[1]\n", + "html = re.sub(r'', r'
', html)\n", + "display(HTML(html))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "> warning: Confusing Name, Beware: The nll in `nll_loss` stands for \"negative log likelihood,\" but it doesn't actually take the log at all! It assumes you have _already_ taken the log. PyTorch has a function called `log_softmax` that combines `log` and `softmax` in a fast and accurate way. `nll_loss` is designed to be used after `log_softmax`." + "Notice how the loss is very large in the third and fourth rows where the predictions are confident and wrong, or in other words have high probabilities on the wrong class. One benefit of using the log to calculate the loss is that our loss function penalizes predictions that are both confident and wrong. This kind of penalty works well in practice to aid in more effective model training. \n", + "\n", + "> s: There are other loss functions such as [focal loss](https://arxiv.org/pdf/1708.02002.pdf) that allow you control this penalty with a parameter. We do not discuss that loss function in this book." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We're calculating the loss from the column containing the correct label. Because there is only one \"right\" answer per example, we don't need to consider the other columns, because by the definition of softmax, they add up to 1 minus the activation corresponding to the correct label. As long as the activation columns sum to 1 (as they will, if we use softmax), then we'll have a loss function that shows how well we're predicting each digit. Therefore, making the activation for the correct label as high as possible must mean we're also decreasing the activations of the remaining columns. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Negative Log Likelihood" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Taking the mean of the negative log of our probabilities (taking the mean of the `loss` column of our table) gives us the *negative log likelihood* loss, which is another name for cross-entropy loss. Recall that PyTorch's `nll_loss` assumes that you already took the log of the softmax, so it doesn't actually do the logarithm for you." ] }, { @@ -1312,7 +1472,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 128, "metadata": {}, "outputs": [], "source": [ @@ -1328,7 +1488,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 129, "metadata": {}, "outputs": [ { @@ -1337,7 +1497,7 @@ "tensor(1.8045)" ] }, - "execution_count": null, + "execution_count": 129, "metadata": {}, "output_type": "execute_result" } @@ -1355,7 +1515,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 130, "metadata": {}, "outputs": [ { @@ -1364,7 +1524,7 @@ "tensor(1.8045)" ] }, - "execution_count": null, + "execution_count": 130, "metadata": {}, "output_type": "execute_result" } @@ -1384,7 +1544,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 131, "metadata": {}, "outputs": [ { @@ -1393,7 +1553,7 @@ "tensor([0.5067, 0.6973, 2.0160, 5.6958, 0.9062, 1.0048])" ] }, - "execution_count": null, + "execution_count": 131, "metadata": {}, "output_type": "execute_result" } @@ -1402,6 +1562,13 @@ "nn.CrossEntropyLoss(reduction='none')(acts, targ)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You will notice these values match the `loss` column in our table exactly." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1655,7 +1822,7 @@ ], "source": [ "learn = cnn_learner(dls, resnet34, metrics=error_rate)\n", - "lr_min,lr_steep = learn.lr_find()" + "lr_min,lr_steep = learn.lr_find(suggest_funcs=(minimum, steep))" ] }, { @@ -2556,11 +2723,23 @@ "split_at_heading": true }, "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.7" } }, "nbformat": 4, - "nbformat_minor": 2 + "nbformat_minor": 4 } diff --git a/clean/04_mnist_basics.ipynb b/clean/04_mnist_basics.ipynb index f6b1016..e303508 100644 --- a/clean/04_mnist_basics.ipynb +++ b/clean/04_mnist_basics.ipynb @@ -396,7 +396,7 @@ "metadata": {}, "outputs": [], "source": [ - "tensor([1,2,3]) + tensor([1,1,1])" + "tensor([1,2,3]) + tensor(1)" ] }, { @@ -956,7 +956,7 @@ "metadata": {}, "outputs": [], "source": [ - "corrects = (preds>0.0).float() == train_y\n", + "corrects = (preds>0.5).float() == train_y\n", "corrects" ] }, @@ -1643,7 +1643,7 @@ "split_at_heading": true }, "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" } diff --git a/clean/05_pet_breeds.ipynb b/clean/05_pet_breeds.ipynb index 3d52d8f..d4a85b4 100644 --- a/clean/05_pet_breeds.ipynb +++ b/clean/05_pet_breeds.ipynb @@ -123,6 +123,7 @@ "dblock1 = DataBlock(blocks=(ImageBlock(), CategoryBlock()),\n", " get_y=parent_label,\n", " item_tfms=Resize(460))\n", + "# Place an image in the 'images/grizzly.jpg' subfolder where this notebook is located before running this\n", "dls1 = dblock1.dataloaders([(Path.cwd()/'images'/'grizzly.jpg')]*100, bs=8)\n", "dls1.train.get_idxs = lambda: Inf.ones\n", "x,y = dls1.valid.one_batch()\n", @@ -341,7 +342,7 @@ "df = pd.DataFrame(sm_acts, columns=[\"3\",\"7\"])\n", "df['targ'] = targ\n", "df['idx'] = idx\n", - "df['loss'] = sm_acts[range(6), targ]\n", + "df['result'] = sm_acts[range(6), targ]\n", "t = df.style.hide_index()\n", "#To have html code compatible with our script\n", "html = t._repr_html_().split('')[1]\n", @@ -371,7 +372,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Taking the Log" + "#### Taking the Log\n", + "\n", + "Recall that cross entropy loss may involve the multiplication of many numbers. Multiplying lots of negative numbers together can cause problems like [numerical underflow](https://en.wikipedia.org/wiki/Arithmetic_underflow) in computers. Therefore, we want to transform these probabilities to larger values so we can perform mathematical operations on them. There is a mathematical function that does exactly this: the *logarithm* (available as `torch.log`). It is not defined for numbers less than 0, and looks like this between 0 and 1:" ] }, { @@ -380,7 +383,38 @@ "metadata": {}, "outputs": [], "source": [ - "plot_function(torch.log, min=0,max=4)" + "plot_function(torch.log, min=0,max=1, ty='log(x)', tx='x')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "plot_function(lambda x: -1*torch.log(x), min=0,max=1, tx='x', ty='- log(x)', title = 'Log Loss when true label = 1')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from IPython.display import HTML\n", + "df['loss'] = -torch.log(tensor(df['result']))\n", + "t = df.style.hide_index()\n", + "#To have html code compatible with our script\n", + "html = t._repr_html_().split('')[1]\n", + "html = re.sub(r'
', r'
', html)\n", + "display(HTML(html))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Negative Log Likelihood" ] }, { @@ -476,7 +510,7 @@ "outputs": [], "source": [ "learn = cnn_learner(dls, resnet34, metrics=error_rate)\n", - "lr_min,lr_steep = learn.lr_find()" + "lr_min,lr_steep = learn.lr_find(suggest_funcs=(minimum, steep))" ] }, { @@ -675,11 +709,11 @@ "split_at_heading": true }, "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" } }, "nbformat": 4, - "nbformat_minor": 2 + "nbformat_minor": 4 }