fastbook/14_deep_conv.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from utils import *"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "[[chapter_deep_conv]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Improving training stability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we are so good at recognizing threes from sevens, let's move onto something harder—recognized all 10 digits. That means we'll need to use `MNIST` instead of `MNIST_SAMPLE`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = untar_data(URLs.MNIST)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "Path.BASE_PATH = path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#2) [Path('testing'),Path('training')]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is in two folders named `training` and `testing`, so we have to tell `GrandparentSplitter` about that (it defaults to `train` and `valid`). We define a function `get_dls` to make it easy to change our batch size later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dls(bs=64):\n",
    "    return DataBlock(\n",
    "        blocks=(ImageBlock(cls=PILImageBW), CategoryBlock), \n",
    "        get_items=get_image_files, \n",
    "        splitter=GrandparentSplitter('training','testing'),\n",
    "        get_y=parent_label,\n",
    "        batch_tfms=Normalize()\n",
    "    ).dataloaders(path, bs=bs)\n",
    "\n",
    "dls = get_dls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Always a good idea to look at your data before you use it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOsAAAD4CAYAAAANSBHgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAdtUlEQVR4nO2deZQV1dHAf5d9QFYJm2ELIhHBJSibfEBYAi5gCKCicFDCEsBEEQ2iAoIQDIugAkERzCdGXEI0iDlGAcNOHAwIKIuon0GBGFQQEFCgvz+G6u7HvHnvzdu671C/czjT9O3XXXNf11TdunXrGsdxUBQl/BQLWgBFURJDlVVRLEGVVVEsQZVVUSxBlVVRLEGVVVEsQZVVUSwhdMpqjOlnjHnXGPO1MeaYMWa7MWakMcYELVuYMcbca4xZf6bfDhpj1hhjugYtlw3Y8s6VCFqAKHwBPAzsBE4A/wPMAU4CjwUoV9jpACwAcoFjwCBgqTGmneM4awOVLPxY8c4ZGzKYjDGvADiO0yNoWWzCGLMVeNNxnJFBy2IbYXznQucG+zF5NAeuBt4OWh6bMMYUA8oDB4KWxSbC/M6F0Q3GGFMR+BwoBRQHxjuO83iwUlnH/UAlYGHQgtiADe9cKJUVOAxcDpQFWgOTjTF7Hcd5Olix7MAYM4w8Ze3uOM5nQctjCaF/52wZs44G7nQcp0bQsoQdY8w9wHjgBsdxlgUtj62E8Z0Lq2U9m2JA6aCFCDvGmAnACOBax3FWBi2P5YTunQudshpjxgOrgY+BkkBbYBTwTJByhR1jzExgCNAH2GmMEYtwzHGcQ8FJFn5seedC5wYbY2YA3YALgOPkdeACYK7jOKeClC3MGGMK+iL/13Gc27Ipi23Y8s6FTlkVRYlOqOdZFUXxUGVVFEtQZVUUS1BlVRRLiDd1c65Hn1JZIqV9lzzad1FQy6oolqDKqiiWoMqqKJagyqoolqDKqiiWoMqqKJagyqoolqDKqiiWELr1rEp6OHUqb2XX9u3b3XOrVq0CYPfu3QBs3rwZgP3797vXLFiwAICWLVtmRU4lcdSyKoolxFvPGkja1+nTpwH4+9//DsDSpUvdtjlz5hT6frm5ue7xT37yEwCKFUvo71SoU+aOHz8OwK5du9xzDz/8MADr1q0DIq2mfNexCs1feOGFAGzZsgWA0qWTrmwSqr6T333u3LkADB8+vMBrhg4dCkCNGoUrvyTvVqdOnQAoU6ZMcsJquqGi2I0qq6JYQuBu8MGDBwGYP3++e+7JJ58EvEBIOnn88by6zXfccUcilwfmyp08eRKAd999N1+buKi///3vAfj444/dNnFxq1WrBsDNN9/sttWqVQuA888/H4DatWsD0LWrt3+VvA9ff/01ABUrVkz2VwiVG/z9998DKbmmCfPQQw8BMGbMmGRvoW6wothMYFM3R44cAaB169YA7NixIyhRQsmaNWsA6NChg3uuoMDQdddd5x6PHz8egEsuuQSIHSCS7yAar7/+OgC33HJLghKHmxIl8l51CSzNmzevwGvFCvv7WzyORAoMPv/880BKljUqalkVxRICs6zFixcHoEqVKkl9PicnB4DvvvsO8JIA4lGhQoWknpctvvjiCwD69++fr03+qv/2t78FvLFm+/btk3rWX//614j7+rnhhhuSumdYESspMQv5GY133nkHgJo1a7rnDh8+DMCUKVMAWLiw4P2+JDaQbtSyKoolqLIqiiUE5gaLGys5qJJxE+2aCy64wD0nLqC4JZKxI1NA0ahUqZJ73Ldv31TEzjhvvfUWAJ99lrdTo3/qZOrUqYAX9ClbtmxanukPpPTo0SOt97aR5s2b5zsnWXAydIhGo0aNAO97SjdqWRXFEgJfdSOWsUmTJu65Rx99FIDp06cD0LlzZ7ft0KG8DdHatGkDxLaowsaNG93jBHOCA6N8+fKA501IfjTAxRdfnNZnffvtt/nOSQAuVv5wUUUCbcuXLwfg17/+tdu2Z88eAI4dOwZ4/TNkyBD3mokTJwJQuXLljMgX7jdXURSXwNMNE+HEiRPucatWrQBvLWY0fvCDHwCwaNEiIHJqo5CWNVQpc+lCUgkbNGgAeN4KeFYl2ekgH6HuuwMHDgCwYcMG99zixYsBePbZZ+N+Po39FA1NN1QUmwl8zCqpXf51l8Lnn38OeCl0ULBFlcR1gEceeQSITNVTPKRihIz369Sp47ZlyFIEwjfffOMeb9q0CYBp06YBsHbtWiDSq4jFNddcA3hJ+s2aNUuXmAmjllVRLEGVVVEsIXA3WNzfunXrJvX56tWrA7B69Wr3nJQmUaKzfv16wJt+6NmzZ5DipJ033ngDgFtvvdU9l8gU39lIggpAu3btAC+nPQjUsiqKJQQ+dSOTzcla1q1btwLe+s00E+rph8IiK3okLU4CMBJsgbSWIM1638nKq1KlSqXwaA9/VQkJcvbq1Qvw3tcMJY/o1I2i2EzgY9Zkkb9wYiWU+MiUllhUmaa57LLLghIprUjCy4ABAwCvYHmySKlXgFGjRkX8lCmgESNGpPSMwqCWVVEsQZVVUSzBugCTuCH33nsvkHxZmASxPsDkz32V4nSCFPbylytNI4H1nbj5L774YoHXSNmacuXKueekPwQpWgfwt7/9DYCvvvoK8AJL/t0errjiilTE9qMBJkWxGessq+S1ynrWDGOdZZUkEwmOdOnSxW378MMPAa+v33//fSBjVSGs67tY7N27F4DGjRsDXqUSSTCB6BUmkkQtq6LYTMqWVdYA+re6kHFlIkgamH/6QKxtNKQm0UcffQScW2NW2V1Pdo2T9brLli1zr5GqGLL9RoRAZ75rGafdfvvtgFfXCuCHP/xhusQNVd+li7Zt2wJeIsmvfvUrt2327NnpeoxaVkWxmZSTIsSXf+CBB/K1JWJht23bBnhRtniIJS1ZsmSiIlqJVHNYsmSJe06im/66TBBZpDuR9DepvSSWwG8R7rrrLsCztrJQwkb83oW8Z5KWWtj3R1IZ/VVLIHodq0yhllVRLEGVVVEsIeUA0xNPPAHAnXfe6Z4777zzAC/o5C+5IowbNw7wyo0m6k7INvODBw9O6PoUyXqQRNZiSonLWMG2WEjutD8/Vr7rWbNmAZ5bLfu9+q+pWrUqAP369XPbxDWuUaNGIiIEFmCS/WqHDRvmnpMCZ7Lf7aWXXlqoe0qChKxrFXTqRlGUfKRsWWWqINpfFdm2ItqkuwRQpGhyNOrVqwdE/nUfOXIkkLXd4LJiHSZNmuQeT5gwAfCCI4kEj+rXr+8ei6fSrVs3IHbpVSlW57cOMhWxc+fOfNfLthB33313gff0kXXLumLFCgD69OkDeOVGAbZv3w7ARRddVODnjx49GnEfWVkDXoDq7IoTalkVRclH2tINZesAgLFjx6YgkpcOJ3toipUIgIxaB0md9G8PcnYyQzTLKltsSFLD5MmT3Wv81Q2SQayRWGiZmgN46qmngNi7qfvIumWVVMBoXsHo0aMBL56yY8cOwEssAa+vY+31Kx7dH//4RwC6d+/utqWxaoRaVkWxGVVWRbGEtLnB/kCRf41govh345Kq5yHInsmoK7dv3z4AGjZs6J6TfrzxxhsBqFmzptsme8tKBX2ZXgkpWXeDpcSPP089XcgeS/PmzQPSv6PfWagbrCg2k7aCabJLOXgJDjNmzACi5w0PHDgQ8Eo8+ifaz5W9QcVqHjlyJGBJigZXX301ENuy9u7dG0isdK0/AaJFixZAwsG1jKCWVVEsIfBKESGnSK7JzBLad8mjY1ZFsRlVVkWxBFVWRbEEVVZFsYTQKWu9evUwxuT7l6Fd4ooUBw4cYOjQodSqVYvSpUtTv359d/2vUjBTp06lVatWVK5cmUqVKtGmTRt3XXGYCN3GVLm5uRGJ1EePHqVp06aZqhpfZDhy5Aht27blggsuYNGiRdStW5d9+/a5y+CUglmxYgUDBgzgqquuIicnh3nz5nH99dezcuVKd+42DMSbugkcY8wgYA5Q13GcvfGuP1cxxowH+gONHMc5Ee96JTbGmK3Am47jjAxaFiF0bnAUhgCvqaLGpSewBphhjNlnjNlhjJlqjMlIuf2ijDGmGFAeOBDv2mwSOjfYjzHmSqAZkD9fUTmbBsCFwItAN6AWMOvMz1sDlMtG7gcqAQuDFsRPqN1gY8w8oCPQwAmzoCHAGHOCPEtQ13Gck2fO9QJeBs53HCexwsznOMaYYcA0oLvjOMviXZ9NQusGG2MqAH2Ap1RRE2If8KEo6hneP/MzsV2/znGMMfcAUwmhokKIlRXoC5QCnglaEEtYDTQwxhT3nWt05uf/ZV8cuzDGTADGAdeGUVEhxG6wMeY9YKfjODcGLYsNGGMuA94BFgAzyRurzgPWOo7TP0jZwo4xZiZ5gcw+wAZf0zHHcQ4FI1V+QqmsxpiWwHqgk+M4y4OWxxaMMR2BR4CmwH7yxqvjHMfJ3oYsFmKMKUgJ/tdxnNuyKUssQqmsiqLkJ8xjVkVRfKiyKoolqLIqiiWosiqKJcRLNzzXo09aRyh5tO+SR2swKYrNqLIqiiWosiqKJaiyKoolhHo9qxKdt99+G4D77rsPgHfeecdtO378OBDsNg9KZlDLqiiWoJbVIr755hsA+vXrB8DXX38NwA033OBeU6yY/v0tqug3qyiWoJY15Pg3qb7++usBbxPmyy67DIBXX301+4IpWUctq6JYglrWkCMRX4C1a9dGtC1YsCDb4ihnkPjBtm3b3HOtW7fO6DPVsiqKJaiyKoolZN0N/vLLLwHo1asXACtXrgTAX16mefPmALRt27bA+9SrVw+Abt26AVCnTp20yxokb731FgDPPJO/uGPNmjUBqFKlSlZlUmDz5s2AN332wQcfuG1ly0ZuftCzZ0/3eMqUKQBUq1Yt6WerZVUUS4hXMC3t6wr//e9/A9CwYUOAlHc5k7S6l156yT0n1jYNZH1N5okTeXtKtWzZEoAtW7bku2bTpk0AXHrppcnKlg2K5HrWq666CoB//etfhfrcgw8+CMD48eMTuVzXsyqKzWR9zCpjy+eeew6Am266KaX7iSW65ZZb3HOHDx9O6Z5BIpv4RrOogozXleyxd2/eJob+MWphiPV9JopaVkWxBFVWRbGEwDKYevfuDXjh7W+/9XZ4WL16NQC1a9cGIsPdO3bsAKB9+/bZEFNRAChfvjzgrXB68cUXC/V5mepJBbWsimIJgecGy/rL8847zz13zTXXFHi95GSeTbly5dIrWAi58847gXPjdw0bYlknTJgAeFM4/sCmfD8vv/wyAI0aNXLb2rVrl7IMalkVxRICt6yJIHWFAEaMGBH1munTp2dLnMCQaa/ixYvHuVLJFBdeeCHgvYfPP/+82yYWVbxFf6LO+eefn/Kz1bIqiiVYYVlnzpzpHr/++usRbfIXa/fu3e65Tz/9NOIaSXwHKFWqVCZETBuSwC9poJUrV3bb7rrrrrQ+69SpUwAsXbrUPWeMiXh+NDp27AhExhnCiP93kHpVkydPBuDIkSNAZM2qPn36AHD11VcDXl9EQ5Ijbr/99nxtkvDTpEmTpGWPhlpWRbEEVVZFsYSsr7oRZLVNtFU3q1atAjx3YtmyZW7bF198Uehn+V3nWNNCUcj6yhFxy8QF87vBBw4cSEEcz/X75z//CXglYwq7gkRWNcn3U4A7nPW+O3nyJOAVPY9VEicWt912GwCPPfaYe05+xw0bNgDelI1/yPXzn/8c8IJOKRRa11U3imIzWbesUlqzR48eALz55pvpfkQ+7r//fvd44sSJhflo4Ja1UqVKbtt//vMfAEqUiB8XlO/VHzySQN0//vGPAj8n95YkAH8pVP8UGkCHDh0ALyh2Flnvu5EjRwLe7+lPHpFKDdddd13EZxYtWuQejx49OqKtVatW7vGYMWMAGD58OACffPIJEFmtY8+ePQCUKVMmGfH9qGVVFJvJumX9+OOPAW9yubBIQoCUfRw4cCAAOTk57jVy7yVLlgBeGhhAxYoVC/O4rFsHsajRtsGQxI9YUzj79+8HYP78+QCMHTvWbTt9+nTEvcUCTJo0yb1Gqk+I1ZwxY4bbds8990R9pkwBnf2rFChkfOL2nfye8+bNc8+J19S9e3cABg8e7LZ17tw57kPFWibybsozhgwZ4p7r2rVr3M8liFpWRbEZVVZFsYSsZzDJGlVZtSCZRzVq1HCvkXKl0aZpqlatCnglTGNx+eWXpyZsAMTKmkkECZj43V9BitRJUE+yvySY5EfKkDz88MMFPktKxgaBDG3+/Oc/u+ckaLlw4UIgsUDPf//7X/d4+fLlca+X7KbZs2cDUKtWrQQlTh21rIpiCYElRZyNVIAAL3h08OBBIHJdoPz1y9JftKwHmB566CEgukVr3Lgx4CU1nF1UGry+Eo9l3LhxbpsEXMSLkeoc/sQUCbLIFIcEcvw8++yzgLerXQFBu4z2nQTJ/B7EAw88AEDJkiUL/Jysh163bh0QGSD67LPP4gomHl2bNm3iXpsCGmBSFJsJzaqbOXPmuMdiUYVBgwa5x9kcIwTBFVdcUWCbrPSQCX4pSSrpceBNZfz0pz/Ndz9JcJDpGEmLSzTdUMZpkm5YoUKFhD6XSTp16uQei5e4ceNGANavX++2iTcie9n6kz2Eiy66CIBdu3YV+DxJfc2wZY2KWlZFsYTAx6yLFy8GIot9y+S9bF7lrySXarS0kGR9zCoJBqNGjQIikxLORsZt/miujD9lPOpPFpF+lcLo0ahbty7gTfrffPPNbpts6ZEgGe07eQ/8iwjE0kcbZwsyzpc+k+QR8LwRGedPmzYt3+cPHTqU77kZQMesimIzqqyKYgmBucHi7knOpn8liKwDlGmaTG//HoPAdkITV9U/NfHCCy8AiU0xRBXozHctLmT9+vUBqF69unuNTMs0aNAgqWf4yGjfSR5utBU/UsanS5cu7jmZthI335+EczbvvfceAM2aNfMEOtN3EsCT52co4KlusKLYTGCWVUo5+lfjCzLA90/oB0So9hgViyr7s8pqmdzc3IQ+/8orrwCeZRWPJR1lMqOQ0b4Tz0wCPn6kKF6qQSD/+laZtjp69CjgrZX1l8ZNcO/VRFDLqig2k3XLKjVsZDJbphj8UzIy+e9PMwyIUFlWyyiSfbdmzRrAq/ckif0QO82xkKhlVRSbUWVVFEvIem7wa6+9BkTuxwqR+a0hcH8VJSpB5AQLalkVxRKybln9pR/9yI7SiqJERy2rolhC1i2r1P9p0aIF4KWEtW/fPtuiKIpVqGVVFEsIfD1ryCmSE/tZQvsueTQpQlFsRpVVUSwhdMq6cOFCmjVrRuXKlcnJyeHiiy9m+vTpxHHXFfIKqBlj8v275JJLghYt1NjSb6GpbihUq1aNMWPG0KhRI0qXLs3q1asZNmwYJUqUiNhgSslPbm5uxCZRR48epWnTphF1lJT82NJv8QJMocAY8wqA4zg9gpbFJowxg4A5QF3HcfYGLY8thLXfQucG+zF5NAeuBt4OWh4LGQK8FqYXzhJC2W+hVFZjTEVjzBHgBLAemOU4zuMBi2UVxpgrgWbAk0HLYhNh7rfQjVnPcBi4HCgLtAYmG2P2Oo7zdLBiWcUQ4BPgzaAFsYzQ9lsoldVxnNPA7jP/3WKMqQxMBFRZE8AYUwHoA0x0bAhKhISw91so3eAoFANKBy2ERfQFSgHPBC2IZYS630JnWY0x44HVwMdASaAtMIqQdmBIGQK86jjOf4IWxDJC3W+hU1agAjAXuAA4Tp7Sjj5zTomDMaYlcClwd9Cy2IQN/WbFPKuiKPaMWRXlnEeVVVEsQZVVUSxBlVVRLCFeNPhcjz5ptYPk0b5LHq0UoSg2o8qqKJagyqoolqDKqiiWoMqqKJagyqoolqDKqiiWoMqqKJaQ0SVy33//PQAnTpxwzx0+fBiA5557LuLaDRs2uMeyedXPfvYzAFq2bFngM4YOHZrvXE5ODgDFixdPRmxFCSVqWRXFEtK2MZXfej722GMAvPHGGwCsXLnSe6CJn4UmMiV7ba9evQCYOnUqALVr1457nwLQlLnk0b5LHk03VBSbUWVVFEtI2g0+fvw4AAMGDABg//79btuqVasirm3atKl73LhxYwD69esHFN5F/eCDDwB49dVX8wQ8I//69evda/bs2RNx7+3bt7ttZcqUKczj1JVLHu275FE3WFFsJmXL2qxZMwC+/fZbt+0Xv/gFAHfccQcA1atXd9vKli2brKwxEWsK0LZt24hzhw4dctvKlStXmNsGbh3mzJmT79zw4cNTuufs2bMLbOvYsSMAjRo1SukZhKDvonHs2DHAm1Z8+mmvbnzdunUBaNCgAQBbtmxx2+69914AfvOb3wDw4IMPAtCiRQv3mnXr1gFQrFjKNlAtq6LYTMpTN2Jh/RRyXJgWVqxY4R5369YN8KaTbLSsYlFTtaLJItZ32LBhyd4isL4Tq/mHP/zBPbdv3z4AHn30UQBOnjxZqHs2b94cgI0bNwJw+vRpAEqX9jaKeO+99wBo2LBhMmL7UcuqKDajyqoolpBybnAQLq+fZcuWAdClS5d8bd27dwegVKlSWZWpKCDudwpucGDIVN2IESNSuo8/MFq+fHnAmyoUrrzySvc4De5vTNSyKoolhHFjqnz4845zc3MB+Mtf/gJ4ecj+3GAJpy9atAiAkiVLZkXOdCJTKLGQKTI/0i+pEu3e5wo1atQAIpN7ZDpHAkoSoBo4cGDW5FLLqiiWkLZVN5lgx44dQOS46exURhmPDhkyxD139915u/alsNpGCOXEfqokMi2Uht0FA+s78cSmTZvmnpOpvR//+McAVKpUCYjeB5K4I9f4EcsqPzdv3uy2/ehHP0pFbD86daMoNhMay+pfCDBz5kwAnnkmb7PzL7/8ssDPjR07NuJnmilSljURiypj1cWLF6f6uFD13XfffQd48YtE1kr7GTduHAATJ04EoH79+gDs3r07XSL6UcuqKDajyqoolpB1N1gG5O+//z4ATzzxBAC7du1yr5Fc3liuihRV69ChQ7pF9BMqVy5VEnH9JKhXVFfdFAa/bkiwUnKMX3jhBQB69+6diUerG6woNpOVpIgpU6a4x6NHj457vaxoiLUusHPnzoAXZpdkCfDSvrQUafT1sGfjX9+aBotqPZLwIFOA4FlU8U6qVKmSdbnUsiqKJWRlzPrhhx+6x7/85S8j2s6ulwSxLeumTZsAb8V/tFKksqr/vvvuA6BixYrJim79uCvWODWN0zRRH53CZwPtu3fffRfw1rD6kaLys2bNyqQIOmZVFJtRZVUUSwg8g+ngwYNA9DzMaHzyySeAV6BNMkukNKmfc7EU6c6dOwEvBzYWacj/jYV1fScsWbIEgB49euRrmzt3LgCDBg3KpAjqBiuKzQS+njVRiypITqbw0ksvAZ5FAbj22msBL3h16tSpVES0iuXLl8e95lxeqxoLCVpOmDAhX5t4KjfeeGNWZfKjllVRLCFwy5oqMr1Tp04d95xYa3/h76KM36vI0oqaIsmYMWMAb3rQj6yHTWEaMGXUsiqKJWTdskoB5nTXRZLtDCBy24NzgViRX//4VC1qfmScCrB69eqINn/lwsLGVjKBWlZFsQRVVkWxhLS5wf5yoVKGZevWrUDkfjhSOLlly5ZJPUfu9dVXXwEwdepUwFvfCl4+bNWqVYG07OoVSnr27Bn3mt/97ndZkMRe/FNdso+NrNaaNGmS2+bf0yYoiuZbrChFkJQtayLlQv0Tyffffz8A27ZtS/gZUqwbvBC6rF+NtuqmXbt2APzpT38CICcnJ+Fn2YCsUY1V0FsCS7o+NTriCUYrtNe3b18AOnXqlFWZ4qGWVVEsIeVEfkmSb9q0acE38T0jkTpA0axlQcgu5127dnXPDR48GEhLuD2UyegBrlEtDKHsO3m35s+fD0QWh5fNp5YuXQpAmzZtMiVGPDSRX1FsRpVVUSwh5QCTBDA++ugj99zLL79c6PtIpXOAw4cPA95Av0mTJvmul927ypUrB9i5U1wi+PN+JTgXixC4v6Emmvsr9O/fHwjU/Y2JWlZFsYTAK0WEnMCDJP5SogWtqAlp/m/gfReNChUqAHD06FEgchWNFKD3r+AKCA0wKYrNWL+eVQmVNbUOiYtAKCxqTNSyKool6Jg1NqEad0kyhIxRJUk/pCmFoeo74aabbgJg7dq1AHz66aduW4i2W9Exq6LYjCqroliCusGxCaUrZwnad8mjbrCi2Ew8y6ooSkhQy6oolqDKqiiWoMqqKJagyqoolqDKqiiWoMqqKJbw/ynY2wK6C+yWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 288x288 with 9 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls.show_batch(max_n=9, figsize=(4,4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our data ready, we can train a simple model on it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A simple baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous chapter, we built a model based on a `conv` function like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def conv(ni, nf, ks=3, act=True):\n",
    "    res = nn.Conv2d(ni, nf, stride=2, kernel_size=ks, padding=ks//2)\n",
    "    if act: res = nn.Sequential(res, nn.ReLU())\n",
    "    return res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with a basic CNN as a baseline. We'll use the same as we had in the last chapter, but with one tweak: we'll use more activations.\n",
    "\n",
    "As we discussed, we generally want to double the number of filters each time we have a stride 2 layer. So, one way to increase the number of filters throughout our network is to double the number of activations in the first layer them – then every layer after that will end up twice as big as the previous version as well.\n",
    "\n",
    "But there is a subtle problem with this. Consider the kernel which is being applied to each pixel. By default, we use a 3x3 pixel kernel. That means that there are a total of 3×3 = 9 pixels that the kernel is being applied to at each location. Previously, our first layer had four filters output. That meant that there were four values being computed from nine pixels at each location. Think about what happens if we double this output to 8 filters. Then when we apply our kernel we would be using nine pixels to calculate eight numbers. That means that it isn't really learning much at all — the output size is almost the same as the input size. Neural networks will only create useful features if they're forced to do so—that is, that the number of outputs from an operation is smaller than the number of inputs.\n",
    "\n",
    "To fix this, we can use a larger kernel in the first layer. If we use a kernel of 5x5 pixels then there are 25 pixels being used at each kernel application — creating eight filters from this will mean the neural net will have to find some useful features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def simple_cnn():\n",
    "    return sequential(\n",
    "        conv(1 ,8, ks=5),        #14x14\n",
    "        conv(8 ,16),             #7x7\n",
    "        conv(16,32),             #4x4\n",
    "        conv(32,64),             #2x2\n",
    "        conv(64,10, act=False),  #1x1\n",
    "        Flatten(),\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you'll see in a moment, we're going to look inside our models while they're training, in order to try to find ways to make them train better. To do this, we use the `ActivationStats` callback, which records the mean, standard deviation, and histogram of activations of every trainable layer (as we've seen, callbacks are used to add behavior to the training loop; we'll see how they work in <<chapter_callbacks>>)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai2.callback.hook import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to train quickly, so that means training at a high learning rate. Let's see how we go at 0.06:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit(epochs=1):\n",
    "    learn = Learner(dls, simple_cnn(), loss_func=F.cross_entropy,\n",
    "                    metrics=accuracy, cbs=ActivationStats(with_hist=True))\n",
    "    learn.fit(epochs, 0.06)\n",
    "    return learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2.307071</td>\n",
       "      <td>2.305865</td>\n",
       "      <td>0.113500</td>\n",
       "      <td>00:16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This didn't train at all well! Let's find out why.\n",
    "\n",
    "One handy feature of callbacks that you pass to `Learner` is that they are made available automatically, with the same name as the callback class, except in `camel_case`. So our `ActivationStats` callback can be accessed through `activation_stats`. In fact--I'm sure you remember `learn.recorder`... can you guess how that is implemented? That's right, it's a callback called `Recorder`!\n",
    "\n",
    "`ActivationStats` includes some handy utilities for plotting the activations during training. `plot_layer_stats(idx)` plots the mean and standard deviation of the activations of layer number `idx`, along with the percent of activations near zero. Here's the first layer's plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs8AAADWCAYAAAAuNG/NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd5iTVfbA8e/JVGDoTUBhpAsiKIgiXVEUdUFdy9p2/a2La1l7wY6uda1rWRV77wUVFQsgvQzSOwy9w8AwM0xN7u+PN28mySSZZCZTkjmf58nD5K03wyTvyX3PPVeMMSillFJKKaXK56jpBiillFJKKRUrNHhWSimllFIqTBo8K6WUUkopFSYNnpVSSimllAqTBs9KKaWUUkqFSYNnpZRSSimlwqTBs1JK1XEiMkxEjIgcWdNtUUqp2k6DZ6WUikMicqQ7IB5W021RqjYQkUtEZJOIHBSRt0UkyWtdgojME5GLa7KNKjZo8KyUUkqpuCYizYG3gXuBwcApwFivTW4FdhhjPq2B5oVNRJLj8VyxRoPnOkJEponImyLyiIjscX/zflREHCLygIjsFpG9IvKo1z6JIjJeRDaKSIGIrBCRa/yOe5OILBaRXBHZJSKfiEgbr/X27eDTRWS6iBwWkZUiMrI6X79S8UpEBonILBHJcT+WuN9fW92bTHW/Bzd57fMvEdnmfj9OBtrXRNuVqkYdgWxjzIfGmGXAN0APABHpAtwEXBvOgdzXxfUiMlpEVotInohMFZFOftv1FZGf3dfHvSLylYh08Fp/tHvZDvd7cZmIXOF3DPva/W8R2QlsD9Kmae73uf/jb17b/Mvd3gIRWSci94pIotf6Te4Y4X8ish+Y5V7exn1tPygi+e5z9QvndxWvNHiuW/4MJAGDsL5l3wN8D6RhfRO/HbhHRM5yb/8GcD5wDXAM8DDwpIj83e+4twO9gPOwLsKfBDj308BjQG8gA/hURJpE7ZUpVQeJSALwLTAPOMH9GA8cdv8McAHQBjjRvc9o4DngWaAP8BnwVHW2W6kasB6oLyL9RKQBMBRYJCICvAncY4zZFcHx2mAF25dh9WI3Ad6yV4pID+B3YA7QDzgVcAK/iEiqe7M04DfgTKxr6ATgbREZ7neui4CWwGnu4wRyvrtN9uMRIA9Y4G7PeKxr9d1Y1/ObsK7tD/od50ZgDzAA+Kv79/MN0B04B+gP7Ha/jhYhf0PxzBijjzrwAKYBi/2WrQCW+S1bghXoHg24gO5+6x/wP47f+uMBA7RzPx/mfn6+1zZHuJeNrOnfiz70EcsPoKn7vTQswLojA60DZgIf+i172r3tkTX9mvShj6p6AOe6r3GZWF8gE4B/AZPc16Vv3OveA9JCHGc8UAK09Fp2ifuamep+/g7wid9+KVhfbMeEOPZE4HWv59OAtYAjgtc5EigEznU/r+8+75l+210JHPR6vgn4zW+b09yfDT38XsdO4IGa/j+tqYenu17VCUv8nu9yP/yXtcL6pixAhvXF0yMR69szYKVlYH2T7YH1zdu+m9EB39tLi+0fjDG7RMQJtK7g61BKAcaYAyLyBjBZRKZg9XR9bYxZE2K3HsDHfstmArdVUTOVqhWMMd8B39nPRSQdGAecBLwArMK6Q/sBcD9wV4jD7TDG7PV6vh3rmtkK2IJ1p6eziOT67ZcKdHGfvz5Wh9S5WL3FyViB6VS/fRYaY1zhvEYR6Yl1N+ku9+sF6AnUA74UEeO1eQKQKiItvV7LfL9D9gT2G2NW2guMMYUiMs+9rk7S4LluKfZ7boIsc1AaBJ+C9Y3VfxtEpD3wA/A+VkrHPqzerl+xPgS8FQVoj6YNKVVJxph/iMh/gTOA04F/i8gNWL1pQXerlsYpVbu9Dow3xmwTkRHAQ8aYEhH5AHionH39r2n2e8rh9e/7wBMB9t3v/vcpYDTWF9fVWGkWzwCN/bbPK++FAIhIK6xUzA+MMc97rbLbdCFWL7a/rHLOFejzQoIsrxM0eFbBLHT/294Y832QbU7E+jZ7szEmH6wBEtXROKVUKWPMcmA58KyIvIpVReBr9+oEv81XAgOB/3ktG1jljVSqFhGRfwBijHndvciBNSYIrM6fynbuZADHARuMO9chgCFYKVSfutvkALpi5RRHRERSsN7zq7Hylr2tAAqAjsaYHyI89AqghYj0sHuf3efqj+9nSJ2iwbMKyBizXkTeAl4XkTuxBj00APpi5Xk9CazD+uZ5m4h8iDUY8IGaarNSdY2IdAb+gXUreivQFmvw7x9Yd4JygTNEZAVQaIw5gNWz9bmIzMe6czQIuCLA4ZWKSyLSDmug3CCvxdOBW90Vp67HSoGqjMewUiA+cN8Z2gukA2OA/xpjMoE1wGgR+RLrvXor1ns44uAZeM2971VAS690y2xjTK6IPAY85l7+C1b81ws43hgTKj1livt1fCQi1wPZWCktqcArFWhnXNDb5iqUsViDKu7F6q36Dfgr1oAKjDFLsQZbXONefztwc420VKm6KQ8rf/ITrNuxXwKzgRvcOZLXY43U3wosAjDGfI11m/hOYClWtYBQF0+l4s1rwBPGmE1ey27ECm4zsN5X5aVthGSMWYWV9pgGTMa6Rr6Odbf2oHuzW4DNWDnOv2HlTX9RwVMOw2r/GqzBfPbjYnd7/u0+39VY459mup9vKud1GKyAfzVWKtgCrMGVpxtj9lWwrTFPgt9NUEoppZRSSnnTnmellFJKKaXCpMGzUkoppZRSYdLgWSmllFJKqTBp8KyUUkoppVSYNHhWSimllFIqTDFV57lFixYmPT29ppuhVK2xcOHCfcaYljXdjkD0/aqUr9r8fgV9zyrlL9h7NqaC5/T0dDIyMmq6GUrVGiKyuabbEIy+X5XyVZvfr6DvWaX8BXvPatqGUkoppZRSYdLgWSmllFJKqTBp8KyUUkoppVSYNHhWSimllFIqTHEVPK/bncMn87fUdDOUUlHgdBle/G0dhwqKa7opStV5uYUl3PnFEmas21vTTVGqxsVV8Hz6c9MZ99Wymm6GUioKfl21m2d+Wcsj36+s6aYoVecZY/gsYxurd+bUdFOUqnFxFTwrpaqOiEwTkQIRyXU/1lTl+QpLXADkFTmr8jRKqTCkJiUAUFCs70el4jJ4drlMTTdBqXh1gzEmzf3oVtONUUpVj6QEB4kOoaBEg2el4jJ4dhoNnpVSSqloqpeUQH6Rq6aboVSNi6vg+ZROzQFroJFSqko8LiL7RGSWiAzzXykiY0UkQ0Qy9u6NzsAiicpRlFKVlZKUoD3PShFnwfPwbq0AKNHgWamqcBfQEWgHTAC+E5FO3hsYYyYYY/oZY/q1bNmyUiczegdJqVolJdFBYbH2PCsVV8FzgsPqo3I69aKrVLQZY+YZY3KMMYXGmHeBWcCoqj6viPY9K1UbJDgEl36pVSq+gufEBOsiW+LSb8ZKVQODZlUoVWc4BA2elSLOgmdPz7OmbSgVVSLSRERGikiqiCSKyGXAEGByTbdNKVU9HA7R66tSQGJNNyCaEh12z7O+uZWKsiTgEaA74ARWA2OMMVVa61kpVXs4RNCOZ6XiLHhOcFgd6frNWKnoMsbsBU6swfOzI7uAxvWSSEuJq48tpWJGgmjOs1IQZ8Gz9jwrFX/mbNjP0Xf/AEDX1mn8fMvQGm6RUnWTiHZOKQVxm/OsAwaVihf7cgs9P6/dncumfXm8P2cTxc6Kv8+LnS427M2NQuuUqjusahs13Qqlal5cBc/a86xU/Ah2d3jY09O4f+IK+j/6a8j98wpLWL3rUMB1j05axWnP/M7O7PzKNlOpOsOhaRtKAXEWPNs9zyVa51mpuHfgcHHI9Td/upgzn59BQXHZGdHmZu63jpEX+hhKqVJaqk4pS1wGz5qTpVTsM4T/Ps7OL2bt7hzenrURsILjX1buBuDxH1aRXU6grZQqn5aqU8oSleBZRJqJyNcikicim0Xk0hDbDheRqSKSLSKbonF+W4KmbShVpzz03QryCkvo/dDPnPHcdB76biUul+GSCXM927w7ZzMPf7+yBlupVHzQUnVKWaLV8/wyUAS0Bi4DXhGRnkG2zQPeAu6I0rk9ErVUnVJx4Y0Zmdzy6ZJyt3t71iZ6Pug7T8vbszeV2a6wxErdWLj5AOnjJrH7UEFU2qlUXZIg2vOsFEQheBaRBsAFwP3GmFxjzEzgW+CKQNsbY+YbY94HMit7bn+lPc9abUOpWPbIpFUV3vffAXqZ1+/JZf2eXN6bswkoP19aKVWWaM6zUkB0ep67Ak5jzFqvZUuAYD3PERGRsSKSISIZe/fuDbltYoLmPCsV61btDFwhozJW78phxLO/4xCJ+rGVipYIUyBPEJHpIpIrIrtF5Cavdenu9MjDIrJaREZEo31WqTq9vioVjeA5Dcj2W5YNNIzCsTHGTDDG9DPG9GvZsmXIbTXnWanYd9Z/Z1TZsf1j50gGJSpVDcJKgRSRFsBPwGtAc6Az8LPXJh8Di9zr7gW+EJHQF9AwWKXqKnsUpWJfucGziEwTERPkMRPIBRr57dYIyKmKBodi13l26btbqbjXuVVaxPsIvtFzqAwvYwyTV+zSzxNVLSJMgbwVmGyM+dAYU2iMyTHGrHIfpytwAvCgMSbfGPMlsMx97Eq2Ue/sKgVhBM/GmGHGGAnyGASsBRJFpIvXbr2BFVXV6GC051mpuuPh0ZFnhv2wbKfP8+nrfFPBpq7ewx9bDrAru4CJi3dwzfsLecdrAOLWrMMs3nqwQu1VqhyRpECeDGSJyGwR2SMi34lIe/e6nkCmMca7AytoKmUkqZEJDsFo2oZSlU/bMMbkAV8BD4tIAxEZCIwG3g+0vYg4RCQVSLKeSqqIJFe2HaDVNpSKN/WSEjw/b3ribDY9cbbneXJC5B9f+X4Tpjw1eQ2XTJiDMYapa/Zw1TsLOP9/szn58d88FTm8ZyEc/J+pjHl5VsTnVSoMkaRAHgn8FbgJaA9sxErViPQ4EaVGOkRwavCsVNRK1V0H1AP2YL2BrzXGrAAQkcEikuu17RAgH/gB602fj2+uVoXZPc/FTq22oVQ8WPnwyDLL/rj/dKbdPoy+HZpyz6juDOjYvFLnmJuZxf68Iq56e4HP8r05hUDwacLX7s7h2yU7KnVupbxEkgKZD3xtjFlgjCkAHgJOEZHGER4nIg6RkKlOStUVidE4iDEmCxgTZN0MrG/C9vNpQJUMeU9JtL4LFOv03ErFBRHhon5H0qVVaadZswbJNGtg3awaO6QTY4d0Ytm2bM59aWaFz3Pm89PLLHtjpjVbocvAvMz9zFq/z2f9Gc9Z+/ypd9sKn1cpL54USGPMOveyYCmQS8FntKv9s7i37ygiDb1SN3oDH1W2gVqqTilLVILn2sIOnu0JEZRSsWXljrJl6v7z597l7pecWLmbaPtyi4Kue2vWRt5yT/utVFUxxuSJiJ0CeTXQBysF8pQAm78NfCkiL2AFy/cDM40xB4GDIrIYeFBE7gPOAo4jGgMGK3sApeJEtNI2agX7AlpYrPeVlIpF3gP4bjytS4gtffkHz3blneoQyQCq5duzycoLHqirOi9gCqR/+qMxZgpwDzDJvW1nwLsm9CVAP+AA8ATwZ2NM6NGASqmwxVnPszW4qLBEg2elYpF3yPvXAR3C3s8/Vg40qOm4IxuzdJv/OKqKefKn1Z6fi5wuz2dPec55cSYdmtfn9zuGR6UdKr4ES4H0T390L3sFeCXIcTYBw6LdPpHgYwCUqkvisue5SINnpWLSu15l4ZqnpYS9n31Bb9ekns9zgFG9jgBg4vUDGX9uD8/y5y/uU+F2vjJtg+fnSD9vNu8/XOHzKlWT/OukK1VXxVXwnOAQkhJEc56VilE7sgsqtF+H5vW5fngn3vt7/zLrXvrLCax/9CxEhL8NPNqzPKkCpe4C2XGwgDdmZDLbPaBwa9Zh8gpL2Jdb6LOd1sdV8UBn5VQqztI2wLogas+zUnWLiHDHyO6e573aNebuUd3JyivC4RAcfj1mF/Y9ssxU3RU10qtSx2fXDOCi1+Z4nm94bJRXCU0NOlRsi9Z7RqlYF3fBc4JDi7grFYu8e2afu7j8ChvB/HjTYNo2qUfjekkB19sTrXyWsbXMOofAS5eewHUf/lGhc3sHzgAHDxd50k9KvArk5hc5qZdcNk86r7CEr/7YxuUnd0A0UlG1kF5elYqztA2wRtnrDINKVQ0R6SIiBSLyQbSPnVdUmm41sucRFT7OMW0aBQ2cveUWlABw46mdmXHncK4f3on1j45iVK82DO7SosLn95aVV8Ss9fs4/dnfySssfX1LtgWe4vuRSSu5f+IKZqzbF3C9UjVJv88pZYm74DnB4aBEg2elqsrLwIJyt6qAnIJiz8/1k6v+ptgl/Y9i7JCO/HNYJ45qVp87RnbH4U6x8C91F04wHsi2A/lc9+EfrNuTyy2fLvYsn71+Hxv35THwiSlMWb0bgAWbsth2wJoKPK+wpELnU6qq6dVVqTgMnhMdQolOz61U1InIJcBB4LeqOP5+90QlT5zfqyoOX0b95ETuGXVMwED94dHH+jw/7sjGvH5lP+4+q3uZbUO56p0FZOdbXwpmes1Q+MKU9Qx/ehrbD+bzf+9ksPtQARe+OsfT46zf/1VtpNU2lLLEXfCc4BDteVYqykSkEfAwcFtVnWPFDqsGc7/0ZlV1irAd1aw+HVs08Dw/lF/M6T1a83+DSqt1pKVEr3d87e4cn+c6bkPVVlo1Rqk4DJ4TEzTnWakq8G/gTWNM2VF2XkRkrIhkiEjG3r2RTWiW485BbtUo/PrOVcqrk+3PfY8EfMvbLX9oZNRO5T8t+X1fL2NrltaDVrWMdjwrBcRj8Kw9z0pFlYj0AUYAz5W3rTFmgjGmnzGmX8uWLSM6zz532kaDash3DkeXVtaEbjPvGs4VA9IDbnPtsE5ROdfjP672eX6ooITbP1/ieT7uy6X856fV/rsBVkWP/X41pZWqKnp1VSoOS9UlOhw4tZ6qUtE0DEgHtrjLp6UBCSLSwxhzQrRO8urv1qx9Cf5zbdeQZy7qw6WbD3Bk0/o+y7sf0ZDVu6w0C1cV3sK2c6UBPllgdfhPXrGL16/sx+aswwzv1gqAPg//ApSW4IvUn1+ZTc+2jXjIL89bKX+1452pVM2Lu+BZc56ViroJwCdez2/HCqavrZHWVJO0lESGdi3be/7dvwZ5gubkILMUJic4KKrkwOWiEheHCor5cdlOz7INe/M49ZnfAfhk7Mk88/Maz7qx72XQulEqD/2pp6dqiL+5mfvp1rohTRske5ZlbD5AxuYDGjyr8OjlVak4TNtIEJwurbahVLQYYw4bY3bZDyAXKDDGRJbUHIYWabUk3zmEpAQHKYnWBCeJjsAfoT/cNLjS58ncl8dx43/mri+XBVx/22dLWLDpgOf5zyt38/7czazbkwvAryt38/Tk0uDa6TJcMmEuV7w1r9JtU3WTTtyjlCXugmfteVaqahljxhtjLo/mMV3u9+zlJ7eP5mGr3GUnt2dAx+Y8dp5VXu/kjs1ISXSQlFD1Qcb2g/kBlxeWWJOxXP1eBi9NXe9Zbs9wuMJvcKJSkdCrq1JxmLahMwwqFXuK3YFdUpA0iNqqRVoKH489GYBLTyoN/Hdm+wa2Y/q05ZvFO8rsf/4J7fjqj+1RbVNOQUnAz0B7mVYaUxWl/c5KWaJypRKRZiLytYjkichmEbk0xLZ3iMhyEckRkY0ickc02mDTnmelYk+Je5Cv/8x+saqZV04xBE5Hefz8Xjx5wXG8fmW/qJ77sjfmccWbZVMzinUgtYoCrfOsVPR6nl8GioDWQB9gkogsMcasCLCtAFcCS4FOwM8istUY80mAbSOW6HCQX+yMxqGUUtXEEzzHWM9zMHZO9PHtmzDimNZcNTCdN2ZuBGDj46NYufMQPdo0QkQ4rXsrBnRszpzM/VE7/+wNvsf6etE2dmYXhL1/bmEJW/YfpkfbRlFrk4p9mvKslKXSwbOINAAuAI41xuQCM0XkW+AKYJz/9saY/3g9XSMiE4GB+I7mrzDteVYq9pSmbcTP1Xn+PafRMDWJeskJPstFhJ5tG3ueOxzCx2NPptM9P3hSKx47rxf3fB14oGCkcgtLuOXTJWWWT1wcPF3kts8WM3nFblY+PDLg9OWq7tKrq1LR6XnuCjiNMWu9li0Bhpa3o1hDdwcDr4XYZiwwFqB9+/IHE1k5z1ptQ6lYUpq2ER89zwCtGqVGtH3Pto1Yui2bOXefSpvG9aIWPB/74OQyy9LHTQq5z6qdVh3rDXvy6HVk45DbqrpD0Jx5pSA6Oc9pQLbfsmygYRj7jne34e1gG0Q6Y1mCQzwXYqVUbCh210SOl5znQG48rQtv/jV4fvM7V/Xnw6tPok3jegD8cGPly91F4nBRCXmF1hTp9d295ee+NLNa26CUUrGg3OBZRKaJiAnymIlV89U/Ma4RkFPOcW/Ayn0+2xgTtbllExM0bUOpWGMHz0mJ8Rs833p6V047pnXQ9c0aJDOwcwvP80D5xv3Tm5VZ9tKlx0elfb0f+pmeD04mK6/IM4MiwLrdIT/KVR0iIhhN3FCq/ODZGDPMGCNBHoOAtUCiiHTx2q03EGiwIAAi8n9Y+dCnGWO2VfZFeEtwOLRUnVIx5lCB1ePZKDWphltSu/x8yxCf572PKptCcXavNlE5l12N45/vL/RZfvpz03l00ko27suLynls2w/mszcnav0mqhrE71dbpSJT6bQNY0we8BXwsIg0EJGBwGjg/UDbi8hlwGPA6caYzMqe31+SQzyTASilYsPBw0UANKmfXM6WdUvX1g1ZdP/pvPd//QEY1q1VmW2iPevb/E1ZZZa9PmMjl78xjw17c5m0dGeAvSI38IkpnPjor1E5lqo+mvOsVPRmGLwOqAfsAT4GrrXL1InIYBHJ9dr2EaA5sEBEct2PV6PUDhIcglNznpWKKTsOWmXUWjWs/dNzV7emDZIZ0rUl6x89yyeto0Pz+p6f7zyzG+2a1KvSdmw/mM9pz/zO9R/9UaXnUbWYdj0rBUSpzrMxJgsYE2TdDKxBhfbzo6NxzmA051mp2LMl6zDJCQ6ObFq1AWAss2tg9+3QlLN7teGS/kdRUGzdZbtuWGeuG9aZvTmF1dKbO2npTvq0b8LAJ6bwl/5HkZzgYPTx7TihfdMqP7eqWdrzrFQcTs+doNNzKxVzikpcpCQ6op6CEI++vPYUz8/+WS4tG6Zw0tHNmLexbOpFNHn3Pn88fysA787ZTMZ9IwLOpqjig2jXs1JA9NI2ao2kBAdFJZrzrFQsKXa6SEqMu4+jGvGX/sHr4Tepbw3IXPLgGVVy7n6P/MqAx3+rkmMrpVRtEXdXq9SkBApKdHpupWJJUYmL5DiZmrumjTm+Hc9c2BuA4d18a+P/dutQpt4+jMb1kvjvJX2q5Pw7swvYmZ3Pa79vwER4j//pyWu44s15VdIuVXl6Y0gpS9xdreolJVDsNJQ4tfdZqVhh9TzrlTlaxhzfjjtGduOFvxzPVQPT6duhKd//axDN01I4ukUDAEb2PIIL+x7Jv0f39Oz3/t/78+C5PTzPxw7pWKHzD3h8Co//uJopq/f4LP9p+S6WbfOdU2tu5n4enLgcgJemrmfGun0VOqeqHpF+IVIqHsVdznNqkvV9oKDERZr2ZCkVEwqdLpL0/Ro1CQ7h+uGdAXjw3J4Bt0lNSuApdw/1hf2OosRlSEtJZGCnFjz03UoA7hl1DBOmV7yi6N/fzWDTE2d7nv/zA6uG9NLxpWkjl0yYC8C9Z5cG7XtzCnl9RiZ3juzmGSipap5+vVXKEnefSvWSrGllC4o1dUOpWFGsaRs1KjUpgbQUqy/FEeUp0mevt3qS7/l6mWfZrZ8uLrPdRa/N8fx81TvzmTA9k39+sJBNUZ6cRVWO9jsrFYc9z2mp1ks6eLhYR30rFSMOFzlJ1gGDcek/k9ew+A3fPOZfV+0ps93irQc9Py/ffsiz3dJt2cy/dwSHi0pITUzg8R9XcXSLNC49KfjASFU1NOdZKUvcXa3aNrbqxO7Mzq/hliilwrV61yE6t0wrf0NVbY5t18jnuV2De+yQjky6cVDYx/EOiitiT04h2fnF9HhgMk/8tJrXZ2z06cVW1UtTnpWK457nvEJN21AqVhwuctKsgU7NXVssf2gkSQlWN+Px7ZuwaMtBJt04mNenZ3LziC4kJjj49dYh5BU6Gf3yrCpvz+M/rALg0wVbPcvW7MphzoZ9XHTiUdRPjrtLWa2kdZ6VssTdJ47DfV9JRwQrFRuMMeQXO6mXnFDTTVFudv4zwKdjB1DiclE/OZHbR3bzLO/cqiEA/zq1My9OWV+l7fl97V7AGhBu31Qc+fx0AMZ/t5Inzu/FJSHqW6voMZr1rFT8pW0kuAe7ODV4ViomFDldGGMNWlO1T3KiI2TP7m1ndCPzsVFlJl6Zd89pnNu7bVTasDO7ACjtHPH36KRVER2v2OnScqYVoDnPSlniLni2P1x1im6lYkNBkRXEaPAcuxwOoXG9JDY9cTYL7xvBH/efTutGqbwQYCKWji0bVPg8dhDtLyFBWLApi4WbD4R117HLvT9yzoszfZat2nmI9HGTtLpHObRfSqm4DJ6tf/UNrlR0icgHIrJTRA6JyFoRuToax813l5W0a7Sr2NY8LcWTvy5+XZWZj42iQRXkJx88XMyFr87hgldmc+tnS0gfN4nXg9SnzikoBmD1rhyf5V8u3AbA5BW7ot6+eKE9z0pZ4u5q5Unb0J5npaLtcSDdGNMI+BPwiIj0rexBcwtLAN88WxU/Xrui9E/E4RDOO74dADee1oUNj43i9zuG8dttQ8vsd4dXfnUkvl60HYBHf1jF7A372LQvj/wi6wvaN4u202v8z2X2ufaDhbwxcyOgKX/l0d+OUnEYPHvSNvQDUKmoMsasMMYU2k/dj06VPW6eO3iuih5JVfNG9jyCh0f35Kk/HwfAVQPTmXfPadx6elcSHEKH5g1Ib142lePaoZX+0+LBiSsY9vQ0jnngJ35ZuZubA0zOAvDj8tLeZqfTsHqXlcIxN3N/pdsQCRFpJiJfi0ieiGwWkUuDbDdeREDhXaIAACAASURBVIpFJNfr0dFrvXEfw173RpRaqHd1lSIeg2d3z/ODE1fUcEuUij8i8j8ROQysBnYCP/itHysiGSKSsXfv3rCOaQfPdplJFX+uHJDOhf2OAqxUjtaNUn3WJziE7kc09Dzf9MTZOBzCkgfO4MebBvPXAR0qdN7N+w97fv7Hexll1qePm8TU1b4TtpS4DLPWW0HzT8urPYXjZaAIaA1cBrwiIoHnV4dPjTFpXg//PJXeXuuikmKlaRtKWeIueE5wv7vzdXpupaLOGHMd0BAYDHwFFPqtn2CM6WeM6deyZcuwjqlpGwrgo3+cDEDjekmeZY3rJ3FMm0Y8NPrYCh2zKIyKGte8v9DnudNlaqTUqYg0AC4A7jfG5BpjZgLfAldUe2NC0q5npeIueHZ4vSItRaRU9BljnO4L+5HAtZU9nh08N9DguU6zvzz9M0i6xo2ndfF53r5Z/aic1z/Afmnqeh5xl74TseqQV9O1pCvgNMas9Vq2BAjW83yuiGSJyAoRCfQ+nC4iu0TkKxFJD3bSSO4WacezUpaoBM/h5mm5t71ZRDLdI/Z3iMhzIhK1q6Z3HdDKTgurlAopkWjmPKdoqbq6LDnRwaYnzubaYYH/pG49vSvz7jnN87xLK2s691cvP6HMtpUph+ftuyU7OfuFmXS+90fAuqZs2X+YLxZuo9f4yVz/4R9ROY9bGpDttywb606Pv8+AY4CWwD+AB0TkL17rhwLpQHdgB/B9sOtspHeLNOdZqej1PEeSp/UdcIJ7xP6xQG/gxii1w5O2AXDxhLnROqxSdZqItBKRS0QkTUQSRGQk8BdgSmWPnVtopVhp2oYqj3fO7VMX9uaJ83sxsucR9OvQ1Ge7u886hj9FYYKWfbmFrNx5CICCYidjXp7FkKemcvvnS8gpKGHSsp2VPoeXXKCR37JGQI7/hsaYlcaYHe67QLOB/wJ/9lo/3RhTZIw5CNwEHI0VbFeK5jwrZal08BxpnpYxZoP7DQ3WXSAX0Lmy7bDZAwZBy9UpFUUGK0VjG3AAeBq42RgzsbIHzisswSFQTydJUeWwK7Kcc1wbmjVI5pL+7RERuh7h2zmbmuTg2Yt6R/Xc475cGnD5LZ8u9tSOrqS1QKKIeOen9AbCGf1uCJ1VUd76sOlVVano9DxHmqeFiFwqIoeAfVgfDq+F2Dai0fsO/WasVNQZY/YaY4YaY5oYYxoZY3oZY16PxrFzC0tokJJYZkINpfw1SElkxp3DecYvMD7p6GY+z5MSHCQmRHdIzzeLdwRc/vWi7Zz5/AxW7jhUqeMbY/KwBuE+LCINRGQgMBp4339bERktIk3F0h/r7u1E97qeItLHfYcoDXgG2A5ENod5AKJZz0oB0QmeI8nTAsAY85E7baMr8CqwO8S2EeVjJWj0rFRMySss0ZQNFbajmtUnJdH3LsXoPu2Yf+9pvHTp8QB0be17+QmnF7phJf4Gtx/MZ9QLMyq8v5frgHrAHuBj4FpjzAoRGSwiuV7bXQKsx0rpeA940hjzrntda+BT4BCQiZX7fI4xJird4zVRiUSp2qbcTwsRmYY1+CCQWcC/CDNPy58xZp2IrAD+B5xf3vbhcGjvlVIxJa+ohPrJmrKhKqdVw1TOOa4t5xxXmuu8dPwZZOUW0b5ZfVqkpdCpVRqPTVrFKZ2bc+/Xy332X/bQSP700kyWbvPvC6o+xpgsYEyA5TOwOqrs53/x38Zr3RSgYtMzlkMvr0pZyu15NsYMM8ZIkMcgKpenBVEasW/zDp6Pbecf0yulapuiEkNyogbPKvoapSaR3qIBDocwpGtL2jWpx8uXncCl/dtzavdWZbb/9oZB9GhjXTeiVQov3mi/s1Jh9DyXxxiTJyJ2ntbVQB+sPK1TAm3v3uZbY8weEekB3A1Mrmw7bN5pG20b14vWYZVSVaTE5SIpQbu0VPUREd7624kAzM3c73Pd2LDXyo7ofkRDtmQdDrh/MNsP5tOuSfxed/RdqpQlWiMqAuZpAQTI1RoILBORPKypfX8A7olSO3wGDOosg0rVfiVOQ6KOVVA15OSOzTkxvXTAYWGJNSHKI+cdy5MX9OLzfw4A4PQercs9Vkpi3M07VoamPCsVhZ5nCJ6n5V7nn6t1VTTOGYyIsOmJs7lkwhwKi3WGQaVqu2KnK+qVEZSqqBuGd+alqetp1TCVi09sD8C6R88iKcHB+j05zFq/n0P5xfTt0JRL35jns2+LtJSaaHK10Yo4Slnidoh7alICWXlFNd0MpVQ5SlyG1CQNnlXtcPvIbtw+0ne8XZL7y13nVg3p3Mqq5FFXry9abUOp6KVt1Dr1khLIL9K0DaVquxKniwRH3H4UqTjlP538BSccWUMtUUpVt7i9YjVrkMy+3MKaboZSqhwlLkOS5jyrGJOSmMCU24bSvEEyABf1qxvBs/Y7KxXHaRutGqZy4HAxxU6X55abUqr2KXEaErXahopBHVumseDeEazceYhj2zWu6eZUOU15VsoSt1FlsnvUs9Ol35OVqs2KXTpgUMUuh0PqRODsoZdUpeI3eLbvAmvwrFTtVuLUtA2lYoEgGjsrRRwHz3bRe5eODFblyCss0cGlNahES9UpFRM0bUMpS9xesex6lK4Klnrec6iAV6Zt0LI8bs/+spb0cZNquhlVoueDk+n/2K813Yw6q9hldIZBpWKEXhOViuPg2b4WV7Tn+YaPF/HkT6tZvSsHsD4w0sdN4tlf1kariWH5dMEWlm/PrtZzBvLCb+sAcMVYGkyJM7xvTzkFJQGX3/75Ev7xXkal2jD65Vk8PXlN2NsXFDvJLQzcnnjkdBmf6ZGVUrWTvkuVssRt8OxwX4ydfsGz02V48bd1ZB8uDrl/njt4sXOm7X9fnrq+wm16/te1EQffd325jHNenFnhc0Yifdwknv05dJDn//usbgs2ZZE+bhJLth4sd9vPM7bS+d4f2XbgcIXP98XCbfyycrfn+eb9eRw8HHpyhP25hT5fMpZsPchLEfzdnPviTI59cHLkjY1RxU4XiVrnWamYEFvdJ0pVjbi9YnnSNoxh4eYDrNhh9d5OWb2HZ35Zy8PfrwzrOHasaMdClfnm/fyv6zw9uJFasyuH8d+u8ARlRSUuXpqyjoLi6ObqvjAldJAXzQGYV709n273/RjRPr+t2gPAGzM3UlxOr/LExTsAyNybV7EGBjD0qWmMeHZ60PV7cgro+8ivPP9rxe5QrNp5iHV7civavJhU4tS0DaVigeY8K2WJ2+A5wSvn+YJXZnP2C1bvrR1slhd02h8Sxv09u7oHHk5cvJ0XvQLtaz9YyDuzN7Ely+pF/Xj+Fp7+eS0TpmdWa7sq+nvYtC+vTK7c1DV7KSyJLCnd/v/4bskOnvhxdcht7bY6Qnzih/oysG53TsDl3pPv5BaW8Owvayl2unhl2gbGvDQLgMkrdgfctzxn/XdGhfaLZSVaqk6pmKEpz0rFcfBsp1AW+QVnnoCqnBxL8etjtver6m/eTpehoNjJTZ8s5hmvFI96ydZUsHZubmGJ0/08dPpJMMu3Z/P+nE1k5RVx9bsZ5aYi2K+7z8O/MGPd3oDb7M0pZOa6fWWWz8vcz7Cnp/F5xrYKtTWYRVsOAHD/N8u5+6tlDHpyis/rKA2egx/juSBpNAXFTvbl+v5OfltVNiB+9ue1vPDbOr5dvIMnf1rNjuwCwAoI/c3N3M8bM6r3y05tZ4yhWEvVKRUTRLuelQLieIZBOzj+z2Tf3kk7oPIMKHSZgLND2Z8RU1fv5eHvVvLWVSdayyuQuLEnp4DDheGlV9zw0R/8uHxXmeUNU63/KjtYtntTSyqYRmHnUe/JKeTXVbt5b87msPYrKnEx/tsVPH7+cbRIS6ZjyzTPuotfm0PmvjzevupElm3L5sbTugCw1t2Du3T7QS468agKtdfD6+XaL/39uaVtn71hP6N6tSG/yMnczCwg9Af+gk1ZAZef++LMMukT8zaW3dbOjf9o/haf5YH+Wy6ZMBeAC/sexepdhzipY/Og7QLIyiuimXvq33hl9/xrz7NSscFo1rNS8dzzbAVM3y/d6bPcTpO117/y+wbOeXEmf7h7MW12uPXcr2vJ2HyAHQfzPSvSx03iug8Xht2W/o/+xrCnp4W1baDAGSA1yep5zitykp1f7KlOUNkc5GVhVvLwDj+NgYtem8Opz/zus03mPiu3+Kq3F/gMjLTbmBCFXgsT5Geb/f/6lFd1i2CnnbpmDxv2Bs4vDpR37H2crVmHefzHVZ4vYws3+/79BOp5tl359nwunjCXYqeL9HGTgg5C/ef74f+NxSr7y59W21Cq9tN3qVKWuO15DtaRZQ+4s3um7TJwOw8WQPvgx9ubY+W52h8ePywLHORWlUR3e6//8A+KnC7+PbonEHnwvGTrQT726iWdtsZKwSgvj01EPBvZQXK4nO5jl5cqY9uTU0BaSiL1k8v58wzQaDsIO5hfmnIRLOf5qrcX+DwvLHFyz1fLuX54p4Dbewf/f31rPpn78ujXoWnAbZ1Ow+wN+2gQ4DXYlULsgPupyWvo2rphmTz8rZWoEhIr7OBZBwwqFRs051mpOA6egwVMTuPbCxq0M9RvRYkzspxnuxJEUpRuR+e50z6K3Md1VGAGxa1Zhxn98qyA6wLdilu96xBdWllBXbAg3ekyFDtdnp7xQFx+Pc8Fxc6Av5eXp67njB6tOf05q5rFpSe1Z+Ki7Xx53Sl0a90QEfEZdBioRYF+3eF2an6xcBtf/rGNQ2HkkdtfIDL8epxtJS7Dpa/PC3kMO40D4B/vZTCws28aR3nVRKqTiKQA/wNGAM2A9cA9xpjIyqX4setwa6k6pWKAfsdVCojjtI1QwR6UBp/esWd+kZPVuw4BZT8j7B6y8nKeDxdZObCDnpxC74d+jrTZQc3J3O/z3G7Hx/O38vWiwAPxPl2whZs+WcTCzVnsOJjP4P9MDXp8/xh87e4cznx+Bs/+soZTn5kWdL+x72XQ/f6fQrbdc2ve3bvY/f6ffNJernxrPku3HeSpyWv486tzPMs/mreFvCInZz4/g9fdA+282xnoe0OgL03+Oc+PfL+SUwOk0dz79XKA0hQdL3mFJRyOYArvilQl8f/bqmVThicCW4GhQGPgfuAzEUmvzEGLndrzrFQs0Y5npaLY8ywizYA3gTOAfcDdxpiPytknGVgKpBljjoxWWwAKigP32gWrwCACN36yiF9W7mbFQyPL7Od057CG6nn+bMFW7vxyKW/+tR+7DxUG3zAKvHuK7/5qGU3qJzO8Wyufbe76chlg1TtukBy8Zxh8g72CYif3f2MFkou2HAz5Wn5bvcezT3nH9k578C7lNn3tXrLdVTKy8wP3+j72w2pG9jzCZ4Dksu3ZzPP7UmGnbYT6kvPGzI1B1wGs2HGozLKeEU5aUpGBnP69//lRruFdGcaYPGC816LvRWQj0BfYVNHj2rnhOmBQqdqvIgPmlYpH0bxivQwUAa2By4BXRKRnOfvcAeyJYhs8ggVzdum6lEQrmPTuIMxwV14oLHGVCZLtHjLvxUu2HiR93CQ277du4d/55VIgcFWGaLPTSMD6ouCfvzvGLz0jr5xeTO/fw2u/Z3peQ7g9qA99F3zSGWcYg8LCOcvQp6bxzuxNPsvenuX7PNAdhwtemV3tvbjRmEymNs+ELiKtga7Aisocx/47TtQBg0rFhlr8uaRUdYlK8CwiDYALgPuNMbnGmJnAt8AVIfY5GrgceDwabfBXUFI2WCpxuvhuiTXrXH2/nlih9Ja/02UCpG3YPc+la65xV0MY+tQ0n21TE0P/Wu1axIu2HOClKet4YOJy3p+7mcd+WBVyP2+B8mHnb8wiO7+YohIXi8OYvtqbd6+n97HDzT742K9Umze7gkjIvOgKjkLxD8iveX8hTpchO9+3RvOO7LKpGFUp0slfAGat31/+RrWAiCQBHwLvGmNW+60bKyIZIpKxd2/geuDeoj02QClVdbTMs1KWaKVtdAWcxhjvGSeWYOVHBvMicA8QMqoRkbHAWID27UOUw/ATKG2j872lY5sO+gVXhtI86I/nb+GPLb7Bp91Dluuu62vtUxrwTV1T2oFeXlWJPg//wsL7RnDe/2aX8yqCK3aWDTYvem0O/dOb8dKlx0d8vCVbS0vW1fP6YlGZqaKLnS6WbjvIqp1WGkSj1OB/bsu3l02VCMekZb6lCAtLXHS978cyPb9b9h/myKb1SKqmgWn+k/PECxFxAO9j3WW6wX+9MWYCMAGgX79+5X4jCueuhFKq9tA6z0pFL3hOA/wLBmcDDQNtLCLnAYnGmK9FZFioA0d6MbYVlpMv+sHcLT6pD6t2HvKUo3s2wKxzgXJYvTtLf1+zl3ZN6rH9YD6dW6WV2dafPVNgRQULzuZvymLlzsgD0ZnrS2cG9O6Vz8oLPfNgKF3u9S3EcP/EFVzYr5KTpIQhUMrEVe9YaS2DOreo8vPHK7Fuu7yJlZo1yhhTsektveiAQaVih75LlbKE1Q0nItNExAR5zARygUZ+uzUCcgIcqwHwH+BflW18KKcd07rcbT5ZsJWfVlgpBf49mP4CBZHeMdo7szfRpH5SmeXBVLanbfvB4DWA/+aX/xypeiHSKypre4BKFtXJ+0uCitgrwDHAucaYqPxHegYMaqk6pWKC1nlWKszg2RgzzBgjQR6DgLVAooh08dqtN4EHE3UB0oEZIrIL+ApoIyK7Klv2ylv/o5uxbPwZYW+fuTf0xB/eM9bZ/PN07ac3frzIs+y3VbsJZFGEOcn+PssIXJ4uGrxTUKJtzobYyOtVvkSkA3AN0AfYJSK57sdllTmu3fOcqD3PStV6IjpeUCmIUtqGMSZPRL4CHhaRq7EusKOBUwJsvhzwvnd/CvAScAJQ/gijCDRMTYrm4coIJ6Xh7+9mBFxeXItzYr3LyEXbfe4SeCq2GGM2UwV3bUt0wKBSSqkYE80r1nVAPazScx8D1xpjVgCIyGARyQUwxpQYY3bZDyALcLmf157CtlXMvl2tIvfaFX25d9QxNd2MKndGj/JTj2KdPZZAS9UpVfsJvrO8KlVXRW2SFGNMFjAmyLoZWIMKA62bBkR1gpSa4l2Jozy/raq61Ih4N7LnERQUO3k0gtJ+0ZLevD6b9gfPN4+m3kc1qZbz1CS7VJ2mbShV+2mpOqUseq80irZkhR9U/byy6lIj6oKqLm024phWAZeHEzinlFPnO1xtm6RG5Ti1mdPT86wfRUrFAu13VkqDZxWjEirZBVKVFUVev7JfpY/x/t/7M6ZPuyi0pnbTAYNKxQ59lypl0eBZxZQxfdoC5U9EU57UpNB/+oEm2fH27zHHBl3nP3tlMA/9Kfjs9YO7tPSZzTJe2bn/OmBQqdigKc9KafCsYkw4NbTDYU8V3q5JvYDry5tF68K+wdP0ww0Ek6OU3hHL7ImKdMCgUjGgDnyhVyoccX/1/uDvJ9V0E6IqVmZii1Zg2DAlkV7tGnueXzGgg8+6irKD55M7Ng+4/vKTOgRcfsmJVpXFlEQHI/wm4klLSWR4t5Yc3bJBWG04vUdrjmoWOHivK4q1VJ1SSqkYE/dXrEFd4ms65h9vGlzTTQjoPxcc5/O8TePIB7s1rmfV5W7dKMWzrF5yAh/+4yQeGXMsX147gBPTm3nWPTymNO2hRVpyROeyB/XZwZu/1CA50Tec2plNT5yNiPDSpcfz8y1DPOuOb9+Et6/qH1Y+9eSbh9AiLYUZd54aUbvjjadUXYx8KVSqLtN3qVKWuA+eAabdPqymmxA17ZrUD2u7ji3C6/2MhiFdW3Li0c18llXkQ/byk9tb/57Ugdeu6AvAnpxCGqUmcfnJHejbwfccxSWlqRVXDTw6onPZ1TqKwpysxiHwv8tO4Mimpb//1KQEurZuWGbbcFIQuh1Rul/zBqWBf+tGKbx91YlhtSke2MFzVVdPUUpFj9Z6VnVdnQie01s0oGFq1Epa16h6YQ5GKy9t4u+DIgs2Q7m431Ehq1/cMbJbWMcRd8htgB5tGpW7fadWpaXD7YD12Hbl7+etyKvnuV+Hpp6fDYZTOlkpHW/+tR8z7jqVUb3aBDzG8e2teswO9++gvIF+/sH1Mxf19vw8/tyeDO8WuExePPLMMKil6pRCRJqJyNcikicim0Xk0iDbjReRYhHJ9Xp09FrfR0QWishh9799otO+aBxFqdhXZ65Yy8aPrOkmRF3HMHNr/X157Sncf06PCp93zt2n8sf9p3PWsUd4lvl/qNoBZPtm9bl+eOdyj/nGlf08xzAGGtcvf2r1vl7BbqI7ZzbcesH+Pc9n9GjNF9eewvnHW+XhUhMTmHBlP77/1yBOO6Z10IGFAP861Xp9wS4s/p2qVw1M93k+rFsrVv/7TDY9cTZnBQnQ41WJlqpTytvLQBHQGrgMeEVEgpXl+dQYk+b1yAQQkWRgIvAB0BR4F5joXl4pdgdHtAZuKxWr6kzwHI5rhnaMKKc4VEAVTdcM7Rhw+YleaQxLx5/hsy5U76d30OnvmCA9vnaJOIA2jevRzCvVQKR0sgvPsqBnKHXnmd345ZYhLLh3BCN6tOa849uRnODgT33akpZs3Sm4upwecjs9xR5IGc6Ayu5HNOSyk6wUkaOa1fe8BoCHRvdk/Lk9GNCpOWkpiRzrNVgxGPsOpveZv/jnAM/PmY+f7fl5/aNncU+AqcWD5VjHu2ItVacUACLSALgAuN8Yk2uMmQl8C1wR4aGGYc0e/LwxptAY8wLWx1OlB1jYb1P/z3ul6hq9YnkZO7hj0OAxkEDb3jKiK+/9X/9KtePU7r637e8+q2ywBb49nY1SfXtqK9KPN/7cHjxzYe+A65rUD91p4fTPgQujAX3bN6VL64a0bGgNEOzYMo21j57F0S0a4HAIGx4bxb1nB37tNnsAoT3Y0Dtd5Zohgb903DyiKxf1O4qfbh7MYPeAUrtHpWFqEn8beHRENZY9wbPXPv3SffOz7zqzO60appCY4KgT9ZvDpaXqlPLoCjiNMWu9li0BgvU8nysiWSKyQkSu9VreE1hqfBOTl4Y4TtgS3Hf2XJrzrOq4+EgEjpK0CPOina6yg81uGtGl0u04OsBgv2cv6k2LNCvIvPus7qQkOli9Kyei49539jEcEaIKxt8GHs22A4Gnn+7RNviXCmPKDiBpkZZC5t48LnaXdgu4XzntDWcQ2cNjevKPIR3ZmZ0PlAbBANcM7cRr0zMBePnSE2iRlszFE+bSt0NTRITuRzRiw548a79KxG726wjV3GuHdeLaYZ0qfpI4Zec864BBpUgDsv2WZQNlRyXDZ8AEYDdwEvCliBw0xnwc4XEQkbHAWID27duHbKD9JbdEe55VHafBs5dk9z2pu87szpM/rS53+5TE6N1q79wqjfV7cgFo4B4UOLhLCx47rxcA559QOinHNUOtIOzur5YGPNYJ7ZtQ6FdF4tbTu3L14MA9sd7sIKZZg2Rm3jUcgE37DnNMm4bc+UXg80HZ30XDlEQ2PDYqZEAZjd6LlMQEOrdKY8fB/DLrvM999nFWLvGmJ8722caeDKUywbNdq/mkowPXjFbBFbsMiQ7R3nilIBfw76VoBJTpJTHGrPR6OltE/gv8Gfg4kuO4jzUBKxCnX79+IT+U7ZldnU4NnlXdpmkbXuwL+DVDOjL9juHlbh9qiuZQ/AM4gF9vHer5uUdbK8/2vOPbeXJyAwsccHx6zYAyy248LbwecbtqhgD1kxOpn5xIj7aNQgY3Ilbu8MuXnsDT7rQPESsQD7VfuIP7IiFiBc2Du7QIawrvQCkXkep+RCNm3DmcqwdHr4JJXeF0GR0sqJRlLZAoIt4f1r2BFWHsayi9IKwAjhPfD7XjwjxOSHbPc5k0PaXqGO15DsDhENo3D11PecptQz25uuUZ06ct3yzeEfb5zzz2CL67YVC5ZdeCxXuRDr7qfkRDHrCrb0joY4dy9nFt+GXl7nK3u2ZoR+onJfqUhosme4BebmFJudval4DKhm+hv+SoYIqdLi1TpxRgjMkTka+Ah0XkaqAPMBo4xX9bERkNTAcOAicCNwL3uFdPA5zAjSLyKvAP9/IplW1jgidtI7z6+ErFK71quQ0OMBNhqOA4JYLqCIkBgtmP/3Ey390wKOg+vY5sXKne0HB2zbhvBJNvHsJPNw/hlM6Rz8QYqPOhNPc5eANaNUzlphFdwuoZroxQtadtdns1baBmlDi151kpL9cB9YA9WCkY1xpjVojIYBHJ9druEmA9VirGe8CTxph3AYwxRcAY4Eqs4Pr/gDHu5ZViB89abUMBuFyGP7YcqNE2bD+YzyJ3G/bmFPL4j6uq5e+zzgfPvY+yJriINHgqL+4rL6VjQKfmdGpVfbMABtIiLcVnpjtfFQtoPKFziN3DKSdX0fN6i+S/tCrCt8tOak+jOJmcp6qUuFwBv1wqVRcZY7KMMWOMMQ2MMe2NMR+5l88wxqR5bfcXY0xzd33n7u5ydN7HWWSM6WuMqWeMOcEYsyga7dPgWXmbMCOT8/83mzkb9gfdZt3uHN6bs6lKzr/nUAEDn5jCef+bDcC9Xy/jtd8zmbl+X5Wcz1udv2pd2NcaiOddxzgc5fVqXnFyB/7z5+M8z5+8oFeZbRyV7O209+7vVxbNWlfBY0fwmRio+cFS4U5ML03RuKhf8Aoc0RROBQd7eu2hXVtG/fyPnteLpXE4OU80FTsNSVppQ6mYkKjBs/Kyxl3xK9CAfds5L87kgYkrgk7pXuJ08dPynQHXr9iRzfo9wauK9X/sN5/ndqEEl8vgdBl+Xbm7yqaSj1rwHO60ou5tQ04tWp0uP7kDGx8f5VPNwvbOVSdy6UmlpXvSvfKgI005uPjEsiWAKpspYO9/Tu+ys9KlpVSsx7N5Wgoje7bm1ctPCLg+2auXMPDfpDsNwm/pJ2NLBzFWxYQggd4g4Xw5OaZNI5Y87ZpDYAAAGF5JREFUeAYX9C37/6+qXolTe56VihUJWqquVtuZnc/y7f5VCqMjc28ub8zILHe7lTsOsWxbaRvsgDbYF67Xpmfyzw/+YPKKXQBsO3CYA3lWhtHZL8xkxLPTw2rf5v15ZOcXe56/Mm09V7+XwZTVe0gfN4n0cZPo9eBk0sdN4r5vlnH9R6XnrIhoXrUimVYUgkwtWh3Oc0/BbAeCwVI2erZt7CkVB/DVdQM9P4cTmNVzB4kN3bfuJ904yOd4Fe4dDrL/6D5taeXO0/7vJX24Y2Q3RhzTmrYhajv7S3AIr13Rr8wkHwAfXX0SU24fWmZ5oFfh/+upiTq+4Z7SnmBFVb8Sd6k6pVTtZ3+OuzR4rjHfL93Bha/OLrN824HDDHh8Cue8ODNq5zLG8FnGVvKLnFz02lwembSK/CIne3MKMcZ4rv13fbmUBycuB2DUCzM496WybShylh1kujM7n7W7rZ7lrDwr8B305FQGPuk7tvX9uZt5d/Ymflm5mwcnLsfpMmz36+0e+tQ0Fm89CMD/vbuA2e5Ukr+/m+HZJsddROCDuVuYtHQnG/flRfor8YhKQqbXtKLHGmNygZkiYk8rOi4a54imcWd15+tF2yPez3tK6nAGo43q1Yad2flcfnIHwArGe7Ytne45UMxwfPsmEbfL9t9Ljvf83KpRKtcP71zhYwUSzqDC8u6QVGegpIMAaz8dMKhU7LB7D1fsOESX1sHGy6iqdMNHVvq602V8OqV+DaPSVaSmr9vHnV8s5dVpGzhcZAWex46fjNNleO2Kvp7tSlyGd+dspkGIO95FJS7qJ1v/2nM8DHi8NEhetv0gS7ZaFcYOFzn576/rPOvu/2a5z7FOO6Y1V741P+i5jMETPIeSkljx/uNo9TxHOq0oBJ9atMrZpdzuGdU94n3t4C+c6loJDmHskE7UTw78B+Xfe710/Bl8MvbkiNtU0yU3TYCfA/WqT7ltKHPuPq1K26IBc9UQkRtEJENECkXknWgdt8TlqpJ630qp6Mty306/+dPFnlld49k172eQPm5StZ7T5TK8OXMjb8/ayDuzNrJiR+A0jCK/idA+X7jN8/O63Tm8PHV90FSJ2Rv2kT5uEmNenkV+kZO8whIKS5wAHC4qYbZ7wN3Bw9b/d+a+PA4XWevtY87LzOIrv07I/03b4Pl5/Z4cpq/d63l+w0eL2Jp1mDOe+53u9//EEncvse3j+VsZ/fIsz/Pnfl1LMC9PXR90XSSSKxE8R6sUQETTgRJ6alEfkUwdGq4EkYATlYTD4RDw+8ZXUf5xXqPUyNIH7P2rKiG+IkonHSm7rmPLtLILo3XeIMsfPLcHAzrpzH9RsAN4BBiJVUorKoqdpkqqryilou8v/dvz0HfW5IbLtmXTpnHUPgqqhDGGNbtz6H5E6DkTgpm8Inhv7updh3C5oEfb0mO7XKbSJVinrd3Dv79f6bPMO14Rsa6zhSVO6rlnIzbGsGLHIc82pz9n5Qm3a1KPUzo3p/+jv/Hcxb0573hrbM93S6x5JxZvPcgPy3Zy2+dLOKZNI368aTD3fLWMbxbvYPodwykJMZPkW7M2hnwd/rnKM9fvY/B/pnqez1hX8YoY8zZmVXhfb5WZJTqssFtEpomICfKYSeTTga40xuwwxjiNMbMBe2rRQNtOMMb0M8b0a9kyShURIvzbfvS8Y+nmvkXl8ASs1r8P/aknl51UsaC+sr2k9t41FTq3bWJ9cNamnGH/3+hVA4+u8AenKmWM+coY8w1Q/r2wCGipOqVih/dg78pWi6oOH8zdzJnPzwhZSi3S463fY5XbPvP5GYx6YYZn3dasw3S85wcmLi7tjbUHqt362WLPILhip4uFm7M8aRAFxU6y8orIyiviho/+YM+hwoDn3rL/ME9PXuN5npVXxNTVe1i185CnV9jfvI376f+oVZHilk+X8K+PF5E+bpLP3Wp7MrFVOw9xqKCY9Xut13fgcBF3fLEkot9PJF6KUu9xZVQmbSOsnmdjzLBQ6905z4ki0sUYYyeqhDutKPhOLVrlIn3PX3ZSBy47ycpbfuy8Xjz2w2rPh8hfT0kH4MN5W6LZxLDUdIrCXWd1o2+HppwSoGc3Bj5XVRWI9E5RsVMHDCoVi0J9xhtjyM4vpkn95OAbeVm4OYuWaanlzuzr7XBRCYkOR8hb78u3W72xm/fnRXQHcm9OYZlJ0pwuw33fLKdhSiLLHiotQfrOrI20bpTquR5/v3Qno/u087kj/NUf2xnQsTmjerXh6Z/X8PasTYDVo/y3t+czNzOLsUM68v3SnWw9EDgd5pbPFrNwc+mEJLd8upgl7qoWj54XeF6Jj+dv9Xlu9zh7e/Db0jDtuPE/08c998XD368k3seFVubaE5UuH2NMHmBPK9pARAZiTSv6fqDtRWS0iDQVS3+sqUUnRqMt4ajMpfr8E44k474RNVI9IpiaytpISUzg7OPa+ATxxlOqrnp/P03dH9JHt6jZiWfqukjvFJU4XRFPJ6+UqnmhgucP5m2hz8O/kLk3N/hGwK7sAvblFnLBK3MY8pR1S//Jn1bzjldKwPUf/cG0NXvK7Nvjgcn0e+QXT0/vnpyCoOexL5H57h7atbtzfHqIvf20fCcnPvor871SA96YkenpKbYrNtjGf7eSaz/8w1Mi7peVu/l+6Y4y1SV+W7WHng9O9gTOAJ9nbGVupnWexVusHOCdAWomz1y3zzPIzrbEqxzcO17HDMcnC7YGXWeXe/MO1Gsje4K7cH3w95P47yV9onb+aE5/dh3wFta0ovtxTysKICKDgR+9Zki6xL1tCrANr6lFq0NN99jGszBm564SfY5qwjtXnaj5zTHGf8S4Uio2BOog+TxjK0c0TmXKKitXeOO+PI5onMq2A/meCals2w4cZtCTU8sc4xX3oLO/DTyaYqeLSUt3MmnpTn67bShHNa3v09N8qKCEmz5ZTLMGyVzx5nzeuLIfI3q05pP5W5i9Yb+nXCzAnA37+cvrc7mo35F8lmENrlu58xBXDkinnTsFsajExfdLdwIwy2uWujdmbPQMlgzGOw3hho8WMeU237KuPwWoKXzHF0s9P8/fZAXRe3LKpm1c/ua8kOdetyf0l5RIVKZ8W3Xq16FpmUGHwXxz/UD6HNWEKat9c9hP7ljxeCFqXT7BphV1r4toatGqVtsu1Rf1q9wEHbXpzkpNtmVYt1aVGgCgqp8OGFQqNt32+RLSx03C6TKeFIU7vljKFW/O93RQFTsNPR6YzBnPTSen4P/bu/PoqMo0j+PfN3vIAokJSwIkhF12EyCSAN20rG6M4DCACMehhR4YRu1zZhQdwWUUx2OPY5/T4wbdNopwZloQRzw6p+2maW3Q9ICC3YICAjJIGxdWUULu/HFvVd1KKkmFkNS9ld/nnHtI3aV47vImb731vs8bmsDi+MlzESvO7swSf9j/BWe/DfXl/cFjW1mxeQ8b3j1cb5B8IC3ZrWur2HHgC+58aTebXV0Uvj1/gd84rdeBijPAU1sPULHqTY6d+IZ1Ow4z55ntwcrzv/86lCrts5PnwjJJlD34P01enwmPbW1yHz9qrMW3sk8eu1dOavH/EU2dyJ06GGCmM9FZalIC//33lcFW5sQEE+yKUtozN2wCuZyM6LoVRXIpW559ozUanm8YURicSac59j80LerJPOryYgN64JeaB0OTFjDGJGH/vkgEEo0xaUCNZVk1jR/ZOKWqE/GnQEts7+VbqOyTx/MLRwe3vfmhXVF1d42Y8vg23rpzAgDzGmhJvfqJ0OQas5/ZXq8v74vvHOHFd47wuCsHMIRaq2stmPX09uD6DVV294SVr4Rnr6jLnW84GtWnG2+Fjmc/HNsLg2HJuv8NrntlaSU9ctMb7eNeWpQT7AqSn5XKu3dfxe0bdrFx51G+1z+fe6+5PPiB47LMVB6YPpgnfv0Rnzst8f+1+EpmPvmH4Pt1SAlvKMtISWT9reX0zO1AQad0BnTNYvuBL1k0LjR5dccOyey5bzK9l2/hjon9WnQd2mfluRWqdj+ZdXF9aVrylXWgn2/WRU7F3RryMu1BFkXNGPghvnAPsML1+ibgPmBlS95Uk6SI+N/vP64Oa+kNeG1PqKvC0a+/4Xf7PmfTrqPsOx5dN4O7N+6JuP7YiYb7N7d3762YxLD73ghb968zhvKPv3q/3r7ThnTlumEF/PnYKSfXcy1rtx8Kbv/byl4MLszm9g2hrBvXDC0AYMm60PsM6R6a/A3g3msuZ1SvXM5fqOVPx05y98Y9pCYlMHd0T17YcZhsZ9blwGR12z6qpiQ/k1llPdhQdQTLgnnlRcwrLwrm2S4rzuWTVVez+9MTPLPtADNKu7Nx51HunjaQNW8dZMmEPnTOCs2mnJSYwMM3DCGS/Q9Na/I6NsU7ta425MUW24uxeHxvcjJSmFHasm4fl1JFnzzWLChjbN9LlFZQPMGyrJW0sKIcyflaDRgU8ZP/XHwlN7paAAOWvbizyWMbmxWuvVr3w9EsWvtHTp1r3pd4KU5l9J2DXwZzPA/r0YmO6cnsfXAKb31czS2/sKemvm54QbDyvG7haPYdP8Xc8qLg794pg7sF39ddeb7tqr68trt+X22Agd2y+fOxkxG33VLZK/hzIBWeMXD/9YMxBuZfWQzY2cuWb9zN7FE9AFg0voQNVUe4fnhB8Pg1C8r4pPps8PWQ7h15YrY9m/LmpZUAjG5B3+WL1S7/asVL5TklKYF55UWeG3A1YUAXVYgkKjVKVSfiKzlRpp+LZ8XON6uBb30Xj+/d5DHf7x9qULrhisLgz8O6d2L3yslh+5Y4WaPumtrwLMiWZbHi2kG8umwsJfn2/i8vqQDsTFjuwXBpyYl8/C9T2bSkgjF98lhQ0avJv9G/+tEYstKSqWkgX91r/zC20eMDAjMsF3RMJzHB8OD0IcGp3eeM7snbd05gxbX2ZNQl+Zl8supqBnYLzc0wYUCXsMq4V7TLGk5bp1ETkchqatVtQ8RPcjqEJsW6blhBI3teWinNaJB58qYr2LH8Bxf9f91U3nCO+omXd+G2q+z+sr+4ZSTzyotYPL4kbJ9HZw4Ne/0fc6+gm5PRY2zfPH7y18N5eUkF868sqtd3F+CuaQNZPb+MhWNLWDS+hGdvLmPz0oqwfc67Zv976UdjeHVZZdj2upXjpMSE4MC5aJQW5QAwfUQB04dHvs/rFo7m6XmlTb7P47OGc9/1gyJuL+iU7svGNnXbEJGYqbmgAYMifuIeFFaY0zbTcz8yYwh/NaI7/e55rcl9B3bLDuuGAPDCwtHMfXYHeZkpTQ72e/vOCRR0SmfbR9Uc+uIsgwqy+bdZw9l5+CtmjQxVqsf1yyc3I4XSotx673FjWY+wNHRTh3Tjd8501JMHdQXsLhYNZa4oyc+gd76doOyuqQPDtg3r3pH3Pj0RrNyCfU/qDtYLfKPX3HzIdXVISeLxvxlBekoS++ukxBvTJy+q95g+orDpnXymfVaeYx2AiADw1dnzankW8ZHEBMPgwmz2HD3JmW9reHTm0LCKotvBh6fR664tF/X/LBpfwlNbDwD2YDF3fudOHZLpmp3Gku/3oTAnnRt+9nbE93h6XimFOekMKujIrnsnkpSYwM2rd3DqXA0Lx/bizQ//wusfhOf+LXBaiF9eUkH16e/o09muxNbNU103VVpgsNuo4tzg8VWHvqJrtj2ILZCJqqlpzf90/+RgV4e69j44hURj+L+vz5Gb2Xj3GWMMm5dWUHRZ8yYOe+zGYVQd+rLe+oYG37VX7bPyrKZnkZjb5SS437r3c7g2xsGISNR+vmAUc5/dzoIxxZx0DXRbPm0AD235MPja/bc2LzOV6tN22rH05ES+OR/K4ewWeI8ZV3TnxxP7c7D6TLAVdl55EWu3H2LXvaFcwufqvI87B/Qkp5UXQi3mL/1dqPvDweqzwHF+PLEfu458TYYrc1Wk1tzGPDJzKI+4umvUbVke1y+f9e8eYUhhx0iH8/pt48hKS2qw4gwE5zGIdhrzod2b3+o8o7S7p5IQeFW7/L5UVWeR2NtxwJ7Y4IBPZrQSEVt+Vipv3D6ekvzMYOV1VHEut44LDZx74/ZxYcdsWVZJRkoiGSmJ7LkvfIAcwIQBnQHo7Qwa69cli5SkBPp3DbX4PjB9MJ+sujrsuFSnRXpksd2N4boG+udGkue03nbtmMbqBSODWRxaw7Qh3Xh/5aR6ad0C+nfNCrZ6i/e105bnWEcgIg2N4hYR/wi0CgcyIvTtnMlHfzlNn/zMsP06Z6fxwf1T6h0/vEcnBnbLCk6GYTXz14Ixhqp7riI7LZkLtRZpydG3CS4YU0xeZmqbDXzMTktueifxhXZVeb6pvCfPbz+sbhsiHrBoXAmPvr6XhR5MQyQi0cnPSg1rDV5/azl7PztFgjNg7dVllZz8pn4O48syUqiptdjkpFc78uVZ0pITqewb3SA0t8DkXM2VlJgQl4PZpPWZunPEe1lZWZlVVVV10cfX1lqcr60N9hsS8TtjzB8tyyqLdRyRRFNev625QHJCQvAPrUg883J5hZb/jW2Omgu1WNRPqSbiJQ2V2XbV8pyQYEhNUMVZxCv0QVakfUpSpVl8TE+viIiIiEiUVHkWEREREYmSKs8iIiIiIlFS5VlEREREJEqqPIuIiIiIRMlXqeqMMZ8Dh5rYLQ+oboNwWovijx0/xl5kWVZ+rIOIROXVFxR/2/JseQWVWR/wc+zgz/gjlllfVZ6jYYyp8nIezaYo/tjxc+x+5fdrrvhjy+/x+5Hfr7mf4/dz7OD/+N3UbUNEREREJEqqPIuIiIiIRCkeK89PxzqAFlL8sePn2P3K79dc8ceW3+P3I79fcz/H7+fYwf/xB8Vdn2cRERERkdYSjy3PIiIiIiKtQpVnEREREZEoxU3l2RiTa4zZaIw5Y4w5ZIyZE+uYAowxqcaY1U5cp4wxO40xU51txcYYyxhz2rX8c51j1xhjThpjPjPG3BGjc/itMeacK8a9rm1znHM7Y4zZZIzJdW2L+X2pc21PG2MuGGN+6mzzxfWPN154LhqjMqsyK+G88Fw0ROVV5bXNWZYVFwvwIrAByAQqgRPAoFjH5cSWAawEirE/sFwDnHJeFwMWkNTAsQ8D24AcYCDwGTAlBufwW2BhhPWDnHMZ51z7dcB6r94X516cBsY5r31x/eNt8dpz0cBzojLrgXujMuuNxWvPRYRnROXVA/elvZTXmAdwCW/Wd0A/17q1wKpYx9ZIzO8DM6J4sI4Ck1yvH3AXnDaMt6GC/RCwzvW6t3Mvsrx4X4D5wAFCg2V9cf3jafHicxFl3CqzsbnuKrMxXrz4XEQRs8prbK57uyiv8dJtox9wwbKsfa5172F/YvMcY0wX7Jg/cK0+ZIz51Bjzc2NMnrNfDlCAfS4BsTyvh40x1caYt4wx33PWDcIVn2VZ+3EKM968L/OBX1pOKXXxw/WPF158LhqlMqsy28558blokMqrymtri5fKcyb2VxVuJ7A/mXmKMSYZeAF4zrKsD7HneR8JFAGl2DG/4Oye6fzrPrdYndc/ASVAIXauxleMMb1p/Np76r4YY3oC44HnXKv9cv3jiaeei6aozKrMireei8aovKq8toWkWAdwiZwGsuusy8buJ+QZxpgE7K9UvgOWAliWdRqocnY5boxZChwzxmRjnxfY53LO9XObn5dlWTtcL58zxswGptH4ta9tZFss3Az83rKsg4EVfrn+ccYX5RVUZutsiwWVWW/wRZlVeQ3bFgvtprzGS8vzPiDJGNPXtW4Y4V/ZxJQxxgCrgS7ADMuyzjewa+CrDmNZ1lfAMexzCfDKeVmAwY4lGJ8xpgRIxb4nXrsvNxP+iTgSv1x/P/PacxGRymyQyqx47bmoR+U1SOW1LcS60/WlWoD12KNOM4AKPDQS2InvSWA7kFln/WigP/YHmcuwR83+xrV9FbAVeyTqAOwHrU1HogKdgMlAGva3FXOBM07cg4CTwFjn2j9P+EhgT9wXYIwTc5bfrn88Ll55LpqIUWVWZVaLx56LRuJTeVV5bbvzjXUAl/DG5QKbnJt3GJgT65hcsRVhf9o6h/01RWCZC8wGDjpxHwN+CXR1HZsKrHEKz3HgjhjEnw+8i/1VytfOL6iJru1znGt+BngZyPXafQGeAtZGWO/56x+Pi1eei0biU5lVmdUSft098Vw0EJvKq8prmy6BVCIiIiIiItKEeOnzLCIiIiLS6lR5FhERERGJkirPIiIiIiJRUuVZRERERCRKqjyLiIiIiERJlWcRERERkSip8iwiIiIiEiVVnkVEREREoqTKs4iIiIhIlP4fgTPMolXoa2wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x216 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.activation_stats.plot_layer_stats(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generally our model should have a consistent, or at least smooth, mean and standard deviation of layer activations during training. Activations near zero are particularly problematic, because it means we have computation in the model that's doing nothing at all (since multiplying by zero gives zero). When you have some zeros in one layer, they will therefore generally carry over to the next layer... which will then create more zeros. Here's the penultimate layer of our network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs8AAADWCAYAAAAuNG/NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de5gcVZ3/8fdnLplJMgmQMAQFQwQTlShhddSVGBfvsKsPuKy/VbKAuhgXFsH1yrqiiK6iq+yqD4Kw3OSqKyCweEEFlODvJwY1YBQCAlHuE0JCZshtZr6/P6o6KZqemZ5MzXR39ef1PPVk+tSp6m/3zEl/+9Q5pxQRmJmZmZnZ6FpqHYCZmZmZWaNw8mxmZmZmViUnz2ZmZmZmVXLybGZmZmZWJSfPZmZmZmZVcvJsZmZmZlYlJ89mZk1O0sGSQtLetY7FzKzeOXk2MysgSXunCfHBtY7FrB5IeqekByStl3SBpPbMvlZJv5T097WM0RqDk2czMzMrNEmzgQuAfwOWAAcByzJVPgQ8HBHfrkF4VZM0pYjP1WicPDcJSTdLOk/S5yQ9nn7z/ndJLZI+JekxSb2S/j1zTJukUyXdL2mzpFWS3l923pMk/VZSn6RHJV0h6TmZ/aXLwW+S9HNJT0v6vaS3TObrNysqSa+RdKukjem2Mm1ff06r3JS2wQcyx3xA0oNpe/wRMLcWsZtNon2BDRFxaUTcCXwP2B9A0nzgJOC4ak6Ufi7eK+kwSXdJ6pd0k6T9yuq9XNIN6edjr6SrJO2T2f/8tOzhtC3eKemosnOUPrs/K+kR4KFhYro5befl27szdT6QxrtZ0j2S/k1SW2b/A2mO8A1JTwC3puXPST/b10valD5XTzXvVVE5eW4ufwe0A68h+Zb9CeB/gS6Sb+IfAT4h6dC0/n8Dfwu8H3gxcBrwRUn/WHbejwAvBd5O8iF8RYXn/jLweWARsAL4tqRdc3tlZk1IUitwLfBL4GXpdirwdPozwBHAc4BXpMccBvwncAZwIPAd4D8mM26zGrgXmCapR9J04K+A30gScB7wiYh4dAznew5Jsr2UpBd7V+D80k5J+wM/A/4v0AO8HhgEfiypM63WBfwUOITkM/Qc4AJJryt7rv8DdANvSM9Tyd+mMZW2zwH9wK/SeE4l+az+V5LP85NIPts/XXaeE4HHgVcDx6Tvz/eAFwFvBV4JPJa+jt1HfIeKLCK8NcEG3Az8tqxsFXBnWdlKkkT3+cAQ8KKy/Z8qP0/Z/r8AAtgrfXxw+vhvM3X2TMveUuv3xZu3Rt6A3dK2dHCFfXtX2gcsBy4tK/tyWnfvWr8mb94magPeln7G3UfyBbIV+ABwffq59L1037eArhHOcyowAHRnyt6ZfmZ2po8vBK4oO66D5Ivt4SOc+xrg3Mzjm4HVQMsYXudbgC3A29LH09LnPaSs3tHA+szjB4CfltV5Q/p/w/5lr+MR4FO1/p3WatveXW9NYWXZ40fTrbxsD5JvygJWJF88t2sj+fYMJMMySL7J7k/yzbt0NWMfnnl56belHyLiUUmDwJydfB1mBkTEk5L+G/iRpBtJerqujoi7Rzhsf+DysrLlwIcnKEyzuhAR1wHXlR5LmgecDLwK+BrwB5IrtJcApwAfH+F0D0dEb+bxQySfmXsAfyK50vMCSX1lx3UC89Pnn0bSIfU2kt7iKSSJ6U1lx9weEUPVvEZJC0muJn08fb0AC4GpwJWSIlO9FeiU1J15LbeVnXIh8ERE/L5UEBFbJP0y3deUnDw3l21lj2OYshZ2JMEHkXxjLa+DpLnA94GLSYZ0rCXp7foJyX8CWVsrxONhQ2bjFBHvk/RV4M3Am4DPSjqBpDdt2MMmJTiz+nYucGpEPCjpjcBnImJA0iXAZ0Y5tvwzrdSmWjL/XgycXuHYJ9J//wM4jOSL610kwyy+AuxSVr9/tBcCIGkPkqGYl0TEf2V2lWJ6B0kvdrl1ozxXpf8vNEx5U3DybMO5Pf13bkT87zB1XkHybfaDEbEJkgkSkxGcme0QEb8DfgecIelsklUErk53t5ZV/z2wGPhGpmzxhAdpVkckvQ9QRJybFrWQzAmCpPNnvJ07K4ADgD9GOtahgteSDKH6dhpTC7CAZEzxmEjqIGnzd5GMW85aBWwG9o2I74/x1KuA3SXtX+p9Tp/rlTzz/5Cm4uTZKoqIeyWdD5wr6WMkkx6mAy8nGef1ReAekm+eH5Z0KclkwE/VKmazZiPpBcD7SC5F/xl4Lsnk31+TXAnqA94saRWwJSKeJOnZ+h9Jt5FcOXoNcFSF05sVkqS9SCbKvSZT/HPgQ+mKU/9MMgRqPD5PMgTikvTKUC8wDzgc+GpE3AfcDRwm6UqStvohkjY85uQZ+GZ67HuA7sxwyw0R0Sfp88Dn0/Ifk+R/LwX+IiJGGp5yY/o6LpP0z8AGkiEtncBZOxFnIfiyuY1kGcmkin8j6a36KXAMyYQKIuIOkskW70/3fwT4YE0iNWtO/STjJ68guRx7JfAL4IR0jOQ/k8zU/zPwG4CIuJrkMvHHgDtIVgsY6cPTrGi+CZweEQ9kyk4kSW5XkLSr0YZtjCgi/kAy7LEL+BHJZ+S5JFdr16fV/gVYQzLG+ack46a/u5NPeTBJ/HeTTOYrbX+fxvPZ9PmOJZn/tDx9/MAoryNIEv67SIaC/YpkcuWbImLtTsba8DT81QQzMzMzM8tyz7OZmZmZWZWcPJuZmZmZVcnJs5mZmZlZlZw8m5mZmZlVycmzmZmZmVmVGmqd59133z3mzZtX6zDM6sbtt9++NiK6ax1HJW6vZs9Uz+0V3GbNyg3XZhsqeZ43bx4rVqyodRhmdUPSmlrHMBy3V7Nnquf2Cm6zZuWGa7O5DtuQNEvS1ZL6Ja2RdOQw9STpi5KeSLcvKXM7HDMzMzOzepR3z/OZwFZgDnAgcL2klRGxqqzeMpI71iwiub3zj0nuWnd2zvGYmZmZmeUmt55nSdOBI4BTIqIvIpYD1wJHVah+DPCViHgwIh4CvgK8O69YzMzMikTSCZJWSNoi6cJR6v6LpEclbZB0vqSOSQrTrCnkOWxjATAYEaszZSuBhRXqLkz3jVYPScvS/zBW9Pb25hasmZlZA3kY+Bxw/kiVJL0FOBl4AzAP2Bf4zEQHZ9ZM8hy20QVsKCvbAMyoou4GoEuSIiKyFSPiHOAcgJ6enmfsK3fuz+9jy8AgJ7x+/lhjNyscSSeQXNF5KXB5RLx7mHrHACcC84GngMuAT0TEQLr/ZuAvgYH0kIci4oXjjW9gcIhv3PxHjl3yfKZNaai5y2aTLiKuApDUA+w9QtVjgPNKwyUlfRa4lCShzs3ZP/sjf3y8L89Tmk2qQ1+6J69/0ZydOjbPT6w+YGZZ2UxgYxV1ZwJ95YnzWP38nl76tgw4eTZLlHqq3gJMHaHeNOCDwC+BbpLhVh8BTs/UOSEi/jvP4K76zUOc8ePVPLVpG5986/55ntqsmS0Ersk8XgnMkTQ7Ip4oryxpGck8JObOnVvVE2wbHOL0H9zFjM42ZnT4i681ppfuvctOH5vnX/1qoE3S/Ii4Jy1bBJRPFiQtWwTcNkq9MRtf+m1WHNX2VEXEWZmHD0m6FHjdBIfHloEhADZtG5zopzJrJpWu7EJyFfhZyfNYru6WbE7b7ElvmM+xS/YdT6xmDSm3Mc8R0Q9cBZwmabqkxcBhwMUVqn8L+JCkvSQ9F/gwcOF4Y5CEc2ezcXstz/4y+wVJayXdKung4Q7cmTkKbrNmuap0ZRcqXwXeKZu3JV98O9p8k2JrTnn/5R9Pcnn4ceBy4LiIWCVpiaTs4KhvAtcBdwK/A65Py8ZF4K5ns3GQ9B6gB/hypvjjJJOO9iLpobpO0n6Vjo+IcyKiJyJ6urvr9kZqZkVWurJbsgh4rNKQjZ1V6nnuaG/N65RmDSXXwUoRsY5k/eby8ltILiWVHgfwsXTLjeReLLOdJelwknHOb4yItaXyiPhlptpFkt4F/DXw9VyeN4+TmBWcpDaSz+xWoFVSJzBQmtib8S3gwnT41SPAJ8nhym7JY09t5q5Hk07sTifP1qQKNdLfH8JmO0fSIcC5wN9ExJ2jVA9ybG7+wmtWlU8Cn848/gfgM5LOB34P7B8Rf4qIH0r6EnATyZXgK8uO22nrn97K4tNvZGAoabW7TG3P47RmDadQyTN41IZZSbU9VZJeT7KU1dsj4rayfbsCrwJ+RrJU3d+TjIn+4MS/AjMriYhTgVOH2d2VfRARZwBn5B3Duv6tDAwF7z5oHn/1wm4W7zc776cwawiFGu2fTBh09myW+iSwiWR9139If/6kpLmS+iSV1qU6BdgF+H5a3ifpB+m+dpLl7nqBtcAHgMMj4u68gvQVI7PGUJoo+Krnz+J1L9yDttZCpRBmVStUz7Nwz7NZSbU9VREx7LJ0EdELvCLXwMqfYyJPbma52TKQTBT0WGdrdoX62ig5eTYzM5sI25eoay9U6mA2ZgVrAV7n2azReNiGWWPY7J5nM6BgyXPS8+z02ayRuMWaNYaBwaS1trcUKnUwG7NCtQD3YJmZmU2MobRzyrmzNbtCNQGPeTZrPP7Sa9YYSld2W+RWa82tWMkzXqrOrNG4xZo1hvTeKE6erekVK3l2ezYzM5sQ24dt+LPWmlyhkmfwsA2zRuPPYbPGUOp5lnuqrMkVKnmWfAnYrNG4zZo1hnDPsxlQtOQZeak6MzOzCTDkCYNmQMGSZ9zzbNZw/DFs1hiGkhsMOnm2pleo5Fng7NmswbjJmjWGUs+zc2drdsVKnuXbc5uZmU2E0qjIFg96tiZXrOQZ357bzMxsInipOrNEsZJnj3k2axj+/DVrLL5JilmiWMkzXufZrFG4qZo1Fo95NksUK3mWb89tZmY2EcJL1ZkBRUueax2AmVXN7dWssXjYhlmiUMkzeNiGWaNwUzVrLJ4waJYoVvIsJ89mZmYTodTzLF83siZXqOTZDdpsB0knSFohaYukC0ep+y+SHpW0QdL5kjoy++ZJuknS05LukvTGXOLL4yRmNmlKY55VqMzBbOwK1QS2DAzy0PpNXuvZLPEw8Dng/JEqSXoLcDLwBmAesC/wmUyVy4HfALOBfwO+K6l7vMG5lZo1lvCYZzOgYMnz/97xCADL711b40jMai8iroqI7wFPjFL1GOC8iFgVEU8CnwXeDSBpAfAy4NMRsSkirgTuBI6YuMjNrBJJsyRdLalf0hpJRw5Tb1dJF0l6PN1OzeP5PebZLNFW6wAmwpNPb6t1CGaNZCFwTebxSmCOpNnpvvsiYmPZ/oXjfVJ//pqN2ZnAVmAOcCBwvaSVEbGqrN5/AtNIriTtAfxU0pqIuGA8T+7VNswShep5LhkYHKp1CGaNpAvYkHlc+nlGhX2l/TMqnUjSsnSc9Yre3t4Rn9TDNsyqJ2k6yRWfUyKiLyKWA9cCR1Wo/jbgSxHxdEQ8AJwHvHe8MfgmKWaJgibP/lg2G4M+YGbmcennjRX2lfZvpIKIOCcieiKip7t73MOizWyHBcBgRKzOlI10FUhlP79kvAH4JilmiUImz1vd82w2FquARZnHi4DHIuKJdN++kmaU7S+/TDxm/vg1G5OxXAX6IXCypBmSXkDS6zyt0knHcrXIwzbMEoVMnj1swwwktUnqBFqBVkmdkirNc/gW8I+S9pe0G/BJ4EKAtJfrt8Cn0+PfDhwAXDne+Hx9yGxMxnIV6ERgE3APyXyGy4EHK510LFeLPGHQLFHM5HnIH8tmJEnwJpJl6P4h/fmTkuZK6pM0FyAifgh8CbgJWJNun86c551AD/AkcDrwdxExcheVmeVtNdAmaX6mrOJVoIhYFxFLI2LPiFhI8ll/23gD2H6TFPc8W5PLJXmudvmctO7r0hsubJD0QB7PX87JsxlExKkRobLt1Ij4U0R0RcSfMnXPiIg5ETEzIt4TEVsy+x6IiIMjYmpEvDAifpJHfP74NateRPQDVwGnSZouaTFwGHBxeV1J+0maLalV0qHAMpI138cbg3udzciv5zm7fM5S4CxJw01i6Ce5acNHc3ruZ/GwDTMzK6DjganA4yRDMY6LiFWSlkjqy9R7Ocl67BuBLwBLKyxnN2ZDER7vbEYO6zxnls95SUT0AcsllZbPObm8fkTcBtyW1y1+K9nm1TbMzKxgImIdcHiF8ltIJhSWHn8H+E7ezz8UnixoBvn0PI91+ZwJt809z2ZmZrkaivAaz2bkkzyP6SYKYzWWZXRKPObZzMwsX+GeZzOgiuRZ0s2SYphtOWO8icJY7cxNF9zzbGZmlq+hIU8YNIMqxjxHxMEj7U/HPLdJmh8R96TFudxEYWf5DoNmZmb58phns8S4h22MZfkcAEkt6Y0b2pOH6pQ0ZbxxZA0MuefZzMwsTx7zbJbIa6m6isvnAFRYQue1JDdr+D4wN/35hpziALzahpmZWZ42bxvkwl88wFObB2odilnNjXupOhh++Zx0X/kSOjczwfdH8JhnMzOz/GzZ5s9Vs5Ji3p7bPc9mZmZmNgGKmTx7zLOZmVl+PNbZbLtiJs/ueTYzMzOzCVDI5HmrxzybmZnlxqtsmO1QyOR5y4CTZzMzMzPLn5NnMzMzG5E7ns12KFTyPKMjWXlvq5NnMzMzM5sAhUqerz9xCS2CrQODtQ7FzMysMORBz2bbFSp5njt7GocduJcnDJqZmZnZhChU8gwwpbXFwzbMzMxy5H5nsx2Klzy3OXk2MzMzs4nh5NmswCTNknS1pH5JayQdOUy9H0jqy2xbJd2Z2f+ApE2Z/TfkFWP4nkZmdc9Dns12aKt1AHmb0tbiMc9mO5wJbAXmAAcC10taGRGrspUi4tDsY0k3AzeWnettEfGTCYzVzMys7hWv57m1hW2DwdCQu7OsuUmaDhwBnBIRfRGxHLgWOGqU4+YBS4CLJzrG5Pkm41nMbDzkUc9m2xUveW5LXpJ7n81YAAxGxOpM2Upg4SjHHQ3cEhH3l5VfKqlX0g2SFuUVpIdtmJlZIylc8tzh5NmspAvYUFa2AZgxynFHAxeWlS0F5gH7ADcBP5K0a/mBkpZJWiFpRW9v787EbGZ1yFeIzHYoXPK8vefZkwbN+oCZZWUzgY3DHSDpNcCewHez5RFxa0RsioinI+ILwHqSoR2U1TsnInoioqe7u7uqIP2hbGZmjaR4yXOrk2ez1GqgTdL8TNkiYNUw9QGOAa6KiL5Rzh3ktPSrh22YmVkjKV7y7J5nMwAioh+4CjhN0nRJi4HDGGYioKSpwDsoG7Ihaa6kxZKmSOqU9FFgd+DW8cTnHmez6o1h2ckOSWdLekzSOknXSdprsuM1K7LiJs8e82wGcDwwFXgcuBw4LiJWSVoiqbx3+XCSMdE3lZXPAM4CngQeAg4BDo2IJ8YTmHuczcYku+zkUuAsSZUm/54EvBo4AHguyRCrr4/3yf1l12yHwq3z3NHWCrjn2QwgItaRJMXl5beQTCjMll1OkmCX111F8kFsZjWQWXbyJemQquWSSstOnlxW/fnAjyLisfTYK4AzJjNes6IrbM/zFifPZnXNPVlmVRvLspPnAYslPVfSNJJe6h+MNwCv82y2Q+F6nj1h0KwxeNiGWdXGsuzkauBPJEOsBoE7gROGO7GkZcAygLlz5+YRq1nhFbbn2WOezcysIMay7ORZQCcwG5hOMml42J7napeX9JUisx0Klzx3eLUNs4bgD2Ozqo1l2clFwIURsS4itpBMFnylpN0nIU6zplC45NlL1Zk1Bg/bMKvOGJed/BVwtKRdJLWTrLjzcESsHU8M/q5rtkPxkufSmOfBwRpHYmZmlptql538CLAZuAfoBf4aePtkB2tWZMWbMOieZ7OG4GEbZtWrdtnJdP31pZMYmlnTKV7Ps5NnMzOzXMnfds22K2zy7HWezczMzCxvxUueW71UnZmZWZ7c72y2Q3GTZ/c8m5mZmVnOCpc8t7SI9lY5eTYzM8uJhzyb7ZBL8ixplqSrJfVLWiPpyBHqflTS7yRtlHS/pI/mEUPWlNYWJ89mZmZmlru8lqo7E9gKzAEOBK6XtDIiKt39SMDRwB3AfsANkv4cEVfkFAtT2lo85tnMzCwnXm3DbIdx9zxLmg4cAZwSEX0RsRy4FjiqUv2I+FJE/DoiBiLibuAaYPF448ia0uaeZzMzMzPLXx7DNhYAgxGxOlO2Elg42oFKvsouASr1UO80J89mZmZmNhHySJ67gA1lZRuAGVUce2oawwXDVZC0TNIKSSt6e3urCmhKawtbPGzDzMzMzHI2avIs6WZJMcy2HOgDZpYdNhPYOMp5TyAZ+/w3EbFluHoRcU5E9ERET3d39+ivCOhoa2XLNifPZmZmZpavUScMRsTBI+1Pxzy3SZofEfekxYsYYSiGpPcCJwOvjYgHqw+3OlOntLJ522DepzUzMzOzJjfuYRsR0Q9cBZwmabqkxcBhwMWV6ktaCnweeFNE3Dfe569kansrm5w8m5mZmVnO8rpJyvHAVOBx4HLguNIydZKWSOrL1P0cMBv4laS+dDs7pziApOf56a1Ons3MzMwsX7kkzxGxLiIOj4jpETE3Ii7L7LslIroyj58fEe0R0ZXZ/imPOEqmtnvYhhlUfwMjSadK2pb5Qtsnad/M/gMl3S7p6fTfAyfvVZiZmdWPwt2eG2DalFae3jpQ6zDM6kH2BkZLgbMkDbeM5LfLvtTeByBpCsl67JcAuwEXAdek5WZmZk2lkMlzZ3srmzxsw5rcWG9gNIKDSSYX/1dEbImIr5HcKfT1ecZrZmbWCAqZPE+b4gmDZoz9BkZvk7RO0ipJx2XKFwJ3RERkyu6odJ6dWZcdYvQqZmZmdaKQyfPU9la2DQbbfKMUa25juYHRd4AXA93A+4BPSXrXWM+zM+uym5mZNZJiJs9TWgHc+2zNruobGEXE7yPi4YgYjIhfAF8F/m6s59k5yuc0ZmZmk6DQyfNmj3u25raa9AZGmbIRb2CUEezIalcBB0jKZrkHVHmeKp/KzMysMRQyeZ6WJs9e69ma2VhuYCTpMEm7KfFK4ESSFTYAbgYGgRMldUg6IS2/ccJfhJmZWZ0pZPI8td3DNsxSFW9gVOHmRe8E7iUZivEt4IsRcRFARGwFDgeOBtYD7wUOT8tz4GEbZmbWONpqHcBEmDoleVnuebZmFxHrSBLf8vJbSCYClh6/q7xOWf3fAC/PPcDk7BNzWjMzswlQ6J5n32XQzMzMzPJUyOTZY57NGomHbZhVQ9IsSVdL6pe0RtKRw9T7gaS+zLZV0p2THa9ZURVy2EanxzybNRAP2zCr0pnAVmAOcCBwvaSVEfGMlW8i4tDsY0k34wm+ZrkpdM/zpq0DNY7EzMxs/CRNB44ATomIvohYDlwLHDXKcfOAJVRYZcfMdk4hk+ftq2142IZZA/CwDbMqLAAGI2J1pmwlsHCU444GbomI+ycsMrMmU8jkeVpHkjz3O3k2awAetmFWhS5gQ1nZBmDGKMcdDVw43E5JyyStkLSit7d3fBGaNYlCJs8dba1MaWvhqc3bah2KmZlZHvqAmWVlM0nWZq9I0muAPYHvDlcnIs6JiJ6I6Onu7s4lULOiK2TyDDCzs42+zR7zbFb/PGzDrAqrgTZJ8zNli4BVw9QHOAa4KiL6RqhjZmNU2OS5q6ONjU6ezRqAh22YjSYi+oGrgNMkTZe0GDiMYSYCSpoKvIMRhmyY2c4pbPI8o7OdjR62YWZmxXE8MBV4HLgcOC4iVklaIqm8d/lwkjHRN01yjGaFV8h1ngFmdLrn2awxeNiGWTUiYh1JUlxefgvJhMJs2eUkCbaZ5azAPc9Ons0ag4dtmJlZ4yhs8tzV0U7fFifPZmZmZpafwibPMzrbvFSdmZmZmeWqsMnzzM42+rYMMDTkS8Jm9Uge62xmZg2osMnzjM52IqB/q4dumNWj8FhnMzNrQIVNnrs6k4VEPGnQzMzMzPJS2OR5Rpo8e9KgWX3ysA0zM2tEhU2eZ3a2A7BhkycNmtUjD9swM7NGVNjkedb0KQCs699a40jMzMzMrCgKmzzP7kqS5yf6nDxb85I0S9LVkvolrZF05DD1Pirpd5I2Srpf0kfL9j8gaZOkvnS7YdyxediGmZk1oMLenntHz/OWGkdiVlNnAluBOcCBwPWSVkbEqrJ6Ao4G7gD2A26Q9OeIuCJT520R8ZO8AvOwDTMza0SF7XnuaGulq6ONJzxsw5qUpOnAEcApEdEXEcuBa4GjyutGxJci4tcRMRARdwPXAIsnN2IzM7P6V9jkGZLeZ495tia2ABiMiNWZspXAwpEOkiRgCVDeO32ppF5JN0haNMyxyyStkLSit7d3xOA8bMPMzBpRoZPn2V1Onq2pdQEbyso2ADNGOe5Ukv8bLsiULQXmAfsANwE/krRr+YERcU5E9ERET3d394hP4mEbZmbWiHJJnqudlJTW/aCk+yQ9JelhSf8paULGXs+ePoW1njBozasPmFlWNhPYONwBkk4gGfv8NxGxfcJARNwaEZsi4umI+AKwnqR32szMrKnk1fOcnZS0FDhL0nCXhq8DXhYRM4GXAIuAE3OK4xmSYRueMGhNazXQJml+pmwRzx6OAYCk9wInA2+IiAdHOXfA+MZdeNiGmZk1onEnz2OZlAQQEX+MiPWlw4Eh4AXjjaOS2V0drOvfytCQLw9b84mIfuAq4DRJ0yUtBg4DLi6vK2kp8HngTRFxX9m+uZIWS5oiqTNdxm534NZxxedhG2Zm1oDy6Hke86QkSUdKegpYS9IT9s0R6lY9Aancc3bpZNtgsLbPvc/WtI4HpgKPA5cDx0XEKklLJPVl6n0OmA38KrOW89npvhnAWcCTwEPAIcChEfHEpL0KMzOzOpHHWOMxT0qKiMuAy9LLyUcDj41Q9xzgHICenp4xdVXttetUAB5av4k9ZnaO5VCzQoiIdcDhFcpvIWm7pcfPH+Ecq4AD8o7NwzbMzKwRjdrzLOlmSTHMtpydmJRUEhH3kIy//MbOBD+avXbbkTybWX3xsA0zM2tEo/Y8R8TBI+1Pxzy3SZqfJsMwwqSkYWLYr8q6Y/LcUs/zk06ezczMzGz8xj3meSyTkgAkHStpj/Tn/WrTRS8AAAsMSURBVIF/BX463jgqmdnZzozONh52z7NZ3fGwDTMza0R5LVVXcVISQIWJSYuBOyX1A99Pt0/kFMez7LXrVA/bMDMzM7Nc5HJzkuEmJaX7yicmvSeP56zWvNnTWf3YqMOvzczMzMxGVejbcwMs2HMGDzzRz+Ztg7UOxczMzMwaXOGT5xftOYOhgHsf7xu9spmZWZ2SNEvS1ZL6Ja2RdOQIdV8m6efpmu2PSTppMmM1K7LCJ88L5iTLTd/9qIdumJlZQzsT2ArMAZYCZ0l61g3JJO0O/JDkBmSzSe7ie8MkxmlWaIVPnufNnkZHWwu/e7j8Pi5mZmaNIV0W9gjglIjoi4jlwLXAURWqfwj4UURcGhFbImJjRPxhMuM1K7LCJ89trS28bO5u3Hb/ulqHYmYZSleqGxzyzVLMqrAAGIyI1ZmylcCzep6BvwTWSfqFpMclXSdpbqWTSlomaYWkFb29vRMQtlnxFD55BnjVvrP4/SNPseHpbbUOxcxSLWnyvGGT26VZFbqA8kuoG4AZFeruDRwDnATMBe4nWUb2WSLinIjoiYie7u7uHMM1K66mSJ5fve9sImD5vWtrHYqZlXnSX2rNqtEHzCwrmwlUmtCzCbg6In4VEZuBzwAHSdplgmM0awpNkTz3zJtF94wOrvntQ7UOxczKPNm/tdYhmDWC1UCbpPmZskXAqgp17wCy46FKP/u2nmY5aIrkubVFHLboudx09+M8umFzrcMxMyDSj/MHnuhn68BQbYMxq3MR0Q9cBZwmabqkxcBhwMUVql8AvF3SgZLagVOA5RGxfvIiNiuupkieAY45aB4R8LUb76l1KGaWsW0w2P9TP+R/Vvy51qGY1bvjganA4yRjmI+LiFWSlkjafjODiLgR+ARwfVr3BcCwa0Kb2djkcnvuRvC8WdM46tX7cMGtD3Dwgm7evHDPWodkZsAr581iy+AQH7vyDmZNn8IbXjyn1iGZ1aWIWAccXqH8FpIJhdmys4CzJik0s6bSNMkzwMcPeRG3r3mS4y/9NR9843yOPmgeMzvbax2WWVMqDcL86rsOZNepU3jHN3/BsotvZ9HeuzC9o40I2DY4xMBQMDA4xLbBYGBoiIHBYNvQ0PZhH2b14vL3/SXPmzWt1mGY2QRrquS5s72VS459FSdfeQdfvmE1X7vxXg7YaxdesEcXs6ZPYddp7bS3ttDWIlpbWmhtgRYJKZ85FnnO1MgpJKtju0xtb4peWCGmTmnl4ve+inNvuY9f/+lJnto8QIugvaWFjrYWujraaG8VbS0ttLWK9tYWtwGrOx3tTTMS0qypNVXyDDCzs51vLH05K/+8nmtXPswdD67nJ394jCef3uabNVhdefFzZjZF8lyy2/QpfOyQF9U6DDMbQc8+u9U6BLOaa7rkuWTR83Zl0fN23f44IujfOshAepl4cCgYGAqGckqo87zEHDjJbwbtrcXuxXrrAc/hoP1ms3vXlFqHYmZVuOffD6XFl3zMmjd5LieJrg6/HVYskmYB5wFvBtYC/xoRl1WoJ+B04Ni06Dzg4xHJ1z5JB6ZlLwb+APxjRPx2PLHN6GxnhuccmDWMon+hN6uWW4JZsZ0JbAXmAEuBsyQtrFBvGcks/kXAAcBbgfcDSJoCXANcAuwGXARck5abmZk1FSfPZgUlaTpwBHBKRPRFxHLgWuCoCtWPAb4SEQ9GxEPAV4B3p/sOJrlK9V8RsSUivkYy//X1E/wSzMzM6o6TZ7PiWgAMRsTqTNlKoFLP88J0X6V6C4E7SkM4UncMcx4zM7NCc/JsVlxdwIaysg3AjCrqbgC60rHQVZ9H0jJJKySt6O3t3enAzczM6pWTZ7Pi6gNmlpXNBDZWUXcm0Jf2Nld9nog4JyJ6IqKnu7t7pwM3MzOrVw21vMTtt9++VtKaUartTrKqQKNy/LXTiLHvM8K+1UCbpPkRcU9atghYVaHuqnTfbRXqrQI+LEmZoRsHkExGHJbba0Nw/JNrpPZac26zda+RY4fGjL9im1UU7B63klZERE+t49hZjr92Gjn24Ui6guRO2McCBwLfBw6KiFVl9f4JOAl4Y1r/x8DXI+LsdFWNe4AzgLOB9wEfBeZHxNZxxtfQ77njr61Gj78RNfp73sjxN3Ls0PjxZ3nYhlmxHQ9MBR4HLgeOi4hVkpZI6svU+yZwHXAn8Dvg+rSMNEE+HDgaWA+8Fzh8vImzmZlZI2qoYRtmNjYRsY4k8S0vv4VkImDpcQAfS7dK5/kN8PIJCtPMzKxhFLHn+ZxaBzBOjr92Gjn2RtXo77njr61Gj78RNfp73sjxN3Ls0Pjxb1e4Mc9mZmZmZhOliD3PZmZmZmYTwsmzmZmZmVmVCpM8S5ol6WpJ/ZLWSDqy1jGVSOqQdF4a10ZJv5F0aLpvnqSQ1JfZTik79nxJT0l6VNKHavQabpa0ORPj3Zl9R6avrV/S9yTNyuyr+e+l7L3tkzQo6evpvoZ4/4umHv4uRuI26zZrz1QPfxfDcXt1e510EVGIjWQZrm+TrCDwGpLbBy+sdVxpbNOBU4F5JF9Y3kpyd7Z56RZA2zDHfgG4BdgNeDHwKHBIDV7DzcCxFcoXpq/ltel7fxlwRb3+XtLfRR/w2vRxQ7z/Rdvq7e9imL8Tt9k6+N24zdbHVm9/FxX+Rtxe6+D30iztteYB5PjL2gosyJRdDJxe69hGiPkO4Igq/rAeAt6cefzZbMOZxHiHa9ifBy7LPN4v/V3MqMffC3AMcB87Jss2xPtfpK0e/y6qjNtttjbvu9tsjbd6/LuoIma319q8703RXosybGMBMBgRqzNlK0m+sdUdSXNIYs7e5W2NpAclXSBp97TebsBzSV5LSS1f1xckrZV0q6SD07KFZOKLiD+SNmbq8/dyDPCtSFtpRiO8/0VRj38XI3KbdZttcvX4dzEst1e314lWlOS5i+RSRdYGkm9mdUVSO3ApcFFE3EVyn/dXkNw//eUkMV+aVi/dxCL72mr1uj4O7AvsRbJW43WS9mPk976ufi+S5gJ/BVyUKW6U979I6urvYjRus26zVl9/FyNxe3V7nQxFucNgHzCzrGwmyTihuiGpheSSylbgBICI6ANWpFUek3QC8IikmSSvC5LXsjnz86S/roj4ZebhRZLeBfw1I7/3QyPsq4WjgeURcX+poFHe/4JpiPYKbrNl+2rBbbY+NESbdXt9xr5aaJr2WpSe59VAm6T5mbJFPPOSTU1JEnAeMAc4IiK2DVO1dKlDEfEk8AjJaympl9cVgEhi2R6fpH2BDpLfSb39Xo7mmd+IK2mU97+R1dvfRUVus9u5zVq9/V08i9vrdm6vk6HWg67z2oArSGadTgcWU0czgdP4zgb+H9BVVv4q4IUkX2Rmk8yavSmz/3TgZyQzUV9E8oc2qTNRgV2BtwCdJFcrlgL9adwLgaeAJel7fwnPnAlcF78X4KA05hmN9v4XcauXv4tRYnSbdZv1Vmd/FyPE5/bq9jp5r7fWAeT4i5sFfC/95f0JOLLWMWVi24fk29ZmkssUpW0p8C7g/jTuR4BvAXtmju0Azk8bz2PAh2oQfzfwK5JLKevT/6DelNl/ZPqe9wPXALPq7fcCfBO4uEJ53b//Rdzq5e9ihPjcZt1mvT3zfa+Lv4thYnN7dXud1K20lIiZmZmZmY2iKGOezczMzMwmnJNnMzMzM7MqOXk2MzMzM6uSk2czMzMzsyo5eTYzMzMzq5KTZzMzMzOzKjl5NjMzMzOrkpNnMzMzM7MqOXk2MzMzM6vS/wfiI2mA+SNXPQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x216 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.activation_stats.plot_layer_stats(-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the problems get worse towards the end of the network, as the instability and zero activations compound over layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Increase batch size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way to make training more stable is to *increase the batch size*. Larger batches have gradients that are more accurate, since they're calculated from more data. On the downside though, a larger batch size means fewer batches per epoch, which means less opportunities for your model to update weights. Let's see if a batch size of 512 helps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls = get_dls(512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2.309385</td>\n",
       "      <td>2.302744</td>\n",
       "      <td>0.113500</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what the penultimate layer looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs8AAADWCAYAAAAuNG/NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxjZZn3/8+VpPalt6re9wWFRpuBVpRNRnTEhRco+uiACM4oDojLOOrMqDgt7j6j8/PxQQRHRBERR1kfEMdRUMAFGqWFBuyGhqYbet+rqmtJcv3+OCfVqXSqKtV1qpLU+b5fr/Oq5OTOyVXpujvXuXOd+zZ3R0REREREhpcodwAiIiIiItVCybOIiIiISImUPIuIiIiIlEjJs4iIiIhIiZQ8i4iIiIiUSMmziIiIiEiJlDyLiMScmZ1uZm5mc8sdi4hIpVPyLCIyAZnZ3DAhPr3csYhUAjN7h5k9a2Z7zey7ZlaT91jSzP5gZm8vZ4xSHZQ8i4iIyIRmZtOA7wKfBE4FTgIuzmvyEeAFd7+pDOGVzMxqJ+JrVRslzzFhZvea2XfM7HNmtj088/68mSXM7NNmts3MdpjZ5/OekzKzVWb2jJl1m9laM3tfwXE/ZGaPmFmHmW01sx+Z2ay8x3NfB7/WzH5jZl1m9riZvW48f3+RicrMTjGzB8zsQLitCfvXprDJPWEffDbvOR8ws81hf/w5ML8csYuMo8XAPne/wd0fBW4FjgEws2XAh4BLSjlQ+Ln4lJmdbWZPmlmnmd1jZksK2p1gZv8dfj7uMLObzWxB3uOLwn0vhH3xUTO7oOAYuc/uz5rZFuD5QWK6N+znhdtFeW0+EMbbbWbrzeyTZpbKe/zZMEf4ppntAh4I988KP9v3mtnB8LVWlvJeTVRKnuPlrUANcArBWfYngP8HNBOciX8U+ISZvT5s/5/AW4D3AUcDVwBfNrO/LzjuR4GXAG8m+BD+UZHX/nfgC8AKYDVwk5lNjuw3E4khM0sCtwN/AI4Pt1VAV3gb4FxgFvCy8DlnA/8BfA04Dvgx8L/HM26RMngKaDSzlWbWBLwK+JOZGfAd4BPuvnUEx5tFkGyfTzCKPRm4NvegmR0D/Br4HbASeDWQAX5hZvVhs2bgl8CZBJ+h1wDfNbO/Lnit/wW0A2eExynmLWFMue1zQCfwUBjPKoLP6n8l+Dz/EMFn+78VHOeDwHbglcCF4ftzK/Bi4E3Ay4Ft4e/RNuQ7NJG5u7YYbMC9wCMF+9YCjxbsW0OQ6C4CssCLCx7/dOFxCh7/K8CBOeH908P7b8lrMzPc97pyvy/atFXzBkwJ+9LpRR6bW+wx4H7ghoJ9/x62nVvu30mbtrHagLPCz7gNBCeQSeADwJ3h59Kt4WPfB5qHOM4qIA205+17R/iZWR/evw74UcHz6ghObM8Z4ti3Ad/Ou38vsA5IjOD3fB3QA5wV3m8MX/fMgnbvAvbm3X8W+GVBmzPC/xuOKfg9tgCfLve/abm2/uF6iYU1Bfe3hlvhvukEZ8oGrA5OPPulCM6egaAsg+BM9hiCM+/ctxkLGPj10iO5G+6+1cwywIwj/D1EBHD3PWb2n8DPzexXBCNdt7j7X4Z42jHAjQX77gf+aYzCFKkI7n4HcEfuvpktBP4FOBH4P8ATBN/Q/gC4HPjnIQ73grvvyLv/PMFn5nTgOYJvepaaWUfB8+qBZeHrNxIMSJ1FMFpcS5CY3lPwnIfdPVvK72hmywm+Tfrn8PcFWA40AD81M89rngTqzaw973d5sOCQy4Fd7v54boe795jZH8LHYknJc7z0Fdz3QfYlOJQEn0RwxlrYBjObD9wFXE9Q0rGTYLTrfwj+E8jXWyQelQ2JjJK7v9fMvg78DfBa4LNmdhnBaNqgTxuX4EQq27eBVe6+2cxeA3zG3dNm9gPgM8M8t/AzLdenEnk/rwe+VOS5u8Kf/xs4m+DE9UmCMouvApMK2ncO94sAmNl0glLMH7j7/5f3UC6mtxGMYhfaPcxrFfv/wgbZHwtKnmUwD4c/57v7/xukzcsIzmY/7O4HIbhAYjyCE5FD3P0x4DHga2b2LYJZBG4JH04WNH8cOBn4Zt6+k8c8SJEKYmbvBczdvx3uShBcEwTB4M9oB3dWAy8Fnvaw1qGI0whKqG4KY0oARxHUFI+ImdUR9PknCeqW860FuoHF7n7XCA+9Fmgzs2Nyo8/ha72cgf+HxIqSZynK3Z8ys2uBb5vZxwkuemgCTiCo8/oysJ7gzPOfzOwGgosBP12umEXixsyWAu8l+Cp6EzCb4OLfPxJ8E9QB/I2ZrQV63H0PwcjWf5nZgwTfHJ0CXFDk8CITkpnNIbhQ7pS83b8BPhLOOPV+ghKo0fgCQQnED8JvhnYAC4FzgK+7+wbgL8DZZvZTgr76EYI+POLkGbg6fO67gfa8cst97t5hZl8AvhDu/wVB/vcS4K/cfajylF+Fv8cPzez9wD6CkpZ64KojiHNC0NfmMpSLCS6q+CTBaNUvgQsJLqjA3f9McLHF+8LHPwp8uCyRisRTJ0H95I8Ivo79KfBb4LKwRvL9BFfqbwL+BODutxB8Tfxx4M8EswUM9eEpMtFcDXzJ3Z/N2/dBguR2NUG/Gq5sY0ju/gRB2WMz8HOCz8hvE3xbuzds9o/ARoIa518S1E3/5Ahf8nSC+P9CcDFfbnt7GM9nw9d7D8H1T/eH958d5vdwgoT/SYJSsIcILq58rbvvPMJYq54N/m2CiIiIiIjk08iziIiIiEiJlDyLiIiIiJRIybOIiIiISImUPIuIiIiIlEjJs4iIiIhIiapqnue2tjZfuHBhucMQqRgPP/zwTndvL3ccxai/igxUyf0V1GdFCg3WZ6sqeV64cCGrV68udxgiFcPMNpY7hsGov4oMVMn9FdRnRQoN1mdVtiEiIiIiUiIlzyIiIiIiJYo0eTazqWZ2i5l1mtlGMztvkHZmZl82s13h9hXLW4hdREREDjGzy8xstZn1mNl1w7T9RzPbamb7zOxaM6sbpzBFYiHqkecrgV5gBnA+cJWZLS/S7mKCtdJXAC8F3gS8L+JYREREJooXgM8B1w7VyMxeB/wLcAawEFgMfGasgxOJk8guGDSzJuBc4Fh37wDuN7PbgQsIOnK+C4Gvuvvm8LlfBd4LfGs0MXz7NxvoSWe47NXLRnMYkVgws8uAi4CXADe6+0WDtLsI+A5wMG/3m9z93ijiuOvRLcxoreeEBVOiOJzIhOTuNwOY2Upg7hBNLwS+4+5rw/afBW7g8M/hMffbp3Zy12NbSJjxwTOW0dZ8ZAPg2/Z3c9W9T9PZk444Qomz179kJq9+8Ywjem6Us20cBWTcfV3evjXAq4q0XR4+lt+u2Aj1iDzw9E52d/YqeRYpTW4k63VAwzBtf+fup4xFEF+5+0leOneykmeRaCwHbsu7vwaYYWbT3H1XYWMzu5jg22Dmz58fWRDP7eri77+3moRBZ2+G6S11/Z/NG3d1sn5bB2ccPZ3BKjY7etLc9sjzHOhO85/3bWB/d5q2ptrI4hN5ydxJR/zcKJPnZmBfwb59QEsJbfcBzWZm7u75DUfSsRtrk2zekxlh2CLxNIKRrDGVcSeT9eEbikgpin2+QvBZfFjy7O7XANcArFy5MpKOmM06H/3JGlIJ478/chqX3vBH7l67lctevYy7Ht3Cx3/yZzp60pxz3Gz+18p5UJA/H+zN8Pm7nmDDjk4Alk1v5sb3voJlM4qlEyLjL8rkuQNoLdjXChwooW0r0FGYOMPIOnZDTYqDvUqeRcbAX5nZTmA3cD3wRXeP5DvUbBayh3d9ETkyxT5fofhn8Zj41ZPbefCZ3XzpLS9h1qQGXn/sTL5w15Pc/MfNfOTHazhu3mROXjqNb977NLc+8kLRY7Q113HDe07k2DmTaKlLkUhoTgGpHFEmz+uAlJktc/f14b4VwNoibdeGjz04TLsRaaxN0tWrmiiRiP0GOBbYSPCV8E1AGvhiYcMj+Qo4k9XIs0iEcp+vPw7vrwC2FSvZGCt3PbqFSQ01nHtC8IXWmctn8YW7nuSj/7WGOZMbuOE9J9JUl+KtJ8xj2/7uosc4emYrkxprxitkkRGJLHl2904zuxm4wszeAxwHnA2cVKT594GPmNldgAP/BHxjtDEEybNGnkWi5O4b8u4+amZXAB+jSPJ8JF8BZ9018iwyDDNLEXxmJ4GkmdUD6SLfAH0fuM7MbgC2AJ8CrhuvOHvTWX7xxDZet3wmNclgQq/50xo5ZlYrj2/Zz+fffCxNdUHqsaitiUVtTeMVmkhkop6q7lKCC4+2AzcCl7j7WjM71cw68tpdDdwBPAo8BtwZ7huVhtokPemsRrFExpZzWJXikcuq5lmkFJ8imPHmX4B3hrc/ZWbzzazDzOYDuPvdwFeAewi+LdoI/Nt4BfnA0zs50J3m9cfOHLD/42e+iE++4WhOf9H08QpFZMxEWbaBu+8mmL+5cP99BBcx5O478PFwi0xjbRKAg30Zmusi/dVEJpxSR7LM7PXAH919m5m9GLgc+K+o4shknYxyZ5EhufsqYNUgDzfn33H3rwFfG+OQirr70a0016U4ZVnbgP2nv2i6EmeZMCbU8tyNtUHCrLpnkZKUNJJFsNjCn82sE7gLuBn4QlRBZB2KXCssIlVm/bYD3PKn53njS2ZRl0qWOxyRMTOhhmf7R55V9ywyrFJHstz9o8BHxyqOrC4YFKl6mazzsZ/8maa6JB8780XlDkdkTE2wkecgedZFgyLVQ/M8i1S//3liG49s2svlbzrmiFcSFKkWEyp5bugv21DyLFItNNuGSPV7answJ8CZBRcKikxEEyp5PjTyrJpnkWoRLJJS7ihEZDQ27e6irbm2/9ojkYlsQiXPDTUq2xCpNirbEKl+m/Z0MXdKY7nDEBkXEyp51gWDItVHZRsi1W/T7oPMm6rkWeJhgiXPqnkWqSbujjsaeRapYpms88Leg8yb0lDuUETGxYRKnhtU8yxSVXJJs5Jnkeq1Zd9B0lnXyLPExoRKnlW2IVJdcjmzqjZEqtem3QcBmKeaZ4mJCZU81yQT1CSNrj4lzyLVIFfrnFH2LFK1Nu3pAmDeVJVtSDxMqOQZghk3NPIsUh1y5RpZlW2IVK3Nu7tIGMyerORZ4mHCJc+NtSnVPItUCY08i1S/TXsOMmtSAzXJCZdSiBQ14f7SG2uTmm1DpEpks+FPJc8iVWvT7i7maqYNiZEJlzw31KpsQ6Ra5Eacc0m0iFSf53Z3MV8zbUiMTLjkWSPPItWjv2xDNc8iVelAdx/bD/SwqL2p3KGIjJsJmDyn6OrL8Ksnt3H1r58udzgiMoTchYKqeRapTht2dAKwpL25zJGIjJ8JmDwnOdib5vrfbeTLdz/Jtv3d5Q5JRAZxqGxDybNINdqwswOAJRp5lhiZcMlzQ22Szp4Mz+3uIutwy5+eL3dIIjKIXM6sCwZFqtPT2ztJJoz5U5U8S3xMuOQ5qHlOs3lPsOLRTx7ejOuDWaQiZbU8t0hV27Czg/lTG6lNTbh0QmRQE+6vvbE2xZ6uPnrSWZbPbuWp7R08+MzucoclIkX0L5Ki3FlkWGY21cxuMbNOM9toZucN0m6ymX3PzLaH26qxiunp7Z0q2ZDYiSR5LrVDh23/2szuMbN9ZvZsFK+fr6Em2X/7H161hBmtdfz991bzqye3Rf1SIjJKmm1DZESuBHqBGcD5wFVmtrxIu/8AGoGFwMuBC8zs3VEHk8k6z+zqZLEuFpSYiWrkudQODdAJXAt8LKLXHqCx9lDyfMzsVm59/8nMm9rIB298RB/QIhUmlzyr5llkaGbWBJwLXO7uHe5+P3A7cEGR5mcBX3H3Lnd/FvgO8HdRx/T8noP0prMsbtPIs8TLqJPnEXZo3P1Bd78e2DDa1y4mlzybwZzJDcya1MBFJy2goyfN5j1dY/GSInKEMlphUKRURwEZd1+Xt28NMNhAlRXcPrZoI7OLzWy1ma3esWPHiAJ6OjfTxnSNPEu8RDHyPNIOPSIj7dgNtSkAZrbWUx+WcCyb0QLAum0dUYQkIhFR2YZIyZqBfQX79gEtRdreDfyLmbWY2VKCUeeiSwC6+zXuvtLdV7a3t48ooAfW7wQ0x7PETxTJ80g69IiNtGPnRp7nTTn0/8Sy8Kx4/fYDUYQkIhHJv2BQs+KIDKkDaC3Y1woU+2D7IHAQWA/cBtwIbI4ymGd2dvK93z3LW0+Yy9Sm2igPLVLxhk2ezexeM/NBtvsZWYcecw255HnqoeS5pb6GWZPqWa+RZ5GKkl+uocFnkSGtA1Jmtixv3wpgbWFDd9/t7ue7+0x3X07wWf9glMF8/s4nqEsl+fiZL4rysCJVITVcA3c/fajHw5rnlJktc/f14e6iHXo8NNbkkueGAfuXzWjRyLNIhclPmLPuJAeUaYpIjrt3mtnNwBVm9h7gOOBs4KTCtma2BNgbbn8DXAy8Ksp47v3Ldi545QKmt9RHeViRqjDqsg137wRyHbrJzE4m6NDXF2tvZgkzqwdqgrtWb2aRfefTVBecD8yfOrC866jpzTy1vUPLAItUkPxaZ9U9iwzrUqAB2E5QinGJu681s1PNLP+r1ROARwm+Af4icL67Rzag5e6ks05LfU1UhxSpKsOOPJfoUoLp57YDuwg7NICZnQr8zN1zVxScBtyT99yDwK+B06MI5JhZraw66xjOPHbmgP3LZjTT3Zdl054uFkzTtDoilWBg2YaSZ5GhuPtu4Jwi++8juP4od//HwI/HKo7ceW4qoW+KJJ4iSZ4H69DhY4Wd+l4Yu+9mEwnjopMXHbY/N+PG+m0dSp5FKkRWI88iVSfXV5NKniWmJtzy3INZGs64sU51zyIVI6MLBkWqTi55TpiSZ4mn2CTPrfU1tLfUsXGnFkoRqRTZbP5tZc8i1SB30quyDYmr2CTPAAunNfLMrs5yhyEiofw654xqnkWqQiYTjjwreZaYilXyvGBaExuVPItUjAFlGxp5FqkKGnmWuItV8rxwWiPb9vfQ1ZsudygiQsEFgxp5FqkK6bDeSiPPElfxSp7bglk2ntutumeRSjBwkZTyxSEipctdq6CRZ4mreCXP4RR1z+5U6YZIJcifnk5lGyLVIfctUVKzbUhMxSp5nj8tWHXw2V0aeRapBAMuGFTyLFIVchcMap5niatYJc+t9TVMa6rVRYMiFUKzbYhUn/6RZyXPElOxSp4BFkxr5BmVbYhUhPzRZlfyLFIVMmHRs5JniavYJc8L25rYqLINkYowsGyjjIGISMlyfVXJs8RV/JLnaU1s2ddNd1+m3KGIlJWZXWZmq82sx8yuG6btP5rZVjPbZ2bXmlldFDHkrzCommeR6tA/VZ0uGJSYil3yPHdKAwDP7z1Y5khEyu4F4HPAtUM1MrPXAf8CnAEsBBYDn4kigAGLpKhsQ6QqaKo6ibsYJs/BjBub9yh5lnhz95vd/VZg1zBNLwS+4+5r3X0P8FngoihiGLBIikaeRSpWNuvsOBAsMpZWzbPEXAyT53DkWcmzSKmWA2vy7q8BZpjZtNEeeOAiKUqeRSrV/u4+Xvb5/+Gmhzb191UlzxJXsUueZ7TWk0oYm/fookGREjUD+/Lu5263FDY0s4vDOurVO3bsGPbAKtsQqQ65RDmTddKa51liLnbJczJhzJpcr7INkdJ1AK1593O3DxQ2dPdr3H2lu69sb28f9sADyzZGGaWIjJlUIkgX0lnXPM8Se7FLngHmTm7UBYMipVsLrMi7vwLY5u7D1UoPSysMilSH/JHnXF9V8ixxFc/keUqDyjYk9swsZWb1QBJImlm9maWKNP0+8PdmdoyZTQE+BVwXRQxaJEWkOuRm1ujLZJU8S+zFMnmeM6WBbft76ElrrmeJtU8BBwmmoXtnePtTZjbfzDrMbD6Au98NfAW4B9gYbv8WRQBanlukNGY21cxuMbNOM9toZucN0q7OzL5lZtvMbLeZ3WFmc0b7+omEkbDghLf/gkHN8ywxFUnyXGqnDtt+zMweM7MDZvaMmX0sihhGIjdd3Za93eP90iIVw91XubsVbKvc/Tl3b3b35/Lafs3dZ7h7q7u/2917ooghv1JDZRsiQ7oS6AVmAOcDV5nZ8iLtPgS8EngpMBvYC3wjigBSiQRpXTAoEtnIc6mdGsCAdwFTgDOBy8zsHRHFUZLcdHW6aFCkvPITZs22IVKcmTUB5wKXu3uHu98P3A5cUKT5IuDn7r7N3buBHxFMNzlqyYQNHHlW8iwxNerkeYSdGnf/irv/0d3T7v4X4Dbg5NHGMRKHkmfVPYuUk2bbECnJUUDG3dfl7VtD8aT4O8DJZjbbzBoJBrR+FkUQqYSRzjhp1TxLzEUx8jySTj2AmRlwKsHV/ONmZms9yYRp5FmkzLRIikhJCudaJ7x/2FzrwDrgOeB5YD9wNHDFYAceydzsyaSRyeqCQZEokueRdOpCq8IYvjtYg5EuulCKVDLBzNZ6XtB0dSJlNWCRFNU8iwymcK51wvuHzbUOXAXUA9OAJuBmhhh5Hsnc7KmEBfM8Z3XBoMTbsMmzmd1rZj7Idj8j69T5x72MoPb5jUNdfDTSRRdKNXtyveZ6FimzAWUbGnkWGcw6IGVmy/L2raD4t7YrgOvcfXf42foN4OVm1jbaIJJh2YZGniXuhk2e3f30Ilfk57ZTGFmnBsDM/o5geqwz3H3zaH+JIzF7cgMv7FPyLFJOWiRFZHju3kkwgnyFmTWZ2cnA2cD1RZo/BLzLzCaZWQ1wKfCCu+8cbRy52TaUPEvcjbpsY4SdGjM7H/gC8Fp33zDa1z9Ssyc3sHVft74qFimj/NFmDTyLDOlSoAHYDtwIXOLua83sVDPryGv3UaAbWA/sAN4AvDmKAFK5muews6aUPEtMFVtN7EhcClxL0Kl3EXZqADM7FfiZuzeHbT9HUIv1kB2ql/qBu/9DRLGUZPbkBvoyzs6OHqa31o/nS4tIaOBsG8qeRQbj7ruBc4rsv4/g2qPc/V0EM2xELllQ85xQ8iwxFUnyPFinDh8r7NiLonjN0ZozOUiYn997UMmzSJkMWCRFQ88iFS0VzvOcS5418ixxFcvluSEYeQZ4QasMipTNgEVSNPIsUtGSBTXPGnmWuIpt8jxrUi551kWDIuUy4IJBjTyLVDSNPIsEYps8t9anaK5Labo6kTLKT5418CxS2ZIJoy9z6ILBhOZ5lpiKbfJsZsyeXM8WTVcnUjaZ7KHprlS2IVLZ+keeM5qqTuIttskzhHM9q+ZZpGyyWacmGXwAa7YNkcqWSoazbbhWGJR4U/Kssg2Rssm6U5NI9N8WkcqVSiT6a57NdMGgxFesk+c5kxvY1dlLd1+m3KGIxFLGnVQ48qzkWaSy5c/zrIsFJc5inzwDbNrdVeZIROIpKNsI/hvKZMscjIgMKah5zpLJui4WlFiLdfK8dHqwdstT2zuGaSkiYyHr9CfPGnkWqWzJhJHOaORZJNbJ85L2ZsxgvZJnkbLIuC4YFKkWuQsG01lXvbPEWqyT54baJHOnNCh5FimTbNZJaeRZpCokwwsGs66RZ4m3WCfPAMumt7B+24FyhyESS/kfwprnWaSypRJGOpslnXXN8SyxFvvkeen0Zjbs7CStq5VExl0mGyxYlEyYlucWqXCphJHJOFklzxJzSp6nN9ObzrJpj+Z7FhlvWXeSiWCxBZ2/ilS2/kVSsq4FUiTWYp88Lwtn3FDphsj4y3rwIZxIqOZZpNIlc8tz64JBibnYJ8+56ep00aDI+AtWKjOSZqp5FqlwqUSif3luXTAocRb75LmlvoZZk+o117NIGQRlG0bCVPMsUumCeZ6zmqpOYi/2yTPA4vYmNuxQ8iwy3rJZwrINjTyLVLpUuDx3VoukSMwpeQYWtwUzbrhGvkTGVcYdMzTbhkgVyNU8p7U8t8SckmdgUVsTB7rT7OrsLXcoIrGSm/IqYYYGnkWGZmZTzewWM+s0s41mdt4g7X5mZh15W6+ZPTra108lE/2zbaSSSp4lvlLlDqASLGpvAuCZnZ20NdeVORqR+DhU86xFUkRKcCXQC8wAjgPuNLM17r42v5G7vz7/vpndC/xqtC+eK9Xoy2Q1VZ3EWiQjz6WeDYdtP2xmG8xsv5m9YGb/YWZlTeKXtAUzbqjuWWR8ZTxvkRQlzyKDMrMm4FzgcnfvcPf7gduBC4Z53kLgVOD60caQWxilJ53VIikSa1GVbeSfDZ8PXGVmywdpewdwvLu3AscCK4APRhTHEZkzpYGapLFhZ2c5wxCJnWzWSRqabUNkeEcBGXdfl7dvDTDYZ23Ou4D73P2Z0QaQUvIsAkSQPI/0bNjdn3b3vbmnA1lg6WjjGI1kwlgwrYlndih5FhlPubKNpGbbEBlOM7CvYN8+oGWY570LuG6wB83sYjNbbWard+zYMeSB+kee+zJKniXWohh5HvHZsJmdZ2b7gZ0EI89XD9G25I49GovbmnhGI88i4yq3SErC0AWDIkPrAFoL9rUCgy6Pa2anADOBnwzWxt2vcfeV7r6yvb19yAByI8+9GnmWmIsieR7x2bC7/zAs2zgK+BawbYi2JXfs0VjU3sTGXV2quxQZR4eW51bZhsgw1gEpM1uWt28FsHaQ9gAXAje7eyQX9CSTQcrQk85qqjqJtWGTZzO718x8kO1+juBsOMfd1xN0/G8eSfBRWtzWRG8my+Y9XeUORSQ2sh58FazluUWG5u6dwM3AFWbWZGYnA2czyIWAZtYAvI0hSjZGqiY38pzJapEUibVhk2d3P93dbZDtFI7sbDhfClgy8tCjtXz2JAD++NyeMkciEh/ZbN4iKUqeRYZzKdAAbAduBC5x97VmdqqZFY4un0PwLfA9Ub24ap5FAqMu2ziCs+H3mNn08PYxwL8CvxxtHKN1zKxWpjbV8pt1O8sdikhsZFyLpIiUyt13u/s57t7k7vPd/Yfh/vvcvbmg7Y3uvsAjXDo3tzBKb0Y1zxJvUU1VV/RsGKDIGfHJwKNm1gncFW6fiCiOI5ZIGKcsbeO+9Tv09bHIODlU8xzcFpHKlUwcqnlW8ixxFsniJO6+m+AromKP3UdwUWHu/rujeM2xcNpR7SHQsxQAABkOSURBVNy+5gWe2Lq/v4xDRMZONhsukmIq2xCpdLk6Z/dDibRIHOmvP89py9oAVLohMk4yWSeZCL750cizSGXLH21OauBZYkzJc57prfW8eGYL960fu/mkRSqFmU01s1vMrNPMNprZeYO0W2VmfWbWkbctjiKG/kVSNPIsUvHyZ9jQyLPEmf76C5y0pI2HN+6hJ50pdygiY+1KoBeYAZwPXGVmgy1udJO7N+dtG6IIIOu5RVI08ixS6QaMPCt7kBjTn3+BVyyeSk86y5pNheu+iEwcZtYEnAtc7u4d7n4/cDtwwXjGkcnmXTCYHc9XFpGRqsnLmDXyLHGmv/4CL180FTP4/YZd5Q5FZCwdBWTcfV3evjXAYCPPZ5nZbjNba2aXDHZQM7vYzFab2eodO4Yvf+pfJEUrDIpUPI08iwT0519gcmMtR89sVfIsE10zwQIK+fYBLUXa/hg4GmgH3gt82sz+tthB3f0ad1/p7ivb29uHDSK3SEpCNc8iFS+/5jmlkWeJMf31F/GKxdNU9ywTXQfQWrCvFThQ2NDdH3f3F9w94+6/Bb4OvDWKIDK5eZ7NiHAtBxEZA/kjzwnTdBsSX0qei1Dds8TAOiBlZsvy9q0A1pbwXAci+eTsn21DZRsiFS9/tFllGxJn+vMv4oQFUwD403N7yhyJyNhw907gZuAKM2sys5OBs4HrC9ua2dlmNsUCLwc+CNwWRRy5RVKCso0ojigiYyWpqepEACXPRU1rrmPe1AYe2bS33KGIjKVLgQZgO3AjcIm7rzWzU82sI6/dO4CnCEo6vg982d2/F0UAGQ8WSUkmgvpnEalcqaQuGBSBiJbnnoiOmzeF1c/uLncYImPG3XcD5xTZfx/BBYW5+0UvDoxCNqx5VtmGSOXTIikiAf31D+K4eZPZsq+bbfu7yx2KyITk7rgHZRumRVJEKt6AmmddMCgxpuR5EMfNmwyg0g2RMZKbmi63PLfKNkQqWzKvbCO/hEMkbpQ8D2L57FZSCVPyLDJGcrmyZtsQqQ4pTVUnAih5HlR9TZKjZ7XyyHNKnkXGQq5MI7dIipbnFqlsyQGLpCh5lvhS8jyE4+dPZs3mvaQ1h5ZI5PrLNsxIGKp5FqlwA0aelTxLjCl5HsIJC6fS1ZvhiS2HLbomIqOUS5b7yzZU8yxS0TTyLBJQ8jyEly0MFkt5SFPWiUQuV6ZhZiQSmm1DpNLV5E3urJFniTMlz0OYNamBOZMbeHijVhoUiVruAsGkBaUbGnkWqWwDVhjUBYMSY0qeh7Fy4RQeenY3rlExkUjlRpoTKtsQKYmZTTWzW8ys08w2mtl5Q7Q93sx+Y2YdZrbNzD402tfPT5hVtiFxFlnyPJJOnfecWjN70sw2RxVH1FYunMr2Az1s3nOw3KGITCi5eZ0TZpiBzk9FhnUl0AvMAM4HrjKz5YWNzKwNuBu4GpgGLAX+e7QvnkgEF/fmbovEVZQjzyV16gIfA7ZHGEPkVi4I6p4ffEZ1zyJRyvjARVI0z7PI4MysCTgXuNzdO9z9fuB24IIizT8C/Nzdb3D3Hnc/4O5PRBFHbpVBjTxLnEWSPI+wU+eeswh4J/DFKGIYKy+a0cKUxhp++/SucociMqHkqjQShso2RIZ3FJBx93V5+9YAxQapXgHsNrPfmtl2M7vDzOYXO6iZXWxmq81s9Y4dO4YNIlf3rJFnibOoRp5H0qlzvgF8AqjoeohEwnjlkmn87umdqnsWiVB+2YZm2xAZVjOwr2DfPqClSNu5wIXAh4D5wDPAjcUO6u7XuPtKd1/Z3t4+bBC5EWeNPEucRZU8j6RTY2ZvBlLufstwBx7pWfFYeOWSNl7Y182zu7rK8voiE1H/IimJ3CIpZQ5IpLJ1AK0F+1qBYgsRHARucfeH3L0b+AxwkplNGm0QyWQ48qzZNiTGSkqezexeM/NBtvsZQacOSzy+AnyglNce6VnxWDh5yTQAfvv0zrK8vshE1D/bhpmmqhMZ3jogZWbL8vatANYWaftnIL9D5W6POuNVzbNIicmzu5/u7jbIdgoj69TLgIXAfWa2FbgZmGVmW81s4Wh+mbGyqK2Jma31/PYp1T2LRCV/qrpc/WRWCbRIUe7eSfB5eYWZNZnZycDZwPVFmn8XeLOZHWdmNcDlwP3uvne0ceSS5qSSZ4mxSMo2RtipHwPmAceF23uAbeHtTVHEEzUz46Sl0/jdhl36cBeJSCZcYTAZjjwDmnFDZGiXAg0Es1TdCFzi7mvN7FQz68g1cvdfEVxTdGfYdikw7PSxpUgqeRYhFeGxLgWuJeiouwg7NYCZnQr8zN2b3T0NbM09ycx2A1l331rkmBXjlYuncfMfn2f99g5eNLNoKbeIjMChso1DV+5nsk5Ncmxe72Bvhv9+fCu3/ul5Ht+yn3TGeesJc3nHy+ezqK1pbF5UJELuvhs4p8j++wiuPcrfdxVwVdQxpJJKnkUiS54H69ThY4d17LzH7iW4MriivWJxUPf8+w27lDyLRCBX4xwsvBB8EI/FwPP6bQf40s+e5L6ndtKbzjJ3SgOnLG2no6eP/7z/Ga7+zQZWzJvMG46dyWuOmcHitiZMF0OJFNU/VZ36iMRYlCPPE9q8qY3MmdzA7zfs4sKTFpY7HJGqlxt5TpqRDAvIoi7buOEPG/nM7Y/TWJfk/BPn89pjZvCKRdP6R7q37e/mtkee5/Y1L/DFnz3JF3/2JHOnNHDSkmmcuGgaxy+YwsJpjUqmRUL9U9Ul1SckvpQ8j8ArFk/jnr9sJ5t1TRAvMkr9i6QkDo1iRTnjxs/XbuVTtz7Gacva+fe3raC9pe6wNjNa67n4tCVcfNoSNu/p4p6/7OC+dTv4+dpt/Hj1ZgAmNdTwkjmTOHpWC0fNCLbF7U201NdEFqtItUiGs21o5FniTMnzCLxyyTR++sfNqnsWiUAmb5GUZMSzbdy/ficf/tEjrJg7masvOIH6Egqp505p5IJXLOCCVywgm3We2tHBwxv3sGbTXta+sJ/v/W4jvelsf/u25joWTGtk3pQG5kxpYM7kRmZOqmNGaz3TW+qZ2lSrulCZcGqSWiRFRMnzCLxi8VQAHnhqp5JnkVHqL9vIq3mOYpXBmx56jk/c8hhL25v59rtWlpQ4F0okrH+U+W9fHqxqnM5k2bi7i6e2d/D0jg427uxi4+5OHnp2D3f8ectho+YJg6lNtUxrqmNyYw1TGmuZ3FjDpIYaWhtqaKlP0VKformuhqa6JE21KZrqkjTUpmisSdJQm6QulVDJiFQUzbYhouR5ROZOaWTZ9GbuXruVvztlUbnDEalqhctzw+hrnq974BlW3fE4px3VzpXn/VWkpRWpZIIl7c0saT/82udM1tm2v5ut+7vZvr+bbft72NnRw86OXnZ39rCns48NOzvY09XHvoN9A0awh1OXSlBfk6S+JkFdKkltKkFtMjHgZ03SqEkmqEkmSCWNVCLYl7udShipZPAzkTBSiWC0Pxnezo3+JxLWX4Oe25dMGBZOJ5ibGSVpRiIRTOOZyHvMcsex3GPhv68ZZoeOmf94fjvIXUDKgOck8toe2jewjTHwmDI2NM+ziJLnEXvDS2bxf361nu0HupneUl/ucESqVqZghUGAbOk55WGu/92zrLrjcV57zAyuPO94alORTGNfkmTCmD25gdmTG0pq392XYX93H509GTq60xzo6eNgb4bO3gwHe9Mc7M3Q1ZehuzdDdzpLd1+Gnr4svZksPekMveksPeksveksXb1pejNZ0hmnL5MlnXX60sHPdDbYl8k66YyTzmZjswx6sQTdKEi6E0HSHdwP9uXuF0vWD7tP+Jzw/jXvWsmcEv8GqpVGnkWUPI/YG186i6//cj0/f2wrF7xyYbnDEalauUQ5GN0Mbh/JyHM6k+Wm1Zu4/La1vObo6Xzz/OOpSY5f4nwkgpHkJJSh+svdyWSdTPgznXWy2UP7slnCn4f2uTtZD0bYM1nHPWwTPpbJhscNn+8cahfcZkDbrDtO8O2DEzwnty/3WrnXcA9fL+v9ZT3ZvDbZMNbguYTHCx8n+JkNHjjseR6+ZiZsD2HMeb/fgOeR97yC47l7LOqAc8tzK3mWOFPyPEJHzWhh6fRm7nx0i5JnkVHIXyTF+keeS0+e12zayy1/ep6fr93Kln3dvHLxNP7veZWfOJebWVjOUe5ApCr1jzyrNEZiTP9/HoE3hqUbz+3qYv60xnKHI1KV+ss2EnllGyWOPG/Zd5C3fet3JBJw8pI2/u2sYzjj6BlKnEXGmGqeRUCfNEfgvBPnk0oY1z7wTLlDEalauVHmZN5UdaXO83z1rzeQdecX//gqvnPRyzjz2FlKnEXGgWqeRZQ8H5EZrfWc9dLZ/Hj1JvZ19ZU7HJGq1L9ISt5sG6WMPO840MONDz7HW46fw7yp+uZHZDzlTlKVPEucKXk+Qu85dTFdvRlueHBjuUMRqUr9i6QkgrrnYN/Qz3F3vnjXE/Rlslxy+tIxjlBECmnkWUTJ8xE7ZnYrpy5r47sPPEtPOlPucESqTv4iKaXWPH/vt89y85+e50NnHMWitqYxj1FEBkrpgkERXTA4Gu87bQnv/M4fuPVPz/P2l80vdzgSsdyUXrm5coN5coO5cnNz6mayTl84f25fxknn9mWdTP++Q89JZ8P5d/Pm5E1nDu3Lv9/eUsf7/3rijq4emm3DaKwL/iv68I8e4ZNvPJrTjmo/rP1T2zv43J1P8Jqjp/OBV0/c90WkkvWPPCeVPEt8KXkehZOXTmP57Fau+c0G3nbCvP66TTnE3enNBIs59GU8/Bks9tDXvz9LbzpIJoPt0O10xsMFIIKEsjeTpS89MFnN7U/nFooIE9C+zMBEN11sfzZLJnxOXyZ3+1CiPJ4SRv8qcKmEcfSs1gmdPPeXbZhx6tI2vvq2FVx5z1O89/ur+eklJ3HsnEkD2n/+zsdpqEnypXNfqr4mUiappEaeRZQ8j4KZcenpS3n/D//Ijx7axHknVs/ocybrdPam6ehO09mTprM3Q1fuZ2+art4MXbnVzvoydPdlg5+9GbrTwWpnPbmVz8KVznrSwe3c/d50kCSPBTOoSeSWIh64NHFN8tByxLlli5MJo74mQaouFSxbnChsW2zfoccKXysZ/sztK3a8ZG6J5NxjiQTJ/vaH4sstnxy3hDBXoZFbFvrcE+byqhe1c9Y37ud91z/Mp954NCctaaO1IcVtj7zAPX/ZwSfe8GLamuvKG7hIjKnmWUTJ86i94SUzOXHRVL5895O8bvkMpo3zB3s6k2V3Vy+7OnrZ3Rlse7p62dvVx56uXvYd7GNfVx/7Dvaxv7uPA91pDnSn6ehJl/waCYOGmiQNtUnqUknqaxLU1ySpSyWoSyVpqU9Rl0pSV5OgNpmgNtxfm8rdDvbXJI3aVDL8mduXoCZ1KMmtSR5qm5/81uTdziWeUt0OjTwf2tfWXMdV7zyBd3/3QS654Y+YwfSWOrbt7+FFM1q48KSF5QlWRACtMCgCSp5Hzcz43DnH8vqv38fbr/k97zxxPq85ZgZzpxz5FFruzoGeNNv397D9QDc7DvT0395+oIcd4bazo4c9Q0yV11yXYlJDTf+2qK2J1voaWupraKlP0VKfoqku2JrrkjTVpmisTdFQm6SpLkljTYr62iCZNX1FJxHL5NU85ztu3mQe/ORreGTTXn739C6e2LKf045q5y3Hz6EulSxHqCIS0gWDIkqeI7FsRgv/97zjufKep1h1x+OsuuNx5kxu4OhZrcyf2sj01jpa6lP9SWg6E5RAdPVm2H8wGCHe3dnHrs4gId55oJeDfYfP4FFfk2B6Sz3tLXUsbm/ixMVTmdZUR1tzLdOa65jSWMvUpmCb3FijRSOkonnebBuFapIJXrZwKi9bOHW8wxKRISSThhmxKzMTyafkOSJnHjuTM4+dyVPbO/j1uh386bk9/GXrAR54amfRRDinNpVgSmMNUxpraWuuY8H8Rtqa65jeWsf0lnqmtwS321vqaa1PaQRYJoxcOXzhyLOIFGdmU4HvAH8D7AT+1d1/WKTdKuCTQE/e7pe6+4bRxjBncgOzJzWM9jAiVU3Jc8SWTm9m6fRmYBEQjK519Wbo6EnTm87iHlytnKshrq/R19AST2etmMXJS6fR1lxb7lBEqsWVQC8wAzgOuNPM1rj72iJtb3L3d0YdwDtPXMDbXzYv6sOKVJXIvtc3s6lmdouZdZrZRjM7b4i2q8ysz8w68rbFUcVSScyMproUM1rrmTe1kfnTGpk9uYEpTbVKnKWsSu2zFviyme0Kt69YBF+BtNTXsGBaky7+FCmBmTUB5wKXu3uHu98P3A5cMJ5xJBKmaw8k9qL81Mo/Iz4fuMrMlg/R/iZ3b87bRv11koiMSKl99mLgHGAF8FLgTcD7xitIEQHgKCDj7uvy9q0BBvucPcvMdpvZWjO7ZLCDmtnFZrbazFbv2LEjynhFJqxIkudKOSMWkdKMsM9eCHzV3Te7+/PAV4GLxi1YEQFoBvYV7NsHtBRp+2PgaKAdeC/waTP722IHdfdr3H2lu69sbz98ZU8ROVxUI88jPSMGnRWLlNNI+uzy8LHh2qm/ioydDqC1YF8rcKCwobs/7u4vuHvG3X8LfB146zjEKBILUSXPIzkjBp0Vi5TbSPpsYdt9QHOxumf1V5Exsw5ImdmyvH0rgGIXCxZyQNPaiESkpOTZzO41Mx9ku58RnBGDzopFKsBI+mxh21agw3MTNYvImHP3TuBm4AozazKzk4GzgesL25rZ2WY2JbzY9+XAB4HbxjdikYmrpKnq3P30oR4P6ydTZrbM3deHu0s9I4YSz4offvjhnWa2cZhmbQTzX0rp9J6NXKW8ZwuO8Hn9o1gl9Nm14WMPDtNuAPXXMaX3beQq4T070v6acylwLbAd2AVc4u5rzexU4Gfu3hy2e0fYrg7YDHzZ3b833MHVZ8eM3rORq5T3rGiftagGj8zsRwRJ8HsI5p+8Czip2PyTZnY28BtgL/Ay4BbgE6V07hLiWO3uK0d7nDjRezZyE+E9K7XPmtk/AB8CXhO2/wXwDXf/VgQxVP37WA5630ZO71k09D6OnN6zkav09yzKqeouBRoIzohvJDwjBjCzU82sI6/tO4CnCL4i/j4lnhWLSKSK9tki/fVq4A7gUeAx4M5wn4iISOxEtsKgu+8mmAu22GP3EVx0lLtf9OJAERk/g/XZIv3VgY+Hm4iISKxNxKW9ril3AFVI79nI6T2Lht7HI6P3beT0nkVD7+PI6T0buYp+zyKreRYRERERmegm4siziIiIiMiYUPIsIiIiIlKiCZM8m9lUM7vFzDrNbKOZnVfumCpRuOBNt5l1hNtf8h47L3zvOs3sVjObWs5Yy8HMLguXl+4xs+sKHjvDzJ40sy4zu8fMFuQ9Vmdm15rZfjPbamYfGffgq4j6a2nUX4enPjs+1GeHp/46vInSXydM8gxcCfQCM4DzgavMbHl5Q6pYl7l7c7i9CCB8r64GLiB4D7uAb5YxxnJ5AfgcwQID/cysjWB1r8uBqcBq4Ka8JquAZQQTqv818HEzO3Mc4q1W6q+lU38dmvrs+FCfLY3669AmRH+dEMmzBSscngtc7u4d7n4/cDvBH6qU5nzgDnf/jbt3EPwBv8XMWsoc17hy95vd/VaC1bvyvQVY6+7/5e7dBB15hZm9OHz8XcBn3X2Puz8BfBu4aJzCrirqr5FQfw2pz4499dlRU38NTZT+OiGSZ+AoIOPu6/L2rQF0VlzcF81sp5k9YGanh/uWE7xnALj70wSjDEeVIb5KVPj+dAJPA8vNbAowO/9x9Pc3FPXXkVF/PTLqs9FRny2d+uuRqar+GtkiKWXWDOwr2LcPiN1ZXQn+GXicoOO+A7jDzI5D7+FwmoEdBfty709z3v3Cx+Rw+lsrnfrrkVOfjY7+3kqj/nrkqqq/TpSR5w6gtWBfK8Hy35LH3f/g7gfcvSdcEv0B4A3oPRzOUO9PR979wsfkcPpbK5H666ioz0ZHf28lUH8dlarqrxMleV4HpMxsWd6+FcDaMsVTTRwwgvdqRW6nmS0G6gjeWzn8/WkClhDUaO0BtuQ/jv7+hqL+euTUX0unPhsd9dkjo/5auqrqrxMieQ5rY24GrjCzJjM7GTgbuL68kVUWM5tsZq8zs3ozS5nZ+cBpwM+BG4CzzOzU8I/2CuBmd4/VmXH4vtQDSSCZe6+AW4Bjzezc8PFPA3929yfDp34f+JSZTQkvcHgvcF0ZfoWKp/5aGvXX0qjPjj312eGpv5ZmwvRXd58QG8HUJrcCncBzwHnljqnSNqAdeIjgq469wO+B1+Y9fl743nUCtwFTyx1zGd6jVQSjBfnbqvCx1wBPAgeBe4GFec+rI5h6Zz+wDfhIuX+XSt7UX0t6j9RfS3uf1GfH531Wnx36/VF/Le19mhD91cKgRERERERkGBOibENEREREZDwoeRYRERERKZGSZxERERGREil5FhEREREpkZJnEREREZESKXkWERERESmRkmcRERERkRIpeRYRERERKZGSZxERERGREv3/iu+ht1/5Ym4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x216 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.activation_stats.plot_layer_stats(-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we've got most of our activations near zero. Let's see what else we can do to improve training stability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1cycle training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our initial weights are not well suited to the task we're trying to solve. Therefore, it is dangerous to begin training with a high learning rate: we may very well make the training diverge instantly, as we've seen above. We probably don't want to end training with a high learning rate either, so that we don't skip over a minimum. But we want to train at a high learning rate for the rest of training, because we'll be able to train more quickly. Therefore, we should change the learning rate during training, from low, to high, and then back to low again.\n",
    "\n",
    "Leslie Smith (yes, the same guy that invented the learning rate finder!) developed this idea in his article [Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://arxiv.org/abs/1708.07120) by designing a schedule for learning rate separated in two phases: one were the learning rate grows from the minimum value to the maximum value (*warm-up*) then one where it decreases back to the minimum value (*annealing*). Smith called this combination of approaches *1cycle training*.\n",
    "\n",
    "1cycle training allows us to use a much higher maximum learning rate than other types of training, which gives two benefits:\n",
    "\n",
    "- By training with higher learning rates, we train faster, a phenomenon Leslie N. Smith named *super-convergence*\n",
    "- By training with higher learning rates, we overfit less because we skip over the sharp local minimas to end-up in a smoother (and therefore more generalizable) part of the loss.\n",
    "\n",
    "The second point is an interesting and subtle idea; it is based on the observation that a model that generalises well is one whose loss would not change very much if you change the input by a small amount. If a model trains at a large learning rate for quite a while, and can find a good loss when doing so, it must have found an area that also generalises well, because it is jumping around a lot from batch to batch (that is basically the definition of a high learning rate). The problem is that, as we have discussed, just jumping to a high learning rate is more likely to result in diverging losses, rather than seeing your losses improve. So we don't just jump to a high learning rate. Instead, we start at a low learning rate, where our losses do not diverge, and we allow the optimiser to gradually find smoother and smoother areas of our parameters, by gradually going to higher and higher learning rates.\n",
    "\n",
    "Then, once we have found a nice smooth area for our parameters, we then want to find the very best part of that area, which means we have to bring out learning rates down again. This is why 1cycle training has a gradual learning rate warmup, and a gradual learning rate cooldown. Many researchers have found that in practice this approach leads to more accurate models, and trains more quickly. That is why it is the approach that is used by default for `fine_tune` in fastai.\n",
    "\n",
    "Later in this book we'll learn all about *momentum* in SGD. Briefly, momentum is a technique where the optimizer takes a step not only in the direction of the gradients, but also continues in the direction of previous steps. Leslie Smith introduced cyclical momentums in [A disciplined approach to neural network hyper-parameters: Part 1](https://arxiv.org/pdf/1803.09820.pdf). It suggests that the momentum vary in the opposite direction of the learning rate: when we are at high learning rate, we use less momentum, and we use more again in the annealing phase.\n",
    "\n",
    "We can use 1cycle training in fastai by calling `fit_one_cycle`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit(epochs=1, lr=0.06):\n",
    "    learn = Learner(dls, simple_cnn(), loss_func=F.cross_entropy,\n",
    "                    metrics=accuracy, cbs=ActivationStats(with_hist=True))\n",
    "    learn.fit_one_cycle(epochs, lr)\n",
    "    return learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.210838</td>\n",
       "      <td>0.084827</td>\n",
       "      <td>0.974300</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're finally making some progress! It's giving us a reasonable accuracy now.\n",
    "\n",
    "We can view the learning rate and momentum throughout training by calling `plot_sched` on `learn.recorder`. `learn.recorder` (as the name suggests) records everything that happens during training, including losses, metrics, and hyperparameters such as learning rate and momentum:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuUAAAD7CAYAAADNeeo8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3yV9fn/8deVvXcIMwsSRoAECFtw7wEuVBSxWrFqqxattbb+3FqtqK1aFfcoqCiKew9AhRBG2DOEQBhZZO/k8/vjHPqNaZAEknOfcT0fj/Oo3PeHc96nmMPlfa77+ogxBqWUUkoppZR1vKwOoJRSSimllKfTolwppZRSSimLaVGulFJKKaWUxbQoV0oppZRSymJalCullFJKKWUxH6sDOEpMTIxJTEy0OoZSSnXaypUri40xsVbncCT9zFZKuaqj/cz2mKI8MTGR7Oxsq2MopVSnicguqzM4mn5mK6Vc1dF+Zmv7ilJKKaWUUhbTolwppZRSSimLaVGulFJKKaWUxbQoV0oppZRSymJalCullFJKKWUxhxXlIhIlIu+LSLWI7BKR6YdZJyLyiIiU2B+Pioi0Ou8tIg+IyF4RqRSR1SIS4aj3oZRSSimlVFdz5EjEZ4AGIA7IAD4RkRxjzIY262YBU4F0wABfAbnAc/bz9wITgPFAPpAG1HV7eqWUUkoppbqJQ4pyEQkGLgSGGmOqgKUi8iEwA7ijzfKZwBxjzB77750DXAs8JyKRwC1AujHm0AzI9Y54D57OGMPm/ZV8u7mQ+sZmAPx9vQkP9CUyyI9eEQH0jQgkNtSfVl9sKKWUQzS3GP7+2SZGxEcyOjGK2FB/qyMppVxMWU0D+aU1FFbUU1rdQGlNA1dPTMLPxzGNJY66Up4KNBtjtrY6lgMc387aNPu51uvS7P88DGgCLhKRPwIVwD+NMc+096IiMgvblXfi4+OP6Q14so/X7uXJr7exvbAKABEwpv21of4+pMSFMKR3GCPjIxmVEEl8VJAW6kqpbrW7tIY3lu3ihSU7ARiXHMU/LkqnX1SQxcmUUs6ooKyWFTtLWbO7jI37Ktiyv5Ly2sb/WTcloze9wgMdkslRRXkIUN7mWDkQ2oG15UCIva+8LxCOrchPAlKAb0RkqzHmq7ZPZIyZC8wFyMzMPEwZqQ6nqbmFR7/YwtzFuaT1DuOBqUM5c2hPokNsV6DqGpspr22ktLqBfeW17DlYy/bCKjbvr+SD1Xt5c1k+AH0jA5mcGsvJg3pwXEoM/j7eVr4tpZQbSowJZu3dp7N+bzk/7yjh2e93cNY/l/DA+UOZktHH6nhKKYvVNzXz4/ZivttcxPdbC9ldWgtAkJ83g3qGcvbwXiTHBNMvKoieYQFEBfsRFexHkJ/jahZHFeVVQFibY2FAZQfWhgFVxhgjIrX2Y/cZY2qBtSLyFnAWtt5z1UUamlq49vVsfthaxJXjE/jb2UP+5+ubAF9vAny9iQsLYHCvX/7xNrcYthVWsmJnKYu3FbNodQHzlucT6u/DqUPiuGhUX8YlR+PlpVfQlVJdw8/Hi5HxkYyMj+S89N7c8vYabn5rDbUNzVw6Rr8tVcrTtLQYftpRwvurC/hy434q65oI8vNmQv8YrpmYxOikKAb1DMPbSWoRRxXlWwEfEUkxxmyzH0sH2t7kif1YOpDVzrq19v/Vq97d7P6PN/LD1iIemDqUK8YldPr3e3sJg3qGMahnGDPGJ9LQ1MKPO4r5bN0+Plu/n4WrC+gbGcj0sfFcNjqeyGC/bngXSilP1S8qiLdnjePq17L52wfriY8KYsKAGKtjKaUcoKSqnvlZ+by1Yjd7DtYSGuDD6Wk9OXt4Lyb0j3bab+zFHK45uKtfyHZF2wC/xTZ95VNgQtvpKyLyO+Bm4BT+b/rKU8aY5+znFwObgJuAZOAH4DJjzDe/9vqZmZkmOzu7S9+Tu3p7RT5/fm8d101O5i9nDe7y569rbOaLDft5K2s3P+eW4O/jxYWj+vK7yf2Jj9b+T6XaEpGVxphMq3M4Uld9ZlfUNXLhv3/iQEUd7984kf6xIV2QTinljHKLqnhhSS4LVxVQ39TChP7RXDomntOGxBHg67hC/Gg/sx05EvEG4GWgECgBrjfGbBCRScBnxphDn5TPYyu219l//aL92CGXAS/Zn6MQuOtIBbnquPUF5dz1wQYmpcRw+xmDuuU1Any9mZLRhykZfdiyv5JXf9rJu9l7eHvFbqZk9Obmk1NIiA7ultdWSnmWsABfXr5qNOc9vZTbFuTw3u8maNucUm5mR1EVT32zjQ9z9uLr7cUFI/tyzXGJDOjR3q2LzsthV8qtplfKj6ylxXDBsz+x52AtX8+eTESQ41pKDlTUMXdxLv9ZvoumZsNlY+K56eQUHWumFHqlvCu8t3IPty7I4aHzhzF9rPaXK+UOCivqeOLrbbyTvRs/by9mjE/g2knJltcOrnClXDm5d1fuYc3uMh6flu7QghwgLiyAu84ZwqzJyfzrm23My8rng9UF3HRyCjMnJDpsRqhSyj1dMLIP72Tv5pHPN3N6Wtx/p0gppVxPfVMzLy7ZydPfbqeppYUZ4xL4/UkDiHHxn2utdBQA5TWN/P3zzWQmRHL+COvGh8WFBfDg+cP46o+TyUyM5MFPN3HGPxezLLfEskxKKdcnIjwwdSjV9U08/Nlmq+MopY7SD1uLOP2Jxfzjiy0cnxrL17OP557z0ly+IActypXdE19vpaymgfumDHWKjX6SY0N45TdjePmqTBqbW7h07jJufzeH8pr/HeyvlFIdkRIXyjXHJfHeqj1sL2xvIq9SylmVVjdwy1urmflyFl5ewutXj+G5GaPc6h40LcoVhRV1zMvKZ1pmP4b0bjtO3lonDYrjy1uO53fH9+e9VQWc9uQPfL+l0OpYSikXdd3x/Qnw8ebf3+2wOopSqoM+X7+fUx7/gU/W7eOmk1P47OZJTE6NtTpWl9OiXPHCklyamlu4/oT+VkdpV6CfN3ecOYgPbphIeKAvV72ygjvfX0dtQ7PV0ZRSLiYq2I8rxsWzKGcvu0qqrY6jlPoVFXWNzH5nDb97cyW9IwL4+A+TmH1qqtPOGT9WWpR7uIPVDfxneT7npfd2+q+AhvUN58PfH8esycnMW57PuU8vZePeCqtjKaVczLWTkvH2Ep77Qa+WK+WscnaXcc6/lrJozV5uOmkA798wkYE9XWvEYWdpUe7hXvlxJzUNzdxw4gCro3RIgK83d541mDevGUt5bSNT//0j87Py8ZTRnkqpY9cjLIBLMvvx7so97C2rtTqOUqoVYwwvLsnloud+ornF8Pasccw+bSC+3u5fsrr/O1SHVV3fxKs/5XHakDhS41zrvz6PS4nh85snMTYpir8sXMdtC9ZqO4tSqsOuOz6Z5hbDvOX5VkdRStlV1Tdx47xVPPDJJk4c2INPbjqOzMQoq2M5jBblHmzRmr1U1DVx3fHJVkc5KtEh/rz6mzHcdHIK763aw8XP/6RXvZRSHdI3MogTB/bg7ezdNDa3WB1HKY+XW1TF1Gd+5PP1+/nLmYN4fsYoh++ZYjUtyj3YvKxdDOoZysj4SKujHDVvL2H2qam8NDOTvOIaznt6Kdl5pVbHUkq5gMvHxVNUWc/XGw9YHUUpj7ZkWxFTn/mRkqp63rxmLNcd398pxjM7mhblHmrtnjLWF1QwfWy8W/yLf/LgOD64cQIh/j5Mf2E5H6wusDqSUi5NRKJE5H0RqRaRXSIy/TDrIkTkNREptD/uOcy640XEiMgD3Rq8E45P7UGfiED+oy0sSlnmjZ/zuOqVFfQKD+TD3x/HhAExVkeyjBblHuo/y/IJ9PVmqoW7d3a1AT1CWXTjcYxMiOCWt9fw5Ndb9QZQpY7eM0ADEAdcDjwrImntrHsCCAISgTHADBH5TesFIuIL/BNY3p2BO8vbS7hkdD+Wbi8mr1jHIyrlSC0thgc/2chdizZwQmos790wgX5RQVbHspQW5R6ooq6RD3P2cl56b8ICfK2O06XCg3x5/eqxXDiyL09+vY0/v7eWJu0XVapTRCQYuBC4yxhTZYxZCnwIzGhn+bnAo8aYGmNMHvAScHWbNbcCXwJOt7/9JaP74e0lzF+hV8uVcpS6xmZ+P38VLyzZyczxCcy9MpMQfx+rY1lOi3IPtGjNXmobm7l8XLzVUbqFn48Xj108nJtOGsA72Xu47o2VOplFqc5JBZqNMVtbHcsB2rtSDiBt/nnof38hkoCtSL/vSC8qIrNEJFtEsouKijqf+ijEhQVw4sBYPlhdQHOLfrOmVHerrGvkqley+HTdfv561mDuOS8Nby/Xb6PtClqUe6APVhcwMC6U4X0jrI7SbUSE2acN5P6pQ/l2SyEzXlpOeW2j1bGUchUhQHmbY+VAe7NTPwfuEJFQERmArQBv/R30v7BfcT/Sixpj5hpjMo0xmbGxjttCe0pGHw5U1JO1U28SV6o7FVfVc+ncZWTnHeTJSzK4dnKyW9zX1lW0KPcwu0trWLnrIFNG9LY6ikPMGJfA05eNJGdPGdNfWEZJVb3VkZRyBVVAWJtjYUBlO2tvAmqBbcAiYD6wB0BEzgVCjTFvd1/UY3fK4DiC/Lz5MEdvEFequ+wrr2Xa8z+zo6iKF2dmutU9bV1Fi3IP82HOXgDOS/eMohzg7OG9mHtlJtsLq7hk7jIKK+qsjqSUs9sK+IhISqtj6cCGtguNMaXGmMuNMT2NMWnY/l7Jsp8+GcgUkf0ish+4BLhFRBZ1c/5OCfTz5rQhcXy6bj8NTXoPilJdLb+khouf+5nCinreuGYsJwzsYXUkp6RFuQcxxvDB6gJGJ0bSN9Kz7nA+cWAPXr96DHvLarl07jL2l2thrtThGGOqgYXAfSISLCITgSnAG23Xikh/EYkWEW8ROROYBRwae3gXtv70DPvjQ+AF4Ddtn8dqUzL6UF7byOKtjullV8pT5BVXM+35n6mqb2LetWMZ7UE7dHaWFuUeZNO+SrYVVnFehmd+ZTQ2OZrXrx5DYWU9l8z9WXf/VOrX3QAEAoXYWlKuN8ZsEJFJItK6P3wUsA5ba8vDwOXGmA0AxphKY8z+Qw9sbS7Vxhina94+LiWGyCDf/36bqJQ6djuLq7l07jIamluYf+04t76XrStoUe5BFuUU4OMlnD2sl9VRLJOZGMXr14yhtKqB6S8s44C2sijVLntbylRjTLAxJt4YM89+fIkxJqTVuneMMb2NMUHGmAxjzBe/8pxXGWP+5oj8neXr7cVZw3rx1cYD1DQ0WR1HKZeXV1zNpXN/pqG5hXnXjmVwr7a3qai2tCj3EMYYPs7Zx6SUGKKC/ayOY6mR8ZG8evUYiirrueyFZRRWamGulLLdf1Lb2MzircVWR1HKpe05WMPlLy6noclWkA/qqQV5R2hR7iHWF1RQUFbLmR58lby1UQm2wnxfWR1XvLicg9UNVkdSSllsTGIU4YG+fLlxv9VRlHJZ+8prmf7CcirrGnnzt1qQd4YW5R7iiw378RLb6C9lMzoxipeuyiSvpIarXsmiql6/slbKk/l4e3Hy4B58s6lQdwJW6iiUVNVzxYvLKa1u4PVrxpLWO9zqSC7FYUW5iESJyPsiUi0iu0Rk+mHWiYg8IiIl9sej0mqyvIgY+3NU2R8vOuo9uLIvNuxnTFKUx7eutDWhfwz/nj6SDXsruObVFdQ16s6fSnmy04bEUV7bSFae092LqpRTs+3UuYI9B2t5aWYmGf30ps7OcuSV8meABiAOuBx4VkTa27J5FjAV20zc4cA5wHVt1qQbY0Lsj992Y2a3sKOoim2FVZye1tPqKE7plCFxzJmWTlZeKX+Yv1qvkCnlwSanxuLv48VXGw9YHUUpl1HX2MxvX8tm074KnrtiFGOTo62O5JIcUpSLSDBwIfatlo0xS7HNq53RzvKZwBxjzB5jTAEwB7jKETnd1RcbbP2Rp2lRflhTMvpwz7lpfLXxAHe+vw5jjNWRlFIWCPLz4bgBMXy54YB+DijVAc0thlveWsPynaXMmZbOiYN0Y6Cj5agr5alAszFma6tjOUB7V8rT7Od+bd1i+w5xC0Uk8XAvKiKzRCRbRLKLijx3Q4gvNhxgeN9w+kQEWh3Fqc2ckMhNJw3gnew9PPblFqvjKKUsclpaHAVltWzcV2F1FKWcmjGGuxat5/MN+/l/5wxhiofug9JVHFWUhwDlbY6VA6EdWFsOhLTqKz8eSAQGAXuBj0XEp70XNcbMNcZkGmMyY2NjjyG+69pfXkfO7jJtXemgP56aymVj+vHMdzt4c9kuq+MopSxw8uA4RNAWFqWO4JnvtjNveT7Xn9Cfq49LsjqOy3NUUV4FtJ2JE4ZtB7gjrQ0Dqoz9e0RjzGJjTIMxpgy4GUgCBnd9ZPfw7eZCAE4dolNXOkJEuH/KUE4cGMv/W7Ser/UvZaU8TkyIP8P7RvD9Fs/9hlWpI3l/9R4e+3Ir54/ow+2nD7Q6jltwVFG+FfARkZRWx9KBDe2s3WA/d6R1hxhAfuW8R/tuSyF9IgJJ6RFy5MUKsI1Fe3r6SIb2CecP81ezbk/bL3mUUu7uhNRYcvaU6R4GSrXjp+3F3P7uWib0j+aRC4fTakieOgYOKcqNMdXAQuA+EQkWkYnAFOCNdpa/DswWkT4i0hu4FXgVQETSRCRDRLxFJATbTaAFwCZHvA9XU9/UzI/bizlxUKz+wHRSsL8PL80cTVSwH1e/toKCslqrIymlHOiEgbEYA4u36dVypVrbXljJdW+uJCkmmGevGIWfj25501Uc+f/kDUAgUAjMB643xmwQkUkiUtVq3fPAR8A6YD3wif0Y2MYpvg1UALnYesvPMcY0OuQduJisnaXUNDRzkt4JfVRiQ/155TejqWto5ppXV1BZp/+aKeUphveNIDLIlx+0hUWp/yqpquc3r67A38ebl68aTXigr9WR3IrDinJjTKkxZqoxJtgYE2+MmWc/vsQYE9JqnTHG3G6MibI/bm/VT/6tMWag/Tl62J9vm6Peg6v5bnMRfj5ejE+OsTqKy0qNC+XfV4xkW2EVN7+1huYWHZGmlCfw9hImpcSyeFsRLfpzrxR1jc3MemMlhRX1vDgzk76RQVZHcjv6nYMb+35LIeOTown087Y6ikublBLLPeel8e3mQv7+mXZKKeUpThgYS3FVAxv26mhE5dmMMdz5/jpW7jrI49MydLfObqJFuZvKK64mt7iaEwd65ijIrjZjXAIzxyfwwpKdvL0i3+o4SikHmJxq+/z8fkuhxUmUstbcxbksXFXA7FNTOXt4L6vjuC0tyt3Uob9EThqkoxC7yl3nDGFSSgx/+2A9K3eVWh1HKdXNYkL8GdYnnO+3al+58lzfbj7A3z/fzDnDe/GHkwZYHcetaVHupn7YWkRyTDDx0drz1VV8vL14+rKR9IkI5Lo3VrFXJ7Io5faOT41lze4yvdFbeaTthVXcNH8Nab3D+MdF6TrJrZtpUe6G6puaWZZbyqQUvcGzq4UH+fLClZnUNjRx3RsrqWtstjqSUqobTRgQTXOLIWunfjumPEt5bSOzXs/G38eL52dk6v1pDqBFuRtatauM2sZmjkvRfvLukBIXypOXjmBdQTl3vr8O+3AgpZQbGhkfSYCvF0u3F1sdRSmHaWkx/PHtNeSX1vDsFaPoExFodSSPoEW5G1q6vQhvL2FccpTVUdzWqUPiuPnkFBauKuD1n3dZHUcp1U0CfL0ZnRjFj1qUKw/y5Dfb+HZzIXefO4QxSVpLOIoW5W5oybZiRvSLIDRAh/p3p5tPTuHkQT24/+ON+tW2Um5s4oAYth6oorCyzuooSnW7rzce4F/fbOOiUX25YlyC1XE8ihblbuZgdQPrCso5TvvJu52Xl/DEpRn0iwrixnmrKKzQv7CVckcT+9s+T3/aXmJxEqW6187iav749hqG9QnngalD9cZOB9Oi3M38tKMEY9CbPB0kLMCX564YRVVdE7+ft5rG5harIymlutiQ3mFEBPlqC4tyazUNTfzujZV4ewvPXjGSAF+9sdPRtCh3M0u3FxHq70N6X91ty1EG9gzl7xcOIyuvlEc+22x1HKVUF/P2EsYnR/Pj9mK9sVu5JWMMf31/PVsLK/nnpSPoG6njlK2gRbkbMcaweGsx4/pH4+Otf7SONCWjDzPHJ/Di0p18vn6f1XGUUl1s4oAY9pbXsbO42uooSnW5/yzP5/3VBdxycirHp+rkNqto5eZG8ktrKCir5bgB2rpihTvPHkx6vwj+tGAtefoXt1JuZUL/aACW5epN3cq9rNtTzn0fbeT41FjdsdNiWpS7kZ932G5Cmjgg2uIknsnfx5tnpo/Ay0u4/j+rdGMhpdxIUkwwPUL9WZarN3sq91Fe28gN81YSE+LHk5dk4OWlN3ZaSYtyN/LTjhJiQ/3pHxtidRSP1TcyiCcvyWDTvgru/Wij1XGUUl1ERBiXHM2y3BLtK1duwRjDnxbksK+sjqemjyQy2M/qSB5Pi3I3YYzh59wSxidH6wgji504qAfXn9Cf+Vn5LFpTYHUcpVQXGZscRWFlPXklNVZHUeqYvfJjHl9uPMAdZw5iVEKk1XEUWpS7jR1FVRRV1jO+v7auOINbT00lMyGSOxeuI7eoyuo4SnWaiESJyPsiUi0iu0Rk+mHWRYjIayJSaH/c0+pcDxGZLyJ7RaRcRH4UkbEOexNdbFzyob5ybWFRri1ndxkPf7aJU4fEcc1xSVbHUXZalLuJQ/3kE7Qodwo+3l48NX0Efj5e3DhvtfaXK1f0DNAAxAGXA8+KSFo7654AgoBEYAwwQ0R+Yz8XAqwARgFRwGvAJyLikj12yTHBxIb6s1yLcuXCymsb+f38VfQIDeAfFw3Xb9ediBblbuKnHSX0Dg8gPkpnizqLXuGBzJmWzqZ9FTz86Sar4yjVYSISDFwI3GWMqTLGLAU+BGa0s/xc4FFjTI0xJg94CbgawBiTa4x53BizzxjTbIyZC/gBAx3yRrqYiDA2KYpluaXaV65ckjGGOxeuY19ZHf+6bAQRQdpH7ky0KHcDLS2GZbkljO8fo//F62ROGhTHb49L4rWfd/H5+v1Wx1Gqo1KBZmPM1lbHcoD2rpQDSJt/HtruIpEMbEX59q4IaYVxydHsr6hjl/aVKxc0P2s3n6zbx22nD9Q+ciekRbkb2Ly/koM1jdpP7qRuP2MQw/uGc/u7ORSU1VodR6mOCAHK2xwrB0LbWfs5cIeIhIrIAGxXyf/nKzsRCQPeAO41xrR97kNrZolItohkFxUVHdMb6C6H+sqX79QWFuVatuyv5N6PNjApJYZZk5KtjqPaoUW5G/jZ3t+oRblz8vPx4qnLRtBi4Ja3VtPU3GJ1JKWOpAoIa3MsDKhsZ+1NQC2wDVgEzAf2tF4gIoHAR8AyY8zDh3tRY8xcY0ymMSYzNtY5dxXsHxtMTIi/biKkXEpdYzN/mL+K0ABfHp+m88idlRblbmB5bgn9ogLpExFodRR1GAnRwTwwdSgr8g7y1Lcu+8298hxbAR8RSWl1LB3Y0HahMabUGHO5MaanMSYN298rWYfOi4g/8AFQAFzXvbG7n4gwOjGSrJ1alCvX8cAnG9l6oIrHp6UTG+pvdRx1GA4ryjsxXktE5BERKbE/HpV2GqVFZKaIGBH5bfend14tLYasvFLGJulVcmc3dUQfLhjRh6e+3abTG5RTM8ZUAwuB+0QkWEQmAlOwtZ/8goj0F5FoEfEWkTOBWcAD9nO+wLvYrqRfaYxxi6+JRidGUVBWy15tR1Mu4PP1+3lzWT6zJiczOdU5v4FSNo68Ut7R8VqzgKnYrsoMB86hzdUVEYkE/kI7V208zdbCSspqGhmbFGV1FNUB900dSr+oIP749hrKaxqtjqPUr7kBCAQKsbWkXG+M2SAik0Sk9fD9UcA6bK0tDwOXG2MOfTZPwPYZfhpQJiJV9sckh72LbjDG/nm7Ik+vlivntq+8ljsWrmVYn3BuO80lhx55FIcU5Z0crzUTmGOM2WOMKQDmAFe1WfMw8C+guPtSu4bl9r7GQzcfKecW4u/DPy8dQWFlPXd+sE7HqimnZW9LmWqMCTbGxBtj5tmPLzHGhLRa944xprcxJsgYk2GM+aLVuR+MMWI/F9LqscSK99RVBvcKI8TfR1tYlFNraTHMfjuHhqYW/nlpBn4+2rHs7Bz1J9SZ8Vpp9nPtrhORMUAm8NyRXtQV7uQ/Vlk7S+kdHkDfSO0ndxUZ/SL446mpfLJ2H++u3HPk36CUcireXsLIhEi9Uq6c2twlufycW8I956aRHOuS+3V5HEcV5Z0Zr9V2bTkQYu819wb+DfyhI72JrnAn/7EwxrB8Zwljk6N1PrmL+d3x/RmbFMU9H25gV0m11XGUUp00JjGSrQeqOFjdYHUUpf7Huj3lzPlyC2cO7cnFmX2tjqM6yFFFeWfGa7VdGwZUGdv3/DcAa40xP3dLShezo6ia4qoG7Sd3Qd5ewuOX2MZS3fL2Gh2TqJSLGZ1o+9zN3nXQ4iRK/VJtQzM3v72a6GB/Hr5gmF60cyGOKso7PF7Lfiz9MOtOBs4Xkf0ish/bTURzROTpbsjs9A5tXjFW+8ldUp+IQB46fxir88t0TKJSLia9XwR+3l7awqKczoOfbiS3qJrHp6UTEeRndRzVCT6OeBFjTLWIHBqv9VsgA9t4rQntLH8dmC0inwIGuBV4yn7uKiCg1dqF2MZtvdRN0Z3a8txSeoT6kxj9P5vnKRdxbnpvvttcyFPfbmNyaqxue6yUiwjw9WZ433C92VM5lW82HeDNZflcOymJCQNirI6jOsmRt+J2dLzW89h2flsHrAc+sR/DGFNmjNl/6IFtxGLF4bZsdmfGGLJ2ljImKUq/mnJx905Jo1d4ILPfWUN1fZPVcZRSHTQ6KYr1BeXUNjRbHUUpiqvq+fN7axncK4zbTtfxh67IYUV5J8ZrGWPM7caYKPvjdnOYuXHGmBOMMS866j04kz0Ha9lfUaf95G4gNMCXJy7JIL+0hvs/3mh1HKVUB2UmRNLUYlizu8zqKMrDGWO44721VNQ18eQlGfj7eFsdSR0FHVrpog59ZTpai3K3MCYpiusm94MYC4AAACAASURBVOetFbv5auMBq+MopTrgULvZyl3awqKs9faK3Xy9qZDbTx/IwJ7tDbZTrkCLche1Iq+U8EBfUnvoD5+7mH1qKkN6hXHHe2sprqq3Oo5S6ggigvwY0CNEJ7AoS+UVV3PfxxuZOCCaqycmWR1HHQMtyl1U1s5SMhMi8fLSfnJ34efjxZOXZlBZ38Qd7+lun0q5gsyESFbtOkhLi/68Ksdram5h9jtr8PES/nFRutYELk6LchdUVFlPbnG1tq64odS4UG4/fSBfbzrAgmzd7VMpZ5eZGEVFXRPbCquOvFipLvb84lxW5Zdx/9Sh9I7Qnb1dnRblLijbPhf30OYVyr1cPTGJcclR3PvRBnaX1lgdRyn1KzLtfeXZ2leuHGx9QTlPfLWVs4f34rz03lbHUV1Ai3IXlJVXSoCvF8P6hFsdRXUDLy/hsYvT8RLh1ndyaNavxZVyWgnRQcSE+LEyT/vKlePUNTYz+501RAX78eDUoToa2U1oUe6CVuSVktEvAj8f/eNzV30jg7j7vDSy8kp5eelOq+MopQ5DRBiVEKk3eyqHevyrrWw9UMWjFw3XXTvdiFZ1LqayrpGNeysYo60rbu/CkX04bUgc//hiC1v2V1odRyl1GJkJUeSX1lBYWWd1FOUBlueW8MKSXC4fG88JA3tYHUd1IS3KXcyq/DJajM4n9wQiwkMXDCM0wIdbF6yhsbnF6khKqXaMSrTPK9cWFtXNquqbuO3dHOKjgrjzrMFWx1FdTItyF5OdV4q3lzAiPtLqKMoBYkL8eeiCYawvqOCpb7dbHUcp1Y6hvcPx8/FipbawqG724Ceb2HOwljkXpxPs72N1HNXFtCh3MSvyShnSK4wQ/WH0GKen9eSCkX145rvt5Oh23ko5HT8fL9L7hrMyX4ty1X2+21LI/Kx8Zk1OJlNbWN2SFuUupKGphTW7y8hM1Kvknubuc9PoEerP7HfWUNfYbHUcpVQbIxMiWV9Qrj+fqluU1TTw53fXkhoXwuxTU62Oo7qJFuUuZP3ecuoaW/QmTw8UHujLoxcNZ0dRNXO+3GJ1HKVUG6PiI2lsNqwvKLc6inJDd3+4gdLqBh6floG/j7fVcVQ30aLchRzaNGiUXin3SJNSYrliXDwvLt1J1k7dqEQpZzLSvomQ9pWrrvbZun0sWrOXP5yUwlDdn8StaVHuQlbkHSQxOogeoQFWR1EW+cuZg+kXGcRtC3Korm+yOo5Syi4mxJ/E6CAtylWXKq6q568frGdYn3BuOLG/1XFUN9Oi3EUYY8jOK9WbOzxcsL8Pj12czu6DNTz82Sar4ygnJiLhInKXiCwUkS9bP6zO5q5GJkSyKv8gxuguvOrYGWP46/vrqKprYs60dHy9tWRzdzrCw0XsKKriYE2j9pMrxiRFcc3EJF5cupPT03oyKSXW6kjKOS0AvIH3gVqLs3iEUQmRLFxVQH5pDQnRwVbHUS7ugzUFfLHhAHecOYjUuFCr4ygH0KLcRaywb0qhk1cUwG2nD+S7LYXc/u5avvjjZMICfK2OpJzPOCDaGNNodRBPMapVX7kW5epY7C+v4+5FGxiVEMm1k5KtjqMcRL8LcREr8kqJDvYjKUY/6BUE+HozZ1oGByrquP+jjVbHUc5pKaBb/jlQSo9QQv19tK9cHRNjDHcsXEtDcwuPXZyOt5dYHUk5iF4pdxHZeQfJTIxERH84lU1GvwiuP6E/z3y3gzOH9eSkQXFWR1LO5SrgUxFZDhxofcIYc58lidyct5eQER+hRbk6Ju9k7+b7LUXcc+4QvRDnYfRKuQsorKgjv7SG0dpPrtq46eQUBvUM5Y731lFW02B1HOVcHgT6AXFASqvHACtDubuR8ZFsOVBJZZ12DanO23Owhvs/3sT45GiuHJ9odRzlYFqUu4DsXYf6ybUoV7/k7+PNYxenU1rdwD0fbrA6jnIulwIZxpiLjDEzWj2utDqYOxuVEIkxkLNbNxFSndPSYvjze2sxxvDoRcPx0rYVj6NFuQtYkVdKgK8Xab3DrI6inNDQPuH8/qQBfLBmL5+v3291HOU8cgG9XOtgGfERiOgmQqrz/rN8Fz9uL+HOswfTLyrI6jjKAg4rykUkSkTeF5FqEdklItMPs05E5BERKbE/HhV7I7WIxIjIj/bjZSLys4hMdNR7sEp23kEy+kXojFJ1WDeeOIC03mH87YN1lFZrG4sC4A3gQxG5TEROav3oyG/uxGd2hIi8JiKF9sc9bc4nish3IlIjIptF5JRjf2vOKyzAl9QeoazK16JcdVx+SQ0PfbqZSSkxTB8Tb3UcZRFHVnnPAA3Y+hsvB54VkbR21s0CpgLpwHDgHOA6+7kq4GogFogEHgE+EhG3vWG1qr6JDXvLtZ9c/Spfby/mTEunvLaRuxattzqOcg43Ar2Ah4CXWj1e7ODv7+hn9hNAEJAIjAFmiMhvWp2fD6wGooG/Au+KiFsP1z+0iVBLi24ipI6spcVw27s5+HgJj1w4XAc6eLAOFeUi4i0iV4uI/9G8iIgEAxcCdxljqowxS4EPgRntLJ8JzDHG7DHGFABzsE0RwBhTZ4zZYoxpAQRoxlacu23Fuia/jBaj/eTqyAb1DOOWU1L5ZO0+Pl671+o4ymLGmKTDPI449LiTn9nnAo8aY2qMMXnYCv+r7c+TCowE7jbG1Bpj3gPW2Z/bbY2Mj6CyrokdRVVWR1Eu4NWf8sjaWcpd5w6hd0Sg1XGUhTpUlBtjmoHHjTH1R/k6qUCzMWZrq2M5QHtXXdLs5w67TkTWAnXY/pJ40RhT2N6LisgsEckWkeyioqKjjG6tFXmleIntQ16pI7lucjLp/SK464P1FFUe7Y+rchci4iMik+0tLJM68a1iZz6zwXaRpPU/D7X/cxqQa4yp7MjzuMNnNvxyEyGlfk1uURWPfrGZkwb14OJRfa2OoyzWmfaVj0Tk3KN8nRCg7a3o5UB7+8a2XVsOhEir73OMMcOBMGA6tg0y2mWMmWuMyTTGZMbGuua3pdm7ShnUM4xQ3bFRdYCPtxdzLh5OdUMzf31/Hcbo1+eeSkQGAZuAecBN2NpINotIRzYU6sxn9ufAHSISKiIDsF0lP3SXWmeexy0+swGSYoKJDPLVvnL1q5pbDLctyMHfx5uHLximbSuqU0V5ALZewO9F5A0Ref3QowO/twpbEd1aGFDZgbVhQJVpU13YW1nmY/vLIL3jb8N1NDW3sDq/jMzESKujKBcyoEcofzptIF9uPMAHawqsjqOs829gLtDPGDPeGNMXeM5+/Eg685l9E1ALbAMWYSv+9xzF87gNEWFkfCSr8susjqKc2ItLclmVX8a956URFxZgdRzlBDpTlK/HdsPQd8B2YEerx5FsBXxEJKXVsXSgvcHKG+znjrTuEF/giD2SrmjjvgpqGpr1Jk/VaVcfl0RmQiR3L9rAgYo6q+Moa2RgaztsfUHjSfvxI+nwZ7YxptQYc7kxpqcxJg3b3ytZ9tMbgGQRCT3S87ibkQmRbC+s0k29VLu2F1Yy56utnJ4Wx5SM3lbHUU7iV4vyNiO0lvzK41cZY6qBhcB9IhJsH2M4BdvIrrZeB2aLSB8R6Q3cCrxqzzNORI4TET8RCRSRP2ObDLC8Y2/XtazIs331qUW56ixvL+EfF6fT0NzCHfbNKJTH2Qsc3+bYJPvxX9WZz2wR6S8i0faBAGdim6D1gP15tgJrgLtFJEBEzsc2Veu9Y3hfLmFkvO0bztW79Wq5+qWm5hZufSeHEH8fHpiqbSvq/xzppp+XOvAcho5dqb4BeBkoBEqA640xG0RkEvCZMSbEvu55+/Ots//6RfsxAH/gX/bzjfY1Zxtj3HLUxIqdpfSLCqRnuH6tpTovKSaYP58xiHs/2siC7D1MG93P6kjKse7ENqf8Y2AXtpGFZwFXdPD3d/QzexS2K/AR2K6wX26MaX0l/FJsF1YOAvnARcYY172Ls4PS+4Xj7SWs2nWQEwf2sDqOciLP/bCDnD3lPDN9JLGhRzXUTrmpXy3KjTFJXfVCxphSbPPH2x5fgu1moEO/NsDt9kfbtT/wy9YWt2WMIXtXKZNTXPdmJ2W9meMT+Xz9fu77eCMTU2Loo+O2PIYx5kMRGQFcgm1e+Vrgb8aYbR38/R39zH4HeOdXnicPOKEz2d1BkJ8PQ3qF6QQW9Qsb91bwz2+2cc7wXpw9vJfVcZST0S0indTO4mqKqxp0Prk6Jl5ewmMXp9NiDH9+V9tYPImIhGO7Sj0K24jD44FnRORLS4N5kJHxEazZXUZTc4vVUZQTaGhq4dYFOYQH+nH/lKFH/g3K42hR7qSy7f3kY5J08oo6Nv2igrjzrMEs3V7Mm8vzrY6jHGcBtivU3wBvAW+3eigHGJkQSU1DM5v3u/WwGdVBT3+7jU37Knjo/KFEBvtZHUc5Ibfdnt7VZeWVEhnkS//YkCMvVuoILh8bzxcb9vPwp5uYnBJDQnSw1ZFU9xsHRBtjGq0O4qkObSK0Kv8gQ/uEW5xGWSlndxnPfL+DC0b24bS0nlbHUU5Kr5Q7qey8UjITo/SubNUlRIRHLhyOtwh/WrCWlhZtY/EAS4GObBSkukmfiEDiwvy1r9zD1TU2c9uCHGJD/Ln73MNtiquUXil3SoWVdeSV1DB9bLzVUZQb6R0RyN3npXHbghxe/nEnv53kluP91f+5CvhURJYDB1qfMMbcZ0kiD/N/mwhpUe7JnvhqK9sKq3j1N6MJD9TdudXh6ZVyJ3Son1xv8lRd7cKRfThlcA8e/WIL2wu1z9XNPQj0w7aXQ0qrxwArQ3maUQmR7C6tpVA38fJI2XmlzF2Sy/Sx8ZygozHVEWhR7oSydpYS6OvN0N7ag6i6lojw0AXDCPLz5tZ3cnQqhHu7FMgwxlxkjJnR6nGl1cE8ychWfeXKs9Q0NHHrghz6RgZy51naSaaOTItyJ5S1s5QR8RH4+egfj+p6PUIDeGDqUHL2lPPs9zusjqO6Ty62TdaUhdJ6h+Hn4/Xfb0CV53j4083kl9bwj4vSCfHXbmF1ZFr1OZmKukY27a9gTJK2rqjuc87w3pyb3pt/frONDXvLrY6juscb2Hb0vExETmr9sDqYJ/H38WZ4n3BW6pVyj7JkWxFvLNvF1ROTGJccbXUc5SK0KHcyK/MOYgyM0X5y1c3un5JGZLAfs9/Oob6p2eo4quvdiG0nz4eAl1o9XrQylCcalRDJ+oJy6hr158wTlNc28qcFaxnQI4Q/nT7Q6jjKhWhR7mSy8krx8RJGxOumQap7RQT58eiFw9lyoJInvurQzuvKhRhjkg7z0LE7DjYqIZLGZsO6Av1WyhPc++EGiqrqeXxaOgG+3lbHUS5Ei3Ink7WzlGF9wwn00x9k1f1OHNSDy8b04/nFO8jOK7U6jlJu6dAmQtpX7v4+X7+PhasLuPHEAQzvG2F1HOVitCh3InWNzazdU6b95Mqh/nr2EPpGBnLrghyq65usjqOU24kO8Sc5JpiVu/Q/fN1ZUWU9d76/nqF9wvjDSTp5VHWeFuVOZHV+GY3NRvvJlUOF+Pvw2EXp5JfW8OCnm6yOo5RbGpkQycpdBzFGd9N1R8YY/rJwLVX1TTwxLQNfby2vVOfpvzVOZEVeKSKQmaBFuXKsscnRXDspmXnL8/luc6HVcZRyO5kJkRysaSS3uNrqKKobLMjew9ebCvnzGYNIiQu1Oo5yUVqUO5GsnaUMjAslPEi34VWON/vUVAbGhXL7e2sprW6wOo5SbiUz0dZXvlL7yt1OfkkN9360gfHJ0fxmQqLVcZQL06LcSTQ0tbBy10GdZ6osE+DrzROXZFBW08DfPlinX7Mr1YWSY0KICPJl5S4tyt1Jc4vh1gVr8BLhsWnpeHmJ1ZGUC9Oi3EmsKyijtrGZccnauqKsM6R3GLNPHcin6/bz/uoCq+Mo5Ta8vISR8ZFk682ebmXu4lxW5B3k3ilp9IkItDqOcnFalDuJZbm2D+oxSXqlXFlr1uRkxiRGcfeiDew5WGN1HKXcRmZiJDuKqrU9zE2sLyjn8a+2cNawnpw/oo/VcZQb0KLcSSzLLSE1LoSoYD+roygP5+0lzJmWjgFmv5NDc4u2sSjVFUbbJ2vpngCur66xmT++vYbIID8enDoMEW1bUcdOi3In0Nis/eTKufSLCuLuc4eQtbOUuYtzrY6jlFsY3jccPx8vsrWv3OU98vlmthVW8djF6UTqxTTVRbQodwLrC8qpaWhmrLauKCdy0ai+nDm0J49/tYX1uj24UsfM38eb9L7hZO3UK+WubMm2Il75MY+Z4xOYnBprdRzlRrQodwKH+snH6k2eyomICA+dP4yoYD9ufms1tQ3NVkdSyuVlJkaxvqBcf55cVGl1A7e+k0NKjxD+ctZgq+MoN+OwolxEokTkfRGpFpFdIjL9MOtERB4RkRL741GxN2uJSKqILBKRIhEpFZEvRGSgo95Dd1m+s4QBPUKICfG3OopSvxAZ7MdjF6ezo6iah3S3T6WO2ZjEKJpaDGt2l1kdRXXSoV07D9Y08OSlGQT4elsdSbkZR14pfwZoAOKAy4FnRSStnXWzgKlAOjAcOAe4zn4uAvgQGGh/nixgUffG7l5NzS1k5x3UUYjKaU1KieWa45J4Y9kuvtl0wOo4Srm0kQmRiNh2cFau5e0Vu/liwwFuP30Qab3DrY6j3JBDinIRCQYuBO4yxlQZY5ZiK65ntLN8JjDHGLPHGFMAzAGuAjDGZBljXjLGlBpjGoEngIEi4rLN2OsKyqmqb9J+cuXU/nT6QAb1DOX2d9dSWFlndRylXFZ4oC8D40K1KHcxO4qquPejjUwcEM01xyVZHUe5KUddKU8Fmo0xW1sdywHau1KeZj93pHUAk4H9xpiS9k6KyCwRyRaR7KKioqOI3f1+2mGLPr6/FuXKeQX4evPUZSOoqm/iTwvW0qJjEpU6aqMTo1i16yBNzS1WR1Ed0NDUwi1vrSHA14vHp2Xorp2q2ziqKA8B2o5vKAdCO7C2HAiRNkNARaQvtpaY2Yd7UWPMXGNMpjEmMzbWOe+Q/nlHCYN6hmo/uXJ6KXGh/O3swfywtYhXfsqzOo5SLiszMZLqhmY27au0OorqgDlfbWFdQTmPXDicuLAAq+MoN+aoorwKCGtzLAxo7xOp7dowoMoY899LcyISC3wJ/NsYM7+LszpMXWMzK/JKmdA/xuooSnXIFeMSOGVwHI98tlnHJCp1lMYk2e4hytIWFqe3ZFsRz/+Qy/Sx8ZyW1tPqOMrNOaoo3wr4iEhKq2PpwIZ21m6wn2t3nYhEYivIPzTGPNgNWR1mdX4Z9U0tTNDWFeUiRIRHLxpOZLAvN721mpqGJqsjqW7SiYlZ/iLynIgcsE/F+khE+rQ6nygin4rIQRHZLyJPi4iP496J8+kVHkh8VBDLc9vtvFROoriqntn28Yd3nT3E6jjKAzikKDfGVAMLgftEJFhEJgJTgDfaWf46MFtE+ohIb+BW4FUAEQkDvgB+NMbc4Yjs3ennHcV4CYzRySvKhUQF+/HEtAx2Fldz74cbrY6juk9HJ2bdDIzHNi2rN1AGPNXq/L+BQqAXkAEcD9zQfbFdw9ikKLLySvX+DCdljOFPC3Ior23kqekjCPTT8Yeq+zlyJOINQCC2D+f5wPXGmA0iMklEqlqtex74CFgHrAc+sR8DOB8YDfxGRKpaPeId9i660E87ShjeN4KwAF+royjVKRMGxHDDCf15O3s3H+bstTqO6mKdnJiVBHxhjDlgjKkD3uKXN+cnAe8YY+qMMfuBzzn8zfseY2xyNGU1jWwt1L5yZ/TS0p18t6WIv509mEE923bfKtU9HFaU28cYTjXGBBtj4o0x8+zHlxhjQlqtM8aY240xUfbH7Yf6yY0xrxljxP4cIa0e+Y56H12lur6JNbvLtHVFuaxbTkllVEIkdy5cR35JjdVxVNfqzMSsl4CJItJbRIKwXVX/rNX5fwKXikiQva3lTGyFuUcba+8rX56rfeXOJmd3GY98vpnT0+KYMS7B6jjKgzjySrlqJSuvlKYWozd5Kpfl6+3FPy/NwEvg9/NX0dCk493cSGcmZm0F8oECoAIYDNzX6vwP2Ir5CmAPkA180N6LusIY267SNzKQ3uEBLN+pfeXOpKKukd/PX0WP0AAevTCdNoPflOpWWpRb5Mdtxfh5ezEqIdLqKEodtb6RQTx6UTpr95Tz9882Wx1HdZ3OTMx6FggAooFgbPcPfQYgIl7Y7gNaaD8XA0QCj7T3oq4wxrariAhjk6PJ2llKq+FiykLGGP6ycB17y+r412UZhAdpa6lyLC3KLbJkWzGjkyL15hHl8s4Y2pOrJiTy8o87+WrjAavjqK7RmYlZ6cCr9hbFemw3eY4RkRggCugHPG2Mqbdv9PYKcFb3xncNY5OiKK5qYEdR1ZEXq2735vJ8Plm7j1tPS2VUgg5gUI6nRbkFDlTUseVAJZNT3PtKkPIcfzlrEMP6hHPbghz2HNT+clfXyYlZK4ArRSRcRHyx3dS/1xhTbIwpBnYC14uIj4hEADP55a7NHmtssu2eomXaV2659QXl3P/RRk4YGMvvJve3Oo7yUFqUW2DxVluv5CQtypWb8Pfx5unpI2hpMdw4b7X2l7uHjk7Mug2oA7YBRdiugp/f6vwFwBn2c9uBJuCP3R/f+SVGB9Ej1J/lO7Uot1JFXSO/n7eKqGA/5lycjpeX9pEra3j0Bg5WWbKtmJgQfwb3au+eKaVcU0J0MP+4eDi/e3MVD326iXvO8/ipdy7NGFMKTG3n+BJsN4Ie+nUJtokrh3ueNcAJ3RDR5YkI4/tH8+P2EowxelOhBYwx/Pndtew+WMv8a8cRHeJvdSTlwfRKuYO1tBiWbi9mckqMfgArt3PG0F5cPTGJV3/K49N1+6yOo5TTm9A/muKqerYVal+5FV75MY/P1u/nz2cMZEyS9pEra2lR7mAb9lZQWt3ApFQdhajc0x1nDmJEfAS3v7tWb2BT6ggOjcX9aXuxxUk8z8pdB3no002cOiSOayclWx1HKS3KHW3xNls/+XEDtJ9cuSc/Hy+emT4SPx8vrn9zJTUNTVZHUspp9YsKol9UID/u0HnljlRcVc+N/1lF74hAHrtY55Er56BFuYMt2VbEkF5hxIZq35pyX70jAvnXpSPYVljFXxau0znMSv2Kif1jWJZbQnOL/pw4QlNzC3+Yt5qDNQ08e8VIwgN1HrlyDlqUO1BFXSPZeQeZnKpXyZX7Oy4lhltPTWXRmr289lOe1XGUclrj+0dTWdfE+oK2m6iq7vCPL7fwc24JD54/jLTe4VbHUeq/tCh3oCVbi2lqMZw8uIfVUZRyiBtOGMApg+N44JNNZOnYN6Xa9d++cm1h6XafrN3H8z/kcvnYeC4a1dfqOEr9ghblDvTt5kIignwZ0S/C6ihKOYSXl/D4Jen0iwrihv+sYn95ndWRlHI6saH+pMaF8NMOvdmzO23ZX8mf3s1hZHwEd5+rI1uV89Gi3EGaWwzfbynkhNRYfLz1/3blOcICfHl+xihqGpr43ZsrqWtstjqSUk5nQv8YVuSVUt+kPx/dobymkeveyCbY34dnrxiFn4/+Paycj/5b6SA5e8ooqW7gpMFxVkdRyuFS40J5fFo6a3aX8bcP1uuNn0q1cdyAGOoaW1iZd9DqKG6nucXwh7dWU1BWy7OXjyQuLMDqSEq1S4tyB/lucyHeXsLxKXqTp/JMZwztxU0np/Duyj288mOe1XGUcirj+kfj6y38YB+bq7rOI59vZvHWIu6bMpTMRN0gSDkvLcod5JtNhYxKiCQ8SEcvKc91y8kpnDYkjgc/3cQSLT6U+q8Qfx9GJUSyeKv2lXel91fvYe7iXK4cn8BlY+KtjqPUr9Ki3AH2ldeycV8FJw/SqSvKs9lu/MwgpUcIN/xnle74qVQrk1Nj2bSvgsIKvSG6K6zcdZA/v7uOcclR3HXOEKvjKHVEWpQ7wFcbDwBwsvaTK0WIvw8vXJmJr7cX176WTXlNo9WRlHIKk+3tjYu36dXyY7XnYA3XvZFNr4gAnr18FL46YEG5AP231AE+W7efAT1CGNAjxOooSjmFflFBPHfFKHYfrOF3b66koanF6khKWW5IrzBiQvxYvFVbu45FVX0Tv30tm/qmFl6aOZrIYD+rIynVIVqUd7OSqnqW7yzhzKE9rY6ilFMZkxTF3y8Yzs+5JdylE1mUwstLmJQSy9LtxbS06M/D0WhqbuH381axrbCKZ6aP1IthyqVoUd7Nvtp4gBYDZ2hRrtT/uHBUX/5w0gDezt7Ncz/kWh1HKcsdnxpLaXUD6/eWWx3F5RhjuPejjXy/pYj7pwxlcqpOO1OuRYvybvbZ+v3ERwUxpFeY1VGUckp/PCWVc9N788jnm1m0psDqOEpZ6riUGAC+36ItLJ310tKdvLFsF7MmJzN9rE5aUa7HYUW5iESJyPsiUi0iu0Rk+mHWiYg8IiIl9sejIiKtzs8VkS0i0iIiVzkq/9Eor2nkpx3FnDm0J63eglKqFS8v4bGLhzMmKYo/LVjLstwSqyMpZZmYEH8y+kXwzaYDVkdxKR/l7OWBTzZx1rCe3HHGIKvjKHVUHHml/BmgAYgDLgeeFZG0dtbNAqYC6cBw4Bzgulbnc4AbgFXdmrYLfL3pAI3NRltXlDoCfx9v5s4YRb+oQGa9ns2W/ZVWR1LKMqcOiSNnT7mORuygZbkl3PpODqMTI3l8WgZeXnoRTLkmhxTlIhIMXAjcZYypMsYsBT4EZrSzfCYwxxizxxhTAMwBrjp00hjzjDHmG8DpP60+WbePXuEBpPeNsDqKUk4vIsiP164eQ4CvNzNfzqKgrNbqSEpZ4hT7+NxvNhdanMT5bdpXwbWvZ9MvKpAXrswkwNfb6khKHTVHXSlPBZqNMVtbHcsB2rtSnmY/d6R1RyQitbLzXgAAF5FJREFUs0QkW0Syi4oc259XXFXPD1uLOC+jt/5Xu1Id1DcyiNeuHkN1QxMzX87iYHWD1ZGUcrjUuBD6RQXy9UZtYfk1u0truPLlLIL9fHj9mrFEBOnoQ+XaHFWUhwBtbyUvB0I7sLYcCJGjaMo2xsw1xmQaYzJjYx17F/ZHOXtpbjFcMKKvQ19XKVc3uFcYL1yZSX5pDVe9uoKq+iarIynlUCLCyYPiWLq9mNqGZqvjOKWiynpmvLSchqYW3rhmzP9v787Dq6ru/Y+/vxkgTAFCSJhHgyiUQSIICKJWWtE6MFTFW1vrzM+pWr1t7/WnhXtvtYP9XUVBWhTFOmDVeoXrUFtrGUSJyhTByAxhDCEhA5nX74+98TkcD0iAnH1O8nk9z3keztqb5LP3OWfle/Zee226tmsRdCSRkxatorwUCJ9+JBWINHA0fN1UoNTF2STGr3+Wz4AuqZzeKdL3DhE5lnP6dGDmNUNZm1/Mzc/lUFGtwkSalovOzKSypo4lG3R3z3BF5VX8YO5H7DlYydM/OpusTP2dlcYhWkV5HpBkZlkhbYOB3Ajr5vrLvmm9mLVhbwmrdxRz5dCuQUcRiVvjB3Tit1MGsWzjfm5/4TOqa3XXT2k6hvdOo01KkoawhCmtrOFHz6xg074y/nBdNsN6tg86ksgpE5Wi3DlXBrwGTDezVmY2GrgcmB9h9eeAe8ysq5l1Ae4F5h1eaGbNzCwFMCDZzFLMLKbmW3/t03wSDC4b0iXoKCJx7cqh3Zh++QDeW7eHu19aSY0Kc2kikhMTGHd6Bu+t26P3va+8qoYfz1vBmvxiZk4d+tWc7iKNRTSL2WlAC2Av8CJwm3Mu18zGmFlpyHpPAW8Ca4C1wCK/7bB3gUPAKGCO/++xDR//+NTWOd5YuZMxWR3JaJMSdByRuHfdyF7824QzWLRmF/f9eTW1uv24NBGXfKsT+8uqWL6pMOgogauoruXGZ3PI2VLI768awvgBmmpYGp+kaP0i51wh3vzj4e2L8S7uPPzcAff7j0g/Z1wDRTwlPsjbS37RIX52sW5eIHKq3DS2D5U1tfz23TwM+M2UwSRqViNp5MadnkGrZoksXL2zSR8Vrqiu5abncvhw035+N2Uwlw3WWWhpnGJq2Edj8NyHW8lo05zv6Fu8yCl1+wVZ3HNRP177LJ/7XlmlI+bS6KUkJ3LRmZm8nbu7yV5TcajKO0K+ZEMBj0waxMSzNKOZNF4qyk+hLQVl/OOLfVwzvAfNkrRrRU61Oy/M4l6/MP/JyyubbKEiTcclg7pQVF7N0iY4C0tZpTeGfOnGAn4zeTDfz+4edCSRBhW14StNwfPLt5KUYEwd0SPoKCKN1h0XZpGclMDDb63nUHUtM6cOpXmS7uInjdPYfum0SUli4epdjDs9I+g4UVN8qJrrn/mYlduLePT7g7lS9/yQJkCHc0+RQ1W1LMjZzncGdiIzVRd4ijSkW8/ryy8vG8BfP9/Djc/mUKYbDJ1yZpZmZq+bWZmZbTWzqUdZr7mZzTazPWZWaGZvmlnXsHWuNrN1/s/aaGZjorMV8a95UiLjz+zEO7m7qaxpGvP1F5RWcs2c5azJL+aJqWepIJcmQ0X5KfLqpzs4WFHDD0f2CjqKSJPww1G9+M3kQSzdUMDUP35EYVlV0JEamyeAKiATuBaYZWYDIqx3FzASGAR0AYqAxw8vNLOLgEeA6/Hu4jwW2NSgyRuZSwd3pqSihvfX7ws6SoPbXljOlNkfsqmglD/+8Gwu/lbnoCOJRI2K8lOgsqaWJ9/fwNAe7Ti7l25kIBItU7K7M/tfhrF+10Emz17G9sLyoCM1CmbWCpgEPOCcK3XOLQH+B/hBhNV7A+845/Y45yqAl4DQ4v2XwHTn3HLnXJ1zLt85l9/Q29CYjDktnYw2zXklZ3vQURrU2vxiJs5aRmFZFX+6cQTn9esYdCSRqFJRfgosyNnBzuIKfvLtfphpmjaRaBo/oBPzbxhBQUklVz65jNU7ioKO1Bj0A2qdc3khbas4stg+bC4w2sy6mFlLvKPqbwGYWSKQDXQ0sw1mtsPMZppZi0i/1MxuNrMcM8vZt6/xHxU+XkmJCUwe1o33v9jL7uKKoOM0iA/y9nH1nOUkJRh/vnUkw3qmBR1JJOpUlJ+kw0fJh/Vsz5gmPI+sSJCG907j1dtG0TwpgaueWq5bk5+81kBxWFsx3vCTcHnANiAfOAicAUz3l2UCycBkYAwwBBgK/HukX+qcm+Ocy3bOZXfsqKOkob6f3Z065w2VbGz+9NFWfjxvBd3TWvLatFFkZUZ6m4k0firKT9LLK7azS0fJRQKXldmG1//PKE7LaM1N83OY/cFGvHuRyQkoBVLD2lKBkgjrzgJSgA5AK+A1/CPleHdcBnjcObfLOVcAPApMOOWJG7le6a04p08aC3K2U9dI5uivqa1j+puf82+vr2VsVjqv3DqSzm0jnkQRaRJUlJ+EovIqHvvbl5zdqz2jT+sQdByRJi+jTQoLbhnJhG915uG31nPvglVUVDeNGStOsTwgycyyQtoGA7kR1h0MzHPOFTrnKvEu8hxuZunOuQPADqBxVJEBu+rs7mzdX87yzfuDjnLSisqr+NEzK3h66WauH92LP1yXTevmmqVZmjYV5SfhPxet40B5Nb+8bKCOkovEiBbNEpl5zdCv7v458UldAFpfzrkyvCPe082slZmNBi4H5kdYfQVwnZm1NbNkYBqw0z8qDvAMcIeZZZhZe+BuYGHDb0Xjc/HAzrRJSeKFj7YFHeWkrM0v5rKZS/l4cyG/njyIB783gKRElSMi+hScoGUbCnjlkx3cNKYPZ3YJP8srIkEyM+68MIu5P8xmx4FyLn18CX9bp3Hm9TQNaAHsBV4EbnPO5ZrZGDMrDVnvp0AF8CWwD29oypUhy2fgFe55wDrgM+A/Gz5+45OSnMg1w3vw1trdcflF0znHyyu2MXHWMqpr63jplnN0l06RECrKT0B5VQ2/eH0NPTu05O5vZ33zfxCRQFx4RiYL7xhD13YtuOHZHKa/+TlVNXVBx4oL/nCUK5xzrZxzPZxzL/jti51zrUPW2++cu9Y5l+Gca+ecO9c593HI8mrn3DR/WSfn3J3+1IlyAq4f3YsEg7lLNgcdpV5KKqq5++WV/OuraxjRO42Fd5zLWT00hbBIKBXl9VRTW8cdL3zGtsJyfjXxW6Qk6/beIrGsRwdvRocfjerF00s3M3HWUr7cE+l6RZHY17ltCy4f0pWXV2znQJzcMOuTrQeY8NhiFq7exU/H92Pe9cPp0Lp50LFEYo6K8npwzvHAG7n8bf1eZlwxkFF9NQWiSDxISU7kocsGMOcHw9hZVMEljy9h7pLNjWYWC2labh7bh0PVtcxfvjXoKMdUUV3Lr95ax5TZy3AOFtwyktsvyCIxQddgiUSiovw4VVTX8uD/5PLix9uYNq4v147oGXQkEamn8QM68c7dYxmblc6MhZ8z5akPddRc4k6/zDZc0D+Decu2UFpZE3SciHK2FPK9x5fw1AebuOrs7rx11xiG9dRwFZFjUVH+DWrrHB9t2s8ljy3muQ+3csO5vbnvO6cHHUtETlDHNs35w3XZ/G7KYDbuK+WSx5bw6LtfcKhKUydK/LjrwiwKy6p48v0NQUc5QnF5Nb94fQ2TZ39IeVUt864/m19NHESblOSgo4nEPE0KehQfbdrPzPc3sHJbESWVNXRum8LzN4zgXN21UyTumRmThnXjvNM7MmPh5zz29w28+mk+D1x6Bt8Z0ElTnErMG9y9HROHduWPSzZzzfAedE9rGWie2jrHCx9v49F3v6D4UDU3ntubn1zUj1aae1zkuOnTchS1dY59JZVcNqQL2b3ac+EZmaTqm75Io5Leujn/ffVQrhnegwffyOXW5z8lu2d7fj7hDJ1ql5h3/3f789ba3fzX/65j1r8MCySDc4731u3lt+98wRd7ShjRO40HvzdAUwWLnAAV5Ucx6rR03r57bNAxRCQKzunTgUV3nsuCnB38/r08Js1axgX9M7jrwiwGd28XdDyRiDq1TWHauL787q95LPmyIKpncp1zfJC3j8f+9iWfbiuid3orZl17Ft8dqDNNIifKnGsasw9kZ2e7nJycoGOISIwrq6zhmaWb+eOSzRSVVzMmK52bxvRhTFZ6YMWGmX3inMsO5JcHRH328amormXCY4s5eKiGRXeeS2ZqSoP+vuraOt5eu5s5/9zEmvxiOrdN4Y4LspiS3Y1k3ZVTBDjxPltFuYhIBKWVNTz34RaeWbqFfSWVZGW0ZuqIHkwc2o22LaM7lE1FuRxL3p4SLp+5lIFdU3nhpnMapDjeWXSIVz/ZwZ8+2sbugxX06tCS28b15cqh3WiWpGJcJJSK8m+gDl5ETkRlTS0LV+1i3rItrMkvpllSAhedmcn3BnVm3OkZUbmBmIpy+SZvrMznrpdWcu2IHsy4fCAJp2Au8KLyKt79fA9vrtrJkg0FOAfnnpbO9aN7cf7pGafkd4g0RifaZ0dtTLmZpQFzgfFAAfDzw7dtDlvPgIeBG/2mucC/Ov/bg5kN8dvOANYBNzjnVjb8FohIU9Q8KZFJw7oxaVg31uYX80rOdhat2cWi1bto2SyRUX3TOb9/R0b26UDv9FYaTyuBuHxIV3J3HmTOPzexr6SS3181pN4zn9TU1rF+dwmLvyzgn3n7WLGlkJo6R7f2Lbjjgiwmn9WNHh2CneVFpDGL5oWeTwBVQCYwBFhkZqucc7lh690MXAEMBhzwV2ATMNvMmgFvAP8PeBK4BXjDzLKcc/Fxv2ERiVsDu7ZlYNe2PHDpmXy4aT/v5u7h7+v38t66PQCkt27GkO7tOKNzKv07pdKzQ0u6p7UkNSVJxbo0uJ9f3J/ObVOYsfBzJj65jGnn9+WiMzNp2ezIP/W1dY69JRXkHzjEpoIy8naXkLvzIKt2FFHuz9ffv1MbbhrbhwkDOzOwa6revyJREJXhK2bWCjgADHTO5flt84F859zPwtZdBsxzzs3xn98A3OScO8fMxgPPAN1CjpxvA252zr19rAw6FSoiDcE5x6aCMj7eXMiKzYWsyS9m475S6kK61uZJCaS1aka/zDY8++Ph9f4dGr4i9fFB3j5+8doa8osO0bJZIl3btQCgqraO4kPVHDxUfcT7MyU5gX6ZbRjavR1n9WzPyD4dyGjgC0ZFGrNYH77SD6g9XJD7VgHnRVh3gL8sdL0BIctWuyO/Saz2279WlJvZzXhH3unRo8cJhxcRORozo2/H1vTt2Jprhnv9TEV1LRv2lrLjQDnbCsvZX1pFYVmVbqQiUXFev44svv98VmwpZNGaXRSUVgKQlJBA2xbJtGuZTKe2KXRp14JeHVrRI60liRofLhK4aP2FaA0Uh7UVA22OY91ioLU/1rw+Pwf/aPsc8I661D+2iEj9pSQnfjXURSQICQnGiD4dGNGnQ9BRROQ4RWseo1Ig/PZeqUDJcaybCpT6R8fr83NEREREROJCtIryPCDJzLJC2gYD4Rd54rcNPsp6ucAgO/KKk0FH+TkiIiIiInEhKkW5c64MeA2YbmatzGw0cDkwP8LqzwH3mFlXM+sC3AvM85f9A6gF7jSz5mZ2u9/+94bMLyIiIiLSkKJ5G65pQAtgL/AicJtzLtfMxphZach6TwFvAmuAtcAivw1/2sMrgOuAIuDHwBWaDlFERERE4lnUpgJwzhXiFdTh7YvxLuA8/NwB9/uPSD/nM2BYA8UUEREREYm6aB4pFxERERGRCFSUi4iIiIgETEW5iIiIiEjA7MibYzZeZrYP2FrP/5YOFDRAnGhR/mDFc/54zg6NL39P51zHoMIE4QT7bIjv1z6es4PyBymes0Pjy39CfXaTKcpPhJnlOOeyg85xopQ/WPGcP56zg/I3ZfG87+I5Oyh/kOI5Oyj/YRq+IiIiIiISMBXlIiIiIiIBU1F+bHOCDnCSlD9Y8Zw/nrOD8jdl8bzv4jk7KH+Q4jk7KD+gMeUiIiIiIoHTkXIRERERkYCpKBcRERERCZiKchERERGRgKkoj8DM0szsdTMrM7OtZjY16ExHY2bNzWyun7PEzD4zs4tDll9oZuvNrNzM3jeznkHmPRYzyzKzCjN7PqRtqr9tZWb2FzNLCzLj0ZjZ1Wa2zs+50czG+O0xvf/NrJeZ/a+ZHTCz3WY208yS/GVDzOwTP/snZjYkBvLebmY5ZlZpZvPClh11X/ufk6fN7KC/nfdEPTxHz29m55jZX82s0Mz2mdkrZtY5ZLmZ2SNmtt9//NrMLIhtiFXqt6NPfXYw4qnfVp9dvz5bRXlkTwBVQCZwLTDLzAYEG+mokoDtwHlAW+ABYIH/oU0HXvPb0oAc4OWggh6HJ4AVh5/4+/wp4Ad4r0U58GQw0Y7OzC4CHgGuB9oAY4FNcbL/nwT2Ap2BIXjvo2lm1gx4A3geaA88C7zhtwdpJ/AfwNOhjcexrx8CsoCewPnA/Wb23SjkDRcxP94+ngP0wstYAjwTsvxm4ApgMDAIuBS4pYGzxhv129GnPjsY8dRvq8+uT5/tnNMj5AG0wuvY+4W0zQceDjpbPbZhNTDJf1MsC9u2Q0D/oDNGyHw1sADvg/i83/ZfwAsh6/T1X5s2QecNy74MuCFCe8zvf2AdMCHk+W/w/qiOB/LxZ2jyl20Dvht0Zj/LfwDzjndf+9syPmT5DOClWMkfYflZQEnYe+zmkOc3AMuDfh1i5aF+O5C86rODyx93/bb67OPrs3Wk/Ov6AbXOubyQtlVArB5xOYKZZeJtQy5e5lWHlznnyoCNxNi2mFkqMB24N2xReP6N+H94o5fu2MwsEcgGOprZBjPb4Z9KbEF87P//Bq42s5Zm1hW4GHgbL+Nq5/cmvtXEVvZQR93XZtYe6BK6nNj/TI/F+wwfdsT2Efv5o039dhSpzw5cY+i31WdHoKL861oDxWFtxXinuGKamSUDfwKedc6tJ362ZQYw1zm3Paw9HvJnAsnAZGAM3qnEocC/Ex/5P8DrKA4CO/BOIf6F+Mge6lh5W4c8D18Wc8xsEPB/gftCmsO3rxhorXHlX4m39+tX4rTfVp8drMbQb6vPjkBF+deVAqlhbal444Vilpkl4J2urQJu95tjflv8i1C+Dfw+wuKYz493ug3gcefcLudcAfAoMIEYz++/Z97BG9fXCkjHGyf3CDGePYJj5S0NeR6+LKaY2WnAW8BdzrnFIYvCty8VKA07ItaUxdv7FYjPflt9drAaUb+tPjsCFeVflwckmVlWSNtgjjwtEVP8b15z8Y4ATHLOVfuLcvGyH16vFd4Yv1jalnF4F0psM7PdwE+BSWb2KV/P3wdojvcaxQTn3AG8IxWRPmixvv/TgO7ATOdcpXNuP96FKhPwMg4K+1Y/iNjJHu6o+9p/jXaFLicGP9P+zAPvATOcc/PDFh+xfcRg/oCp346ecajPDlJj6bfVZ0cS9OD/WHwALwEv4n0LHY132mFA0LmOkXc2sBxoHdbe0c8+CUjB+yYdUxeHAS2BTiGP3wJ/9rMfPj03xn8tnifACz2OsQ3T8WYgyMA7YrEY7/RuPOz/TcDP8GaDaAe8jncqvRmwFbgL74/q7f7zZgHnTfL35a/wjjCm+G3H3NfAw3infNsD/fE6/Khf/HSM/F3xxlPed5T/dyvexV1d8cZa5gK3Bv3+iaWH+u2o5VafHXz+uOm31WfXr88O/M0Viw+8b6J/AcrwrlyeGnSmY2TtifeNvwLvdMnhx7X+8m8D6/FO2f0D6BV05m/Ynofwr+T3n0/1X4MyvKme0oLOGCFzMt4UVUXAbuAxICUe9j/eeMp/AAeAAuAVIMNfNhT4xM/+KTA0BvI+5L/fQx8PfdO+9v9APY1XMOwB7oml/MCD/r9DP8OlIf/PgF8Dhf7j14TMsKCH+u0At0V9dvTzx02/rT67fn22+f9ZREREREQCojHlIiIiIiIBU1EuIiIiIhIwFeUiIiIiIgFTUS4iIiIiEjAV5SIiIiIiAVNRLiIiIiISMBXlIiIiIiIBU1EuIiIiIhKw/w/mTPgGRC83DgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_sched()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Smith's original 1cycle paper used a linear warm-up and linear annealing. As you see above, we adapted the approach in fastai by combining it with another popular approach: cosine annealing. `fit_one_cycle` provides the following parameters you can adjust:\n",
    "\n",
    "- `lr_max`: The highest learning rate that will be used (this can also be a list of learning rates for each layer group, or a python `slice` object containing the first and last layer group learning rates)\n",
    "- `div`: How much to divide `lr_max` by to get the starting learning rate\n",
    "- `div_final`:  How much to divide `lr_max` by to get the ending learning rate\n",
    "- `pct_start`: What % of the batches to use for the warmup\n",
    "- `moms`: A tuple `(mom1,mom2,mom3)` where mom1 is the initial momentum, mom2 is the minimum momentum, and mom3 is the final momentum.\n",
    "\n",
    "Let's take a look at our layer stats again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsQAAADWCAYAAADW1JQ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3hcZ5X48e+ZLo16tyXLcu9xXNIdEpNOJ6EEkgC7QIBQf5AldLK0pbP0JQllN4QSIAklJAHSnAQSx3bcu2XZVu8jzWhGmvL+/rgzsnqxJY00Op/nmcfWnTt33rE1M+eee97zijEGpZRSSimlZitbsgeglFJKKaVUMmlArJRSSimlZjUNiJVSSiml1KymAbFSSimllJrVNCBWSimllFKzmgbESimllFJqVtOAWCmlUoyIXC4iRkTKkj0WpZSaCTQgVkqpGUBEyuJB7uXJHotS04WI3CgiVSLSLiI/FxFnn/vsIvKCiLw5mWNUM4MGxEoppZSacUQkH/g58GngUuBi4NY+u3wUqDXG/DYJwxszEXGl4nPNNBoQz2Ai8pSI/FREviQijfEz5C+LiE1EPiciDSLSJCJf7vMYh4jcKSLHRSQkIvtE5D0DjvthEdkpIn4RqReR34jInD73Jy7HXiUiW0SkS0T2i8g1U/n6lUpFIrJJRJ4Tkc74bVf8vXUqvsuT8fdfVZ/HfFBEquPvxceA8mSMXakpthDwGWPuM8bsAR4CVgKIyBLgw8D7xnKg+PfiURF5rYgcFJGAiDwpIosG7LdBRP4W/35sEpEHRGR+n/sXxLfVxt+Pe0TklgHHSHx3f1FE6oCaYcb0VPy9PvD2jj77fDA+3pCIHBGRT4uIo8/9VfEY4Uci0gI8F98+J/7d3i4iwfhzbRzLv1Wq0oB45nsD4AQ2YZ0Nfwr4C5CBdcZ8O/ApEbkuvv89wPXAe4AVwBeAr4nIOwcc93ZgDfB6rC/X3wzx3N8EvgKsBbYBvxWRnAl7ZUrNMiJiB/4EvACsj9/uBLrifwe4AZgDnBd/zGuB7wDfBs4F7ge+MZXjVipJjgLpIrJRRLzAZcBLIiLAT4FPGWPqx3G8OVgB9E1Y2eYc4GeJO0VkJfA08C9gI/ByIAr8XUQ88d0ygMeBa7G+Q+8Cfi4imwc815uAQuCK+HGGcn18TInbl4AA8GJ8PHdifVd/Euv7/MNY3+2fH3CcDwGNwEXA2+P/Pg8By4FXAecDDfHXUTDiv1AqM8bobYbegKeAnQO27QP2DNi2Cyt4XQDEgOUD7v/cwOMMuH8dYIDS+M+Xx3++vs8+JfFt1yT730VvepupNyA3/j66fIj7yoa6D3gWuG/Atm/G9y1L9mvSm94m8wa8Ov4dV4l1YmgHPgg8HP9eeih+3/8BGSMc504gAhT22XZj/DvTE//5F8BvBjzOjXXC+roRjv1H4O4+Pz8FHAZs43id1wDdwKvjP6fHn/faAfu9DWjv83MV8PiAfa6Ifz6sHPA66oDPJfv/NFm33rS6mrF2Dfi5Pn4buK0I64xWgG3WCWIvB9ZZLmCVRGCdca7EOkNOXEmYT/9LOzsTfzHG1ItIFCg+w9eh1KxnjGkTkXuAx0TkCaxs1IPGmEMjPGwl8OsB254FPjZJw1Rq2jDG/Bn4c+JnEakAPgFcAHwPOIB1JfWXwGeBO0Y4XK0xpqnPzzVY35lFwEmsqzKLRcQ/4HEeYEn8+dOxkkyvxsrqurCCzScHPGa7MSY2ltcoIquwrvzcEX+9AKuANOAPImL67G4HPCJS2Oe1bB1wyFVAizFmf2KDMaZbRF6I3zcraUA884UH/GyG2WbjdGB7MdaZ5cB9EJFy4K/AvVjlFM1Ymal/YL2x++oZYjxahqPUWTDGvFtEvgtcDVwFfFFEPoCV8Rr2YVMyOKWmv7uBO40x1SJyJfCfxpiIiPwS+M9RHjvwOy3xvrL1+fNe4KtDPLYl/uc3gNdinZAexCpx+BaQPWD/wGgvBEBEirDKIH9pjPnvPnclxvRGrGzzQK2jPNdQnxkyzPZZQQPi2WV7/M9yY8xfhtnnPKyzzo8YY4JgTSKYisEppSzGmL3AXuDbIvI/WDPnH4zfbR+w+37gEuBHfbZdMumDVGqaEZF3A2KMuTu+yYY1xwashM7ZJmy2AecAx0y8zmAIL8MqYfptfEw2YClWje64iIgb631/EKsOuK99QAhYaIz56zgPvQ8oEJGViSxx/LnOp//nyKyiAfEsYow5KiI/A+4WkY9jTQzwAhuw6qa+BhzBOkP8mIjchzVh7nPJGrNSs4mILAbejXUJ+BQwF2ty7A6sqzV+4GoR2Qd0G2PasLJPvxORrVhXdzYBtwxxeKVSloiUYk0m29Rn8xbgo/FOS+/HKkE6G1/BKj/4ZfwqThNQAbwO+K4xphI4BLxWRP6A9X79KNb7eNwBMfCT+GP/DSjsU+roM8b4ReQrwFfi2/+OFdOtAdYZY0YqDXki/jp+JSLvB3xY5SQe4MdnMM6UoJe3Z59bsSYefBors/Q48HasSQcYY3ZjTUh4T/z+24GPJGWkSs0+AaxaxN9gXQb9A/BP4APxesP3Y81OPwW8BGCMeRDr8uzHgd1YM+RH+jJUKhX9BPiqMaaqz7YPYQWs27DeW6OVTIzIGHMAq+QwA3gM6zvybqyrqu3x3f4fcAKrZvhxrDrk35/hU16ONf5DWBPeErc3x8fzxfjzvQtrPtGz8Z+rRnkdBiuIP4hVivUi1gTEq4wxzWc41hlPhs/6K6WUUkoplfo0Q6yUUkoppWY1DYiVUkoppdSspgGxUkoppZSa1TQgVkoppZRSs5oGxEoppZRSalZLeh/igoICU1FRkexhKDVtbN++vdkYU5jscQxF369K9afvV6VmjpHer0kPiCsqKti2bVuyh6HUtCEiJ5I9huHo+1Wp/vT9qtTMMdL7VUsmlFJKKaXUrKYBsVJKKaWUmtU0IFZKKaWUUrPahAbEIvIBEdkmIt0i8ouJPLZSSimllFKTYaIn1dUCXwKuAdIm+NhjZozhJ1sqee25c5mTnbRhKKXUlPq/f1Wxbl4ua8qykz0UpdQECEdjPHmwke0n23DabCyfk4kgLC7KYFlJZrKHl1ImNCA2xjwAICIbgbKJPPZ4VDYH+OojB4kZw22XL07WMJRSaso0dob43B/3sbYsm4fefwkikuwhKaXOQmugh9vu287zla247DaixhCNGQBE4A3ry3j/5sVUFHiTPNLUkJS2ayJyK3ArQHl5+YQfv7IpAEBde2jCj62UUtPRM4ebAdhV7WPr8VYuWJif5BGp6a47EuWBHTW8eeM8bDY9gZounjjYwE+ermRPjY9IzPD1G87htevmYszp+OahnTX8/Lnj/G57NZctLeSmC8qZm5NGTrqTstz0JL+CmSkpAbEx5i7gLoCNGzeaiT5+ZZMfgNr24EQfWimlpqWnDzdRkOEiZuDuZyo1IFaj+vOuOj75wB4e21fPf7/5XHLSXcke0qzVGujhsX31vFjVygM7alhQ4OWNG8p448Z5rC49XQK1cm5W75/v2rSAX209ya9eOMmt927v3efChXmcvyCfZcWZXLWyGJdD+yeMRdIX5pgMx+IBcY0GxEqpWSAaMzxzpInNy4uYl5vOdx8/Qp0vqHMo1IhuWF9KMBzlC3/ex6ce3MOPbtqQ7CHNSsea/Lz9Z1upbgvidth456YFfPzaZbgd9hEfV5Tl4SNXLuX9mxfzfGULoXCMww2dPLCjmh88cYSYgbnZHt6woYwLFuYTCkex2wSn3cbOU+1UtwUJR2OsnJPF2nnZzMtNJxwzhMJRBMhOc5Kb7sJmE7p6ItT7QqS7HBRmurEPuKIQCkc52dpFa6CHjmCYmAGP08Zj++o51hTg8mWF5KW7CEdjzM/3UucLsremA4BILEZPxJDpcdARDHOsyY+IMCfbw/ryXB58qYaTrV3c+rKF5KQ72Vvj40RLF3leF4uLMmjoCBEzkOd1cfMF8ynPP7MMeUoGxIlLCpohVkrNBntqfLR1hblsaSGLizL47uNHeL6yhdevS9pUDjUDiAi3XDifbVWtbD/RluzhzDq17UHue+EE//fPE7gcNn733ovYUJ477vIVp93GpUus1YivWlnM+zcvpicS47ljzdy9pZIfPHmU7z1xdNDjCjLciMDvt1cPe2ybgNftoDMU6d2W6XGwam4WjR3d2GxCUaabHSfbCIVjgx6f5rRTUeDl648eGnRfpseBwyY47DZcdhsdoTBpTjtLizOx2YSdp9p5ZG89FfnpnFeRy7f/fhiAfK+LigIve2t9PLK3nsJMN06b0BLo4VXnzKGcaRAQi4gjfkw7YBcRDxAxxkRGfuTEqmwOIAIdoQidoTCZHudUPr1SSk2pZw43IQKXLikkJ81JdpqT54+1akCsxmRhQQZ/2lVLKBzF4xw5K6lGFwpH2VvjwxcMk+ays748t/ffNRYzPLavnnuePc72E22IwNUri/n0K1aecWZzKC6Hjc3Liti8rIjWQA/7azvI8DiIxgzBniir5maR67VKZGrbgxys76CmLYjTbiPNZccYaO/qoTXQgy8YJj/DTVluGsFwlL01HRyo62BZSSaRmKHOF+RNG+exsSKPfK+L7DQr5uoIhVldmk2Wx0ljZ4hI1GAT4XhzgPwMF0uKMkac/GuMoaY9SEmWB4fdxtHGTjxOO6U5ab2PC0djOO22fo85UxOdIf4M8Pk+P98M/Cdw5wQ/z7DaAtZ/4NqybHZV+6jzhTQgVkqltK1VrSwrziQv/gV3wYI8/lXZkuRRqdGISB7wU+BqoBn4pDHmV0Ps5wa+C7wecALPAe81xtRMxDgWFXkxBo43B1gxJ2siDpnyWuOxxqJCb29wFgpH+c7fD/Pz56roiZ7OlnqcNvK9btq6egiGoxgDCwq8fPzaZbxqzdwJDYSHkud1sWlJwbD3z81JY27O5JZXFWV6ev9eku0ZYc/TRKTfBMHFRYPbzPUNhhOPOVMT3XbtTqYw+B1KZbNVP7xpSQG7qn3UtAdZWqy9+pRSqSkSjbHjRBvXrz+dDb5wYT5/299ATXuQ0kn+olNn5YdAD1AMnAs8LCK7jDH7Buz3YeAi4BzAB9wNfB+4fiIGsbAgA7BqWTUg7i/QHaGpsxu300ZxpoeqlgA/ebqSB3fW0BOJsajQSzhqaOrsxiYQ6Ilyw/oyrl1dQlGmm9ZAD08fbqIjGCbP6yLdZWdpSSbXrZ4zqA5XJVfK1RAfi9cPb1pcyA+fPKZ1xEqplHawvpNAT5SNFbm92y6Md5h4/lgLN2zQsonpSES8wA3AamOMH3hWRP4E3AJ8YsDuC4DHjDEN8cf+Bvj2RI1lQbyPbWL+zWxV2x7kqUNNZHgcBLoj7DrVzp921dLVEwXA67ITDEdxOWy8aWMZS4oy+ceBBrI8TuZkewhFoly9soSXLS3sd9zNy4uS8XLUOKVgQOzHaRfWz8/BbhMNiJVSKe3FqlYAzqvI6922vCSTnHQnW4+3akA8fS0FosaYw3227QIuG2LfnwLfFZG5QDtwE/DIRA0kzWXVZSZals4EnaEwNhG8biuM+fXWk9z3wgmWFWdx/fpSLllcwM5T7dy/7RQvVLawrjyXy5YWMj8/nRVzshDgqUNN8S4Ifgoz3Tx1qInuSP9Sh9esncsFC/IJRaIcru8kK83J2y+uoCDDDcDbL65IwqtXkyH1AuJGPxX5XtwOOyVZHmp1cQ6lVArbVtVG6YAaQJtNWFOazb46XxJHpkaRgVX+0JcPGKrG7zBwEqgBosAe4ANDHfRMF75aWOjtvcI6nf19fwM/eOIIu2t8GANZHgfz8tLZV9vBsuJMnjjYwB92VLO8JJOD9Z2kOe1smJ/Lo3vre7spZKc5SXfZqfOFyPI4WDEniwN1nbxyzRzed/kiwDpJSEzmUrNDygXEhxo6WVuWA8DcHI/2IlZKpSxjDC9WtXLRosGLcKyck8XPn6saNAtbTRt+YGDBbhbQOcS+PwY8QD4QAD6OlSG+YOCOZ7rw1aLCDH637RTGmGm77Pc9z1Ty5b8eYHFhBh++Ygluh51TbV0crOvgI1cu4YMvX0I4GuOuLZX8eVctt1+9lH+7ZAFet4PuSJRjjQEqm/08fqCRjmCYz796FVesKNL3hwJSLCAOdEc41RrkTRvmAdbMSe2tqJRKVW1dYRo7u1nTZyWrhBVzsuiJxqhsCrCsRCcWT0OHAYeILDHGHIlvWwsMnFCX2P5pY0wrgIh8H/iCiBQYY5onYjALC70EeqI0dHSPuQvAVNpd3c6XHj7AtatK+O8bzx22PZzdZudDVyzhQ1cs6bfd7bCzcm4WK+dm8apz5k7FkNUMk1KnRYcbrBPrxId/aU4a9b4Q0diErw6tlFJJV+ezroAN1Uki0S1gv5ZNTEvGmADwAFZg6xWRS4DXAvcOsfuLwNtEJFtEnMBtQO1EBcNwutNEolPTdPPz56rwuux8443naK9kNSlSOiAuy00nEjM0dGgdsVIq9SQ+24qHyOgtLPTictg4UDfUFXg1TdwGpAGNwK+B9xlj9onIpSLSNzK9HQgBR4Am4BVYPYknTFGWNUmsxd8zkYedEI2dIf6yu5Y3bpyn6wqoSZNSJROH6v14nDbmxRs5l+VaWZPqtuCkN51WSqmpVu/rBqAka3BA7LTbWFqcwYG6jqkelhqjeAnE64bY/gzWpLvEzy1YnSUmTU66FWi2d02/gPiX/zpBOGq0o4OaVCmVIT7U0NG7Bjb0DYi7kjkspZSaFPW+IDaBwkz3kPevKMlif23HWS1nqmaHnDRrlcO2rnDSxtAdiRLps8IbWGVBdz9znGtXlfT2S1ZqMqRWQFzvZ1mfVekSWeHqNu00oZRKPfUdIQoy3MPOkl85N4uWQA+Nnd1TPDI107gcNjLdDloDyckQ17YHueJbT3PTPS/0C4q/8teDxIzh069ckZRxqdkjZQLi1kAPzf7ufrOpPU47RZluzRArpVJS/SgdAc4ps7pPvHRSu+2o0eV4nUkpmfB1hXnbz7bS2NnNC8db+cGTRwGrs8Sfd9XynssWMS8vfcrHpWaXlAmI99RYM6kHrsNelpumGWKl1LR1qrWLe58/cUZlDfW+4JD1wwlrSnNwO2xsPa4BsRpdbrorKSUTD+2s4Wijn1+84zyuX1fK9x4/wo6Tbfzs2eNkuB28+9IFUz4mNfukTED80sk2RE5nRBLKctM1IFZKTVv3PFPJZx/ay6GG8XeDqPeFRswQuxw21pXnsLWq5YzHt+NkG++9dztNWnaR8nLSXbQlIUP80sk2ijLdXLQon/987SqKszx87P5d/GV3HW/SzhJqiqRMQLzjZDvLijMHvXHKctOobQ9qL2Kl1LT0fGUrAH/ZVTeux3X1ROgIRSgeIUMMcP6CfPbXdtAZGn/mr6Y9yK3/t41H99Xzhb/sH/fj1cySl+5MSkC8q9rHufNyEBEyPU6+/PrVHG8OEDWGd2hnCTVFUiIgjsUMO0+2sa48d9B92otYKTVdtQZ6ejPDD++pG1fZRL3P+kybM8qqYhcsyCNmGPeqnUcb/bz9Z1vpDsd408Yy/ryrlicONozrGGpmyUl30R6Y2pKJ9q4ejjcHWDsvp3fby5cX857LFvLuSxdSnq+1w2pqpERAXNnspyMUYV15zqD7+vYiVkqp6WTrcSs7fMP6Mo43B9g/jp7B9fGT/JFqiAHWlefgsEnvc43FlsNNvPr7z9Ia6OEnb9vAl163hoWFXr7998NjPoaaeXLTXXR2R+iJxEbfeYLsqrbm/6yb1//7+5PXreBTr9DOEmrqpERAvONkOwDrh8wQay9ipdT09MLxFjxOG3dcuwy7TXh0b/2YHzvSKnV9pbscrCnL5rmjY1vl92RLFx/41Q7m56fz1w9dysWLCnA5bNxy4Xz21nSMuNBHdVsXv3juOPe/eIrbf7eLt979PLXtQY40dPL1Rw/yyJ46QuHomF+jmlp53vjiHMGpK5vYebIdEVgzYP6PUlMtJVaqe+lkG1keBwuHaNo9NycNEahq0YBYKTW9vFDZyvryXIqyPCws8HKofuwT6+p8Y8sQA1y5ophvPHZo1El44WiM2361HYC7btnYb9/XnlvKV/56gD9sr+Yzr1o55OM/89BenjrUBECmx0EsZnjDj/+JLxgm0GMFwi9fXsTP3nHe2F6kmlI56dbiHO1dYYoyR/+9mgg7T7WxuDBDJ86ppJvxGeLuSJS/7WvgksUFvSvU9eVx2llSlMHu6vYkjE4ppYYWCkc5UN/Bxoo8ACoKvFS1BMb8+AZfiEyPA6979LzGNauKAfj7/pEz0Pf+6wR7azr42g3nDKrdzPO62LysiId21hCODr6kfqCug6cONfGhly/mqdsvZ/tnruI3t15EMBxlcVEGz96xmQ+9fDFPHGwcdz2zmhq58YB4Khfn2FPj45yyweWOSk21GR8QP7q3npZAD285v3zYfdaX57LjRBsx7TShlJomatqDGAMLCqzAc0GBlxMtXWP+nGrs7B61w0TC4qJMFhZ6eWzf8JPimjq7+c7fD/OypYVcu7pkyH3efN48mv093HT3CxyJTwb82756vvf4Eb752CHSXXbeuWkhFQVeXA4ba8qyefaOl/PAbZdQlpvOey9fRJ7XxXcfPzKmcauplZsomZiiThMt/m6a/T2smJM5+s5KTbIZHxDf9/xJ5uens2lxwbD7rJ+fS0coQmWzfwpHppRSw6uJT/QtzbEC4vn56XRHYr2T5UbT0BGiKNM95ue7ZlUJz1e24Bti4QVjDJ95aA+hSJTPv3olIoOvtoFV7vC1G9ZwsL6Dq/97C6/+/rPceu92vv33wzx+sJEbzysnO73/pW+v24E9fvUu3eXg1pctZMvhJt577/YR65HV1EtkiKdqcY4jjdZ38pJiDYhV8k1oQCwieSLyoIgEROSEiLx1Io8/0Esn29ha1cpbzy8fslwiYcN8a7KdXqZTSk0XNe1WQDw3x8ryLsi35kBUNY+tbGI8GWKw6ogjMcNzxwZPrvv5c1U8tq+BO65dzqLCjGGPISK8+bxynrz9ct572SI6QmE+cuUStn76Cv7n5vV87Oqlo47j3y9ZwAc2L+ZflS3cdM8LNPt1wY/pYqpLJhJXGZYWD/87p9RUmegM8Q+BHqAYuAn4sYismuDnAKxLLe+/bwelOWnceN7w5RIACwu85KQ7NSBWSk0bNW1B7DbpnRRXEZ8UfHwMdcTGGBo7useVIT6nLJs0p31Q+7W/7avnK389wJUrinnnprEtkZuf4eaOa5fz9H9s5iNXLqUo08O1q+eMqZ7Z5bBx+zXL+N17L8IfivDpB/ec0bLVauKluey4HbYpK5k43OAn0+0Y08RQpSbbhAXEIuIFbgA+a4zxG2OeBf4E3HI2xx04ecMXDPPInjrecvfztAR6+MktGwZdohtibGwoz9WAWCk1bdS0BynJ8uCwWx/DJVke3A7bmDLEvmCYnmiMonEEEk67tYzzi1VWQByLGe5/8RS33beD1aXZfPvNa4ctlZgMS4sz+djVS3lsXwOPjKPdnJpceV7XlJVMHG7oZHFxxpT+3ik1nIlsu7YUiBpj+nZu3wVcdjYHXf35xzAG3A4bkZghGO9hWZGfzv/cvIHVpWPrXbixIo/HDzZypKFT65WUUklX0xakNCet92ebTZifnz6mFpENHVaZwXgyxGB9Dv7giSPU+0J84Fc72HaijfMr8rjnHRvJSkLbq3ddupAHX6rhvx45wMuXF+Fx2qd8DKq/nHQXbVNUMnG00c+VK4qn5LmUGs1EBsQZgG/ANh8wKPoUkVuBWwHKy4cvdzDG8IHNiwn0ROmORHHYhMJMN0uLM7l0SWHvRI2xePN58/jRk0f52qOHuOftG8f8OKVU6mnq7CY7zYnLkbx5xTXtQc5fkNdvW0W+l+NjyBA3dloT78YbEJ9fYS3jfNt929lxsp3/un4Nb944b8Q5GJPJbhM+88qV3PzTF7j5nhc41NDJh69YwrsuXZiU8SjITXfSNgUlEy3+bloCPSzR+mE1TUxkQOwHsgZsywIGdZo3xtwF3AWwcePGYYvHRIQPXrFkQgaX53Xx3ssX8Y3HDvFCZQsXLMyfkOMqpWaWHSfbeMtdz/OhK5bw/s2LkzKGSNTqJtE3QwxWHfFTh5uIxcyIQWoiQzyeSXVgLeNstwk7TrbzmrVzR2xXOVU2LSng6pXFPH24iYWFGXzp4QO4HDbedlFFsoc2K2WnOXtXQZxMhxusDhNL9YqtmiYmMj1yGHCISN8Idi2wbwKf46z8+yULmJvt4eN/2I0vODU1Ukqp6eNUaxfv/t9tdEdiVMfbniVDQ2c30ZihNLd/QLygwEtPJNbbgWI4vRnirPFliL1uB6vmZuGy2/iPa5aNb9CT6AdvXc+Oz17FH99/CVcsL+Jzf9zHe+7dRku8A8XRRj+docn7zK5qDnD1d54etIDTbOxd73U76OqZ/OW1D9VbLfc0Q6ymiwkLiI0xAeAB4Asi4hWRS4DXAvdO1HOcrTSXne+9ZR01bUE+dv8undms1Cxz3wsn6QiFKchw4QtO3WpcA53uQdw/IF5eYmXL9o/Sn7exo5tMt4N01/gv8n36FSv43lvWMS8vffSdp4jLYcPrduBy2PjJLRv4xHXLefJQE7f/bhdHGjp5xXef4d9/8WJvgGqMobpt9FrrsfrttlMcbvDz8d/v7p3Ifc8zldx67zZ6IoNX5UtlGW4H/u7IpD/Ps0dbKMtN0w4TatqY6AK624A0oBH4NfA+Y8y0yRCDNankE9ct5x8HGnj6cFOyh6OUmkK17UHmZKdRke+lLZC8q0Q17VYwNzBDvLwkC5vA/tpRAuLOEIXjzA4nXLAwf9iV6KYDh93Gey9bxB3XWkHxzT99AYPhxao2frX1JAB3P1PJpq89yV/31A17nLZAD2//2VZ+++JJ2rt6+NCvX+KuLccGZX1jMcMfX6qhNCeNg/WdfOqBPXz1kYN86eEDOO0zfu2qcUt32enqiU5qwqg7EuW5o81sXlakHSbUtDGRNcQYY1qB103kMSfD2y6q4CdbKvn5c1Vcvqwo2cNRSk2R+o4QJVkestKco5YlTKbhMuCDSSIAACAASURBVMRpLjsLCzPYN1pAPM4exDPROy6u4I87a9hd7eNbb1zLAy9V819/PUAoHOWbf7OaGX3hz/t52dJC0p127t92io5QmHduWkjMGN7/qx3881gLTx9u4pt/O0yzv5s/7arl/m3VRGOG+fnp3HjePDLcTmp9Ib73lnX861gLv44H3a9ZO5dvv2ltb1u82cLrdhCNGbojsUnr+rH1eCvBcJTNywsn5fhKnYkJDYhnCpfDxtsunM+3/n6Yo42dLC7Son41e8Xr/vcAvzfG3Jzs8Uymel+ItfNycDts7K8d2BRn6tS0h8j3uoYMOFbOyRq1Z3pDZ4j15bmTNbxpwW4TfvjW9fzrWAvXry/lokX5vO+X2/nSwwfISXfyg7es49Z7t3PTPS9gF9hx0qr/ffxAI8FwlN3VPr56/Rq2nWjjiYON/ObdF3KipYvfbT9FQYabl062895f7sBuE7wuO1etKOY1a+fyiWuX09AZYnFhRtK6byRTRnxxFX93ZNIC4qcONeFy2LhoYcGkHF+pMzErA2KAt15QzvefPMr//vMEX3zd6mQPR6lk+iHwYrIHMdmMMdR3hLgmy40xTNniA0OpbQ8OKpdIWDU3iz/tqqW9q4ec+FK6fZ3JKnUz1by89N5a57k5aTxw2yX8btspFhVlcF5FHp+8bjkPvlRDdyTGV16/BoAv/mU/8/PT+a/r13Dj+eXceH450ZjBbhMuWJjPm86bB0A0ZnjiYCO/3nqSDfNzSXNZwV92unPUxZ5SWWK1wa7uqNVMdRI8eaiRCxfm9/6bKzUdzNqAOD/DzdUri3lkbx13vmbVuHoaK5UqRORGoB34J5CcHmRTpL0rTE8kRkl2GqFwlGA4SigcTcpiELXtQRYVDh1trJxrda/cX9vBxYsHZ9A6ghG6I7Fxt1xLBXabcGOfVnHvuWwR77lsUb99bjxvcF/loT7f7TbhqpXFXLVSF4boyxsPUidrYl1te5DKpgBvnQYt/5Tqa3YVRw1w9aoSmv09vHRSl3RWs4+IZAFfAD6W7LFMhfp4b9WSLA/ZaVYGsCMJ7ReNMdS2B5mbM3SGeOWceEA8TKeJWp9Vf1ySPfsC4rGYjWUOEymRIQ70TE5A/MLxFgAuWqRrAajpZVYHxJuXFeK0C3/b35DsoSiVDF8EfmqMOTXSTiJyq4hsE5FtTU0ztzNLvS8eEGe7yYlfEk9G2URHMEKgJ8rcnKED2vwMNyVZnmEn1p1osVayq8j3TtoY1ezVGxBPUob4+WOtZKc5WVEycB0vpZJrVgfEmR4nFy8q4LF99dqTWM0qInIucCXwndH2NcbcZYzZaIzZWFg4c2eFJzLExVkectKs2tz2KViidqBEd4uBHSb6Wj4nk4P1gxb5BKCqxWrZVp4/ffoIq9ThdVslE4HuyVmc4/njLZy/IE8z+WramdUBMcDVq4o50dLF0UZ/soei1FS6HKgATopIPXA7cIOI7EjmoCZTvS+ECBRlenozxO1jLJl48lAjd2+pnJBx1MYD4uFKJgCWlWRyrNHfu0hEXydaAuR7XWR5Zu/ELzV5vK7JyxDXtgc50dLFhQu1XEJNP7M+IN4Un7Sytao1ySNRakrdBSwCzo3f/gd4GLgmmYOaTA0dIfK9blwO2+mAeIwZ4m88eoivPHKAU61nvzpazRgC4uUlmfREY1Q1BwbdV9XcxXzNDqtJkjGJNcSJ+uELF+ZN+LGVOluzPiAuz0sn3+tix4n20XdWKkUYY7qMMfWJG+AHQsaYGVUk7AuGOdLQSVNn96j71neEKMm2WpUl2pm1j6GG+ERLgP11HRgD928bsdx6kHueqeSeZ/pnlmvbg7gcNvK9g1uqJSwttnqjD1U2caIloPXDatKk95ZMTGxA3BOJcdeW4xRnuVmu9cNqGpq1bdcSRIT183PZoZ0m1CxmjLkz2WM4E2+563n213VgE3j6Pzb39qwdSr0vRFm896/XZcdhkzGVTDyytx6A1aVZ/PbFU3z4iiVjWr2sqjnAVx85iNft4N8uWdDb+qumPcjcbM+INZSLizKw24RD9Z28eu3p7aFwlFpfiPkaEKtJ4nbYcdoF/wTUEO881U5VcwAR2H6ijQN1Hdz9to3a5lRNS7M+QwywvjyX480BWgNTP8FGKXXmqloCLCr0EjNwfIjygr6sDLHV2UFEyEl3jalk4pE9dawty+bDVyylsbObh/fUARCLjTwR95t/O0QkZvAFw+yuPn0FaqSWawluh50FBd5BGeJEyUZFgZZMqMnjdTvOOkP8+IEGXvfD5/jIb3fy4d/s5P/+dYI3bCjTvs9q2tKAGNgw31oCdccoy6UqpaaPQHeErp4oF8Qn6DTEu0gMJRSO0t4VpqTPYhY56c5RSybqfEF2Vfu4dvUcNi8rZHVpFp/74z5+v72ajV/+B595aA+xmOHRvXX9AvIdJ9v4y+46br6wHBHYcri5977a9tCoATFYE+sON/QPiBMdJjRDrCaT1+U4qxricDTGl/96gIUFXv7x0cv42/97Gfe+83y+pKvCqmlMA2LgnLJsHDbRsgmlZpBE3fCq+MpujSPUEfvipRF9l0LOSRs9IN563Jpse9nSQhx2Gz9863piMcPtv9sFwC+fP8nmbz3Fe3+5g/fcu41INEawJ8rt9+9ibraHO65dzjml2TxzxCrNDkdjNHSOLSBeXpzJydaufpm60z2INUOsJo/XbT+rDPFvXjxFZVOAT1y3nMVFGSwtzuTSJYVJWRVSqbHSgBjwOO2smpvFds0QKzVjNPmtAHhebjrZac4RM8SdISvwzfScnjaRk+4ctYZ4+4k2vC47y0qsSW7z87386Ob1vOPiCrZ8fDPv3LSA5s5ubjxvHocb/PxkSyWffGA3lc0BvvnGtWR6nLxsaSEvnWqnIxTmREsXxsC83LFliAEO1p9eoKOqJUB2mrNfYK/URLNKJs68hvi+50+wdl6OlkeoGWXWT6pLWFeey29fPEU4GsM5hgkzSqnkSmSICzPdFGe5RwmIrWxX3969OemuYVeD8wXDZKc52X6ijXXluf0mAV26pJBLl1gLlHz2VSv55HXLsduE6rYg33jsEAAffPliLo63dLx0SSHff+Iozx1ppiMemK8rzx319a2dlwPAzlM+NszPIxYzPHOkuTcjrtRkyXA78J9hhrjeF+JgfSd3XLscEZ08p2YOjfziNszPJRiOcrBu6NWhlFLTS/+A2ENDx/AlE4mAuF+GeJiSid3V7az7wt94ZE8dB+o6WD9/5ODVYbchInzl9Wu4+cJy/vLBTXzs6mW9968vzyHT4+Dpw028WNVGbrqTRYWj1wAXZ3mYk+1h1ylrQt7zx1s40dLFGzeWjfpYpc5GuuvMSya2HLbKgy5fNnNXtVSzk2aI4xJfejtOtrGmLDvJo1FKjaapsxu7TchLd1GU6eFYY/Ow+yYC4owBJRPBcJRQONqvtnHr8VZiBj7++93EzOlJt6Mpz0/nS69bM2i7w25j0+ICnjrUhNtpY2NF3pgzZ2vLctgZD4h/++IpMj0Orls9Z0yPVepMnU3JxFOHGynJ8rA8XvKj1EyhGeK4udkeSrI8Wkes1AzR1NlNQYYLm00oznLT2Nk9bCu00zXEp0smCjOtRToaB2SW99b4rMd0RxCBdeU5Zz3Wy5cVUt8R4kRLF+dVjC3ABji3PIeTrV0ca/LzyN56Xr+uVCcmpRARyRORB0UkICInROStI+y7XkS2iIhfRBpE5MOTNa4M95l1mYhEYzxzpJnLlhZquYSacTRDHGct0JGjAbFSM0STv7s3qC3O8hCJGVq7eijIcA/ad6iSiUTrsuMtAcr7dG3YV9vB5csKqWkL4rTb+tUdn6nLlhb1/n1jxdiXrT03Xkf8/vt2EInGuOXC+Wc9FjWt/BDoAYqxllB/WER2GWP29d1JRAqAR4H/B/wecAGTVjuT7jqzPsR7anx0hiK8bKmWS6iZRzPEfawvz6WmPTji5Byl1PTQ1NlNYUYiILb+HO692xn/cs9wnQ6IFxRYAXFVn/7BwZ4ox5r8nFOazS/fdQF3vW3DhIy1JNu6hOx22Fg9d+wlWWtKs7GJtYTzzRfOZ0mxXoZOFSLiBW4APmuM8RtjngX+BNwyxO4fBR4zxtxnjOk2xnQaYw5M1tgy3HbCUUN3ZHxlE0ca/YC1qqNSM40GxH0k6ohf0n7ESk17TZ2nM8RF8QU3BpY/JHSGwmS4Hf2WSy7KdJPusvdbUONAfQcxA6tKsynO8lCWO3H9fj9y5VJuv3oZLsfYP3a9bgdLizPJTXfy0auWTthY1LSwFIgaYw732bYLWDXEvhcCrSLyTxFpFJE/i0j5UAcVkVtFZJuIbGtqajqjgXnd1olj1zjriKuaAzhsQukY+mwrNd1oyUQfK+dk4bAJu+MrUymlpqdYzNA8oGQCrOWZh9IZivQrlwCrTGp+vrd3sQugtw3bZLQ2u3Z1yRk97ptvXEvMGO09nHoyAN+AbT5gqMsAZcB64CpgD/B14NfAJQN3NMbcBdwFsHHjxpHXFx+GN34lxd8dIdc79t+7qpYA8/LScWjrUjUDTchvrYh8IH5G2i0iv5iIYyaDx2k14N9TM/AzSik1nbQHw0RiprdkIvHnsCUTofCggBhgQUF673LIAPtqfOSkO6dVhmt1aTbnlJ39xD417fiBgWdeWcBQvT+DwIPGmBeNMSHgP4GLRWRSWiIlMsTjnVh3vLlLV1FUM9ZEncbVAl8CfjZBx0uac8qy2V3tw5gzOrFWSk2B0z2Ircywy2Ej3+sathexvzvSr8NEQkW+l1OtXUSiMQB2V/tYNTdLZ8irqXAYcIjIkj7b1gL7hth3N9D3Synx90n5RfW6rU4m42m9ZozhREuAioLRe2wrNR1NSEBsjHnAGPMQ0DIRx0umNaU5+IJhTrUGkz0UpdQw+i7KkVCU5aFxhJKJDPfgDHFFgZdIzFDdFqQt0MOB+g7Or8ifnEEr1YcxJgA8AHxBRLwicgnwWuDeIXb/OfB6ETlXRJzAZ4FnjTHtkzG23gzxODpNNHZ209UT7Z2sqtRMk5RCn4ko+p8s58QX5dhdM/hzpjsSxTfEylZKqanV5LcC374BcWGmmyb/cJPqBtcQQ59OEy0Bnq9swRi4ZLEGxGrK3AakAY1YNcHvM8bsE5FLRcSf2MkY8wTwKeDh+L6LgWF7Fp+tRA3xeALixOTURDtDpWaapEyqm4ii/8mytDgTl93GnmofrzpnLgDRmOHjv9/NX/fU4bALL376Sm2Or1QStQWsE9Pc9D4LbWS4OdIw9NLrVg3x0CUTYM2OP9rkJ91lZ+08rddVU8MY0wq8bojtz2BNuuu77cfAj6diXL0lEz1jL5lITE5doAGxmqFGzRCLyFMiYoa5PTsVg5xKLoeNFXMye5dLBXh0bz1/2FHNspJMOkMRjjb6RziCUmqyJTJX3j5lEIWZbpr93UPW/3eEImQNkSEuyHCR4XbwYlUb/zzawvkL8nDqDHk1y6W5rIA4FB57QHy8uQunXZib45msYSk1qUb95DfGXG6MkWFum6ZikFPtZUsL2VrVyr5aa3Ldj546ysICL19/wzkAHKofOgullJoa/u4IboetX/BamOkmHDX4gv3LmrojUXoisSFriEWEmy4o5+E9dVQ2B7hkUcGkj12p6S7NOf6AuKpZW66pmW2i2q45RMQD2AG7iHhEZMb2OH7XpQvJSXPy5YcP8Pvt1eyr7eC9ly1iYYEXl93G4WEuyyqlpoa/e/AkuYIMq19qYsJd775DLNvc1x3XLuf69aXYBF1yVinoLQkMjqNk4mRrF/PztOWamrkm6lTuM1h9Ej8B3Bz/+2cm6NhTLjvNyYevWMI/j7XwH7/fzaJCL69bV4rDbmNRUQYHNUOsVFL5uyP9yiXg9AS7gQFxZ29APLiGGMBmE775hrVs+fhmlpXo0shKOe02nHYhOI4McX1HiJLs6dO/W6nxmpAsrjHmTuDOiTjWdHHThfNpCfSwpDiTq1cW9y63urwkk+crZ3x3OaVmtMAQGeKiREDsHy4gHv7jzmaTCV2mWamZzuO0jzkg7o5EaQ30UJKl9cNq5pqxZQ2TzWm38bGrlw3avrQ4kwdfqsEXDJOdNnTGSSk1uYYqmSjMsL6MB2WIu62a4owRAmKlVH9pTvuYa4gb4wvilGS7R9lTqelLq9/HaXn8kqrWESuVPP7uyKAANyvNgctuGzZDnDVMyYRSarA0l33MNcT18QVxijVDrGYwDYjHaWk8INZOE0olT6A7OqiGWEQoyHDR3NnTb/tYSiaUUv2ljaNkot5nBcQl2RoQq5lLA+JxmpvtIdPtGHYBAKXU5LOWYh68OM5Qq9V1hqySieEm1SmlBrNqiGNj2rchniHWGmI1k2lAPE4iQnl+Oidau5I9FKVmraEm1UE8IB6my8RQ+yulhpbmtBMaa8mEL4TbYdN5NWpG04D4DMzPT+dkiwbESiVDNGYIhgeXTAAUZFir1fWVWMQj0SlGKTW6NNc4SiY6QpRkexCRSR6VUpNHvyHOwLy8dKrbgkRjg5eIVUpNLn/38Bnfwkw3Lf7ufu/NzlBYyyWUGqfx1BA3dIR0Qp2a8TQgPgPz87z0RGO9M2uVUlMnMEpAHDPQGjg9sa4tECY3XQNipcbD4xxflwmtH1YznQbEZ2B+vtXA/0RLIMkjUWr2SWSIhyqZKMwYvFpda1cPuV7X1AxOqRSR5rKNqQ+xMYaGjm7maIcJNcNpQHwGyuPrtZ/SiXVKTbnRSiag/2p1bYEe8tI1IFZqPMZaMtHWFaYnEtOSCTXjaUB8BuZke3DYhBM6sU6pKddbMjFEX+HEl3KD73Q5U1tXmFyvlkwoNR6JgNiYkefKaA9ilSo0ID4DDruN0tw0bb2mVBL4423UvK7BAXFRlpUhTtT3G2No6+ohVzPESo2Lx2XHGOiOjNyLuEFXqVMpQgPiM1Sel64lE0olQaJkYqiV59wOO/leF3XxrFVHKEI0ZsjTGmKlxiXNaS18M1odcUt8AmtBhr7H1MymAfEZmp+friUTSiXBSJPqwMpUJbJWbfEva80QKzU+iYB4tDpiXQlSpQoNiM9QeV46vmAYX1c42UNRalYJ9AbEg5duBqvGP1HX2NZlBcSaIVZqfNJc8YB4lNZriZUgh7pio9RMogHxGertNNGmWWKlppK/O4rLbsPtGDogLs729NYQJwLiHO1DrNS4eMaRIfY4bTjtGk6omU1/g89QaY4VEFe3BZM8EqVmF393eNjsMMCcLA+tgR66I1FaA9YVHM0QKzU+Y60h7gxFtFxCpQS9xnGGynLTAKgeIkP8l921PLKnnvL8dO64dvlUD02plBbojg5bPwxWhhigsaOb9niGWBfmUGp8TpdMjNxlwgqINZRQM5/+Fp+hnHQnXpedmvb+GeLnK1v4wK9e6u3h+IYNZSwqzEjSKJVKPZ2hyJCLciQklpCt84VoDfTgsAmZI+yvlBpsrJPqOkJhzRCrlKAlE2dIRCjLTR9UMvHPo83YBB79yKW4HDZ+9uzxJI1QqdQU6B45IE4sIVvfEaKtq4ecdBciMlXDUyoljL2GOEKWZohVCtCA+CyU5qYNCohfON7K6tJs5ud7uX5dKX/YUU1rvPWTUursBXoiQ65Sl5AomWjwhWgLhMnTVeqUGrdEyURo1C4TYS2ZUCnhrANiEXGLyE9F5ISIdIrISyJy3UQMbrory03rV0McCkd56VQ751fkAfBvlywgFI7x8J66ZA1RqZTjD0VGrCHOdDtId9mtkgldpU6pMzL2PsQRMt160qlmvonIEDuAU8BlQDbwWeB+EamYgGNPa2W5aXSGIviC1kz23dU+eiIxLliYD8DS4gyKMt1sq2pN5jCVGmQmn8j6uyNkDLFsc4KIUJJtLc7RFujRDhNKnYFxBcSaIVYp4KwDYmNMwBhzpzGmyhgTM8b8BTgObDj74U1vZblW67WaeNnEC5UtiMB5FbmA9cV8XkUe26rakjZGpYYxrU5k733+BHdvqRzTvv7ukUsmwJpYd7K1i7ausHaYUOoMuB1WeDDSwhzhaIxgOKqT6lRKmPAaYhEpBpYC+0bY51YR2SYi25qamiZ6CFNmYOu1rVWtLCvOJKfPJdoN83OpaQ9S59N+xWr6mG4nsr/bdoov//UA/zzaPOJ+oXCUrp4ouaMstHH5skL21Pho9nePuq9SajCbTfA4bSP2IfbrKnUqhUxoQCwiTuA+4H+NMQeH288Yc5cxZqMxZmNhYeFEDmFKleYkAuIgsZhh56l21s/P7bfPxni2eFtVGx2hMMaYCXv+mvbgqE3TlRqLkU5kp+IEtj2+BPp//H43/vjSzEM5vRSze8Tjve2iCublWe9PrSFW6swk2ocOR5dtVqlk1IBYRJ4SETPM7dk++9mAe4Ee4AOTOOZpI8/rIs1pp7otSGVzgM5QhHPn5fTbZ+WcLNJddu55ppKNX/wHP3zy6IQ8d7AnyjXf2cJXHxn2vEOpMRntRHYqTmDbunpYMSeLmvYgTx5sHHa/RMeW0TpHeJx2PnndCgCK432JlVLjk+a0j1gy0RGyTmS1ZEKlglEDYmPM5cYYGea2CUCsJp8/BYqBG4wx4Uke97QgIiws9LK3xseuU+0AgwJih93GufNy2FXtI2YMP9lS2TsJ72w8f7wFf3eEB3ZUa5ZYnbHpcCIbicboDEW4fFkhNoHDDZ3D7tsWX4p5LFnf61aX8JtbL+TqVcUTNlalZhOPa2wZYu1DrFLBRJVM/BhYAbzaGDOrimWvXFHMiyda+fv+BjLcjiFXpXv7xRXcsL6MX737QjpDEf73n1Vn/bxbDluXrjtCEf6+v2Fcjz3a6KcnMvJynCr1TZcT2Y74l2pxppuKAu+IAXFLoBtgTJ0jRIQLF+bjdtgnZqBKzTJpTvuICZdOzRCrFDIRfYjnA+8BzgXqRcQfv9101qObAV55zhyMgUf31bOmNBu7bfCKWNesKuFbb1rL+QvyuGplMfc8U0mzv7v3fmMMsdjg2mJjzLA1x1sON3HpkgJKc9K4f9upMY/3aKOfq7/zNO+5dxuRqAbFs9y0OJFtj9cF53pdLC3K5HCDf9h923pLJrQuWKnJpjXEajaZiLZrJ+LlEx5jTEaf230TMcDpbmlxJkuKrKzwueU5o+wNd1y7jGA4yhf/sh+wJsZd9Z0tnPflf/DR3+7k8QMN9ERidIbC3PLTrVzz31vYU+3rd4ya9iDHmgJctrSQ69eX8uzR5jGXYfxpZw0xA08eauJzfxq2EYhKcdPpRLYtPqEuO83J0uIMTrQEhs1KtXaFEbH2VUpNrjTXyDXEpzPEGhCrmU9/iyfAK9bM4buPH2Ft2egB8eKiTG67fDHfffwI6S4H/zrWTIu/h83Li3jiUCMPvFRDdpqTnHQnNW1Bcr0uXv+j5/j8a1Zxy4XzgdPlEi9bWkhlUwBj4FRrF9ml2SM+tzGGh3bWsmlxASvnZnHXlkquX1fKxvjKemr2MMacAAZfzkgCX9DK+uaku1hakknMwLEmP6vmDv59bgv0kJ3mxGHXVeeVmmwep51mf8+w95/OEOsJqpr5NCCeAG+9oJx6X4hNSwrGtP9tmxexq7qdh16qIc1l5xf/fj4b5ufSE4nxzJEmHt5dx/66Dn5yywY2zM/lo/fv4rMP7aWmLcgnrlvOAzuqmZ+fzpKijN5a4FOtXaweJSDeeaqdk61dfPDli3nlOXN48KUavvboQe5/z0VY5aRKTb1Ey7WcNCfprkzAmlg3VEDcqivPKTVlRq0h7o7gdthwOfQEVc18GhBPgOIsD197wzlj3t/tsPOLfzs/XiNsNUAHcDlsXLGimCtW9J8Vf/fbNvLpB/fwP08foyTLzYtVbXzmlSsQEeblWavlnYovDjKSP+2qxeWwcc3qEtJdDj50xRI++9BenjrUxOblReN4xUpNnERAnJvuIs1lx2GTYeuIWwM95GlfYaWmRJrTTlfP8H3BO0NhzQ6rlKGndUkkIr3B8EjsNuHzr17F3GwPd/55Px6njTdumAdYtZSZHgfVbaPPiXr6cBMXL8onK/4BduN588hOc/LYvvqzeyFKnYX2oFUXnOlx4HLYWFjo5cgwnSbaunp0KWaVMkQkT0QeFJGAiJwQkbeOsr9LRA6KSPVUjC/dbaere6Q+xBFtuaZShgbEM0Say86nX7kSgNedW0p2n+Vo5+Wmc6p15AxxbXuQyqYAmxafLutwxnsk74z3UFYqGdq7rLrgxMnh4qIMjjUFhtxXM8QqxfwQqwd4MXAT8GMRWTXC/v8BDL9yzQTLcDvw90SG7XbUGYrohDqVMjQgnkFesaaEb7zhHD569dJ+2+flpXFqlAzxs0ebAQbVOa+dl8Phhk4CIyyXq9Rkau8Kk9Ona0RpThq17cFBX8LGGNq6esjL0IBYzXwi4gVuAD5rjPEbY54F/gTcMsz+C4Cbgf+aqjF63Q6Mga5hOk1oyYRKJRoQzyAiwhs3zqMos/9StGW56VS3dQ17Fg/w3NFmCjLcLCvO7Ld93bwcYgb21PiGeaRSk6s9GCa7T9Z3bk4a3ZFY7zLNCZ3dEcJRoxlilSqWAlFjzOE+23YBw2WIvw98Chgx+yEit4rINhHZ1tTUdFYDzHBb2d/hEiadoQhZaZohVqlBA+IUMC83jVA4Nmx7nFjM8NzRZjYtzh/UTWJtfKnpXVo2oZLE19VDbp8SoDnZaQDUtof67ZdYlENriFWKyAAGZiJ8QObAHUXk9YDDGPPgaAc1xtxljNlojNlYWFh4dgOMB8SdwwTE/lCkdx+lZjoNiFNAWe7InSYONXTS7O/hksWD28LleV2U56VrHbFKmrYBJRNzc6wrILW+/omwRMY4XwNilRr8QNaAbVlAvxml8dKKrwMfnKJx9RotQ+zvjpDh1pIJlRo0IE4BidZrw3WaeG6Y+uEEPT+QdAAAFo1JREFUnVinkqm9q4ecASUTAHXtQwfEmiFWKeIw4BCRJX22rQUGLiG6BKgAnhGReuABYI6I1ItIxWQO0BsPiP2hwQFxLGasgFgn1akUoQFxCijLtQKI4TpNPHu0mUWF3t5L0QOtLs2izheivWv4FYmUmgzRmKEjFOm3FHO+14XLYaPW179kIhEQaw2xSgXGmABWcPsFEfGKyCXAa4F7B+y6F5iHtcz6ucC7gIb4309N5hgTGWL/EBnirviCHZlaMqFShAbEKcDrdpDndQ0ZEPdEYrxQ2dqv3dpAiZKLgTWbSk22jmBiUY7TAbGIMDfbQ+2ADHFbVyJDrJdoVcq4DUjDaqX2a+B9xph9InKpiPgBjDERY0x94ga0ArH4z8M3CZ4AiexvYIjFORJZY80Qq1Shv8kpYkGBl8rmwb1bd5xsIxiOsmnJ8JMrei9R+4KsnDuwpE2pyZMIcnMGZH3nZKdRNyBDXNMWJMPt0Ek8KmUYY1qB1w2x/RmsSXdDPeYpoGxyR2bxuu3A0CUT/u5wfB99P6rUoBniFLGo0EvlEIsZPHe0GbtNuGBh3rCPnZsdn8TUPvpqd0pNpPZ4hrjvQjNgnaQN/H082uRnUaF3UKcUpdTkyIxPmPMPsVpdZzxI1pIJlSo0IE4RCwszaPZ344sHGAlbj7eyujS7d7nmoRRkuHHaZVDNplKTLVG33rfLBFidJho6QkSisd5txxoDLCocMmmmlJoEHqcNmwzdZSIQD5K1ZEKlCg2IU0QiUKhs8vduM8awv66DNaUjl0HYbELJEDWbSk22RFlESXb/xWbmZKcRM9DY2Q1Yk3rqO0IsKtKAWKmpIiJ43Y4hJ9X1lky4NCBWqUED4hSxsNAL0K9sorotSGcowoo5o9cFz8lOo04n1akpVtcewm6TQasv9vYijp+kHWu0TvQ0Q6zU1MocJiDuLZnQDLFKEfqbnCLK89Jx2IRjfTLEB+o6AMYUEJfmpPFiVeukjW8m6InE+OojB9l+ohWv28HXbjiHOl+IP++qRQTeuGEea8qyh318JBrjSKOfOl+QixcV4HHap3D0M1OtL0hxphu7rX9dcGKiZ017kI3Q+3u9uMg71UNUalbzuh1DTqpLlFHoJFeVKvQ3OUU47TbK89P7ZYgP1HUiAstLBq0EOsicbA/1vhDRmBkUnKSS6rYu3A47hZlujjcH+Nmzx3nyUCOfeeVKjjR08rPnjnPxonz21vi47rvP4O+O4HXZ/397dx5ddXnncfz9vbnZ90A2AgkKgSAIiGjdaEGo2rpVcQWdOtNWj9baxdbac6ZqnbZ2OT3T1rq0dZwqbadOHRdcOnYclxZFRiiCBSEGBFmSkBCy3IQkJPeZP24SE7LdhEvuks/rnHuO+f3u75cnj883fO+T5/d96HSO5zdXsupzp/Od1Vvp8Pu57JQitlc3kZYYz5KZudy9egvbqgKbTN26ZDpfP39mmH/ayFdZ30phVv/62FMnpJIU72HTngYunV/EjhofXo9RMkEJschYSkvyDlx2rSshVpUJiRUayTFkWm5anxnirZUNTJ2QSkoQa7wKs5Lp8DtqfW3kZyQN+/5otPtgMxffv4aUBC8/v/YUbv7tBnxtHUxMS+S2P2wE4MK5hTywYgE7a3zc8eRmFpRk85Vlpeyvb+WSX6zhovvXkBDnoSAziW8/u4XUhDhaO/w8/PoOclIT+MHlJ/PSlioeW7uLT59cyBceX09ivIePl+byjfNn6h+Po1Q2HGZOUf9Z9wSvh/lTsnr+alFxwEfxhBTi47TKS2QspQ22ZKKtgwSvhwSvYlJig/51jiEn5qby+vaanlne9yqbmDPMA3XdirrWbO6rPxyTCfHug83ctGoDZoavrYOrfrmWzOR4XrhtETmpCVzx0JvU+tq4++KTgEDVjidvPqvn+ul5afz4inn8+KVt/OiKeZxaks3OGh8lE1Kpamjluc37Wb5gMgWZScwpyuSi+9dw2YNvkOj1UFaQw+Nrd7Hugzr+7bMLe5YDjHfOOSobWjlvdsGA50+bmsMDr1bga+tgR40qTIiEQ2qCl6oBKhD5WjtUck1iSkhGs5n9FlgKpAJVwI+cc4+E4t4SvGm5abR3+vmg1kd+RhIf1rVw5anB1W/v3ta5sr4Vikf3/Z1zbNnfyKSsZHJSI2N73YbDR7hp1Xre2lmH12M8esNpmMHdq7fwvc+czPSuqgXP3no2vraOfg939Xbh3EIunFvY83VpfmApSvGEFL64ZHrP8TlFmZxblsfr5TU8snIBi0pzeXX7Ab70+4389OVyfnTFvOP000aXuuZ22jr8FGYO3OenTc3B7+Av5TXsPtjMsln5Y9xCEUlL8g5Sdq1DJdckpoRqNN8HfM4512ZmZcBrZrbRObchRPeXIJx54gQAXtte07PjXDAP1EHf3epGau2Og7xRUcvL71WzraqJzOR47rroJJYHmYyHWqffce9zWzAzNu6pZ+v+Bu78VBkXzS3s2ab6ldsX97kmPSme9CFqNY/Uv149n72HWpg9KbAcYMnMPJ754lkUZGp2uFt3ybXCQfpkQUk2HoM7/2sznX7HRb0+jIjI2EhL9NI0YNm1DpVck5gSktHsnNvS+8uu1zRACfEYmpKTQllBOi+/V015dRMpCXGcMW1CUNdmJHlJTYhj3whrEdc1t7PykbcwM+ZMyuA7l8zm+c37uf2Pm2hp7+D6M6cGdZ9Ov+PFdyv5sK6FWxZPO6bdyB57cxePrd1NQpwHh+MXKxZw/iB/lj9eMpPjyUzuuzZ2et7wDzeOJ90l1bpLrB0tLdHLSZMy+Pu+Rm44a+qAa41F5PhKSwzMEDvn+vxebmrVDLHElpCNZjN7ELgBSAY2Ai+G6t4SvGWz8nno9R1s3tvARXMLgy6JY2ZMykpm36GRJcRv7TyI38GTN53BwqmB7aGvO6OEm1at5+7VW5ick8KSmXlD3qPigI9bf/+3ngoNmcnxXHdGyYja0e3Dgy38+KXtLJmZy8PXn0p7hz+kM78SOt0J8WAzxABLy/JpOHyE28+bMVbNEpFeUhO9+B20HvGTnPBRKcnm9g7yh1hiJhJtQvZ4qHPuFiAdWAQ8BbQN9l4zu9HM1pvZ+pqamlA1QYCls/Lo9Dta2ju5auGUEV07OTt5xDPEa3ccJDUhjnlTsnqOxXmMn11zCtPz0rj3ua10+t2g16/beZDLHniDmqY27r/2FBaVTuT7L77HV594h6t+uZam1iODXtubc443K2pZ/vCbeD3G9y47mURvnJLhCFbZ0EpCnIcJQ6w3/8qyUl69fbH+P4qESVpiIAluauv7u9jX2qGqORJThk2Izew1M3ODvNb0fq9zrtM5twaYDNw82D2dc79yzi10zi3Mzc099p9CesybnEVueiIn5qZyakn2iK4tGkVC/OaOWk47IadfOazURC9fXjqDD2qb+Z+tVYNef9+ftpGZEs/qL53DxfMm8cPlc/F6jJe3VvN/H9TxxNt7hm3Dc5v2c/YPXmHFI+tIT/Tyx5vPVCWHKLC/oZXCrCQ8Q9S9NjO8KrUmEjbdyyKa2zr7HPfpoTqJMcOOZufc4lHed9oorpNj5PEYD61cQHJC3IjX4RZlpVDfciTwiy6IT/7Vja3sqGnm6tMGnom+YE4BxTkpPPz6Ts6fXdCvPeXVTbyzp55/vnAWRV0J7KSsZP56x7kkJXi47pF1/Psbu7jhrKmDJkUHGlv51lPvUpyTwhfPnc6l84u0c1KUqKw/PGiFCRGJDN0Pzh29W12Tyq5JjDnmqRczyzOza8wszczizOx84FrglWNvnozGwqk5PdUNRqIou2u73CDXEb+18yAAZ544ccDzcR7jC4tO4J099fzkz+U413fpxBNv7yE+zrjslKI+xzNT4kn0xvH5RSeyr/4wX/vPTdz17N+pONDEmvdr+fxjb/Pu3gYAvv/ie7R3+Hlw5QJWfqxEyXAU2V3X0lP1Q0QiU/cscO/NOY50+mnr8Ov3rcSUUIxmR2B5xMMEEuzdwFecc8+G4N4yhiZ3J8T1LcwMYrvn18tryEyO7ynxNpBrTy9my/5GfvFqBQeaWvnh8rmYGYfbO3l64z6WzcpnQlrigNcum5VPWUE6f/p7JXEe43frPqTT7zALrF1eUJLNX9+v5UvnTmfqRG3pG03qmtupaWpjZr4qb4hEsu6kt3ct4mZt2ywx6JhHs3OuBvhECNoiYTY5K/gZ4iOdfv73vQMsnZVH3BBrQL1xHu67/GTy0hP5+SsVFGQkcdvSUm77w0YOtbTzj2efMOi1cR7jhdsWAVDf0s79r1SQlujlyoWTuWnVBjbvbeCbF5Tx+UWD30Mi0/auiiLBfPASkfDpTnp7zxA3dS2f0BpiiSUazdJjYloiCXEe9gaREK/dcZCGw0e4IIj6vmbGVz85g6rGVn7+SgUP/2Un7R1+vnPJbE4/IWfIa7uT7Qlpidxzyeye46tvPQe/cyTFxw12qUSw8molxCLRIH2AhLj7v7WGWGKJRrP08HiMSVlJ7A2i0sR/b6kiJSGOj88IrkqIWaAU2oLibLZVNVFWkM41p49yj2ggwavKA9FsW1UTWSnx5KUPvFxGRCJDRnKg5GHD4Y/KrmnJhMQijWbpY3J2yrBLJjr9jj9vqWJJWd6IZmjj4zzHlARL7Nhe1ciM/PRj2pFQRI6/pPg4slLiqeraah3o2cpZSyYklmiaTfooyhq+FvEL71ZS62vn4rmTxqhVEkucc5RX+yjTcgmRqFCQkURlr4S4uwSblkxILFFCLH0UZSdT09RG65HOAc93+h0/e7mcGflpnHdS/hi3TmLBvvrD+No6mKEKEyJRoTAziarGjyZKmjVDLDFICbH0UZwTqAu762DzgOef37yfHTXNfHnpjCF3GBMZTHeFCc0Qi0SHgszkPksmqhvbAMhKHnzbdZFoo4RY+pg7ObChx8YP6wc8/+gbu5iel8an5gxfXUJkIGsqaknwephVOHj9ahGJHIWZSdT62mnrCPzlsKLGx5ScZJITVOVHYocSYunjhImpZKfE87fdh/qdqzjQxKY99Vxz2hTNDsuo+P2OF9+tZPGMXD2hLhIlCrq2WD/QNTP8fnUT03PTwtkkkZBTQix9mBkLirP524f9E+InN+wjzmNcOr9ogCtFhrd+9yGqG9u4cG5huJsiIkEq7EqIKxta6fQ7dtY2U6pnACTGKCGWfhaUZLOjppn6lvaeY51+x9Mb97JkZi65qh0ro/TC5v0kej0sm6UHMkWixUcJ8WH21LXQ3uFnep5miCW2KCGWfk4pzgJg456P1hH/9OVyqhvbuHLhlHA1S6JcU+sRnt9cyZKZeVouIRJFCjKTAahqaOX9Az4ASpUQS4xRQiz9zJuchcdgY9c64ic37OX+Vyq4euEUlVqTUfvJn8upa2nn5sXTwt0UERmBtEQvaYleKhtaef9AoEqMZogl1ighln5SE73MnZzF8+9W0th6hO++sJXTT8jhu5fN0c5iMiobdtfx+NpdXPexEuZNyQp3c0RkhAoyk6hqaKWi2kdBRhLpSfHhbpJISCkhlgHd+PET2VnTzOd+8zb1LUe481NlxMdpuMQSM8sxs6fNrNnMdpvZilB/j7aOTlat3cW1v15HYWYyXz9vZqi/hYiMgcLMJCobA0smSvM1OyyxRwv5ZEAXzC6grCCdt3cd4pzpE1lQnB3uJknoPQC0A/nAfOAFM9vknNsy2huu31XH+wd81DW3s72qib++X8OhliMsKp3IT6+eT2aKZpVEolFBRhLrdu6n0zn+6eyp4W6OSMgpIZYBeTzGN86fyY2rNnDb0tJwN0dCzMxSgeXAHOecD1hjZquB64E7R3vf3761m2fe2Q9AUVYy55TmcuWpkzln+kTVrhaJYidNyuCZd/Zx1cLJ3LJ4eribIxJySohlUEtn5bPxrk+SobVisWgG0OmcK+91bBPwiWO56bc+PYs7LigjKyWelAT9ehGJFZ89cyrXnl5MUrx2p5PYpH+xZEhKhmNWGtBw1LEGoF+1fTO7EbgRoLi4eMib5mckhah5IhJJPB4jyaNkWGKXnpISGZ98QMZRxzKApqPf6Jz7lXNuoXNuYW5u7pg0TkREZCwpIRYZn8oBr5n1XiA+Dxj1A3UiIiLRSgmxyDjknGsGngLuNbNUMzsbuBRYFd6WiYiIjD0lxCLj1y1AMnAA+A/g5mMpuSYiIhKt9FCdyDjlnKsDPhPudoiIiISbOefC2wCzGmD3MG+bCNSOQXNiifps5CKlz0qccxH59Jri9bhRn41cpPSZ4nV8Ur+NXCT02aDxGvaEOBhmtt45tzDc7Ygm6rORU5+Fhvpx5NRnI6c+Cw314+io30Yu0vtMa4hFREREZFxTQiwiIiIi41q0JMS/CncDopD6bOTUZ6Ghfhw59dnIqc9CQ/04Ouq3kYvoPouKNcQiIiIiIsdLtMwQi4iIiIgcF0qIRURERGRci+iE2MxyzOxpM2s2s91mtiLcbYo0ZvaambWama/rtb3XuRVd/dZsZs+YWU442xouZnarma03szYz+81R55aa2TYzazGzV82spNe5RDN71MwazazKzL425o2PIorX4Chmh6Z4HRuK1+AoXocXKzEb0Qkx8ADQDuQDK4GHzGx2eJsUkW51zqV1vWYCdPXTL4HrCfRfC/BgGNsYTvuB7wKP9j5oZhOBp4BvAznAeuCJXm+5BygFSoAlwB1mdsEYtDdaKV6Dp5gdnOJ1bCheg6d4HVpMxGzEJsRmlgosB77tnPM559YAqwkMPhneSuA559xfnHM+AgPycjNLD3O7xpxz7inn3DPAwaNOXQ5scc790TnXSiA455lZWdf5fwD+xTl3yDn3HvBr4IYxanZUUbyGhGIWxetYULyGhOK1S6zEbMQmxMAMoNM5V97r2CZAn2D7u8/Mas3sDTNb3HVsNoH+AsA5t4PAbMCMMLQvUh3dR83ADmC2mWUDk3qfR+NvKIrXkVHMjpziNXQUryOjeB2dqIpZb7i+cRDSgIajjjUA4+7T1zC+CWwlEIjXAM+Z2XzUf8FIA2qOOtbdR2m9vj76nPSn8RY8xezoKF5DR2MteIrX0YuqmI3kGWIfkHHUsQygKQxtiVjOuXXOuSbnXJtz7jHgDeDTqP+CMVQf+Xp9ffQ56U/jLUiK2VFTvIaOxlqQFK/HJKpiNpIT4nLAa2alvY7NA7aEqT3RwgFGoJ/mdR80sxOBRAL9KgFH91EqMI3AmqdDQGXv82j8DUXxOnqK2eAoXkNH8Tp6itfgRVXMRmxC3LXW5CngXjNLNbOzgUuBVeFtWeQwsywzO9/MkszMa2YrgY8DLwG/Ay42s0Vdg/Be4Cnn3Lj79NrVN0lAHBDX3V/A08AcM1vedf4uYLNzblvXpY8D/2xm2V0PAXwB+E0YfoSIp3gNjmJ2eIrX40/xGhzFa3BiJmadcxH7IlCm4xmgGfgQWBHuNkXSC8gF3ibwJ4Z64C3gk73Or+jqt2bgWSAn3G0OUz/dQ+BTfe/XPV3nlgHbgMPAa8DUXtclEigj0whUA18L988SyS/Fa1B9pJgdvo8Ur2PTz4rX4ftI8RpcP8VEzFpXo0RERERExqWIXTIhIiIiIjIWlBCLiIiIyLimhFhERERExjUlxCIiIiIyrikhFhEREZFxTQmxiIiIiIxrSohFREREZFxTQiwiIiIi45oSYhEREREZ1/4fwTGrw9LcOLMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x216 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.activation_stats.plot_layer_stats(-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The % of non-zero weights is getting much better, although it's still quite high.\n",
    "\n",
    "We can see even more about what's going on in our training using `color_dim`, passing it a layer index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAADNCAYAAAC8XqoPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2dzY5kSZqWzf8iIzIrc6qqK5lWA6JpDaAZaYbZgAQIiQ0bJJbcAkskuBtug9sYwYLesGAxQgIBhdQz9ZcZ4X8sanok/77Hw9/0yO7OMD3Pzk/asWPHjp0Tlu7PeW1xPB6HiIiIyMwsf9cNEBEREflN44RHREREpscJj4iIiEyPEx4RERGZHic8IiIiMj1OeERERGR61o/9479c/ptn8876//u3/6RtW+xPP3/1X79rZY5/9surj7n+xc9PPv/w99+2Ml//w00/5qrXVdu6+baXufm2X44vf/mXvf7N6QG++cWrVubdV32ue6DRAFPizTen7VjAKFk9wEbYtLtdXDwecoBjbk8/b/tpj9X7vm0BdY1F37TclfPe9zJE3W+MMbYvT0/05vveiPs3vTOO0K7t69ONy4dehsbc+ofeLrqW6/enG6kN1F8raMe7L08LHm76jlg/cKzjFdpO13b9LnyslWLLXS9C2+g+qtuOMM6p7w8r6Ix6y2z7jodN32+xz673YV32pe4Kx8CyPAd2d1Sob1q97wc9Lk/3pWcMjR0a+/Qsvf/8tCGre+jXR/9inme5hY1hv27Kfbp9BYVgnNN5E3UsLuG5RudNfVif56v7vt/+BbQhaCvVdeh/Xsd/+Y//4ewTxG94REREZHqc8IiIiMj0OOERERGR6XHCIyIiItNzpYL16bF72T2lzfenUtVh0+d3oR+JHG9vTj4/vIb6SQqkaWYpd7jpRZYgHW6/uG3b1t+dGnIoUYLslQp5tW0krxH7F723X/3fU0Pu+5+CvUZ9CP1Tz4nkXTrvKgWOAeLmGOO4KMItyHckMqOkXg75/vdAUCaRj6TG7y+LoSS2Nul3jDGg/VVEfPW/u6n7/sve2C3ckxWSio9wjXDf2lY6HI0dEnpBLN/8cPoZJXgQWweMnXqeNL7o/luAFN33ywTlKv2OEQrioQhMInbdGSVvuN57kNnrf8+PIHRXSXoM7p/7N5f7jNpA503HXJeXIw70DKCxD2OgSsoovNOLFwCJv7uXp5/xuQntevgM+r9c390dtAHq3wfjifqQ/qY8ht/wiIiIyPQ44REREZHpccIjIiIi0+OER0RERKZnGmmZqLLl/q6f7lM64HhzujdKbjClTGRUkl+3IKNWQZkgSZOSP1ORuQpyKBiG/PDVaWdgejEl6AZTdRLaViTvksxJYuD+chkS65Z7EiRPP5M4TdIvyflVgOa02UzAHJRK+93pAe4/74UoETgRNY8kW2L6L5RrlfdNmDpNwxXGwO7laUOqxDzGGFsQNxMRmMbhgdx/CiwvEi6mmqPE2svta9L5GGNRGoJlSKYO3l2gZ0yakF3PiZ+t2b1M6ef1OYASNrSVkolrmjCJ05iQjcL7aTkSv+mrC3qWUspxhZKcqb+wHa0RfdOuv2eD160moqMwDn34GH7DIyIiItPjhEdERESmxwmPiIiITM+zdHhWb/uq5PTbcPU87r/4uA7P8pvTH/XX96+j/ej36fp76x4Cm26+6dvuf9J/EK2/+ZJTsIMQNfytm1bOLb+30srGFJKYBJ+Rq7EHd4m8hersrGBVbOr7PYU8wu/TtRyVodA8dFmSEDXymWih5NJn9Ns9ulhQbkXhliWYbHV/2SMag/tnV/pnT7/n03/DaFu9luS7hOGNGMxYyu3DQESqvwcPQhkKfSRvr4xrdi4ebeFfQ8GJ9T5KV/rGIL1gFWz0CeGebOGscD3ovDH8FbzGVgauUQ35PFeuhpQS6EbRPV+gMYEru4PDWP82jNG9oTq+xuC+Jv+ueZOhYkPtam0I7+XH8BseERERmR4nPCIiIjI9TnhERERkepzwiIiIyPQ8S2k5pQpgFI72FOpq6fdvYMVrCqcjYbEIWet3vQwJnqv7brnVlZ8XYNKSyJwKvZsqmJEUTcIcyoOnn0lgJEGZQvnWIE9XqA9xBWcMEKyf+/EoGBBl2iJwo3RIIj6FidVrBOOLAvgICpGsQm8ayJaMMZQ0ScwOAjxRnIbVoTGwDoT6FsAHsigG0QVhaHSvJSLtGDDGqL8wpA3qCs4pEY/H4Odau7dCsZyk5TqG6XwoWA9fEgm2ofxP1ygIjKR2sQANL3aUbenK6ATdI7X/6TrSeePfIxCekzbQedNK660ueAY/ht/wiIiIyPQ44REREZHpccIjIiIi0+OER0RERKbnWUrLx7/Zk5ZJCqtyIsqWT+HrX518XBx+0tsAQt5xSdbWqbRFwmpd/X2MMe6/6AUjyRQgQTlZkXh3RyZc30SSYUugpURPcjlhW0uIRUOvb0qSkMcYY1/uFl7ZuO9H1y0RrCl9GVcND1xXGoco+YJwW88JJW8YOyh9lnYc4XxIVkwSaGmc0HhavYf6Iem1yaKwYjQKpOS11gBakKRx7ONK34/XTWXG4PR2bH+Vlmkl7lAEbi8lULvCBOsmqQdjYowzz79grGACdPAyw48HLYcLJfXkWUf3KKU2E8dAsMYV56H69Q/QjpvLgjW9gEDUl1XohYoPjVr2Gx4RERGZHic8IiIiMj1OeERERGR6nPCIiIjI9DxLaXn5zQ9t22Hzedu2KsmcT0mojMBEXShHInAxSCkV+trZKaaphv4XSmdl1KxRAoX64QS2r04PuvkO0otfkQUKm6qkHiZrU5IzyYmtDImCaZJsbRsdj6qnlO7S1ySfYzgyyJbHBQiLReZEwZckb5IygycOyqJwTq2vqe9hHGJb6R6psjZckAUIpElicvqCAF24QxOgoS7o51T+r+OJX7yAdoF8XIXeNAkZXy6o4zCUtekcMaX55vHPY/S/KWOMsQvS25Pn6Llyta9J3qVrhH0RPJ/onqH7b18H4oAxQH0PfUjtry+h4IsXcB0fw294REREZHqc8IiIiMj0OOERERGR6XmWDs/x2+/7Ngjzq8FwD28+7mrpu7/3s9KGXua4ynyH9tsqNDV1TepvpOlvxbgSN/zmXqfJabAe1V+31d9txzjjENDIrecULqRLAYLU/zffnFa4IzeH3InAzcBVf+k3eOif5Hfz1AeCn+XbNYqcpDHYVWs+Qi+D7Q/cA3QDyDVJwxvrNlh5nXZMVypv+4EDQ+GQbZxTCCf5J9B+XLm6toPagM+nvu1jhgVWB4nCQfFZl5zjGD3gD9q6e9m3Ub/WdlAb8HlLAZ7VqYLzppXX0fWhsNSyK65UT14dOVvlmUXXYwkBnsfEB0od2UfwGx4RERGZHic8IiIiMj1OeERERGR6nPCIiIjI9DxLaXm8/aJtQpGyBlWFwlzK7rPTA6DgC0QrIGPYF4ldIKaVMLT1u257vX8JdZFAStRANhTaQI4DsbyKs2lQHK7EXUczBetR34Nsiauelz5LV2HG8LgqNUKKGl0PlHDrWKEy0KwV9Q+Fe5VyJKmnK68vi5x4fAVtIKE0EGep7Xg9gtWtxxhjWWRUklhRyoT7ocquFApH0i+uoF6D4uiawQWn59PVojGJwMHLESiMh7QxRhIr1Q99nYyV9Lyvhl4QuLJ+et7i85xehKhtgDKpyLwvIYzrnhGMgbA0zmugI8rh9Kx4BL/hERERkelxwiMiIiLT44RHREREpscJj4iIiEzPs5SWH37/s7YNkznLNhKcVm/ftm37r7+Oyj0UiRgTPamHg3JLWtE5lKK3L0933n6WpRcf1mQBQjtuL583niMkfzablqRikqJJwKxSG6ULB6ugjzHGDmTaVVkVPkrMPrctWPEaIbm2ppuG0m+S6krbaEV7SsimpNcqNeLq0GGiaiL649ihciQk15ceKC03XFm6ysFbGF8ErnjdGtE30croUV0DROMgUfdsO6pgHa4IT5Z9krSMidZBajO2LUyYpvF0se7B9x+2v0rkJBVjonFWLiL921auG60kn1Lvtyhd/wJ+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZnmcpLW8/I0OrbzqWZN/tK4pdzVi87pbhYX1a3+6OTEFKF+7bFodqsfaqMHkXDlll0VTUXeyz/mkyOFwOSnemNNDd3elnlCFDObgJknQ5KK2T5F1KIS71YXIt1J8Ii1WIHiOXj2v/J2nJY7AEuHwH5cp5P7yBpOVQHqwiYipOcx+eVra6D+9vKpaMJ+hXFE9JFg0EaEzRpvTism8izY5x7kUFKFdTwMOEaTxm8l9qksFJGA7GGJ4PJbWHY6yVoXTk4PlEx+OEeihX5Xw4xz29kAMp2kk6NbULXxoI2p+mgBP1WYoCdDj2f43f8IiIiMj0OOERERGR6XHCIyIiItPjhEdERESm51lKy5TqSinBVQQmIYxk5AFJy8dvv2/bljsy68p+NKXERNJj+QzpyLDf7rZvrKmxJPIdbkCmTpODywmQYFjTmM+1IxHYUJAM/FSUNINE7jHOSL5l/KCcSmI51L8uknJNIB6DE1WpXGvDJouIxVRXkjLrNaLqw/86tTFMEja0PxFK93d9v9W7TGRG+Xh/uQwmU8PYaYL75UfHj8UC0ZjGCdYVjv0KpklDuzDpugq38fMQygV9Rn2P5YJEdLzXaLwG9afyLl7vC3WPMcaK7mUAX4SoZULxO0qQp+cJHTN40SK91x7Db3hERERkepzwiIiIyPQ44REREZHpeZYOTw38G4PD/I4PJYCPgr3evIyOSSuoP7z+xWkZCKLDVXmD36draOIYY4wFeDHkJdVd6XdUCBkkrwddluXjn8+ROARp6FWyMj3WRb9P0zUKfuumvkEnCeqvq2UnYXVjnBnD6OzUHbO6aAxXsK00poOxj35I4hGN7kuRy0LXO3Ve6jnRedO9fKQxENyTMaX+NFj02lXV8dqGq563ILowBBAJ/C/sC1rhHIrVex7HSbiyezJ2rl25nK4ZuYN03hSMWtuahF2OccYHSq4R/R2A61aDBuNx8gh+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZnmcpLSMkDxaR+bgE2XmTGX+Lf/THbdv6/Wn9GNhEIWqry5IpBQ+SJEYhjItDNaD7fhjuRiIl9mtpA4W2hfJxbxfsR20gsbWGnNF+6YrUJCTXVaopiC6Uj2v/YN8TyX9RUGqEUL73vSCt2l6vdx1eY5wJnwyk6/1tlsCXBJ9haNuRUtr6puUOghnLatOpnE/3Q1t5PQy7RLm57kuBbOFq4Mmq7alkSi9a1H7llw1oQPVNbQVy2o2Md6qK7vl6T9Jfx+xdktY/FHiK1y14ESIdh/gsSl7aCPo+LYdjJ3zWXR1K+wh+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZnmcpLb/7qs/TjqtuaB1rIjPIUvc/6UviUmjl8vseUflQV1qPVxsPyqViF8miRWRGmZNSMklgJCGvnie1NUzL3d+VMqnsHEiAKFamqxaTcFv7hwRMWmGZLtL28gXG60bCeyCZYlIxnWPwRFhCWmsqybZzIsH6CStL90J90+qBRH/Yta5KDkJpmkpbBeg67n88IGzC+68UpH6GHfH+C1K6eWVxkLxhTLeXI0IhlpLg68seqUebklzvNDm4rZZOz8Pk2XpmWytD922YrL2qYxOeYZgeTocMytF9Sy+5YHr0E/EbHhEREZkeJzwiIiIyPU54REREZHqc8IiIiMj0PEtpmaTGASmfVXJLU11Xb9/2um56V+1qyjEJbZAiSttqQuhTUoJbim8WJo3TX0pBrfVT2nNsFFbJLRSgua+hXLDfHlKIMbG1jDEUKyk1FqhCHgmSqazdknGTdN4x8J5Z3pNdeeHzOCPvkigdCNbpOKwnisJkIv2eaUeVcGPhHepqIigJxDAOE/mfZOHdHYxpkqkDeZ5S2SluO3lJAOVUFG6za9TK4PXI7u9WDqT+RModo/crCuMvIP0chPrW/ySkh88PvCevlfOpXOkzut4EJkCX9uM5pn/bft2eDysuIiIi8vxwwiMiIiLT44RHREREpud5OjwvYCMEsrWgKjjb3V2f8y1qoOAY44ef9W31Z2Z0hGhKiavr1pXds9+wW7jiGGNUhydcJZlC7ZLAulXqn9B5Vzcj+W19nPEpysrMtF8aiIi/+9emkttAm+h35iDEkMcJbKsqWbgyM60avgf3Y/Uu8MuorRRsWMd5mhSH5WqoHfgP4ROO/IDq1CzB2aIVqdHRq/uG9yR5YvU+3b0CF+uhbcLOZjeqfKTxRGM6cKgw8BSInh/0DAuCFM/t3DzK8BmceIHoT2G4IlRfh04YwknH5DDQy/dkXfX+xwNcrmuE+9E4bOGNH2G24jc8IiIiMj1OeERERGR6nPCIiIjI9DjhERERken55KVlCgHEEKREWgZZ6v3nfc73+rYv03pc9p23n51uI2GuSVxjYOBblXVRXoPzxuDBFu6WyXcHCuAD6awKkVT/4YZktctCIYYApvLd5rIomIZqYfhdLUJ1kTwI1VdJb4nCLVw3qCtZIjoV1xcQPFhl/NV7ClykdgVCLIbVheJp2fcIYXi8ynq2knhra7jidbTSd/hfTZQ5S1upXXtaaRoD+KBcct6pZF+fRfRiBImtRHXUw+cC3acc4FlPPKs/ake68joFvZYwQhTNaTxRv9LfnqCuNLS3jif8mxgGo9YQSQyLTJ8V5w8jIiIiMhdOeERERGR6nPCIiIjI9DjhERERken55KVlglYCRhmrJS33MtvPsjnf7mUvt6+JzyjyhW2tVZGwCrIlJZfu72obehlOCc5W/yaBu0Kr5FL7acXgviNsCzzHujL3GGMs0pRgkOHqCsgoO8eCda08TNamhNtAtkShFE6c5MRlSROmMZfKg3VfTvzu23g16CKp40lCI8Jr1P47uAM5H5LOk5TuKH37TLtS4bm1AVKhE9E4Hk9QLEnSxrGT7EhF6FlHxdIxUPdLX1So8i69aIOJ3NCu4N7C7gqT85tYHqbRI8F51+fJGBiwD2ne8Iz8wBmM3/CIiIjI9DjhERERkelxwiMiIiLT44RHREREpudZSst7kJaX625V7ben8zkSqEhU+/YffNG2be8uS3SUVDwoFRMFrdL+h94wlCEp1bVsIrELRbhQAqyJxktI3kX5ddvLNXmQ/DwSNyMZErZRuSDNdowuemPqKlwPTLgtHYvSXkgkW6bSYSARL6EQiZuREEsCfxrjW+tKzzEd53Ubpp9fTpMeY7Sk3TS1GeXmuh+dNyWFxzJ4LZQlfuPYb7Y2FLkyQRdFYDifBb2MQfduTbBOhXG6bqVt6f1H17ueJ94emNoMddFtVF4wwTGH5ndwTHqehMn2re4nPCM/4DAiIiIizxsnPCIiIjI9TnhERERkepzwiIiIyPQ8S2mZpKoVmFyHKr6BYHi46XUtjr3c+n3ftn9RLKpQaFskScsg7aVJxauH03Io2mHiMCXEklB4+hnTNCFpmdpaBegFiM2JuPnjzkHabyi+oVhXq6LUUkywhk2lrdT0xQ76guqvZTC1GYRxkqnhmA1ISsXkWtq3FEwFxui88Z6B6h9oXyi3vXwfLRPpd2SJsCx+X365IEn6PUdL/KYy4T3DicmnH0lsPpD4HZCkBv9YMK2/XO8o/ffM2K8vjoTXFu+/6n3TOYaeP6Vt17+BT5GDa9tQig8T6pvv/hG+nvEbHhEREZkeJzwiIiIyPU54REREZHo+eYfnu3/2d9u2IwT8LWtw3xhjUcIIm9MzxtiDw/MAK6jff05BgOUzrU4LgYgUkngoIYm00i06CrBa86GuWAv9hSGGaQDY7rLbQCtqJ/Aq8VAOVwK+/ONz/Ps0eVal/niV58SVocOF1639Vk+/59PYCYPVlsUJ27+AJmDA2HVuRupZRfvR6tngA6FbUgMXw5WysR31kBR8Fw6nxC9DfyN0XloAH67qnS2X3u5dCjEM+6KG39FzgVcbh7qAdj8/IayzuSxUFz5vL7cLryP1BfhABwiETcJfsX5o/7VjJ7kneYX7D3vG+A2PiIiITI8THhEREZkeJzwiIiIyPU54REREZHo+eWmZIFl0CfJS3XbYgCwM0vIOVkbHldbrNpLvQqlqWeSu/Q72C1cNbwF/YSgVSrjBNpQCr13hPAiYO19/cN6hVIz1gxhfSTPOarYlSoHhyvEtmIxkYZJYk9WtR79HsF0gixLt+qI4nbW/75i14erVuTHIDeqi/q87Yzhk3y0S46kMNBZlZ5SIiyyKJw7NCsYdPgPovFG8T9oATcAw06RfoS7aLZCWWd7t+6EoXfsnlanD+yh6kYP2O1yunwRrehkD6y/tP4ay9mP4DY+IiIhMjxMeERERmR4nPCIiIjI9TnhERERkej55afnbn/UmLm7v27b1uhtgu12Nu+z10wre+00m6dV9j5CgTO0ijofSVpQaL682PsboicAkiYHAjVYmprPWVeh7mXjV8yoPYspnmKpcN6EQmxp/QCJdk8GICcAlPZXk10DkQ0geDVczX+BAv3xITIgNpMlEFj5LrZ8kbIDumcXD5X1RKoZ7C8XWep54beGgtK3KouE9w3XBtnICeN/iCwFUV/kcjhMUy5M3AugywrhAiThJHE6kYiJM0aZ7t/bFAf5m4SHDlzb6quSp/X85aRkFZXp2X5sgH6Trnxzmg0qLiIiIPEOc8IiIiMj0OOERERGR6XHCIyIiItPzyUvLh03ftiA5GGTUzebUttxBUi7Ja/vbLkLtX0Dbau+RcEZyIvmEpW3LNSSlLrrxR+mpi32RDq9Nrh0jSvalutJU11Z1mBJMgviHCmx/DaauQrlaP54PNTYoRv0cptK2dtD1rgL/mfpRIK1SI0rYUFcaO33pgCmpAE1CbJrw3XaEuqBY60Nqa3o9ykMlln6Dusbo121B5xiK8f2AsA0TdIP60/RwaisJ6FUGx1smFONLX+P1uMleoKgvQuALG0QivA+Q5dPHGsYol7FJL5zASxA4Nuu1xOfhh923fsMjIiIi0+OER0RERKbHCY+IiIhMzyfv8Nx/2betIMxvDX7Otrgs6xsIJ4RgpN2rfkwK36qhSgsI81uRNxQsT7ugqSj9Fr2F32QTtwgcmCMln+FK4uV3WtgPVwJOXJxo6d4z1LZSVbgCeegj1HKYDBj6UokXQ6D7EZx3/Ft35n5cTQvgS4MaYVsdwul+4SrVx7pKPPkhO6gLqI4Q+k3hdWtDDEPt6L6FupKxCc8w3JHqqtebtJVr/wqFuhzeM3QtAwcM+wvDG8t+qfOEz6LLzlZM4ArSOOfx1De1cvjMD55h4/pV3B/Db3hERERkepzwiIiIyPQ44REREZHpccIjIiIi0/PJS8sUXHQD8vFmBULy6nQ+t1t383gLgWO0gjquZFxEqyWIwEsUzC6LiEsIUjyAFH2EVbZb9WEg2xL6lQTrYw2+SmVIogXwZbJlFHJGkDAHUGBkawLJimm7ajkKE6NwxaBdC7jeuHJyKjdTaGGr63oJt+0Wjtdjsy3TVZ5p2+X7CMVNWg06PWYlDcBs8nx4vFTUredJY5NEZipXq0//i52IuVQkvL8HBU3W5lN/wXMZzzvpw/QZWcMCUZwOVyDH8VSKhIIyknQ//kkM2v8Rvp7xGx4RERGZHic8IiIiMj1OeERERGR6nPCIiIjI9Hzy0vLuZTecbmlldNj2UFesRekQ5OAbWJUcJLfj3em+a0iAJvmYROAqN+9SeZckvSLbUaryksRWcmTpmGVfFGLJTEMxN5FYoQ/3tJRxUBcKq5BEjQm3pV/pANRfwerGR4prTVZGH2euUS1D4xclU9hWZfmnBC9Xz5jk17SqdHX0AiadJ8nE6SrVRLIrXcckLZz2e8Kq4Y30GgXjFd1wOu0gcfgp0NivY/GIjQ3F79plwXN6jJGNAepn6sS0DxOx/Nq6UqGe/4ic7gZ9iM+wR/AbHhEREZkeJzwiIiIyPU54REREZHqc8IiIiMj0fPLS8uFlF+ZebHZ926pv265P53P7my443b/Y9GPe9G6hJe0XJZmYUpXXlMwJ7A6nbSUBer/rMvWehLlA1iaZmqa/JHgeigR9PPR2oawGbW1CdeihLje9f1oTQsmRBG6UWGuXwfVmzzE4qSTaeZwRdeuma1Oux2B5sPQjioJXytTUBny5INiXuvAAYigmikOadOsKlEUfad9JQ0o74EUC7K9EWk5E8w+h1pdcxzH6/TEyKR3vGZRRy3MtbVd60HracI1SSba+JID7wWMzSRw+wphepF9dBPcWSuThyx513/QaoUT+EVPZf43f8IiIiMj0OOERERGR6XHCIyIiItPzyTk861/8/OTz4q67ObdrcHhg2/3+9PRqEOEYY6zgN+btLf0YDfsWr4fcojWF5qFbcvm37iWsCH9YXw5/WoHvEioj+IPuosyTyadhz6Nvqv7MU34/joqQz4QrfV8OaTse+v8XyD+hALPkl2f8XZ5cmaANhyQkjCqjY6JDEPonAVe7GRToSSvO065wzDouMOyS6qKuSMIbQ5+prhIfhzdiwF9YsJagQ5KjV+/vcOxEwYN0j4ahktG9lQZUUlW1qRhkSnteDtdbpn5WehvVtqZ1Xds9tN+VIaIfit/wiIiIyPQ44REREZHpccIjIiIi0+OER0RERKbnk5OWv/nT3z/5fHP3QytDgvLNkuTm7cnnLUimmxsILHzRu4XE2bovCcorCPjbQzvWgbS1QTn48pyVAhFJ1j4EdY0xxmJxeVXhtK4qYtN+6NEGQXfBArx/VRe0C/rsUOTBNYydHQTYUV21bUf4v8eCwiGvFClJzKZjJpDgSyTOMtWEoZjAYV+S2wKh+xwoNzdJNmvXEoP0aqFeJpXzl4GNmorrKOwHpCJ+q59WDU8JxkX6IgHWVQM2of70ZY9679JzjZ7B2P66ijtcR3wRJh1PZRu+4ADgM6Xs+zHHIZ1jHFL66+N8UGkRERGRZ4gTHhEREZkeJzwiIiIyPU54REREZHo+PWn5b5+KiC9vH1qZuyIjjzHGLayWviureG/XfXnaG1iVfHvb6xLrHkQAAAiESURBVCJuSrLyDcjUBCUmb4uASe0i2XkP5SokO5PsRRLdHtJlF8XcO+DKw9lK4lXoXS6zVOgkkTkV2lCAxtTbyzIfiuWBBLi/fBnHGGMsg1Rd6hsSp3cwXJO+jhdjD5KDUxGRyq3K2E9XcScWV8rgqUDaEsWhrlggLuVSkXYN0jW29cqk6zglvVBfBvhxv0TMhjbA84PAcZdI0aHQW1nDcxr7HsrV/qHnNHFtW5eUrh+mgNe20n4rGP3JPb+Io6PP4zc8IiIiMj1OeERERGR6nPCIiIjI9DjhERERken55KTldz89FZN+dnvfyrzZvG/bXkDS8sPqVATe7PvpvnzRpegdyMEkgL0o0vItSMskGhO19j0cLxWZqwC2BhkvFUNHIMiR2LyEuFmqf1P6kCXpfsxEzE2lZZQHcd/TtqWJwMkxPzQx9LQdJSkVJFA6x5sbEikvHy9N0U5I+zBKzQ4Srcc4J2BeFnqpX5ehCJyQJHKP0aXV9Hhp/ZX0/kuOSX246u+SXC1Ap+26Vtam+hPxnq9RJu/W/kmvN43phKeMpy4tX/8MTtr/ofea3/CIiIjI9DjhERERkelxwiMiIiLT44RHREREpueTk5a3f+M0RfnNiy4ov1p3kXkFctT2eDqf260hqfgI20DKpCXtX92cCs+UZFoTVs/VVaELEyexFlYghlIbUABDibjIwXBMaimm5da6whRRIumfJ4nMkJCd1E/jqV4TTKu+FvhvDAmAdEQaF/WcWH7N0nKT/k9FRBJgr4W7v0rw16dCJ/ulabad65Jrx+BrWUulT53kObCC5+G14xCfJyTShsJwfbmDXhxJSfa8tl+Tvx9jPO1FiEp+f3+8lzGuHYeP4Tc8IiIiMj1OeERERGR6nPCIiIjI9PxOHZ7ln/5R2/b6J9+ffH774rtW5vPNu7bt/tBP5W5VgsPA16FtFDxIfsirTQ8trDzse6rWkn7hLYdMfZ1FsEI71UW/TyeuyRj9N+Q1eDd5ONZ1Tse1vw3T+VD9seMUtIu8gt8k+zB4kK5bWl9Sf+LwpKuNE9V7Sr2x9N6q5fYQpnmtf0KkxkhtV+p00Ni/1kGiY67Bcbs2pDJpa7xSPZwjBV7WfdfpMxhXDb8cUpreM7Uc5DReXReRtpWea/VZkfTNOZLr/aF/B/yGR0RERKbHCY+IiIhMjxMeERERmR4nPCIiIjI9v1Np+X/8q8/btr/ze39+8vkrkJZfr3oY4Ri3bUsV6w6rbH5HKyATN0XSS+VBYlHkZlrhfAXtouDECrVrA+W2IFiT3PybHDS/Xb33rwDxjc6xlkpWqj9HFSlp5KTBZ13mo3bBeArlxJt1CUl8imhcytG9toWwS64sKBcKq0R1lMk9R5kzqPtjhsKl/2sF5/qjtoM57WsSXfmlgV5Tbf8KJOn0xYtE6E2f5yRFH6IXFdJnxeXxmgaeXk923W7Wl/8mrpaXA1zHyOR8uraP1vlBpUVERESeIU54REREZHqc8IiIiMj0OOERERGR6fmtScurt2/btu2fdCH5D15/ffL5b938qpXZg+KZyLskl92telry3eqmbcNk0SJfvdt3FZhSm3dHSF8ubbtZ9gTl9QKSoqEvmqwdpiofwGpcJqtnkxgaypBdYu37pcnXta6PmfZM9VVpPd0vPeY6TIBuZWIpN7tuSVtTQfJasZ/2qzJnmix7rVieyMhjnGvr5WOm+yX3DPGhgudj7SJQ4C7Ce3I+5+qq0HWktGdiRWOlbEteXBgjWyX++tdZroek7rT/K9eu7P4U6jE/xmTFb3hERERkepzwiIiIyPQ44REREZHpccIjIiIi0/Nbk5b/+7//g7btn//8l23bH738Xyefv1x3sfkv9q/atu2qK4U1mXiz6FIxpRe/WfckZxK77g+Xu2936O3aHfsxd0VIZiEWJDTQyarI/AAJyg+L3nZM6wQZvPYZpUKnIiUJyZeOd67+SL57Qhr2tZBYXq8bCaUkAu8COT9NgKY+XF+Z6noE4Z3Ou0Lp3tQuSp2+VoAmYTUZr9eOOdo33S8pl95r175IQC24ti6CrgclZCc8RcSvz4anpFAndaVC/bVjJ71GlaesGHBtGxJxPe3DR9vzQaVFREREniFOeERERGR6nPCIiIjI9HwUh+f//Lt/evL523/8rpX513/4Z23bv3jz39q2n67+8uTzA8R9rcBbuT90P2dTPJiXyx4yuFlQOBOs4Axhgd/t+wrtlQM4FxRQWKHgwU3osrTjgbu0PvS63kO7DuAb1baRw5O4IGOM8VAcJ/otl/qQrlEtlzgk545JXPtbelLXGsbhYdnPu/bXGODswH9j6HrU++NcudrXT3FZHor3Rg4anSMFYCZtSP2Na30mIlqJOxybieNGPCUMtPIU36het4/hYTxWF5H2Re1req5dC43za1lEK7HnVE9zE65m/lF9SKiqXt+PcTy/4REREZHpccIjIiIi0+OER0RERKbHCY+IiIhMz6PS8n/6n100JpbjP5fPsGourPS9PXY56leH09C/P9/1lcs3iy70ElVufrnqgYJvll2wvl1u27YHkJa/Xrw5+UwhfSQ7k9xXRTGSll+E22r9JKF9B/1KkiYJpDelvrtV7684DC0QuFNpMgnlS1djJ6rEmO5H7a911T4d48z5wNDf1VW9QTSn86YxhvJuaWsqYGLg4pXibHJtX5DIHoQ+jsHBj10sT1ehh2DRYyLn03Oz13XtSuLXsgpd0aRdqax9reT7lPNerS5f32tl8FSoT8o95aUBYj2uk7OvFb+prfQsTaTxDz1vv+ERERGR6XHCIyIiItPjhEdERESmxwmPiIiITM/iePy4qY0iIiIinxp+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZHic8IiIiMj1OeERERGR6/j8jStwSQ8E4SwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.activation_stats.color_dim(-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`color_dim` was developed by fast.ai in conjunction with a student, Stefano Giomo. Stefano, who refers to the idea as the *colorful dimension*, has a [detailed explanation](https://forums.fast.ai/t/the-colorful-dimension/42908) of the history and details behind the method. The basic idea is to create a histogram of the activations of a layer, which we would hope would follow a smooth pattern such as the normal distribution shown by Stefano here:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/colorful_dist.jpeg\" id=\"colorful_dist\" caption=\"Histogram in 'colorful dimension'\" alt=\"Histogram in 'colorful dimension'\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create `color_dim`, we take the histgram shown on the left here, and convert it into just the colored representation shown at the bottom. Then we flip it on its side, as shown on the right. We found that the distribution is clearer if we take the `log` of the histogram values. Then, Stefano describes:\n",
    "\n",
    "> : The final plot for each layer is made by stacking the histogram of the activations from each batch along the horizontal axis. So each vertical slice in the visualisation represents the histogram of activations for a single batch. The color intensity corresponds to the height of the histogram, in other words the number of activations in each histogram bin.\n",
    "\n",
    "This is Stefano's picture of how this all fits together:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/colorful_summ.png\" id=\"colorful_summ\" caption=\"Summary of 'colorful dimension'\" alt=\"Summary of 'colorful dimension'\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So with that in mind, let's take another look at the result for the penultimate layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAADNCAYAAAC8XqoPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2dzY5kSZqWzf8iIzIrc6qqK5lWA6JpDaAZaYbZgAQIiQ0bJJbcAkskuBtug9sYwYLesGAxQgIBhdQz9ZcZ4X8sanok/77Hw9/0yO7OMD3Pzk/asWPHjp0Tlu7PeW1xPB6HiIiIyMwsf9cNEBEREflN44RHREREpscJj4iIiEyPEx4RERGZHic8IiIiMj1OeERERGR61o/9479c/ptn8876//u3/6RtW+xPP3/1X79rZY5/9surj7n+xc9PPv/w99+2Ml//w00/5qrXVdu6+baXufm2X44vf/mXvf7N6QG++cWrVubdV32ue6DRAFPizTen7VjAKFk9wEbYtLtdXDwecoBjbk8/b/tpj9X7vm0BdY1F37TclfPe9zJE3W+MMbYvT0/05vveiPs3vTOO0K7t69ONy4dehsbc+ofeLrqW6/enG6kN1F8raMe7L08LHm76jlg/cKzjFdpO13b9LnyslWLLXS9C2+g+qtuOMM6p7w8r6Ix6y2z7jodN32+xz673YV32pe4Kx8CyPAd2d1Sob1q97wc9Lk/3pWcMjR0a+/Qsvf/8tCGre+jXR/9inme5hY1hv27Kfbp9BYVgnNN5E3UsLuG5RudNfVif56v7vt/+BbQhaCvVdeh/Xsd/+Y//4ewTxG94REREZHqc8IiIiMj0OOERERGR6XHCIyIiItNzpYL16bF72T2lzfenUtVh0+d3oR+JHG9vTj4/vIb6SQqkaWYpd7jpRZYgHW6/uG3b1t+dGnIoUYLslQp5tW0krxH7F723X/3fU0Pu+5+CvUZ9CP1Tz4nkXTrvKgWOAeLmGOO4KMItyHckMqOkXg75/vdAUCaRj6TG7y+LoSS2Nul3jDGg/VVEfPW/u6n7/sve2C3ckxWSio9wjXDf2lY6HI0dEnpBLN/8cPoZJXgQWweMnXqeNL7o/luAFN33ywTlKv2OEQrioQhMInbdGSVvuN57kNnrf8+PIHRXSXoM7p/7N5f7jNpA503HXJeXIw70DKCxD2OgSsoovNOLFwCJv7uXp5/xuQntevgM+r9c390dtAHq3wfjifqQ/qY8ht/wiIiIyPQ44REREZHpccIjIiIi0+OER0RERKZnGmmZqLLl/q6f7lM64HhzujdKbjClTGRUkl+3IKNWQZkgSZOSP1ORuQpyKBiG/PDVaWdgejEl6AZTdRLaViTvksxJYuD+chkS65Z7EiRPP5M4TdIvyflVgOa02UzAHJRK+93pAe4/74UoETgRNY8kW2L6L5RrlfdNmDpNwxXGwO7laUOqxDzGGFsQNxMRmMbhgdx/CiwvEi6mmqPE2svta9L5GGNRGoJlSKYO3l2gZ0yakF3PiZ+t2b1M6ef1OYASNrSVkolrmjCJ05iQjcL7aTkSv+mrC3qWUspxhZKcqb+wHa0RfdOuv2eD160moqMwDn34GH7DIyIiItPjhEdERESmxwmPiIiITM+zdHhWb/uq5PTbcPU87r/4uA7P8pvTH/XX96+j/ej36fp76x4Cm26+6dvuf9J/EK2/+ZJTsIMQNfytm1bOLb+30srGFJKYBJ+Rq7EHd4m8hersrGBVbOr7PYU8wu/TtRyVodA8dFmSEDXymWih5NJn9Ns9ulhQbkXhliWYbHV/2SMag/tnV/pnT7/n03/DaFu9luS7hOGNGMxYyu3DQESqvwcPQhkKfSRvr4xrdi4ebeFfQ8GJ9T5KV/rGIL1gFWz0CeGebOGscD3ovDH8FbzGVgauUQ35PFeuhpQS6EbRPV+gMYEru4PDWP82jNG9oTq+xuC+Jv+ueZOhYkPtam0I7+XH8BseERERmR4nPCIiIjI9TnhERERkepzwiIiIyPQ8S2k5pQpgFI72FOpq6fdvYMVrCqcjYbEIWet3vQwJnqv7brnVlZ8XYNKSyJwKvZsqmJEUTcIcyoOnn0lgJEGZQvnWIE9XqA9xBWcMEKyf+/EoGBBl2iJwo3RIIj6FidVrBOOLAvgICpGsQm8ayJaMMZQ0ScwOAjxRnIbVoTGwDoT6FsAHsigG0QVhaHSvJSLtGDDGqL8wpA3qCs4pEY/H4Odau7dCsZyk5TqG6XwoWA9fEgm2ofxP1ygIjKR2sQANL3aUbenK6ATdI7X/6TrSeePfIxCekzbQedNK660ueAY/ht/wiIiIyPQ44REREZHpccIjIiIi0+OER0RERKbnWUrLx7/Zk5ZJCqtyIsqWT+HrX518XBx+0tsAQt5xSdbWqbRFwmpd/X2MMe6/6AUjyRQgQTlZkXh3RyZc30SSYUugpURPcjlhW0uIRUOvb0qSkMcYY1/uFl7ZuO9H1y0RrCl9GVcND1xXGoco+YJwW88JJW8YOyh9lnYc4XxIVkwSaGmc0HhavYf6Iem1yaKwYjQKpOS11gBakKRx7ONK34/XTWXG4PR2bH+Vlmkl7lAEbi8lULvCBOsmqQdjYowzz79grGACdPAyw48HLYcLJfXkWUf3KKU2E8dAsMYV56H69Q/QjpvLgjW9gEDUl1XohYoPjVr2Gx4RERGZHic8IiIiMj1OeERERGR6nPCIiIjI9DxLaXn5zQ9t22Hzedu2KsmcT0mojMBEXShHInAxSCkV+trZKaaphv4XSmdl1KxRAoX64QS2r04PuvkO0otfkQUKm6qkHiZrU5IzyYmtDImCaZJsbRsdj6qnlO7S1ySfYzgyyJbHBQiLReZEwZckb5IygycOyqJwTq2vqe9hHGJb6R6psjZckAUIpElicvqCAF24QxOgoS7o51T+r+OJX7yAdoF8XIXeNAkZXy6o4zCUtekcMaX55vHPY/S/KWOMsQvS25Pn6Llyta9J3qVrhH0RPJ/onqH7b18H4oAxQH0PfUjtry+h4IsXcB0fw294REREZHqc8IiIiMj0OOERERGR6XmWDs/x2+/7Ngjzq8FwD28+7mrpu7/3s9KGXua4ynyH9tsqNDV1TepvpOlvxbgSN/zmXqfJabAe1V+31d9txzjjENDIrecULqRLAYLU/zffnFa4IzeH3InAzcBVf+k3eOif5Hfz1AeCn+XbNYqcpDHYVWs+Qi+D7Q/cA3QDyDVJwxvrNlh5nXZMVypv+4EDQ+GQbZxTCCf5J9B+XLm6toPagM+nvu1jhgVWB4nCQfFZl5zjGD3gD9q6e9m3Ub/WdlAb8HlLAZ7VqYLzppXX0fWhsNSyK65UT14dOVvlmUXXYwkBnsfEB0od2UfwGx4RERGZHic8IiIiMj1OeERERGR6nPCIiIjI9DxLaXm8/aJtQpGyBlWFwlzK7rPTA6DgC0QrIGPYF4ldIKaVMLT1u257vX8JdZFAStRANhTaQI4DsbyKs2lQHK7EXUczBetR34Nsiauelz5LV2HG8LgqNUKKGl0PlHDrWKEy0KwV9Q+Fe5VyJKmnK68vi5x4fAVtIKE0EGep7Xg9gtWtxxhjWWRUklhRyoT7ocquFApH0i+uoF6D4uiawQWn59PVojGJwMHLESiMh7QxRhIr1Q99nYyV9Lyvhl4QuLJ+et7i85xehKhtgDKpyLwvIYzrnhGMgbA0zmugI8rh9Kx4BL/hERERkelxwiMiIiLT44RHREREpscJj4iIiEzPs5SWH37/s7YNkznLNhKcVm/ftm37r7+Oyj0UiRgTPamHg3JLWtE5lKK3L0933n6WpRcf1mQBQjtuL583niMkfzablqRikqJJwKxSG6ULB6ugjzHGDmTaVVkVPkrMPrctWPEaIbm2ppuG0m+S6krbaEV7SsimpNcqNeLq0GGiaiL649ihciQk15ceKC03XFm6ysFbGF8ErnjdGtE30croUV0DROMgUfdsO6pgHa4IT5Z9krSMidZBajO2LUyYpvF0se7B9x+2v0rkJBVjonFWLiL921auG60kn1Lvtyhd/wJ+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZnmcpLW8/I0OrbzqWZN/tK4pdzVi87pbhYX1a3+6OTEFKF+7bFodqsfaqMHkXDlll0VTUXeyz/mkyOFwOSnemNNDd3elnlCFDObgJknQ5KK2T5F1KIS71YXIt1J8Ii1WIHiOXj2v/J2nJY7AEuHwH5cp5P7yBpOVQHqwiYipOcx+eVra6D+9vKpaMJ+hXFE9JFg0EaEzRpvTism8izY5x7kUFKFdTwMOEaTxm8l9qksFJGA7GGJ4PJbWHY6yVoXTk4PlEx+OEeihX5Xw4xz29kAMp2kk6NbULXxoI2p+mgBP1WYoCdDj2f43f8IiIiMj0OOERERGR6XHCIyIiItPjhEdERESm51lKy5TqSinBVQQmIYxk5AFJy8dvv2/bljsy68p+NKXERNJj+QzpyLDf7rZvrKmxJPIdbkCmTpODywmQYFjTmM+1IxHYUJAM/FSUNINE7jHOSL5l/KCcSmI51L8uknJNIB6DE1WpXGvDJouIxVRXkjLrNaLqw/86tTFMEja0PxFK93d9v9W7TGRG+Xh/uQwmU8PYaYL75UfHj8UC0ZjGCdYVjv0KpklDuzDpugq38fMQygV9Rn2P5YJEdLzXaLwG9afyLl7vC3WPMcaK7mUAX4SoZULxO0qQp+cJHTN40SK91x7Db3hERERkepzwiIiIyPQ44REREZHpeZYOTw38G4PD/I4PJYCPgr3evIyOSSuoP7z+xWkZCKLDVXmD36draOIYY4wFeDHkJdVd6XdUCBkkrwddluXjn8+ROARp6FWyMj3WRb9P0zUKfuumvkEnCeqvq2UnYXVjnBnD6OzUHbO6aAxXsK00poOxj35I4hGN7kuRy0LXO3Ve6jnRedO9fKQxENyTMaX+NFj02lXV8dqGq563ILowBBAJ/C/sC1rhHIrVex7HSbiyezJ2rl25nK4ZuYN03hSMWtuahF2OccYHSq4R/R2A61aDBuNx8gh+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZnmcpLSMkDxaR+bgE2XmTGX+Lf/THbdv6/Wn9GNhEIWqry5IpBQ+SJEYhjItDNaD7fhjuRiIl9mtpA4W2hfJxbxfsR20gsbWGnNF+6YrUJCTXVaopiC6Uj2v/YN8TyX9RUGqEUL73vSCt2l6vdx1eY5wJnwyk6/1tlsCXBJ9haNuRUtr6puUOghnLatOpnE/3Q1t5PQy7RLm57kuBbOFq4Mmq7alkSi9a1H7llw1oQPVNbQVy2o2Md6qK7vl6T9Jfx+xdktY/FHiK1y14ESIdh/gsSl7aCPo+LYdjJ3zWXR1K+wh+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZnmcpLb/7qs/TjqtuaB1rIjPIUvc/6UviUmjl8vseUflQV1qPVxsPyqViF8miRWRGmZNSMklgJCGvnie1NUzL3d+VMqnsHEiAKFamqxaTcFv7hwRMWmGZLtL28gXG60bCeyCZYlIxnWPwRFhCWmsqybZzIsH6CStL90J90+qBRH/Yta5KDkJpmkpbBeg67n88IGzC+68UpH6GHfH+C1K6eWVxkLxhTLeXI0IhlpLg68seqUebklzvNDm4rZZOz8Pk2XpmWytD922YrL2qYxOeYZgeTocMytF9Sy+5YHr0E/EbHhEREZkeJzwiIiIyPU54REREZHqc8IiIiMj0PEtpmaTGASmfVXJLU11Xb9/2um56V+1qyjEJbZAiSttqQuhTUoJbim8WJo3TX0pBrfVT2nNsFFbJLRSgua+hXLDfHlKIMbG1jDEUKyk1FqhCHgmSqazdknGTdN4x8J5Z3pNdeeHzOCPvkigdCNbpOKwnisJkIv2eaUeVcGPhHepqIigJxDAOE/mfZOHdHYxpkqkDeZ5S2SluO3lJAOVUFG6za9TK4PXI7u9WDqT+RModo/crCuMvIP0chPrW/ySkh88PvCevlfOpXOkzut4EJkCX9uM5pn/bft2eDysuIiIi8vxwwiMiIiLT44RHREREpud5OjwvYCMEsrWgKjjb3V2f8y1qoOAY44ef9W31Z2Z0hGhKiavr1pXds9+wW7jiGGNUhydcJZlC7ZLAulXqn9B5Vzcj+W19nPEpysrMtF8aiIi/+9emkttAm+h35iDEkMcJbKsqWbgyM60avgf3Y/Uu8MuorRRsWMd5mhSH5WqoHfgP4ROO/IDq1CzB2aIVqdHRq/uG9yR5YvU+3b0CF+uhbcLOZjeqfKTxRGM6cKgw8BSInh/0DAuCFM/t3DzK8BmceIHoT2G4IlRfh04YwknH5DDQy/dkXfX+xwNcrmuE+9E4bOGNH2G24jc8IiIiMj1OeERERGR6nPCIiIjI9DjhERERken55KVlCgHEEKREWgZZ6v3nfc73+rYv03pc9p23n51uI2GuSVxjYOBblXVRXoPzxuDBFu6WyXcHCuAD6awKkVT/4YZktctCIYYApvLd5rIomIZqYfhdLUJ1kTwI1VdJb4nCLVw3qCtZIjoV1xcQPFhl/NV7ClykdgVCLIbVheJp2fcIYXi8ynq2knhra7jidbTSd/hfTZQ5S1upXXtaaRoD+KBcct6pZF+fRfRiBImtRHXUw+cC3acc4FlPPKs/ake68joFvZYwQhTNaTxRv9LfnqCuNLS3jif8mxgGo9YQSQyLTJ8V5w8jIiIiMhdOeERERGR6nPCIiIjI9DjhERERken55KVlglYCRhmrJS33MtvPsjnf7mUvt6+JzyjyhW2tVZGwCrIlJZfu72obehlOCc5W/yaBu0Kr5FL7acXgviNsCzzHujL3GGMs0pRgkOHqCsgoO8eCda08TNamhNtAtkShFE6c5MRlSROmMZfKg3VfTvzu23g16CKp40lCI8Jr1P47uAM5H5LOk5TuKH37TLtS4bm1AVKhE9E4Hk9QLEnSxrGT7EhF6FlHxdIxUPdLX1So8i69aIOJ3NCu4N7C7gqT85tYHqbRI8F51+fJGBiwD2ne8Iz8wBmM3/CIiIjI9DjhERERkelxwiMiIiLT44RHREREpudZSst7kJaX625V7ben8zkSqEhU+/YffNG2be8uS3SUVDwoFRMFrdL+h94wlCEp1bVsIrELRbhQAqyJxktI3kX5ddvLNXmQ/DwSNyMZErZRuSDNdowuemPqKlwPTLgtHYvSXkgkW6bSYSARL6EQiZuREEsCfxrjW+tKzzEd53Ubpp9fTpMeY7Sk3TS1GeXmuh+dNyWFxzJ4LZQlfuPYb7Y2FLkyQRdFYDifBb2MQfduTbBOhXG6bqVt6f1H17ueJ94emNoMddFtVF4wwTGH5ndwTHqehMn2re4nPCM/4DAiIiIizxsnPCIiIjI9TnhERERkepzwiIiIyPQ8S2mZpKoVmFyHKr6BYHi46XUtjr3c+n3ftn9RLKpQaFskScsg7aVJxauH03Io2mHiMCXEklB4+hnTNCFpmdpaBegFiM2JuPnjzkHabyi+oVhXq6LUUkywhk2lrdT0xQ76guqvZTC1GYRxkqnhmA1ISsXkWtq3FEwFxui88Z6B6h9oXyi3vXwfLRPpd2SJsCx+X365IEn6PUdL/KYy4T3DicmnH0lsPpD4HZCkBv9YMK2/XO8o/ffM2K8vjoTXFu+/6n3TOYaeP6Vt17+BT5GDa9tQig8T6pvv/hG+nvEbHhEREZkeJzwiIiIyPU54REREZHo+eYfnu3/2d9u2IwT8LWtw3xhjUcIIm9MzxtiDw/MAK6jff05BgOUzrU4LgYgUkngoIYm00i06CrBa86GuWAv9hSGGaQDY7rLbQCtqJ/Aq8VAOVwK+/ONz/Ps0eVal/niV58SVocOF1639Vk+/59PYCYPVlsUJ27+AJmDA2HVuRupZRfvR6tngA6FbUgMXw5WysR31kBR8Fw6nxC9DfyN0XloAH67qnS2X3u5dCjEM+6KG39FzgVcbh7qAdj8/IayzuSxUFz5vL7cLryP1BfhABwiETcJfsX5o/7VjJ7kneYX7D3vG+A2PiIiITI8THhEREZkeJzwiIiIyPU54REREZHo+eWmZIFl0CfJS3XbYgCwM0vIOVkbHldbrNpLvQqlqWeSu/Q72C1cNbwF/YSgVSrjBNpQCr13hPAiYO19/cN6hVIz1gxhfSTPOarYlSoHhyvEtmIxkYZJYk9WtR79HsF0gixLt+qI4nbW/75i14erVuTHIDeqi/q87Yzhk3y0S46kMNBZlZ5SIiyyKJw7NCsYdPgPovFG8T9oATcAw06RfoS7aLZCWWd7t+6EoXfsnlanD+yh6kYP2O1yunwRrehkD6y/tP4ay9mP4DY+IiIhMjxMeERERmR4nPCIiIjI9TnhERERkej55afnbn/UmLm7v27b1uhtgu12Nu+z10wre+00m6dV9j5CgTO0ijofSVpQaL682PsboicAkiYHAjVYmprPWVeh7mXjV8yoPYspnmKpcN6EQmxp/QCJdk8GICcAlPZXk10DkQ0geDVczX+BAv3xITIgNpMlEFj5LrZ8kbIDumcXD5X1RKoZ7C8XWep54beGgtK3KouE9w3XBtnICeN/iCwFUV/kcjhMUy5M3AugywrhAiThJHE6kYiJM0aZ7t/bFAf5m4SHDlzb6quSp/X85aRkFZXp2X5sgH6Trnxzmg0qLiIiIPEOc8IiIiMj0OOERERGR6XHCIyIiItPzyUvLh03ftiA5GGTUzebUttxBUi7Ja/vbLkLtX0Dbau+RcEZyIvmEpW3LNSSlLrrxR+mpi32RDq9Nrh0jSvalutJU11Z1mBJMgviHCmx/DaauQrlaP54PNTYoRv0cptK2dtD1rgL/mfpRIK1SI0rYUFcaO33pgCmpAE1CbJrw3XaEuqBY60Nqa3o9ykMlln6Dusbo121B5xiK8f2AsA0TdIP60/RwaisJ6FUGx1smFONLX+P1uMleoKgvQuALG0QivA+Q5dPHGsYol7FJL5zASxA4Nuu1xOfhh923fsMjIiIi0+OER0RERKbHCY+IiIhMzyfv8Nx/2betIMxvDX7Otrgs6xsIJ4RgpN2rfkwK36qhSgsI81uRNxQsT7ugqSj9Fr2F32QTtwgcmCMln+FK4uV3WtgPVwJOXJxo6d4z1LZSVbgCeegj1HKYDBj6UokXQ6D7EZx3/Ft35n5cTQvgS4MaYVsdwul+4SrVx7pKPPkhO6gLqI4Q+k3hdWtDDEPt6L6FupKxCc8w3JHqqtebtJVr/wqFuhzeM3QtAwcM+wvDG8t+qfOEz6LLzlZM4ArSOOfx1De1cvjMD55h4/pV3B/Db3hERERkepzwiIiIyPQ44REREZHpccIjIiIi0/PJS8sUXHQD8vFmBULy6nQ+t1t383gLgWO0gjquZFxEqyWIwEsUzC6LiEsIUjyAFH2EVbZb9WEg2xL6lQTrYw2+SmVIogXwZbJlFHJGkDAHUGBkawLJimm7ajkKE6NwxaBdC7jeuHJyKjdTaGGr63oJt+0Wjtdjsy3TVZ5p2+X7CMVNWg06PWYlDcBs8nx4vFTUredJY5NEZipXq0//i52IuVQkvL8HBU3W5lN/wXMZzzvpw/QZWcMCUZwOVyDH8VSKhIIyknQ//kkM2v8Rvp7xGx4RERGZHic8IiIiMj1OeERERGR6nPCIiIjI9Hzy0vLuZTecbmlldNj2UFesRekQ5OAbWJUcJLfj3em+a0iAJvmYROAqN+9SeZckvSLbUaryksRWcmTpmGVfFGLJTEMxN5FYoQ/3tJRxUBcKq5BEjQm3pV/pANRfwerGR4prTVZGH2euUS1D4xclU9hWZfmnBC9Xz5jk17SqdHX0AiadJ8nE6SrVRLIrXcckLZz2e8Kq4Y30GgXjFd1wOu0gcfgp0NivY/GIjQ3F79plwXN6jJGNAepn6sS0DxOx/Nq6UqGe/4ic7gZ9iM+wR/AbHhEREZkeJzwiIiIyPU54REREZHqc8IiIiMj0fPLS8uFlF+ZebHZ926pv265P53P7my443b/Y9GPe9G6hJe0XJZmYUpXXlMwJ7A6nbSUBer/rMvWehLlA1iaZmqa/JHgeigR9PPR2oawGbW1CdeihLje9f1oTQsmRBG6UWGuXwfVmzzE4qSTaeZwRdeuma1Oux2B5sPQjioJXytTUBny5INiXuvAAYigmikOadOsKlEUfad9JQ0o74EUC7K9EWk5E8w+h1pdcxzH6/TEyKR3vGZRRy3MtbVd60HracI1SSba+JID7wWMzSRw+wphepF9dBPcWSuThyx513/QaoUT+EVPZf43f8IiIiMj0OOERERGR6XHCIyIiItPzyTk861/8/OTz4q67ObdrcHhg2/3+9PRqEOEYY6zgN+btLf0YDfsWr4fcojWF5qFbcvm37iWsCH9YXw5/WoHvEioj+IPuosyTyadhz6Nvqv7MU34/joqQz4QrfV8OaTse+v8XyD+hALPkl2f8XZ5cmaANhyQkjCqjY6JDEPonAVe7GRToSSvO065wzDouMOyS6qKuSMIbQ5+prhIfhzdiwF9YsJagQ5KjV+/vcOxEwYN0j4ahktG9lQZUUlW1qRhkSnteDtdbpn5WehvVtqZ1Xds9tN+VIaIfit/wiIiIyPQ44REREZHpccIjIiIi0+OER0RERKbnk5OWv/nT3z/5fHP3QytDgvLNkuTm7cnnLUimmxsILHzRu4XE2bovCcorCPjbQzvWgbS1QTn48pyVAhFJ1j4EdY0xxmJxeVXhtK4qYtN+6NEGQXfBArx/VRe0C/rsUOTBNYydHQTYUV21bUf4v8eCwiGvFClJzKZjJpDgSyTOMtWEoZjAYV+S2wKh+xwoNzdJNmvXEoP0aqFeJpXzl4GNmorrKOwHpCJ+q59WDU8JxkX6IgHWVQM2of70ZY9679JzjZ7B2P66ijtcR3wRJh1PZRu+4ADgM6Xs+zHHIZ1jHFL66+N8UGkRERGRZ4gTHhEREZkeJzwiIiIyPU54REREZHo+PWn5b5+KiC9vH1qZuyIjjzHGLayWviureG/XfXnaG1iVfHvb6xLrHkQAAAiESURBVCJuSrLyDcjUBCUmb4uASe0i2XkP5SokO5PsRRLdHtJlF8XcO+DKw9lK4lXoXS6zVOgkkTkV2lCAxtTbyzIfiuWBBLi/fBnHGGMsg1Rd6hsSp3cwXJO+jhdjD5KDUxGRyq3K2E9XcScWV8rgqUDaEsWhrlggLuVSkXYN0jW29cqk6zglvVBfBvhxv0TMhjbA84PAcZdI0aHQW1nDcxr7HsrV/qHnNHFtW5eUrh+mgNe20n4rGP3JPb+Io6PP4zc8IiIiMj1OeERERGR6nPCIiIjI9DjhERERken55KTldz89FZN+dnvfyrzZvG/bXkDS8sPqVATe7PvpvnzRpegdyMEkgL0o0vItSMskGhO19j0cLxWZqwC2BhkvFUNHIMiR2LyEuFmqf1P6kCXpfsxEzE2lZZQHcd/TtqWJwMkxPzQx9LQdJSkVJFA6x5sbEikvHy9N0U5I+zBKzQ4Srcc4J2BeFnqpX5ehCJyQJHKP0aXV9Hhp/ZX0/kuOSX246u+SXC1Ap+26Vtam+hPxnq9RJu/W/kmvN43phKeMpy4tX/8MTtr/ofea3/CIiIjI9DjhERERkelxwiMiIiLT44RHREREpueTk5a3f+M0RfnNiy4ov1p3kXkFctT2eDqf260hqfgI20DKpCXtX92cCs+UZFoTVs/VVaELEyexFlYghlIbUABDibjIwXBMaimm5da6whRRIumfJ4nMkJCd1E/jqV4TTKu+FvhvDAmAdEQaF/WcWH7N0nKT/k9FRBJgr4W7v0rw16dCJ/ulabad65Jrx+BrWUulT53kObCC5+G14xCfJyTShsJwfbmDXhxJSfa8tl+Tvx9jPO1FiEp+f3+8lzGuHYeP4Tc8IiIiMj1OeERERGR6nPCIiIjI9PxOHZ7ln/5R2/b6J9+ffH774rtW5vPNu7bt/tBP5W5VgsPA16FtFDxIfsirTQ8trDzse6rWkn7hLYdMfZ1FsEI71UW/TyeuyRj9N+Q1eDd5ONZ1Tse1vw3T+VD9seMUtIu8gt8k+zB4kK5bWl9Sf+LwpKuNE9V7Sr2x9N6q5fYQpnmtf0KkxkhtV+p00Ni/1kGiY67Bcbs2pDJpa7xSPZwjBV7WfdfpMxhXDb8cUpreM7Uc5DReXReRtpWea/VZkfTNOZLr/aF/B/yGR0RERKbHCY+IiIhMjxMeERERmR4nPCIiIjI9v1Np+X/8q8/btr/ze39+8vkrkJZfr3oY4Ri3bUsV6w6rbH5HKyATN0XSS+VBYlHkZlrhfAXtouDECrVrA+W2IFiT3PybHDS/Xb33rwDxjc6xlkpWqj9HFSlp5KTBZ13mo3bBeArlxJt1CUl8imhcytG9toWwS64sKBcKq0R1lMk9R5kzqPtjhsKl/2sF5/qjtoM57WsSXfmlgV5Tbf8KJOn0xYtE6E2f5yRFH6IXFdJnxeXxmgaeXk923W7Wl/8mrpaXA1zHyOR8uraP1vlBpUVERESeIU54REREZHqc8IiIiMj0OOERERGR6fmtScurt2/btu2fdCH5D15/ffL5b938qpXZg+KZyLskl92telry3eqmbcNk0SJfvdt3FZhSm3dHSF8ubbtZ9gTl9QKSoqEvmqwdpiofwGpcJqtnkxgaypBdYu37pcnXta6PmfZM9VVpPd0vPeY6TIBuZWIpN7tuSVtTQfJasZ/2qzJnmix7rVieyMhjnGvr5WOm+yX3DPGhgudj7SJQ4C7Ce3I+5+qq0HWktGdiRWOlbEteXBgjWyX++tdZroek7rT/K9eu7P4U6jE/xmTFb3hERERkepzwiIiIyPQ44REREZHpccIjIiIi0/Nbk5b/+7//g7btn//8l23bH738Xyefv1x3sfkv9q/atu2qK4U1mXiz6FIxpRe/WfckZxK77g+Xu2936O3aHfsxd0VIZiEWJDTQyarI/AAJyg+L3nZM6wQZvPYZpUKnIiUJyZeOd67+SL57Qhr2tZBYXq8bCaUkAu8COT9NgKY+XF+Z6noE4Z3Ou0Lp3tQuSp2+VoAmYTUZr9eOOdo33S8pl95r175IQC24ti6CrgclZCc8RcSvz4anpFAndaVC/bVjJ71GlaesGHBtGxJxPe3DR9vzQaVFREREniFOeERERGR6nPCIiIjI9HwUh+f//Lt/evL523/8rpX513/4Z23bv3jz39q2n67+8uTzA8R9rcBbuT90P2dTPJiXyx4yuFlQOBOs4Axhgd/t+wrtlQM4FxRQWKHgwU3osrTjgbu0PvS63kO7DuAb1baRw5O4IGOM8VAcJ/otl/qQrlEtlzgk545JXPtbelLXGsbhYdnPu/bXGODswH9j6HrU++NcudrXT3FZHor3Rg4anSMFYCZtSP2Na30mIlqJOxybieNGPCUMtPIU36het4/hYTxWF5H2Re1req5dC43za1lEK7HnVE9zE65m/lF9SKiqXt+PcTy/4REREZHpccIjIiIi0+OER0RERKbHCY+IiIhMz6PS8n/6n100JpbjP5fPsGourPS9PXY56leH09C/P9/1lcs3iy70ElVufrnqgYJvll2wvl1u27YHkJa/Xrw5+UwhfSQ7k9xXRTGSll+E22r9JKF9B/1KkiYJpDelvrtV7684DC0QuFNpMgnlS1djJ6rEmO5H7a911T4d48z5wNDf1VW9QTSn86YxhvJuaWsqYGLg4pXibHJtX5DIHoQ+jsHBj10sT1ehh2DRYyLn03Oz13XtSuLXsgpd0aRdqax9reT7lPNerS5f32tl8FSoT8o95aUBYj2uk7OvFb+prfQsTaTxDz1vv+ERERGR6XHCIyIiItPjhEdERESmxwmPiIiITM/iePy4qY0iIiIinxp+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZHic8IiIiMj1OeERERGR6/j8jStwSQ8E4SwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.activation_stats.color_dim(-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows a classic picture of \"bad training\". We start with nearly all activations at zero--that's what we see at the far left, with nearly all the left hand side dark blue; the bright yellow at the bottom are the near-zero activations. Then over the first few batches we see the number of non-zero activations exponentially increasing. But it goes too far, and collapses! We see the dark blue return, and the bottom becomes bright yellow again. It almost looks like training restarts from scratch. Then we see the activations increase again, and then it collapses again. After repeating a few times, eventually we see a spread of activations throughout the range.\n",
    "\n",
    "It's much better if training can be smooth from the start. The cycles of exponential increase and then collapse that we see above tend to result in a lot of near-zero activations, resulting in slow training, and poor final results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batch normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To fix this, we need to both fix the initial large percentage of near-zero activations, and then try to maintain a good distribution of activations throughout training. In the abstract, they describe just the problem that we've seen:\n",
    "\n",
    "Sergey Ioffe and Christian Szegedy showed a solution to this problem in the 2015 paper [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://arxiv.org/abs/1502.03167). \n",
    "\n",
    "> : \"Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization... We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs.\"\n",
    "\n",
    "Their solution, they say is:\n",
    "\n",
    "> : \"...making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization.\"\n",
    "\n",
    "The paper caused great excitement as soon as it was released, because they showed this chart, which clearly demonstrated that batch normalization could train a model that was even more accurate than the current state of the art (the *inception* architecture), around 5x faster:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"Impact of batch normalization\" width=\"553\" caption=\"Impact of batch normalization\" id=\"batchnorm\" src=\"images/att_00046.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The way batch normalization (often just called *batchnorm*) works is that it takes an average of the mean and standard deviations of the activations of a layer, and uses those to normalize the activations. However, this can cause problems because the network might really want some activations to be really high in order to make accurate predictions, they also add two learnable parameters (meaning they will be updated in our SGD step), usually called `gamma` and `beta`; after normalizing the activations to get some new activation vector `y`, a batchnorm layer returns `gamma*y + beta`.\n",
    "\n",
    "That why our activations can have any mean or variance, which is independant from the mean and std of the results of the previous layer. Those statistics are learned separately, making training easier on our model. The behavior is different during traning and validation: during training, we use the mean and standard deviation of the batch to normalize the data. During validation, we instead use a running mean of the statistics calculated during training.\n",
    "\n",
    "Let's add a batchnorm layer to `conv`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def conv(ni, nf, ks=3, act=True):\n",
    "    layers = [nn.Conv2d(ni, nf, stride=2, kernel_size=ks, padding=ks//2)]\n",
    "    layers.append(nn.BatchNorm2d(nf))\n",
    "    if act: layers.append(nn.ReLU())\n",
    "    return nn.Sequential(*layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and fit our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.130036</td>\n",
       "      <td>0.055021</td>\n",
       "      <td>0.986400</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's a great result! Let's take a look at `color_dim`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAADNCAYAAAC8XqoPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2dy5IcR5peIy91AwgQIBtg07ppo+nWQhrrkWkxO61legK9ip5MT6B30E47mcmM1JimmySAuuRFC3aPdf7/qcoPUVm8eJ+zq6CHh4eHR8CZceLzxX6/n0RERERGZvlTN0BERETkqXHCIyIiIsPjhEdERESGxwmPiIiIDI8THhERERkeJzwiIiIyPOuH/uN/Xv5Xv1n/kVj9278/3PDdu1Zm87sv+37f3bRt17/55HC/Z6tW5uy7Tdt2+6oPh/X7Xdu2vD3cdvO677dfLnpb7/pwunl5fM692PZtOxi5F98dtmtz2duw7Kc9TTDK99CsfenG1U3fkc6b2rEu+5593/v57nlvxNm7Xm5z1euv7NbZ9Vj06qft+eG+dD6r217X1Td3bVsdKwvo+yW06/pVH8PnpS/urnp/ra/7CW0vYFz0pvbjfdcH4vayH/PiX3pld88Pz7uOpWmapt1ZbxeN/Xo96DoSS7hGd59Q+w9vkt1Z9v/Fu3MY5+8OT4D6i+6Zxe74vUXjkPr17H2vawXjotZP55P2Ybsf4BLROdLYr+3a0nMN9tvA/VCv7Q/1HZaj67F+3wfi9Wf9IbzcHLbj/Nu+H/17RNeNnkUVuo7/47//t3sfiP7CIyIiIsPjhEdERESGxwmPiIiIDI8THhERERmeB6Vl+fHYX14c/L198zLa793ve7kqcpHcNy36pV9dd/Ht9iXYZMUwu3tO0iHsBtPrKrltQdxcgyi4uu113RZ58OqfQbR73c+HxFYS/uo5bS9aEZSD6zlO0zTtSzFq1wLWuXv3617uk/9zKCK+fwu3NUqTUOy8b7t9USTZ616G6r/+/Kxtq4L1Fo538SfoL5Iay+UleXe/ykRg2rfKwdsLGBNbumd6/2+qrAu3JI0TEqyrLE/CKomtJOFS+2/KxwskpPMHASD0vji8cHXcTxOL+Nef0X16WD9ds1u4RrfwfLqA867Pzfdv+vg9h48LaOzXsUP9vF/2ti7P4EOCMl7pQ5LNMxD2P/RyH97AByaLw7ZekGj8HK4HfrRx+Dd90EJ9QWOsnjdJ/SRYP4S/8IiIiMjwOOERERGR4XHCIyIiIsPjhEdERESGR2n5Z8Li+jAxeffmWSuzvO4yGQmYVUy7+CNIgSDJ7j7tdVFKaZVYMaUWXDIST69fH865UZAE05GSllcldPrDr/oBSawkQXn94bhASn2/CB06klFbXWEq9Pe/KenFsB/1fZqYXK/vEqTDD296H56DfFxFb+rDNCH7w+eHx6RxSG2luuiYtW3Uh5SGTeL65vLw7+df93uSxnQiB68gAfrD590Gv/i2HxPHeWlrFXDvY0mp0EU0pfFF4ildo3o9rl/1vr+EZx21/w4k3/ocoGcfSfBESxz+Yz8hEoFvXxwfT6sbeB6C0EvPippOPk3TdPtJkNROyeAwzs+KUE0foUzQh3fP4P4r7af7Y00fUDyAv/CIiIjI8DjhERERkeFxwiMiIiLDo8PzM+HD7z8/WoYcHuLs3eHLT1z1F1wWem++69lb0/n3JSwQ6qeAMVrV++qfD9/53r6k99O9rsTXWL+DUDV4R06+0c2nFNZYjkehdvDefPWhb9tclbrgfHiFc2hWWXUZryPc6VQ/r+Jd/4ZwSHBB6LrtSp+dQXgZXY+7rrRNZ+9q3XA8qIzGJq6yXXyH6ptNE/tSFOhY20r+ycV34PXQOZU+/P63kN4IvAfPitta7iMIJKV7mQLl1mXbBwgUJFcmGa/U9puXwcrl0z3jvLhEFJRaQx+niYNXq1ty92W/RhScSOdUww5pHJ5/ezwEcJru8fZujj/PW3DmxO2/KUG15NBtgjDNaZqag0S+Hz3DHsJfeERERGR4nPCIiIjI8DjhERERkeFxwiMiIiLDo7T8xKy+eNu27d981rat3x8aWbefduPz+m1fnhuF4SLfkThG0h5BoVRVyKNVw0mYIyGvSsq0X5U0/1wSjnlYjuQ4EmIxxBBWY28re5OYfdm3UchZ23eXybsocG+OB0GiQEyLqi+Ot3UHjuzypm+j1d7rNhI+cWVpGAObqxLAB23A8QTtP/u+H7OGwFG/0thf0tgp46KKwdM0TXcQgEkSfA15ROET7jVeob1vq6GY9IEDB0b29tfnDB2Px0AvV+XpLYxp6i8KtcO+qAGbtPp7ENL3w76Hf++XML5IdiYpuowL7HuQyAkMdi370kcJl/9CQZm9XAut3GXnSP8e1X/b6F4j4f0h/IVHREREhscJj4iIiAyPEx4REREZHic8IiIiMjxKyyeEBOXt19+0bYuvoNzF4dyTE2+zZN9NFZkxnbdvI7kvEWCpLhI3SRZtZSg5k5Jxk9Bp8tlQBM5WY9/XbSSLEuQTBiIwkaQop0nCJAKTMNyuCQmx8L9Oe2hIlYhp/G5phXAYTzXlmFK6KYF2olXu4ZyqpE5tpbFDEnG9jyjxtsrCP+xHdR1uo+u9A4l1hYnJfd+aCn0DqdA0BqgP6zhPP2ag8Vr7rN2P0zStaPV3+LCDPkroYn82DumcmpBM5w33Mo2Ldn/DM5LGCUn89Iyn/ql8+ByE9OBaUj/fvsj+nalyNv6bBeL3Q/gLj4iIiAyPEx4REREZHic8IiIiMjxOeERERGR4lJZPCAnKyz/8u7bt5lWPjFzdHBpZ17+CGFEAU1BLmi0mDlNdII4RLU0zlC1Jkq1CXpoSTHJfTRFFyRtEQZQ5Qe6r0hy1gWRLEveq9EltpT6kpNcqau5AQiQZmaRulMZru6APWYqGbaV6EoEpSRal7sVhZZSWTGOHuP7s+PWmsbOABN0k1ZqEUroe3IclgRbSvUnWprpYBK37ZWL2LvjQgtq1eXZ8v2mapkV51uHzhPoLxjSOi1KMkn3pAwQSpet9ih92wLM7Sb6++FM25qj9JCjXduAzAO7JOg6nqaf6032L140Gf7ln6GMGelY8hL/wiIiIyPA44REREZHhccIjIiIiw6PDc0JwZXQod/7H/gL//W/gRXbhDsLEzsFbuHl5fDVleudLPgWFmvXVlGm/46FztC1d1Ts9ZqsLA/7CttZilJFIOXfBitrkh3AwINRf3m2jAwOeFZ0jnhMFytW6KOww+N8pvLbkFsF1q+eJzgWAwYborxUXDvypNfkbwerfHCwKTYDVpiu4Ajm0FR2eIIQRxxw4Nhiw2drVt62uadtx14f6a0VOR+hG1XJ0OuQITfBcruOJPKvqoE0Tj+F6H1FwH0H3La2WXsuhMwlO4/rdvNXk+blwvA/T59VD+AuPiIiIDI8THhERERkeJzwiIiIyPE54REREZHiUlk/Ji+dt0+Z1X4749tNuhW2uDueeFEp19r4f8vp1n7PW8CcS9CgsEOVmCKqq9W1pZfEgdG6aIJQvlNBoledV7YpQpCVxFldVD0ISSVqmclXARGmWvFAKAAvOG1d/DyRyIg0LTGRRPB61FVe5b60IyrDMuQGpdLmtS333MiRAoxBby9D50H7BExpXGw/HOY7hKi2TYB0GbLbzXPYTp+BBfFbUbz3wAwEQxsOPMdbl+UdyMMrmMPZrv5JYjgT3FgrQaV0gwdexQv1F9wfeb8HYT59P7XrT8zw43F/jLzwiIiIyPE54REREZHic8IiIiMjwOOERERGR4VFaDqEU5Sopb968bEW2F31OiUJvKUYrcdM2EhZJ3GvHgyu/JQmNUoKLTLYFgRiTgylFuW4LU1FRkCzlqG8wnRVWrt5SInNpG5031bWBY1ZpEkVXuoxBCjEJmQT1IaYVl2OiYEh9ncjgYaoyybW1rXRtOUEZmhXIoiTKo/RLK15X6TO9RjQ26zXCFbyhMmg/tqNsIxE1lXDrea+hXTROUMSuq2fDWI2k/nvK3Zb0aEy+BlD+L22lexLHHN1HgQi8DBKzp+me1ctL26hda0jDputWt+E50r8NUNfdi1oI9vvIn2z8hUdERESGxwmPiIiIDI8THhERERkeJzwiIiIyPErLIduvv2nbVlMRmUFa3p33OeXds+PzzLvn8wXlKrmREEag0AZNrcmoKI6R8Aftn4rARqIgCb0kizZhjo4H4ttdD8hmKTMQbik1ls67to2aiknLwf+ioBQP20goTARSEj634XWr50R17c57w/aQ0Fv3RYkyFJlJgK3lUgkUhdtAqCexlcZm25fG9Cd9G4mteL/VMZaOkyDRmNpFIjMl+9b6sQ1hwjTJ4FFCNpWhuN/S1/gMJrkdnh+LILUZk9RhTNOzelk+XqAk51Qsb8ejbaEM3mTt8COOh/AXHhERERkeJzwiIiIyPE54REREZHic8IiIiMjwKC0DmKpMlKTlm8+7JXb3vM8pSTROEiNJZEaKyEWiHaV3osQayMck5FGaZpJ8jP0A+5HMWafv1AaS3FACpFToItMub8LrAcVqXYtdL1Rlwmm6R06sknr4vzGYJg1t7Qm3vRNX0Bc4nhIRMaSN6yAteZruETdJgi/9g+OJ2kUJwFVSD6VfPsDhn5T4jRZ8OlyDthIoqJZ9aUyToEzUexKFd7hvMdmXnon1fgivByY5B+M8FYGbrB2kwE/TfYn+veDuLEjmp2cK7Vb/baC2BrLzNE3twxdqu0nLIiIiIgUnPCIiIjI8TnhERERkeHR4Qja/+7Jtu357+LJ+c9nnj5ur/qJz/aG/i7z95LDc9jx74Z6s4EyhcDjVxXfi8N40aNoCguKSlasxII/e+VL767tuep9P4Y1rOMfAcdqBy4LAMZfbulx6L7N53jcu7yhospQLVsCepnw16OpKLDe9ss0zaCu5E+W6ofdBfgVco/WHsro1BRbCebe+v6f+2tdp8CDRXANypXBH2FSOSW1fwDnSOKfxVPu/ja9pmpYLSsXsm6p3Qb4R3mvkXpX66ToSeI2ClMQ9PPtir6f+yxqujE4BmLSt1QX9lTwjp2ma9tVNhPsb/x2g5yu1o5aB+tHPCcI6DR4UERERKTjhERERkeFxwiMiIiLD44RHREREhkdpmXjRU+3uXnZDq0rKFIJE225edWnr7pPjAh4JyiRtJZIbke5XpbY9SI0omKE4ezyAD4U5CvvaHm8XNQIlVpDoWhvS0DyScIOUOQwZpHbVoEkK1gsD+JIFiantdDppsGFSF68kXsYO1UUru5OES/Ju6Z+7F9l5Y7Ba7evw/iD5uI5rKkMNoz7Ee6vKoiSiQl0USFk/VMAwvHQF8jbOIUQvDXTEoL4i797Cc4HOkQJCa0hiELp6X7tqX6PQHQbJ4qrtBb4/ejnMD6xtpTbQMyz40CINuH0If+ERERGR4XHCIyIiIsPjhEdERESGxwmPiIiIDI/Scsh+BatB70qK6EWfP+Kq2+vjgmSaEozibNmWrPw9TSzDcRpoTebM6qe21mPu09RjTL2tMmcvQyJzuhp0q4s2kgicrNYcCtC4OnCQNooyZyijtm2hDIkCdB2bYUpwIj8uSDxOU6dhvEb7UTEU6ku7SFhFC/T4/cByfgfFUxr7wf8G0zFRii7iehqMS0HOSUo3tZ0SplH0Lmye9wtCKcT4sUctRn0TPtdawjt9zPCY+6iKxr3I7A9hYjE7gfYj4f0B/IVHREREhscJj4iIiAyPEx4REREZHic8IiIiMjx/89Ly6ou3bdvmzcu2bXdGiaeH20hC24HsfNurj0RjFFYDCw2FNpJ+w9FQ900Th1eQXNok1lCmxsTQ0q5U/CbonJqoGZ747vJ4si+mHlMgaSAap9cjSV39oeDhn7tnqZA+LyUYJdxAdMTE3sdQx3komeJ9WquGe3L1gZJ9g2OGbagC8TRl4iymC4O8i88ZKJdAKd3tvJNY8OmecUE3V0l5x1Ro6msar6UudOfTBOi2I+1HNyA0KxDqU5kaz7vY5vhRAj2Dg4dWcMmO4i88IiIiMjxOeERERGR4nPCIiIjI8PzNOzy0Mvrmee+Wi//XU7tu/+7wBXuy4vk0TfesNFzLZAF89P6+7poE/k3TNE1pGFoNAKP6Yb8NuCytreGK8/j+u7YrDOkj9yD1A1oReq+NPsXxyjBILxgDeD4Y3AfHnPl+nUj6Iln9/YdydIBkR9gvvk2PL/28X1Igaa+rh2JCgN0lNSJZPhuOB6ugp0F0Sf+k9ddSqU9D7k8d1zh+6RzDgM3WDFzhHMYAeSrln4tt6sVgQGiwBHkabEg+ZHUA0RWFbfBcaCGG4SrxtOL8VJ4NqDzp8IiIiIgc4oRHREREhscJj4iIiAyPEx4REREZHqXl7961TZvnb9q2D2+7aXXz8nC+SMFeFKCFoYJVyCO5LxB1fyh2XO7D/UjkowC+QBTDMtjWcjyS3CgAjA4a2Np7kA7TlcTbouG0sjucN60m30RBqItCtSIBPVwhnMMVqf7StnDsIGjJliJw4iw3H5d3o/0mXkm8StcseYfjqUi42C4Aw++C1dFRPifpOjCU00BHCpmrY5/HF9SVrF6P43CeAP3nraWu402YpntWXk+ekdT+QOqm649tCFclT4IyMaR0H4QKkl+NH8IEH+k84gOEfz3MxxUXERER+eXhhEdERESGxwmPiIiIDI8THhERERmevylpefFPf2jbbl90q5hWRt9ckvhWylzBQUG02kLicEK+CnZJdSVpL115HazlKlyiZIqr5B6vi4TuVNbu+1EyLpQjmZPuDBIDKyDfkcyerCKdyokkYrf9ILV5ClNv62rvsaSJonQSVx1uqzJnmKhLleE1Cq43SsWBkMwrTc9rA4HiNK6qTmPseLsQOKd9XYGcxgSlNqfierAfCtDU1vpsCJPasX/KdaNhj/f3xy7//UAbqP7kXx58xjxCsu8HgP1Qbg4+SlBaFhERETnECY+IiIgMjxMeERERGR4nPCIiIjI8f1PS8vJ/f9O3/e7Ltu361UXbRjLt5tnxY26fhUmsRSjEJNBUmKuBwyAFYsonbFtukjThTPrdQyptO0+S3khGJVoUclBmyuXBmmCNbSUxlK5lJF1D9Rf9AK2t1AYSN0N3vkqMKNzStkTAxIDYrGFN/CaBmJJ3QxE4kcFRZE/akYrZRK2K5FT6aOCWbvDg/k779Y6iqMt+Z8H4naZpt4Zyta9pSNOzjtoapLA/Ss6v50n3AmzancN53wX3Xziekn1jmRpl7fI3fiQC2+jDmnq5qW8+Uur3Fx4REREZHic8IiIiMjxOeERERGR4nPCIiIjI8AwtLa++eHvw9/U/ftXKbC+7kXfxpy5Hvfuyl6siMyUoY3IwgAnAtQxdrUR0TGVnStmF9leBbZfIndOEo60KcpjyGaY2N0ESk6NhU5gU3WRgumZJ0uh0jzzYCj1iWy1CSankmGKSczXqjx/vh8pgW21HKk5vemPxGgWkqbHteifnc0+5RRucUFco1Ld9wyRyEt6z6N1sGwnJzQ1+hKQeSeQp9CFEHZr03ElTiGtd4f0Xi8YBsdwcXCP8EIJk8FURrMP7Az96qJtmhlD/Nf7CIyIiIsPjhEdERESGxwmPiIiIDM/QDs/uq0OHZ7/qLwFvXvU5393zXu72Ra9/e1FW+obe3IWhc+19axxC1tvaMs4eEToXhWPB+914peHkXTftBr5A64vQ4cG6MHSsbEtdkCTADFcRp1A7CFabu7o1sIfAt2is4PUOwgjRbaCgTBpkAelu2GdBR6I/E4T54fLZ8y5cC2CcJjxvDNtLxnDodMz+3+c0ZLUl0UGZ0E3EVb3rvUX71WcAtYvaQY4KPRcSdxCvI1yQ2SuvZ0GyWK5uovFF543jMAiahDDCh/AXHhERERkeJzwiIiIyPE54REREZHic8IiIiMjwDC0tL643B3/fvO6ne/1Zn/NR2F4SKpgGoSVBVQj5ciSZ1tWtU7k2PGaT1UiKJkiii4ITSXKDclXcI5kz9fhIMqz7PqZfa/0oGFIbgrGT/m9MOg5r06itaV31HqHzgfoXMM6bIJmOk/Sy1XPC8w7rr+eJ43BeON3+gvoQdkuvUS1HkikFqlJdyThPxdZVIKiiQBymdSbjPA2frFDT048l2rMoFN7p34YEOm/q+7RcJRTLo4DNj8RfeERERGR4nPCIiIjI8DjhERERkeFxwiMiIiLDM7S0vH1xcfD3LSQok6C8uezbKGmySso7kgfTVZHnCrCUvFtXFU5X2E5X5a3TZBLOYHXraGXpVOZMzolWV6YVfuNA0sN99yTXUsouJQ5XyTAUDDHotXZ16iqGSaxNGKYxB4L4AiR+6rO2H41D6FfIm+37zU2bBfA6xgJ3kBKMEmgg9NLxML04HK9JGnaaFB3J2gAJvcnxdlm6Oo2xdnnTttIzZRk8K9J/B5IEeequue1P+2sD5cq4o+dCtDL6NDUBGvvrI+9vf+ERERGR4XHCIyIiIsPjhEdERESGxwmPiIiIDM8vUlpe/NMf2rbd2apt+/b3Vwd/by+64LQ97/Vvr7pBRUnLSbJyE4in6Z7kz5mgABYIktB2SrNNQPExdMn2lCxa66d2zZRRF5AEuoBpP/mprR9D23lJwl9pP0qB1DdBIPOCBHvygNfbo+1C4OIuL7K6lue7o2UIHGPb431Iwm18zEUVMPtAwbEJ/bPvg6fvB+3ne+twWyzdw3MHZdp6TlQG7qOsX0OZmpK12/0A/QXP2z09jOix3IqF/RV87EHXEaG+KH2932bjMBLSifDjFerrXhfsNzetmq5Z+qHNXw7zUaVFREREfoE44REREZHhccIjIiIiw/Ozd3hWX7xt2/bXPfHo5tfP+rZPD+dzm+e9/rsXx1dBn6Zp2pNjUbfR9JHe8SevuuMVZSlMLFmxNvMFiPZOPHwniy4LvOPt7YKNQXBY6jZUV+PPW4NN4btuaGvtC2wBqQf0Hrv2NaksoaeE78Rrv2K4IhyTggfrLUN9A+N3C95Cq5/GdG8Wez1UrrBK7qtpmnYQyEbn2doQim/L6halXgk1YXn8mEsIAdxDV9D1TvaLx9NZdVnC8w6PWZ+lc8c0Ej5iqK8rmHUZOU98zOS5P99BCvcLvNb4ej+Av/CIiIjI8DjhERERkeFxwiMiIiLD44RHREREhudnJy1XSfn2H37bynz/m54WuIFQwbtPSpnnmTS7uwQBDEPNyt9JmekeASxZVRglUwo+q8eDuohQil4uijwIZVbn3dzc7XpDVonomMqctV8ftTI6taNsgLGzh3MkqijNcm0WalfPk/q5iq7TNE07um5BoBxdD+pXkufn9v8qWQI+vN68cnUgsdIhYb/lBfR1vSbBfXXfMVsbKLCQyoVjIGFFci0FLpZtC7pnwnHYn5HHjzdNHLBJ1Ht3SW0I9iP4wwgoF4QFLuHZis+FmdeIoL7YUQBiuefT8UXPp9aujxSU8TiPrkFERETkZ44THhERERkeJzwiIiIyPE54REREZHh+dtLy/s1nB3/fvO6xx5urLkLdvuzbqqS8o5XRaRX0dIXzItwuQBIjCQ3FzSA9tUm599CE2DDJNGWueLoGebDWtYPzpoRbkgCr5EaJvakAzWJulYMp8hRWDQ+kTKorvd6V3S7bj/5vh9pRxw/JtSlzx05dIXyasmtJ4yRNZ23J3bQ6dyijLmFcZG2Y12GxJDuzD9Ny25KOu4K+x/soqJ+fAVFVXP9q3jXawf1Q711MNU8vbZKEDG3AeznoszTh/Qzk6TpeKUj/lM+6dGz+63FmHUVERETkF4QTHhERERkeJzwiIiIyPE54REREZHh+NGm5JihP0zTtvurbNleHkvL7N31ORoLy7UuSj0vdL0BiXYOASenIF11oWwbyI4mzKEDXImR7AZRIWkUuTB8NReZErEMxNJTJ5kqZcyG5L6VL0ZRaeny/aerXbQ2J05iYDOOplsOUWoDkwTU8ERIRNBUdk/2IuTJqIp/fVy7JOab7KJFFH9Ou5N7K08lJ7K9lSEamsdnLrcpzjK7jEp63mMoejGtMk8brke2b1FXPkeqaex3vO2aFpd+sD5Oxwtdt7kcV8z4cmdv2v8ZfeERERGR4nPCIiIjI8DjhERERkeH50RyeGig4TdN0/faqbfv27w6bdPspBAo+g3fKsG13Xt6j0uqx4E4sz8HXwQDBw20r8IHSd591G73fXZMPNBNaxZZWrCXaeZPHELosPcwPVuCd2a7UgSGSd8Nb8gyg/qQcHW+9hlWRgRqYRnXh+YT/u1PHSjpOiMwvO+4ppWyhfvZDjofOxTlxJ/zfyHlm0T3nSP0a+H2YkwohoqvEgQmfO4vF8VA7gp4Vm0V2Qerzle7bNTmfwKbsewbji8YmnWHiitK1xRXIYd/k35V01fME6sOkL+h8PrZd/sIjIiIiw+OER0RERIbHCY+IiIgMjxMeERERGZ4HpWUKC9x+/c3Rctf/+FUrc/eyH4pCBW9eH0pImysQlGHbjlY9Pz+UoxYgI9PKyeszWtUbyhX5isqQaEXb1kGoViodnhIWCh+xJPFRshWLSWCsfXF5dnu0zH11YcuKiLheZVIxSd1JH6aSXi2Hq1aDgPmYMLSEpH46R5S8QZJNjkcPuLlCLBEuTD/VjMptvJL1zOBBeFbM/biA9ptL2l90zNqutL9W4Ur1tW1rEI0ha5Sf+7vD/qfzWdKq4TMf59Sv58E9k5KHdR5C7aJzvAg/9ngs/sIjIiIiw+OER0RERIbHCY+IiIgMjxMeERERGZ6Hk5ZfPG+bVlMXmW//4beHf7/q1b77oi8pe/2rfsjtxaHltHkOstQFCL60mnlJVsaEY5CFz0BaJmG4Sm1n4SrVidycpJZO0/zE5FOm5aZ1kYT21JJsJe0vKrcrY+AxfZhwimTRv3ABAmZaV21HTZFN94sJUo+nKUuATtNsqaVJ71D9c887lTTpWXRK6hmlfTO3D1OihGmSlsPE4blQ/TQu2n6hodw+SsiaheeYtPVpP4NhqL/qv6+neN76C4+IiIgMjxMeERERGR4nPCIiIjI8TnhERERkeB6Ulm+/et223bzu0vL1q2Up0wWk6zPjgYsAAAYOSURBVM9BwLwAgepZEfLOYdl7SExeBonJK5CKL87uel1gbZ2vezprLbYGmXBu+vISlLMd6GQk6a0Xp5Mak4TNVDw+CxJP08TbKG02rIv6lfr/dnco3qcp2sk1ojYQ6bhIWME4wQTrUv8ZJdCGGuhmf/ismJsmTcdM++Ex4vqcMtMU3jPQr3MToB+T2vyUMv7cjyymaX7y7tMmwzP9E53OKZPOH/PcPCvb5o5pqj9NUqek5eSjhI/9iMNfeERERGR4nPCIiIjI8DjhERERkeFxwiMiIiLD86C0/H//42Xbtj3v5bZXh3/fQTry9nkX8hbPING4pCOfX3Sp+AyWva+pjNM0TauSrEyyJSVPUjkSgWviKUnLj5H02n4nzQedz1xJlpgrnlJfVCEW9wvFN0oTvlofjkWqi/Y7o4TvQBRM21qlQ+Kk4/AR8uDZdHhvnbJdaQI0XQ9sxxSIxsApReMtjOnkniEhPaW2n5Kdnzo1nUg+enhq5l7b9MOIhFRIn71veIpz27GGj4fm1rWAe/kh/IVHREREhscJj4iIiAyPEx4REREZngcdnu/+DbwHhoC//WUJ+Lvq71ovznpwH61KXv2Zc/B1riAskPyZ+o4UVzyHbefL3lbyCpKAvzW8d97Re/lS12aXRFfd45EELgsx16fAlcWfONyN+vBydVfKZPthqB2M8zqe6jWbpmk6h66f66TMXc187orqVFcKhivCvVU9m1OuCE99n9xr91H3TYM/ibnXaD0db+spwxunqZ8nuibwXKN+pf5P9iOSunC/mV5g6t0k9T/mnkw4ZVjkUz8/0rFZnxX0PPlY/IVHREREhscJj4iIiAyPEx4REREZHic8IiIiMjwPSstnv37ft4FoXFccvwRB+RzC/C5Wvdxl2UbS7zlso3I1VJCEM17xOpOjKLSw1TVTcqP97kDaS0S+VGJOj5lA7UKJeHFckEyFvEiAPqGIeEpIgKfrloSVnTLkbJpOK0TWezcVXef2RSr0cp9tjpaZK2A+5YrkH8NcaTxv/+nCAnf7x0urf+GUsnxSP4ViPvUYSINRa7lT9s2j6sq+2/ko/IVHREREhscJj4iIiAyPEx4REREZHic8IiIiMjwPSsv//tdft22X655yXOVBkpEpvfgMhMUqH58tKNGzi1ArSCT9scXAx0hvd/vjhtYF1L8N5qxpajNJmRfBftsw0Xhu/5CQTud0ysThhJ9C7nvqYxK1/rnp4feV6/v1ZwyRjute/+meC6lMvVvN+1Dhx5bnqR2PacPcupK+oOdO8iFJekz6YGN2/eFQPfU5zWlHOg7nfggxt65TtMFfeERERGR4nPCIiIjI8DjhERERkeFxwiMiIiLD86C0/J8+/19t2wqTcHdHy5wturScyIMkIyeibgrJwiRWbjFldfnRZe5jW8qlcl/db5r6Oe2WpxUfa/9TemeyHxFf71UXW2nfp4TaNbcNjxnTP/Z5n5LHCMRPLaUn/FzGwI/N3OdyKuKfkrStp7xuP9e66COgek1+ivOZe8yPvWd+OXeYiIiIyEyc8IiIiMjwOOERERGR4XnQ4fkvn/zPWZXSe1ryW4haLvVp7vb9VGo5KnMLDk9SF7WN6trBnDLxbtI2EHPdImoX8ZSO0GPC12r7V+Gq9+l5J/VRXbRfPc9TtzVpw1we019zr9FPQXKeyTneV+4peUwb5vqEc4PoUubef/J40nF+yrpquVPcV44OERERGR4nPCIiIjI8TnhERERkeJzwiIiIyPA8KC3/h/PLWZVu97BKMghtd3sIQZq2pUwPmLuDum73VH/dj2TnPue7Bvn4GiTiKhZf78+gXVRXL1frokBEkrbmhn2lgU1JMOPcVbFPTSp1V0650jdBdVUorPOUJG1I25H2M9X1lNcovT7ptU377KfmlP1MPOZeTvrwMdctYW77qQ/Tvq7lnrqux3DK+ufW9dRj+C/4C4+IiIgMjxMeERERGR4nPCIiIjI8TnhERERkeBZ7kH1FRERERsJfeERERGR4nPCIiIjI8DjhERERkeFxwiMiIiLD44RHREREhscJj4iIiAzP/wfEY39V6lGjrwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.activation_stats.color_dim(-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is just what we hope to see: a smooth development of activations, with no \"crashes\". Batchnorm has really delivered on its promise here! In fact, batchnorm has been so successful that we see it (or something very similar) today in nearly all modern neural networks.\n",
    "\n",
    "An interesting observation about models containing batch normalisation layers is that they tend to generalise better than models that don't contain them. Although we haven't as yet seen a rigourous analysis of what's going on here, most researchers believe that the reason for this is that batch normalisation add some extra randomness to the training process. Each mini batch will have a somewhat different mean and standard deviation to each other mini batch. Therefore, the activations will be normalised by different values each time. In order for the model to make accurate predictions, it will have to learn to become insensitive to these variations. In general, adding additional randomisation to the training process often helps.\n",
    "\n",
    "Since things are going so well, let's train for a few more epochs and see how it goes. In fact, let's even *increase* the learning rate, since the abstract of the batchnorm paper claimed we should be able to \"train at much higher learning rates\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.191731</td>\n",
       "      <td>0.121738</td>\n",
       "      <td>0.960900</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.083739</td>\n",
       "      <td>0.055808</td>\n",
       "      <td>0.981800</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.053161</td>\n",
       "      <td>0.044485</td>\n",
       "      <td>0.987100</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.034433</td>\n",
       "      <td>0.030233</td>\n",
       "      <td>0.990200</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.017646</td>\n",
       "      <td>0.025407</td>\n",
       "      <td>0.991200</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = fit(5, lr=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.183244</td>\n",
       "      <td>0.084025</td>\n",
       "      <td>0.975800</td>\n",
       "      <td>00:13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.080774</td>\n",
       "      <td>0.067060</td>\n",
       "      <td>0.978800</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.050215</td>\n",
       "      <td>0.062595</td>\n",
       "      <td>0.981300</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.030020</td>\n",
       "      <td>0.030315</td>\n",
       "      <td>0.990700</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.015131</td>\n",
       "      <td>0.025148</td>\n",
       "      <td>0.992100</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = fit(5, lr=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, I think it's fair to say we know how to recognize digits! It's time to move on to something harder..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questionnaire"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. What method can we use to see that data in DataLoaders?\n",
    "1. Why do we double the number of filters after each stride 2 conv?\n",
    "1. Why do we use a larger kernel in the first conv with MNIST (with `simple_cnn`)?\n",
    "1. What information does `ActivationStats` save for each layer?\n",
    "1. How can we access a learner's callback after training?\n",
    "1. What are the three statistics plotted by `plot_layer_stats`? What does the x-axis represent?\n",
    "1. Why are activations near zero problematic?\n",
    "1. What are the upsides and downsides of training with a larger batch size?\n",
    "1. Why should we avoid using a high learning rate at the start of training?\n",
    "1. What is 1cycle training?\n",
    "1. What are the benefits of training with a high learning rate?\n",
    "1. Why do we want to use a low learning rate at the end of training?\n",
    "1. What is cyclical momentum?\n",
    "1. What callback tracks hyperparameter values during training (along with other information)?\n",
    "1. What does one column of pixels in the `color_dim` plot represent?\n",
    "1. What does \"bad training\" look like in `color_dim`? Why?\n",
    "1. What trainable parameters does a batch normalization layer contain?\n",
    "1. What statistics are used to normalize in batch normalization during training? How about during validation?\n",
    "1. Why do models with batch normalization layers generalize better?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further research"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. There are other normalization layers available in PyTorch. Try them out and see what works best. Learn about why other normalization layers have been developed, and how they differ from batch normalization.\n",
    "1. Try moving the activation function after the batch normalization layer in `conv`. Does it make a difference? See what you can find out about what order is recommended, and why.\n",
    "1. Batch normalization isn't defined for a batch size of one, since the standard deviation isn't defined for a single item. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "split_at_heading": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}