From e477ef2d85e617a490cb2dd6080bcbe7933c121c Mon Sep 17 00:00:00 2001 From: Jeremy Howard Date: Wed, 4 Mar 2020 12:19:41 -0800 Subject: [PATCH] merge --- 01_intro.ipynb | 2 +- 07_sizing_and_tta.ipynb | 2 +- 10_nlp.ipynb | 1026 --------------------------------------- 13_convolutions.ipynb | 989 ++++++++++++++++++++++++++++++++++++- 4 files changed, 989 insertions(+), 1030 deletions(-) diff --git a/01_intro.ipynb b/01_intro.ipynb index 4ec8c34..089f8de 100644 --- a/01_intro.ipynb +++ b/01_intro.ipynb @@ -947,7 +947,7 @@ "\n", "Let us take these concepts one by one, in order to understand how they fit together in practice. First, we need to understand what Samuel means by a *weight assignment*.\n", "\n", - "Weights are just variables, and a weight assignment is a particular choice of values for those variables. The program's inputs are values that it processes in order to products its results -- for instance, taking image pixels as inputs, and returning the classification \"dog\" as a result. But the program's weight assignments are other values which define how the program will operate.\n", + "Weights are just variables, and a weight assignment is a particular choice of values for those variables. The program's inputs are values that it processes in order to produce its results -- for instance, taking image pixels as inputs, and returning the classification \"dog\" as a result. But the program's weight assignments are other values which define how the program will operate.\n", "\n", "Since they will affect the program they are in a sense another kind of input, so we will update our basic picture of <> and replace it with <> in order to take this into account:" ] diff --git a/07_sizing_and_tta.ipynb b/07_sizing_and_tta.ipynb index 4b0080e..bcf431e 100644 --- a/07_sizing_and_tta.ipynb +++ b/07_sizing_and_tta.ipynb @@ -835,7 +835,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "This all done for us inside fastai by adding a `Callback` to our `Learner`. `Callback`s are what is used inside fastai to inject custom behavior in the training loop (like a learning rate schedule, or training in mixed precision). We'll be learning all about callbacks, including how to make your own, in <>. For now, all you need to know is that you use the `cbs` parameter to `Learner` to pass callbacks.\n", + "This all done for us inside fastai by adding a `Callback` to our `Learner`. `Callback`s are what is used inside fastai to inject custom behavior in the training loop (like a learning rate schedule, or training in mixed precision). We'll be learning all about callbacks, including how to make your own, in <>. For now, all you need to know is that you use the `cbs` parameter to `Learner` to pass callbacks.\n", "\n", "Here is how you train a model with Mixup:\n", "\n", diff --git a/10_nlp.ipynb b/10_nlp.ipynb index b239bdd..aea83d3 100644 --- a/10_nlp.ipynb +++ b/10_nlp.ipynb @@ -2190,1007 +2190,6 @@ "Many people assume or hope that algorithms will come to our defence here. The hope is that we will develop classification algorithms which can automatically recognise auto generated content. The problem, however, is that this will always be an arms race, in which better classification (or discriminator) algorithms can be used to create better generation algorithms." ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "TK make transition if it stays in this chapter or conclusion of NLP chapter" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Data munging with fastai's mid-level API" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We have seen what `Tokenizer` or a `Numericalize` do to a collection of texts, and how they're used inside the data block API, which handles those transforms for us directly using the `TextBlock`. But what if we want to only apply one of those transforms, either to see intermediate results or because we have already tokenized texts. More generally, what can we do when the data block API is not flexible enough to accommodate our particular use case? For this, we need to use fastai's *mid-level API* for processing data. The data block API is built on top of that layer, so it will allow you to do everything the data block API does, and much much more." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Going deeper into fastai's layered API" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The fastai library is built on a *layered API*. At the very top layer, there are *applications* that allow us to train a model in five lines of codes, as we saw in <>. In the case of creating `DataLoaders` for a text classifier, for instance, we used the line:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from fastai2.text.all import *\n", - "\n", - "dls = TextDataLoaders.from_folder(untar_data(URLs.IMDB), valid='test')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The factory method `TextDataLoaders.from_folder` is very convenient when your data is arranged the exact same way as the IMDb dataset, but in practice, that often won't be the case. The data block API offers more flexibility. As we saw in the last chapter, we can ge the same result with:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "path = untar_data(URLs.IMDB)\n", - "dls = DataBlock(\n", - " blocks=(TextBlock.from_folder(path),CategoryBlock),\n", - " get_y = parent_label,\n", - " get_items=partial(get_text_files, folders=['train', 'test']),\n", - " splitter=GrandparentSplitter(valid_name='test')\n", - ").dataloaders(path)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "But it's sometimes not flexible enough. For debugging purposes for instance, we might need to apply just parts of the transforms that come with this data block. Or, we might want to create `DataLoaders` for some application that isn't directly supported by fastai. In this section, we'll dig into the pieces that are used inside fastai to implement the data block API. By understanding these pieces, you'll be able to leverage the power and flexibility of this mid-tier API." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> note: The mid-level API in general does not only contain functionality for creating `DataLoaders`. It also has the *callback* system that we will study in <>, which allows us to customize the training loop any way we like, and the *general optimizer* that we will cover in <>." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Transforms" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "When we studied tokenization and numericalization in the last chapter, we started by grabbing a bunch of texts:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "files = get_text_files(path, folders = ['train', 'test'])\n", - "txts = L(o.open().read() for o in files[:2000])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We then showed how to tokenize them with a `Tokenizer`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "(#228) ['xxbos','xxmaj','this','movie',',','which','i','just','discovered','at'...]" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "tok = Tokenizer.from_folder(path)\n", - "tok.setup(txts)\n", - "toks = txts.map(tok)\n", - "toks[0]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([ 2, 8, 20, 27, 11, 88, 18, 53, 3286, 45])" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "num = Numericalize()\n", - "num.setup(toks)\n", - "nums = toks.map(num)\n", - "nums[0][:10]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And how to numericalize, including automatically creating the vocab for our corpus:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([ 2, 8, 20, 27, 11, 88, 18, 53, 3286, 45])" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "num = Numericalize()\n", - "num.setup(toks)\n", - "nums = toks.map(num)\n", - "nums[0][:10]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The classes also have a *decode* method. For instance, `Numericalize.decode` gives us back the string tokens:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "(#10) ['xxbos','xxmaj','this','movie',',','which','i','just','discovered','at']" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "nums_dec = num.decode(nums[0][:10]); nums_dec" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "...and `Tokenizer.decode` turns this back into a single string (it may not, however, be exactly the same as the original string; this depends on whether the tokenizer is *reversible*, which the default word tokenizer is not at the time we're writing this book):" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'xxbos xxmaj this movie , which i just discovered at'" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "tok.decode(nums_dec)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "`decode` is used by fastai's `show_batch` and `show_results`, as well as some other inference methods, to convert predictions and mini-batches into a human-understandable representation.\n", - "\n", - "For each of `tok` or `num` above, we created an object, called the setup method (which trains the tokenizer if needed for `tok` and creates the vocab for `num`), applied it to our raw texts (by calling the object as a function), and then finally decoded it back to an understandable representation. These steps are needed for most data preprocessing tasks, so fastai provides a class that encapsulates them. This is the `Transform` class. Both `Tokenize` and `Numericalize` are `Transform`s.\n", - "\n", - "In general, a `Transform` is an object that behaves like a function, has an optional *setup* that will initialize some inner state (like the vocab inside `num` for instance), and has an optional *decode* that will reverse the function (this reversal may not be perfect, as we saw above for `tok`).\n", - "\n", - "A good example of `decode` is found in the `Normalize` transform that we saw in <>: to be able to plot the images its `decode` method undoes the normalization (i.e. it multiplies by the std and adds back the mean). On the other hand, data augmentation transforms do not have a `decode` method, since we want to show the effects on images, to make sure the data augmentation is working as we want." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The second special behavior of `Transform`s is that they always get applied over tuples: in general, our data is always a tuple `(input,target)` (sometimes with more than one input or more than one target). When applying a transform on an item like this, such as `Resize`, we don't want to resize the tuple, but resize the input (if applicable) and the target (if applicable). It's the same for the batch transforms that do data augmentation: when the input is an image and the target is a segmentation mask, the transform needs to be applied (the same way) to the input and the target.\n", - "\n", - "We can see this behavior if we pass a tuple of texts to `tok`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "((#374) ['xxbos','xxmaj','well',',','\"','cube','\"','(','1997',')'...],\n", - " (#207) ['xxbos','xxmaj','conrad','xxmaj','hall','went','out','with','a','bang'...])" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "tok((txts[0], txts[1]))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Writing your own Transform" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If you want to write a custom transform to apply to your data, the easiest way is to write a function:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "3" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "def f(x): return x+1\n", - "tfm = Transform(f)\n", - "tfm(2)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "`tfm` will automatically convert `f` to a `Transform` with no setup and no decode method. If you need either of those, you will need to subclass `Transform`. When writing this subclass, you need to implement the actual function in `encodes`, then (optionally), the setup behavior in `setups` and the decoding behavior in `decodes`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class NormalizeMean(Transform):\n", - " def setups(self, items): self.mean = sum(items)/len(items)\n", - " def encodes(self, x): return x-self.mean\n", - " def decodes(self, x): return x+self.mean" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here `NormalizeMean` will initialize some state during the setup (the mean of all elements passed), then the transformation is to subtract that mean. For decoding purposes, we implement the reverse of that transformation by adding the mean. Here is an example of `NormalizeMean` in action:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "(3.0, 5.0, 2.0)" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "tfm = NormalizeMean()\n", - "tfm.setup([1,2,3,4,5])\n", - "start = 2\n", - "y = tfm(start)\n", - "z = tfm.decode(y)\n", - "tfm.mean,y,z" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Note that the method called and the method implemented are different, for each of these methods:\n", - "\n", - "```asciidoc\n", - "[options=\"header\"]\n", - "|======\n", - "| Class | To call | To implement\n", - "| `nn.Module` (PyTorch) | `()` (i.e. call as function) | `forward`\n", - "| `Transform` | `()` | `encodes`\n", - "| `Transform` | `decode()` | `decodes`\n", - "| `Transform` | `setup()` | `setups`\n", - "|======\n", - "```\n", - "\n", - "So, for instance, you would never call `setups` directly, but instead would call `setups`. The reason for this is that `setup` does some work before and after calling `setups` for you. To learn more about `Transform`s and how you can use them to have different behavior depending on the type of the input, be sure to check the tutorials in the fastai docs." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Pipeline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To compose several transforms together, fastai provides `Pipeline`. We define a `Pipeline` by passing it a list of `Transform`s; it will then compose the transforms inside it. When you call a `Pipeline` on an object, it will automatically call the transforms inside, in order:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([ 2, 8, 76, 10, 23, 3112, 23, 34, 3113, 33, 10, 8, 4477, 22, 88, 32, 10, 27, 42, 14])" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "tfms = Pipeline([tok, num])\n", - "t = tfms(txts[0]); t[:20]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And you can call decode on the result of your encoding, to get back something you can display and analyze:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'xxbos xxmaj well , \" cube \" ( 1997 ) , xxmaj vincenzo \\'s first movie , was one of the most interesti'" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "tfms.decode(t)[:100]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The only part that doesn't work the same way as in `Transform` is the setup. To properly setup a `Pipeline` of `Transform`s on some data, you need to use a `TfmdLists`." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## TfmdLists and Datasets: Transformed collections" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Your data is usually a set of raw items (like filenames, or rows in a dataframe) to which you want to apply a succession of transformations. We just saw that the succession of transformations was represented by a `Pipeline` in fastai. The class that groups together this pipeline with your raw items is called `TfmdLists`." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### TfmdLists" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here is the short way of doing the transformation we saw in the previous section:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "tls = TfmdLists(files, [Tokenizer.from_folder(path), Numericalize])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "At initialization, the `TfmdLists` will automatically call the setup method of each transform in order, providing them not with the raw items but the items transformed by all the previous `Transform`s in order. We can get the result of our pipeline on any raw element just by indexing into the `TfmdLists`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([ 2, 8, 91, 11, 22, 5793, 22, 37, 4910, 34, 11, 8, 13042, 23, 107, 30, 11, 25, 44, 14])" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "t = tls[0]; t[:20]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And the `TfmdLists` knows how to decode for showing purposing:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'xxbos xxmaj well , \" cube \" ( 1997 ) , xxmaj vincenzo \\'s first movie , was one of the most interesti'" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "tls.decode(t)[:100]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In fact, it even has a `show` method:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "xxbos xxmaj well , \" cube \" ( 1997 ) , xxmaj vincenzo 's first movie , was one of the most interesting and tricky ideas that xxmaj i 've ever seen when talking about movies . xxmaj they had just one scenery , a bunch of actors and a plot . xxmaj so , what made it so special were all the effective direction , great dialogs and a bizarre condition that characters had to deal like rats in a labyrinth . xxmaj his second movie , \" cypher \" ( 2002 ) , was all about its story , but it was n't so good as \" cube \" but here are the characters being tested like rats again . \n", - "\n", - " \" nothing \" is something very interesting and gets xxmaj vincenzo coming back to his ' cube days ' , locking the characters once again in a very different space with no time once more playing with the characters like playing with rats in an experience room . xxmaj but instead of a thriller sci - fi ( even some of the promotional teasers and trailers erroneous seemed like that ) , \" nothing \" is a loose and light comedy that for sure can be called a modern satire about our society and also about the intolerant world we 're living . xxmaj once again xxmaj xxunk amaze us with a great idea into a so small kind of thing . 2 actors and a blinding white scenario , that 's all you got most part of time and you do n't need more than that . xxmaj while \" cube \" is a claustrophobic experience and \" cypher \" confusing , \" nothing \" is completely the opposite but at the same time also desperate . \n", - "\n", - " xxmaj this movie proves once again that a smart idea means much more than just a millionaire budget . xxmaj of course that the movie fails sometimes , but its prime idea means a lot and offsets any flaws . xxmaj there 's nothing more to be said about this movie because everything is a brilliant surprise and a totally different experience that i had in movies since \" cube \" .\n" - ] - } - ], - "source": [ - "tls.show(t)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `TfmdLists` is named with an \"s\" because it can handle a training and validation set with a splits argument. You just need to pass the indices of which elemets are in the training set, and which are in the validation set:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "cut = int(len(files)*0.8)\n", - "splits = [list(range(cut)), list(range(cut,len(files)))]\n", - "tls = TfmdLists(files, [Tokenizer.from_folder(path), Numericalize], splits=splits)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can then access them through the `train` and `valid` attribute:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([ 2, 8, 20, 30, 87, 510, 1570, 12, 408, 379, 4196, 10, 8, 20, 30, 16, 13, 12216, 202, 509])" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "tls.valid[0][:20]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If you have manually written a `Transform` that returns your whole data (input and target) from the raw items you had, then `TfmdLists` is the class you need. You can directly convert it to a `DataLoaders` object with the `dataloaders` method. This is what we will do in our Siamese example further in this chapter.\n", - "\n", - "In general though, you have two (or more) parallel pipelines of transforms: one for processing your raw items into inputs and one to process your raw items into targets. For instance, here, the pipeline we defined only processes the input. If we want to do text classification, we have to process the labels as well. \n", - "\n", - "Here we need to do two things: first take the label name from the parent folder. There is a function `parent_label` for this:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "(#50000) ['pos','pos','pos','pos','pos','pos','pos','pos','pos','pos'...]" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "lbls = files.map(parent_label)\n", - "lbls" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Then we need a `Transform` that will grab the unique items and build a vocab with it during setup, then will transform the string labels into integers when called. fastai provides this transform, it's called `Categorize`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "((#2) ['neg','pos'], TensorCategory(1))" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "cat = Categorize()\n", - "cat.setup(lbls)\n", - "cat.vocab, cat(lbls[0])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To do the whole setup automatically on our list of files, we can create a `TfmdLists` as before:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "TensorCategory(1)" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "tls_y = TfmdLists(files, [parent_label, Categorize()])\n", - "tls_y[0]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "But then we end up with two separate objects for our inputs and targets, which is not what we want. This is where `Datasets` comes to the rescue." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Datasets" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "`Datasets` will apply two (or more) pipelines in parallel to the same raw object and build a tuple with the result. Like `TfmdLists`, it will automatically do the setup for us, and when we index into a `Datasets`, it will return us a tuple with the results of each pipeline:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x_tfms = [Tokenizer.from_folder(path), Numericalize]\n", - "y_tfms = [parent_label, Categorize()]\n", - "dsets = Datasets(files, [x_tfms, y_tfms])\n", - "x,y = dsets[0]\n", - "x[:20],y" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Like a `TfmdLists`, we can pass along `splits` to a `Datasets` to split our data between training and validation:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "(tensor([ 2, 8, 20, 30, 87, 510, 1570, 12, 408, 379, 4196, 10, 8, 20, 30, 16, 13, 12216, 202, 509]),\n", - " TensorCategory(0))" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "x_tfms = [Tokenizer.from_folder(path), Numericalize]\n", - "y_tfms = [parent_label, Categorize()]\n", - "dsets = Datasets(files, [x_tfms, y_tfms], splits=splits)\n", - "x,y = dsets.valid[0]\n", - "x[:20],y" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "It can also decode any processed tuple or show it directly:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "('xxbos xxmaj this movie had horrible lighting and terrible camera movements . xxmaj this movie is a jumpy horror flick with no meaning at all . xxmaj the slashes are totally fake looking . xxmaj it looks like some 17 year - old idiot wrote this movie and a 10 year old kid shot it . xxmaj with the worst acting you can ever find . xxmaj people are tired of knives . xxmaj at least move on to guns or fire . xxmaj it has almost exact lines from \" when a xxmaj stranger xxmaj calls \" . xxmaj with gruesome killings , only crazy people would enjoy this movie . xxmaj it is obvious the writer does n\\'t have kids or even care for them . i mean at show some mercy . xxmaj just to sum it up , this movie is a \" b \" movie and it sucked . xxmaj just for your own sake , do n\\'t even think about wasting your time watching this crappy movie .',\n", - " 'neg')" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "t = dsets.valid[0]\n", - "dsets.decode(t)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The last step is to convert your `Datasets` object to a `DataLoaders`, which can be done with the `dataloaders` method. Here we need to pass along special arguments to take care of the padding problem (as we saw in the last chapter). This needs to happen just before we batch the elements, so we pass it to `before_batch`: " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "dls = dsets.dataloaders(bs=64, before_batch=pad_input)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "`dataloaders` directly calls `DataLoader` on each subset of our `Datasets`. fastai's `DataLoader` expands the PyTorch class of the same name and is responsible for collating the items from our datasets into batches. It has a lot of points of customization but the most important you should know are:\n", - "\n", - "- `after_item`: applied on each item after grabbing it inside the dataset. This is the equivalent of the `item_tfms` in `DataBlock`.\n", - "- `before_batch`: applied on the list of items before they are collated. This is the ideal place to pad items to the same size.\n", - "- `after_batch`: applied on the batch as a whole after its construction. This is the equivalent of the `batch_tfms` in `DataBlock`." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As a conclusion, here is the full code necessary to prepare the data for text classification:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "tfms = [[Tokenizer.from_folder(path), Numericalize], [parent_label, Categorize]]\n", - "files = get_text_files(path, folders = ['train', 'test'])\n", - "splits = GrandparentSplitter(valid_name='test')(files)\n", - "dsets = Datasets(files, tfms, splits=splits)\n", - "dls = dsets.dataloaders(dl_type=SortedDL, before_batch=pad_input)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The two differences with what we had above is the use of `GrandParentSplitter` to split our training and validation data, and the `dl_type` argument. This is to tell `dataloaders` to use the `SortedDL` class of `DataLoader`, and not the usual one. This is the class that will handle the construction of batches by putting samples of roughly the same lengths into batches.\n", - "\n", - "This does the exact same thing as our `DataBlock` from above:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "path = untar_data(URLs.IMDB)\n", - "dls = DataBlock(\n", - " blocks=(TextBlock.from_folder(path),CategoryBlock),\n", - " get_y = parent_label,\n", - " get_items=partial(get_text_files, folders=['train', 'test']),\n", - " splitter=GrandparentSplitter(valid_name='test')\n", - ").dataloaders(path)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "...except that now, you know how to customize every single piece of it!\n", - "\n", - "Let's practice what we just learned on this mid-level API for data preprocessing on a computer vision example now, with a Siamese Model input pipeline." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Applying the mid-tier data API: SiamesePair" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "A Siamese model takes two images and has to determine if they are of the same classe or not. For this example, we will use the pets dataset again, and prepare the data for a model that will have to predict if two images of pets are of the same breed or not. TK see if we train that model later in the book. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from fastai2.vision.all import *\n", - "path = untar_data(URLs.PETS)\n", - "files = get_image_files(path/\"images\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class SiameseImage(Tuple):\n", - " def show(self, ctx=None, **kwargs): \n", - " img1,img2,same_breed = self\n", - " dim = 2 if isinstance(img1, Tensor) else 1\n", - " return show_image(torch.cat([tensor(img1),tensor(img2)], dim=dim), \n", - " title=same_breed, ctx=ctx)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "" - ] - }, - "execution_count": null, - "metadata": {}, - "output_type": "execute_result" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAASUAAAB8CAYAAAAvkab8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOy9a7Bt2VXf9xtjzrX23udxX31vP24/1C2puyVZAgkhsEBEDsK8MRhQTGEwJk4c48qDqlRwKkBQQtlViSuVD64kxuUPKapiXE5SFWJXFAoIAov3Q+j9QK1+t7r73tv3ntd+rDXnGPkw5trnilitiiDQxGdU3b639zln77XmGnOM//iP/5hH3J0zO7MzO7NXiumf9gWc2Zmd2ZndbmdB6czO7MxeUXYWlM7szM7sFWVnQenMzuzMXlF2FpTO7MzO7BVlZ0HpzM7szF5RdhaUzuzMzuwVZWdB6cwAEBH/An+e+NO+xjP718Pyn/YFnNkrxu657d9fAfxs+/vp9lr9V/2QiPTuPvx/fG1n9q+RnSGlMwPA3Z+f/gAvtZev3fb6NQAReV5EfkJE/pGIvAT8oojMG5r67tvfU0TeLyL/8Lb/70Xk74rIkyKyEpGPiMgP/ond5Jn9mbAzpHRmX4z9x8B/BXwl/+986KeBR4B/G/gM8Hbgp0RkcPf/6Y/9Ks/sz6SdBaUz+2LsX7r7353+R0TmX+gHROR1wF8BXu3uj7eXHxeRNwL/AXAWlM4MOAtKZ/bF2W99ET/ztvb3h0Xk9tczcPJHvqIz+/+NnQWlM/ti7A8HEWt/yx96vbvt3wo4EZzGz/PzZ3ZmZ0HpzP7o5u6DiBwAV6fXRGQHeBT4QHvpd4igda+7/8Kf/FWe2Z8VOwtKZ/bHZb8A/Psi8hvAEvgJbuvuuvtHReSfAP+jiPwI8JvAPvDlwHl3/2/+FK75zF6BdiYJOLM/Lvth4NNEcPoXwHuBD/2h7/kB4H8A3gN8HPh54K8Cj/2JXeWZveJNzk6ePLMzO7NXkp0hpTM7szN7RdlZUDqzMzuzV5SdBaUzO7Mze0XZWVA6szM7s1eUnQWlMzuzM3tF2cvqlHZl4TWvmUtGspOSAIr0wMaopmw8IVpQT3TiCEYSxV2wDNkLJhkRwd1xd4SEophVqoKYkxSoStUBtYyqYgLqBek6cEVsxBAMJ7lgotSxoDlhZqQsqBmGbD9vMqtNbqweP0/G1VBPOJWcla5LRJw23IWUEmjCyggqeDVUQVUZx8p81qFZweOzzCyu2wzNiW6xQz/fY7aYM5vNOH/hIjkpVRJ7+/u4CmKCacHGQi2ODRu6nTlSjOLG+mRJSh1936NdjntqYxrDuAYTVEFMWB6fsHPuPK4FzKnVKaUwrJaIJHZ2dig4fdexXq9RN1wTqoqXynoo0HWwWjKOFc1tDb2yHEZsLBwcHJC6BWk2Z33zGn7jeXKfePZGwcRwHelzT9aKiEBSdDSqOKua2nOp7KSMM6IkkihFFaUgIiiJUh3R8JfkjingHcUrGUc81r1qgSrknCE57pXOOkyMKpVworgPMaW64e6oKklBHMwF0YohYPG1MhqdQEHQBGAIXXifwKxv+VwSIUhXVBVQilQ6lKGMzGcZqwJemc1muApWKi5sPwsgLxZ0/ZxutqDvey7feQVyj1lh0c/QxYKcM2aGlYEyGm4FT5lM+HkdRkopdN0MV2c+32G9XtMtdijrE8qmIIuMFGNYDYgIi/09qoXAvqxGxnFDKYWce/q+J/UddRgZ6oC6xvNUpWwGqmTMCmW9QklIgjJUqhvDsOb46AA8412HzuesPvEhLMX+eeza8R9W/2/tZZGSzwtd7ti4sCnOcgPr0VgdOcvR2Vih1DVmzsYGhuKMLpwUZ4mhrphkMMdKRU1RydRacQqkSp4GDFyxVBHLFI15BHfHU8JKRWxk4y0giWI41IJnxashItjomGpsJPPYANVwExCjaHyOtFjlBUSd5hckV6oXigiWBBfCCVow7rIiCkkys/40nkewFVQEASQLbpUyVKwMqDh9PwM3nAgi63HgeH2M5QyDsKkgObG7u0uSTMUBJQF9nxmGNSerYzxlFvM5pJ4yjHQuZI+NLSJsVis6ncfGq4JaJmtHXR/z/FOfpjis12uqGZhg7kjbTOuTA4aTFWM1HEVniapGl3ehglejLNcMyxWp69Cy3s6HeC5odlwyYzFWg7AahPXaWFZYj7GWtY6MxVgXY1UTqwLHY6FUwy0hHkFdMGpxcKXU8F9xJ7tQWtIpqaKekAylFNwEkUSR2BhuiZTS9nmrRaDU5iOG45oQwF3CT4Dq8cyLKq5xHYbgVNAIcmLxpi5QVLcJrEpsOrdEl3rEjZQiaLo4VGuf53SioAnHKJs1wzAwlg3agk+tlVQNT5my3rAaNyjGUJSchH62oJ9NiTQ+N+cMGCfrI1bjQLfYQZKSTBFzeulIJBTh+OQALNGnBVIU9RlZOxLw4jOf5uD4gHGorMu4PU1LRBBxzODGtRfxCnWM/ec6xzOIWwSpAsdHNxmHyp5nzEfEnOELyJBeVqc067OrxwKaEA89gZrgfwiJOJWUM9qyeCmFLLFIxQtUY9b3zFUoRKAQF6o66qAI1ePmKk5qY1RZFDJI9W2QEAkHcipObMbJ8WqtqCraCVKd6gaueAYtMKjToTgVIZFTBImUpoyXUY0MmZvz1lpRhNwnpF1jBKEIfJITXo1a42GZGeO4od9dcOH8Jfb29pidO8/uufN4GXnp+WdIB9eZqXM8P8+lVz1M3/fklOj6ni71bMY1w/KEW7duMVz7LDvrm4CxkQ45d5l8/jJpNme+WJBzj9WRkxc+y+qpTzGcv8KFex5gZ+8CYCyPDykvPIUdXudkVVhJ4c43vZ1OlINbL7G7c57l+oTxxedYHr3E7M57yfsXqJuCpcrq2kucHN1kfv4KHFxHzVglxUenp5KS8uTBCgBzJzm4nm7yrWkgEdEIhNKcvNaKmTHLHdplymaNamaRFFHHXFAUd8OIn1cLJN0+NAJF84NINEqxeG4QSBkJXzA8vi9FQtOcUDfEnCKB5GmBA8A1nvnkLyIN2UsgPzTQnYgwWEXGisyFzgUkrgsgqWAikVRroVQFHxlGx6xw8c7L7Cz22L14kb0LF7brcvj0E+zUY7wI3PMQ+5ev0HUdSZXcLTAbcDOObt3k4Prz6MFNFnZCIbHJc/TiPVy8dBdFjXnu0NyzOrzJ8VOfYCxGPXeFi3deZWdnhzqOHDz/NPObzzEcD1yzDXrxMnc/+AaWx4d4NVKXOT64SX3xGY6HgSuPfAXr1U2SwrqMHDz9FHmxj+SOtDrAaqGmBayOyKqszXn61vLzIqWXDUp9l1w9ELAQJRJiJBL1toMIVfJ2kwOkHA6pKaBxr1FGuCZ2Zx2UEec2hJME9QalcRDDULJEUMgCnoTkgV68GmUKZLVu4XmRCDjK6WebR2lZ1eirUjR+zs0Q1YDm6mQVVHMEGhJFlTIMjNVhNNI8sdPPQKNEyKKoCifLkflOR+4SRy8tGYYBb87Y7864fOcVFnu7XLh0hXPnzjEOA/WJj7OXYoON6xUvWc/i4S9j99yC8XiNiHCyvMX1zzzOostclCUjwrqOyOAMQ+UWlcuPvJmDp59htMrdr341drLkpWc+xeZ44Ggs3Pnga7l019288NRnWN94gWqGO3jK0M9ZjQN+csxs9xzrUujqgDiUfoaIUldHUBPdPNNZ4nhYs7u3oKuFmjpSrWQpJO144mAT5RYwRYsoNSMQiDouhpeA/yKRhLImRKMkFVe6JNQ6UjzKpHPdDCcQUfXAZQmhKHQuUaLV8KHs8bmaItiZ2TbZWY3XXG5Lsg5ZhCxQxakFcoqva/N3b4FTAERIOYLSaSKLexmrUcfCYI5qYm8e6BNNZDFUhbHEnum6js2msl6uGMZIoqLOxSuX2Tu3z/mLl7lwxyUkJ248+xT3rQ8pmzXMEscHa06uvoZ7HniIzbhmM6xQ73jhsT9gc3yTO7rCrOs5Wa9apaBcPzlk9+ojpNTx/LXPcscdlzl/6Q6OHv8Ym83I0fEKdhbc97o3khCe+ODvIFTEhCrCqjrW99TVBlXwPKcOK/Y6WNeMZcE3I6kokpzdnZ6j1QrmC3YoJFeoCdGRRVZOyh8hKHUpO+2hABg1shtp62wTByMa76MOJCUhjFaDL0rS4Kow351ThgExpfnQKf9jDQ0JW6SEOdq1TEMEPwjU5maUWqkqmApUI6kEQtPEUEbGhqya329r+KjonNm8I7XXRZyTjcW1jyORXA0QJCs7855u3pHMUDGON87x8epz+CsRsFY6qgqX77zChUuXeO75z+IO48mSvayoGINFiduJs0kJcse4WqIWjp4VxhZ8exfGVLb3nkjUWpHGpw1lpOvnaDGKVPDY/GMtdCrx5KpR8cZlRRk8EhyLC1R3cuPscAFxXLTxdI6iIFGmLhYLbDPQdRWn4/rR5v/J41ERV5I4tkWnXfiLSCCmhiQcsFrbczdEYb9fMJbNKXIRmei0eP9SqVnoTeJ5SkNiOdYM10hwDSUN5tt7K270KYMbs6RYhbEWrH2AOqQUCc4lKgFJHbMu0FhKgouyXsUzGcexBeL42L7v2ZkrXU5kV1a1cnSywkvge7eGAq0i0nilnLjvwVcxjiO3Dg+RYthmxU7OFBuDXxNlAGb759kc3Aok7zDvZ2ClrUGrVmSM9a2JtRiZQGmlFMiKetx/adXFVIUgUdo7MLYCPRnB/4rgwSzE9bXg7gQQdgTHkJaY3Sqz2ZyEoFI4P+t5frnh4HjzRSKl3HkEm0YOOkDGveIaG90liGJr5QxA9oTI6fsqRk6AK4ssDJJwC4J6rJWUOsSCKC5EBkQFGoGXRAMiS8UtNuHGCuZtkawGZeoEEZ0EK4aLoxUsMP0WHU2L5uJ0qWPedcyyoFm4dmvJYBLO6I54xVCSxGbp5h07846j5YiPJR6eFkSgum6J3OwpNqUkSM68QX1VJclIQmLtBKop2UtwEZIa56bbTWtWEEmNYTKMinlLDABVtqR9sUqShLmAjkgLcEWc5BboUdgizm0DQgjCHtsGJDzQBklxlEyg0VSNOxY9AMPojBgHa0MxXD24oVaWuwtoBKSpyeHu28/NE3IN5peskCR8yQVmmhmIkqy6xzpYrM/kuoGGakMu8f+xdo5VZWMlSvCWBUsSzGVLEAO4WcuvDrc1SkyFFFQQmip92mFnpiy6zMk4cuNkGShABPFKdSFLBim4JM4tFDfhcF1QH4EZJob42GqLFgQbegRDRQLB5UDunVQUZXTdkvSKsfRC74oZaOpwxlMCvZWTLsaUaiB4tnWq9KMgKQKRV0eTb31DPdCmjU7ViTM9fWZOS1gePiKumBQE3VIvEw1DhXlyLs171o0+2YyVg+XwxQWlnLNP2SfJ6UVN/NK0UaeNuIW5BNJRbTV+y64ToWxTGaZRd689yL9a6zYTx41BpwlzR8Wj21YrtLJtsMjE1pxMW/lmxLUWL6cktIPJRL6f3mOUh8pi3pHEOViN29e1cSPQECAaHSaLbo16oEQTbV9vZYOAWG0dHgFRVILDmEoMaesG1ngPbQEKstB4D2veEeWCEoG9NIcIsr/dTAqepLp9Dgq9/T6tIcaqvuVhIhGGo6lArYa0jLdFPu74hIibsy00sdcleon3fepo2CJqF0guKNbQrwYK01OeJpKWbtch+ykKSm2DisRGERE8B1Fb3GIjeaANF2OWMjSuKXkEbtcIflRj40YtEoEC3aK2rnFBIxbXU+NZUk+fTZX2dBqN4e4susx8PuNkM1KHMZjNieZoazz5ikig0mmzA5/z7+1GFCepoJqi6msBayLlk8aeU6lbX5m4sUBztk0yU/WS6DApVIIzq3bqo0GReARRI5pL7rhD4AfdkvkVO+UDRSOUe1QQE3CYAtXU+fB2n4aTDPZmHedyCgBhxpOHXyRS2pslz5JILpTGIk1QW6ZLaRcuDb/Vlm1EnNSIx9Q2rUhwPS8N1so5pwBaY1FFwpEqwSMtuj6W15zRYfAam9PBsWi1AmjGqNCyyOQYrkIppX12FA/qUDkNsOravtZQnrTsfpvzVNr72mnGsBA14JIwD9icGroKZBUOk3K8buL0krafG1DYSbXB3NtJ1VYeb9vVjbyVVk6JnSKP9kMtE0Ii4RpIKhy4na4mIT8ochqQxEF8DMTWsuHk8JNN9zt9hklkw9jvRkqJRTaO1xXVRKcpgt5on9OwMHHUBXebQBhKux6IYJ1C6jGtJWIMwMkY/iAEYV19QhYV9wiSs6T0KUXzxGDNiLvQaSSJWh2xQK4Vj0RHQluLOgJYCSnLv+L+xQRX3yYiEYnSWk7vUTDGIKFQOy2Tp6pi8il3J7Xu6jaQeyVp8JRT8poSOkBtZHsyPfUVic8vNNQ3PSMP9K6aG8OvtyWD4Hu3SVFPm0RJ/LRTjWyfU3BsrXLQoAYmxFan79kGxCmwEDKOGjFBK4zidDmxyM5zt77IoLQ/zy4GXcrhPPX237IjyAT7b4uYE7LSBtmVhLcsnj2C0vV12ZLYW4fgFMlMzhC6qNZK9rRFayKBftwrJrqF6dVDIzW1+ROJoZZ2uaeOMn2eukaobfyLeI1AVFvQIEW2hxYo9XSTMmV92yJBv20lVASXSpIcDpbjQfUqjdea9DIZr6WVq4rit0H04AsmB8StoYOKSKZS6V0YEFLT39R2/zlalZjKlhexKYu1kvRz9SA24RakGGODtSJxDd70PlisTZDAEsSuCJ1M5X2UHaUUYAriIcOYOq3TKonehvT0tPyHSGRJ4n4Oh+AmJ18RDS3a5/7SJ6PLiWpOcbabfovOLb7H2jM0bxuTRO7Ylpq11oaODbGEtRJZLJKNqAdprqfoaAo4BSentOVmpHGLU8Le+t124XV7zy7BFwGRvNTpZGrBC9I6j9ManL5X46OsJXii9qpu2+deqxNtdKXzCDhJ4hmiaYvo1IlStZWAoJhHxZHadVhzoqlDjTmjBNpyMcSVaqfBdNSG4BvKRTrwynozft6g9AUPeRMBd9t69CmZOYkQ9XNEg6iBtILAwZIhBkkTEwnQNb5ItWXOqHBaV2xCDM2ZRHF1sk+ReHJc22a0INfBq1M92H53xdQiU1r7MRFEG0fvguhUOk0ZQEiRnlpp6mjbhIlQ2oUP+bb8oJGw2tBM8CY51ky7bRDNgKiQNQhs19Sc1kAzQ6lxPd4yC07BSUnjelvnCAUnJAidKFWcuQhDW2PFsHb/KhGEUyuNvF3fsC01PcpKA0lNmOeQsxBqlyYLlJb1zDAJIrwKUMMtQiQ5PRfHa73tOTX/mTYpAhJlt7siGn6VPTb1llvxUGnFo/VWu7YH6cG5eK5BGEvCRCgAmlCvSOM1t0hHhArbgC/u1EjllDGRO0WT4l6aTze0z6mTpRZApL/NQad90jaTE80WTUEIn8bZCZ3Xbfe6Exo3Nt2TUhVyCmpAFaLH42hK20BUEZx4czGHJNQUPhvBE3LTP7k7XeOORAMJqTsuU3JsPJ9VqoQ3ZxRJk9jZorvujlDJKhScWcpUs/ANEbI61VMgfQ3AIh7rbE28U03D37/A4ccvG5SiTu1Bx/hgHNMErUviokit7FmiepRPUgT66GYUM1wV9xH11H5WMKlbMmzioQaLhzUR5Gl62MVIWUHCgcVlW9NGGRffLQIkRWpsglriQWoOIR5yisCqODlNJUk8dFRQC6LSdQo3guW0LRc7FGuBNKtQLL5HmqK3uJFVW/MiskiXQ48lyVBJZDWyZqo7CEgrh3KDxylFOVuAThVB0ORojgCgJNyinA7eLPi4ec6tpBe8OkUqJtG1m+UonzfVopTQzJxC9dQEhE3QJ0oWA1NMQ9eFK57GQDuSYztIJclUMDoDNYSajSdUVQY3ZiIMJmSrKJliAzMhZAbtfsyNdYnOjmqOz0GQbkOqiZG07caKEhyigrkGcS0RPNRpSSOetWn8jKOYV6JPHHyjS4hGk8ZUgkqiFkNTPNcg3PPWX0SJ4MfExwguheyZ2hTe4g3/eTQdkkWg1BT7ZUrmsYaGaqI23k0lRQIBuobiNAXhnwRCNc72eadW9nYmWKe4QSIU6TlFAG/LQK4dAxZdVE0Ujfo3iaBJqTWuPyh4xRMsRCmt/DIzihXyxMUJSNOM5WiLMPPAGyKCbh2nx2WD0JGYutEhrnZ5+aj0suXbj7/rDr/6QM9ddy3Yv3iOnd05lEMW5+9FhoFZ31FsQ+6UWlZY3aBpRpKOEeHGwbPMZ3scrxPr5S2e+uSa9/7qAb/y7Ig7qAoQXTCvvi17wmGkNUKmYvxzS71J5yJbeVy8NtgptHV3Ok3hBFkDSVkEOTdiJTUWOblTaYHRoSQLFeznlGoE1xPhKprkrUukrWk1tcFxJamRUkeSRnqLkokRhalWr01saRIyCpqAL0/q9TQFvciqwYdEKXp7DT85K2wbZ1uonVqWnTQ7QczKbYLVU4lEJO2mmq5GFQJVeKilIe4xiOamARLh33vrLpevdLzqocz++YvMdzv6PnFufwGlI2kEykylWqGMJw3KR6khvaDJONkoxwe3eOGzhU98YskvfviIT986LRtcfFt41kZ4w2k7eotmG1eyLaUawsNO0b6pRGnTeBQRYZamEj0qGJna4NWxScogTq4pAmlDYrURupYcausANoSiia2QdPKXgCVR/oeGx0OzJU5WJQlkVbocr1tLv2OxhpSgM7CWcKbnr8m29z9642v9c/fE7RaJMJD31CCauL6JT01oa1hMNMupr4jA2FBRUYIiccfIQKFU2d5/sbqVEBytv8jy7Wu//hI661jMMoYw60aou/SsKclZj0t6lFE78IGyBk/HSFownyXOLy5RgX52wnA849yFgbc+OuODxzscHh5tW7AyZbzWcROinU4LOeYEnDRg+h5ROmd7kxBq4j6aJ22jQ/HCLHd4rcxzCiWsZNabkcFrIx3DR6btLxo1sqk2djWEn0CMNTg0twPie2sLHlm6Vs5GnnYdMe+3wsIxBWObidRqSUAyXium0dZOmhFr81GuW+gusSQgjmpA6whCSi+xZtHADHkAqXVQGu9jVLIksgveFdyUVFvwQ6J179YCaKzHWhwx6EQCaUU0jfdTJ5FI5nz9t9+JdsreQpj1+1H+mOFjIec5QzlCqFSkoRxlvVrR7+wCG+pG0dkCK0somQuXC/df7fiKw45P3lifatpMsFZ2J8m4VGrjM6LKCyV3bog+JC1N92VRIkIg7Ew0IsbBWvZOVLNoxZuxM8t0Kcaijm0gtVJLSFhq6vAU/htooCGIlFtbKNZJmpNGQAuuSlLTbXkHrVRCo93iKjhCdacI9KUGIS8Zkt52X0Bt0pesuDneOJtSQ+QsNK5WYn+ISxMAT9cTyM2SNk1eWB+wN8QEPuIp/H3aL+jUUY31zS4MW8lIoLzBgrwPYj0aIem2ps4XFZSEGTYW6Jy+y9FOZ0016Loe3ySGcck8D1Sf0c0rZSOMtTDmc1g6ACuUw4E0SyADSUbOp4FDIDXYPpVPk3ZIWwdBdPulRiUImchKZoa2UL3tpLVg5RpZ01uWHAo8eOU8b3tkwcngXN5VUj/jvkce4Tc+csiv/dpvUYrTS44NKoZpR/YIlK5ddF6qBPnRRHnJhVHiQXcy0dxKSrSM16Ea3JmKxgyUApojWHjF3NDOyZ2g3rUgE7A4SUYo9Cg10TQzlYzintC2DgYkr/F3EqBrXEyQy1laFw/BzFExiinq4UTmTt/KyGkTTtl2jlLVcYxZFxlzrIFyx2oNPUBipC4HJO8x2ib2jkVHVcsJXZ4hZR3DnxXybIdOK3UspFlHnscsXxqNrneWJ4nMwLxvhbxPcpNY59hUwV9GxzTmEt2dnDSCQI5S9/ZZt+AxY+QjoVRA1aiNMBhqzOe9+x33o7PEHONkXbhwcZdXf+XX8qM/+Q/JBinlKBktNFHVorPapSiDO4hRmBa8Jj5K6XBp105qXcjw+ZQyqhKDwhID4T00ymDEqHQizDqoKJJbh2wq+TQSUk7BxVarjTENxDaTxNC6ccmtcYYhlOyQaG5syfmoHMRjXtVqjYBIkOulNZWowW1N8oQsinZRls+0w4rHxIYJQ/usL2QvG5TGzTUWi32WxwP7i8rSnPnOAreBcQwyT9OM9bAk+RqpGe0Tnc1wXUNV3DLznX02y0NAmfUde7OhaY8mRn/K5K3u99qGW4PQmzoZrjFzJz6RhnGd6t7a/5WUQ+Q3ipNz6Jp2Zwsu3KEs0kA6N+PBC8L+uZ6jpz/Iu77qLfzE3/tf+Ka/8F1I1zOzgqQZIpUYafZte3PSHplDJlMF9k9bKaHyLRXVjElorPDaCOdpbi6hVvAsuCXmE2EqESAmfoLU5qrIW2I2tEOZ5M6m6bWyaQyBtlJz9PAdq5OOpRW4LtsgJcZW4pBRpLXNvTgbryHhaAFATTAPLMXUUZxIVQ9e0d1ZHa2Z5cxwuCEvCt3ORcxXdGkHNJHUsL6gZUZKA+PJEf1M8VnHUJ1+3jFQkNRDKeSsSBrouwgeowaas4l9b5kfnGwapYWz1WI17ji4Rrz9LduOGQiDBJcXosuYW3QTXv2qu/jM9RPeevUc/V7mtXcpR0eHyGO/zo/9h9/KtaMZP/PT70VS6HfiVIza+NAZLoHktCbIFZrQVQQygpCDH5UYyh1bZy3jMV2Qu233K77HcWZMWkAkSkWaNAPYtveTBScqDrnrwIOyqNXpPeE5NGeZGFYf1OlJjF5C9BxdGPrkDCWuyy0ogFEcNZgmL2q1aAQ1vkCFLRLeNsOSt2ty5mRGMeSPgpScvSB+i1P9GNEdbBhBMtJXxHrMjb7bj9EMBjqFUqHvF6yKQCrYutKRSZ2SemMR6xedAo+Mbg5TU33aMKa0ebco2RRwuy2YTR05d4oHsT2hphQtQHKnLFfHpMWrePLFZ3jLl11l5sfcWnc8favy1ltP8c9/6r/kJ37yb/Pf/9c/TUk9XkKd6im1Vm1k4uS3E5Ye3SjY8hcK0KfW8YjTDVRBRTFxUk7BaaQYXHY1SuUU5eEkzVhTtyNCakEwWuZRwppE290muYM2QhVC4UsbGPXSSk1pQZZiWLUAACAASURBVCZHYKqVZFMDIUW7GNAkbKyACyOV5DDgjVeJP1biGVS1FpAq1aPRMNtVzPrg7sZj3BTLqxA6+AIrHdaU/GlnB+0ztayYdwmd75CtghzDZo2tocsLUj6gSmR22jV4K+Hx2CS1Xb9LjEOIxMZMPkmyJnV+dDATULbOH4MR6oS6PMONg2O+5k138eRzxle/5jw5QbEZnz0YuXf+KS6/9l2cS4k6T3in9MVxElWEpBJkvRmdGuNU7k+nVkhqrXMaVwTZFbUaCNoj6mrWz0m+qiEWjadJczbHPFCheWioPMqF1gm27X5KqiytMIszYDDV0EO1zm4nXSBkib5rlUTq2lrViPKzqCFBoktcUryeDTBrnGpLZiq4K10NHrAKoZqX4GJfzl72q5IMrxtyB6ujgo4DSaIOF0+kWSL14WiWKjlnakp4ik6Pra+DrOjmUZemlOi6RKZuW+hZggBOBPRLtE5LTnR2SmOn1svX26bBp+M6pvGNTqQFtoDnOSlqzqXL5xiv3eB6WfDZ59d87KPPkIZdHn70ET76xC1s2LB87Nf53u/7i+ztJPpOyL0w10LSQueFmdQgrDHEK1iMhXRJ6FF6PR3U7HMCqzFRrhM5G6MvrQkYmUQkNmjrLlakTY44koROgrMKMaFGNkpyyjGJ4l1iTkJTilMastB3ib5LdNKRUkdOQpeVnCFnjRMJciZ1INmRFBk5ZaHP0CfoUyC0eVbmuWeWEjkpfU70OdNJ0yY1p+9nO5wcOZQ1w1DYHA9QNuSsUNpg9aynm8/JOLnvwEdgzpgXLJcvIJtDlF3A6WcAIc7MDS0n2lxj85FOU/OdaBIkY6v1SY1vo82vpZS2zQZpwWPSTKmmCCaa2F/ssBnXzLWwfxEe/9Q1nvv0Eff8uTdxfnfG48/1+Au/z9/4m+/itXfv0AmxebOTukKWkUyh05gmyGZbjVVKiU6MXiZ1eyW1LlZKKVBSG0732kj8SZun03lNgLQRG03t6zBLKfgbiaSdJD4vd6nJE5wFCTpFuo4uZ1LOzGcdfZfihIrUI8nCd7KG7+RMlzPdLCOdQxf+l9TYzcosJ7Rr8oKk9Kr0Gr7S5RT/VmVB8JQh/nz5Eu5lg9IwDKjP8cHoF3NszJTxiK435ot96gg5j8z3O2p1NusT3GfMFj11tsF0QVkZ4+oYHwvFK+sV3BoznqO9DnyOgnjqKKXqlESggKmb0jpj5uWU+PTgpTq0aaW2IoGAuMOMcVjzzLjhDVcytR5z8coOj1+7hi0u89Ddr+HwxoqTzQ4Hn/kE9165SJckNCQaLXXQ09Kr8S4uqc1iCUl9K9xUosvgOU45sBTEcmpdjqxC32W65hiSlSw9eaYhFWgK46wpJt9FyCmRszCf9XQpg8fxILWGqNC6dpaTEmf0tIHo3Aldis9KOZNzjqDUQd8lUoqgutO+hsQaT8gkS+tyEl244GVkS4arNiekcuvWLTQ543rDbNFTikUfY+NopzFbVdekDPSZ5fIl3IR+t2Pez5jPzlNQsCN8EIaNQW9sNpsI/nab+JBTAWGM1kXgznKqRo8jbZoglklfxjbBafXTA/lqdK5yNUYb2J8t+Nhjh8xT5byc8PyNY1742Mc5OXKuzDKfeWLDfLzFX/zOb2SXEDnORJhrCGWzTOr66J52ObQ/WSDldooFEThUJgRs4TdIkO1J0VS3a5xSIudM33Ux5JsTcaBLoD/JKRJZynGGVFK6lFsZGR3GIWr6CMhZyV1CVCOZpdBq9blrARxyDkoidYTfSmLWCalP5L6jeJNANPJ8CkzahKJWA4VGZRnatz4Lsy8gCXh58WSpbNIJsjb6bkbNFVlnTvpCztdY7O0zrIVhs2Gx2GVjILUybNbB2tdCxliVAc1CWQ1cO3Aeu2ZNOJa2bUafxh4aceqi9AY0iUBohRostzwJlEL5SmQda06bW7a0JJzMBnbzOV59ZY+yWSHLxOV/8x3ct77FZ373l7lyzxXuuf8eLt95hU+ujrlyofLCs8IqOTMSNQdPVJm4q4yKtdZxlFwuyqzL2zZgVm3iwMpemrF/zyVe9eh9fMkb38JHP/wRfv3/+mV2dvboRMjFkQSVjjRzvBZqbUK93DUtSCAZl7g3T20UR7uo6auTcivSksSJkkmb8jGCaTCeoVtIDp5lq/COI0TaVk4JagjdktUmJUhxYmKQLkwnNDoNvquy6EKDNlSYb5Z0vbKpiTQeInkBMoe8YFwNaK/s7d9FYYWbcXzwPKlboA7rkxU5K8VGytL41DMxoDqm29T+QuMWo8WOGG5Q9BQtRaep0QETPJVpTjOEhKPFWVC1dZSKgg3wHe/+Gp780O9yvKzcffUCDz90D8kHVvWElTp33nWBw5MTnvrQL/GaN72RJz7xcUqt9CmGh63RB9PoUjJoDh+yl65rXeZJ21MRz/EeOztceuA+3vKWR/nyP//V/Pjf/mFml86hozfE04SdCboaPGxoqdppDE1aMnXXEoL3ORBbycElMSHvaeC7hi8r4I6rboeXYzY0hM8mcdxQUkdUAnWJIFXwok0mEw9IVRHaREKN8m0aUcv6RyjfNqMw65U8CxFa8or3xkIGxAtl3ND1oDlxslwznqxwOwxVXF1yfHLE5niFmXJ84ly7kXj/YwVyPVVjN6F3TItH+5BJjdyyQOvGY1LBx+3DjBuIaxMSo0m0MiW0ONWNRep429c8wnMHG/bvvpOrD17kud/5OMcHxzzydd/Kt/zQT/Ls00/R7xs3bt7gofsusykjuXiwxZLAjTRUZpLIPqLFyCbMk9Nj7GBQCzNxMpU8Kl2pnDt3gZKBuqSuNjz5gV/ibQ/u8zP/2//K/rmL0dJOFfocim+HrIZoAs3R1tYca+OhrSpubboiyrtF19H1imjCNbps8eXgpbzNSnXOVlDnGogia4cmoeq47U6BMc+JndxGXTR2VSe2LU9Ty9zSOqKGs1orm80mjizuujhOVp2jo4pgqCyYpcRiLxDm6vAGZTliw5rUFfANm+WSWitHJ0s2m56Pfnrk917IgUItTkucCNRkIDXkELUNY6cmP3exOBpXLEqSxrHQjmXGo9/VSwhjZ/QUqRTPXNnf4b3v/1Us91y5o+P5J24is8q5Kw8xu3CFh97xLnb37uSlWwccM+fFZ55gGAbqsAE3skscSzsOzF2Z48xSopdKR2zoWR3pfCBTsTKi0kXy3V/Q7865oxt49pMf4Pfe/z/z4//gP+Pv/9Q/Ju/NSK3xou5kbyr0eX9a1m0Dzammz5qUJlnwk/Oc6bJgNVD1dqRJGkFtIbVxCV2SptZ5kWjU5CT05Kb/iiA/z4EOU2qjQqlur0GJz52SydQtfDlL73nPez7vFz/5vv/2PZpnrIYRiCM1U6+UNGezGtDcoWpYKXS9kxYdNsBmNXLzpSWlOMdr4fqtkes3jY984pjffbJQG1ckarin6ESYtpZtazE2ibo2os8bWJ023ES8OiBmmEYnidaF6fuO+x59HfdcXfDoa17P3d0x128tuTxU1kcvwmbDd/3wj/CBX3sfr/qyt1BvHXPhgV1efPzT7Ct86dvezON/8FkW80zfhe4ki2ISJVNOeUvWh4pXyDm0QZmC753jtZecv/yW80g55qEvez337u3wzGOf4v3vfx/f+IN/jZ27H+Lpj300wqqEw2Xp470J1GhuaOripEYzUht50RRnTFdNkZmyRqctteBDQ0QtRlWfWlKRyTvigDRFyaQ2jR+fKkCRBIx4Tfg0qe+lNRp825WMUZXEN7x9xs7+HqvB2c/OeiwkyXR7O6zWzny3sjxZYTagUun25miC1fGG1cmGw1vHjDXx4kuF6y8tePL5NZ98svLYjbIleklsS5YYbZnU003nJtrQUaxFCCArMbwNotqOgQl/gtDvlGxcvvsqX/ttf4mnP/N7vPtbv5QyvMT+ubs4z5p665CH3vWNvDQ7z1c88ijrkxfYmUF9/hrzeceD95znwt130J+/g8SG+RiSgTjGJY50waNTFqeVNsI4KfOuozt/B4/eC/een/PVD8XJm9/+/T/Ekx/4NT78u49xXG8ii/t49tOPARXNOdC1WShUUsyFpnZUr7bB79SaNIGihC4lRgEkSklkYmCnQWmnV4njaj08QVrDKPzKEc1UD/X4NEAfXc/wg2lI3gG3inhBcwMQ7qQcspf/6D/9sf/i88Wdl0VKhwcjy/UxXZ6Bj6yzsinCMBjSGbXGca3Hx8ccrwu3rh/x4vER12+NrNaV5TDjhYPCyXqXzUliPlfG6lvIjSsJ27azo2N025k44hi1nSQITrRXJSBB8AI1vidXEImTEwM6Bifz1HMvcOnChgcfvpO3veVudu7tOX/+Am//Wz/CL/7zn+XyPefxzYybqw3n5Dy7+/dy4eFXI/Md/pO//x7e9e7vpptluqx8yVe/mTsuXYlfBLDomS3mUctLcE9Jg7R8wyN3c++ecH7/Ah/rH+DCnVe59QeP89jRmtXG2LnnEstrn+Xg+vNcve9+sILoiEhi0A14wVPFJbXxiBIdm5wpTaQHYBrlb28xU6WqaKWdy1RjQLS1/5O3UqF1o4rGxo2aImaShEqSdvyJj6HByRZzgVZJLjEGgVBGmMCk1crxgXBw64Su77l+0jEWYa0Dy9UxJmuuv3CLzbBkuVxycnzIrZsnHBwO3DjacHQMt9aZa0tYLSulCgsP9CTEmUK4QlXEKmOMgkapX6UNfFsg6e38YRscJRBWToJU3/JJvTSdjcdBaHm+oMvKlUtXee7JE669uMH9BgOZ+7/9ezjyGd/3rd/G+3/9vVy4+w3sLq5ytLfLq67u0125xDu/+Tv4O//5j3N0NHAyrugv7HLprnPs7O3hWcNXJBo42QWpA+rG/nyH+2bH3HFhwcad1aN/gTmJj3/4fXzd3/gR7r8Edy56vu4b3sH9f+4N5JwR32BeSCmFglyG6JJbJKJoDMQwc/XTEzOqGmmrgD/lXlsDO0q9RphDq0ZqO4OrNAEm0SnOwbHHYC9TGThACglM8jgXTTVHUiNhEr9MwezlOaWXRUq//DN/7z21zBg3PSfrgWHs2YywXCduvnRIZcF66CgyZ9w4J6NQliPjYKTFHj4M9AlyV/CxsFoP/PZjpY12OBO/7eJt7IOY17CWidtUOu14ka2Qsi0CItvMU4mzdFzi0Ku+F8ax8PoH7+Krv+KN/PzPf5Dv/+v/DqTKa97xLRw+8Wn2r5zjoS/5Vj7xwfdxUM5x8b77eOs3/SB3XLiHhx66ysHNF+n6xD0XKnfdscPu3p3szAb2L17k9a95gK985xv5+m//Lh577DHENqxl5I33nScvLrHLis/efIE3P3CVk95481d9E3sn19n78m/k4A9+i5Mbz3D/69/BB371fcRhauB4oBaNQcushkmicybc2xaM7akEOLEp2xCwNh2Xegzi0ileCgVnWI+sxw3rapTNwGqs1FIptTJUsNEZN4VhNJabymasDKOxqc66wFhhaTCQ2JizRhhdGVDeer/iJeOiHN7cQEoM4w7rUVgeFQrKsMosrcPKnFWpHB2MlMEYUVQ7OquQFR9HxuLcuFb51I2BRJMD3NaF80nF1rps7qE1kuYjui0TagtfoX2jkfZTYBfpEAvB643r1/grf/MHkNUT3LnnvPMv/7uMBwf80vt+gb/2vT/ARz/0++w+8Aa+9pu+k/f/y/fxtq/5FnZe/RYeuO9hzu8qN158hje87kHuvtiRNkfsnN9l79yCq1ev8o6veRPf9b3v5mQwxBPDYFy9a4d7rxi7u4nNKnHf6+9h9dSTfOcP/xjPfvJT/ObP/nc88m/8VT72y+9lde4yv/nLv8qewYgEVS5CMqfzjOT4zTBIZZKZZkJLFoO27cgZ2tlRTcbhGmslk17LLXRNpbBebthYZRhHSoWNj3Hk72ZkGOKcqtUwUjbOaigMgzGUmGUci7EyZ12MsfnLxpTRhLXB3/nRz4+UXjYo/d7P/YP37Ox1mMKsn4NWhk1hZ+7Mzu2BdJRakTri5YQOZb5zjvkOlE1BtdClGTkJtQwUUz7y+MhQY35JJAXp5o61MZOIWEonOUg8rcEVNMV0alzTVDNrK+q2R+iaIb1SB6efCd/3t/46H/ilf8Hr7tvjqY98kLsffpCPfPJjvPnPv5NLD76FX/o//hmv/pK3M/cTLj/wGsQLicrq8Bo3X1qyOXyGe1/zpdx8+uOsX3qWA+Y8fPdFRh0YN87DX/lO5NonuLA/557dgs4XvP5t9/Pwl7yd17/mNfz+89e5cf06j/3Wb3L/7IB/8ou/wqa7xLnL9/HC4Ybjz15jHFbkmpjFeRJUDT0L7Wyf2D/aNCyOFWeolXEcKKWyWm9Ybtash8J6qByvlpyUwnIoLJeFo6EwDkY156TG0RtLFzZDDZ2UJjZmjK4UlJONUV3RrmMYYRgiWKyHijcVr8RRjEF8GrzzLXvx64F24vjTOo50s4SKMNvfIedE7XsYR3JWfH3CfHfG3rkdynjCjBIoIkUZMqyNw1trPvF8xadjadt/3SW4uNoGuDW2nBC/MUSnkwtEQDLuBWn6mNROBw1OpM2EyUDfz3jovqt8z7u/jsMXPsVYCtx4jjd+w3dx9dWP8n/+3D/lK9/13Tz72CepRyuuPvAQlx54NX3ZsNkccPOFFxjGEekSz3/8U1y96wLXToTd/fPce3nGOBqbNOPO+67AtY+S92c8et44tDv45u/7IVabJ7iUznP3V30Tv/G//zPS8eN88Ddf4NmbJ9zYvYs3P/wWRqncfOpppuHWBUrJAmOhagh+q3eQQvTqZcTNGMeKV2MzVMZxZLkZWA0j683AZrnm1nrNZhw5XI2sVyPLobIZ4zSJkxJzj6tqLFfGxiop9WyIw97Wg7MaDFKClFlvnIoxVqjWRsPMmgi3jbm48SM/9uOfNyi9bPfNlmtYKIudjGcnqTLb7WNeqQw4G1Id2d0PFbNTMTkms4AdY1wF9PcVzBeJ/bnw6N0zPvDMhmIj2U8P3YoziWhnFktrN9bgASSOsO2kndGk8TvCco5BWlTJHjejmikqbKRALYyHN7jrrnPcdf4yH356zfd/zV/itz/1j/m7P/rD/KN/+gvY0ugM7ti7m1vXb/DxD/wcczvk6v2v5YGHrvDRjz7Lx373V7h876s4WkM2obt4he/+jn+LZz7+CT7z+CdJe/dw66Wb3H3H/dxz54w77nodVx55Ezc/8WHu/tDznDz8ZRx+8Ld56vlbfM83fxsfeOJZ0uFzfPBDf8B4+BKMGwYKtooHXWulFENSKNKDTJYWlELclhA2nsjquCvF40gKVTgeFRuVC/PMchgoVUnT0Rv1/ybtPWMtu84zzWeFHU+4+VbVrcgqlpgpijkoUsGWJctBlizLLY/b9qCD23B72qntxkDTYUZjDAaG0T0DeNTTaNnjINmWbAUrUImSGESKZDEUi8WqYuW6+Z6400rzY191/xo1oDnAxf1/sM/a6/ve931emGB3vUeaXpbQuAYtNMJDowKzUQxa0xQ1eQaDsWA+zyhkQRCKabErNjjfXvU1RDbgzQQtOjgtyfI+cSJw/vuoVZBmQhpLtHJksx28g2DHdHsZvqraHGUQGFPTSSJmZ2OSHhRT0xpYd/cbQondbN/uqBbs7t6rfW7d9ykPu+M/KtoNZttWaUQQRS1opAWVafI04gO/9AtcuLDKaO00z5/3nHINR+8u8fIaP/9L/yOf/vM/5r0/+iaS2RleOvE8xclnOf/KKXq55P43v4uzp57n6uUxizceplq9yOvuuBOC4ebbH2Lfob2ceuLLLB95PZsnnqGnxwhmONZVrF05w5lBzfHlhtHJ7xKNt1jYk/PWn7mHR743Rdp1rpY1GkXpA8YWOCuZ+FaFNcaAqLC+/S38V27WbvRklzH1Xw7qoHYjQJLGgfGKWElEMNS2dV135PejSFBrSYYgV5D1MlzjSOOIcTWhP5OjpaAsLCI4dNxioeO4DSyPx6a9qYrdog7ZYnZ+0OcHHkqR3nVSS0uexpSNJFaSXDWoyBNUQjQTY2sJwZJEEZZWIcvyDhPhKCZT0kxQ145uT3PvsYRXJwmTrQFOQqLaBWtrwmtV7Fi2o0ikY5yAmFZlEbqdUdt7g0RLgQugvAcVESuJtZ4ygIu65L055rrzPPLSGne+8SgPvu02njl9hiXneNdP/hz//T/9BX7zX3yUu+++j4/9699hcV/Ee3/mA+hklo2rpzBbV7jz3ocZ1orxYI18NMKsvkY52eDJv/m/OHr3w0Rnr7H/nnuZed2tMDjFdLRNsT7i9jtm+NxTT3LzG97Czo4hdAQ7k5iLj36DheP3kC9dx+v0hK985m/QUiO8wnpPTKAOAufavrdpaYmEo9PJmdaGTlezvlXQ6yQkEiKtkUFQVBUaRRYrumkMOiJKPPFEsj1swAQ6vQicpy8jbGgXAs619EZhPTISdGSENTWiacdqFSdYDIWpSNOU8bhob74OtGofbkxAK08cCZJY4ExNmkQYp0gjRy/LWwXVty5hW9dIRog8wdWBNBaEuMdkUqBCTRoDMw39nuHthxWfP2lbLIds5eQ2drMLJiOA0sjQBmkJDq8CUWjNld8fT9rGkPY5+j5/yyPQIbBjBNvbA8g7XDv5bR47u8AH3n4rRggmmaG+2Gb5bn/gLWw1OS/+5Z+zsBLxvg/9K8aDK6ytbzO9doXD19/M3tdlROkcLz/1ZbZXz3Pkdbdz/sQjSP0QHTXl6stP8o4P/TLPvPBt5vqHsdU1Zjsx7/rxnyMn4tzjX6JQOZtXL3ChWOYN+woumZwzT3yd5868SjEqaWy7ZzSujaY0XpJ2U4ppQew8JG3/XpZrhpN23zSTaxJaFLSwgcoaklTRTSKCkkSxwNmY4cQQSo+IImZS1fqndkkSSirq2rSTS2XpyBTtA01Vk6JACaoQUTU1WRTTNA06CgS3+xvfFRqi/4aj+wcHcoMFPEqmYAO9JCaOG2IVoaI2piBElzzTECRVuY7yDU4IqqIkTWPiSLJ2dYtIdIhkw9xCzL0HHF8fCjKl6MT/1cj3/XiIFKJFyKg2cyOcbd+qQYBs4w/Bt65SJzQu2LbnRDi0gqT21LLigbe9jbps+Kmf/QhZucELzz/Be37iQ6wu9dh37Gb+53d9GIPl03/5f+OrNR58y6/z4nMnSGa7zHcPUaiGSKYs753n3vveyDe/9mUaqZhNHTvnn+PCM4+z97p5TLVF5izLN7+Bju6xsbXNN7/1Dd75Ex+kaSzn/vTfM9Pvs+kkhe9x4NITnOns4+Dr3oQrA2nsqJxAB0uUJkgdsFMHtqIfC7Ksy6Sq0dZiCsFsPyVWikgJbGnJOzGVEZSVJe9mBBqSJKEpamKl6Xc81lqyPGqX11GMrSuEcZjGEiUK7zwijmgau+uRErjGUxY1eRzR7SUAzNHBxYLJsKGq2xtTAKyr0C4iuJq5fqAoAvPzs+CnaGlwTtHpLLUjUzzAu4RiMiDr5ngbY4MhzRSm1JS2Qomc5QXwusdnXrxAJ42JlacTC2Sk0F4RtES58F9iFW3D8W7CwLcHUKsyeYJtiyG8b5f/NgRkpAjGURhFHkdsj3e4trrFr/7qL3Lh2c8xd/2DbDx3ipELPPvcE+xZnOG2297Aia9+ggff+e9YvXyJK2dOsGkaVuaWMZVhcWGF7p7DFI1h+cBlbFOSzu5n8/RJxmaDxT03Mag2mOkd5eC+JbqzNyG0QRjHX3z6L/iZn/45PvmH/xM7A8d2aOhliyyJKd84cRbjwVYNrfTfBnnTXkyY1IhySldLyAKJTinHFXXh6EQxMmkP8eB2s44R1HV7OGW5whpPnKRM6pJeoqgEdLIYU9XkvQ6uMRjvoHIkSVsqGWIgBGoHSaKRRrJVTnFBMdPJkGlgNs4JQVAHx85WsTsFhF1k0f/35wfulF786z/46MreGOcUs7M9ItUQa02sU6qmRIkcKdtMi1RTIgSym5FlHUw5pRgNkar9kSE9URThnaGTpDz60pgsEcRakcbtG1/hiSJA7IZYnUNEELNrMJOAELtxEvA+EEkPQbWGMicZOs9wChMFH/m1f8ZzTz6Ls+u8eP4MN13/VvYdOMKTJ0+zZ3Ef48EWZy+dJ08MQs1y9ZXvIquSnbU1+rMZdtTgRcn80h5inXHzzbeycvQI569eYe/yccZS8Pq7343TOYPJDrXNGQ13SIQjnlsgjjsszCxwfmuH0bV16qJi1c9x220ryP4NPPPYCYYbqwjROsinXiEiGI9rKgJRmlAbz2BiqIwnyTReKNI4oioNHa1IuxneBWLj6M3EbI1KxlWgsYZJCZPJBBEktbHoXNPrZTSmoihNWweuJVgDUhElisnEUtQVRoF1Hgz4SFI3nvHY4LxnZ1yTZ5qyNuRZRCThbYcU/Z4kigN5d4Y01UQSYp3s3oJjENWuyBwjQ0Xa7ZP15pHEjDevoIXGBtN6cVQgimLKakxdZ6zuTEhiTRJBFkXtCgPXUjLFruyOQGpPvOvkVrJVREWQBGFbpS60OxLrPDY4NiqoK0cdJyz3Mt790C2E2hLP7ufeB+7DlBsk3b3ce/9DvPbaZaaDDQajES+/+Awnn3+c2qyRecnC8hLX1q6xee0s1x05Djpi7/4D7D16C0v7j7CxdY0jKzez7/Ahct3nxltvJZrN6cYzSCnYHBfYUPHcI5/l4M33cuHcC+xd6fPCywNuvutulg4c57EvfZM8S1q6AxCUpGkElXHIVOFVRFFYBpMGJyDJYorGkmiFqw3dboJOYqST9KQj72RcXB1jRWA8qiinBu8spW3bibpzKXGiGVcFxkDeUURBEqRCqYBrJNOixpjWjQ607TEqUI0Dg/GUum4omtZJriOIdAxYfvP3fsidklRQu4CsLWa8TRKlCGExbojSs+gkwZYSkbYL6uC2EXVAJBFJd5a0k9LYEuskOqahxgAAIABJREFUrhHURUGWa/JxYGE2xVcNKniiIFDCY7RvBVwlwLTSgHOBarfSufEBY92uyuDbOuddxKeTILzAICAVdHTCU1/9NpfWXuPeO+4iYcSefUuMteemlUWWVvZSDg097znz/LPU506RL+TY5f0kUczGa8/SXzjGzuoGC3sO4fuWE09+mWa8RjdOyHtLHMBx+sQjLK4c4XWHridEggunz7J83UNUtubqy0/z4lgQrp5n5q43kl18mV//lV/nV/7xv+DuB/bx2gvPUTlDLCJkGhPHhtJbdJLSSTyDYUPWiQmhYU+/y6WtkjRx2KEh0pKdUU00LvFaI5QiCZralcx1NaF2WGtag6Z1RCIw3pqyulnSjyV7Z7o0uzEL4zQGxcbmiATwMsJNw+5i2OOtI5Eaqdvs1nys8EawmKVtdVEaobUk6cakQlGOpmSxx6mYOM8xQZL1E0KIqIoR6cyh9vZbTygdRJEkn10mihSqrrFmzLSY4EzGwlzOof4mr+gYaVtAnpKmTa+7lk/kUa2uZNsQs3E1lWnz+C6E3eC0arlLweOlJBIxmECuJEXk2VwfMrP/AF/5zilm9SZrm+C14pufeZT3/6N/xGjzMtddd5jLr13hzDOPsdjvoLvHkOmIwkqGG7PEnRm2Ns6wtr2BM1M21rYoNte57taHWFlcZHtzlamvKDE8/p0vcuuD72GwfhHnLZOi4I0Pv4/BHW/iU7/5i+w5fgxnYu5++3380cf/lrc/fD+dWDKqDFke0U0iNgc1vX5rEvaVpLA1exYSxhNLUIKyqImcIEiDtY5r6zv0Uo0QCh8n2GFJfzYhIWBqR4mnsIpcepqm4cqFCpdq9nYiuv0YG1S7+4xgMKoRZYv0HXhILLsjWkA0LTVVJTHR9x3msUdYgccSkuSHvymd+NS//ai2GkJJEJD0ekRxBqGDylKS+eNkK2+mM7+fui6Is2VEiHHWEqUKHeUomaIjTVFMUVFEWTviKHDpSuDSsK0zMkFSukBtxO6fpHaCMiiq2mODZlI7SiNonKD0bb/8tPEo17Jd0BFVYaitY8+Ro1y6tsl3HnuC+978etbXpkwriU4tduMy+47eRaezxCOf+wRzvRmCL+jkDXZrk2J9jemwAmuxoaEF+TuuO3Yj5WCNq1dfQ8uI/tIBLl86wc76Bj7yHD18kEe/8FVWVq7n4qVnWDt/CpfO8fM/+0+oMHziE5/iw7/4y3zr258FOcNjj34XW5Zo2nzbtHTE3qGQpFpgrW+JAjjmZhPGjSVyoELAiDZkqXb3cI3z6FixvV0QfKCTJeikdS+rEHCRBqXpxjGJVqRKU1lLbQK+duAblLF0tCQ4Sx5phPAob+lmKZFsc2VprDBOIGJJ4zyBQJRqmhreeTO4icOFBmU03bkZZJShowyihHTmKPHsLXRW3okjJtKAysG3doCkG4MwBCGpq6aFpcURxkjMtOJbZ0Nr3pSKwkLZtC+s2kgaoygCWCMpjaeymtpJjIOpCzQNNI3ABocxHp0lOBeY1BbvBftvv5FIwf6Dh5HmMgeO3cwdDz3En/7vf8h1t+7l0KGjnLs6wQvBznjInBRkccL06jnWL67S1GMmo02kFwg7YmnlOIcOH2frtZcYjzd48aUTdOZn2Lxwjvmsx/4bH+K2B9/K45/7OL5x3Hj9zUgleeLrf8Uz33sM17melVtu5+gdD3DLsZt47qkTvPriSwTviZXENAFXOvK4DXAnEmrj2+cmSGb7ClNYlGsxyi4I4kQTBYH3GhlLpoVl2hgiJcnzBBlFbTGIEqBj4kiTdSJy0canpo1v8R/WY6c1uVIt/955ugoSLdHBkSgJ3tLNU3xj2267TFGZgFQt/NB7yT//nX/5w1kCnvrT/+WjSRwIcUyStMHQoBT54j7ypfuZed1PM3fr+5hZuZ8wdz066aHqCu/GKGFwwhFoqJv2Wui9wtUSYw2vXKy4NHAtQEq0FUqlgdJC2YCIJGXjqQpP2Tisb0FzAkXsW6Ncv5e36oppb1CJhCzWDMdjRBKDgv0HFtjT73P76+9j88LLNKOSwdYVLl45g5uOuXbhHPF4nTOXCz7/7St85cQmt8waZub7lI2iP9PDmJqllQMkWUozdVx95ZtMbJfrjt9JsrgfszWiaDwP/tgH2Tr/MnsO3sPKoddxdfUs8/v2Mx1skespL3zvUe5/xz8krwoe/dZTVE5iUUgtKJp2yR3ruFUmRUDF7WKxth7ROKTwyFgTxWlrgUg1SS7pxgmpEywkmoVOTJCCpAEhPVEcoRtHniSExiAEWBuwu8CwxhuaIMlSQWEsMtKkicbUjjhNsEATDGiBCS1ATCiId8NbUkpkJLh3wSGVIdMxMhJEaUsvT2ZmSOZuI9v7ZuZe/156R+9l/th9TGvAGKQoUKLGB9fWciuFkmArRdNMsFWgMpKvn67RKsYFR5CawgSMD0yrttxUac1WUePqQGkEWSrbxl3bxlxErMm6CkwrkQu7G3pNJNvbU6bjhrWLq7zx3ffx9HNnOHx4nm7ecPjYm7m4scP6K1/ivb/4+zz2p3/EyRde4dTzp/mzx66xN4voCI8TnixO6PU6zC7uZ3bPMTavvsrq5fPUxZClQ3cyt7xEMrOH5578DHu68/QO34SpYiq3xde+/PccOXacn//5X+UbX/o4X/zKC/zSL3yIP/30J7l48VWqYYvatSHCek8ZHEmyy9CwEOcRtXUoHMMpCGGJo4i4E7exIhmTZglJmqC8YD6WLHRy8lQhjEeYmiSPiUzb0eZcoBoX9HJF48F5QxQnbag3BGSsMQGSWNCElrhqo92ERaxovEVECqnbCFmKa1lZosUJ/dpv/u4PN77pCJwzRN5jG4W1Y9Koi5YpxdYqcnGVeH4V2T+ESlLMpKYxl1A6xvmAmezgrCWJBCaSWFu23hahSIQgSmK6URup2DKemV7E1qBkeS5FR5J6s6aXeITU6CxiPKnJpGBoAovzHTa3xzgviPKITqoYjBtmY42X4ErPbGeGYnOLt33wQ4w3TrNVjOl151BlTF6OWTl6HcXLzzApPPXYU9WW238y44UnK2Q65fjdh1lYXGZSbFBNCxCK7Z1TTNZ2GK59ic3zT6F6muXl2xGqJUv2j9yKU4qra6fZ2tjm0W89QmyGzC0uES8f4auPfYu1l55jUprd9L8kjiMO5iltw2lEn6gdZ41nKYkYI+mawMiVICEOBiklFZCJmGndFkH6qG2shYDMYkxp0VKiewprDD4XCOPQEuJU01hPZybDVa25MUpB4gmJxHlNlCbkoaas4l36gkelEXVpQEIqJVXd4LVGykCsJTIWOF9QVxlJWtLUAjfeIGSXyCf3Eqcen3TATNFhg4oxUit8WVNPHVK3fXU6tSQ+ZzpyOOp2d6ShH2dUKpDtqnC1qdm/N2V1bcSMEOgEsp5iNLVI5ym8Js3BN4bBdsB5wcx8SnCeoqiYjzKIM7yfcm3tKuVWwdLKEnKzYr6/wnTrAlcvnmTl8C189J23ctPdh7j9hj388YubLN5+gKX3vZvLn/k0+zqaoBsWlm7B+wY3XUV3Z3CTHcL2Nmcf+ytU1CfqZyws3si5KxfozB/l9ruO88LLF7n/zT/DTnENIyIeeviDXHfsWU48+yjf+fbTvP8dd/B/fvwbZDIlSWAhy4gi0RZnWkGSBWxpmYlbyGJVt2HtprZMRxVLvS4T19BMLPuW51kzNd57up2c8XDYIkqynHpaEvcijDWoRNJJYwpjSaL28FORJdFydy8nUcFihCdSESJuUUPKtR4nGcAIR6IiGueRWhEa20IEg/pBx85/wxIQqRbVmmZknQjp6lb+kwYZrSOLF1l/dUAUAmH8Gq5+ER3n1MMNhK2QiSKKYWpaS3yYSsAzHVaUVlLZhtgH9i7NEtdTohDTXdD4YNkY1DTOEbQmjtosVdEIClrH7nBiqH2bjM57CYm3LPdipIpQSYcsdlTeMt52PPbdx7nphhvYd/gwzaTgrnvu4tTT3+WFT/01Kzcs0sk0XV2S9zKab5dcd99t7Dswi1dd9Owhrj98A0mS4RBcO7nK1qUx64OLKKUoK8/bfipjMEmJ7ngnK/v38trFS6xeqRhe2+L5p1/kJ97zdq6/62HOvvgtejMVL18cMBsHPApdGxZnUoaVwTrPwhIMB+331enlXFnfYWG2S6kc9USQJC36VCUpqhhTVIZOP8VaR5QnZI2jTnYZNsHQeIdPE7K+JliNsSXClJg6oLzBV1D7gHYgI9GykAaWWClsUdEIQbCWyEUY5RmXNVkSo1RA2jYhLgIkud4d2T0iyohiAdajE0WUWmR1hvLqpym2DxLsBrp8DaMCGoOdDCDtkc90qKYT0IpmUFELj48kZRkojSPxGt2NmM9iykFBgmBmscdgu6KuAlUIzHQ0azut870xoKSjKNqcVuUNvTQlTxXTiaXTz6ilZ35liepcwcJ8n5eee5lxM2XceZzZfTdiakNP53Q6Q65/8z2UO9cQ2ZSl5T7rF6Y8+vFP8aEP3I2fWiKd0V/Zy3hUMJ5UFEXBaGvE2RfPsjOBYwsR07k57rlforuHKc2U4bTPzs41Gr/DvrkDfPkzf8dsL+LBh97N7/3BH/Ku2/bzJ3/zGMJHJNrT8wJhLYVQzGURRgSq2qGS1g7jS0+UaLanFbFy6H5EEwuUkegQM5gM6fVigpXEuaAf5VQIci/xRkFfoRpFpDKkbSjLGu8sognt/1RhjKGF5Qakb6NNxpQt+55W+AoBpBHU0pFEGp0JGtO07CcZ/cBD6QcHcv/+Dz4aZZok1Qhp0VGXKIuRwqN1jK13iOx5VLMKzSbO7BDJGBc1uxVMHkRJPa2Zjgsmk4omaMbDhi++aJlMGq6bn4MkIHZqdrRDq4hOv0NmDVUwLPQ7hKlFNAKDZ7Gf0uvEuKZicSbC2ECvNIzKmhhJCRy8/nqklBw4dJCpDVy8dIaXLq2RlwWL/Q7nX3kekVTs2b8PilWc7DEZbPHjP/5urjvW48iR67j/R36E173+QVZfe5HaOjaunEFEM6xdvcaVzR2uxheZv2nK5x+zLKaOgOPUyac5dueDPP/4N9nZ2eTGWw7zjofew8XNM/TnuiwuLPLI158lVYZXzu8QZ5Kkl1Eag/Bml8YgiIUg6yc0jafT1VRlgxSBrKfo9GbJ0hjyCCEcQimc8fiytUSovBUQciFxlaG0Bl81VEWDcA1VY3fREpDEORiH9tFuf70GAelsSiwjkkihEaRpgscjrQMBrnLUtq0alXFbHPCO44E0j4gyTZpFQDtmalXhnUKGGmWn2OlFGF3Aj88SpMfh0XEXGQqaWqCEZTwcI5E0laOqNC+/WnBqLTCrNP2FHIZT4iRluylZ6HfBBpQKzPUThIFeCIytpdeJmet3yaNAlEr6SUw5bRB1g/UCrCXu9omSLv2ZGeRMnztvPcZk7RK33XErwTt+79/8B85vXubs8yeQm+vksx0mI8NDDz3MsSMRb7jjMMsrN3LX23+EgzfchkUz3rhA2p1h9eoqRdnw2JNneOs/vZ9PfudFmmlMv9jk4Q98BDcZc+7iZYQZkDjJ2VfPcf7aJebyhjRfJl65l3zwFE+fG2EKQ6+TEncjpk1DokIbL/IOqRxxkuJ9IEQGZwxpqun2u2RJiujEpFqhItEyzKYtSiROI4qiIRcKGwy2NpjC4Y3HTgrqpib4Fuuio5hAQBqPFjFKxggCnTghjmPiNCdREq0EaZIiatNigS14L6gtoFt2uXeOf/7bv//D7ZROfPYPPtrp7lZiExBR27KBq5HCIEVFIMFbj3UFIVZgB0iRg6spmgHBOJq6YTppME6yvl5zbUPz1VcNMtUI6ajHlng+JUQRzgd2dkaMxg0dKWkqQ9pJcLEkI2AaR9UE0sSjnSbtKEwQ1NZRoZlfnmVnOGVx3wF+7APvY2s4Ys/iIhuvnuGDP/dhbrj9QbKsy86V0yQJbFxbZ2Qy7nzPf0c2E7Fv337uePBhiuGAYnOHvNPl9CunWNizjBIlc0t78Wmfuw7exsf/08sktsfBqEFVG/zib/1LHvn24/jpNmVTkSZ9LgwuQuXodTJeubDF5UHNVx97AVNWzC11mZ1P6XQzZKSJlaCpLaXzdGe7bWddaEjiDFM11FZSjkuKqsGOKkoD0rbMGq883gWsdaRe4oxBxhFaarpZ3srBkSIVGu0CUaxp8DghsR6stegkQskYtzPGuxaDHKzFNA4XQhss3QWsKSFaXAsSlca88bhrVVMhUFpiTI2QYHwgjTxaWlCSIEYYMyHEGc40xFFGYwbU4wJLTVNXVNOGnWHJ9gh2xp7HzwgGpWOKQ1RtD6yejaitpyoaJpMSZxxht3a7u9TD1w3KQ2MMZWOJTCBJNDZYrPNMDSzuXSQIjQmOt7z7x0izDmefeoojR+dR2R727NvH8cPHuHLuOXIvCFGELGMOvO29JIszFDsv8573/zOSmYjV86+ys3qJalgymUxxdcme/fuwXtDdk/Hv/8N30YXk4L4DVOsXKaohd7z9pznx7DPYnQHziwscOXqAw/tWWJ6f5cLqaT7/lW/RdWNuvvV6Ll7apttPWVhZJMahlSYQKJ1BxBF5RxO8pZMr8Ipm2lCUBtMEmmlB2QRsYxHe4yKF9RY3DS0szlikleisbfqZ6WbEkSTr5ojS0Z3p7FZpSUxo2WJ4UGlGFRqqcY10Db5suUyToiQkUXvbtq6FvjlH6kFlMV5Jfv1/+CF3SjZ4iqkj7WjiFCLtEFrgfYWQMULENMUaaZ4TXIOwKc7E2DCgrqa4Cib1lGASqjqiNIHh2HD6So3AsCdPoGl/gLZ2rG2PyTJFb6aHFCXBemLdwsjdpMQHRSMCi3OayjmsAVd5vJPM9brtFzYpSGYX2HNkP5/8s78mlxEX1s/ylje+gVfPvMDCgcPc+/Z3cOXEFxnuOKJ8D/f91Id58XvPcmTecej+d/L0o19iz/wyhQl4b7np1jdw7vFPM9UJqhzzoV/5XT7/+b/nH7z3TVx67Rz790aYSLGzNURPG5yKOH50kTjqkE0dO9NthKvoze7hhSe/wBvuf4Anv/ZNRtMGX9cEqbCVR0kPso1kTDZHCCkxjUaqCqEjIu9BaSINzgbyTtxeqyUIpci0xPkaYSSNbeFvQggKZRGFI3JQO0skHXbcYmj6S7MU21Pifo9iWuG9IZ7toeMIZwUUNfFiB18YRts7+Cymn3YZjCZorSicRRlHWUOvp9oRAkOSRkhlSROJCw1BarwdYBuQWiCMISAoiw1EUBjrqaqqDXZPofEZO+OS6TjhuxcGLMx0WFaBoCNAMlyv2Skss3Ma3UvbuMuopEkFw02DNS3qN4k8c4szjNeGGGtJZErW1/REoBqO0GnK0pGjPPmdp7ntntt4cjDlpuYwd93/Ztx0nb/75lfouAlJnlFvX2DfGz/MysH9nH/8z3n/R36XUycfZ/PSNfrLRxjvXEJPttm6uErY5zl9EnKhOXDgEB/8keOcOT/h7v0WwnWcfvZ7/OhHRhzdf5D1K4ZzF86xuLyEUhlDPyGfuYVDhxu2z25w74MPMfnkU1hrcVWDRVBNpsRxa9OoC8OOaevn66p9wYgkIzGGOBa4ELcRnjRBVR4tPSpPUKENJ/siYGhROCZ4yrrCN54walX36c4YpwT9XgeZdigGDVZZbFnT7+eExOGjDLM1Ik4T4sUek60RhlYs0QiqxlILjyoaVP7/Y3x74lP/20eDDigdcE4h4wRv6/be1Ci8TBACTLBUdQPUWOMoqynNNDAaFRBiNgYl06FjY1vy8lXJnz83IAmCyjqiLGkZ1Zkk1i05MJWwMDtLP4vpRZpO4umnMyjhSFyg14modhrm8xzbNMz0c2zZsNDpUHnD1IGKFOiI0JT88od/jLf+6E+yvG+JT33y/+HBe17PyWe/jHcWtXQT2k7Zu9Dnjgfew2c+9hsEs4Wod7h87Qqf/Yu/4VAvUMkMW05QeQaLt5JnklfOPsPhuQwRS6jHbIY+/fllrK2YWzzIdFRSTYZU04qqGvL5r36Xp771PS6ffY1uR7F/eZZeqnfbTQNZnKBqz2yeE0lFvjiDxtHp5/T3dTFjD1qR9xOCVkgLUTdBTEqiWFGKQNbtsrkxZlDV4GBxZYF+osnilHI4xamILddSGhEQj6eEVCJ6CTpRdBZ6rF/eRgpBU00prMfUDcIG+nO91iGtLXMHF4mcRzQe5+GugwLjA1HmUCrBOI9MIpqyAu9pmpIgErzz+AhM3eBMu7OwxjMeDykmnmmjMEaxfa3A2Jy/e3bA9sDhrKdqPLPzsxhnEcLS66ekIqafpsxErbo0nyd00wzhalIBmdpFzo4r8riDU5ZyWrEy02kLJiPF7MEVhFCsXrpEU9YcvfsIUir27NvLPfc9zODV53ntzCkO3fujvPrst7j1jruZW9zLN/72T7j8zUdw2lBvrPH8E9/jwqtnmTqFm1aY6RYLh2/k+nvfyl4duPfhhzl/8kniXszN7/xZDix3KD3tMjnuUBYFW1vrHJrv8NmvPsrC/kPURclffOKviHsJB5ZnUbLlxWRJSqolmYcsSGYW55jvd1HK053rMrfcZTyoSedTRBwhTCDVCi8cupPRVBVxr8twWjEaVS31cr5HRysWVxYorm0TxxEDAzqNMFVbNtEMxsheTGemQ9JPKZsGM65xpkXKVLUh0PrJOmmKkpDOJnTmclTRVj454/j13/q9H258+8p/+jcf1YlARm0ldOMabBBoqSld2bZeCocdT7BBMp566rJmNK2Z1FOc1TQhYbjpGWxbhjuCEy8PGFUtMD9TMQtxQqIDfrsh9QLtBDIJbK/voFNJ7Sx1UFhTofoZQrZqW2++x9XtMZ00Be2QMwnbWyMaIZjbu5cagxpUPPC2O7j7tkV8/wDXLp3mwTvvQ6NZP3+a7vUPct8DD5HlGYdXjvHJj/0GtrHECwuMas1nPvciO03Ok0+foitL8iyidhFXnnmEm+/5UfYfu5lSR1TbG8wsLZN2Eq6/4Sb2Hz3G2rXzeOGoNjc4depVZFHzib96msMHZjFNg3SaVNXo+TkGg22WDx8klzWhbJe7PhEwqlp4/3IfN3S4yYQkjYizHCljginozfbaCvIkJu0qptaReE8WZfR7ChFF2Koh2IBOJaGsSAX0JUhnEVKRJDFpnFKPpkQqQcQCZx3SSXIhW4VwNm3ryIPDloFiPEVITzqXISeeG5ZBqkDe7bf15Umbd4yiBCtbF7UxNdZagnBMxg2omknpcYWnMoraK2oPw2uWSaP4y29vsCfrcWlYE2lNN1VkQhBHAT02qKAZTsYELFVVoVJNbS3T8ZSZ5YyJFSzO9xlXJXplns31MXNLfZJeyva0ZtTU+KA4esPNDLeHxMHyr37nI1y8eJV3veUthKbAGM3Lr3yWarLDqJph/sABbr3lIf72P/9rJucuovoJ1XTI6bPX+PbJKVc3xvi64Ybj83ifUhiPHK+y54GfYePCGUaXXyI0gq0Lr6KW5jlw8PVcOvcyQljSfp8rJ79LPVqllMtceuUsj379WSrr6AhQsWY6bcfTXjfGNk1ro0gVvpC4aYXsdUjylI0L2yTS0e3MttlKU2IyT7/fp5mWZDMZzgYi61FJi25Jk4g0iWhGNUkno57WpNKjrCWNdjHLXhItZEyvjOhkOWknpa4N3hukbem0GZKk26esK5yDaljS1BXJbEIiNLa2/Npv/5A+pT//w499tNPpoIgQcYKtJT7EOA1l1XalizihDprgZOtDIqcxAhsSdjYqttcHXL1muHSlZHFWUhjDKxst4yYQ0Imgv9DH5ZpRabBCkMxFJCrCCkscKYS1zO1JaZtfG+I8wdcFeQRp5onzmLm9i+SppPKC2x96K6efP0vcTbjxUODRJ8/zuhtvJUdw5bXnuLK6xXDnAu//h7/PYDDlhptu5P/42G/Tj2sa4ZmmR/iTv3mKkdEcPrrCVhV47OSI9z58N4PLpwmTLT77uc/w8I88QLk9RHhN3OmytXmJY7e/iYuvvcrOxpjReIIvRmxcusjprTHXXX8LJ58/CVKwbykj0RGjtQoaga4LmmyBuANaB2zdMHYB1wT8pEF3UuJOl7pwuLohy6GcNuT9lGrisE2NHBsiE5HMdUhEIJ/pU4vAaGdI2TQkuWJaOhauXyaJUzpZytQ7Or0exWRCZ26GsqjI85QZ2SEISzybks128V4TXA2VhUQS9xKmOzWNDag8IbM1WZJhrUDqiKYOBJfSiIB1GlPLlhue9bBWoOlRmQ7WdxiPJ9Q2YjoyXDlb8uKpAYScN99s+MLzllFtUVJS28Dini5eKtLlWda2JizsmcUIT5bFLcxRC1IJupvTEQ7jGhb3L2MGO8ylilh7Zvd26c0lxHFGtrDETjnF2YrV4RbXv/4h3vcTP8tg5zRnXn2Fl57+KvM6ZkrGz/+DD3P//e/i7MvPsPHSs+Br3vDGh/nEF07z6CsjrOxw8OB+vnN2lfOXN7n/xmUunTrNtTMnWVzucOj4TRSjDVxTsLiyiG1K6miBXpwiqZhubnHhtTW217ZYrVOe+dZTjOuGLIlY2Z9DJcCWKFuxtV2x9+ASIUC3G7M5HeAjiRtVmLph4che6rJmuDUhzdpSAS8F0iuaqiZuoJ42GOOZ27sI1pIszTLZGDKcjkEJev2I7caycut1yNoQzfcYTyq6UU5nNicIxXg8YmamS6gsOhbk8x2SbgdjSmJ2scs9DcYzHFWkeYxWEf/4N37rhyNPfu2lKeevVKyuFmxeKSnrhMp6xtsWqTKmtWTrasV4x1LbHqWIGBZjGqMYbEy5emFEMdYszsGbH+gxv2y4OIxBBazwNLRhSo9F2yH7FhL2HOqyONtBR45elhEv7SU9MM/IdajjWWZX9mOMQ4eEuDPHWEQUjeDc2UuMrcBXFS8+8wy59AzWxvzZ517l+M0Q+fa7AAAgAElEQVT3s3b+EldeeRJharQ06Nk9PPvNL7JnYZ7tzR0O5A3R4g1EAq5ducrUO7Y3d3jbW9+IjFNCInn23AVCMITZPbz3x99FABYPH+P8xRdwScPyXA+k5sDRW5hbWsRPBwy3Vzm/NmBUZzz93W/Q76QsZQk6RAy3GkbDHboLmrKwmNUrVOMRE+OYW15heT4j6TjimYRiZ4gLloU9CWnWY2tck8/nNN4xnJSMpxP0/AL5dTMoDUk3oVjbwV7bYW5phnRmBqMS4m6XOLQufRtHLC4u4JQhSVNEXZFqx3Q6ZRIM8UyKzlNULJBNASHQme/iXVu71J3PWeympN7xjVOWZ8+WbA8DW1caiolgXEIxbWgaReUV4zJivNowGXjGvmBia4bTgmJqWL9c8NrpLaSUvPG+ee65s+a1y5odAw5P6SxKO5z3dLTAFxsc2tdhfm+f+VxiXUl3ro+Ym0EuzVMrRefQAkmvx+blDWTSw0Y5E6k5d3bI+rrFTmoW5vtsXLjM1dUN7lyK+ON/+0f004zv/f2XaLZX6WCphOZND3+Y//X3fosTrzzPo49+kjpdYvXaFkYmbIwcooEkSrn9rgfo9+ZZu2aZP3Kcbq6YP34zn/qP/5F4ssoTj59ic2tI42J21je59867mFlaoBjUNH5ML2nIZnt88XPfIKjAnm6X+VQz2TRsrY8pixh8QjCW9YtrjKqGcSPZv7SEFgqnBHEaM728wfLKAgtLfdbHY+KOQgwgyTRh5PHdDrP751m6YT+mKRFC4k5eYW6+S39pgcY5dgJ0uzmuLghK0k07LCzPYGKgNvhmTKo1w+kE0Y1JO5ooj4licHVDFEuiTgauDXSvLM6gRAyu+UHHzg9edN9zfYeEhms7EWvXatK+I9OeqCOJ1YSZxR51NUGHiEacoxtnZKlD6ob5XuDIA0u4esBwrNhYn7Az6dBNSnRQLdp2F4uwNaroLiwzntTICsrG4WSfvBkSDa6Szs+wOdpEkDKcauq6QaQpTT0l0gHrAzNRgtgeE3nFeGuL2T37uPu2G/ncX/0tK/MRz3ztP3Pd8WPcet+PceWlr7G4cJzT5x4jWVxCNVNe2xphzp0kPXIT99x/PU+cGnHvwZjtJ77EjTfdwtWtR7jr9ttYSD1N1CUq1/j6Iyd425vuYjxSXHvpJeYPHyTzJRMvmVQlwwunmQhFf98CX/nCCWZ7PYauQipBpyM4sDCPm6aMXaC2hrmFhNFmg+sYrgyusLCYE6aBohgiEsH2+hZNnBJnEf0optkqUJEiiyNUEIzWV7GrbW9D0k+QiUaiMDs1WZ4ivKKqarbXtlBxjHCGphqRdHsYU5KuLDCdTkmCxdsKOwyYUCERFGVDpDRVOcG6wPaWJVhPQcA7x/Jiwtvv288Lz19mXUF2WZLMNiRKkHVL4lgR9wQ0JVpbjBEk2tHtJRjruO66jNffssLmlYvsTCPOn02olSaPDJVRbamB1IwKQzTTpe7sYWM0oLM9IniJNQI52GKmF1OJGLM1ZX0bnAokcUzVVGQyIi8FTln6qhUCXn7uJAtHjnJ1bZVj993Pt0/+PS+fOcWVCxeZXZDsP/4W0ljxhT/5Q97/T36T1TOnqS6d5dpqzfLeA5w7s8btxwTbd/+/pL1XrG7pfZ/3vGX1r+1vt7NP7+fMnOkzrEOqUFSNSIlU7MSMYge+EtKQWAhS5MCKkMAJnABxYASyY8mWAglyREmWRYVhsckR67DMDA85Z2ZO77vvr6/6llysLeQmGgPWvv+u9sK73vX//37P85O4d97gn/3W/8VBOeXFJ1YZbU555uWPMM4PcC//HEV3jV/+H/4u//C//g85/+Qz+HQNHQv6/SWilWPcun4T6Uq+f6vkeD+jkp5ZNUenA9ZPLuOmc7rac3MrJ4kV08LSywW1mTHLYqjnqFpT7Ve4LOTxzU10pFntdCj2DVZCfrAA6SkPRlReUnqP9I4oDfDLXRbjgiSKcCX4SOIWnmrmqfBU4xGqtgxX+xR2TtjrU+UFSRFiyxJTGZr5AlO3W7iZaSiaBVJK5v7Qfyf8v9b79q6H0v9zc8bHnljimZMRq8csy6sdhIjo9LtYCXHSwZYOQY2UGlvOkcown8+oG8/eXol14JDgEnTgGEYO4xxOtSXaomxaIeNBQRAafCkII4VRlmIGs4Wg19T0l47gxwu8h+VeB5V1qKsphRC4QrB6KuP+93eoNTgtmUxmTKsJstPl2PE+t4+scvDwIbfX3qEoJWvecPLYSbavfYmD2RJbPyhoooZ+eZVmZZ2XL3X4F1/Z4jt3DLPqEX/9Z19k7/Ft9KkzpEHOVMRcODXk9Te+z4d++kfYvXedB48cpak4fuYF7ty8g8u6TK6/zcUXXuYrX3ibWtQM05g4htnenDwwHD/XJbk7Jz6aEWroJAEP783Iepp8uiC3glB7IgdB0FBaQzMRrV3COhpjMKhDI3DLZiRoAXvKgQtoawhVTiUkSmrKhUdVC5QIcZVlUc1wtmFxZwsjJcbUBIfoXWtb224YCuqmQBlIhMS41pzrnaKRjpubDZ/5wn0+cMVx5GiH1aEi6WeoTkasFYQhSZjha4fSCaaYUBU7OAGLuUXJkq1twbxZxpQlQTfnSKE424eDWavBrgtLFdZM1IRIB3QDRYTANxofR1gRUm43JKIh6nXQWtI0jnipy0xMiQoQacog7VIUc3b2CrLVVXYeb3PlytMc7GnClSHX3/42R544g14YLp7tU7pVgh/7GFu3HjJ55xX+5q/9fX7lr/4ij2clMlvhiafO8c//+LNsj1oH3wcvnuBDLx1lYhYs6ctMmgkXT4e89c2vIdM+2coGTnRorCFWS6wsN2xvf5Z8y7NQy3zr6tdJuzFLRmDTAOMND+9sk3UCjE9ZX45pEAyPOmYzgd2bkVtLJGNKVxMHEpdXiERjy4KyzjFWoIRiUZd4oQlN21+UyuGkIM8bVAC+bl88tQSdtyqn/d2SkBaa10jB7r0dZDdm/GAPLMSynU06KXB1g1YCUxkCpcikAAyld60ezHvsX8b7VjjFbJrz4BHt1in2ODOidgVehaRVg7c1xpYIQwth85LRxBJGGVJUECqkdWT9CrFwbHQsK72IB5OcLAyRVpBYT9CRxJVG6JB0tctkUdA7mhLrirqQ+JklONnn4PEChMSMZgzWVvF2ShM4pvsVq2f7zB8WGKnxvqF0kk99/Kd48cplHt1/D698+as81xnwYDTlxNETLPIRN279S179Xs7f/s3f4r/5G7/I8lrKtavvsLx6lH/vUyfZfO0OF97TZ7l3kpNnT3PtK58nPXKMTj9lvH+N177xGs9+6EVSucaRkwZR5cQK8rJB9o5g9ZjP/f5XQGmoDLWDzrBHN1LUpcCWEh0E2EJz/9Euy6cyTCAJkFht6ckWli/coXlUKoxoYVlKt764EE/j/xzgZQm8QnqHQSDbzjNetjxnpQxdDUZ4rKvxIQhf45zFWk90qEWPnKTB4g6VcYqWcSVFCwnBeQKhsMK2zOtYYrxlaz+jNg4VJQwSR1YIJm5EksU4W1DOa6Ry2KomLw1ap+Szdi2tcW0ivOOhVBSJ5PxKxNXdkqIUKCFIpaJXR0SiXfsnKzEkHie7xIOAtKloZpIkg7yoII3IpxPcICPMQqbFgmjh6CQdOud73C9DenXIhecu8+YXP8dav4vdvcPJc09w+9Wv8nA75+23/5Rnnr7A8ZOn+Y0/mPCZX/irHGwXnDu1wu69TTq9p/mFj/8YmwePGWqNTJbQUYj3DTfe+Dy9jVXG1TK9YZ/7d65x5vkf5cY73+TUMz/JvJhT2hrfVDz95Aa/+bvXCXVAYDy1lKBD1ocdpnv7ZFGHwXpG/bDh0eMJUaror2SUlSTtK7RtCGKwoiXbhY2nVhJpLSpqB9WRCZFaUjvfCk+dIPSSxjtEI7DKIZwg0ao1OitIlWtv3IdChtxb5DwnwhKqAI/E2zbLJpREOI/Qgsa3ZXYhBOGhINV4j/3LkCeNl6Salr8zhc7AEUUpee4IAoGJPNV0RlNZ4k5GpAKqQpFlClSNUg6FoixyvKypa8vpEyln7o0Yh2CFIQhbo2dZWHxRkCUwu7+FsILRgx1UJ4OiZPnkEB1oMgVyOGDyaJPJtevYMELXcOL8MkXV4I3AmorltVN0goQ/+fyf8Km/8n4++BOfREnN4wc3MU3F5//gH/LX/pP/il//7x/SHUT80R9+mmNPXeTB9bcJdM586wF51iUwJW98a5NjJwT3r7/DvJgQb09IOhnxSo9nXnwfew+uc/KJLv3hOUzaZXd3lzhKGaQhX7y9zYODPdSh0uboeo9eL8N4SKspVsgWG9wxrJwa0OtqTKEobE5kHcJpaiUR0hBYiZEeKRw0AhkHuGmDUxBGAuMNTqmWmiAEmhYaL5zHWI9Wre+rEQKEJ9ZQmUNFuAxbjImAEImTLQtdI8BarHQEshVwet+G7hrbKqACFMaXOCS9vqebhUync1QQ4ESMNe1NqRnNEd4QhwFeB8RZijOGwbKkajxKhuAMua9QViIyz4lVzzPiNN99fYsobpGuubNUlUMIgx5NMbXBakf1sKUE9HsxWXcJN6vIVjrckwXR/X1GVY6LNGtLXYJaIpRlNm1orGF99QhfmFvs5kN66y9w5ugVxPOe6V7NmctP8dRTP8SX//R/x5pNXnjhBZ5/7w/x2d/5J4y8YvTqN6iXJY9HAQ/H+1y8dJqtYkaUJlhr2R3N6GT7XHj2GTavXyV+8QpnLr/AuTMXebT9iHox4sTxs7x99SoPb90jUhKHaMO8q8toDYFeI9QOO7NM5g0rJ7v4WjCIJBMahHVINNIGWG3QXtDItoOGaUkA9TynKSDoeMThRhSgbrXTrUbLGqyXOGNa/6JtJQVK+8P6iCA8tJ1IGVA7RyJF+4xax5+LZ7RrZZxSqUOLjm8PJOMPdfT/hoeSlQ5ZQNQPiEKFcRItajqDDrYCHQZkG+vgxggbIQNBKCqsC5HKUToJlGAaukkIA8dit2bQ6fBvP7HMn762xzAOOfbECZJOxuNv/wBbL3CJopYhhQ3pO0Pckczuj2EUMTi5ysHDbfrDDi5JIfCI3CA6MWJWMXdzQjnk8pULjMd7/MhHfpb//G//PX7lV/4O/UGPJ87+JH/ymd/BoLBGcfbUBkdffJo7Dx9jx/fY6EZM84a19TWm3nPlA++jnE750me/RGewxNKww/3tfYbLHXrdjKMn+9y/BUGwzIWTF5BBh8nemCyKqboDojhBGFA64sgwoHSOYaZZPNxvQ5DzBdvjHDNzxAMIo4zF/oyVocIjMcISONG+qUTrzjLWkXQivCnwqcaJQ2PsIZHQWUEcBLjGIhRo5fG21XA33iCkxluP1Y4AjUXgvGtvU7R0xkbAYX2NQLdhVSUlEZBjkJ4WNAY44YjDkJMrClc5aj1naTUjzAKMreh3FUI1JGuOSHUQLsK6EldLrDQIoQiDAFPnNJWlG8cczMboIERKwQsblm+84dkIFceOLZMd24DphL23biKiGroBioh9Fqz3A+x8wf3rOadfOM/08R6rSiOOr5BWJZQlvSggOd3lxquPqKXG6ZS7N9/hwrnzbO7eYykaUBQH5GbKxukTJIli6/HbPLj1gCCN6R87xvWrbzDLPVE8JVSKwdEnaMIdkiMd7t7eJq8sTo4YDrr4Mqc+v8TV17/Jg9tzvPoaT579Kbwo+b3f+zTnL12C0SY3r32HQkiUcqz0YxpviLoJZT5F+QoWigdbD5nkDiaCi+cHbB/k9Dtxa+QV4MMa5wRK6Nb+ay1CWsIsoKgkEo8IBNK0UtKiEiRZuwu3wiNkW6wNkHjhaA7NQs61zwKypcFaPN55AikxgHS+Tez7P/cwSqwzhB6st+0z5tyhfPYvQZ78X//Or/3qT38ww88L+j1JFCnSTBMHjsB5Ai1QTUWV12jZI19MSZOgZW07iZYWW83AtQyVujZMp3DqREqYNDQjxyLTTGdl65WfF8QBlAjGectgTlo+PHVhGJ5dZ3R3kzg1LCYFaTfiYG/GyrkjOKsZvbVFutFjXHjiLOHejRu88fprvHR8iAsDnLYs6jkLO+OHfvpTfPYf/T0GR9b5zp99hb5qUaGLWY6nomo85TRnd+ceiwcT+uvLZEtLbN97TCNhqaMJItjaHTN/uEW6NAThqUzIq9/7CjLLuPutL/DVb7zFovGsDwNOHzlK2ORMHi6YLgyh0oymJatDyfKRmPmeJ8gb+oOwZdwJjzGSEM3koMJ2NGkno8g91hqM0ASdmLQfQW5xxrXKaxm01pBMoVWAcZZAqpZ/Llq1dOu+9jgtKezhw+YFgZfU3hGI9k2qPDQSdGUpK4sOFcqDPZSFCVq55cIZfuZTP0l08DaxkwSRJThEHaeRIotU27VbKMpGQy1Qak6WDvE+RGtDFIYI52kKg5eSuvYUpeNIr8exfsOWicibnGpRYPfGKBEiOoLtcYmpJYlOCEWDKQzDjXWaeQFlSVHleFeQyhDV6SLXjrD/gx2ixDJzklRqZlXJaPMxddlQHdzmgz/1CSb7U37mZ36BfHzA61/45xjZ4bn3v8zO1h6337yGpmrX4qM508mEZjQlTiROalIxJxABwgt8qknDmCjrsdQVEFQ8+/4f4507t3jmmZfAebY2H/PZz71OK44RPHnuJJmwMCvZujfF1I66tnSyiPVjCUki2X1QMAxBxr4tXntJYx2i0kybhu7aAF84amGZzyusyuiu9AkCjTus63jlaEwrtsy6GbZo0KHGGYuVrfLc+VYCqpyEqPW3OWfRQtEcmondodgU177c6oUhCvShnrwtbQvvW/+c8/zH/8Vf3H1710jAR690cBY63QhbVKymEt0YlG6vls4ckPVTut01glgx7B0nVEfAKWLpqPOcOI5JuhLpLHkZsrrSoduRLCt48qjgiQvnufDsaZaGHTqXj1KUgnDWcFYqjktHojzSKlY+dIk6irBaMzEROgm4d28b001Z7M1xswVNDHZc4E3D1q17eC+59PR5bj0u+dwf/SHPX3kR7zQ//uFP8uCtb/DO3Qcsxls8ffk0UmqapmLj7HkG/RTpJEvDAFlFmKhE2pLR7duMFw5LQBprVFVyIkvQvYz1E8dJ1o7zZ6/8Ec+cvch6N+GPP/82ziqGgy6xUrx+8x77taPbFQyGIQvvkHFINfNMHswY9ixRtzVsaC0ROmB4pIfvCY5cGhBLMJMFqTHUkxoxL2mKivHWnNI5Kq9wKkI0FnUoXsxnJVoqSu+pfMC4NohZBaK1esTLSywOSnRtaSrD3rwhcJ5GKYJI08I9BTKLCTsdhBE08tC55w/5OV5zIetz+3Ofo6kkQgQksSYwC7JAtwKDaBnfhCTdDp1OTbcXk3UG2KokjMA7iSsKpK6o6gLlNZ6a1VWJ0hPW0pBMxzz53FOcOHaU9RcvkB3rwdiyFnqSoiYqFshccuSFS8gjKZuTMXPfIKIAozSvPd5CJBa7u40MHGEQoLxl5cwlUiUZz8eky0Ou3vP81m//BqPJPnk1J+51mBaWpV7Aze98g/E7rzLH4ssS5XNIQqpQ41VDVc5x0zGWCKcDqqZkSUmS0HLvxiOWlk9gTJdvfef79JbPEWcJW1u38XWOVB5VVCwnETduP2Brz6IyzcZKTGeQkRvDaFIyfjBH5DXrK4JIewIRkAUJ/UATLQ1J11KOHh1S7Y/AGMTE4OaWrp/S7M+Y7k9wYUQtJCIMUUIy6CXMZg2NAIOnCdrbZ547cO08qggkrhIU23lrMFk41BxMpFBRiAtAhiHeQ7qyzKJ0OC0xsuVuoSRGtM/lv/FN6Qd/+mu/WlQFqknodiHuSOLQEqoQU0f0h8eQQqN1i7VAHcHXC4w3qLBBakG5MOSlJwgSQhp8MCfuCJwt8U4yGpzA7s+Y74wo92asXzhF0MvQ3ZDdgznHTq+jtKZeFIjpPoNLZ8h6MWGa0etlJLagmUlWjnVJkw7jaU6tNFHcY21jjYsXz/DN197kmedfoJMsI7OGAEE+ndKPEpRsOHLiJBLHw/sjuh2ojGUwXGV9rc94umBleUBdzCkWFWnapZc5Lj5/hencs7m/zdrZ40Rhj9e+8202Tl9gfzrl/rXXefXVO9SNIRGOntJ0tUI37rCI6oiwKG1JQkESBygpsULgrEErkFZSVznSBZimaR14uvVzdvsxTksi4dHWECvR/sY4IulaFpYRUHkWjUUqhbINQSxIh10IE/J5DjiWji0xG+eEWtFJNU6ptgKUZvggwQmJcRJbF0gE1kvqwrEwIIwgTiQvHKtZGWTkhSHwhtU1RZqmmLwgDAakgxhMjAoapFjB6iHeZjRW0kx2cbJBqPamGicJZVMSqR5JIgnSBt94tu7uEQ2WIE0Yv3GLJEpx3ZReZ5myzqlDRa8T0Tm2ysH1W2wc6xEdWwMcYdxjbb2D2WzI+iloh4hS9scLRtvbDDc2eOrpK0x2xzhb0Fnq8/GPfwLhHQ/u3uF7r3yRy+99iXo2RwQpWw/3yUKBsUELSpOCo6sJWibEicQ3lqaoUSrj6MXz7G/eYikKWT/RJ0mWeOql9/Ldb34ZZx27d2/yx//si9zfmxB7iQoF/bDlD4m6wZgG4Up6WhGnkixuGWRSaYzXNL5q9VjOo8oSqTxNs0AqAUoRRQGqJ3GyBhRCeSQG4T2RaFoEdWGQTmCwTPOGYRTglYfIE/d61LXB1Y6lIwMqGeKrgk4SYCPA2tao0u2xQFJaA1XZboQBUwuKvKEqLZHQ6EjyS3/rL050v+tM6dYNy7GlBKUspREUVqJCj24cQc8wHd8jCzOMTWjMnFDleFVS1zVOxNAIgiQBUWEWJdlqhsoFVnniLCRKDA9fu0/3SIifVnTOrxIOYlxhOPjWLvGRJYpa0DtzhPmtR3S7fWajx1QLQRT3Gb2zTbCesLbc5fH3HlH0NLURJKlmvhhzvn+Ct+/f44nLJ3jr5n2efv8DPnTsZUaTx8RRj6XVM0TBKl7F3PnyK5RVxaLu0Osuk08PWEQpy+sddh6M0ZEk6/aYoXnxQy/x8O41hieP8t5zH2VRLSjyhg//yEe59vpV3rxxA7f7iKIoWOlo+gOFxFBpR9eFNFKCaYu0XRW0skAnsMqSdFNsVVKFPUJCksBihEXJgN3RlKVeB6FrbO3IuhE7ed1uOmrLoBtBXRMiaIRGNIawI4mEwDjXhtqcpskXYB1pt0MQamTuiPsJvZVVOkrx1r1HnDq6Srk9xkaefL/C9RWRkoggpNeJGM9q0spgrSXoDdm5twv9GScGDmfhYOLwccDqakLQ95hFg3H7NLXEyS18ExEEFcKlpN2U0oGvczrpKpODEUudATVtX85WNTr2rC6FbM497s59jBb4lZjuUoeDL17l2AcuML75iHS9j68K0rSDSfssHm8jwyWK7QN2t3Oe+NBJpjdGTBclLEfQCETq2Xt0nxdefg/vXL2O7mq+8bXrvPba3+LT/+TvU832OXY04cln3sfO3WsspV1eWKxRRprFdI61U5JknXJ8k8JHpGmHolwghODSi5f4+Cd/ji9+IUDagNHC8dIHX+ZfffbTOB2ysfEjfPvzf8IPtib0Qxh2FJ3YY4VhMDhsTTSHItZI4aoaLxRGOeIooBMlbOUevbxG4CWxL2msanXbOLx1mFlFtz9kNMkxZoERCb1UEFvbJvCVoxES7w2xMeg0QFiBdQVahrjpPoGK0RmU8wWdLEEsrRJkCWY0oyMlNi+opmP6UlFWLR7XaEG2kjBfWGajhkwEWO0x5t0hb+96U/rT/+XXfrUjGyLlEHPL6nJIGMfoJDgkIHawQmNcQZheJt14Lyee/SV0/xmKvXdQTtLph+jAEncDdAjICNs01JVjXli+d9MQ9/rsbu3gZYPdPEALh+5HnDi3TDWb0dQ1uW8oDqZIlaBEe7IvDTu4vZKRzplTEtYVnUAwKSuSTsZzLz3HuTPnuXP7Hu/98Pv4w9//Ij945waXLpynpxtOn1tj6+EdZK9HL+0j6zkXP/jjPN68hRyeIGtmrF56guXTG8iy5s69Td733jPES8f42L//1wmTVR5uP2Dn4Q0+/HMf56v/92c4evYkZucRX/3+iEkxZ6Wb0g09KIUXregg8AKlgrYt72rwgnAtZnX9GG/e2iMdDrn91g5bB1NOnzqCnxWIhWBRGW7c2kPnDQ8PCqytORjXjKaW0Dk2xwVF3rC61CHs9XB1TRImUJvWOmNBCoO2cdvQnxuy1QHNzj4Y8PmCYjohGnapdiY417R66NAhXYCSIJTBuAgxnqFjj3OtKHLdz1mJDHEYEjhBV1iSGEQAKgwwdYMkx7ouyg3Ili7Tu/Tz9E78DOVehJ09IMw8LqgYDlNMUKN1DOqw5OkSRtOKSd2haAqCNEXMJpjFnG43JtSeYDnBzgqUThlvjyhHU5wKGGQdamNY7QU8fnOLYDWgso6ucCSBp4wGvPDsk+wdzFs9k4Xjp4/ziY99kl5k+dYPXkWaBRcuvp/O2mmmewf4TtsjUysDYply/MwxKhwb/YhC9CnGU3TgeObF56msxAWSpz70I1ilkTLg3LMvMdnf5Hd/63f5wlfv0JGCUCo6WQgh9KTGA96YFsNcQxJrrIQw0/SGSwRpzGsPdimnigf3d0lHOcsXVik2Dwi05Pqbjyimlv1ZxebeAVjP3rRhtmjIi4rNRUlUwfLqEoP+EiwswgiU9PjQEwcaJaCRgsYJJpXFlx7qAplX+KpkqZcyGU1wRZvg9h6cbnBSIaVDxAHjrZylfvuprPodFrsT/rNf+YttJu96KH3pH//dX91Y6xCjWFpXCOdoZg3ljqP2y8xGmk42oOEIdC7h+8sc3H+TevOr5Iu7WNPKBZNeD3Go7KmbHCXa03Q8r7h2xzA4mrF2+QxJ2uX+wYimyRCZp7e+TpR1KO+P6K9uIK3E5w1mllPMG1dkXpMAACAASURBVIoqRw9CillFF6ic5ezzH+L6rRs8deUpktVVlNZcuniOrYdbvPD8Mzx6cI/3Pv0U1oxxleOFD/9bvHP9u1x57n1kg4QPffSn+MZXv0Ew7LN+6Rj13gHR4Djj+9dZzObMZxMGT11gpbfM2tHT7Oxu8dKP/Tz3bt3DNZrPfeZLfPE7b3H83Gny0YiNYz3CMKSsW5RoKBTpchfwJINV0o7izryk4yNkNyQwnqZoOFiUKCN5uLnPg1HOo/0Fo7xACMHMWqRUnN4YEnnHkeUIYwVHljOENMxz2N6cs1/kNN7TW+tD0zrslVKULREHApgWFquARKG8ax181qCziNpButZBuaBNeHuPQtGYHBG0A9AoDCid4UJsGKaaQU8QRTXxICKUCjeWzBcrLCZQm4ww7OOyZwk2PgIss/f9X4d6h6YeITWEWqGTEO0NxkNV55RlzSIXHOSag6DLYH2ZeKVLJRWb21OSfozt9uhmS6goYn7nAHopOooR04KqMFSLgkVVIQYhqizIBkNcmNJEApEucfL8KX7+3/kFvv2tb/LSiy9w6fJlAmH42reu8dzzT3P8xAbTyS6NDNi8eY0wC7jy8ifI0pRvvnOPI2fPceLYBnHUEIea3YdjVk+vc+697+fnPvE3+dp3v8ZsZ59iukl39Thf+eIf0smWMEXBifNPMZ7PSGLLYNhBmgaUBC2J+xlZJ6Gqa5LlPvO6oqwEzCRRFmNLQVNMwQRMm4a7D/fYmnq29iZUWlI5S2MdG8uK2MccGUZ0IkuaRHRCibOC/UnJvc0RhbMMjg8JVYArW9V7oCXUhkC1Gu66coSpIhv2oMiZzRd0ehk2CkEJOit9mkVLElFCIIo2lNnKQgNcY7BRyH/6y38xT+ldD6Uv/oP/7le7HUgSR5IpdGgJk4jOiiMIpqysV+Bq0kDS5B5ZSbKOJu4cg/oOYSiQgaWqbVvSqw0eSe0Es3lD5WJu3q+QTYwd54wf7dMloD7IQdfkewv2tvYoqdjfH1E2hlLUFKYiEA1WKzSOLoIKR5LGvPjDP8nIFfS7AU9ceYEPv+/DfOXrX+f8M2dhf8bJy8/QO7HBb/9vv87Jc+eYzHKmkxrrJEfPPs/W7bdIeqvEAWysH+Ng8zFytstsMeeDP/MTXH75Q0z3twnpcu3qv8JbcJT8wT/9bbodwetXb6CzNeJQMh+PSMOI1SNLRD6kHhni413svCLPc4pZTlnVlFPBcD3l9atb7G7P2JsUBA7SNCSqLY0EIT3GtYB24R2XNzKmpSFKQ248mlIYx6JuSOKMpaWE9eWYWEmgzY3JWDNZGOpQMOh3aSYVrnF0AxBKIpumXfN7i2g8zni8NZhFiTUGh8TNGwYrQ6pFgQ6DNj6AR9WOI6FlMIxIU+h0NGkaIENPEEK339BfqYi0pxNJtNlhNNtlaX1AuvIepLmNEKa1lTiLThWlB+8MTS2ZLzyOkMkC9rZh8fiA8iBHLyCqKuZ1TbO/YH9/xGI6xSjP/iJnWsxRgNEN1nuU0iSixgG+Mayde5Yn3/NeAqc4cfI0O4/vEMUxd6//gGpvm5WNDX74R3+KL//xH/D61dusrA4IQs3Syjpnn3wvRi9RecVSb5n/4FN/jZX1M9y+vcPdH7zG8UsbzHKDFo6rb3wPO9sELM99+JNc+/aX2VsY3vze29zZa2ic5+7VG4RJzPIwI00GlDtz5HqfLNbMtsc4a5hPCrpZyNhApxPznTfus1MU2FripWNwmB3j0I8ovcIJz+nlhCwL8a5mauHxqKasLbWVBLFm0AsYdiWmEnQDhY0s3ktsGFDh6CZ9qmmFVpBlmsBr6qLAYtFC0BQ13lpoLPV8AWgoDelgQD5doA5tE054jLVE3vFLv/wXo0uEP6wT/P/9/UdPRP7y+ZggcvT7AUSWpSRBBJJ0MCRe7uLdClm4QrP0BF4tU9z/NFrcRSIOzauKxlR476nqmsm4pipqJqOS3VnKZ68umNUdtDAEgcBbEE0JOiATYCJJhCOXkHnRri+FwvoGZ0BaT4NBa42XKTpULG08xcblE/RXBrzz6iucPHGBxuek/YhKJkSmg08dD9++yo//yEd5+PA6nV6fK1eu8Porn0EeeY4NO+L+6ICehIPJHrOtMb1M4gOBCVMiLKNJzsnnXuL29Te48XDC9MYY08m49Nx7eOv7bxEWe9jRgs1xgVUKvOLCcxsE+xNMoknTmPnBmCQNsU4wN5r60RhTN2wd8m9CKchrSxPCudUO88bQjcGLgEXtuL+5AGERQuKlQDQSq1qshxSOTicgSRLSCDqxpqoaplOLNJZuJ6TeWxAsqf8ve+IFCk85b9BZgJD2sKsIi7lp+U/9Jcx8Tn99nc0HD3F4nkprLp6KWOkZeoM2EBsHkBzLiCx0106gpCOI17HiJGpwBSUD7N5vIHJDo3N0GFIWNToQzOYL8oWnKBbkPuTxnTmjWcrvv16wlAVYU6MDqEvBUhgilKFyjiRq+1wdKagbTyklsqpJpKdCITxtxUloZF1QhBnHLl3iY//uL3Ln2ve5ff0tVk+dZa23RBhKrrzv/bz2xmtce/V18v0tfuwn348m4tz587z2vW9RlwavNKuDDm+88RofvHyeN69+l0IFKB/TTw1eVnRXTzPa2UJ6x9LwOP/o915BCMFHfvrneeLKJf6n//K/5dyV48zeecim09B4NkLJEx8+w/69ESQO7z1CxehqwUKFzPYLotwwmzfsCliVMNeeam6QHc/KUpeuk6hUUZY198cNVWlQQrbePN8qj4y3SC9IIwVo1tdSRNMQhBpjLY8mOceloj9cYjwaEWQC4SyuUfhQ4GuHta2h2AiL9oIyb0jjlDhLKZxrw5pG4Jo544OCd6bVXxhWetdBd45gkVtWM4HQgm7ShQTiXkKUKKSrCaMd5tN7BMXX25PXSxqTE0QpphE421DVJVJCsWhQCqyT1D4Ab0kImBYF2UAjrQMNUocY66icI/YKrCSONM60PbmqMu23q/cIDYGIkFaQVyWnnvkApW14/oWXeO2rn+PoiZNsnD1FN+vR7S1x5+41FqXg4oWnuHjyHPPZDsan7E1zkizj4Tt3WQ+H7EQpzz5xhSiG3/mtf4qYj9jcV1w8c47FfMLcWITq8/bXv8obdwyhsGxcukSpApLhEg9uvEOnCRiJBtBIYTmpQ+JRzVyGTMeWqpwRW081qcmlZlQ3DFYkxVyRjhwdJEurKa5ylHmBdA0raUSDYbKw3D+YY7XgWNShpGJUGNCe2EmyQJF0EnbnFfP5ApwlV5bYCiKl6aUR3lh6x5dRwrVpcCvwRU5xYFg+t8ZkbxsVRDRCIlSOFopsbUiDbLlSDx7Q7y2x92gXEzqa2rf/n1LSX4KwE9JPOjgBzszQaYipH+Crx4jmVfKqIhQOj8C6BmlhMZ8QxRl4h6fAiwAaSSAjXBziywlZeqif8pYsFjhdobwmcNBYT6I13jm0FgQeLJLKeSrtCbzCOw+hRq4epadTGmK+/mev0PU5K0c2yJRmaSWjNzzJq//ymwSx59SpU3SfOsfWzozRwV3OXrrMZDJB5jXxQDHaK/jkT3+Ea9evEfaWSBcFwWrGbO8RFRH9YYALlplubfHpr7zKkaUuz374h7FS8JWvvIJ2jttXt6gDCQp6VrE2TLj2Z3c5/cJR7j3cYTkMMfKAymka2RDHIQ5YlCXLtWD5+BD9+IDB0SG1nLUcNNpP381xhSsFRnpORYqHiwor22VIJwoY9jP2FwXzoiZ/UCG8pBYVgUxYFhITQu0t2SDBNQ6ykGZSIJ2n0+9RzEq8a3BKgneEYYgToFe6NJu7LRk1CJlOIRThux07/5rum5HUUpEXrUY5TCydOMI0noWtCGODswFhAqZuoeGuaRBBH+dKqnqOIMBZQW1qgkhTzl3bkdGCMErYmu8xbWDZaYIoaCFg3hMFIc43IAVSS4QTeCXxOJKgnd5b53GNwQdQA+lwhTv3bnPszBneuXMNZ2cEYR8aSxSW7OzcRYcRKQWPHt3g3LkrXP3uNyiNIYtifuMf/DonL38Agim1innwcIvX376FjpcpqpDTl57koJhw6+GMleUVtu8/Jjo25PR7jvLySy9z4+Y7LEZ7PP3EZf7PwpJrz4ZSnD6/wt03t3mgah7f3OTpc6v4boCaehYqIAwaBv0U8WDK2slVrt3Z5uSTA+y0JheKzc0J3guOiYC3yxFYDTiUEqwgWRaWYLXDS8eXuPFnd+ic3uDm/W38yHCiH9MfhNzdX3C21+HhdkkgPJlWqCjG5CXpmVUiYRALh+5kbNs95vMClfXIFw6RpQyWlymYUWyO8WvL2LkhXVmiOihJOhGl9EznDb2JoNf3NE1CJEVbo6lqGgPOhSSdiJoxttRIoancAoegmtZEkUXIiDwvccrQWI1QEh1rDCU4ReEsB5XgeFcjdASmDfFJJbEIwkDjnMMJkFITGQtZu+GMabNVjQNbu3bwnyqW+z36tv1svvTkUcqq4WB7F2nhwoUT3Lp9m1OnTjKeTuh1HTI4yrde/z4ur5jWNdyGH/r4+/jCl77N9s6jtosYa/zmAVl8hMXCs3X1LpOi4Md//m9ws/4X/MTPfJJ+qPjN3/w/uHX9IbUVJLrheBITB5oHZc7rmyVrVpAg6KQhVSFRnS6+sSytxszvTwmO9Fg+uU65tWC3GFEFIY+3R5zoRexsTtp2vpB4LZGy4WKYsXZuyOlBwmJcsrs1oZxXbO1OORpqTvViFqliXhpCJON9h+8KXJZQjhcMjg8QnfYLKEhS9rd24bD8q6KM+e6C9eUE7yTVeEIzrRAyJA0iGuNwDejo3XNK73ooldaxWCyIg5jYQl3X5LkgSyQisCxmFdZH1NMGoRLCoMBZjbOQpSlxFjA5mGGFpK66hFGFb2pcIIGIsjA8GhtCHWGcRaEINBij8O20AikbmsajyNACCiER3rcFUS0I0BTSkSRL9NePEwqDDELq8QHeOV58+Sd4+M7rTEe7uDKgqCpOnz7KnbtjbFWwtnKkHf6WOyx/4KOErqGeamql+c6NB1RWEQ2Osn6qz37p2J0fcObyOXycceTkSY6fu8jXv/51Xv/6K5x/+gXkypDJ4/scH8Zc2ugz3h8zuTlm+USf4dTSO7XM1v6U06sp98qCjWHIV7+/R3B3ToDjhD3C5VNrzBcV1oGqDCdW+kTW8dbBnCNpTC094ylgG/Y1DKOAjbPrFLMDjj97lDoQyHsRtTJUeUOw2kWrnEleU1nHyZMZD+7OScY1T/zwUYqHOTs7c4qiYJDE9HoZTe1JA0HUb8H692/ex5eSZD2lJwqCYYYyhlnkiKRm1Ag2nIfQtfJCaykLmM1mRHGKDjz5YkLlA0xe4JUn0K2r3vsCoQOKekZdJRgfEkWKupgiRYjXBoSjbmz7EnIxVikkDqklOI+lQSpN4xwKRyw1jRAopSlMQyA0tROgG4RLGS4voZaXQAf88I//KD/42iucPrVEFHdJkhpbjSlmnp2dO3i6bG7fZWV5g92Hd1k9fhkZhFTzMVl5wImnn+QLX/gazjjOPfdepvubhP01Th8/xhvf/xb5ZI/+YEB8/AS7j+9xcuMYjx7e424xZ2VlhbtXf8CTG6usLIeMH8+QBtZVysrFJcZ1xZv3Rpw8EZMnsP9gzqPxHHELLq4PGESax4t9OnGAH3uO9mNWuzHbewuCQJEFnqqQiNrRFYIHpuRo5TG2oZdGRGcH7N6bslNaZtaSldBZSRktPItZjkwUqyrl/k7OlSfWyCLNzt0ZO7Oa1EEQgpISVTtkYFnrhjQHOQdFjVUKV+YEPU1T5FgZ0ltJmS+qdz2U3nXQ/Y//5//xVztakknQsSPqxUgpCdKAclHgVBv4U0GIUx6pNFIbrG8oiwXGC6y14DRWLHA2wnrVspjzBftTxSvXZkRKMogUSdhWIaSQhEq2bEorUQEEsi306cNDyTlH0kuhsvQ7McYYzjz7Ps5duMDP/eKneOv73+Wl9/woSdi09g1CxvM9oiRlPJlC7bA0JHFKXUxpLHzkpz7O5v1HTA42Wdk4iREFZy8+w8lT56DI6Q2W6Q7X+fGPfYKb177Fj/7sX2G0u8OlCxfZ27nP+3/0J/j0b/8On/3MZxj2U5QQNDqmntc0qg2Dbj864MkzaywO5hTacWpjldndEVUIDri+ecD40ZjjvS57u3OOP7lCkGrC1ZAzx1ZYXesw3l4QaU8vSZg3lr2yIpkZ0kFMZRxZ1OX+9j7Se7SCYlLS1JalUBE5z+1xgVECiSdr4MadLeaNI5Se/aZhdzancg2UjlBrusMuK0d6LB/p4+aG0XZJfTBjbzxD9jroboaYTOkEjo4OCaUiTCFIYnQMpilpat8+wU2N7GSAwEvwLqSpHF4IlNQ428oVKwRl7g813prxVPH23Tk3txs6EXRj2c66DudpSqjDeILGOdvGLwTYQ0OuEhIXNFgSwvD/Je09fm7b7/u8Z/1WX2uv3fd+ez393HPObeS9LGKRKMqULCF0BkaQSYoU2EEQQ4gTyLE9uEAUI/YkihEYMILEsOCBAyeSFVuyTZEUHZoiby+nl/e8vexe1l69/DJ4maE0oP+BNVr44vctn+eRVJdXefMLX6Kzuo1VlFQaLVobV3j56F2SOKXVajKejjE1A9u1kFKgiUtpwXwxZ2PzcklSFglxqLO8uYPpONS6q7h5xMVkzEp3Hbe2hOPVWLt+j9nZAU7FYn19mw/e/YivffMXePnoQ/xpTHfJQqo2k2mIH6aEecbFOKCpm6wt2fizHOEJVuoO+30fVVUZhTEnZz6rkUIWgWUL2ndXkWXG5vUuG5ttbN3ETC7jKRdJCggm/SmdbgNEjqoapGHOYBFgCoFt6EymITIKub7mMhkXXGQpOQXqYEERljwZzdDTDF8UjIOcOM3IVI186FNZbqLVDZpLVRrVCsOjPv44I5zm+EpGpiroiuCv/OafTZ7882dKSk6MQVIUFEWBCHKEIy5fO4qJXalQyJy8VNEUQRjPcGwLvaKxmOSUyeXUXZYKoF9uz8IIzVRRiiqBH6GaFmWZoWtVdFWQFQUol+FAKSWKKCkzhd5eQHujiVRCUCWqLghnMVc/d4vh6QBVCPxpD0VNefdf/TFCUal4dZ48/RjL1TBVG0XqSH+O0GtUWw2SeEp1eQulyAjHF7z3wx+iKBpXrt9lOJ8SjmdMlQGZ46BU26RxRLde53v/8v+m2dzg337nn+N5OseHEdfvvc3es0OyeEaRw/7JDE0RlKjoKtRmI9q6idtyGCcRdsXm6tUqs/vn7G62SY8HzAuJ+OlQ/8f7PVTTpPFySBxldK+4VFZa+IuIzRttoCBYpMS9grqh09qtoQpYXMz5eNxHldqlPDKTuLpgc6VGUWRYhrg84FRgMY54eDLBLQUoktWmSaIJorhkNA44kwVKEGNO5jgZhDnMRI4iU5qxyc6dLorrko6mxLHF3EqIyxQtUakkGsnUx3BsshzctkOhKKjaJS5Zp4DSI9ciCmFSrejMJyFZOqOQlZ9qnRSKtCCKCqIo46QfUsjLga+qKWiolJSkFJCLyy1PqWMQM+sLNDdBMRQMoaCpCtJs0a6pWJ0uQWTx9PEz3n7zcyxkynzUY5z4fO4Lf5Gz888YTEOUQhJHKRfjF6yvXEdXJXa3S8Vy6V9c4DgetfoNPMfm8YMHLBYB7dYKZXuVt26/wcNPP+HG5hViw8RWUu69/gWqns1Z/4zXbjdZWd5k//GI0SxjMArQVY2iKHCUkmXdRq+b1F2dUZKzslFDMaD/3ik7ps5hnKIqlx3FwySB1KclbTzvjPkspb5UQ1d1alUL3chRggTNrbGYJFxZa6FbCuFM4dmzfaZcut1CKRjOY5Y7FtVGg9zTWVsMsVt1xr0pflhw6gc4AqxCsrHlcr6/QJLTn0TkecLFg0NatkWSC8bF5YJLI2Kz1aCy2WD2+Iz6TuvPfSn9uUXpZFSw5pYEpYKXG8zznGxRIOwmpSqIgxjd0inSjIIM3fBIcijmKabQSHIu5QJ5gaqZpMGCNFNRUQmzkpFfYuYp6CoJGZk00cQlWkPJFBQBSqkiTFi+Z9F7OOXz3/4y97/zA7ytBpFQOd3fo97eYam7gmG6PPjoJ1x0V/nGL3yTw+M9sjBHJhnrb7xO5E+xnQb+fIQiS5baKyiqjhAqShoj44hCNdGtOvZiRpHE9PvHvHL3TXTTIlIFXsVld3eXzz74gGqtRc1doV3XUYXC04+/i+XUuP3KCr0PTtCkxlxm3NtcITUgmickccDmvRtkwwvU4YKszJFRjoaBoqRUUVlGYVRISApORzlzoXB6f0H02QPapUqn41LEKQd5ikBlGIQcvh9SomI5kjIv0FVBSypUG5ccG8U2iCcJimuj5jlNRyf2E3QJulrSNgX+oiT3VBTVYKmSceqXxKWEUFJtqNgpTMIStVTxOjaf7I2RDJClwm7HZByqDGYSva0RJwp2wyKXOsLSiZMUTdMoC4FQJSUWUFLkAvKCcCEAgdCrxHGJEBpKWSI1nThMOZ9kfHySYJnKZeK8gFwHJCglSEpQDChilLqJOykIV1fQnvbQlkwSQ2U8TehNYccWDGenfO3mVzk8OaTbbjJfDLi38xXe//BPeOXVL1IsnlFf2WU+93ErdVRV0qw5tFa32Ts8IhwfE6YJmllhOgGZzdGVjPnsHMNp0T/tU7EFmmNzvH+f5cYKXrvK0tJdDDnk6hvf4I/+z99FzQPuVnUezzKUsmS5KLn21VsE+2NkDsaqzVrHRZ0mZOOA+mqF8d4cgUJW5GyhYsWSZ4ZGKgue9DL8oODgey9wFQXHFFQ9k9NpiFGzmRU5774YouyNySS4osTIFXRVZaWi4JoVNM8iilMszyJRFTqqJDBVgjjDkOAIiWnpTKYZmaPTVATFPGKqGqTAJM/p1kzmY4WAAkeBg2BG8uEEihL36N8DXWJZFoNZjK2ZuHZGvdO8hKf5KYbj4raayDxAFRLd9EjTGNMQxPFloFNTFeIcykIliQqKXEHTDeISJiOf/rygVBSkVDAVFcMQICVqUoKuoxo5VddmNgh48Dhj894697//Y7ylCseDiNffuMv58z2srsH08CXV5Q637r7O2lKXg+efUJYlzU6XXJo8e//fcPvNt9l78Qx/6mNaLpPZjE63iuVUKBYhqdMjLlQ+PPkEJUpASpzyUr/sqhYz3yfLUiquycr2NWQRUl+qs/f4KcEiwp/EnB4fcPfKEiN3xCDJ8aSGnxacHPaRLZu3v36H8OgcqQpyReHZUUCEjiYLtj2dxM9YCGhpCgMpaZYKpS6xioJEQl5RGPYDRiJjydRotQz2zjISqaKbktdvtFBVlcGpz2mQcTG/PMKUizE3XJ3wdMZZoaBKSUtTUCjAMzjxJTfbGrMgY+gvaNYNrrUMno8D7l1fwaqpTM4jRmFG1zAokpiyiMmFwZs3O5wfjsgti1miUE8gLiCIcoI4or2yhEDieTXKPELhkpipaSpFlKCqKnl++aNmeUkSpZeS0qQklSpRJlnEGrqqoCiSsgDHMigUUKVKWapkosCqqEhV5+yzMacKfL5UyeoGsywj0DRWljzyImB+/oJmZ4WKp+Pmkun8hJu7t3n66Xe4fu3zDA+e0Ow08CoVUqGzmPVxNZuZH5EeHzHt7yFkTr3iYVeXOHxxnySKKYRBvzdkbcdjZ/caL5+GAKxtbKKXBp3ONs+eP2QyGOL7PppMyPUURbfQFxEZl6ag6cmY/fGI26+sUO16jB69oHbrJsMnp9zvJWiU1HQVL5Ys9BLbNVgNC+bABgXHSomuCnwtpZLrHPR9IlXhbilxLZujIkfmMXevNKjYAls3OBzkvLjoI4ICZRrTkBpb2oxgnvLhbEQVgW2ZULHoj6asJxLHFsz8lKmh0G27KIMIqcH1VzcxhUZ/+JKGaeClOa5hcpKm1Ko6S+W/RyD3D/7h77zzJMkwC0nNENi6RqaDzDJQVNI0xa6aZKFPFM6QUgfLJs8Fiq6SZzp5npCmJXmckZUlSVaw8KE/LjgeFAyjAt3SWG3omIZAVVVKCkwdDN3kh++fIUqdzZaJkiUsPJvlm9vUzZR4vCDXBU/3Txktpnztm7/I9pUbHB49QcQJuibIspTxdEytWkGjju24LK2ukQZzVM1hEcwx7C6T8z3c6hKDwRHSn+K5NobTRLE0ShyEJmjWqwwHE7orq6TjPrZX58nHf0qz22bv0SNEHBKmCa6asXmjgb3IyS0dq6JgyIJuw+Hgw2PqdQPbMvnwR6cEOuRKiQl4iULhCbL4MtahK4JQSqZFiaFoJJpCLS7oGQIVFUtVqbsKO9tNrKLEFRoHJ3NeDhcUUUl7tYZZamw7BZ26xaN+QATUbJ0bV6uojoNhCq5tehz0FohUwdJU3KrKosxxqi2SIGEQJ0RhxMvzObkAl8v7FqeAVsNkGuYs2ZKizC83nQIcU6BbFoauIRVIkwzDdsiLlGARIwv9krkjJRgGCjp5CnlSkMU5aQqLKCYrNWbDmON+zMUiJywU2o5K3TPRFVA0FU3XcBoWs4+mPBgGXOm0aVcNzvKA5vVdtnZr2P0JYRyxdz5D9SU1M8eyK5TVCv3P7hPMJjSaHRpLOxiGwWA4x/A8prMFlqYy9xfYrksYhiRBwMXJGV67w8GTj7EBp2Jj1rpYmo0wHJQ8p1QFlmmRpDFxVtIbX2CjEE0vmA6GxPMRSRazumlza2sVRxREmsb8bMr2ao3kZM7ibEJ9Y4n+oxOeDQpKpcAoJdWsZGZbODpMowJbCgJZ0ktzEqHhITEKQYog1FQ0RaFSKqztODRtcNEIZzlHw5jzWYSOQrddp5pkrLYtCDKejhNyCa9cbWF2XIQtqHgGV2o2T/yEai6p6DnSdBjnGR4GmapyOhhzMZjilxKjAFNVUClZ8iziQlJvuPzGf/3Xf7aL7n/4iUiNHwAAIABJREFUD377nTIqmM4LaraGRo6haxiqDrpGEoUkSYJmO1Sb66CCMC10oSEUkyhaUKQKeSTJNQ0UG3+SEscZ/qLAT1MaQmEcp2y3PTShQJajmgYiV8hynZ2tNuvXu2RFRCkS6kKwuLggPimw17tUW5J47DOaxrhVk0c/+gnHJy/JoxQhwavVmI5n1Fc2GE7mPPzT71JdWub04AV7B0dsbN8gjnyE7ZImAULX2NzepdJos3XjCxSyYNY7ptNp4zba7G5ucHBywmzm025ZJFnOxf09FrMRvWjC808vGBzNGR/njIucLCiZTAP6ieRoEVFbq6GMA17sjRkaAooCT0ITlaGWkadgo2CXkttf2mHnCztc21lmZWeJ6MkxG/eWcTLBIEsJy4JRWFIkEZZhI+cRa5aOWTUxQknVzGks28xiUBybq60qTVUymWUcTGKiOGXTNZhPQ+pSp71RIxvNab+6zdpKg/nFhO2tFmYe0V6ukeQFhqaSFCrjPGZJ0VFLgWZrVAxBrCg8niZYZYlnq+gStJoDSHTDZjH2SYqESrOLbukIYaNpOppmk0QJWZEQBjlJIanUHGJfIwoihKIxCwqqepV+XNB1NJoVHUmJJjTytECr1MjqVW7fWEE4OXkRUjM0sos+8ShkdJiz9flNoumMcVKwdWWFYDbiy9/6D7n/w+9j6AYKkIQ+H7//PtdfvcPLh49Qipg4mKHoJrVqh0JRyEsF03GJ/SHhbIJd63AxCdi+8ipFNGE+HZIqChvr2yhIas02mR8xOTtlOO7T6w0psinnz2c8fdFjehjTO53SG8XEi4QJBQdhyiiFO6+3OP5wxLNFQkSBrkBDglYxiJKSMs3xBeyuOLz6Szd57Y3rrNQc8jBmZalCfblCbzAnFpDngkVUkEuJnQJZTL1h4yoKjSjH66o4tot0dQxD8Opra5T9Kc8HEaNJwKplYhUKcRCyvdYk9UOWtzdp7nZpZlCWKZsbNdQip7NU53wastpyOQ9yPKmjU2Ij2ZtG/Nbf/LN5Sn9uUfrffufvvlNKhUEEniGoWmAKjUwtkFl5GUXIErKfTucNq0IWKKRxSZaUaEKSZilS1cmymChIiaKYOMhwKgaGqnI+T1BLi25To7rc5izQCQJJmcaIsGC+8Ok/H9Pe7tC8eYve0SmOZmB6KlnoM+zPqdguXtXCzAKysMdqu4tlOlTrdRbTMcV4QOvKLbKhj+U5VNvL1L0K/f6Q5bUNyjJnPrrA8DyS2Zhxr0e1vs2jRz+mWu/S29vHNEzOD18wmR9jomDqKmcnFyh5xtnxIVmS8+SzCwKlpNQUXr3j4Z/PsB2LXhqj6Jc88tAPqRkmTUVl0zNwgoJTReKrAlkKmgK8zSZqmSMsnWrH4dEff0K10WT11jJK1cAzHKbHQ37h7dtsbjfp7U3Zm4dMNZVZWjKMisstnJB4nSXiOGExCQnzjErVQJQ5WSJRUOhstKl2KiTDOasbW+h1mzwHMcrxT+ek1QJRFIxGGTXxU54zJYtEIIXEMCDQNAw1u8SahBm2UHENkBQYikJJSTQPKIoUVdFI8xzVEJSqSRqWyEIlSxYIRSUpSsooJi4UsihBFRpFqZGXksUiZ7DIWa0JKhWH2s4yDw9nMA7J/QUqCf2TU86fz7j+i1+gv0gx8phCQm3NIujPcTUuPWVKipQGH/7xH9DprlIqJV69Su/kBBKfaBKyem0Xr7mKPxty3BsSxSlV1yKJA+K8RFcKFpM+ttdkZeMqTz78CZ1uDVURyGROgcWf/PN/xuDsnEwm6LrGybPHmJqOCBZ8570DhK3RXrZYaVnkySWt1ahI8uwSTXvRn7DZcrh2exlvFjKNcuZCMMsLTEVQk3B1s02lWaVqOpx/tEf/aER3tYF5pUZjqcniyQVdNN78+nVcz+LliwkXaUyqO/QXCbOkwExLnGaNTMB0sGAUpFQQ2A2TYJFQFAXJLObmV24RjMesXN9GC3OwTKJFjJkLDnoTchsqhkEyjVm3LcI4JywS/KLAFgKrU2Eyi/mtv/W3f7ai9H/8zv/4jlAgzCVLQqBr8lL+qCgIAYoA3XJBcbB1jVKYqD9dBZvVCnkeE0cZigTd0pB5jCYV6k0b3YBmo4qRpAwzya3Xd7h/v8dyRaDLAFvXSCoS1U6pejZPHwzRpU9ebSPiAM1UKWSBrRsYhsF/8lf+Kg/e+zFVVWeYxdiGSRYsiKdDpGKyt/+IxB9x661v8qPv/RGf+8JXaHTqXHvlNfxZH9et0XBcTvY+o72yhdWs01m6wuj0Ocs725w8+YTOzi6zoyPGB/v4ixg1X5BHPsEkZBZOmM1jfuWLuzTLgGgQcxQUbNzucPdmBW/P57SERsNiu6LT7NQZRQHH8+LysE1R2ESgFhn5IOXGV3fRLbj4+AQjVRg8PyPp+Vy86NP2XFpVDelnxEaJOwrZXWtw89VVWqXF6WCCkFCkgtn5nOE8ZNPVMFWF+6cBZaly604Dven9lA6pERzN6B0PKZGsXlsi1XRsIVFsgyCA/iRg8+YaMk0RkUR3VGIhadguSVrSsDUMURIk8HQaseyoeBUTQySXL2dN4po2KCW6oWHpFdBVVKtEc0yEZqEoBcQpumkRpws816XIC+LUp7tUp6mrXAxzWjWH/dzGlQlNNaNaNTCbOnkOumWgWwp7D4659tYNLvaPMDUVhf8fLiZZv30LS9eJgwkOlxEV07CZDk4p4gjPWyEtVU73HvDqF36eLAq499pb2K7L8u4uRRRjqyrjyZgsSNi6eZfxPKSzusLO9dd49pMfoDc6GOUcVdGodpo8/+R9ZLTg+OSUw6OA+08vuLnV5gv36lijgPPDiNMk4623lthuVgnDED8WLNUtlnSBUuQcDQMyJDkqFRRur9TxJwuur3WxrngcffiYiiWI+gtOTwcs9mfUmw7OIGL9526y/28/pfHKGvHphLe/fJW1612Gz/somsa4LImnGePhkCoaKysGR3tz9scB7arN7o1Nqks2tuMS9adEp3OCvKAU0FxfZno2oL7dQSgqp/tT9GUPXRZYlDimi6IDmoo/DRCawX/zN35GQ+7//jt/9x1dVchyiUGCbmc0bAunUaKKy+gHUqLroFUd9EodWQiUIiFbLC4hUrpCIWJswyVXJK2lKkmRoxcJcV7iCYWX/ZRJpcZuTRKnKa4haLTbvP+kT6X0eNGb03Vg9+fusf/pMyqejiwS9FIhU1TCKGScz6m2VhBKQbVuEY0njMcLrIqGqqukQUwuVdJ0jqcJ/t2PfoRuuHTaS3z60Qcc7z1kMjqhogg6GzcYXOxz8PIpuW5QM2xaq10WgwGmltNwDZ5/+ohe74w0WrB59Sbf+8EL7txoMXh0zmfDgotUQaoFjuuh5JKBUjIPc5pJwbNFyeLCR1vkbF8x6EcqnUzyxf/gdby2jauaxJOAXHcg8klEShnpxEmIJkGtujTW2ly8PME0dGo3Vhk/OEHbbjF4sM/bf+ltdjabdNaq1CsV/CjiRZBSW0gmSoZlCxrtLk23oNZsk4YRRZYh5xmxTJGNBicfPWeh5Gxf3SQ/GXLj7VuoSY5V0ZhGORfjmJpt4hUBVgSGJyhLhYySMitZ8wwsW2FpyUKKDEtzyJQSHYVat0VhOsgsvdwi+TMMFRTJpTSyDDCEQSFTVFPBslVMzUTmAb28pOt1ePLshO2lKrnI0QqVi6hg0kswpMV4kPLWr7zBJx8f07UyyDQoCnRZkpSCr/zKt7nonaGocPdzr3L0/DnzaYAhYqxagzCYIMuESnOH3uEzPtl7RjCdMJhGPHj0nJoFi+EFQZLg1KrodhXd8dha2eTZJ3/Kys4G/nTEytouF88fMDjaR1VStLJg9cYuyILZZMhuzeH9+wP2QkmAJNUUVtoNtGjBg7OEtqNyMU24mOWIfoFnCLpbVSbjlFttl81fuktTVzm/mJJNJTXLJTU1cj/FySWKYmBULVLTIDo5Ze21XYLxFDUuKAwNcTxl+80tbry2zZrn0l7zUKeSgzAnnBeIMmOhCK5sdqEMqDVrBKM+luGQ+zEyKTgZTAmnMdM4YuXmJtYoobVTo2VbyNJBrRlMQ4XhLGKrbqLMClxF8lf/+z87kPvnFqV/9Du//Y4qFKRIaRmClqHRqhvoqopZsSgp0EyBaVkgF0TzAMcz0C0XyzZQdAXVdtFQqFQcShaIssDUJZoscY0ckRf8/oOE1Z0OaW9BSY4hBGm4YK3r0N5ZYXPbo5AQXvSpGgJKcIVBYZR49gbdG1e59dodgsE5jWaVuD9BUU1M43IYr8kS1/YoSsFoFPPp432UeIE/9fnyV7/C7/3Tf0JZhHRqdcI450cPH/LKq69x+NlHNNsbCFGippc0Qi2ZMw0WHB+d0nZNXM9gnNv8p7/+y7z3Bz8kLgS3OyZ5WNCMM7ZWBa5iMT8JWHEUVjdMjELjMIhJPYtoJKgoBXmR46cFxcGcakVDa1WQuc/4KMTymlS6HuMkI9NUIiXl8LyHtBWGvmR2NuXKL9ykblaZPL+g6jn0PtrHMBxOB0OiaU5R5mzfrbO12aGz7PH84JSD0wUHe33Wlrs0rndxr61SzEbI0QKzbqH5CfPDGZXlFoOHp6SRzzyKyaOILdtAVjQu4pxQl9T0y0NN11BZUhQqVolnXook6vUqbsuGPAVdkAYxWTRD0xTcegPHraBaJnmR41QsHF3FbtUw5BzdclGjAE3NcEyF3/vRhO52m/39hKtrNmVZoqo6nlRordfoXuvSWnaZHZ5TUyISRUdTS7BKkqJEVZs8ff4JpWGy2m2TxzNGJyO8io2q6ORFgKXqRErJs7NTWq0WR0+foyQB67stXn/9ixx++mPm2Yx45uM4NlZri9nzD/nuv/4jvvKLv8rx0/vYjo4QEpEESCGZXcxYzAOuvnKFkwGcPnmJo6hctQXLtsE4KXjd1fGWFIbDEGYpN19p01xI7JrgYZqSFwXzUUwrlfie4OyjQ7RFwtruCooYsRgtELOCwoCFKRmkJZCSlAXSMDnsXXB2HtG9ssTyxjIiUxiHE/JnExJdZXRwQSEVJkVIZ9tjec3lxrU1/HnAWS/k8bMBWR7T3F6ChkDVFFqNOuViQMOuM3h0Sp4ElH7MYDhE5gV5GpJMply7scL53Oc4uoQT/lf/7W/9bEXpd//Xv/eOVHQsYMPV8PSSStXAqJRUXIdm18ZwNFSRgyKwLINs4RNGI0wV7KpLHoR4nktJCXlEPp8AgrSMcKXBkyMIIkE7T8mUKv1UpaKqqBbE05hwPkELFHIyrEKlECWmooJT4D/PEdfquKvrfOcPv4uuVFjuuuTziCjJ0cQUxzQuqXthjO4I9g/P2L21QZiauJ0uTz78E7I0p9Ws4l+MWNm9wYOPP8GotFk2XVTXIknn6I4CWEwnI2QQMZ6FNDtVFHQ+eXzKR48P8SolO3cr/OTxjFBI6grkQYbesUhsqLUc1Mig3nZRpj4iL5kLwc5anfVOg1uvb0ABk/6MRX+MarpkKhwEMbpYsHt1DWdZ0PN1skzBa7RYdkIcVePxox7x6QXte12cRo1c1ciKiLwfYNiwXanx4KDPsBfiByHr11bZWfJYX6pxfDhl/+CCaDin5taw9BLVFdRvr5KUIWrVQNpQXW7g931OEjiPcpJFhFupoCFxNEDKS/1OEbNTNXBtlUZDw61amI5AdQWWpWCZAkUTFHlMnvpIGaFRUqnVyaMhQrWo1x3iYIgSpFgVE7KCUk/5N+8ltB2N3bbD84nKZB5SVUsMsyQMA6L+HBGEFCVIXdAs5WXwNEzJJjobX7rH6198i+y0j1vVSZOIaDim2TJ+KriIcetNUt9HkPNH3/8IR3hUGl1eng8ZPLrPRw/38BQfqg0+ePcB5eiI3Tu36PkhohxRXWqjIcmlgqPA0fk5ghTDs7EI+af/4kP+1t/5a4j0jPNC5+HFgkae4xguIo8xTJNG26azvsTs6IJmrUa1lEyzEl8xeetqm5tvXcHOC/BMxidD1FwjTCSBl1MIheWtCkbbZHm1zZOXc3TPYnOjTctMyRdTescLFqcDtr96m9LQSJ+dUm/VOblYsNauEhwteDGKyYKE3izm1TsdlmsOtW6Ng70Ro+Ec22mgVAV2TUNp16mbCfqVFhgqtUaFSrfBp4czYs3g/DxEIKg3VMZBxH/3N37WmdL/8nff0TUoS40lLWe5qVC1CmpVC1sv0FwdoVxCwWs1HV3J0YWg0WyRZQXkOa5noBkl0XiMqaWomofbcLA0jfNRwm///pA0ltz7/CZZMKOuF5hayviFj1F1SFQVDY1s4JPHCfUbywTTiFJoxC99rrx5h0c/+DFWGXHlWgdZSg5eXtBqS1Y3b3M2CFDVANWoUuQZe6c+xihl6veRMuHxJy8RqsLgJOT+wZDv//gjxgOJPzxkmitYlmRz5wZXt2+gZgG65XFx+IKL8ylXXtnl9OCEHJieDNi9eZOnDy4IkxIBqKVCpeYhewH17RrxLEKrKjx/MGJaXMLZ7y3ZrH31Gt71Fcq4IOiHuKJk6coGwjN5cDLgtTUX1VI5H495+HLB6zcdTs8jBpMppWvTdjRUA4QjOTqJiBcRWW/I0pV1ZJpzMJkTxRmdGyus1RVcNcOYKsSDHCVMUY2SdanjLZvQsAlLSTKd8+DpEF0x2Xsx5mweQRIQ5Zdcraqn4vsFq6qg4lXQ1ZI0uWwN10RJ3S2p21CpSBwDSiWj6oFumuh6gakLXMtCaBJLNdDUAl0zEKqBqafEkx6aoaELgV71iIqI99+TaEYTY6mCnkd0XUGnWjKfpPjjObbl0I8KYiEwhxntK01m8wgLQdaLSRWV9q1lvv97f8jFcIxXaxD2z3AdleWNdaqddbIUijjizi//Kp/9+BEXkzkbyzZhGBNMJ0x7M5Iw5tmLMR9/ts/FacDjZ6eIeI7rumxeeYW618FRSkyvxrB/RjQbYXY7OErA4cmQ9lKban2ZF4dDHj0/QxMFpbDJlJhqrmNXXfrRAt2wCP2U/Ys54yTDReGrP3+d+u0VYtWg0m1S+gFLGzu4yy5BMEPRLEy9oEgUHh8ucB2LPJdEoc/eyzHe6hp2AIG6wKkYPHpwgTad4260iAdzLM9iNl5wJnO+9a1XsOczOq5K3M8RiwIsHSvMWHYtGjfXMJOcYBZy8mxIqRn0TxY8Pl4QLC6/f3424+61DkEcYYUpV+5eo8hK/su/9jOeBPyjv/933jF0nVyUXLETqo6KZWnYRokgxzYKXEfFsUAlREiB6emYssAQBaouMYSOXEyodeq4G9dYu/cllLTgxcMLfvRJyLduNmh7GsejgGbDvcQiSBVRA8cwIS2AkHQo2Xj9FuaVFTy7pJj6NK/ViXshjSt1dl67wfOP7pOlPoYCnZVVljfXyWYn3Pz6t3Gqa/zkvfeZzRJGoxmDeUbvdEyYp/QmEcPZJS40i3NsLcWPLx13GztbXLt2DYqAYDokmU1J04C+H3Hj9k0ef/YYP8y5s+kyfXbG9mqHupMjXIOzMMEOSpbvtrj4eMLLSUw9TOjuekwmKd1MYDdcBj/c44MH+ywpGpPzE5Y/f4vhvE809InTBMu99HCVVh3P0RFpSaNjEsxCau06qmViFzEvhypX2jpPzkJaVZfkZEruL1h2XTpLNbpWjTT0GZ2nWEsqgwufjd1t8rxE7NYwI5VyMENmguXbN5mPe8ymEVEpyMoSP5BEacH2VpXVjkccJZz6Ge48xWlqlLIkz0sMYMmVl/+GfanlMrVLyJpjgCYsbAMsw0CkCaarYmkWml5Qpuc4toswTAxLcP0X/wt6gz0+/SDgw0czNusqRxOLmh7iba8z+WBEdUNDyU3MikJFdxFliRkIap+/Q62pUsoEo25g1Aw+++yAL3/tDdxqlbd+7ufo7+2hqCpL164xOXqBjM9ZefXr7L72Nb73r/4lbbvO3smck7MxaVxwMeoxjnLCLEfXDVxNRag5p2cDJnHCW2++RrehY1dcFouI6cUp48ERamlxerjg/YMRTgm1Rk7x8IDdnTpSSPw0oUgFDUtBreo8fBlQns5prVjUqyVHC8ldq4LdqPKT/+dDssenFDJH8SOad5Z4+WiPxXnIaVyw3TVJUXE8F38S0OqY6JqJ5QikqlKrWIwnBUbHIfMDjqYlnbpLOJtRbVRoOQarusH5y3OkYSAMgziLWJzMqZgWjatNjHaHydGI4fM+G5+/iqvCWd9ntkjIlMuFRzCakWsW612FpY0OT4+nDI+HdHTBb/z1n7F9+8f/4O+9owB6Ar1+wO11G1XPcKsWlqWh6QpFXmBpAkoV27HRVRVb3wZhohKDkoChkPsLRHeL8emIF4/2+eC9C+ajgs01A8cUPDyN8SyBrhrkAhz70r6pFArpecnWN++g120mvT6d1S5n3zslEjHj05CYhPOjMV/+2j3CMCCYa3ztW1/m2cP7uMs3Gb98TqprWEKj1XDoZJKN114huBigCQFCw0KjJnW+9fYqX/7W17lxbQ2hmFy5usXV27e5ePGIQlHRDAvFNJi9eIpZN/ng0z5v7a4RjUJWdlbIUh+3ZnDj7TdY6XioTs71t2+w/tourhqyN4XT0wUzaaCvlBzPE3ZrNt/4y9+i9DTyMED6ksGzHjLOOE9TthoWsaKj5FNqps1kMudwWhItUm5e2aSm6eS2TtuVpFHGy1nAZsXA7jZor7SovrlDPouYjc4pKzXcLOGsn5EkgoenF1zZ6VBvt3j8w6ecBSFrKw2ePz3FmecQK7zxpW3OLwJ0cjJVEAUppa4hypxpWBCVJZ2qhpTyMufomWykGW5Dw7UVnIpA1wwMU0VmKbrIMUwTWarY7Q6mt41MZuRJiFvtkC+mJNmCSvcKD/7fP+KTH8148dTni/cs6q6gIgomCZTzOcayh9AUVEtQloJkFpIdJ6z+8i0EKs1ul/HJkCLKOX/ZZ/faOu+++4K1ps7eo3fxTIvqkoVlGEjNIA0khWlw+NF7VOsK6uk5ndt3CcYTzCwhVy0M20IrC97QDL7xrVfYuL3N+tYmmxsrdJaX0JEc7D3BtEyqtRpnz/dRKBhlJbcy6LouaqeOpyjQ87nza19gZWeNg9Nzrn9xm86NdTaWq2hiwbuHIaNpwe5OlR4xB89GfPvX/wLrt1eoVCpMhj4HHz3BKGx0qyQwFBq2xvEw4vaqh0PC/iBh4ud4lotupihZiOJCrcwJAoVQZrQck7VWG+vqMvqyR9DvYdsadrtB6Ss8OTlnpJpkJSytNpifTDh50ae6WUGdSR4fDWlEKbYNKzWVSVySFgpS0SjynKbd5Kg/JhQlMsj5zb/9M7Zv/+Tv/0/vaBISWyUOSyZBTqsm8QwFlBxBiVvRETpouoMimojur9H+3H/O5GwI6XOSNKBUdPrDjMcf3MfvLzh4OqBTc3jzVZs8L9G0gofHJQQBwWRKkQvSiynNzbv4p+cYpkOs5fQuEhY/ekJZgm55nBQK2kaLWrNOzS5QZyGqArvLTfSaeynLC46Y9HoYhsJnn+7Rrlc4HJzz2dNTClXj57+0xKYi2bBMXDvn+MWC83zK0YsxXsWmWhNsbF3lxXvfJ0kDFouAyXROupjy8uiCX/+N/4i8WcGu14iTgMXjGeVxjK3ZcDag2gDVNZkOI37wJ8+Ii4K/8Jfu8cb1BrNhzvQ8Yh7nPHj/GQ/2zlCOpqzc26TZqKHZDvVBxKMyxZ8lLLUcRJZQa7Yx84KwKNg7HFC6Gr39MWpZoDkl1T5Y7S4br63Se3CEMk7QN6o0dtY5+3dP0UyX9TfXWFqqs7xawzIUxoc90jRkZ3eLFw+PcCR0by6TjwI0x+RoNKOUko6ULHcs5n7KeJLQ1jVurruU5JSlxK045EnOd49DOq5BzckxTYlj6uiaRFUlXt0Famj2Lquv/01y/Q6Z/5gy7SNMnaOzOYuFZO95zMnTY3rHIW++YeEaFmG4gELnyX5CsFjgzxfIHPxQpZgmyF7Kjf/4K7x47wDmQ8YvT5kchTSvrvF0mrHxyjZpMKa9bJP74HVsmo1VwnRAtd3FqznouuDl8ZDZOONs4HM2nSNTFVsX/NrP7+I9OGe5VicKI55EKR893sPxmjTsgis3XmH/o+9SqzoUwuLFo8ecH5yQF5Kt3XWW1tusfOWL+Cf7xH6BlulEFwn+h8945bUd6rZKdXmVP/y/fsDxTOPGhs3Xf/Xz1GyF3nHCRbLg8L193n9+xMODCzY0jc7GClanjktJ6M/Ym+SsuTpmCbXbW8iZj5pL0jJn0F8wniSkSYZ0KuzUXWZHY9bbFZI8Q84WCNekstkgm0qS2YLOnW22bmxjhT63X11llAbIYx+v4cEi5PDwnBUFtr5wh+mLU8aqSpjmaKrOtXqJkCrpYoReqmzVKvgy5zd/62c8nvzd//l/eEc4GmlSYsYJ56FAKDqabpAWUGl6aJa4TDcjMbWYbH6fwcN/RhI9pNSqYLUYjnMGR0OSucb4ZMH6dY3VJY0kEBT5JS/p6QDarkS37EsZoqMxG/e4+5e/wfHhkBtfe5Pmep3Kkop7+zqH+ZDlcYStJmSU/PK3v43acpgf7fG5X/vPuPX6l/ALlUxR2b7xGkGRYtkdOusOlWaNK1cbHB/1ODpe8Noby4w+GtLe8phGBdLQaG6t0V62uPnq56jXWpy+fA+BSjIe8uj5CWfnExK/4Du//y737t1kev8l7a/eYqjrqIdnxHnIoj9Fv1LF9LaQ8wmHpwsQJet1idfewjHB7lrcvL7E1XtbDKKQjarBlV96g6Q3Y7E/BFHw+tfeZPp0n0mucT7NSBYLglJQK0tWFR0/KzGmKVoY0NxqUenaqEbK/JMjaleWkEsNzt99QjQYU+moNDc3GX5ySJ7mnD/rcX405TwpaNcbjAKfdBiitgwWpzOOyMjGU27f2aFYJMzIaGiCqZ+ilSloFqYOsiyQCIokJkskywbsDwu7q+BtAAAgAElEQVRWujphDKouUXQNYYHMJaqaUqYDRvv/Ar/3r8nSGaXWYByrjE8HDA5BpBMujlN+7i9uoIQFcZyj6ioHpwuiMqPp2ohCousC2y7Z/Pm3uf7FVzj43n12vnmH0rNoXN/ixls3GD07xp0vaG23+MYvfpPO+hbFYsD6eovWzhv8f5y9+ZNl532f95z9nHvuvvTtfe+e6ZmeHTPAYCeIVaQkiKJIUXQky6KSOKWknEhOKrGrwoorLpedxJVURZElVyTLMiSKlAgCIkiQ2IHBALMvPb3v27199/3es+eH8a9mqvhPfN/3/bzfz/Okp+d59AuvoOspHCXEX/7gpzz++Awz8ycYGlURjRBrmztU6i59BgwNRWnttcmOp2h2XEYnp5ieHkQLRfFKexRKJVr7uxiqx4d3Criuz8VH55HVLDvvv096Zo7w3CCN7UMqe0VMQ8ERHcJjCXTNJLe4T7nr8tzTx+n12nSbAcm4zJmnz9GfVDn+2BmE+3nGL4ziVG30Wh1jbJT0UIJEoNCr11lvWdQ26nQ0F10LaOcszk2MUi/1SHRs+k0ZIxknGYtRLVWIRXWip4bJ39hBsjSK+3nCGYGjG0e0KhWcao/FtRzF3TZ+yyU0kWUrVyQTCFiuiI/I7WKVibEU/WqMXL2MqcmoksyR7RMyVPbLNcKawO/945/zpvS9P/7X3y6u1ckkh9B8i9m0iOf7HBw8xGbUuwKNXkClpVBtilSrLsWaTb3m0WoK5Dbq1PdLNI6aBHbA0EySsSmJCD5+x6XabCMhUyh5tFQdt+1gKCKW56ELMmpYJnF8isGxfrydPHp/DNNUWPqzdzk+HqH/sVn0dJTR/gTb29vQ65EaG2H96jvktu5R3N/g6O4dYlNncVolrlxbw9BA3NjF2SgyP5JlrdTDNBUmj2Uo3SxhjJic+fJL7O/v06x2ePILL1M92iMQ2jQO9wkZElPnLnDn5hqT8Shf+affYmlrDTeucO8vr7G5lWN6MExHdx+ejtNTZMbj7Ozusr/X4PnTI0hWB0WBqJHEVHQauwXsTg9jt0Jovo+d713FN32MuMbALz7GyhtXCI9HOPfoJGZHJjFsYtQchs4N0VipEQ57iC0XA4GxsxMU7h3SPzfO4v0cE6cmMYSAbk9gdfuI1OlpKoUGclpCi4Ux++OoIoxMZqg+OCCuqODbtAOJVDjK8GyEeFRlt1ClVnfod31008D0A4an+pE1i25gowYB3a6AW5Qw5DZZU2PchPU9G9eBRtPE1xQOCja2H6ZQsCk1XWoVm3rdpdcQOFgrUN48wrVkUjGH4TOjTIz7+K0e7XoAeDRrHntl6HghZNFDFMWHUk5Rpbi+S/zULLFEFDkZxgw6NFZz5D9dJHthmqGTMeTAoFDc4mBnj4kT01QKPaqbt2lWC+Tvf8L+zi7xoZMkccCTaW3fRdjvckzvMXJulv3tKkPxCF7NIR2XOCzZ/IN/8nus3llDUny+/Ku/SjcAr1PBqjfxJJmBgTCd+3XcbIi+YzEkM80nb3zM1jv3aHcssmkdUTARMwJW0yE0NsDCtU3GJqOkwwL13SOsByUmzhyn9ukiWkKjtLzG4HCYZrNOlwbd7Sap5+eICSH2FlZInR3nzOl5/FoJ4bDH2efPUtk8wMrVEXEwdAU3JCNHBMr5ApIbgmgIVZRRwxFWry7S//gMvmYipUTUlE52KMLAxCBCp0d6KE7pzjaTjx1jaa9GIhEhMhhmfMigVmpSbTm4QsCArONYFpIk0JdViEQSqF7At/7bn9Nm8if/+z/7tp7QEaUOMcFCd0WiYsDMKYVuU8TtNXFboLgOTsfH6lg4TQevYdMsdQksC1mSGRxWmT4VR/Pa9OoQEGD7PoooUCv4tByoWg8Nrh4Cgg+C7KOZIj05SjQpYwz3Q+mQu39xlbPfeoHw0DjL3/2E3WqHxmGb/a0tHnn5P0N16+ysrhF4Lt1uA0Jh6rkd/J7L/lGBu9dLzKQUuvku5kyU9d0yngfpAQ2x3UZuK5QLJaLJfsaGZIxQmGptj9H+GTrtAkooTOD0iMQNDvMVWu0W5p1DovEkwdoR/S2HqVefoHrzgNmvPYYcMvnB33zA+nqd5x5NMnR2Dn8/QOuPY9erNFSdq9c3GRqXaK1UGZ2fwDQjNLw2I4NTlCoVhqayHNwukDw3QSyZwi7UUBIJFF2heVTk5CtnqVXKdDoiktZh8MxpFE1gffWQe9sl6ocV6tUqxCXcQgecKmuHLgnNInR8iKWVAzyrRd/4BM1ShdR8PwPjCQJRIjLUR7vaYPOghWBICKpMSIdOIHNYbaM0XaotkVhIJKRJmEkZ2XeIKj6aIHDphES7q1KpWNhNB831sLsWTsfG60C31KRT9+nVWriuTTjjMzap0jeYhmYNq+Xj2gKK6uPYPsWyS9ODju085MHjI9sghmT0sMTQ6AyRQQOrWaOxso0RjzL2wgU6my1quQq7i1tUWl3OPfcK5598le2bbyJIAt1Gg0qhitPrcnBvgXgqyiefXWcsm8WqtWmttzgKRTncOyAyEMXttEjNpAgclwdb26wcHjIUkpg59wTZZD/5/fv0T53Es9pEs1mOqnVm+hLUc03Wf/Q5IQmMtkRqPERQdWFQJDOaRI7189lfv092QOLZr75Mq1hG6RqEjw1j7eYoBz5EAuprTYYvnEOouWSP9VPfL5MYGQLFReg5GKE4YiZMcXmbRDZDaG6EytIusfE48UyYer6D0+4x8tRpsqMjNDbz3MxV2d6qUskVaOsaqlelWWyxvNmAbo/M9BSH11cpuRKpYYNmuYJShhNPDeDpICgeoUSCwl6dw1aHdMxAVVVU1Wet5GJ6PrVql5oT8I/+h5/TZvLEZDgIRAEjCBgKXDKaSzQqkwyDYWjEsuJDALss4Nk2eiyOHBUI8lVENYN+5kXElsfRtTcJGw6YIkpIxPM8mjWLTltgL2fT7MqsNWRUXCQhwHckJBGcjku9oXHyxQliMYVuMYeUnKO+cUj/I7Nsv3+X7DMXqX70OZYHuVCDbHKYmfljhI0eyx/fRTMjVGsN2jZM9UeJxBX+4idb+DUbR3GZUhSMwTSdvRZrlkVcl3GRCA1FePnyOP1TZxg9dor16z9maGIOu9PmYHODk5ce5zt/+ofYBy00OYSPgCxp+L0aYsZAUQIGpmfZe3CPYrvL5LFJJDVCcOBjpk2M8QTSXplqWOXd164S6lnMnu0nOjwICZOQrLLy3Q+QswpWzyc1Pow81oe9XSF1bIjrf/Exjudy9leexG/WEWMq9XKd2HCKxb/4nJUgoG9QJprJcPdeDpmAcyMaKwWYnRqEap6DkktDsElqBnokYDPfZTqikRqIoGsJlrdzlKodDEWh5doIsvRwX0gWURwfTxLwcFGQmetXkGUZVRRxKxVm4gpCN2BiSCCZNlFUHz2mokQEet0A1ZAJQjJh10UWZLzoNNGpJ8nfeBuxvU801qPl+piRMN12h2ZDJn/QotqWKPbgoAJRQ8J2HQJXxNZE6gd1WnaUX/r9Zyj87QIDz8/x4M2bjH7xDKJjUl9aRRzQ2L+dRxzyefnV3+Du7buMjvexc/sGXaGHCuSqHorrce7sIG2i/NVr95FVhwlBRqJLWVHpWB66rFMVfIywzuPPPYoZ1PnFv/9fs3jlTfqHZqmVd6lV6oRCIfaPDvj8vRuMoRMEwkPyhOLS9+gQVqGCOTlJq1xl4f0FRkZMjKEsA9FRdj66RfqFeW69dYVH588inkzx1//bW8x7ATO//ixKMoQnOmx+9waRE2m6xRxdWSGlDJE+NcR+Pkfv/iGflqo8/cwxYpEYG+/cZeDZCYxIlF6hxedv3cYyFTLDGkclqLZsYhGRmCIix0J0q1W6Pah3RCKSw0gyxlHdwxEtLp7rx1IM/J0yH2yXMQMRCwFRAgsf1ffwRJlQz8PSgUBCkxQOG42fz2aiizKdno2rCbQdj7GwSVj1EEMqttWmtC8RjoAgG4STfchCP5HMSYzxk7jhMMWNO3DwExRTRExJhDQVX3TBFrA7Is22gyqLIHrIkonveUgI2IFDSJCJD8Q4/UtPUsgtsPSdbU78xhwoLUJn+qjXNxm9PEZPrNN1XFKDSR6sOfSaVZ5+YYBWT+CZ35xn4eoNKu0ubqFLr1pDfmQAp+mQwiZrycx9/VH0WITv/+Fb+KIBrostgdRoM3/pJRbu3uPkxRSpZJaD5TsMjk5QK29y53OB+ctPsXL/Lo3tEudffYzOXhFj8Dyf/fhzZsYzVHYK3Fx92GqPj49Tub1P0IDYeILl9z+ldrvGia+dZGQwQd+UwfDsFK//yXuMqTrRmSQjl0+R/+w+oeNRQrP9vPXaDaZMja1bm5SjHjnbZ+fvrhOxm9iKRL+kM3pe5DAQEGU4e34aZJm15RKeG2D1ROYGXNY3D4mHfVDA7QSUBQe9IyCKMjtNn61WEdGv4gUuBDLjUZGqGyFuBjgNj2K7gazplBAI4xOJ6riuRRA85DP5koYRDkjGXfS4QeC5dFoitmWh1gVEvQ/FTCHLWVLTz2KFo0QTSZY/+HM05xAtBo4ikkylcNwGtKFW6eEJD9E2nufS6XmoqosYyASqhKHD5JeeIpLSsdZymP0aLbfLzK9fwm5bqO1tYpdHMV2B5s193L5xfvijd9g/yHFq/hsok8dIRiTaa+uE2i3CRQ1nB65tbxCVAnRR5NyTE8hTad55a4XOYZl0s03BkKEaMDs7ST2/R7NRJ5HKsnbrU8bPPE9h7TU27rc5/8Sz9H0pwt0bi0TH+mD9kMhYHx1HQfIl3IbFrQ+W2G32mDWHCVkGPgGiY6O2unRzPXLSNuMnk5wZijD14nFKDwqs3ixyri9NYrKf5mYefSwNqs8nq7uY1xcYmxlmp/OQhvrhh2sIng+yROv1FYbiAh09QVGTiZsipy/M8+DGDu12CcH1SUR1fLuNH8hEYgpN36HlKWzWfHQloNy1eOfaIdgiQiAiSCD5DqOJEH1pjfWdJhFVxxY9ejyUgvSPmRzutX7W2Pn/W578598OhABBMFBwiJigqD5RTcANZERcRFlBlkRERcT1LQRPpm0fUG9uYTbv42s+kViPaLQPx21jtcvUtSfw6/vYjoLj+Hi2ykGlheSJ+KKI4kh0JI8mESbTFq3NNnN//1HajR5/+/oK5YM2o31hwqkYTrdCoavz2WKOJx4dYGyyn/WPP8Z0Ojx490PajsOV64fk2z6r1Q74Lk8dG+bi154lEmkTy2T57EaOZBpmmz6D6RDHwhLDFvgZiU4nIB4X2b57jYP1TUpHRdq7RyyvrDMzf4LiwiLX71fILx/QF09iFet8dHOHleUyOwcVkhGbi5dnkYoKgqRjpEVWv3eFbtFiAwGlVCI+ksAQDd5+8zZDXZ+pVy5RWtph+qUL6DNRQpFBfvTWbc4ek3Ekja5rMZaNkC9JpMIe46N9nHhsikSfQt3pcPHpU4jVEobv8867G6AK+K5HqeUxPBDnxNkU/RNDxGMCRzmPc9MGI6cmaRZbtG0H1xWZcKE/oeJKDi1HphdYRBIStiyRikepVZoMhUMohspofxy72USWVVDACQSiso/oCoS0hzok2+qh6jKCIKEpAYJuE4lKVJw67s4BjdJdYloeUeoSy/ThSRaC1cD2BTY3OmgIWCg0Oh5BIFFo+yjIIGrIno3sQdeyCe1XiczNwoCB7Jl89PYyna5LVApIR6IcNKv88Oo2Tzx1itkTWfrDcZZufcKgrpNKjtOUHN7+0SYHPZelYpWnnxllIh1j6otzKP1JDE3lxvsLPBrPMJhRePaZaQadLuVmm9ZhjoHjpylu3aC0ss7R2jUOy216lTJWILG+t0VpN8fn1w4ZH1PQ+kf4m+98yP2VCvcX92l0HR6fDmN6YQJH5aM3rxKVYeXmBjISK5Uqg4k46ajB4huLrO8ckaVLNd9k6Pwo6SdOYBUOID1KuJjHMWXadpNUMsJBs4MqK4RDAV94op++4wOUQgGZEZ35U6dIpWxufppnP19GkBS6PZdSx+Xk8QyZ6SEyWQNVFBAsi+PTMiFdpV6xkAMNXXCYTaokYhqtQCaTEvFkiYGRgYfcdE3Gbvlk0xF8xaPXcfiD//HnDLr/z//129/WVA3H7jIZV9FCIEs+gRJghn0iURUjKuMJFqF0BEHxkPQECTOLl2si+DdQcSEQ8b0SrfTX6eaW8II2piGyWu4SVVWado9SDfBlXMFFjIeJSCIXf+0XWXntAwa/MMfK/SJyrUiwXGRYdbl61Ob9dxfYXG2wtd3l137jMr3dde7e3aeipPEPtui7cAo38FlezmMHEiHRo1JrsbNWI/fhInPf/CV0SWF4NMbrb94nnIkzd/JhsDvwzBR7y+tYPR+reIeupSHTIhSPYI6OMziWodz1+PjTdV7+0jHatse7n67QrDSxHR/Rlgi8AN3Q6RdsYhdmURQbodgmOpIlNppF2DnCbtpYosrNzTwzbZ+e5BDrtbCnRzAJ6Dag69TpVcv0WgJNBxADHMdjJBpir9ojX+rSrtbJ520cu0vTcsgcm+Lg0GXsZJqjXIXAeygBUDJJ7lzNkY4HdJbLDIdllNEB1t64zfnLM+zs1/FEn4oIdVcgacoMxxXyPZu5yRFkUaHQqGOENUzVxbQ8bu21SIcNunYPt+PhIzAaETBMEVGUMAyfUFQnFAuQQiK6AnJCRZAVFMfEDgJ6lRU0jrDkME4jR3TsLKv3Gtheg2jaJOjL4DVtbLtHxxIpd8SHkDcV9MBH7NOYvXSKg892iMyNUi/s072+TUpwWe+0+PTzA25f32RxPc/s1CARqUp9e5sbNw8gHCcq23SsNu/+6AGO72G7FpIfsLvYwl/fZ8LM0pCqyHqWWlBnZa/C8ceGqVeq6E0HfUyjU26jqHkc26PVOKLTFcgMZhi5/CSyLDM8PY9rlzh7JsNascOHP1khIYdpCAGaEBAIApOjSQZmshjH+hiOCwxcmMeM6wTVBilHYW95jY09B3lQJlF38a2A/pcuU93No7d6eKZKMVemcNSkJwgcv3iR/c1dRhNQrFk07YDcjk29UqNS7nB/qUS8T4FokmhCpdbo4TsOPj6u61Brw/LCLlODKZpLeY6dGiUIp4mpKnuFNpbn4QYC+Z5L0wqYSokclGxGBhN4rS624FJpdehL6ihSj/J+B0vV+cc/gxLwMzOlv/pnF4PAblF8cMRmDjTJR3J7GCEZVZaIRgMiURVN07CCNqaiokcl7I6AIHRRVPUhf7sdoVU4IjUzgGho9NoVQmEDApmb1QjurWXubFpIusmne2XmMgmGR0J4uz2OvfI4ubcXGP3NS/QOm/jRKMVimfxfXuGG7iI+ZCIgCAIvfOEClO8iihAeOYUeEvjhW6v0um08z+W0IxPRJY79N7+EbbUIXI+t715l/tXLOLKHK4o0HixhWxZer4bWn6TSFEiacfRYCN+xcSWJkZEJakENrWvwxl9/Qjqtc+ZsHCGs88HVIu1ak3LJJap4eC60PBdXlnjxUor+M5e4+x8+I+700JAQZQtBDXHvoMyJ8QTppEmzAXbcwQ6pDM3P8OF3P+Gw0aFnSzwyGeKo7pIrO8yeitGquewfNtEUAcuVePLZYwxPjPGTP/kJo75DVZCoaxYXnr1ExNDRsmFe/8N30GWZsWMqcgeCuEntxi7Tz83z+fYK54Zn2b+7TfrMENfu5BhEeKg46oKRNlD0LqU9i4LvgyShiwL/9FvHGR/sZ2/zHvmVDh2vR2CJSLJH1NCQBI9En4JpKAgKKIKPHFWg5yEpOoJmoGBTFCYQqgeEwi2EMIRDCRq1Gtlsllajzv33yuw3HJbzHj3RYmR4nnarxvGnp9n5958z+8plqvk8ffMT2MUeZbdB7/NNwudT/ODTIk7HRhZ9JEniK7/9OKXr9xgYn4WwT+DL/Nl3P8dEJSEGzHR8zv/+L9PRQLJs7HyH0lufMfrbT+PrAbW1PLJs4xbLhIcT9FBwjtqIyTg7BzUm+hOIhkStaRNWItz4q+vsShbPPz3O+LEx1re3WdpssLvh0W/bNHUPVwnouhKzIyLPvHyZH3/nGqqok6WL0hY4+c1LfPTvPiaoOUw8cpz06TTr//5j/BMJTnzpBRZ++hYLGy41y6U/EqCrPoW6Qt+ISeC4lJtdvKZMTwBZcvnaNx5nYaGEdnOTitejRsCl584THk9Qr3WQEPjojfdIJE3mjo+yttIklTVp3SkgjoQYmUpw8MEaiZfnOFw9ZOewRTQQMS2fQ9WhG+gc7xdYPPQQfQ9BgFTcYGm//J/MlH7mULr+nX8YtEqrtHKHbHWiHN68j9cVkBEIKQGq7pIZMJEkAVEDVZGQJOVhl0cAPS6Ry/cQ3SQhwyKSkHFsSPU9lNmZ4YeA8tb4C/zp//T/8pNlm68+OsP441N0Do5o18qkz5xn64/fYcF32ZU8EoHAKBqJiMutrkIQONiBhxCI4PnopspLTw0TEmQOrh/gWF0GLg5zWLO5vXaE2A5IdQUC2efV3/0FzJEsle0dwuMpivvrlBe3sdJDbHy0wcmzE0SEPF46i9IDTw7xpa//Drv5Iz748PsEH2xw7CvP8qevvQWWRCTq8tKzp3nr4y1O9SnYgkMiHmL//QJH2TCO4zAaFjj3a5c5unuLuJEgdf4M+5uHWIqBXSgwcfoMlXev0YsKfHx7k0bLIZmNcvnCEH/31l1Qwlw4m2BkegzdVfn4k2Vc4WEnqZgr8rW/9wv88R/+LVkhzJgg4fsupcAnOmHitrq89PUXHyJ27zbQp+JsvHObUF+H1f0ekusRPz5L984K00+P8MOf7pEdNFBkgb6IQbdjsbLZYCKssGY5yB74kohsw/f/+Jsg2dS2F2nXa9z+cB9XEQk6D9v9iqyRHhIR8QiZCqLoo+gKkgCBJCF4PpIYUCjqhIZlBuP9tHpLqMYguHUkSSEcMggNf5nX/sX/zadlgaWSzS+EY8z+2jnu/tEnaNkIo790iff+7XvkQzJS4BPuOozIYbYTNu2mCLIE+NiejS5r2L0mv/uNR1n5bO0/mnV8lIzKZskif9jG67hMugoTmkv2y08SP96PH1PI37iDi0g4muFwa5Or1w544ewEahxark80mWIgM8rFLz7Pn/zbPyfr+MROZXjrT94mV2+j6yoXzg4Qj0l89MkBmZBMui+Ebvl8vNvE8h0iUY0nBw0iZ0/hFXaxKw2yz7zI/uYes8eHaR0dIG5beCGTD2/dYCcv4ghdxkd0VFVnc62E5em8+mIWoR2j4DXZWimjmBoHhQrPv3SJkBjhB2+9w2w3wBclVMshNxxmIiVz1HB48defZev+DsP9BrVAZe27HxLLmhRdmboA8e0e2eeHaRS63FutMDEdRnN9ZFllY6dE0xIYkxUOLB9X8vE8D9kPKPT+09run/l8q+38+NsiAo5o0z7oIKVHaR8doioCqi6hyCKSGOD7YDsWiD6iEKBqOoIsIhIQkhXW19uEIiFSo6MgthBcD8ex0U2DSCyF2BW48c4SpaaMe1TicGUXq2cz+swZdn6yyK1mA0PwiOGiewKWILFiW/RE8AIQPBFZlvCDgFceGePdqxuEDQ8xHWcwHWF745CIGkJPqqRKbR599XHmv3SRfLuIYWo4iktle4doZoC+4/Nk4hmG+j1yr99n/JXHMGSV9oFL38URtrbus3v9E0bGJhh/8iSrb39OrlimLknMSBqGLCJJTdr3euhDIuV8QDIdwsZhelymzzB5+80V/JhEqWhhHRQRBIGgUObm7RyHS0sMTWa5vbBLte5jiQoxw2Pu/DEuPjPF8eNDjPX3o6giuffuM9eXYvaZk3z00T2+qMU48DqEj9qkR8OkBzXiAyr1vEM0Y5CKRTlc3sS3G9T1gNZKkdrGCg0zjOcHRFMKh5s5pi5NYXUbDIz3YcoKpXKD6KDB9laLwSEFp6UyPZzAsLr4gYPhynzjV0+jyBKa10NQYW+zg5yO4bS6GLqIpASIgo0kiIiChICHIEhImvTQdmOKyL7H5l6DVtkhnDQZPvMSXnMJUZDQtRBCyMRp5xh46WU+f+MmgQVLokD2sIppBsz8l1/g5v/xFruRCHOuy3AkQsPp0vQ8yo5CgAueRwDgg0vA80+M8dmHuxhJkz5ZRKxZtPMNTk0k2CjazAQeZ7/1LGNfvIAUVwgpBhs/+TFmOsXo5Dz1nX0GTh0jU6zR3a6iD+sIXZ+UGQVBYWXpc9qHd8nOjiF0RQ6vbmIZPpoYZ6jQRFZ9xHUPR3cJZJFuy+aRKYOoJnAsa7B1u8XyRv5hzcgJaO8uERJEGkt3eOftHUKKRaOwy51tG0vxUXx47NwYs7NxHvnCOaaCFunZUcprJbScz9O/8giuVWfYSNK4doec7JLdqeNOKmSHZGzXwVdUhjIam1tFyB+hxiKIRKi8cZNG1ESOivQChaDpoh6bIG645JtFzs31Uz+yOSr1sF2fVtdjdDyJddDh9PkJcoUjFNEnjcp/9U9+zo3uwoMr3/bqXaLmadz2Aus3trCFhz52ZIFQXMAPSWiBTzQZxkyGESUBFxdREQiHIwSWy0HOplSqkMyEmTzxKnZjCT2kI5kqciCimTDy9Ass7u5x6defpn5th6nHRjH7dATgkRfPM/f0NHNPXUZSbMxKnUHbJGn1qIgP4fGu66IoCvf2ywwi4ysyixsdhHSMuqJyEAQcLFYJ+xLZbhshEiZVCpAGTW790Xus3ytiX1ln8OIUPb9LfWuDzNA4xsgQsdE0haMV4v3jXP0XP+bBVgH7doFrnyxyr9KkI6pMTCfYKdgcVBscH9XQBwzaPYXTT46zU+giSh7qXoAxLtG2Rdb3e5SqLkt7dfYLbRa3Gzx/VqW+53Jj7YhIQmd2QCYodNhvw8qDAsfGwoTTCY7+Zovtm8tIvsqmIVI+LBHvelz63ZeJaApRA6YvncJpFc34UvkAACAASURBVNBiEWKRgHDGRPFcdFMklukjPTKIdVgje+IkRzv7NGwPX1MYMV38Wo+jezV6zRpuqci5Lz6KbpU4KjnUSxbHjw0SiC1kWaHpB5y6dJrnL/bhSj0wUngRHWF4mKNrDwgICAJQQjKhhEygiGiqiN4XItafot1so2gGkmIjuBq5/R6O77CxXGNqwiQ09xheaZtwLInouyiajN5o8NG9A87+8hPMRgOEmkP01Ys4lRaZYxNcenKAmZefIDGTovfxGnHXIyoK9AtQEQR8ARRFQUdgdbvBE/0+d7Z6XC9XGHviOF46xicrFdqlOmc9hdnHztHZzaGHDaxmmR/87R3c1Ty+Z6EOxZAPj3BqLgNPnyMIqmjpJIMjE6xfv80Hby3S2vG5/dkKq/e2yMmA5zE4GeJuqcFA0iAV17lScPnKb54nnjbJ9ZLIeyXkiEb/fJRcMeCg0CRfs9gqOCxvN4hoEmNxkxuHHbYqLmendCY1hVzX5cZaEaluMTyTIpo9xuKfvUN8epjUU7P85Hs/RdUTDMz3kb08R3S5wfhXz5B0XTRdgrjG8FgWt9sknTCJJUL0Z2OIaoRCu06i4VJUPHw7oM9QEYIq9xeqTKazFG8UGT/Zx/jJUUr5MtVOl1jZYuaRKQ46PbRA4MRsH16pw+/8vJC3ujv1bc1QaOWusLO5wvJKG0EOEPAwVIlA8TBkCSWqY7dc5LiK03ZRNRV8GSQRx/UoFXrYvkLlsMbYZIrQ3OME9XWEIIoeTtGqFUiG2vwv/9c9zkVt5FAE9Via+rt3sWttuh0BtyGS/+GHpGfHObqX4zPFJieKSL6IJ2sPqYJBwIwPEyMGA0mNsWERDY+kVCcajbNRbNBAYq3R5cbqDve29rn+8TpDhsYr//2vkDyfonZ3j+jIIJIjsfjGXTKXJ4hEohx97y7dmEXyiVmcj/e4pbv0JA0zqvBbv/MMx88cwz485OyAiivA3S2P6UvDrF3boOH7pJIS92st6pUe/S2Pnu0hGCCLHkKvS1wWEDSNaBLGYzohQyFmmGwftfFUEcG1yAyZ7K3vkpz0mLw8j+irdGolQqKNrlqUPlihcmsX4/Qs92/dZeL8SRzZxMpXCXyPrtcmM9rH8oM9BiYG0fsMauU67VqR6QEdt+PgxAUOCiKnTs9AsUH6whg7t/e5ttRitOcy2BehsnqIKuqs16r4bQmjXeCXf+8NksNfobbzPp61i1jz2VnYJfACTE1ElnzUmIwqqgQECIqAbTkokoIgukiyjqj4HOz18BT9oQlE9ZkcPYfTXMWxfLREDN+ziQ5McfzyPG/9m/fp3s0RenwQsxEgdCtY5SZWsYtleQitOnZd5FavRc6QqKkCYuCDpNILAnwRnnEFhJMGo/0qg0cijtrCkG3WVhsIksqWLvDBjQWur+yx+Pkq6zcO+dq3XmTk5QvocpdYdoh6r0792gFH15cwpzP0lhsYk1kiSRUzqbOwd4QtyzRl+PKXRnnk5S/gNUqcmM/SftDgg3yJTExmyIhx5+NNDLXLVruH6yiEBQWtaHMgOAiOh+UrjHgS0biKJ/lMDYfoVwQMXaCer1P0FHTNp227dF0Lq1Jm6rnTeL7Fu6+9z4XpFK7ZofD9TSKuQmN1CWtwlHQiQOjrx61V8Bo9RMFDQiE23o/gtlFlmeyF0xws7yK7XfqiEe4eNdFkncFIjEQiyfC5JLVmg4XFIxJ5h8GICiMpigt55s7Psbua42i3xNDJPr75n/+jn28ovfOvHvk2/jaNWoVOt4owMkh1vY4nKSiSj6EraKEASQTZEPDxMI0oiuLjBy6S5eB2LAoVEXwfH5PNxU0uXX4Ep7aNpAt4gYMggiQn+NrXv8iVP/4UR3SJmGn+w5VVOsS4t7HDQNtlYbdEZ7HCgmLjewKzEyGef3qUrt/isNbg0skRRgZFooNJWlaT0OQ5gnoZZUfgYPGQS8+Ncz5sYwcq6dEQ9hGc1jwufOsVBENDC8LImRii6JG7vkttr0x2JIo6Osza391hfTHHI8fHSZ8bZWC/xl63y9e/MY8Wz7CzsMTNT3cwrIBQJsH4gM/CT/cwY2EC2yNuCmwVbFxXJucFuIaE5zpcGIxjySGCIEAzHCL9SQqugNdsYiZlQobC3ESIZ5+dITN2ltRwhs2/XKb2yQ61/QbatIZpqNhaEi/iQFInK5qkojHcagNFq2OHdPZyPeYuncZVAyanT9JpduneL1C5uc75X36KOzcPmRjqI2lk6Dwo0v/C0MNS5pFPus9k/aCA3hM49sqTVNtFdpsOta7NoGHgxVXOZ97nzmu/T3IwiX10wMYnizzYa+DLAZr80BKsGwqq4aPqGj4e2DZaSCbwBeQggG5A1zWwWxZiEJArNUlGXIzzXybiF2nVSqghA9/1iYstonMp8mWLkZNzfL5W5sZSju3DHvndBjs3Vri1kmfJaiMrIXzX46svjnD6zBDXb+3hOzK/89tfQMy4OBUJIZli5olTxPvjrL+xyqUnphnIyJw4PYhuBtSOLJ70RAZSIcZ+4RyGanL3n79O9sVzBLrMykerhFST4S9/kcDq8fqf/Rjn5j7zX3+CR56+QOH9JeKjEmeeuoRsN7jy/jpHC0WGpkLMz8XRbJnW0h4LHY/54TDbNQu3aHGvbFPARkIhI7hMjxpYgU06FSMyOUIiGnDkyYTVgORgHF+A5x8d4OIvvkRcFgmKPXb/doHa/QM2FInMgEhS1TkUfNpKlWZNYHKmn5XXbpG8NIHqdVjbg5nnL2Dt7pK5cI7qYov9j/ZpFvNEUxr9ZwbIX69y7hdmEbfrJMYTxOIyjUoNJTHIcF+E1e0CJ56eRm17rNfKbO0U0V2LIKnRbdv8w//u56yZfPpv/udvo8sgB5TulWgENoWiQxgBSfZRdAFBCnDbLqGYhogA/9GUoZkart1D8AVKBQcPaDoOvugQzcrETz6HX8/jWi0UPUIslCQqBiwU91j3ZNRKlXSjw0RU53hflDeLBZrI7OIgST6GG1DrdJieSjOZCSE4IS698ASdnTU6gY3vh1l6sEKorOKnJM784lmiYQd9fIhEyqP3Tg5RcKkYAcUf3aP80QOqG3tEzo3z+b/8PuuHXcohH/nuLlZQZ/6rz1H5cIlrzSIrn+7yWbsJsk/j0ObDn95ieauGL4pUXZelXIv1nEWHMCOqy5DssVD2sSSJlt2l5wX4ooztCwSKS8YQ6FU8Jqb7SCUMpqaSWBt1bMdFxMUrCrx+dZPdhS10u8HE4xMMvHSaxOVj+GaUXKdFWFPp+BpuUqd6sEdPbiJrWRLaOIeuw9nZE6hhne/9xXsMDM0QjghUF3YZ/vXH8KwOWS1Kc32X8IkYQ49OUr5+iJkK0+vW8U2BkydPkZyJ8vGb1zj96Axnnj5NebeAWbUp+SJT3RwD01nscoOGobK/UeaoCJImYioikhwgqwq+6yLKENIUPElEcAVEUcETLFR0fFeg0rJBVPB6MquLO5ydH8Vvr6JoUQIBBDSUqIkSnWdtrcrqtVVGbJfxuo/WbBIeT7LU8bCQkGQfz3VQhIBaIWBscpBeu0l/NoNb20VqtVD0gHq3xdtvLjOVSBMb1xm4OEm6P4LgeDhX88z0aWizA6wdlKktbLP6nc955A++zJ1//QO2r6xxRwmI+gHhRpXBF8+ifrDObcmhsp/n/bfv0lRVjqoOm7e2uXZtG9vxaUiwWg5Y2alRagVIXZdToya+rLKwdYStG8Q0mU4nQJZ9fF0lZiqYnoqhO6SGVAQ/oFZ0ae7ViaWiON0G128eIdSbXL2yyOR8kvmvvUh0PsL006fY3mswd+oRRCNgd71EZDxGt1JE1nRGkwNYsSh9IymO/uom8plj/PDtK8ycnKVyf5vh5yYIH5+j28rTuVVk4MIwidEMDRl2P1pn9NgkG2urxDIhQhMJWq0ah1eOuPj3HmekP0p3uURHldAbDv/Fz1As/czftz/6eiYIJWMcbeQ4zDl80g2T7jYYjylEohBLauhqgCzL6KZIOKkj4+GJPqqgINkevmPzYM3DwoNAxXIcnG6P3/knv4Uv1NF7HZRwHE3T6DgiRSfBO6/dpLy6T/aZWY4+2qBudzH8Lm8e1AmkgGOuQt8ZjeBBDy8kc9vxGegKdByHpqwgaRb9DY9Lrz7K+9+/QS2soUoiiugw3vC5/NWzOAkfIdZH9bN1ZNemsFri+Fee5yd//h57qkMgyWQyOuUth37NQfQFDiUNgx5zU2F6osSt5Tqq7TM8neKLL53FHBpgc+OA/qSB3esQzo7SqVZ5/S/fw/UFOtUamqbxq796DMEOUamU+Pj2EbOdHrHJfvSIB45Ao2Oxmysz1BclMZrl/lqVmbEQiDKLS0ecP5sknEqgRGap/OgaC5t5MnMKEcPnw1WH41GDcNjm+JOPgdWiXXIJzQ5SXjqgcnWd9NlJ7l9ZoP/xEVJTp+jsLyEEEpFIhK39Age5OudOppD6R6js7rLw+TptIYxr+3i+yGMzGQi6XN/NkfQ1DK/Lb38xQXQsRmG3iCaJ3F21WCiJTKZ0EmKTZNrEDHkPSaSyjx5WiURDBIGDKIqInoXVstF1g9v3W/i+SMcNUBSFF35lnsGzT+EdfkIg6yiOizl4jvbBff7wp3mkBZ/BF2eovLdJTQo4uLLI+pCG1dQwPZgN+SRPxuncqVMdNqlslFFdh6geZcG1ODeRRG04RCWRW4U2Jd1G9XTGh2KM7rUZ+uYEWv8EbivP0r+7TfTcKEpHw6sXubpXoSvKIAqMDakcHnRI9QKqYkBHVFFkiS89n+aj6xWaVhsfmemxNI88exo3kNlc3GP2WJKwmcSVDLbXN/nhm5+i2xItzeG5031cfOYsrZ7PxrUNZC8gJvV449Y+T59LYJgR7i1WiJsapZ5Ktely4rhO7ajH5GSaSFLEd2Bochb7yCF/Y42t/AF9syFCgcan2yXGw2GOApfnHp9BCCURUmGceoPF/+cTDBNC2RG26gc88c3nuP7jz5g8lSaQdbaXCjxYLfD4xSlkUyIU1ynvV/ng2j4iEqLkYboOF+emCIZifPbuXezA4djUBB/cWvn5VgL+5Vf7gnr5oYxyOR9wrWRzPi3TF7UYDGuENZtM2sBxHKIZHSOqoUgCsijgSw9t8Xajy71li0CScXwZL/DxpYBv/PZXULOjpIIH1Es9tJBCtxsg+gZ/vexQXN5n/817rOoiti7x4qlJ/u76DuHAYrAjEvgOwxdTBNsttISMOGiy+1mVciZMy+7htSEtuRx6Ihoqgi7h2z2GJjLoOy2e/AfP4FY3CA1OUVlZZ+f2DumuwI96NrJk4HS7/PKrp/GDBm/8dBmxJ3Hm9ACi67OwUsa1XH7rD75OMj7A1mdvY/bFiA8O4woy+2//iFCmn/D5R4mGFTY/vknfyWkcz+L6B4usPsjjSzYpU6DYFjmr2hx6GiNJleuHFo/YLolzEfRIiB99tM/lJyfY3K5z6uwwktEhO3qCxvV9JEK4XoHEEydp3dll85M1hl6ZYef1W8x/ZZ69XEDryj5T3ziObYg0ii2EUIzczRXu7lU5d3yIxcVDRFHkqUcn+Lv3ltBjOp2281DjIzhcPD3K5m6RvXKHuZk47RrsFWqomsjJSBbfq/H/tXdnP3rd933H3+d3tmdf55l93zjD4XC4SdwpURIlS7LjJXFqpK6bpDEK10mAtkCQXrgRkKAtkiJI0DqNBSRNahS2a1u2a7WWtdqURIkUSZEiOeTs2zPLM88zz76c/fSCN71pL3TR6mJe/8A5V5+Dzxe/7/l5ts+nH9fY22thWgrb+SYvLxl0RwMcTEskAi6d8QCRqEsoIFCDglgqgqwJZN9DUQWW6aA6Pqrs8cEtA08FDwXDtLHqDr/9jS+iUibkSngBF9mLYPuwVQ7x47fXWHnzDmauxrYCSctlpy2K1TDpdGyStkr7mU7UukF+rkJ8JIIiu7x3r4oeD5MzbCKWR1QKU9ZbOMj4ToOQHqSnN82oGmbkuWkcw6Rpt2jeuUc8GOWld3IYromr2Tx1foRQ9xBXfvER1Y0yoYyCrCp4sk9jx2R4KsZTX/g8wpNY+umPCHX00Dk9QW1ugepuDj8VIzlxBMeoE8DDD4Zptep8+69exVdVLN8m4rlkmgrxRAjRZuH5QZazRZ6e6saMKkRUi3fvuzx5YYSFrU3icZeerjHspkPx6hy76yV6Lk2h6galN8q0XRyl7rmEdxsEJlXu//0sceEQ/NwoRUujM9VPQodv/e1r9ARd0r0jLGWXSEWDDHbEeON6Hlc2URQFyfV4+mwPq2WDlVmLnkGN0l4Ls9mkZEMHgjZkBh6fYuHVG3zQMj5eKKWDul8yHHRNxbcdPNnlSEJmLK2TjErEgh7RZICAClpQIhwL4domuiahBwSYPsLzuTdnPawqElgeOJKDmfP4zT94mmi4itOUEAGN5naVVq3MspRhcd0iMXKA+W+9wv2dEluBEJNOk8GpDiwZKg2fYG+Ajds53AJsxQQNTyaJx+TBCPduGnRZZXaFREBXKTouX/+jX6O++BHf/s48ESnMZ08dRh5ysZUwVn2b8ivbZDMyy9tVOtqCnLvQgxtuw9aiSKUN3nl1Bb3oMpKSqW1UmPids3QNTzL/6k9obtUQbpj6Vgs/1KJ9tJ3amoE+HSGuKjiuROLcef72T79DPNlGuVzF9h+uYRyfGObqbA7DqRFSJFxfwkamzTVQbZN2KYSnOizKAsmWONQVZcM1iStROlOC7qPtlK/WiR0eItKf4M5fvMXY7zxO1K1hNkuowmZpdoc7a3WW8g2SQsWQPIQQuJ6M77tIssnFAx2UKi3m1ywuPDZCeW+HzKFD3L+5jsYOA9OjzF/f5sFKkRgSMzMZsis1VqomSDalpokv6aiaC4ZLOCh4pF2jM+aTCevooYdnbwIRiYCmEoyqqJKLYVgEgzpBZDzT4KNZG8sXmJKDJFR8T+VX/8kx2qa+SPnGD9E1B9uzwXFpbMN3rm0RTCv0jp/i+r9/iTnLR1UE6Q6P3o4wFdOhvSPJK+/vcsRscVto1HwJDY8TQwlKWOwulBkkDJbFXEbjwlMztHeHyN5b5O2rFT7dGaX3S8cor6wgp5JULq/T6kjx2tUHBGyLX3l2gljKpH38OQqrt6mXCvzwpXkGTMHpLxzFyeaJPHGEvaVV7I0NWrMVXEVg7NXR0kHszgCZHhnfDxPXFSJT47z91jb35u8SthPUrCouNidPDnD1o00k5+EivOVJhB2fgK8gWnUmAmHKYUGhbFKVXUwFBuI66bYQma4MTrNMuqMTu66QOJKm+uYihdldjv/h59ibvYmLielpGLFufva9y9RbHkHFo2l6KJqO4ks0DYdU1GGsUyMRjfGg6DIeM7BbEB8e56U3b/GZp7qxbZnLb69jOjJjEnQfGuLG8gbtXQGu3t75eOeUvvGNP3lBVl1cyUXzQOCTCGgkwjqa5KAqMgHVRygQ1BVkyUMIHz34cL4kyT5ey8WywXZ8HA9kCVQUHF0mrMqMPvPPqSy8SW57m1pR8Gcv3uWbP80xkBaYd3JkH+QxVY88LgFfI5etIfaa9AwqSGUbr2mQHo7TnjcouDbjhkq3LjN8xOfop44z/ewZBiIR7i8XKGxUKddMCsUKLUliIVdgdHwQLRph5focQnj0nu5mYXmPYtHh7lyF9YUSwzGVK7+4z7nHhwjO7TD6Dy8SHQuRHB1h7e9eZW+uyol/9VuUfrnM5D99lLazj5AeOkz85AAxuU5xq4GKR31zk6nzI5x56nE21rMcPxSjvV3nys09bGHT4Uh01zxUXZBwbKa0AGOPDzDxzAx1PcRIymYy3EMmkGby6WMkkjFWfn4XJROn57FHcd097PVZTBKMnJpi8+5N/vsbG1y5mSObb1EutVBQUGSXTCbN0cNJggGPA706B8ZjWMUm/Rce5dQTh5HCcWIZnXK5RqKyx25VEAnK3LibxfclXOFz/PAEobDFifNTvHNtHUWV8SWLgODh1dZCkA76RPSHz9RDGiFdRpEcVE0G2QdfoKkC+eEN5ghfsF108IQPqoZrOQjZ44PXFzg42YZortOo1ZCMBv/lb2bZqVh8WNHIpDVWf/iAVt1g2TdpqTpmyaIzC2a+hhUL0O/5xCcTdFsSrbrHIeFjbZuMzKSYPtvF4HMH6Ln4CMqtDa58tEZ3bzdhpc7CepmlqkFi2yQ5NYbVqlAt2QxMxbj34Sq+5DO71OCjO3tk792iuN3E8mBsMMFYewYxFsGNquy9M4uzk0fabDFw8RjF21sc/qNfp+vMDOqNHfo+8xi6YlHeKlLZ3KRT83j0V88zOT3G2sp9zp8fw64LqoU6tuPT0/RJV+tUYhEerTuc+r1LSGGXU79xCbG3wfO/+Su07q0z0R1j+NPPk/3+Fe5s1pg4GiU9NEpucZnIZAI/qNKULG5c2+L1a2sszO8x+94qj4x1sZavENAULj42RH6rxPlHuwjHbErFJqd+/VNEIwrdI4NUb20w8dxZNhdnOXbuCIWtXa6/V6Ah+eiShCEkxqa7yLTpdGVifP5LX/t4g+5/+yd//AJCQjgePakQ7arADWjEPAdNdQkHVTTZR9UFqiJwfQ89rCI8Gccz0BUFo2URjkvsFQSuL3CEg+ereJLD2tIek5OTBBSLv/x3l6k6DuOTcRZyHhMjSay1PEsxjZwjo+JRcG0ausykq2KndK7f3WUVBX+rSdWRaAjBltagfbNB44GJpyo0Pprn1uVtClKVydEou7sl+vpTDPV1srSTQ76zhpbSGJ4+hB5r0toVHD87zOFHu7j64S6W32JzvUHThmEpjlks0ffsGWq5VRofzmNncxz43d+gurfL5tsfYYod6usqC6+8xc9v3UWyLTqH40hagIZToaO9E9lrceCxY+TLFpu1Bo2dGmfa00yOJkleOMjQ8TGk2R36J7oIDKV5//ISszfzTIx3Ee9KsNVaIxSNoWkhKsYOnV3t5BYXMEoWP3k1y5GzKaRQmnq1wb0HGxw63c3GWp7jmQBdk1GODPrs5Bt09sW5fqNEbsenvrRHRzLG0psL7MlFrlz5kK2dCsEHBTqfOsrG9i5z63kaVRtPcultD3P5TpZmSaCWPNaKeyD5eJKMK0s80hEjKil4nkMsIBEJPJwxqLKPqimA/7De2DZC8pBlgZB87HqTXMHHljw8T0LIPp4ioQc0pk5kSI2d4QcvXuHmbB49IeHbPqawaVypE52K8X6hjIGP49tYKnR7HtJEhOauw7W9Glbewd9rshAwqboWu4pGKmegNjS8koxvb/HqtTypLo2g6nD9WoEzj6ZYX65SqhqYV9bonpgk0WZCzubguWNMj4dZfVBC0jWaLY983mBzc5dDfe3IDfCTKv7aKrX7VQ585RkinQlyH2xQ9Zskxjq48cJ/ZfgrU3z7P11jaCCMF9PRhIQrTCJ9h6Da4PSnTlBtCV7/2YcMtzye+ewpWrbPud9/npPnjmA9uEO8p4uCU+LKyzcYeWyceCxOVAj8Ph9ndZHd2RpPPD+Mm+5CEy4/+ekNmrkaQ8cyuKbH21eWkYVKoi9OodTk4qUD4LZIeS6RdICYq/PeRxuUqoLplsPijXt4uyW21mDDsVD1OH52h3vv7cBQlO3dGkGhI1QTodmsL9aYn9umslTm69/4Py/k/l/r2+On+3xzq0JmoIvN69t0HgqzlS/R5UAq5JNKysSCPvFEkKDuIVSIRTVkWUbIIBk2vu/jSoJ7sw6OLCEUBV/yaTZsJA36IiFydoNkMEIkESYcha9+c520ptIW1mg4DqoCsiowTRur5ZJSFCxNwfM8TBxiQlAHVN/miTM9XHmvjOu0cCQdSXHQQkHqTYfhVJjNWoFTj4zQnpL4nz/LYrR8FNWlw9V49nAfXDqIXF7CqMG22kHazGLGJaz3Wxz4ynM0Zq+T22yRfW0B2WpiHO/lyNlxEulePnjhB4z93jTf/fPLNJNtXDiYRMTD9HYo1Gou7aEwi5eXGfnyaUTLhViSzXKBxrfe4Ngff5X6Vp4b/+3n6Hvgh2RErYKfjpOcbicasqlsG9y8k8NSg/zav3wGs+kjWTJGrsSPfnwZNRLANAXhiENhu0Y0FWNgrJteamjpEMV3Nzn51c9hhl2u/dnrRPokAj1JLn+QRZMF8bjPmScfRygS61evM3DpHPdefAU1FcSTmxhumOVdm+npBPcW8hSqNt2yTKJNpShbJB2ZWDKCoulkb+2QGdIoF00GEjKZuIQmfNpSMqoiEY0rSDgEgjKaLgioAdyWiWdY3F2SsXFA07AsG0kJ4LouUd0BwySaSSFpMoGwoLDR5C/fKhEPh5ElQctsEtB1PGwsRyDrEpOmwoZSp+WHkYWHUCSEKxEN+mQyYW5vWAQ8HxubiKRjSy2O9yXo7EhQllQSSZ9itcnbl3PIskZbq0VHQnD2tz5PI7tOMG2TfX8FZWKcSrGAv1Xn9Bc+TbG2SkCPYGwVmPvxLQ6fPMRmc46upy7h3tvhwStvc+oP/zH/4V9/l1NPdrO4WiAZ8jk004/kSyy8uUj/oTSisw+1f5AgKt/56+/z5a/9A+ZefpOo1k4+exdVVSnNN4mmFSrdSQ4ORdCFwq0fr1MNekSOxTk+c4Rgez/5pXmWvvsuCxFBy5WwqZNJdlAtVLAs+PLXLrFz7TqBTAavWCJz8ST+8irz7y4RP9rF1tIud+frTIxGGH/8JLqv09rapVqbJTV+grkfvk9qKIJZb2LKLtfm6lw41s8Hd0soTZO2Tlgt+ORqjY9X3773n//NC3o6TCytIjSJet4A20QgiIUEuqIQ0mUkCXzfQxEPv2xCVfF9CUWW0bUQKj7ZnI/Nw9+XmrZAkn2EBJm2IGEh0daVANFAqdV4+XaNgc4ofWWXhi6jKCq4cPZsHxefHGFmsov5lSqoEkKW6ZRUgpZPOimzsQeO5+Kg2b4aDQAABjJJREFUoCg+piEjOTKu63HykQjnLgwyeuwopVyJkarN5DOHiHgG4d0m16tVCq/PsrphIW/6zL1zF1UP8rNXV+ndMei4MEN0dBBNkmi+dpfo9CB9ySb9Z06y84t5lLhB9eVNMseT6CLAwlyZcrPJ1btFBrtCOCs52vrDZGZO4giJN9+4wvwbK2hVm4XrK7z+y4+Y7I6xm6ux2W4zlUgx/c/Osn4jz3bJoHCnTndniO6+MHM/uEsxu8LgYBpnz2N2+WF3P/1kH5cunWawXcZVggwOthOolJFjGj2fOU+tIPjJ9y9z9PwJIv2C1IHDhCp5Vnfr5Co+e+tbWHtZKkaLaFsHxZUcNws1Oo9Ms7xVoVxvslf1OD2TpD2dRrctQmFID3ajhz36xyYwigVEwCXXsnGagmTARxIOejCAUEEGVNlFFiqBSADfF8gauIaD6wkKJQffkbB9Cc8HSZbwfJd4QBDtiKIEIBIJorigIPHBWpGkrZICbA881aOzS+ez53o48dgjJPpk7tzfQhESddOlzbCpOirPf+YAm9k65d0KsgSyrOLJNXB85LjK1PFROro6UMMgNho8/duPMz4YZedOnoqkULw+y/IHC8zPN/FWywSWa1xer5Iq2kQyQRo3l4kfP8ziW+8w2DXIgw9vEhpN0X/oKFf/9Hv0PHeI9//851jjUYxrOVQtgmHrmNUW7RoYcy36PnuSdM8gXsVl9s4iW3eXSUWCbNwusmKu0lcWbCgKgwfCdI63oyf6uH95hd5DndxczRIMCeJZi51mHXNziXhGo3I7S1YJkE7CF794mPGhNNPTMcotDc1uITk2oYOj9E4d5v2/eZemkSfzaD+p/j6GD5zAKaxwc6nJratrDHXC3twSqfEh/Iogey9LwdMpVD1qdhzXarCcNzl9oh0j1yLmyvR0x/ny1//Fx6tvf/1XL76QSUSIuQ5760XqLRNFCeE6PmHZI6z5KOLhBrmmyijKw8VLPSCB7eG6Pq7voEg+hZKL54Ljg+faIGmEVUH3UBRF8ommw0i+Qiit89I7FTK6wvhEho5tn+H+EPV8ncV8nbs3c5w9NUD/gTCxUJDtbIGpsMbMuTbGjx9HS8YxdInCbp2O8U7KxQozB0P0xoO0LIWgrWFtFdCjAjMcJRbLYK3uYBVMDh/rJnOwh05TYk602K55rFV2SaKiez6Z4RTF7A4776+wur6LFldpOzzML3+5SGB9k9T0KN2fvcjosSkGzo5x9PxRjh/u5pELM7RLGvM/e0D4gEJlaYOIW2NobJLobJHExXYkv8i4JLMcD3HiRJrDx0bwN4v4k5OUfnyVkSeO0DfcQ/9jY+y+sYQ0GKWxVodGjexby6iuT12B5dVddhd3CWge3R0BAgH40ZvLDCXbkZoSrdXbKJKgoz+CHsyweO0q2VyZatPCk1QsFFrNBut7KtpekbVcDVNRGGkTbG+W0XzBSMqlUrCRJJ98sUXIU0mNdBKLtWHkl6k0WxglG4Gg2KoTVWUSqoRiuWiKhxrU0NSHi7GK7iNsGdt0kFWFQFwnv2FjKw8DCVnFsV1k36c7pROMqEiWS0C1iIR0FDxeveVw+tIAge08wYbDpBZivmpwc65GR8Cis7eDg+MdbKzXmWi5nLjQRc/xARZX6xw+N0PvQIzN3QJCDaA4Lk8/MYRvSjT2DLrSQeyWgR5JIsoGhqEwdWqQEVw6z40Q2i4jOy3Wo4J5V8cQDgdaHi3ToOt4H6asEG25XH/nNl0n06RnJvjef3yZ8YkUI09coP9TR5k5NcnMs2dJGQ2OfOlRMgN9PHjxXaLdCWoxm9r1D0hNzNDT2caYCCN1KATrRcqbBod/92kGuiN4y2ViR/pom5ki8MpNTFtl5HwnoycPcH8+T19M4b0HRcbjCV7P5tF8n6YhuHUry956kQPTE6TbNZyqRqi9g7aQzTvf/CkjR1Jkqy0mjpxFxuX6az+nZYSoNOoowuRAb5j/8eEee/ObbG7XMGybzZpBog3WNmv0IzHeH6FZaVLCQi81kcMy/+j3/+Dj1bd9+/bt+39N/P9+gX379u373+2H0r59+z5R9kNp3759nyj7obRv375PlP1Q2rdv3yfKfijt27fvE+V/AULlS7eHg/J9AAAAAElFTkSuQmCC\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "img = PILImage.create(files[0])\n", - "s = SiameseImage(img, img, True)\n", - "s.show();" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "tst = ToTensor()(s)\n", - "tst.show();" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "All in one transform" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class SiamesePair(Transform):\n", - " def __init__(self,items,labels):\n", - " self.items,self.labels,self.assoc = items,labels,self\n", - " sortlbl = sorted(enumerate(labels), key=itemgetter(1))\n", - " # dict of (each unique label) -- (list of indices with that label)\n", - " self.clsmap = {k:L(v).itemgot(0) for k,v in itertools.groupby(sortlbl, key=itemgetter(1))}\n", - " self.idxs = range_of(self.items)\n", - " \n", - " def encodes(self,i):\n", - " \"x: tuple of `i`th image and a random image from same or different class; y: True if same class\"\n", - " othercls = self.clsmap[self.labels[i]] if random.random()>0.5 else self.idxs\n", - " otherit = random.choice(othercls)\n", - " return SiameseImage(self.items[i], self.items[otherit], self.labels[otherit]==self.labels[i])" - ] - }, { "cell_type": "markdown", "metadata": {}, @@ -3240,31 +2239,6 @@ "1. See what you can learn about language models and disinformation. What are the best language models today? Have a look at some of their outputs. Do you find them convincing? How could a bad actor best use this to create conflict and uncertainty?\n", "1. Given the limitation that models are unlikely to be able to consistently recognise machine generated texts, what other approaches may be needed to handle large-scale disinformation campaigns that leveraged deep learning?" ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Becoming a deep learning practitioner" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Congratulations — you've completed all of the chapters in this book which cover the key practical parts of training and using deep learning! You know how to use all of fastai's built in applications, and how to customise them using the data blocks API and loss functions. You even know how to create a neural network from scratch, and train it! (And hopefully you now know some of the questions to ask to help make sure your creations help improve society too.)\n", - "\n", - "The knowledge you already have is enough to create full working prototypes of many types of neural network application. More importantly, it will help you understand the capabilities and limitations of deep learning models, and how to design a system which best handles these capabilities and limitations.\n", - "\n", - "In the rest of this book we will be pulling apart these applications, piece by piece, to understand all of the foundations they are built on. This is important knowledge for a deep learning practitioner, because it is the knowledge which allows you to inspect and debug models that you build, and to create new applications which are customised for your particular projects." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { diff --git a/13_convolutions.ipynb b/13_convolutions.ipynb index 72ee2e8..c3c1331 100644 --- a/13_convolutions.ipynb +++ b/13_convolutions.ipynb @@ -2637,6 +2637,962 @@ "This is taking the 3 slices of the convolutional kernel, for each output feature, and displaying them as images. We can see that even although the creators of the neural net never explicitly created kernels to find edges, for instance, the neural net automatically discovered these features using SGD." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Improving training stability" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Since we are so good at recognizing threes from sevens, let's move onto something harder—recognizing all 10 digits. That means we'll need to use `MNIST` instead of `MNIST_SAMPLE`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "path = untar_data(URLs.MNIST)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "#hide\n", + "Path.BASE_PATH = path" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(#2) [Path('testing'),Path('training')]" + ] + }, + "execution_count": null, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "path.ls()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The data is in two folders named `training` and `testing`, so we have to tell `GrandparentSplitter` about that (it defaults to `train` and `valid`). We define a function `get_dls` to make it easy to change our batch size later:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def get_dls(bs=64):\n", + " return DataBlock(\n", + " blocks=(ImageBlock(cls=PILImageBW), CategoryBlock), \n", + " get_items=get_image_files, \n", + " splitter=GrandparentSplitter('training','testing'),\n", + " get_y=parent_label,\n", + " batch_tfms=Normalize()\n", + " ).dataloaders(path, bs=bs)\n", + "\n", + "dls = get_dls()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Always a good idea to look at your data before you use it:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOsAAAD4CAYAAAANSBHgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAdtUlEQVR4nO2deZQV1dHAf5d9QFYJm2ELIhHBJSibfEBYAi5gCKCicFDCEsBEEQ2iAoIQDIugAkERzCdGXEI0iDlGAcNOHAwIKIuon0GBGFQQEFCgvz+G6u7HvHnvzdu671C/czjT9O3XXXNf11TdunXrGsdxUBQl/BQLWgBFURJDlVVRLEGVVVEsQZVVUSxBlVVRLEGVVVEsQZVVUSwhdMpqjOlnjHnXGPO1MeaYMWa7MWakMcYELVuYMcbca4xZf6bfDhpj1hhjugYtlw3Y8s6VCFqAKHwBPAzsBE4A/wPMAU4CjwUoV9jpACwAcoFjwCBgqTGmneM4awOVLPxY8c4ZGzKYjDGvADiO0yNoWWzCGLMVeNNxnJFBy2IbYXznQucG+zF5NAeuBt4OWh6bMMYUA8oDB4KWxSbC/M6F0Q3GGFMR+BwoBRQHxjuO83iwUlnH/UAlYGHQgtiADe9cKJUVOAxcDpQFWgOTjTF7Hcd5Olix7MAYM4w8Ze3uOM5nQctjCaF/52wZs44G7nQcp0bQsoQdY8w9wHjgBsdxlgUtj62E8Z0Lq2U9m2JA6aCFCDvGmAnACOBax3FWBi2P5YTunQudshpjxgOrgY+BkkBbYBTwTJByhR1jzExgCNAH2GmMEYtwzHGcQ8FJFn5seedC5wYbY2YA3YALgOPkdeACYK7jOKeClC3MGGMK+iL/13Gc27Ipi23Y8s6FTlkVRYlOqOdZFUXxUGVVFEtQZVUUS1BlVRRLiDd1c65Hn1JZIqV9lzzad1FQy6oolqDKqiiWoMqqKJagyqoolqDKqiiWoMqqKJagyqoolqDKqiiWELr1rEp6OHUqb2XX9u3b3XOrVq0CYPfu3QBs3rwZgP3797vXLFiwAICWLVtmRU4lcdSyKoolxFvPGkja1+nTpwH4+9//DsDSpUvdtjlz5hT6frm5ue7xT37yEwCKFUvo71SoU+aOHz8OwK5du9xzDz/8MADr1q0DIq2mfNexCs1feOGFAGzZsgWA0qWTrmwSqr6T333u3LkADB8+vMBrhg4dCkCNGoUrvyTvVqdOnQAoU6ZMcsJquqGi2I0qq6JYQuBu8MGDBwGYP3++e+7JJ58EvEBIOnn88by6zXfccUcilwfmyp08eRKAd999N1+buKi///3vAfj444/dNnFxq1WrBsDNN9/sttWqVQuA888/H4DatWsD0LWrt3+VvA9ff/01ABUrVkz2VwiVG/z9998DKbmmCfPQQw8BMGbMmGRvoW6wothMYFM3R44cAaB169YA7NixIyhRQsmaNWsA6NChg3uuoMDQdddd5x6PHz8egEsuuQSIHSCS7yAar7/+OgC33HJLghKHmxIl8l51CSzNmzevwGvFCvv7WzyORAoMPv/880BKljUqalkVxRICs6zFixcHoEqVKkl9PicnB4DvvvsO8JIA4lGhQoWknpctvvjiCwD69++fr03+qv/2t78FvLFm+/btk3rWX//614j7+rnhhhuSumdYESspMQv5GY133nkHgJo1a7rnDh8+DMCUKVMAWLiw4P2+JDaQbtSyKoolqLIqiiUE5gaLGys5qJJxE+2aCy64wD0nLqC4JZKxI1NA0ahUqZJ73Ldv31TEzjhvvfUWAJ99lrdTo3/qZOrUqYAX9ClbtmxanukPpPTo0SOt97aR5s2b5zsnWXAydIhGo0aNAO97SjdqWRXFEgJfdSOWsUmTJu65Rx99FIDp06cD0LlzZ7ft0KG8DdHatGkDxLaowsaNG93jBHOCA6N8+fKA501IfjTAxRdfnNZnffvtt/nOSQAuVv5wUUUCbcuXLwfg17/+tdu2Z88eAI4dOwZ4/TNkyBD3mokTJwJQuXLljMgX7jdXURSXwNMNE+HEiRPucatWrQBvLWY0fvCDHwCwaNEiIHJqo5CWNVQpc+lCUgkbNGgAeN4KeFYl2ekgH6HuuwMHDgCwYcMG99zixYsBePbZZ+N+Po39FA1NN1QUmwl8zCqpXf51l8Lnn38OeCl0ULBFlcR1gEceeQSITNVTPKRihIz369Sp47ZlyFIEwjfffOMeb9q0CYBp06YBsHbtWiDSq4jFNddcA3hJ+s2aNUuXmAmjllVRLEGVVVEsIXA3WNzfunXrJvX56tWrA7B69Wr3nJQmUaKzfv16wJt+6NmzZ5DipJ033ngDgFtvvdU9l8gU39lIggpAu3btAC+nPQjUsiqKJQQ+dSOTzcla1q1btwLe+s00E+rph8IiK3okLU4CMBJsgbSWIM1638nKq1KlSqXwaA9/VQkJcvbq1Qvw3tcMJY/o1I2i2EzgY9Zkkb9wYiWU+MiUllhUmaa57LLLghIprUjCy4ABAwCvYHmySKlXgFGjRkX8lCmgESNGpPSMwqCWVVEsQZVVUSzBugCTuCH33nsvkHxZmASxPsDkz32V4nSCFPbylytNI4H1nbj5L774YoHXSNmacuXKueekPwQpWgfwt7/9DYCvvvoK8AJL/t0errjiilTE9qMBJkWxGessq+S1ynrWDGOdZZUkEwmOdOnSxW378MMPAa+v33//fSBjVSGs67tY7N27F4DGjRsDXqUSSTCB6BUmkkQtq6LYTMqWVdYA+re6kHFlIkgamH/6QKxtNKQm0UcffQScW2NW2V1Pdo2T9brLli1zr5GqGLL9RoRAZ75rGafdfvvtgFfXCuCHP/xhusQNVd+li7Zt2wJeIsmvfvUrt2327NnpeoxaVkWxmZSTIsSXf+CBB/K1JWJht23bBnhRtniIJS1ZsmSiIlqJVHNYsmSJe06im/66TBBZpDuR9DepvSSWwG8R7rrrLsCztrJQwkb83oW8Z5KWWtj3R1IZ/VVLIHodq0yhllVRLEGVVVEsIeUA0xNPPAHAnXfe6Z4777zzAC/o5C+5IowbNw7wyo0m6k7INvODBw9O6PoUyXqQRNZiSonLWMG2WEjutD8/Vr7rWbNmAZ5bLfu9+q+pWrUqAP369XPbxDWuUaNGIiIEFmCS/WqHDRvmnpMCZ7Lf7aWXXlqoe0qChKxrFXTqRlGUfKRsWWWqINpfFdm2ItqkuwRQpGhyNOrVqwdE/nUfOXIkkLXd4LJiHSZNmuQeT5gwAfCCI4kEj+rXr+8ei6fSrVs3IHbpVSlW57cOMhWxc+fOfNfLthB33313gff0kXXLumLFCgD69OkDeOVGAbZv3w7ARRddVODnjx49GnEfWVkDXoDq7IoTalkVRclH2tINZesAgLFjx6YgkpcOJ3toipUIgIxaB0md9G8PcnYyQzTLKltsSFLD5MmT3Wv81Q2SQayRWGiZmgN46qmngNi7qfvIumWVVMBoXsHo0aMBL56yY8cOwEssAa+vY+31Kx7dH//4RwC6d+/utqWxaoRaVkWxGVVWRbGEtLnB/kCRf41govh345Kq5yHInsmoK7dv3z4AGjZs6J6TfrzxxhsBqFmzptsme8tKBX2ZXgkpWXeDpcSPP089XcgeS/PmzQPSv6PfWagbrCg2k7aCabJLOXgJDjNmzACi5w0PHDgQ8Eo8+ifaz5W9QcVqHjlyJGBJigZXX301ENuy9u7dG0isdK0/AaJFixZAwsG1jKCWVVEsIfBKESGnSK7JzBLad8mjY1ZFsRlVVkWxBFVWRbEEVVZFsYTQKWu9evUwxuT7l6Fd4ooUBw4cYOjQodSqVYvSpUtTv359d/2vUjBTp06lVatWVK5cmUqVKtGmTRt3XXGYCN3GVLm5uRGJ1EePHqVp06aZqhpfZDhy5Aht27blggsuYNGiRdStW5d9+/a5y+CUglmxYgUDBgzgqquuIicnh3nz5nH99dezcuVKd+42DMSbugkcY8wgYA5Q13GcvfGuP1cxxowH+gONHMc5Ee96JTbGmK3Am47jjAxaFiF0bnAUhgCvqaLGpSewBphhjNlnjNlhjJlqjMlIuf2ijDGmGFAeOBDv2mwSOjfYjzHmSqAZkD9fUTmbBsCFwItAN6AWMOvMz1sDlMtG7gcqAQuDFsRPqN1gY8w8oCPQwAmzoCHAGHOCPEtQ13Gck2fO9QJeBs53HCexwsznOMaYYcA0oLvjOMviXZ9NQusGG2MqAH2Ap1RRE2If8KEo6hneP/MzsV2/znGMMfcAUwmhokKIlRXoC5QCnglaEEtYDTQwxhT3nWt05uf/ZV8cuzDGTADGAdeGUVEhxG6wMeY9YKfjODcGLYsNGGMuA94BFgAzyRurzgPWOo7TP0jZwo4xZiZ5gcw+wAZf0zHHcQ4FI1V+QqmsxpiWwHqgk+M4y4OWxxaMMR2BR4CmwH7yxqvjHMfJ3oYsFmKMKUgJ/tdxnNuyKUssQqmsiqLkJ8xjVkVRfKiyKoolqLIqiiWosiqKJcRLNzzXo09aRyh5tO+SR2swKYrNqLIqiiWosiqKJaiyKoolhHo9qxKdt99+G4D77rsPgHfeecdtO378OBDsNg9KZlDLqiiWoJbVIr755hsA+vXrB8DXX38NwA033OBeU6yY/v0tqug3qyiWoJY15Pg3qb7++usBbxPmyy67DIBXX301+4IpWUctq6JYglrWkCMRX4C1a9dGtC1YsCDb4ihnkPjBtm3b3HOtW7fO6DPVsiqKJaiyKoolZN0N/vLLLwHo1asXACtXrgTAX16mefPmALRt27bA+9SrVw+Abt26AVCnTp20yxokb731FgDPPJO/uGPNmjUBqFKlSlZlUmDz5s2AN332wQcfuG1ly0ZuftCzZ0/3eMqUKQBUq1Yt6WerZVUUS4hXMC3t6wr//e9/A9CwYUOAlHc5k7S6l156yT0n1jYNZH1N5okTeXtKtWzZEoAtW7bku2bTpk0AXHrppcnKlg2K5HrWq666CoB//etfhfrcgw8+CMD48eMTuVzXsyqKzWR9zCpjy+eeew6Am266KaX7iSW65ZZb3HOHDx9O6Z5BIpv4RrOogozXleyxd2/eJob+MWphiPV9JopaVkWxBFVWRbGEwDKYevfuDXjh7W+/9XZ4WL16NQC1a9cGIsPdO3bsAKB9+/bZEFNRAChfvjzgrXB68cUXC/V5mepJBbWsimIJgecGy/rL8847zz13zTXXFHi95GSeTbly5dIrWAi58847gXPjdw0bYlknTJgAeFM4/sCmfD8vv/wyAI0aNXLb2rVrl7IMalkVxRICt6yJIHWFAEaMGBH1munTp2dLnMCQaa/ixYvHuVLJFBdeeCHgvYfPP/+82yYWVbxFf6LO+eefn/Kz1bIqiiVYYVlnzpzpHr/++usRbfIXa/fu3e65Tz/9NOIaSXwHKFWqVCZETBuSwC9poJUrV3bb7rrrrrQ+69SpUwAsXbrUPWeMiXh+NDp27AhExhnCiP93kHpVkydPBuDIkSNAZM2qPn36AHD11VcDXl9EQ5Ijbr/99nxtkvDTpEmTpGWPhlpWRbEEVVZFsYSsr7oRZLVNtFU3q1atAjx3YtmyZW7bF198Uehn+V3nWNNCUcj6yhFxy8QF87vBBw4cSEEcz/X75z//CXglYwq7gkRWNcn3U4A7nPW+O3nyJOAVPY9VEicWt912GwCPPfaYe05+xw0bNgDelI1/yPXzn/8c8IJOKRRa11U3imIzWbesUlqzR48eALz55pvpfkQ+7r//fvd44sSJhflo4Ja1UqVKbtt//vMfAEqUiB8XlO/VHzySQN0//vGPAj8n95YkAH8pVP8UGkCHDh0ALyh2Flnvu5EjRwLe7+lPHpFKDdddd13EZxYtWuQejx49OqKtVatW7vGYMWMAGD58OACffPIJEFmtY8+ePQCUKVMmGfH9qGVVFJvJumX9+OOPAW9yubBIQoCUfRw4cCAAOTk57jVy7yVLlgBeGhhAxYoVC/O4rFsHsajRtsGQxI9YUzj79+8HYP78+QCMHTvWbTt9+nTEvcUCTJo0yb1Gqk+I1ZwxY4bbds8990R9pkwBnf2rFChkfOL2nfye8+bNc8+J19S9e3cABg8e7LZ17tw57kPFWibybsozhgwZ4p7r2rVr3M8liFpWRbEZVVZFsYSsZzDJGlVZtSCZRzVq1HCvkXKl0aZpqlatCnglTGNx+eWXpyZsAMTKmkkECZj43V9BitRJUE+yvySY5EfKkDz88MMFPktKxgaBDG3+/Oc/u+ckaLlw4UIgsUDPf//7X/d4+fLlca+X7KbZs2cDUKtWrQQlTh21rIpiCYElRZyNVIAAL3h08OBBIHJdoPz1y9JftKwHmB566CEgukVr3Lgx4CU1nF1UGry+Eo9l3LhxbpsEXMSLkeoc/sQUCbLIFIcEcvw8++yzgLerXQFBu4z2nQTJ/B7EAw88AEDJkiUL/Jysh163bh0QGSD67LPP4gomHl2bNm3iXpsCGmBSFJsJzaqbOXPmuMdiUYVBgwa5x9kcIwTBFVdcUWCbrPSQCX4pSSrpceBNZfz0pz/Ndz9JcJDpGEmLSzTdUMZpkm5YoUKFhD6XSTp16uQei5e4ceNGANavX++2iTcie9n6kz2Eiy66CIBdu3YV+DxJfc2wZY2KWlZFsYTAx6yLFy8GIot9y+S9bF7lrySXarS0kGR9zCoJBqNGjQIikxLORsZt/miujD9lPOpPFpF+lcLo0ahbty7gTfrffPPNbpts6ZEgGe07eQ/8iwjE0kcbZwsyzpc+k+QR8LwRGedPmzYt3+cPHTqU77kZQMesimIzqqyKYgmBucHi7knOpn8liKwDlGmaTG//HoPAdkITV9U/NfHCCy8AiU0xRBXozHctLmT9+vUBqF69unuNTMs0aNAgqWf4yGjfSR5utBU/UsanS5cu7jmZthI335+EczbvvfceAM2aNfMEOtN3EsCT52co4KlusKLYTGCWVUo5+lfjCzLA90/oB0So9hgViyr7s8pqmdzc3IQ+/8orrwCeZRWPJR1lMqOQ0b4Tz0wCPn6kKF6qQSD/+laZtjp69CjgrZX1l8ZNcO/VRFDLqig2k3XLKjVsZDJbphj8UzIy+e9PMwyIUFlWyyiSfbdmzRrAq/ckif0QO82xkKhlVRSbUWVVFEvIem7wa6+9BkTuxwqR+a0hcH8VJSpB5AQLalkVxRKybln9pR/9yI7SiqJERy2rolhC1i2r1P9p0aIF4KWEtW/fPtuiKIpVqGVVFEsIfD1ryCmSE/tZQvsueTQpQlFsRpVVUSwhdMq6cOFCmjVrRuXKlcnJyeHiiy9m+vTpxHHXFfIKqBlj8v275JJLghYt1NjSb6GpbihUq1aNMWPG0KhRI0qXLs3q1asZNmwYJUqUiNhgSslPbm5uxCZRR48epWnTphF1lJT82NJv8QJMocAY8wqA4zg9gpbFJowxg4A5QF3HcfYGLY8thLXfQucG+zF5NAeuBt4OWh4LGQK8FqYXzhJC2W+hVFZjTEVjzBHgBLAemOU4zuMBi2UVxpgrgWbAk0HLYhNh7rfQjVnPcBi4HCgLtAYmG2P2Oo7zdLBiWcUQ4BPgzaAFsYzQ9lsoldVxnNPA7jP/3WKMqQxMBFRZE8AYUwHoA0x0bAhKhISw91so3eAoFANKBy2ERfQFSgHPBC2IZYS630JnWY0x44HVwMdASaAtMIqQdmBIGQK86jjOf4IWxDJC3W+hU1agAjAXuAA4Tp7Sjj5zTomDMaYlcClwd9Cy2IQN/WbFPKuiKPaMWRXlnEeVVVEsQZVVUSxBlVVRLCFeNPhcjz5ptYPk0b5LHq0UoSg2o8qqKJagyqoolqDKqiiWoMqqKJagyqoolqDKqiiWoMqqKJaQ0SVy33//PQAnTpxwzx0+fBiA5557LuLaDRs2uMeyedXPfvYzAFq2bFngM4YOHZrvXE5ODgDFixdPRmxFCSVqWRXFEtK2MZXfej722GMAvPHGGwCsXLnSe6CJn4UmMiV7ba9evQCYOnUqALVr1457nwLQlLnk0b5LHk03VBSbUWVVFEtI2g0+fvw4AAMGDABg//79btuqVasirm3atKl73LhxYwD69esHFN5F/eCDDwB49dVX8wQ8I//69evda/bs2RNx7+3bt7ttZcqUKczj1JVLHu275FE3WFFsJmXL2qxZMwC+/fZbt+0Xv/gFAHfccQcA1atXd9vKli2brKwxEWsK0LZt24hzhw4dctvKlStXmNsGbh3mzJmT79zw4cNTuufs2bMLbOvYsSMAjRo1SukZhKDvonHs2DHAm1Z8+mmvbnzdunUBaNCgAQBbtmxx2+69914AfvOb3wDw4IMPAtCiRQv3mnXr1gFQrFjKNlAtq6LYTMpTN2Jh/RRyXJgWVqxY4R5369YN8KaTbLSsYlFTtaLJItZ32LBhyd4isL4Tq/mHP/zBPbdv3z4AHn30UQBOnjxZqHs2b94cgI0bNwJw+vRpAEqX9jaKeO+99wBo2LBhMmL7UcuqKDajyqoolpBybnAQLq+fZcuWAdClS5d8bd27dwegVKlSWZWpKCDudwpucGDIVN2IESNSuo8/MFq+fHnAmyoUrrzySvc4De5vTNSyKoolhHFjqnz4845zc3MB+Mtf/gJ4ecj+3GAJpy9atAiAkiVLZkXOdCJTKLGQKTI/0i+pEu3e5wo1atQAIpN7ZDpHAkoSoBo4cGDW5FLLqiiWkLZVN5lgx44dQOS46exURhmPDhkyxD139915u/alsNpGCOXEfqokMi2Uht0FA+s78cSmTZvmnpOpvR//+McAVKpUCYjeB5K4I9f4EcsqPzdv3uy2/ehHP0pFbD86daMoNhMay+pfCDBz5kwAnnkmb7PzL7/8ssDPjR07NuJnmilSljURiypj1cWLF6f6uFD13XfffQd48YtE1kr7GTduHAATJ04EoH79+gDs3r07XSL6UcuqKDajyqoolpB1N1gG5O+//z4ATzzxBAC7du1yr5Fc3liuihRV69ChQ7pF9BMqVy5VEnH9JKhXVFfdFAa/bkiwUnKMX3jhBQB69+6diUerG6woNpOVpIgpU6a4x6NHj457vaxoiLUusHPnzoAXZpdkCfDSvrQUafT1sGfjX9+aBotqPZLwIFOA4FlU8U6qVKmSdbnUsiqKJWRlzPrhhx+6x7/85S8j2s6ulwSxLeumTZsAb8V/tFKksqr/vvvuA6BixYrJim79uCvWODWN0zRRH53CZwPtu3fffRfw1rD6kaLys2bNyqQIOmZVFJtRZVUUSwg8g+ngwYNA9DzMaHzyySeAV6BNMkukNKmfc7EU6c6dOwEvBzYWacj/jYV1fScsWbIEgB49euRrmzt3LgCDBg3KpAjqBiuKzQS+njVRiypITqbw0ksvAZ5FAbj22msBL3h16tSpVES0iuXLl8e95lxeqxoLCVpOmDAhX5t4KjfeeGNWZfKjllVRLCFwy5oqMr1Tp04d95xYa3/h76KM36vI0oqaIsmYMWMAb3rQj6yHTWEaMGXUsiqKJWTdskoB5nTXRZLtDCBy24NzgViRX//4VC1qfmScCrB69eqINn/lwsLGVjKBWlZFsQRVVkWxhLS5wf5yoVKGZevWrUDkfjhSOLlly5ZJPUfu9dVXXwEwdepUwFvfCl4+bNWqVYG07OoVSnr27Bn3mt/97ndZkMRe/FNdso+NrNaaNGmS2+bf0yYoiuZbrChFkJQtayLlQv0Tyffffz8A27ZtS/gZUqwbvBC6rF+NtuqmXbt2APzpT38CICcnJ+Fn2YCsUY1V0FsCS7o+NTriCUYrtNe3b18AOnXqlFWZ4qGWVVEsIeVEfkmSb9q0acE38T0jkTpA0axlQcgu5127dnXPDR48GEhLuD2UyegBrlEtDKHsO3m35s+fD0QWh5fNp5YuXQpAmzZtMiVGPDSRX1FsRpVVUSwh5QCTBDA++ugj99zLL79c6PtIpXOAw4cPA95Av0mTJvmul927ypUrB9i5U1wi+PN+JTgXixC4v6Emmvsr9O/fHwjU/Y2JWlZFsYTAK0WEnMCDJP5SogWtqAlp/m/gfReNChUqAHD06FEgchWNFKD3r+AKCA0wKYrNWL+eVQmVNbUOiYtAKCxqTNSyKool6Jg1NqEad0kyhIxRJUk/pCmFoeo74aabbgJg7dq1AHz66aduW4i2W9Exq6LYjCqroliCusGxCaUrZwnad8mjbrCi2Ew8y6ooSkhQy6oolqDKqiiWoMqqKJagyqoolqDKqiiWoMqqKJbw/ynY2wK6C+yWAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "dls.show_batch(max_n=9, figsize=(4,4))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now that we have our data ready, we can train a simple model on it." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### A simple baseline" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In the previous section, we built a model based on a `conv` function like this:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def conv(ni, nf, ks=3, act=True):\n", + " res = nn.Conv2d(ni, nf, stride=2, kernel_size=ks, padding=ks//2)\n", + " if act: res = nn.Sequential(res, nn.ReLU())\n", + " return res" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's start with a basic CNN as a baseline. We'll use the same as we had in the last chapter, but with one tweak: we'll use more activations.\n", + "\n", + "As we discussed, we generally want to double the number of filters each time we have a stride 2 layer. So, one way to increase the number of filters throughout our network is to double the number of activations in the first layer – then every layer after that will end up twice as big as the previous version as well.\n", + "\n", + "But there is a subtle problem with this. Consider the kernel which is being applied to each pixel. By default, we use a 3x3 pixel kernel. That means that there are a total of 3×3 = 9 pixels that the kernel is being applied to at each location. Previously, our first layer had four filters output. That meant that there were four values being computed from nine pixels at each location. Think about what happens if we double this output to 8 filters. Then when we apply our kernel we would be using nine pixels to calculate eight numbers. That means that it isn't really learning much at all — the output size is almost the same as the input size. Neural networks will only create useful features if they're forced to do so—that is, that the number of outputs from an operation is smaller than the number of inputs.\n", + "\n", + "To fix this, we can use a larger kernel in the first layer. If we use a kernel of 5x5 pixels then there are 25 pixels being used at each kernel application — creating eight filters from this will mean the neural net will have to find some useful features." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def simple_cnn():\n", + " return sequential(\n", + " conv(1 ,8, ks=5), #14x14\n", + " conv(8 ,16), #7x7\n", + " conv(16,32), #4x4\n", + " conv(32,64), #2x2\n", + " conv(64,10, act=False), #1x1\n", + " Flatten(),\n", + " )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As you'll see in a moment, we're going to look inside our models while they're training in order to try to find ways to make them train better. To do this, we use the `ActivationStats` callback, which records the mean, standard deviation, and histogram of activations of every trainable layer (as we've seen, callbacks are used to add behavior to the training loop; we'll see how they work in <>)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from fastai2.callback.hook import *" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We want to train quickly, so that means training at a high learning rate. Let's see how we go at 0.06:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def fit(epochs=1):\n", + " learn = Learner(dls, simple_cnn(), loss_func=F.cross_entropy,\n", + " metrics=accuracy, cbs=ActivationStats(with_hist=True))\n", + " learn.fit(epochs, 0.06)\n", + " return learn" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
02.3070712.3058650.11350000:16
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "learn = fit()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This didn't train at all well! Let's find out why.\n", + "\n", + "One handy feature of the callbacks passed to `Learner` is that they are made available automatically, with the same name as the callback class, except in `camel_case`. So our `ActivationStats` callback can be accessed through `activation_stats`. In fact--I'm sure you remember `learn.recorder`... can you guess how that is implemented? That's right, it's a callback called `Recorder`!\n", + "\n", + "`ActivationStats` includes some handy utilities for plotting the activations during training. `plot_layer_stats(idx)` plots the mean and standard deviation of the activations of layer number `idx`, along with the percent of activations near zero. Here's the first layer's plot:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs8AAADWCAYAAAAuNG/NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd5iTVfbA8e/JVGDoTUBhpAsiKIgiXVEUdUFdy9p2/a2La1l7wY6uda1rWRV77wUVFQsgvQzSOwy9w8AwM0xN7u+PN28mySSZZCZTkjmf58nD5K03wyTvyX3PPVeMMSillFJKKaXK56jpBiillFJKKRUrNHhWSimllFIqTBo8K6WUUkopFSYNnpVSSimllAqTBs9KKaWUUkqFSYNnpZRSSimlwqTBs1JK1XEiMkxEjIgcWdNtUUqp2k6DZ6WUikMicqQ7IB5W021RqjYQkUtEZJOIHBSRt0UkyWtdgojME5GLa7KNKjZo8KyUUkqpuCYizYG3gXuBwcApwFivTW4FdhhjPq2B5oVNRJLj8VyxRoPnOkJEponImyLyiIjscX/zflREHCLygIjsFpG9IvKo1z6JIjJeRDaKSIGIrBCRa/yOe5OILBaRXBHZJSKfiEgbr/X27eDTRWS6iBwWkZUiMrI6X79S8UpEBonILBHJcT+WuN9fW92bTHW/Bzd57fMvEdnmfj9OBtrXRNuVqkYdgWxjzIfGmGXAN0APABHpAtwEXBvOgdzXxfUiMlpEVotInohMFZFOftv1FZGf3dfHvSLylYh08Fp/tHvZDvd7cZmIXOF3DPva/W8R2QlsD9Kmae73uf/jb17b/Mvd3gIRWSci94pIotf6Te4Y4X8ish+Y5V7exn1tPygi+e5z9QvndxWvNHiuW/4MJAGDsL5l3wN8D6RhfRO/HbhHRM5yb/8GcD5wDXAM8DDwpIj83e+4twO9gPOwLsKfBDj308BjQG8gA/hURJpE7ZUpVQeJSALwLTAPOMH9GA8cdv8McAHQBjjRvc9o4DngWaAP8BnwVHW2W6kasB6oLyL9RKQBMBRYJCICvAncY4zZFcHx2mAF25dh9WI3Ad6yV4pID+B3YA7QDzgVcAK/iEiqe7M04DfgTKxr6ATgbREZ7neui4CWwGnu4wRyvrtN9uMRIA9Y4G7PeKxr9d1Y1/ObsK7tD/od50ZgDzAA+Kv79/MN0B04B+gP7Ha/jhYhf0PxzBijjzrwAKYBi/2WrQCW+S1bghXoHg24gO5+6x/wP47f+uMBA7RzPx/mfn6+1zZHuJeNrOnfiz70EcsPoKn7vTQswLojA60DZgIf+i172r3tkTX9mvShj6p6AOe6r3GZWF8gE4B/AZPc16Vv3OveA9JCHGc8UAK09Fp2ifuamep+/g7wid9+KVhfbMeEOPZE4HWv59OAtYAjgtc5EigEznU/r+8+75l+210JHPR6vgn4zW+b09yfDT38XsdO4IGa/j+tqYenu17VCUv8nu9yP/yXtcL6pixAhvXF0yMR69szYKVlYH2T7YH1zdu+m9EB39tLi+0fjDG7RMQJtK7g61BKAcaYAyLyBjBZRKZg9XR9bYxZE2K3HsDHfstmArdVUTOVqhWMMd8B39nPRSQdGAecBLwArMK6Q/sBcD9wV4jD7TDG7PV6vh3rmtkK2IJ1p6eziOT67ZcKdHGfvz5Wh9S5WL3FyViB6VS/fRYaY1zhvEYR6Yl1N+ku9+sF6AnUA74UEeO1eQKQKiItvV7LfL9D9gT2G2NW2guMMYUiMs+9rk7S4LluKfZ7boIsc1AaBJ+C9Y3VfxtEpD3wA/A+VkrHPqzerl+xPgS8FQVoj6YNKVVJxph/iMh/gTOA04F/i8gNWL1pQXerlsYpVbu9Dow3xmwTkRHAQ8aYEhH5AHionH39r2n2e8rh9e/7wBMB9t3v/vcpYDTWF9fVWGkWzwCN/bbPK++FAIhIK6xUzA+MMc97rbLbdCFWL7a/rHLOFejzQoIsrxM0eFbBLHT/294Y832QbU7E+jZ7szEmH6wBEtXROKVUKWPMcmA58KyIvIpVReBr9+oEv81XAgOB/3ktG1jljVSqFhGRfwBijHndvciBNSYIrM6fynbuZADHARuMO9chgCFYKVSfutvkALpi5RRHRERSsN7zq7Hylr2tAAqAjsaYHyI89AqghYj0sHuf3efqj+9nSJ2iwbMKyBizXkTeAl4XkTuxBj00APpi5Xk9CazD+uZ5m4h8iDUY8IGaarNSdY2IdAb+gXUreivQFmvw7x9Yd4JygTNEZAVQaIw5gNWz9bmIzMe6czQIuCLA4ZWKSyLSDmug3CCvxdOBW90Vp67HSoGqjMewUiA+cN8Z2gukA2OA/xpjMoE1wGgR+RLrvXor1ns44uAZeM2971VAS690y2xjTK6IPAY85l7+C1b81ws43hgTKj1livt1fCQi1wPZWCktqcArFWhnXNDb5iqUsViDKu7F6q36Dfgr1oAKjDFLsQZbXONefztwc420VKm6KQ8rf/ITrNuxXwKzgRvcOZLXY43U3wosAjDGfI11m/hOYClWtYBQF0+l4s1rwBPGmE1ey27ECm4zsN5X5aVthGSMWYWV9pgGTMa6Rr6Odbf2oHuzW4DNWDnOv2HlTX9RwVMOw2r/GqzBfPbjYnd7/u0+39VY459mup9vKud1GKyAfzVWKtgCrMGVpxtj9lWwrTFPgt9NUEoppZRSSnnTnmellFJKKaXCpMGzUkoppZRSYdLgWSmllFJKqTBp8KyUUkoppVSYNHhWSimllFIqTDFV57lFixYmPT29ppuhVK2xcOHCfcaYljXdjkD0/aqUr9r8fgV9zyrlL9h7NqaC5/T0dDIyMmq6GUrVGiKyuabbEIy+X5XyVZvfr6DvWaX8BXvPatqGUkoppZRSYdLgWSmllFJKqTBp8KyUUkoppVSYNHhWSimllFIqTHEVPK/bncMn87fUdDOUUlHgdBle/G0dhwqKa7opStV5uYUl3PnFEmas21vTTVGqxsVV8Hz6c9MZ99Wymm6GUioKfl21m2d+Wcsj36+s6aYoVecZY/gsYxurd+bUdFOUqnFxFTwrpaqOiEwTkQIRyXU/1lTl+QpLXADkFTmr8jRKqTCkJiUAUFCs70el4jJ4drlMTTdBqXh1gzEmzf3oVtONUUpVj6QEB4kOoaBEg2el4jJ4dhoNnpVSSqloqpeUQH6Rq6aboVSNi6vg+ZROzQFroJFSqko8LiL7RGSWiAzzXykiY0UkQ0Qy9u6NzsAiicpRlFKVlZKUoD3PShFnwfPwbq0AKNHgWamqcBfQEWgHTAC+E5FO3hsYYyYYY/oZY/q1bNmyUiczegdJqVolJdFBYbH2PCsVV8FzgsPqo3I69aKrVLQZY+YZY3KMMYXGmHeBWcCoqj6viPY9K1UbJDgEl36pVSq+gufEBOsiW+LSb8ZKVQODZlUoVWc4BA2elSLOgmdPz7OmbSgVVSLSRERGikiqiCSKyGXAEGByTbdNKVU9HA7R66tSQGJNNyCaEh12z7O+uZWKsiTgEaA74ARWA2OMMVVa61kpVXs4RNCOZ6XiLHhOcFgd6frNWKnoMsbsBU6swfOzI7uAxvWSSEuJq48tpWJGgmjOs1IQZ8Gz9jwrFX/mbNjP0Xf/AEDX1mn8fMvQGm6RUnWTiHZOKQVxm/OsAwaVihf7cgs9P6/dncumfXm8P2cTxc6Kv8+LnS427M2NQuuUqjusahs13Qqlal5cBc/a86xU/Ah2d3jY09O4f+IK+j/6a8j98wpLWL3rUMB1j05axWnP/M7O7PzKNlOpOsOhaRtKAXEWPNs9zyVa51mpuHfgcHHI9Td/upgzn59BQXHZGdHmZu63jpEX+hhKqVJaqk4pS1wGz5qTpVTsM4T/Ps7OL2bt7hzenrURsILjX1buBuDxH1aRXU6grZQqn5aqU8oSleBZRJqJyNcikicim0Xk0hDbDheRqSKSLSKbonF+W4KmbShVpzz03QryCkvo/dDPnPHcdB76biUul+GSCXM927w7ZzMPf7+yBlupVHzQUnVKWaLV8/wyUAS0Bi4DXhGRnkG2zQPeAu6I0rk9ErVUnVJx4Y0Zmdzy6ZJyt3t71iZ6Pug7T8vbszeV2a6wxErdWLj5AOnjJrH7UEFU2qlUXZIg2vOsFEQheBaRBsAFwP3GmFxjzEzgW+CKQNsbY+YbY94HMit7bn+lPc9abUOpWPbIpFUV3vffAXqZ1+/JZf2eXN6bswkoP19aKVWWaM6zUkB0ep67Ak5jzFqvZUuAYD3PERGRsSKSISIZe/fuDbltYoLmPCsV61btDFwhozJW78phxLO/4xCJ+rGVipYIUyBPEJHpIpIrIrtF5Cavdenu9MjDIrJaREZEo31WqTq9vioVjeA5Dcj2W5YNNIzCsTHGTDDG9DPG9GvZsmXIbTXnWanYd9Z/Z1TZsf1j50gGJSpVDcJKgRSRFsBPwGtAc6Az8LPXJh8Di9zr7gW+EJHQF9AwWKXqKnsUpWJfucGziEwTERPkMRPIBRr57dYIyKmKBodi13l26btbqbjXuVVaxPsIvtFzqAwvYwyTV+zSzxNVLSJMgbwVmGyM+dAYU2iMyTHGrHIfpytwAvCgMSbfGPMlsMx97Eq2Ue/sKgVhBM/GmGHGGAnyGASsBRJFpIvXbr2BFVXV6GC051mpuuPh0ZFnhv2wbKfP8+nrfFPBpq7ewx9bDrAru4CJi3dwzfsLecdrAOLWrMMs3nqwQu1VqhyRpECeDGSJyGwR2SMi34lIe/e6nkCmMca7AytoKmUkqZEJDsFo2oZSlU/bMMbkAV8BD4tIAxEZCIwG3g+0vYg4RCQVSLKeSqqIJFe2HaDVNpSKN/WSEjw/b3ribDY9cbbneXJC5B9f+X4Tpjw1eQ2XTJiDMYapa/Zw1TsLOP9/szn58d88FTm8ZyEc/J+pjHl5VsTnVSoMkaRAHgn8FbgJaA9sxErViPQ4EaVGOkRwavCsVNRK1V0H1AP2YL2BrzXGrAAQkcEikuu17RAgH/gB602fj2+uVoXZPc/FTq22oVQ8WPnwyDLL/rj/dKbdPoy+HZpyz6juDOjYvFLnmJuZxf68Iq56e4HP8r05hUDwacLX7s7h2yU7KnVupbxEkgKZD3xtjFlgjCkAHgJOEZHGER4nIg6RkKlOStUVidE4iDEmCxgTZN0MrG/C9vNpQJUMeU9JtL4LFOv03ErFBRHhon5H0qVVaadZswbJNGtg3awaO6QTY4d0Ytm2bM59aWaFz3Pm89PLLHtjpjVbocvAvMz9zFq/z2f9Gc9Z+/ypd9sKn1cpL54USGPMOveyYCmQS8FntKv9s7i37ygiDb1SN3oDH1W2gVqqTilLVILn2sIOnu0JEZRSsWXljrJl6v7z597l7pecWLmbaPtyi4Kue2vWRt5yT/utVFUxxuSJiJ0CeTXQBysF8pQAm78NfCkiL2AFy/cDM40xB4GDIrIYeFBE7gPOAo4jGgMGK3sApeJEtNI2agX7AlpYrPeVlIpF3gP4bjytS4gtffkHz3blneoQyQCq5duzycoLHqirOi9gCqR/+qMxZgpwDzDJvW1nwLsm9CVAP+AA8ATwZ2NM6NGASqmwxVnPszW4qLBEg2elYpF3yPvXAR3C3s8/Vg40qOm4IxuzdJv/OKqKefKn1Z6fi5wuz2dPec55cSYdmtfn9zuGR6UdKr4ES4H0T390L3sFeCXIcTYBw6LdPpHgYwCUqkvisue5SINnpWLSu15l4ZqnpYS9n31Bb9ekns9zgFG9jgBg4vUDGX9uD8/y5y/uU+F2vjJtg+fnSD9vNu8/XOHzKlWT/OukK1VXxVXwnOAQkhJEc56VilE7sgsqtF+H5vW5fngn3vt7/zLrXvrLCax/9CxEhL8NPNqzPKkCpe4C2XGwgDdmZDLbPaBwa9Zh8gpL2Jdb6LOd1sdV8UBn5VQqztI2wLogas+zUnWLiHDHyO6e573aNebuUd3JyivC4RAcfj1mF/Y9ssxU3RU10qtSx2fXDOCi1+Z4nm94bJRXCU0NOlRsi9Z7RqlYF3fBc4JDi7grFYu8e2afu7j8ChvB/HjTYNo2qUfjekkB19sTrXyWsbXMOofAS5eewHUf/lGhc3sHzgAHDxd50k9KvArk5hc5qZdcNk86r7CEr/7YxuUnd0A0UlG1kF5elYqztA2wRtnrDINKVQ0R6SIiBSLyQbSPnVdUmm41sucRFT7OMW0aBQ2cveUWlABw46mdmXHncK4f3on1j45iVK82DO7SosLn95aVV8Ss9fs4/dnfySssfX1LtgWe4vuRSSu5f+IKZqzbF3C9UjVJv88pZYm74DnB4aBEg2elqsrLwIJyt6qAnIJiz8/1k6v+ptgl/Y9i7JCO/HNYJ45qVp87RnbH4U6x8C91F04wHsi2A/lc9+EfrNuTyy2fLvYsn71+Hxv35THwiSlMWb0bgAWbsth2wJoKPK+wpELnU6qq6dVVqTgMnhMdQolOz61U1InIJcBB4LeqOP5+90QlT5zfqyoOX0b95ETuGXVMwED94dHH+jw/7sjGvH5lP+4+q3uZbUO56p0FZOdbXwpmes1Q+MKU9Qx/ehrbD+bzf+9ksPtQARe+OsfT46zf/1VtpNU2lLLEXfCc4BDteVYqykSkEfAwcFtVnWPFDqsGc7/0ZlV1irAd1aw+HVs08Dw/lF/M6T1a83+DSqt1pKVEr3d87e4cn+c6bkPVVlo1Rqk4DJ4TEzTnWakq8G/gTWNM2VF2XkRkrIhkiEjG3r2RTWiW485BbtUo/PrOVcqrk+3PfY8EfMvbLX9oZNRO5T8t+X1fL2NrltaDVrWMdjwrBcRj8Kw9z0pFlYj0AUYAz5W3rTFmgjGmnzGmX8uWLSM6zz532kaDash3DkeXVtaEbjPvGs4VA9IDbnPtsE5ROdfjP672eX6ooITbP1/ieT7uy6X856fV/rsBVkWP/X41pZWqKnp1VSoOS9UlOhw4tZ6qUtE0DEgHtrjLp6UBCSLSwxhzQrRO8urv1qx9Cf5zbdeQZy7qw6WbD3Bk0/o+y7sf0ZDVu6w0C1cV3sK2c6UBPllgdfhPXrGL16/sx+aswwzv1gqAPg//ApSW4IvUn1+ZTc+2jXjIL89bKX+1452pVM2Lu+BZc56ViroJwCdez2/HCqavrZHWVJO0lESGdi3be/7dvwZ5gubkILMUJic4KKrkwOWiEheHCor5cdlOz7INe/M49ZnfAfhk7Mk88/Maz7qx72XQulEqD/2pp6dqiL+5mfvp1rohTRske5ZlbD5AxuYDGjyr8OjlVak4TNtIEJwurbahVLQYYw4bY3bZDyAXKDDGRJbUHIYWabUk3zmEpAQHKYnWBCeJjsAfoT/cNLjS58ncl8dx43/mri+XBVx/22dLWLDpgOf5zyt38/7czazbkwvAryt38/Tk0uDa6TJcMmEuV7w1r9JtU3WTTtyjlCXugmfteVaqahljxhtjLo/mMV3u9+zlJ7eP5mGr3GUnt2dAx+Y8dp5VXu/kjs1ISXSQlFD1Qcb2g/kBlxeWWJOxXP1eBi9NXe9Zbs9wuMJvcKJSkdCrq1JxmLahMwwqFXuK3YFdUpA0iNqqRVoKH489GYBLTyoN/Hdm+wa2Y/q05ZvFO8rsf/4J7fjqj+1RbVNOQUnAz0B7mVYaUxWl/c5KWaJypRKRZiLytYjkichmEbk0xLZ3iMhyEckRkY0ickc02mDTnmelYk+Je5Cv/8x+saqZV04xBE5Hefz8Xjx5wXG8fmW/qJ77sjfmccWbZVMzinUgtYoCrfOsVPR6nl8GioDWQB9gkogsMcasCLCtAFcCS4FOwM8istUY80mAbSOW6HCQX+yMxqGUUtXEEzzHWM9zMHZO9PHtmzDimNZcNTCdN2ZuBGDj46NYufMQPdo0QkQ4rXsrBnRszpzM/VE7/+wNvsf6etE2dmYXhL1/bmEJW/YfpkfbRlFrk4p9mvKslKXSwbOINAAuAI41xuQCM0XkW+AKYJz/9saY/3g9XSMiE4GB+I7mrzDteVYq9pSmbcTP1Xn+PafRMDWJeskJPstFhJ5tG3ueOxzCx2NPptM9P3hSKx47rxf3fB14oGCkcgtLuOXTJWWWT1wcPF3kts8WM3nFblY+PDLg9OWq7tKrq1LR6XnuCjiNMWu9li0Bhpa3o1hDdwcDr4XYZiwwFqB9+/IHE1k5z1ptQ6lYUpq2ER89zwCtGqVGtH3Pto1Yui2bOXefSpvG9aIWPB/74OQyy9LHTQq5z6qdVh3rDXvy6HVk45DbqrpD0Jx5pSA6Oc9pQLbfsmygYRj7jne34e1gG0Q6Y1mCQzwXYqVUbCh210SOl5znQG48rQtv/jV4fvM7V/Xnw6tPok3jegD8cGPly91F4nBRCXmF1hTp9d295ee+NLNa26CUUrGg3OBZRKaJiAnymIlV89U/Ma4RkFPOcW/Ayn0+2xgTtbllExM0bUOpWGMHz0mJ8Rs833p6V047pnXQ9c0aJDOwcwvP80D5xv3Tm5VZ9tKlx0elfb0f+pmeD04mK6/IM4MiwLrdIT/KVR0iIhhN3FCq/ODZGDPMGCNBHoOAtUCiiHTx2q03EGiwIAAi8n9Y+dCnGWO2VfZFeEtwOLRUnVIx5lCB1ePZKDWphltSu/x8yxCf572PKptCcXavNlE5l12N45/vL/RZfvpz03l00ko27suLynls2w/mszcnav0mqhrE71dbpSJT6bQNY0we8BXwsIg0EJGBwGjg/UDbi8hlwGPA6caYzMqe31+SQzyTASilYsPBw0UANKmfXM6WdUvX1g1ZdP/pvPd//QEY1q1VmW2iPevb/E1ZZZa9PmMjl78xjw17c5m0dGeAvSI38IkpnPjor1E5lqo+mvOsVPRmGLwOqAfsAT4GrrXL1InIYBHJ9dr2EaA5sEBEct2PV6PUDhIcglNznpWKKTsOWmXUWjWs/dNzV7emDZIZ0rUl6x89yyeto0Pz+p6f7zyzG+2a1KvSdmw/mM9pz/zO9R/9UaXnUbWYdj0rBUSpzrMxJgsYE2TdDKxBhfbzo6NxzmA051mp2LMl6zDJCQ6ObFq1AWAss2tg9+3QlLN7teGS/kdRUGzdZbtuWGeuG9aZvTmF1dKbO2npTvq0b8LAJ6bwl/5HkZzgYPTx7TihfdMqP7eqWdrzrFQcTs+doNNzKxVzikpcpCQ6op6CEI++vPYUz8/+WS4tG6Zw0tHNmLexbOpFNHn3Pn88fysA787ZTMZ9IwLOpqjig2jXs1JA9NI2ao2kBAdFJZrzrFQsKXa6SEqMu4+jGvGX/sHr4Tepbw3IXPLgGVVy7n6P/MqAx3+rkmMrpVRtEXdXq9SkBApKdHpupWJJUYmL5DiZmrumjTm+Hc9c2BuA4d18a+P/dutQpt4+jMb1kvjvJX2q5Pw7swvYmZ3Pa79vwER4j//pyWu44s15VdIuVXl6Y0gpS9xdreolJVDsNJQ4tfdZqVhh9TzrlTlaxhzfjjtGduOFvxzPVQPT6duhKd//axDN01I4ukUDAEb2PIIL+x7Jv0f39Oz3/t/78+C5PTzPxw7pWKHzD3h8Co//uJopq/f4LP9p+S6WbfOdU2tu5n4enLgcgJemrmfGun0VOqeqHpF+IVIqHsVdznNqkvV9oKDERZr2ZCkVEwqdLpL0/Ro1CQ7h+uGdAXjw3J4Bt0lNSuApdw/1hf2OosRlSEtJZGCnFjz03UoA7hl1DBOmV7yi6N/fzWDTE2d7nv/zA6uG9NLxpWkjl0yYC8C9Z5cG7XtzCnl9RiZ3juzmGSipap5+vVXKEnefSvWSrGllC4o1dUOpWFGsaRs1KjUpgbQUqy/FEeUp0mevt3qS7/l6mWfZrZ8uLrPdRa/N8fx81TvzmTA9k39+sJBNUZ6cRVWO9jsrFYc9z2mp1ks6eLhYR30rFSMOFzlJ1gGDcek/k9ew+A3fPOZfV+0ps93irQc9Py/ffsiz3dJt2cy/dwSHi0pITUzg8R9XcXSLNC49KfjASFU1NOdZKUvcXa3aNrbqxO7Mzq/hliilwrV61yE6t0wrf0NVbY5t18jnuV2De+yQjky6cVDYx/EOiitiT04h2fnF9HhgMk/8tJrXZ2z06cVW1UtTnpWK457nvEJN21AqVhwuctKsgU7NXVssf2gkSQlWN+Px7ZuwaMtBJt04mNenZ3LziC4kJjj49dYh5BU6Gf3yrCpvz+M/rALg0wVbPcvW7MphzoZ9XHTiUdRPjrtLWa2kdZ6VssTdJ47DfV9JRwQrFRuMMeQXO6mXnFDTTVFudv4zwKdjB1DiclE/OZHbR3bzLO/cqiEA/zq1My9OWV+l7fl97V7AGhBu31Qc+fx0AMZ/t5Inzu/FJSHqW6voMZr1rFT8pW0kuAe7ODV4ViomFDldGGMNWlO1T3KiI2TP7m1ndCPzsVFlJl6Zd89pnNu7bVTasDO7ACjtHPH36KRVER2v2OnScqYVoDnPSlniLni2P1x1im6lYkNBkRXEaPAcuxwOoXG9JDY9cTYL7xvBH/efTutGqbwQYCKWji0bVPg8dhDtLyFBWLApi4WbD4R117HLvT9yzoszfZat2nmI9HGTtLpHObRfSqm4DJ6tf/UNrlR0icgHIrJTRA6JyFoRuToax813l5W0a7Sr2NY8LcWTvy5+XZWZj42iQRXkJx88XMyFr87hgldmc+tnS0gfN4nXg9SnzikoBmD1rhyf5V8u3AbA5BW7ot6+eKE9z0pZ4u5q5Unb0J5npaLtcSDdGNMI+BPwiIj0rexBcwtLAN88WxU/Xrui9E/E4RDOO74dADee1oUNj43i9zuG8dttQ8vsd4dXfnUkvl60HYBHf1jF7A372LQvj/wi6wvaN4u202v8z2X2ufaDhbwxcyOgKX/l0d+OUnEYPHvSNvQDUKmoMsasMMYU2k/dj06VPW6eO3iuih5JVfNG9jyCh0f35Kk/HwfAVQPTmXfPadx6elcSHEKH5g1Ib142lePaoZX+0+LBiSsY9vQ0jnngJ35ZuZubA0zOAvDj8tLeZqfTsHqXlcIxN3N/pdsQCRFpJiJfi0ieiGwWkUuDbDdeREDhXaIAACAASURBVIpFJNfr0dFrvXEfw173RpRaqHd1lSIeg2d3z/ODE1fUcEuUij8i8j8ROQysBnYCP/itHysiGSKSsXfv3rCOaQfPdplJFX+uHJDOhf2OAqxUjtaNUn3WJziE7kc09Dzf9MTZOBzCkgfO4MebBvPXAR0qdN7N+w97fv7Hexll1qePm8TU1b4TtpS4DLPWW0HzT8urPYXjZaAIaA1cBrwiIoHnV4dPjTFpXg//PJXeXuuikmKlaRtKWeIueE5wv7vzdXpupaLOGHMd0BAYDHwFFPqtn2CM6WeM6deyZcuwjqlpGwrgo3+cDEDjekmeZY3rJ3FMm0Y8NPrYCh2zKIyKGte8v9DnudNlaqTUqYg0AC4A7jfG5BpjZgLfAldUe2NC0q5npeIueHZ4vSItRaRU9BljnO4L+5HAtZU9nh08N9DguU6zvzz9M0i6xo2ndfF53r5Z/aic1z/Afmnqeh5xl74TseqQV9O1pCvgNMas9Vq2BAjW83yuiGSJyAoRCfQ+nC4iu0TkKxFJD3bSSO4WacezUpaoBM/h5mm5t71ZRDLdI/Z3iMhzIhK1q6Z3HdDKTgurlAopkWjmPKdoqbq6LDnRwaYnzubaYYH/pG49vSvz7jnN87xLK2s691cvP6HMtpUph+ftuyU7OfuFmXS+90fAuqZs2X+YLxZuo9f4yVz/4R9ROY9bGpDttywb606Pv8+AY4CWwD+AB0TkL17rhwLpQHdgB/B9sOtspHeLNOdZqej1PEeSp/UdcIJ7xP6xQG/gxii1w5O2AXDxhLnROqxSdZqItBKRS0QkTUQSRGQk8BdgSmWPnVtopVhp2oYqj3fO7VMX9uaJ83sxsucR9OvQ1Ge7u886hj9FYYKWfbmFrNx5CICCYidjXp7FkKemcvvnS8gpKGHSsp2VPoeXXKCR37JGQI7/hsaYlcaYHe67QLOB/wJ/9lo/3RhTZIw5CNwEHI0VbFeK5jwrZal08BxpnpYxZoP7DQ3WXSAX0Lmy7bDZAwZBy9UpFUUGK0VjG3AAeBq42RgzsbIHzisswSFQTydJUeWwK7Kcc1wbmjVI5pL+7RERuh7h2zmbmuTg2Yt6R/Xc475cGnD5LZ8u9tSOrqS1QKKIeOen9AbCGf1uCJ1VUd76sOlVVano9DxHmqeFiFwqIoeAfVgfDq+F2Dai0fsO/WasVNQZY/YaY4YaY5oYYxoZY3oZY16PxrFzC0tokJJYZkINpfw1SElkxp3DecYvMD7p6GY+z5MSHCQmRHdIzzeLdwRc/vWi7Zz5/AxW7jhUqeMbY/KwBuE+LCINRGQgMBp4339bERktIk3F0h/r7u1E97qeItLHfYcoDXgG2A5ENod5AKJZz0oB0QmeI8nTAsAY85E7baMr8CqwO8S2EeVjJWj0rFRMySss0ZQNFbajmtUnJdH3LsXoPu2Yf+9pvHTp8QB0be17+QmnF7phJf4Gtx/MZ9QLMyq8v5frgHrAHuBj4FpjzAoRGSwiuV7bXQKsx0rpeA940hjzrntda+BT4BCQiZX7fI4xJird4zVRiUSp2qbcTwsRmYY1+CCQWcC/CDNPy58xZp2IrAD+B5xf3vbhcGjvlVIxJa+ohPrJmrKhKqdVw1TOOa4t5xxXmuu8dPwZZOUW0b5ZfVqkpdCpVRqPTVrFKZ2bc+/Xy332X/bQSP700kyWbvPvC6o+xpgsYEyA5TOwOqrs53/x38Zr3RSgYtMzlkMvr0pZyu15NsYMM8ZIkMcgKpenBVEasW/zDp6Pbecf0yulapuiEkNyogbPKvoapSaR3qIBDocwpGtL2jWpx8uXncCl/dtzavdWZbb/9oZB9GhjXTeiVQov3mi/s1Jh9DyXxxiTJyJ2ntbVQB+sPK1TAm3v3uZbY8weEekB3A1Mrmw7bN5pG20b14vWYZVSVaTE5SIpQbu0VPUREd7624kAzM3c73Pd2LDXyo7ofkRDtmQdDrh/MNsP5tOuSfxed/RdqpQlWiMqAuZpAQTI1RoILBORPKypfX8A7olSO3wGDOosg0rVfiVOQ6KOVVA15OSOzTkxvXTAYWGJNSHKI+cdy5MX9OLzfw4A4PQercs9Vkpi3M07VoamPCsVhZ5nCJ6n5V7nn6t1VTTOGYyIsOmJs7lkwhwKi3WGQaVqu2KnK+qVEZSqqBuGd+alqetp1TCVi09sD8C6R88iKcHB+j05zFq/n0P5xfTt0JRL35jns2+LtJSaaHK10Yo4Slnidoh7alICWXlFNd0MpVQ5SlyG1CQNnlXtcPvIbtw+0ne8XZL7y13nVg3p3Mqq5FFXry9abUOp6KVt1Dr1khLIL9K0DaVquxKniwRH3H4UqTjlP538BSccWUMtUUpVt7i9YjVrkMy+3MKaboZSqhwlLkOS5jyrGJOSmMCU24bSvEEyABf1qxvBs/Y7KxXHaRutGqZy4HAxxU6X55abUqr2KXEaErXahopBHVumseDeEazceYhj2zWu6eZUOU15VsoSt1FlsnvUs9Ol35OVqs2KXTpgUMUuh0PqRODsoZdUpeI3eLbvAmvwrFTtVuLUtA2lYoEgGjsrRRwHz3bRe5eODFblyCss0cGlNahES9UpFRM0bUMpS9xesex6lK4Klnrec6iAV6Zt0LI8bs/+spb0cZNquhlVoueDk+n/2K813Yw6q9hldIZBpWKEXhOViuPg2b4WV7Tn+YaPF/HkT6tZvSsHsD4w0sdN4tlf1kariWH5dMEWlm/PrtZzBvLCb+sAcMVYGkyJM7xvTzkFJQGX3/75Ev7xXkal2jD65Vk8PXlN2NsXFDvJLQzcnnjkdBmf6ZGVUrWTvkuVssRt8OxwX4ydfsGz02V48bd1ZB8uDrl/njt4sXOm7X9fnrq+wm16/te1EQffd325jHNenFnhc0Yifdwknv05dJDn//usbgs2ZZE+bhJLth4sd9vPM7bS+d4f2XbgcIXP98XCbfyycrfn+eb9eRw8HHpyhP25hT5fMpZsPchLEfzdnPviTI59cHLkjY1RxU4XiVrnWamYEFvdJ0pVjbi9YnnSNoxh4eYDrNhh9d5OWb2HZ35Zy8PfrwzrOHasaMdClfnm/fyv6zw9uJFasyuH8d+u8ARlRSUuXpqyjoLi6ObqvjAldJAXzQGYV709n273/RjRPr+t2gPAGzM3UlxOr/LExTsAyNybV7EGBjD0qWmMeHZ60PV7cgro+8ivPP9rxe5QrNp5iHV7civavJhU4tS0DaVigeY8K2WJ2+A5wSvn+YJXZnP2C1bvrR1slhd02h8Sxv09u7oHHk5cvJ0XvQLtaz9YyDuzN7Ely+pF/Xj+Fp7+eS0TpmdWa7sq+nvYtC+vTK7c1DV7KSyJLCnd/v/4bskOnvhxdcht7bY6Qnzih/oysG53TsDl3pPv5BaW8Owvayl2unhl2gbGvDQLgMkrdgfctzxn/XdGhfaLZSVaqk6pmKEpz0rFcfBsp1AW+QVnnoCqnBxL8etjtver6m/eTpehoNjJTZ8s5hmvFI96ydZUsHZubmGJ0/08dPpJMMu3Z/P+nE1k5RVx9bsZ5aYi2K+7z8O/MGPd3oDb7M0pZOa6fWWWz8vcz7Cnp/F5xrYKtTWYRVsOAHD/N8u5+6tlDHpyis/rKA2egx/juSBpNAXFTvbl+v5OfltVNiB+9ue1vPDbOr5dvIMnf1rNjuwCwAoI/c3N3M8bM6r3y05tZ4yhWEvVKRUTRLuelQLieIZBOzj+z2Tf3kk7oPIMKHSZgLND2Z8RU1fv5eHvVvLWVSdayyuQuLEnp4DDheGlV9zw0R/8uHxXmeUNU63/KjtYtntTSyqYRmHnUe/JKeTXVbt5b87msPYrKnEx/tsVPH7+cbRIS6ZjyzTPuotfm0PmvjzevupElm3L5sbTugCw1t2Du3T7QS468agKtdfD6+XaL/39uaVtn71hP6N6tSG/yMnczCwg9Af+gk1ZAZef++LMMukT8zaW3dbOjf9o/haf5YH+Wy6ZMBeAC/sexepdhzipY/Og7QLIyiuimXvq33hl9/xrz7NSscFo1rNS8dzzbAVM3y/d6bPcTpO117/y+wbOeXEmf7h7MW12uPXcr2vJ2HyAHQfzPSvSx03iug8Xht2W/o/+xrCnp4W1baDAGSA1yep5zitykp1f7KlOUNkc5GVhVvLwDj+NgYtem8Opz/zus03mPiu3+Kq3F/gMjLTbmBCFXgsT5Geb/f/6lFd1i2CnnbpmDxv2Bs4vDpR37H2crVmHefzHVZ4vYws3+/79BOp5tl359nwunjCXYqeL9HGTgg5C/ef74f+NxSr7y59W21Cq9tN3qVKWuO15DtaRZQ+4s3um7TJwOw8WQPvgx9ubY+W52h8ePywLHORWlUR3e6//8A+KnC7+PbonEHnwvGTrQT726iWdtsZKwSgvj01EPBvZQXK4nO5jl5cqY9uTU0BaSiL1k8v58wzQaDsIO5hfmnIRLOf5qrcX+DwvLHFyz1fLuX54p4Dbewf/f31rPpn78ujXoWnAbZ1Ow+wN+2gQ4DXYlULsgPupyWvo2rphmTz8rZWoEhIr7OBZBwwqFRs051mpOA6egwVMTuPbCxq0M9RvRYkzspxnuxJEUpRuR+e50z6K3Md1VGAGxa1Zhxn98qyA6wLdilu96xBdWllBXbAg3ekyFDtdnp7xQFx+Pc8Fxc6Av5eXp67njB6tOf05q5rFpSe1Z+Ki7Xx53Sl0a90QEfEZdBioRYF+3eF2an6xcBtf/rGNQ2HkkdtfIDL8epxtJS7Dpa/PC3kMO40D4B/vZTCws28aR3nVRKqTiKQA/wNGAM2A9cA9xpjIyqX4setwa6k6pWKAfsdVCojjtI1QwR6UBp/esWd+kZPVuw4BZT8j7B6y8nKeDxdZObCDnpxC74d+jrTZQc3J3O/z3G7Hx/O38vWiwAPxPl2whZs+WcTCzVnsOJjP4P9MDXp8/xh87e4cznx+Bs/+soZTn5kWdL+x72XQ/f6fQrbdc2ve3bvY/f6ffNJernxrPku3HeSpyWv486tzPMs/mreFvCInZz4/g9fdA+282xnoe0OgL03+Oc+PfL+SUwOk0dz79XKA0hQdL3mFJRyOYArvilQl8f/bqmVThicCW4GhQGPgfuAzEUmvzEGLndrzrFQs0Y5npaLY8ywizYA3gTOAfcDdxpiPytknGVgKpBljjoxWWwAKigP32gWrwCACN36yiF9W7mbFQyPL7Od057CG6nn+bMFW7vxyKW/+tR+7DxUG3zAKvHuK7/5qGU3qJzO8Wyufbe76chlg1TtukBy8Zxh8g72CYif3f2MFkou2HAz5Wn5bvcezT3nH9k578C7lNn3tXrLdVTKy8wP3+j72w2pG9jzCZ4Dksu3ZzPP7UmGnbYT6kvPGzI1B1wGs2HGozLKeEU5aUpGBnP69//lRruFdGcaYPGC816LvRWQj0BfYVNHj2rnhOmBQqdqvIgPmlYpH0bxivQwUAa2By4BXRKRnOfvcAeyJYhs8ggVzdum6lEQrmPTuIMxwV14oLHGVCZLtHjLvxUu2HiR93CQ277du4d/55VIgcFWGaLPTSMD6ouCfvzvGLz0jr5xeTO/fw2u/Z3peQ7g9qA99F3zSGWcYg8LCOcvQp6bxzuxNPsvenuX7PNAdhwtemV3tvbjRmEymNs+ELiKtga7Aisocx/47TtQBg0rFhlr8uaRUdYlK8CwiDYALgPuNMbnGmJnAt8AVIfY5GrgceDwabfBXUFI2WCpxuvhuiTXrXH2/nlih9Ja/02UCpG3YPc+la65xV0MY+tQ0n21TE0P/Wu1axIu2HOClKet4YOJy3p+7mcd+WBVyP2+B8mHnb8wiO7+YohIXi8OYvtqbd6+n97HDzT742K9Umze7gkjIvOgKjkLxD8iveX8hTpchO9+3RvOO7LKpGFUp0slfAGat31/+RrWAiCQBHwLvGmNW+60bKyIZIpKxd2/geuDeoj02QClVdbTMs1KWaKVtdAWcxhjvGSeWYOVHBvMicA8QMqoRkbHAWID27UOUw/ATKG2j872lY5sO+gVXhtI86I/nb+GPLb7Bp91Dluuu62vtUxrwTV1T2oFeXlWJPg//wsL7RnDe/2aX8yqCK3aWDTYvem0O/dOb8dKlx0d8vCVbS0vW1fP6YlGZqaKLnS6WbjvIqp1WGkSj1OB/bsu3l02VCMekZb6lCAtLXHS978cyPb9b9h/myKb1SKqmgWn+k/PECxFxAO9j3WW6wX+9MWYCMAGgX79+5X4jCueuhFKq9tA6z0pFL3hOA/wLBmcDDQNtLCLnAYnGmK9FZFioA0d6MbYVlpMv+sHcLT6pD6t2HvKUo3s2wKxzgXJYvTtLf1+zl3ZN6rH9YD6dW6WV2dafPVNgRQULzuZvymLlzsgD0ZnrS2cG9O6Vz8oLPfNgKF3u9S3EcP/EFVzYr5KTpIQhUMrEVe9YaS2DOreo8vPHK7Fuu7yJlZo1yhhTsektveiAQaVih75LlbKE1Q0nItNExAR5zARygUZ+uzUCcgIcqwHwH+BflW18KKcd07rcbT5ZsJWfVlgpBf49mP4CBZHeMdo7szfRpH5SmeXBVLanbfvB4DWA/+aX/xypeiHSKypre4BKFtXJ+0uCitgrwDHAucaYqPxHegYMaqk6pWKC1nlWKszg2RgzzBgjQR6DgLVAooh08dqtN4EHE3UB0oEZIrIL+ApoIyK7Klv2ylv/o5uxbPwZYW+fuTf0xB/eM9bZ/PN07ac3frzIs+y3VbsJZFGEOcn+PssIXJ4uGrxTUKJtzobYyOtVvkSkA3AN0AfYJSK57sdllTmu3fOcqD3PStV6IjpeUCmIUtqGMSZPRL4CHhaRq7EusKOBUwJsvhzwvnd/CvAScAJQ/gijCDRMTYrm4coIJ6Xh7+9mBFxeXItzYr3LyEXbfe4SeCq2GGM2UwV3bUt0wKBSSqkYE80r1nVAPazScx8D1xpjVgCIyGARyQUwxpQYY3bZDyALcLmf157CtlXMvl2tIvfaFX25d9QxNd2MKndGj/JTj2KdPZZAS9UpVfsJvrO8KlVXRW2SFGNMFjAmyLoZWIMKA62bBkR1gpSa4l2Jozy/raq61Ih4N7LnERQUO3k0gtJ+0ZLevD6b9gfPN4+m3kc1qZbz1CS7VJ2mbShV+2mpOqUseq80irZkhR9U/byy6lIj6oKqLm024phWAZeHEzinlFPnO1xtm6RG5Ti1mdPT86wfRUrFAu13VkqDZxWjEirZBVKVFUVev7JfpY/x/t/7M6ZPuyi0pnbTAYNKxQ59lypl0eBZxZQxfdoC5U9EU57UpNB/+oEm2fH27zHHBl3nP3tlMA/9Kfjs9YO7tPSZzTJe2bn/OmBQqdigKc9KafCsYkw4NbTDYU8V3q5JvYDry5tF68K+wdP0ww0Ek6OU3hHL7ImKdMCgUjGgDnyhVyoccX/1/uDvJ9V0E6IqVmZii1Zg2DAlkV7tGnueXzGgg8+6irKD55M7Ng+4/vKTOgRcfsmJVpXFlEQHI/wm4klLSWR4t5Yc3bJBWG04vUdrjmoWOHivK4q1VJ1SSqkYE/dXrEFd4ms65h9vGlzTTQjoPxcc5/O8TePIB7s1rmfV5W7dKMWzrF5yAh/+4yQeGXMsX147gBPTm3nWPTymNO2hRVpyROeyB/XZwZu/1CA50Tec2plNT5yNiPDSpcfz8y1DPOuOb9+Et6/qH1Y+9eSbh9AiLYUZd54aUbvjjadUXYx8KVSqLtN3qVKWuA+eAabdPqymmxA17ZrUD2u7ji3C6/2MhiFdW3Li0c18llXkQ/byk9tb/57Ugdeu6AvAnpxCGqUmcfnJHejbwfccxSWlqRVXDTw6onPZ1TqKwpysxiHwv8tO4Mimpb//1KQEurZuWGbbcFIQuh1Rul/zBqWBf+tGKbx91YlhtSke2MFzVVdPUUpFj9Z6VnVdnQie01s0oGFq1Epa16h6YQ5GKy9t4u+DIgs2Q7m431Ehq1/cMbJbWMcRd8htgB5tGpW7fadWpaXD7YD12Hbl7+etyKvnuV+Hpp6fDYZTOlkpHW/+tR8z7jqVUb3aBDzG8e2teswO9++gvIF+/sH1Mxf19vw8/tyeDO8WuExePPLMMKil6pRCRJqJyNcikicim0Xk0iDbjReRYhHJ9Xp09FrfR0QWishh9799otO+aBxFqdhXZ65Yy8aPrOkmRF3HMHNr/X157Sncf06PCp93zt2n8sf9p3PWsUd4lvl/qNoBZPtm9bl+eOdyj/nGlf08xzAGGtcvf2r1vl7BbqI7ZzbcesH+Pc9n9GjNF9eewvnHW+XhUhMTmHBlP77/1yBOO6Z10IGFAP861Xp9wS4s/p2qVw1M93k+rFsrVv/7TDY9cTZnBQnQ41WJlqpTytvLQBHQGrgMeEVEgpXl+dQYk+b1yAQQkWRgIvAB0BR4F5joXl4pdgdHtAZuKxWr6kzwHI5rhnaMKKc4VEAVTdcM7Rhw+YleaQxLx5/hsy5U76d30OnvmCA9vnaJOIA2jevRzCvVQKR0sgvPsqBnKHXnmd345ZYhLLh3BCN6tOa849uRnODgT33akpZs3Sm4upwecjs9xR5IGc6Ayu5HNOSyk6wUkaOa1fe8BoCHRvdk/Lk9GNCpOWkpiRzrNVgxGPsOpveZv/jnAM/PmY+f7fl5/aNncU+AqcWD5VjHu2ItVacUACLSALgAuN8Yk2uMmQl8C1wR4aGGYc0e/LwxptAY8wLWx1OlB1jYb1P/z3ul6hq9YnkZO7hj0OAxkEDb3jKiK+/9X/9KtePU7r637e8+q2ywBb49nY1SfXtqK9KPN/7cHjxzYe+A65rUD91p4fTPgQujAX3bN6VL64a0bGgNEOzYMo21j57F0S0a4HAIGx4bxb1nB37tNnsAoT3Y0Dtd5Zohgb903DyiKxf1O4qfbh7MYPeAUrtHpWFqEn8beHRENZY9wbPXPv3SffOz7zqzO60appCY4KgT9ZvDpaXqlPLoCjiNMWu9li0BgvU8nysiWSKyQkSu9VreE1hqfBOTl4Y4TtgS3Hf2XJrzrOq4+EgEjpK0CPOina6yg81uGtGl0u04OsBgv2cv6k2LNCvIvPus7qQkOli9Kyei49539jEcEaIKxt8GHs22A4Gnn+7RNviXCmPKDiBpkZZC5t48LnaXdgu4XzntDWcQ2cNjevKPIR3ZmZ0PlAbBANcM7cRr0zMBePnSE2iRlszFE+bSt0NTRITuRzRiw548a79KxG726wjV3GuHdeLaYZ0qfpI4Zec864BBpUgDsv2WZQNlRyXDZ8AEYDdwEvCliBw0xnwc4XEQkbHAWID27duHbKD9JbdEe55VHafBs5dk9z2pu87szpM/rS53+5TE6N1q79wqjfV7cgFo4B4UOLhLCx47rxcA559QOinHNUOtIOzur5YGPNYJ7ZtQ6FdF4tbTu3L14MA9sd7sIKZZg2Rm3jUcgE37DnNMm4bc+UXg80HZ30XDlEQ2PDYqZEAZjd6LlMQEOrdKY8fB/DLrvM999nFWLvGmJ8722caeDKUywbNdq/mkowPXjFbBFbsMiQ7R3nilIBfw76VoBJTpJTHGrPR6OltE/gv8Gfg4kuO4jzUBKxCnX79+IT+U7ZldnU4NnlXdpmkbXuwL+DVDOjL9juHlbh9qiuZQ/AM4gF9vHer5uUdbK8/2vOPbeXJyAwsccHx6zYAyy248LbwecbtqhgD1kxOpn5xIj7aNQgY3Ilbu8MuXnsDT7rQPESsQD7VfuIP7IiFiBc2Du7QIawrvQCkXkep+RCNm3DmcqwdHr4JJXeF0GR0sqJRlLZAoIt4f1r2BFWHsayi9IKwAjhPfD7XjwjxOSHbPc5k0PaXqGO15DsDhENo3D11PecptQz25uuUZ06ct3yzeEfb5zzz2CL67YVC5ZdeCxXuRDr7qfkRDHrCrb0joY4dy9nFt+GXl7nK3u2ZoR+onJfqUhosme4BebmFJudval4DKhm+hv+SoYIqdLi1TpxRgjMkTka+Ah0XkaqAPMBo4xX9bERkNTAcOAicCNwL3uFdPA5zAjSLyKvAP9/IplW1jgidtI7z6+ErFK71quQ0OMBNhqOA4JYLqCIkBgtmP/3Ey390wKOg+vY5sXKne0HB2zbhvBJNvHsJPNw/hlM6Rz8QYqPOhNPc5eANaNUzlphFdwuoZroxQtadtdns1baBmlDi151kpL9cB9YA9WCkY1xpjVojIYBHJ9druEmA9VirGe8CTxph3AYwxRcAY4Eqs4Pr/gDHu5ZViB89abUMBuFyGP7YcqNE2bD+YzyJ3G/bmFPL4j6uq5e+zzgfPvY+yJriINHgqL+4rL6VjQKfmdGpVfbMABtIiLcVnpjtfFQtoPKFziN3DKSdX0fN6i+S/tCrCt8tOak+jOJmcp6qUuFwBv1wqVRcZY7KMMWOMMQ2MMe2NMR+5l88wxqR5bfcXY0xzd33n7u5ydN7HWWSM6WuMqWeMOcEYsyga7dPgWXmbMCOT8/83mzkb9gfdZt3uHN6bs6lKzr/nUAEDn5jCef+bDcC9Xy/jtd8zmbl+X5Wcz1udv2pd2NcaiOddxzgc5fVqXnFyB/7z5+M8z5+8oFeZbRyV7O209+7vVxbNWlfBY0fwmRio+cFS4U5ML03RuKhf8Aoc0RROBQd7eu2hXVtG/fyPnteLpXE4OU80FTsNSVppQ6mYkKjBs/Kyxl3xK9CAfds5L87kgYkrgk7pXuJ08dPynQHXr9iRzfo9wauK9X/sN5/ndqEEl8vgdBl+Xbm7yqaSj1rwHO60ou5tQ04tWp0uP7kDGx8f5VPNwvbOVSdy6UmlpXvSvfKgI005uPjEsiWAKpspYO9/Tu+ys9KlpVSsx7N5Wgoje7bm1ctPCLg+2auXMPDfpDsNwm/pJ2NLBzFWxYQggd4g4Xw5OaZNI5Y87ZpDYAAAGF5JREFUeAYX9C37/6+qXolTe56VihUJWqquVtuZnc/y7f5VCqMjc28ub8zILHe7lTsOsWxbaRvsgDbYF67Xpmfyzw/+YPKKXQBsO3CYA3lWhtHZL8xkxLPTw2rf5v15ZOcXe56/Mm09V7+XwZTVe0gfN4n0cZPo9eBk0sdN4r5vlnH9R6XnrIhoXrUimVYUgkwtWh3Oc0/BbAeCwVI2erZt7CkVB/DVdQM9P4cTmNVzB4kN3bfuJ904yOd4Fe4dDrL/6D5taeXO0/7vJX24Y2Q3RhzTmrYhajv7S3AIr13Rr8wkHwAfXX0SU24fWmZ5oFfh/+upiTq+4Z7SnmBFVb8Sd6k6pVTtZ3+OuzR4rjHfL93Bha/OLrN824HDDHh8Cue8ODNq5zLG8FnGVvKLnFz02lwembSK/CIne3MKMcZ4rv13fbmUBycuB2DUCzM496WybShylh1kujM7n7W7rZ7lrDwr8B305FQGPuk7tvX9uZt5d/Ymflm5mwcnLsfpMmz36+0e+tQ0Fm89CMD/vbuA2e5Ukr+/m+HZJsddROCDuVuYtHQnG/flRfor8YhKQqbXtKLHGmNygZkiYk8rOi4a54imcWd15+tF2yPez3tK6nAGo43q1Yad2flcfnIHwArGe7Ytne45UMxwfPsmEbfL9t9Ljvf83KpRKtcP71zhYwUSzqDC8u6QVGegpIMAaz8dMKhU7LB7D1fsOESX1sHGy6iqdMNHVvq602V8OqV+DaPSVaSmr9vHnV8s5dVpGzhcZAWex46fjNNleO2Kvp7tSlyGd+dspkGIO95FJS7qJ1v/2nM8DHi8NEhetv0gS7ZaFcYOFzn576/rPOvu/2a5z7FOO6Y1V741P+i5jMETPIeSkljx/uNo9TxHOq0oBJ9atMrZpdzuGdU94n3t4C+c6loJDmHskE7UTw78B+Xfe710/Bl8MvbkiNtU0yU3TYCfA/WqT7ltKHPuPq1K26IBc9UQkRtEJENECkXknWgdt8TlqpJ630qp6Mty306/+dPFnlld49k172eQPm5StZ7T5TK8OXMjb8/ayDuzNrJiR+A0jCK/idA+X7jN8/O63Tm8PHV90FSJ2Rv2kT5uEmNenkV+kZO8whIKS5wAHC4qYbZ7wN3Bw9b/d+a+PA4XWevtY87LzOIrv07I/03b4Pl5/Z4cpq/d63l+w0eL2Jp1mDOe+53u9//EEncvse3j+VsZ/fIsz/Pnfl1LMC9PXR90XSSSKxE8R6sUQETTgRJ6alEfkUwdGq4EkYATlYTD4RDw+8ZXUf5xXqPUyNIH7P2rKiG+IkonHSm7rmPLtLILo3XeIMsfPLcHAzrpzH9RsAN4BBiJVUorKoqdpkqqryilou8v/dvz0HfW5IbLtmXTpnHUPgqqhDGGNbtz6H5E6DkTgpm8Inhv7updh3C5oEfb0mO7XKbSJVinrd3Dv79f6bPMO14Rsa6zhSVO6rlnIzbGsGLHIc82pz9n5Qm3a1KPUzo3p/+jv/Hcxb0573hrbM93S6x5JxZvPcgPy3Zy2+dLOKZNI368aTD3fLWMbxbvYPodwykJMZPkW7M2hnwd/rnKM9fvY/B/pnqez1hX8YoY8zZmVXhfb5WZJTqssFtEpomICfKYSeTTga40xuwwxjiNMbMBe2rRQNtOMMb0M8b0a9kyShURIvzbfvS8Y+nmvkXl8ASs1r8P/aknl51UsaC+sr2k9t41FTq3bWJ9cNamnGH/3+hVA4+u8AenKmWM+coY8w1Q/r2wCGipOqVih/dg78pWi6oOH8zdzJnPzwhZSi3S463fY5XbPvP5GYx6YYZn3dasw3S85wcmLi7tjbUHqt362WLPILhip4uFm7M8aRAFxU6y8orIyiviho/+YM+hwoDn3rL/ME9PXuN5npVXxNTVe1i185CnV9jfvI376f+oVZHilk+X8K+PF5E+bpLP3Wp7MrFVOw9xqKCY9Xut13fgcBF3fLEkot9PJF6KUu9xZVQmbSOsnmdjzLBQ6905z4ki0sUYYyeqhDutKPhOLVrlIn3PX3ZSBy47ycpbfuy8Xjz2w2rPh8hfT0kH4MN5W6LZxLDUdIrCXWd1o2+HppwSoGc3Bj5XVRWI9E5RsVMHDCoVi0J9xhtjyM4vpkn95OAbeVm4OYuWaanlzuzr7XBRCYkOR8hb78u3W72xm/fnRXQHcm9OYZlJ0pwuw33fLKdhSiLLHiotQfrOrI20bpTquR5/v3Qno/u087kj/NUf2xnQsTmjerXh6Z/X8PasTYDVo/y3t+czNzOLsUM68v3SnWw9EDgd5pbPFrNwc+mEJLd8upgl7qoWj54XeF6Jj+dv9Xlu9zh7e/Db0jDtuPE/08c998XD368k3seFVubaE5UuH2NMHmBPK9pARAZiTSv6fqDtRWS0iDQVS3+sqUUnRqMt4ajMpfr8E44k474RNVI9IpiaytpISUzg7OPa+ATxxlOqrnp/P03dH9JHt6jZiWfqukjvFJU4XRFPJ6+UqnmhgucP5m2hz8O/kLk3N/hGwK7sAvblFnLBK3MY8pR1S//Jn1bzjldKwPUf/cG0NXvK7Nvjgcn0e+QXT0/vnpyCoOexL5H57h7atbtzfHqIvf20fCcnPvor871SA96YkenpKbYrNtjGf7eSaz/8w1Mi7peVu/l+6Y4y1SV+W7WHng9O9gTOAJ9nbGVupnWexVusHOCdAWomz1y3zzPIzrbEqxzcO17HDMcnC7YGXWeXe/MO1Gsje4K7cH3w95P47yV9onb+aE5/dh3wFta0ovtxTysKICKDgR+9Zki6xL1tCrANr6lFq0NN99jGszBm564SfY5qwjtXnaj5zTHGf8S4Uio2BOog+TxjK0c0TmXKKitXeOO+PI5onMq2A/meCals2w4cZtCTU8sc4xX3oLO/DTyaYqeLSUt3MmnpTn67bShHNa3v09N8qKCEmz5ZTLMGyVzx5nzeuLIfI3q05pP5W5i9Yb+nXCzAnA37+cvrc7mo35F8lmENrlu58xBXDkinnTsFsajExfdLdwIwy2uWujdmbPQMlgzGOw3hho8WMeU237KuPwWoKXzHF0s9P8/fZAXRe3LKpm1c/ua8kOdetyf0l5RIVKZ8W3Xq16FpmUGHwXxz/UD6HNWEKat9c9hP7ljxeCFqXT7BphV1r4toatGqVtsu1Rf1q9wEHbXpzkpNtmVYt1aVGgCgqp8OGFQqNt32+RLSx03C6TKeFIU7vljKFW/O93RQFTsNPR6YzBnPTSen4P/bu/PoqMo0j+PfN3vIAokJSwIkhF12EyCSAN20rG6M4DCACMehhR4YRu1zZhQdwWUUx2OPY5/T4wbdNopwZloQRzw6p+2maW3Q9ICC3YICAjJIGxdWUULu/HFvVd1KKkmFkNS9ld/nnHtI3aV47vImb731vs8bmsDi+MlzESvO7swSf9j/BWe/DfXl/cFjW1mxeQ8b3j1cb5B8IC3ZrWur2HHgC+58aTebXV0Uvj1/gd84rdeBijPAU1sPULHqTY6d+IZ1Ow4z55ntwcrzv/86lCrts5PnwjJJlD34P01enwmPbW1yHz9qrMW3sk8eu1dOavH/EU2dyJ06GGCmM9FZalIC//33lcFW5sQEE+yKUtozN2wCuZyM6LoVRXIpW559ozUanm8YURicSac59j80LerJPOryYgN64JeaB0OTFjDGJGH/vkgEEo0xaUCNZVk1jR/ZOKWqE/GnQEts7+VbqOyTx/MLRwe3vfmhXVF1d42Y8vg23rpzAgDzGmhJvfqJ0OQas5/ZXq8v74vvHOHFd47wuCsHMIRaq2stmPX09uD6DVV294SVr4Rnr6jLnW84GtWnG2+Fjmc/HNsLg2HJuv8NrntlaSU9ctMb7eNeWpQT7AqSn5XKu3dfxe0bdrFx51G+1z+fe6+5PPiB47LMVB6YPpgnfv0Rnzst8f+1+EpmPvmH4Pt1SAlvKMtISWT9reX0zO1AQad0BnTNYvuBL1k0LjR5dccOyey5bzK9l2/hjon9WnQd2mfluRWqdj+ZdXF9aVrylXWgn2/WRU7F3RryMu1BFkXNGPghvnAPsML1+ibgPmBlS95Uk6SI+N/vP64Oa+kNeG1PqKvC0a+/4Xf7PmfTrqPsOx5dN4O7N+6JuP7YiYb7N7d3762YxLD73ghb968zhvKPv3q/3r7ThnTlumEF/PnYKSfXcy1rtx8Kbv/byl4MLszm9g2hrBvXDC0AYMm60PsM6R6a/A3g3msuZ1SvXM5fqOVPx05y98Y9pCYlMHd0T17YcZhsZ9blwGR12z6qpiQ/k1llPdhQdQTLgnnlRcwrLwrm2S4rzuWTVVez+9MTPLPtADNKu7Nx51HunjaQNW8dZMmEPnTOCs2mnJSYwMM3DCGS/Q9Na/I6NsU7ta425MUW24uxeHxvcjJSmFHasm4fl1JFnzzWLChjbN9LlFZQPMGyrJW0sKIcyflaDRgU8ZP/XHwlN7paAAOWvbizyWMbmxWuvVr3w9EsWvtHTp1r3pd4KU5l9J2DXwZzPA/r0YmO6cnsfXAKb31czS2/sKemvm54QbDyvG7haPYdP8Xc8qLg794pg7sF39ddeb7tqr68trt+X22Agd2y+fOxkxG33VLZK/hzIBWeMXD/9YMxBuZfWQzY2cuWb9zN7FE9AFg0voQNVUe4fnhB8Pg1C8r4pPps8PWQ7h15YrY9m/LmpZUAjG5B3+WL1S7/asVL5TklKYF55UWeG3A1YUAXVYgkKjVKVSfiKzlRpp+LZ8XON6uBb30Xj+/d5DHf7x9qULrhisLgz8O6d2L3yslh+5Y4WaPumtrwLMiWZbHi2kG8umwsJfn2/i8vqQDsTFjuwXBpyYl8/C9T2bSkgjF98lhQ0avJv9G/+tEYstKSqWkgX91r/zC20eMDAjMsF3RMJzHB8OD0IcGp3eeM7snbd05gxbX2ZNQl+Zl8supqBnYLzc0wYUCXsMq4V7TLGk5bp1ETkchqatVtQ8RPcjqEJsW6blhBI3teWinNaJB58qYr2LH8Bxf9f91U3nCO+omXd+G2q+z+sr+4ZSTzyotYPL4kbJ9HZw4Ne/0fc6+gm5PRY2zfPH7y18N5eUkF868sqtd3F+CuaQNZPb+MhWNLWDS+hGdvLmPz0oqwfc67Zv976UdjeHVZZdj2upXjpMSE4MC5aJQW5QAwfUQB04dHvs/rFo7m6XmlTb7P47OGc9/1gyJuL+iU7svGNnXbEJGYqbmgAYMifuIeFFaY0zbTcz8yYwh/NaI7/e55rcl9B3bLDuuGAPDCwtHMfXYHeZkpTQ72e/vOCRR0SmfbR9Uc+uIsgwqy+bdZw9l5+CtmjQxVqsf1yyc3I4XSotx673FjWY+wNHRTh3Tjd8501JMHdQXsLhYNZa4oyc+gd76doOyuqQPDtg3r3pH3Pj0RrNyCfU/qDtYLfKPX3HzIdXVISeLxvxlBekoS++ukxBvTJy+q95g+orDpnXymfVaeYx2AiADw1dnzankW8ZHEBMPgwmz2HD3JmW9reHTm0LCKotvBh6fR664tF/X/LBpfwlNbDwD2YDF3fudOHZLpmp3Gku/3oTAnnRt+9nbE93h6XimFOekMKujIrnsnkpSYwM2rd3DqXA0Lx/bizQ//wusfhOf+LXBaiF9eUkH16e/o09muxNbNU103VVpgsNuo4tzg8VWHvqJrtj2ILZCJqqlpzf90/+RgV4e69j44hURj+L+vz5Gb2Xj3GWMMm5dWUHRZ8yYOe+zGYVQd+rLe+oYG37VX7bPyrKZnkZjb5SS437r3c7g2xsGISNR+vmAUc5/dzoIxxZx0DXRbPm0AD235MPja/bc2LzOV6tN22rH05ES+OR/K4ewWeI8ZV3TnxxP7c7D6TLAVdl55EWu3H2LXvaFcwufqvI87B/Qkp5UXQi3mL/1dqPvDweqzwHF+PLEfu458TYYrc1Wk1tzGPDJzKI+4umvUbVke1y+f9e8eYUhhx0iH8/pt48hKS2qw4gwE5zGIdhrzod2b3+o8o7S7p5IQeFW7/L5UVWeR2NtxwJ7Y4IBPZrQSEVt+Vipv3D6ekvzMYOV1VHEut44LDZx74/ZxYcdsWVZJRkoiGSmJ7LkvfIAcwIQBnQHo7Qwa69cli5SkBPp3DbX4PjB9MJ+sujrsuFSnRXpksd2N4boG+udGkue03nbtmMbqBSODWRxaw7Qh3Xh/5aR6ad0C+nfNCrZ6i/e105bnWEcgIg2N4hYR/wi0CgcyIvTtnMlHfzlNn/zMsP06Z6fxwf1T6h0/vEcnBnbLCk6GYTXz14Ixhqp7riI7LZkLtRZpydG3CS4YU0xeZmqbDXzMTktueifxhXZVeb6pvCfPbz+sbhsiHrBoXAmPvr6XhR5MQyQi0cnPSg1rDV5/azl7PztFgjNg7dVllZz8pn4O48syUqiptdjkpFc78uVZ0pITqewb3SA0t8DkXM2VlJgQl4PZpPWZunPEe1lZWZlVVVV10cfX1lqcr60N9hsS8TtjzB8tyyqLdRyRRFNev625QHJCQvAPrUg883J5hZb/jW2Omgu1WNRPqSbiJQ2V2XbV8pyQYEhNUMVZxCv0QVakfUpSpVl8TE+viIiIiEiUVHkWEREREYmSKs8iIiIiIlFS5VlEREREJEqqPIuIiIiIRMlXqeqMMZ8Dh5rYLQ+oboNwWovijx0/xl5kWVZ+rIOIROXVFxR/2/JseQWVWR/wc+zgz/gjlllfVZ6jYYyp8nIezaYo/tjxc+x+5fdrrvhjy+/x+5Hfr7mf4/dz7OD/+N3UbUNEREREJEqqPIuIiIiIRCkeK89PxzqAFlL8sePn2P3K79dc8ceW3+P3I79fcz/H7+fYwf/xB8Vdn2cRERERkdYSjy3PIiIiIiKtQpVnEREREZEoxU3l2RiTa4zZaIw5Y4w5ZIyZE+uYAowxqcaY1U5cp4wxO40xU51txcYYyxhz2rX8c51j1xhjThpjPjPG3BGjc/itMeacK8a9rm1znHM7Y4zZZIzJdW2L+X2pc21PG2MuGGN+6mzzxfWPN154LhqjMqsyK+G88Fw0ROVV5bXNWZYVFwvwIrAByAQqgRPAoFjH5cSWAawEirE/sFwDnHJeFwMWkNTAsQ8D24AcYCDwGTAlBufwW2BhhPWDnHMZ51z7dcB6r94X516cBsY5r31x/eNt8dpz0cBzojLrgXujMuuNxWvPRYRnROXVA/elvZTXmAdwCW/Wd0A/17q1wKpYx9ZIzO8DM6J4sI4Ck1yvH3AXnDaMt6GC/RCwzvW6t3Mvsrx4X4D5wAFCg2V9cf3jafHicxFl3CqzsbnuKrMxXrz4XEQRs8prbK57uyiv8dJtox9wwbKsfa5172F/YvMcY0wX7Jg/cK0+ZIz51Bjzc2NMnrNfDlCAfS4BsTyvh40x1caYt4wx33PWDcIVn2VZ+3EKM968L/OBX1pOKXXxw/WPF158LhqlMqsy28558blokMqrymtri5fKcyb2VxVuJ7A/mXmKMSYZeAF4zrKsD7HneR8JFAGl2DG/4Oye6fzrPrdYndc/ASVAIXauxleMMb1p/Np76r4YY3oC44HnXKv9cv3jiaeei6aozKrMireei8aovKq8toWkWAdwiZwGsuusy8buJ+QZxpgE7K9UvgOWAliWdRqocnY5boxZChwzxmRjnxfY53LO9XObn5dlWTtcL58zxswGptH4ta9tZFss3Az83rKsg4EVfrn+ccYX5RVUZutsiwWVWW/wRZlVeQ3bFgvtprzGS8vzPiDJGNPXtW4Y4V/ZxJQxxgCrgS7ADMuyzjewa+CrDmNZ1lfAMexzCfDKeVmAwY4lGJ8xpgRIxb4nXrsvNxP+iTgSv1x/P/PacxGRymyQyqx47bmoR+U1SOW1LcS60/WlWoD12KNOM4AKPDQS2InvSWA7kFln/WigP/YHmcuwR83+xrV9FbAVeyTqAOwHrU1HogKdgMlAGva3FXOBM07cg4CTwFjn2j9P+EhgT9wXYIwTc5bfrn88Ll55LpqIUWVWZVaLx56LRuJTeVV5bbvzjXUAl/DG5QKbnJt3GJgT65hcsRVhf9o6h/01RWCZC8wGDjpxHwN+CXR1HZsKrHEKz3HgjhjEnw+8i/1VytfOL6iJru1znGt+BngZyPXafQGeAtZGWO/56x+Pi1eei0biU5lVmdUSft098Vw0EJvKq8prmy6BVCIiIiIiItKEeOnzLCIiIiLS6lR5FhERERGJkirPIiIiIiJRUuVZRERERCRKqjyLiIiIiERJlWcRERERkSip8iwiIiIiEiVVnkVEREREoqTKs4iIiIhIlP4fgTPMolXoa2wAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "learn.activation_stats.plot_layer_stats(0)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Generally our model should have a consistent, or at least smooth, mean and standard deviation of layer activations during training. Activations near zero are particularly problematic, because it means we have computation in the model that's doing nothing at all (since multiplying by zero gives zero). When you have some zeros in one layer, they will therefore generally carry over to the next layer... which will then create more zeros. Here's the penultimate layer of our network:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs8AAADWCAYAAAAuNG/NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de5gcVZ3/8fdnLplJMgmQMAQFQwQTlShhddSVGBfvsKsPuKy/VbKAuhgXFsH1yrqiiK6iq+yqD4Kw3OSqKyCweEEFlODvJwY1YBQCAlHuE0JCZshtZr6/P6o6KZqemZ5MzXR39ef1PPVk+tSp6m/3zEl/+9Q5pxQRmJmZmZnZ6FpqHYCZmZmZWaNw8mxmZmZmViUnz2ZmZmZmVXLybGZmZmZWJSfPZmZmZmZVcvJsZmZmZlYlJ89mZk1O0sGSQtLetY7FzKzeOXk2MysgSXunCfHBtY7FrB5IeqekByStl3SBpPbMvlZJv5T097WM0RqDk2czMzMrNEmzgQuAfwOWAAcByzJVPgQ8HBHfrkF4VZM0pYjP1WicPDcJSTdLOk/S5yQ9nn7z/ndJLZI+JekxSb2S/j1zTJukUyXdL2mzpFWS3l923pMk/VZSn6RHJV0h6TmZ/aXLwW+S9HNJT0v6vaS3TObrNysqSa+RdKukjem2Mm1ff06r3JS2wQcyx3xA0oNpe/wRMLcWsZtNon2BDRFxaUTcCXwP2B9A0nzgJOC4ak6Ufi7eK+kwSXdJ6pd0k6T9yuq9XNIN6edjr6SrJO2T2f/8tOzhtC3eKemosnOUPrs/K+kR4KFhYro5befl27szdT6QxrtZ0j2S/k1SW2b/A2mO8A1JTwC3puXPST/b10valD5XTzXvVVE5eW4ufwe0A68h+Zb9CeB/gS6Sb+IfAT4h6dC0/n8Dfwu8H3gxcBrwRUn/WHbejwAvBd5O8iF8RYXn/jLweWARsAL4tqRdc3tlZk1IUitwLfBL4GXpdirwdPozwBHAc4BXpMccBvwncAZwIPAd4D8mM26zGrgXmCapR9J04K+A30gScB7wiYh4dAznew5Jsr2UpBd7V+D80k5J+wM/A/4v0AO8HhgEfiypM63WBfwUOITkM/Qc4AJJryt7rv8DdANvSM9Tyd+mMZW2zwH9wK/SeE4l+az+V5LP85NIPts/XXaeE4HHgVcDx6Tvz/eAFwFvBV4JPJa+jt1HfIeKLCK8NcEG3Az8tqxsFXBnWdlKkkT3+cAQ8KKy/Z8qP0/Z/r8AAtgrfXxw+vhvM3X2TMveUuv3xZu3Rt6A3dK2dHCFfXtX2gcsBy4tK/tyWnfvWr8mb94magPeln7G3UfyBbIV+ABwffq59L1037eArhHOcyowAHRnyt6ZfmZ2po8vBK4oO66D5Ivt4SOc+xrg3Mzjm4HVQMsYXudbgC3A29LH09LnPaSs3tHA+szjB4CfltV5Q/p/w/5lr+MR4FO1/p3WatveXW9NYWXZ40fTrbxsD5JvygJWJF88t2sj+fYMJMMySL7J7k/yzbt0NWMfnnl56belHyLiUUmDwJydfB1mBkTEk5L+G/iRpBtJerqujoi7Rzhsf+DysrLlwIcnKEyzuhAR1wHXlR5LmgecDLwK+BrwB5IrtJcApwAfH+F0D0dEb+bxQySfmXsAfyK50vMCSX1lx3UC89Pnn0bSIfU2kt7iKSSJ6U1lx9weEUPVvEZJC0muJn08fb0AC4GpwJWSIlO9FeiU1J15LbeVnXIh8ERE/L5UEBFbJP0y3deUnDw3l21lj2OYshZ2JMEHkXxjLa+DpLnA94GLSYZ0rCXp7foJyX8CWVsrxONhQ2bjFBHvk/RV4M3Am4DPSjqBpDdt2MMmJTiz+nYucGpEPCjpjcBnImJA0iXAZ0Y5tvwzrdSmWjL/XgycXuHYJ9J//wM4jOSL610kwyy+AuxSVr9/tBcCIGkPkqGYl0TEf2V2lWJ6B0kvdrl1ozxXpf8vNEx5U3DybMO5Pf13bkT87zB1XkHybfaDEbEJkgkSkxGcme0QEb8DfgecIelsklUErk53t5ZV/z2wGPhGpmzxhAdpVkckvQ9QRJybFrWQzAmCpPNnvJ07K4ADgD9GOtahgteSDKH6dhpTC7CAZEzxmEjqIGnzd5GMW85aBWwG9o2I74/x1KuA3SXtX+p9Tp/rlTzz/5Cm4uTZKoqIeyWdD5wr6WMkkx6mAy8nGef1ReAekm+eH5Z0KclkwE/VKmazZiPpBcD7SC5F/xl4Lsnk31+TXAnqA94saRWwJSKeJOnZ+h9Jt5FcOXoNcFSF05sVkqS9SCbKvSZT/HPgQ+mKU/9MMgRqPD5PMgTikvTKUC8wDzgc+GpE3AfcDRwm6UqStvohkjY85uQZ+GZ67HuA7sxwyw0R0Sfp88Dn0/Ifk+R/LwX+IiJGGp5yY/o6LpP0z8AGkiEtncBZOxFnIfiyuY1kGcmkin8j6a36KXAMyYQKIuIOkskW70/3fwT4YE0iNWtO/STjJ68guRx7JfAL4IR0jOQ/k8zU/zPwG4CIuJrkMvHHgDtIVgsY6cPTrGi+CZweEQ9kyk4kSW5XkLSr0YZtjCgi/kAy7LEL+BHJZ+S5JFdr16fV/gVYQzLG+ack46a/u5NPeTBJ/HeTTOYrbX+fxvPZ9PmOJZn/tDx9/MAoryNIEv67SIaC/YpkcuWbImLtTsba8DT81QQzMzMzM8tyz7OZmZmZWZWcPJuZmZmZVcnJs5mZmZlZlZw8m5mZmZlVycmzmZmZmVmVGmqd59133z3mzZtX6zDM6sbtt9++NiK6ax1HJW6vZs9Uz+0V3GbNyg3XZhsqeZ43bx4rVqyodRhmdUPSmlrHMBy3V7Nnquf2Cm6zZuWGa7O5DtuQNEvS1ZL6Ja2RdOQw9STpi5KeSLcvKXM7HDMzMzOzepR3z/OZwFZgDnAgcL2klRGxqqzeMpI71iwiub3zj0nuWnd2zvGYmZmZmeUmt55nSdOBI4BTIqIvIpYD1wJHVah+DPCViHgwIh4CvgK8O69YzMzMikTSCZJWSNoi6cJR6v6LpEclbZB0vqSOSQrTrCnkOWxjATAYEaszZSuBhRXqLkz3jVYPScvS/zBW9Pb25hasmZlZA3kY+Bxw/kiVJL0FOBl4AzAP2Bf4zEQHZ9ZM8hy20QVsKCvbAMyoou4GoEuSIiKyFSPiHOAcgJ6enmfsK3fuz+9jy8AgJ7x+/lhjNyscSSeQXNF5KXB5RLx7mHrHACcC84GngMuAT0TEQLr/ZuAvgYH0kIci4oXjjW9gcIhv3PxHjl3yfKZNaai5y2aTLiKuApDUA+w9QtVjgPNKwyUlfRa4lCShzs3ZP/sjf3y8L89Tmk2qQ1+6J69/0ZydOjbPT6w+YGZZ2UxgYxV1ZwJ95YnzWP38nl76tgw4eTZLlHqq3gJMHaHeNOCDwC+BbpLhVh8BTs/UOSEi/jvP4K76zUOc8ePVPLVpG5986/55ntqsmS0Ersk8XgnMkTQ7Ip4oryxpGck8JObOnVvVE2wbHOL0H9zFjM42ZnT4i681ppfuvctOH5vnX/1qoE3S/Ii4Jy1bBJRPFiQtWwTcNkq9MRtf+m1WHNX2VEXEWZmHD0m6FHjdBIfHloEhADZtG5zopzJrJpWu7EJyFfhZyfNYru6WbE7b7ElvmM+xS/YdT6xmDSm3Mc8R0Q9cBZwmabqkxcBhwMUVqn8L+JCkvSQ9F/gwcOF4Y5CEc2ezcXstz/4y+wVJayXdKung4Q7cmTkKbrNmuap0ZRcqXwXeKZu3JV98O9p8k2JrTnn/5R9Pcnn4ceBy4LiIWCVpiaTs4KhvAtcBdwK/A65Py8ZF4K5ns3GQ9B6gB/hypvjjJJOO9iLpobpO0n6Vjo+IcyKiJyJ6urvr9kZqZkVWurJbsgh4rNKQjZ1V6nnuaG/N65RmDSXXwUoRsY5k/eby8ltILiWVHgfwsXTLjeReLLOdJelwknHOb4yItaXyiPhlptpFkt4F/DXw9VyeN4+TmBWcpDaSz+xWoFVSJzBQmtib8S3gwnT41SPAJ8nhym7JY09t5q5Hk07sTifP1qQKNdLfH8JmO0fSIcC5wN9ExJ2jVA9ybG7+wmtWlU8Cn848/gfgM5LOB34P7B8Rf4qIH0r6EnATyZXgK8uO22nrn97K4tNvZGAoabW7TG3P47RmDadQyTN41IZZSbU9VZJeT7KU1dsj4rayfbsCrwJ+RrJU3d+TjIn+4MS/AjMriYhTgVOH2d2VfRARZwBn5B3Duv6tDAwF7z5oHn/1wm4W7zc776cwawiFGu2fTBh09myW+iSwiWR9139If/6kpLmS+iSV1qU6BdgF+H5a3ifpB+m+dpLl7nqBtcAHgMMj4u68gvQVI7PGUJoo+Krnz+J1L9yDttZCpRBmVStUz7Nwz7NZSbU9VREx7LJ0EdELvCLXwMqfYyJPbma52TKQTBT0WGdrdoX62ig5eTYzM5sI25eoay9U6mA2ZgVrAV7n2azReNiGWWPY7J5nM6BgyXPS8+z02ayRuMWaNYaBwaS1trcUKnUwG7NCtQD3YJmZmU2MobRzyrmzNbtCNQGPeTZrPP7Sa9YYSld2W+RWa82tWMkzXqrOrNG4xZo1hvTeKE6erekVK3l2ezYzM5sQ24dt+LPWmlyhkmfwsA2zRuPPYbPGUOp5lnuqrMkVKnmWfAnYrNG4zZo1hnDPsxlQtOQZeak6MzOzCTDkCYNmQMGSZ9zzbNZw/DFs1hiGkhsMOnm2pleo5Fng7NmswbjJmjWGUs+zc2drdsVKnuXbc5uZmU2E0qjIFg96tiZXrOQZ357bzMxsInipOrNEsZJnj3k2axj+/DVrLL5JilmiWMkzXufZrFG4qZo1Fo95NksUK3mWb89tZmY2EcJL1ZkBRUueax2AmVXN7dWssXjYhlmiUMkzeNiGWaNwUzVrLJ4waJYoVvIsJ89mZmYTodTzLF83siZXqOTZDdpsB0knSFohaYukC0ep+y+SHpW0QdL5kjoy++ZJuknS05LukvTGXOLL4yRmNmlKY55VqMzBbOwK1QS2DAzy0PpNXuvZLPEw8Dng/JEqSXoLcDLwBmAesC/wmUyVy4HfALOBfwO+K6l7vMG5lZo1lvCYZzOgYMnz/97xCADL711b40jMai8iroqI7wFPjFL1GOC8iFgVEU8CnwXeDSBpAfAy4NMRsSkirgTuBI6YuMjNrBJJsyRdLalf0hpJRw5Tb1dJF0l6PN1OzeP5PebZLNFW6wAmwpNPb6t1CGaNZCFwTebxSmCOpNnpvvsiYmPZ/oXjfVJ//pqN2ZnAVmAOcCBwvaSVEbGqrN5/AtNIriTtAfxU0pqIuGA8T+7VNswShep5LhkYHKp1CGaNpAvYkHlc+nlGhX2l/TMqnUjSsnSc9Yre3t4Rn9TDNsyqJ2k6yRWfUyKiLyKWA9cCR1Wo/jbgSxHxdEQ8AJwHvHe8MfgmKWaJgibP/lg2G4M+YGbmcennjRX2lfZvpIKIOCcieiKip7t73MOizWyHBcBgRKzOlI10FUhlP79kvAH4JilmiUImz1vd82w2FquARZnHi4DHIuKJdN++kmaU7S+/TDxm/vg1G5OxXAX6IXCypBmSXkDS6zyt0knHcrXIwzbMEoVMnj1swwwktUnqBFqBVkmdkirNc/gW8I+S9pe0G/BJ4EKAtJfrt8Cn0+PfDhwAXDne+Hx9yGxMxnIV6ERgE3APyXyGy4EHK510LFeLPGHQLFHM5HnIH8tmJEnwJpJl6P4h/fmTkuZK6pM0FyAifgh8CbgJWJNun86c551AD/AkcDrwdxExcheVmeVtNdAmaX6mrOJVoIhYFxFLI2LPiFhI8ll/23gD2H6TFPc8W5PLJXmudvmctO7r0hsubJD0QB7PX87JsxlExKkRobLt1Ij4U0R0RcSfMnXPiIg5ETEzIt4TEVsy+x6IiIMjYmpEvDAifpJHfP74NateRPQDVwGnSZouaTFwGHBxeV1J+0maLalV0qHAMpI138cbg3udzciv5zm7fM5S4CxJw01i6Ce5acNHc3ruZ/GwDTMzK6DjganA4yRDMY6LiFWSlkjqy9R7Ocl67BuBLwBLKyxnN2ZDER7vbEYO6zxnls95SUT0AcsllZbPObm8fkTcBtyW1y1+K9nm1TbMzKxgImIdcHiF8ltIJhSWHn8H+E7ezz8UnixoBvn0PI91+ZwJt809z2ZmZrkaivAaz2bkkzyP6SYKYzWWZXRKPObZzMwsX+GeZzOgiuRZ0s2SYphtOWO8icJY7cxNF9zzbGZmlq+hIU8YNIMqxjxHxMEj7U/HPLdJmh8R96TFudxEYWf5DoNmZmb58phns8S4h22MZfkcAEkt6Y0b2pOH6pQ0ZbxxZA0MuefZzMwsTx7zbJbIa6m6isvnAFRYQue1JDdr+D4wN/35hpziALzahpmZWZ42bxvkwl88wFObB2odilnNjXupOhh++Zx0X/kSOjczwfdH8JhnMzOz/GzZ5s9Vs5Ji3p7bPc9mZmZmNgGKmTx7zLOZmVl+PNbZbLtiJs/ueTYzMzOzCVDI5HmrxzybmZnlxqtsmO1QyOR5y4CTZzMzMzPLn5NnMzMzG5E7ns12KFTyPKMjWXlvq5NnMzMzM5sAhUqerz9xCS2CrQODtQ7FzMysMORBz2bbFSp5njt7GocduJcnDJqZmZnZhChU8gwwpbXFwzbMzMxy5H5nsx2Klzy3OXk2MzMzs4nh5NmswCTNknS1pH5JayQdOUy9H0jqy2xbJd2Z2f+ApE2Z/TfkFWP4nkZmdc9Dns12aKt1AHmb0tbiMc9mO5wJbAXmAAcC10taGRGrspUi4tDsY0k3AzeWnettEfGTCYzVzMys7hWv57m1hW2DwdCQu7OsuUmaDhwBnBIRfRGxHLgWOGqU4+YBS4CLJzrG5Pkm41nMbDzkUc9m2xUveW5LXpJ7n81YAAxGxOpM2Upg4SjHHQ3cEhH3l5VfKqlX0g2SFuUVpIdtmJlZIylc8tzh5NmspAvYUFa2AZgxynFHAxeWlS0F5gH7ADcBP5K0a/mBkpZJWiFpRW9v787EbGZ1yFeIzHYoXPK8vefZkwbN+oCZZWUzgY3DHSDpNcCewHez5RFxa0RsioinI+ILwHqSoR2U1TsnInoioqe7u7uqIP2hbGZmjaR4yXOrk2ez1GqgTdL8TNkiYNUw9QGOAa6KiL5Rzh3ktPSrh22YmVkjKV7y7J5nMwAioh+4CjhN0nRJi4HDGGYioKSpwDsoG7Ihaa6kxZKmSOqU9FFgd+DW8cTnHmez6o1h2ckOSWdLekzSOknXSdprsuM1K7LiJs8e82wGcDwwFXgcuBw4LiJWSVoiqbx3+XCSMdE3lZXPAM4CngQeAg4BDo2IJ8YTmHuczcYku+zkUuAsSZUm/54EvBo4AHguyRCrr4/3yf1l12yHwq3z3NHWCrjn2QwgItaRJMXl5beQTCjMll1OkmCX111F8kFsZjWQWXbyJemQquWSSstOnlxW/fnAjyLisfTYK4AzJjNes6IrbM/zFifPZnXNPVlmVRvLspPnAYslPVfSNJJe6h+MNwCv82y2Q+F6nj1h0KwxeNiGWdXGsuzkauBPJEOsBoE7gROGO7GkZcAygLlz5+YRq1nhFbbn2WOezcysIMay7ORZQCcwG5hOMml42J7napeX9JUisx0Klzx3eLUNs4bgD2Ozqo1l2clFwIURsS4itpBMFnylpN0nIU6zplC45NlL1Zk1Bg/bMKvOGJed/BVwtKRdJLWTrLjzcESsHU8M/q5rtkPxkufSmOfBwRpHYmZmlptql538CLAZuAfoBf4aePtkB2tWZMWbMOieZ7OG4GEbZtWrdtnJdP31pZMYmlnTKV7Ps5NnMzOzXMnfds22K2zy7HWezczMzCxvxUueW71UnZmZWZ7c72y2Q3GTZ/c8m5mZmVnOCpc8t7SI9lY5eTYzM8uJhzyb7ZBL8ixplqSrJfVLWiPpyBHqflTS7yRtlHS/pI/mEUPWlNYWJ89mZmZmlru8lqo7E9gKzAEOBK6XtDIiKt39SMDRwB3AfsANkv4cEVfkFAtT2lo85tnMzCwnXm3DbIdx9zxLmg4cAZwSEX0RsRy4FjiqUv2I+FJE/DoiBiLibuAaYPF448ia0uaeZzMzMzPLXx7DNhYAgxGxOlO2Elg42oFKvsouASr1UO80J89mZmZmNhHySJ67gA1lZRuAGVUce2oawwXDVZC0TNIKSSt6e3urCmhKawtbPGzDzMzMzHI2avIs6WZJMcy2HOgDZpYdNhPYOMp5TyAZ+/w3EbFluHoRcU5E9ERET3d39+ivCOhoa2XLNifPZmZmZpavUScMRsTBI+1Pxzy3SZofEfekxYsYYSiGpPcCJwOvjYgHqw+3OlOntLJ522DepzUzMzOzJjfuYRsR0Q9cBZwmabqkxcBhwMWV6ktaCnweeFNE3Dfe569kansrm5w8m5mZmVnO8rpJyvHAVOBx4HLguNIydZKWSOrL1P0cMBv4laS+dDs7pziApOf56a1Ons3MzMwsX7kkzxGxLiIOj4jpETE3Ii7L7LslIroyj58fEe0R0ZXZ/imPOEqmtnvYhhlUfwMjSadK2pb5Qtsnad/M/gMl3S7p6fTfAyfvVZiZmdWPwt2eG2DalFae3jpQ6zDM6kH2BkZLgbMkDbeM5LfLvtTeByBpCsl67JcAuwEXAdek5WZmZk2lkMlzZ3srmzxsw5rcWG9gNIKDSSYX/1dEbImIr5HcKfT1ecZrZmbWCAqZPE+b4gmDZoz9BkZvk7RO0ipJx2XKFwJ3RERkyu6odJ6dWZcdYvQqZmZmdaKQyfPU9la2DQbbfKMUa25juYHRd4AXA93A+4BPSXrXWM+zM+uym5mZNZJiJs9TWgHc+2zNruobGEXE7yPi4YgYjIhfAF8F/m6s59k5yuc0ZmZmk6DQyfNmj3u25raa9AZGmbIRb2CUEezIalcBB0jKZrkHVHmeKp/KzMysMRQyeZ6WJs9e69ma2VhuYCTpMEm7KfFK4ESSFTYAbgYGgRMldUg6IS2/ccJfhJmZWZ0pZPI8td3DNsxSFW9gVOHmRe8E7iUZivEt4IsRcRFARGwFDgeOBtYD7wUOT8tz4GEbZmbWONpqHcBEmDoleVnuebZmFxHrSBLf8vJbSCYClh6/q7xOWf3fAC/PPcDk7BNzWjMzswlQ6J5n32XQzMzMzPJUyOTZY57NGomHbZhVQ9IsSVdL6pe0RtKRw9T7gaS+zLZV0p2THa9ZURVy2EanxzybNRAP2zCr0pnAVmAOcCBwvaSVEfGMlW8i4tDsY0k34wm+ZrkpdM/zpq0DNY7EzMxs/CRNB44ATomIvohYDlwLHDXKcfOAJVRYZcfMdk4hk+ftq2142IZZA/CwDbMqLAAGI2J1pmwlsHCU444GbomI+ycsMrMmU8jkeVpHkjz3O3k2awAetmFWhS5gQ1nZBmDGKMcdDVw43E5JyyStkLSit7d3fBGaNYlCJs8dba1MaWvhqc3bah2KmZlZHvqAmWVlM0nWZq9I0muAPYHvDlcnIs6JiJ6I6Onu7s4lULOiK2TyDDCzs42+zR7zbFb/PGzDrAqrgTZJ8zNli4BVw9QHOAa4KiL6RqhjZmNU2OS5q6ONjU6ezRqAh22YjSYi+oGrgNMkTZe0GDiMYSYCSpoKvIMRhmyY2c4pbPI8o7OdjR62YWZmxXE8MBV4HLgcOC4iVklaIqm8d/lwkjHRN01yjGaFV8h1ngFmdLrn2awxeNiGWTUiYh1JUlxefgvJhMJs2eUkCbaZ5azAPc9Ons0ag4dtmJlZ4yhs8tzV0U7fFifPZmZmZpafwibPMzrbvFSdmZmZmeWqsMnzzM42+rYMMDTkS8Jm9Uge62xmZg2osMnzjM52IqB/q4dumNWj8FhnMzNrQIVNnrs6k4VEPGnQzMzMzPJS2OR5Rpo8e9KgWX3ysA0zM2tEhU2eZ3a2A7BhkycNmtUjD9swM7NGVNjkedb0KQCs699a40jMzMzMrCgKmzzP7kqS5yf6nDxb85I0S9LVkvolrZF05DD1Pirpd5I2Srpf0kfL9j8gaZOkvnS7YdyxediGmZk1oMLenntHz/OWGkdiVlNnAluBOcCBwPWSVkbEqrJ6Ao4G7gD2A26Q9OeIuCJT520R8ZO8AvOwDTMza0SF7XnuaGulq6ONJzxsw5qUpOnAEcApEdEXEcuBa4GjyutGxJci4tcRMRARdwPXAIsnN2IzM7P6V9jkGZLeZ495tia2ABiMiNWZspXAwpEOkiRgCVDeO32ppF5JN0haNMyxyyStkLSit7d3xOA8bMPMzBpRoZPn2V1Onq2pdQEbyso2ADNGOe5Ukv8bLsiULQXmAfsANwE/krRr+YERcU5E9ERET3d394hP4mEbZmbWiHJJnqudlJTW/aCk+yQ9JelhSf8paULGXs+ePoW1njBozasPmFlWNhPYONwBkk4gGfv8NxGxfcJARNwaEZsi4umI+AKwnqR32szMrKnk1fOcnZS0FDhL0nCXhq8DXhYRM4GXAIuAE3OK4xmSYRueMGhNazXQJml+pmwRzx6OAYCk9wInA2+IiAdHOXfA+MZdeNiGmZk1onEnz2OZlAQQEX+MiPWlw4Eh4AXjjaOS2V0drOvfytCQLw9b84mIfuAq4DRJ0yUtBg4DLi6vK2kp8HngTRFxX9m+uZIWS5oiqTNdxm534NZxxedhG2Zm1oDy6Hke86QkSUdKegpYS9IT9s0R6lY9Aancc3bpZNtgsLbPvc/WtI4HpgKPA5cDx0XEKklLJPVl6n0OmA38KrOW89npvhnAWcCTwEPAIcChEfHEpL0KMzOzOpHHWOMxT0qKiMuAy9LLyUcDj41Q9xzgHICenp4xdVXttetUAB5av4k9ZnaO5VCzQoiIdcDhFcpvIWm7pcfPH+Ecq4AD8o7NwzbMzKwRjdrzLOlmSTHMtpydmJRUEhH3kIy//MbOBD+avXbbkTybWX3xsA0zM2tEo/Y8R8TBI+1Pxzy3SZqfJsMwwqSkYWLYr8q6Y/LcUs/zk06ezczMzGz8xj3meSyTkgAkHStpj/Tn/WrTRS8AAAsMSURBVIF/BX463jgqmdnZzozONh52z7NZ3fGwDTMza0R5LVVXcVISQIWJSYuBOyX1A99Pt0/kFMez7LXrVA/bMDMzM7Nc5HJzkuEmJaX7yicmvSeP56zWvNnTWf3YqMOvzczMzMxGVejbcwMs2HMGDzzRz+Ztg7UOxczMzMwaXOGT5xftOYOhgHsf7xu9spmZWZ2SNEvS1ZL6Ja2RdOQIdV8m6efpmu2PSTppMmM1K7LCJ88L5iTLTd/9qIdumJlZQzsT2ArMAZYCZ0l61g3JJO0O/JDkBmSzSe7ie8MkxmlWaIVPnufNnkZHWwu/e7j8Pi5mZmaNIV0W9gjglIjoi4jlwLXAURWqfwj4UURcGhFbImJjRPxhMuM1K7LCJ89trS28bO5u3Hb/ulqHYmYZSleqGxzyzVLMqrAAGIyI1ZmylcCzep6BvwTWSfqFpMclXSdpbqWTSlomaYWkFb29vRMQtlnxFD55BnjVvrP4/SNPseHpbbUOxcxSLWnyvGGT26VZFbqA8kuoG4AZFeruDRwDnATMBe4nWUb2WSLinIjoiYie7u7uHMM1K66mSJ5fve9sImD5vWtrHYqZlXnSX2rNqtEHzCwrmwlUmtCzCbg6In4VEZuBzwAHSdplgmM0awpNkTz3zJtF94wOrvntQ7UOxczKPNm/tdYhmDWC1UCbpPmZskXAqgp17wCy46FKP/u2nmY5aIrkubVFHLboudx09+M8umFzrcMxMyDSj/MHnuhn68BQbYMxq3MR0Q9cBZwmabqkxcBhwMUVql8AvF3SgZLagVOA5RGxfvIiNiuupkieAY45aB4R8LUb76l1KGaWsW0w2P9TP+R/Vvy51qGY1bvjganA4yRjmI+LiFWSlkjafjODiLgR+ARwfVr3BcCwa0Kb2djkcnvuRvC8WdM46tX7cMGtD3Dwgm7evHDPWodkZsAr581iy+AQH7vyDmZNn8IbXjyn1iGZ1aWIWAccXqH8FpIJhdmys4CzJik0s6bSNMkzwMcPeRG3r3mS4y/9NR9843yOPmgeMzvbax2WWVMqDcL86rsOZNepU3jHN3/BsotvZ9HeuzC9o40I2DY4xMBQMDA4xLbBYGBoiIHBYNvQ0PZhH2b14vL3/SXPmzWt1mGY2QRrquS5s72VS459FSdfeQdfvmE1X7vxXg7YaxdesEcXs6ZPYddp7bS3ttDWIlpbWmhtgRYJKZ85FnnO1MgpJKtju0xtb4peWCGmTmnl4ve+inNvuY9f/+lJnto8QIugvaWFjrYWujraaG8VbS0ttLWK9tYWtwGrOx3tTTMS0qypNVXyDDCzs51vLH05K/+8nmtXPswdD67nJ394jCef3uabNVhdefFzZjZF8lyy2/QpfOyQF9U6DDMbQc8+u9U6BLOaa7rkuWTR83Zl0fN23f44IujfOshAepl4cCgYGAqGckqo87zEHDjJbwbtrcXuxXrrAc/hoP1ms3vXlFqHYmZVuOffD6XFl3zMmjd5LieJrg6/HVYskmYB5wFvBtYC/xoRl1WoJ+B04Ni06Dzg4xHJ1z5JB6ZlLwb+APxjRPx2PLHN6GxnhuccmDWMon+hN6uWW4JZsZ0JbAXmAEuBsyQtrFBvGcks/kXAAcBbgfcDSJoCXANcAuwGXARck5abmZk1FSfPZgUlaTpwBHBKRPRFxHLgWuCoCtWPAb4SEQ9GxEPAV4B3p/sOJrlK9V8RsSUivkYy//X1E/wSzMzM6o6TZ7PiWgAMRsTqTNlKoFLP88J0X6V6C4E7SkM4UncMcx4zM7NCc/JsVlxdwIaysg3AjCrqbgC60rHQVZ9H0jJJKySt6O3t3enAzczM6pWTZ7Pi6gNmlpXNBDZWUXcm0Jf2Nld9nog4JyJ6IqKnu7t7pwM3MzOrVw21vMTtt9++VtKaUartTrKqQKNy/LXTiLHvM8K+1UCbpPkRcU9atghYVaHuqnTfbRXqrQI+LEmZoRsHkExGHJbba0Nw/JNrpPZac26zda+RY4fGjL9im1UU7B63klZERE+t49hZjr92Gjn24Ui6guRO2McCBwLfBw6KiFVl9f4JOAl4Y1r/x8DXI+LsdFWNe4AzgLOB9wEfBeZHxNZxxtfQ77njr61Gj78RNfp73sjxN3Ls0PjxZ3nYhlmxHQ9MBR4HLgeOi4hVkpZI6svU+yZwHXAn8Dvg+rSMNEE+HDgaWA+8Fzh8vImzmZlZI2qoYRtmNjYRsY4k8S0vv4VkImDpcQAfS7dK5/kN8PIJCtPMzKxhFLHn+ZxaBzBOjr92Gjn2RtXo77njr61Gj78RNfp73sjxN3Ls0Pjxb1e4Mc9mZmZmZhOliD3PZmZmZmYTwsmzmZmZmVmVCpM8S5ol6WpJ/ZLWSDqy1jGVSOqQdF4a10ZJv5F0aLpvnqSQ1JfZTik79nxJT0l6VNKHavQabpa0ORPj3Zl9R6avrV/S9yTNyuyr+e+l7L3tkzQo6evpvoZ4/4umHv4uRuI26zZrz1QPfxfDcXt1e510EVGIjWQZrm+TrCDwGpLbBy+sdVxpbNOBU4F5JF9Y3kpyd7Z56RZA2zDHfgG4BdgNeDHwKHBIDV7DzcCxFcoXpq/ltel7fxlwRb3+XtLfRR/w2vRxQ7z/Rdvq7e9imL8Tt9k6+N24zdbHVm9/FxX+Rtxe6+D30iztteYB5PjL2gosyJRdDJxe69hGiPkO4Igq/rAeAt6cefzZbMOZxHiHa9ifBy7LPN4v/V3MqMffC3AMcB87Jss2xPtfpK0e/y6qjNtttjbvu9tsjbd6/LuoIma319q8703RXosybGMBMBgRqzNlK0m+sdUdSXNIYs7e5W2NpAclXSBp97TebsBzSV5LSS1f1xckrZV0q6SD07KFZOKLiD+SNmbq8/dyDPCtSFtpRiO8/0VRj38XI3KbdZttcvX4dzEst1e314lWlOS5i+RSRdYGkm9mdUVSO3ApcFFE3EVyn/dXkNw//eUkMV+aVi/dxCL72mr1uj4O7AvsRbJW43WS9mPk976ufi+S5gJ/BVyUKW6U979I6urvYjRus26zVl9/FyNxe3V7nQxFucNgHzCzrGwmyTihuiGpheSSylbgBICI6ANWpFUek3QC8IikmSSvC5LXsjnz86S/roj4ZebhRZLeBfw1I7/3QyPsq4WjgeURcX+poFHe/4JpiPYKbrNl+2rBbbY+NESbdXt9xr5aaJr2WpSe59VAm6T5mbJFPPOSTU1JEnAeMAc4IiK2DVO1dKlDEfEk8AjJaympl9cVgEhi2R6fpH2BDpLfSb39Xo7mmd+IK2mU97+R1dvfRUVus9u5zVq9/V08i9vrdm6vk6HWg67z2oArSGadTgcWU0czgdP4zgb+H9BVVv4q4IUkX2Rmk8yavSmz/3TgZyQzUV9E8oc2qTNRgV2BtwCdJFcrlgL9adwLgaeAJel7fwnPnAlcF78X4KA05hmN9v4XcauXv4tRYnSbdZv1Vmd/FyPE5/bq9jp5r7fWAeT4i5sFfC/95f0JOLLWMWVi24fk29ZmkssUpW0p8C7g/jTuR4BvAXtmju0Azk8bz2PAh2oQfzfwK5JLKevT/6DelNl/ZPqe9wPXALPq7fcCfBO4uEJ53b//Rdzq5e9ihPjcZt1mvT3zfa+Lv4thYnN7dXud1K20lIiZmZmZmY2iKGOezczMzMwmnJNnMzMzM7MqOXk2MzMzM6uSk2czMzMzsyo5eTYzMzMzq5KTZzMzMzOzKjl5NjMzMzOrkpNnMzMzM7MqOXk2MzMzM6vS/wfiI2mA+SNXPQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "learn.activation_stats.plot_layer_stats(-2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As expected, the problems get worse towards the end of the network, as the instability and zero activations compound over layers." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Increase batch size" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "One way to make training more stable is to *increase the batch size*. Larger batches have gradients that are more accurate, since they're calculated from more data. On the downside though, a larger batch size means fewer batches per epoch, which means less opportunities for your model to update weights. Let's see if a batch size of 512 helps:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "dls = get_dls(512)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
02.3093852.3027440.11350000:08
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "learn = fit()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's see what the penultimate layer looks like:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs8AAADWCAYAAAAuNG/NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxjZZn3/8+VpPalt6re9wWFRpuBVpRNRnTEhRco+uiACM4oDojLOOrMqDgt7j6j8/PxQQRHRBERR1kfEMdRUMAFGqWFBuyGhqYbet+rqmtJcv3+OCfVqXSqKtV1qpLU+b5fr/Oq5OTOyVXpujvXuXOd+zZ3R0REREREhpcodwAiIiIiItVCybOIiIiISImUPIuIiIiIlEjJs4iIiIhIiZQ8i4iIiIiUSMmziIiIiEiJlDyLiMScmZ1uZm5mc8sdi4hIpVPyLCIyAZnZ3DAhPr3csYhUAjN7h5k9a2Z7zey7ZlaT91jSzP5gZm8vZ4xSHZQ8i4iIyIRmZtOA7wKfBE4FTgIuzmvyEeAFd7+pDOGVzMxqJ+JrVRslzzFhZvea2XfM7HNmtj088/68mSXM7NNmts3MdpjZ5/OekzKzVWb2jJl1m9laM3tfwXE/ZGaPmFmHmW01sx+Z2ay8x3NfB7/WzH5jZl1m9riZvW48f3+RicrMTjGzB8zsQLitCfvXprDJPWEffDbvOR8ws81hf/w5ML8csYuMo8XAPne/wd0fBW4FjgEws2XAh4BLSjlQ+Ln4lJmdbWZPmlmnmd1jZksK2p1gZv8dfj7uMLObzWxB3uOLwn0vhH3xUTO7oOAYuc/uz5rZFuD5QWK6N+znhdtFeW0+EMbbbWbrzeyTZpbKe/zZMEf4ppntAh4I988KP9v3mtnB8LVWlvJeTVRKnuPlrUANcArBWfYngP8HNBOciX8U+ISZvT5s/5/AW4D3AUcDVwBfNrO/LzjuR4GXAG8m+BD+UZHX/nfgC8AKYDVwk5lNjuw3E4khM0sCtwN/AI4Pt1VAV3gb4FxgFvCy8DlnA/8BfA04Dvgx8L/HM26RMngKaDSzlWbWBLwK+JOZGfAd4BPuvnUEx5tFkGyfTzCKPRm4NvegmR0D/Br4HbASeDWQAX5hZvVhs2bgl8CZBJ+h1wDfNbO/Lnit/wW0A2eExynmLWFMue1zQCfwUBjPKoLP6n8l+Dz/EMFn+78VHOeDwHbglcCF4ftzK/Bi4E3Ay4Ft4e/RNuQ7NJG5u7YYbMC9wCMF+9YCjxbsW0OQ6C4CssCLCx7/dOFxCh7/K8CBOeH908P7b8lrMzPc97pyvy/atFXzBkwJ+9LpRR6bW+wx4H7ghoJ9/x62nVvu30mbtrHagLPCz7gNBCeQSeADwJ3h59Kt4WPfB5qHOM4qIA205+17R/iZWR/evw74UcHz6ghObM8Z4ti3Ad/Ou38vsA5IjOD3fB3QA5wV3m8MX/fMgnbvAvbm3X8W+GVBmzPC/xuOKfg9tgCfLve/abm2/uF6iYU1Bfe3hlvhvukEZ8oGrA5OPPulCM6egaAsg+BM9hiCM+/ctxkLGPj10iO5G+6+1cwywIwj/D1EBHD3PWb2n8DPzexXBCNdt7j7X4Z42jHAjQX77gf+aYzCFKkI7n4HcEfuvpktBP4FOBH4P8ATBN/Q/gC4HPjnIQ73grvvyLv/PMFn5nTgOYJvepaaWUfB8+qBZeHrNxIMSJ1FMFpcS5CY3lPwnIfdPVvK72hmywm+Tfrn8PcFWA40AD81M89rngTqzaw973d5sOCQy4Fd7v54boe795jZH8LHYknJc7z0Fdz3QfYlOJQEn0RwxlrYBjObD9wFXE9Q0rGTYLTrfwj+E8jXWyQelQ2JjJK7v9fMvg78DfBa4LNmdhnBaNqgTxuX4EQq27eBVe6+2cxeA3zG3dNm9gPgM8M8t/AzLdenEnk/rwe+VOS5u8Kf/xs4m+DE9UmCMouvApMK2ncO94sAmNl0glLMH7j7/5f3UC6mtxGMYhfaPcxrFfv/wgbZHwtKnmUwD4c/57v7/xukzcsIzmY/7O4HIbhAYjyCE5FD3P0x4DHga2b2LYJZBG4JH04WNH8cOBn4Zt6+k8c8SJEKYmbvBczdvx3uShBcEwTB4M9oB3dWAy8Fnvaw1qGI0whKqG4KY0oARxHUFI+ImdUR9PknCeqW860FuoHF7n7XCA+9Fmgzs2Nyo8/ha72cgf+HxIqSZynK3Z8ys2uBb5vZxwkuemgCTiCo8/oysJ7gzPOfzOwGgosBP12umEXixsyWAu8l+Cp6EzCb4OLfPxJ8E9QB/I2ZrQV63H0PwcjWf5nZgwTfHJ0CXFDk8CITkpnNIbhQ7pS83b8BPhLOOPV+ghKo0fgCQQnED8JvhnYAC4FzgK+7+wbgL8DZZvZTgr76EYI+POLkGbg6fO67gfa8cst97t5hZl8AvhDu/wVB/vcS4K/cfajylF+Fv8cPzez9wD6CkpZ64KojiHNC0NfmMpSLCS6q+CTBaNUvgQsJLqjA3f9McLHF+8LHPwp8uCyRisRTJ0H95I8Ivo79KfBb4LKwRvL9BFfqbwL+BODutxB8Tfxx4M8EswUM9eEpMtFcDXzJ3Z/N2/dBguR2NUG/Gq5sY0ju/gRB2WMz8HOCz8hvE3xbuzds9o/ARoIa518S1E3/5Ahf8nSC+P9CcDFfbnt7GM9nw9d7D8H1T/eH958d5vdwgoT/SYJSsIcILq58rbvvPMJYq54N/m2CiIiIiIjk08iziIiIiEiJlDyLiIiIiJRIybOIiIiISImUPIuIiIiIlEjJs4iIiIhIiapqnue2tjZfuHBhucMQqRgPP/zwTndvL3ccxai/igxUyf0V1GdFCg3WZ6sqeV64cCGrV68udxgiFcPMNpY7hsGov4oMVMn9FdRnRQoN1mdVtiEiIiIiUiIlzyIiIiIiJYo0eTazqWZ2i5l1mtlGMztvkHZmZl82s13h9hXLW4hdREREDjGzy8xstZn1mNl1w7T9RzPbamb7zOxaM6sbpzBFYiHqkecrgV5gBnA+cJWZLS/S7mKCtdJXAC8F3gS8L+JYREREJooXgM8B1w7VyMxeB/wLcAawEFgMfGasgxOJk8guGDSzJuBc4Fh37wDuN7PbgQsIOnK+C4Gvuvvm8LlfBd4LfGs0MXz7NxvoSWe47NXLRnMYkVgws8uAi4CXADe6+0WDtLsI+A5wMG/3m9z93ijiuOvRLcxoreeEBVOiOJzIhOTuNwOY2Upg7hBNLwS+4+5rw/afBW7g8M/hMffbp3Zy12NbSJjxwTOW0dZ8ZAPg2/Z3c9W9T9PZk444Qomz179kJq9+8Ywjem6Us20cBWTcfV3evjXAq4q0XR4+lt+u2Aj1iDzw9E52d/YqeRYpTW4k63VAwzBtf+fup4xFEF+5+0leOneykmeRaCwHbsu7vwaYYWbT3H1XYWMzu5jg22Dmz58fWRDP7eri77+3moRBZ2+G6S11/Z/NG3d1sn5bB2ccPZ3BKjY7etLc9sjzHOhO85/3bWB/d5q2ptrI4hN5ydxJR/zcKJPnZmBfwb59QEsJbfcBzWZm7u75DUfSsRtrk2zekxlh2CLxNIKRrDGVcSeT9eEbikgpin2+QvBZfFjy7O7XANcArFy5MpKOmM06H/3JGlIJ478/chqX3vBH7l67lctevYy7Ht3Cx3/yZzp60pxz3Gz+18p5UJA/H+zN8Pm7nmDDjk4Alk1v5sb3voJlM4qlEyLjL8rkuQNoLdjXChwooW0r0FGYOMPIOnZDTYqDvUqeRcbAX5nZTmA3cD3wRXeP5DvUbBayh3d9ETkyxT5fofhn8Zj41ZPbefCZ3XzpLS9h1qQGXn/sTL5w15Pc/MfNfOTHazhu3mROXjqNb977NLc+8kLRY7Q113HDe07k2DmTaKlLkUhoTgGpHFEmz+uAlJktc/f14b4VwNoibdeGjz04TLsRaaxN0tWrmiiRiP0GOBbYSPCV8E1AGvhiYcMj+Qo4k9XIs0iEcp+vPw7vrwC2FSvZGCt3PbqFSQ01nHtC8IXWmctn8YW7nuSj/7WGOZMbuOE9J9JUl+KtJ8xj2/7uosc4emYrkxprxitkkRGJLHl2904zuxm4wszeAxwHnA2cVKT594GPmNldgAP/BHxjtDEEybNGnkWi5O4b8u4+amZXAB+jSPJ8JF8BZ9018iwyDDNLEXxmJ4GkmdUD6SLfAH0fuM7MbgC2AJ8CrhuvOHvTWX7xxDZet3wmNclgQq/50xo5ZlYrj2/Zz+fffCxNdUHqsaitiUVtTeMVmkhkop6q7lKCC4+2AzcCl7j7WjM71cw68tpdDdwBPAo8BtwZ7huVhtokPemsRrFExpZzWJXikcuq5lmkFJ8imPHmX4B3hrc/ZWbzzazDzOYDuPvdwFeAewi+LdoI/Nt4BfnA0zs50J3m9cfOHLD/42e+iE++4WhOf9H08QpFZMxEWbaBu+8mmL+5cP99BBcx5O478PFwi0xjbRKAg30Zmusi/dVEJpxSR7LM7PXAH919m5m9GLgc+K+o4shknYxyZ5EhufsqYNUgDzfn33H3rwFfG+OQirr70a0016U4ZVnbgP2nv2i6EmeZMCbU8tyNtUHCrLpnkZKUNJJFsNjCn82sE7gLuBn4QlRBZB2KXCssIlVm/bYD3PKn53njS2ZRl0qWOxyRMTOhhmf7R55V9ywyrFJHstz9o8BHxyqOrC4YFKl6mazzsZ/8maa6JB8780XlDkdkTE2wkecgedZFgyLVQ/M8i1S//3liG49s2svlbzrmiFcSFKkWEyp5bugv21DyLFItNNuGSPV7answJ8CZBRcKikxEEyp5PjTyrJpnkWoRLJJS7ihEZDQ27e6irbm2/9ojkYlsQiXPDTUq2xCpNirbEKl+m/Z0MXdKY7nDEBkXEyp51gWDItVHZRsi1W/T7oPMm6rkWeJhgiXPqnkWqSbujjsaeRapYpms88Leg8yb0lDuUETGxYRKnhtU8yxSVXJJs5Jnkeq1Zd9B0lnXyLPExoRKnlW2IVJdcjmzqjZEqtem3QcBmKeaZ4mJCZU81yQT1CSNrj4lzyLVIFfrnFH2LFK1Nu3pAmDeVJVtSDxMqOQZghk3NPIsUh1y5RpZlW2IVK3Nu7tIGMyerORZ4mHCJc+NtSnVPItUCY08i1S/TXsOMmtSAzXJCZdSiBQ14f7SG2uTmm1DpEpks+FPJc8iVWvT7i7maqYNiZEJlzw31KpsQ6Ra5Eacc0m0iFSf53Z3MV8zbUiMTLjkWSPPItWjv2xDNc8iVelAdx/bD/SwqL2p3KGIjJsJmDyn6OrL8Ksnt3H1r58udzgiMoTchYKqeRapTht2dAKwpL25zJGIjJ8JmDwnOdib5vrfbeTLdz/Jtv3d5Q5JRAZxqGxDybNINdqwswOAJRp5lhiZcMlzQ22Szp4Mz+3uIutwy5+eL3dIIjKIXM6sCwZFqtPT2ztJJoz5U5U8S3xMuOQ5qHlOs3lPsOLRTx7ejOuDWaQiZbU8t0hV27Czg/lTG6lNTbh0QmRQE+6vvbE2xZ6uPnrSWZbPbuWp7R08+MzucoclIkX0L5Ki3FlkWGY21cxuMbNOM9toZucN0m6ymX3PzLaH26qxiunp7Z0q2ZDYiSR5LrVDh23/2szuMbN9ZvZsFK+fr6Em2X/7H161hBmtdfz991bzqye3Rf1SIjJKmm1DZESuBHqBGcD5wFVmtrxIu/8AGoGFwMuBC8zs3VEHk8k6z+zqZLEuFpSYiWrkudQODdAJXAt8LKLXHqCx9lDyfMzsVm59/8nMm9rIB298RB/QIhUmlzyr5llkaGbWBJwLXO7uHe5+P3A7cEGR5mcBX3H3Lnd/FvgO8HdRx/T8noP0prMsbtPIs8TLqJPnEXZo3P1Bd78e2DDa1y4mlzybwZzJDcya1MBFJy2goyfN5j1dY/GSInKEMlphUKRURwEZd1+Xt28NMNhAlRXcPrZoI7OLzWy1ma3esWPHiAJ6OjfTxnSNPEu8RDHyPNIOPSIj7dgNtSkAZrbWUx+WcCyb0QLAum0dUYQkIhFR2YZIyZqBfQX79gEtRdreDfyLmbWY2VKCUeeiSwC6+zXuvtLdV7a3t48ooAfW7wQ0x7PETxTJ80g69IiNtGPnRp7nTTn0/8Sy8Kx4/fYDUYQkIhHJv2BQs+KIDKkDaC3Y1woU+2D7IHAQWA/cBtwIbI4ymGd2dvK93z3LW0+Yy9Sm2igPLVLxhk2ezexeM/NBtvsZWYcecw255HnqoeS5pb6GWZPqWa+RZ5GKkl+uocFnkSGtA1Jmtixv3wpgbWFDd9/t7ue7+0x3X07wWf9glMF8/s4nqEsl+fiZL4rysCJVITVcA3c/fajHw5rnlJktc/f14e6iHXo8NNbkkueGAfuXzWjRyLNIhclPmLPuJAeUaYpIjrt3mtnNwBVm9h7gOOBs4KTCtma2BNgbbn8DXAy8Ksp47v3Ldi545QKmt9RHeViRqjDqsg137wRyHbrJzE4m6NDXF2tvZgkzqwdqgrtWb2aRfefTVBecD8yfOrC866jpzTy1vUPLAItUkPxaZ9U9iwzrUqAB2E5QinGJu681s1PNLP+r1ROARwm+Af4icL67Rzag5e6ks05LfU1UhxSpKsOOPJfoUoLp57YDuwg7NICZnQr8zN1zVxScBtyT99yDwK+B06MI5JhZraw66xjOPHbmgP3LZjTT3Zdl054uFkzTtDoilWBg2YaSZ5GhuPtu4Jwi++8juP4od//HwI/HKo7ceW4qoW+KJJ4iSZ4H69DhY4Wd+l4Yu+9mEwnjopMXHbY/N+PG+m0dSp5FKkRWI88iVSfXV5NKniWmJtzy3INZGs64sU51zyIVI6MLBkWqTi55TpiSZ4mn2CTPrfU1tLfUsXGnFkoRqRTZbP5tZc8i1SB30quyDYmr2CTPAAunNfLMrs5yhyEiofw654xqnkWqQiYTjjwreZaYilXyvGBaExuVPItUjAFlGxp5FqkKGnmWuItV8rxwWiPb9vfQ1ZsudygiQsEFgxp5FqkK6bDeSiPPElfxSp7bglk2ntutumeRSjBwkZTyxSEipctdq6CRZ4mreCXP4RR1z+5U6YZIJcifnk5lGyLVIfctUVKzbUhMxSp5nj8tWHXw2V0aeRapBAMuGFTyLFIVchcMap5niatYJc+t9TVMa6rVRYMiFUKzbYhUn/6RZyXPElOxSp4BFkxr5BmVbYhUhPzRZlfyLFIVMmHRs5JniavYJc8L25rYqLINkYowsGyjjIGISMlyfVXJs8RV/JLnaU1s2ddNd1+m3KGIlJWZXWZmq82sx8yuG6btP5rZVjPbZ2bXmlldFDHkrzCommeR6tA/VZ0uGJSYil3yPHdKAwDP7z1Y5khEyu4F4HPAtUM1MrPXAf8CnAEsBBYDn4kigAGLpKhsQ6QqaKo6ibsYJs/BjBub9yh5lnhz95vd/VZg1zBNLwS+4+5r3X0P8FngoihiGLBIikaeRSpWNuvsOBAsMpZWzbPEXAyT53DkWcmzSKmWA2vy7q8BZpjZtNEeeOAiKUqeRSrV/u4+Xvb5/+Gmhzb191UlzxJXsUueZ7TWk0oYm/fookGREjUD+/Lu5263FDY0s4vDOurVO3bsGPbAKtsQqQ65RDmTddKa51liLnbJczJhzJpcr7INkdJ1AK1593O3DxQ2dPdr3H2lu69sb28f9sADyzZGGaWIjJlUIkgX0lnXPM8Se7FLngHmTm7UBYMipVsLrMi7vwLY5u7D1UoPSysMilSH/JHnXF9V8ixxFc/keUqDyjYk9swsZWb1QBJImlm9maWKNP0+8PdmdoyZTQE+BVwXRQxaJEWkOuRm1ujLZJU8S+zFMnmeM6WBbft76ElrrmeJtU8BBwmmoXtnePtTZjbfzDrMbD6Au98NfAW4B9gYbv8WRQBanlukNGY21cxuMbNOM9toZucN0q7OzL5lZtvMbLeZ3WFmc0b7+omEkbDghLf/gkHN8ywxFUnyXGqnDtt+zMweM7MDZvaMmX0sihhGIjdd3Za93eP90iIVw91XubsVbKvc/Tl3b3b35/Lafs3dZ7h7q7u/2917ooghv1JDZRsiQ7oS6AVmAOcDV5nZ8iLtPgS8EngpMBvYC3wjigBSiQRpXTAoEtnIc6mdGsCAdwFTgDOBy8zsHRHFUZLcdHW6aFCkvPITZs22IVKcmTUB5wKXu3uHu98P3A5cUKT5IuDn7r7N3buBHxFMNzlqyYQNHHlW8iwxNerkeYSdGnf/irv/0d3T7v4X4Dbg5NHGMRKHkmfVPYuUk2bbECnJUUDG3dfl7VtD8aT4O8DJZjbbzBoJBrR+FkUQqYSRzjhp1TxLzEUx8jySTj2AmRlwKsHV/ONmZms9yYRp5FmkzLRIikhJCudaJ7x/2FzrwDrgOeB5YD9wNHDFYAceydzsyaSRyeqCQZEokueRdOpCq8IYvjtYg5EuulCKVDLBzNZ6XtB0dSJlNWCRFNU8iwymcK51wvuHzbUOXAXUA9OAJuBmhhh5Hsnc7KmEBfM8Z3XBoMTbsMmzmd1rZj7Idj8j69T5x72MoPb5jUNdfDTSRRdKNXtyveZ6FimzAWUbGnkWGcw6IGVmy/L2raD4t7YrgOvcfXf42foN4OVm1jbaIJJh2YZGniXuhk2e3f30Ilfk57ZTGFmnBsDM/o5geqwz3H3zaH+JIzF7cgMv7FPyLFJOWiRFZHju3kkwgnyFmTWZ2cnA2cD1RZo/BLzLzCaZWQ1wKfCCu+8cbRy52TaUPEvcjbpsY4SdGjM7H/gC8Fp33zDa1z9Ssyc3sHVft74qFimj/NFmDTyLDOlSoAHYDtwIXOLua83sVDPryGv3UaAbWA/sAN4AvDmKAFK5muews6aUPEtMFVtN7EhcClxL0Kl3EXZqADM7FfiZuzeHbT9HUIv1kB2ql/qBu/9DRLGUZPbkBvoyzs6OHqa31o/nS4tIaOBsG8qeRQbj7ruBc4rsv4/g2qPc/V0EM2xELllQ85xQ8iwxFUnyPFinDh8r7NiLonjN0ZozOUiYn997UMmzSJkMWCRFQ88iFS0VzvOcS5418ixxFcvluSEYeQZ4QasMipTNgEVSNPIsUtGSBTXPGnmWuIpt8jxrUi551kWDIuUy4IJBjTyLVDSNPIsEYps8t9anaK5Labo6kTLKT5418CxS2ZIJoy9z6ILBhOZ5lpiKbfJsZsyeXM8WTVcnUjaZ7KHprlS2IVLZ+keeM5qqTuIttskzhHM9q+ZZpGyyWacmGXwAa7YNkcqWSoazbbhWGJR4U/Kssg2Rssm6U5NI9N8WkcqVSiT6a57NdMGgxFesk+c5kxvY1dlLd1+m3KGIxFLGnVQ48qzkWaSy5c/zrIsFJc5inzwDbNrdVeZIROIpKNsI/hvKZMscjIgMKah5zpLJui4WlFiLdfK8dHqwdstT2zuGaSkiYyHr9CfPGnkWqWzJhJHOaORZJNbJ85L2ZsxgvZJnkbLIuC4YFKkWuQsG01lXvbPEWqyT54baJHOnNCh5FimTbNZJaeRZpCokwwsGs66RZ4m3WCfPAMumt7B+24FyhyESS/kfwprnWaSypRJGOpslnXXN8SyxFvvkeen0Zjbs7CStq5VExl0mGyxYlEyYlucWqXCphJHJOFklzxJzSp6nN9ObzrJpj+Z7FhlvWXeSiWCxBZ2/ilS2/kVSsq4FUiTWYp88Lwtn3FDphsj4y3rwIZxIqOZZpNIlc8tz64JBibnYJ8+56ep00aDI+AtWKjOSZqp5FqlwqUSif3luXTAocRb75LmlvoZZk+o117NIGQRlG0bCVPMsUumCeZ6zmqpOYi/2yTPA4vYmNuxQ8iwy3rJZwrINjTyLVLpUuDx3VoukSMwpeQYWtwUzbrhGvkTGVcYdMzTbhkgVyNU8p7U8t8SckmdgUVsTB7rT7OrsLXcoIrGSm/IqYYYGnkWGZmZTzewWM+s0s41mdt4g7X5mZh15W6+ZPTra108lE/2zbaSSSp4lvlLlDqASLGpvAuCZnZ20NdeVORqR+DhU86xFUkRKcCXQC8wAjgPuNLM17r42v5G7vz7/vpndC/xqtC+eK9Xoy2Q1VZ3EWiQjz6WeDYdtP2xmG8xsv5m9YGb/YWZlTeKXtAUzbqjuWWR8ZTxvkRQlzyKDMrMm4FzgcnfvcPf7gduBC4Z53kLgVOD60caQWxilJ53VIikSa1GVbeSfDZ8PXGVmywdpewdwvLu3AscCK4APRhTHEZkzpYGapLFhZ2c5wxCJnWzWSRqabUNkeEcBGXdfl7dvDTDYZ23Ou4D73P2Z0QaQUvIsAkSQPI/0bNjdn3b3vbmnA1lg6WjjGI1kwlgwrYlndih5FhlPubKNpGbbEBlOM7CvYN8+oGWY570LuG6wB83sYjNbbWard+zYMeSB+kee+zJKniXWohh5HvHZsJmdZ2b7gZ0EI89XD9G25I49GovbmnhGI88i4yq3SErC0AWDIkPrAFoL9rUCgy6Pa2anADOBnwzWxt2vcfeV7r6yvb19yAByI8+9GnmWmIsieR7x2bC7/zAs2zgK+BawbYi2JXfs0VjU3sTGXV2quxQZR4eW51bZhsgw1gEpM1uWt28FsHaQ9gAXAje7eyQX9CSTQcrQk85qqjqJtWGTZzO718x8kO1+juBsOMfd1xN0/G8eSfBRWtzWRG8my+Y9XeUORSQ2sh58FazluUWG5u6dwM3AFWbWZGYnA2czyIWAZtYAvI0hSjZGqiY38pzJapEUibVhk2d3P93dbZDtFI7sbDhfClgy8tCjtXz2JAD++NyeMkciEh/ZbN4iKUqeRYZzKdAAbAduBC5x97VmdqqZFY4un0PwLfA9Ub24ap5FAqMu2ziCs+H3mNn08PYxwL8CvxxtHKN1zKxWpjbV8pt1O8sdikhsZFyLpIiUyt13u/s57t7k7vPd/Yfh/vvcvbmg7Y3uvsAjXDo3tzBKb0Y1zxJvUU1VV/RsGKDIGfHJwKNm1gncFW6fiCiOI5ZIGKcsbeO+9Tv09bHIODlU8xzcFpHKlUwcqnlW8ixxFsniJO6+m+AromKP3UdwUWHu/rujeM2xcNpR7SHQsxQAABkOSURBVNy+5gWe2Lq/v4xDRMZONhsukmIq2xCpdLk6Z/dDibRIHOmvP89py9oAVLohMk4yWSeZCL750cizSGXLH21OauBZYkzJc57prfW8eGYL960fu/mkRSqFmU01s1vMrNPMNprZeYO0W2VmfWbWkbctjiKG/kVSNPIsUvHyZ9jQyLPEmf76C5y0pI2HN+6hJ50pdygiY+1KoBeYAZwPXGVmgy1udJO7N+dtG6IIIOu5RVI08ixS6QaMPCt7kBjTn3+BVyyeSk86y5pNheu+iEwcZtYEnAtc7u4d7n4/cDtwwXjGkcnmXTCYHc9XFpGRqsnLmDXyLHGmv/4CL180FTP4/YZd5Q5FZCwdBWTcfV3evjXAYCPPZ5nZbjNba2aXDHZQM7vYzFab2eodO4Yvf+pfJEUrDIpUPI08iwT0519gcmMtR89sVfIsE10zwQIK+fYBLUXa/hg4GmgH3gt82sz+tthB3f0ad1/p7ivb29uHDSK3SEpCNc8iFS+/5jmlkWeJMf31F/GKxdNU9ywTXQfQWrCvFThQ2NDdH3f3F9w94+6/Bb4OvDWKIDK5eZ7NiHAtBxEZA/kjzwnTdBsSX0qei1Dds8TAOiBlZsvy9q0A1pbwXAci+eTsn21DZRsiFS9/tFllGxJn+vMv4oQFUwD403N7yhyJyNhw907gZuAKM2sys5OBs4HrC9ua2dlmNsUCLwc+CNwWRRy5RVKCso0ojigiYyWpqepEACXPRU1rrmPe1AYe2bS33KGIjKVLgQZgO3AjcIm7rzWzU82sI6/dO4CnCEo6vg982d2/F0UAGQ8WSUkmgvpnEalcqaQuGBSBiJbnnoiOmzeF1c/uLncYImPG3XcD5xTZfx/BBYW5+0UvDoxCNqx5VtmGSOXTIikiAf31D+K4eZPZsq+bbfu7yx2KyITk7rgHZRumRVJEKt6AmmddMCgxpuR5EMfNmwyg0g2RMZKbmi63PLfKNkQqWzKvbCO/hEMkbpQ8D2L57FZSCVPyLDJGcrmyZtsQqQ4pTVUnAih5HlR9TZKjZ7XyyHNKnkXGQq5MI7dIipbnFqlsyQGLpCh5lvhS8jyE4+dPZs3mvaQ1h5ZI5PrLNsxIGKp5FqlwA0aelTxLjCl5HsIJC6fS1ZvhiS2HLbomIqOUS5b7yzZU8yxS0TTyLBJQ8jyEly0MFkt5SFPWiUQuV6ZhZiQSmm1DpNLV5E3urJFniTMlz0OYNamBOZMbeHijVhoUiVruAsGkBaUbGnkWqWwDVhjUBYMSY0qeh7Fy4RQeenY3rlExkUjlRpoTKtsQKYmZTTWzW8ys08w2mtl5Q7Q93sx+Y2YdZrbNzD402tfPT5hVtiFxFlnyPJJOnfecWjN70sw2RxVH1FYunMr2Az1s3nOw3KGITCi5eZ0TZpiBzk9FhnUl0AvMAM4HrjKz5YWNzKwNuBu4GpgGLAX+e7QvnkgEF/fmbovEVZQjzyV16gIfA7ZHGEPkVi4I6p4ffEZ1zyJRyvjARVI0z7PI4MysCTgXuNzdO9z9fuB24IIizT8C/Nzdb3D3Hnc/4O5PRBFHbpVBjTxLnEWSPI+wU+eeswh4J/DFKGIYKy+a0cKUxhp++/SucociMqHkqjQShso2RIZ3FJBx93V5+9YAxQapXgHsNrPfmtl2M7vDzOYXO6iZXWxmq81s9Y4dO4YNIlf3rJFnibOoRp5H0qlzvgF8AqjoeohEwnjlkmn87umdqnsWiVB+2YZm2xAZVjOwr2DfPqClSNu5wIXAh4D5wDPAjcUO6u7XuPtKd1/Z3t4+bBC5EWeNPEucRZU8j6RTY2ZvBlLufstwBx7pWfFYeOWSNl7Y182zu7rK8voiE1H/IimJ3CIpZQ5IpLJ1AK0F+1qBYgsRHARucfeH3L0b+AxwkplNGm0QyWQ48qzZNiTGSkqezexeM/NBtvsZQacOSzy+AnyglNce6VnxWDh5yTQAfvv0zrK8vshE1D/bhpmmqhMZ3jogZWbL8vatANYWaftnIL9D5W6POuNVzbNIicmzu5/u7jbIdgoj69TLgIXAfWa2FbgZmGVmW81s4Wh+mbGyqK2Jma31/PYp1T2LRCV/qrpc/WRWCbRIUe7eSfB5eYWZNZnZycDZwPVFmn8XeLOZHWdmNcDlwP3uvne0ceSS5qSSZ4mxSMo2RtipHwPmAceF23uAbeHtTVHEEzUz46Sl0/jdhl36cBeJSCZcYTAZjjwDmnFDZGiXAg0Es1TdCFzi7mvN7FQz68g1cvdfEVxTdGfYdikw7PSxpUgqeRYhFeGxLgWuJeiouwg7NYCZnQr8zN2b3T0NbM09ycx2A1l331rkmBXjlYuncfMfn2f99g5eNLNoKbeIjMChso1DV+5nsk5Ncmxe72Bvhv9+fCu3/ul5Ht+yn3TGeesJc3nHy+ezqK1pbF5UJELuvhs4p8j++wiuPcrfdxVwVdQxpJJKnkUiS54H69ThY4d17LzH7iW4MriivWJxUPf8+w27lDyLRCBX4xwsvBB8EI/FwPP6bQf40s+e5L6ndtKbzjJ3SgOnLG2no6eP/7z/Ga7+zQZWzJvMG46dyWuOmcHitiZMF0OJFNU/VZ36iMRYlCPPE9q8qY3MmdzA7zfs4sKTFpY7HJGqlxt5TpqRDAvIoi7buOEPG/nM7Y/TWJfk/BPn89pjZvCKRdP6R7q37e/mtkee5/Y1L/DFnz3JF3/2JHOnNHDSkmmcuGgaxy+YwsJpjUqmRUL9U9Ul1SckvpQ8j8ArFk/jnr9sJ5t1TRAvMkr9i6QkDo1iRTnjxs/XbuVTtz7Gacva+fe3raC9pe6wNjNa67n4tCVcfNoSNu/p4p6/7OC+dTv4+dpt/Hj1ZgAmNdTwkjmTOHpWC0fNCLbF7U201NdEFqtItUiGs21o5FniTMnzCLxyyTR++sfNqnsWiUAmb5GUZMSzbdy/ficf/tEjrJg7masvOIH6Egqp505p5IJXLOCCVywgm3We2tHBwxv3sGbTXta+sJ/v/W4jvelsf/u25joWTGtk3pQG5kxpYM7kRmZOqmNGaz3TW+qZ2lSrulCZcGqSWiRFRMnzCLxi8VQAHnhqp5JnkVHqL9vIq3mOYpXBmx56jk/c8hhL25v59rtWlpQ4F0okrH+U+W9fHqxqnM5k2bi7i6e2d/D0jg427uxi4+5OHnp2D3f8ectho+YJg6lNtUxrqmNyYw1TGmuZ3FjDpIYaWhtqaKlP0VKformuhqa6JE21KZrqkjTUpmisSdJQm6QulVDJiFQUzbYhouR5ROZOaWTZ9GbuXruVvztlUbnDEalqhctzw+hrnq974BlW3fE4px3VzpXn/VWkpRWpZIIl7c0saT/82udM1tm2v5ut+7vZvr+bbft72NnRw86OXnZ39rCns48NOzvY09XHvoN9A0awh1OXSlBfk6S+JkFdKkltKkFtMjHgZ03SqEkmqEkmSCWNVCLYl7udShipZPAzkTBSiWC0Pxnezo3+JxLWX4Oe25dMGBZOJ5ibGSVpRiIRTOOZyHvMcsex3GPhv68ZZoeOmf94fjvIXUDKgOck8toe2jewjTHwmDI2NM+ziJLnEXvDS2bxf361nu0HupneUl/ucESqVqZghUGAbOk55WGu/92zrLrjcV57zAyuPO94alORTGNfkmTCmD25gdmTG0pq392XYX93H509GTq60xzo6eNgb4bO3gwHe9Mc7M3Q1ZehuzdDdzpLd1+Gnr4svZksPekMveksPeksveksXb1pejNZ0hmnL5MlnXX60sHPdDbYl8k66YyTzmZjswx6sQTdKEi6E0HSHdwP9uXuF0vWD7tP+Jzw/jXvWsmcEv8GqpVGnkWUPI/YG186i6//cj0/f2wrF7xyYbnDEalauUQ5GN0Mbh/JyHM6k+Wm1Zu4/La1vObo6Xzz/OOpSY5f4nwkgpHkJJSh+svdyWSdTPgznXWy2UP7slnCn4f2uTtZD0bYM1nHPWwTPpbJhscNn+8cahfcZkDbrDtO8O2DEzwnty/3WrnXcA9fL+v9ZT3ZvDbZMNbguYTHCx8n+JkNHjjseR6+ZiZsD2HMeb/fgOeR97yC47l7LOqAc8tzK3mWOFPyPEJHzWhh6fRm7nx0i5JnkVHIXyTF+keeS0+e12zayy1/ep6fr93Kln3dvHLxNP7veZWfOJebWVjOUe5ApCr1jzyrNEZiTP9/HoE3hqUbz+3qYv60xnKHI1KV+ss2EnllGyWOPG/Zd5C3fet3JBJw8pI2/u2sYzjj6BlKnEXGmGqeRUCfNEfgvBPnk0oY1z7wTLlDEalauVHmZN5UdaXO83z1rzeQdecX//gqvnPRyzjz2FlKnEXGgWqeRZQ8H5EZrfWc9dLZ/Hj1JvZ19ZU7HJGq1L9ISt5sG6WMPO840MONDz7HW46fw7yp+uZHZDzlTlKVPEucKXk+Qu85dTFdvRlueHBjuUMRqUr9i6QkgrrnYN/Qz3F3vnjXE/Rlslxy+tIxjlBECmnkWUTJ8xE7ZnYrpy5r47sPPEtPOlPucESqTv4iKaXWPH/vt89y85+e50NnHMWitqYxj1FEBkrpgkERXTA4Gu87bQnv/M4fuPVPz/P2l80vdzgSsdyUXrm5coN5coO5cnNz6mayTl84f25fxknn9mWdTP++Q89JZ8P5d/Pm5E1nDu3Lv9/eUsf7/3rijq4emm3DaKwL/iv68I8e4ZNvPJrTjmo/rP1T2zv43J1P8Jqjp/OBV0/c90WkkvWPPCeVPEt8KXkehZOXTmP57Fau+c0G3nbCvP66TTnE3enNBIs59GU8/Bks9tDXvz9LbzpIJoPt0O10xsMFIIKEsjeTpS89MFnN7U/nFooIE9C+zMBEN11sfzZLJnxOXyZ3+1CiPJ4SRv8qcKmEcfSs1gmdPPeXbZhx6tI2vvq2FVx5z1O89/ur+eklJ3HsnEkD2n/+zsdpqEnypXNfqr4mUiappEaeRZQ8j4KZcenpS3n/D//Ijx7axHknVs/ocybrdPam6ehO09mTprM3Q1fuZ2+art4MXbnVzvoydPdlg5+9GbrTwWpnPbmVz8KVznrSwe3c/d50kCSPBTOoSeSWIh64NHFN8tByxLlli5MJo74mQaouFSxbnChsW2zfoccKXysZ/sztK3a8ZG6J5NxjiQTJ/vaH4sstnxy3hDBXoZFbFvrcE+byqhe1c9Y37ud91z/Mp954NCctaaO1IcVtj7zAPX/ZwSfe8GLamuvKG7hIjKnmWUTJ86i94SUzOXHRVL5895O8bvkMpo3zB3s6k2V3Vy+7OnrZ3Rlse7p62dvVx56uXvYd7GNfVx/7Dvaxv7uPA91pDnSn6ehJl/waCYOGmiQNtUnqUknqaxLU1ySpSyWoSyVpqU9Rl0pSV5OgNpmgNtxfm8rdDvbXJI3aVDL8mduXoCZ1KMmtSR5qm5/81uTdziWeUt0OjTwf2tfWXMdV7zyBd3/3QS654Y+YwfSWOrbt7+FFM1q48KSF5QlWRACtMCgCSp5Hzcz43DnH8vqv38fbr/k97zxxPq85ZgZzpxz5FFruzoGeNNv397D9QDc7DvT0395+oIcd4bazo4c9Q0yV11yXYlJDTf+2qK2J1voaWupraKlP0VKfoqku2JrrkjTVpmisTdFQm6SpLkljTYr62iCZNX1FJxHL5NU85ztu3mQe/ORreGTTXn739C6e2LKf045q5y3Hz6EulSxHqCIS0gWDIkqeI7FsRgv/97zjufKep1h1x+OsuuNx5kxu4OhZrcyf2sj01jpa6lP9SWg6E5RAdPVm2H8wGCHe3dnHrs4gId55oJeDfYfP4FFfk2B6Sz3tLXUsbm/ixMVTmdZUR1tzLdOa65jSWMvUpmCb3FijRSOkonnebBuFapIJXrZwKi9bOHW8wxKRISSThhmxKzMTyafkOSJnHjuTM4+dyVPbO/j1uh386bk9/GXrAR54amfRRDinNpVgSmMNUxpraWuuY8H8Rtqa65jeWsf0lnqmtwS321vqaa1PaQRYJoxcOXzhyLOIFGdmU4HvAH8D7AT+1d1/WKTdKuCTQE/e7pe6+4bRxjBncgOzJzWM9jAiVU3Jc8SWTm9m6fRmYBEQjK519Wbo6EnTm87iHlytnKshrq/R19AST2etmMXJS6fR1lxb7lBEqsWVQC8wAzgOuNPM1rj72iJtb3L3d0YdwDtPXMDbXzYv6sOKVJXIvtc3s6lmdouZdZrZRjM7b4i2q8ysz8w68rbFUcVSScyMproUM1rrmTe1kfnTGpk9uYEpTbVKnKWsSu2zFviyme0Kt69YBF+BtNTXsGBaky7+FCmBmTUB5wKXu3uHu98P3A5cMJ5xJBKmaw8k9qL81Mo/Iz4fuMrMlg/R/iZ3b87bRv11koiMSKl99mLgHGAF8FLgTcD7xitIEQHgKCDj7uvy9q0BBvucPcvMdpvZWjO7ZLCDmtnFZrbazFbv2LEjynhFJqxIkudKOSMWkdKMsM9eCHzV3Te7+/PAV4GLxi1YEQFoBvYV7NsHtBRp+2PgaKAdeC/waTP722IHdfdr3H2lu69sbz98ZU8ROVxUI88jPSMGnRWLlNNI+uzy8LHh2qm/ioydDqC1YF8rcKCwobs/7u4vuHvG3X8LfB146zjEKBILUSXPIzkjBp0Vi5TbSPpsYdt9QHOxumf1V5Exsw5ImdmyvH0rgGIXCxZyQNPaiESkpOTZzO41Mx9ku58RnBGDzopFKsBI+mxh21agw3MTNYvImHP3TuBm4AozazKzk4GzgesL25rZ2WY2JbzY9+XAB4HbxjdikYmrpKnq3P30oR4P6ydTZrbM3deHu0s9I4YSz4offvjhnWa2cZhmbQTzX0rp9J6NXKW8ZwuO8Hn9o1gl9Nm14WMPDtNuAPXXMaX3beQq4T070v6acylwLbAd2AVc4u5rzexU4Gfu3hy2e0fYrg7YDHzZ3b833MHVZ8eM3rORq5T3rGiftagGj8zsRwRJ8HsI5p+8Czip2PyTZnY28BtgL/Ay4BbgE6V07hLiWO3uK0d7nDjRezZyE+E9K7XPmtk/AB8CXhO2/wXwDXf/VgQxVP37WA5630ZO71k09D6OnN6zkav09yzKqeouBRoIzohvJDwjBjCzU82sI6/tO4CnCL4i/j4lnhWLSKSK9tki/fVq4A7gUeAx4M5wn4iISOxEtsKgu+8mmAu22GP3EVx0lLtf9OJAERk/g/XZIv3VgY+Hm4iISKxNxKW9ril3AFVI79nI6T2Lht7HI6P3beT0nkVD7+PI6T0buYp+zyKreRYRERERmegm4siziIiIiMiYUPIsIiIiIlKiCZM8m9lUM7vFzDrNbKOZnVfumCpRuOBNt5l1hNtf8h47L3zvOs3sVjObWs5Yy8HMLguXl+4xs+sKHjvDzJ40sy4zu8fMFuQ9Vmdm15rZfjPbamYfGffgq4j6a2nUX4enPjs+1GeHp/46vInSXydM8gxcCfQCM4DzgavMbHl5Q6pYl7l7c7i9CCB8r64GLiB4D7uAb5YxxnJ5AfgcwQID/cysjWB1r8uBqcBq4Ka8JquAZQQTqv818HEzO3Mc4q1W6q+lU38dmvrs+FCfLY3669AmRH+dEMmzBSscngtc7u4d7n4/cDvBH6qU5nzgDnf/jbt3EPwBv8XMWsoc17hy95vd/VaC1bvyvQVY6+7/5e7dBB15hZm9OHz8XcBn3X2Puz8BfBu4aJzCrirqr5FQfw2pz4499dlRU38NTZT+OiGSZ+AoIOPu6/L2rQF0VlzcF81sp5k9YGanh/uWE7xnALj70wSjDEeVIb5KVPj+dAJPA8vNbAowO/9x9Pc3FPXXkVF/PTLqs9FRny2d+uuRqar+GtkiKWXWDOwr2LcPiN1ZXQn+GXicoOO+A7jDzI5D7+FwmoEdBfty709z3v3Cx+Rw+lsrnfrrkVOfjY7+3kqj/nrkqqq/TpSR5w6gtWBfK8Hy35LH3f/g7gfcvSdcEv0B4A3oPRzOUO9PR979wsfkcPpbK5H666ioz0ZHf28lUH8dlarqrxMleV4HpMxsWd6+FcDaMsVTTRwwgvdqRW6nmS0G6gjeWzn8/WkClhDUaO0BtuQ/jv7+hqL+euTUX0unPhsd9dkjo/5auqrqrxMieQ5rY24GrjCzJjM7GTgbuL68kVUWM5tsZq8zs3ozS5nZ+cBpwM+BG4CzzOzU8I/2CuBmd4/VmXH4vtQDSSCZe6+AW4Bjzezc8PFPA3929yfDp34f+JSZTQkvcHgvcF0ZfoWKp/5aGvXX0qjPjj312eGpv5ZmwvRXd58QG8HUJrcCncBzwHnljqnSNqAdeIjgq469wO+B1+Y9fl743nUCtwFTyx1zGd6jVQSjBfnbqvCx1wBPAgeBe4GFec+rI5h6Zz+wDfhIuX+XSt7UX0t6j9RfS3uf1GfH531Wnx36/VF/Le19mhD91cKgRERERERkGBOibENEREREZDwoeRYRERERKZGSZxERERGREil5FhEREREpkZJnEREREZESKXkWERERESmRkmcRERERkRIpeRYRERERKZGSZxERERGREv3/iu+ht1/5Ym4AAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "learn.activation_stats.plot_layer_stats(-2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Again, we've got most of our activations near zero. Let's see what else we can do to improve training stability." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1cycle training" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Our initial weights are not well suited to the task we're trying to solve. Therefore, it is dangerous to begin training with a high learning rate: we may very well make the training diverge instantly, as we've seen above. We probably don't want to end training with a high learning rate either, so that we don't skip over a minimum. But we want to train at a high learning rate for the rest of training, because we'll be able to train more quickly. Therefore, we should change the learning rate during training, from low, to high, and then back to low again.\n", + "\n", + "Leslie Smith (yes, the same guy that invented the learning rate finder!) developed this idea in his article [Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://arxiv.org/abs/1708.07120) by designing a schedule for learning rate separated in two phases: one were the learning rate grows from the minimum value to the maximum value (*warm-up*), and then one where it decreases back to the minimum value (*annealing*). Smith called this combination of approaches *1cycle training*.\n", + "\n", + "1cycle training allows us to use a much higher maximum learning rate than other types of training, which gives two benefits:\n", + "\n", + "- By training with higher learning rates, we train faster, a phenomenon Leslie N. Smith named *super-convergence*\n", + "- By training with higher learning rates, we overfit less because we skip over the sharp local minimas to end-up in a smoother (and therefore more generalizable) part of the loss.\n", + "\n", + "The second point is an interesting and subtle idea; it is based on the observation that a model that generalises well is one whose loss would not change very much if you change the input by a small amount. If a model trains at a large learning rate for quite a while, and can find a good loss when doing so, it must have found an area that also generalises well, because it is jumping around a lot from batch to batch (that is basically the definition of a high learning rate). The problem is that, as we have discussed, just jumping to a high learning rate is more likely to result in diverging losses, rather than seeing your losses improve. So we don't just jump to a high learning rate. Instead, we start at a low learning rate, where our losses do not diverge, and we allow the optimiser to gradually find smoother and smoother areas of our parameters, by gradually going to higher and higher learning rates.\n", + "\n", + "Then, once we have found a nice smooth area for our parameters, we then want to find the very best part of that area, which means we have to bring out learning rates down again. This is why 1cycle training has a gradual learning rate warmup, and a gradual learning rate cooldown. Many researchers have found that in practice this approach leads to more accurate models, and trains more quickly. That is why it is the approach that is used by default for `fine_tune` in fastai.\n", + "\n", + "Later in this book we'll learn all about *momentum* in SGD. Briefly, momentum is a technique where the optimizer takes a step not only in the direction of the gradients, but also continues in the direction of previous steps. Leslie Smith introduced cyclical momentums in [A disciplined approach to neural network hyper-parameters: Part 1](https://arxiv.org/pdf/1803.09820.pdf). It suggests that the momentum varies in the opposite direction of the learning rate: when we are at high learning rate, we use less momentum, and we use more again in the annealing phase.\n", + "\n", + "We can use 1cycle training in fastai by calling `fit_one_cycle`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def fit(epochs=1, lr=0.06):\n", + " learn = Learner(dls, simple_cnn(), loss_func=F.cross_entropy,\n", + " metrics=accuracy, cbs=ActivationStats(with_hist=True))\n", + " learn.fit_one_cycle(epochs, lr)\n", + " return learn" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.2108380.0848270.97430000:08
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "learn = fit()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We're finally making some progress! It's giving us a reasonable accuracy now.\n", + "\n", + "We can view the learning rate and momentum throughout training by calling `plot_sched` on `learn.recorder`. `learn.recorder` (as the name suggests) records everything that happens during training, including losses, metrics, and hyperparameters such as learning rate and momentum:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuUAAAD7CAYAAADNeeo8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3yV9fn/8deVvXcIMwsSRoAECFtw7wEuVBSxWrFqqxattbb+3FqtqK1aFfcoqCiKew9AhRBG2DOEQBhZZO/k8/vjHPqNaZAEknOfcT0fj/Oo3PeHc96nmMPlfa77+ogxBqWUUkoppZR1vKwOoJRSSimllKfTolwppZRSSimLaVGulFJKKaWUxbQoV0oppZRSymJalCullFJKKWUxH6sDOEpMTIxJTEy0OoZSSnXaypUri40xsVbncCT9zFZKuaqj/cz2mKI8MTGR7Oxsq2MopVSnicguqzM4mn5mK6Vc1dF+Zmv7ilJKKaWUUhbTolwppZRSSimLaVGulFJKKaWUxbQoV0oppZRSymJalCullFJKKWUxhxXlIhIlIu+LSLWI7BKR6YdZJyLyiIiU2B+Pioi0Ou8tIg+IyF4RqRSR1SIS4aj3oZRSSimlVFdz5EjEZ4AGIA7IAD4RkRxjzIY262YBU4F0wABfAbnAc/bz9wITgPFAPpAG1HV7eqWUUkoppbqJQ4pyEQkGLgSGGmOqgKUi8iEwA7ijzfKZwBxjzB77750DXAs8JyKRwC1AujHm0AzI9Y54D57OGMPm/ZV8u7mQ+sZmAPx9vQkP9CUyyI9eEQH0jQgkNtSfVl9sKKWUQzS3GP7+2SZGxEcyOjGK2FB/qyMppVxMWU0D+aU1FFbUU1rdQGlNA1dPTMLPxzGNJY66Up4KNBtjtrY6lgMc387aNPu51uvS7P88DGgCLhKRPwIVwD+NMc+096IiMgvblXfi4+OP6Q14so/X7uXJr7exvbAKABEwpv21of4+pMSFMKR3GCPjIxmVEEl8VJAW6kqpbrW7tIY3lu3ihSU7ARiXHMU/LkqnX1SQxcmUUs6ooKyWFTtLWbO7jI37Ktiyv5Ly2sb/WTcloze9wgMdkslRRXkIUN7mWDkQ2oG15UCIva+8LxCOrchPAlKAb0RkqzHmq7ZPZIyZC8wFyMzMPEwZqQ6nqbmFR7/YwtzFuaT1DuOBqUM5c2hPokNsV6DqGpspr22ktLqBfeW17DlYy/bCKjbvr+SD1Xt5c1k+AH0jA5mcGsvJg3pwXEoM/j7eVr4tpZQbSowJZu3dp7N+bzk/7yjh2e93cNY/l/DA+UOZktHH6nhKKYvVNzXz4/ZivttcxPdbC9ldWgtAkJ83g3qGcvbwXiTHBNMvKoieYQFEBfsRFexHkJ/jahZHFeVVQFibY2FAZQfWhgFVxhgjIrX2Y/cZY2qBtSLyFnAWtt5z1UUamlq49vVsfthaxJXjE/jb2UP+5+ubAF9vAny9iQsLYHCvX/7xNrcYthVWsmJnKYu3FbNodQHzlucT6u/DqUPiuGhUX8YlR+PlpVfQlVJdw8/Hi5HxkYyMj+S89N7c8vYabn5rDbUNzVw6Rr8tVcrTtLQYftpRwvurC/hy434q65oI8vNmQv8YrpmYxOikKAb1DMPbSWoRRxXlWwEfEUkxxmyzH0sH2t7kif1YOpDVzrq19v/Vq97d7P6PN/LD1iIemDqUK8YldPr3e3sJg3qGMahnGDPGJ9LQ1MKPO4r5bN0+Plu/n4WrC+gbGcj0sfFcNjqeyGC/bngXSilP1S8qiLdnjePq17L52wfriY8KYsKAGKtjKaUcoKSqnvlZ+by1Yjd7DtYSGuDD6Wk9OXt4Lyb0j3bab+zFHK45uKtfyHZF2wC/xTZ95VNgQtvpKyLyO+Bm4BT+b/rKU8aY5+znFwObgJuAZOAH4DJjzDe/9vqZmZkmOzu7S9+Tu3p7RT5/fm8d101O5i9nDe7y569rbOaLDft5K2s3P+eW4O/jxYWj+vK7yf2Jj9b+T6XaEpGVxphMq3M4Uld9ZlfUNXLhv3/iQEUd7984kf6xIV2QTinljHKLqnhhSS4LVxVQ39TChP7RXDomntOGxBHg67hC/Gg/sx05EvEG4GWgECgBrjfGbBCRScBnxphDn5TPYyu219l//aL92CGXAS/Zn6MQuOtIBbnquPUF5dz1wQYmpcRw+xmDuuU1Any9mZLRhykZfdiyv5JXf9rJu9l7eHvFbqZk9Obmk1NIiA7ultdWSnmWsABfXr5qNOc9vZTbFuTw3u8maNucUm5mR1EVT32zjQ9z9uLr7cUFI/tyzXGJDOjR3q2LzsthV8qtplfKj6ylxXDBsz+x52AtX8+eTESQ41pKDlTUMXdxLv9ZvoumZsNlY+K56eQUHWumFHqlvCu8t3IPty7I4aHzhzF9rPaXK+UOCivqeOLrbbyTvRs/by9mjE/g2knJltcOrnClXDm5d1fuYc3uMh6flu7QghwgLiyAu84ZwqzJyfzrm23My8rng9UF3HRyCjMnJDpsRqhSyj1dMLIP72Tv5pHPN3N6Wtx/p0gppVxPfVMzLy7ZydPfbqeppYUZ4xL4/UkDiHHxn2utdBQA5TWN/P3zzWQmRHL+COvGh8WFBfDg+cP46o+TyUyM5MFPN3HGPxezLLfEskxKKdcnIjwwdSjV9U08/Nlmq+MopY7SD1uLOP2Jxfzjiy0cnxrL17OP557z0ly+IActypXdE19vpaymgfumDHWKjX6SY0N45TdjePmqTBqbW7h07jJufzeH8pr/HeyvlFIdkRIXyjXHJfHeqj1sL2xvIq9SylmVVjdwy1urmflyFl5ewutXj+G5GaPc6h40LcoVhRV1zMvKZ1pmP4b0bjtO3lonDYrjy1uO53fH9+e9VQWc9uQPfL+l0OpYSikXdd3x/Qnw8ebf3+2wOopSqoM+X7+fUx7/gU/W7eOmk1P47OZJTE6NtTpWl9OiXPHCklyamlu4/oT+VkdpV6CfN3ecOYgPbphIeKAvV72ygjvfX0dtQ7PV0ZRSLiYq2I8rxsWzKGcvu0qqrY6jlPoVFXWNzH5nDb97cyW9IwL4+A+TmH1qqtPOGT9WWpR7uIPVDfxneT7npfd2+q+AhvUN58PfH8esycnMW57PuU8vZePeCqtjKaVczLWTkvH2Ep77Qa+WK+WscnaXcc6/lrJozV5uOmkA798wkYE9XWvEYWdpUe7hXvlxJzUNzdxw4gCro3RIgK83d541mDevGUt5bSNT//0j87Py8ZTRnkqpY9cjLIBLMvvx7so97C2rtTqOUqoVYwwvLsnloud+ornF8Pasccw+bSC+3u5fsrr/O1SHVV3fxKs/5XHakDhS41zrvz6PS4nh85snMTYpir8sXMdtC9ZqO4tSqsOuOz6Z5hbDvOX5VkdRStlV1Tdx47xVPPDJJk4c2INPbjqOzMQoq2M5jBblHmzRmr1U1DVx3fHJVkc5KtEh/rz6mzHcdHIK763aw8XP/6RXvZRSHdI3MogTB/bg7ezdNDa3WB1HKY+XW1TF1Gd+5PP1+/nLmYN4fsYoh++ZYjUtyj3YvKxdDOoZysj4SKujHDVvL2H2qam8NDOTvOIaznt6Kdl5pVbHUkq5gMvHxVNUWc/XGw9YHUUpj7ZkWxFTn/mRkqp63rxmLNcd398pxjM7mhblHmrtnjLWF1QwfWy8W/yLf/LgOD64cQIh/j5Mf2E5H6wusDqSUi5NRKJE5H0RqRaRXSIy/TDrIkTkNREptD/uOcy640XEiMgD3Rq8E45P7UGfiED+oy0sSlnmjZ/zuOqVFfQKD+TD3x/HhAExVkeyjBblHuo/y/IJ9PVmqoW7d3a1AT1CWXTjcYxMiOCWt9fw5Ndb9QZQpY7eM0ADEAdcDjwrImntrHsCCAISgTHADBH5TesFIuIL/BNY3p2BO8vbS7hkdD+Wbi8mr1jHIyrlSC0thgc/2chdizZwQmos790wgX5RQVbHspQW5R6ooq6RD3P2cl56b8ICfK2O06XCg3x5/eqxXDiyL09+vY0/v7eWJu0XVapTRCQYuBC4yxhTZYxZCnwIzGhn+bnAo8aYGmNMHvAScHWbNbcCXwJOt7/9JaP74e0lzF+hV8uVcpS6xmZ+P38VLyzZyczxCcy9MpMQfx+rY1lOi3IPtGjNXmobm7l8XLzVUbqFn48Xj108nJtOGsA72Xu47o2VOplFqc5JBZqNMVtbHcsB2rtSDiBt/nnof38hkoCtSL/vSC8qIrNEJFtEsouKijqf+ijEhQVw4sBYPlhdQHOLfrOmVHerrGvkqley+HTdfv561mDuOS8Nby/Xb6PtClqUe6APVhcwMC6U4X0jrI7SbUSE2acN5P6pQ/l2SyEzXlpOeW2j1bGUchUhQHmbY+VAe7NTPwfuEJFQERmArQBv/R30v7BfcT/Sixpj5hpjMo0xmbGxjttCe0pGHw5U1JO1U28SV6o7FVfVc+ncZWTnHeTJSzK4dnKyW9zX1lW0KPcwu0trWLnrIFNG9LY6ikPMGJfA05eNJGdPGdNfWEZJVb3VkZRyBVVAWJtjYUBlO2tvAmqBbcAiYD6wB0BEzgVCjTFvd1/UY3fK4DiC/Lz5MEdvEFequ+wrr2Xa8z+zo6iKF2dmutU9bV1Fi3IP82HOXgDOS/eMohzg7OG9mHtlJtsLq7hk7jIKK+qsjqSUs9sK+IhISqtj6cCGtguNMaXGmMuNMT2NMWnY/l7Jsp8+GcgUkf0ish+4BLhFRBZ1c/5OCfTz5rQhcXy6bj8NTXoPilJdLb+khouf+5nCinreuGYsJwzsYXUkp6RFuQcxxvDB6gJGJ0bSN9Kz7nA+cWAPXr96DHvLarl07jL2l2thrtThGGOqgYXAfSISLCITgSnAG23Xikh/EYkWEW8ROROYBRwae3gXtv70DPvjQ+AF4Ddtn8dqUzL6UF7byOKtjullV8pT5BVXM+35n6mqb2LetWMZ7UE7dHaWFuUeZNO+SrYVVnFehmd+ZTQ2OZrXrx5DYWU9l8z9WXf/VOrX3QAEAoXYWlKuN8ZsEJFJItK6P3wUsA5ba8vDwOXGmA0AxphKY8z+Qw9sbS7Vxhina94+LiWGyCDf/36bqJQ6djuLq7l07jIamluYf+04t76XrStoUe5BFuUU4OMlnD2sl9VRLJOZGMXr14yhtKqB6S8s44C2sijVLntbylRjTLAxJt4YM89+fIkxJqTVuneMMb2NMUHGmAxjzBe/8pxXGWP+5oj8neXr7cVZw3rx1cYD1DQ0WR1HKZeXV1zNpXN/pqG5hXnXjmVwr7a3qai2tCj3EMYYPs7Zx6SUGKKC/ayOY6mR8ZG8evUYiirrueyFZRRWamGulLLdf1Lb2MzircVWR1HKpe05WMPlLy6noclWkA/qqQV5R2hR7iHWF1RQUFbLmR58lby1UQm2wnxfWR1XvLicg9UNVkdSSllsTGIU4YG+fLlxv9VRlHJZ+8prmf7CcirrGnnzt1qQd4YW5R7iiw378RLb6C9lMzoxipeuyiSvpIarXsmiql6/slbKk/l4e3Hy4B58s6lQdwJW6iiUVNVzxYvLKa1u4PVrxpLWO9zqSC7FYUW5iESJyPsiUi0iu0Rk+mHWiYg8IiIl9sej0mqyvIgY+3NU2R8vOuo9uLIvNuxnTFKUx7eutDWhfwz/nj6SDXsruObVFdQ16s6fSnmy04bEUV7bSFae092LqpRTs+3UuYI9B2t5aWYmGf30ps7OcuSV8meABiAOuBx4VkTa27J5FjAV20zc4cA5wHVt1qQbY0Lsj992Y2a3sKOoim2FVZye1tPqKE7plCFxzJmWTlZeKX+Yv1qvkCnlwSanxuLv48VXGw9YHUUpl1HX2MxvX8tm074KnrtiFGOTo62O5JIcUpSLSDBwIfatlo0xS7HNq53RzvKZwBxjzB5jTAEwB7jKETnd1RcbbP2Rp2lRflhTMvpwz7lpfLXxAHe+vw5jjNWRlFIWCPLz4bgBMXy54YB+DijVAc0thlveWsPynaXMmZbOiYN0Y6Cj5agr5alAszFma6tjOUB7V8rT7Od+bd1i+w5xC0Uk8XAvKiKzRCRbRLKLijx3Q4gvNhxgeN9w+kQEWh3Fqc2ckMhNJw3gnew9PPblFqvjKKUsclpaHAVltWzcV2F1FKWcmjGGuxat5/MN+/l/5wxhiofug9JVHFWUhwDlbY6VA6EdWFsOhLTqKz8eSAQGAXuBj0XEp70XNcbMNcZkGmMyY2NjjyG+69pfXkfO7jJtXemgP56aymVj+vHMdzt4c9kuq+MopSxw8uA4RNAWFqWO4JnvtjNveT7Xn9Cfq49LsjqOy3NUUV4FtJ2JE4ZtB7gjrQ0Dqoz9e0RjzGJjTIMxpgy4GUgCBnd9ZPfw7eZCAE4dolNXOkJEuH/KUE4cGMv/W7Ser/UvZaU8TkyIP8P7RvD9Fs/9hlWpI3l/9R4e+3Ir54/ow+2nD7Q6jltwVFG+FfARkZRWx9KBDe2s3WA/d6R1hxhAfuW8R/tuSyF9IgJJ6RFy5MUKsI1Fe3r6SIb2CecP81ezbk/bL3mUUu7uhNRYcvaU6R4GSrXjp+3F3P7uWib0j+aRC4fTakieOgYOKcqNMdXAQuA+EQkWkYnAFOCNdpa/DswWkT4i0hu4FXgVQETSRCRDRLxFJATbTaAFwCZHvA9XU9/UzI/bizlxUKz+wHRSsL8PL80cTVSwH1e/toKCslqrIymlHOiEgbEYA4u36dVypVrbXljJdW+uJCkmmGevGIWfj25501Uc+f/kDUAgUAjMB643xmwQkUkiUtVq3fPAR8A6YD3wif0Y2MYpvg1UALnYesvPMcY0OuQduJisnaXUNDRzkt4JfVRiQ/155TejqWto5ppXV1BZp/+aKeUphveNIDLIlx+0hUWp/yqpquc3r67A38ebl68aTXigr9WR3IrDinJjTKkxZqoxJtgYE2+MmWc/vsQYE9JqnTHG3G6MibI/bm/VT/6tMWag/Tl62J9vm6Peg6v5bnMRfj5ejE+OsTqKy0qNC+XfV4xkW2EVN7+1huYWHZGmlCfw9hImpcSyeFsRLfpzrxR1jc3MemMlhRX1vDgzk76RQVZHcjv6nYMb+35LIeOTown087Y6ikublBLLPeel8e3mQv7+mXZKKeUpThgYS3FVAxv26mhE5dmMMdz5/jpW7jrI49MydLfObqJFuZvKK64mt7iaEwd65ijIrjZjXAIzxyfwwpKdvL0i3+o4SikHmJxq+/z8fkuhxUmUstbcxbksXFXA7FNTOXt4L6vjuC0tyt3Uob9EThqkoxC7yl3nDGFSSgx/+2A9K3eVWh1HKdXNYkL8GdYnnO+3al+58lzfbj7A3z/fzDnDe/GHkwZYHcetaVHupn7YWkRyTDDx0drz1VV8vL14+rKR9IkI5Lo3VrFXJ7Io5faOT41lze4yvdFbeaTthVXcNH8Nab3D+MdF6TrJrZtpUe6G6puaWZZbyqQUvcGzq4UH+fLClZnUNjRx3RsrqWtstjqSUqobTRgQTXOLIWunfjumPEt5bSOzXs/G38eL52dk6v1pDqBFuRtatauM2sZmjkvRfvLukBIXypOXjmBdQTl3vr8O+3AgpZQbGhkfSYCvF0u3F1sdRSmHaWkx/PHtNeSX1vDsFaPoExFodSSPoEW5G1q6vQhvL2FccpTVUdzWqUPiuPnkFBauKuD1n3dZHUcp1U0CfL0ZnRjFj1qUKw/y5Dfb+HZzIXefO4QxSVpLOIoW5W5oybZiRvSLIDRAh/p3p5tPTuHkQT24/+ON+tW2Um5s4oAYth6oorCyzuooSnW7rzce4F/fbOOiUX25YlyC1XE8ihblbuZgdQPrCso5TvvJu52Xl/DEpRn0iwrixnmrKKzQv7CVckcT+9s+T3/aXmJxEqW6187iav749hqG9QnngalD9cZOB9Oi3M38tKMEY9CbPB0kLMCX564YRVVdE7+ft5rG5harIymlutiQ3mFEBPlqC4tyazUNTfzujZV4ewvPXjGSAF+9sdPRtCh3M0u3FxHq70N6X91ty1EG9gzl7xcOIyuvlEc+22x1HKVUF/P2EsYnR/Pj9mK9sVu5JWMMf31/PVsLK/nnpSPoG6njlK2gRbkbMcaweGsx4/pH4+Otf7SONCWjDzPHJ/Di0p18vn6f1XGUUl1s4oAY9pbXsbO42uooSnW5/yzP5/3VBdxycirHp+rkNqto5eZG8ktrKCir5bgB2rpihTvPHkx6vwj+tGAtefoXt1JuZUL/aACW5epN3cq9rNtTzn0fbeT41FjdsdNiWpS7kZ932G5Cmjgg2uIknsnfx5tnpo/Ay0u4/j+rdGMhpdxIUkwwPUL9WZarN3sq91Fe28gN81YSE+LHk5dk4OWlN3ZaSYtyN/LTjhJiQ/3pHxtidRSP1TcyiCcvyWDTvgru/Wij1XGUUl1ERBiXHM2y3BLtK1duwRjDnxbksK+sjqemjyQy2M/qSB5Pi3I3YYzh59wSxidH6wgji504qAfXn9Cf+Vn5LFpTYHUcpVQXGZscRWFlPXklNVZHUeqYvfJjHl9uPMAdZw5iVEKk1XEUWpS7jR1FVRRV1jO+v7auOINbT00lMyGSOxeuI7eoyuo4SnWaiESJyPsiUi0iu0Rk+mHWRYjIayJSaH/c0+pcDxGZLyJ7RaRcRH4UkbEOexNdbFzyob5ybWFRri1ndxkPf7aJU4fEcc1xSVbHUXZalLuJQ/3kE7Qodwo+3l48NX0Efj5e3DhvtfaXK1f0DNAAxAGXA8+KSFo7654AgoBEYAwwQ0R+Yz8XAqwARgFRwGvAJyLikj12yTHBxIb6s1yLcuXCymsb+f38VfQIDeAfFw3Xb9ediBblbuKnHSX0Dg8gPkpnizqLXuGBzJmWzqZ9FTz86Sar4yjVYSISDFwI3GWMqTLGLAU+BGa0s/xc4FFjTI0xJg94CbgawBiTa4x53BizzxjTbIyZC/gBAx3yRrqYiDA2KYpluaXaV65ckjGGOxeuY19ZHf+6bAQRQdpH7ky0KHcDLS2GZbkljO8fo//F62ROGhTHb49L4rWfd/H5+v1Wx1Gqo1KBZmPM1lbHcoD2rpQDSJt/HtruIpEMbEX59q4IaYVxydHsr6hjl/aVKxc0P2s3n6zbx22nD9Q+ciekRbkb2Ly/koM1jdpP7qRuP2MQw/uGc/u7ORSU1VodR6mOCAHK2xwrB0LbWfs5cIeIhIrIAGxXyf/nKzsRCQPeAO41xrR97kNrZolItohkFxUVHdMb6C6H+sqX79QWFuVatuyv5N6PNjApJYZZk5KtjqPaoUW5G/jZ3t+oRblz8vPx4qnLRtBi4Ja3VtPU3GJ1JKWOpAoIa3MsDKhsZ+1NQC2wDVgEzAf2tF4gIoHAR8AyY8zDh3tRY8xcY0ymMSYzNtY5dxXsHxtMTIi/biKkXEpdYzN/mL+K0ABfHp+m88idlRblbmB5bgn9ogLpExFodRR1GAnRwTwwdSgr8g7y1Lcu+8298hxbAR8RSWl1LB3Y0HahMabUGHO5MaanMSYN298rWYfOi4g/8AFQAFzXvbG7n4gwOjGSrJ1alCvX8cAnG9l6oIrHp6UTG+pvdRx1GA4ryjsxXktE5BERKbE/HpV2GqVFZKaIGBH5bfend14tLYasvFLGJulVcmc3dUQfLhjRh6e+3abTG5RTM8ZUAwuB+0QkWEQmAlOwtZ/8goj0F5FoEfEWkTOBWcAD9nO+wLvYrqRfaYxxi6+JRidGUVBWy15tR1Mu4PP1+3lzWT6zJiczOdU5v4FSNo68Ut7R8VqzgKnYrsoMB86hzdUVEYkE/kI7V208zdbCSspqGhmbFGV1FNUB900dSr+oIP749hrKaxqtjqPUr7kBCAQKsbWkXG+M2SAik0Sk9fD9UcA6bK0tDwOXG2MOfTZPwPYZfhpQJiJV9sckh72LbjDG/nm7Ik+vlivntq+8ljsWrmVYn3BuO80lhx55FIcU5Z0crzUTmGOM2WOMKQDmAFe1WfMw8C+guPtSu4bl9r7GQzcfKecW4u/DPy8dQWFlPXd+sE7HqimnZW9LmWqMCTbGxBtj5tmPLzHGhLRa944xprcxJsgYk2GM+aLVuR+MMWI/F9LqscSK99RVBvcKI8TfR1tYlFNraTHMfjuHhqYW/nlpBn4+2rHs7Bz1J9SZ8Vpp9nPtrhORMUAm8NyRXtQV7uQ/Vlk7S+kdHkDfSO0ndxUZ/SL446mpfLJ2H++u3HPk36CUcireXsLIhEi9Uq6c2twlufycW8I956aRHOuS+3V5HEcV5Z0Zr9V2bTkQYu819wb+DfyhI72JrnAn/7EwxrB8Zwljk6N1PrmL+d3x/RmbFMU9H25gV0m11XGUUp00JjGSrQeqOFjdYHUUpf7Huj3lzPlyC2cO7cnFmX2tjqM6yFFFeWfGa7VdGwZUGdv3/DcAa40xP3dLShezo6ia4qoG7Sd3Qd5ewuOX2MZS3fL2Gh2TqJSLGZ1o+9zN3nXQ4iRK/VJtQzM3v72a6GB/Hr5gmF60cyGOKso7PF7Lfiz9MOtOBs4Xkf0ish/bTURzROTpbsjs9A5tXjFW+8ldUp+IQB46fxir88t0TKJSLia9XwR+3l7awqKczoOfbiS3qJrHp6UTEeRndRzVCT6OeBFjTLWIHBqv9VsgA9t4rQntLH8dmC0inwIGuBV4yn7uKiCg1dqF2MZtvdRN0Z3a8txSeoT6kxj9P5vnKRdxbnpvvttcyFPfbmNyaqxue6yUiwjw9WZ433C92VM5lW82HeDNZflcOymJCQNirI6jOsmRt+J2dLzW89h2flsHrAc+sR/DGFNmjNl/6IFtxGLF4bZsdmfGGLJ2ljImKUq/mnJx905Jo1d4ILPfWUN1fZPVcZRSHTQ6KYr1BeXUNjRbHUUpiqvq+fN7axncK4zbTtfxh67IYUV5J8ZrGWPM7caYKPvjdnOYuXHGmBOMMS866j04kz0Ha9lfUaf95G4gNMCXJy7JIL+0hvs/3mh1HKVUB2UmRNLUYlizu8zqKMrDGWO44721VNQ18eQlGfj7eFsdSR0FHVrpog59ZTpai3K3MCYpiusm94MYC4AAACAASURBVOetFbv5auMBq+MopTrgULvZyl3awqKs9faK3Xy9qZDbTx/IwJ7tDbZTrkCLche1Iq+U8EBfUnvoD5+7mH1qKkN6hXHHe2sprqq3Oo5S6ggigvwY0CNEJ7AoS+UVV3PfxxuZOCCaqycmWR1HHQMtyl1U1s5SMhMi8fLSfnJ34efjxZOXZlBZ38Qd7+lun0q5gsyESFbtOkhLi/68Ksdram5h9jtr8PES/nFRutYELk6LchdUVFlPbnG1tq64odS4UG4/fSBfbzrAgmzd7VMpZ5eZGEVFXRPbCquOvFipLvb84lxW5Zdx/9Sh9I7Qnb1dnRblLijbPhf30OYVyr1cPTGJcclR3PvRBnaX1lgdRyn1KzLtfeXZ2leuHGx9QTlPfLWVs4f34rz03lbHUV1Ai3IXlJVXSoCvF8P6hFsdRXUDLy/hsYvT8RLh1ndyaNavxZVyWgnRQcSE+LEyT/vKlePUNTYz+501RAX78eDUoToa2U1oUe6CVuSVktEvAj8f/eNzV30jg7j7vDSy8kp5eelOq+MopQ5DRBiVEKk3eyqHevyrrWw9UMWjFw3XXTvdiFZ1LqayrpGNeysYo60rbu/CkX04bUgc//hiC1v2V1odRyl1GJkJUeSX1lBYWWd1FOUBlueW8MKSXC4fG88JA3tYHUd1IS3KXcyq/DJajM4n9wQiwkMXDCM0wIdbF6yhsbnF6khKqXaMSrTPK9cWFtXNquqbuO3dHOKjgrjzrMFWx1FdTItyF5OdV4q3lzAiPtLqKMoBYkL8eeiCYawvqOCpb7dbHUcp1Y6hvcPx8/FipbawqG724Ceb2HOwljkXpxPs72N1HNXFtCh3MSvyShnSK4wQ/WH0GKen9eSCkX145rvt5Oh23ko5HT8fL9L7hrMyX4ty1X2+21LI/Kx8Zk1OJlNbWN2SFuUupKGphTW7y8hM1Kvknubuc9PoEerP7HfWUNfYbHUcpVQbIxMiWV9Qrj+fqluU1TTw53fXkhoXwuxTU62Oo7qJFuUuZP3ecuoaW/QmTw8UHujLoxcNZ0dRNXO+3GJ1HKVUG6PiI2lsNqwvKLc6inJDd3+4gdLqBh6floG/j7fVcVQ30aLchRzaNGiUXin3SJNSYrliXDwvLt1J1k7dqEQpZzLSvomQ9pWrrvbZun0sWrOXP5yUwlDdn8StaVHuQlbkHSQxOogeoQFWR1EW+cuZg+kXGcRtC3Korm+yOo5Syi4mxJ/E6CAtylWXKq6q568frGdYn3BuOLG/1XFUN9Oi3EUYY8jOK9WbOzxcsL8Pj12czu6DNTz82Sar4ygnJiLhInKXiCwUkS9bP6zO5q5GJkSyKv8gxuguvOrYGWP46/vrqKprYs60dHy9tWRzdzrCw0XsKKriYE2j9pMrxiRFcc3EJF5cupPT03oyKSXW6kjKOS0AvIH3gVqLs3iEUQmRLFxVQH5pDQnRwVbHUS7ugzUFfLHhAHecOYjUuFCr4ygH0KLcRaywb0qhk1cUwG2nD+S7LYXc/u5avvjjZMICfK2OpJzPOCDaGNNodRBPMapVX7kW5epY7C+v4+5FGxiVEMm1k5KtjqMcRL8LcREr8kqJDvYjKUY/6BUE+HozZ1oGByrquP+jjVbHUc5pKaBb/jlQSo9QQv19tK9cHRNjDHcsXEtDcwuPXZyOt5dYHUk5iF4pdxHZeQfJTIxERH84lU1GvwiuP6E/z3y3gzOH9eSkQXFWR1LO5SrgUxFZDhxofcIYc58lidyct5eQER+hRbk6Ju9k7+b7LUXcc+4QvRDnYfRKuQsorKgjv7SG0dpPrtq46eQUBvUM5Y731lFW02B1HOVcHgT6AXFASqvHACtDubuR8ZFsOVBJZZ12DanO23Owhvs/3sT45GiuHJ9odRzlYFqUu4DsXYf6ybUoV7/k7+PNYxenU1rdwD0fbrA6jnIulwIZxpiLjDEzWj2utDqYOxuVEIkxkLNbNxFSndPSYvjze2sxxvDoRcPx0rYVj6NFuQtYkVdKgK8Xab3DrI6inNDQPuH8/qQBfLBmL5+v3291HOU8cgG9XOtgGfERiOgmQqrz/rN8Fz9uL+HOswfTLyrI6jjKAg4rykUkSkTeF5FqEdklItMPs05E5BERKbE/HhV7I7WIxIjIj/bjZSLys4hMdNR7sEp23kEy+kXojFJ1WDeeOIC03mH87YN1lFZrG4sC4A3gQxG5TEROav3oyG/uxGd2hIi8JiKF9sc9bc4nish3IlIjIptF5JRjf2vOKyzAl9QeoazK16JcdVx+SQ0PfbqZSSkxTB8Tb3UcZRFHVnnPAA3Y+hsvB54VkbR21s0CpgLpwHDgHOA6+7kq4GogFogEHgE+EhG3vWG1qr6JDXvLtZ9c/Spfby/mTEunvLaRuxattzqOcg43Ar2Ah4CXWj1e7ODv7+hn9hNAEJAIjAFmiMhvWp2fD6wGooG/Au+KiFsP1z+0iVBLi24ipI6spcVw27s5+HgJj1w4XAc6eLAOFeUi4i0iV4uI/9G8iIgEAxcCdxljqowxS4EPgRntLJ8JzDHG7DHGFABzsE0RwBhTZ4zZYoxpAQRoxlacu23Fuia/jBaj/eTqyAb1DOOWU1L5ZO0+Pl671+o4ymLGmKTDPI449LiTn9nnAo8aY2qMMXnYCv+r7c+TCowE7jbG1Bpj3gPW2Z/bbY2Mj6CyrokdRVVWR1Eu4NWf8sjaWcpd5w6hd0Sg1XGUhTpUlBtjmoHHjTH1R/k6qUCzMWZrq2M5QHtXXdLs5w67TkTWAnXY/pJ40RhT2N6LisgsEckWkeyioqKjjG6tFXmleIntQ16pI7lucjLp/SK464P1FFUe7Y+rchci4iMik+0tLJM68a1iZz6zwXaRpPU/D7X/cxqQa4yp7MjzuMNnNvxyEyGlfk1uURWPfrGZkwb14OJRfa2OoyzWmfaVj0Tk3KN8nRCg7a3o5UB7+8a2XVsOhEir73OMMcOBMGA6tg0y2mWMmWuMyTTGZMbGuua3pdm7ShnUM4xQ3bFRdYCPtxdzLh5OdUMzf31/Hcbo1+eeSkQGAZuAecBN2NpINotIRzYU6sxn9ufAHSISKiIDsF0lP3SXWmeexy0+swGSYoKJDPLVvnL1q5pbDLctyMHfx5uHLximbSuqU0V5ALZewO9F5A0Ref3QowO/twpbEd1aGFDZgbVhQJVpU13YW1nmY/vLIL3jb8N1NDW3sDq/jMzESKujKBcyoEcofzptIF9uPMAHawqsjqOs829gLtDPGDPeGNMXeM5+/Eg685l9E1ALbAMWYSv+9xzF87gNEWFkfCSr8susjqKc2ItLclmVX8a956URFxZgdRzlBDpTlK/HdsPQd8B2YEerx5FsBXxEJKXVsXSgvcHKG+znjrTuEF/giD2SrmjjvgpqGpr1Jk/VaVcfl0RmQiR3L9rAgYo6q+Moa2RgaztsfUHjSfvxI+nwZ7YxptQYc7kxpqcxJg3b3ytZ9tMbgGQRCT3S87ibkQmRbC+s0k29VLu2F1Yy56utnJ4Wx5SM3lbHUU7iV4vyNiO0lvzK41cZY6qBhcB9IhJsH2M4BdvIrrZeB2aLSB8R6Q3cCrxqzzNORI4TET8RCRSRP2ObDLC8Y2/XtazIs331qUW56ixvL+EfF6fT0NzCHfbNKJTH2Qsc3+bYJPvxX9WZz2wR6S8i0faBAGdim6D1gP15tgJrgLtFJEBEzsc2Veu9Y3hfLmFkvO0bztW79Wq5+qWm5hZufSeHEH8fHpiqbSvq/xzppp+XOvAcho5dqb4BeBkoBEqA640xG0RkEvCZMSbEvu55+/Ots//6RfsxAH/gX/bzjfY1Zxtj3HLUxIqdpfSLCqRnuH6tpTovKSaYP58xiHs/2siC7D1MG93P6kjKse7ENqf8Y2AXtpGFZwFXdPD3d/QzexS2K/AR2K6wX26MaX0l/FJsF1YOAvnARcYY172Ls4PS+4Xj7SWs2nWQEwf2sDqOciLP/bCDnD3lPDN9JLGhRzXUTrmpXy3KjTFJXfVCxphSbPPH2x5fgu1moEO/NsDt9kfbtT/wy9YWt2WMIXtXKZNTXPdmJ2W9meMT+Xz9fu77eCMTU2Loo+O2PIYx5kMRGQFcgm1e+Vrgb8aYbR38/R39zH4HeOdXnicPOKEz2d1BkJ8PQ3qF6QQW9Qsb91bwz2+2cc7wXpw9vJfVcZST0S0indTO4mqKqxp0Prk6Jl5ewmMXp9NiDH9+V9tYPImIhGO7Sj0K24jD44FnRORLS4N5kJHxEazZXUZTc4vVUZQTaGhq4dYFOYQH+nH/lKFH/g3K42hR7qSy7f3kY5J08oo6Nv2igrjzrMEs3V7Mm8vzrY6jHGcBtivU3wBvAW+3eigHGJkQSU1DM5v3u/WwGdVBT3+7jU37Knjo/KFEBvtZHUc5Ibfdnt7VZeWVEhnkS//YkCMvVuoILh8bzxcb9vPwp5uYnBJDQnSw1ZFU9xsHRBtjGq0O4qkObSK0Kv8gQ/uEW5xGWSlndxnPfL+DC0b24bS0nlbHUU5Kr5Q7qey8UjITo/SubNUlRIRHLhyOtwh/WrCWlhZtY/EAS4GObBSkukmfiEDiwvy1r9zD1TU2c9uCHGJD/Ln73MNtiquUXil3SoWVdeSV1DB9bLzVUZQb6R0RyN3npXHbghxe/nEnv53kluP91f+5CvhURJYDB1qfMMbcZ0kiD/N/mwhpUe7JnvhqK9sKq3j1N6MJD9TdudXh6ZVyJ3Son1xv8lRd7cKRfThlcA8e/WIL2wu1z9XNPQj0w7aXQ0qrxwArQ3maUQmR7C6tpVA38fJI2XmlzF2Sy/Sx8ZygozHVEWhR7oSydpYS6OvN0N7ag6i6lojw0AXDCPLz5tZ3cnQqhHu7FMgwxlxkjJnR6nGl1cE8ychWfeXKs9Q0NHHrghz6RgZy51naSaaOTItyJ5S1s5QR8RH4+egfj+p6PUIDeGDqUHL2lPPs9zusjqO6Ty62TdaUhdJ6h+Hn4/Xfb0CV53j4083kl9bwj4vSCfHXbmF1ZFr1OZmKukY27a9gTJK2rqjuc87w3pyb3pt/frONDXvLrY6juscb2Hb0vExETmr9sDqYJ/H38WZ4n3BW6pVyj7JkWxFvLNvF1ROTGJccbXUc5SK0KHcyK/MOYgyM0X5y1c3un5JGZLAfs9/Oob6p2eo4quvdiG0nz4eAl1o9XrQylCcalRDJ+oJy6hr158wTlNc28qcFaxnQI4Q/nT7Q6jjKhWhR7mSy8krx8RJGxOumQap7RQT58eiFw9lyoJInvurQzuvKhRhjkg7z0LE7DjYqIZLGZsO6Av1WyhPc++EGiqrqeXxaOgG+3lbHUS5Ei3Ink7WzlGF9wwn00x9k1f1OHNSDy8b04/nFO8jOK7U6jlJu6dAmQtpX7v4+X7+PhasLuPHEAQzvG2F1HOVitCh3InWNzazdU6b95Mqh/nr2EPpGBnLrghyq65usjqOU24kO8Sc5JpiVu/Q/fN1ZUWU9d76/nqF9wvjDSTp5VHWeFuVOZHV+GY3NRvvJlUOF+Pvw2EXp5JfW8OCnm6yOo5RbGpkQycpdBzFGd9N1R8YY/rJwLVX1TTwxLQNfby2vVOfpvzVOZEVeKSKQmaBFuXKsscnRXDspmXnL8/luc6HVcZRyO5kJkRysaSS3uNrqKKobLMjew9ebCvnzGYNIiQu1Oo5yUVqUO5GsnaUMjAslPEi34VWON/vUVAbGhXL7e2sprW6wOo5SbiUz0dZXvlL7yt1OfkkN9360gfHJ0fxmQqLVcZQL06LcSTQ0tbBy10GdZ6osE+DrzROXZFBW08DfPlinX7Mr1YWSY0KICPJl5S4tyt1Jc4vh1gVr8BLhsWnpeHmJ1ZGUC9Oi3EmsKyijtrGZccnauqKsM6R3GLNPHcin6/bz/uoCq+Mo5Ta8vISR8ZFk682ebmXu4lxW5B3k3ilp9IkItDqOcnFalDuJZbm2D+oxSXqlXFlr1uRkxiRGcfeiDew5WGN1HKXcRmZiJDuKqrU9zE2sLyjn8a+2cNawnpw/oo/VcZQb0KLcSSzLLSE1LoSoYD+roygP5+0lzJmWjgFmv5NDc4u2sSjVFUbbJ2vpngCur66xmT++vYbIID8enDoMEW1bUcdOi3In0Nis/eTKufSLCuLuc4eQtbOUuYtzrY6jlFsY3jccPx8vsrWv3OU98vlmthVW8djF6UTqxTTVRbQodwLrC8qpaWhmrLauKCdy0ai+nDm0J49/tYX1uj24UsfM38eb9L7hZO3UK+WubMm2Il75MY+Z4xOYnBprdRzlRrQodwKH+snH6k2eyomICA+dP4yoYD9ufms1tQ3NVkdSyuVlJkaxvqBcf55cVGl1A7e+k0NKjxD+ctZgq+MoN+OwolxEokTkfRGpFpFdIjL9MOtERB4RkRL741GxN2uJSKqILBKRIhEpFZEvRGSgo95Dd1m+s4QBPUKICfG3OopSvxAZ7MdjF6ezo6iah3S3T6WO2ZjEKJpaDGt2l1kdRXXSoV07D9Y08OSlGQT4elsdSbkZR14pfwZoAOKAy4FnRSStnXWzgKlAOjAcOAe4zn4uAvgQGGh/nixgUffG7l5NzS1k5x3UUYjKaU1KieWa45J4Y9kuvtl0wOo4Srm0kQmRiNh2cFau5e0Vu/liwwFuP30Qab3DrY6j3JBDinIRCQYuBO4yxlQZY5ZiK65ntLN8JjDHGLPHGFMAzAGuAjDGZBljXjLGlBpjGoEngIEi4rLN2OsKyqmqb9J+cuXU/nT6QAb1DOX2d9dSWFlndRylXFZ4oC8D40K1KHcxO4qquPejjUwcEM01xyVZHUe5KUddKU8Fmo0xW1sdywHau1KeZj93pHUAk4H9xpiS9k6KyCwRyRaR7KKioqOI3f1+2mGLPr6/FuXKeQX4evPUZSOoqm/iTwvW0qJjEpU6aqMTo1i16yBNzS1WR1Ed0NDUwi1vrSHA14vHp2Xorp2q2ziqKA8B2o5vKAdCO7C2HAiRNkNARaQvtpaY2Yd7UWPMXGNMpjEmMzbWOe+Q/nlHCYN6hmo/uXJ6KXGh/O3swfywtYhXfsqzOo5SLiszMZLqhmY27au0OorqgDlfbWFdQTmPXDicuLAAq+MoN+aoorwKCGtzLAxo7xOp7dowoMoY899LcyISC3wJ/NsYM7+LszpMXWMzK/JKmdA/xuooSnXIFeMSOGVwHI98tlnHJCp1lMYk2e4hytIWFqe3ZFsRz/+Qy/Sx8ZyW1tPqOMrNOaoo3wr4iEhKq2PpwIZ21m6wn2t3nYhEYivIPzTGPNgNWR1mdX4Z9U0tTNDWFeUiRIRHLxpOZLAvN721mpqGJqsjqW7SiYlZ/iLynIgcsE/F+khE+rQ6nygin4rIQRHZLyJPi4iP496J8+kVHkh8VBDLc9vtvFROoriqntn28Yd3nT3E6jjKAzikKDfGVAMLgftEJFhEJgJTgDfaWf46MFtE+ohIb+BW4FUAEQkDvgB+NMbc4Yjs3ennHcV4CYzRySvKhUQF+/HEtAx2Fldz74cbrY6juk9HJ2bdDIzHNi2rN1AGPNXq/L+BQqAXkAEcD9zQfbFdw9ikKLLySvX+DCdljOFPC3Ior23kqekjCPTT8Yeq+zlyJOINQCC2D+f5wPXGmA0iMklEqlqtex74CFgHrAc+sR8DOB8YDfxGRKpaPeId9i660E87ShjeN4KwAF+royjVKRMGxHDDCf15O3s3H+bstTqO6mKdnJiVBHxhjDlgjKkD3uKXN+cnAe8YY+qMMfuBzzn8zfseY2xyNGU1jWwt1L5yZ/TS0p18t6WIv509mEE923bfKtU9HFaU28cYTjXGBBtj4o0x8+zHlxhjQlqtM8aY240xUfbH7Yf6yY0xrxljxP4cIa0e+Y56H12lur6JNbvLtHVFuaxbTkllVEIkdy5cR35JjdVxVNfqzMSsl4CJItJbRIKwXVX/rNX5fwKXikiQva3lTGyFuUcba+8rX56rfeXOJmd3GY98vpnT0+KYMS7B6jjKgzjySrlqJSuvlKYWozd5Kpfl6+3FPy/NwEvg9/NX0dCk493cSGcmZm0F8oECoAIYDNzX6vwP2Ir5CmAPkA180N6LusIY267SNzKQ3uEBLN+pfeXOpKKukd/PX0WP0AAevTCdNoPflOpWWpRb5Mdtxfh5ezEqIdLqKEodtb6RQTx6UTpr95Tz9882Wx1HdZ3OTMx6FggAooFgbPcPfQYgIl7Y7gNaaD8XA0QCj7T3oq4wxrariAhjk6PJ2llKq+FiykLGGP6ycB17y+r412UZhAdpa6lyLC3KLbJkWzGjkyL15hHl8s4Y2pOrJiTy8o87+WrjAavjqK7RmYlZ6cCr9hbFemw3eY4RkRggCugHPG2Mqbdv9PYKcFb3xncNY5OiKK5qYEdR1ZEXq2735vJ8Plm7j1tPS2VUgg5gUI6nRbkFDlTUseVAJZNT3PtKkPIcfzlrEMP6hHPbghz2HNT+clfXyYlZK4ArRSRcRHyx3dS/1xhTbIwpBnYC14uIj4hEADP55a7NHmtssu2eomXaV2659QXl3P/RRk4YGMvvJve3Oo7yUFqUW2DxVluv5CQtypWb8Pfx5unpI2hpMdw4b7X2l7uHjk7Mug2oA7YBRdiugp/f6vwFwBn2c9uBJuCP3R/f+SVGB9Ej1J/lO7Uot1JFXSO/n7eKqGA/5lycjpeX9pEra3j0Bg5WWbKtmJgQfwb3au+eKaVcU0J0MP+4eDi/e3MVD326iXvO8/ipdy7NGFMKTG3n+BJsN4Ie+nUJtokrh3ueNcAJ3RDR5YkI4/tH8+P2EowxelOhBYwx/Pndtew+WMv8a8cRHeJvdSTlwfRKuYO1tBiWbi9mckqMfgArt3PG0F5cPTGJV3/K49N1+6yOo5TTm9A/muKqerYVal+5FV75MY/P1u/nz2cMZEyS9pEra2lR7mAb9lZQWt3ApFQdhajc0x1nDmJEfAS3v7tWb2BT6ggOjcX9aXuxxUk8z8pdB3no002cOiSOayclWx1HKS3KHW3xNls/+XEDtJ9cuSc/Hy+emT4SPx8vrn9zJTUNTVZHUspp9YsKol9UID/u0HnljlRcVc+N/1lF74hAHrtY55Er56BFuYMt2VbEkF5hxIZq35pyX70jAvnXpSPYVljFXxau0znMSv2Kif1jWJZbQnOL/pw4QlNzC3+Yt5qDNQ08e8VIwgN1HrlyDlqUO1BFXSPZeQeZnKpXyZX7Oy4lhltPTWXRmr289lOe1XGUclrj+0dTWdfE+oK2m6iq7vCPL7fwc24JD54/jLTe4VbHUeq/tCh3oCVbi2lqMZw8uIfVUZRyiBtOGMApg+N44JNNZOnYN6Xa9d++cm1h6XafrN3H8z/kcvnYeC4a1dfqOEr9ghblDvTt5kIignwZ0S/C6ihKOYSXl/D4Jen0iwrihv+sYn95ndWRlHI6saH+pMaF8NMOvdmzO23ZX8mf3s1hZHwEd5+rI1uV89Gi3EGaWwzfbynkhNRYfLz1/3blOcICfHl+xihqGpr43ZsrqWtstjqSUk5nQv8YVuSVUt+kPx/dobymkeveyCbY34dnrxiFn4/+Paycj/5b6SA5e8ooqW7gpMFxVkdRyuFS40J5fFo6a3aX8bcP1uuNn0q1cdyAGOoaW1iZd9DqKG6nucXwh7dWU1BWy7OXjyQuLMDqSEq1S4tyB/lucyHeXsLxKXqTp/JMZwztxU0np/Duyj288mOe1XGUcirj+kfj6y38YB+bq7rOI59vZvHWIu6bMpTMRN0gSDkvLcod5JtNhYxKiCQ8SEcvKc91y8kpnDYkjgc/3cQSLT6U+q8Qfx9GJUSyeKv2lXel91fvYe7iXK4cn8BlY+KtjqPUr9Ki3AH2ldeycV8FJw/SqSvKs9lu/MwgpUcIN/xnle74qVQrk1Nj2bSvgsIKvSG6K6zcdZA/v7uOcclR3HXOEKvjKHVEWpQ7wFcbDwBwsvaTK0WIvw8vXJmJr7cX176WTXlNo9WRlHIKk+3tjYu36dXyY7XnYA3XvZFNr4gAnr18FL46YEG5AP231AE+W7efAT1CGNAjxOooSjmFflFBPHfFKHYfrOF3b66koanF6khKWW5IrzBiQvxYvFVbu45FVX0Tv30tm/qmFl6aOZrIYD+rIynVIVqUd7OSqnqW7yzhzKE9rY6ilFMZkxTF3y8Yzs+5JdylE1mUwstLmJQSy9LtxbS06M/D0WhqbuH381axrbCKZ6aP1IthyqVoUd7Nvtp4gBYDZ2hRrtT/uHBUX/5w0gDezt7Ncz/kWh1HKcsdnxpLaXUD6/eWWx3F5RhjuPejjXy/pYj7pwxlcqpOO1OuRYvybvbZ+v3ERwUxpFeY1VGUckp/PCWVc9N788jnm1m0psDqOEpZ6riUGAC+36ItLJ310tKdvLFsF7MmJzN9rE5aUa7HYUW5iESJyPsiUi0iu0Rk+mHWiYg8IiIl9sejIiKtzs8VkS0i0iIiVzkq/9Eor2nkpx3FnDm0J63eglKqFS8v4bGLhzMmKYo/LVjLstwSqyMpZZmYEH8y+kXwzaYDVkdxKR/l7OWBTzZx1rCe3HHGIKvjKHVUHHml/BmgAYgDLgeeFZG0dtbNAqYC6cBw4Bzgulbnc4AbgFXdmrYLfL3pAI3NRltXlDoCfx9v5s4YRb+oQGa9ns2W/ZVWR1LKMqcOiSNnT7mORuygZbkl3PpODqMTI3l8WgZeXnoRTLkmhxTlIhIMXAjcZYypMsYsBT4EZrSzfCYwxxizxxhTAMwBrjp00hjzjDHmG8DpP60+WbePXuEBpPeNsDqKUk4vIsiP164eQ4CvNzNfzqKgrNbqSEpZ4hT7+NxvNhdanMT5bdpXwbWvZ9MvKpAXrswkwNfb6khKHTVHXSlPBZqNMVtbHcsB2rtSnmY/d6R1RyQitbLzXgAAF5FJREFUs0QkW0Syi4oc259XXFXPD1uLOC+jt/5Xu1Id1DcyiNeuHkN1QxMzX87iYHWD1ZGUcrjUuBD6RQXy9UZtYfk1u0truPLlLIL9fHj9mrFEBOnoQ+XaHFWUhwBtbyUvB0I7sLYcCJGjaMo2xsw1xmQaYzJjYx17F/ZHOXtpbjFcMKKvQ19XKVc3uFcYL1yZSX5pDVe9uoKq+iarIynlUCLCyYPiWLq9mNqGZqvjOKWiynpmvLSchqYW3rhmzP9v787Dq6ru/Y+/vxkgTAFCSJhHgyiUQSIICKJWWtE6MFTFW1vrzM+pWr1t7/WnhXtvtYP9XUVBWhTFOmDVeoXrUFtrGUSJyhTByAxhDCEhA5nX74+98TkcD0iAnH1O8nk9z3keztqb5LP3OWfle/Zee226tmsRdCSRkxatorwUCJ9+JBWINHA0fN1UoNTF2STGr3+Wz4AuqZzeKdL3DhE5lnP6dGDmNUNZm1/Mzc/lUFGtwkSalovOzKSypo4lG3R3z3BF5VX8YO5H7DlYydM/OpusTP2dlcYhWkV5HpBkZlkhbYOB3Ajr5vrLvmm9mLVhbwmrdxRz5dCuQUcRiVvjB3Tit1MGsWzjfm5/4TOqa3XXT2k6hvdOo01KkoawhCmtrOFHz6xg074y/nBdNsN6tg86ksgpE5Wi3DlXBrwGTDezVmY2GrgcmB9h9eeAe8ysq5l1Ae4F5h1eaGbNzCwFMCDZzFLMLKbmW3/t03wSDC4b0iXoKCJx7cqh3Zh++QDeW7eHu19aSY0Kc2kikhMTGHd6Bu+t26P3va+8qoYfz1vBmvxiZk4d+tWc7iKNRTSL2WlAC2Av8CJwm3Mu18zGmFlpyHpPAW8Ca4C1wCK/7bB3gUPAKGCO/++xDR//+NTWOd5YuZMxWR3JaJMSdByRuHfdyF7824QzWLRmF/f9eTW1uv24NBGXfKsT+8uqWL6pMOgogauoruXGZ3PI2VLI768awvgBmmpYGp+kaP0i51wh3vzj4e2L8S7uPPzcAff7j0g/Z1wDRTwlPsjbS37RIX52sW5eIHKq3DS2D5U1tfz23TwM+M2UwSRqViNp5MadnkGrZoksXL2zSR8Vrqiu5abncvhw035+N2Uwlw3WWWhpnGJq2Edj8NyHW8lo05zv6Fu8yCl1+wVZ3HNRP177LJ/7XlmlI+bS6KUkJ3LRmZm8nbu7yV5TcajKO0K+ZEMBj0waxMSzNKOZNF4qyk+hLQVl/OOLfVwzvAfNkrRrRU61Oy/M4l6/MP/JyyubbKEiTcclg7pQVF7N0iY4C0tZpTeGfOnGAn4zeTDfz+4edCSRBhW14StNwfPLt5KUYEwd0SPoKCKN1h0XZpGclMDDb63nUHUtM6cOpXmS7uInjdPYfum0SUli4epdjDs9I+g4UVN8qJrrn/mYlduLePT7g7lS9/yQJkCHc0+RQ1W1LMjZzncGdiIzVRd4ijSkW8/ryy8vG8BfP9/Djc/mUKYbDJ1yZpZmZq+bWZmZbTWzqUdZr7mZzTazPWZWaGZvmlnXsHWuNrN1/s/aaGZjorMV8a95UiLjz+zEO7m7qaxpGvP1F5RWcs2c5azJL+aJqWepIJcmQ0X5KfLqpzs4WFHDD0f2CjqKSJPww1G9+M3kQSzdUMDUP35EYVlV0JEamyeAKiATuBaYZWYDIqx3FzASGAR0AYqAxw8vNLOLgEeA6/Hu4jwW2NSgyRuZSwd3pqSihvfX7ws6SoPbXljOlNkfsqmglD/+8Gwu/lbnoCOJRI2K8lOgsqaWJ9/fwNAe7Ti7l25kIBItU7K7M/tfhrF+10Emz17G9sLyoCM1CmbWCpgEPOCcK3XOLQH+B/hBhNV7A+845/Y45yqAl4DQ4v2XwHTn3HLnXJ1zLt85l9/Q29CYjDktnYw2zXklZ3vQURrU2vxiJs5aRmFZFX+6cQTn9esYdCSRqFJRfgosyNnBzuIKfvLtfphpmjaRaBo/oBPzbxhBQUklVz65jNU7ioKO1Bj0A2qdc3khbas4stg+bC4w2sy6mFlLvKPqbwGYWSKQDXQ0sw1mtsPMZppZi0i/1MxuNrMcM8vZt6/xHxU+XkmJCUwe1o33v9jL7uKKoOM0iA/y9nH1nOUkJRh/vnUkw3qmBR1JJOpUlJ+kw0fJh/Vsz5gmPI+sSJCG907j1dtG0TwpgaueWq5bk5+81kBxWFsx3vCTcHnANiAfOAicAUz3l2UCycBkYAwwBBgK/HukX+qcm+Ocy3bOZXfsqKOkob6f3Z065w2VbGz+9NFWfjxvBd3TWvLatFFkZUZ6m4k0firKT9LLK7azS0fJRQKXldmG1//PKE7LaM1N83OY/cFGvHuRyQkoBVLD2lKBkgjrzgJSgA5AK+A1/CPleHdcBnjcObfLOVcAPApMOOWJG7le6a04p08aC3K2U9dI5uivqa1j+puf82+vr2VsVjqv3DqSzm0jnkQRaRJUlJ+EovIqHvvbl5zdqz2jT+sQdByRJi+jTQoLbhnJhG915uG31nPvglVUVDeNGStOsTwgycyyQtoGA7kR1h0MzHPOFTrnKvEu8hxuZunOuQPADqBxVJEBu+rs7mzdX87yzfuDjnLSisqr+NEzK3h66WauH92LP1yXTevmmqVZmjYV5SfhPxet40B5Nb+8bKCOkovEiBbNEpl5zdCv7v458UldAFpfzrkyvCPe082slZmNBi4H5kdYfQVwnZm1NbNkYBqw0z8qDvAMcIeZZZhZe+BuYGHDb0Xjc/HAzrRJSeKFj7YFHeWkrM0v5rKZS/l4cyG/njyIB783gKRElSMi+hScoGUbCnjlkx3cNKYPZ3YJP8srIkEyM+68MIu5P8xmx4FyLn18CX9bp3Hm9TQNaAHsBV4EbnPO5ZrZGDMrDVnvp0AF8CWwD29oypUhy2fgFe55wDrgM+A/Gz5+45OSnMg1w3vw1trdcflF0znHyyu2MXHWMqpr63jplnN0l06RECrKT0B5VQ2/eH0NPTu05O5vZ33zfxCRQFx4RiYL7xhD13YtuOHZHKa/+TlVNXVBx4oL/nCUK5xzrZxzPZxzL/jti51zrUPW2++cu9Y5l+Gca+ecO9c593HI8mrn3DR/WSfn3J3+1IlyAq4f3YsEg7lLNgcdpV5KKqq5++WV/OuraxjRO42Fd5zLWT00hbBIKBXl9VRTW8cdL3zGtsJyfjXxW6Qk6/beIrGsRwdvRocfjerF00s3M3HWUr7cE+l6RZHY17ltCy4f0pWXV2znQJzcMOuTrQeY8NhiFq7exU/H92Pe9cPp0Lp50LFEYo6K8npwzvHAG7n8bf1eZlwxkFF9NQWiSDxISU7kocsGMOcHw9hZVMEljy9h7pLNjWYWC2labh7bh0PVtcxfvjXoKMdUUV3Lr95ax5TZy3AOFtwyktsvyCIxQddgiUSiovw4VVTX8uD/5PLix9uYNq4v147oGXQkEamn8QM68c7dYxmblc6MhZ8z5akPddRc4k6/zDZc0D+Decu2UFpZE3SciHK2FPK9x5fw1AebuOrs7rx11xiG9dRwFZFjUVH+DWrrHB9t2s8ljy3muQ+3csO5vbnvO6cHHUtETlDHNs35w3XZ/G7KYDbuK+WSx5bw6LtfcKhKUydK/LjrwiwKy6p48v0NQUc5QnF5Nb94fQ2TZ39IeVUt864/m19NHESblOSgo4nEPE0KehQfbdrPzPc3sHJbESWVNXRum8LzN4zgXN21UyTumRmThnXjvNM7MmPh5zz29w28+mk+D1x6Bt8Z0ElTnErMG9y9HROHduWPSzZzzfAedE9rGWie2jrHCx9v49F3v6D4UDU3ntubn1zUj1aae1zkuOnTchS1dY59JZVcNqQL2b3ac+EZmaTqm75Io5Leujn/ffVQrhnegwffyOXW5z8lu2d7fj7hDJ1ql5h3/3f789ba3fzX/65j1r8MCySDc4731u3lt+98wRd7ShjRO40HvzdAUwWLnAAV5Ucx6rR03r57bNAxRCQKzunTgUV3nsuCnB38/r08Js1axgX9M7jrwiwGd28XdDyRiDq1TWHauL787q95LPmyIKpncp1zfJC3j8f+9iWfbiuid3orZl17Ft8dqDNNIifKnGsasw9kZ2e7nJycoGOISIwrq6zhmaWb+eOSzRSVVzMmK52bxvRhTFZ6YMWGmX3inMsO5JcHRH328amormXCY4s5eKiGRXeeS2ZqSoP+vuraOt5eu5s5/9zEmvxiOrdN4Y4LspiS3Y1k3ZVTBDjxPltFuYhIBKWVNTz34RaeWbqFfSWVZGW0ZuqIHkwc2o22LaM7lE1FuRxL3p4SLp+5lIFdU3nhpnMapDjeWXSIVz/ZwZ8+2sbugxX06tCS28b15cqh3WiWpGJcJJSK8m+gDl5ETkRlTS0LV+1i3rItrMkvpllSAhedmcn3BnVm3OkZUbmBmIpy+SZvrMznrpdWcu2IHsy4fCAJp2Au8KLyKt79fA9vrtrJkg0FOAfnnpbO9aN7cf7pGafkd4g0RifaZ0dtTLmZpQFzgfFAAfDzw7dtDlvPgIeBG/2mucC/Ov/bg5kN8dvOANYBNzjnVjb8FohIU9Q8KZFJw7oxaVg31uYX80rOdhat2cWi1bto2SyRUX3TOb9/R0b26UDv9FYaTyuBuHxIV3J3HmTOPzexr6SS3181pN4zn9TU1rF+dwmLvyzgn3n7WLGlkJo6R7f2Lbjjgiwmn9WNHh2CneVFpDGL5oWeTwBVQCYwBFhkZqucc7lh690MXAEMBhzwV2ATMNvMmgFvAP8PeBK4BXjDzLKcc/Fxv2ERiVsDu7ZlYNe2PHDpmXy4aT/v5u7h7+v38t66PQCkt27GkO7tOKNzKv07pdKzQ0u6p7UkNSVJxbo0uJ9f3J/ObVOYsfBzJj65jGnn9+WiMzNp2ezIP/W1dY69JRXkHzjEpoIy8naXkLvzIKt2FFHuz9ffv1MbbhrbhwkDOzOwa6revyJREJXhK2bWCjgADHTO5flt84F859zPwtZdBsxzzs3xn98A3OScO8fMxgPPAN1CjpxvA252zr19rAw6FSoiDcE5x6aCMj7eXMiKzYWsyS9m475S6kK61uZJCaS1aka/zDY8++Ph9f4dGr4i9fFB3j5+8doa8osO0bJZIl3btQCgqraO4kPVHDxUfcT7MyU5gX6ZbRjavR1n9WzPyD4dyGjgC0ZFGrNYH77SD6g9XJD7VgHnRVh3gL8sdL0BIctWuyO/Saz2279WlJvZzXhH3unRo8cJhxcRORozo2/H1vTt2Jprhnv9TEV1LRv2lrLjQDnbCsvZX1pFYVmVbqQiUXFev44svv98VmwpZNGaXRSUVgKQlJBA2xbJtGuZTKe2KXRp14JeHVrRI60liRofLhK4aP2FaA0Uh7UVA22OY91ioLU/1rw+Pwf/aPsc8I661D+2iEj9pSQnfjXURSQICQnGiD4dGNGnQ9BRROQ4RWseo1Ig/PZeqUDJcaybCpT6R8fr83NEREREROJCtIryPCDJzLJC2gYD4Rd54rcNPsp6ucAgO/KKk0FH+TkiIiIiInEhKkW5c64MeA2YbmatzGw0cDkwP8LqzwH3mFlXM+sC3AvM85f9A6gF7jSz5mZ2u9/+94bMLyIiIiLSkKJ5G65pQAtgL/AicJtzLtfMxphZach6TwFvAmuAtcAivw1/2sMrgOuAIuDHwBWaDlFERERE4lnUpgJwzhXiFdTh7YvxLuA8/NwB9/uPSD/nM2BYA8UUEREREYm6aB4pFxERERGRCFSUi4iIiIgETEW5iIiIiEjA7MibYzZeZrYP2FrP/5YOFDRAnGhR/mDFc/54zg6NL39P51zHoMIE4QT7bIjv1z6es4PyBymes0Pjy39CfXaTKcpPhJnlOOeyg85xopQ/WPGcP56zg/I3ZfG87+I5Oyh/kOI5Oyj/YRq+IiIiIiISMBXlIiIiIiIBU1F+bHOCDnCSlD9Y8Zw/nrOD8jdl8bzv4jk7KH+Q4jk7KD+gMeUiIiIiIoHTkXIRERERkYCpKBcRERERCZiKchERERGRgKkoj8DM0szsdTMrM7OtZjY16ExHY2bNzWyun7PEzD4zs4tDll9oZuvNrNzM3jeznkHmPRYzyzKzCjN7PqRtqr9tZWb2FzNLCzLj0ZjZ1Wa2zs+50czG+O0xvf/NrJeZ/a+ZHTCz3WY208yS/GVDzOwTP/snZjYkBvLebmY5ZlZpZvPClh11X/ufk6fN7KC/nfdEPTxHz29m55jZX82s0Mz2mdkrZtY5ZLmZ2SNmtt9//NrMLIhtiFXqt6NPfXYw4qnfVp9dvz5bRXlkTwBVQCZwLTDLzAYEG+mokoDtwHlAW+ABYIH/oU0HXvPb0oAc4OWggh6HJ4AVh5/4+/wp4Ad4r0U58GQw0Y7OzC4CHgGuB9oAY4FNcbL/nwT2Ap2BIXjvo2lm1gx4A3geaA88C7zhtwdpJ/AfwNOhjcexrx8CsoCewPnA/Wb23SjkDRcxP94+ngP0wstYAjwTsvxm4ApgMDAIuBS4pYGzxhv129GnPjsY8dRvq8+uT5/tnNMj5AG0wuvY+4W0zQceDjpbPbZhNTDJf1MsC9u2Q0D/oDNGyHw1sADvg/i83/ZfwAsh6/T1X5s2QecNy74MuCFCe8zvf2AdMCHk+W/w/qiOB/LxZ2jyl20Dvht0Zj/LfwDzjndf+9syPmT5DOClWMkfYflZQEnYe+zmkOc3AMuDfh1i5aF+O5C86rODyx93/bb67OPrs3Wk/Ov6AbXOubyQtlVArB5xOYKZZeJtQy5e5lWHlznnyoCNxNi2mFkqMB24N2xReP6N+H94o5fu2MwsEcgGOprZBjPb4Z9KbEF87P//Bq42s5Zm1hW4GHgbL+Nq5/cmvtXEVvZQR93XZtYe6BK6nNj/TI/F+wwfdsT2Efv5o039dhSpzw5cY+i31WdHoKL861oDxWFtxXinuGKamSUDfwKedc6tJ362ZQYw1zm3Paw9HvJnAsnAZGAM3qnEocC/Ex/5P8DrKA4CO/BOIf6F+Mge6lh5W4c8D18Wc8xsEPB/gftCmsO3rxhorXHlX4m39+tX4rTfVp8drMbQb6vPjkBF+deVAqlhbal444Vilpkl4J2urQJu95tjflv8i1C+Dfw+wuKYz493ug3gcefcLudcAfAoMIEYz++/Z97BG9fXCkjHGyf3CDGePYJj5S0NeR6+LKaY2WnAW8BdzrnFIYvCty8VKA07ItaUxdv7FYjPflt9drAaUb+tPjsCFeVflwckmVlWSNtgjjwtEVP8b15z8Y4ATHLOVfuLcvGyH16vFd4Yv1jalnF4F0psM7PdwE+BSWb2KV/P3wdojvcaxQTn3AG8IxWRPmixvv/TgO7ATOdcpXNuP96FKhPwMg4K+1Y/iNjJHu6o+9p/jXaFLicGP9P+zAPvATOcc/PDFh+xfcRg/oCp346ecajPDlJj6bfVZ0cS9OD/WHwALwEv4n0LHY132mFA0LmOkXc2sBxoHdbe0c8+CUjB+yYdUxeHAS2BTiGP3wJ/9rMfPj03xn8tnifACz2OsQ3T8WYgyMA7YrEY7/RuPOz/TcDP8GaDaAe8jncqvRmwFbgL74/q7f7zZgHnTfL35a/wjjCm+G3H3NfAw3infNsD/fE6/Khf/HSM/F3xxlPed5T/dyvexV1d8cZa5gK3Bv3+iaWH+u2o5VafHXz+uOm31WfXr88O/M0Viw+8b6J/AcrwrlyeGnSmY2TtifeNvwLvdMnhx7X+8m8D6/FO2f0D6BV05m/Ynofwr+T3n0/1X4MyvKme0oLOGCFzMt4UVUXAbuAxICUe9j/eeMp/AAeAAuAVIMNfNhT4xM/+KTA0BvI+5L/fQx8PfdO+9v9APY1XMOwB7oml/MCD/r9DP8OlIf/PgF8Dhf7j14TMsKCH+u0At0V9dvTzx02/rT67fn22+f9ZREREREQCojHlIiIiIiIBU1EuIiIiIhIwFeUiIiIiIgFTUS4iIiIiEjAV5SIiIiIiAVNRLiIiIiISMBXlIiIiIiIBU1EuIiIiIhKw/w/mTPgGRC83DgAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "learn.recorder.plot_sched()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Smith's original 1cycle paper used a linear warm-up and linear annealing. As you see above, we adapted the approach in fastai by combining it with another popular approach: cosine annealing. `fit_one_cycle` provides the following parameters you can adjust:\n", + "\n", + "- `lr_max`: The highest learning rate that will be used (this can also be a list of learning rates for each layer group, or a python `slice` object containing the first and last layer group learning rates)\n", + "- `div`: How much to divide `lr_max` by to get the starting learning rate\n", + "- `div_final`: How much to divide `lr_max` by to get the ending learning rate\n", + "- `pct_start`: What % of the batches to use for the warmup\n", + "- `moms`: A tuple `(mom1,mom2,mom3)` where mom1 is the initial momentum, mom2 is the minimum momentum, and mom3 is the final momentum.\n", + "\n", + "Let's take a look at our layer stats again:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsQAAADWCAYAAADW1JQ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3hcZ5X48e+ZLo16tyXLcu9xXNIdEpNOJ6EEkgC7QIBQf5AldLK0pbP0JQllN4QSIAklJAHSnAQSx3bcu2XZVu8jzWhGmvL+/rgzsnqxJY00Op/nmcfWnTt33rE1M+eee97zijEGpZRSSimlZitbsgeglFJKKaVUMmlArJRSSimlZjUNiJVSSiml1KymAbFSSimllJrVNCBWSimllFKzmgbESimllFJqVtOAWCmlUoyIXC4iRkTKkj0WpZSaCTQgVkqpGUBEyuJB7uXJHotS04WI3CgiVSLSLiI/FxFnn/vsIvKCiLw5mWNUM4MGxEoppZSacUQkH/g58GngUuBi4NY+u3wUqDXG/DYJwxszEXGl4nPNNBoQz2Ai8pSI/FREviQijfEz5C+LiE1EPiciDSLSJCJf7vMYh4jcKSLHRSQkIvtE5D0DjvthEdkpIn4RqReR34jInD73Jy7HXiUiW0SkS0T2i8g1U/n6lUpFIrJJRJ4Tkc74bVf8vXUqvsuT8fdfVZ/HfFBEquPvxceA8mSMXakpthDwGWPuM8bsAR4CVgKIyBLgw8D7xnKg+PfiURF5rYgcFJGAiDwpIosG7LdBRP4W/35sEpEHRGR+n/sXxLfVxt+Pe0TklgHHSHx3f1FE6oCaYcb0VPy9PvD2jj77fDA+3pCIHBGRT4uIo8/9VfEY4Uci0gI8F98+J/7d3i4iwfhzbRzLv1Wq0oB45nsD4AQ2YZ0Nfwr4C5CBdcZ8O/ApEbkuvv89wPXAe4AVwBeAr4nIOwcc93ZgDfB6rC/X3wzx3N8EvgKsBbYBvxWRnAl7ZUrNMiJiB/4EvACsj9/uBLrifwe4AZgDnBd/zGuB7wDfBs4F7ge+MZXjVipJjgLpIrJRRLzAZcBLIiLAT4FPGWPqx3G8OVgB9E1Y2eYc4GeJO0VkJfA08C9gI/ByIAr8XUQ88d0ygMeBa7G+Q+8Cfi4imwc815uAQuCK+HGGcn18TInbl4AA8GJ8PHdifVd/Euv7/MNY3+2fH3CcDwGNwEXA2+P/Pg8By4FXAecDDfHXUTDiv1AqM8bobYbegKeAnQO27QP2DNi2Cyt4XQDEgOUD7v/cwOMMuH8dYIDS+M+Xx3++vs8+JfFt1yT730VvepupNyA3/j66fIj7yoa6D3gWuG/Atm/G9y1L9mvSm94m8wa8Ov4dV4l1YmgHPgg8HP9eeih+3/8BGSMc504gAhT22XZj/DvTE//5F8BvBjzOjXXC+roRjv1H4O4+Pz8FHAZs43id1wDdwKvjP6fHn/faAfu9DWjv83MV8PiAfa6Ifz6sHPA66oDPJfv/NFm33rS6mrF2Dfi5Pn4buK0I64xWgG3WCWIvB9ZZLmCVRGCdca7EOkNOXEmYT/9LOzsTfzHG1ItIFCg+w9eh1KxnjGkTkXuAx0TkCaxs1IPGmEMjPGwl8OsB254FPjZJw1Rq2jDG/Bn4c+JnEakAPgFcAHwPOIB1JfWXwGeBO0Y4XK0xpqnPzzVY35lFwEmsqzKLRcQ/4HEeYEn8+dOxkkyvxsrqurCCzScHPGa7MSY2ltcoIquwrvzcEX+9AKuANOAPImL67G4HPCJS2Oe1bB1wyFVAizFmf2KDMaZbRF6I3zcraUA884UH/GyG2WbjdGB7MdaZ5cB9EJFy4K/AvVjlFM1Ymal/YL2x++oZYjxahqPUWTDGvFtEvgtcDVwFfFFEPoCV8Rr2YVMyOKWmv7uBO40x1SJyJfCfxpiIiPwS+M9RHjvwOy3xvrL1+fNe4KtDPLYl/uc3gNdinZAexCpx+BaQPWD/wGgvBEBEirDKIH9pjPnvPnclxvRGrGzzQK2jPNdQnxkyzPZZQQPi2WV7/M9yY8xfhtnnPKyzzo8YY4JgTSKYisEppSzGmL3AXuDbIvI/WDPnH4zfbR+w+37gEuBHfbZdMumDVGqaEZF3A2KMuTu+yYY1xwashM7ZJmy2AecAx0y8zmAIL8MqYfptfEw2YClWje64iIgb631/EKsOuK99QAhYaIz56zgPvQ8oEJGViSxx/LnOp//nyKyiAfEsYow5KiI/A+4WkY9jTQzwAhuw6qa+BhzBOkP8mIjchzVh7nPJGrNSs4mILAbejXUJ+BQwF2ty7A6sqzV+4GoR2Qd0G2PasLJPvxORrVhXdzYBtwxxeKVSloiUYk0m29Rn8xbgo/FOS+/HKkE6G1/BKj/4ZfwqThNQAbwO+K4xphI4BLxWRP6A9X79KNb7eNwBMfCT+GP/DSjsU+roM8b4ReQrwFfi2/+OFdOtAdYZY0YqDXki/jp+JSLvB3xY5SQe4MdnMM6UoJe3Z59bsSYefBors/Q48HasSQcYY3ZjTUh4T/z+24GPJGWkSs0+AaxaxN9gXQb9A/BP4APxesP3Y81OPwW8BGCMeRDr8uzHgd1YM+RH+jJUKhX9BPiqMaaqz7YPYQWs27DeW6OVTIzIGHMAq+QwA3gM6zvybqyrqu3x3f4fcAKrZvhxrDrk35/hU16ONf5DWBPeErc3x8fzxfjzvQtrPtGz8Z+rRnkdBiuIP4hVivUi1gTEq4wxzWc41hlPhs/6K6WUUkoplfo0Q6yUUkoppWY1DYiVUkoppdSspgGxUkoppZSa1TQgVkoppZRSs5oGxEoppZRSalZLeh/igoICU1FRkexhKDVtbN++vdkYU5jscQxF369K9afvV6VmjpHer0kPiCsqKti2bVuyh6HUtCEiJ5I9huHo+1Wp/vT9qtTMMdL7VUsmlFJKKaXUrKYBsVJKKaWUmtU0IFZKKaWUUrPahAbEIvIBEdkmIt0i8ouJPLZSSimllFKTYaIn1dUCXwKuAdIm+NhjZozhJ1sqee25c5mTnbRhKKXUlPq/f1Wxbl4ua8qykz0UpdQECEdjPHmwke0n23DabCyfk4kgLC7KYFlJZrKHl1ImNCA2xjwAICIbgbKJPPZ4VDYH+OojB4kZw22XL07WMJRSaso0dob43B/3sbYsm4fefwkikuwhKaXOQmugh9vu287zla247DaixhCNGQBE4A3ry3j/5sVUFHiTPNLUkJS2ayJyK3ArQHl5+YQfv7IpAEBde2jCj62UUtPRM4ebAdhV7WPr8VYuWJif5BGp6a47EuWBHTW8eeM8bDY9gZounjjYwE+ermRPjY9IzPD1G87htevmYszp+OahnTX8/Lnj/G57NZctLeSmC8qZm5NGTrqTstz0JL+CmSkpAbEx5i7gLoCNGzeaiT5+ZZMfgNr24EQfWimlpqWnDzdRkOEiZuDuZyo1IFaj+vOuOj75wB4e21fPf7/5XHLSXcke0qzVGujhsX31vFjVygM7alhQ4OWNG8p448Z5rC49XQK1cm5W75/v2rSAX209ya9eOMmt927v3efChXmcvyCfZcWZXLWyGJdD+yeMRdIX5pgMx+IBcY0GxEqpWSAaMzxzpInNy4uYl5vOdx8/Qp0vqHMo1IhuWF9KMBzlC3/ex6ce3MOPbtqQ7CHNSsea/Lz9Z1upbgvidth456YFfPzaZbgd9hEfV5Tl4SNXLuX9mxfzfGULoXCMww2dPLCjmh88cYSYgbnZHt6woYwLFuYTCkex2wSn3cbOU+1UtwUJR2OsnJPF2nnZzMtNJxwzhMJRBMhOc5Kb7sJmE7p6ItT7QqS7HBRmurEPuKIQCkc52dpFa6CHjmCYmAGP08Zj++o51hTg8mWF5KW7CEdjzM/3UucLsremA4BILEZPxJDpcdARDHOsyY+IMCfbw/ryXB58qYaTrV3c+rKF5KQ72Vvj40RLF3leF4uLMmjoCBEzkOd1cfMF8ynPP7MMeUoGxIlLCpohVkrNBntqfLR1hblsaSGLizL47uNHeL6yhdevS9pUDjUDiAi3XDifbVWtbD/RluzhzDq17UHue+EE//fPE7gcNn733ovYUJ477vIVp93GpUus1YivWlnM+zcvpicS47ljzdy9pZIfPHmU7z1xdNDjCjLciMDvt1cPe2ybgNftoDMU6d2W6XGwam4WjR3d2GxCUaabHSfbCIVjgx6f5rRTUeDl648eGnRfpseBwyY47DZcdhsdoTBpTjtLizOx2YSdp9p5ZG89FfnpnFeRy7f/fhiAfK+LigIve2t9PLK3nsJMN06b0BLo4VXnzKGcaRAQi4gjfkw7YBcRDxAxxkRGfuTEqmwOIAIdoQidoTCZHudUPr1SSk2pZw43IQKXLikkJ81JdpqT54+1akCsxmRhQQZ/2lVLKBzF4xw5K6lGFwpH2VvjwxcMk+ays748t/ffNRYzPLavnnuePc72E22IwNUri/n0K1aecWZzKC6Hjc3Liti8rIjWQA/7azvI8DiIxgzBniir5maR67VKZGrbgxys76CmLYjTbiPNZccYaO/qoTXQgy8YJj/DTVluGsFwlL01HRyo62BZSSaRmKHOF+RNG+exsSKPfK+L7DQr5uoIhVldmk2Wx0ljZ4hI1GAT4XhzgPwMF0uKMkac/GuMoaY9SEmWB4fdxtHGTjxOO6U5ab2PC0djOO22fo85UxOdIf4M8Pk+P98M/Cdw5wQ/z7DaAtZ/4NqybHZV+6jzhTQgVkqltK1VrSwrziQv/gV3wYI8/lXZkuRRqdGISB7wU+BqoBn4pDHmV0Ps5wa+C7wecALPAe81xtRMxDgWFXkxBo43B1gxJ2siDpnyWuOxxqJCb29wFgpH+c7fD/Pz56roiZ7OlnqcNvK9btq6egiGoxgDCwq8fPzaZbxqzdwJDYSHkud1sWlJwbD3z81JY27O5JZXFWV6ev9eku0ZYc/TRKTfBMHFRYPbzPUNhhOPOVMT3XbtTqYw+B1KZbNVP7xpSQG7qn3UtAdZWqy9+pRSqSkSjbHjRBvXrz+dDb5wYT5/299ATXuQ0kn+olNn5YdAD1AMnAs8LCK7jDH7Buz3YeAi4BzAB9wNfB+4fiIGsbAgA7BqWTUg7i/QHaGpsxu300ZxpoeqlgA/ebqSB3fW0BOJsajQSzhqaOrsxiYQ6Ilyw/oyrl1dQlGmm9ZAD08fbqIjGCbP6yLdZWdpSSbXrZ4zqA5XJVfK1RAfi9cPb1pcyA+fPKZ1xEqplHawvpNAT5SNFbm92y6Md5h4/lgLN2zQsonpSES8wA3AamOMH3hWRP4E3AJ8YsDuC4DHjDEN8cf+Bvj2RI1lQbyPbWL+zWxV2x7kqUNNZHgcBLoj7DrVzp921dLVEwXA67ITDEdxOWy8aWMZS4oy+ceBBrI8TuZkewhFoly9soSXLS3sd9zNy4uS8XLUOKVgQOzHaRfWz8/BbhMNiJVSKe3FqlYAzqvI6922vCSTnHQnW4+3akA8fS0FosaYw3227QIuG2LfnwLfFZG5QDtwE/DIRA0kzWXVZSZals4EnaEwNhG8biuM+fXWk9z3wgmWFWdx/fpSLllcwM5T7dy/7RQvVLawrjyXy5YWMj8/nRVzshDgqUNN8S4Ifgoz3Tx1qInuSP9Sh9esncsFC/IJRaIcru8kK83J2y+uoCDDDcDbL65IwqtXkyH1AuJGPxX5XtwOOyVZHmp1cQ6lVArbVtVG6YAaQJtNWFOazb46XxJHpkaRgVX+0JcPGKrG7zBwEqgBosAe4ANDHfRMF75aWOjtvcI6nf19fwM/eOIIu2t8GANZHgfz8tLZV9vBsuJMnjjYwB92VLO8JJOD9Z2kOe1smJ/Lo3vre7spZKc5SXfZqfOFyPI4WDEniwN1nbxyzRzed/kiwDpJSEzmUrNDygXEhxo6WVuWA8DcHI/2IlZKpSxjDC9WtXLRosGLcKyck8XPn6saNAtbTRt+YGDBbhbQOcS+PwY8QD4QAD6OlSG+YOCOZ7rw1aLCDH637RTGmGm77Pc9z1Ty5b8eYHFhBh++Ygluh51TbV0crOvgI1cu4YMvX0I4GuOuLZX8eVctt1+9lH+7ZAFet4PuSJRjjQEqm/08fqCRjmCYz796FVesKNL3hwJSLCAOdEc41RrkTRvmAdbMSe2tqJRKVW1dYRo7u1nTZyWrhBVzsuiJxqhsCrCsRCcWT0OHAYeILDHGHIlvWwsMnFCX2P5pY0wrgIh8H/iCiBQYY5onYjALC70EeqI0dHSPuQvAVNpd3c6XHj7AtatK+O8bzx22PZzdZudDVyzhQ1cs6bfd7bCzcm4WK+dm8apz5k7FkNUMk1KnRYcbrBPrxId/aU4a9b4Q0diErw6tlFJJV+ezroAN1Uki0S1gv5ZNTEvGmADwAFZg6xWRS4DXAvcOsfuLwNtEJFtEnMBtQO1EBcNwutNEolPTdPPz56rwuux8443naK9kNSlSOiAuy00nEjM0dGgdsVIq9SQ+24qHyOgtLPTictg4UDfUFXg1TdwGpAGNwK+B9xlj9onIpSLSNzK9HQgBR4Am4BVYPYknTFGWNUmsxd8zkYedEI2dIf6yu5Y3bpyn6wqoSZNSJROH6v14nDbmxRs5l+VaWZPqtuCkN51WSqmpVu/rBqAka3BA7LTbWFqcwYG6jqkelhqjeAnE64bY/gzWpLvEzy1YnSUmTU66FWi2d02/gPiX/zpBOGq0o4OaVCmVIT7U0NG7Bjb0DYi7kjkspZSaFPW+IDaBwkz3kPevKMlif23HWS1nqmaHnDRrlcO2rnDSxtAdiRLps8IbWGVBdz9znGtXlfT2S1ZqMqRWQFzvZ1mfVekSWeHqNu00oZRKPfUdIQoy3MPOkl85N4uWQA+Nnd1TPDI107gcNjLdDloDyckQ17YHueJbT3PTPS/0C4q/8teDxIzh069ckZRxqdkjZQLi1kAPzf7ufrOpPU47RZluzRArpVJS/SgdAc4ps7pPvHRSu+2o0eV4nUkpmfB1hXnbz7bS2NnNC8db+cGTRwGrs8Sfd9XynssWMS8vfcrHpWaXlAmI99RYM6kHrsNelpumGWKl1LR1qrWLe58/cUZlDfW+4JD1wwlrSnNwO2xsPa4BsRpdbrorKSUTD+2s4Wijn1+84zyuX1fK9x4/wo6Tbfzs2eNkuB28+9IFUz4mNfukTED80sk2RE5nRBLKctM1IFZKTVv3PFPJZx/ay6GG8XeDqPeFRswQuxw21pXnsLWq5YzHt+NkG++9dztNWnaR8nLSXbQlIUP80sk2ijLdXLQon/987SqKszx87P5d/GV3HW/SzhJqiqRMQLzjZDvLijMHvXHKctOobQ9qL2Kl1LT0fGUrAH/ZVTeux3X1ROgIRSgeIUMMcP6CfPbXdtAZGn/mr6Y9yK3/t41H99Xzhb/sH/fj1cySl+5MSkC8q9rHufNyEBEyPU6+/PrVHG8OEDWGd2hnCTVFUiIgjsUMO0+2sa48d9B92otYKTVdtQZ6ejPDD++pG1fZRL3P+kybM8qqYhcsyCNmGPeqnUcb/bz9Z1vpDsd408Yy/ryrlicONozrGGpmyUl30R6Y2pKJ9q4ejjcHWDsvp3fby5cX857LFvLuSxdSnq+1w2pqpERAXNnspyMUYV15zqD7+vYiVkqp6WTrcSs7fMP6Mo43B9g/jp7B9fGT/JFqiAHWlefgsEnvc43FlsNNvPr7z9Ia6OEnb9vAl163hoWFXr7998NjPoaaeXLTXXR2R+iJxEbfeYLsqrbm/6yb1//7+5PXreBTr9DOEmrqpERAvONkOwDrh8wQay9ipdT09MLxFjxOG3dcuwy7TXh0b/2YHzvSKnV9pbscrCnL5rmjY1vl92RLFx/41Q7m56fz1w9dysWLCnA5bNxy4Xz21nSMuNBHdVsXv3juOPe/eIrbf7eLt979PLXtQY40dPL1Rw/yyJ46QuHomF+jmlp53vjiHMGpK5vYebIdEVgzYP6PUlMtJVaqe+lkG1keBwuHaNo9NycNEahq0YBYKTW9vFDZyvryXIqyPCws8HKofuwT6+p8Y8sQA1y5ophvPHZo1El44WiM2361HYC7btnYb9/XnlvKV/56gD9sr+Yzr1o55OM/89BenjrUBECmx0EsZnjDj/+JLxgm0GMFwi9fXsTP3nHe2F6kmlI56dbiHO1dYYoyR/+9mgg7T7WxuDBDJ86ppJvxGeLuSJS/7WvgksUFvSvU9eVx2llSlMHu6vYkjE4ppYYWCkc5UN/Bxoo8ACoKvFS1BMb8+AZfiEyPA6979LzGNauKAfj7/pEz0Pf+6wR7azr42g3nDKrdzPO62LysiId21hCODr6kfqCug6cONfGhly/mqdsvZ/tnruI3t15EMBxlcVEGz96xmQ+9fDFPHGwcdz2zmhq58YB4Khfn2FPj45yyweWOSk21GR8QP7q3npZAD285v3zYfdaX57LjRBsx7TShlJomatqDGAMLCqzAc0GBlxMtXWP+nGrs7B61w0TC4qJMFhZ6eWzf8JPimjq7+c7fD/OypYVcu7pkyH3efN48mv093HT3CxyJTwb82756vvf4Eb752CHSXXbeuWkhFQVeXA4ba8qyefaOl/PAbZdQlpvOey9fRJ7XxXcfPzKmcauplZsomZiiThMt/m6a/T2smJM5+s5KTbIZHxDf9/xJ5uens2lxwbD7rJ+fS0coQmWzfwpHppRSw6uJT/QtzbEC4vn56XRHYr2T5UbT0BGiKNM95ue7ZlUJz1e24Bti4QVjDJ95aA+hSJTPv3olIoOvtoFV7vC1G9ZwsL6Dq/97C6/+/rPceu92vv33wzx+sJEbzysnO73/pW+v24E9fvUu3eXg1pctZMvhJt577/YR65HV1EtkiKdqcY4jjdZ38pJiDYhV8k1oQCwieSLyoIgEROSEiLx1Io8/0Esn29ha1cpbzy8fslwiYcN8a7KdXqZTSk0XNe1WQDw3x8ryLsi35kBUNY+tbGI8GWKw6ogjMcNzxwZPrvv5c1U8tq+BO65dzqLCjGGPISK8+bxynrz9ct572SI6QmE+cuUStn76Cv7n5vV87Oqlo47j3y9ZwAc2L+ZflS3cdM8LNPt1wY/pYqpLJhJXGZYWD/87p9RUmegM8Q+BHqAYuAn4sYismuDnAKxLLe+/bwelOWnceN7w5RIACwu85KQ7NSBWSk0bNW1B7DbpnRRXEZ8UfHwMdcTGGBo7useVIT6nLJs0p31Q+7W/7avnK389wJUrinnnprEtkZuf4eaOa5fz9H9s5iNXLqUo08O1q+eMqZ7Z5bBx+zXL+N17L8IfivDpB/ec0bLVauKluey4HbYpK5k43OAn0+0Y08RQpSbbhAXEIuIFbgA+a4zxG2OeBf4E3HI2xx04ecMXDPPInjrecvfztAR6+MktGwZdohtibGwoz9WAWCk1bdS0BynJ8uCwWx/DJVke3A7bmDLEvmCYnmiMonEEEk67tYzzi1VWQByLGe5/8RS33beD1aXZfPvNa4ctlZgMS4sz+djVS3lsXwOPjKPdnJpceV7XlJVMHG7oZHFxxpT+3ik1nIlsu7YUiBpj+nZu3wVcdjYHXf35xzAG3A4bkZghGO9hWZGfzv/cvIHVpWPrXbixIo/HDzZypKFT65WUUklX0xakNCet92ebTZifnz6mFpENHVaZwXgyxGB9Dv7giSPU+0J84Fc72HaijfMr8rjnHRvJSkLbq3ddupAHX6rhvx45wMuXF+Fx2qd8DKq/nHQXbVNUMnG00c+VK4qn5LmUGs1EBsQZgG/ANh8wKPoUkVuBWwHKy4cvdzDG8IHNiwn0ROmORHHYhMJMN0uLM7l0SWHvRI2xePN58/jRk0f52qOHuOftG8f8OKVU6mnq7CY7zYnLkbx5xTXtQc5fkNdvW0W+l+NjyBA3dloT78YbEJ9fYS3jfNt929lxsp3/un4Nb944b8Q5GJPJbhM+88qV3PzTF7j5nhc41NDJh69YwrsuXZiU8SjITXfSNgUlEy3+bloCPSzR+mE1TUxkQOwHsgZsywIGdZo3xtwF3AWwcePGYYvHRIQPXrFkQgaX53Xx3ssX8Y3HDvFCZQsXLMyfkOMqpWaWHSfbeMtdz/OhK5bw/s2LkzKGSNTqJtE3QwxWHfFTh5uIxcyIQWoiQzyeSXVgLeNstwk7TrbzmrVzR2xXOVU2LSng6pXFPH24iYWFGXzp4QO4HDbedlFFsoc2K2WnOXtXQZxMhxusDhNL9YqtmiYmMj1yGHCISN8Idi2wbwKf46z8+yULmJvt4eN/2I0vODU1Ukqp6eNUaxfv/t9tdEdiVMfbniVDQ2c30ZihNLd/QLygwEtPJNbbgWI4vRnirPFliL1uB6vmZuGy2/iPa5aNb9CT6AdvXc+Oz17FH99/CVcsL+Jzf9zHe+7dRku8A8XRRj+docn7zK5qDnD1d54etIDTbOxd73U76OqZ/OW1D9VbLfc0Q6ymiwkLiI0xAeAB4Asi4hWRS4DXAvdO1HOcrTSXne+9ZR01bUE+dv8undms1Cxz3wsn6QiFKchw4QtO3WpcA53uQdw/IF5eYmXL9o/Sn7exo5tMt4N01/gv8n36FSv43lvWMS8vffSdp4jLYcPrduBy2PjJLRv4xHXLefJQE7f/bhdHGjp5xXef4d9/8WJvgGqMobpt9FrrsfrttlMcbvDz8d/v7p3Ifc8zldx67zZ6IoNX5UtlGW4H/u7IpD/Ps0dbKMtN0w4TatqY6AK624A0oBH4NfA+Y8y0yRCDNankE9ct5x8HGnj6cFOyh6OUmkK17UHmZKdRke+lLZC8q0Q17VYwNzBDvLwkC5vA/tpRAuLOEIXjzA4nXLAwf9iV6KYDh93Gey9bxB3XWkHxzT99AYPhxao2frX1JAB3P1PJpq89yV/31A17nLZAD2//2VZ+++JJ2rt6+NCvX+KuLccGZX1jMcMfX6qhNCeNg/WdfOqBPXz1kYN86eEDOO0zfu2qcUt32enqiU5qwqg7EuW5o81sXlakHSbUtDGRNcQYY1qB103kMSfD2y6q4CdbKvn5c1Vcvqwo2cNRSk2R+o4QJVkestKco5YlTKbhMuCDSSIAACAASURBVMRpLjsLCzPYN1pAPM4exDPROy6u4I87a9hd7eNbb1zLAy9V819/PUAoHOWbf7OaGX3hz/t52dJC0p127t92io5QmHduWkjMGN7/qx3881gLTx9u4pt/O0yzv5s/7arl/m3VRGOG+fnp3HjePDLcTmp9Ib73lnX861gLv44H3a9ZO5dvv2ltb1u82cLrdhCNGbojsUnr+rH1eCvBcJTNywsn5fhKnYkJDYhnCpfDxtsunM+3/n6Yo42dLC7Son41e8Xr/vcAvzfG3Jzs8Uymel+ItfNycDts7K8d2BRn6tS0h8j3uoYMOFbOyRq1Z3pDZ4j15bmTNbxpwW4TfvjW9fzrWAvXry/lokX5vO+X2/nSwwfISXfyg7es49Z7t3PTPS9gF9hx0qr/ffxAI8FwlN3VPr56/Rq2nWjjiYON/ObdF3KipYvfbT9FQYabl062895f7sBuE7wuO1etKOY1a+fyiWuX09AZYnFhRtK6byRTRnxxFX93ZNIC4qcONeFy2LhoYcGkHF+pMzErA2KAt15QzvefPMr//vMEX3zd6mQPR6lk+iHwYrIHMdmMMdR3hLgmy40xTNniA0OpbQ8OKpdIWDU3iz/tqqW9q4ec+FK6fZ3JKnUz1by89N5a57k5aTxw2yX8btspFhVlcF5FHp+8bjkPvlRDdyTGV16/BoAv/mU/8/PT+a/r13Dj+eXceH450ZjBbhMuWJjPm86bB0A0ZnjiYCO/3nqSDfNzSXNZwV92unPUxZ5SWWK1wa7uqNVMdRI8eaiRCxfm9/6bKzUdzNqAOD/DzdUri3lkbx13vmbVuHoaK5UqRORGoB34J5CcHmRTpL0rTE8kRkl2GqFwlGA4SigcTcpiELXtQRYVDh1trJxrda/cX9vBxYsHZ9A6ghG6I7Fxt1xLBXabcGOfVnHvuWwR77lsUb99bjxvcF/loT7f7TbhqpXFXLVSF4boyxsPUidrYl1te5DKpgBvnQYt/5Tqa3YVRw1w9aoSmv09vHRSl3RWs4+IZAFfAD6W7LFMhfp4b9WSLA/ZaVYGsCMJ7ReNMdS2B5mbM3SGeOWceEA8TKeJWp9Vf1ySPfsC4rGYjWUOEymRIQ70TE5A/MLxFgAuWqRrAajpZVYHxJuXFeK0C3/b35DsoSiVDF8EfmqMOTXSTiJyq4hsE5FtTU0ztzNLvS8eEGe7yYlfEk9G2URHMEKgJ8rcnKED2vwMNyVZnmEn1p1osVayq8j3TtoY1ezVGxBPUob4+WOtZKc5WVEycB0vpZJrVgfEmR4nFy8q4LF99dqTWM0qInIucCXwndH2NcbcZYzZaIzZWFg4c2eFJzLExVkectKs2tz2KViidqBEd4uBHSb6Wj4nk4P1gxb5BKCqxWrZVp4/ffoIq9ThdVslE4HuyVmc4/njLZy/IE8z+WramdUBMcDVq4o50dLF0UZ/soei1FS6HKgATopIPXA7cIOI7EjmoCZTvS+ECBRlenozxO1jLJl48lAjd2+pnJBx1MYD4uFKJgCWlWRyrNHfu0hEXydaAuR7XWR5Zu/ELzV5vK7JyxDXtgc50dLFhQu1XEJNP7M+IN4Un7Sytao1ySNRakrdBSwCzo3f/gd4GLgmmYOaTA0dIfK9blwO2+mAeIwZ4m88eoivPHKAU61nvzpazRgC4uUlmfREY1Q1BwbdV9XcxXzNDqtJkjGJNcSJ+uELF+ZN+LGVOluzPiAuz0sn3+tix4n20XdWKkUYY7qMMfWJG+AHQsaYGVUk7AuGOdLQSVNn96j71neEKMm2WpUl2pm1j6GG+ERLgP11HRgD928bsdx6kHueqeSeZ/pnlmvbg7gcNvK9g1uqJSwttnqjD1U2caIloPXDatKk95ZMTGxA3BOJcdeW4xRnuVmu9cNqGpq1bdcSRIT183PZoZ0m1CxmjLkz2WM4E2+563n213VgE3j6Pzb39qwdSr0vRFm896/XZcdhkzGVTDyytx6A1aVZ/PbFU3z4iiVjWr2sqjnAVx85iNft4N8uWdDb+qumPcjcbM+INZSLizKw24RD9Z28eu3p7aFwlFpfiPkaEKtJ4nbYcdoF/wTUEO881U5VcwAR2H6ijQN1Hdz9to3a5lRNS7M+QwywvjyX480BWgNTP8FGKXXmqloCLCr0EjNwfIjygr6sDLHV2UFEyEl3jalk4pE9dawty+bDVyylsbObh/fUARCLjTwR95t/O0QkZvAFw+yuPn0FaqSWawluh50FBd5BGeJEyUZFgZZMqMnjdTvOOkP8+IEGXvfD5/jIb3fy4d/s5P/+dYI3bCjTvs9q2tKAGNgw31oCdccoy6UqpaaPQHeErp4oF8Qn6DTEu0gMJRSO0t4VpqTPYhY56c5RSybqfEF2Vfu4dvUcNi8rZHVpFp/74z5+v72ajV/+B595aA+xmOHRvXX9AvIdJ9v4y+46br6wHBHYcri5977a9tCoATFYE+sON/QPiBMdJjRDrCaT1+U4qxricDTGl/96gIUFXv7x0cv42/97Gfe+83y+pKvCqmlMA2LgnLJsHDbRsgmlZpBE3fCq+MpujSPUEfvipRF9l0LOSRs9IN563Jpse9nSQhx2Gz9863piMcPtv9sFwC+fP8nmbz3Fe3+5g/fcu41INEawJ8rt9+9ibraHO65dzjml2TxzxCrNDkdjNHSOLSBeXpzJydaufpm60z2INUOsJo/XbT+rDPFvXjxFZVOAT1y3nMVFGSwtzuTSJYVJWRVSqbHSgBjwOO2smpvFds0QKzVjNPmtAHhebjrZac4RM8SdISvwzfScnjaRk+4ctYZ4+4k2vC47y0qsSW7z87386Ob1vOPiCrZ8fDPv3LSA5s5ubjxvHocb/PxkSyWffGA3lc0BvvnGtWR6nLxsaSEvnWqnIxTmREsXxsC83LFliAEO1p9eoKOqJUB2mrNfYK/URLNKJs68hvi+50+wdl6OlkeoGWXWT6pLWFeey29fPEU4GsM5hgkzSqnkSmSICzPdFGe5RwmIrWxX3969OemuYVeD8wXDZKc52X6ijXXluf0mAV26pJBLl1gLlHz2VSv55HXLsduE6rYg33jsEAAffPliLo63dLx0SSHff+Iozx1ppiMemK8rzx319a2dlwPAzlM+NszPIxYzPHOkuTcjrtRkyXA78J9hhrjeF+JgfSd3XLscEZ08p2YOjfziNszPJRiOcrBu6NWhlFLTS/+A2ENDx/AlE4mAuF+GeJiSid3V7az7wt94ZE8dB+o6WD9/5ODVYbchInzl9Wu4+cJy/vLBTXzs6mW9968vzyHT4+Dpw028WNVGbrqTRYWj1wAXZ3mYk+1h1ylrQt7zx1s40dLFGzeWjfpYpc5GuuvMSya2HLbKgy5fNnNXtVSzk2aI4xJfejtOtrGmLDvJo1FKjaapsxu7TchLd1GU6eFYY/Ow+yYC4owBJRPBcJRQONqvtnHr8VZiBj7++93EzOlJt6Mpz0/nS69bM2i7w25j0+ICnjrUhNtpY2NF3pgzZ2vLctgZD4h/++IpMj0Orls9Z0yPVepMnU3JxFOHGynJ8rA8XvKj1EyhGeK4udkeSrI8Wkes1AzR1NlNQYYLm00oznLT2Nk9bCu00zXEp0smCjOtRToaB2SW99b4rMd0RxCBdeU5Zz3Wy5cVUt8R4kRLF+dVjC3ABji3PIeTrV0ca/LzyN56Xr+uVCcmpRARyRORB0UkICInROStI+y7XkS2iIhfRBpE5MOTNa4M95l1mYhEYzxzpJnLlhZquYSacTRDHGct0JGjAbFSM0STv7s3qC3O8hCJGVq7eijIcA/ad6iSiUTrsuMtAcr7dG3YV9vB5csKqWkL4rTb+tUdn6nLlhb1/n1jxdiXrT03Xkf8/vt2EInGuOXC+Wc9FjWt/BDoAYqxllB/WER2GWP29d1JRAqAR4H/B/wecAGTVjuT7jqzPsR7anx0hiK8bKmWS6iZRzPEfawvz6WmPTji5Byl1PTQ1NlNYUYiILb+HO692xn/cs9wnQ6IFxRYAXFVn/7BwZ4ox5r8nFOazS/fdQF3vW3DhIy1JNu6hOx22Fg9d+wlWWtKs7GJtYTzzRfOZ0mxXoZOFSLiBW4APmuM8RtjngX+BNwyxO4fBR4zxtxnjOk2xnQaYw5M1tgy3HbCUUN3ZHxlE0ca/YC1qqNSM40GxH0k6ohf0n7ESk17TZ2nM8RF8QU3BpY/JHSGwmS4Hf2WSy7KdJPusvdbUONAfQcxA6tKsynO8lCWO3H9fj9y5VJuv3oZLsfYP3a9bgdLizPJTXfy0auWTthY1LSwFIgaYw732bYLWDXEvhcCrSLyTxFpFJE/i0j5UAcVkVtFZJuIbGtqajqjgXnd1olj1zjriKuaAzhsQukY+mwrNd1oyUQfK+dk4bAJu+MrUymlpqdYzNA8oGQCrOWZh9IZivQrlwCrTGp+vrd3sQugtw3bZLQ2u3Z1yRk97ptvXEvMGO09nHoyAN+AbT5gqMsAZcB64CpgD/B14NfAJQN3NMbcBdwFsHHjxpHXFx+GN34lxd8dIdc79t+7qpYA8/LScWjrUjUDTchvrYh8IH5G2i0iv5iIYyaDx2k14N9TM/AzSik1nbQHw0RiprdkIvHnsCUTofCggBhgQUF673LIAPtqfOSkO6dVhmt1aTbnlJ39xD417fiBgWdeWcBQvT+DwIPGmBeNMSHgP4GLRWRSWiIlMsTjnVh3vLlLV1FUM9ZEncbVAl8CfjZBx0uac8qy2V3tw5gzOrFWSk2B0z2Ircywy2Ej3+sathexvzvSr8NEQkW+l1OtXUSiMQB2V/tYNTdLZ8irqXAYcIjIkj7b1gL7hth3N9D3Synx90n5RfW6rU4m42m9ZozhREuAioLRe2wrNR1NSEBsjHnAGPMQ0DIRx0umNaU5+IJhTrUGkz0UpdQw+i7KkVCU5aFxhJKJDPfgDHFFgZdIzFDdFqQt0MOB+g7Or8ifnEEr1YcxJgA8AHxBRLwicgnwWuDeIXb/OfB6ETlXRJzAZ4FnjTHtkzG23gzxODpNNHZ209UT7Z2sqtRMk5RCn4ko+p8s58QX5dhdM/hzpjsSxTfEylZKqanV5LcC374BcWGmmyb/cJPqBtcQQ59OEy0Bnq9swRi4ZLEGxGrK3AakAY1YNcHvM8bsE5FLRcSf2MkY8wTwKeDh+L6LgWF7Fp+tRA3xeALixOTURDtDpWaapEyqm4ii/8mytDgTl93GnmofrzpnLgDRmOHjv9/NX/fU4bALL376Sm2Or1QStQWsE9Pc9D4LbWS4OdIw9NLrVg3x0CUTYM2OP9rkJ91lZ+08rddVU8MY0wq8bojtz2BNuuu77cfAj6diXL0lEz1jL5lITE5doAGxmqFGzRCLyFMiYoa5PTsVg5xKLoeNFXMye5dLBXh0bz1/2FHNspJMOkMRjjb6RziCUmqyJTJX3j5lEIWZbpr93UPW/3eEImQNkSEuyHCR4XbwYlUb/zzawvkL8nDqDHk1y6W5rIA4FB57QHy8uQunXZib45msYSk1qUb95DfGXG6MkWFum6ZikFPtZUsL2VrVyr5aa3Ldj546ysICL19/wzkAHKofOgullJoa/u4IboetX/BamOkmHDX4gv3LmrojUXoisSFriEWEmy4o5+E9dVQ2B7hkUcGkj12p6S7NOf6AuKpZW66pmW2i2q45RMQD2AG7iHhEZMb2OH7XpQvJSXPy5YcP8Pvt1eyr7eC9ly1iYYEXl93G4WEuyyqlpoa/e/AkuYIMq19qYsJd775DLNvc1x3XLuf69aXYBF1yVinoLQkMjqNk4mRrF/PztOWamrkm6lTuM1h9Ej8B3Bz/+2cm6NhTLjvNyYevWMI/j7XwH7/fzaJCL69bV4rDbmNRUQYHNUOsVFL5uyP9yiXg9AS7gQFxZ29APLiGGMBmE775hrVs+fhmlpXo0shKOe02nHYhOI4McX1HiJLs6dO/W6nxmpAsrjHmTuDOiTjWdHHThfNpCfSwpDiTq1cW9y63urwkk+crZ3x3OaVmtMAQGeKiREDsHy4gHv7jzmaTCV2mWamZzuO0jzkg7o5EaQ30UJKl9cNq5pqxZQ2TzWm38bGrlw3avrQ4kwdfqsEXDJOdNnTGSSk1uYYqmSjMsL6MB2WIu62a4owRAmKlVH9pTvuYa4gb4wvilGS7R9lTqelLq9/HaXn8kqrWESuVPP7uyKAANyvNgctuGzZDnDVMyYRSarA0l33MNcT18QVxijVDrGYwDYjHaWk8INZOE0olT6A7OqiGWEQoyHDR3NnTb/tYSiaUUv2ljaNkot5nBcQl2RoQq5lLA+JxmpvtIdPtGHYBAKXU5LOWYh68OM5Qq9V1hqySieEm1SmlBrNqiGNj2rchniHWGmI1k2lAPE4iQnl+Oidau5I9FKVmraEm1UE8IB6my8RQ+yulhpbmtBMaa8mEL4TbYdN5NWpG04D4DMzPT+dkiwbESiVDNGYIhgeXTAAUZFir1fWVWMQj0SlGKTW6NNc4SiY6QpRkexCRSR6VUpNHvyHOwLy8dKrbgkRjg5eIVUpNLn/38Bnfwkw3Lf7ufu/NzlBYyyWUGqfx1BA3dIR0Qp2a8TQgPgPz87z0RGO9M2uVUlMnMEpAHDPQGjg9sa4tECY3XQNipcbD4xxflwmtH1YznQbEZ2B+vtXA/0RLIMkjUWr2SWSIhyqZKMwYvFpda1cPuV7X1AxOqRSR5rKNqQ+xMYaGjm7maIcJNcNpQHwGyuPrtZ/SiXVKTbnRSiag/2p1bYEe8tI1IFZqPMZaMtHWFaYnEtOSCTXjaUB8BuZke3DYhBM6sU6pKddbMjFEX+HEl3KD73Q5U1tXmFyvlkwoNR6JgNiYkefKaA9ilSo0ID4DDruN0tw0bb2mVBL4423UvK7BAXFRlpUhTtT3G2No6+ohVzPESo2Lx2XHGOiOjNyLuEFXqVMpQgPiM1Sel64lE0olQaJkYqiV59wOO/leF3XxrFVHKEI0ZsjTGmKlxiXNaS18M1odcUt8AmtBhr7H1MymAfEZmp+friUTSiXBSJPqwMpUJbJWbfEva80QKzU+iYB4tDpiXQlSpQoNiM9QeV46vmAYX1c42UNRalYJ9AbEg5duBqvGP1HX2NZlBcSaIVZqfNJc8YB4lNZriZUgh7pio9RMogHxGertNNGmWWKlppK/O4rLbsPtGDogLs729NYQJwLiHO1DrNS4eMaRIfY4bTjtGk6omU1/g89QaY4VEFe3BZM8EqVmF393eNjsMMCcLA+tgR66I1FaA9YVHM0QKzU+Y60h7gxFtFxCpQS9xnGGynLTAKgeIkP8l921PLKnnvL8dO64dvlUD02plBbojg5bPwxWhhigsaOb9niGWBfmUGp8TpdMjNxlwgqINZRQM5/+Fp+hnHQnXpedmvb+GeLnK1v4wK9e6u3h+IYNZSwqzEjSKJVKPZ2hyJCLciQklpCt84VoDfTgsAmZI+yvlBpsrJPqOkJhzRCrlKAlE2dIRCjLTR9UMvHPo83YBB79yKW4HDZ+9uzxJI1QqdQU6B45IE4sIVvfEaKtq4ecdBciMlXDUyoljL2GOEKWZohVCtCA+CyU5qYNCohfON7K6tJs5ud7uX5dKX/YUU1rvPWTUursBXoiQ65Sl5AomWjwhWgLhMnTVeqUGrdEyURo1C4TYS2ZUCnhrANiEXGLyE9F5ISIdIrISyJy3UQMbrory03rV0McCkd56VQ751fkAfBvlywgFI7x8J66ZA1RqZTjD0VGrCHOdDtId9mtkgldpU6pMzL2PsQRMt160qlmvonIEDuAU8BlQDbwWeB+EamYgGNPa2W5aXSGIviC1kz23dU+eiIxLliYD8DS4gyKMt1sq2pN5jCVGmQmn8j6uyNkDLFsc4KIUJJtLc7RFujRDhNKnYFxBcSaIVYp4KwDYmNMwBhzpzGmyhgTM8b8BTgObDj74U1vZblW67WaeNnEC5UtiMB5FbmA9cV8XkUe26rakjZGpYYxrU5k733+BHdvqRzTvv7ukUsmwJpYd7K1i7ausHaYUOoMuB1WeDDSwhzhaIxgOKqT6lRKmPAaYhEpBpYC+0bY51YR2SYi25qamiZ6CFNmYOu1rVWtLCvOJKfPJdoN83OpaQ9S59N+xWr6mG4nsr/bdoov//UA/zzaPOJ+oXCUrp4ouaMstHH5skL21Pho9nePuq9SajCbTfA4bSP2IfbrKnUqhUxoQCwiTuA+4H+NMQeH288Yc5cxZqMxZmNhYeFEDmFKleYkAuIgsZhh56l21s/P7bfPxni2eFtVGx2hMMaYCXv+mvbgqE3TlRqLkU5kp+IEtj2+BPp//H43/vjSzEM5vRSze8Tjve2iCublWe9PrSFW6swk2ocOR5dtVqlk1IBYRJ4SETPM7dk++9mAe4Ee4AOTOOZpI8/rIs1pp7otSGVzgM5QhHPn5fTbZ+WcLNJddu55ppKNX/wHP3zy6IQ8d7AnyjXf2cJXHxn2vEOpMRntRHYqTmDbunpYMSeLmvYgTx5sHHa/RMeW0TpHeJx2PnndCgCK432JlVLjk+a0j1gy0RGyTmS1ZEKlglEDYmPM5cYYGea2CUCsJp8/BYqBG4wx4Uke97QgIiws9LK3xseuU+0AgwJih93GufNy2FXtI2YMP9lS2TsJ72w8f7wFf3eEB3ZUa5ZYnbHpcCIbicboDEW4fFkhNoHDDZ3D7tsWX4p5LFnf61aX8JtbL+TqVcUTNlalZhOPa2wZYu1DrFLBRJVM/BhYAbzaGDOrimWvXFHMiyda+fv+BjLcjiFXpXv7xRXcsL6MX737QjpDEf73n1Vn/bxbDluXrjtCEf6+v2Fcjz3a6KcnMvJynCr1TZcT2Y74l2pxppuKAu+IAXFLoBtgTJ0jRIQLF+bjdtgnZqBKzTJpTvuICZdOzRCrFDIRfYjnA+8BzgXqRcQfv9101qObAV55zhyMgUf31bOmNBu7bfCKWNesKuFbb1rL+QvyuGplMfc8U0mzv7v3fmMMsdjg2mJjzLA1x1sON3HpkgJKc9K4f9upMY/3aKOfq7/zNO+5dxuRqAbFs9y0OJFtj9cF53pdLC3K5HCDf9h923pLJrQuWKnJpjXEajaZiLZrJ+LlEx5jTEaf230TMcDpbmlxJkuKrKzwueU5o+wNd1y7jGA4yhf/sh+wJsZd9Z0tnPflf/DR3+7k8QMN9ERidIbC3PLTrVzz31vYU+3rd4ya9iDHmgJctrSQ69eX8uzR5jGXYfxpZw0xA08eauJzfxq2EYhKcdPpRLYtPqEuO83J0uIMTrQEhs1KtXaFEbH2VUpNrjTXyDXEpzPEGhCrmU9/iyfAK9bM4buPH2Ft2egB8eKiTG67fDHfffwI6S4H/zrWTIu/h83Li3jiUCMPvFRDdpqTnHQnNW1Bcr0uXv+j5/j8a1Zxy4XzgdPlEi9bWkhlUwBj4FRrF9ml2SM+tzGGh3bWsmlxASvnZnHXlkquX1fKxvjKemr2MMacAAZfzkgCX9DK+uaku1hakknMwLEmP6vmDv59bgv0kJ3mxGHXVeeVmmwep51mf8+w95/OEOsJqpr5NCCeAG+9oJx6X4hNSwrGtP9tmxexq7qdh16qIc1l5xf/fj4b5ufSE4nxzJEmHt5dx/66Dn5yywY2zM/lo/fv4rMP7aWmLcgnrlvOAzuqmZ+fzpKijN5a4FOtXaweJSDeeaqdk61dfPDli3nlOXN48KUavvboQe5/z0VY5aRKTb1Ey7WcNCfprkzAmlg3VEDcqivPKTVlRq0h7o7gdthwOfQEVc18GhBPgOIsD197wzlj3t/tsPOLfzs/XiNsNUAHcDlsXLGimCtW9J8Vf/fbNvLpB/fwP08foyTLzYtVbXzmlSsQEeblWavlnYovDjKSP+2qxeWwcc3qEtJdDj50xRI++9BenjrUxOblReN4xUpNnERAnJvuIs1lx2GTYeuIWwM95GlfYaWmRJrTTlfP8H3BO0NhzQ6rlKGndUkkIr3B8EjsNuHzr17F3GwPd/55Px6njTdumAdYtZSZHgfVbaPPiXr6cBMXL8onK/4BduN588hOc/LYvvqzeyFKnYX2oFUXnOlx4HLYWFjo5cgwnSbaunp0KWaVMkQkT0QeFJGAiJwQkbeOsr9LRA6KSPVUjC/dbaere6Q+xBFtuaZShgbEM0Say86nX7kSgNedW0p2n+Vo5+Wmc6p15AxxbXuQyqYAmxafLutwxnsk74z3UFYqGdq7rLrgxMnh4qIMjjUFhtxXM8QqxfwQqwd4MXAT8GMRWTXC/v8BDL9yzQTLcDvw90SG7XbUGYrohDqVMjQgnkFesaaEb7zhHD569dJ+2+flpXFqlAzxs0ebAQbVOa+dl8Phhk4CIyyXq9Rkau8Kk9Ona0RpThq17cFBX8LGGNq6esjL0IBYzXwi4gVuAD5rjPEbY54F/gTcMsz+C4Cbgf+aqjF63Q6Mga5hOk1oyYRKJRoQzyAiwhs3zqMos/9StGW56VS3dQ17Fg/w3NFmCjLcLCvO7Ld93bwcYgb21PiGeaRSk6s9GCa7T9Z3bk4a3ZFY7zLNCZ3dEcJRoxlilSqWAlFjzOE+23YBw2WIvw98Chgx+yEit4rINhHZ1tTUdFYDzHBb2d/hEiadoQhZaZohVqlBA+IUMC83jVA4Nmx7nFjM8NzRZjYtzh/UTWJtfKnpXVo2oZLE19VDbp8SoDnZaQDUtof67ZdYlENriFWKyAAGZiJ8QObAHUXk9YDDGPPgaAc1xtxljNlojNlYWFh4dgOMB8SdwwTE/lCkdx+lZjoNiFNAWe7InSYONXTS7O/hksWD28LleV2U56VrHbFKmrYBJRNzc6wrILW+/omwRMY4XwNilRr8QNaAbVlAvxml8dKKrwMfnKJx9RotQ+zvjpDh1pIJlRo0IE4BidZrw3WaeG6Y+uEEPT+QdAAAFo1JREFUnVinkqm9q4ecASUTAHXtQwfEmiFWKeIw4BCRJX22rQUGLiG6BKgAnhGReuABYI6I1ItIxWQO0BsPiP2hwQFxLGasgFgn1akUoQFxCijLtQKI4TpNPHu0mUWF3t5L0QOtLs2izheivWv4FYmUmgzRmKEjFOm3FHO+14XLYaPW179kIhEQaw2xSgXGmABWcPsFEfGKyCXAa4F7B+y6F5iHtcz6ucC7gIb4309N5hgTGWL/EBnirviCHZlaMqFShAbEKcDrdpDndQ0ZEPdEYrxQ2dqv3dpAiZKLgTWbSk22jmBiUY7TAbGIMDfbQ+2ADHFbVyJDrJdoVcq4DUjDaqX2a+B9xph9InKpiPgBjDERY0x94ga0ArH4z8M3CZ4AiexvYIjFORJZY80Qq1Shv8kpYkGBl8rmwb1bd5xsIxiOsmnJ8JMrei9R+4KsnDuwpE2pyZMIcnMGZH3nZKdRNyBDXNMWJMPt0Ek8KmUYY1qB1w2x/RmsSXdDPeYpoGxyR2bxuu3A0CUT/u5wfB99P6rUoBniFLGo0EvlEIsZPHe0GbtNuGBh3rCPnZsdn8TUPvpqd0pNpPZ4hrjvQjNgnaQN/H082uRnUaF3UKcUpdTkyIxPmPMPsVpdZzxI1pIJlSo0IE4RCwszaPZ344sHGAlbj7eyujS7d7nmoRRkuHHaZVDNplKTLVG33rfLBFidJho6QkSisd5txxoDLCocMmmmlJoEHqcNmwzdZSIQD5K1ZEKlCg2IU0QiUKhs8vduM8awv66DNaUjl0HYbELJEDWbSk22RFlESXb/xWbmZKcRM9DY2Q1Yk3rqO0IsKtKAWKmpIiJ43Y4hJ9X1lky4NCBWqUED4hSxsNAL0K9sorotSGcowoo5o9cFz8lOo04n1akpVtcewm6TQasv9vYijp+kHWu0TvQ0Q6zU1MocJiDuLZnQDLFKEfqbnCLK89Jx2IRjfTLEB+o6AMYUEJfmpPFiVeukjW8m6InE+OojB9l+ohWv28HXbjiHOl+IP++qRQTeuGEea8qyh318JBrjSKOfOl+QixcV4HHap3D0M1OtL0hxphu7rX9dcGKiZ017kI3Q+3u9uMg71UNUalbzuh1DTqpLlFHoJFeVKvQ3OUU47TbK89P7ZYgP1HUiAstLBq0EOsicbA/1vhDRmBkUnKSS6rYu3A47hZlujjcH+Nmzx3nyUCOfeeVKjjR08rPnjnPxonz21vi47rvP4O+O4HXZ/397dx5ddXnncfz9vbnZ90A2AgkKgSAIiGjdaEGo2rpVcQWdOtNWj9baxdbac6ZqnbZ2OT3T1rq0dZwqbadOHRdcOnYclxZFRiiCBSEGBFmSkBCy3IQkJPeZP24SE7LdhEvuks/rnHuO+f3u75cnj883fO+T5/d96HSO5zdXsupzp/Od1Vvp8Pu57JQitlc3kZYYz5KZudy9egvbqgKbTN26ZDpfP39mmH/ayFdZ30phVv/62FMnpJIU72HTngYunV/EjhofXo9RMkEJschYSkvyDlx2rSshVpUJiRUayTFkWm5anxnirZUNTJ2QSkoQa7wKs5Lp8DtqfW3kZyQN+/5otPtgMxffv4aUBC8/v/YUbv7tBnxtHUxMS+S2P2wE4MK5hTywYgE7a3zc8eRmFpRk85Vlpeyvb+WSX6zhovvXkBDnoSAziW8/u4XUhDhaO/w8/PoOclIT+MHlJ/PSlioeW7uLT59cyBceX09ivIePl+byjfNn6h+Po1Q2HGZOUf9Z9wSvh/lTsnr+alFxwEfxhBTi47TKS2QspQ22ZKKtgwSvhwSvYlJig/51jiEn5qby+vaanlne9yqbmDPMA3XdirrWbO6rPxyTCfHug83ctGoDZoavrYOrfrmWzOR4XrhtETmpCVzx0JvU+tq4++KTgEDVjidvPqvn+ul5afz4inn8+KVt/OiKeZxaks3OGh8lE1Kpamjluc37Wb5gMgWZScwpyuSi+9dw2YNvkOj1UFaQw+Nrd7Hugzr+7bMLe5YDjHfOOSobWjlvdsGA50+bmsMDr1bga+tgR40qTIiEQ2qCl6oBKhD5WjtUck1iSkhGs5n9FlgKpAJVwI+cc4+E4t4SvGm5abR3+vmg1kd+RhIf1rVw5anB1W/v3ta5sr4Vikf3/Z1zbNnfyKSsZHJSI2N73YbDR7hp1Xre2lmH12M8esNpmMHdq7fwvc+czPSuqgXP3no2vraOfg939Xbh3EIunFvY83VpfmApSvGEFL64ZHrP8TlFmZxblsfr5TU8snIBi0pzeXX7Ab70+4389OVyfnTFvOP000aXuuZ22jr8FGYO3OenTc3B7+Av5TXsPtjMsln5Y9xCEUlL8g5Sdq1DJdckpoRqNN8HfM4512ZmZcBrZrbRObchRPeXIJx54gQAXtte07PjXDAP1EHf3epGau2Og7xRUcvL71WzraqJzOR47rroJJYHmYyHWqffce9zWzAzNu6pZ+v+Bu78VBkXzS3s2ab6ldsX97kmPSme9CFqNY/Uv149n72HWpg9KbAcYMnMPJ754lkUZGp2uFt3ybXCQfpkQUk2HoM7/2sznX7HRb0+jIjI2EhL9NI0YNm1DpVck5gSktHsnNvS+8uu1zRACfEYmpKTQllBOi+/V015dRMpCXGcMW1CUNdmJHlJTYhj3whrEdc1t7PykbcwM+ZMyuA7l8zm+c37uf2Pm2hp7+D6M6cGdZ9Ov+PFdyv5sK6FWxZPO6bdyB57cxePrd1NQpwHh+MXKxZw/iB/lj9eMpPjyUzuuzZ2et7wDzeOJ90l1bpLrB0tLdHLSZMy+Pu+Rm44a+qAa41F5PhKSwzMEDvn+vxebmrVDLHElpCNZjN7ELgBSAY2Ai+G6t4SvGWz8nno9R1s3tvARXMLgy6JY2ZMykpm36GRJcRv7TyI38GTN53BwqmB7aGvO6OEm1at5+7VW5ick8KSmXlD3qPigI9bf/+3ngoNmcnxXHdGyYja0e3Dgy38+KXtLJmZy8PXn0p7hz+kM78SOt0J8WAzxABLy/JpOHyE28+bMVbNEpFeUhO9+B20HvGTnPBRKcnm9g7yh1hiJhJtQvZ4qHPuFiAdWAQ8BbQN9l4zu9HM1pvZ+pqamlA1QYCls/Lo9Dta2ju5auGUEV07OTt5xDPEa3ccJDUhjnlTsnqOxXmMn11zCtPz0rj3ua10+t2g16/beZDLHniDmqY27r/2FBaVTuT7L77HV594h6t+uZam1iODXtubc443K2pZ/vCbeD3G9y47mURvnJLhCFbZ0EpCnIcJQ6w3/8qyUl69fbH+P4qESVpiIAluauv7u9jX2qGqORJThk2Izew1M3ODvNb0fq9zrtM5twaYDNw82D2dc79yzi10zi3Mzc099p9CesybnEVueiIn5qZyakn2iK4tGkVC/OaOWk47IadfOazURC9fXjqDD2qb+Z+tVYNef9+ftpGZEs/qL53DxfMm8cPlc/F6jJe3VvN/H9TxxNt7hm3Dc5v2c/YPXmHFI+tIT/Tyx5vPVCWHKLC/oZXCrCQ8Q9S9NjO8KrUmEjbdyyKa2zr7HPfpoTqJMcOOZufc4lHed9oorpNj5PEYD61cQHJC3IjX4RZlpVDfciTwiy6IT/7Vja3sqGnm6tMGnom+YE4BxTkpPPz6Ts6fXdCvPeXVTbyzp55/vnAWRV0J7KSsZP56x7kkJXi47pF1/Psbu7jhrKmDJkUHGlv51lPvUpyTwhfPnc6l84u0c1KUqKw/PGiFCRGJDN0Pzh29W12Tyq5JjDnmqRczyzOza8wszczizOx84FrglWNvnozGwqk5PdUNRqIou2u73CDXEb+18yAAZ544ccDzcR7jC4tO4J099fzkz+U413fpxBNv7yE+zrjslKI+xzNT4kn0xvH5RSeyr/4wX/vPTdz17N+pONDEmvdr+fxjb/Pu3gYAvv/ie7R3+Hlw5QJWfqxEyXAU2V3X0lP1Q0QiU/cscO/NOY50+mnr8Ov3rcSUUIxmR2B5xMMEEuzdwFecc8+G4N4yhiZ3J8T1LcwMYrvn18tryEyO7ynxNpBrTy9my/5GfvFqBQeaWvnh8rmYGYfbO3l64z6WzcpnQlrigNcum5VPWUE6f/p7JXEe43frPqTT7zALrF1eUJLNX9+v5UvnTmfqRG3pG03qmtupaWpjZr4qb4hEsu6kt3ct4mZt2ywx6JhHs3OuBvhECNoiYTY5K/gZ4iOdfv73vQMsnZVH3BBrQL1xHu67/GTy0hP5+SsVFGQkcdvSUm77w0YOtbTzj2efMOi1cR7jhdsWAVDf0s79r1SQlujlyoWTuWnVBjbvbeCbF5Tx+UWD30Mi0/auiiLBfPASkfDpTnp7zxA3dS2f0BpiiSUazdJjYloiCXEe9gaREK/dcZCGw0e4IIj6vmbGVz85g6rGVn7+SgUP/2Un7R1+vnPJbE4/IWfIa7uT7Qlpidxzyeye46tvPQe/cyTFxw12qUSw8molxCLRIH2AhLj7v7WGWGKJRrP08HiMSVlJ7A2i0sR/b6kiJSGOj88IrkqIWaAU2oLibLZVNVFWkM41p49yj2ggwavKA9FsW1UTWSnx5KUPvFxGRCJDRnKg5GHD4Y/KrmnJhMQijWbpY3J2yrBLJjr9jj9vqWJJWd6IZmjj4zzHlARL7Nhe1ciM/PRj2pFQRI6/pPg4slLiqeraah3o2cpZSyYklmiaTfooyhq+FvEL71ZS62vn4rmTxqhVEkucc5RX+yjTcgmRqFCQkURlr4S4uwSblkxILFFCLH0UZSdT09RG65HOAc93+h0/e7mcGflpnHdS/hi3TmLBvvrD+No6mKEKEyJRoTAziarGjyZKmjVDLDFICbH0UZwTqAu762DzgOef37yfHTXNfHnpjCF3GBMZTHeFCc0Qi0SHgszkPksmqhvbAMhKHnzbdZFoo4RY+pg7ObChx8YP6wc8/+gbu5iel8an5gxfXUJkIGsqaknwephVOHj9ahGJHIWZSdT62mnrCPzlsKLGx5ScZJITVOVHYocSYunjhImpZKfE87fdh/qdqzjQxKY99Vxz2hTNDsuo+P2OF9+tZPGMXD2hLhIlCrq2WD/QNTP8fnUT03PTwtkkkZBTQix9mBkLirP524f9E+InN+wjzmNcOr9ogCtFhrd+9yGqG9u4cG5huJsiIkEq7EqIKxta6fQ7dtY2U6pnACTGKCGWfhaUZLOjppn6lvaeY51+x9Mb97JkZi65qh0ro/TC5v0kej0sm6UHMkWixUcJ8WH21LXQ3uFnep5miCW2KCGWfk4pzgJg456P1hH/9OVyqhvbuHLhlHA1S6JcU+sRnt9cyZKZeVouIRJFCjKTAahqaOX9Az4ASpUQS4xRQiz9zJuchcdgY9c64ic37OX+Vyq4euEUlVqTUfvJn8upa2nn5sXTwt0UERmBtEQvaYleKhtaef9AoEqMZogl1ighln5SE73MnZzF8+9W0th6hO++sJXTT8jhu5fN0c5iMiobdtfx+NpdXPexEuZNyQp3c0RkhAoyk6hqaKWi2kdBRhLpSfHhbpJISCkhlgHd+PET2VnTzOd+8zb1LUe481NlxMdpuMQSM8sxs6fNrNnMdpvZilB/j7aOTlat3cW1v15HYWYyXz9vZqi/hYiMgcLMJCobA0smSvM1OyyxRwv5ZEAXzC6grCCdt3cd4pzpE1lQnB3uJknoPQC0A/nAfOAFM9vknNsy2huu31XH+wd81DW3s72qib++X8OhliMsKp3IT6+eT2aKZpVEolFBRhLrdu6n0zn+6eyp4W6OSMgpIZYBeTzGN86fyY2rNnDb0tJwN0dCzMxSgeXAHOecD1hjZquB64E7R3vf3761m2fe2Q9AUVYy55TmcuWpkzln+kTVrhaJYidNyuCZd/Zx1cLJ3LJ4eribIxJySohlUEtn5bPxrk+SobVisWgG0OmcK+91bBPwiWO56bc+PYs7LigjKyWelAT9ehGJFZ89cyrXnl5MUrx2p5PYpH+xZEhKhmNWGtBw1LEGoF+1fTO7EbgRoLi4eMib5mckhah5IhJJPB4jyaNkWGKXnpISGZ98QMZRxzKApqPf6Jz7lXNuoXNuYW5u7pg0TkREZCwpIRYZn8oBr5n1XiA+Dxj1A3UiIiLRSgmxyDjknGsGngLuNbNUMzsbuBRYFd6WiYiIjD0lxCLj1y1AMnAA+A/g5mMpuSYiIhKt9FCdyDjlnKsDPhPudoiIiISbOefC2wCzGmD3MG+bCNSOQXNiifps5CKlz0qccxH59Jri9bhRn41cpPSZ4nV8Ur+NXCT02aDxGvaEOBhmtt45tzDc7Ygm6rORU5+Fhvpx5NRnI6c+Cw314+io30Yu0vtMa4hFREREZFxTQiwiIiIi41q0JMS/CncDopD6bOTUZ6Ghfhw59dnIqc9CQ/04Ouq3kYvoPouKNcQiIiIiIsdLtMwQi4iIiIgcF0qIRURERGRci+iE2MxyzOxpM2s2s91mtiLcbYo0ZvaambWama/rtb3XuRVd/dZsZs+YWU442xouZnarma03szYz+81R55aa2TYzazGzV82spNe5RDN71MwazazKzL425o2PIorX4Chmh6Z4HRuK1+AoXocXKzEb0Qkx8ADQDuQDK4GHzGx2eJsUkW51zqV1vWYCdPXTL4HrCfRfC/BgGNsYTvuB7wKP9j5oZhOBp4BvAznAeuCJXm+5BygFSoAlwB1mdsEYtDdaKV6Dp5gdnOJ1bCheg6d4HVpMxGzEJsRmlgosB77tnPM559YAqwkMPhneSuA559xfnHM+AgPycjNLD3O7xpxz7inn3DPAwaNOXQ5scc790TnXSiA455lZWdf5fwD+xTl3yDn3HvBr4IYxanZUUbyGhGIWxetYULyGhOK1S6zEbMQmxMAMoNM5V97r2CZAn2D7u8/Mas3sDTNb3HVsNoH+AsA5t4PAbMCMMLQvUh3dR83ADmC2mWUDk3qfR+NvKIrXkVHMjpziNXQUryOjeB2dqIpZb7i+cRDSgIajjjUA4+7T1zC+CWwlEIjXAM+Z2XzUf8FIA2qOOtbdR2m9vj76nPSn8RY8xezoKF5DR2MteIrX0YuqmI3kGWIfkHHUsQygKQxtiVjOuXXOuSbnXJtz7jHgDeDTqP+CMVQf+Xp9ffQ56U/jLUiK2VFTvIaOxlqQFK/HJKpiNpIT4nLAa2alvY7NA7aEqT3RwgFGoJ/mdR80sxOBRAL9KgFH91EqMI3AmqdDQGXv82j8DUXxOnqK2eAoXkNH8Tp6itfgRVXMRmxC3LXW5CngXjNLNbOzgUuBVeFtWeQwsywzO9/MkszMa2YrgY8DLwG/Ay42s0Vdg/Be4Cnn3Lj79NrVN0lAHBDX3V/A08AcM1vedf4uYLNzblvXpY8D/2xm2V0PAXwB+E0YfoSIp3gNjmJ2eIrX40/xGhzFa3BiJmadcxH7IlCm4xmgGfgQWBHuNkXSC8gF3ibwJ4Z64C3gk73Or+jqt2bgWSAn3G0OUz/dQ+BTfe/XPV3nlgHbgMPAa8DUXtclEigj0whUA18L988SyS/Fa1B9pJgdvo8Ur2PTz4rX4ftI8RpcP8VEzFpXo0RERERExqWIXTIhIiIiIjIWlBCLiIiIyLimhFhERERExjUlxCIiIiIyrikhFhEREZFxTQmxiIiIiIxrSohFREREZFxTQiwiIiIi45oSYhEREREZ1/4fwTGrw9LcOLMAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "learn.activation_stats.plot_layer_stats(-2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The % of non-zero weights is getting much better, although it's still quite high.\n", + "\n", + "We can see even more about what's going on in our training using `color_dim`, passing it a layer index:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAADNCAYAAAC8XqoPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2dzY5kSZqWzf8iIzIrc6qqK5lWA6JpDaAZaYbZgAQIiQ0bJJbcAkskuBtug9sYwYLesGAxQgIBhdQz9ZcZ4X8sanok/77Hw9/0yO7OMD3Pzk/asWPHjp0Tlu7PeW1xPB6HiIiIyMwsf9cNEBEREflN44RHREREpscJj4iIiEyPEx4RERGZHic8IiIiMj1OeERERGR61o/9479c/ptn8876//u3/6RtW+xPP3/1X79rZY5/9surj7n+xc9PPv/w99+2Ml//w00/5qrXVdu6+baXufm2X44vf/mXvf7N6QG++cWrVubdV32ue6DRAFPizTen7VjAKFk9wEbYtLtdXDwecoBjbk8/b/tpj9X7vm0BdY1F37TclfPe9zJE3W+MMbYvT0/05vveiPs3vTOO0K7t69ONy4dehsbc+ofeLrqW6/enG6kN1F8raMe7L08LHm76jlg/cKzjFdpO13b9LnyslWLLXS9C2+g+qtuOMM6p7w8r6Ix6y2z7jodN32+xz673YV32pe4Kx8CyPAd2d1Sob1q97wc9Lk/3pWcMjR0a+/Qsvf/8tCGre+jXR/9inme5hY1hv27Kfbp9BYVgnNN5E3UsLuG5RudNfVif56v7vt/+BbQhaCvVdeh/Xsd/+Y//4ewTxG94REREZHqc8IiIiMj0OOERERGR6XHCIyIiItNzpYL16bF72T2lzfenUtVh0+d3oR+JHG9vTj4/vIb6SQqkaWYpd7jpRZYgHW6/uG3b1t+dGnIoUYLslQp5tW0krxH7F723X/3fU0Pu+5+CvUZ9CP1Tz4nkXTrvKgWOAeLmGOO4KMItyHckMqOkXg75/vdAUCaRj6TG7y+LoSS2Nul3jDGg/VVEfPW/u6n7/sve2C3ckxWSio9wjXDf2lY6HI0dEnpBLN/8cPoZJXgQWweMnXqeNL7o/luAFN33ywTlKv2OEQrioQhMInbdGSVvuN57kNnrf8+PIHRXSXoM7p/7N5f7jNpA503HXJeXIw70DKCxD2OgSsoovNOLFwCJv7uXp5/xuQntevgM+r9c390dtAHq3wfjifqQ/qY8ht/wiIiIyPQ44REREZHpccIjIiIi0+OER0RERKZnGmmZqLLl/q6f7lM64HhzujdKbjClTGRUkl+3IKNWQZkgSZOSP1ORuQpyKBiG/PDVaWdgejEl6AZTdRLaViTvksxJYuD+chkS65Z7EiRPP5M4TdIvyflVgOa02UzAHJRK+93pAe4/74UoETgRNY8kW2L6L5RrlfdNmDpNwxXGwO7laUOqxDzGGFsQNxMRmMbhgdx/CiwvEi6mmqPE2svta9L5GGNRGoJlSKYO3l2gZ0yakF3PiZ+t2b1M6ef1OYASNrSVkolrmjCJ05iQjcL7aTkSv+mrC3qWUspxhZKcqb+wHa0RfdOuv2eD160moqMwDn34GH7DIyIiItPjhEdERESmxwmPiIiITM+zdHhWb/uq5PTbcPU87r/4uA7P8pvTH/XX96+j/ej36fp76x4Cm26+6dvuf9J/EK2/+ZJTsIMQNfytm1bOLb+30srGFJKYBJ+Rq7EHd4m8hersrGBVbOr7PYU8wu/TtRyVodA8dFmSEDXymWih5NJn9Ns9ulhQbkXhliWYbHV/2SMag/tnV/pnT7/n03/DaFu9luS7hOGNGMxYyu3DQESqvwcPQhkKfSRvr4xrdi4ebeFfQ8GJ9T5KV/rGIL1gFWz0CeGebOGscD3ovDH8FbzGVgauUQ35PFeuhpQS6EbRPV+gMYEru4PDWP82jNG9oTq+xuC+Jv+ueZOhYkPtam0I7+XH8BseERERmR4nPCIiIjI9TnhERERkepzwiIiIyPQ8S2k5pQpgFI72FOpq6fdvYMVrCqcjYbEIWet3vQwJnqv7brnVlZ8XYNKSyJwKvZsqmJEUTcIcyoOnn0lgJEGZQvnWIE9XqA9xBWcMEKyf+/EoGBBl2iJwo3RIIj6FidVrBOOLAvgICpGsQm8ayJaMMZQ0ScwOAjxRnIbVoTGwDoT6FsAHsigG0QVhaHSvJSLtGDDGqL8wpA3qCs4pEY/H4Odau7dCsZyk5TqG6XwoWA9fEgm2ofxP1ygIjKR2sQANL3aUbenK6ATdI7X/6TrSeePfIxCekzbQedNK660ueAY/ht/wiIiIyPQ44REREZHpccIjIiIi0+OER0RERKbnWUrLx7/Zk5ZJCqtyIsqWT+HrX518XBx+0tsAQt5xSdbWqbRFwmpd/X2MMe6/6AUjyRQgQTlZkXh3RyZc30SSYUugpURPcjlhW0uIRUOvb0qSkMcYY1/uFl7ZuO9H1y0RrCl9GVcND1xXGoco+YJwW88JJW8YOyh9lnYc4XxIVkwSaGmc0HhavYf6Iem1yaKwYjQKpOS11gBakKRx7ONK34/XTWXG4PR2bH+Vlmkl7lAEbi8lULvCBOsmqQdjYowzz79grGACdPAyw48HLYcLJfXkWUf3KKU2E8dAsMYV56H69Q/QjpvLgjW9gEDUl1XohYoPjVr2Gx4RERGZHic8IiIiMj1OeERERGR6nPCIiIjI9DxLaXn5zQ9t22Hzedu2KsmcT0mojMBEXShHInAxSCkV+trZKaaphv4XSmdl1KxRAoX64QS2r04PuvkO0otfkQUKm6qkHiZrU5IzyYmtDImCaZJsbRsdj6qnlO7S1ySfYzgyyJbHBQiLReZEwZckb5IygycOyqJwTq2vqe9hHGJb6R6psjZckAUIpElicvqCAF24QxOgoS7o51T+r+OJX7yAdoF8XIXeNAkZXy6o4zCUtekcMaX55vHPY/S/KWOMsQvS25Pn6Llyta9J3qVrhH0RPJ/onqH7b18H4oAxQH0PfUjtry+h4IsXcB0fw294REREZHqc8IiIiMj0OOERERGR6XmWDs/x2+/7Ngjzq8FwD28+7mrpu7/3s9KGXua4ynyH9tsqNDV1TepvpOlvxbgSN/zmXqfJabAe1V+31d9txzjjENDIrecULqRLAYLU/zffnFa4IzeH3InAzcBVf+k3eOif5Hfz1AeCn+XbNYqcpDHYVWs+Qi+D7Q/cA3QDyDVJwxvrNlh5nXZMVypv+4EDQ+GQbZxTCCf5J9B+XLm6toPagM+nvu1jhgVWB4nCQfFZl5zjGD3gD9q6e9m3Ub/WdlAb8HlLAZ7VqYLzppXX0fWhsNSyK65UT14dOVvlmUXXYwkBnsfEB0od2UfwGx4RERGZHic8IiIiMj1OeERERGR6nPCIiIjI9DxLaXm8/aJtQpGyBlWFwlzK7rPTA6DgC0QrIGPYF4ldIKaVMLT1u257vX8JdZFAStRANhTaQI4DsbyKs2lQHK7EXUczBetR34Nsiauelz5LV2HG8LgqNUKKGl0PlHDrWKEy0KwV9Q+Fe5VyJKmnK68vi5x4fAVtIKE0EGep7Xg9gtWtxxhjWWRUklhRyoT7ocquFApH0i+uoF6D4uiawQWn59PVojGJwMHLESiMh7QxRhIr1Q99nYyV9Lyvhl4QuLJ+et7i85xehKhtgDKpyLwvIYzrnhGMgbA0zmugI8rh9Kx4BL/hERERkelxwiMiIiLT44RHREREpscJj4iIiEzPs5SWH37/s7YNkznLNhKcVm/ftm37r7+Oyj0UiRgTPamHg3JLWtE5lKK3L0933n6WpRcf1mQBQjtuL583niMkfzablqRikqJJwKxSG6ULB6ugjzHGDmTaVVkVPkrMPrctWPEaIbm2ppuG0m+S6krbaEV7SsimpNcqNeLq0GGiaiL649ihciQk15ceKC03XFm6ysFbGF8ErnjdGtE30croUV0DROMgUfdsO6pgHa4IT5Z9krSMidZBajO2LUyYpvF0se7B9x+2v0rkJBVjonFWLiL921auG60kn1Lvtyhd/wJ+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZnmcpLW8/I0OrbzqWZN/tK4pdzVi87pbhYX1a3+6OTEFKF+7bFodqsfaqMHkXDlll0VTUXeyz/mkyOFwOSnemNNDd3elnlCFDObgJknQ5KK2T5F1KIS71YXIt1J8Ii1WIHiOXj2v/J2nJY7AEuHwH5cp5P7yBpOVQHqwiYipOcx+eVra6D+9vKpaMJ+hXFE9JFg0EaEzRpvTism8izY5x7kUFKFdTwMOEaTxm8l9qksFJGA7GGJ4PJbWHY6yVoXTk4PlEx+OEeihX5Xw4xz29kAMp2kk6NbULXxoI2p+mgBP1WYoCdDj2f43f8IiIiMj0OOERERGR6XHCIyIiItPjhEdERESm51lKy5TqSinBVQQmIYxk5AFJy8dvv2/bljsy68p+NKXERNJj+QzpyLDf7rZvrKmxJPIdbkCmTpODywmQYFjTmM+1IxHYUJAM/FSUNINE7jHOSL5l/KCcSmI51L8uknJNIB6DE1WpXGvDJouIxVRXkjLrNaLqw/86tTFMEja0PxFK93d9v9W7TGRG+Xh/uQwmU8PYaYL75UfHj8UC0ZjGCdYVjv0KpklDuzDpugq38fMQygV9Rn2P5YJEdLzXaLwG9afyLl7vC3WPMcaK7mUAX4SoZULxO0qQp+cJHTN40SK91x7Db3hERERkepzwiIiIyPQ44REREZHpeZYOTw38G4PD/I4PJYCPgr3evIyOSSuoP7z+xWkZCKLDVXmD36draOIYY4wFeDHkJdVd6XdUCBkkrwddluXjn8+ROARp6FWyMj3WRb9P0zUKfuumvkEnCeqvq2UnYXVjnBnD6OzUHbO6aAxXsK00poOxj35I4hGN7kuRy0LXO3Ve6jnRedO9fKQxENyTMaX+NFj02lXV8dqGq563ILowBBAJ/C/sC1rhHIrVex7HSbiyezJ2rl25nK4ZuYN03hSMWtuahF2OccYHSq4R/R2A61aDBuNx8gh+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZnmcpLSMkDxaR+bgE2XmTGX+Lf/THbdv6/Wn9GNhEIWqry5IpBQ+SJEYhjItDNaD7fhjuRiIl9mtpA4W2hfJxbxfsR20gsbWGnNF+6YrUJCTXVaopiC6Uj2v/YN8TyX9RUGqEUL73vSCt2l6vdx1eY5wJnwyk6/1tlsCXBJ9haNuRUtr6puUOghnLatOpnE/3Q1t5PQy7RLm57kuBbOFq4Mmq7alkSi9a1H7llw1oQPVNbQVy2o2Md6qK7vl6T9Jfx+xdktY/FHiK1y14ESIdh/gsSl7aCPo+LYdjJ3zWXR1K+wh+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZnmcpLb/7qs/TjqtuaB1rIjPIUvc/6UviUmjl8vseUflQV1qPVxsPyqViF8miRWRGmZNSMklgJCGvnie1NUzL3d+VMqnsHEiAKFamqxaTcFv7hwRMWmGZLtL28gXG60bCeyCZYlIxnWPwRFhCWmsqybZzIsH6CStL90J90+qBRH/Yta5KDkJpmkpbBeg67n88IGzC+68UpH6GHfH+C1K6eWVxkLxhTLeXI0IhlpLg68seqUebklzvNDm4rZZOz8Pk2XpmWytD922YrL2qYxOeYZgeTocMytF9Sy+5YHr0E/EbHhEREZkeJzwiIiIyPU54REREZHqc8IiIiMj0PEtpmaTGASmfVXJLU11Xb9/2um56V+1qyjEJbZAiSttqQuhTUoJbim8WJo3TX0pBrfVT2nNsFFbJLRSgua+hXLDfHlKIMbG1jDEUKyk1FqhCHgmSqazdknGTdN4x8J5Z3pNdeeHzOCPvkigdCNbpOKwnisJkIv2eaUeVcGPhHepqIigJxDAOE/mfZOHdHYxpkqkDeZ5S2SluO3lJAOVUFG6za9TK4PXI7u9WDqT+RModo/crCuMvIP0chPrW/ySkh88PvCevlfOpXOkzut4EJkCX9uM5pn/bft2eDysuIiIi8vxwwiMiIiLT44RHREREpud5OjwvYCMEsrWgKjjb3V2f8y1qoOAY44ef9W31Z2Z0hGhKiavr1pXds9+wW7jiGGNUhydcJZlC7ZLAulXqn9B5Vzcj+W19nPEpysrMtF8aiIi/+9emkttAm+h35iDEkMcJbKsqWbgyM60avgf3Y/Uu8MuorRRsWMd5mhSH5WqoHfgP4ROO/IDq1CzB2aIVqdHRq/uG9yR5YvU+3b0CF+uhbcLOZjeqfKTxRGM6cKgw8BSInh/0DAuCFM/t3DzK8BmceIHoT2G4IlRfh04YwknH5DDQy/dkXfX+xwNcrmuE+9E4bOGNH2G24jc8IiIiMj1OeERERGR6nPCIiIjI9DjhERERken55KVlCgHEEKREWgZZ6v3nfc73+rYv03pc9p23n51uI2GuSVxjYOBblXVRXoPzxuDBFu6WyXcHCuAD6awKkVT/4YZktctCIYYApvLd5rIomIZqYfhdLUJ1kTwI1VdJb4nCLVw3qCtZIjoV1xcQPFhl/NV7ClykdgVCLIbVheJp2fcIYXi8ynq2knhra7jidbTSd/hfTZQ5S1upXXtaaRoD+KBcct6pZF+fRfRiBImtRHXUw+cC3acc4FlPPKs/ake68joFvZYwQhTNaTxRv9LfnqCuNLS3jif8mxgGo9YQSQyLTJ8V5w8jIiIiMhdOeERERGR6nPCIiIjI9DjhERERken55KVlglYCRhmrJS33MtvPsjnf7mUvt6+JzyjyhW2tVZGwCrIlJZfu72obehlOCc5W/yaBu0Kr5FL7acXgviNsCzzHujL3GGMs0pRgkOHqCsgoO8eCda08TNamhNtAtkShFE6c5MRlSROmMZfKg3VfTvzu23g16CKp40lCI8Jr1P47uAM5H5LOk5TuKH37TLtS4bm1AVKhE9E4Hk9QLEnSxrGT7EhF6FlHxdIxUPdLX1So8i69aIOJ3NCu4N7C7gqT85tYHqbRI8F51+fJGBiwD2ne8Iz8wBmM3/CIiIjI9DjhERERkelxwiMiIiLT44RHREREpudZSst7kJaX625V7ben8zkSqEhU+/YffNG2be8uS3SUVDwoFRMFrdL+h94wlCEp1bVsIrELRbhQAqyJxktI3kX5ddvLNXmQ/DwSNyMZErZRuSDNdowuemPqKlwPTLgtHYvSXkgkW6bSYSARL6EQiZuREEsCfxrjW+tKzzEd53Ubpp9fTpMeY7Sk3TS1GeXmuh+dNyWFxzJ4LZQlfuPYb7Y2FLkyQRdFYDifBb2MQfduTbBOhXG6bqVt6f1H17ueJ94emNoMddFtVF4wwTGH5ndwTHqehMn2re4nPCM/4DAiIiIizxsnPCIiIjI9TnhERERkepzwiIiIyPQ8S2mZpKoVmFyHKr6BYHi46XUtjr3c+n3ftn9RLKpQaFskScsg7aVJxauH03Io2mHiMCXEklB4+hnTNCFpmdpaBegFiM2JuPnjzkHabyi+oVhXq6LUUkywhk2lrdT0xQ76guqvZTC1GYRxkqnhmA1ISsXkWtq3FEwFxui88Z6B6h9oXyi3vXwfLRPpd2SJsCx+X365IEn6PUdL/KYy4T3DicmnH0lsPpD4HZCkBv9YMK2/XO8o/ffM2K8vjoTXFu+/6n3TOYaeP6Vt17+BT5GDa9tQig8T6pvv/hG+nvEbHhEREZkeJzwiIiIyPU54REREZHo+eYfnu3/2d9u2IwT8LWtw3xhjUcIIm9MzxtiDw/MAK6jff05BgOUzrU4LgYgUkngoIYm00i06CrBa86GuWAv9hSGGaQDY7rLbQCtqJ/Aq8VAOVwK+/ONz/Ps0eVal/niV58SVocOF1639Vk+/59PYCYPVlsUJ27+AJmDA2HVuRupZRfvR6tngA6FbUgMXw5WysR31kBR8Fw6nxC9DfyN0XloAH67qnS2X3u5dCjEM+6KG39FzgVcbh7qAdj8/IayzuSxUFz5vL7cLryP1BfhABwiETcJfsX5o/7VjJ7kneYX7D3vG+A2PiIiITI8THhEREZkeJzwiIiIyPU54REREZHo+eWmZIFl0CfJS3XbYgCwM0vIOVkbHldbrNpLvQqlqWeSu/Q72C1cNbwF/YSgVSrjBNpQCr13hPAiYO19/cN6hVIz1gxhfSTPOarYlSoHhyvEtmIxkYZJYk9WtR79HsF0gixLt+qI4nbW/75i14erVuTHIDeqi/q87Yzhk3y0S46kMNBZlZ5SIiyyKJw7NCsYdPgPovFG8T9oATcAw06RfoS7aLZCWWd7t+6EoXfsnlanD+yh6kYP2O1yunwRrehkD6y/tP4ay9mP4DY+IiIhMjxMeERERmR4nPCIiIjI9TnhERERkej55afnbn/UmLm7v27b1uhtgu12Nu+z10wre+00m6dV9j5CgTO0ijofSVpQaL682PsboicAkiYHAjVYmprPWVeh7mXjV8yoPYspnmKpcN6EQmxp/QCJdk8GICcAlPZXk10DkQ0geDVczX+BAv3xITIgNpMlEFj5LrZ8kbIDumcXD5X1RKoZ7C8XWep54beGgtK3KouE9w3XBtnICeN/iCwFUV/kcjhMUy5M3AugywrhAiThJHE6kYiJM0aZ7t/bFAf5m4SHDlzb6quSp/X85aRkFZXp2X5sgH6Trnxzmg0qLiIiIPEOc8IiIiMj0OOERERGR6XHCIyIiItPzyUvLh03ftiA5GGTUzebUttxBUi7Ja/vbLkLtX0Dbau+RcEZyIvmEpW3LNSSlLrrxR+mpi32RDq9Nrh0jSvalutJU11Z1mBJMgviHCmx/DaauQrlaP54PNTYoRv0cptK2dtD1rgL/mfpRIK1SI0rYUFcaO33pgCmpAE1CbJrw3XaEuqBY60Nqa3o9ykMlln6Dusbo121B5xiK8f2AsA0TdIP60/RwaisJ6FUGx1smFONLX+P1uMleoKgvQuALG0QivA+Q5dPHGsYol7FJL5zASxA4Nuu1xOfhh923fsMjIiIi0+OER0RERKbHCY+IiIhMzyfv8Nx/2betIMxvDX7Otrgs6xsIJ4RgpN2rfkwK36qhSgsI81uRNxQsT7ugqSj9Fr2F32QTtwgcmCMln+FK4uV3WtgPVwJOXJxo6d4z1LZSVbgCeegj1HKYDBj6UokXQ6D7EZx3/Ft35n5cTQvgS4MaYVsdwul+4SrVx7pKPPkhO6gLqI4Q+k3hdWtDDEPt6L6FupKxCc8w3JHqqtebtJVr/wqFuhzeM3QtAwcM+wvDG8t+qfOEz6LLzlZM4ArSOOfx1De1cvjMD55h4/pV3B/Db3hERERkepzwiIiIyPQ44REREZHpccIjIiIi0/PJS8sUXHQD8vFmBULy6nQ+t1t383gLgWO0gjquZFxEqyWIwEsUzC6LiEsIUjyAFH2EVbZb9WEg2xL6lQTrYw2+SmVIogXwZbJlFHJGkDAHUGBkawLJimm7ajkKE6NwxaBdC7jeuHJyKjdTaGGr63oJt+0Wjtdjsy3TVZ5p2+X7CMVNWg06PWYlDcBs8nx4vFTUredJY5NEZipXq0//i52IuVQkvL8HBU3W5lN/wXMZzzvpw/QZWcMCUZwOVyDH8VSKhIIyknQ//kkM2v8Rvp7xGx4RERGZHic8IiIiMj1OeERERGR6nPCIiIjI9Hzy0vLuZTecbmlldNj2UFesRekQ5OAbWJUcJLfj3em+a0iAJvmYROAqN+9SeZckvSLbUaryksRWcmTpmGVfFGLJTEMxN5FYoQ/3tJRxUBcKq5BEjQm3pV/pANRfwerGR4prTVZGH2euUS1D4xclU9hWZfmnBC9Xz5jk17SqdHX0AiadJ8nE6SrVRLIrXcckLZz2e8Kq4Y30GgXjFd1wOu0gcfgp0NivY/GIjQ3F79plwXN6jJGNAepn6sS0DxOx/Nq6UqGe/4ic7gZ9iM+wR/AbHhEREZkeJzwiIiIyPU54REREZHqc8IiIiMj0fPLS8uFlF+ZebHZ926pv265P53P7my443b/Y9GPe9G6hJe0XJZmYUpXXlMwJ7A6nbSUBer/rMvWehLlA1iaZmqa/JHgeigR9PPR2oawGbW1CdeihLje9f1oTQsmRBG6UWGuXwfVmzzE4qSTaeZwRdeuma1Oux2B5sPQjioJXytTUBny5INiXuvAAYigmikOadOsKlEUfad9JQ0o74EUC7K9EWk5E8w+h1pdcxzH6/TEyKR3vGZRRy3MtbVd60HracI1SSba+JID7wWMzSRw+wphepF9dBPcWSuThyx513/QaoUT+EVPZf43f8IiIiMj0OOERERGR6XHCIyIiItPzyTk861/8/OTz4q67ObdrcHhg2/3+9PRqEOEYY6zgN+btLf0YDfsWr4fcojWF5qFbcvm37iWsCH9YXw5/WoHvEioj+IPuosyTyadhz6Nvqv7MU34/joqQz4QrfV8OaTse+v8XyD+hALPkl2f8XZ5cmaANhyQkjCqjY6JDEPonAVe7GRToSSvO065wzDouMOyS6qKuSMIbQ5+prhIfhzdiwF9YsJagQ5KjV+/vcOxEwYN0j4ahktG9lQZUUlW1qRhkSnteDtdbpn5WehvVtqZ1Xds9tN+VIaIfit/wiIiIyPQ44REREZHpccIjIiIi0+OER0RERKbnk5OWv/nT3z/5fHP3QytDgvLNkuTm7cnnLUimmxsILHzRu4XE2bovCcorCPjbQzvWgbS1QTn48pyVAhFJ1j4EdY0xxmJxeVXhtK4qYtN+6NEGQXfBArx/VRe0C/rsUOTBNYydHQTYUV21bUf4v8eCwiGvFClJzKZjJpDgSyTOMtWEoZjAYV+S2wKh+xwoNzdJNmvXEoP0aqFeJpXzl4GNmorrKOwHpCJ+q59WDU8JxkX6IgHWVQM2of70ZY9679JzjZ7B2P66ijtcR3wRJh1PZRu+4ADgM6Xs+zHHIZ1jHFL66+N8UGkRERGRZ4gTHhEREZkeJzwiIiIyPU54REREZHo+PWn5b5+KiC9vH1qZuyIjjzHGLayWviureG/XfXnaG1iVfHvb6xLrHkQAAAiESURBVCJuSrLyDcjUBCUmb4uASe0i2XkP5SokO5PsRRLdHtJlF8XcO+DKw9lK4lXoXS6zVOgkkTkV2lCAxtTbyzIfiuWBBLi/fBnHGGMsg1Rd6hsSp3cwXJO+jhdjD5KDUxGRyq3K2E9XcScWV8rgqUDaEsWhrlggLuVSkXYN0jW29cqk6zglvVBfBvhxv0TMhjbA84PAcZdI0aHQW1nDcxr7HsrV/qHnNHFtW5eUrh+mgNe20n4rGP3JPb+Io6PP4zc8IiIiMj1OeERERGR6nPCIiIjI9DjhERERken55KTldz89FZN+dnvfyrzZvG/bXkDS8sPqVATe7PvpvnzRpegdyMEkgL0o0vItSMskGhO19j0cLxWZqwC2BhkvFUNHIMiR2LyEuFmqf1P6kCXpfsxEzE2lZZQHcd/TtqWJwMkxPzQx9LQdJSkVJFA6x5sbEikvHy9N0U5I+zBKzQ4Srcc4J2BeFnqpX5ehCJyQJHKP0aXV9Hhp/ZX0/kuOSX246u+SXC1Ap+26Vtam+hPxnq9RJu/W/kmvN43phKeMpy4tX/8MTtr/ofea3/CIiIjI9DjhERERkelxwiMiIiLT44RHREREpueTk5a3f+M0RfnNiy4ov1p3kXkFctT2eDqf260hqfgI20DKpCXtX92cCs+UZFoTVs/VVaELEyexFlYghlIbUABDibjIwXBMaimm5da6whRRIumfJ4nMkJCd1E/jqV4TTKu+FvhvDAmAdEQaF/WcWH7N0nKT/k9FRBJgr4W7v0rw16dCJ/ulabad65Jrx+BrWUulT53kObCC5+G14xCfJyTShsJwfbmDXhxJSfa8tl+Tvx9jPO1FiEp+f3+8lzGuHYeP4Tc8IiIiMj1OeERERGR6nPCIiIjI9PxOHZ7ln/5R2/b6J9+ffH774rtW5vPNu7bt/tBP5W5VgsPA16FtFDxIfsirTQ8trDzse6rWkn7hLYdMfZ1FsEI71UW/TyeuyRj9N+Q1eDd5ONZ1Tse1vw3T+VD9seMUtIu8gt8k+zB4kK5bWl9Sf+LwpKuNE9V7Sr2x9N6q5fYQpnmtf0KkxkhtV+p00Ni/1kGiY67Bcbs2pDJpa7xSPZwjBV7WfdfpMxhXDb8cUpreM7Uc5DReXReRtpWea/VZkfTNOZLr/aF/B/yGR0RERKbHCY+IiIhMjxMeERERmR4nPCIiIjI9v1Np+X/8q8/btr/ze39+8vkrkJZfr3oY4Ri3bUsV6w6rbH5HKyATN0XSS+VBYlHkZlrhfAXtouDECrVrA+W2IFiT3PybHDS/Xb33rwDxjc6xlkpWqj9HFSlp5KTBZ13mo3bBeArlxJt1CUl8imhcytG9toWwS64sKBcKq0R1lMk9R5kzqPtjhsKl/2sF5/qjtoM57WsSXfmlgV5Tbf8KJOn0xYtE6E2f5yRFH6IXFdJnxeXxmgaeXk923W7Wl/8mrpaXA1zHyOR8uraP1vlBpUVERESeIU54REREZHqc8IiIiMj0OOERERGR6fmtScurt2/btu2fdCH5D15/ffL5b938qpXZg+KZyLskl92telry3eqmbcNk0SJfvdt3FZhSm3dHSF8ubbtZ9gTl9QKSoqEvmqwdpiofwGpcJqtnkxgaypBdYu37pcnXta6PmfZM9VVpPd0vPeY6TIBuZWIpN7tuSVtTQfJasZ/2qzJnmix7rVieyMhjnGvr5WOm+yX3DPGhgudj7SJQ4C7Ce3I+5+qq0HWktGdiRWOlbEteXBgjWyX++tdZroek7rT/K9eu7P4U6jE/xmTFb3hERERkepzwiIiIyPQ44REREZHpccIjIiIi0/Nbk5b/+7//g7btn//8l23bH738Xyefv1x3sfkv9q/atu2qK4U1mXiz6FIxpRe/WfckZxK77g+Xu2936O3aHfsxd0VIZiEWJDTQyarI/AAJyg+L3nZM6wQZvPYZpUKnIiUJyZeOd67+SL57Qhr2tZBYXq8bCaUkAu8COT9NgKY+XF+Z6noE4Z3Ou0Lp3tQuSp2+VoAmYTUZr9eOOdo33S8pl95r175IQC24ti6CrgclZCc8RcSvz4anpFAndaVC/bVjJ71GlaesGHBtGxJxPe3DR9vzQaVFREREniFOeERERGR6nPCIiIjI9HwUh+f//Lt/evL523/8rpX513/4Z23bv3jz39q2n67+8uTzA8R9rcBbuT90P2dTPJiXyx4yuFlQOBOs4Axhgd/t+wrtlQM4FxRQWKHgwU3osrTjgbu0PvS63kO7DuAb1baRw5O4IGOM8VAcJ/otl/qQrlEtlzgk545JXPtbelLXGsbhYdnPu/bXGODswH9j6HrU++NcudrXT3FZHor3Rg4anSMFYCZtSP2Na30mIlqJOxybieNGPCUMtPIU36het4/hYTxWF5H2Re1req5dC43za1lEK7HnVE9zE65m/lF9SKiqXt+PcTy/4REREZHpccIjIiIi0+OER0RERKbHCY+IiIhMz6PS8n/6n100JpbjP5fPsGourPS9PXY56leH09C/P9/1lcs3iy70ElVufrnqgYJvll2wvl1u27YHkJa/Xrw5+UwhfSQ7k9xXRTGSll+E22r9JKF9B/1KkiYJpDelvrtV7684DC0QuFNpMgnlS1djJ6rEmO5H7a911T4d48z5wNDf1VW9QTSn86YxhvJuaWsqYGLg4pXibHJtX5DIHoQ+jsHBj10sT1ehh2DRYyLn03Oz13XtSuLXsgpd0aRdqax9reT7lPNerS5f32tl8FSoT8o95aUBYj2uk7OvFb+prfQsTaTxDz1vv+ERERGR6XHCIyIiItPjhEdERESmxwmPiIiITM/iePy4qY0iIiIinxp+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZHic8IiIiMj1OeERERGR6/j8jStwSQ8E4SwAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "learn.activation_stats.color_dim(-2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "`color_dim` was developed by fast.ai in conjunction with a student, Stefano Giomo. Stefano, who refers to the idea as the *colorful dimension*, has a [detailed explanation](https://forums.fast.ai/t/the-colorful-dimension/42908) of the history and details behind the method. The basic idea is to create a histogram of the activations of a layer, which we would hope would follow a smooth pattern such as the normal distribution shown by Stefano here:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\"Histogram" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To create `color_dim`, we take the histogram shown on the left here, and convert it into just the colored representation shown at the bottom. Then we flip it on its side, as shown on the right. We found that the distribution is clearer if we take the `log` of the histogram values. Then, Stefano describes:\n", + "\n", + "> : The final plot for each layer is made by stacking the histogram of the activations from each batch along the horizontal axis. So each vertical slice in the visualisation represents the histogram of activations for a single batch. The color intensity corresponds to the height of the histogram, in other words the number of activations in each histogram bin.\n", + "\n", + "This is Stefano's picture of how this all fits together:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\"Summary" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "So with that in mind, let's take another look at the result for the penultimate layer:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAADNCAYAAAC8XqoPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2dzY5kSZqWzf8iIzIrc6qqK5lWA6JpDaAZaYbZgAQIiQ0bJJbcAkskuBtug9sYwYLesGAxQgIBhdQz9ZcZ4X8sanok/77Hw9/0yO7OMD3Pzk/asWPHjp0Tlu7PeW1xPB6HiIiIyMwsf9cNEBEREflN44RHREREpscJj4iIiEyPEx4RERGZHic8IiIiMj1OeERERGR61o/9479c/ptn8876//u3/6RtW+xPP3/1X79rZY5/9surj7n+xc9PPv/w99+2Ml//w00/5qrXVdu6+baXufm2X44vf/mXvf7N6QG++cWrVubdV32ue6DRAFPizTen7VjAKFk9wEbYtLtdXDwecoBjbk8/b/tpj9X7vm0BdY1F37TclfPe9zJE3W+MMbYvT0/05vveiPs3vTOO0K7t69ONy4dehsbc+ofeLrqW6/enG6kN1F8raMe7L08LHm76jlg/cKzjFdpO13b9LnyslWLLXS9C2+g+qtuOMM6p7w8r6Ix6y2z7jodN32+xz673YV32pe4Kx8CyPAd2d1Sob1q97wc9Lk/3pWcMjR0a+/Qsvf/8tCGre+jXR/9inme5hY1hv27Kfbp9BYVgnNN5E3UsLuG5RudNfVif56v7vt/+BbQhaCvVdeh/Xsd/+Y//4ewTxG94REREZHqc8IiIiMj0OOERERGR6XHCIyIiItNzpYL16bF72T2lzfenUtVh0+d3oR+JHG9vTj4/vIb6SQqkaWYpd7jpRZYgHW6/uG3b1t+dGnIoUYLslQp5tW0krxH7F723X/3fU0Pu+5+CvUZ9CP1Tz4nkXTrvKgWOAeLmGOO4KMItyHckMqOkXg75/vdAUCaRj6TG7y+LoSS2Nul3jDGg/VVEfPW/u6n7/sve2C3ckxWSio9wjXDf2lY6HI0dEnpBLN/8cPoZJXgQWweMnXqeNL7o/luAFN33ywTlKv2OEQrioQhMInbdGSVvuN57kNnrf8+PIHRXSXoM7p/7N5f7jNpA503HXJeXIw70DKCxD2OgSsoovNOLFwCJv7uXp5/xuQntevgM+r9c390dtAHq3wfjifqQ/qY8ht/wiIiIyPQ44REREZHpccIjIiIi0+OER0RERKZnGmmZqLLl/q6f7lM64HhzujdKbjClTGRUkl+3IKNWQZkgSZOSP1ORuQpyKBiG/PDVaWdgejEl6AZTdRLaViTvksxJYuD+chkS65Z7EiRPP5M4TdIvyflVgOa02UzAHJRK+93pAe4/74UoETgRNY8kW2L6L5RrlfdNmDpNwxXGwO7laUOqxDzGGFsQNxMRmMbhgdx/CiwvEi6mmqPE2svta9L5GGNRGoJlSKYO3l2gZ0yakF3PiZ+t2b1M6ef1OYASNrSVkolrmjCJ05iQjcL7aTkSv+mrC3qWUspxhZKcqb+wHa0RfdOuv2eD160moqMwDn34GH7DIyIiItPjhEdERESmxwmPiIiITM+zdHhWb/uq5PTbcPU87r/4uA7P8pvTH/XX96+j/ej36fp76x4Cm26+6dvuf9J/EK2/+ZJTsIMQNfytm1bOLb+30srGFJKYBJ+Rq7EHd4m8hersrGBVbOr7PYU8wu/TtRyVodA8dFmSEDXymWih5NJn9Ns9ulhQbkXhliWYbHV/2SMag/tnV/pnT7/n03/DaFu9luS7hOGNGMxYyu3DQESqvwcPQhkKfSRvr4xrdi4ebeFfQ8GJ9T5KV/rGIL1gFWz0CeGebOGscD3ovDH8FbzGVgauUQ35PFeuhpQS6EbRPV+gMYEru4PDWP82jNG9oTq+xuC+Jv+ueZOhYkPtam0I7+XH8BseERERmR4nPCIiIjI9TnhERERkepzwiIiIyPQ8S2k5pQpgFI72FOpq6fdvYMVrCqcjYbEIWet3vQwJnqv7brnVlZ8XYNKSyJwKvZsqmJEUTcIcyoOnn0lgJEGZQvnWIE9XqA9xBWcMEKyf+/EoGBBl2iJwo3RIIj6FidVrBOOLAvgICpGsQm8ayJaMMZQ0ScwOAjxRnIbVoTGwDoT6FsAHsigG0QVhaHSvJSLtGDDGqL8wpA3qCs4pEY/H4Odau7dCsZyk5TqG6XwoWA9fEgm2ofxP1ygIjKR2sQANL3aUbenK6ATdI7X/6TrSeePfIxCekzbQedNK660ueAY/ht/wiIiIyPQ44REREZHpccIjIiIi0+OER0RERKbnWUrLx7/Zk5ZJCqtyIsqWT+HrX518XBx+0tsAQt5xSdbWqbRFwmpd/X2MMe6/6AUjyRQgQTlZkXh3RyZc30SSYUugpURPcjlhW0uIRUOvb0qSkMcYY1/uFl7ZuO9H1y0RrCl9GVcND1xXGoco+YJwW88JJW8YOyh9lnYc4XxIVkwSaGmc0HhavYf6Iem1yaKwYjQKpOS11gBakKRx7ONK34/XTWXG4PR2bH+Vlmkl7lAEbi8lULvCBOsmqQdjYowzz79grGACdPAyw48HLYcLJfXkWUf3KKU2E8dAsMYV56H69Q/QjpvLgjW9gEDUl1XohYoPjVr2Gx4RERGZHic8IiIiMj1OeERERGR6nPCIiIjI9DxLaXn5zQ9t22Hzedu2KsmcT0mojMBEXShHInAxSCkV+trZKaaphv4XSmdl1KxRAoX64QS2r04PuvkO0otfkQUKm6qkHiZrU5IzyYmtDImCaZJsbRsdj6qnlO7S1ySfYzgyyJbHBQiLReZEwZckb5IygycOyqJwTq2vqe9hHGJb6R6psjZckAUIpElicvqCAF24QxOgoS7o51T+r+OJX7yAdoF8XIXeNAkZXy6o4zCUtekcMaX55vHPY/S/KWOMsQvS25Pn6Llyta9J3qVrhH0RPJ/onqH7b18H4oAxQH0PfUjtry+h4IsXcB0fw294REREZHqc8IiIiMj0OOERERGR6XmWDs/x2+/7Ngjzq8FwD28+7mrpu7/3s9KGXua4ynyH9tsqNDV1TepvpOlvxbgSN/zmXqfJabAe1V+31d9txzjjENDIrecULqRLAYLU/zffnFa4IzeH3InAzcBVf+k3eOif5Hfz1AeCn+XbNYqcpDHYVWs+Qi+D7Q/cA3QDyDVJwxvrNlh5nXZMVypv+4EDQ+GQbZxTCCf5J9B+XLm6toPagM+nvu1jhgVWB4nCQfFZl5zjGD3gD9q6e9m3Ub/WdlAb8HlLAZ7VqYLzppXX0fWhsNSyK65UT14dOVvlmUXXYwkBnsfEB0od2UfwGx4RERGZHic8IiIiMj1OeERERGR6nPCIiIjI9DxLaXm8/aJtQpGyBlWFwlzK7rPTA6DgC0QrIGPYF4ldIKaVMLT1u257vX8JdZFAStRANhTaQI4DsbyKs2lQHK7EXUczBetR34Nsiauelz5LV2HG8LgqNUKKGl0PlHDrWKEy0KwV9Q+Fe5VyJKmnK68vi5x4fAVtIKE0EGep7Xg9gtWtxxhjWWRUklhRyoT7ocquFApH0i+uoF6D4uiawQWn59PVojGJwMHLESiMh7QxRhIr1Q99nYyV9Lyvhl4QuLJ+et7i85xehKhtgDKpyLwvIYzrnhGMgbA0zmugI8rh9Kx4BL/hERERkelxwiMiIiLT44RHREREpscJj4iIiEzPs5SWH37/s7YNkznLNhKcVm/ftm37r7+Oyj0UiRgTPamHg3JLWtE5lKK3L0933n6WpRcf1mQBQjtuL583niMkfzablqRikqJJwKxSG6ULB6ugjzHGDmTaVVkVPkrMPrctWPEaIbm2ppuG0m+S6krbaEV7SsimpNcqNeLq0GGiaiL649ihciQk15ceKC03XFm6ysFbGF8ErnjdGtE30croUV0DROMgUfdsO6pgHa4IT5Z9krSMidZBajO2LUyYpvF0se7B9x+2v0rkJBVjonFWLiL921auG60kn1Lvtyhd/wJ+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZnmcpLW8/I0OrbzqWZN/tK4pdzVi87pbhYX1a3+6OTEFKF+7bFodqsfaqMHkXDlll0VTUXeyz/mkyOFwOSnemNNDd3elnlCFDObgJknQ5KK2T5F1KIS71YXIt1J8Ii1WIHiOXj2v/J2nJY7AEuHwH5cp5P7yBpOVQHqwiYipOcx+eVra6D+9vKpaMJ+hXFE9JFg0EaEzRpvTism8izY5x7kUFKFdTwMOEaTxm8l9qksFJGA7GGJ4PJbWHY6yVoXTk4PlEx+OEeihX5Xw4xz29kAMp2kk6NbULXxoI2p+mgBP1WYoCdDj2f43f8IiIiMj0OOERERGR6XHCIyIiItPjhEdERESm51lKy5TqSinBVQQmIYxk5AFJy8dvv2/bljsy68p+NKXERNJj+QzpyLDf7rZvrKmxJPIdbkCmTpODywmQYFjTmM+1IxHYUJAM/FSUNINE7jHOSL5l/KCcSmI51L8uknJNIB6DE1WpXGvDJouIxVRXkjLrNaLqw/86tTFMEja0PxFK93d9v9W7TGRG+Xh/uQwmU8PYaYL75UfHj8UC0ZjGCdYVjv0KpklDuzDpugq38fMQygV9Rn2P5YJEdLzXaLwG9afyLl7vC3WPMcaK7mUAX4SoZULxO0qQp+cJHTN40SK91x7Db3hERERkepzwiIiIyPQ44REREZHpeZYOTw38G4PD/I4PJYCPgr3evIyOSSuoP7z+xWkZCKLDVXmD36draOIYY4wFeDHkJdVd6XdUCBkkrwddluXjn8+ROARp6FWyMj3WRb9P0zUKfuumvkEnCeqvq2UnYXVjnBnD6OzUHbO6aAxXsK00poOxj35I4hGN7kuRy0LXO3Ve6jnRedO9fKQxENyTMaX+NFj02lXV8dqGq563ILowBBAJ/C/sC1rhHIrVex7HSbiyezJ2rl25nK4ZuYN03hSMWtuahF2OccYHSq4R/R2A61aDBuNx8gh+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZnmcpLSMkDxaR+bgE2XmTGX+Lf/THbdv6/Wn9GNhEIWqry5IpBQ+SJEYhjItDNaD7fhjuRiIl9mtpA4W2hfJxbxfsR20gsbWGnNF+6YrUJCTXVaopiC6Uj2v/YN8TyX9RUGqEUL73vSCt2l6vdx1eY5wJnwyk6/1tlsCXBJ9haNuRUtr6puUOghnLatOpnE/3Q1t5PQy7RLm57kuBbOFq4Mmq7alkSi9a1H7llw1oQPVNbQVy2o2Md6qK7vl6T9Jfx+xdktY/FHiK1y14ESIdh/gsSl7aCPo+LYdjJ3zWXR1K+wh+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZnmcpLb/7qs/TjqtuaB1rIjPIUvc/6UviUmjl8vseUflQV1qPVxsPyqViF8miRWRGmZNSMklgJCGvnie1NUzL3d+VMqnsHEiAKFamqxaTcFv7hwRMWmGZLtL28gXG60bCeyCZYlIxnWPwRFhCWmsqybZzIsH6CStL90J90+qBRH/Yta5KDkJpmkpbBeg67n88IGzC+68UpH6GHfH+C1K6eWVxkLxhTLeXI0IhlpLg68seqUebklzvNDm4rZZOz8Pk2XpmWytD922YrL2qYxOeYZgeTocMytF9Sy+5YHr0E/EbHhEREZkeJzwiIiIyPU54REREZHqc8IiIiMj0PEtpmaTGASmfVXJLU11Xb9/2um56V+1qyjEJbZAiSttqQuhTUoJbim8WJo3TX0pBrfVT2nNsFFbJLRSgua+hXLDfHlKIMbG1jDEUKyk1FqhCHgmSqazdknGTdN4x8J5Z3pNdeeHzOCPvkigdCNbpOKwnisJkIv2eaUeVcGPhHepqIigJxDAOE/mfZOHdHYxpkqkDeZ5S2SluO3lJAOVUFG6za9TK4PXI7u9WDqT+RModo/crCuMvIP0chPrW/ySkh88PvCevlfOpXOkzut4EJkCX9uM5pn/bft2eDysuIiIi8vxwwiMiIiLT44RHREREpud5OjwvYCMEsrWgKjjb3V2f8y1qoOAY44ef9W31Z2Z0hGhKiavr1pXds9+wW7jiGGNUhydcJZlC7ZLAulXqn9B5Vzcj+W19nPEpysrMtF8aiIi/+9emkttAm+h35iDEkMcJbKsqWbgyM60avgf3Y/Uu8MuorRRsWMd5mhSH5WqoHfgP4ROO/IDq1CzB2aIVqdHRq/uG9yR5YvU+3b0CF+uhbcLOZjeqfKTxRGM6cKgw8BSInh/0DAuCFM/t3DzK8BmceIHoT2G4IlRfh04YwknH5DDQy/dkXfX+xwNcrmuE+9E4bOGNH2G24jc8IiIiMj1OeERERGR6nPCIiIjI9DjhERERken55KVlCgHEEKREWgZZ6v3nfc73+rYv03pc9p23n51uI2GuSVxjYOBblXVRXoPzxuDBFu6WyXcHCuAD6awKkVT/4YZktctCIYYApvLd5rIomIZqYfhdLUJ1kTwI1VdJb4nCLVw3qCtZIjoV1xcQPFhl/NV7ClykdgVCLIbVheJp2fcIYXi8ynq2knhra7jidbTSd/hfTZQ5S1upXXtaaRoD+KBcct6pZF+fRfRiBImtRHXUw+cC3acc4FlPPKs/ake68joFvZYwQhTNaTxRv9LfnqCuNLS3jif8mxgGo9YQSQyLTJ8V5w8jIiIiMhdOeERERGR6nPCIiIjI9DjhERERken55KVlglYCRhmrJS33MtvPsjnf7mUvt6+JzyjyhW2tVZGwCrIlJZfu72obehlOCc5W/yaBu0Kr5FL7acXgviNsCzzHujL3GGMs0pRgkOHqCsgoO8eCda08TNamhNtAtkShFE6c5MRlSROmMZfKg3VfTvzu23g16CKp40lCI8Jr1P47uAM5H5LOk5TuKH37TLtS4bm1AVKhE9E4Hk9QLEnSxrGT7EhF6FlHxdIxUPdLX1So8i69aIOJ3NCu4N7C7gqT85tYHqbRI8F51+fJGBiwD2ne8Iz8wBmM3/CIiIjI9DjhERERkelxwiMiIiLT44RHREREpudZSst7kJaX625V7ben8zkSqEhU+/YffNG2be8uS3SUVDwoFRMFrdL+h94wlCEp1bVsIrELRbhQAqyJxktI3kX5ddvLNXmQ/DwSNyMZErZRuSDNdowuemPqKlwPTLgtHYvSXkgkW6bSYSARL6EQiZuREEsCfxrjW+tKzzEd53Ubpp9fTpMeY7Sk3TS1GeXmuh+dNyWFxzJ4LZQlfuPYb7Y2FLkyQRdFYDifBb2MQfduTbBOhXG6bqVt6f1H17ueJ94emNoMddFtVF4wwTGH5ndwTHqehMn2re4nPCM/4DAiIiIizxsnPCIiIjI9TnhERERkepzwiIiIyPQ8S2mZpKoVmFyHKr6BYHi46XUtjr3c+n3ftn9RLKpQaFskScsg7aVJxauH03Io2mHiMCXEklB4+hnTNCFpmdpaBegFiM2JuPnjzkHabyi+oVhXq6LUUkywhk2lrdT0xQ76guqvZTC1GYRxkqnhmA1ISsXkWtq3FEwFxui88Z6B6h9oXyi3vXwfLRPpd2SJsCx+X365IEn6PUdL/KYy4T3DicmnH0lsPpD4HZCkBv9YMK2/XO8o/ffM2K8vjoTXFu+/6n3TOYaeP6Vt17+BT5GDa9tQig8T6pvv/hG+nvEbHhEREZkeJzwiIiIyPU54REREZHo+eYfnu3/2d9u2IwT8LWtw3xhjUcIIm9MzxtiDw/MAK6jff05BgOUzrU4LgYgUkngoIYm00i06CrBa86GuWAv9hSGGaQDY7rLbQCtqJ/Aq8VAOVwK+/ONz/Ps0eVal/niV58SVocOF1639Vk+/59PYCYPVlsUJ27+AJmDA2HVuRupZRfvR6tngA6FbUgMXw5WysR31kBR8Fw6nxC9DfyN0XloAH67qnS2X3u5dCjEM+6KG39FzgVcbh7qAdj8/IayzuSxUFz5vL7cLryP1BfhABwiETcJfsX5o/7VjJ7kneYX7D3vG+A2PiIiITI8THhEREZkeJzwiIiIyPU54REREZHo+eWmZIFl0CfJS3XbYgCwM0vIOVkbHldbrNpLvQqlqWeSu/Q72C1cNbwF/YSgVSrjBNpQCr13hPAiYO19/cN6hVIz1gxhfSTPOarYlSoHhyvEtmIxkYZJYk9WtR79HsF0gixLt+qI4nbW/75i14erVuTHIDeqi/q87Yzhk3y0S46kMNBZlZ5SIiyyKJw7NCsYdPgPovFG8T9oATcAw06RfoS7aLZCWWd7t+6EoXfsnlanD+yh6kYP2O1yunwRrehkD6y/tP4ay9mP4DY+IiIhMjxMeERERmR4nPCIiIjI9TnhERERkej55afnbn/UmLm7v27b1uhtgu12Nu+z10wre+00m6dV9j5CgTO0ijofSVpQaL682PsboicAkiYHAjVYmprPWVeh7mXjV8yoPYspnmKpcN6EQmxp/QCJdk8GICcAlPZXk10DkQ0geDVczX+BAv3xITIgNpMlEFj5LrZ8kbIDumcXD5X1RKoZ7C8XWep54beGgtK3KouE9w3XBtnICeN/iCwFUV/kcjhMUy5M3AugywrhAiThJHE6kYiJM0aZ7t/bFAf5m4SHDlzb6quSp/X85aRkFZXp2X5sgH6Trnxzmg0qLiIiIPEOc8IiIiMj0OOERERGR6XHCIyIiItPzyUvLh03ftiA5GGTUzebUttxBUi7Ja/vbLkLtX0Dbau+RcEZyIvmEpW3LNSSlLrrxR+mpi32RDq9Nrh0jSvalutJU11Z1mBJMgviHCmx/DaauQrlaP54PNTYoRv0cptK2dtD1rgL/mfpRIK1SI0rYUFcaO33pgCmpAE1CbJrw3XaEuqBY60Nqa3o9ykMlln6Dusbo121B5xiK8f2AsA0TdIP60/RwaisJ6FUGx1smFONLX+P1uMleoKgvQuALG0QivA+Q5dPHGsYol7FJL5zASxA4Nuu1xOfhh923fsMjIiIi0+OER0RERKbHCY+IiIhMzyfv8Nx/2betIMxvDX7Otrgs6xsIJ4RgpN2rfkwK36qhSgsI81uRNxQsT7ugqSj9Fr2F32QTtwgcmCMln+FK4uV3WtgPVwJOXJxo6d4z1LZSVbgCeegj1HKYDBj6UokXQ6D7EZx3/Ft35n5cTQvgS4MaYVsdwul+4SrVx7pKPPkhO6gLqI4Q+k3hdWtDDEPt6L6FupKxCc8w3JHqqtebtJVr/wqFuhzeM3QtAwcM+wvDG8t+qfOEz6LLzlZM4ArSOOfx1De1cvjMD55h4/pV3B/Db3hERERkepzwiIiIyPQ44REREZHpccIjIiIi0/PJS8sUXHQD8vFmBULy6nQ+t1t383gLgWO0gjquZFxEqyWIwEsUzC6LiEsIUjyAFH2EVbZb9WEg2xL6lQTrYw2+SmVIogXwZbJlFHJGkDAHUGBkawLJimm7ajkKE6NwxaBdC7jeuHJyKjdTaGGr63oJt+0Wjtdjsy3TVZ5p2+X7CMVNWg06PWYlDcBs8nx4vFTUredJY5NEZipXq0//i52IuVQkvL8HBU3W5lN/wXMZzzvpw/QZWcMCUZwOVyDH8VSKhIIyknQ//kkM2v8Rvp7xGx4RERGZHic8IiIiMj1OeERERGR6nPCIiIjI9Hzy0vLuZTecbmlldNj2UFesRekQ5OAbWJUcJLfj3em+a0iAJvmYROAqN+9SeZckvSLbUaryksRWcmTpmGVfFGLJTEMxN5FYoQ/3tJRxUBcKq5BEjQm3pV/pANRfwerGR4prTVZGH2euUS1D4xclU9hWZfmnBC9Xz5jk17SqdHX0AiadJ8nE6SrVRLIrXcckLZz2e8Kq4Y30GgXjFd1wOu0gcfgp0NivY/GIjQ3F79plwXN6jJGNAepn6sS0DxOx/Nq6UqGe/4ic7gZ9iM+wR/AbHhEREZkeJzwiIiIyPU54REREZHqc8IiIiMj0fPLS8uFlF+ZebHZ926pv265P53P7my443b/Y9GPe9G6hJe0XJZmYUpXXlMwJ7A6nbSUBer/rMvWehLlA1iaZmqa/JHgeigR9PPR2oawGbW1CdeihLje9f1oTQsmRBG6UWGuXwfVmzzE4qSTaeZwRdeuma1Oux2B5sPQjioJXytTUBny5INiXuvAAYigmikOadOsKlEUfad9JQ0o74EUC7K9EWk5E8w+h1pdcxzH6/TEyKR3vGZRRy3MtbVd60HracI1SSba+JID7wWMzSRw+wphepF9dBPcWSuThyx513/QaoUT+EVPZf43f8IiIiMj0OOERERGR6XHCIyIiItPzyTk861/8/OTz4q67ObdrcHhg2/3+9PRqEOEYY6zgN+btLf0YDfsWr4fcojWF5qFbcvm37iWsCH9YXw5/WoHvEioj+IPuosyTyadhz6Nvqv7MU34/joqQz4QrfV8OaTse+v8XyD+hALPkl2f8XZ5cmaANhyQkjCqjY6JDEPonAVe7GRToSSvO065wzDouMOyS6qKuSMIbQ5+prhIfhzdiwF9YsJagQ5KjV+/vcOxEwYN0j4ahktG9lQZUUlW1qRhkSnteDtdbpn5WehvVtqZ1Xds9tN+VIaIfit/wiIiIyPQ44REREZHpccIjIiIi0+OER0RERKbnk5OWv/nT3z/5fHP3QytDgvLNkuTm7cnnLUimmxsILHzRu4XE2bovCcorCPjbQzvWgbS1QTn48pyVAhFJ1j4EdY0xxmJxeVXhtK4qYtN+6NEGQXfBArx/VRe0C/rsUOTBNYydHQTYUV21bUf4v8eCwiGvFClJzKZjJpDgSyTOMtWEoZjAYV+S2wKh+xwoNzdJNmvXEoP0aqFeJpXzl4GNmorrKOwHpCJ+q59WDU8JxkX6IgHWVQM2of70ZY9679JzjZ7B2P66ijtcR3wRJh1PZRu+4ADgM6Xs+zHHIZ1jHFL66+N8UGkRERGRZ4gTHhEREZkeJzwiIiIyPU54REREZHo+PWn5b5+KiC9vH1qZuyIjjzHGLayWviureG/XfXnaG1iVfHvb6xLrHkQAAAiESURBVCJuSrLyDcjUBCUmb4uASe0i2XkP5SokO5PsRRLdHtJlF8XcO+DKw9lK4lXoXS6zVOgkkTkV2lCAxtTbyzIfiuWBBLi/fBnHGGMsg1Rd6hsSp3cwXJO+jhdjD5KDUxGRyq3K2E9XcScWV8rgqUDaEsWhrlggLuVSkXYN0jW29cqk6zglvVBfBvhxv0TMhjbA84PAcZdI0aHQW1nDcxr7HsrV/qHnNHFtW5eUrh+mgNe20n4rGP3JPb+Io6PP4zc8IiIiMj1OeERERGR6nPCIiIjI9DjhERERken55KTldz89FZN+dnvfyrzZvG/bXkDS8sPqVATe7PvpvnzRpegdyMEkgL0o0vItSMskGhO19j0cLxWZqwC2BhkvFUNHIMiR2LyEuFmqf1P6kCXpfsxEzE2lZZQHcd/TtqWJwMkxPzQx9LQdJSkVJFA6x5sbEikvHy9N0U5I+zBKzQ4Srcc4J2BeFnqpX5ehCJyQJHKP0aXV9Hhp/ZX0/kuOSX246u+SXC1Ap+26Vtam+hPxnq9RJu/W/kmvN43phKeMpy4tX/8MTtr/ofea3/CIiIjI9DjhERERkelxwiMiIiLT44RHREREpueTk5a3f+M0RfnNiy4ov1p3kXkFctT2eDqf260hqfgI20DKpCXtX92cCs+UZFoTVs/VVaELEyexFlYghlIbUABDibjIwXBMaimm5da6whRRIumfJ4nMkJCd1E/jqV4TTKu+FvhvDAmAdEQaF/WcWH7N0nKT/k9FRBJgr4W7v0rw16dCJ/ulabad65Jrx+BrWUulT53kObCC5+G14xCfJyTShsJwfbmDXhxJSfa8tl+Tvx9jPO1FiEp+f3+8lzGuHYeP4Tc8IiIiMj1OeERERGR6nPCIiIjI9PxOHZ7ln/5R2/b6J9+ffH774rtW5vPNu7bt/tBP5W5VgsPA16FtFDxIfsirTQ8trDzse6rWkn7hLYdMfZ1FsEI71UW/TyeuyRj9N+Q1eDd5ONZ1Tse1vw3T+VD9seMUtIu8gt8k+zB4kK5bWl9Sf+LwpKuNE9V7Sr2x9N6q5fYQpnmtf0KkxkhtV+p00Ni/1kGiY67Bcbs2pDJpa7xSPZwjBV7WfdfpMxhXDb8cUpreM7Uc5DReXReRtpWea/VZkfTNOZLr/aF/B/yGR0RERKbHCY+IiIhMjxMeERERmR4nPCIiIjI9v1Np+X/8q8/btr/ze39+8vkrkJZfr3oY4Ri3bUsV6w6rbH5HKyATN0XSS+VBYlHkZlrhfAXtouDECrVrA+W2IFiT3PybHDS/Xb33rwDxjc6xlkpWqj9HFSlp5KTBZ13mo3bBeArlxJt1CUl8imhcytG9toWwS64sKBcKq0R1lMk9R5kzqPtjhsKl/2sF5/qjtoM57WsSXfmlgV5Tbf8KJOn0xYtE6E2f5yRFH6IXFdJnxeXxmgaeXk923W7Wl/8mrpaXA1zHyOR8uraP1vlBpUVERESeIU54REREZHqc8IiIiMj0OOERERGR6fmtScurt2/btu2fdCH5D15/ffL5b938qpXZg+KZyLskl92telry3eqmbcNk0SJfvdt3FZhSm3dHSF8ubbtZ9gTl9QKSoqEvmqwdpiofwGpcJqtnkxgaypBdYu37pcnXta6PmfZM9VVpPd0vPeY6TIBuZWIpN7tuSVtTQfJasZ/2qzJnmix7rVieyMhjnGvr5WOm+yX3DPGhgudj7SJQ4C7Ce3I+5+qq0HWktGdiRWOlbEteXBgjWyX++tdZroek7rT/K9eu7P4U6jE/xmTFb3hERERkepzwiIiIyPQ44REREZHpccIjIiIi0/Nbk5b/+7//g7btn//8l23bH738Xyefv1x3sfkv9q/atu2qK4U1mXiz6FIxpRe/WfckZxK77g+Xu2936O3aHfsxd0VIZiEWJDTQyarI/AAJyg+L3nZM6wQZvPYZpUKnIiUJyZeOd67+SL57Qhr2tZBYXq8bCaUkAu8COT9NgKY+XF+Z6noE4Z3Ou0Lp3tQuSp2+VoAmYTUZr9eOOdo33S8pl95r175IQC24ti6CrgclZCc8RcSvz4anpFAndaVC/bVjJ71GlaesGHBtGxJxPe3DR9vzQaVFREREniFOeERERGR6nPCIiIjI9HwUh+f//Lt/evL523/8rpX513/4Z23bv3jz39q2n67+8uTzA8R9rcBbuT90P2dTPJiXyx4yuFlQOBOs4Axhgd/t+wrtlQM4FxRQWKHgwU3osrTjgbu0PvS63kO7DuAb1baRw5O4IGOM8VAcJ/otl/qQrlEtlzgk545JXPtbelLXGsbhYdnPu/bXGODswH9j6HrU++NcudrXT3FZHor3Rg4anSMFYCZtSP2Na30mIlqJOxybieNGPCUMtPIU36het4/hYTxWF5H2Re1req5dC43za1lEK7HnVE9zE65m/lF9SKiqXt+PcTy/4REREZHpccIjIiIi0+OER0RERKbHCY+IiIhMz6PS8n/6n100JpbjP5fPsGourPS9PXY56leH09C/P9/1lcs3iy70ElVufrnqgYJvll2wvl1u27YHkJa/Xrw5+UwhfSQ7k9xXRTGSll+E22r9JKF9B/1KkiYJpDelvrtV7684DC0QuFNpMgnlS1djJ6rEmO5H7a911T4d48z5wNDf1VW9QTSn86YxhvJuaWsqYGLg4pXibHJtX5DIHoQ+jsHBj10sT1ehh2DRYyLn03Oz13XtSuLXsgpd0aRdqax9reT7lPNerS5f32tl8FSoT8o95aUBYj2uk7OvFb+prfQsTaTxDz1vv+ERERGR6XHCIyIiItPjhEdERESmxwmPiIiITM/iePy4qY0iIiIinxp+wyMiIiLT44RHREREpscJj4iIiEyPEx4RERGZHic8IiIiMj1OeERERGR6/j8jStwSQ8E4SwAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "learn.activation_stats.color_dim(-2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This shows a classic picture of \"bad training\". We start with nearly all activations at zero--that's what we see at the far left, with nearly all the left hand side dark blue; the bright yellow at the bottom are the near-zero activations. Then over the first few batches we see the number of non-zero activations exponentially increasing. But it goes too far, and collapses! We see the dark blue return, and the bottom becomes bright yellow again. It almost looks like training restarts from scratch. Then we see the activations increase again, and then it collapses again. After repeating a few times, eventually we see a spread of activations throughout the range.\n", + "\n", + "It's much better if training can be smooth from the start. The cycles of exponential increase and then collapse that we see above tend to result in a lot of near-zero activations, resulting in slow training, and poor final results." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Batch normalization" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To fix this, we need to both fix the initial large percentage of near-zero activations, and then try to maintain a good distribution of activations throughout training. In the abstract, they describe just the problem that we've seen:\n", + "\n", + "Sergey Ioffe and Christian Szegedy showed a solution to this problem in the 2015 paper [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://arxiv.org/abs/1502.03167). \n", + "\n", + "> : \"Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization... We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs.\"\n", + "\n", + "Their solution, they say is:\n", + "\n", + "> : \"...making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization.\"\n", + "\n", + "The paper caused great excitement as soon as it was released, because they showed this chart, which clearly demonstrated that batch normalization could train a model that was even more accurate than the current state of the art (the *inception* architecture), around 5x faster:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\"Impact" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The way batch normalization (often just called *batchnorm*) works is that it takes an average of the mean and standard deviations of the activations of a layer, and uses those to normalize the activations. However, this can cause problems because the network might really want some activations to be really high in order to make accurate predictions, they also add two learnable parameters (meaning they will be updated in our SGD step), usually called `gamma` and `beta`; after normalizing the activations to get some new activation vector `y`, a batchnorm layer returns `gamma*y + beta`.\n", + "\n", + "That why our activations can have any mean or variance, which is independent from the mean and std of the results of the previous layer. Those statistics are learned separately, making training easier on our model. The behavior is different during training and validation: during training, we use the mean and standard deviation of the batch to normalize the data. During validation, we instead use a running mean of the statistics calculated during training.\n", + "\n", + "Let's add a batchnorm layer to `conv`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def conv(ni, nf, ks=3, act=True):\n", + " layers = [nn.Conv2d(ni, nf, stride=2, kernel_size=ks, padding=ks//2)]\n", + " layers.append(nn.BatchNorm2d(nf))\n", + " if act: layers.append(nn.ReLU())\n", + " return nn.Sequential(*layers)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "...and fit our model:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.1300360.0550210.98640000:10
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "learn = fit()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "That's a great result! Let's take a look at `color_dim`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAADNCAYAAAC8XqoPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2dy5IcR5peIy91AwgQIBtg07ppo+nWQhrrkWkxO61legK9ip5MT6B30E47mcmM1JimmySAuuRFC3aPdf7/qcoPUVm8eJ+zq6CHh4eHR8CZceLzxX6/n0RERERGZvlTN0BERETkqXHCIyIiIsPjhEdERESGxwmPiIiIDI8THhERERkeJzwiIiIyPOuH/uN/Xv5Xv1n/kVj9278/3PDdu1Zm87sv+37f3bRt17/55HC/Z6tW5uy7Tdt2+6oPh/X7Xdu2vD3cdvO677dfLnpb7/pwunl5fM692PZtOxi5F98dtmtz2duw7Kc9TTDK99CsfenG1U3fkc6b2rEu+5593/v57nlvxNm7Xm5z1euv7NbZ9Vj06qft+eG+dD6r217X1Td3bVsdKwvo+yW06/pVH8PnpS/urnp/ra/7CW0vYFz0pvbjfdcH4vayH/PiX3pld88Pz7uOpWmapt1ZbxeN/Xo96DoSS7hGd59Q+w9vkt1Z9v/Fu3MY5+8OT4D6i+6Zxe74vUXjkPr17H2vawXjotZP55P2Ybsf4BLROdLYr+3a0nMN9tvA/VCv7Q/1HZaj67F+3wfi9Wf9IbzcHLbj/Nu+H/17RNeNnkUVuo7/47//t3sfiP7CIyIiIsPjhEdERESGxwmPiIiIDI8THhERERmeB6Vl+fHYX14c/L198zLa793ve7kqcpHcNy36pV9dd/Ht9iXYZMUwu3tO0iHsBtPrKrltQdxcgyi4uu113RZ58OqfQbR73c+HxFYS/uo5bS9aEZSD6zlO0zTtSzFq1wLWuXv3617uk/9zKCK+fwu3NUqTUOy8b7t9USTZ616G6r/+/Kxtq4L1Fo538SfoL5Iay+UleXe/ykRg2rfKwdsLGBNbumd6/2+qrAu3JI0TEqyrLE/CKomtJOFS+2/KxwskpPMHASD0vji8cHXcTxOL+Nef0X16WD9ds1u4RrfwfLqA867Pzfdv+vg9h48LaOzXsUP9vF/2ti7P4EOCMl7pQ5LNMxD2P/RyH97AByaLw7ZekGj8HK4HfrRx+Dd90EJ9QWOsnjdJ/SRYP4S/8IiIiMjwOOERERGR4XHCIyIiIsPjhEdERESGR2n5Z8Li+jAxeffmWSuzvO4yGQmYVUy7+CNIgSDJ7j7tdVFKaZVYMaUWXDIST69fH865UZAE05GSllcldPrDr/oBSawkQXn94bhASn2/CB06klFbXWEq9Pe/KenFsB/1fZqYXK/vEqTDD296H56DfFxFb+rDNCH7w+eHx6RxSG2luuiYtW3Uh5SGTeL65vLw7+df93uSxnQiB68gAfrD590Gv/i2HxPHeWlrFXDvY0mp0EU0pfFF4ildo3o9rl/1vr+EZx21/w4k3/ocoGcfSfBESxz+Yz8hEoFvXxwfT6sbeB6C0EvPippOPk3TdPtJkNROyeAwzs+KUE0foUzQh3fP4P4r7af7Y00fUDyAv/CIiIjI8DjhERERkeFxwiMiIiLDo8PzM+HD7z8/WoYcHuLs3eHLT1z1F1wWem++69lb0/n3JSwQ6qeAMVrV++qfD9/53r6k99O9rsTXWL+DUDV4R06+0c2nFNZYjkehdvDefPWhb9tclbrgfHiFc2hWWXUZryPc6VQ/r+Jd/4ZwSHBB6LrtSp+dQXgZXY+7rrRNZ+9q3XA8qIzGJq6yXXyH6ptNE/tSFOhY20r+ycV34PXQOZU+/P63kN4IvAfPitta7iMIJKV7mQLl1mXbBwgUJFcmGa/U9puXwcrl0z3jvLhEFJRaQx+niYNXq1ty92W/RhScSOdUww5pHJ5/ezwEcJru8fZujj/PW3DmxO2/KUG15NBtgjDNaZqag0S+Hz3DHsJfeERERGR4nPCIiIjI8DjhERERkeFxwiMiIiLDo7T8xKy+eNu27d981rat3x8aWbefduPz+m1fnhuF4SLfkThG0h5BoVRVyKNVw0mYIyGvSsq0X5U0/1wSjnlYjuQ4EmIxxBBWY28re5OYfdm3UchZ23eXybsocG+OB0GiQEyLqi+Ot3UHjuzypm+j1d7rNhI+cWVpGAObqxLAB23A8QTtP/u+H7OGwFG/0thf0tgp46KKwdM0TXcQgEkSfA15ROET7jVeob1vq6GY9IEDB0b29tfnDB2Px0AvV+XpLYxp6i8KtcO+qAGbtPp7ENL3w76Hf++XML5IdiYpuowL7HuQyAkMdi370kcJl/9CQZm9XAut3GXnSP8e1X/b6F4j4f0h/IVHREREhscJj4iIiAyPEx4REREZHic8IiIiMjxKyyeEBOXt19+0bYuvoNzF4dyTE2+zZN9NFZkxnbdvI7kvEWCpLhI3SRZtZSg5k5Jxk9Bp8tlQBM5WY9/XbSSLEuQTBiIwkaQop0nCJAKTMNyuCQmx8L9Oe2hIlYhp/G5phXAYTzXlmFK6KYF2olXu4ZyqpE5tpbFDEnG9jyjxtsrCP+xHdR1uo+u9A4l1hYnJfd+aCn0DqdA0BqgP6zhPP2ag8Vr7rN2P0zStaPV3+LCDPkroYn82DumcmpBM5w33Mo2Ldn/DM5LGCUn89Iyn/ql8+ByE9OBaUj/fvsj+nalyNv6bBeL3Q/gLj4iIiAyPEx4REREZHic8IiIiMjxOeERERGR4lJZPCAnKyz/8u7bt5lWPjFzdHBpZ17+CGFEAU1BLmi0mDlNdII4RLU0zlC1Jkq1CXpoSTHJfTRFFyRtEQZQ5Qe6r0hy1gWRLEveq9EltpT6kpNcqau5AQiQZmaRulMZru6APWYqGbaV6EoEpSRal7sVhZZSWTGOHuP7s+PWmsbOABN0k1ZqEUroe3IclgRbSvUnWprpYBK37ZWL2LvjQgtq1eXZ8v2mapkV51uHzhPoLxjSOi1KMkn3pAwQSpet9ih92wLM7Sb6++FM25qj9JCjXduAzAO7JOg6nqaf6032L140Gf7ln6GMGelY8hL/wiIiIyPA44REREZHhccIjIiIiw6PDc0JwZXQod/7H/gL//W/gRXbhDsLEzsFbuHl5fDVleudLPgWFmvXVlGm/46FztC1d1Ts9ZqsLA/7CttZilJFIOXfBitrkh3AwINRf3m2jAwOeFZ0jnhMFytW6KOww+N8pvLbkFsF1q+eJzgWAwYborxUXDvypNfkbwerfHCwKTYDVpiu4Ajm0FR2eIIQRxxw4Nhiw2drVt62uadtx14f6a0VOR+hG1XJ0OuQITfBcruOJPKvqoE0Tj+F6H1FwH0H3La2WXsuhMwlO4/rdvNXk+blwvA/T59VD+AuPiIiIDI8THhERERkeJzwiIiIyPE54REREZHiUlk/Ji+dt0+Z1X4749tNuhW2uDueeFEp19r4f8vp1n7PW8CcS9CgsEOVmCKqq9W1pZfEgdG6aIJQvlNBoledV7YpQpCVxFldVD0ISSVqmclXARGmWvFAKAAvOG1d/DyRyIg0LTGRRPB61FVe5b60IyrDMuQGpdLmtS333MiRAoxBby9D50H7BExpXGw/HOY7hKi2TYB0GbLbzXPYTp+BBfFbUbz3wAwEQxsOPMdbl+UdyMMrmMPZrv5JYjgT3FgrQaV0gwdexQv1F9wfeb8HYT59P7XrT8zw43F/jLzwiIiIyPE54REREZHic8IiIiMjwOOERERGR4VFaDqEU5Sopb968bEW2F31OiUJvKUYrcdM2EhZJ3GvHgyu/JQmNUoKLTLYFgRiTgylFuW4LU1FRkCzlqG8wnRVWrt5SInNpG5031bWBY1ZpEkVXuoxBCjEJmQT1IaYVl2OiYEh9ncjgYaoyybW1rXRtOUEZmhXIoiTKo/RLK15X6TO9RjQ26zXCFbyhMmg/tqNsIxE1lXDrea+hXTROUMSuq2fDWI2k/nvK3Zb0aEy+BlD+L22lexLHHN1HgQi8DBKzp+me1ctL26hda0jDputWt+E50r8NUNfdi1oI9vvIn2z8hUdERESGxwmPiIiIDI8THhERERkeJzwiIiIyPErLIduvv2nbVlMRmUFa3p33OeXds+PzzLvn8wXlKrmREEag0AZNrcmoKI6R8Aftn4rARqIgCb0kizZhjo4H4ttdD8hmKTMQbik1ls67to2aiknLwf+ioBQP20goTARSEj634XWr50R17c57w/aQ0Fv3RYkyFJlJgK3lUgkUhdtAqCexlcZm25fG9Cd9G4mteL/VMZaOkyDRmNpFIjMl+9b6sQ1hwjTJ4FFCNpWhuN/S1/gMJrkdnh+LILUZk9RhTNOzelk+XqAk51Qsb8ejbaEM3mTt8COOh/AXHhERERkeJzwiIiIyPE54REREZHic8IiIiMjwKC0DmKpMlKTlm8+7JXb3vM8pSTROEiNJZEaKyEWiHaV3osQayMck5FGaZpJ8jP0A+5HMWafv1AaS3FACpFToItMub8LrAcVqXYtdL1Rlwmm6R06sknr4vzGYJg1t7Qm3vRNX0Bc4nhIRMaSN6yAteZruETdJgi/9g+OJ2kUJwFVSD6VfPsDhn5T4jRZ8OlyDthIoqJZ9aUyToEzUexKFd7hvMdmXnon1fgivByY5B+M8FYGbrB2kwE/TfYn+veDuLEjmp2cK7Vb/baC2BrLzNE3twxdqu0nLIiIiIgUnPCIiIjI8TnhERERkeHR4Qja/+7Jtu357+LJ+c9nnj5ur/qJz/aG/i7z95LDc9jx74Z6s4EyhcDjVxXfi8N40aNoCguKSlasxII/e+VL767tuep9P4Y1rOMfAcdqBy4LAMZfbulx6L7N53jcu7yhospQLVsCepnw16OpKLDe9ss0zaCu5E+W6ofdBfgVco/WHsro1BRbCebe+v6f+2tdp8CDRXANypXBH2FSOSW1fwDnSOKfxVPu/ja9pmpYLSsXsm6p3Qb4R3mvkXpX66ToSeI2ClMQ9PPtir6f+yxqujE4BmLSt1QX9lTwjp2ma9tVNhPsb/x2g5yu1o5aB+tHPCcI6DR4UERERKTjhERERkeFxwiMiIiLD44RHREREhkdpmXjRU+3uXnZDq0rKFIJE225edWnr7pPjAh4JyiRtJZIbke5XpbY9SI0omKE4ezyAD4U5CvvaHm8XNQIlVpDoWhvS0DyScIOUOQwZpHbVoEkK1gsD+JIFiantdDppsGFSF68kXsYO1UUru5OES/Ju6Z+7F9l5Y7Ba7evw/iD5uI5rKkMNoz7Ee6vKoiSiQl0USFk/VMAwvHQF8jbOIUQvDXTEoL4i797Cc4HOkQJCa0hiELp6X7tqX6PQHQbJ4qrtBb4/ejnMD6xtpTbQMyz40CINuH0If+ERERGR4XHCIyIiIsPjhEdERESGxwmPiIiIDI/Scsh+BatB70qK6EWfP+Kq2+vjgmSaEozibNmWrPw9TSzDcRpoTebM6qe21mPu09RjTL2tMmcvQyJzuhp0q4s2kgicrNYcCtC4OnCQNooyZyijtm2hDIkCdB2bYUpwIj8uSDxOU6dhvEb7UTEU6ku7SFhFC/T4/cByfgfFUxr7wf8G0zFRii7iehqMS0HOSUo3tZ0SplH0Lmye9wtCKcT4sUctRn0TPtdawjt9zPCY+6iKxr3I7A9hYjE7gfYj4f0B/IVHREREhscJj4iIiAyPEx4REREZHic8IiIiMjx/89Ly6ou3bdvmzcu2bXdGiaeH20hC24HsfNurj0RjFFYDCw2FNpJ+w9FQ900Th1eQXNok1lCmxsTQ0q5U/CbonJqoGZ747vJ4si+mHlMgaSAap9cjSV39oeDhn7tnqZA+LyUYJdxAdMTE3sdQx3komeJ9WquGe3L1gZJ9g2OGbagC8TRl4iymC4O8i88ZKJdAKd3tvJNY8OmecUE3V0l5x1Ro6msar6UudOfTBOi2I+1HNyA0KxDqU5kaz7vY5vhRAj2Dg4dWcMmO4i88IiIiMjxOeERERGR4nPCIiIjI8PzNOzy0Mvrmee+Wi//XU7tu/+7wBXuy4vk0TfesNFzLZAF89P6+7poE/k3TNE1pGFoNAKP6Yb8NuCytreGK8/j+u7YrDOkj9yD1A1oReq+NPsXxyjBILxgDeD4Y3AfHnPl+nUj6Iln9/YdydIBkR9gvvk2PL/28X1Igaa+rh2JCgN0lNSJZPhuOB6ugp0F0Sf+k9ddSqU9D7k8d1zh+6RzDgM3WDFzhHMYAeSrln4tt6sVgQGiwBHkabEg+ZHUA0RWFbfBcaCGG4SrxtOL8VJ4NqDzp8IiIiIgc4oRHREREhscJj4iIiAyPEx4REREZHqXl7961TZvnb9q2D2+7aXXz8nC+SMFeFKCFoYJVyCO5LxB1fyh2XO7D/UjkowC+QBTDMtjWcjyS3CgAjA4a2Np7kA7TlcTbouG0sjucN60m30RBqItCtSIBPVwhnMMVqf7StnDsIGjJliJw4iw3H5d3o/0mXkm8StcseYfjqUi42C4Aw++C1dFRPifpOjCU00BHCpmrY5/HF9SVrF6P43CeAP3nraWu402YpntWXk+ekdT+QOqm649tCFclT4IyMaR0H4QKkl+NH8IEH+k84gOEfz3MxxUXERER+eXhhEdERESGxwmPiIiIDI8THhERERmevylpefFPf2jbbl90q5hWRt9ckvhWylzBQUG02kLicEK+CnZJdSVpL115HazlKlyiZIqr5B6vi4TuVNbu+1EyLpQjmZPuDBIDKyDfkcyerCKdyokkYrf9ILV5ClNv62rvsaSJonQSVx1uqzJnmKhLleE1Cq43SsWBkMwrTc9rA4HiNK6qTmPseLsQOKd9XYGcxgSlNqfierAfCtDU1vpsCJPasX/KdaNhj/f3xy7//UAbqP7kXx58xjxCsu8HgP1Qbg4+SlBaFhERETnECY+IiIgMjxMeERERGR4nPCIiIjI8f1PS8vJ/f9O3/e7Ltu361UXbRjLt5tnxY26fhUmsRSjEJNBUmKuBwyAFYsonbFtukjThTPrdQyptO0+S3khGJVoUclBmyuXBmmCNbSUxlK5lJF1D9Rf9AK2t1AYSN0N3vkqMKNzStkTAxIDYrGFN/CaBmJJ3QxE4kcFRZE/akYrZRK2K5FT6aOCWbvDg/k779Y6iqMt+Z8H4naZpt4Zyta9pSNOzjtoapLA/Ss6v50n3AmzancN53wX3Xziekn1jmRpl7fI3fiQC2+jDmnq5qW8+Uur3Fx4REREZHic8IiIiMjxOeERERGR4nPCIiIjI8AwtLa++eHvw9/U/ftXKbC+7kXfxpy5Hvfuyl6siMyUoY3IwgAnAtQxdrUR0TGVnStmF9leBbZfIndOEo60KcpjyGaY2N0ESk6NhU5gU3WRgumZJ0uh0jzzYCj1iWy1CSankmGKSczXqjx/vh8pgW21HKk5vemPxGgWkqbHteifnc0+5RRucUFco1Ld9wyRyEt6z6N1sGwnJzQ1+hKQeSeQp9CFEHZr03ElTiGtd4f0Xi8YBsdwcXCP8EIJk8FURrMP7Az96qJtmhlD/Nf7CIyIiIsPjhEdERESGxwmPiIiIDM/QDs/uq0OHZ7/qLwFvXvU5393zXu72Ra9/e1FW+obe3IWhc+19axxC1tvaMs4eEToXhWPB+914peHkXTftBr5A64vQ4cG6MHSsbEtdkCTADFcRp1A7CFabu7o1sIfAt2is4PUOwgjRbaCgTBpkAelu2GdBR6I/E4T54fLZ8y5cC2CcJjxvDNtLxnDodMz+3+c0ZLUl0UGZ0E3EVb3rvUX71WcAtYvaQY4KPRcSdxCvI1yQ2SuvZ0GyWK5uovFF543jMAiahDDCh/AXHhERERkeJzwiIiIyPE54REREZHic8IiIiMjwDC0tL643B3/fvO6ne/1Zn/NR2F4SKpgGoSVBVQj5ciSZ1tWtU7k2PGaT1UiKJkiii4ITSXKDclXcI5kz9fhIMqz7PqZfa/0oGFIbgrGT/m9MOg5r06itaV31HqHzgfoXMM6bIJmOk/Sy1XPC8w7rr+eJ43BeON3+gvoQdkuvUS1HkikFqlJdyThPxdZVIKiiQBymdSbjPA2frFDT048l2rMoFN7p34YEOm/q+7RcJRTLo4DNj8RfeERERGR4nPCIiIjI8DjhERERkeFxwiMiIiLDM7S0vH1xcfD3LSQok6C8uezbKGmySso7kgfTVZHnCrCUvFtXFU5X2E5X5a3TZBLOYHXraGXpVOZMzolWV6YVfuNA0sN99yTXUsouJQ5XyTAUDDHotXZ16iqGSaxNGKYxB4L4AiR+6rO2H41D6FfIm+37zU2bBfA6xgJ3kBKMEmgg9NLxML04HK9JGnaaFB3J2gAJvcnxdlm6Oo2xdnnTttIzZRk8K9J/B5IEeequue1P+2sD5cq4o+dCtDL6NDUBGvvrI+9vf+ERERGR4XHCIyIiIsPjhEdERESGxwmPiIiIDM8vUlpe/NMf2rbd2apt+/b3Vwd/by+64LQ97/Vvr7pBRUnLSbJyE4in6Z7kz5mgABYIktB2SrNNQPExdMn2lCxa66d2zZRRF5AEuoBpP/mprR9D23lJwl9pP0qB1DdBIPOCBHvygNfbo+1C4OIuL7K6lue7o2UIHGPb431Iwm18zEUVMPtAwbEJ/bPvg6fvB+3ne+twWyzdw3MHZdp6TlQG7qOsX0OZmpK12/0A/QXP2z09jOix3IqF/RV87EHXEaG+KH2932bjMBLSifDjFerrXhfsNzetmq5Z+qHNXw7zUaVFREREfoE44REREZHhccIjIiIiw/Ozd3hWX7xt2/bXPfHo5tfP+rZPD+dzm+e9/rsXx1dBn6Zp2pNjUbfR9JHe8SevuuMVZSlMLFmxNvMFiPZOPHwniy4LvOPt7YKNQXBY6jZUV+PPW4NN4btuaGvtC2wBqQf0Hrv2NaksoaeE78Rrv2K4IhyTggfrLUN9A+N3C95Cq5/GdG8Wez1UrrBK7qtpmnYQyEbn2doQim/L6halXgk1YXn8mEsIAdxDV9D1TvaLx9NZdVnC8w6PWZ+lc8c0Ej5iqK8rmHUZOU98zOS5P99BCvcLvNb4ej+Av/CIiIjI8DjhERERkeFxwiMiIiLD44RHREREhudnJy1XSfn2H37bynz/m54WuIFQwbtPSpnnmTS7uwQBDEPNyt9JmekeASxZVRglUwo+q8eDuohQil4uijwIZVbn3dzc7XpDVonomMqctV8ftTI6taNsgLGzh3MkqijNcm0WalfPk/q5iq7TNE07um5BoBxdD+pXkufn9v8qWQI+vN68cnUgsdIhYb/lBfR1vSbBfXXfMVsbKLCQyoVjIGFFci0FLpZtC7pnwnHYn5HHjzdNHLBJ1Ht3SW0I9iP4wwgoF4QFLuHZis+FmdeIoL7YUQBiuefT8UXPp9aujxSU8TiPrkFERETkZ44THhERERkeJzwiIiIyPE54REREZHh+dtLy/s1nB3/fvO6xx5urLkLdvuzbqqS8o5XRaRX0dIXzItwuQBIjCQ3FzSA9tUm599CE2DDJNGWueLoGebDWtYPzpoRbkgCr5EaJvakAzWJulYMp8hRWDQ+kTKorvd6V3S7bj/5vh9pRxw/JtSlzx05dIXyasmtJ4yRNZ23J3bQ6dyijLmFcZG2Y12GxJDuzD9Ny25KOu4K+x/soqJ+fAVFVXP9q3jXawf1Q711MNU8vbZKEDG3AeznoszTh/Qzk6TpeKUj/lM+6dGz+63FmHUVERETkF4QTHhERERkeJzwiIiIyPE54REREZHh+NGm5JihP0zTtvurbNleHkvL7N31ORoLy7UuSj0vdL0BiXYOASenIF11oWwbyI4mzKEDXImR7AZRIWkUuTB8NReZErEMxNJTJ5kqZcyG5L6VL0ZRaeny/aerXbQ2J05iYDOOplsOUWoDkwTU8ERIRNBUdk/2IuTJqIp/fVy7JOab7KJFFH9Ou5N7K08lJ7K9lSEamsdnLrcpzjK7jEp63mMoejGtMk8brke2b1FXPkeqaex3vO2aFpd+sD5Oxwtdt7kcV8z4cmdv2v8ZfeERERGR4nPCIiIjI8DjhERERkeH50RyeGig4TdN0/faqbfv27w6bdPspBAo+g3fKsG13Xt6j0uqx4E4sz8HXwQDBw20r8IHSd591G73fXZMPNBNaxZZWrCXaeZPHELosPcwPVuCd2a7UgSGSd8Nb8gyg/qQcHW+9hlWRgRqYRnXh+YT/u1PHSjpOiMwvO+4ppWyhfvZDjofOxTlxJ/zfyHlm0T3nSP0a+H2YkwohoqvEgQmfO4vF8VA7gp4Vm0V2Qerzle7bNTmfwKbsewbji8YmnWHiitK1xRXIYd/k35V01fME6sOkL+h8PrZd/sIjIiIiw+OER0RERIbHCY+IiIgMjxMeERERGZ4HpWUKC9x+/c3Rctf/+FUrc/eyH4pCBW9eH0pImysQlGHbjlY9Pz+UoxYgI9PKyeszWtUbyhX5isqQaEXb1kGoViodnhIWCh+xJPFRshWLSWCsfXF5dnu0zH11YcuKiLheZVIxSd1JH6aSXi2Hq1aDgPmYMLSEpH46R5S8QZJNjkcPuLlCLBEuTD/VjMptvJL1zOBBeFbM/biA9ptL2l90zNqutL9W4Ur1tW1rEI0ha5Sf+7vD/qfzWdKq4TMf59Sv58E9k5KHdR5C7aJzvAg/9ngs/sIjIiIiw+OER0RERIbHCY+IiIgMjxMeERERGZ6Hk5ZfPG+bVlMXmW//4beHf7/q1b77oi8pe/2rfsjtxaHltHkOstQFCL60mnlJVsaEY5CFz0BaJmG4Sm1n4SrVidycpJZO0/zE5FOm5aZ1kYT21JJsJe0vKrcrY+AxfZhwimTRv3ABAmZaV21HTZFN94sJUo+nKUuATtNsqaVJ71D9c887lTTpWXRK6hmlfTO3D1OihGmSlsPE4blQ/TQu2n6hodw+SsiaheeYtPVpP4NhqL/qv6+neN76C4+IiIgMjxMeERERGR4nPCIiIjI8TnhERERkeB6Ulm+/et223bzu0vL1q2Up0wWk6zPjgYsAAAYOSURBVM9BwLwAgepZEfLOYdl7SExeBonJK5CKL87uel1gbZ2vezprLbYGmXBu+vISlLMd6GQk6a0Xp5Mak4TNVDw+CxJP08TbKG02rIv6lfr/dnco3qcp2sk1ojYQ6bhIWME4wQTrUv8ZJdCGGuhmf/ismJsmTcdM++Ex4vqcMtMU3jPQr3MToB+T2vyUMv7cjyymaX7y7tMmwzP9E53OKZPOH/PcPCvb5o5pqj9NUqek5eSjhI/9iMNfeERERGR4nPCIiIjI8DjhERERkeFxwiMiIiLD86C0/H//42Xbtj3v5bZXh3/fQTry9nkX8hbPING4pCOfX3Sp+AyWva+pjNM0TauSrEyyJSVPUjkSgWviKUnLj5H02n4nzQedz1xJlpgrnlJfVCEW9wvFN0oTvlofjkWqi/Y7o4TvQBRM21qlQ+Kk4/AR8uDZdHhvnbJdaQI0XQ9sxxSIxsApReMtjOnkniEhPaW2n5Kdnzo1nUg+enhq5l7b9MOIhFRIn71veIpz27GGj4fm1rWAe/kh/IVHREREhscJj4iIiAyPEx4REREZngcdnu/+DbwHhoC//WUJ+Lvq71ovznpwH61KXv2Zc/B1riAskPyZ+o4UVzyHbefL3lbyCpKAvzW8d97Re/lS12aXRFfd45EELgsx16fAlcWfONyN+vBydVfKZPthqB2M8zqe6jWbpmk6h66f66TMXc187orqVFcKhivCvVU9m1OuCE99n9xr91H3TYM/ibnXaD0db+spwxunqZ8nuibwXKN+pf5P9iOSunC/mV5g6t0k9T/mnkw4ZVjkUz8/0rFZnxX0PPlY/IVHREREhscJj4iIiAyPEx4REREZHic8IiIiMjwPSstnv37ft4FoXFccvwRB+RzC/C5Wvdxl2UbS7zlso3I1VJCEM17xOpOjKLSw1TVTcqP97kDaS0S+VGJOj5lA7UKJeHFckEyFvEiAPqGIeEpIgKfrloSVnTLkbJpOK0TWezcVXef2RSr0cp9tjpaZK2A+5YrkH8NcaTxv/+nCAnf7x0urf+GUsnxSP4ViPvUYSINRa7lT9s2j6sq+2/ko/IVHREREhscJj4iIiAyPEx4REREZHic8IiIiMjwPSsv//tdft22X655yXOVBkpEpvfgMhMUqH58tKNGzi1ArSCT9scXAx0hvd/vjhtYF1L8N5qxpajNJmRfBftsw0Xhu/5CQTud0ysThhJ9C7nvqYxK1/rnp4feV6/v1ZwyRjute/+meC6lMvVvN+1Dhx5bnqR2PacPcupK+oOdO8iFJekz6YGN2/eFQPfU5zWlHOg7nfggxt65TtMFfeERERGR4nPCIiIjI8DjhERERkeFxwiMiIiLD86C0/J8+/19t2wqTcHdHy5wturScyIMkIyeibgrJwiRWbjFldfnRZe5jW8qlcl/db5r6Oe2WpxUfa/9TemeyHxFf71UXW2nfp4TaNbcNjxnTP/Z5n5LHCMRPLaUn/FzGwI/N3OdyKuKfkrStp7xuP9e66COgek1+ivOZe8yPvWd+OXeYiIiIyEyc8IiIiMjwOOERERGR4XnQ4fkvn/zPWZXSe1ryW4haLvVp7vb9VGo5KnMLDk9SF7WN6trBnDLxbtI2EHPdImoX8ZSO0GPC12r7V+Gq9+l5J/VRXbRfPc9TtzVpw1we019zr9FPQXKeyTneV+4peUwb5vqEc4PoUubef/J40nF+yrpquVPcV44OERERGR4nPCIiIjI8TnhERERkeJzwiIiIyPA8KC3/h/PLWZVu97BKMghtd3sIQZq2pUwPmLuDum73VH/dj2TnPue7Bvn4GiTiKhZf78+gXVRXL1frokBEkrbmhn2lgU1JMOPcVbFPTSp1V0650jdBdVUorPOUJG1I25H2M9X1lNcovT7ptU377KfmlP1MPOZeTvrwMdctYW77qQ/Tvq7lnrqux3DK+ufW9dRj+C/4C4+IiIgMjxMeERERGR4nPCIiIjI8TnhERERkeBZ7kH1FRERERsJfeERERGR4nPCIiIjI8DjhERERkeFxwiMiIiLD44RHREREhscJj4iIiAzP/wfEY39V6lGjrwAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "learn.activation_stats.color_dim(-4)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This is just what we hope to see: a smooth development of activations, with no \"crashes\". Batchnorm has really delivered on its promise here! In fact, batchnorm has been so successful that we see it (or something very similar) today in nearly all modern neural networks.\n", + "\n", + "An interesting observation about models containing batch normalisation layers is that they tend to generalise better than models that don't contain them. Although we haven't as yet seen a rigourous analysis of what's going on here, most researchers believe that the reason for this is that batch normalisation add some extra randomness to the training process. Each mini batch will have a somewhat different mean and standard deviation to each other mini batch. Therefore, the activations will be normalised by different values each time. In order for the model to make accurate predictions, it will have to learn to become insensitive to these variations. In general, adding additional randomisation to the training process often helps.\n", + "\n", + "Since things are going so well, let's train for a few more epochs and see how it goes. In fact, let's even *increase* the learning rate, since the abstract of the batchnorm paper claimed we should be able to \"train at much higher learning rates\":" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.1917310.1217380.96090000:11
10.0837390.0558080.98180000:10
20.0531610.0444850.98710000:10
30.0344330.0302330.99020000:10
40.0176460.0254070.99120000:10
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "learn = fit(5, lr=0.1)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
epochtrain_lossvalid_lossaccuracytime
00.1832440.0840250.97580000:13
10.0807740.0670600.97880000:12
20.0502150.0625950.98130000:12
30.0300200.0303150.99070000:12
40.0151310.0251480.99210000:12
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "learn = fit(5, lr=0.1)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "At this point, I think it's fair to say we know how to recognize digits! It's time to move on to something harder..." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -2655,6 +3611,13 @@ "Convolutions are by far the most common pattern of connectivity we see in neural nets (along with regular linear layers, which we refer to as *fully connected*), but it's likely that many more will be discovered." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "TK add last section to conclusion" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -2686,7 +3649,26 @@ "1. Run conv-example.xlsx yourself and experiment with \"trace precedents\".\n", "1. Have a look at Jeremy or Sylvain's list of recent Twitter \"like\"s, and see if you find any interesting resources or ideas there.\n", "1. How is a color image represented as a tensor?\n", - "1. How does a convolution work with a color input?" + "1. How does a convolution work with a color input?\n", + "1. What method can we use to see that data in DataLoaders?\n", + "1. Why do we double the number of filters after each stride 2 conv?\n", + "1. Why do we use a larger kernel in the first conv with MNIST (with `simple_cnn`)?\n", + "1. What information does `ActivationStats` save for each layer?\n", + "1. How can we access a learner's callback after training?\n", + "1. What are the three statistics plotted by `plot_layer_stats`? What does the x-axis represent?\n", + "1. Why are activations near zero problematic?\n", + "1. What are the upsides and downsides of training with a larger batch size?\n", + "1. Why should we avoid using a high learning rate at the start of training?\n", + "1. What is 1cycle training?\n", + "1. What are the benefits of training with a high learning rate?\n", + "1. Why do we want to use a low learning rate at the end of training?\n", + "1. What is cyclical momentum?\n", + "1. What callback tracks hyperparameter values during training (along with other information)?\n", + "1. What does one column of pixels in the `color_dim` plot represent?\n", + "1. What does \"bad training\" look like in `color_dim`? Why?\n", + "1. What trainable parameters does a batch normalization layer contain?\n", + "1. What statistics are used to normalize in batch normalization during training? How about during validation?\n", + "1. Why do models with batch normalization layers generalize better?" ] }, { @@ -2700,7 +3682,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "1. What features other than edge detectors have been used in computer vision (especially before deep learning became popular)?" + "1. What features other than edge detectors have been used in computer vision (especially before deep learning became popular)?\n", + "1. There are other normalization layers available in PyTorch. Try them out and see what works best. Learn about why other normalization layers have been developed, and how they differ from batch normalization.\n", + "1. Try moving the activation function after the batch normalization layer in `conv`. Does it make a difference? See what you can find out about what order is recommended, and why.\n", + "1. Batch normalization isn't defined for a batch size of one, since the standard deviation isn't defined for a single item. " ] }, {