![\"Entity](\"images/att_00018.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The insight is important because we already know how to train linear layers, so this shows that from the point of view of the architecture and our training algorithm the embedding layer is just another layer. We also saw this in practice in the last chapter, when we built a collaborative filtering neural network that looks exactly like this diagram.\n",
"\n",
"Where we analyzed the embedding weights for movie reviews, the authors of the entity embeddings paper analyzed the embedding weights for their sales prediction model. What they found was quite amazing, and illustrates their second key insight. This is that the embedding makes the categorical variables into something which is both continuous and also meaningful.\n",
"\n",
"The images below illustrate these ideas. They are based on the approaches used in the paper, along with some analysis we have added.\n",
"\n",
"On the left in the image below is a plot of the embedding matrix for the possible values of the `State` category. For a categorical variable we call the possible values of the variable its \"levels\" (or \"categories\" or \"classes\"), so here one level is \"Berlin,\" another is \"Hamburg,\" etc.. On the right is a map of Germany. The actual physical locations of the German states were not part of the provided data; yet, the model itself learned where they must be, based only on the behavior of store sales!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![\"State](\"images/att_00015.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Do you remember how we talked about *distance* between embeddings? The authors of the paper plotted the distance between embeddings between stores against the actual geographic distance between the stores in practice. They found that they matched very closely!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![\"Store](\"images/att_00016.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We've even tried plotted the embeddings for days of the week and months of the year, and found that days and months that are near each other on the calendar ended up close as embeddings too."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![\"Date](\"images/att_00017.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What stands out in these two examples is that we provide the model fundamentally categorical data about discrete entities (German states or days of the week), and then the model learns an embedding for these entities which defines a continuous notion of distance between them. Because the embedding distance was learned based on real patterns in the data, that distance tends to match up with our intuitions.\n",
"\n",
"In addition, it is also valuable in its own right that embeddings are continuous. It is valuable because models are better at understanding continuous variables. This is unsurprising considering models are built of many continuous parameter weights, continuous activation values, all updated via gradient descent, a learning algorithm for finding the minimums of continuous functions.\n",
"\n",
"Is is also valuable because we can combine our continuous embedding values with truly continuous input data in a straightforward manner: we just concatenate the variables, and feed the concatenation into our first dense layer. In other words, the raw categorical data is transformed by an embedding layer, before it interacts with the raw continuous input data. This is how fastai, and the entity embeddings paper, handle tabular models containing continuous and categorical variables.\n",
"\n",
"This concatenation approach is, for instance, how Google do their recommendations on Google Play, as they explained in their paper [Wide & Deep Learning for Recommender Systems](https://arxiv.org/abs/1606.07792), and as shown in this figure from their paper:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![\"The](\"images/att_00019.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Interestingly, Google are actually combining both the two approaches we saw in the previous chapter: the *dot product* (which Google call *Cross Product*) and neural network approach."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Beyond deep learning"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"But let's pause for a moment. So far, the solution to all of our modelling problems has been: *train a deep learning model*. And indeed, that is a pretty good rule of thumb for complex unstructured data like images, sounds, natural language text, and so forth. Deep learning also works very well for collaborative filtering. But it is not always the best starting point for analysing tabular data.\n",
"\n",
"Most machine learning courses will throw dozens of different algorithms at you, with a brief technical description of the math behind them and maybe a toy example. You're left confused by the enormous range of techniques shown and have little practical understanding of how to apply them.\n",
"\n",
"The good news is that modern machine learning can be distilled down to a couple of key techniques that are of very wide applicability. Recent studies have shown that the vast majority of datasets can be best modeled with just two methods:\n",
"\n",
"1. Ensembles of decision trees (i.e. Random Forests and Gradient Boosting Machines), mainly for structured data (such as you might find in a database table at most companies)\n",
"1. Multi-layered neural networks learnt with SGD (i.e. shallow and/or deep learning), mainly for unstructured data (such as audio, vision, and natural language)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Although deep learning is nearly always clearly superior for unstructured data, these two approaches tend to give quite similar results for many kinds of structured data. But ensembles of decision trees tend to train faster, are often easier to interpret, do not require special GPU hardware for inference at scale, and often require less hyperparameter tuning. They have been popular for quite a lot longer than deep learning, so there is a more mature ecosystem for tooling and documentation around them.\n",
"\n",
"Most importantly, the critical step of interpreting a model of tabular data is significantly easier for decision tree ensembles. There are tools and methods for answering the pertinent questions. For instance, which columns in the dataset were the most important for your predictions? How are they related to the dependent variable? How do they interact with each other? And which particular features were most important for some particular observation?\n",
"\n",
"Therefore, ensembles of decision trees are our first approach for analysing a new tabular dataset.\n",
"\n",
"The exception to this guideline is when the dataset meets one of these conditions:\n",
"\n",
"- There are some high cardinality categorical variables that are very important (\"cardinality\" refers to the number of discrete levels representing categories, so a high cardinality categorical variable is something like a ZIP Code, which can take on thousands of possible levels)\n",
"- There are some columns which contain data which would be best understood with a neural network, such as plaintext data.\n",
"\n",
"In practice, when we deal with datasets which meet these exceptional conditions, we would always try both decision tree ensembles and deep learning to see which works best. For instance, in our case of collaborative filtering you by definition have at least two high cardinality categorical variables, the users and the movies, so it is likely that deep learning would be a useful approach. But in practice things tend to be less cut and dried, and there will often be a mixture of high and low cardinality categorical variables and continuous variables.\n",
"\n",
"Either way, it's clear that we are going to need to add decision tree ensembles to our modelling toolbox!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Up to now we've used PyTorch and fastai for pretty much all of our heavy lifting. But these libraries are mainly designed for algorithms that do lots of matrix multiplication and derivatives (that is, stuff like deep learning!) Decision trees don't depend on these operations at all, so PyTorch isn't much use.\n",
"\n",
"Instead, we will be largely relying on a library called scikit-learn (also known as *sklearn*). Scikit-learn is a popular library for creating machine learning models, using approaches that are not covered by deep learning. In addition, we'll need to do some tabular data processing and querying, so we'll want to use the Pandas library. Finally, we'll also need numpy, since that's the main numeric programming library that both sklearn and Pandas rely on.\n",
"\n",
"We don't have time to do a deep dive on all these libraries in this book, so we'll just be touching on some of the main parts of each. For a far more in depth discussion, we strongly suggest Wes McKinney's [Python for Data Analysis, 2nd ed](https://www.amazon.com/Python-Data-Analysis-Wrangling-IPython/dp/1491957662/ref=asap_bc?ie=UTF8). Wes is the creator of Pandas, so you can be sure that the information is accurate!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will be looking at the Blue Book for Bulldozers Kaggle Competition: \"The goal of the contest is to predict the sale price of a particular piece of heavy equipment at auction based on its usage, equipment type, and configuration. The data is sourced from auction result postings and includes information on usage and equipment configurations.\"\n",
"\n",
"This is a very common type of dataset and prediction problem, and similar to what you may see in your project or workplace."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Kaggle Competitions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Kaggle is an awesome resource for aspiring data scientists or anyone looking to improve their machine learning skills. There is nothing like getting hands-on practice and receiving real-time feedback to help you improve your skills.\n",
"\n",
"Kaggle provides:\n",
"\n",
"1. Interesting datasets\n",
"2. Feedback on how you're doing\n",
"3. A leader board to see what's good, what's possible, and what's state-of-art.\n",
"4. Blog posts by winning contestants sharing useful tips and techniques.\n",
"\n",
"Until now all our datasets have been available to download through fastai's integrated dataset system. However, the dataset we will be using in this chapter is only available from Kaggle. Therefore, you will need to sign up to Kaggle, then you need to go to the [page for the competition](https://www.kaggle.com/c/bluebook-for-bulldozers). On that page click on \"rules\", and then \"I understand and accept\". (Although the competition has finished, and you will not be entering it, you still have to agree to the rules to be allowed to download the data).\n",
"\n",
"The easiest way to download Kaggle datasets is to use the Kaggle API. You can install this using pip by running this in a notebook cell:\n",
"\n",
" !pip install kaggle\n",
"\n",
"You need an API key to use the Kaggle API; to get one, go to \"my account\" on the Kaggle website, and click \"create new API token\". This will save a file called `kaggle.json` to your PC. We need to create this on your GPU server. To do so, open the file you downloaded, copy the contents, and paste them inside `''` below, e.g.: `creds = '{\"username\":\"xxx\",\"key\":\"xxx\"}'`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"creds = ''"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"...then execute this cell (only needs to be run once):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cred_path = Path('~/.kaggle/kaggle.json').expanduser()\n",
"if not cred_path.exists():\n",
" cred_path.parent.mkdir(exist_ok=True)\n",
" cred_path.write(creds)\n",
" cred_path.chmod(0o600)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now you can download datasets from Kaggle! We'll pick a path to download the dataset to:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Path('/home/jhoward/.fastai/archive/bluebook')"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"path = URLs.path('bluebook')\n",
"path"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#hide\n",
"Path.BASE_PATH = path"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"...and use the Kaggle API to download the dataset to that path, and extract it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(#7) [Path('TrainAndValid.csv'),Path('Machine_Appendix.csv'),Path('random_forest_benchmark_test.csv'),Path('Test.csv'),Path('median_benchmark.csv'),Path('ValidSolution.csv'),Path('Valid.csv')]"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"if not path.exists():\n",
" path.mkdir()\n",
" api.competition_download_cli('bluebook-for-bulldozers', path=path)\n",
" file_extract(path/'bluebook-for-bulldozers.zip')\n",
"\n",
"path.ls(file_type='text')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Look at the data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Kaggle provides info about some of the fields of our dataset; on the [Kaggle Data info](https://www.kaggle.com/c/bluebook-for-bulldozers/data) page they say that the key fields in `train.csv` are:\n",
"\n",
"- SalesID: the unique identifier of the sale\n",
"- MachineID: the unique identifier of a machine. A machine can be sold multiple times\n",
"- saleprice: what the machine sold for at auction (only provided in train.csv)\n",
"- saledate: the date of the sale\n",
"\n",
"In any sort of data science work, it's **important to look at your data directly** to make sure you understand the format, how it's stored, what type of values it holds, etc.. Even if you've read descriptions about your data, the actual data may not be what you expect. We'll start by reading the training set into a Pandas DataFrame; note that we have to tell Pandas which columns contain dates. Generally it's a good idea to also specify `low_memory=False` unless Pandas actually runs out of memory and returns an error. The `low_memory` parameter, which is `True` by default, tells Pandas to only look at a few rows of data at a time to figure out what type of data is in each column. This means that Pandas can actually end up using a different data type for different rows, which generally leads to data processing errors or model training problems later."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(path/'TrainAndValid.csv', low_memory=False)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Index(['SalesID', 'SalePrice', 'MachineID', 'ModelID', 'datasource',\n",
" 'auctioneerID', 'YearMade', 'MachineHoursCurrentMeter', 'UsageBand',\n",
" 'saledate', 'fiModelDesc', 'fiBaseModel', 'fiSecondaryDesc',\n",
" 'fiModelSeries', 'fiModelDescriptor', 'ProductSize',\n",
" 'fiProductClassDesc', 'state', 'ProductGroup', 'ProductGroupDesc',\n",
" 'Drive_System', 'Enclosure', 'Forks', 'Pad_Type', 'Ride_Control',\n",
" 'Stick', 'Transmission', 'Turbocharged', 'Blade_Extension',\n",
" 'Blade_Width', 'Enclosure_Type', 'Engine_Horsepower', 'Hydraulics',\n",
" 'Pushblock', 'Ripper', 'Scarifier', 'Tip_Control', 'Tire_Size',\n",
" 'Coupler', 'Coupler_System', 'Grouser_Tracks', 'Hydraulics_Flow',\n",
" 'Track_Type', 'Undercarriage_Pad_Width', 'Stick_Length', 'Thumb',\n",
" 'Pattern_Changer', 'Grouser_Type', 'Backhoe_Mounting', 'Blade_Type',\n",
" 'Travel_Controls', 'Differential_Type', 'Steering_Controls'],\n",
" dtype='object')"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.columns"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That's a lot of columns for us to look at! Try looking through the dataset to get a sense of what kind of information is in each one. We'll shortly see how to \"zero in\" on the most interesting bits.\n",
"\n",
"At the point, a good next step is to handle *ordinal columns*. This refers to columns containing strings or similar, but where those strings have a natural ordering. For instance, here are the levels of `ProductSize`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([nan, 'Medium', 'Small', 'Large / Medium', 'Mini', 'Large', 'Compact'], dtype=object)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df['ProductSize'].unique()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can tell Pandas about a suitable ordering of these levels like so:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sizes = 'Large','Large / Medium','Medium','Small','Mini','Compact'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df['ProductSize'] = df['ProductSize'].astype('category')\n",
"df['ProductSize'].cat.set_categories(sizes, ordered=True, inplace=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It's important to note what metric is being used for a project. Generally, selecting the metric is an important part of the project setup. In many cases, choosing a good metric will require more than just selecting a variable that already exists. It is more like a design process. You should think carefully about which metric, or set of metric, actually measures the notion of model quality which matters to you. If no variable represents that metric, you should say if you can build the metric from the variables which are available.\n",
"\n",
"However, in this case Kaggle tells us what metric to use: RMSLE (root mean squared log error) between the actual and predicted auction prices. Here we need do only a small amount of processing to use this: we take the log of the prices, so that m_rmse of that value will give us what we ultimately need."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"dep_var = 'SalePrice'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df[dep_var] = np.log(df[dep_var])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Decision trees"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Decision tree ensembles, as the name suggests, rely on decision trees. So let's start there! A decision tree asks a series of binary (that is, yes or no) questions about the data, and on that basis makes a prediction. For instance, the first question might be \"was the equipment manufactured before 1990?\" The second question will depend on the result of the first question (which is why this is a tree); for equipment manufactured for 1990 the second question might be \"was the auction after 2005?\" And so forth…\n",
"\n",
"This sequence of questions is now a procedure for taking any data item, whether an item from the training set or a new one, and assigning that item to a group. Namely, after asking and answering the questions, we can say the item belongs to the group of all the other training data items which yielded the same set of answers to the questions. But what good is this? the goal of our model is to predict values for items, not to assign them into groups from the training dataset. The value of this is that we can now assign a prediction value for each of these groups--for regression, we take the target mean of the items in the group.\n",
"\n",
"Let's consider how we find the right questions to ask. Of course, we wouldn't want to have to create all these questions ourselves — that's what computers are for! The basic steps to train a decision tree can be written down very easily:\n",
"\n",
"1. Loop through each column of the dataset in turn\n",
"1. For each column, loop through each possible level of that column in turn\n",
"1. Try splitting the data into two groups, based on whether they are greater than or less than that value (or if it is a categorical variable, based on whether they are equal to or not equal to that level of that categorical variable)\n",
"1. Find the average sale price for each of those two groups, and see how close that is to the actual sale price of each of the items of equipment in that group. That is, treat this as a very simple \"model\" where our predictions are simply the average sale price of the item's group\n",
"1. After looping through all of the columns and possible levels for each, pick the split point which gave the best predictions using our very simple model\n",
"1. We now have two different groups for our data, based on this selected split. Treat each of these as separate datasets, and find the best split for each, by going back to step one for each group\n",
"1. Continue this process recursively, and until you have reached some stopping criterion for each group — for instance, stop splitting a group further when it has only 20 items in it.\n",
"\n",
"Although this is an easy enough algorithm to implement yourself (and it is a good exercise to do so) we can save some time by using the implementation built into sklearn.\n",
"\n",
"But even before using sklearn, we have to prepare our data somewhat before we can use it."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> A: Here's a productive question to ponder. If you consider that the procedure for defining decision tree essentially chooses _sequence of splitting questions about variables_, you might ask yourself, how do we know this procedure chooses the _correct sequence_? The rule is to choose the splitting question which produces the best split, and then to apply the same rule to groups that split produces, and so on (this is known in computer science as a \"greedy\" approach). Can you imagine a scenario in which asking a “less powerful” splitting question would enable a better split down the road (or should I say down the trunk!) and lead to a better result overall?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Handling dates"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The first piece of data preparation we need to do is to enrich our representation of dates. The fundamental basis of the decision tree which we just described is bisection -- dividing up a group into two. We look at the ordinal variables and divide up the dataset based on whether the variable's value is greater (or lower) than a threshhold, and we look at the categorical variables and divided up the dataset based on whether the variable's level is a particular level. So this algorithm has a way of dividing up the dataset based on both ordinal and categorical data.\n",
"\n",
"How does this apply to a common data type, the date? You might want to treat a date as an ordinal value, because it is meaningful to say that one date is greater than another. However, dates are a bit different from most ordinal values in that some dates are qualitatively different from others in a way that that is often relevant to the systems we are modelling.\n",
"\n",
"So in order to help the above algorithm handle dates intelligently, we'd like our model to know more than whether a date is more recent or less recent. We might want our model to make decisions based on that date's day of week, on whether a day is a holiday, on what month it is in, and so forth. To do this, we replace every date column with a set of date metadata columns, such as holiday, day of week, and month. These columns provide categorical data that we suspect will be useful.\n",
"\n",
"Fastai comes with a function that will do this for us — we just have to pass a column name which contains dates:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = add_datepart(df, 'saledate')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's do the same for the test set while we're there:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df_test = pd.read_csv(path/'Test.csv', low_memory=False)\n",
"df_test = add_datepart(df_test, 'saledate')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that there are now lots of new columns in our DataFrame:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'saleYear saleMonth saleWeek saleDay saleDayofweek saleDayofyear saleIs_month_end saleIs_month_start saleIs_quarter_end saleIs_quarter_start saleIs_year_end saleIs_year_start saleElapsed'"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"' '.join(o for o in df.columns if o.startswith('sale'))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Using TabularPandas and TabularProc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A second piece of preparatory processing is to be sure we can handle strings and missing data. Out of the box, sklearn cannot do either. Instead we will use fastai's class `TabularPandas`, which wraps a Pandas data frame and provides a few conveniences. To populate a `TabularPandas`, we will use two `TabularProc`s, `Categorify` and `FillMissing`. A `TabularProc` is like a regular `Transform`, except that:\n",
"\n",
"- It returns the exact same object that's passed to it, after modifying the object *in-place*, and\n",
"- It runs the transform once, when data is first passed in, rather than lazily as the data is access.\n",
"\n",
"`Categorify` is a `TabularProc` which replaces a column with a numeric categorical column. `FillMissing` is an `TabularProc` which replaces missing values with the median of the column, and creates a new boolean column that is set to True for any row where the value was missing. These two transforms are needed for nearly every tabular dataset you will use, so it's a good starting point for your data processing."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"procs = [Categorify, FillMissing]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`TabularPandas` will also handle splitting into training vs validation datasets for us. \n",
"\n",
"We need to be very careful about our validation set here. In particular we want to design it so that it is like the *test set* which Kaggle will use to judge the contest.\n",
"\n",
"Recall the distinction between a validation set and a test set, as discussed in <| \n", " | UsageBand | \n", "fiModelDesc | \n", "fiBaseModel | \n", "fiSecondaryDesc | \n", "fiModelSeries | \n", "fiModelDescriptor | \n", "ProductSize | \n", "fiProductClassDesc | \n", "state | \n", "ProductGroup | \n", "ProductGroupDesc | \n", "Drive_System | \n", "Enclosure | \n", "Forks | \n", "Pad_Type | \n", "Ride_Control | \n", "Stick | \n", "Transmission | \n", "Turbocharged | \n", "Blade_Extension | \n", "Blade_Width | \n", "Enclosure_Type | \n", "Engine_Horsepower | \n", "Hydraulics | \n", "Pushblock | \n", "Ripper | \n", "Scarifier | \n", "Tip_Control | \n", "Tire_Size | \n", "Coupler | \n", "Coupler_System | \n", "Grouser_Tracks | \n", "Hydraulics_Flow | \n", "Track_Type | \n", "Undercarriage_Pad_Width | \n", "Stick_Length | \n", "Thumb | \n", "Pattern_Changer | \n", "Grouser_Type | \n", "Backhoe_Mounting | \n", "Blade_Type | \n", "Travel_Controls | \n", "Differential_Type | \n", "Steering_Controls | \n", "saleIs_month_end | \n", "saleIs_month_start | \n", "saleIs_quarter_end | \n", "saleIs_quarter_start | \n", "saleIs_year_end | \n", "saleIs_year_start | \n", "SalesID_na | \n", "MachineID_na | \n", "ModelID_na | \n", "datasource_na | \n", "auctioneerID_na | \n", "YearMade_na | \n", "MachineHoursCurrentMeter_na | \n", "saleYear_na | \n", "saleMonth_na | \n", "saleWeek_na | \n", "saleDay_na | \n", "saleDayofweek_na | \n", "saleDayofyear_na | \n", "saleElapsed_na | \n", "SalesID | \n", "MachineID | \n", "ModelID | \n", "datasource | \n", "auctioneerID | \n", "YearMade | \n", "MachineHoursCurrentMeter | \n", "saleYear | \n", "saleMonth | \n", "saleWeek | \n", "saleDay | \n", "saleDayofweek | \n", "saleDayofyear | \n", "saleElapsed | \n", "SalePrice | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "Low | \n", "521D | \n", "521 | \n", "D | \n", "#na# | \n", "#na# | \n", "#na# | \n", "Wheel Loader - 110.0 to 120.0 Horsepower | \n", "Alabama | \n", "WL | \n", "Wheel Loader | \n", "#na# | \n", "EROPS w AC | \n", "None or Unspecified | \n", "#na# | \n", "None or Unspecified | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "2 Valve | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "None or Unspecified | \n", "None or Unspecified | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "Standard | \n", "Conventional | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "1139246 | \n", "999089 | \n", "3157 | \n", "121 | \n", "3.0 | \n", "2004 | \n", "68.0 | \n", "2006 | \n", "11 | \n", "46 | \n", "16 | \n", "3 | \n", "320 | \n", "1163635200 | \n", "11.097410 | \n", "
| 1 | \n", "Low | \n", "950FII | \n", "950 | \n", "F | \n", "II | \n", "#na# | \n", "Medium | \n", "Wheel Loader - 150.0 to 175.0 Horsepower | \n", "North Carolina | \n", "WL | \n", "Wheel Loader | \n", "#na# | \n", "EROPS w AC | \n", "None or Unspecified | \n", "#na# | \n", "None or Unspecified | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "2 Valve | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "23.5 | \n", "None or Unspecified | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "Standard | \n", "Conventional | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "1139248 | \n", "117657 | \n", "77 | \n", "121 | \n", "3.0 | \n", "1996 | \n", "4640.0 | \n", "2004 | \n", "3 | \n", "13 | \n", "26 | \n", "4 | \n", "86 | \n", "1080259200 | \n", "10.950807 | \n", "
| 2 | \n", "High | \n", "226 | \n", "226 | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "Skid Steer Loader - 1351.0 to 1601.0 Lb Operating Capacity | \n", "New York | \n", "SSL | \n", "Skid Steer Loaders | \n", "#na# | \n", "OROPS | \n", "None or Unspecified | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "Auxiliary | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "None or Unspecified | \n", "None or Unspecified | \n", "None or Unspecified | \n", "Standard | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "#na# | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "False | \n", "1139249 | \n", "434808 | \n", "7009 | \n", "121 | \n", "3.0 | \n", "2001 | \n", "2838.0 | \n", "2004 | \n", "2 | \n", "9 | \n", "26 | \n", "3 | \n", "57 | \n", "1077753600 | \n", "9.210340 | \n", "
\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" SalesID SalePrice MachineID ModelID ... saleDay_na saleDayofweek_na \\\n",
"0 1139246 11.097410 999089 3157 ... 1 1 \n",
"1 1139248 10.950807 117657 77 ... 1 1 \n",
"2 1139249 9.210340 434808 7009 ... 1 1 \n",
"\n",
" saleDayofyear_na saleElapsed_na \n",
"0 1 1 \n",
"1 1 1 \n",
"2 1 1 \n",
"\n",
"[3 rows x 79 columns]"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"to.items.head(3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The conversion of categorical columns to numbers is done by simply replacing each unique level with a number. The numbers associated with the levels are chosen consecutively as they are seen in a column. So there's no particular meaning to the numbers in categorical columns after conversion. The exception is if you first convert a column to a pandas ordered category (as we did for `ProductSize` above), in which case the ordering you chose is used. We can see the mapping by looking at the `classes` attribute:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(#7) ['#na#','Large','Large / Medium','Medium','Small','Mini','Compact']"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"to.classes['ProductSize']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since it takes a minute or so to process the data to get to this point, we should save it - that way in the future we can continue our work from here without rerunning the previous steps. fastai provides a `save` method that uses Python's *pickle* system to save nearly any Python object."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"(path/'to.pkl').save(to)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To read this back later, you would type:\n",
"\n",
"```python\n",
"to = (path/'to.pkl').load()\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Creating the decision tree"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can read in our data (only needed if you're coming back to this notebook after a break, and don't want to recreate the `TabularPandas` object), and define our independent and dependent variables."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"to = (path/'to.pkl').load()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"xs,y = to.train.xs,to.train.y\n",
"valid_xs,valid_y = to.valid.xs,to.valid.y"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that our data is all numeric, and there are no missing values, we can create a decision tree:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"m = DecisionTreeRegressor(max_leaf_nodes=4)\n",
"m.fit(xs, y);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To see what it's learned, we can display the tree. To keep it simple, we've told sklearn to just create four *leaf nodes*."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n",
"\n",
"\n"
],
"text/plain": [
"| \n", " | SalesID | \n", "SalePrice | \n", "MachineID | \n", "ModelID | \n", "... | \n", "saleDay_na | \n", "saleDayofweek_na | \n", "saleDayofyear_na | \n", "saleElapsed_na | \n", "
|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "1139246 | \n", "11.097410 | \n", "999089 | \n", "3157 | \n", "... | \n", "1 | \n", "1 | \n", "1 | \n", "1 | \n", "
| 1 | \n", "1139248 | \n", "10.950807 | \n", "117657 | \n", "77 | \n", "... | \n", "1 | \n", "1 | \n", "1 | \n", "1 | \n", "
| 2 | \n", "1139249 | \n", "9.210340 | \n", "434808 | \n", "7009 | \n", "... | \n", "1 | \n", "1 | \n", "1 | \n", "1 | \n", "
3 rows × 79 columns
\n", "![\"sklearn](\"images/sklearn_features.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To see the impact of `n_estimators`, let's get the predictions from each individual tree in our forest (these are in the `estimators_` attribute):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"preds = np.stack([t.predict(valid_xs) for t in m.estimators_])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see, `preds.mean(0)` gives the same results as our random forest:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.233502"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"r_mse(preds.mean(0), valid_y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> question: Why does this give the same result as our random forest?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here's how RMSE improves as we add more and more trees. As you can see, the improvement levels off quite a bit after around 30 trees:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD7CAYAAABt0P8jAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAfiUlEQVR4nO3de3xdZZ3v8c9v35LsnXuatNDSBkovXBQGihwuHaRcVNTBI44Hy0s9Kl7gxRnH4+DgnOHIOBxnhnMcHWfwwvGGoOIgMAg6HkVRuQ8tUqDQFmib0nubJmnuyd77d/7YK2E3Jk2aplm7Wd/367VfyX7W2ju/PE2/+9nPWvtZ5u6IiEh0xMIuQEREppeCX0QkYhT8IiIRo+AXEYkYBb+ISMQkwi5gImbNmuXNzc1hlyEiclRZvXr1XndvHNl+VAR/c3Mzq1atCrsMEZGjipm1jNauqR4RkYhR8IuIRIyCX0QkYhT8IiIRo+AXEYkYBb+ISMQo+EVEImZGB/99v9/KnU+OehqriEhkzejg/+lzO/n+U1vCLkNEpKTM6OCvzyTZ190fdhkiIiVlRgd/XSZFW/cgusqYiMjrZnTwN2RSDOTydPVnwy5FRKRkzOjgr0unAGjrHgy5EhGR0jGjg7+hshD8+3oGQq5ERKR0zOjgHxrx6wCviMjrZnTw12eGgl9TPSIiQyIS/Brxi4gMmdHBX1mWIBk3jfhFRIrM6OA3M+ozKdq6dXBXRGTIjA5+KBzgbVXwi4gMm/HBX59J0abTOUVEhkUi+PdpxC8iMkzBLyISMZEI/o7eQbK5fNiliIiUhEgEP0Bbj07pFBGBCAT/8EJtOsArIgJEIPgbghF/a5eCX0QEIhD8dRmN+EVEis344G8YXq9HwS8iAhEI/tq0gl9EpNiMD/5UIkZVeULBLyISmPHBD/oQl4hIsUgEf11a6/WIiAyJRPA3aMQvIjIsEsFfp+AXERkWieAfGvG7e9iliIiELhLBX5dJ0Z/N0zOQC7sUEZHQRSL463Uuv4jIsGgEv5ZtEBEZFongH1qvR9feFRGJSPAPrdfTpuAXEYlG8NdpoTYRkWETCn4zqzez+8ys28xazGzlGPv9uZltNLP9ZrbdzL5kZomi7c1m9rCZ9ZjZOjO7eKp+kYOpLk+QiJmCX0SEiY/4bwUGgNnAVcDXzOyUUfZ7ADjD3auBU4HTgD8r2v5D4PdAA/A/gB+bWeMka58wM6Muo2UbRERgAsFvZhngCuBGd+9y90eBnwDvH7mvu7/q7u1DDwXywInB8ywGzgA+5+697n4P8Hzw3EdcfTqlq3CJiDCxEf9iIOfuG4ra1gCjjfgxs5Vmth/YS2HE/41g0ynARnfvnODzfMzMVpnZqj179kygzIOr14hfRASYWPBXAh0j2jqAqtF2dvcfBFM9i4GvA7sm+Ty3ufsyd1/W2Hj4s0H1mZRO5xQRYWLB3wVUj2irBjpH2XeYu78MrAW+ejjPM1XqMkmdzikiwsSCfwOQMLNFRW2nUQj18SSAhcH3a4ETzKx4hD/R5zls9Zky2nsHyeW1UJuIRNu4we/u3cC9wOfNLGNm5wGXA3eM3NfMrjazpuD7k4HPAr8KnmcD8CzwOTMrN7P/DLwRuGeqfpmDqU8ncYd2zfOLSMRN9HTOa4EKYDeFUzKvcfe1ZrbczLqK9jsPeN7MuoGfBbe/Ktp+JbAMaAP+HniPux/+kdsJqK8sA7Rej4hIYvxdwN33Ae8apf0RCgdth+5/aJzn2Qy8+ZAqnCJDK3S2dg1wYlMYFYiIlIZILNkAhYO7oBG/iEhkgr8hU5jq2dc9GHIlIiLhikzwD43493X3h1yJiEi4IhP8ZYk4lWUJjfhFJPIiE/xQGPVrxC8iURep4K9Pp9jXoxG/iERbtII/k9KyDSISeZEK/rpMShdjEZHIi1TwNyj4RUSiFfx1mRS9gzl6B3JhlyIiEppIBf/Qsg379OldEYmwaAV/phD8OsArIlEWyeDXlbhEJMoiGfwa8YtIlEUy+DXiF5Eoi1TwV5cnicdMI34RibRIBX8sZtSlkzqrR0QiLVLBD1CXTrGvS8EvItEVueCvz6Q04heRSItm8GuOX0QiLJLBr4O7IhJl0Qz+ngHyeQ+7FBGRUEQu+OvSKfIOHb26IIuIRFPkgr+hUgu1iUi0RS7464ZW6NQ8v4hEVOSCf2jZBgW/iERVZINfZ/aISFRFNvi1UJuIRFXkgr88GSedimvELyKRFbngh2C9HgW/iERUJIO/oVLr9YhIdEUy+OvSWrZBRKIrksFfn0np4K6IRFZkg18jfhGJqsgGf/dAjr7BXNiliIhMu8gGP0CbDvCKSARFMvi1Xo+IRFkkg1/r9YhIlE0o+M2s3szuM7NuM2sxs5Vj7He9mb1gZp1mtsnMrh+x/XQze8TMOsxsq5n9z6n4JQ6Vgl9EomyiI/5bgQFgNnAV8DUzO2WU/Qz4AFAHvBW4zsyuLNr+A+B3QD1wAXCNmf3JJGufNAW/iETZuMFvZhngCuBGd+9y90eBnwDvH7mvu9/i7s+4e9bd1wP3A+cV7dIMfN/dc+7+KvAoMNoLyBFVU5EkZlqhU0SiaSIj/sVAzt03FLWtYZzANjMDlgNri5q/DHzAzJJmtgQ4B3hojMd/zMxWmdmqPXv2TKDMiYvHjNq0lm0QkWiaSPBXAh0j2jqAqnEed1Pw/N8pansQeA/QC6wDvuXuT4/2YHe/zd2XufuyxsbGCZR5aOrSSU31iEgkTST4u4DqEW3VQOdYDzCz6yjM9b/d3fuDtnrg58DngXLgOOAtZnbtJOo+bA2ZMgW/iETSRIJ/A5Aws0VFbadx4BTOMDP7MHADcJG7by3adAKFKaPvBccAtgJ3AZdNrvTDU5fRiF9Eomnc4Hf3buBe4PNmljGz84DLgTtG7mtmVwFfAC5x940jNm8o7GIrzSxmZnOA/0LheMG0q8+Usa97MIwfLSISqomeznktUAHsBn4IXOPua81suZl1Fe13M9AAPG1mXcHt6wDuvh94N/ApoA14FngB+F9T86scmvpMkraeAdw9jB8vIhKaxER2cvd9wLtGaX+EwsHfofvHj/M8vwbOOsQaj4i6dIpc3tnfm6UmnQy7HBGRaRPJJRugcBUuQKd0ikjkRDb4X1+orT/kSkREpldkg78hUwagA7wiEjmRDf66TGFeXyN+EYmayAa/RvwiElWRDf6KVJzyZExX4RKRyIls8APMq0uzbueYK0+IiMxIkQ7+CxY38uTGVnoGsmGXIiIybSId/CuWNjGQzfPYK61hlyIiMm0iHfxnNddTWZbg4fW7wy5FRGTaRDr4U4kY5584i4fX7daaPSISGZEOfoALlzayo6NPB3lFJDIU/EuaAPj1Ok33iEg0RD74m6rLOXVuNQ8r+EUkIiIf/AArljTxzJY22nRFLhGJAAU/cOHSJvIOv3t5T9iliIgccQp+4LR5tTRkUpruEZFIUPADsZhxwZJGfrthD7m8TusUkZlNwR9YsbSJtp5Bnn2tLexSRESOKAV/YPmiRuIx02mdIjLjKfgDNRVJzlxQx6/X6QCviMxsCv4iK5Y28dKO/ezo6A27FBGRI0bBX2TF0sKneH+zXqN+EZm5FPxFFjVVMre2QvP8IjKjKfiLmBkrljbx2Ct76c/mwi5HROSIUPCPsGJpEz0DOZ7auC/sUkREjggF/wjnLGygPBnTdI+IzFgK/hHKk3HOXTiLh9fr4iwiMjMp+Edx4ZJGWlp72LS3O+xSRESmnIJ/FBcu1cVZRGTmUvCPYl5dmsWzK3URdhGZkRT8Y7hwaRP/sWkfXf3ZsEsREZlSCv4xrFjSxGDO+a0+xSsiM4yCfwxnLqhjbm0F33tic9iliIhMKQX/GBLxGB86r5mnNu3j+a0dYZcjIjJlFPwH8d6zjqOyLME3H90YdikiIlNGwX8Q1eVJrjzrOB58bgfb27VUs4jMDAr+cXzo/OMB+O7jm8MtRERkikwo+M2s3szuM7NuM2sxs5Vj7He9mb1gZp1mtsnMrh9ln08G27rN7CUzW3y4v8SRNLe2gsvecAw/fGoLnX2DYZcjInLYJjrivxUYAGYDVwFfM7NTRtnPgA8AdcBbgevM7MrhjWZXAx8B3g5UAu8A9k66+mly9fnH09mf5V9XbQ27FBGRwzZu8JtZBrgCuNHdu9z9UeAnwPtH7uvut7j7M+6edff1wP3AecHzxIDPAZ9y9xe94FV3L/n1j087rpY3Ndfz7Uc3kc3lwy5HROSwTGTEvxjIufuGorY1wGgj/mFmZsByYG3QNC+4nWpmrwXTPX8TvCCUvKuXH8+29l5+vnZn2KWIiByWiYRuJTDyRPYOoGqcx90UPP93gvvzgq+XAm8ALgTeR2Hq5w+Y2cfMbJWZrdqzJ/xPz1500myaG9L830c2ablmETmqTST4u4DqEW3VQOdYDzCz6yjM9b/d3fuD5qHzIW9x93Z33wx8A7hstOdw99vcfZm7L2tsbJxAmUdWPGZ85PzjWfNaO6ta2sIuR0Rk0iYS/BuAhJktKmo7jdencA5gZh8GbgAucvfio6HrKRwgPmqHy1ecOY/adJJvPqIPdInI0Wvc4Hf3buBe4PNmljGz84DLgTtG7mtmVwFfAC5x940jnqcH+BHwGTOrMrN5wEeBBw//15ge6VSCq86ezy9e3MVmXaRFRI5SEz2wei1QAewGfghc4+5rzWy5mXUV7Xcz0AA8bWZdwe3rRduvozB1tB14AvgB8O3D/SWm0wfPaSYZi/HtxzaFXYqIyKQkJrJTcMrlu0Zpf4TCwd+h+8eP8zz7gSsPtk+pa6ou509OP5a7V23lv1+ymNp0KuySREQOyVFxKmWpuXr58fQO5vj+U1vCLkVE5JAp+Cdh6Zxqli+axe2Pb6Y/mwu7HBGRQ6Lgn6RPXLCQ3Z39/P2/rwu7FBGRQ6Lgn6TzTpzFh887nu88tpkH1mwPuxwRkQlT8B+Gz162lDMX1HHDPc/xyu4xP88mIlJSFPyHIRmPcevKMyhPxvnEnc/Q3Z8NuyQRkXEp+A/TnJpy/vl9f8TGPV3ccO/zWsdHREqegn8KnHviLD596RIeWLOd7z3REnY5IiIHpeCfItdcsJCLT2ri5p++yGot4iYiJUzBP0ViMeOLf3o6c2rKue4Hz9Da1T/+g0REQqDgn0I16SRfu+pMWrsH+ORdz5LLa75fREqPgn+KnTq3hpsvP5VHX9nLlx/aMP4DRESmmYL/CHjvWcfx3mXz+JeHX+E/NpX8JYVFJGIU/EfI5955CvPqKrj+x2voGdD5/SJSOhT8R0imLMEtV5xGS2sPt/x8fdjliIgMU/AfQecsbOCD5yzgu49v5qmNrWGXIyICKPiPuL9821Lm16e5/sfPacpHREqCgv8IS6cS3PKeN7Jln6Z8RKQ0KPinwX86oYH/em4z3318M09qykdEQqbgnyafeesSFjSk+YymfEQkZAr+aZJOJbjlisKUzz/oql0iEiIF/zQ6O5jyuf2JFp54VVM+IhIOBf80G57yuWeNLtwiIqFQ8E+zoSmf1/b18lf3Pa/5fhGZdgr+EJx9QgOfvGgR9z+7nUv+8Xf8v7U7deUuEZk2Cv6QfOqSxdz9iXOoLEvw8TtW85HbV7GltSfsskQkAhT8ITqruZ4H/+x8/vrtJ/HUxlYu+dJv+edfvUx/Nhd2aSIygyn4Q5aMx7h6+Qk89OkLuPik2Xzxlxt425cf4dGX9475GHcnr4u8iMgk2dEwt7xs2TJftWpV2GVMi99u2MPn7n+Bza091KaT5PKFkM+5k89DNp8n72AGl548m7+4dAmLZleFXbaIlCAzW+3uy/6gXcFfevoGc9z5ZAtb9vUQMyMeK7qZEYsZnX2D3L1qKz0DWd59xjz+/OJFzKtLh126iJQQBf8MtK97gK8+/Arfe7IFHFaePZ/rVpzIrMqysEsTkRKg4J/Btrf38pVfvczdq7dSlojxkfOP56N/fALV5cmwSxORECn4I2Djni7+8ZcbePC5HVSXJ7j0lDlcfNJsli+aRaYsEXZ5IjLNFPwR8sK2Dr75yEZ+vW43+/uypBIxzl3YwMUnzeaik5o4pqYi7BJFZBoo+CNoMJdn1eY2HnppFw+9tIuW4ANip86t5i0nz+G9Zx3H7OrykKsUkSNFwR9x7s6re7r45Yu7eeilXTyzpY24GW89dQ4fPLeZZQvqMLOwyxSRKaTglwNs3tvNnU+28K+rXmN/X5aTjqnmg+cs4PLT51KRioddnohMAQW/jKpnIMv9z27n9sc3s25nJzUVSd67bB5Xnb2A5lmZsMsTkcOg4JeDcnee3tzG7U9s5ucv7CSXd04/rpbLTz+Wd7zxWBqr9NkAkaPNWME/obV6zKzezO4zs24zazGzlWPsd72ZvWBmnWa2ycyuH2O/C8zMzezmQ/s15EgxM950fD23rjyDx29YwWfftpSBbJ6/eeBFzv7CQ7z/W09xz+qtdPYNhl2qiBymiZ7cfSswAMwGTgd+amZr3H3tiP0M+ADwHLAQ+IWZvebudw3vYJYE/gl46nCLlyNjdnU5H79gIR+/YCEv7+rk/me3c/+abXz67jWU3Rfj4pNnc9mpx3D+olnUVOhDYiJHm3GneswsA7QBp7r7hqDtDmCbu98wzmO/EvyM/1bUdgNQDzQBW939r8crUlM94XN3ntnSzk+e3caDz+2gtXuAeMxYtqCOC5c2sWJpE4uaKnVmkEgJmfQcv5n9EfC4u1cUtf0FcIG7v/MgjzPgGeAb7v71oG0B8EvgDOBfOEjwm9nHgI8BzJ8//8yWlpaD/4YybbK5PM++1s7D63fz8Lo9vLhjPwBzayt485JGVixtYtmCemrSejcgEqaxgn8iUz2VQMeItg5gvLWAb6JwDOE7RW1fAW50967xRobufhtwGxRG/BOoU6ZJIh5jWXM9y5rruf4tS9nZ0cdv1u/m1+t2c9/vt/H9p7YAML8+zRvm1nDq3BreENz0YiASvokEfxdQPaKtGugc6wFmdh2Fuf7l7t4ftL0TqHL3H02yVilRc2rKufJN87nyTfPpz+ZY3dLGmtc6eH5bO89ta+enz+8Y3nfoxWDpnCqWzKnipGOqmVtbQSymKSKR6TKR4N8AJMxskbu/HLSdBow8sAuAmX0YuAH4Y3ffWrTpImCZme0M7tcAOTN7g7tfPrnypdSUJeKcu3AW5y6cNdzW3jPAC9v289y2dl7Y1sHz2zoOeDHIpOIsnlPF0jlVLJ1TTfOsDMmYYWbEDGKxwtfCfaOyLM68ujTlSX3QTGQyJnQev5ndBThwNYWzen4GnDvyrB4zuwr4InChu780YlsVUPyJoH8CtgN/6+77DvbzdXB35unqz7JhVyfrd3aybsd+1u3sZP2uTtp7Jna6qBkcU13O/IY0zQ0Z5jekWVCfYUFDmhObKvWiIMLhzfEDXAt8G9gNtALXuPtaM1sO/Lu7Vwb73Qw0AE8XzeHf6e6fcPdOiqaHzKwX6B4v9GVmqixLcMb8Os6YXzfc5u7s2t/P1raewiUnPbi+sEPenbw7DnT0DNLS2kNLazct+3p46KVd7O0aGH6eRMw4ZW4NZ86v48wFdSxrrtNidCJF9MldmRG6+rOFF4LWHp7f1hEcZ2inP5sHCmccnbmgjjPm17KwqZJjays4tqZC6xLJjKYlGyRyBrJ5Xtqxn9Utbaze0sbqzW3s3N93wD516STH1lZwTE0Fc2vLmVtXwcLGShbPrtJBZznqHe5Uj8hRJ5WIcdpxtZx2XC0f5ngAdnb00dLazY6OPra197K9vZcdHX1sbevhqU2tdPZlhx+fTsU5samSRU1VLJ5deDE4oTFDbTpFVVlCLwpy1FLwS6TMqSlnTs3Y8/0dvYO8sruTDbu62LCrk5d3dfHIy3u455mtB+wXM6gqT1JTceAtlYjR3Z+ldzBHd3+WnoFc0S1LWSJGU1U5TdVlNFaW0Rh8baoup7GyjIpUnETMiMds+OvQLRmPUZ6Iky6Lk4xPaJktkVEp+EWK1FQkOXNBPWcuqD+gvaNnkA27O9m8t5uO3sFRb9s7ehnM5cmkElSk4mRSCWZVlpFOxUmXJUgn4/Rlc+ze38+ern427ulmT1c/A8FxiEORjBsVyTjpVIJ0Kk5FKk5VeYJ5dWnm1xduxwVfZ1WmtJSGHEDBLzIBNekkZzXXc1Zz/fg7HwJ3Z39vlj1dfezu7Kd/ME827+TyQ1+dbM7JuTOYy9M7kKN3IEfPYPB1oPCuoncgR3vvII+8vIdd+/sP+BkVyTjz69McW1tOfaaMhsoUdekUDZkU9ZkU9ZWF7zNliaJ3G7HgXYbpRWMGUvCLhMjMqEknqUknObFpvFVQJqZvMMfWth627OthS2sPr7X1smVfDzs6elm/s5PW7oHhs50mImaQiMVIl8WpLEtQVZ6kqjxBVVmi8DW4P6emnGNqKji2tpxjayqoTSf1olGiFPwiM0x5Ms6JTVUHfSHpGcjS2jXAvu7CrbV7gN7BHNlcvvAuI+9kc6+/6xjMOT0DWTr7hm6D7OjoY8PuQbr6suzvy5LLH3iGYEUyzjHBi0BTdWHKqywRpzxZOFZRloxRnoxTnohTWZ5gdnU5x9SU01RVRkLHMI4oBb9IBKVTCdL1CY6rT0/J8+Xzzt7ufra397GjvZftHX3BGVO9bG/vY9PGbvoGc4VbNv8HLxLFzKCxsqxwIL66cDA+nUowmMsP3wayfsD9mBUOfifjwddEjGTs9e/jZsSKDpjHrPB9LGak4kZ1RZK6dGEKrDadpC6TIpOKH/F3LO7++pRe8LXwfZ58HrL5PLOry6f8YL6CX0QOWyxmhbOVqso5/bjacffP5vL0ZfP0Deboz+bp6Blk1/4+dnT0sXN/Hzs7etm5v5/Nrd08sbGV/myeVHGwx2OkEoX7iViMfHAMJJt3BrN5BnLB/VyegeBdzEFea0aVjBu16RSVZYWYLP7M02hPZRSm7iy4M3S/8I5p6EXqwO8P9gI45FefvoCFjZXj7ncoFPwiMu0S8RiV8dhwqM6treDkY0cuAjy13IMRtfvwaDqfh/5cjo6eQdp7B2nrHqC9Z5C2ngHaegbp6B2gqz/H0Li/+A1A8XsBB9yHvhaWFsHBceKx4AUrFiOZeP2Fa+hFKxkvfjcSG34nMvTuZFbl1F/vWsEvIpFgZiTiVhR6Q8t1JGmqitZaTjqCIiISMQp+EZGIUfCLiESMgl9EJGIU/CIiEaPgFxGJGAW/iEjEKPhFRCLmqLj0opntAVom+fBZwN4pLGcqqbbJUW2To9om52iubYG7N45sPCqC/3CY2arRrjlZClTb5Ki2yVFtkzMTa9NUj4hIxCj4RUQiJgrBf1vYBRyEapsc1TY5qm1yZlxtM36OX0REDhSFEb+IiBRR8IuIRIyCX0QkYmZs8JtZvZndZ2bdZtZiZivDrmmImf3GzPrMrCu4rQ+xluvMbJWZ9ZvZd0dsu8jM1plZj5k9bGYLSqE2M2s2My/qvy4zu3Ea6yozs28Ff1edZvZ7M3tb0fbQ+u1gtYXdb0ENd5rZDjPbb2YbzOzqom1h/72NWlsp9FtRjYuC7LizqG1l8O/dbWb/Zmb14z6Ru8/IG/BD4EdAJXA+0AGcEnZdQW2/Aa4Ou46glncD7wK+Bny3qH1W0Gd/CpQD/xt4skRqa6ZwedNESH2WAW4K6ogB7wA6g/uh9ts4tYXab0F9pwBlwfdLgZ3AmWH32zi1hd5vRTX+AngEuLOo5k7gj4Os+wFw13jPMyOvuWtmGeAK4FR37wIeNbOfAO8Hbgi1uBLj7vcCmNkyYF7RpncDa9397mD7TcBeM1vq7utCri1U7t5NIVyHPGhmmyiERAMh9ts4ta0+0j9/PO6+tvhucFtIob6w/97Gqq11On7+eMzsSqAdeBw4MWi+CnjA3X8X7HMj8JKZVbl751jPNVOnehYDOXffUNS2hsKrY6n4OzPba2aPmdmbwy5mFKdQ6DNgOFBepbT6sMXMtprZd8xsVlhFmNlsCn9zaymxfhtR25BQ+83MvmpmPcA6YAfwM0qk38aobUho/WZm1cDngU+P2DSy314FBij8m49ppgZ/JYW3jcU6gKoQahnNXwInAHMpfADjATNbGG5Jf6CU+3AvcBawgMJIsQr4fhiFmFky+Nm3ByPTkum3UWoriX5z92uDn70cuBfop0T6bYzaSqHf/hb4lru/NqJ9Uv02U4O/C6ge0VZNYS4sdO7+lLt3unu/u98OPAZcFnZdI5RsH7p7l7uvcvesu+8CrgMuDUZF08bMYsAdFEZY1wXNJdFvo9VWKv0W1JJz90cpTOFdQ4n022i1hd1vZnY6cDHwpVE2T6rfZuQcP7ABSJjZInd/OWg7jQPf7pYSByzsIkZYC3xw6E5w3GQhpdmHQx8/n7Y+NDMDvgXMBi5z98FgU+j9dpDaRpr2fhtFgtf7p9T+3oZqG2m6++3NFA4wbyn801IJxM3sZODnFLKtUJDZCUAZhQwcW9hHqY/g0e+7KJzZkwHOo0TO6gFqgbdQOHMhQeHgTDewJKR6EkEtf0dhhDhUV2PQZ1cEbf/A9J9lMVZtZwNLKLxjbaBw9tbD01zb14EngcoR7aXQb2PVFmq/AU3AlUPBFfw/6AYuD7vfxqkt7H5LA3OKbv8H+HHQZ6cA+ylMTWWAO5nAWT3T9sc43TegHvi34B9vC7Ay7JqCuhqBpym8FWsP/oNeEmI9N/H6GQxDt5uCbRdTOMjVS+EU1OZSqA14H7Ap+LfdAXwPmDONdS0Iaumj8FZ76HZV2P12sNpKoN8agd8Gf/f7geeBjxZtD7Pfxqwt7H4bpdabCE7nDO6vDDKuG7gfqB/vObRIm4hIxMzUg7siIjIGBb+ISMQo+EVEIkbBLyISMQp+EZGIUfCLiESMgl9EJGIU/CIiEfP/AfUVFNIxYB8qAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot([r_mse(preds[:i+1].mean(0), valid_y) for i in range(40)]);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Out-of-bag error"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Is our validation set worse than our training set because we're over-fitting, or because the validation set is for a different time period, or a bit of both? With the existing information we've shown, we can't tell. However, random forests have a very clever trick called *out-of-bag (OOB) error* which can handle this (and more!)\n",
"\n",
"The idea is to calculate error on the training set, but only include the trees in the calculation of a row's error where that row was *not* included in training that tree. This allows us to see whether the model is over-fitting, without needing a separate validation set.\n",
"\n",
"This also has the benefit of allowing us to see whether our model generalizes, even if we have such a small amount of data that we want to avoid removing items to create a validation set. The OOB predictions are available in the `oob_prediction_` attribute. Note that we compare to *training* labels, since this is being calculated on the OOB trees on the training set.\n",
"\n",
"> A: My intuition for this is that, since every tree was trained with a different randomly selected subset of rows, out-of-bag error is a little like imagining that every tree therefore also has its own validation set. That validation set is simply the rows that were notselected for that tree's training."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.210686"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"r_mse(m.oob_prediction_, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that our OOB error is much lower than our validation set error. This means that something else is causing our error, in *addition* to normal generalization error. We'll discuss the reasons for this later in this chapter."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> question: Make a list of reasons why this model's validation set error on this dataset might be worse than the OOB error. How could you test your hypotheses?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model interpretation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For tabular data, model interpretation is particularly important. For a given model, the things we are most likely to be interested in are:\n",
"\n",
"- How confident are we in our projections using a particular row of data?\n",
"- For predicting with a particular row of data, what were the most important factors, and how did they influence that prediction?\n",
"- Which columns are the strongest predictors, which can we ignore?\n",
"- Which columns are effectively redundant with each other, for purposes of prediction?\n",
"- How do predictions vary, as we vary these columns?\n",
"\n",
"As we will see, random forests are particularly well suited to answering these questions. Let's start with the first one!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Tree variance for prediction confidence"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We saw how the model averages predictions across the trees to get a prediction, that is, an estimate of the value. But how can we know the confidence of the estimate? One simple way is to use the standard deviation of predictions across the trees, instead of just the mean. This tells us the *relative* confidence of predictions. That is, for rows where trees give very different results, you would want to be more cautious of using those results, compared to cases where they are more consistent.\n",
"\n",
"In the last lesson we saw how to get predictions over the validation set, using a Python list comprehension to do this for each tree in the forest:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"preds = np.stack([t.predict(valid_xs) for t in m.estimators_])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(40, 7988)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"preds.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"No we have a prediction for every tree and every auction, for 160 trees and 7988 auctions in the validation set.\n",
"\n",
"Using this we can get the standard deviation of the predictions over all the trees, for each auction:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"preds_std = preds.std(0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here are the prediction standard deviations for the first 5 auctions, that is, the first 5 rows of the validation set:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0.21529149, 0.10351274, 0.08901878, 0.28374773, 0.11977206])"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"preds_std[:5]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see, the confidence of the predictions varies widely. For some auctions, there is a low standard deviation because the trees agree. For others, it's higher, as the trees don't agree. This is information that would be useful to use in a production setting; for instance, if you were using this model to decide what items to bid on at auction, a low-confidence prediction may cause you to look more carefully into an item before you made a bid"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Feature importance"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It's not normally enough to just to know that a model can make accurate predictions -- we also want to know *how* it's making predictions. The most important way to see this is with *feature importance*. We can get these directly from sklearn's random forest, by looking in the `feature_importances_` attribute. Here's a simple function we can use to pop them into a DataFrame and sort them:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def rf_feat_importance(m, df):\n",
" return pd.DataFrame({'cols':df.columns, 'imp':m.feature_importances_}\n",
" ).sort_values('imp', ascending=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The feature importances for our model show that the first few most important columns have a much higher importance score than the rest, with (not surprisingly) `YearMade` and `ProductSize` being at the top of the list:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" cols imp\n",
"69 YearMade 0.182890\n",
"6 ProductSize 0.127268\n",
"30 Coupler_System 0.117698\n",
"7 fiProductClassDesc 0.069939\n",
"66 ModelID 0.057263\n",
"77 saleElapsed 0.050113\n",
"32 Hydraulics_Flow 0.047091\n",
"3 fiSecondaryDesc 0.041225\n",
"31 Grouser_Tracks 0.031988\n",
"1 fiModelDesc 0.031838"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fi = rf_feat_importance(m, xs)\n",
"fi[:10]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A plot of the feature importances shows the relative importances more clearly:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAzcAAAGeCAYAAACpXf/VAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdaZSdVZn28f9FgDCHeUgYgjIKKGIBSkODoo1MgrYyI9gCDk3bONAgrRhsQLEVFdFGoJUhBnECxcCL0MgMygmiiIIyFIQkICFkDlNyvR/2LjkUpypVSQ1J5fqtVYtznj0+Rz/kXnvve8s2ERERERERS7vlBnsCERERERERfSHBTUREREREDAkJbiIiIiIiYkhIcBMREREREUNCgpuIiIiIiBgSlh/sCcTQsO6663r06NGDPY2IiIiIGOImTJgw1fZ6rcoS3ESfGD16NI1GY7CnERERERFDnKTHuyrLtrSIiIiIiBgSsnKzjJFkYEvbD/dlv/dPmsHoU8d3Wd7+5f37criIiIiIiNfIys0gknSEpIak2ZKmSLpO0u6DPa+IiIiIiKVRgptBIulTwDeAs4ENgE2B7wAHDea8uiIpq3wRERERsURLcDMIJI0Avgj8q+2f2Z5j+yXb19g+WdJwSd+QNLn+fUPS8Nr2WEm3d+rPkraony+RdIGkGyTNknSLpM26mMdwSV+V9ISkp2u7lWvZXpKelHSKpKeA7/frjxIRERERsZgS3AyOtwErAVd1Uf6fwFuBHYE3AbsAn+tF/0cC/wWsC9wH/KCLeucAW9VxtgBGAac3lW8IrA1sBpzQubGkE+q2usb8uTN6Mb2IiIiIiL6X4GZwrANMtf1yF+VHAl+0/TfbzwBnAEf3ov/xtm+1/QIlUHqbpE2aK0gScDzwSdvTbM+ibJE7rKnaAuALtl+wPa/zILYvtN1mu23YKiN6Mb2IiIiIiL6XcxSD41lgXUnLdxHgjASa83c/Xp/11MSOD7ZnS5pW209sqrMesAowocQ5AAgY1lTnGdvP92LciIiIiIhBk+BmcNwFPA8cDPykRflkylawB+r3TeszgDmUoAQASRu2aL9JU/lqlK1lkzvVmQrMA7azPamLebrbt2iyw6gRNJLuOSIiIiIGUbalDQLbMyhnW74t6WBJq0haQdK+kr4CXAF8TtJ6ktatdcfW5r8HtpO0o6SVgDEththP0u6SVqScvfmN7eZVG2wvAC4Cvi5pfQBJoyTt0w+vHBERERHR7xLcDBLb5wKfoiQKeIayZexE4GrgTKAB/AG4H7i3PsP2XyiZ1m4E/grc3rlvYBzwBWAa8BbKGZ5WTgEeBu6WNLP2ufXiv11ERERExMCT3eOdR7EUkHQJ8KTt3mRXW2xtbW1uNBoDOWRERERELIMkTbDd1qosKzcRERERETEkJLgZAJJOk3TxYM8jIiIiImIoy7a0PiZpL2Cs7Y0Hey4LI2kMsIXto+p3A3MpWdJeoFwAeqHtKxfW1/CNtvRGx3xjoWO2J6NaRERERCyGbEuL15DUVRrwN9lejZJY4BLgfElfGLCJRUREREQsomU+uJF0qqRHJM2S9CdJ763Px0ga21RvtCR3BAWS1pb0fUmTJT0n6WpJqwLXASMlza5/I1v09R5JD0iaLulmSds2lbVL+oykP0iaIenKmvK5o/wASffVtndKemNT2UhJP5X0jKTHJH2iqWyMpJ9IGlszox3b3e9ie6rty4GPAZ+VtM6i/8oREREREf1vmQ9ugEeAPYARwBnAWEkb9aDd5ZTLNLcD1ge+bnsOsC8w2fZq9e9Vl2dK2opyj81JwHrAtcA19U6aDocA7wY2B95IDUQk7QR8D/gIsA7wXeAXkoZLWg64hnIPzihgb+CkTvfWHES5NHRN4Ac9eEeAn1Mue92lc4GkEyQ1JDXmz53Rw+4iIiIiIvrHMh/c2P6x7cm2F9SzJX+lxT/km9XgZ1/go7afs/2S7Vt6OOShwHjbN9h+CfgqsDKwW1Od8+qcplEClh3r8+OB79r+je35ti+lnI15K7AzsJ7tL9p+0fajlEs6D2vq9y7bV9d3ndeTydY5TgXWblF2oe02223DVhnRw9ePiIiIiOgfXZ27WGZI+iDlMs3R9dFqwLoLabYJMM32c4sw5Ejg8Y4vthdImkhZbenwVNPnubUNwGbAMZL+ral8xVo+n7IdbnpT2TDgtqbvE3s7WUkrUFaYpvW2bURERETEQFqmgxtJm1FWN/amrGrMl3QfIGAOZdtZhw2bPk8E1pa0pu3mYAJKprHuTAZ2aJqDKMHSpB5MeSJwlu2zWrzL24DHbG/ZTftFSY13EPAy8NvuKu0wagSNZEKLiIiIiEG0rG9LW5XyD/5nACR9CNi+lt0H/KOkTSWNAD7b0cj2FErigO9IWkvSCpL+sRY/DaxT27TyI2B/SXvXVZFPU7aW3dmD+V4EfFTSripWlbS/pNUpwcdMSadIWlnSMEnbS9q5F7/H39WECUcC3wbOsf3sovQTERERETFQlungxvafgK8Bd1GCkh2AO2rZDcCVwB+ACcAvOzU/GngJeBD4GyVBALYfpCQMeLRmNBvZ3Mj2Q8BRwLcoZ1kOBA60/WIP5tugnLs5H3gOeJiabMD2/NrXjsBjte+LKYkSeuP3kmbXvo8DPmn79F72EREREREx4HKJZ/SJtrY2NxqNwZ5GRERERAxxucQzIiIiIiKGvAQ3ERERERExJCS4GWIkHSnpV03f/0HSXyXNlnSwpOskHTOYc4yIiIiI6A85czOAJO0OfAXYjnIvzZ+Bk2zf049j/h/wC9vf7K8xAIZvtKU3OuYbC63XnnTREREREbEYujtzs0zfczOQJK1Bybj2MUo66BWBPShpoPtqjOVtv9zp8WbAA/3Ud0RERETEEiPb0gbOVgC2r7A93/Y827+y/QcAScdL+rOkWZL+JGmn+vxUSY80PX9vR4eSjpV0h6SvS5oGjKnPbq/ljwCvA66p29KGS7pZ0nFNffxLHfc5SdfXi007yizpXyX9FfjrQPxIERERERGLKsHNwPkLMF/SpZL2lbRWR4GkDwBjgA8CawDvATouzXyEssIzAjgDGCtpo6Z+dwUeBdYHzmoe0PbrgSco9+isZvtVq0SSDgZOA94HrAfcRrmjp9nBdYw3dH4hSSdIakhqzJ87o6e/Q0REREREv0hwM0BszwR2BwxcBDwj6ReSNqBclvkV2/e4eNj247Xdj21Ptr3A9pWUFZRdmrqebPtbtl+2Pa+X0/oI8CXbf65bzs4Gdmxevanl01r1bftC222224at0tu7QiMiIiIi+laCmwFUg4hjbW8MbA+MBL4BbEJZoXkNSR+UdJ+k6ZKm13brNlWZuBhT2gz4ZlPf0wABo/qo/4iIiIiIAZOEAoPE9oOSLqGsnkwEXt+5Tl1BuQjYG7jL9nxJ91ECkL93tRjTmAicZfsH3U21Jx3tMGoEjWRCi4iIiIhBlJWbASJpG0mflrRx/b4JcDhwN3Ax8BlJb1GxRQ1sVqUEF8/UNh+irNz0lQuAz0rarvY/op7/iYiIiIhY6mTlZuDMohzM/5SkNYHplNTQJ9ueKWkdYBxlS1g7cLTt30n6GnAXsAC4DLijryZk+ypJqwE/rMHUDOAG4Md9NUZERERExEDJJZ7RJ9ra2txoNAZ7GhERERExxHV3iWe2pUVERERExJCQ4CYiIiIiIoaEBDeDTJIlbTHY82gm6RJJZw72PCIiIiIieiMJBZYSkh4CPm/7R/X7PwC3A4d2enY9sGa9lHPA3D9pBqNPHd/j+u1JGx0RERERfSwrN0uPW4E9m77/I/Bgi2d3DnRgExERERGxJEhw04cknSJpkqRZkh6StLekXSTdJWm6pCmSzpe0Yhfth0v6qqQnJD0t6QJJK9fiWynBS4c9gHNaPLu1qb9/kfRnSc9Jur6me+4o20bSDZKm1bke0sWcVpf0a0nnSVKrOhERERERS4IEN31E0tbAicDOtlcH9qHcVzMf+CSwLvA2YG/g4110cw6wFbAjsAXlzpvTa9ktwHaS1pa0HNAGXAms2fRsN2pwI+lg4DTgfcB6wG3AFbVsVcp9NuOA9SmXiX6n4zLPpndaB/g/4A7bn3CnvOGSTpDUkNSYP3dG736wiIiIiIg+luCm78wHhgNvkLSC7Xbbj9ieYPtu2y/bbge+y6u3kgFQV0WOBz5pe5rtWcDZwGEAtp8AnqCszrwJ+KvteZRLPTuerQT8pnb5EeBLtv9ct6mdDexYV28OANptf7/O617gp8D7m6Y0khJQ/dj251q9sO0LbbfZbhu2yohF/NkiIiIiIvpGEgr0EdsPSzoJGENZYbke+BSwGnAuZaVlFcpvPqFFF+vV8glNu78EDGuq07E17QnKSgyUpAIdz35j+4X6fDPgm5K+1tRelNWgzYBdJU1vKlseuLzp+/7AbOCCHrx+RERERMSgS3DTh2yPA8ZJWoOyQnMOZQXkd8DhtmfVAOj9LZpPBeYB29me1MUQt1JWZB4Hvl+f3QYcU5/d2lR3InCW7R907qSu3txi+13dvM5FwFrAtZLebXtON3XZYdQIGsmAFhERERGDKNvS+oikrSW9Q9Jw4HlKoDIfWB2YCcyWtA3wsVbtbS+gBBRfl7R+7XOUpH2aqt0KvJmyre2O+ux+YHPg7bw6uLkA+GzHORpJIyR9oJb9EthK0tGSVqh/O0vattO0TgQeAn7ZlNggIiIiImKJlOCm7wwHvkxZgXmKclD/NOAzwBHALErwcmU3fZwCPAzcLWkmcCOwdUeh7b8AfwOm2J5eny0AfgusAdzZVPcqysrRD2tffwT2rWWzgH+inOeZXOd7Tn0HmvowcAJlFejnklbq5W8SERERETFg1CkBVsQiaWtrc6PRGOxpRERERMQQJ2mC7bZWZVm5iYiIiIiIISHBTUREREREDAnJlraYJJ0JbGz72MGey+KQ9CRwlO2bF6X9/ZNmMPrU8b1u154MaxERERHRR5bKlRtJs5v+Fkia1/T9yMGeXyuShkv6oqSHJc2R1C7pYkmb9kHfT0raqw+mGRERERGx1Foqgxvbq3X8US6vPLDpWat7XQZ1hUrlVs6fUbKVHQqMAHYE/gC8YwDGzwpdRERERAx5S2VwszCSzpR0paQrJM0CjpL0Nkl3S5ouaYqk8yStUOtfLOnLnfoYL+kT9fPGkq6S9IykxyT9ay+ntA/lHpqDbU+w/bLt6bbPs31J0xi/lDRN0l8l/Uun97lC0lhJsyT9UdJOtewKykWh19WVq09J2kKSJX1I0hPAr2rdgyU9UH+DmyRt3Xmitd5bJd0raaakpyX9dxf1TpDUkNSYP3dGL3+SiIiIiIi+NSSDm+q9wDjKKsmVwMvAvwPrAv8AvBv4SK07DjisrrAgaR3KisqVkoZRLr28BxgFvAs4WdLevZjLO4G7bE/qps6VwGOUQOVQ4CuS9mwqPxi4HFgTuA44D8D24ZS7avatK1fnNrX5R2AbYP96QedY4N+A9Sh36FzTEeB18i3gv22vAWwB/KTVhG1faLvNdtuwVUZ0+wNERERERPS3oRzc3G77GtsLbM+zfY/t39RVk0eBC4GO4OFmYAXgbfX7IcBttp8G3gqsYfts2y/afhj4X8oFmD21DjClq0JJmwO7AKfaft72vcD3gaObqt1i+3rb8ylBzo49GPcLtufanlfn+wvbN9l+iXLh6BrAri3avQRsKWkd27Ns/6YnLxkRERERMZiGcnAzsfmLpG3qVrOnJM0EvkhZxcH2AsrKyeG1+hFAx9mdzYBN61au6ZKmA/8BbNiLuTwLbNRN+Uhgqu05Tc8ep6wUdXiq6fNcYNUejNv8G4ysfQJ/f+cnO43R4UPAG4CHJP1W0n49GCsiIiIiYlAN5YPm7vT9u8DdwKG2Z0v6DHBAU/kVwC8lnQvsBFxVn08E/mp728WYy43AxyWNtD25RflkYF1JqzYFOJsC3W1ja9b5XctDu/n5ZGDLji+SlgM2bjWG7Yco2/SWAz4A/FTSWraf72oCO4waQSNpnSMiIiJiEA3llZvOVgdmAHPq+ZOPNBfavqeWXwhca3tmLboLeFHSpyWtJGmYpB0kvaUXY18P/Bq4StKbax9rSPq4pGNsPwY0gLNryugdKasnr8n81oWngdctpM6PgPdI2queszkZmAW8ZsuZpKMlrVtXd2ZQgqcFPZxLRERERMSgWJaCm08Dx1D+Qf9dyja0zq6gHP4f1/HA9svAfpQzMe3A1Np+jZ4OXFdQ3kfJWvYTYCZwP+XczE212qGUlZWnap3TbP+6h0OcDZxRt82d1MUcHqC8//8Az1ASKrynnr/pbD/gzzXT3Fcpq10v9nAuERERERGDQq/euRSxaNra2txoNAZ7GhERERExxEmaYLutVdmytHITERERERFD2DIX3EjaWtLv6mWYCyR9vo/6PaZeotn57ylJY3vYx82SjuuL+bTo+zRJF/dH3xERERERS4KhnC2tK/8B3Gz7zc0PJe1FPfRv+31Nz98E3Ee5Z2avrjq1fSlwaefnksZQLsLsFUkrAl+inMVZk3JO5mrbn+xtX3V+Zy9Ku566f9IMRp86fpHatifLWkRERET0gWVu5YZyb80DXZQ9A+wmaZ2mZ8cAf+n3Wb3WZ4E2SiKD1YG3A79blI4kLYtBbEREREQsY5ap4EbSTZQg4fy6ZWycpDObqrwIXA0cVusPAw6hU0pmSbtJukfSjPrf3ZrKNpd0S932dgP1otCm8rdKurNmNvt9XTFqZWfKKtJkF+22L2vqZ6Skn0p6RtJjkj7RVDZG0k8kja0Xlh5bn41tqtPlPCQdK+nR+g6PSTqyRz9wRERERMQgWqaCG9vvAG4DTrS9GiWY6ewy4IP18z6UVZ6/X7wpaW1gPHAesA5wLjC+abVnHDCBEtT8F2Xlp6PtqNr2TGBt4DOUCzLXazGPu4FP1btwdpCkpn6WA64Bfg+MAvYGTpK0T1P7gygppdfktcFZl/OQtGp9t31trw7sRtmW9xqSTpDUkNSYP3dGqyoREREREQNmmQpuesL2ncDakramBDmXdaqyP/BX25fbftn2FcCDwIGSNqWsuHze9gu2b6UEIR2OolwQeq3tBbZvoFzeuV+LqXwJOAc4staZJKkjUNoZWM/2F22/aPtR4CLqilN1l+2r6zjzOvW9sHksALaXtLLtKfWOnFa/1YW222y3DVtlRKsqEREREREDJsFNa5cDJ1K2sF3VqWwk8HinZ49TVlBGAs/ZntOprMNmwAfqVrDpkqYDuwMbdZ6A7fm2v237HyirL2cB35O0be1nZKd+TgM2aOpiYjfv1+U86twPBT4KTJE0XtI23fQVEREREbFEyEHz1i4HHgYusz23aUcYlC1qm3Wqvynw/4ApwFqSVm0KcDYFOm5KnQhcbvv43kymrrx8W9IZwBtqP4/Z3rK7Zt2UdTsP29cD10tambJ17SJgj+7muMOoETSS9SwiIiIiBlFWblqw/RiwJ/CfLYqvBbaSdISk5SUdSgk4fmn7ccr2rjMkrShpd+DAprZjKdvX9pE0TNJKkvaStHHnQSSdVMtWruMcQ8ma9jvgt8BMSafU8mGStpe0cw9fsct5SNpA0nvq2ZsXgNnA/B72GxERERExaBLcdMH27bYnt3j+LHAA8GngWcq9OQfYnlqrHAHsCkwDvkDTmR3bEykH/U+jpJ2eCJxM6/8d5gFfA54CpgL/Cvyz7Udtz6cETTsCj9Xyi4EeHXxZyDyWq+82ub7DnsDHe9JvRERERMRgkt3d7qWInmlra3Oj0RjsaURERETEECdpgu22VmVZuYmIiIiIiCEhwU1ERERERAwJCW6WApIsaYvBnkdERERExJIsqaCHEEl7ATcBc+uj6cCdwH/bvqc/x75/0gxGnzp+kdu3J410RERERCymrNwMPZNtr0ZJG/1W4EHgNkl7D+60IiIiIiL6V4KbAVbvppkkaZakhyTtLWkXSXdJmi5piqTzJa3YRfvhkr4q6QlJT0u6oF62+SounrR9OiVN9DlNfXxT0kRJMyVNkLRHfb6hpLmS1mmq+xZJz0haoe9/jYiIiIiIvpPgZgBJ2ho4EdjZ9urAPkA75ZLMTwLrAm8D9qbru2XOAbai3HGzBTAKOH0hQ/8M2KlezAlwT22/NjAO+LGklWw/BdwMHNLU9ijgh7ZfavE+J0hqSGrMnztjIVOIiIiIiOhfCW4G1nxgOPAGSSvYbrf9iO0Jtu+2/bLtduC7lMszX0WSgOOBT9qeZnsWcDZw2ELGnQwIWBPA9ljbz9bxvlbntHWteykloEHSMOBw4PJWndq+0Hab7bZhq/To/tCIiIiIiH6ThAIDyPbDkk4CxgDbSboe+BSwGnAu0AasQvnfZUKLLtar5RNKnAOUoGXYQoYeBZiSYABJnwaOA0bW52tQVo0Afg5cIOl1lBWiGbZ/29t3jYiIiIgYaAluBpjtccA4SWtQVmjOoQQZvwMOtz2rBkDvb9F8KjAP2M72pF4M+17gXttz6vmaUyhb3x6wvUDSc5QgCdvPS/oRcCSwDV2s2nS2w6gRNJLxLCIiIiIGUbalDSBJW0t6h6ThwPOUQGU+JbPZTGC2pG2Aj7Vqb3sBcBHwdUnr1z5HSdqnxViqZV+grNKcVotWB14GngGWl3Q6ZeWm2WXAscB7gLGL8coREREREQMmwc3AGg58mbIC8xSwPiXo+AxwBDCLErxc2U0fpwAPA3dLmgncyCvnZQBGSpoNzKYkDtgB2Mv2r2r59cB1wF+AxylB1sTmAWzfASygrPa0L+K7RkREREQMKNke7DnEEkjSTcA42xf3pH5bW5sbjUY/zyoiIiIilnWSJthua1WWMzfxGpJ2BnYCDhrsuURERERE9FS2pcWrSLqUstXtpJpqOiIiIiJiqZCVmyFE0hhgC9tH9bD+XsBY2xt3PLN9zKKMff+kGYw+dfyiNP279mRbi4iIiIjFkJWbiIiIiIgYEhLcLKUknSJpkqRZkh6StD8l89qhkmZL+n2t9yFJf671HpX0kfp8VUrWtJG1/mxJIyUtJ+lUSY9IelbSjyStPXhvGhERERHRMwlulkKStgZOBHa2vTqwD/AgcDZwpe3VbL+pVv8bcADlLpsPUe7I2cn2HGBfYHKtv5rtycAngIOBPSmXiz4HfLuLeZwgqSGpMX/ujH5734iIiIiInkhws3SaT7kz5w2SVrDdbvuRVhVtj7f9iItbgF8Be3TT90eA/7T9pO0XgDHA+yW95nyW7Qttt9luG7bKiMV+qYiIiIiIxZHgZilk+2HgJErg8TdJP5Q0slVdSftKulvSNEnTgf2AdbvpfjPgKknTa/0/U4KpDfr0JSIiIiIi+liCm6WU7XG2d6cEIwbOqf/9O0nDgZ8CXwU2sL0mcC2gjm5adD0R2Nf2mk1/K9me1F/vEhERERHRF5IKeilUz9yMAu4AngfmUQLVp4F3SVrO9gJgRcr2tWeAlyXtC/wT8Mfa1dPAOpJG2O44NHMBcJakY2w/Lmk9YDfbP+9uTjuMGkEjqZwjIiIiYhBl5WbpNBz4MjAVeApYn5Ip7ce1/FlJ99ZLOD8B/IiSGOAI4Bcdndh+ELgCeLRuQxsJfLPW+ZWkWcDdwK4D8lYREREREYtBdqudSRG909bW5kajMdjTiIiIiIghTtIE222tyrJyExERERERQ0KCm4iIiIiIGBIS3DSRZElbDPY8FkbSeyVNlDRb0psHez4REREREUuCZEtbBJL2Am4C5tZH04E7gf+2fc8ATOGrwIkLy2A2kO6fNIPRp45frD7ak20tIiIiIhZDVm4W3WTbqwGrA28FHgRuk7T3AIy9GfDAAIzzd5ISCEdERETEEm3IBjeSTpE0SdIsSQ9J2lvSLpLuqmmPp0g6X9KKXbQfLumrkp6Q9LSkCySt3Lmeiydtnw5cTLlMs6OPb9btYzMlTZC0R32+oaS5ktZpqvsWSc9IWkHScpI+J+lxSX+TdJmkEXVOs4FhwO8lPSLpZEk/7TT3b0n6Rv08QtL/1vedJOlMScNq2esl3STpWUlTJf1A0ppN/bTX3/EPwJwEOBERERGxJBuSwU295PJEYGfbqwP7AO3AfOCTwLrA24C9gY930c05wFbAjsAWlEszT1/I0D8DdpK0av1+T22/NjAO+LGklWw/BdwMHNLU9ijgh7ZfAo6tf28HXgesBpxv+4W6WgTwJtuvB8YC7+4ISmoAcihwea13KfByfYc3Uy7xPK7jpwK+BIwEtgU2AcZ0eqfDgf2BNW2/3Fwg6QRJDUmN+XNnEBERERExmIZkcEMJYoYDb5C0gu1224/YnmD7btsv224Hvgvs2bmxJAHHA5+0Pa1ehnk2cNhCxp1MCRjWBLA91vazdbyv1TltXeteSgloqCsph/NKQHIkcK7tR23PBj4LHNZq5cT2FOBW4AP10buBqbYnSNoA2Bc4yfYc238Dvt7xHrYftn1DDZqeAc5t8XucZ3ui7Xktxr7QdpvttmGrjFjITxMRERER0b+G5DYj2w9LOomyCrGdpOuBT1FWQM4F2oBVKO8/oUUX69XyCSXOAUrQMmwhQ48CTEkwgKRPU1ZJRtbna1BWjQB+Dlwg6XWUFaIZtn9by0YCjzf1+3id6wbApBbjXgp8DLiIEjB1BEmbASsAU5reYzlgYp3f+sB5wB6Us0PLAc916nviQt45IiIiImKJMCSDGwDb44BxktagrNCcQwkafgccbntWDYDe36L5VGAesJ3tVsFEV94L3Gt7Tj1fcwpl69sDthdIeo4SJGH7eUk/oqzSbMMrAQmUFaDNmr5vStla9nQX414N/I+k7YEDgP+ozycCLwDrdt5SVn2JEnS90fazkg4Gzu9Uxz14b3YYNYJGsp1FRERExCAaktvSJG0t6R2ShgPPUwKV+ZTViZnAbEnbUFY7XsP2AsoqyNfr6gaSRknap8VYqmVfoKzSnFaLVqcEJM8Ay0s6nbJy0+wyytma91DOznS4AvikpM0lrUbZEndlFwEKtp8HfkI51/Nb20/U51OAXwFfk7RGTVTwekkdW89WB2YD0yWNAk5u1X9ERERExNJgSAY3lLMtX6aswLPHTaUAACAASURBVDwFrE8JOj4DHAHMogQvV3bTxynAw8DdkmYCN/LKeRmAkTVz2WxK4oAdgL1s/6qWXw9cB/yFsq3seTpt8bJ9B7CAstrT3lT0PcpKzq3AY7Xtvy3knS+tc7i80/MPAisCf6JsOfsJsFEtOwPYCZgBjKckRIiIiIiIWCrJ7tGuo+gnkm4Cxtm+eDH72ZRy186Gtmf2yeR6oa2tzY1GY6CHjYiIiIhljKQJtttalQ3ZMzdLA0k7U1ZODlrMfpajJEz44WAENhERERERS4IEN4NE0qXAwcC/11TTi9rPqpREA49T0kBHRERERCyTEtwMEtvH9FE/cygpriMiIiIilmkJbgaJpNOA19k+bgDH3AO42PbWC63cS/dPmsHoU8f3WX/tSSsdEREREb00VLOlDTpJs5v+Fkia1/T9SNtn90dgI2k7Sb+S9Jyk6ZImSNoPwPZt/RHYREREREQsCbJy009s/32rmKR24DjbN/akraTlu7rTpgeuAf6HcpknwM7Ui0MjIiIiIoayrNwMEkljJI2tn0dLsqQPS3oCuKk+f6ukO+sKzO8l7bWQPtcFNgcusv1i/bvD9u21fC9JT9bPh3ZaXXpB0s21bLikr0p6QtLTki6QtHKL8U6Q1JDUmD93Rh/+OhERERERvZfgZsmyJ7AtsI+kUZSLNc8E1qZcQPpTSet10/5ZysWjYyUdLGmDriravtL2anWFaSTwKHBFLT4H2ArYEdgCGAWc3qKPC2232W4btsqIXr5qRERERETfSnCzZBlje47tecBRwLW2r7W9wPYNQAPYr6vGLjeyvh1oB74GTJF0q6Qtu2pT78gZB9xs+7uSBBwPfNL2tJqm+mzgsD56x4iIiIiIfpEzN0uWiU2fNwM+IOnApmcrAL/urgPbTwInAkjaBLgQuAx4WxdNzgJWBz5Rv68HrAJMKHEOUM7sDOtu3B1GjaCRDGcRERERMYgS3CxZ3PR5InC57eMXuTN7oqRv88p2s1eRdBhwOLCz7Zfq46nAPGA725MWdeyIiIiIiIGWbWlLrrHAgZL2kTRM0ko1IcDGXTWQtJakMyRtIWm5mmDgX4C7W9R9M/At4GDbz3Q8t70AuAj4uqT1a91Rkvbp4/eLiIiIiOhTCW6WULYnAgcBpwHPUFZyTqb7/81eBEYDNwIzgT8CLwDHtqh7ELAWcHtTxrTratkplMQEd0uaWfvL/TgRERERsURTOYMesXja2trcaDQGexoRERERMcRJmmC7rVVZVm4GWfN9N/3UvyVtUT9fIOnz/TVWRERERMRgSkKBRSCpHTjO9o1Nz46tz3YfgPFnd1G0r+3bumpn+6P9NCXunzSD0aeO76/u/649GdkiIiIiogsJbpZgkpa3/XLn5/XizYiIiIiIaJJtaX1M0smSftrp2bckfaN+3lzSLZJmSboBWLep3ui6jezDkp4AbqrPfyzpKUkz6qWc2zW1uVnScU3fj5V0exdzu0TSmU3fD5J0n6SZkh6R9O6mPh6tc3xM0pF98+tERERERPSfBDd9byzwbklrQll9AQ4FLq/l44AJlKDmv4BjWvSxJ7At0JF++TpgS2B94F7gB4s7SUm7UC73PBlYE/hHoF3SqsB5lC1uqwO7Afd10ccJkhqSGvPnzljcKUVERERELJZsS1t0V0tq3jK2InCv7SmSbgU+QLkv5t3AVNsTJG0K7Ay80/YLwK2SrmnR9xjbczq+2P5ex2dJY4DnJI2wvTgRxYeB79m+oX6fVPtfFVgAbC/pCdtTgCmtOrB9IXAhwPCNtkzavYiIiIgYVFm5WXQH216z4w/4eFPZpcBR9fNRvLJqMxJ4rjlwAR5v0ffEjg/1As8v121jM4H2WrRui3a9sQnwSOeHdW6HAh8FpkgaL2mbxRwrIiIiIqLfJbjpH1cDb5S0PXAAr2wjmwKsVVdHOmzaon3zKsgRlAs33wmMoFzSCaD63znAKk31N+zhHCcCr29VYPt62+8CNgIepKxARUREREQs0bItrR/Yfl7STyjna35r+4n6/HFJDeAMSacBuwAHAr/oprvVgReAZylBzNmdyu8D3ifpYsrK0IeBp3swzf8FfiXpl8CvKYHM6sBzwK7A/wHzgNnA/IV1tsOoETSSpjkiIiIiBlFWbvrPpcAOvLIlrcMRlOBhGvAFyqH+7lxG2bo2CfgTcHen8q8DL1ICmkvpYbIB278FPlTbzwBuATaj/H/i08DkOsc9efWWu4iIiIiIJZLsnAPvDzV5wIPAhrZnDvZ8+ltbW5sbjcZgTyMiIiIihjhJE2y3tSrLyk0/kLQc8Cngh8tCYBMRERERsSTImZs+VpMFPE3ZSvbuQZ5ORERERMQyI8FNFyRdAEyy/V+9aVdTKa/WP7OKiIiIiIiuLBNnbiS1AxsAL1Myf/2JclD/QtsLBmlO21EO8+9MSev8CPB529cuRp83A2NtX9wnk+yF4Rtt6Y2O+cZAD/sq7cnWFhERETHk5cxNcaDt1SkZwb4MnEJJh/wakoYNwHyuAW6gBF3rA58Acj4nIiIiImIRLUvBDQC2Z9j+BXAocIyk7SVdIul/JF0raQ7w9vrsTABJf5Z0QEcfkpaXNFXSTvX7WyXdKWm6pN9L2qu7OUhaF9gcuMj2i/XvDtu31/I/Sjqwqf4KdbwdJa0kaaykZ+t490jaQNJZwB7A+ZJmSzq/tt1G0g2Spkl6SNIhTf1eIuk7kq6rbe6QtKGkb0h6TtKDkt7cN798RERERET/WuaCmw71npcnKQEBlPtnzqJcZHl7p+pXAIc3fd8HmGr7XkmjgPHAmcDawGeAn0par5vhnwUeBsZKOljSBp3KLwOOavq+HzDF9n3AMcAIYBNgHeCjwDzb/wncBpxoezXbJ9bkBjdQLhNdv77Dd+qWuA6HAJ8D1qVcFnoXcG/9/hPg3K5eQtIJkhqSGvPnzujmdSMiIiIi+t8yG9xUkykBCcDP6+rJAtvPd6o3DniPpFXq9yPqMyhByLW2r61tbwAalICkJZeDTm8H2oGvAVMk3Sppy1plLLCfpDXq96N55TLQlyhBzRa259ue0E266QOAdtvft/2y7XuBnwLvb6pzVe3jeeAq4Hnbl9meD1wJdLlyY/tC222224atMqKrahERERERA2JZD25GAdPq54ldVbL9MPBn4MAa4LyHV4KbzYAP1C1i0yVNB3YHNupuYNtP2j7R9utrH3MoKzbYngzcAfyzpDWBfYEf1KaXA9cDP5Q0WdJXJK3QxTCbAbt2mtuRwIZNdZ5u+jyvxfdkfouIiIiIpcIymwpa0s6U4OZ2YFdgYWnjOramLQf8qQY8UIKiy20fv6hzsT1R0rfrGB0uBY6j/G90l+1Jte5LwBnAGZJGA9cCD1GSI3R+h4nALbbftahz66kdRo2gkWxlERERETGIlrmVG0lr1OQAP6SkTb6/h01/CPwT8DFeWbWBsoXsQEn7SBpWD/zvJWnjbuawlqQzJG0habmaYOBfgLubql0N7AT8O3VFp7Z9u6Qdaka3mZRtavNr8dPA65r6+CWwlaSja1KCFSTtLGnbHr5zRERERMRSY1kKbq6RNIuymvGflIPyH+ppY9tTKIftd6OcRel4PhE4CDgNeKb2fzLd/7YvAqOBGykByh8ph/mPbep3HuV8zObAz5rabkg56D+TslXuFkqABfBN4P0109l5tmdRArLDKOeLngLOAYb39L0jIiIiIpYWy8QlnksrSacDW9k+aqGVB1lbW5sbjcZgTyMiIiIihrjuLvFcZs/cLOkkrQ18mJIpLSIiIiIiFmJZ2pY24OrFmK3+9lhIu+Mp29uus33rwMw2IiIiImLplpWbfmS712mUJV0CPGl71S7KZwNvtP3oYk4vIiIiImJISXCzCCS1AyOBkbanNj2/D3gTsLnt9v4Ye1ECplaagqjP1ZTSj1Hu2qH+9x7gm/VS0oW6f9IMRp86vi+mNuDak8I6IiIiYkjItrRF9xjl3hsAJO0ArDx40+kTa9bg6U3ADcBVko4d3ClFRERERPRMgptFdznwwabvx/Dq+2j2l/Q7STMlTZQ0prmxpN0l3Slpei0/tql4LUnjJc2S9BtJr29qZ0lb1M+XSPp2N3W3kXSDpGmSHpJ0SE9ezPZTtr8JjAHOkZT/n0RERETEEi//aF10dwNrSNq2Xqh5KK/cNwNla9cHgTWB/YGPSToYQNKmwHXAt4D1gB2B+5raHg6cAawFPAyc1c08WtaVtCpl9WUcsH6t9x1J2/XiHX9W227dqlDSCZIakhrz587oRbcREREREX0vwc3i6Vi9eRfwIDCpo8D2zbbvt73A9h+AK4A9a/GRwI22r7D9ku1nbTcHNz+z/VvbLwM/oAQ/Xemq7gFAu+3v237Z9r2US0Hf34v3m1z/u3arQtsX2m6z3TZslRG96DYiIiIiou8locDiuRy4Fdicpi1pAJJ2Bb4MbA+sCAwHflyLNwEe6abfp5o+zwW6SyLQVd3NgF0lTW8qX77OuadG1f9O60WbiIiIiIhBkeBmMdh+XNJjwH6UCzebjQPOB/a1/bykbwDr1rKJwC79PL2JwC2237UYfbwX+Bvw0MIq7jBqBI1kHYuIiIiIQZRtaYvvw8A7bM/p9Hx1YFoNbHYBjmgq+wHwTkmHSFpe0jqSutt6tih+CWwl6WhJK9S/nSVtu7CGkjaQdCLwBeCzthf08dwiIiIiIvpcgpvFZPsR240WRR8HvihpFnA68KOmNk9QVns+Tdny1XE/Tl/OaxbwT8BhlLMzTwHnULbHdWW6pDnA/XV+H7D9vb6cV0REREREf5HtwZ5DDAFtbW1uNFrFeBERERERfUfSBNttrcqychMREREREUPCEhPcSNq6Xno5S9ICSZ/v5/HGSBq78Jog6WZJx/XnfBaFpAck7TXY84iIiIiIWBIsSdnS/gO42fabmx/Wf7z/GrjK9vuanr+JclblFtt7DdQkJR0L/C8wrz56BrgZ+JLtvwzUPABs9/hCTkntwHG2b+yPudw/aQajTx3fH10vkdqTGS4iIiJiibPErNxQ7mV5oIuyZ4DdJK3T9OwYYECDiSZ32V4NGAG8kxLoTJC0/UAMLmlAg9KBHi8iIiIiYlEsEcGNpJuAtwPnS5otaZykM5uqvAhcTcn8haRhwCGUlMrN/ewm6R5JM+p/d2sq21zSLXXb2w28cudMR/lbJd0pabqk3/dku5ft+TVb2seBW4AxPelP0rGSHq1zeUzSkU1lx0v6cy37k6Sd6vN2SadI+gMwp6aQbpf0zlo+RtJPJF1Z295bV7eQdDmwKXBN/X3/oz5/T93aNr1uvdu2aR6vGW9hv0dERERExGBaIoIb2+8AbgNOrCsiL7aodhnwwfp5H8oqz+SOQklrA+OB84B1gHOB8U2rPeOACZSg5r8oKz8dbUfVtmcCawOfAX4qab1evMbPgD0W1p+kVesc97W9OrAbZXsdkj5ACZA+CKwBvAd4tmmMw4H9gTVtv9xiDgcBP65jjgOulrSC7aOBJ4ADba9m+yuStgKuAE4C1gOupQQ/K/ZivIiIiIiIJcYSEdz0hO07gbUlbU35x/9lnarsD/zV9uW2X7Z9BfAgcKCkTYGdgc/bfsH2rcA1TW2PAq61fa3tBbZvABqUu156ajIlqOhJfwuA7SWtbHuK7Y7teMcBX7F9j4uHbT/eNMZ5tifankdrE2z/xPZLlOBuJeCtXdQ9FBhv+4Za/6vAypRgq0fjSTpBUkNSY/7cGV39LhERERERA2KpCW6qy4ETKVvYrupUNhJ4vNOzx4FRtew523M6lXXYDPhA3Z41XdJ0YHdgo17MbRTlQs5u+6tzOBT4KDBF0nhJ29R2mwCPdDPGxIXM4e/lthcAT1LevZVX/V61/sT6Hj0az/aFtttstw1bZcRCphYRERER0b+WtnMUlwMPA5fZniupuWwyJahotinw/4ApwFqSVm0KcDYFOm4wnQhcbvv4xZjbeylb6xban+3rgeslrUzZunYRZUvbROD13YyxsBtXN+n4IGk5YGNe2brXue1kYIem+qrtJ/VivIiIiIiIJcZSFdzYfkzSnsCjLYqvBb4l6QjgR8A/A28Afml7qqQGcIak04BdgAOBX9S2Y4F7JO0D3AisQNnO9bDtJ7uaT01ssCnwKWAv4G0L6w94CdgV+D9KlrXZwPza7mLgXEm3A/dSAp2XOm1N685bJL2vvtcngBeAu2vZ08Drmur+CDhV0t7ArcC/1/p39nCsV9lh1AgaSY8cEREREYNoaduWhu3bbU9u8fxZ4ADg05RD+P8BHGB7aq1yBCWomAZ8gaYzO7YnUg7jn0ZJOz0ROJmuf5+3SZoNzKTccbMGsLPt+3vQ33J1jpPrXPYEPl7b/Rg4i5IMYBYlQ1zHOZ6e+Dlly9tzwNHA++p5GoAvAZ+r2+Q+Y/shytmgbwFTKcHegbZbJXOIiIiIiFjiyc7Oo6FA0hhgC9tHDcb4bW1tbjQagzF0RERERCxDJE2w3daqbKlbuYmIiIiIiGglwU1ERERERAwJS1VCge5I2gsYa3vjwZ7LYLA9ZrDnEBERERExmAY8uJHUDmxAyRA2h5Ll7N9szx7oubQi6WZKkHRx0zMB/wqcAGxBSSTwIHCB7R8Owhz3Am4C5tZH0ylZzv7b9j0DPR+A+yfNYPSp4wdj6CVKezLGRURERAyaHm9Lk3S4pG3r560l3SrppqYLKHvjQNurATsBOwOf6zSW6j0tS4rzgJMoWc7WoVx0+Tng3a0qD9D8J9ffcHVKmukHgdtqaueIiIiIiGVOb/4BfiYldTHAV4HfUu5H+c6iDm57EnAdsL2kmyWdJekOyorE6ySNlPQLSdMkPSzp75diSlpZ0iWSnpP0J0qQRFO5JW3R9P0SSWc2fT9I0n2SZkp6RNK7JZ1FuUzzfEmzJZ0vaStKqubDbN9ge57t+TUl9bFN/fV2/p3ns5ekJ5u+t0v6rKQ/1Xf8vqSVWvyGtv2k7dMp9+Sc09THNpJuqOM/JOmQprL9at+zJE2S9JnufpuF/W8ZERERETHYerMtbT3bT9d/YO8OvJ9yIeXU7pt1TdImwH7AzyhBxdHAvsBDgCgXYD4AjAS2AW6Q9Kjt/6PcVfP6+rcqJUjq6bi7UO65eT/lMs2NgNVt/z9J/0DTtjRJHwUm2u5JnuPezL8njgT2oWzfu4ayWvS5bur/DPi4pFXr9xuA0+uc3gj8StIDth8A/hc4xPZtktYCNq/v2/K3aTWYpBMoW/UYtsZ6PXyliIiIiIj+0ZuVm2f+P3v3GaZXVbZ9/H8S6QkJMQgkhEQBUYqiDio2UOGhKIKPBamiFBG7qCA2UBD1saCiL1KULiAClqCAYhCkyARBpKiUhCGJQAip9OR8P6w1sLmdycykzUw4f8dxH9z3Xnuvfe2dL3Ox1rpWHQnZGbjB9uPAapQ/4vvqYkmzgKuBK4Gv1+On2b7V9lPAepQk6nDbj9m+iTIysW89973AsbZn1k0zf9CH+x8A/LSOxCy0PdX2Hd2cOwr4T/OApPvqZpiPSRrXaOpL/L1xgu0O2zMpm3vu2cP50yj/HiMoG5pOtv0z20/ZvhH4JSVpgZKYbiZpLdsP13bow7uxfZLtNtttQ9YY3ofHioiIiIhY+vqS3HwNmET5P/7/V4+9Fbh5Me67u+0RtsfZPtT2o/V4R+Oc0cBM23Mbx6ZQ1rt0tne0tPXWWOCuXp77EGX04mm1ItsoYFWendz1Jf7eaH2+0T2cPwYwpcDAOOA1NQmbVZPJvSlJF8C7KKNmUyRdKWmberwv7yYiIiIiYsDo9bQ026dJOr9+76zSdT3wvqUYjxvfpwEjJQ1rJAgbAlPr9+mUP8RvbbQ1PQKs0fi9HtC5pqWDMp2tpxigVCU7QVJbL6am9SX++V3E12ps4/uGtc9FeSdwo+35kjqAK23v0GWgparabpJWBj4KnF/vt6h3060txwynPZXCIiIiIqIfLXLkRtJKzQ/wGPBY4/cM4IFlEVidanYNcJyk1SS9jDJl6ux6yvnA5yWtLWkD4GMtXdwE7CVpSF0Qv22j7VTgA5LeWp9lTKPq2/3Aixpx/BP4CXCupB1qIYMhwOuWMP6bgF0kjZS0HqUaW6uPSNpA0kjgSOC81hNqZbYxkr4CHFjPA/gt8GJJ+0pauX62lvRSSatI2lvScNtPUkpbL+jFu4mIiIiIGLB6mpb2FGVtRnefzvZlZU9gPGXE4iLgK7Yvr21HU6Zq3QNcBpzZcu0ngF0pU7T2Bi7ubLD9V+ADwPeA2ZR1P51rZ74PvLtWKOtcx/MRypqe71Iqxt1Hmaa3B3DvYsZ/JmVK3+Qa/38lLsA5te3u+jmm0TZa0jxgHnADsCWwne3L6jPOBf6HMrI2jbJu6JuUqXRQ1v5MljQHOATYpxfvJiIiIiJiwJLdOgur0fjsxfLdst2X9S7RCyqbnR5o+w/9HUtvtLW1ub29NwXlIiIiIiIWn6RJttu6alvkmpuukpY6HW1d4H7bC5dOiBEREREREUum19XSJK0l6QzKupupwKOSTpeUGsAREREREdHv+lIK+geUzTK3AFanrPFYg77tLxO9tz9wWj/HEBERERExaPS6FDSwE/CiRhnof0n6ANkTBXh6jcy6PFN1DMqmnh/tn4iWr1umzmb8ERP6O4wBY3LKYkdEREQsd31Jbh4D1uHZm2WOAh5fqhENbrsOlgIAXZE0xPaCns+MiIiIiBh4+jIt7RTgckmHSNpZ0iHApcDJyya0FYOk/SVdLenbtbz0PZJ2brSPlPQzSdNq+8Xd9PNSSRMlzZJ0q6R3NNp2kXSbpLmSpkr6TPPeLf1Y0sb1+2mS/p+kSyTNB94sadUa672S7pd0oqTVl8nLiYiIiIhYivoycnMspZDA3sBoyt4p37J96rIIbAXzGuB0ykjXwcCpksa41OE+k7JXzeb1v/+1OaiklYHfAD+l7F3zBuBXktrqJqOnAu+1fZWktYEX9iG2vYBdgLcDq1D2wnkRsBVlD6NzgC8Dn+8iroPr8zBkrXX6cMuIiIiIiKWvLyM33wf+aXt725vZ3h64XdLxyyi2wejiOrLS+TmoHp9i++Q65et0YH1gXUnrAzsDh9h+2PaTtq/sot/XAkOBb9h+wvYVwG8pm4RCSUI2k7RW7efGPsT8K9t/qWW9HwcOAj5le2bdCPTrlI1A/4vtk2y32W4bskaK5kVERERE/+pLcrMn0LpL4yTK//mPYnfbIxqfzil7/+k8oVGQYSgwFphp++Ee+h0NdLTsKzQFGFO/v4sy+jJF0pWStulDzB2N7+tQKuBN6kzQgN/X4xERERERA1pfpqUZGNJybAh9S5Di2TqAkZJG2J61iPOmAWMlrdRIcDYE/gVg+wZgtzp97aPA+ZTEaT4lWQFA0npd9O3G9xnAo8Dmtqf25UG2HDOc9lQIi4iIiIh+1JfE5Crga5JWAqj/Paoej8VgezrwO+DHktaWtLKkN3Vx6vWUROVz9ZztgF2BcyWtImlvScNtPwnM4Zly1DcDm0vaStJqlH+vRcWzkFIg4nuSXgAgaYykHZf8aSMiIiIilq2+JDefALYHpkv6K2U0YQfgY8sisEHqN5LmNT4X9eKafSlrZu4AHgA+2XqC7SeAd1DW58wAfgzsZ/uORh+TJc0BDgH2qdf9C/gq8Afg38DV9Oxw4E7gutrfH4BNe3FdRERERES/UinY1cuTy2jNqylTnjqAv7asA4nnqLa2Nre3ty7JioiIiIhYuiRNst3WVVtf1tx0Tlu6rn4iIiIiIiIGjBQDGKCam21GRERERETP+jRyEwOLpHWBW4F3257YOP4zYDXbe3Z37dJ2y9TZjD9iwvK63aA0OdXkIiIiIpapJDeDmO37JX0KOFnSy2w/KumtwNuAzZfmvSQ9z/ZTS7PPiIiIiIilKdPSlgNJh0uaKmmupH9KequkV0u6tm6WOV3SCZJW6eb6VSV9W9K9ku6XdKKk1QFsnwn8E/hqPfYT4OO2H6zXbiDpIkkPSrpH0kca/W4j6bpGDD+oe+Ug6Xl1atyhku6kVHOLiIiIiBiwktwsY5I2pWysubXtYcCOwGTKXjSfAkYB2wBvBQ7tpptvAi8GtgI2BsYAX260HwJ8EDgX+Iftc+u9hwC/BW6o1+wAfLaO7gA8RSnxPQp4PbAT8KGWe78D2BrYss8PHxERERGxHCW5WfYWAKsCm0la2fZk23fZnmT7OttP2Z5MGXHZtvViSQIOAj5le6btucDXgfd1nmP7Pkqysz3w4cblrwXWsv1120/YvhM4tfNa2zfYvr7GcDdwUhcxfN32w7Yf7SK2gyW1S2pf8MjsxXw9ERERERFLR9bcLGO275T0SeAoYHNJlwKfBoYC3wXagDUo/xaTuuhindo+qeQ5AAgY0nLercDDtqc3jo0DNpQ0q3FsCDARQNJLgO8Ar2rEcH1Lvx2LeLaTKAkRq66/Se83TIqIiIiIWAYycrMc2D7H9hsoyYYp08z+H2Udyya21wKOpCQtrWYAjwKb2x5RP8NtD+3FrTuAfzeuG2F7mO1da/tPgH8AG9cYvtxFDElaIiIiImJQyMjNMlbX3IwB/gI8RklUVgKGAXOAeXUE5cPAg63X214o6WTge5I+avsBSWOALWxf2sPtrwWekHQY8CPgSWAzYBXbk2oMs4H5kl5KWW8zdXGec8sxw2lPqeOIiIiI6EcZuVn2VgW+QRmB+Q/wAsoozWeAvYC5wMnAeYvo43DgTuA6SXOAPwCb9nTjWrp5F+DVlCIGMyijNWvVUw4D3l9j+EkPMUREREREDGiyM+sollxbW5vb29v7O4yIiIiIWMFJmmS7rau2jNxERERERMQKIclNRERERESsEJLcNEjaVNLfJM2VtFDSl/o7pmVBkiVt3IvztpN03/KIKSIiIiJiSaVa2rN9Dpho+xXNg5K2u6DUFQAAIABJREFUA64AHqmHZgOn2P7KsgxG0kTKpppb2b65cfxiYDfgzbYnLssYeuuWqbMZf8SE/g5jUJqcKnMRERERS0VGbp5tHGUzzK5Msz207i/zBuAASbsvh5j+BezX+UPS84HX0kXZ6IiIiIiI57IkN5WkK4A3AydImifpHEnHdHWu7XuAayh7xnRe/31JHZLmSJok6Y2NtldLaq9t90v6bqPttZKukTRL0s11lKjpbGAPSUPq7z2Bi4AnGn2sKul4SdPq53hJqzbaPytpem37YMtzryrp25LurbGdKGn1vr29iIiIiIj+l+Smsv0W4Crgo3V05onuzpW0CfB64LrG4RuArYCRwDnALyStVtu+D3zf9lrARsD5tZ8xwATgmHrdZ4BfSlqn0e804Dbgf+rv/YAzWkL6AmU0Zyvg5ZR9bb5Y77FT7XcHYBNg+5Zrvwm8uF67MWXD0S939+wt7+HgmrS1L3hkdm8uiYiIiIhYZpLc9N7oOroyhzJV7Hrg6s5G22fZfsj2U7a/Q9m8s3OjzSeBjSWNsj3PdmdStA9wie1LbC+0fTnQTtl4s+kMYD9JmwIjbF/b0r438FXbD9h+EDga2Le2vRf4me1/2J4PHNV5kSQBBwGfsj3T9lzg68D7evNCbJ9ku81225A1hvfmkoiIiIiIZSbJTe9Nsz2ijr6MAB4FTu9slHSYpNslzZY0CxgOjKrNB1BGR+6QdIOkt9fj44D31KRpVr3uDcD6Lfe+EHgL8DHgzC5iGw1MafyeUo91tnW0tHVaB1gDmNS4/+/r8YiIiIiIQSXV0haD7dmSzgHOA6jraw4H3grcanuhpIcB1fP/DewpaSXgf4ELamGADuBM2wf1cL9HJP0O+DBlWluraTy7GMKG9RjAdGBs49wNG99nUJK0zW1P7dXDd2PLMcNpT9WviIiIiOhHGblZDJKGUqZudSYTw4CnKBXMnifpy8BajfP3kbSO7YXArHp4AXAWsKukHSUNkbRa3Vtmgy5ueySwre3JXbT9HPiipHUkjaKsmTmrtp0P7C9pM0lrAE+Xr67xnAx8T9ILaqxjJO3Y55cSEREREdHPktz03uhaRW0eZWrXSMpaF4BLgd9R1uJMAR7j2VPBdgJurdd+H3if7cdsd1D2qzmSkhh1AJ+li38X29NsX916vDqGslbn78AtwI31GLZ/BxxP2afnzvrfpsPr8evqeqI/8MxaoYiIiIiIQUO2+zuGWAG0tbW5vb29v8OIiIiIiBWcpEm227pqy8hNRERERESsEJLcRERERETECiHJzRKqBQDu6+84IiIiIiKe654zpaAlTQbWpVQpmw9cAnzM9rz+jKuTpInAWbZPaRwT8BHgYGBjYA5wB3Ci7XP7I87u3DJ1NuOPmNDfYQxak1NGOyIiImKJPddGbna1PRR4JbA18MVmo4qB9E5+AHwSOAx4PjCGEvNOXZ08AOOPiIiIiFhunpN/CNcNK38HbCFpoqRjJf0FeAR4kaTRkn4taaakOyU9vcmmpNUlnSbpYUm3UZIkGu2WtHHj92mSjmn83k3STZLmSLpL0k6SjgXeCJxQy02fIOnFwKGUstGX237U9gLbV9vev9FfX+NvjedZ0+okTZb0eUm31Wf8maTVlvytR0REREQsW8+ZaWlNksYCuwAXUpKKfYGdgX8Couz1ciswGngJcLmku23/kbIJ5kb1syYlSertfV8NnAG8G/gjsD4wzPbvJb2exrQ0SYcAHbZ7U1+5L/H3xt7AjpTpe7+hjBZ9sfUkSQdTpswxZK11etl1RERERMSy8VwbublY0izgauBK4Ov1+Gm2b7X9FLAe8Abg8LrR5k3AKZQEAuC9wLG2Z9ZNOH/Qh/sfAPy0jsQstD3V9h3dnDsK+E/zgKT7JM2S9JikcY2mvsTfGyfY7rA9EzgW2LOrk2yfZLvNdtuQNYb3ofuIiIiIiKXvuTZys7vtPzQPlDX7dDQOjQZm2p7bODYFaGu0d7S09dZYSiGD3niIMrLzNNsbSHoe8CRlhKZTX+LvjdbnG92HayMiIiIi+sVzLbnpjhvfpwEjJQ1rJAgbAlPr9+mUJOXWRlvTI8Aajd/rAZ1rWjoo09l6igHgCsoanLZeTE3rS/zzu4iv1djG9w1rn4u05ZjhtKfiV0RERET0o+fatLQe1alm1wDHSVpN0sso08nOrqecD3xe0tqSNgA+1tLFTcBekoZI2gnYttF2KvABSW+VtJKkMZJeUtvuB17UiOOfwE+AcyXtUAsZDAFet4Tx3wTsImmkpPUo1dhafUTSBpJGAkcC5y3qnhERERERA0GSm67tCYynjFhcBHzF9uW17WjKVK17gMuAM1uu/QSwKzCLsjD/4s4G238FPgB8D5hNWffTuXbm+8C7a4WyznU8H6Gs6fkuMJMyAvQ1YA/g3sWM/0zgZmByjb+rxOWc2nZ3/RzTxTkREREREQOK7NbZUPFcVjc7PbB1bVJP2tra3N7em8JuERERERGLT9Ik212uJ8/ITURERERErBCS3ERERERExAoh1dIGqcWdPtYT2+MX57pbps5m/BETlmYo0TA5legiIiIiepSRm34m6Q2SrpE0W9JMSX+RtPVyuvdESQfW79tJWihpXv3cJ+n85RVLRERERMSSSnLTjyStBfwW+CEwEhhDqcb2eD+FNM32UGAY8FrgDuAqSW/tp3giIiIiInotyU3/ejGA7Z/bXmD7UduX2f67pI0kXSHpIUkzJJ0taURXndQ9c46QdFc9//y6Rw11r5uz6vFZkm6QtO6ignJxn+0vA6cA31zaDx4RERERsbQluelf/wIWSDpd0s6S1m60CTgOGA28FBgLHNVNPx8HdqdsGDoaeBj4UW17PzC8Xv984BDg0T7EeCHwSklrtjZIOlhSu6T2BY/M7kOXERERERFLX5KbfmR7DvAGwMDJwIOSfi1pXdt32r7c9uO2H6Rs5LltN119CPhCHW15nJIEvVvS84AnKUnNxnV0aFK9b29NoyRa/zVqZPsk222224asMbwPXUZERERELH1JbvqZ7dtt7297A2ALysjL8ZJeIOlcSVMlzQHOAkZ108044KI67WwWcDuwAFgXOBO4FDhX0jRJ35K0ch9CHENJvmYt3hNGRERERCwfKQU9gNi+Q9JplJGY4yhJxctsPyRpd+CEbi7tAD5o+y/dtB8NHC1pPHAJ8E/g1F6G9U7gRtvzF3XSlmOG055yxRERERHRjzJy048kvUTSYZI2qL/HAnsC11Eqls0DZkkaA3x2EV2dCBwraVztZx1Ju9Xvb5a0paQhwBzKNLUFPcQlSWMkfQU4EDhyiR40IiIiImI5SHLTv+YCrwGulzSfktT8AziMMtrySmA2MIGysL873wd+DVwmaW7t5zW1bT3gAkpicztwJWWKW1dGS5pHSapuALYEtrN92eI+YERERETE8iLb/R1DrADa2trc3t7e32FERERExApO0iTbbV21ZeQmIiIiIiJWCEluGiRtKulvkuZKWijpS8v4fkdJ6m6KWOu5EyUduCzjiYiIiIgYzFIt7dk+B0y0/YrmQUnbAX8CLrL9v43jLwduAq60vd3yClLS/pRqZ52bcT4ITASOs/2v5RVH0y1TZzP+iAn9cetYQpNT5S4iIiJWEBm5ebZxwK3dtD0IvE7S8xvH3g/0SzIBXGt7KDAc2J6S6EyStEU/xRMRERER0a+S3FSSrgDeDJwgaZ6kcyQd0zjlCeBi4H31/CHAe4GzW/p5naQbJM2u/31do+2Fkq6s094up2VTTkmvlXRN3Yzz5jpitEi2F9i+y/ahlEpoR/WmP0n7S7q7xnKPpL0bbQdJur223SbplT2+wIiIiIiIfpbkprL9FuAq4KN1ROSJLk47A9ivft+RMsozrbNR0khK2eYfAM8HvgtMaIz2nANMoiQ1X6OM/HReO6ZeewwwEvgM8EtJ6/ThMS4E3thTf5LWrDHubHsY8DrK9DokvYeSIO0HrAW8A3ioq5tJOlhSu6T2BY/M7kOYERERERFLX5KbPrB9DTBS0qaUP/7PaDnlbcC/bZ9p+ynbPwfuAHaVtCGwNfAl24/b/jPwm8a1+wCX2L7E9kLblwPtwC59CHEaJZHpTX8LgS0krW57uu3O6XgHAt+yfYOLO21P6eZ9nGS7zXbbkDWG9yHMiIiIiIilL8lN350JfJQyhe2ilrbRQGsiMAUYU9setj2/pa3TOOA9dQrZLEmzgDcA6/chtjHAzJ76qzHsARwCTJc0QdJL6nVjgbv6cM+IiIiIiAEh1dL67kzgTuAM249IarZNoyQVTRsCvwemA2tLWrOR4GwIdO6i2gGcafugJYjtnZSpdT32Z/tS4FJJq1Omrp1MmdLWAWzU1xtvOWY47am6FRERERH9KCM3fWT7HmBb4AtdNF8CvFjSXpKeJ2kPYDPgt3VqVztwtKRVJL0B2LVx7VmU6Ws7ShoiaTVJ20naYFHx1HNfKOmHwHbA0T31J2ldSe+oa28eB+YBC+p1pwCfkfQqFRtLak3YIiIiIiIGnCQ3i8H21bandXH8IeDtwGGURfifA95ue0Y9ZS/gNZSpY1+hsWbHdgewG3Akpex0B/BZuv832kbSPGAOZY+btYCtbd/Si/5WqjFOq7FsCxxar/sFcCyl+MFcSoW4znU8EREREREDlmz3fFZED9ra2tze3t7fYURERETECk7SJNttXbVl5CYiIiIiIlYISW4iIiIiImKFkOQmIiIiIiJWCCkF/Rwl6SzgTttHLY3+bpk6m/FHTFgaXUU/mJwy3hEREbECGPQjN5LeJ+l6SfMlPVC/H6qWDWgGOkkbSprX+Lg+U+fvN/Z3jBERERERA9mgTm4kHQZ8H/g/YD1gXeAQ4PXAKl2cP2S5BtgFSV2Oltm+1/bQzk89/PLGsatarxkIzxMRERERMVAM2uRG0nDgq8Chti+wPdfF32zvbftxSadJ+n+SLpE0H3izpOGSzpD0oKQpkr4oaaXa51F1ulbnPcbXEZTn1d/7S7pb0lxJ90jau3HuByXdLulhSZc2N76sfXxE0r+Bfy/BM58l6UeSfl+f5411M86bakz3SvpSyzVvknSdpNmSOiTt20W/a0n6s6Tv1Y07316fZa6k+yR9qpt4DpbULql9wSOzF/exIiIiIiKWisG85mYbYFXgVz2ctxewC2VzzVWAk4DhwIuA5wOXAdOBUxfViaQ1gR9QNsr8p6T1qZtbStqdslnmrpTk5Qjg58DrGl3sTtnA89FeP+Gin+d6YGXKKNU+wG3AlsAfJf3N9m8lvRCYABwAXAiMADZoea5RwO+B33auv5H0M2A329dIGgmM7yoQ2ydR3ierrr9JNkyKiIiIiH41aEdugFHADNtPdR6QdI2kWZIelfSmevhXtv9ieyHwJLAH8Pk60jMZ+A7wX6MZ3VgIbCFpddvTbd9aj38IOM727TWerwNbNUdvavtM20ua3Fxk+1rbC20/bvsK2/+ov28GzgW2refuA/ze9vm2n7I9w/ZNjb7GAFcCZ7cUFngS2EzSsBrzjUsYc0RERETEMjeYR24eAkZJel5ngmP7dQCS7uOZxK2jcc0oyujNlMaxKZQ/8hfJ9nxJewCfAU6V9BfgMNt3AOOA70v6TuMS1X4779XB0vGsfiRtAxwHbE55tlUpo0YAY4G7FtHXO4DZwMktx98JfAH4P0k3A4fbvn5RQW05ZjjtqbgVEREREf1oMI/cXAs8DuzWw3nN6VIzKKMSzRGVDYGp9ft8YI1G23rP6si+1PYOwPrAHTyTFHQAH7I9ovFZ3fY13cSxJFr7ORf4JTDW9nDgFEpi1RnXRovo60TgT8AESU8/t+3rbb8DeAHw23qPiIiIiIgBbdAmN7ZnAUcDP5b0bklDJa0kaStgzW6uWQCcDxwraVidNvZpoLOIwE3Am2pZ5uHA5zuvlbRuXby/JiWpmgcsqM0nAp+XtHk9d7ik9yz1h+7aMGCm7cckvRZ4X6PtLGAnSe+S9DxJoyS9vNFuSnW5u4FfS1pN0uqS9pK0lu0ngbk885wREREREQPWoE1uAGx/i5KcfA54ALgf+AlwOHBNN5d9jDJCczdwNXAO8NPa3+XAecDfgUmUUYtOKwGHAdOAmZR1LYfW6y4CvgmcK2kO8A9g56X0mD35MHCcpLmUogbndzbYvodS5ODwGvONlKIDNM4xpeDAA8BFlKlt7wem1Gc5gN6vSYqIiIiI6Dcqf9tGLJm2tja3t7f3dxgRERERsYKTNMl2W1dtg3rkJiIiIiIiotOAr5YmaVPKgvaNKWtpvmL7a/0bVe9ImgicZfuUxrE3Ar/r6nzbQ5dTaEvdLVNnM/6ICf0dRgxSk1NpLyIiIpaCwTBy8zlgou1htldqJjaSjpR0j6R5ku6TdF4/xtkrtq+yPbSrT1/6kTRR0mOS5kqaI2mSpCMkrbqsYo+IiIiIGMgGQ3IzDri19aCk91MWum9fE4M24I/LObZlQlJvR9Q+ansYpTT1YZRKaZdI0qIvi4iIiIhY8Qzo5EbSFcCbgRPq6Mw5ko6pzVsDl9q+C8D2f2yf1Lh2uKRTJU2XNFXSMZKGNNoPknR7Hfm4TdIr6/GX1lGRWZJulfSOxjWnSfqRpAn1uuslbdRo30HSHZJmSzqBZ/abQdJGkq6Q9JCkGZLOljSi0T5Z0uGS/g7Ml/RZSb9seR8/lHR863uyPd/2RMqmnNsAb6vnr1RHc+6q9z1f0sjatpqks+rxWZJukLRubRsp6WeSpkl6WNLFffuXi4iIiIhY/gZ0cmP7LcBVlBGKocATjebrgP1qEtDWTFyq04GnKGt1XgH8D3AgQN2D5ihgP2AtSlLwkKSVgd8Al1E2sPwYcHZd99NpT8r+OmsDdwLH1j5HUTbT/CIwCrgLeH3jOgHHAaOBlwJjawxNe1ISkxE8s0fNiNr/84A9gDMX8b7uBdqBN9ZDHwd2p5StHg08DPyotr0fGF7jeD5lv5tHa9uZlM1MN6/v4Xtd3U/SwZLaJbUveGR2d2FFRERERCwXAzq5WRTbZ1GSjx2BK4EHJB0BZcNNyj4zn6yjGg9Q/kDv3ODyQOBbtm9wcaftKcBrgaHAN2w/YfsKyl43ezZufaHtv9p+Cjgb2Koe3wW4zfYFdfPL44H/NOK90/blth+3/SDwXUrS0fQD2x22H7U9Hfgz0LkZ6E7ADNuTeng104CR9fuHgC/Yvs/245Rk6t01UXqSktRsbHuB7Um250hav767Q2w/bPtJ21d2dSPbJ9lus902ZI3hPYQVEREREbFsDfhqaYti+2zKyMrKlBGKsyX9jTJCsTIwvbH8ZCWgo34fSxlZaTUa6LC9sHFsCjCm8fs/je+PUJKhp69txGZJT/+W9ALgB5RRlWE1nodb7t/R8vt0yiadJwP7sIhRm4YxPLOB6TjgIknN51kArFv7GkvZeLRzpOgL9dhM262xRUREREQMaIM6uelUR0p+IelwYAvgHOBxYFQdYWnVAWzUxfFpwFhJKzUSnA2Bf/UijOmUxACAuqh/bKP9OMDAy2w/JGl34ITWR2n5fTHw/yRtAbydUjmuW5LGAq8CvlkPdQAftP2Xbi45Gjha0njgEuCf9b8jJY2wPWtR92vacsxw2lPONyIiIiL60aCdliZpf0lvkzSsLpzfmbJG5Po6pesy4DuS1qrtG0nqnAZ2CvAZSa9SsbGkccD1wHzgc5JWlrQdsCtln52eTAA2l/S/ddrXx4H1Gu3DgHnALEljgM/21KHtx4ALKMnaX+uamq7exRr12X4F/JWSoACcCBxbnw1J60jarX5/s6Qt61qlOZRpagvqu/sd8GNJa9f38KZePH9ERERERL8atMkN5Q/yI4F7gVnAt4AP2766tu8HrALcRpn+dQGlZDK2f0EpBHAOMJcyQjLS9hOU4gI7AzOAHwP72b6jp2Bsz6Csj/kG8BCwCdAcMTkaeCUwm5IIXdjL5zwd2JKup6SdIGkucD9ljc8vgZ0ao07fB34NXFbPuw54TW1bj/JO5gC3U9YtnVXb9qUkO3cADwCf7GWsERERERH9RnbrTKgYSCRtSEky1rM9p7/j6U5bW5vb29v7O4yIiIiIWMFJmmS7rau2wTxys8KTtBLwaeDcgZzYREREREQMBM/p5EbSUZLO6vnMxe7fkjau30+U9KU+XLsmZcrYDsBXFvP+p+mZTU8jIiIiIlZog65amqTJwIG2/9A4tn899ob+iqsntg/p4/nzeabMdLfqs5/KMxtwApxm+6N9CnAJ3TJ1NuOPmLA8bxnRrcmp3BcREfGcNOiSm+VF0vO6KSM9EF07kBO7iIiIiIjlYYWalibps5J+2XLsh5KOr99fKOlKSXMlXQ6Mapw3vk4jO0DSvcAV9fgvJP1H0mxJf5a0eeOaiZIObPzeX9LVdKF1ipik3STdJGmOpLsk7dTo4+4a4z2S9l46bwckHSTpTkkzJf1a0uh6/GhJP6zfV5Y0X9K36u/VJT0mae2lFUdERERExLKwQiU3lFLGO0kaAWX0BdiDZ8oonwNMoiQ1XwPe30Uf2wIvBXasv39HKev8AuBG4OwlDVLSq4EzKHvdjADeBEyu62x+AOxsexjwOuCmJb1fvedbKBuJvpdSEnsKz+zfcyWwXf2+NfAfynsA2Ab4p+2Hu+jzYEntktoXPDJ7aYQZEREREbHYBuu0tIslNaeMrQLcaHu6pD9T9ps5GdgJmGF7Ui2pvDWwve3HgT9L+k0XfR9V17sAYPunnd8lHQU8LGm47SX5a/4A4Ke2L6+/p9b+1wQWAltIurduqDm9F/29VtKsxu+dbF/Xcs7e9Z431nt9vj7LeOBaYBNJz6ckWqcCh0oaSklyruzqprZPAk4CWHX9TVJTPCIiIiL61WAdudnd9ojOD3Boo+10YJ/6fR+eGbUZDTzcTFwooxetOjq/SBoi6Rt12tgcYHJtGtXFdX0xFrir9WCNbQ/gEGC6pAmSXtKL/q5rvo8uEhsoz//089qeR9lsdIztR4F2SiLzJkoycw3wehaR3EREREREDCSDNblZlIuBl0naAng7z0wjmw6sXUdHOm3YxfXNEYi9gN2A7YHhwPh6XPW/84E1Guev18sYO4CNumqwfantHShTx+6gjEAtDdOAcZ0/6nt4PnXUiJLAvAV4BXBD/b0j8Grgz0sphoiIiIiIZWawTkvrlu3HJF1AWV/zV9v31uNTJLUDR0s6kvJH+67ArxfR3TDgccoIxxrA11vabwL+V9IplJGRA4D7exHmqcBlkn4L/ImSyAwDHgZeA/yRUtp5HrCgF/31xjnAuZLOAW6nPMv1tifX9iuBC4AbbD8haSJljc49th/sqfMtxwynPeV3IyIiIqIfrYgjN1Cmpm3JM1PSOu1FSR5mUjbGPKOHfs6gTOWaCtwGtE73+h7wBCWhOZ1eFhuw/VfgA/X62ZTEYhzl3+MwyijLTMqUsEO76aZPbP8R+BLwS8oo1kbA+xqnXAOszjOjNLcBj5FRm4iIiIgYJGSveOvAa/GAO4D1bM/p73ieC9ra2tze3t7fYURERETECk7SJNttXbWtcCM3klYCPg2cm8QmIiIiIuK5Y4VKbuoi+TnADpRpZysESSdKmtfF58T+ji0iIiIiYqBYoZIb2/NtD7W9ue2Onq9YeiRZ0sZLoZ+Jkg5sHrN9SH2u1s8hS3q/XsSznaT7lvV9IiIiIiKW1ApXLW2gk7QdcAXwSEvTDravXf4RLR23TJ3N+CMm9HcYEX02OVX+IiIiVhhJbvrHNNsb9HcQERERERErkhVqWtrSIulwSVMlzZX0T0lvlfRqSddKmiVpuqQTJK3SzfWrSvq2pHsl3V/XzKy+GHFsJOkKSQ9JmiHpbEkjGu2TJX1e0m2SHpb0M0mr1bZRkn5b450p6apabAFJoyX9UtKDku6R9PFGn6tLOq32dxuwdZ9fYEREREREP0hy00LSpsBHga1tDwN2BCZTNtP8FDAK2AZ4K93vQfNN4MXAVsDGwBjgy4sTDmUjzdHAS4GxwFEt5+xdY9yo3vOL9fhhwH3AOsC6wJGAa4LzG+DmGtdbgU9K2rFe95Xa10a13/d3G5x0sKR2Se0LHpm9GI8XEREREbH0JLn5bwuAVYHNJK1se7Ltu2xPsn2d7adsTwZ+Qtlk81kkCTgI+JTtmbbnAl/n2Rtmjq4jKs3Pmq192b7T9uW2H7f9IPDdLu55gu0O2zOBY4E96/EngfWBcbaftH2Vy6ZGWwPr2P6q7Sds3w2c3IjvvcCxNfYO4AfdvSjbJ9lus902ZI3hi3ilERERERHLXtbctLB9p6RPUkZINpd0KWXfnKGU5KINWIPy7iZ10cU6tX1SyXOAMgIzpHFOr9bcSHoBJbl4IzCMkow+3HJasyrcFMooD8D/1We4rMZxku1vAOOoyVXjuiHAVfX76C76jIiIiIgY8JLcdMH2OcA5ktaijNB8k/JH/9+APW3PrQnQu7u4fAbwKLC57alLGMpxgIGX2X5I0u7ACS3njG183xCYVp9hLmVq2mGSNgf+JOkGSuJyj+1Nurnn9NrnrY0+e7TlmOG0p+pURERERPSjTEtrIWlTSW+RtCrwGCVRWUAZOZkDzJP0EuDDXV1veyFlmtf36sgLksY01rT0xTBgHjBL0hjgs12c8xFJG0gaSVlXc16959slbVynyc2pz7AA+CswpxZNWF3SEElbSOosHHA+8HlJa0vaAPjYYsQdEREREbHcJbn5b6sC36CMwPwHeAElafgMsBcwl5K8nLeIPg4H7gSukzQH+AOwaaN9tKR5LZ93ddHP0cArgdnABODCLs45B7gMuLt+jqnHN6n3nQdcC/zY9kTbC4BdKcUO7qnPeQrQuWjmaMpUtHtqv2cu4jkjIiIiIgYMlTXmMRhJmgwcaPsP/R1LW1ub29vb+zuMiIiIiFjBSZpku62rtozcREQtxUEbAAAcLElEQVRERETECiHJTURERERErBCS3AxitscDd0qypB4r30naX9LVyz6yiIiIiIjlL6Wgl7O6TmY0MNr2jMbxm4CXAy+sm4Quj1gMbFL39jkK+AKlQhyUktCXUTb0nN5TX7dMnc34IyYss1gjBrPJKZMeERGxXGTkpn/cA+zZ+UPSlsDq/RfO086zPQwYCbwTWI+yGen6/RtWRERERETPktz0jzOB/Rq/3w+c0flD0nBJZ0h6UNIUSV+UtFJtGyLp25JmSLobeNb/Eq7XnippuqSpko6RNKQvwdl+0vatwB7Ag5TNQCMiIiIiBrQkN/3jOmAtSS+ticcewFmN9h9S9p15EbAtJRH6QG07CHg78AqgDXh3S9+nA08BG9dz/gc4cHGCrHvi/Ap4Y1ftkg6W1C6pfcEjsxfnFhERERERS02Sm/7TOXqzA3AHMLUe70x2Pm97bl1/8x1g39r+XuB42x22ZwLHdXYoaV1gZ+CTtufbfgD4HvC+JYhzGmWa2n+xfZLtNtttQ9YY3tUpERERERHLTQoK9J8zgT8DL6QxJQ0YBawCTGkcmwKMqd9HAx0tbZ3GASsD0yV1Hlup5fy+GgPMXILrIyIiIiKWiyQ3/cT2FEn3ALsABzSaZgBPUhKV2+qxDXlmZGc6MLZx/oaN7x3A48Ao208taYx1nc+uwB96OnfLMcNpT0WoiIiIiOhHmZbWvw4A3mJ7fuPYAuB84FhJwySNAz7NM2tyzgc+LmkDSWsDR3ReWEs2XwZ8R9JaklaStJGkbfsSlKSVJb0U+DmlYtp3F/cBIyIiIiKWlyQ3/cj2Xbbbu2j6GDAfuBu4GjgH+GltOxm4FLgZuBG4sOXa/SjT2m4DHgYuAHpbynkPSfOAWcCvgYeAV9me1ttnioiIiIjoL7Ld3zHECqCtrc3t7V3laRERERERS4+kSbbbumrLyE1ERERERKwQlktyI2lTSX+TNFfSQklfWk73PU3SMcvjXl3c25I27o97R0REREQ8Fy2vammfAybafkXzoKTtgCuARwBT9lT5hu2fLae4eiRpPHAPsHKzApmk9YFjKNXOhlKqmZ0HfKulQMCyiGl/4FTg0XroQWAicJztfy3Le3fnlqmzGX/EhP64dUQsZZNT+TAiIgap5TUtbRxwazdt02wPBdYCDgdOlrRZ60mSBkzZakkjgWuB1YFtbA+jbMY5AthoOYVxbX1vw4HtKYnOJElbLKf7R0REREQMKMs8uZF0BfBm4ARJ8ySd09VUMRcXUyp8bSZpfJ3adYCkeykjPEh6h6RbJc2SNLGWLO681ysk3Vinv50HrNZo21/S1S2xPT11TNLqkr4jaYqk2ZKulrQ6ZaNNgFk1/m0opZnnAvvYnlzj77D9Cdt/7+IdvK1Oy5sjqUPSUY221SSdJemh+kw3SFq3EfPd9XnukbR3F+9tQa26dihwJdDs+7WSrqn93lxHyuipb0kHSbq9tt0m6ZWt942IiIiIGGiWeXJj+y3AVcBH60jDE12dV/dkeSdl9OOWRtO2wEuBHSW9mLL3yieBdYBLgN9IWkXSKsDFwJnASOAXwLv6EOq3gVcBr6vXfw5YCLypto+wPdT2tZSRkgttL+xl3/MpJZpHAG8DPixp99r2fsroy1jg+cAhwKOS1gR+AOxcR4ZeB9zUw30uBN4IIGkMMIEydW4k8Bngl5LWWVTfkt5DSZD2o4ymvYNSEvq/SDpYUruk9gWPzO7lq4iIiIiIWDYGQrW00ZJmATOArwD72v5no/0o2/NtPwrsAUywfbntJykJyeqUP85fC6wMHG/7SdsXADf0JgBJKwEfBD5he2odDbnG9uPdXPJ8YHpvH9D2RNu32F5YR3Z+TknaAJ6s/W1c7zvJ9pzathDYQtLqtqfb7m5qX6dplEQGYB/gEtuX1PteDrRT1ggtqu8DKeuGbqijaXfantLNc51ku81225A1hvf2dURERERELBMDIbmZZnuE7ZG2t7J9bkt7R+P7aODpP7TryEkHMKa2TfWzN+7p8o/yLoyiTGG7q5fnP0TvN8ZE0msk/UnSg5JmU0ZnRtXmMymbcp4raZqkb0lauRYl2KOeO13SBEkv6eFWY4CZ9fs44D11StqsmkC+AVi/h77H0vv3EBERERExYAyE5KYnzWRlGuWPdgAkifLH+FTKSMqYeqzTho3v84E1Gteu12ibATxG18UAutrl9A/AO+uIT2+cA/waGGt7OHAiIIA6ynS07c0oI1Bvp0wJw/altnegJFJ3ACf3cJ93UqYAQkn6zqyJY+dnTdvf6KHvDpZfUYSIiIiIiKVmwFQg66XzgSMkvZWy0P8TwOPANbX9KeDjkn5EWSvyauBPte1mYHNJW1H+mD+qs1PbCyX9FPiupH2B++u1N1LKLC8EXgR0lln+LmXa1+mSvmh7Sl3jchhwWhdFBYYBM20/JunVwF7AZQCS3kxJrm4D5lCmqS2oRQVeA/yRUgltHrCg9YVIGkJJ4j4NbAdsU5vOAm6QtCMlGVuZMnXvznqP7vo+pb6Hq+vzbwQ82d3UtE5bjhlOe8rHRkREREQ/GgwjN0+ra3H2AX5ISQh2BXa1/YTtJ4D/BfanVFzbg7LAvvPafwFfpfyh/2/gWZXTKAvub6Gs05kJfBNYyfYjwLHAX+r0rtfankkZZXkSuF7SXEqiMJuSPLQ6FPhqPe/LlCSt03rABZTE5nZKxbOzKP82h1FGq2ZS1ugc2rhuG0nz6nUTKYv/t7Z9S33eDmA34EhKgtYBfLb2223ftn9Rn/ccSkW4i3lmHU9ERERExIClZy9RiVg8bW1tbm9v7+8wIiIiImIFJ2mS7bau2gbVyE1ERERERER3ktxERERERMQKIclNL0iypI37O46IiIiIiOjeYKuWhqS9KJXBXkJZ8H4TcKzt1gIBA56kI4GDgHWAWcBfbO+xBP1tB5xle4OlE2Hv3TJ1NuOPmLC8bxsREc8hk1OVMyJ6MKhGbiR9Gjge+DqwLqUE8o8pVcEGHEndJo+S3g/sC2xveyjQRqm4FhERERERi2HQJDeShlNKOX/E9oW259cNMH9j+7OSVpV0vKRp9XO8pFXrtfvXfVua/T091UzSaZJOlHS5pLmSrpQ07r+jgHqfb0u6V9L99brVa9t2ku6TdLik/wA/W8QjbQ1cavsuANv/sX1S7ec9kia13PcwSRfX77tIuq3GOlXSZyStCfwOGC1pXv2MlrSSpCMk3SXpIUnnSxpZ+xlf38MHJHVIeljSIZK2lvT3Wvr6hD7+U0VERERE9ItBk9xQNqdcDbiom/YvUDap3Ap4OWUTzi/2of+9ga8BoyhT3c7u5rxvAi+u99kYGEPZu6bTepR9YcYBBy/iftcB+0n6rKS2uhlnp18DL5T00saxfYAz6/dTgQ/ZHgZsAVxhez6wMzDN9tD6mQZ8HNidspfNaMoeQD9qieU1wCaUvYGOp7zL7YHNgfdK2rarB5B0sKR2Se0LHpm9iEeNiIiIiFj2BlNy83xghu2numnfG/iq7QdsPwgcTZn21VsTbP/Z9uOUP+63kTS2eYIkUdbIfMr2TNtzKVPk3tc4bSHwFduP2360u5vZPgv4GLAjZePOByQdUdseB86jJDRI2hwYD/y2Xv4ksJmktWw/bPvGRTzXh4Av2L6v9nsU8O6WKXNfs/2Y7cuA+cDP63ucClwFvKKbZzjJdpvttiFrDF9ECBERERERy95gSm4eAkYtYh3LaGBK4/eUeqy3Ojq/2J4HzOzi+nWANYBJdcrWLOD39XinB20/1psb2j7b9vbACOAQ4KuSdqzNpwN71YRqX+D8mpwAvAvYBZhSp9Bts4jbjAMuasR7O7CAsmap0/2N74928Xtob54nIiIiIqI/DaZqadcCj1GmWF3QRfs0yh/y/7+9Ow+WsrrTOP59RFARxBUNKBKjxgQ1LjdmahSDSyYuFTWjY01w3GJEraCOOo4JSQo1LgmV0onbpNRRRnGJ+xITNXHf9Rq34ChRRBFXUFkNRvjNH+c0vLTdlwu327735flUvUX3Wd7ld89t+txz3vNOzO+H5DRIoxF9KwUlbVCj/kaF/H6kqWVvVZWZTvqyPyyPatQSHV5FrQoRfweul3QKaZrZXRHxuKRPgOHAyLxVyj8F7CupNzAauC6ff61jTwW+HxGPVGdIGrqs51rPVoMH0O5VbMzMzMyshXrMyE1EzCTd23KhpP0k9ZXUW9KeksYB1wA/lbSepHVz2Qm5+nPAMEnbSFqVNDWr2l6SdpLUh3TvzRMRMbVYICIWApcA50oaCCBpcGG0pdPyIgd7S+qfb/rfk3SPyxOFYlcAFwCfVpa6ltRH0kGSBuRO0SzSSAykEZd18uILFb8BzqwskJDj0y1XlzMzMzMz64oe07kBiIhzSM+4+SnwPmlUYjRwC3AG0A48D7wA/DmnERGTSCut/Qn4K1DrmThXA2NJ09G2J93DU8spwCvA45Jm5X1+eTkuZxYwBniD9IybccAxVc/ruZI0knNlVd2DgSn5+EeT782JiJdInbzJeRraIODXpAUK7pY0m7SQwTeW43zNzMzMzLo1RSzzLKrSkTQeeDMilmV1tabLS0y/B2wXEX9t9fl0pK2tLdrb21t9GmZmZmZWcpKejoi2Wnk9auRmBXQM8FR379iYmZmZmXUHPWlBgR5H0hjS1LNqD0XEnkupOwUQaQEFMzMzMzNbCndugIg4rEn7PYv0HJxFJI1g8UIHHdUd2shzyR2tTSLiB43cr5mZmZlZd+HOTR155GR90kpkc4HfA8fmZ+C0nKT7gQkRcWkhbV/Sw0s3AT4hrRJ3RERMyR2tpnlh2kyG/uiOZh7CzMys5ab4sQdm3ZrvuenYdyKiH7Ad8HXSKm2LKOkWMZS0KWnp6JOAAcAXgYuAha08LzMzMzOzz0u3+GLe3eUHdv4B2FLS/ZLOlPQIMA/YRNIgSbdJ+kDSK5KOrNSVtJqk8ZI+lPQiqZNEIT9yx6TyfrykMwrv95X0rKRZkl6VtIekM0kP97xA0hxJFwDbAK9FxD2RzI6IGyPijbyfUyVNyK8r9Srbp5JOzXmDJN0o6X1Jr0k6rjlRNTMzMzNrLHduOkHSRsBewDM56WBgFNAfeJ30bJk3gUHAAcBZknbLZccCX8rbt4FDl+G4O5BGY04G1gR2BqZExE+Ah4DREdEvIkaTnuuzhaRzJe0iqV+9/UZEpV4/YCfgQ+DWPAp1O2k622BgN+Df6z2kVNIoSe2S2hfMm9nZyzIzMzMzawp3bjp2i6SPSA/9fIDFiwOMj4iJEfEpsAGpg3BKRPwtIp4FLiV1gAAOBM6MiA8iYipw3jIc/wjgsoj4Y0QsjIhp+UGdnxERk4ERpE7JdcD0PApUt5MjaT3SA1CPjYhnSKNK60XE6RHxSd7nJcC/1jnmxRHRFhFtvfoOWIbLMjMzMzNrPC8o0LH9IuJPxQRJAFMLSYOADyJidiHtdaCtkD+1Kq+zNiItZNApEfE4qTOFpK8DvwV+Avy4uqyk3sANwNURcW1O3hgYlDt0Fb1Io0RmZmZmZt2aOzfLJwqv3wLWltS/0MEZAkzLr98mdVImFvKK5gF9C+83IE1xg9Qp+lInzuGzmRFPSboJ2LJOkfOB2Sy5SMJU0n07m3W071q2GjyAdq8gY2ZmZmYt5GlpXZSnmj0KnC1pVUlbk6aTXZWLXAf8WNJakjYEjq3axbPASEm9JO0BfLOQ9z/A4ZJ2k7SSpMGStsh575KWfAZA0k6SjpQ0ML/fAtgHeLz6nCUdlY8zMiKKq6k9CcySdEpeCKGXpC3zKJCZmZmZWbfmzk1jfA8YShrFuRkYGxF/zHmnkaaivQbcDVxZVfd44DvAR8BBpHtgAIiIJ4HDgXOBmaT7fjbO2b8GDsirsJ2X6+8DvCBpDnBnPpdxdc53E+CtwoppYyJiQT6XbfL5TifdP+QbaszMzMys21NEh7ObzDqlra0t2tvbW30aZmZmZlZykp6OiLZaeR65MTMzMzOzUnDnZgUnaULlAZ5mZmZmZj2ZV0vrAklXAfMj4vuFtG8CNwFbRsTbDTzWBNI9OXtHxO8L6RcAPwQOjogJjTresnph2kyG/uiOVh3ezMzMzD4nU7rxCrkeuema44C9JH0LQNKqpIdentTgjk2v/HIScGghvTewPzC5UccyMzMzM+up3LnpgoiYQVra+WJJqwNjgVcjYnxeunmMpFclTZd0raS1AHLeDZLekfSRpPslfaWy3zxV7EJJd0qaCwzPWbcAIyRVVi/bG2gH3i/U3UzSfZJm5ONeWSiPpO0lPStptqRrgFWK1yRpH0nP5fN6WFK95+SYmZmZmXUr7tx0UURcDzwNXAOMAo7KWSeSOh87AxsCc4HzClV/B2xGemjnX/jsEtEjSctI9wcey2kfA3cAB+b3hwBXVNUTcAbwBeCrpCWffwYgaRXgVuAyYO38er9FFdPzbC4BfgCsk8vdKqlPrWuXNEpSu6T2BfNm1ipiZmZmZva5ceemMX4I7AqcHhFv5LSjgDERMS0i/gacChwoaaWIWBgR4yNidiFv+zz6U3FzRDyWy84vpF8BHCJpbeAfgduKJxIRkyLinoj4JCLeIz0jp/Jg0B2BAM6PiL9HxLXAM4Xqo4CLIuKpiFgQEZfl9JoP8YyIiyOiLSLaevX1o3DMzMzMrLW8oEADRMS7kqYDEwvJQ4DbJS0sFgUGSnofOBs4AFgXqJRZlzTCAzC1zuEeII0EjQFujYj5khZlStqANEK0I2nUZyUWT1sbBLwZSz7c6PXC642BgySdUEjrAwyucy5mZmZmZt2GR26a503gWxGxZmFbNSLeIU0n24s02jMA2DTXUaF+zaer5o7JVaRpb9VT0gB+CcwHtoqINYDDCvt9m9QxKhpSeD0VOK3qnPtGxHWdu2QzMzMzs9bxyE3z/AY4S9LhEfGGpIHAP0TEbaQRlfnADKAvcOYy7vtc4L6IeKRGXn/gPWCmpI2A/yjkPQysJGl0Pr/vAtsBf8j5FwPXS7qXtFDB6sAuwL0RMZcObDV4AO3deFlAMzMzMys/j9w0zznAncA9kmYDj7L43pXLgbfyNjHndVpEzIiIe+pkjwV2AGaS7se5sVBvPqlDcyTwIfDPpBXYKvlPAMcA/53zJwH/tiznZmZmZmbWKlry9guz5dPW1hbt7e2tPg0zMzMzKzlJT0dEW808d26sEfLo1MutPo+SWheY3uqTKDHHt3kc2+ZxbJvHsW0ex7Z5VrTYbhwR69XK8D031igv1+tBW9dIandsm8fxbR7Htnkc2+ZxbJvHsW0ex3Yx33NjZmZmZmal4M6NmZmZmZmVgjs31igXt/oESsyxbS7Ht3kc2+ZxbJvHsW0ex7Z5HNvMCwqYmZmZmVkpeOTGzMzMzMxKwZ0bMzMzMzMrBXduzMzMzMysFNy5sUUkrS3pZklzJb0uaWSdcpL0S0kz8jZOkgr520h6WtK8/O82na1bVo2IraTNJd0q6X1JH0i6S9KXC3UPk7RA0pzCNuJzusSWaWC7jbyPSuwu7WzdMmtQ2x1e1S7n5Hjvn/PddjuO7S6S7pM0U9KUGvlDc/48SS9J2r0q/wRJ7+T6l0lapUmX1G00IraSBkq6RtJbOf8RSd8o5I+QtLCq3R7a5EtruQa22ymSPi7E7u6qfLfb5Wu3Q+p83p6U88vfbiPCmzciAuAa4LdAP2AnYCYwrEa5o4CXgQ2BwcCLwNE5rw/wOnACsApwXH7fZ2l1y7w1KLY7AEcAawO9gZ8DLxXqHgY83Opr7YmxzfkBbFrnGCtku21kfKvKjgBmA6vn9267Hcd2B+BgYBQwpUb+Y8A5wGrA/sBHwHo579vAu8AwYC3gfuAXrb72nhBbYBPgROALQK9cZjrQL+ePAN5s9bX2xNjm/CnA7nWO4XbbhdhWlf0isAAYmt+Xvt22/AS8dY8NWB34BNi8kHZlrQ8T4FFgVOH9EcDj+fU/AdPIK/HltDeAPZZWt6xbo2Jbo+zapC/k6+T3h7GCfUFsZGzpuHOzwrXbRse3quzlwOWF9267HcS2kL979RcZYHNgPtC/kPYQi/8ocjVwViFvN+CdVl9/T4htnXKzgO3z6xGU/EtiM2NLx50bt9suxLaqzFjgvsL70rdbT0uzis2BBRExqZD2HOmvJtWG5bxa5YYBz0f+Dcqer8qvV7esGhXbajuTPuxnFNK2lTRd0iRJP5O0cldOvAdodGwfzNMgbpI0dBnrllHD266kvsABwP9WZbntLl+7GgZMjojZdfZT6+eyvqR1lvE4PUmjYrsEpSnWfYBXCskDJb0r6TVJ50pavSvH6AEaHdurlKZa3y3pa4V0t9ukEf/XHMJnP29L3W7dubGKfqThz6KZQP9OlJ0J9JOkTuyno7pl1ajYLiJpQ+BC0pSJigeBLYGBpKkp3wNO7tKZd3+NjO03gaHAFsBbwO8KX7BXxHYLTWi7pLY5HXigkOa2m9SLbVf2U+vnwnIcpydpVGwXkbQG6a/op0VEZd8vAduQpq3tCmxPmh5YZo2M7UGkz9yNgfuAuyStWec4brfLQdJwYH3ghkJy6dutOzdWMQdYoyptDdK8+KWVXQOYk0drlrafjuqWVaNiC4Ck9YC7gYsi4ppKekRMjojXImJhRLwAnE76C3mZNSy2EfFgRHwSER8Bx5PmKX+lM3VLrKFtNzsUuKKY7ra7SL3YdmU/tX4uLMdxepJGxRYASasBt5OmWZ5dSY+IdyLixdxuXwP+E7fbTouIRyLi44iYl+P6ETC8znHcbpfPocCNETGnkrAitFt3bqxiErCypM0KaV8DJtYoOzHn1So3Edi66q+1W1fl16tbVo2KLZLWInVsbouIM5dy3ADKPrLQsNjWUIzfithuocHxlbQRab73FUs5rttu500ENpFU/Otu9Wdy9c/l3arprGXTqNiSV+i6hXQv6VFLKe522zVL+8x1u10GuVP+L3x2Slq18rXbVt/04637bMC1pJU6Vgd2pP4qHUcD/0daEWkQ6ReverW040mrpY1mydXS6tYt89ag2K4BPAlcUOcYewLr59dbAH8Bxrb62ntIbIeRhul7kaYG/Bdp5a/eS6tb9q0R8S2UGQM8WKOu227HsV0JWDXH6fX8uk8h/3HgVzn9uyy5WtoewDvAV0mrTt3LirHqVJdjS1qV8nZS52blGnVHAENIXww3Ik2turzV195DYjsk1+2T008G3mfxAjlut134TMhlRuY8VaWXvt22/AS8dZ+NtPrWLcBc0gpnI3P6cNL0kko5AeOAD/I2jiVXR9sWeBr4GPgzsG1n65Z1a0RsScPLkfcxp7ANyfm/Ii2dOReYTJra07vV195DYrsrqTMzF3gv72+zFb3dNiq+hTIvAUfUOIbbbsexHZF/94vb/YX8oaSlcj/O7Xj3quOcmOM7i7RS3SqtvvaeEFvSfXgBzKv6zB1eiOu0nD8VOJ/CqnVl3RoU22GkxYbmAjOAe4A2t9vGfCbkMncBP69xjNK328p/7GZmZmZmZj2a77kxMzMzM7NScOfGzMzMzMxKwZ0bMzMzMzMrBXduzMzMzMysFNy5MTMzMzOzUnDnxszMzMzMSsGdGzMzMzMzKwV3bszMzMzMrBT+H4tSZ8IJvrrgAAAAAElFTkSuQmCC\n",
"text/plain": [
"| \n", " | cols | \n", "imp | \n", "
|---|---|---|
| 69 | \n", "YearMade | \n", "0.182890 | \n", "
| 6 | \n", "ProductSize | \n", "0.127268 | \n", "
| 30 | \n", "Coupler_System | \n", "0.117698 | \n", "
| 7 | \n", "fiProductClassDesc | \n", "0.069939 | \n", "
| 66 | \n", "ModelID | \n", "0.057263 | \n", "
| 77 | \n", "saleElapsed | \n", "0.050113 | \n", "
| 32 | \n", "Hydraulics_Flow | \n", "0.047091 | \n", "
| 3 | \n", "fiSecondaryDesc | \n", "0.041225 | \n", "
| 31 | \n", "Grouser_Tracks | \n", "0.031988 | \n", "
| 1 | \n", "fiModelDesc | \n", "0.031838 | \n", "
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"def plot_fi(fi):\n",
" return fi.plot('cols', 'imp', 'barh', figsize=(12,7), legend=False)\n",
"\n",
"plot_fi(fi[:30]);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The way these importances are calculated is quite simple yet elegant. The feature importance algorithm loops through each tree, and then recursively explores each branch. At each branch, it looks to see what feature was used for that split, and how much the model improves as a result of that split. The improvement (weighted by the number of rows in that group) is added to the importance score for that feature. This is added across all branches of all trees, and finally the scores are normalized such that they add to 1.0."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Removing low-importance variables"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It seems likely that we could use just a subset of the columns and still get good results. Let's try just keeping those with a feature importance greater than 0.005."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"21"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"to_keep = fi[fi.imp>0.005].cols\n",
"len(to_keep)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll retrain our model, using just this subset of the columns."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"xs_imp = xs[to_keep]\n",
"valid_xs_imp = valid_xs[to_keep]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"m = rf(xs_imp, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"...and here's the result:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(0.181208, 0.232323)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m_rmse(m, xs_imp, y), m_rmse(m, valid_xs_imp, valid_y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our accuracy is about the same, but we have far less columns to study:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(78, 21)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(xs.columns), len(xs_imp.columns)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We've found that generally the first step to improving a model is simplifying it. 78 columns are too many for us to study them all in depth! Furthermore, in practice often a simpler, more interpretable model is easier to roll out and maintain.\n",
"\n",
"It also makes our feature importance plot easier to interpret. Let's look at it again:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAzYAAAGeCAYAAABGn5TrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeZRdVZ3+//dDRCAkJMQgQxiChElAbbtQRPwRFWWQqbtFZtQWaEAcUaHVVmjBadk4odKgzZAwKohisDXINyAiQoVGaSYbJFAkAYEMkIEpeX5/nF16uFalbiVVdesmz2utu3LOns7nnOSPfNbeZx/ZJiIiIiIiop2t1eoAIiIiIiIiVlUSm4iIiIiIaHtJbCIiIiIiou0lsYmIiIiIiLaXxCYiIiIiItrey1odQKwexo8f74kTJ7Y6jIiIiIhYzc2cOfNJ2xs1liexiQExceJEOjs7Wx1GRERERKzmJD3cU3mWokVERERERNtLYhMREREREW0viU1ERERERLS9vGPTC0nnArNtf6HVsbSDu2YvZOJp04bkWrO+/K4huU5EREREtI81YsZG0ixJSyU9I2mBpFsknSCp1/u3fcJgJjWSdpL0S0nzS0wzJe23imPOkHTsQMUYEREREdEu1ojEpjjA9mhgK+DLwKnAD3pqKGnEEMRzLTAd2Bh4JfBh4OkhuG5ERERExGpnTUpsALC90PZPgUOB90raWdKFkr4n6TpJi4G3lrIzASTdK2n/7jEkvUzSk5JeX853K7NACyT9XtLkFcUgaTywNXC+7efL7ze2by71/yvpgFr7tcv1XidpXUlTJT1Vrne7pI0lnQW8BThH0iJJ55S+O0iaLmmepPslvac27oWSvivp56XPbyRtIukbZSbpPkl/NzBPPiIiIiJi8KxxiU0327cBj1IlAwBHAGcBo4GbG5pfBhxeO98beNL2HZImANOAM4FxwCeAqyT9zUeDap4CHgCmSjpY0sYN9RcDR9XO9wPm2r4TeC8wBtgCeAVwArDU9meAXwMn2x5l+2RJ61PNCl1KNSt0OPBdSTvVxn4P8FlgPPAc8FvgjnL+I+Ds3m5C0vGSOiV1LluycAW3GxERERExuNbYxKaYQ5WMAPykzJost/1sQ7tLgQMljSznR5QyqBKQ62xfV/pOBzqpkpEe2TbwVmAW8B/AXEk3Sdq2NJkK7Cdpg3J+NDClHL9AldBMsr3M9kzbvS1h2x+YZfsC2y/avgO4Cnh3rc2PyxjPAj8GnrV9se1lwBVArzM2ts+z3WG7Y8TIMb01i4iIiIgYdGt6YjMBmFeOu3prZPsB4F7ggJLcHMhfE5utgEPKsrAFkhYAewCbrujCth+1fbLtbcoYi6lmarA9B/gN8E+SxgL7ApeUrlOAXwCXS5oj6auS1u7lMlsBb2yI7Uhgk1qbx2vHS3s4H7Wi+4iIiIiIGA7W2O2eJe1KldjcDLwRcB9dupejrQXcU5IdqBKiKbaPW9lYbHdJ+k65RreLgGOp/o5+a3t2afsCcAZwhqSJwHXA/VQbITTeQxdwo+13rGxsERERERHtYI2bsZG0QdkI4HJgqu27mux6OfBO4ET+OlsD1bKxAyTtLWlEebl/sqTNVxDDhpLOkDRJ0lplM4F/Bm6tNbsGeD3wEcpMTun7Vkm7lJ3bnqZamrasVD8OvKo2xs+A7SQdXTYgWFvSrpJ2bPKeIyIiIiLawpo0Y3OtpBeB5cA9VC/Fn9tsZ9tzJf0W2JPqhfvu8i5JBwFfpZpxWQbcRpUA9eZ5YCJwPdVL+ouA/wd8qDbuUklXUc0SXV3ru0mJe/PS7wqq5Argm8BFkk6kmkX6sKR3lns9myqR/T3w8Wbvu1m7TBhDZz6cGREREREtouo99hiOJH0O2M72UX02brGOjg53dna2OoyIiIiIWM1Jmmm7o7F8TZqxaSuSxgEfoNoRLSIiIiIiVmCNe8dmKJWPXvb0e0sf/Y6jevH/57ZvGppoIyIiIiLaV2ZsBpHtldoq2fb5wPkDHE5ERERExGorMzYREREREdH2kthERERERETbS2ITERERERFtL4lNRERERES0vWwesIokTQam2t681bG00l2zFzLxtGmtDmOlzcrHRSMiIiLa2hozYyNplqSlZbvlxyVdIGmldi0bDJJmSDq2oUySTpb0B0lLJD1W2h3WqjgjIiIiIoajNSaxKQ4oWzC/HtgV+Gy9siQSw+mZfAv4KHAK8ApgAlXM+/TUeBjGHxERERExJNbI/wTbng38HNi5zICcJek3wBLgVZI2k/RTSfMkPVA+mAmApPUkXShpvqR7qBIkavWWNKl2fqGkM2vnB0m6U9LTkh6UtI+ks4C3AOeUGaVzJG0HnAQcZnu67aW2l9m+2fb7auP1N/7GeCZLerR2PkvSv0q6p9zjBZLWXfWnHhERERExeNbId2wkbQHsB1xNlVAcDewL3A8IuB64G9gM2AGYLulPtn8FfB7YpvzWp0qQmr3uG4CLgXcDvwI2BUbb/m9Jb6Z6V+f7pe0JQJftziaG7k/8zTgS2BtYDFxLNUv02cZGko4HjgcYscFGTQ4dERERETHw1rQZm2skLQBuBm4EvljKL7R9t+0XgU2APYBTbT9r+07g+1TJA8B7gLNsz7PdRbVcrFkfAP6rzMAstz3b9n29tB0PPFYvkPSopAWSnpW0Va2qP/E34xzbXbbnAWcBh/fUyPZ5tjtsd4wYOaYfw0dEREREDKw1bcbmYNvX1wskAXTVijYD5tl+plb2MNBRq+9qqGvWFsB1TbZ9impG5y9sby7pZcALVDMz3foTfzMa72+zfvSNiIiIiBhya9qMTW9cO54DjJM0ula2JTC7HM+lSlDqdXVLgJG1801qx11US9j6igHgBmBzSc0kJP2Jf/EK4uvWeH9zmoghIiIiIqJlktg0KMvLbgG+JGldSa+hWkJ2SWlyJfCvkjaUtDnwoYYh7gSOkDRC0j7AnrW6HwDvl/R2SWtJmiBph1L3OPCqWhz3A/8JXC7pHWXTghHA7qsY/53AfpLGSdqEate1Rh+UtLmkccCngStWdM2IiIiIiFZb05aiNetw4FyqmYr5wOdtTy91Z5S6h0r9BcBHan0/AlwEfBC4pvwAsH2bpPcDXwe2pkpmPgjcB3wTuEjSicAU2x8udR8CzgYmAQuAPwKHAo+sZPxTgL2AWeV3AdV20nWXAr+kWoL2E+BM+rDLhDF05iOXEREREdEishtXQMWaTNIs4NjGd5H60tHR4c7OZjZwi4iIiIhYeZJm2v6b1zWyFC0iIiIiItpeEpuIiIiIiGh7eccmXsL2xFbHEBERERHRX5mxiYiIiIiItpfEJiIiIiIi2l4Sm4iIiIiIaHtJbCIiIiIiou2tNpsHSJoMTLW9eatjWRPdNXshE0+b1uowVsmsfGA0IiIiom0N+YyNpFmSlkpaJOlxSRdIGjXUcfRG0gxJxzaUSdLJkv4gaYmkx0q7w1oU42RJy8szXCTpUUlXStq1FfFERERERLRaq5aiHWB7FPB6YFfgs/XKkkgMp2Vy3wI+CpwCvAKYQBXzPj01HqL455RnOBrYDbgP+LWktw/ydSMiIiIihp2WJg+2ZwM/B3YuMyBnSfoNsAR4laTNJP1U0jxJD0g6rruvpPUkXShpvqR7qBIkavWWNKl2fqGkM2vnB0m6U9LTkh6UtI+ks4C3AOeUmZBzJG0HnAQcZnu67aW2l9m+2fb7auP1N/7GeCZLerR2PkvSv0q6p9zjBZLW7eEZ2vajtj8HfB/4Sm2MHSRNL9e/X9J7anX7lbGfkTRb0idW9Gz6+ruMiIiIiGillr5jI2kLYD/gaqqE4mhgX+B+QMD1wN3AZsAOwHRJf7L9K+DzwDbltz5VgtTsdd8AXAy8G/gVsCkw2vZ/S3oz1bs63y9tTwC6bHc2MXR/4m/GkcDewGLgWqpZos+uoP3VwEmS1i/n04HPlZheA/xS0t227wZ+ALzH9q8lbQhsXe63x2fTZLwRERERES3RqhmbayQtAG4GbgS+WMovtH237ReBTYA9gFNtP2v7TqoZiaNL2/cAZ9meZ7uLarlYsz4A/FeZgVlue7bt+3ppOx54rF5Q3mlZIOlZSVvVqvoTfzPOsd1lex5wFnB4H+3nUCVUY4H9gVm2L7D9ou07gKuoEhaAF4BXS9rA9vxSD/14NpKOl9QpqXPZkoX9uK2IiIiIiIHVqsTmYNtjbW9l+yTbS0t5V63NZsA828/Uyh6mer+lu76roa5ZWwAPNtn2KapZi78oO6+NB9ahSiS69Sf+ZjTe32Z9tJ8AGFgAbAW8sSRgC0oieSRVwgXwT1SzZQ9LulHSm0p508/G9nm2O2x3jBg5prk7ioiIiIgYBMPpBX2o/lPebQ4wTlJ9GdSWwOxyPJfqP+H1urolwMja+Sa14y6qJWx9xQBwA7C5pI4VxN1T377iX7yC+Lo13t+cPq7/D8AdthdT3eONJYHs/o2yfSKA7dttHwS8ErgGuLKMsaJnExERERExLA23xOYvyvKyW4AvSVpX0muolkldUppcCfyrpA0lbQ58qGGIO4EjJI0oL7/vWav7AfB+SW+XtJakCZJ2KHWPA6+qxXE/8J/A5ZLeUTYtGAHsvorx3wnsJ2mcpE2odl1r9EFJm0saB3wauKKxQdmBbYKkzwPHlnYAPwO2k3S0pLXLb1dJO0p6uaQjJY2x/QLwNLCsiWcTERERETEsDfcPdB4OnEs1UzEf+Lzt6aXujFL3UKm/APhIre9HgIuAD1LNSFzTXWH7NknvB75O9dL846XdfcA3gYsknQhMsf3hUvch4GxgEtVSrz8ChwKPrGT8U4C9gFnldwHVdtJ1lwK/pFqC9hPgzFrdZpIWUS2FW0iVRE22fWu5x2ckvbPEfDZVEvt74OOl/9FUu7+NoNrs4Kgmnk2vdpkwhs584DIiIiIiWkR248qrGA4kzQKOtX19q2NpRkdHhzs7m9k4LiIiIiJi5UmaaftvXhMZtkvRIiIiIiIimpXEJiIiIiIi2t5wf8dmjWV7YqtjiIiIiIhoF5mxiYiIiIiItpfEJiIiIiIi2l4Sm4iIiIiIaHtJbCIiIiIiou1l84AYEHfNXsjE06a1OowhMSsfIo2IiIgYdjJjExERERERbS+JTYtI+rSk7w/xNd8i6f6hvGZERERExFBIYjNIJC2q/ZZLWlo7P9L2F20fOwjX3UnSLyXNl7RA0kxJ+wHY/rXt7Qf6mhERERERrZZ3bAaJ7VHdx5JmAcfavr6ZvpJeZvvFlbz0tcD3gP3L+a6AVnKsiIiIiIi2kBmbFpF0uqSp5XiiJEv6gKRHgBtK+W6SbikzL7+XNLmPMccDWwPn236+/H5j++ZSP1nSo+X40IZZpeckzSh160j6mqRHJD0u6VxJ6/VwveMldUrqXLZk4QA+nYiIiIiI/kliM7zsCewI7C1pAjANOBMYB3wCuErSRivo/xTwADBV0sGSNu6toe0rbI8qM0ubAX8CLivVXwG2A14HTAImAJ/rYYzzbHfY7hgxckw/bzUiIiIiYuAksRleTre92PZS4CjgOtvX2V5uezrQCezXW2fbBt4KzAL+A5gr6SZJ2/bWR9JawKXADNv/KUnAccDHbM+z/QzwReCwAbrHiIiIiIgBl3dshpeu2vFWwCGSDqiVrQ38vxUNYPtR4GQASVsA5wEXA2/qpctZwGjgw+V8I2AkMLPKcYDqHZ0RTd9FRERERMQQS2IzvLh23AVMsX3cSg9md0n6Dn9dYvYSkg4DDgd2tf1CKX4SWArsZHt2s9faZcIYOvPhyoiIiIhokSxFG76mAgdI2lvSCEnrlpf/N++tg6QNJZ0haZKktcpmAv8M3NpD278Dvg0cbPuJ7nLby4Hzga9LemVpO0HS3gN8fxERERERAyaJzTBluws4CPg08ATVDM4nWfHf2fPAROB64Gngf4HngPf10PYgYEPg5trOaD8vdadSbUJwq6Sny3j5/k1EREREDFuq3jePWDUdHR3u7OxsdRgRERERsZqTNNN2R2N5ZmwiIiIiIqLtZfOANiRpUS9V+9r+9ZAGExERERExDCSxaUPlo5oREREREVFkKVpERERERLS9JDYREREREdH2shQtBsRdsxcy8bRprQ5jyMzKx0gjIiIihpXM2Awzki6UdOYK6hdJetVQxhQRERERMdwlsVkJkmZJel7S+IbyOyVZ0sTBurbtUbb/tKrj1BMoSRNL3N0f6nxc0s8kvWPVI46IiIiIGHxJbFbeQ8Dh3SeSdgHWa104A2Js2XHttcB04MeS3tfakCIiIiIi+pbEZuVNAY6pnb8XuLj7RNK7JP2PpKcldUk6vd5Z0h6SbpG0oNS/r1a9oaRpkp6R9DtJ29T6WdKkcnyhpO+soO0OkqZLmifpfknvaebGbD9m+5vA6cBXJOXfSUREREQMa/kP68q7FdhA0o6SRgCHAlNr9YupEp+xwLuAEyUdDCBpS+DnwLeBjYDXAXfW+h4OnAFsCDwAnLWCOHpsK2l9qlmXS4FXlnbflbRTP+7x6tJ3+54qJR0vqVNS57IlC/sxbERERETEwEpis2q6Z23eAdwHzO6usD3D9l22l9v+A3AZsGepPhK43vZltl+w/ZTtemJzte3bbL8IXEKV+PSmt7b7A7NsX2D7Rdt3AFcB7+7H/c0pf47rqdL2ebY7bHeMGDmmH8NGRERERAysbPe8aqYANwFbU1uGBiDpjcCXgZ2BlwPrAD8s1VsAD65g3Mdqx0uAUSvRdivgjZIW1OpfVmJu1oTy57x+9ImIiIiIGHKZsVkFth+m2kRgP6plW3WXAj8FtrA9BjgXUKnrArZhcHUBN9oeW/uNsn1iP8b4B+DPwP2DE2JERERExMDIjM2q+wCwoe3FkurPczQwz/azkt4AHAH8stRdAny6vMx/NTCGKgGqL0dbVT8DvizpaODyUvY6YJHte1fUUdLGwCHA54GP2F7e18V2mTCGzny0MiIiIiJaJDM2q8j2g7Y7e6g6Cfh3Sc8AnwOurPV5hGqW5xSqZV53Um2xPJBxPQO8EziM6l2Zx4CvUC2J680CSYuBu0p8h9j+r4GMKyIiIiJiMMh2q2OI1UBHR4c7O3vK7yIiIiIiBo6kmbY7GsszYxMREREREW0viU1ERERERLS9JDYREREREdH2kthERERERETbS2ITERERERFtL4lNRERERES0vXygMwbEXbMXMvG0aa0OY1iZlQ+WRkRERAyZYTNjI2l7Sf8j6RlJyyX92yBf73RJU5tsO0PSsYMZz8qQdLekya2OIyIiIiKi1YZNYgN8Cphhe7TttWx/AUDSZEmWdHW9saTXlvIZQxmkpPdJWiZpUfk9JOkCSdsNZRwAtneyPaOZtpJmSdprkEOKiIiIiGiJ4ZTYbAXc3UvdE8Dukl5RK3sv8MdBj6pnv7U9ChgD7AUsBWZK2nkoLi5pSJcQDvX1IiIiIiL6a1gkNpJuAN4KnFNmQS6VdGatyfPANcBhpf0I4D3AJQ3j7C7pdkkLy5+71+q2lnRjWeo2HRjf0Hc3SbdIWiDp980s8bK9zPaDtk8CbgROb2a8MuvzpxLLQ5KOrNUdJ+neUnePpNeX8lmSTpX0B2CxpJfVZ2HK0rofSbqi9L1D0mtL3RRgS+Da8nw/VcoPLMvZFpTldjvW4vib6/X1PCIiIiIiWmVYJDa23wb8Gji5zIQ830Ozi4FjyvHeVLM7c7orJY0DpgHfAl4BnA1Mq83yXArMpEpovkA149Pdd0LpeyYwDvgEcJWkjfpxG1cDb+lrPEnrlxj3tT0a2B24s/Q7hCo5OgbYADgQeKp2jcOBdwFjbb/YQwwHAT8s17wUuEbS2raPBh4BDrA9yvZXy9K5y4CPAhsB11ElPi9v9nqSjpfUKalz2ZKF/XhUEREREREDa1gkNs2wfQswTtL2VP/xv7ihybuA/7M9xfaLti8D7gMOkLQlsCvwb7afs30TcG2t71HAdbavs73c9nSgE9ivHyHOoUoomhlvObCzpPVsz7XdvQTvWOCrtm935QHbD9eu8S3bXbaX9hLDTNs/sv0CVWK3LrBbL20PBabZnl7afw1YjyrRaup6ts+z3WG7Y8TIMb09l4iIiIiIQdc2iU0xBTiZatnajxvqNgMebih7GJhQ6ubbXtxQ120r4JCyJGuBpAXAHsCm/YhtAjCvr/FKDIcCJwBzJU2TtEPptwXw4Aqu0dVHDH+pt70ceJTq3nvykudV2neV+2j2ehERERERw0K7vTcxBXgAuNj2Ekn1ujlUCUXdlsB/A3OBDSWtX0tutgRcjruAKbaPW4XY/oFqOV2f49n+BfALSetRLVc7n2oZWxewzQqu4RXUQZUYASBpLWBz/rpcr7HvHGCXWnuV/rP7cb2IiIiIiGGhrRIb2w9J2hP4Uw/V1wHflnQEcCXwT8CrgZ/ZflJSJ3CGpE8DbwAOAH5a+k4Fbpe0N3A9sDbVEq4HbD/aWzxlE4MtgY8Dk4E39TUe8ALwRuBXVLupLQKWlX7fB86WdDNwB1WS80LDcrQV+XtJ/1ju68PAc8Ctpe5x4FW1tlcCp0l6O3AT8JHS/pYmr/USu0wYQ2c+SBkRERERLdJuS9GwfbPtOT2UPwXsD5xC9cL9p4D9bT9ZmhxBlVDMAz5P7R0d211UL95/mmpr6S7gk/T+fN4kaRHwNDCD6kX/XW3f1cR4a5UY55RY9gROKv1+CJxF9eL/M1Q7wXW/t9OMn1Atc5sPHA38Y3l/BuBLwGfL0rhP2L6f6l2gbwNPUiV6B9juaeOGiIiIiIhhTXZWG60OJJ0OTLJ9VCuu39HR4c7OzlZcOiIiIiLWIJJm2u5oLG+7GZuIiIiIiIhGSWwiIiIiIqLttdXmAdE726e3OoaIiIiIiFbJjE1ERERERLS9JDYREREREdH2kthERERERETbS2ITERERERFtL5sHtNhgf39GkoFtbT8g6Vxgtu0vDPR17pq9kImnTRvoYVdrs778rlaHEBEREbHayIzNSpA0S9JeDWXvk3Rzq2Jqhu0TBiOpiYiIiIhotSQ2w5ikzKhFRERERDQhic0Ak/RJSVc1lH1b0jfK8daSbpT0jKTpwPhau4mSLOkDkh4BbijlP5T0mKSFkm6StFOtzwxJx9bOe505knShpDNr5wdJulPS05IelLRPbYw/lRgfknTkwDydiIiIiIjBkcRm4E0F9pE0Fv4y63IoMKXUXwrMpEpovgC8t4cx9gR2BPYu5z8HtgVeCdwBXLKqQUp6A3Ax8ElgLPD/AbMkrQ98C9jX9mhgd+DOXsY4XlKnpM5lSxauakgRERERESstS51W3jWSXqydvxy4w/ZcSTcBhwDnA/sAT9qeKWlLYFdgL9vPATdJuraHsU+3vbj7xPZ/dR+XzQbmSxpje1WyiQ8A/2V7ejmfXcZfH1gO7CzpEdtzgbk9DWD7POA8gHU23darEEtERERExCrJjM3KO9j22O4fcFKt7iKge5ezo/jrbM1mwPx60gI83MPYXd0HkkZI+nJZKvY0MKtUje+hX39sATzYWFhiOxQ4AZgraZqkHVbxWhERERERgyqJzeC4BniNpJ2B/fnr0rG5wIZlVqTblj30r89+HAEcBOwFjAEmlnKVPxcDI2vtN2kyxi5gm54qbP/C9juATYH7qGaeIiIiIiKGrSQ2g8D2s8CPqN6nuc32I6X8YaATOEPSyyXtARzQx3CjgeeAp6gSmC821N8J/KOkkZImUS0xa8YPgPdLeruktSRNkLSDpI0lHViSr+eARcCyJseMiIiIiGiJvGMzeC4CjgX+uaH8iFI3D/gt1Qv8Y1cwzsVUmwjMLn3+DTixVv91qvd2Hgf+QDU7tBd9sH2bpPeX/luX/h8EFgKnUC2fM1XidFJv43TbZcIYOvPByYiIiIhoEdl553swlI0C7gM2sf10q+MZbB0dHe7s7Gx1GBERERGxmpM003ZHY3mWog0CSWsBHwcuXxOSmoiIiIiIVstStAFW3k15nGq3s31aHE5ERERExBohic0AK9slj2p1HBERERERa5IsRYuIiIiIiLaXxCYiIiIiItpeEpuIiIiIiGh7SWwiIiIiIqLtZfOAYUqSgW1tP9DqWJpx1+yFTDxtWqvDWK3MygdPIyIiIpqWGZs2JmljSU9KmtxQfoGky1oUVkRERETEkMuMTRuz/bikjwHnS3qN7aWS3g68C9hpIK8l6WW2XxzIMSMiIiIiBkpmbIaApFMlzZb0jKT7Jb1d0hsk/VbSAklzJZ0j6eW99F9H0tckPSLpcUnnSloPwPYU4H7g30vZfwIftv1E6bu5pB9LekLSQ5I+WBv3TZJurcXwLUlrl7qXSbKkkyQ9ANw3yI8pIiIiImKlJbEZZJK2B04GdrU9GtgbmAUsAz4GjAfeBLwdOKmXYb4CbAe8DpgETAA+V6s/Afhn4HLgf21fXq49AvgZcHvp8w7gk2VWB+BF4CMlhjcD+wD/0nDtA4FdgV16uLfjJXVK6ly2ZGETTyMiIiIiYnAksRl8y4B1gFdLWtv2LNsP2p5p+1bbL9qeRTXTsmdjZ0kCjgM+Znue7WeALwKHdbex/ShVorMXcGKt+27ABra/aPv5shHBD7r72r7d9u9KDH8Czushhi/anm97aWNsts+z3WG7Y8TIMSv5eCIiIiIiVl3esRlkth+Q9FHgdGAnSb8APg6MAs4GOoCRVH8XM3sYYqNSP7PKcQAQMKKh3d3AfNtza2VbAVtKWlArGwHMAJC0A/AfwN/XYvhdw7hdTd5qRERERETLZMZmCNi+1PYeVImGqZaWfY/qvZVtbW8AfJoqYWn0JLAU2Mn22PIbY3tUE5fuAv6v1m+s7dG2Dyj1/wn8LzCpxPC5HmJwP283IiIiImLIJbEZZJK2l/Q2SesAz1IlKcuA0cDTwKIyc3JiT/1tLwfOB74u6ZVlzAmS9m7i8r8Fnpd0iqR1JY2QtIukvy/1o4GFwGJJO/K379dERERERLSFLEUbfOsAXwZ2BF4AbgGOp9oE4DzgU8D/AFcAb+tljFOpZlNulTQemE014/OLFV3Y9ouS9qNa8vbJEsu9wGdKk1OAc6lmi+4oMVETrAQAACAASURBVOyxMje5y4QxdOaDkhERERHRIrKz0ihWXUdHhzs7O1sdRkRERESs5iTNtN3RWJ6laBERERER0faS2ERERERERNtLYhMREREREW0viU1ERERERLS9JDYREREREdH2kthERERERETbS2ITERERERFtLx/orJG0PXA51ccz1wc+b/sLrY1q4EkysK3tB/poNxmYanvzvsa8a/ZCJp42bYAijEaz8vHTiIiIiBXKjM1LfQqYYXu07bW6kxpJkyUtl7So/GZLOmOwg5E0Q5Ilvbah/JpSPnmwY4iIiIiIaAdJbF5qK+DuXurm2B5lexSwB/ABSQcPQUx/BI7pPpH0CmA34IkhuHZERERERFtIYlNIugF4K3BOmZW5VNKZPbW1/RBwC/DqWv9vSuqS9LSkmZLeUqt7g6TOUve4pLNrdbtJukXSAkm/72EW5hLgUEkjyvnhwI+B52tjrCPpG5LmlN83JK1Tq/+kpLml7p8b7nsdSV+T9EiJ7VxJ6/Xv6UVEREREtFYSm8L224BfAyeXWZnne2sraVvgzcCtteLbgdcB44BLgR9KWrfUfRP4pu0NgG2AK8s4E4BpwJml3yeAqyRtVBt3DnAP8M5yfgxwcUNIn6GaxXkd8FrgDcBnyzX2KeO+A9gW2Kuh71eA7UrfScAE4HO93XvDczi+JGydy5YsbKZLRERERMSgSGLTvM3KrMrTVMvDfgfc3F1pe6rtp2y/aPs/gHWA7Uv1C8AkSeNtL7LdnRAdBVxn+zrby21PBzqB/RqufTFwTNncYKzt3zbUHwn8u+0/234COAM4utS9B7jA9v/aXgyc3t1JkoDjgI/Znmf7GeCLwGHNPBDb59nusN0xYuSYZrpERERERAyKJDbNm2N7bJl1GQssBS7qrpR0iqR7JS2UtAAYA4wv1R+gmhW5T9LtkvYv5VsBh5SEaUHptwewacO1rwbeBnwImNJDbJsBD9fOHy5l3XVdDXXdNgJGAjNr1//vUh4RERER0Tay3fNKsL1Q0qXAFQDlfZpTgbcDd9teLmk+oNL+/4DDJa0F/CPwo7IJQBcwxfZxfVxviaSfAydSLWVrNIeXbnywZSkDmAtsUWu7Ze34SaoEbSfbs5u6+YiIiIiIYSgzNitB0iiq5VrdicRo4EWqncpeJulzwAa19kdJ2sj2cmBBKV4GTAUOkLS3pBGS1i1bS/f03ZhPA3vantVD3WXAZyVtJGk81TsyU0vdlcD7JL1a0kjg892dSjznA1+X9MoS6wRJe/f7oUREREREtFBmbJq3maRF5fg5qo0DjiznvwB+TvXuzWLg67x0+dc+wNklsXgYOMz2s0CXpIOAr1IlJ8uA26hmZl7C9hz+OgvT6EyqROoP5fyHpQzbP5f0DeAGYDnVpgJH1vqeSpUI3VqSotnA98o9NW2XCWPozEckIyIiIqJFZLvVMcRqoKOjw52dna0OIyIiIiJWc5Jm2u5oLM9StIiIiIiIaHtJbCIiIiIiou0lsYmIiIiIiLaXxCYiIiIiItpeEpuIiIiIiGh7SWwiIiIiIqLtJbGJiIiIiIi2lw90tilJs4BjbV/f6lgA7pq9kImnTWt1GLECs/IB1YiIiFiNZcamxSTtIekWSQslzZP0G0m7DtG1Z0g6thxPlrRc0qLye1TSlUMVS0RERETEqkhi00KSNgB+BnwbGAdMAM4AnmtRSHNsjwJGA7sB9wG/lvT2FsUTEREREdGUJDattR2A7ctsL7O91PYvbf9B0jaSbpD0lKQnJV0iaWxPg0haS9Jpkh4s7a+UNK7UrStpailfIOl2SRuvKChXHrX9OeD7wFcG+sYjIiIiIgZSEpvW+iOwTNJFkvaVtGGtTsCXgM2AHYEtgNN7GefDwMHAnqX9fOA7pe69wJjS/xXACcDSfsR4NfB6Ses3Vkg6XlKnpM5lSxb2Y8iIiIiIiIGVxKaFbD8N7AEYOB94QtJPJW1s+wHb020/Z/sJ4GyqxKUn/wJ8psyyPEeVAL1b0suAF6gSmkllVmhmuW6z5lAlWX8zW2T7PNsdtjtGjBzTjyEjIiIiIgZWEpsWs32v7ffZ3hzYmWrG5RuSXinpckmzJT0NTAXG9zLMVsCPy1KzBcC9wDJgY2AK8AvgcklzJH1V0tr9CHECVeK1YOXuMCIiIiJi8CWxGUZs3wdcSJXgfIkqoXiN7Q2Ao6hmTnrSBexre2ztt67t2bZfsH2G7VcDuwP7A8f0I6x/AO6wvXglbysiIiIiYtAlsWkhSTtIOkXS5uV8C+Bw4FaqnckWAQskTQA+uYKhzgXOkrRVGWcjSQeV47dK2kXSCOBpqqVpy/qIS5ImSPo8cCzw6VW60YiIiIiIQZYPdLbWM8AbgY+XHc8WUG3//Emql/0vBhYCD1AtKftYL+N8k2o255eSNgP+DFwB/ATYhCrx2ZwqUbqCallbTzaTtKiMtRC4BZhs+9a+bmSXCWPozAcgIyIiIqJFZLvVMcRqoKOjw52dna0OIyIiIiJWc5Jm2u5oLM9StIiIiIiIaHtJbCIiIiIiou0lsYmIiIiIiLbXdGIj6XBJO5bj7SXdJOkGSTsMXngRERERERF968+MzZnAvHL8NeA24CbguwMdVERERERERH/0Z7vnjWw/LmldYA/g3VTfRHlyUCKLiIiIiIhoUn8SmyckTQJ2AW63/ZykkVTfPImIiIiIiGiZ/iQ2XwBmUn21/tBS9nbg9wMdVKtI2h64HJgErA983vYXBvF6pwOTbB/VRNsZwFTb3x+seFbFXbMXMvG0aa0OI4aRWflga0RERAyhpt+xsX0hsCmwue3ppfh3wGGDEFerfAqYYXu07bW6kxpJkyVZ0tX1xpJeW8pnDGWQkt4naZmkReX3kKQLJG03lHFERERERAwXK0xsJK1V/wHPAs/Wzp8E/jwUgQ6RrYC7e6l7Athd0itqZe8F/jjoUfXst7ZHAWOAvYClwExJO7conoiIiIiIlulrxuZFqg0Cevt117c9STcAbwXOKbMgl0o6s9bkeeAaygyVpBHAe4BLGsbZXdLtkhaWP3ev1W0t6UZJz0iaDoxv6LubpFskLZD0e0mT+4rb9jLbD9o+CbgROL2Z8cqsz59KLA9JOrJWd5yke0vdPZJe3+cDjIiIiIhoob4Sm62BV63g113f9my/Dfg1cHKZCXm+h2YXA8eU472pZnfmdFdKGgdMA74FvAI4G5hWm+W5lOo9pfFU7yy9t9Z3Qul7JjAO+ARwlaSN+nEbVwNv6Ws8SeuXGPe1PRrYHbiz9DuEKjk6BtgAOBB4qh8xREREREQMuRVuHmD74caysgRtY+Bx28sHK7DhyPYtksaVTQaOoUp01qs1eRfwf7anlPPLJH0YOKDMCO0K7GX7OeAmSdfW+h4FXGf7unI+XVInsB9wUZMhzqFKYvoa70fAcmBnSY/YngvMLe2OBb5q+/Zy/kBvF5N0PHA8wIgN+pN/RUREREQMrKY3D5C0gaSLqd6zmQ0slXSRpDGDFt3wNAU4mWrZ2o8b6jYDGpPBh4EJpW6+7cUNdd22Ag4py8YWSFpA9b2gTfsR2wT++hHVXscrMRwKnADMlTRN0g6l3xbAg81czPZ5tjtsd4wYuab9M4iIiIiI4aTpxIZq6dL6wM5UsxS7ACNL+ZpkCnAS1WzIkoa6OVQJRd2WVIngXGDDsgysXtetC5hie2ztt77tL/cjtn+gWk7X53i2f2H7HVSJ033A+bV+2/TjmhERERERLdefxGYf4Gjbf7T9nO0/Au8v5WsM2w8BewKf6aH6OmA7SUdIepmkQ4FXAz8ry/o6gTMkvVzSHsABtb5TqZas7S1phKR1yzbTm68ontJ2a0nfBiYDZ/Q1nqSNJR1YkqzngEVU3ycC+D7wCUl/r8okSY3JWkRERETEsNKfD3Q+C2zES5dPjaf6j/EaxfbNvZQ/JWl/4JvA96jeT9nf9pOlyRFU78vMA35L9Y7O2NK3S9JBwFeBy6gSjduAE3sJ402SFgGi2nZ7BrCr7XubGG8t4BSq2SdTbRxwUun3w7LZwaVUS9tmAUfzt0vsXmKXCWPozAcZIyIiIqJFZLu5htJnqV6YP5vqP7lbAR8DpnZ/yDLWXB0dHe7s7Gx1GBERERGxmpM003ZHY3l/ZmzOonpX5EiqF+HnUO2e9YOBCTEiIiIiImLl9Ocdm28C99vey/arbe8F3CvpG4MUW0RERERERFP6k9gcTvXye91MqvdGIiIiIiIiWqY/iY2BEQ1lI/o5RkRERERExIDrT1Lya+ALktYCKH+ezl+/mxIREREREdES/dk84CPAz6i+VP8w1ccl5/LSb7FEREREREQMuaYTG9uPSno98AZgC6ov1N9me/lgBRcREREREdGMpr9jE7Ei62y6rTd9bzbIi8E1Kx+BjYiIWOP19h2bvPg/TEmaLOnRVscREREREdEOktgMEEmzJC2VtKj2O6fVcUVERERErAn6s3lA9O0A29e3OoiVJWmE7WWtjiMiIiIior8yYzPIJL1P0s2SviZpvqSHJO1bqx8n6QJJc0r9Nb2Ms6OkGZIWSLpb0oG1uv0k3SPpGUmzJX2ifu2GcSxpUjm+UNL3JF0naTHwVknrlFgfkfS4pHMlrTcoDyciIiIiYoAksRkabwTuB8YDXwV+IEmlbgowEtgJeCXw9cbOktYGrgV+Wdp8CLhE0valyQ+Af7E9GtgZuKEfsR0BnAWMBm4GvgJsB7wOmARMAD7XU0dJx0vqlNS5bMnCflwyIiIiImJgJbEZWNeUGZXu33Gl/GHb55dlXhcBmwIbS9oU2Bc4wfZ82y/YvrGHcXcDRgFftv287Ruovil0eKl/AXi1pA3KOHf0I+af2P5N2bb7OeA44GO259l+BvgicFhPHW2fZ7vDdseIkWP6ccmIiIiIiIGVxGZgHWx7bO13fil/rLuB7SXlcBTV94Dm2Z7fx7ibAV0N3wx6mGo2BeCfgP2AhyXdKOlN/Yi5q3a8EdXs0czu5Az471IeERERETFsJbFprS5gnKSxfbSbA2whqf73tSUwG8D27bYPolqmdg1wZWmzmCpRAUDSJj2MXf+Q0ZPAUmCnWnI2xvao/txURERERMRQy65oLWR7rqSfA9+V9EFgEfAm2zc1NP0dVZLyKUn/AbwZOADYVdLLgUOAn9leKOlpoHtns98DO0l6HXAfcHof8SyXdD7wdUkn2/6zpAnAzrZ/saK+u0wYQ2c+nhgRERERLZIZm4F1bcN3bH7cRJ+jqd6RuQ/4M/DRxga2nwcOpHof50ngu8Axtu+rjTGrJDUnAEeVfn8E/h24Hvg/qs0B+nIq8ABwaxnvemD7FXeJiIiIiGgt2e67VUQfOjo63NnZ2eowIiIiImI1J2mm7Y7G8szYRERERERE20tiExERERERbS+JTUREREREtL0kNhERERER0faS2ERERERERNtLYhMREREREW0vH+iMAXHX7IVMPG1aq8OI6LdZ+bBsRETEamHYz9hI2l7S/0h6RtJySf/W6piaJWmGpGNbHUdERERExOpu2Cc2wKeAGbZH217L9he6KyR9WtJDkhZJelTSFS2Mc0iVpOnZkvA9LWmmpNMkrdPq2CIiIiIihlo7JDZbAXc3Fkp6L3A0sJftUUAH8Kshjm1QSGp2ieDJtkcDmwKnAIcB10nSoAUXERERETEMDevERtINwFuBc8qszKWSzizVuwK/sP0ggO3HbJ9X6ztG0g8kzZU0W9KZkkbU6o+TdG+Z8bhH0utL+Y5lNmSBpLslHVjrc6Gk70iaVvr9TtI2tfp3SLpP0kJJ5wCq1W0j6QZJT0l6UtIlksbW6mdJOlXSH4DFkj4p6aqG5/FtSd9ofE62F9ueARwIvAl4V2m/VpnFebBc90pJ40rdupKmlvIFkm6XtHGpGyfpAklzJM2XdE3//uYiIiIiIobWsE5sbL8N+DXVzMQo4Pla9a3AMSUB6KgnLcVFwIvAJODvgHcCxwJIOgQ4HTgG2IAqIXhK0trAtcAvgVcCHwIukbR9bdzDgTOADYEHgLPKmOOBq4DPAuOBB4E31/oJ+BKwGbAjsEWJoe5wqqRkLDAV2Kc7+SmzOIcCU1bwvB4BOoG3lKIPAwcDe5brzge+U+reC4wpcbwCOAFYWuqmACOBncpz+HpP15N0vKROSZ3LlizsLayIiIiIiEE3rBObFbE9lSrx2Bu4EfizpNMAyszDvsBHy2zGn6n+c35Y6X4s8FXbt7vygO2Hgd2AUcCXbT9v+wbgZ1QJR7erbd9m+0XgEuB1pXw/4B7bP7L9AvAN4LFavA/Ynm77OdtPAGdTJRx137LdZXup7bnATcAhpW4f4EnbM/t4NHOAceX4X4DP2H7U9nNUidS7S5L0AlVCM8n2MtszbT8tadPy7E6wPd/2C7Zv7OlCts+z3WG7Y8TIMX2EFRERERExeNp6u2fbl1DNqKxNNTNxiaT/oZqZWBuYW3vdZC2gqxxvQTWj0mgzoMv28lrZw8CE2vljteMlVInQX/rWYrOkv5xLeiXwLarZlNElnvkN1+9qOL8IOBE4HziKFczW1EwAbinHWwE/llS/n2XAxmWsLYDLy6zQVOAzpWye7cbYIiIiIiKGrbadsakrswo/BP4A7EyVIDwHjLc9tvw2sL1T6dIFbNPDUHOALSTVn8uWwOwmwphLlRQA/P/t3XmcXFWd/vHPQ2QLCYkQtkRCRGBkc1AbFBVBxWFxw3EddgcE3BVUFvXHooAyCi7oMKCC7KAoqFEBUTZZpIMoA4IDJCEkYQkhKxAgeX5/3NNwKas7naTS1ZU879frvrx1zj3nnnu401PfnKUoC/g3ruWfDBh4le21qQKVxkX+bvh8OfAqSdsA76QaIeqVpI2B11JN34PqOfeo9cFI22vYnlr67HjbWwFvKPXvX8qsU1//ExEREREx2HXsiI2kA4HHqKZrzaeakrY1cKvt6ZKuAr5VfvdmHvBy4GVlWtUPgVMl3QjcThXkPAvcWur6oqRvUa2ReRfVRgWLM55qk4N/B34JfALYsJY/HJgNzJI0BvjC4iq0/bSknwEXAn8ua2ia9cXQ0sbTgD8DvylZZwAnSjrA9mRJ6wFvsH2FpLcAM4C7gTnl+ReWvvst8ANJn6Dqux1tX99XW7cdM4Lu/NBhRERERLRJJ4/YzAGOAR4EZgGnAB+zfWPJ3x9YjeqL+xPAz6i2RaaM7pxIFTDMpRoZWcf2M1QbCexB9aX/B8D+tu9ZXGNsz6BaD/N14HFgc+BPtUuOB15DFdyMB37ez+f8CbAtzaehnS5pLvAI1Zqey4Dda1PpvkMVZF1VrrsFeF3J25CqT+YAf6dap3R+yduPKtC5B3gU+Gw/2xoRERER0RayG2c/xWAiaSxVgLGh7Tntbk9vurq63N3d3e5mRERERMQKTtIE212N6Z08YrPCK2t9DgcuHsxBTUREREREu3XsGpsVnaS1qKaYTaba6jkiIiIiInqRwGaQsj2fF7aSjoiIiIiIPmQqWkREREREdLwENhERERER0fES2ERERERERMfLGptoiTunzmbcUePb3YyItpuUH6qNiIhoi4zYtIgkS9qsBfVcK+ngVrSpFSTtIumhdrcjIiIiIqIvCWwGWAkUFkma13Ds2O62RURERER0qkxFa49ptl/W7kZERERERKwoMmLThKQjJU2VNFfSvZLeJmkHSTdLmiVpuqTTJa3WS/nVJX1T0oOSHpF0hqQ1l6Idr5D0B0mPS5oh6QJJI2v5kyQdLeluSU9IOlvSGiVvlKRfl/bOlHSDpFVK3mhJl0l6TNJESZ+u1bmmpHNKfXcD2y9xB0ZEREREDLAENg0k/QvwSWB728OB3YBJwELgc8AoYEfgbcDHe6nmG8AWwHbAZsAY4P8tTXOAk4HRwJbAxsBxDdfsU9r4inLPL5f0I4CHgPWADYBjAJfg5lfAX0u73gZ8VtJupdyxpa5XlHoP6LVx0iGSuiV1L3xy9lI8XkREREREaySw+WcLgdWBrSStanuS7fttT7B9i+3nbE8C/gfYubGwJAEfBT5ne6btucBJwIdrl40uIyn1Y63GumzfZ/tq2wtsPwac2uSep9ueYnsmcCLwHyX9WWAjYBPbz9q+wbapRmDWs32C7WdsPwCcVWvfB4ETS9unAN/traNsn2m7y3bXkKEj+ujSiIiIiIjlK2tsGti+T9JnqUZGtpZ0JXA4MIwqsOgChlL13YQmVaxX8idUMQ5QjbwMqV3TrzU2ktanCix2AoZTBaJPNFw2pXY+mWp0B+C/yjNcVdpxpu2vA5tQAqtauSHADeV8dJM6IyIiIiIGtYzYNGH7QttvogoCTDW17L+Be4DNba9NNbVLTYrPAJ4CtrY9shwjbA9biqacXO7/qnLPfZvcc+Pa+VhgWnmGubaPsL0p8C7gcElvowpaJtbaNtL2cNt7ljqmN6kzIiIiImJQy4hNg7LGZgzwJ+BpqiBlFaoRkznAPEmvBD4GPNZY3vYiSWcBp0n6pO1HJY0BtrF95RI2ZzgwG5hV6vhCk2s+IenXwJNUwdYl5TneSRWI3V/avbAcfwbmSDqSajToGar1O2vavg24FDha0q3AWsCn+tPQbceMoDs/TBgRERERbZIRm3+2OvB1qpGXh4H1qQKGzwN7A3Op1qRc0kcdRwL3AbdImgP8HviXWv7oJr9j874m9RwPvIYquBkP/LzJNRcCVwEPlONrJX3zct95wM3AD2xfa3sh1QjOdsDE8pw/BHoWyRxPNf1sYqn3vD6eMyIiIiJiUFC1njw6kaRJwMG2f9/utnR1dbm7u7vdzYiIiIiIFZykCba7GtMzYhMRERERER0vgU1ERERERHS8bB7QwWyPa3cbIiIiIiIGg4zYREREREREx0tgExERERERHS+BTUREREREdLyssYmWuHPqbMYdNb7dzYgY1CblR2wjIiKWm4zYdDhJ4yRZ0mKDVEkHSrpxINoVERERETGQEtgMMEmTJD0jaVRD+h0lQBk3gG2xpM3K+XGSnpU0txz/kHS6pI0Gqj0REREREUsrgU17TAT+o+eDpG2BNdvXnOddYns4sA7wXmBDYEKCm4iIiIgY7BLYtMd5wP61zwcA5/Z8kDRC0rmSHpM0WdKXJa1S8oZI+qakGZIeAF40ab+U/ZGk6ZKmSvqapCFL0jjbz9q+C/gQ8BhwxFI+Z0RERETEgEhg0x63AGtL2rIEHR8Czq/lfw8YAWwK7EwVBH2k5H0UeCfwaqALeH9D3T8BngM2K9f8G3Dw0jTS9kLgCmCnZvmSDpHULal74ZOzl+YWEREREREtkcCmfXpGbd4O3ANMLek9gc7RtufangR8C9iv5H8Q+LbtKbZnAif3VChpA2AP4LO259t+FDgN+PAytHMa1dS0f2L7TNtdtruGDB2xDLeIiIiIiFg22e65fc4DrgdeTm0aGjAKWA2YXEubDIwp56OBKQ15PTYBVgWmS+pJW6Xh+iU1Bpi5DOUjIiIiIpa7BDZtYnuypInAnsBBtawZwLNUQcrdJW0sL4zoTAc2rl0/tnY+BVgAjLL93LK2sazreRfw+2WtKyIiIiJiecpUtPY6CHir7fm1tIXApcCJkoZL2gQ4nBfW4FwKfFrSyyS9FDiqp6Dt6cBVwLckrS1pFUmvkLTzkjRK0qqStgQuotoZ7dSlfcCIiIiIiIGQEZs2sn1/L1mfotpA4AHgaeAs4Mcl7yxgC+CvwBzgm8Bba2X3B75ONdozvNTxjX426UOS9gJEtbbmauC1tqctruC2Y0bQnV9Vj4iIiIg2ke12tyFWAF1dXe7u7m53MyIiIiJiBSdpgu2uxvRMRYuIiIiIiI6XwCYiIiIiIjpeApuIiIiIiOh4CWwiIiIiIqLjJbCJiIiIiIiOl8AmIiIiIiI6XgKbiIiIiIjoeCv1D3RKOg7YzPa+y6l+A5vbvk/SGcBU219dHvfq5f7nAA/Z/vLyvtedU2cz7qjxy/s2ESutSfkB3IiIiD513IiNpEmSdm1IO1DSje1qU3/YPmx5BDXl2RdKmlc7Tm/1fSIiIiIiBrOVesSmL5JeYvu5drejn262/aZ2NyIiIiIiol06bsSmL5K+IOmyhrTvSfp2OX+5pOskzZV0NTCqdt04SZZ0kKQHgT+U9J9KeljSbEnXS9q6VuZaSQfXPvc6ciTpHElfq31+j6Q7JM2RdL+k3Wt1PFDaOFHSPq3pHZD0UUn3SZop6ZeSRpf04yV9r5yvKmm+pFPK5zUlPS3ppa1qR0REREREq61QgQ1wPrC7pJFQjboAHwLOK/kXAhOoApqvAgc0qWNnYEtgt/L5t8DmwPrA7cAFy9pISTsA5wJfAEYCbwYmSVoL+C6wh+3hwBuAO5b1fuWebwVOBj4IbARMBi4u2dcBu5Tz7YGHqfoBYEfgXttPNKnzEEndkroXPjm7Fc2MiIiIiFgqnToV7XJJ9WliqwG3254u6XrgA8BZwO7ADNsTJI2l+tK+q+0FwPWSftWk7uNsz+/5YPvHPedls4EnJI2wvSzf5A8Cfmz76vJ5aql/LWARsI2kB21PB6b3o77XS5pV+7y77Vsartmn3PP2cq+jy7OMA24GNpe0LlWQ9SPg45KGUQU41zW7qe0zgTMBVt9oc/ejnRERERERy0WnjtjsZXtkzwF8vJb3E6Bnl7N9eWG0ZjTwRD1ooRq1aDSl50TSEElfL1PF5gCTStaoJuWWxMbA/Y2JpW0fAg4DpksaL+mV/ajvlnp/NAlqoHr+55/X9jzgcWCM7aeAbqog5s1UgcxNwBvpI7CJiIiIiBgsOjWw6cvlwKskbQO8kxemjk0HXlpGRXqMbVK+PvKwN/AeYFdgBDCupKv873xgaO36DfvZxinAK5pl2L7S9tuppovdQzXy1ArTgE16PpR+WJcyWkQVvLwVeDVwW/m8G7ADcH2L2hARERERsVyshP7fNAAAIABJREFUcIGN7aeBn1Gtp/mz7QdL+mSqUYnjJa0m6U3AuxZT3XBgAdXIxlDgpIb8O4B/lzRU0mZUU8z640fARyS9TdIqksZIeqWkDSS9uwQdC4B5wMJ+1rk4F5Z7bidp9fIst9qeVPKvA/YH7rb9DHAtcDAw0fZjLWpDRERERMRy0alrbBbnJ1Rfyv+zIX3vkjeTal3JuVSL93tzLtWoxdRS5ivAx2r5p1Gt23kE+BvV6NCuLIbtP0v6SCn/8lL+E8Bs4Aiq6XOmCpw+3ls9S8L2NZK+AlwGvJRqqtmHa5fcBKzJC6MzdwNP08/Rmm3HjKA7PyAYEREREW0ie8Vb8102CrgH2ND2nHa3Z2XQ1dXl7u7udjcjIiIiIlZwkibY7mpMX+GmoklaBTgcuDhBTURERETEymGFCmzK2pQ5wNuBY9vcnJaRdIakeU2OM9rdtoiIiIiIwWCFWmNTtkse1u52tJrtw6i2gI6IiIiIiCZWqBGbiIiIiIhYOSWwiYiIiIiIjpfAJiIiIiIiOl4Cm4iIiIiI6Hgr1OYBy4skA5vbvq/dbRms7pw6m3FHjW93MyKinyblB3UjImIF03EjNpL2ltRdtjueLum3kt7U7nYtDUnHSJpYnuUhSZcsY327SHqoVe2LiIiIiOgUHRXYSDoc+DZwErABMBb4AfCedrarN5J6HRGTdACwH7Cr7WFAF3DNQLUtIiIiImJF0jGBjaQRwAnAJ2z/3PZ828/a/pXtL0haXdK3JU0rx7clrV7KHijpxob6LGmzcn5O+RHMqyXNlXSdpE16acfqkr4p6UFJj5Rya5a8XcrIy5GSHgbO7uORtgeutH0/gO2HbZ9Z6vmApAkN9z1C0uXlfE9Jd5e2TpX0+fLjpL8FRtd+wHO0pFUkHSXpfkmPS7pU0jqlnnGlHz4iaYqkJyQdJml7SX+TNEvS6Uv4nyoiIiIiYsB1TGAD7AisAfyil/wvAa8HtgP+FdgB+PIS1L8P8FVgFHAHcEEv130D2KLcZzNgDPD/avkbAusAmwCH9HG/W4D9JX1BUpekIbW8XwIvl7RlLW1f4Lxy/iPgUNvDgW2AP5QfJ90DmGZ7WDmmAZ8G9gJ2BkYDTwDfb2jL64DNgQ9RjYh9CdgV2Br4oKSdmz2ApEPKtMDuhU/O7uNRIyIiIiKWr04KbNYFZth+rpf8fYATbD9q+zHgeKqpXv013vb1thdQfbHfUdLG9QskCfgo8DnbM23PpZoW9+HaZYuAY20vsP1UbzezfT7wKWA34DrgUUlHlbwFwCVUwQyStgbGAb8uxZ8FtpK0tu0nbN/ex3MdCnzJ9kOl3uOA9zdMk/uq7adtXwXMBy4q/TgVuAF4dS/PcKbtLttdQ4aO6KMJERERERHLVycFNo8Do/pYtzIamFz7PLmk9deUnhPb84CZTcqvBwwFJpRpWrOA35X0Ho/Zfro/N7R9ge1dgZHAYcAJknYr2T8B9i7B1H7ApSUwAXgfsCcwuUyb27GP22wC/KLW3r8DC6nWKPV4pHb+VJPPw/rzPBERERER7dJJgc3NwNNU06qamUb1Jb7H2JIG1SjE0J4MSRs2Kb9xLX8Y1XSyaQ3XzKD6or+17ZHlGFEW//dwP57lRcpaoZ8Cf6OaWobtW4BngJ2AvXlhGhq2b7P9HmB94HLg0j7uPQXYo9bekbbXKKMxERERERErhI4JbGzPplrL8n1Je0kaKmlVSXtIOgW4CPiypPUkjSrXnl+K/xXYWtJ2ktagmo7VaE9Jb5K0GtVam1ttT6lfYHsRcBZwmqT1ASSNqY2y9FvZ0OAdkoaXBf57UK1pubV22bnA6cBztm8s5VaTtI+kEbafBeZQjcBANdKybtlooccZwIk9myGU/hmUu8hFRERERCytjvqBTtunSnqEalOAC4C5wATgROB2YG2qUQ+AnwJfK+X+IekE4PdUIy5HU609qbsQOJZqk4LbqdbsNHMkVdB0SwmgpgL/DVy5hI8zBziGKvgaQjV17mM9AUxxHlWQ9dWGsvsBp5cNB+6lrMWxfY+ki4AHSt5WwHcAAVdJGg08SrV+54olbG+fth0zgu784F9EREREtInsJZ45tcKRdA7wkO0l2UVtuSvbSD8KvMb2/7W7PX3p6upyd3d3u5sRERERESs4SRNsdzWmd8xUtJXUx4DbBntQExERERHRbh01Fa3TSDqGarpZoxts77GYspOoppD1tllCREREREQUCWwA2wcup3pPovqdm6UpO661rYmIiIiIWHFlKlpERERERHS8BDYREREREdHxEthERERERETHS2ATEREREREdL5sHREvcOXU2444a3+5mRMQgNik/4hsREctRRmxWUpLOl3Rcu9sREREREdEKHR/YSPqwpFslzZf0aDn/uCS1u21LQtJYSfNqh8sz9Xzeqd1tjIiIiIgYrDo6sJF0BPAd4L+ADYENgMOANwKrNbl+yIA2sAlJTaf/2X7Q9rCeoyT/ay3thiZ1tf15IiIiIiIGg44NbCSNAE4APm77Z7bnuvIX2/vYXiDpHEn/Lek3kuYDb5E0QtK5kh6TNFnSlyWtUuo8TtL5tXuMKyMnLymfD5T0gKS5kiZK2qd27X9K+rukJyRdKWmTWp4lfULS/wH/twzPfL6k70v6XXmenSS9W9IdpU0PSvpKQ5k3S7pF0mxJUyTt16TetSVdL+k0Vd5ZnmWupIckfa6X9hwiqVtS98InZy/tY0VERERELLNO3jxgR2B14IrFXLc3sCfwTqpRnDOBEcCmwLrAVcB04Ed9VSJpLeC7wPa275W0EbBOydsLOAZ4F1XgchRwEfCGWhV7Aa8Dnur3E/b9PLcCq1KNTu0L3A1sC1wj6S+2fy3p5cB44CDg58BI4GUNzzUK+B3wa9vHlbSzgffYvknSOsC4Zg2xfSZVf7L6Rpt7GZ8rIiIiImKpdeyIDTAKmGH7uZ4ESTdJmiXpKUlvLslX2P6T7UXAs8CHgKPLCM8k4FvAP41i9GIRsI2kNW1Pt31XST8UONn230t7TgK2q4/alPyZtpc1sPmF7ZttL7K9wPYfbP9v+fxX4GJg53LtvsDvbF9q+znbM2zfUatrDHAdcEFPUFM8C2wlaXhp8+3L2OaIiIiIiOWqkwObx4FR9TUrtt9ge2TJ63m2KbUyo6hGbSbX0iZTfcHvk+35VEHRYcB0SeMlvbJkbwJ8pwRVs4CZgBrqnUJrvKgeSTtKurZMrZsNHEz1nAAbA/f3Ude7qUZ9zmpIf2/Je7DU/brWND0iIiIiYvno5MDmZmAB8J7FXFefIjWDajSiPpIyFphazucDQ2t5G76oIvtK228HNgLu4YWAYApwqO2RtWNN2zf10o5l0VjPxcBlwMa2RwA/pAqqetr1ij7qOgP4IzBe0vPPbftW2+8G1gd+Xe4RERERETFodewaG9uzJB0P/KBs7fw74EngVcBavZRZKOlS4ERJ+1OtkTkc+Ga55A7gSEljgdnA0T1lJW1AtUbmGqp1MvOAhSX7DOCrku6wfVfZ2ODfbP+0pQ/d3HBgpu2nJb0e+DBVMAJwPvA3Se+jWos0EhhTpqxBFSQdBvwY+KWkd1IFRe+lWnMzR9Lc2nP2atsxI+jOj+9FRERERJt08ogNtk+hCky+CDwKPAL8D3AkcFMvxT5FNTLzAHAjcCHVF3tsXw1cAvwNmMALAQJUfXUEMI1qqtnOwMdLuV8A3wAuljQH+F9gjxY95uJ8DDi5BCDHAJf2ZNieSLWhwZGlzbdTbTBA7RpTbS7wKPALqql6BwCTy7McRP/XIEVEREREtIWq77URy6arq8vd3d3tbkZERERErOAkTbDd1Zje0SM2ERERERER0MFrbDqVpJ2A3zbLsz1sgJsTEREREbFCSGAzwGzfACSAiYiIiIhooUxFi4iIiIiIjpfAJiIiIiIiOl4Cm4iIiIiI6HgDssZG0r9Q/Xr9ZlQ/nnms7a8OwH3PAR6y/eXlfa8m9zawue37Bvre7XDn1NmMO2p8u5sRER1uUn7oNyIiltJAjdh8EbjW9nDbq/QENZJ2kbRI0jxJcyXdK+kjA9SmfpE0TpIlvaQhfSNJP5I0vbT9HknHS1prANp0oKSFpd/mSZoo6WxJWyzve0dEREREDEYDFdhsAtzVS960ss3x2sCRwFmStmq8qDGwaCdJ6wA3A2sCO9oeDrwdGAm8YoCacXPptxHArsBTwARJ2wzQ/SMiIiIiBo3lHthI+gPwFuD0MrpwoaSvNV7nyuXAE8BWtZGSgyQ9CPyh1PduSXdJmiXpWklb1u71akm3lxGUS4A1ankHSrqxoW2WtFk5X1PStyRNljRb0o2S1gSuL5fPKu3fETgcmAvsa3tSaf8U25+x/bcmffAOSX+RNEfSFEnH1fLWkHS+pMfLM90maYNamx8ozzNR0j5N+m2h7fttfxy4DqjX/XpJN5V6/yppl4b+aFq3pI9K+nvJu1vSaxrvGxERERExmCz3wMb2W4EbgE+WEYZnml0naRVJ76Ua9bizlrUzsCWwW5lqdRHwWWA94DfAryStJmk14HLgPGAd4KfA+5agqd8EXgu8oZT/IrAIeHPJH2l7mO2bqUZIfm57UT/rng/sX57tHcDHJO1V8g6gGnXZGFgXOAx4qkxp+y6wRxkRegNwx2Lu83NgJwBJY4DxwNfK83weuEzSen3VLekDVMHR/lSjaO8GHu/nc0ZEREREtMVg2BVttKRZwAzgWGA/2/fW8o+zPd/2U8CHgPG2r7b9LFUwsibVF/PXA6sC37b9rO2fAbf1pwGSVgH+E/iM7allFOQm2wt6KbIuML2/D2j7Wtt32l5URnQuogrYAJ4t9W1W7jvB9pyStwjYRtKatqfb7m06X49pVEEMwL7Ab2z/ptz3aqAb2HMxdR8MnGL7tjKKdp/tyc1uJukQSd2Suhc+Obu/3RERERER0XKDIbCZZnuk7XVsb2f74ob8KbXz0cDzX7LLiMkUYEzJm2rbteubfiFvYhTVtLX7+3n948BG/bwWSa+T9EdJj0maTTUqM6pknwdcCVwsaZqkUyStans+VSB3GDBd0nhJr1zMrcYAM8v5JsAHyjS0WSV4fBOw0WLq3ph+9oPtM2132e4aMnREv/oiIiIiImJ5GAyBzeLUA5VpVF/YAZAkqi/iU6lGUMaUtB5ja+fzgaG1shvW8mYAT9N84b+bpP0eeG8Z6emPC4FfAhvbHgGcAQigjC4db3srqpGnd1JNA8P2lbbfThVE3QOctZj7vJdq2h9UAd95JWjsOday/fXF1D2FgdsAISIiIiKiJTohsKm7FHiHpLdJWhU4AlgA3ES1S9lzwKclvUTSvwM71Mr+Fdha0naS1qC2yL6M/PwYOFXSaElDJO0oaXXgMappW5vW6jqVav3JTyRtAtWaFkmnSnpVk3YPB2baflrSDsDePRmS3iJpW0lDgDlUU9MWStqgbJSwVnnGecDCxopLW18u6XvALsDxJet84F2SdivXrKFqe+2XLabuHwKfl/RaVTbrecaIiIiIiMFq0Gyh3B+275W0L/A9qmlXdwDvsv0MQAlmzqJaMP8bqsX0PWX/IekEqtGWp4CjgUNr1X8eOJlqXc4wqkBoN9tPSjoR+FMJpna3fYukN5T73FoChKlUa2ea/SDnx4FvSTqdaueyS6k2EgDYkGoE52VUAcYlVEHJelSB23lUo0Z3lHp67ChpHtXIzwzgWmB7238vzztF0nuAU0q7FgJ/Bj5GFdA2rdv2TyWtSzXKNAaYBOzHYqb1bTtmBN35Yb2IiIiIaBO9eElKxNLp6upyd3d3u5sRERERESs4SRNsdzWmd9pUtIiIiIiIiH+SwCYiIiIiIjpeApuIiIiIiOh4CWwiIiIiIqLjJbCJiIiIiIiOl8AmIiIiIiI6XgKbiIiIiIjoeB31A50xeN05dTbjjhrf7mZEREQMqEn5ceqIQSMjNgNM0i6SHmrDfY+R9MOBvm9ERERExEBIYNMLSZMkPSVpnqRHJJ0taVi729VD0rWSDm5Ie4+kOyTNkTRD0jWSxgHYPsn2wc3qioiIiIjodAls+vYu28OA1wDbA1+uZ6oyKPpQ0mbAucARwAjg5cAPgEXtbFdERERExEAYFF/KBzvbU4HfAtuUkZITJf0JeBLYVNJoSb+UNFPSfZI+2lNW0pqSzpH0hKS7qQIkavkuQUnP53Mkfa32uT4Kc7+k3SWdCOwEnF5GlE4HtgMm2r7Glbm2L7P9YKnnOEnnl/Oecj3Hc5KOK3mjJV0m6TFJEyV9evn0akRERERE6ySw6QdJGwN7An8pSfsBhwDDgcnARcBDwGjg/cBJkt5Wrj0WeEU5dgMOWIL77kA1CvMFYCTwZmCS7S8BNwCftD3M9ieB24FXSjpN0lv6mjZnu6fcMOBNwBPAFWX06VfAX4ExwNuAz0rarZf2HSKpW1L3widn9/exIiIiIiJaLoFN3y6XNAu4EbgOOKmkn2P7LtvPARtSBQdH2n7a9h3AD6mCH4APAifanml7CvDdJbj/QcCPbV9te5HtqbbvaXah7QeAXagCkkuBGWX0p9cAR9J6wOXAp2z/hWo0aT3bJ9h+ptR5FvDhXu55pu0u211Dho5YgseKiIiIiGitbPfct71s/76eIAlgSi1pNDDT9txa2mSgq5Y/pSGvvzYGftPfi23fQhVIIWl74BLgS8DRjddKWhX4GXCh7YtL8ibA6BLM9RhCNToUERERETFoJbBZOq6dTwPWkTS8FtyMBaaW8+lUAcpdtby6J4Ghtc8bUk1rgyogekU/2vDPmfZtkn4ObNPLJd8D5vLiDRGmUK3T2byvuiMiIiIiBpsENsvI9hRJNwEnS/o8sAXVFLJ9yyWXAkdLuhVYC/hUQxV3AHtLugt4O7Az0F3yfgRcJenXwB+BjYDhZTraI8CmPZVIehOwJXCF7UclvRJ4N/CTxjZLOrTc53W267um/RmYI+lIqilzz5Q617R9W1/9sO2YEXTnR8oiIiIiok2yxqY1/gMYRzV68wvgWNtXl7zjqaafTQSuAs5rKPsZ4F3ALGAfqjUvANj+M/AR4DRgNtU6n01K9neA95fd1r5byr8buFPSPOB3pS2n9NLeTYFptZ3RjrG9sLRlu9LeGVTrhbKAJiIiIiIGNdl9zmiK6Jeuri53d3cv/sKIiIiIiGUgaYLtrsb0jNhERERERETHS2ATEREREREdL4FNRERERER0vAQ2ERERERHR8RLYREREREREx0tgExERERERHS8/0BktcefU2Yw7any7mxERERERy9mkQfqj7BmxWclJOl/Sce1uR0RERETEskhgswwkXSDpxw1pO0t6XNJGLb7X+ZIsac+G9NNL+r6tvF9ERERERCdJYLNsPg3sKentAJLWAM4CjrA9vVU3kTSknP4DOKCWvirwPuCBVt0rIiIiIqITJbBZBrYfBz4FnClpLeBY4H7b50haRdIxku6XNEPSxZJeClDyfibpYUmzJF0racueesvozPcl/U7SfGCnknU5sIukEeXzO4Bu4LFa2c0l/bGMGs2QdF7teiS9VtIdkuZKughYvf5Mkt4t6a+lXTdK2qblHRcRERER0WIJbJaR7Z8CE4CLgEOAQ0vW4VSBx5uBlwHzge/Wiv4a2BzYEPhf4LyGqvcGjgeGAzeXtKeA8cAHy+f9gXMbygn4GrARsBWwKfAVAEmrA1cAPwbWKed7PV9Q2p5qxOlgYN1y3RWSVmv27JIOkdQtqXvhk7ObXRIRERERMSAS2LTGJ4C3AifYfrCkHQocY3uq7aeB44APSlrF9iLb59ieW8t7bRn16fEL2zeXaxfU0s8F9pe0DvAG4Jf1htj+h+1rbD9j+1HgNGDnkv1GwMD3bD9r+2LgL7XihwA/sH2b7YW2e9YPbd/soW2fabvLdteQoSOaXRIRERERMSCy3XML2H5E0gzgrlryWOBXkhbVLwXWl/QYcDLwfmAU0HPNKKqRHYApvdzuOqoRoGOAK2wvkPR8pqQNqUaG3kg12rMKL0xVGw08ZNu1+ibXzjcB9pH0uVraasCYXtoSERERETEoZMRm+XkIeLvtkbVjDdsPU00h25NqlGcEsFkpo1p500QJSi6gmurWOA0N4BvAAmBb22sDB9bqnU4VFNWNrZ1PAY5vaPNQ25f275EjIiIiItojIzbLzxnASZI+YvtBSesDr7f9S6qRlAXA48BQ4MQlrPs04I+2/9QkbzjwKDBb0sbA52t5NwKrSPpkad97gdcAvy35ZwI/lfQHqk0J1gLeAvzB9nz6sO2YEXQP0h9rioiIiIgVX0Zslp9Tgd8B10iaC9zEC2tVzgamleOuktdvth+3fU0v2ccCOwCzqdbfXFYrt4AqmPko8ATw71Q7rfXk3wp8DPjvkv8PIL+PExERERGDnl683CJi6XR1dbm7u7vdzYiIiIiIFZykCba7GtMzYhMRERERER0vgU1ERERERHS8TEWLlijriO5tdztWAqOAGe1uxEog/Tww0s8DI/08cNLXAyP9PDAGcz9vYnu9xsTsihatcm+zuY7RWpK608/LX/p5YKSfB0b6eeCkrwdG+nlgdGI/ZypaRERERER0vAQ2ERERERHR8RLYRKuc2e4GrCTSzwMj/Tww0s8DI/08cNLXAyP9PDA6rp+zeUBERERERHS8jNhERERERETHS2ATEREREREdL4FNRERERER0vAQ28TxJ60j6haT5kiZL2ruX6yTpG5IeL8cpklTL307SBElPlv/drr9lVwat6GdJW0i6QtJjkmZKulLSv9TKHihpoaR5tWOXAXrEQaGF77NLHT39+MP+ll0ZtOh93qnhXZ1X+v19JT/vc//7+S2S/ihptqRJTfLHlfwnJd0jadeG/M9JeriU/7Gk1ZfTIw1KrehnSetLukjStJL/J0mvq+XvImlRw/t8wHJ+tEGlhe/zJElP1frxqob8vM/L/j6P7eXv8xElf3C9z7Zz5MA2wEXAJcAw4E3AbGDrJtcdCtwLvAwYA9wNHFbyVgMmA58DVgc+XT6vtriyK8vRon7eATgIWAdYFfgqcE+t7IHAje1+1k7v55JvYLNe7pH3uUX93HDtLsBcYK3yOe9z//t5B2A/4BBgUpP8m4FTgTWB9wGzgPVK3m7AI8DWwEuBa4Gvt/vZO62fgU2Bw4GNgCHlmhnAsJK/C/BQu5+10/u55E8Cdu3lHnmfW9TPDde+HFgIjCufB9X73PYG5BgcB7AW8AywRS3tvGZ/BICbgENqnw8Cbinn/wZMpey4V9IeBHZfXNmV4WhVPze5dh2qL+Drls8HshJ/EWxlP9N3YJP3efm8z2cDZ9c+533uZz/X8ndt/IICbAEsAIbX0m7ghX8wuRA4qZb3NuDhdj9/p/VzL9fNAV5bzndhEH0R7OR+pu/AJu/zcnifgWOBP9Y+D6r3OVPRoscWwELb/6il/ZXqXzoabV3yml23NfA3l7e9+FtDfm9lVwat6udGb6b6g/14Le3VkmZI+oekr0h6ybI0vMO0up+vL9MZfi5p3BKWXZG1/H2WNBR4P/CThqy8z/3r575sDTxge24v9TT7b7SBpHWX8D6dqlX9/CKqpmOvBtxXS15f0iOSJko6TdJay3KPDtPqfr5A1bTsqyT9ay097/NyeJ+B/fnnv8+D5n1OYBM9hlENUdbNBob349rZwDBJ6kc9fZVdGbSqn58n6WXA96mmPvS4HtgGWJ9qusl/AF9YppZ3llb2887AOOCVwDTg17Uv1XmfW/w+U72vM4Draml5n/vfz8tST7P/RizFfTpVq/r5eZLWpvpX8uNt99R9D7Ad1VS1twKvpZoeuLJoZT/vQ/X3eRPgj8CVkkb2cp+8z8v+Pu8EbAD8rJY8qN7nBDbRYx6wdkPa2lTz3Bd37drAvDJKs7h6+iq7MmhVPwMgaT3gKuAHti/qSbf9gO2JthfZvhM4gepfwVcWLetn29fbfsb2LOAzVPOLt+xP2ZVAS9/n4gDg3Hp63ucl6udlqafZfyOW4j6dqlX9DICkNYFfUU25PLkn3fbDtu8u7/NE4IvkfV6qfrb9J9tP2X6y9PEsYKde7pP3eRne5+IA4DLb83oSBtv7nMAmevwDeImkzWtp/wrc1eTau0pes+vuAl7V8K+wr2rI763syqBV/Yykl1IFNb+0feJi7mtgZRlFgBb2cxP1vsz73MJ+lrQx1Xztcxdz37zPS/eu3QVsKqn+L7aNf78b/xs90jDFdUXWqn6m7L51OdWa00MXc3ne59b97Vzc3+e8z0vZzyVQ/wD/PA2tUXvf53Yv8skxeA7gYqodNNYC3kjvu2ccBvydamej0VT/R9K4K9pnqHZF+yQv3hWt17Iry9Gifl4b+DNwei/32APYoJy/Evhf4Nh2P3sH9vPWVEPsQ6iG9b9NtbPXqosru7Icrejn2jXHANc3KZv3uf/9vAqwRumzyeV8tVr+LcA3S/p7efGuaLsDDwNbUe0i9QdWvl2klrmfqXaq/BVVYPOSJmV3AcZSffnbmGoK1dntfvYO7OexpexqJf0LwGO8sIlO3ucW/d0o1+xd8tSQPqje57Z3eo7Bc1DtrHU5MJ9qJ7O9S/pOVFNGeq4TcAowsxyn8OJd0F4NTACeAm4HXt3fsivD0Yp+phoOdqljXu0YW/K/SbXN5XzgAaqpO6u2+9k7sJ/fShXIzAceLfVt3p+yK8vRqr8b5Zp7gIOa3CPvc//7eZfyt6F+XFvLH0e17e1T5d3eteE+h5e+nkO1O93q7X72TutnqnV5Bp5s+Pu8U62Pp5b8KcD3qO1UtzIcLernrak2J5oPPA5cA3Q13Cfvcwv+bpRrrgS+2uQeg+p97vl/3hERERERER0ra2wiIiIiIqLjJbCJiIiIiIiOl8AmIiIiIiI6XgKbiIiIiIjoeAlsIiIiIiKi4yWwiYiIiIiIjpfAJiIiIiLsXnLIAAAAEklEQVQiOl4Cm4iIiIiI6Hj/H3Cm8MC9SmE1AAAAAElFTkSuQmCC\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plot_fi(rf_feat_importance(m, xs_imp));"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Removing redundant features"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One thing that makes this harder to interpret is that there seem to be some variables with very similar meanings. Let's try to remove redundent features. We can do this using \"*hierarchical cluster analysis*\", which find pairs of columns that are the most similar, and replaces them with the average of those columns. It does this recursively, until there's just one column. It plots a \"*dendrogram*\", which shows which columns were combined in which order, and how far away they are from each other.\n",
"\n",
"Here's how it looks:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArkAAAFoCAYAAABT1OOjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzde7hVVb3/8fcH8g6CiqIbEUw0UysqvJXKPmnHMk07p7RQ01JJzTqVlWmmWF6yU1lqZZiXvGyN8q6ZlzwbLS8JsvOXpoUKbtiIINeNiArf3x9jbJmt9mVt2JsFa31ez7Me5xpjzDG/c/k89fW7x5xDEYGZmZmZWTXpU+kAzMzMzMx6mpNcMzMzM6s6TnLNzMzMrOo4yTUzMzOzquMk18zMzMyqjpNcMzMzM6s6byt34KBBg2L48OG9GIqZmZlZz5g8efLciNiy0nFY5ZSd5A4fPpxJkyb1ZixmZmZmPULS9ErHYJXl5QpmZmZmVnWc5JqZmZn1IElXSzq3k/5WSW9fkzHVIie5ZmZmVnMkTZP0uqRBJe1NkkLS8N66dkT0i4jnV3eeYjItaXiOuzV/Zku6U9KHVz/idZOTXDMzM6tVLwCfafsi6V3ARpULp0cMjIh+wHuA+4BbJB1b2ZAqo+wHz8xqQUvLeGbPbqh0GGZmtmZcC3wWuCR/Pwa4Bmirjn4sH+8ALASuiIhxbSdL2gf4AbALsBj4TkRcnbs3k3QXsB/wNDAmIp7L5wWwY0RMlXQ1sAQY3sHYnXN87wfm5GtM6OrGIuIl4KeS1gMulHRNRKzo5u+zTnMl16xg9uwGWlubKh2GmZmtGY8Cm0p6p6S+wBHAdYX+JaQkeCDwMeAkSYcBSNoOuJuUgG4JjASK/wfyGeAcYDNgKnBeJ3G0O1bSJqRqbAOwVR73c0m7duMeb87nvqMb51QFV3LNSvTrN5L3vrex0mGYmdlqUbkD26q5E4FngJltHRHRWBj3pKQbgNHArcCRwP0RcUPufyV/2twcEX8BkHQ98ONOYuho7MHAtIi4Kn9/QtJNwCeBp8q8v5b8z83LHF81nOSamZlZLbsWeBDYnrRU4S2S9gS+D+wGrA9sAPw2dw8Fnutk3pcKx68C/VZh7DBgT0kLCv1vyzGXa0j+57xunFMVvFzBzMzMalZETCc9gHYQ6U/7RQ3A7cDQiBgAXMbKEnEzaa1ub2oGJkbEwMKnX0Sc1I05PgG8DDzbOyGuvZzkmpmZWa07DvhQRCwpae8PzIuI1yTtAYwp9F0PHCDpcElvk7SFpJE9HNedwE6Sjpa0Xv7sLumdXZ0oabCkU4CzgdNr7aEzcJJrZmZmNS4inouISe10nQx8V9Ji4CxgQuGcF0nV31NJSwGaSK/t6sm4FgP/CXyatLb2JeBC0rKJjiyQtAT4fzm+T0XElT0Z17pCEVHWwFGjRsWkSe39+zerHlOm1AP4wTMzs3WcpMkRMarScVjl+MEzW2uMHw8NFX5FbWvrT7j88q9UNggzMzNbbV6uYGuNhgZo8itqzczMrAe4kmtrlZEjobGxctefMsVVXDMzs2rgSq6ZmZmZVR0nuWZmZmarSdJwSSGpy7+SSzpW0p/WRFy1zEmumZmZ1RxJ0yS9LmlQSXtTTlaHr8FYQtKIfDxO0huSFufPPyRdKmmbNRVPtXCSa2ZmZrXqBeAzbV8kvQvYqHLhvOU3EdEf2Jy0Y9nWwGQnut3jJNfMzMxq1bXAZwvfjwGuafsiaYCkayTNkTRd0pmS+uS+vpJ+KGmupOeBjxUnzudeIWmWpJmSzpXUtzvBRcQbEfEUcAQwh7TxhJXJb1cwK9Ha2vTWphBmZlbVHgWOztvk/oOUTO4DnJv7LwEGAG8HtgDuBWYBVwAnAAcD7wWWADeVzP1rYDYwAtiEtEVvM/DL7gYZEcsl3QYc2N1za5mTXLOCwYPHdD3IzMyqSVs1dyLwDDAzt/clJb3vzdvrLpb0I+BoUpJ7OPCTiGgGkHQBUJ+PBwMfBQZGxFJgiaSLgLGsQpKbtZCWL1iZnOSaFdTVjaWubmylwzAzs9WmcgdeCzwIbE9hqQIwCFgfmF5omw4Mycd1pMpssa/NMGA9YJb0Vhx9SsZ31xBg3mqcX3Oc5JqZmVnNiojpkl4ADgKOK3TNBd4gJaxP57btWFnpnQUMLYzfrnDcDCwDBkXEm6sbY14HfAhw/+rOVUv84JmZmZnVuuOAD0XEkkLbcmACcJ6k/pKGAV8Drsv9E4AvS9pW0mbAt9pOjIhZpPW7P5K0qaQ+knaQNLo7QUlaL68XvoH0hoUfr+oN1iInuWZmZlbTIuK5iJjUTteXSA+VPQ/8CWgArsx9lwP3AH8FngBuLjn3s6TlDk8D84HfAeW+AuwISa3AAuB24BXg/RHRUu49GSgiyho4atSomDSpvX//Zj2jvj79s7GxklGYmVk1kDQ5IkZVOg6rHFdyzczMzKzqOMk1MzMzs6rjJNfMzMzMqo6TXDMzMzOrOk5yzczMzKzqOMk1MzMzK5OkkDSi0nFY15zkmpmZmfUgSYMlzZVUX9J+laQbKhRWzfG2vmZmZmY9KCJmS/oqcLmkd0fEUkn7Ax8Ddu3Ja0l6W09sHVyNXMk1MzOzqtEyvoUp9VPKGivpNEkzJS2W9Kyk/SXtIekRSQskzZJ0qaT1Ozh/A0k/lPSipNmSLpO0EUBEXAs8C3w3t/0S+HJEzMnnbivpFklzJL0g6YuFefeW9GghhoslrZf73paXTJwsaSrwzOr8XtXMSa6ZmZlVjdkNs1k4cWGX4yS9AzgF2D0i+gMHAtOA5cBXgUHA3sD+wMkdTHMhsBMwEhgBDAHOKvSfCHweuBH4W0TcmK/dF7gTeDyf82HgG7naC/Am8D85hg8CHwG+UHLtjwO7A+/q8mZrlJNcMzMzqyoDRg8oZ9hyYANgF0nrRcS0iHguIiZHxKMR8WZETCNVYEeXnixJwAnAVyNiXkQsBs4HPt02JiJmkJLeA4CTCqfvBWwaEedHxOsRMRW4ou3ciHg8Ih7LMTwPjG8nhvMjYn5ELC3nZmuR1+SamZlZzYmIqZK+AowDdpV0D/A1oB/wY2AUsDEpV5rczhRb5v7JKd8FQEDfknFPAfMjYlahbRiwnaQFhba+QCOApJ2BHwHvL8TwWMm8zWXeas1yJdfMzMxqUkQ0RMQ+pKQzSMsPfkFa57pjRGwKnEFKXkvNBZYCu0bEwPwZEBH9yrh0M/DPwnkDI6J/RByS+38J/A0YkWM4q50Yopu3W3NcybVeN348NDR0Pa6pCUaO7P14zMzM8prcIcCfgddICWsfoD+wCGjNFdWTgDml50fECkmXAxdJOiUiXpY0BNgtIu7p4vKPAK9LOhX4GfAGsAuwfkRMzjEsBJZIeidpPe7M1b7pGuNKrvW6hoaUwJqZma1FNgC+T6rIvgRsRarafh0YAywGLgd+08kcpwFTgUclLQLuB97R1YXzK78OAvYgPew2l1S93TQPORU4Jsfwyy5isA64kmtrxMiR0NjY+Zj6+jURiZmZGUTEk6Qks1QLsHNJ21tvTIgIFY5fIyXGZ3RynUZg23baZwJHdHDO/9FBspwT5PaWT1gJJ7lmZr2gZXwLsxtmVzoMs5rT2tRKv5HlLIu1auflCmZmvWB2w2xam1orHYaZWc1yJdfMrJf0G9mP9za+t9JhmNWUcnc7s+rnSq6ZmZmZVR0nuWZmZmZWdZzkmpmZmZVJUkga0QPzNEo6vidi6gmS6iXNqHQcPclJrpmZmVkPy0njCkmtJZ+9Kx1brfCDZ2ZmZma9oyUi/u0dubZmuJJrZmZmVWXhxIVljZN0mqSZkhZLelbS/pL2kPSIpAWSZkm6VNL6HZy/gaQfSnpR0mxJl0naqLvxStpB0gOSXpE0V9L1kgYW+qdJOl3S05LmS7pK0oa5b5CkO3O88yQ9JKlP7quTdJOkOZJekPTlwpwbSbo6z/c0sHt3417bOck1MzOzqjF4zGAGjB7Q5ThJ7wBOAXaPiP7AgaQtdpcDXwUGAXsD+wMndzDNhcBOwEhgBDCEwu5o3SDgAqAOeCcwFBhXMubIHOMO+Zpn5vZTgRnAlsBg0u5rkRPdO4C/5rj2B74i6cB83tl5rh3yvMesQtxrNSe5ZmZmVjXqxtaV+37q5cAGwC6S1ouIaRHxXERMjohHI+LNiJgG/BIYXXqyJAEnAF+NiHkRsRg4H/h0MZxcYS1+NimdKyKmRsR9EbEsIuYAP27nmpdGRHNEzAPOAz6T298AtgGGRcQbEfFQRASpMrtlRHw3Il6PiOeBywvxHQ6cl2NvBi4u50dbl3hNrpmZmdWciJgq6Sukiumuku4Bvgb0IyWZo4CNSbnS5Ham2DL3T075LpAqsn0LY8pakytpK1KSuS/Qn1SEnF8yrLlwPJ1U9QX433wP9+Y4xkfE94Fh5CS7cF5f4KF8XNfOnFXFSa6ZrRNaxrcwu2F2pcMoW2tTK/1G9qt0GGbWiYhoABokbUqq2F5ISv6mAJ+JiMU5Ef5kO6fPBZYCu0bEzNUM5QIggHdHxCuSDgMuLRkztHC8HdCS72ExacnCqZJ2Bf5P0uOkBPaFiNixg2vOynM+VZizqni5gpmtE2Y3zKa1qbXSYZhZlZD0DkkfkrQB8BopYV1OqqQuAlol7Qyc1N75EbGC9Of/i3IlFklDCmteu6M/0AoskDQE+EY7Y74oaVtJm5PW3f4mX/NgSSPy8olF+R6WA38BFuWH6zaS1FfSbpLaHjCbAJwuaTNJ2wJfWoW412qu5JrZOqPfyH7lrrWruCn1Uyodgpl1bgPg+6QHvd4AHgbGkh4gGw98k1TR/Q3woQ7mOI30oNmjkgYBM4FfAPfk/jpJpf91fkxE3FTSdg5wDbAQmApcS3r4ragBuJdUab4NODe370iq+m5JWuLw84hoBJB0CPAj4IV8v8+y8oG1c4DLcl8LcBXwPx3c5zrJSa6ZmZnVnIh4Etijna4WYOeStrfemBARKhy/RqqqntHO/I108hfziKgvHD8FvL9kyI9Kvj8eERe0M89FwEUdXKOFlQ+olfa9Cny2pPl/O4p3XeTlCmZmZmZWdZzkmpmZmVnV8XIFMzMzs7VYRAyvdAzrIldyzczMzKzqOMk1MzMz6wZJ0yQdUOk4rHNOcs3MzKwmSdpH0sOSFkqaJ+nPhffI9va1GyUdn4/rJa2Q1Jo/MyRNWFOxVCsnuWZmZlZz8i5ndwKXAJsDQ0jvjl1WoZBaIqIfaWOIvYBngIck7V+heNZ5TnLNzMysFu0EEBE3RMTyiFgaEfdGxJOSdpD0gKRXJM2VdL2kge1NIqmPpG9Jei6Pn5B3JUPShpKuy+0LJD0uaXBnQUUyIyLOAn5F2mrYVoHfrmBm1ktam1q985nZ2usfwHJJvwZuBB6NiPm5T8AFwIPApsBNwDjgK+3M82XgMGA0MAe4GPgZaROGY4ABwFBShXgkafvgct0MnCxpk4hY0p2bMye5tpZpaoL6+kpHYWuj1qYRXD5yaqXDKNvgMZ0Wa8yswiJikaR9SFvzXg5sLen3wAkRMZW0vS7AHEk/Bs7uYKovAKdExAwASeOAFyUdTdoueAtgRN5hbXI3w2whJdwDASe53eQk19YaY8ZUOgKznlM3to66sXWVDsOsdqnrIRHxd+BYAEk7A9cBP5H0P6SK7L6kNbJ9gPkdTDMMuEXSikLbcmAwcC2pintjXu5wHfDtiHijzLsYAgSwoMzxVuAk19YaY8emj1l7ptSvO1VcM1v3RMQzkq4mVWYvICWX746IVyQdBlzawanNwOcj4s8d9J8DnCNpOPB74FngijLD+gTwhJcqrBo/eGZmZmY1R9LOkk6VtG3+PpS0jvZRUvW2FVggaQjwjU6mugw4T9KwPM+Wkg7Nx/8h6V2S+gKLSMsXlncRlyQNkXQ2cDxwxmrdaA1zkmtmZma1aDGwJ/CYpCWk5PZvwKmk6uv7gIXAXaQHwDryU+B24F5Ji/M8e+a+rYHfkRLcvwMTSUsW2lMnqZWUXD8OvAuoj4h7V/UGa52XK5iZmVnNiYiZwOEddD8FvL+k7UeFc4cXjlcAP86f0mvcANzQwfXrC8eNuPDY4/yDmpmZmVnVcZJrZmZmZlXHSa6ZmZmZVR0nuWZmZmZWdZzkmpmZmVnVcZJrZmZmto6RdF3eQtg64CTXzMzMaoqk6yVdWdI2WtIrkrbp4WtdJykkHVTSfmluP6onr2crOck1MzOzWvNl4CBJHwaQtCFwOXBqRMzqqYvknc4A/gEcU2hfD/hv4Pmeupb9O28GYWbrjNamVqbUT6l0GGa2jouIVyR9CRgvaTfgTOC5iLhaUh/gW8BxwADgfuCkiJif+yYA+wAbAk257++QqrakXdJ2APYFPpYveStwrKQBEbEwt08CtmyLSdKOwHjg3UAAdwOn5PFIej9wRZ77TkpyOEkfB74HDCPt3HZiRPyth36ydZIruWa2Thg8ZjD9RvardBhmViUi4rfAZNKOZGOBL+Sur5GS0P2AbYElwMWFU+8EdiRt2fs34NqSqceQtgXuDzyS25aStgdu22Hts8A1JecJOBfYBtgFeDvwHQBJGwC3AVcCm+fjw946UdqdVIk+Htgij7tN0vrl/RrVyZVcM1sn1I2to25sXaXDMLN1hcoa9UXgOeDbEfFibvsCcHze9pf8cNdUScfkLXyvfusSqW+OpE0iYkluviUi2pLbZdJbgVwDfE/STcAHgM8Ap7Z1RsQ/SMsaAF6WdBFwWv7+QVJ195KICOBGSV8r3MdY4OcR8Xj+fqWkbwO7A38u65eoQk5y15Dx46GhodJRVEZTE4wcWekozMzM/lVEzJY0F3iq0LwdcIekFcWhwFaS5gAXAJ8EBgFtYwaRKr4AzR1cbiKpMnwGcFtEFBNgJG1Nqhh/kFQF7gPMyd11wIyc4LaZXjgeBhwp6auFtvWBIR3EUhO8XGENaWhIyZ6ZmZmt1WYAH46IgYXPhhHxEmmZwUHAh0jrdUfkc4p146AdOUG9nrQconSpAsCFwDLgXRGxKXBsYd5ZpAS5aLvCcTNwTknMG0fEhPJuuTq5krsGjRwJjY2VjmLNq6+vdARmZmZluww4X9LnIuJFSVsBe0XE7aQK6zLgFWBj4Lxuzn0R8H8R0d4Sgv7Ay8BCSUOBrxf6/gT0kXRKju8TwPtID6dBemDtt5IeID3QtgnwH8ADhWUUNceVXDMzM7OVfgz8AfijpMXAw6S1rQBXAS3581TuK1tEvBIRf+yg+2xgD9LbGW4Hbiqct4yU2J4AzAf+i/TGhrb+x4CTgF/k/n8ANf/+Xf3r8o6OjRo1KiZNmtTL4VSvtmpmLVdya/HezcysMiRNjohRlY7DKseVXDMzMzOrOk5yzczMzKzqOMk1MzMzs6rjJNfMzMzMqo6TXDMzMzOrOk5yzczMzKxbJF2XtzVeaznJNTMzs5ol6dOSHpO0RNLL+fhkFffcXQdI2k5Sa+ET+Z7avu9b6RjXNCe5ZmZmVpMknQr8FPhfYGtgMHAi8EFg/XbG912jAbZDUru71UbEixHRr+2Tm99TaHuonbkqfj+9yUmumZmZ1RxJA4DvAidHxO8iYnEkUyLiyIhYJulqSb+Q9HtJS4D/kDRA0jWS5kiaLulMSX3ynOMkXVe4xvBcUX1b/n6spOclLZb0gqQjC2M/L+nvkuZLukfSsEJfSPqipH8C/1yNe75O0s8k/SHfz76SPi6pKcf0oqTvlJyzn6RHJS2U1Czp6Hbm3VTSg5IuUnJwvpfFkmZI+uqqxrw62v2vATMzM7N1Vf2UKeUM2xvYALiti3FjgIOAg0nV3fHAAODtwBbAvcAs4IrOJpG0CXAxsHtEPCtpG2Dz3HcYcAZwCCmJ/RZwA/CBwhSHAXsCS8u5uTLu5zFgPVLV+ijgaeBdpO2Mp0TEnZK2B+4CjgNuBgYC25bc1yDSNsh3RsS43HYVcGhEPCxpc2D4asa8SlzJNTMzs1o0CJgbEW+2NUh6WNICSUsl7Zebb4uIP0fECuAN4Ajg9Fz5nQb8CPi36mYHVgC7SdooImZFxFO5/QvABRHx9xzP+cDIYjU398+LiNVNcm+JiEciYkVELIuIByLib/n7X4EbgdF57FHAHyJiQkS8GRFzI6KpMNcQYCJwfVuCm70B7CKpf475idWMeZU4yTUzM7Na9AowqLjGNSI+EBEDc19bjtRcOGcQqZo7vdA2nZTsdSoilpAS5BOBWZLukrRz7h4G/DQn2AuAeYBK5m2mZ/zLPJL2ltSYl18sBI4n3SfAUOC5Tub6OKkafHlJ+ydy34t57j17JvTucZJrZmZmtegRYBlwaBfjonA8l1SlLFZYtwNm5uMlwMaFvq3/ZaKIeyLiw8A2wDOsTA6bgS9ExMDCZ6OIeLiDOFZH6Tw3AjcBQyNiAPArUoLdFtcOncx1GfB/wF2S3rrviHgsIj4ObAXcma+xxjnJNTMzs5oTEQuAc4CfS/qkpH6S+kgaCWzSwTnLgQnAeZL65+UEXwPaHjZrAvbLr/MaAJzedq6kwfkhr01IyXUrsDx3XwacLmnXPHaApE/1+E23rz8wLyJek7QX8OlC33XARyT9t6S3SRok6T2F/iBVpp8Hbpe0oaSNJI2RtGlEvAEsZuV9rlFOcs3MzKwmRcQPSEnqN4GXgdnAL4HTgIc7OO1LpIrt88CfgAbgyjzffcBvgCeByaQqZps+wKlAC2k5wmjg5HzeLcCFwI2SFgF/Az7aQ7fZlZOACyQtJj38NqGtIyJeID0Md1qO+QnSw2kUxgTpwbSXgVtIyzmOAabnezmO8tcs9yil2Lo2atSomDRpUi+HU73q69M/GxsrGUVl1PK9m5nZmjW+pYVvPvccC/fbb3JEjKp0PFY5ruSamZlZ1WiYPZuFyyvy13FbyzjJNTMzs6oyoG/1buQlad+S7Xvf+lQ6trWNN4MwMzMzW0fk7Xn7dTnQXMk1MzMzs+rjJNfMzMxsNUgaJ+m6rkeu8vwhaUQ+vkzSd3rrWh1c/2pJ567Ja/YEJ7lmZmZWcyRNk3RASduxkv5UqZjKEREnRsT3enrefO/LS9b5XtrT11mTvCbXzMzMrJdIeltEvFnpOMr0SETsU+kgeooruWZmZmYFkr4h6aaStksk/SQfby9poqTFku4DBhXGDc/LC46T9CLwQG7/raSXJC2U9GDb7ma5r1HS8YXvHVaUS5cOSDpUUpOkRZKek/SRwhzP5xhfkHRkz/w6IOkESVMlzZN0u6S63H6OpEvy8XqSlkj6Qf6+kaTXJG3WU3F0xUmumZmZVZWR/Vb75QNt29kOhFSNBY4Ars39DaQdzQYB3yPt8FVqNPBO4MD8/W5gR2Ar0s5h169ukJL2AK4BvgEMBPYDpuWtgy8GPhoR/YEPkLYcXm2SPgRcABwObANMB27M3ROB+ny8O/AS6XcA2Bt4NiLm90Qc5fByBTMzM6tVt0oqLiVYH3giImZJehD4FHA58BFgbkRMlrQdKYE7ICKWAQ9KuqOducdFxJK2LxFxZduxpHHAfEkDImLhasR/HHBl3k4YYGaefxNgBbCbpBcjYhYwq4z59pK0oPD9IxHxaMmYI/M1n8jXOj3fy3DgEWBHSVuQEu4rgJMl9SMluxNX4R5XmSu5ZmZmVqsOi4iBbR/g5ELfr4Gj8vFRrKzi1gHziwksqZpZqrntQFJfSd/PywkWAdNy16B2zuuOocBzpY05tiOAE4FZku6StHMZ8z1a/D3aSXAh3f9b9xsRrcArwJCIWApMIiW0+5GS2oeBD1KBJNeVXLMaMr6lhYbZsysdhplZr2lqbe2J5QoAtwK/kLQbcDDwzdw+C9hM0iaFRHc7IErOL34fAxwKHEBKcAcA8wHl/iXAxoXxW5cZYzOwQ3sdEXEPcI+kjYBzSRXpfcuctzMtwLC2L7lqvAW5ikxKZD8EvBd4PH8/ENgDeLAHrl82V3LNakjD7Nk0tXrnRzOzrkTEa8DvSOtv/xIRL+b26aRq5TmS1pe0D3BIF9P1B5aRKp4bA+eX9DcB/yVp4/w+3OPKDPMK4HOS9pfUR9IQSTtLGizp4zkBXQa0AsvLnLMrDfmaIyVtkO/lsYiYlvsnAp8Fno6I14FG4HjghYiY00MxlMWVXLMaM7JfPxrf+95Kh2Fm1ivqp0zpyel+TUrQPl/SPib3zSOtQ72G9OBXR64hVTNn5nO+A5xU6L+ItM53NvAk6aG0A+hCRPxF0ufy+dvn878ILAROJS2xCFISfXJH83RHRPwxb0ZxE7AZaTnCpwtDHgY2YmXV9mngNdZwFRdAEaXV9faNGjUqJk2a1MvhVK/6+vTPxsZKRlEZtXzva5u2//F3kmtm1artf+cmvu99kyNi1OrMlR8yewbYOiIW9UB4tgZ5uYKZmZlZCUl9gK8BNzrBXTc5yTUzMzMryGtZFwEfBs6ucDg9RtJlJdv2tn0uq3RsvcFrcs3MzMwK8lsTeuQVDWuTiDiR9FqxmuBKrpmZmZlVHSe5ZmZmZmuQpMivCrNe5CTXzMzMapakMZIm5bWpsyTdnd99u86RdIakF/K9zJD0m9Wcr17SjJ6Kb01zkmtmZmY1SdLXgJ+QNjQYTNq57Oek3cnWOpI6fJZK0jHA0cABEdEPGAX8cU3FtjZykmtmZmY1R9IA4LvAFyPi5ohYEhFvRMQdEfENSRtI+omklvz5Sd7hC0nHSvpTyXxvLUGQdHV+k8F9khZLmihp2L9HAfk6P5T0oqTZ+byNcl99rsieJukl4KpObml34J6IeA4gIl6KiPF5nk9Jmlxy3VMl3ZqPD5L0dI51pqSv5zdM3A3UFd7CUJd3VvuWpOckvSJpgqTN8zzD8+/wOUnNkuZLOlHS7pKelLRA0qXd/Fe1ypzkmpmZWVWZuHBhOcP2BjYEbumg/9vAXsBI4D3AHsCZ3QjjSOB7wCDSjmPXdzDuQmCnfJ0RwBDgrEL/1sDmwDBgbCfXexT4rKRvSBolqW+h73Zge0nvLLQdRdoRDdL2wF+IiP7AbsAD+Q0THwVaIqJf/rQAXwYOA0YDdcB84GclsewJ7AgcQaqUf5u0g9uuwOGSRndyHz3GSa6ZmZlVjTGDBxBFB8EAACAASURBVDN6wIByhm4BzI2INzvoPxL4bkS8HBFzgHNIywHKdVdEPBgRy0hJ3t6ShhYHSBJwAvDViJgXEYtJSyeK2+SuAM6OiGURsbSji0XEdcCXSNsHTwRelvSt3LcM+A0psUXSrsBw4M58+hvALpI2jYj5EfFEJ/f1BeDbETEjzzsO+GTJUorvRcRrEXEvsAS4If+OM4GHgDWy7abfk2trRFPTyu19rXKaWkcw8vKplQ7DzKzXjK2rY2xdHep66CvAIElv6yDRrQOmF75Pz23lam47iIhWSfPy+c2FMVsCGwOTU74LgIBiFXZORLxWzgUj4nrgeknrkaqt10uaEhH3AL8GbpB0JilZn5CTVID/JlWpvy/pSeBbEfFIB5cZBtwiaUWhbTlpTXOb2YXjpe18XyPvIHYl13rdmDEwcmSlozAzM/sXjwCvkZLB9rSQEro22+U2SNXJjds6JG3dzvlDC/39SEsOWkrGzCUlfbtGxMD8GZAfHGsTZdzLv8hri38LPElafkBEPAq8DuwLjGHlUgUi4vGIOBTYCrgVmNDJtZuBjxbiHRgRG+Yq7VrFlVzrdWPHpo9VXv0UV3HNzAAiYqGks4CfSXoTuJf0Z/sDgP8AbgDOlPQ4Kdk7C7gun/5XYFdJI4FnSH+yL3VQfhXZX0hrcx+LiGIVl4hYIely4CJJp0TEy5KGALvl6mvZJB0LzAEeJCXhB5LWwD5WGHYNcCnwZkT8KZ+3PvAp4M78mywiVWYhVWC3kDQgItoWOl8GnCfpmIiYLmlL4AMRcVt34l0TXMk1MzOzmhQRPwa+RvpT/RxSlfIUUjXzXGASqRr6/4AnchsR8Q/SmxnuB/4J/Kl0bqABOBuYB7yftMa3PacBU4FHc4J5P/COVbidRcAZwIvAAuAHwEltyWx2Lamye23JuUcD0/L1TySv3Y2IZ0jJ/vP5zQh1wE9JD7LdK2kx6YG3PVch3l6niPKq4KNGjYpJkyb1cjjVq209amNjJaOwWlc/ZQoAje9dI2v+zcwqRtLkiBhVoWtfDcyIiO68jaHX5VeTvQy8LyL+Wel4epsruWZmZma14STg8VpIcMFrcs3MzMzWCZLOIC1JKPVQRHy0i3Onkd7c0NGDdlXHSa6ZmZlZD4qIY3tp3vNJ79FdlXOH92w0az8vVzAzMzOzquMk18zMzMyqjpNcMzMzs9UkqV7SjErHYSs5yTUzM7OaI2mapKWSWgufSysdl/UcP3hmZmZmteqQiLi/0kGsKkl9I2J51yNrk5NcsxrT1Nr61qYQZmb2r/L2uMeTdvI6jrR72MkRcXfu3xz4EWnb3I2AiRHxb6/lkvRO4BfASGAmcHpE3J77DgJ+CAwl7VR2UUT8sO3aEbFPYZ4AdoyIqXmTiaXAMGA0cKikh4DzgMOBDYBbgK9GxNIe/FnWSV6uYFZDxgwezMh+/SodhpnZ2m5P4FlgEGl73CskKfddC2wM7ApsBVxUerKk9YA7gHvzmC8B10tq2673CuALEdGftM3uA92IbQwpqe1P2k74QmAnUjI9AhgCnNWN+aqWK7lmNWRsXR1j6+oqHYaZWa9T10MAbpX0ZuH7N4A3gOkRcTmApF8DPwcG50T3o8AWETE/nzOxnXn3AvoB34+IFcADku4EPgOMy9fYRdJf8zzz25mjI7dFxJ9zbMuAE4B3R8S83HY+0ACc3o05q5IruWZmZlarDouIgYXP5bn9pbYBEfFqPuxHWl4wr5DgdqQOaM4JbpvppCorwH8DBwHTJU2UtHc3Ym4uHG9JqipPlrRA0gLgD7m95jnJNTMzMytPM7C5pIFdjGsBhkoq5lnbkdbmEhGPR8ShpKUMtwIT8pglpKQVAElbtzN3FI7nktbo7lpI1AdEhNel4STXzMzMrCwRMQu4G/i5pM0krSdpv3aGPkZKWL+Zx9QDhwA3Slpf0pGSBkTEG6QHz9rekPBXYFdJIyVtSFra0Fk8K4DLgYskbQUgaYikA1f/btd9TnLNzMysVt1R8p7cW8o452jSmtpngJeBr5QOiIjXgY+T1u/OJa3p/WxEPFOYY5qkRcCJwFH5vH8A3wXuB/5JerCsK6cBU4FH83z3A+/o/JTaoIjoehQwatSomDRpUi+HU73q69M/GxsrGYWZmVltkDQ5IkZVOg6rnLXu7Qrjx0NDQ6Wj6HlNTTByZKWjMDMzM6sNa91yhYaGlBCamZmZma2qta6SC6niWW1/1m9brmBmZmZmvW+tq+SamZmZma0uJ7lmZmZmq0HSOEnX9eL8IWlEPr5M0nd661rVxEmumZmZ1RxJ0yQdUNJ2rKRyXttVMRFxYkR8r9JxrAuc5JqZmZn1Eklr5fNPtcBJrpmZmVmBpG9Iuqmk7RJJP8nH20uaKGmxpPuAQYVxw/PyguMkvQg8kNt/K+klSQslPShp18I5jZKOL3zvsKIs6WpJ5xa+HyqpSdIiSc9J+khhjudzjC9IOrJnfp11h//rwsx6VrW+7NrMasl1wDhJAyNiQa7GHkHawQygAXgE+E9gT+Au4LaSOUYD7wRW5O93A58HXgcuBK4HVusN+pL2AK4BPgn8EdgG6C9pE+BiYPeIeFbSNsDmq3OtdZGTXDPrWW0vu/buJ2a29rtV0puF7+sDT0TELEkPAp8CLgc+AsyNiMmStgN2Bw6IiGXAg5LuaGfucRGxpO1LRFzZdixpHDBf0oCIWLga8R8HXBkR9+XvM/P8m5CS690kvRgRs4BZq3GddZKTXDPredX4smszW7dI5Yw6LCLuX3mKjgXalg38GjiJlOQeBVyb2+uA+cUEFpgODC2Zu7kwb1/gPFLSvCUrq7uDgNVJcocCvy9tjIglko4Avg5cIenPwKkR8cxqXGud4zW5ZmZmZv/uVuDdknYDDiYtL4BUEd0sV0vbbNfO+VE4HgMcChwADACG5/a2THwJsHFh/NZlxtgM7NBeR0TcExEfJi1heIaUrNcUJ7lmZmZmJSLiNeB3pPW3f4mIF3P7dGAScI6k9SXtAxzSxXT9gWXAK6Rk9vyS/ibgvyRtnN+He1yZYV4BfE7S/pL6SBoiaWdJgyV9PCfiy4BWYHmZc1YNJ7lmZmZm7fs18C5WLlVoM4b0wNk84GzSw1+duYa0pGEm8DTwaEn/RaQH0mbna15PGSLiL8Dn8vkLgYnAMFJ+dyrQkmMcDZxczpzVxGtyzczMrOZExPB22q4Gri40vQgsBW4qGfc8sG8H805j5TKEtrZW0nKFomsK/XNJb2ooGlfoV+H42JK5bwFuaSeU0e3FV0uc5Jp1xK/CWjV+s4KZVQFJfYCvATdGxKJKx2Pd5+UKZh1pexWWmZnVlLyWdRHwYdJyBFsHuZJr1hm/Cqv76usrHYGZ2WrJrwfrV+k4bPW4kmtmZmZmVcdJrpmZmZlVHSe5ZmZmZj1EUr2kGRW47hmSfrWmr7s2c5JrZmZmNUfSNElLJbVKmi3pKklrzTpcSY2Sji9pO1RSk6RFkuZK+qOk4QARcX5EHN/eXLXKSa6ZmZnVqkMioh/wPmB34Mxip5K1IlfKO6FdQ9rkYQCwPfBzYEUl41qb+e0KZtbzmpr8lgUzW2dExExJdwO7SWoE/gzUk5Lfd0l6FbgM2Ie0g9iFEXE5gKSNgF+QNnuYBVxVnFtSADtGxNT8/WpgRkScmb8fCpwDvB2YA3yRtNHEvsBekn5C2qCiEXghIv6Yp15MYZMKSeOAERFxlKRLgWMLYWwInBsR4yTVAZcA+5G2+70oIi5etV9u7eYk18x61pgxlY7AzKxbJA0FDgJuJiWXRwMfBZ4l7V52P/AUUAfsDNwn6fmccJ4N7JA/mwB3d+O6e5Cqs58E/ghsA/SPiD9I+iBwXUT8Ko99O7CzpIuA24HH805q/yYiTgFOyeeNBO4DbstV6TuA24DPANsC90t6NiLuKTfudYWTXDPrWWPHpo+ZWSVJXY+BWyW9CSwE7gLOJyWpV0fEU2kaDSVVcA+OiNeApvyA19GkxPRw4OSImAfMk3QxcFaZUR4HXBkR9+XvMzsaGBHPS6on7cI2Aegv6UbglI6SXUlbArcCX4qIKZL2BLaMiO/mIc9Luhz4NOAk18zMzKxKHBYR9xcblJLj5kJTHTAvIhYX2qYDowr9zSV95RoK/L7cwRHxKCmpRtLuwG+AbwOnl46VtB7wO6AhIm7MzcOAOkkLCkP7Ag91I+Z1hpNcWyuNH5921a2opp/QOPIrFQ7CzMwqIArHLcDmkvoXEt3tWFl1nUVKVp8q9BW9Cmxc+L410PaKsWbSMoeuYvj3zojHJd0M7NbBkEtI63aLD9M1k9b17tjZ3NVirXhi0KxUQ0N6dsnMzKySIqIZeBi4QNKGkt5NWmZwfR4yAThd0maStgW+VDJFEzBGUl9JHwFGF/quAD4naX9JfSQNkbRz7ptNehgNAEn7SDpB0lb5+87Ax4FHS2OW9IV8nTERUXz7wl+ARZJOk7RRjmm3XBWuOq7k2lpr5EhobKxgAPWu4pqZGZAe0rqMVNWdD5xdWEd7Tu57IfdfBfxP4dz/AX5NemvCrfkDQET8RdLngItIrwSbncc9A/wU+LWkk4BrgfGkpPZcSZsAc0nLFX7QQbxvB1q0cm3y+RFxvqRDgB/leDcgPVx3ZjtzrPOc5JqZmVnNiYjhHbTXt9M2Azi4g/GvAp8taf7fQv8kYNdO4rgFuKWd9keAnUqaD+lknnGF4/pOxrWQkuCq5+UKZmZmZlZ1nOSamZmZWdVxkmtmZmZmVcdJrpmZmZlVHSe5ZmZmVnMkvUPSFEmLJa2Q9J1evt44SdeVObZR0vG9Gc+qkPRU3nVtneAk18zMzGrRN4HGiOgfEX0i4nsAkuolRd5o4S2S3pPbG9dkkJKOlbRcUmv+vCDpKkmlb17odRGxa0Q0ljNW0jRJB/RySJ1ykmtmZma1aBgrdykrNQf4gKQtCm3HAP/o9aja90hE9AMGAAcAS4HJkjra7axHSVqjr5ztqes5yTUzM7OaIukB4D+AS3N1tEHSuYUhr5M2bfh0Ht8XOJyVu5y1zfMBSY9LWpj/+YFC3/aSJublEPcBg0rO3UvSw5IWSPprOcsAImJ5RDwXEScDE4Fx5cyXq8HP51hekHRkoe8ESX/PfU9Lel9un5Z3RnsSWCLpbcXqbF5+8TtJv8nnPiHpPbnvWtL2xnfk3/ebuf3jecnDgrwk452FOP7tel39Hl3xZhBmnWlqgvr6SkdhZmY9KCI+lJcdXBcRv5J0dTvDriHtRPYz4EBS1belrVPS5sBdwJeBG4BPAXdJGhERrwANwCPAfwJ75rG35XOH5O9HA38A9gdukrRzRMwp8zZuBi7oaj7gVeBiYPeIeFbSNsDm+bxPkRLlw4BJwA7AG4VrfAb4GDA3It4s7J7W5tA85ijSzm63StopIo6WtC9wfETcn6+1U/6dDgMaga+SkuBdIuL19q5X5u/QIVdyzToyZkzaW9jMzGpORDwMbC7pHaQdza4pGfIx4J8RcW1EvBkRN5C24z1E0nbA7sB3ImJZRDwI3FE49yjg9xHx+4hYkbcIngQc1I0QW8jJahnzrQB2k7RRRMyKiLZlGscDP4iIxyOZGhHTC9e4OCKaI2JpBzFMjojfRcQbwI+BDYG9Ohh7BHBXRNyXx/8Q2Aj4QGFMV9frFldyzToydmz6mJnZuuffq46r4lrgFNLShs8DYwp9dcD0kvHTgSG5b35ELCnpG5qPhwGfklTcpnc94P+6EdsQYF5X80XEEklHAF8HrpD0Z+DUiHgmx/NcJ9do7iKGt/ojYoWkGaR7b8+//F55fHO+j3Kv1y1Ocs3MzMzady0wFbgmIl4t+XN9Cym5LNqOtFxgFrCZpE0Kie52QOTjZuDaiDhhNWL7BPBQOfNFxD3APZI2As4FLgf2zeft0Mk1opM+WJm0I6kPsC0rl3SUntsCvKswXvn8md24Xrd4uYKZmZlZOyLiBWA08O12un8P7CRpTH4o6whgF+DO/Cf/ScA5ktaXtA9QrLJeR1rWcKCkvpI2zK8u27azePLY7SVdAtQD53Q1n6TB+YGvTYBlQCuwPJ/3K+Drkt6vZISk0sS9M++X9F/5IbGv5PkfzX2zgbcXxk4APiZpf0nrAafm8Q9343rd4iTXzMzMrAMR8aeIaGmn/RXgYFKy9grpvbsHR8TcPGQM6YGzecDZFNb0RkQz6aGtM0ivK2sGvkHHedneklqBRaSHtjYlPUj2/8qYr0+OsSXHMho4OZ/3W+A80kNyi0lvlGhb51uO20hrbeeTHnr7r7zeFtJDcWfmNyl8PSKeJa0dvgSYS0r6Dyk8dNbjFFFeZXjUqFExadKk3orjLW0Psjc29vql1qhqva/e4t/LzMxWh6TJETGq0nFUK0njgBERcVSlY+mIK7lmZmZmVnWc5JqZmZlZ1fHbFczMzMysWyJiXKVj6IoruWZmZmZWdZzkmpmZmVnVcZJrZmZm1kPy+2lnVDoOc5JrZmZmNUjSNElLJbVKmi3pKkn9Kh1XG0mNko4vaZOkUyQ9KelVSS/lcZ+uUIz1klbk37BV0gxJEyTtXol4SjnJNTMzs1p1SET0A94H7A6cWezMSeXalCtdTNpZ7FRgC2AIKeaPtDd4DcXfkn/D/sBewDPAQ5L27+Xrdmlt+hdnZmZmtsZFxEzgbmC3XBk9T9KfgVeBt0uqk3S7pHmSpko6oe1cSRtJulrSfElPk5JlCv0haUTh+9WSzi18P1RSk6RFkp6T9BFJ5wH7ApfmCumlknYi7VT26Yi4LyKWRsTyvCPbsYX5uht/aTz/stwiV7xPl/R0vserJG3Yzm8YETEjIs4ibRd8YWGOnSXdl6//rKTDC30H5bkXS5op6eud/TZd/bss8ivEzMzMrKrUX13frfGShgIHATeTksujgY8CzwIC7geeAuqAnYH7JD0fEX8kbdm7Q/5sQkqWy73uHqTtfj8J/BHYBugfEX+Q9EHguoj4VR57ItAcEeVsP9ud+MtxJHAgsAS4g1Q9PrOT8TcDJ0vaJH+/Dzgrx/Ru4F5JT0XEU8AVwOER8ZCkzYDt8/22+9uUGS/gSq6ZmZnVrlslLQD+BEwEzs/tV0fEUxHxJrA1sA9wWkS8FhFNpErl0Xns4cB5ETEvIppJSwrKdRxwZa7MroiImRHxTAdjBwEvFRvyGtgFkl6TNKzQ1Z34y3FpRDRHxDzgPOAzXYxvISXXA4GDgWkRcVVEvBkRTwA3kZJXgDeAXSRtGhHzcz9077dpl5NcMzMzq1WHRcTAiBgWESdHxNLc3lwYUwfMi4jFhbbppPWwbf3NJX3lGgo8V+bYV0jVzLdExLak5HcDUlLZpjvxl6P0/uq6GD8ECGABMAzYMyfjC/J/VBxJSr4B/ptURZ8uaaKkvXN7d36bdjnJNTMzM/tXUThuATaXVPxT+XbAzHw8i5SQFfuKXgU2LnzfunDcTFrm0FUMAA8A20oa1Unc7Z3bVfxLOomvTen9tXRx/U8AT0TEEtI9Tsz/MdH26RcRJwFExOMRcSiwFXArMCHP0dlvUxYnuWZmZmYdyEsQHgYukLShpHeT/pR+fR4yAThd0maStgW+VDJFEzBGUt/84NToQt8VwOck7S+pj6QhknbOfbOBtxfieBb4JXCjpA/nB976Ah9YzfibgIMkbS5pa9LbG0p9UdK2kjYHzgB+Uzogv8lhiKSz+f/t3XmYXFW1/vHvmzAJCUMIEhLCIAGUQaI0CAjSODEGcEAgQAAJuaAoXiYVURIBB36KTF4VUAIJDU7AFQgyiB2ZFDragEzKEGhIgkAIGQhTsn5/7F2kqFudrkrSVd3V7+d56knV2fucs85OP+mVXeucDWNzP4AbgS0kHSFp5fzaQdIHJK0i6TBJa0XEW8BcYFEFY1MRJ7lmZmZmS3cosAlpBvM64MyIuC23TSB9hf80cCswqWTfE4FRpK/uDyPNVgIQEfcBRwM/AV4l1QUXamsvAD6fn2hQqPP9Mqnm9zxgNvAccBZwMPDsMsY/CXgAmJ7j/z8JLNCS257Kr7OL2oZKmg/MB+4HtgWaI+LWfI3zgE8Dh+TzzyI9eWHVvP8RwHRJc4HjgMMrGJuKKKJ0Nry8pqamaGur5Ia+5dPcnP5sbe32U9VUo15Xd/F4mZnZsmqe2MzUo6dOi4hKvtq3pZA0HRgbEbfXO5Zq+RFiZmZmDe6SaZfQ8lBLvcOomfZZ7fUOwXoAlyuYmZk1uJaHWpz4WZ/jmVwzM7M+YOSQkbQe1VrvMGqieWIzU5la7zAaQkRsUu8YlpVncs3MzMys4Xgmt4ba25fcUGVL194OI0fWOwozM7PqSGomLce7Yb1j6es8k1sjo0c7aTMzM+spJE2XtFDSfEkvSLpc0oB6x1UgqVXS2JJtknSCpAclvSZpVu53SL3i7Mk8k1sj48all1XGM95mZlYDoyLidknDgFuAM4BvFBolifS41cX1CrDEhcDewPHAXcCbwM6kxReuKe3cA+OvKc/kmpmZWZ8WEc8DNwPb5JnRcyTdTVqS932Shkr6g6TZkp6QdGxh37zy2MS8aMMjwA7Fx5YUkkYUfZ4o6eyizwdIapc0V9KTkvaSdA6wG3Bxnmm+WNIWwJeAQyLitohYGBGLIuKuiDiq6HjVxl8aT7Ok54o+T5f0TUmP5Gu8XNJqyz/q3c8zuWZmZtanSRoO7ANcS0oujyDNmD4OCLgdeBgYCrwfuE3SUxHxJ+BMYLP8WoOULFd63h2BK4HPA38CNgAGRsQfJX2UVNt7We57HNAREZWszFVN/JU4DNgTWADcQJrxPqPCfevGM7lmZmbWUKp4VNr1kuaQvvqfCnwvb58YEQ9HxNvAEGBX4OsR8XpEtAOXkRJJgC8A50TE7IjoIJUUVOoY4Fd5ZnZxRDwfEY910ncwaUncd0h6TtIcSa9LKl7ytpr4K3FxRHRExGzgHNIywT2eZ3LNzMysrzqwdLnaVMZKR9GmocDsiJhXtO0ZoKmovaOkrVLDgSkV9n2ZNNP7jojYUNJKwFukGduCauKvROn1Da1i37rxTK6ZmZnZu0XR+xnAIEkDi7ZtBDyf388kJavFbcVeA1Yv+jyk6H0HqcyhqxgA7gA2lFRJclpN/AuWEl9B6fXNqCCGuvNMrpnZcrhk2iW0PNRS7zDMlqp9Vjsjh/g5lssiIjok3QN8X9IpwBakMoPDc5ffAN+U9DdSTe5XSg7RDoyW9DDwKWB3oFBX+0vgVkk3An9mSU3uY8ALwPuK4nhc0i+AayQVP11hl+WMvx04Od98tgrwtTKH+XKO8TXgdODXSztnT+GZXDOz5dDyUAvts9rrHYaZda9DgU1IM5jXAWdGxG25bQLpK/yngVuBSSX7ngiMAuaQbuC6vtAQEfcBRwM/AV4l1QUXamsvAD6fn2hQqPP9Mqnm9zxgNvAccBZwMPDsMsY/CXgAmJ7jL5fAtuS2p/Lr7DJ9ehxFlM6Gl9fU1BRtbZXc0Ld8Cs9HbW3t9lNZD+afA+stmic2A1Xd6GJWc33x51TStIiopu7UypA0HRhbWrvcG3gm18zMzMwajpNcMzMzM2s4vvHMzMzMzMqKiE3qHcOy8kyumZmZmTUcJ7lmZmZm1nCc5JqZmVmfI2lLSf+QNE/SYknfrtF5J+Zn0tacpJA0oh7nrgcnuWZmZtYXnQa0RsTAiOgXEWcBSGrOSe/8nAA/LunoOsf6LpI2yQnrSiXbN5D0S0kzc+yPSZogaY0axHSUpEV53OZLelrS5ZK26O5zd8ZJrpmZmfVFGwMPd9I2IyIGAGsCXwculbRVaafSJLOeJA0C7gXeA+wcEQNJK6ytTedLB69o9+ZxWwv4JLAQmCZpmxqd/116zF+OmZmZdZ/2We3vLArR10m6g7S87q6Szgf+ADwVEWcU94u0Ytb1kl4BtpL0Gmlls7HAmaRVwj4maX/g+8Aw0jK5x0fEo/lcHyIt37s5MAV4ZxUuSUeRFlrYtWhbAJtHxBOS3kNaXezzpGT1IVLi+pfcfY4k8rZ9gXnA4RGxOMffQVpxrdwY7JuPvRlptbVfRsT43LYacBmwN9Af+DewX0S8kGP+DrAe8BJwRkRcVTJui4AngS9J2ggYn68BSTuRVmzbirRS3IkR0Vo0HmWPLelY4CRgQ6AjX+ffy11bgWdyzczMGtzobUczcsjIeofRY0TEx4E7gRPyzOOb5fpJ6ifpMyxJMAt2Bz4A7Jm/jr8a+BopOZsC3CBpFUmrkJbxnQQMAn4LfK6KUH8EbA/skvc/DVgMfCy3rx0RAyLiXtLM6bWFBLcCC4Ax+dr2BY6XdGBuO5I0GzscWBc4DliYyx4uBPbOM8W7kJL6pbkW2A1A0jDgJlJyPQg4Bfi9pPWWdmxJB5ES5TGk2fX9gZe7ukDP5Jr1UpdMu4SWh1rqHUaf1z6r3cmD9Xjjth/HuO3H1TuMmtLRWp7dh0qaQ0oonwWOiIjHJW2S28dHxAIASQcDN0XEbfnzj0izp7vk/VcGzs+zwr+TdFJF8Uv9gC8CO0XE83nzPbmt3C7rAjMrvcDC7Gn2oKSrScn79cBb+XgjIuJBYFo+7xr5mraR9GxEzKzgnDNICS3A4cCUiJiSP98mqQ3YB/jdUo49Fjg3Iu7Pn5+o5Bo9k2vWS7U81EL7rK7+A21mZstgRkSsHRGDImJkRFxT0t5R9H4o6Wt3APJMagepdGEo8HxOcAueoTKDgdVIX/tX4mVggwr7Iukjkv4s6UVJr5Jmawfn5knALcA1kmZIOlfSyjmxPzj3nSnpJknv7+JUw4DZ+f3GwEGS5hRewK7ABl0ceziVj8M7PJNr1ouNHDKS1qNa6x1Gn+YaR7M+qThpnQFsW/igNM06HHg+9xsmSUWJ7kYsSdgWAKsX7Tuk6LgvAa+TamYfWMr5C24HPiNpQoUlCy3AxaTygNdzbfJggIh4C5gATMizmeJVFgAAG/JJREFU11OAx0l1u7cAtxTVC19KLkfoxGdIpSGQkv9JEXFsuY5LOXYHy3DznGdyzczMzJbdb4B9JX1C0srAycAbpNKCe4G3ga9KWknSZ4Edi/Z9ANha0sh8s9f4QkNOVH8FnCdpqKT+knaWtCrwIumr/fcVHes8Ur3qFZI2hlQDK+k8SR8sE/dAYHZOcHcERhcaJO0haVtJ/YG5pPKFRZLWl7R/Llt4A5gPLCo9cI51U0kXAc2khBlgMjBK0p65z2r5kW0bdnHsy4BTJG2vZEThGpfGSa6ZmZnZMoqIx0m1pheRZl9HAaMi4s2IeBP4LHAU8Arp6/hri/b9F/Bd0izsv4G7Sg5/CumGt/tJX/n/EOgXEa8B5wB356/9d4qI2aQ64LeAv0maB/yJ9OSEcjWsXwK+m/t9h5SsFwwh1cjOBR4FppIS1H6kJH5Gjmf3fJyCnSXNz/u1kpLuHSLioXy9HcABwOmkRL0DODUft9NjR8Rv8/W2kJ4gcT1L6nw7pXeXiXSuqakp2traKuq7PJqb05+trd1+KuvB/HPQtcLX5C5XqC//PZj1TJKmRURTveOw+vFMrpmZmZk1HCe5ZmZmZtZwnOSamZmZWcNxkmtmZmZmDcdJrpmZmZk1HCe5ZmZmZkUknS7pshqfczdJj9fynI3OSa6ZmZn1KZLmF70WS1pY9PmwiPheRIzthvNuLelWSa/k59tOk7QPQETcGRFbruhz9mVe1tfMzMz6lIgYUHgvaTowNiJur2RfSStFxNvLeOobgJ8B++XPOwBaxmNZF5zkmpktp/ZZ7e8sCmFmvZ+k8cCIiDhc0ibA08BY4ExgOvAxSTuRltLdCngGODEiWpdyzMHApsCleSU0gLuL2puByRGxoaSDgV8W7b4ycG9ENOdlfc8BvgCsClwH/HdELFy+q248LlcwM1sOo7cdzcghI+sdhpl1v92BDwB7ShoG3AScTVpe9hTg95LWW8r+L5OW150s6UBJ63fWMSJ+HRED8ozzUOAp4Orc/ENgC2AkMAIYRlqW10p4JtfMbDmM234c47YfV+8wzKyEjl7hVQDjI2IBgKTDgSkRMSW33SapDdgHuKLczhERkvYAvgH8GNhU0l3AMRHx77LXIPUDWoDWiPiFJAHHAh+MiNm5z/dyn2+uqAttFE5yzczMzLrWUfR+Y+AgSaOKtq0M/HlpB4iI54ATACQNBy4BrgR27mSXc4CBwFfz5/WA1YFpKd8FUk1v/4qvog9xkmtmZmbWtSh63wFMiohjl/lgER2SfsqSMoR3kXQIcCiwQ0S8lTe/BCwEto6I55f13H2Fk1zrsdrbobm53lH0XO2zzmfkN75W7zDMzPqiycD9kvYEbifN4u4EPJFna/8PSesAXwMmkWpsBwFfBP5apu+HgIuAT0XEi4XtEbFY0qXATySdEBH/yfXB20TELSv0ChuAbzyzHmn0aBjpe3nMzKwHiogO4ADgdOBF0szuqSw9r3oT2ISUFM8F/gm8ARxVpu8BwDrAXUXP7705t32ddAPbXyXNzcfz83XLUER03QtoamqKtra2bg5nycxda2u3n8qsVys8sqr1qNa6xmFm1hNJmhYRTfWOw+rHM7lmZmZm1nBck2tmZma2gkia30nT3hFxZ02D6eOc5JqZmZmtIMVLBlt9uVzBzMzMzBqOk1wzMzPrcyRtKekfkuZJWizp2/WOqTtICkkjKujXLKns4896Kye5ZmZm1hedRloud2BE9IuIs+CdZG9x0aO7npc0obuDkdSaE9LtSrZfn7c3d3cMjcZJrpmZmfVFGwMPd9I2IyIG5PraXYFjJB1Yg5j+BYwpfJC0LmmRiRc73cM65STXzMzM+hRJdwB7ABfn2doWSWeX6xsRTwP3AFsV7X+BpA5JcyVNk7RbUduOktpy2wuSzitq20nSPZLmSHqgzOzsVcDBkvrnz4cC15EWkigcY1VJ50uakV/nS1q1qP1USTNz2xdLrntVST+S9GyO7eeS3lPd6PUeTnLNzMysoRQWy+lMRHwcuBM4Ic/WvtlZX0mbAx/l3cvv3g+MJC3N2wL8VtJque0C4IKIWBPYDPhNPs4w4Cbg7LzfKcDvJa1XdNwZwCPAp/PnMcCVJSF9izS7OxLYDtgROCOfY6983E8BmwOfLNn3h8AWed8RwDDgO51de2/nJNfMzMzs3Ybm2da5pBKCvwF3FRojYnJEvBwRb0fEj4FVWbK07lvACEmDI2J+RBSS48OBKRExJSIWR8RtQBuwT8m5rwTGSNoSWDsi7i1pPwz4bkT8JyJeBCYAR+S2LwCXR8Q/I2IBML6wkyQBxwL/HRGzI2Ie8D3gkGUdpJ7OSa6ZmZnZu82IiLXzbOzawELgikKjpJMlPSrpVUlzgLWAwbn5GNJs6WOS7pe0X96+MXBQTp7n5P12BTYoOfe1wMeBrwCTysQ2FHim6PMzeVuhraOkrWA9YHVgWtH5/5i3NyQvBmFmZmbWiYh4VVIL8GuAXH/7deATwMMRsVjSK4By/38Dh0rqB3wW+F2+gawDmBQRx3Zxvtck3QwcTyp3KDWDd980t1HeBjATGF7Ud6Oi9y+RkvWtI+L5ii6+l/NMrpmZmVknJA0gfaVfSCoHAm+TnniwkqTvAGsW9T9c0noRsRiYkzcvAiYDoyTtKam/pNXy48o2LHPa04HdI2J6mbargTMkrSdpMKmmdnJu+w1wlKStJK0OnFnYKcdzKfATSe/NsQ6TtGfVg9JLOMk1MzMze7ehhefkkr7yH0SqhQW4BbiZVKv7DPA67y4R2At4OO97AXBIRLweER3AAaQE9sW8z6mUycUiYkZE3FW6PTubVMv7IPAQ8Pe8jYi4GTgfuAN4Iv9Z7Ot5+19zvfHtLKklbjiKiIo6NjU1RVtbWzeHA83N6c/W1m4/lVmvVrh7uPWo1rrGYWbW0zRPbGbq0VOnRURTvWOx+nFNrlkv1j6rvctH5ZiZ9TXts9rrHYL1AE5yzXqp0duOrncIZmZmPZaTXLNeatz24xi3/bh6h2Fm1uM0T2xmKlPrHYbVmW88MzMzM7OG4yTXzMzMzBqOk1wzMzPrcyRtKekfkuZJWizp2918vvGSJnfdEyS1ShrbnfH0BU5yzczMrC86DWiNiIER0S8izgLICzSEpGuLO0vaLm9vrWWQko6StKjw3F5JT0u6XNIWtYyjN3KSa2ZmZn1R8dK4pV4EdsnL8RYcSVoAoh7ujYgBwFrAJ0nL806TtE2d4ukVnOSamZlZnyLpDmAP4OI8O9oi6eyiLm8C15OW80VSf+ALwFUlx9lF0v2SXs1/7lLUtqmkqbkc4jZgcMm+O0m6R9IcSQ9Iau4q7ohYFBFPRsSXgKnA+EqOl2eDn8qxPC3psKK2YyU9mtsekfThLgewl3CSa2ZmZg2lq5UgI+LjwJ3ACXmG9M0y3a4ExuT3e5JmfWcUGiUNAm4CLgTWBc4Dbiqa/W0BppGS27NIM8GFfYflfc8mLRl8CvB7SetVcZnXArt1dTxJa+QY946IgcAuQHve7yBSojwGWBPYH3i5ihh6NCe5ZmZmZiUi4h5gkKQtSUnglSVd9gX+HRGTIuLtiLgaeAwYJWkjYAfg2xHxRkT8BbihaN/DgSkRMSUiFkfEbUAbsE8VIc4gJbSVHG8xsI2k90TEzIgolGmMBc6NiPsjeSIinqkihh7NSa6ZmZlZeZOAE0ilDdeVtA0FShPCZ4Bhue2ViFhQ0lawMXBQLi2YI2kOsCuwQRWxDQNmd3W8HMPBwHHATEk3SXp/3m848GQV5+xVvOKZmZmZWXmTgCeAKyPiNUnFbTNIyWWxjYA/AjOBdSStUZTobgREft8BTIqIY5cjts+QSi66PF5E3ALcIuk9pJKGS0mlDh3AZssRQ4/mmVwzMzOzMiLiaWB34FtlmqcAW0gaLWklSQcDWwE35q/824AJklaRtCswqmjfyaSyhj0l9Ze0Wn502YZLiyf33VTSRUAzMKGr40laX9L+uTb3DWA+sCjvdxlwiqTtlYyQVJq491pOcs3MzMw6ERF3RcSMMttfBvYDTibdrHUasF9EvJS7jAY+QiopOJOimt6I6AAOAE4nPa6sAziVzvOynSXNB+YCraSbxHaIiIcqOF6/HOOMHMvuwJfyfr8FziHdJDeP9ESJQp1vr6eI6LoX0NTUFG1tbd0cDjQ3pz9bW7v9VGZmZtagJE2LiKZ6x2H145lcMzMzM2s4TnLNzMzMrOE4yTUzMzOzhuMk18zMzMwajpNcMzMzswpI+rmkb9c7DquMk1wzMzPrcyRNl7RQ0ry8Stg9ko6T1GluFBHHRcRZ3RjT1pJulfRKjmmapGqW+i13zFZJY1dUjL2Jk1wzMzPrq0ZFxEDSymU/AL4O/LJcR0n9axDPDcBtwPrAe4Gvkp6Na8vASa6ZmZn1aRHxakT8ATgYOFLSNpImSvqZpCmSFgB75G1nA0h6VNJ+hWPkVc9ekvTh/HmnPDs8R9IDkpqXFoOkwcCmwKUR8WZ+3R0Rd+X2f0oaVdR/5Xy+kXmFs8mSXs7nuz+vdHYOafneiyXNl3Rx3vf9km6TNFvS45K+UHTciZL+R9LNeZ+7JQ2RdH6eYX5M0odWzMh3r5XqHYBZrV1yCbS01DsKMzPraSLiPknPkRJDSKuW7UNa2WwV4PCi7lcDhwI35s97Ai9FxN8lDQNuAo4A/gh8Avi9pPdHxIudnP5l4AlgsqTLgHsj4oWi9ivz+W/In/cBZkZEu6T/AtYChpOW7h0JLIyIb0n6KDA5Ii4DyMv73gZ8B9gb+CBwq6SHI+LhfOwv5Ot5mLR88b2kVdtOJi0lfB6wx9JHs/48k2t9TksLtLfXOwozM+uhZrBkadv/zbOpiyPi9ZJ+LcD+klbPn0fnbZCS0SkRMSXvexvQRkpMy4q0BO0ewHTgx8BMSX+RtHnuMhnYR9Ka+fMRwKT8/i1gXWBERCyKiGkR0VmZw37A9Ii4PCLejoi/A78HPl/U57p8jNeB64DXI+LKiFgE/BrwTK5ZTzVypJeONjNrZNIy7zoMmJ3fd3TWKSKekPQoMErSDcD+LEn+NgYOKi4vAFYG/ry0E0fEc8AJAJKGA5eQZnB3jogZku4GPifpOtIs7Il510mkWdxrJK1NSoi/FRFvlTnNxsBHJM0p2rYSSxJmgOIZ5IVlPg9Y2nX0FE5yzczMzABJO5CS3LuAjwDRxS6FkoV+wCMR8UTe3gFMiohjlzWWiOiQ9NN8joIrgLGk/O3eiHg+932LVEYwQdImpBKDx0k30ZVeQwcwNSI+tayx9RYuVzAzM7M+TdKa+Saya0j1qw9VuOs1wKeB41lSqgBpJnWUpD0l9c83hjVL2nApMawjaYKkEZL65RvRvgj8tajb9cCHSTO4Vxbtu4ekbfMTIOaSyhcW5eYXgPcVHeNGYAtJR+Sb11aWtIOkD1R4zb2Gk1wzMzPrq26QNI80u/kt0g1VR1e6c0TMJN2UtQupVrWwvQM4ADgdeDEf/1SWnne9CWwC3E5KVP9JuonsqKLjLiTVz24KXFu07xDgd3m/R4GppEQb4ALg8/nJCBdGxDxSYn4Iqf54FvBDYNVKr7u3UKpz7lpTU1O0tbV1czjQ3Jz+dL2kdRf/jJmZNT5J0yKiqd5xrGiSvgNsERGHd9m5j+uRNbnt7UsSEbMVrb093XhmZmbWm0gaBBxDerKCdaHHlSuMHu0ExMzMzBpTXmCh3Gu3LvY7llT2cHNE/KU20fZuPa5cway7uVzBzKzxNWq5glWux83kmpmZmZktLye5ZmZmZtZwnOSamZlZnyNpS0n/kDRP0mJJ3653TJWS1CppbL3j6Omc5JqZmVlfdBrQGhEDI6JfRJxVaJB0uqSn8w1hz0n69VKO01ByAv16Tv7nSpom6RuSet1zdJ3kmpmZWV+0MfBw6UZJR5Ie0fXJiBgANAF/qnFs3UJSpY+OPSEiBgIbACeTFo6YIkndFlw36JHPyTXrbn4Ws5lZ3yXpDmB3YFdJ5wN/AJ6KiDOAHYBbIuJJgIiYBVxStO9apJXR9gEWA5cDZ0bEotx+LHASsCHpkV+HR8Tf87K5PwNGAs8D34yIP+R9JgILSCuefQx4BBhdiEHSp4CLSEnnJOCdZFPSZsClwHZAALcAX46IObl9ej7vYcCWks4AdoqIzxUd4yJgUUR8rXicImIB0Cppf+AxYF/gRkn9SDPhxwJrk/4TcFxEzJa0GnAZsDfQH/g3sF9EvJCf8/tjYE/gPcDUiDiwq7+vZeWZXOtz/CxmM7O+LSI+DtxJmrEcQFpSt+CvwBhJp0pqktS/ZPcrgLeBEcCHSEvkjgWQdBAwHhgDrAnsD7wsaWXgBuBW4L3AV4CrJG1ZdNxDgQnAOsATwDn5mINJS/meAQwGngQ+WrSfgO8DQ4EPAMNzDMUOJSWoa5OW+91L0tr5+CsBB5OS587G61mgDSg8y/erwIGk/ygMBV4BfprbjgTWynGsCxwHLMxtk4DVga3zOPyks3OuCJ7JtT5n3Lj0MjOzxrWsX6xHxGRJARxNShZfl/T/IuIHktYnzVCuHRELgQWSfgKMA35BSnbPjYj78+GeSLFoN2AA8IOIWAzcIelGUvI5Pve9NiLuy/2vIs0WQ5oxfiQifpfbzieVEBTifaJwHuBFSecBZ5Zc1oUR0ZHfL5T0F+Ag0gzwXsBLETGti6GZAQzK7/+L9B+E53JM44FnJR0BvEVKbkdExIPAtNxngzx260bEK/k4U7s453JxkmtmZmZWJCKuIs20rkyasbxK0j9IM5YrAzOLylP7kcoSIM1ePlnmkEOBjpzgFjwDDCv6PKvo/WukpPidfYtiC0nvfJb0XuBC0izrwBzPK7xbR8nnK4DjSUnu4SxlFrfIMOCe/H5j4DpJxdezCFg/H2s4cE2eLZ4MfCtvm12U4HY7lyuYmZmZlRERb0XEb4EHgW1IyeIbwOCIWDu/1oyIrfMuHcBmZQ41Axiea1kLNiLV5nZlJilBBCDf/DW8qP37pFrcD0bEmqSktXQeu3R52+uBD0raBtgPuGppAUgaDmxPKvGAdJ17F43B2hGxWkQ8n8dsQkRsBeySjz8m7zOoUCZRC05yzczMzDJJR0naV9JASf0k7U2qIf1bRMwk1dX+WNKauX0zSbvn3S8DTpG0vZIRkjYG/ka6sew0SStLagZGAddUENJNwNaSPpvrZ78KDClqHwjMB+ZIGgac2tUBI+J14HdAC3BfrrktNxar52v7X+A+YEpu+jlwTr42JK0n6YD8fg9J2+Za5rmk8oVFeexuBv5H0jp5HD5WwfUvMye5ZmZmZkvMBU4HngXmAOcCx0fEXbl9DLAK6QkIr5CSxQ0A8qzvOaTkcR5pxnRQRLxJugltb+Al4H+AMRHxWFfBRMRLpPrZHwAvA5sDdxd1mQB8GHiVlBBfW+F1XgFsS/lShYslzQNeAM4n3fi2V1G5xQWkJ1Lcmvv9FfhIbhtCGpO5wKOkutvJua1Qs/sY8B/gXU9zWNEUUTqDXV5TU1O0tbV1ZyxmZmZmK4SkaRHRVO84eipJG5GSzSERMbfe8XQHz+SamZmZ9SG5Nvgk4JpGTXChiplcSS+S7gQsZzBp+t1qw+NdOx7r2vJ415bHu7Y83rW1ZV61y4pIWoNUhvAMqQSh9MkLDaPiR4hFxHqdtUlq81cCtePxrh2PdW15vGvL411bHu/akuQayzLyKmYDuuzYAFyuYGZmZmYNx0mumZmZmTWcFZXkXrKCjmOV8XjXjse6tjzeteXxri2Pd215vPu4im88MzMzMzPrLVyuYGZmZmYNx0mumZmZmTWcipJcSSdIapP0hqSJS+l3pKRpkuZKek7SuXmdZatQpWOd+/63pFmSXpX0K0mr1ijMhiJpkKTrJC2Q9Iyk0Z30W1XSzyW9IGm2pBvyOuFWhUrHO/f9sKS/SJqfx/3EWsbaCKoZ79x/FUmPSXquVjE2kir+PTlV0j8lzZP0tKRTax1rb1fFWEvSDyW9nF/nSlKt47Xaq3QmdwZwNvCrLvqtTlqHeDBpDeNPAKcsc3R9U0VjLWlP4BukMd4EeB9p/Wqr3k+BN4H1gcOAn0nauky/E4GdgQ8CQ0lrml9UqyAbSEXjLWkw8EfgF8C6wAjg1hrG2Sgq/fkuOJW0prwtm0rHW8AYYB1gL+AESYfULMrGUOlYjwMOBLYj/fu9H/BftQrS6qeqG88knQ1sGBFHVdj/JGCPiBi1bOH1XV2NtaQWYHpEnJ4/fwK4KiKG1C7K3i+v/PIKsE1E/CtvmwQ8HxHfKOn7M2BeRJyWP+8LnBcRW9Y47F6ryvH+HjA8Io6ofaSNoZrxzm2bAlNIy31eGhEb1jLe3q7a8S7Z90LS7+SvdH+kvV+V/5bcA0yMiEvy52OAYyNipxqHbTXW3TW5HwMe7uZz9FVbAw8UfX4AWF/SunWKp7faAlhU+Ecye4A0vqV+CXxU0lBJq5NmDm6uQYyNpJrx3gmYLekeSf/J5SEb1STKxlHNeEP6ZuJ0YGF3B9agqh1vIH2dDuyGf19Wo5qxLvf7cql/J9YYui3JlXQ00AT8qLvO0ccNAF4t+lx473W6q1M6juTP5cbxX8CzwPPAXOADwHe7NbrGU814bwgcSSoT2Qh4Gri6W6NrPBWPt6TPACtFxHW1CKxBVfPzXWw86ffx5d0QU6OqZqzL/b4c4LrcxtctSa6kA4EfAHtHxEvdcQ5jPrBm0efC+3l1iKU3Kx1H8udy4/gzYDVSfegawLV4Jrda1Yz3QuC6iLg/Il4n1ZzvImmtbo6xkVQ03vmr33MBf1W+fKr5+QbSzcak2tx9I+KNboyt0VQz1uV+X84PLxTQ8FZ4kitpL+BSYFREPLSij2/veJhURF+wHfBCRLxcp3h6q38BK0navGjbdpT/2nA7Ul3X7PzL6CJgx3yDlFWmmvF+ECj+JVR479mXylU63puTbmC9U9Is0n/gNshPb9mkBnE2imp+vpH0RfINxBHhp1lUp5qxLvf70qUhfUCljxBbSdJqQH+gv6TVVObRYJI+DlwFfC4i7luxofYNlY41cCVwjKStJK0DnAFMrGGoDSEiFpB+oX9X0hqSPgocAEwq0/1+YIyktSStDHwJmOFvKypX5XhfDnxG0sg83t8G7oqIObWLuHerYrz/CQwHRubXWOCF/L6jdhH3btX8fEs6DPge8KmIeKq2kfZ+Vf5bciVwkqRhkoYCJ+Pfl31DRHT5ItULRclrPKlObj6wUe73Z+DtvK3wurmSc/hV3VjnvieRfhHNJSUEq9Y7/t74AgYB1wMLSDW3o/P23UhfaRX6rUv6T9x/SI8PuwvYsd7x97ZXpeOdtx1PqoF+BbiB9LSFul9Db3pVM95F+zQDz9U79t74quLfk6eBt0p+X/683vH3plcVYy1SOc7s/DqX/HQpvxr7VdUjxMzMzMzMegMv62tmZmZmDcdJrpmZmZk1HCe5ZmZmZtZwnOSamZmZWcNxkmtmZmZmDcdJrpmZmZk1HCe5ZmZmZtZwnOSamZmZWcNxkmtmZmZmDef/AyKjej63sgf1AAAAAElFTkSuQmCC\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"cluster_columns(xs_imp)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> note: The most similar pairs are found by calculating the *rank correlation*, which means that all the values are replaced with their *rank* (i.e. first, second, third, etc within the column), and then the *correlation* is calculated. (Feel free to skip over this minor detail though, since it's not going to come up again in the book!)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the chart above, you can see pairs of columns that were extremely similar as the ones that were merged together early, far away from the \"root\" of the tree at the left. For example, the fields `ProductGroup` and `ProductGroupDesc` were merged quite early, as were `saleYear` and `saleElapsed`, and as were `fiModelDesc` and `fiBaseModel`. These might be so closely correlated they are practically synonyms for each other.\n",
"\n",
"Let's try removing some of these closely related features to see if the model can be simplified without impacting the accuracy. First, we create a function that quickly trains a random forest and returns the OOB score, by using a lower `max_samples` and higher `min_samples_leaf` . The *score* is a number returned by sklearn that is 1.0 for a perfect model, and 0.0 for a random model. (In statistics it's called *R^2*, although the details aren't important for this explanation). We don't need it to be very accurate--we're just going to use it to compare different models, based on removing some of the possibly redundent columns."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def get_oob(df):\n",
" m = RandomForestRegressor(n_estimators=40, min_samples_leaf=15,\n",
" max_samples=50000, max_features=0.5, n_jobs=-1, oob_score=True)\n",
" m.fit(df, y)\n",
" return m.oob_score_"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here's our baseline."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8771039618198545"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"get_oob(xs_imp)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we try removing each variable one at a time."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'saleYear': 0.8759666979317242,\n",
" 'saleElapsed': 0.8728423449081594,\n",
" 'ProductGroupDesc': 0.877877012281002,\n",
" 'ProductGroup': 0.8772503407182847,\n",
" 'fiModelDesc': 0.8756415073829513,\n",
" 'fiBaseModel': 0.8765165299438019,\n",
" 'Hydraulics_Flow': 0.8778545895742573,\n",
" 'Grouser_Tracks': 0.8773718142788077,\n",
" 'Coupler_System': 0.8778016988955392}"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"{c:get_oob(xs_imp.drop(c, axis=1)) for c in (\n",
" 'saleYear', 'saleElapsed', 'ProductGroupDesc','ProductGroup',\n",
" 'fiModelDesc', 'fiBaseModel',\n",
" 'Hydraulics_Flow','Grouser_Tracks', 'Coupler_System')}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's try dropping multiple variables. We'll drop one from each of the tightly aligned pairs we noticed above. Let's see what that does."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8739605718147015"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"to_drop = ['saleYear', 'ProductGroupDesc', 'fiBaseModel', 'Grouser_Tracks']\n",
"get_oob(xs_imp.drop(to_drop, axis=1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Looking good! This is really not much worse than the model with all the fields. Let's create DataFrames without these columns, and save them:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"xs_final = xs_imp.drop(to_drop, axis=1)\n",
"valid_xs_final = valid_xs_imp.drop(to_drop, axis=1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"(path/'xs_final.pkl').save(xs_final)\n",
"(path/'valid_xs_final.pkl').save(valid_xs_final)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can load them back later with:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"xs_final = (path/'xs_final.pkl').load()\n",
"valid_xs_final = (path/'valid_xs_final.pkl').load()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can check our RMSE again, to confirm it is still a similar accuracy."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(0.183263, 0.233846)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m = rf(xs_final, y)\n",
"m_rmse(m, xs_final, y), m_rmse(m, valid_xs_final, valid_y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Partial dependence"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The two most important predictors are `ProductSize` and `YearMade`. We'd like to understand the relationship between these predictors and sale price. It's a good idea to first check the count of values per category (provided by the Pandas `value_counts` method), to see how common each category is:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAcYAAAD7CAYAAADw8TTuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAaG0lEQVR4nO3dfbRddX3n8feHgMEIhASoQHgIImqHUmgNVWd8YLVaFWu1ddT6BLRVprauWR3Blum0mqFQW6et6SztWKwFJUXxkYpgR51KrVjaXlBBLLWlEgNE5CkhJIASv/PH3rf9ebxPuU/n3PB+rXVWzt6/3/7t79n33vPJb599705VIUmSOnsNuwBJkkaJwShJUsNglCSpYTBKktQwGCVJauw97AK0ew4++OBau3btsMuQpCXl2muvvauqDplJX4NxiVm7di1jY2PDLkOSlpQkm2ba11OpkiQ1DEZJkhoGoyRJDYNRkqSGwShJUsNglCSp4a9rLDE33LaNtedcMewyNEO3/O4Lhl2CpN3kjFGSpIbBKElSw2CUJKlhMEqS1NhjgjHJK5OMJbk/yZYkn0zy9GHXNRNJbkny7GHXIUnaQ4IxyRuBDcDvAI8FjgL+GHjRMOuSJC09Sz4Yk6wEzgV+pao+WlU7quo7VXV5Vb0pyfIkG5Lc3j82JFneb3tKkluT/FqSb/UzzRcnOTXJ15Lck+Q3mn2tT/LhJJcm2Z7kuiQnNu3nJLm5b/tqkp8ZqPV1Sf6xaf/RJBfTBfnl/Wz31xbnyEmSJrLkgxF4GrAv8LFJ2v8H8FTgJOBE4MeA32zaD+23XwO8GXg38GrgycAzgDcneVzT/0XAh4DVwCXAZUn26dtu7rdZCfxPYGOSwwCSvBRYD5wGHAD8NHB3Vb0G+Abwwqrar6reNvgCkpzZnyYe27Vz2wwPiyRpNvaEYDwIuKuqHp6k/VXAuVX1raq6ky6wXtO0fwc4v6q+A3wAOBj4o6raXlU3AjcCP9z0v7aqPtz3/0O6UH0qQFV9qKpur6rvVtWlwD/TBTHAa4G3VdU/VOdfqmpG9werqguqal1VrVu2YuVMNpEkzdKeEIx3Awcnmeyv+BwOtAG0qV/3b9tX1a7++QP9v3c07Q8A+zXLm8efVNV3gVvHx0tyWpIvJdmaZCvwQ3RBC3Ak3YxSkjTC9oRg/FvgQeDFk7TfDhzdLB/Vr5utI8efJNkLOAK4PcnRdKdh3wAcVFUHAl8B0nffDBw7yZg1h3okSfNoyQdjVW2j+2zwnf2FMyuS7JPk+UneBrwf+M0khyQ5uO+7cQ67fHKSn+1nqL8KPARcAzyGLuDuBEjy83QzxnF/Cpyd5MnpPL4PU+hmqO3nmJKkIVnywQhQVX8IvJHuopo76WZnbwAuA84DxoDrgRuA6/p1s/UXwMuBe+k+q/zZ/irYrwJ/QDeDvQM4Abi6qfFDwPl0F+xs72tb3Te/lS68tyY5ew61SZLmKFWexZupJOuBx1fVq4dVw/LDjqvDTt8wrN1rN3l3DWk0JLm2qtbNpO8eMWOUJGm+eD/GJeaENSsZcxYiSQvGYNwNVbV+2DVIkhaWp1IlSWoYjJIkNQxGSZIaBqMkSQ2DUZKkhsEoSVLDYJQkqWEwSpLUMBglSWoYjJIkNQxGSZIaBqMkSQ2DUZKkhnfXWGJuuG0ba8+5Yso+3hxXkmbPGaMkSQ2DUZKkhsEoSVLDYFxASd6V5Lfmu68kaeF48c0sJbkFOBw4vKruatZ/CTgROKaqfmmm4+1OX0nSwnHGODdfB14xvpDkBODRwytHkjRXBuPcXAyc1iyfDrxvfCHJRUnO65+fkuTWJGcl+VaSLUl+fqK+kqThMRjn5hrggCQ/mGQZ8HJg4xT9DwVWAmuAXwTemWTVdDtJcmaSsSRju3Zum4+6JUmTMBjnbnzW+BzgJuC2Kfp+Bzi3qr5TVVcC9wNPnG4HVXVBVa2rqnXLVqycj5olSZPw4pu5uxj4HHAMzWnUSdxdVQ83yzuB/RaqMEnS7nPGOEdVtYnuIpxTgY8OuRxJ0hw5Y5wfvwisqqodSTymkrSE+SY+D6rq5mHXIEmaHwbjLFXV2knWPwykXzyjWX8VcMRkY1TVGUiShs7PGCVJajhjXGJOWLOSMe+3KEkLxhmjJEkNg1GSpIbBKElSw2CUJKlhMEqS1DAYJUlqGIySJDUMRkmSGgajJEkNg1GSpIbBKElSw2CUJKlhMEqS1DAYJUlqeNupJeaG27ax9pwrhl2GgFu8/Ze0R3LGKElSw2CUJKlhMEqS1DAYhyTJ2iSVZO9++aokrx12XZL0SGcwTiDJ05N8Icm2JPckuTrJycOuS5K08LwqdUCSA4BPAK8HPgg8CngG8NAw65IkLQ5njN/vCQBV9f6q2lVVD1TVp6rq+iRn9LPHtyfZmuRfk/zHfv3mJN9Kcvr4QElekOSLSe7r29cP7VVJkmbEYPx+XwN2JXlvkucnWTXQ/hTgeuAg4BLgA8DJwOOBVwPvSLJf33cHcBpwIPAC4PVJXry7BSU5M8lYkrFdO7fN6kVJkmbGYBxQVfcBTwcKeDdwZ5KPJ3ls3+XrVXVhVe0CLgWOBM6tqoeq6lPAt+lCkqq6qqpuqKrvVtX1wPuBZ82ipguqal1VrVu2YuXcX6QkaVIG4wSq6h+r6oyqOgL4IeBwYEPffEfT9YG+/+C6/QCSPCXJZ5PcmWQb8EvAwQv+AiRJs2YwTqOqbgIuogvI3XUJ8HHgyKpaCbwLyPxVJ0mabwbjgCRPSnJWkiP65SOBVwDXzGK4/YF7qurBJD8GvHIeS5UkLQCD8fttp7vA5u+S7KALxK8AZ81irF8Gzk2yHXgz3a9/SJJGWKpq2DVoNyw/7Lg67PQN03fUgvPuGtLSkeTaqlo3k77OGCVJaviXb5aYE9asZMyZiiQtGGeMkiQ1DEZJkhoGoyRJDYNRkqSGwShJUsNglCSpYTBKktQwGCVJahiMkiQ1DEZJkhoGoyRJDYNRkqSGwShJUsNglCSp4W2nlpgbbtvG2nOuGHYZ88Ib/UoaRc4YJUlqGIySJDUMRkmSGgbjbkhSSR7fP39Xkt8adk2SpPm1x158k+QW4HDg8Kq6q1n/JeBE4JiqumW241fVL821RknS6NnTZ4xfB14xvpDkBODRwytHkjTq9vRgvBg4rVk+HXjf+EKS5Ul+P8k3ktzRnx59dNP+piRbktye5BfagZNclOS8/vkZST4/0N6edr0oyR8n+WSS+5NcneTQJBuS3JvkpiQ/sgCvX5K0m/b0YLwGOCDJDyZZBrwc2Ni0/x7wBOAk4PHAGuDNAEmeB5wNPAc4Dnj2HGt5GfCbwMHAQ8DfAtf1yx8G/nCyDZOcmWQsydiundvmWIYkaSp7ejDCv88anwPcBNzWrw/wOuC/VdU9VbUd+B3g5/r2lwEXVtVXqmoHsH6OdXysqq6tqgeBjwEPVtX7qmoXcCkw6Yyxqi6oqnVVtW7ZipVzLEOSNJU99uKbxsXA54BjaE6jAocAK4Brk4yvC7Csf344cG3Tf9Mc67ijef7ABMv7zXF8SdI82OODsao2Jfk6cCrwi03TXXSBdHxV3TbBpluAI5vlo6bYzQ66kAUgyaGzr1iSNEyPhFOp0AXij/enRMd9F3g38PYkPwCQZE2S5/btHwTOSPIfkqwA3jLF+F8Gjk9yUpJ9mftpV0nSkDwigrGqbq6qsQmafh34F+CaJPcBnwGe2G/zSWAD8Fd9n7+aYvyvAef22/8z8PnJ+kqSRluqatg1aDcsP+y4Ouz0DcMuY154dw1JiyXJtVW1biZ9HxEzRkmSZmqPv/hmT3PCmpWMOdOSpAXjjFGSpIbBKElSw2CUJKlhMEqS1DAYJUlqGIySJDUMRkmSGgajJEkNg1GSpIbBKElSw2CUJKlhMEqS1DAYJUlqGIySJDW87dQSc8Nt21h7zhULMrY3DpYkZ4ySJH0Pg1GSpIbBKElSw2DcDUnWJqkke/fLn0xy+rDrkiTNn2mDMcktSZ69GMXsriSHJ7l1krZKcsd4iPXr9k7yrSQ1H/uvqudX1XvnYyxJ0mhYlBljOguxr1OBv5yifSvw/IH+9y5AHZKkPcSswyrJqiSfSHJnknv750c07VclOT/J1cBO4HFJjknyuSTbk3wmyTuTbGy2eWqSLyTZmuTLSU6ZpoxTgSunaL8YOK1ZPg1438DrWJnkPUm2JLktyXlJlvVty5L8fpK7kvwr8IKBba9K8tr++fqB1zJ42vWqfuwvJLk/yeVJDkry50nuS/IPSdZO83olSQtsLrO4vYALgaOBo4AHgHcM9HkNcCawP7AJuAT4e+AgYH3fDkCSNcAVwHnAauBs4CNJDplo50n2AZ4JfHqKGi8DnpnkwCQHAs8A/mKgz3uBh4HHAz8C/CTw2r7tdcBP9evXAf95in3NxM/RveY1wLHA39Idw9XAPwJvmWijJGcmGUsytmvntjmWIEmayqyDsarurqqPVNXOqtoOnA88a6DbRVV1Y1U9DBwGnAy8uaq+XVWfBz7e9H01cGVVXVlV362qTwNjdLPCiTwT+HK/78k8CFwOvJwulD7erwMgyWPpTrX+alXtqKpvAW/v+wK8DNhQVZur6h7grVMflWldWFU3V9U24JPAzVX1mf74fIgugL9PVV1QVeuqat2yFSvnWIIkaSqz/ss3SVbQhcjzgFX96v2TLKuqXf3y5maTw4F7qmpns24zcGT//GjgpUle2LTvA3x2khKmO4067n10gRbg1wfaju73sSXJ+Lq9mroPH3gNm2awv6nc0Tx/YILl/eY4viRpjubyJ+HOAp4IPKWqvpnkJOCLdAE0rr36cwuwOsmKJhyPbNo3AxdX1etmuP9TgZ+ZQb+/oZutFvB5ulOY7T4fAg7uZ22DtgzUeNQU+9kBrGiWD51BbZKkETPTU6n7JNm3eexN97nhA8DWJKuZ5POxcVW1ie7U6Pokj0ryNKCdHW4EXpjkuf1FL/smOaW9oGdckmOA5VV103SFV1X1+/np/nnbtgX4FPAHSQ5IsleSY5OMnxL+IPBfkxyRZBVwzhS7+hLd55lHJVkJ/PfpapMkjZ6ZBuOVdCE4/lgPbAAeDdwFXMPUvzYx7lXA04C76S6yuZRuxkZVbQZeBPwGcCfdbO5Nk9T4AmZ2GpV+7Bur6sZJmk8DHgV8le5XOT5MN8MEeDfwf4EvA9cBH51iH5/uX8/1wLXAJ2ZanyRpdGRgErW4O08uBW6qqilnmxNsdyXwjqqacTjuKZYfdlwddvqGBRnbu2tI2lMlubaq1s2k76L+SbgkJ/enKvdK8jy6GeJlsxjqKia/KEeSpFlb7PsxHkp3OvIg4Fbg9VX1xd0dpKreNt+FLRUnrFnJmDM7SVowixqMVXU53e8VSpI0kry7hiRJDYNRkqSGwShJUsNglCSpYTBKktQwGCVJahiMkiQ1DEZJkhoGoyRJDYNRkqSGwShJUsNglCSpYTBKktRY7NtOaY5uuG0ba8+5YthlTMgbHUvaEzhjlCSpYTBKktQwGCVJahiMkiQ1DMYBSW5J8uxh1yFJGg6DcZ6l43GVpCXKN/AZSLIqySeS3Jnk3v75EU37VUnOT3I1sBN4XJJjknwuyfYkn0nyziQbm22emuQLSbYm+XKSU4bw0iRJAwzGmdkLuBA4GjgKeAB4x0Cf1wBnAvsDm4BLgL8HDgLW9+0AJFkDXAGcB6wGzgY+kuSQiXae5MwkY0nGdu3cNn+vSpL0fQzGGaiqu6vqI1W1s6q2A+cDzxrodlFV3VhVDwOHAScDb66qb1fV54GPN31fDVxZVVdW1Xer6tPAGHDqJPu/oKrWVdW6ZStWzvvrkyT9O4NxBpKsSPInSTYluQ/4HHBgkmVNt83N88OBe6pq5yTtRwMv7U+jbk2yFXg6XaBKkobIPwk3M2cBTwSeUlXfTHIS8EUgTZ9qnm8BVidZ0YTjkU37ZuDiqnrdQhYtSdp9zhgntk+SfccfwCq6zxW3JlkNvGWqjatqE92p0fVJHpXkacALmy4bgRcmeW6SZf1+Tmkv6JEkDYfBOLEr6YJw/HEg8GjgLuAa4C9nMMargKcBd9NdZHMp8BBAVW0GXgT8BnAn3QzyTfj1kKSh81TqgKpaO8Ouf9Jsc8oE49wMPGN8OcmlwE1N+9/x/RfwSJKGzBnKAklycpJjk+yV5Hl0M8TLhl2XJGlqzhgXzqHAR+l+j/FW4PVV9cW5DnrCmpWMed9DSVowBuMCqarLgcuHXYckafd4KlWSpIbBKElSw2CUJKlhMEqS1DAYJUlqGIySJDUMRkmSGgajJEkNg1GSpIbBKElSw2CUJKlhMEqS1DAYJUlqGIySJDW87dQSc8Nt21h7zhXDLkOSFtUti3gfWmeMkiQ1DEZJkhoGoyRJDYNxCkn+PslxSR6X5LoFGP+DSZ6TZHmSb873+JKk3WcwTiLJPsDRwL8ATwbmPRibcX8Y+MoCjC9J2k0G4+R+CPhqVRWwjiYYk9yS5Owk1yfZluTSJPv2bauSfCLJnUnu7Z8fMTh4klVAquruwfElScNjMA5I8vNJtgJXA0/rn58F/F6SrUmO6bu+DHgecAzdjO+Mfv1ewIV0s82jgAeAdzTj/0Q/5mbgiP75HwG/0o//rAlqOjPJWJKxXTu3zf+LliT9G4NxQFVdWFUHAtcCT+XfT3MeUFUHVtXX+67/u6pur6p7gMuBk/rt766qj1TVzqraDpwPPKsZ///1418GvBRYA9wCHNKP/9cT1HRBVa2rqnXLVqxcqJcuScJf8P8eSVYD/woE2A+4CljeN9+bZH1VbeiX24tldgKH92OsAN5ON5tc1bfvn2RZVe1Kcms/9v7ATwH70H0dbk/yZ1X1xoV6fZKk6TljbFTVPf1s7r8Af9o//0vghf1sbsPUIwDdadcnAk+pqgOAZ/br0+/jCLrQ/Ew//gXAr/TjG4qSNGQG48Taq1B/hO606kztT/e54tZ+BvqWacb/UWBslnVKkuaZwTixJwPXJTkI2FVV9+7GthuARwN3AdfQzTgnGz/Ak4Ab51ivJGme+BnjBKrqx5vFYydoXzuwvL55fjtwysAmfzLQ/xeaxcfOskxJ0gJwxihJUsMZ4xJzwpqVjC3i7Vck6ZHGGaMkSQ2DUZKkhsEoSVLDYJQkqWEwSpLUMBglSWoYjJIkNdLdh1dLRZLtwD8Nu45pHEz3J/FG1ajXB6Nf46jXB6Nf46jXB3tWjUdX1SEzGdBf8F96/qmq1g27iKkkGRvlGke9Phj9Gke9Phj9Gke9Pnjk1uipVEmSGgajJEkNg3HpuWDYBczAqNc46vXB6Nc46vXB6Nc46vXBI7RGL76RJKnhjFGSpIbBKElSw2CUJKlhMC4RSVYn+ViSHUk2JXnlEGq4KsmDSe7vH//UtL2yr2tHksuSrF7o2pO8IclYkoeSXDTQ9hNJbkqyM8lnkxzdtC1P8mdJ7kvyzSRvnOm281VjkrVJqjmW9yf5rcWusd/Pe/qvy/YkX0zy/JnsZxRqHKHjuDHJln4/X0vy2pnsY5G/FyescVSOYTPeceneZzY262b9/jLVtpOqKh9L4AG8H7gU2A94OrANOH6Ra7gKeO0E648HtgPP7Ou7BPjAQtcO/CzwYuD/ABc16w/u9/FSYF/gfwHXNO1vBf4GWAX8IPBN4Hkz2XYea1wLFLD3JNstSo3AY4D1fT17AT/Vfy3XjspxnKbGUTmOxwPL++dP6vfz5FE5htPUOBLHsNnfp/r9bWzqntX7y3TbTlrDXF6Aj8V59G8M3wae0Ky7GPjdRa7jKiYOxt8BLmmWj+3r3X8xagfO43tD50zgCwPH7wHgSf3ybcBPNu2/Pf7DMt2281jjdG9Gi15jM971wEtG8ThOUOPIHUfgicAW4GWjegwHahyZYwj8HPBBuv8IjQfjrN9fptp2qjo8lbo0PAHYVVVfa9Z9me5/Q4vtrUnuSnJ1klP6dcf39QBQVTfTf7MynNoH69kB3Awcn2QVcHjbPlDPpNsuUK2bktya5MIkBwMMs8Ykj6X7mt041X5GqMZxQz+OSf44yU7gJrrQuXKqfQzjGE5S47ihHsMkBwDnAmcNNM3l/WWqbSdlMC4N+9GdHmhto/sf02L6deBxwBq6X6q9PMmxTF3fMGqfrh4G2tt6Fqveu4CTgaPpTmftD/x5U8Oi15hkn76G91bVTdPsZ1RqHJnjWFW/3G/7DOCjwEPT7GPRj+EkNY7KMfxt4D1VtXlg/VzeX2ZVn8G4NNwPHDCw7gC6c+eLpqr+rqq2V9VDVfVe4Grg1GnqG0bt09XDQHtbz6LUW1X3V9VYVT1cVXcAbwB+sv9f86LXmGQvulNQ3+5rmW4/I1HjqB3HqtpVVZ8HjgBeP80+hvK9OFjjKBzDJCcBzwbePkHzXN5fZlWfwbg0fA3YO8lxzboT+d5TScNQQPo6ThxfmeRxwHK6uodR+2A9j6H7bOHGqrqX7hTSiU3/tp5Jt13AeqE7ltD9NapFrTFJgPcAjwVeUlXfmW4/I1TjoKEdxwF7N2ONxDGcosZBwziGp9B91vmNJN8EzgZekuS6CfaxO+8vU207ubl+iOtjcR7AB+iuvnoM8J9Y5KtSgQOB59JdebY38CpgB92H+McD99GdnnkMsJHvvWpsQWrv69iX7qq5i5vaDun38ZJ+3e/xvVcC/i7w13RX2T2J7gd//Cq7Kbedxxqf0h+7vYCD6K6q++yQanwXcA2w38D6UTqOk9U49OMI/ADdRSP7Acv6n5MdwItG5RhOU+MoHMMVwKHN4/eBD/fjz/r9ZbptJ61ntt+oPhb3AawGLuu/mb8BvHKR938I8A90pyC29m9Sz2naX9nXtQP4C2D1QtdOd+VaDTzW923PprvA4AG6q2nXNtstB/6s/4G5A3jjwLiTbjtfNQKvAL7eH5MtwPuAQxe7RrrPlQp4kO600/jjVaNyHKeqcRSOI93Pxl/T/VzcB9wAvG4m+1jEYzhpjaNwDCf5udnYLM/6/WWqbSd7+EfEJUlq+BmjJEkNg1GSpIbBKElSw2CUJKlhMEqS1DAYJUlqGIySJDUMRkmSGv8fgG3NpoQZaYMAAAAASUVORK5CYII=\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"p = valid_xs_final['ProductSize'].value_counts(sort=False).plot.barh()\n",
"c = to.classes['ProductSize']\n",
"plt.yticks(range(len(c)), c);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The largrest group is `#na#`, which is the label fastai applies to missing values.\n",
"\n",
"Let's do the same thing for `YearMade`. However, since this is a numeric feature, we'll need to draw a histogram, which groups the year values into a few discrete bins:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD7CAYAAABnoJM0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAXe0lEQVR4nO3df5BdZX3H8fcHAkncTSAQXIrW3UIDYZIQLKtoGZtNsQIyVGQ7HSBKIwNRnDh2iNCMBkgFFEWmUxlEwxADiIDUBERsWmlzUYax06SYwEpgmpKokMQAa8gNJBD99o9zlh5u98fd3bv35ub5vGbu5N7zPc+z53nuzX72/NizigjMzCxdBzV6A8zMrLEcBGZmiXMQmJklzkFgZpY4B4GZWeLGNXoDhmvq1KnR0dFRk752795NS0tLTfpqVqnPQerjB88BpDEH69atezEijuqv1nRB0NHRwdq1a2vSV6lUoqurqyZ9NavU5yD18YPnANKYA0lbBqr50JCZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeKa7jeLzcwaqWPxww372ptvOHtM+vUegZlZ4hwEZmaJcxCYmSXOQWBmljgHgZlZ4hwEZmaJcxCYmSXOQWBmljgHgZlZ4hwEZmaJcxCYmSXOQWBmljgHgZlZ4hwEZmaJqyoIJH1H0lZJr0h6VtIlhdrpkjZKelXSGknthdp4ScvzdtskXV7R74BtzcysPqrdI/gy0BERk4G/BK6TdIqkqcBK4CrgCGAtcF+h3VJgGtAOzAWulHQmQBVtzcysDqr6wzQR0VN8mT+OA04BeiLifgBJS4EXJU2PiI3ARcAnIqIX6JV0GzAfWA2cN0RbMzOrg6r/Qpmkb5B9E58IPAH8CLgeWN+3TkTslrQJmCFpO3BMsZ4/Pzd/PmOgtsBbgkDSAmABQFtbG6VSqdrNHlS5XK5ZX80q9TlIffzgOYDhzcGiWfvGdmMGMVbvU9VBEBGflvQZ4P1AF7AXaAV2VKy6E5iU1/peV9YYom3l114GLAPo7OyMrq6uajd7UKVSiVr11axSn4PUxw+eAxjeHMxv5J+qnNc1Jv0O66qhiPhdRDwGvBO4DCgDkytWmwzsymtU1PtqDNHWzMzqZKSXj44jO0fQA8zuWyippW95fl5ga7GeP+873zBg2xFuk5mZjcCQQSDp7ZLOl9Qq6WBJZwAXAP8OrAJmSuqWNAG4GthQONl7J7BE0hRJ04FLgRV5bai2ZmZWB9XsEQTZYaBfA73A14C/jYgHI2IH0E120rgXOBU4v9D2GmATsAV4FLgxIlYDVNHWzMzqYMiTxfk37DmD1B8Bpg9Q2wtcnD+G1dbMzOrDt5gwM0ucg8DMLHEOAjOzxDkIzMwS5yAwM0ucg8DMLHEOAjOzxDkIzMwS5yAwM0ucg8DMLHEOAjOzxFX9h2nMzPYnHTX8AzGLZu1r6B+caTTvEZiZJc5BYGaWOAeBmVniHARmZolzEJiZJc5BYGaWOAeBmVniHARmZolzEJiZJW7IIJA0XtLtkrZI2iXpCUln5bUOSSGpXHhcVdF2uaRXJG2TdHlF36dL2ijpVUlrJLXXfohmZjaYam4xMQ74FTAH+CXwYeB7kmYV1jk8Ivb103YpMA1oB44G1kj6RUSsljQVWAlcAjwEXAvcB7xvhGMxM7MRGHKPICJ2R8TSiNgcEb+PiB8CzwGnVNH/RcC1EdEbEU8DtwHz89p5QE9E3B8Re8hCY7ak6SMZiJmZjYwiYngNpDZgC3AysIcsFF4AAvgxcEVEvChpCvAycHREbM/b/hVwTUTMkvSPwKERcVmh76fy+vcrvuYCYAFAW1vbKffee++IBlupXC7T2tpak76aVepzkPr4oXnn4Mnnd9asr7aJsP21mnU3Zma947ARt507d+66iOjsrzasu49KOgS4G7gjIjZKagXeA/wcOBK4Ja+fAfR9sorv1k5gUv68FdhR8SWK9TdFxDJgGUBnZ2d0dXUNZ7MHVCqVqFVfzSr1OUh9/NC8c1DLu4UumrWPm57c/2/GvHle15j0W/XIJR0E3AW8DiwEiIgysDZfZbukhcBWSZOBcr58MtmeQ9/zXfnzcv66qFg3M7M6qOryUUkCbgfagO6IeGOAVfuOMykieoGtwOxCfTbQkz/vKdYktQDHFepmZlYH1f4ewa3AicA5EfHmkTRJp0o6QdJBko4Evg6UIqLvcNCdwBJJU/KTwJcCK/LaKmCmpG5JE4CrgQ0RsXH0wzIzs2pV83sE7cAnyU4Obyv8vsA84FhgNdnhnKeAvcAFhebXAJvITi4/CtwYEasBImIH0A1cD/QCpwLn12hcZmZWpSHPEUTEFkCDrHLPIG33Ahfnj/7qjwC+XNTMrIF8iwkzs8Q5CMzMEucgMDNLnIPAzCxxDgIzs8Q5CMzMEucgMDNLnIPAzCxxDgIzs8Q5CMzMEucgMDNLnIPAzCxxDgIzs8Q5CMzMEucgMDNLnIPAzCxxDgIzs8Q5CMzMEucgMDNLnIPAzCxxDgIzs8Q5CMzMEjdkEEgaL+l2SVsk7ZL0hKSzCvXTJW2U9KqkNZLaK9oul/SKpG2SLq/oe8C2ZmZWH9XsEYwDfgXMAQ4DrgK+J6lD0lRgZb7sCGAtcF+h7VJgGtAOzAWulHQmQBVtzcysDsYNtUJE7Cb7ht7nh5KeA04BjgR6IuJ+AElLgRclTY+IjcBFwCciohfolXQbMB9YDZw3RFszM6uDIYOgkqQ24HigB7gMWN9Xi4jdkjYBMyRtB44p1vPn5+bPZwzUFnhLEEhaACwAaGtro1QqDXez+1Uul2vWV7NKfQ5SHz807xwsmrWvZn21Taxtf2NlrN6nYQWBpEOAu4E7ImKjpFZgR8VqO4FJQGvhdWWNvD5Q27eIiGXAMoDOzs7o6uoazmYPqFQqUau+mlXqc5D6+KF552D+4odr1teiWfu46clh/1xcd5vndY1Jv1VfNSTpIOAu4HVgYb64DEyuWHUysCuvUVHvqw3V1szM6qSqIJAk4HagDeiOiDfyUg8wu7BeC3Ac2bH/XmBrsZ4/7xmq7YhGYmZmI1LtHsGtwInAORHxWmH5KmCmpG5JE4CrgQ2Fk713AkskTZE0HbgUWFFlWzMzq4Nqfo+gHfgkcDKwTVI5f8yLiB1AN3A90AucCpxfaH4NsAnYAjwK3BgRqwGqaGtmZnVQzeWjWwANUn8EmD5AbS9wcf4YVlszM6sP32LCzCxxDgIzs8Q5CMzMEucgMDNLnIPAzCxxDgIzs8Q5CMzMEucgMDNLnIPAzCxxDgIzs8Q5CMzMEucgMDNLnIPAzCxxDgIzs8Q5CMzMEucgMDNLnIPAzCxxDgIzs8Q5CMzMEucgMDNLnIPAzCxxVQWBpIWS1kraK2lFYXmHpJBULjyuKtTHS1ou6RVJ2yRdXtHv6ZI2SnpV0hpJ7TUbmZmZVWVcleu9AFwHnAFM7Kd+eETs62f5UmAa0A4cDayR9IuIWC1pKrASuAR4CLgWuA9437BGYGZmo1LVHkFErIyIB4CXhtn/RcC1EdEbEU8DtwHz89p5QE9E3B8Re8hCY7ak6cP8GmZmNgrV7hEMZYukAH4MXBERL0qaAhwDrC+stx44N38+o1iLiN2SNuXLNxY7l7QAWADQ1tZGqVSqyUaXy+Wa9dWsUp+D1McPzTsHi2b1dxBiZNom1ra/sTJW79Nog+BF4D3Az4EjgVuAu8kOIbXm6+wsrL8TmJQ/bwV2VPRXrL8pIpYBywA6Ozujq6trlJudKZVK1KqvZpX6HKQ+fmjeOZi/+OGa9bVo1j5uerJWPxePnc3zusak31GNPCLKwNr85XZJC4GtkiYD5Xz5ZGBP4fmu/Hk5f11UrJuZWR3U+vLRyP9VRPQCW4HZhfpsoCd/3lOsSWoBjivUzcysDqq9fHScpAnAwcDBkibky06VdIKkgyQdCXwdKEVE3+GgO4ElkqbkJ4EvBVbktVXATEnded9XAxsi4i3nB8zMbGxVu0ewBHgNWAx8LH++BDgWWE12OOcpYC9wQaHdNcAmYAvwKHBjRKwGiIgdQDdwPdALnAqcP7rhmJnZcFV1jiAilpJd3tmfewZptxe4OH/0V38E8OWiZmYN5FtMmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpa4qoJA0kJJayXtlbSiona6pI2SXpW0RlJ7oTZe0nJJr0jaJunyatuamVl9VLtH8AJwHbC8uFDSVGAlcBVwBLAWuK+wylJgGtAOzAWulHRmlW3NzKwOqgqCiFgZEQ8AL1WUzgN6IuL+iNhD9o1/tqTpef0i4NqI6I2Ip4HbgPlVtjUzszoYN8r2M4D1fS8iYrekTcAMSduBY4r1/Pm5Q7UFNha/iKQFwAKAtrY2SqXSKDc7Uy6Xa9ZXs0p9DlIfPzTvHCyata9mfbVNrG1/Y2Ws3qfRBkErsKNi2U5gUl7re11ZG6rtW0TEMmAZQGdnZ3R1dY1qo/uUSiVq1VezSn0OUh8/NO8czF/8cM36WjRrHzc9Odpvh2Nv87yuMel3tFcNlYHJFcsmA7vyGhX1vtpQbc3MrE5GGwQ9wOy+F5JagOPIjv33AluL9fx5z1BtR7lNZmY2DNVePjpO0gTgYOBgSRMkjQNWATMldef1q4ENEdF3jP9OYImkKflJ4EuBFXltqLZmZlYH1e4RLAFeAxYDH8ufL4mIHUA3cD3QC5wKnF9odw2wCdgCPArcGBGrAapoa2ZmdVDV2ZGIWEp2eWd/tUeAfi/5jIi9wMX5Y1htzcysPnyLCTOzxDkIzMwS5yAwM0vc/v8bFGa2X+uo4S92WWN4j8DMLHEOAjOzxDkIzMwS5yAwM0ucg8DMLHEOAjOzxPnyURszjbqscPMNZzfk65o1K+8RmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpa4mgSBpJKkPZLK+eOZQu1CSVsk7Zb0gKQjCrUjJK3Ka1skXViL7TEzs+rVco9gYUS05o8TACTNAL4FfBxoA14FvlFocwvwel6bB9yatzEzszoZ65vOzQMeioifAEi6Cnha0iTg90A3MDMiysBjkn5AFhqLx3i7zMwsp4gYfSdSCZgBCHgG+EJElCQ9CDweEV8prFsG5pAFweMRMbFQ+xwwJyLOqeh/AbAAoK2t7ZR777131NsMUC6XaW1trUlfzWos5+DJ53eOSb9DmfWOw6pe15+B0c9Bo97nWmqbCNtfa/RWDG04n+1Kc+fOXRcRnf3VarVH8HfAL8gO85wPPCTpZKAVqPyU7AQmAb8bpPYWEbEMWAbQ2dkZXV1dNdnoUqlErfpqVmM5B/MbdRvqeV1Vr+vPwOjnoFHvcy0tmrWPm57c/+/KP5zP9nDUZOQR8R+Fl3dIugD4MFAGJlesPhnYRbZHMFDNzMzqZKwuHw2yw0Q9wOy+hZKOBcYDz+aPcZKmFdrNztuYmVmdjDoIJB0u6QxJEySNkzQP+DPgX4C7gXMkfUBSC/BFYGVE7IqI3cBK4IuSWiSdBnwEuGu022RmZtWrxaGhQ4DrgOlkx/03AudGxDMAkj5FFghHAo8Anyi0/TSwHPgN8BJwWUR4j8DMrI5GHQQRsQN4zyD17wLfHaD2MnDuaLfBzMxGzreYMDNLnIPAzCxxDgIzs8Q5CMzMEucgMDNLnIPAzCxxDgIzs8Q5CMzMEucgMDNL3P5/31UzG1LHKG4FvWjWvgPiVtI2ct4jMDNLnIPAzCxxDgIzs8Q5CMzMEucgMDNLnIPAzCxxvnzUrIZGcxmnWaN4j8DMLHEOAjOzxDkIzMwS5yAwM0ucg8DMLHENDwJJR0haJWm3pC2SLmz0NpmZpWR/uHz0FuB1oA04GXhY0vqI6GnsZpmZpaGhQSCpBegGZkZEGXhM0g+AjwOLx+JrFq/zruftdzffcHZdvo4N71p+34LZDBQRjfvi0ruBxyNiYmHZ54A5EXFOYdkCYEH+8gTgmRptwlTgxRr11axSn4PUxw+eA0hjDtoj4qj+Co0+NNQK7KxYthOYVFwQEcuAZbX+4pLWRkRnrfttJqnPQerjB88BeA4afbK4DEyuWDYZ2NWAbTEzS1Kjg+BZYJykaYVlswGfKDYzq5OGBkFE7AZWAl+U1CLpNOAjwF112oSaH25qQqnPQerjB88BJD4HDT1ZDNnvEQDLgb8AXgIWR8R3G7pRZmYJaXgQmJlZYzX6HIGZmTWYg8DMLHEHRBBIWihpraS9klZU1C6R9N+SypJWSzqmULtC0lOSdkl6TtIVFW07JK2R9KqkjZI+WKchDdtI56CwzqH5GH9dsfxkSevyOVgn6eQxHsqIjGb8kv5E0k/y+nZJny3UDvjPgKTxkr6Zj/1lSQ9Jekeh3jT3A8vHcnu+nbskPSHprEL99Px9fDV/X9sr2i6X9IqkbZIur+h7wLbN7oAIAuAF4Dqyk85vkjQH+BLZlUhHAM8B9xRXAS4CpgBnAgslnV+o3wM8ARwJfAH4J0n9/mbefmCkc9DnCuA3FW0PBR4EvkM2R3cAD+bL9zcjGr+kqcBq4Ftk7/MfA/9a6CKFz8BngfcDJwHHAL8Fbi7Ui/cDmwfcKmnG2Axh1MYBvwLmAIcBVwHfywN9KtlVileRzcNa4L5C26XANKAdmAtcKelMePNzMljb5hYRB8yD7D/BisLrrwG3FF4fAwRw3ADtvw7cnD8/HtgLTCrUfwp8qtHjrPUcAH8EPA2cBfy6sPxDwPPkFxXky34JnNnocdZq/GTfIO8aoK8kPgPArcBXC/WzgWfy5y1kIXB8oX4XcEOjxzmM+dhAdk+zBWS3tOlb3gK8BkzPXz8PfKhQvxa4N38+aNtmfxwoewQDUf4ovgaY+f9WlAR8gP/7ZbYZwP9ERPG3nNfny5tJNXNwM/B5sg920QxgQ+Sf/NwGmmsOhhr/+4CXJT0u6Tf5YZF35bVUPgO3A6dJOkbS28h+6v/nvHY88LuIeLbQvmnmQFIb2Rh6yLZ5fV8tst9j2gTMkDSFLCDXF5oXxzlg27Hc/no50IPgR8BfSzpJ0kTgarKfhN7Wz7pLyebj2/nrqu6D1AQGnQNJHwXGRcSqftoeCHMw1GfgncDfkB0eeRdvPWxyIIwfhp6DZ8n29J4HXgFOBL6Y15p2DiQdAtwN3BERGxl8LK2F15U1hmjb9A7oIIiIfwOuAb4PbAE2k93HqPKE6EKycwVnR8TefPEBcR+kweZA2W3Avwp8ZoDmTT8HVXwGXgNWRcR/RsQe4O+BP5V0GAfA+KGqObgVmEB2HqSF7Fh43x5BU86BpIPIDmG9DizMFw82lnLhdWVtqLZN74AOAoCIuCUipkXE28n+I4wDnuqrS7qY7G8fnB4RxYDoAY6VVEz8prwP0iBzMA3oAH4qaRvZN4A/yK+Y6CAb60n5YbM+J9FkczDEZ2AD2U/Hb66e/yvS+AxANqYVEfFy/oPQzcB78xOkTXc/sPzzejvZye3uiHgjL/WQbXvfei3AcUBPRPQCW4t13jrOAduO0TDqq9EnKWrxIPtQTwC+TPZTwITCsplk/6nfBZSALxXazQO2AScO0O/PyE60TQA+SnY1xVGNHm+t5iCvH114nEd25cnRwMHAoWQ/QX4WGE/2k9UW4NBGj7eGn4E/B3rJ/jreIcA/AD9N5TOQt/s2WTgcls/B54HnC/V7yQ6XtQCnkR0SmdHo8Q4yD9/M37fWiuVH5dvenc/JV4CfFeo3AI+SXSE3nSwYzqymbbM/Gr4BNXrjl5L9JFd8LAUOJ/uJbzfZN/wvAwcX2j0HvEG229f3+Gah3pH/p3mN7I/hfLDRY631HFT00UXhqqF82buBdfkc/Bfw7kaPtdbjBy4jOz7eCzwE/GFKnwGyQ0J3k10+/FvgMeC9hfoRwAN5+18CFzZ6rIPMQXs+7j0V/6/n5fUPAhvz97MEdBTajie79PYVYDtweUXfA7Zt9ofvNWRmlrgD/hyBmZkNzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiftfGkutOUh/ddgAAAAASUVORK5CYII=\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"ax = valid_xs_final['YearMade'].hist()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Other than the special value 1950 which we used for coding missing year values, most of the data is after 1990.\n",
"\n",
"Now we're ready to look at *partial dependence plots*. Partial dependence plots try to answer the question: if a row varied on nothing other than the feature in question, how would it impact the dependent variable?\n",
"\n",
"For instance, how does `YearMade` impact sale price, all other things being equal?\n",
"\n",
"To answer this question, we can't just take the average sale price for each `YearMade`. The problem with that approach is that many other things vary from year to year as well, such as which products are sold, how many products have air-conditioning, inflation, and so forth. So merely averaging over all the auctions that have the same `YearMade` would also capture the effect of how every other field also changed along with `YearMade` and how that overall change affected price.\n",
"\n",
"Instead, what we do is replace every single value in the `YearMade` column with 1950, and then calculate the predicted sale price for every auction, and take the average over all auctions. Then we do the same for 1951, 1952, and so forth until our final year of 2011. This isolates the effect of only `YearMade` (even if it does so by averaging over some imagined records where we assign a `YearMade` value that might never actually exist alongside some other values). \n",
"\n",
"> A: If you are philosophically minded it is somewhat dizzying to contemplate the different kinds of hypotheticality that we are juggling to make this calculation. First, there's the fact that *every* prediction is hypothetical, because we are not noting empirical data. Second, there's the point that we're *not* merely interested in asking how would sale price change if we changed `YearMade` and everything else along with it. Rather, we're very specifically asking, how would sale price change in a hypothetical world where only `YearMade` changed. Phew! It is impressive that we can ask such questions. I recommend Judea Pearl's recent book on causality, *The Book of Why*, if you're interested in more deeply exploring formalisms for analyzing these subtleties.\n",
"\n",
"With these averages, we can then plot each of these years on the x-axis, versus each of the predictions on the Y axis. This, finally, is a partial dependence plot. Let's take a look:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAt4AAAEOCAYAAAC+QhDtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3xUZdr/8c+VXgmBJLQAofcasIPYUbEg7Oo+9vrsrt3VdX+uuth2fbbp7urquruKZa2AqKjoWrFL70U6SYBQA0lIv39/zARDTGBSJmeSfN+v13nNzDlnznyHkMmVk/tctznnEBERERGR4ArzOoCIiIiISGugwltEREREpAmo8BYRERERaQIqvEVEREREmoAKbxERERGRJqDCW0RERESkCajwFhERERFpAhFN9UJmdgNwBTAEeMk5d0WVbacAjwPdgG+AK5xzmw5zrJuBW4A0YDNwnnNuzeFePyUlxWVkZDTsTYiI1FP23gPsLSxlYKc2mNXtufPnz9/pnEsNTrLQpM9sEWmuDveZ3WSFN5ADPAicAcRWrjSzFGAGcA3wFvAA8ApwTE0HMbNrgKuBs4GVQE9gz5FePCMjg3nz5jXsHYiI1INzjmN+9yHnd0vmiUsy6/x8M6v1RERLpc9sEWmuDveZ3WSFt3Nuhj/MKCC9yqYLgOXOudf826cAO82sv3NuVdVjmFkY8Bt8Z8RX+FevC3Z2EZGGWJa9j+37ijllQAevo4iIiIdCYYz3IGBx5QPnXAG+YnpQDfum+5fBZrbFzDaY2X3+gvwHzOw6M5tnZvN27NgRjOwiIkf0wcrtmMFJ/VrVaJE602e2iLR0oVB4JwB51dblAYk17Ft5pvx0fGPFTwJ+gm/oyQ84555yzo1yzo1KTdUPPBHxxoertjOyWzLtE6K9jhLS9JktIi1dKBTe+UCbauvaAPtr2PeA//b3zrm9zrmNwD+As4IXT0Sk/rblFbEsex+nDEjzOoqIiHgsFArv5cCwygdmFg/08q+vbjVQArimiSYi0jAfrcoF4JT+Gt8tItLaNVnhbWYRZhYDhAPhZhZjZhHA6/jGbE/yb78XWFL9wkoA51whvo4nvzSzRDNLB64FZjXV+xARqYsPV24nPTmWvh0SvI4iIiIea8oz3nfjGyryK+AS//27nXM7gEnAQ/jaAh4NXFT5JDN70syerHKcG/ANT8kBvgJeBJ5uijcgIlIXB0rK+XztTk4d0AGra/NuERFpcZqyneAUYEot2z4A+tey7afVHu+jSmEuIhKqvli7k+KyCo3vFhERIDTGeIuItEgfrtpOQnQER/do73UUEREJASq8RUSCwDnHhytzGds3hagIfdSKiIgKbxGRoFiWvY/c/cWcrG4mIiLip8JbRCQINFuliIhUp8JbRCQINFuliIhUp8JbRKSRabZKERGpiQpvEZFGVjlb5akDNL5bRES+p8JbRKSRVc5W2SdNs1WKiMj3VHiLiDQizVYpIiK1UeEtItKINFuliIjURoW3iEgj0myVIiJSGxXeIiKNpKJCs1WKiEjt9JNBRKSRLM/xzVZ5imarFBGRGqjwFhFpJJWzVY7TbJUiIlIDFd4iIo1Es1WKiMjhqPAWEWkEmq1SRESORIW3iEgj+HDVdkCzVYqISO1UeIuINIKPVubStZ1mqxQRkdqp8BYRaaDK2SpP6a/ZKkVEpHYqvEVEGkizVYqISCBUeIuINJBmqxQRkUCo8BYRaQDNVikiIoHSTwkRkQZYlpOn2SpFRCQgKrxFRBrgw5W5mMFJ/TW+W0REDk+Ft4hIA1TOVtkuPsrrKCIiEuJUeIuI1JNmqxQRkbpQ4S0iUk+arVJEROpChbeISD19qNkqRUSkDlR4i4jUw4GScr7QbJUiIlIHKrxFROpBs1WKiEhdqfAWEakHzVYpIiJ1pcJbRKSONFuliIjUh35iiIjUkWarFBGR+lDhLSJSRx+szCVMs1WKiEgdqfAWEamjjzRbpYiI1IMKbxGROqicrfJkdTMREZE6arLC28xuMLN5ZlZsZlOrbTvFzFaZWaGZfWxm3QM43olm5szswaCFFhGpRrNViohIfTXlGe8c4EHg6aorzSwFmAHcA7QD5gGvHO5AZhYJ/AX4JihJRURqodkqRUSkvpqs8HbOzXDOzQR2Vdt0AbDcOfeac64ImAIMM7P+hzncL4D3gVVBCSsiUoPCkjLNVikiIvUWCmO8BwGLKx845wqAdf71P+AfhnIVcP+RDmxm1/mHt8zbsWNHI8UVkdbqvyu2U1xWwfjBHb2O0iLpM1tEWrpQKLwTgLxq6/KAxFr2/ytwj3Mu/0gHds495Zwb5ZwblZqa2sCYItLazVyYTeekGI7KaOd1lBZJn9ki0tKFQuGdD7Sptq4NsL/6jmZ2DpDonDvsGHARkca2M7+YOd/t5LwRXQgL0zATERGpuwivAwDLgcsrH5hZPNDLv766U4BRZrbN/zgJKDezIc6584KeVERarVmLcyivcEwc0cXrKCIi0kwFdMbbzKLN7CEzW29mef51p5vZDYG+kJlFmFkMEA6Em1mMmUUArwODzWySf/u9wBLnXE0XTt4D9AWG+5c3gX8CVwaaQ0SkPl5flMOATm3o26G2UXAiIiKHF+hQk0eAwcDFgPOvWw78rA6vdTdwAPgVcIn//t3OuR3AJOAhYA9wNHBR5ZPM7EkzexLAObffObetcvEfo8A5t7sOOURE6mTDzgIWb9nLxBGdvY4iIiLNWKBDTSYCvZ1zBWZWAeCcyzazgP/m6pybgq9VYE3bPgBqbB/onPvpYY55RaCvLyJSXzMXZmMG5w7TMBNpfCVlFeQdKCU6Moyo8DCiI8LUrlKkhQq08C6pvq+ZpfLDntwiIi2Kc46Zi7I5tmd7OibFeB1HWpjcfUVMfvIrNu8uPGR9VHgYURG+Jdp/GxUedrA4960PP2Sf6IP7hFfZp+pxwg85TnQtxznkfrh+CRBpTIEW3q8Bz5rZrQBm1gl4FHg5WMFERELBoi172bSrkOtP6u11FGlhCkvKuPrZeezML+bXZw3ADIrLKiguq6DEvxSXlfvul1dQXOq7rVy/t7Dk4L7FB/cp992WVeDckTMEovIsfE3FeXxUBMlxUSTHR5IUG0VyXCTJcVG0jYskOd73uG1cFG1jI4kID4VGaiLeCrTwvgv4PbAUiAO+w3dR4xEnsRERac5mLswmOiJMk+ZIoyqvcNz00kKW5+Txr8tHcXL/Do16fOccZRWuSgHvL+bLyyk6pIA/fJFfXH1b2aHrCorLWbcjnz2bStlbWEJZRe3VfmKMv0iPiyQprlqRXuW2auEeHxWuM+7SogRUeDvnSoBbgFv8Q0x2OtdYv0uLiISm0vIK3lqylVMHdqBNTKTXcaQFeWDWCj5Ymcv95w1q9KIbwMyIDDciw8OIj270w9fIOUd+cRl7C0vZW1jKnsIS9hSWHLz//Tpfkb5xZwF7CkvYX1RW6zEjw+2HZ9LjomgbH3mwiK88o54c//32SJ1dlxAVUOFtZpcBi5xzS/xdSDCzYcBQ59zzwQwoIuKVz77bwe6CEs4frosqpfE888UGpn65katP6MFlx2Z4HafRmBmJMZEkxkTStQ6Tu5aW+y4u3esvyvcUHFqk5x0oYU+B7/GmXYUs2rKXvYWllJRX1HrMhOiIGs6kf3+mfWh6EpndNQOtNL1Ah5o8gK9vdlVb8PXRVuEtIi3SzIU5tI2L5MS+mr5cGsd/V2zn/lkrOGNQB+46a4DXcUJCZHgYKQnRpCQEfmreOUdhSfnBM+nfn1X3F+/VzrBv3l3I3sJS8g6UHjzGFcdl8Ksz+xMTGR6MtyVSo0AL7zbAvmrr8oC2jRtHRCQ05BeX8f6KbUzOTCcqQn+2loZbmpXHTS8tZGiXJB69cAThYRq7XF9mRnx0BPHREaQnB/688grHnsISHvtoLVO/3MhX63bx15+MoF9HTYwlTSPQnyYr8E1yU9VEYGXjxhERCQ3vLdtGUWmFhplIo8jaU8hVz86lXXwU/7p8NLFROsvqhfAwIyUhminnDuKZK0ezq6CEcx77nKlfbECXrklTCLTwvhP4l5lNN7Pfm9kM4N/AL4IXTUTEOzMXZZOeHEtm9zqcThOpwb6iUq6aOpei0nKmXjma1MQmutpRDuukfmnMvmUMJ/ROYcpbK7hy6lx27C/2Opa0cAEV3s65z/FNGT8XiAe+BQY7574IYjYREU/k7ivii7U7mTiii1qZSYOUllfw8xcWsH5HAU9ekkmfDhrSEEpSEqL59+WjuP+8QXy1bhfjH53DR6u2ex1LWrBAx3jjnNsMPBzELCIiIeHNxTlUODhPw0ykAZxz3P36Mj5fu5M/TB7K8b1TvI4kNTAzLjs2g2N6tuemlxZy1dR5XHZsd+46a4AuvJRGF2g7wXbA7fg6myRU3eacGxuEXCIinpm5KJshXZLonZZw5J1FavH3T9bxyrwt3Hhyb340qqvXceQI+nZIZOb1x/OH91bz7883HLzwckCnNl5HkxYk0DPeLwLRwKtAYfDiiIh4a23ufpZl7+OeCQO9jiLN2JuLc/jDe6s5b3hnbjutr9dxJEAxkeHcM2EgY/umcvtriznvsS+488z+XHlcBmHqQiONINDC+zgg1Tmnqw5EpEWbuTCHMINzhnXyOoo0U/M27ub21xYzOiOZ308equsEmqET+6Yy++Yx3Dl9CQ/MWsGna3bwx8lDSWsT43U0aeYC7WqyBEgPZhAREa9VVDhmLsrmhD6ppCXqB6zU3cadBVz73Dy6tI3lqUtHER2hMcLNVfuEaP552SgeOH8w36zfxfi/fMYHK3ThpTRMoGe8PwJmm9kzwLaqG5xzTzd6KhERD8zfvIesPQc0NEDqZU9BCVdOnQvAM1eMJjk+yuNE0lBmxqXHdOfYnu246aVFXPPcPC45phu/PmugerFLvQRaeI8BsoDTqq13gApvEWkRZi7MJjYynDMGdfQ6ijQzxWXl/O/z88nee4AXrzmajJR4ryNJI+qdlsjr1x/HH99bzT8/28DX63fzl4uGM6hzktfRpJkJqPB2zp0U7CAiIl4qKatg1pKtnD6oA/HRAXdaFcE5xy+nLeHbjbv5209GMCqjndeRJAiiI8L59dkDObFvGre9uoiJj3/JL8f346rje+jCSwlYoGO8MbP2Znapmd3hf9zZzDTuW0RahE9W55J3oFRTxEud/fm/a3hjUQ53nNGPc4Z19jqOBNkJfVKYfctYTuyXyoNvr+TyZ75l+74ir2NJMxFQ4W1mJwKrgYuBe/2r+wBPBCmXiEiTemNRDu3jozihjyY5kcC9Om8Lf/toLReO6srPx/XyOo40kXbxUTx1aSa/nTiEuRt3M/7ROby/fNuRnyitXqBnvB8FLnTOjQfK/Ou+AY4KSioRkSa0r6iU/67czjnDOhMZHvAfAqWV+2LtTu6asZQxfVJ4cOJgtQ1sZcyM/zm6G7NuHEOX5Fiue34+d72+lAMl5V5HkxAW6E+YDOfch/77zn9bQh2mnBcRCVWzl26jpKyC80domIkE5rvt+/npC/PplZrA4xeP1C9srVjvtARm/Ox4/vfEnrz07WbO/ttnLMvO8zqWhKhAPylWmNkZ1dadCixt5DwiIk3u9YXZZLSPY1i6OhTIkeXuL+KKZ+YSExnO01eOpk1MpNeRxGNREWH8vzMH8J+rj6aguIyJf/+Cf3y6jooKd+QnS6sSaOH9C+A/ZvYsEGtm/wCmAncEK5iISFPYmneArzfs4vwRXTRUQI7oQEk51z47j90FJTx9+Wi6tI31OpKEkON6pzD75rGc3D+N3727ikv+/Q3b8nThpXwvoMLbOfc1MAxYjq9v9wbgKOfc3CBmExEJujcX5eAc6mYiR1Re4bj55YUsyc7jrz8ZwRD9hURqkBwfxZOXZPLwBUNYuHkv4/8yh9nLdOGl+AQ8Rts5lw38PohZRESa3OsLsxneta0mPJEj+u07K3l/xXZ+c85AThvYwes4EsLMjIuO6sZRPdpx88uL+OkL87lodFfuPWcgcVG6PK41q/Wrb2bP8/2FlLVyzl3WqIlERJrIqm37WLVtP/edO8jrKBLinvtqI//+fANXHJfBlcf38DqONBM9UxOY/rPjeOSDNTz56Tq+3bCbRy8aztD0tl5HE48cbqjJWmCdf8kDzgfC8U0dHwacB+wNdkARkWCZuTCH8DBjwtBOXkeREPbhyu1MeXM5pw7owD0TBnodR5qZqIgw7hzfnxevOYYDpeVc8PcveeKTdZTrwstWqdYz3s65+yrvm9l7wNnOuc+qrDsBuCe48UREgqOiwvHGomxO7JtK+4Ror+NIiFqWnceNLy1kUOck/vqT4YRranCpp2N7tefdm8dw1+tL+b/Zq/h0TS6PXDicTkm6QLc1CbSryTHA19XWfQMc27hxRESaxjcbdrM1r4jzhmuKb6lZzt4DXDV1LslxUfz78lEamysN1jYuisf/ZyS/nzyUJVl5jH/0M95dutXrWNKEAi28FwK/NbNYAP/tQ8CiYAUTEQmmNxZlEx8VzukDO3odRULQ/qJSrpo6lwMl5Tx9xWjS2sR4HUlaCDPjx6O68s5NY8hoH8fP/rOAX05bTEFx2ZGfLM1eoIX3FcDxQJ6Zbcc35vsE4PIg5RIRCZqi0nLeXrqVMwZ3JDYq3Os4EmJKyyu4/sWFrM3N5++XjKRfx0SvI0kLlJESz7SfHccNJ/XmtflZnP3Xz1i8RZfOtXSB9vHe6Jw7DugFnAv0ds4d55zbENR0IiJB8PGqXPYXlal3t/yAc45731jOnDU7eGjiYMb0SfU6krRgkeFh3H5GP16+9hhKyiqY9MSXPP7xWl142YIFesa7UjGwA4gws55m1jMImUREgmrmomxSEqI5rld7r6NIiPnHnPW89O1mfj6uFxeO7uZ1HGklju7ZnndvHssZgzvyh/dW85N/fk323gNex5IgCKjwNrPxZpYNbMPXZrBy+S6I2UREGt3+olI+Xr2DCUM7ERFe13MP0pK9vWQrD7+7iglDO3H76f28jiOtTFJcJI/9ZAR//NEwlmfnceajc5i1JMfrWNLIAv2p8zjwABDvnAursmhwpIg0Kx+s3E5JWYV6d8sh5m/aw62vLmJU92T++KNhhKltoHjAzJicmc47N4+hZ2oCN7y4kF+8uph8XXjZYgRaeCcD/3DO1fvvHmZ2g5nNM7NiM5tabdspZrbKzArN7GMz617LMdLM7CUzyzGzPDP7wsyOrm8mEWl9Zi3eSqekGEZ2S/Y6ioSITbsKuPa5eXROiuGpy0YRE6lzSuKt7u3jee2nx3LTyb15fWEWZ/3lMxZu3uN1LGkEgRbe/waubOBr5QAPAk9XXWlmKcAMfJPxtAPmAa/UcowEYC6Q6d/3WeBtM0toYDYRaQXyCkuZ890Ozh7SSWc0BYC9hSVcOXUuFc7xzJVH0S4+yutIIoDvwsvbTu/HK/97LOUVjslPfsVTc9Z5HUsaqC4T6DxhZmvMbE7VJdAXcs7NcM7NBHZV23QBsNw595pzrgiYAgwzs/41HGO9c+7Pzrmtzrly59xTQBSgwXgickTvrdhGabljwjBNmiNQXFbOdc/PJ2v3Af552Sh6pMR7HUnkB0ZntOOdm8dwcv80fvvOKjbtKvA6kjRAoNNw/cu/BMMgYHHlA+dcgZmt869fdbgnmtlwfIX32lq2XwdcB9Ctm65OF2ntZi3ZStd2sQxLT/I6itSgKT+znXP8avpSvt2wm79cNJzRGe2C+noiDZEUG8n95w3ig5Xbmb4gm9tO6+t1JKmngApv59yzQcyQgK9FYVV5wGFnLDCzNsDzwH3Oubya9vGfEX8KYNSoUWqKKdKK7S4o4Yu1O7l2TE/MNMwkFDXlZ/ajH3zH6wuzuf30vpynfu7SDHRKiuWE3inMWJDFLaf00XC5ZirQdoJmZtea2UdmtsS/bqyZ/bgRMuQDbaqtawPsP0yeWOAt4Gvn3O8aIYOItHCzl22jvMKpm4kwfX4Wf/nwO348Kp3rT+rtdRyRgE3OTCdrzwG+2bDb6yhST4GO8b4fuBrfmYjKv/9lAXc2QoblwLDKB2YWj2+GzOU17Wxm0cBMIBv430Z4fRFpBWYtyaFHSjyDOlf/PV9ak6/W7eJXM5ZwfO/2PDRxiP76Ic3K6QM7khgdwbT5WV5HkXoKtPC+ApjgnHsZqPzz3wYg4JkrzSzCzGKAcCDczGLMLAJ4HRhsZpP82+8FljjnfjC+28wigWnAAeAy51xFoK8vIq3Xjv3FfL1+FxOGdlKh1Yqtzd3P/z4/j4z28fz94kwiNYGSNDOxUeGcPbQT7y7bSoF6ezdLgX7qhOMbEgLfF94JVdYF4m58BfOvgEv89+92zu0AJgEPAXuAo4GLKp9kZk+a2ZP+h8cBE4DTgb1mlu9fxtQhh4i0Mu8u20qFgwlD1c2ktdqZX8yVU+cSFRHOM1eOJik20utIIvUyOTOdwpJy3l22zesoUg+BdjV5B/izmd0KvjHf+GayfCvQF3LOTcHXKrCmbR8AP2gf6N/20yr3PwV0ukpE6mTW4q30SUugX8fDXrMtLVRRaTnXPDuPHfuLeeW6Y0lPjvM6kki9ZXZPJqN9HNPmb2FyZrrXcaSOAj3jfRvQGV+3kSR8Z7q70zhjvEVEgmZbXhFzN+3W2e5WqqLCcesri1ictZe/XjSCYV3beh1JpEHMjEkj0/l6/W627C70Oo7UUUCFt3Nun3PufHwXVh4D9HLOTXTO1dp5REQkFLy9dCvOwYRh6mbSGj08exXvLtvG3WcP5PRBHb2OI9IoLshMxwxmLMj2OorUUcBXlphZW+A0YBxwipklByuUiEhjmbUkhwGd2tArNcHrKNLEXvh6E0/NWc9lx3bnquMzvI4j0mi6tI3luF7tmbZgCxUVmqakOQm0j/fJwEbgJmA0cCOwwcxOCV40EZGGydpTyMLNe9W7uxX6eHUu976xjJP7p3HvhIHqZiMtzuTMdLbsPsDcjerp3ZwEesb7MeA659zRzrkfO+eOAa4FHg9eNBGRhnl7yVYAztH47lZlRc4+bvjPAgZ0asPffjKCCLUNlBbojEEdiY8KV0/vZibQT6POwPRq614HNGBORELWrCVbGZqeRLf26mLRWmzNO8BVU+fSJjaSp68YTXx0oM27RJqXuKgIzh7aiXeWbqWwRD29m4tAC+/ngOurrfuZf72ISMjZuLOApdl5GmbSiuQXl3HV1HnkF5fx9BWj6dAmxutIIkE1ObMrBSXlzFZP72Yj0MJ7JPAnM8sys2/MLAv4EzDCzOZULsGLKSJSN28v9Q0zOVvDTFqFsvIKbnhxAWu27+fxi0cyoFMbryOJBN3ojGS6tYvTcJNmJNC/wf3Tv4iINAtvLc5hZLe2dGkb63UUaQL5xWXsyi/hgfMGc2LfVK/jiDSJyp7ej364hqw9hZocqhkIqPB2zj0b7CAiIo1lbW4+q7bt594JA72OIk2kbVwUM35+HJG6kFJamQtGduGRD9bw+oJsbjylj9dx5AgCbSdoZnatmX1kZkv868aa2Y+DG09EpO7eXrIVMzhb47tbFRXd0hp1bRfHsT3bM21BFs6pp3eoC/RT6n7gauApfLNXAmShKeNFJATNWpLD6Ix2urhORFqFSZnpbNpVyLxNe7yOIkcQaOF9BTDBOfcyUPnr1AagZzBCiYjU1+pt+/kuN59zdLZbRFqJMwd3JC4qnOm6yDLkBVp4hwP5/vuVhXdClXUiIiFh1pIcwgzGD1bhLSKtQ3x0BGcN6cSsJVs5UFLudRw5jEAL73eAP5tZNPjGfAMPAG8FK5iISF0555i1ZCvH9mpPamK013FERJrMpJHp5BeX8d5y9fQOZYEW3rfhm70yD0jCd6a7OxrjLSIhZHnOPjbsLGCCeneLSCtzdI92pCfHMn2BhpuEsoAKb+fcPufc+fiK7WOAXs65ic65/UFNJyJSB7OWbCUizBg/qKPXUUREmlRYmK+n9+drd5Kz94DXcaQWtRbeZhZWfQF2APOB3CrrREQ85xtmksPxvVNIjo/yOo6ISJObNDId5+D1hdleR5FaHK5wLgNKA1hERDy3OCuPrD0HmKBuJiLSSnVrH8dRPdoxbb56eoeqwxXePfC1C+wJ3Ah8CowHBvhvPwZuCHZAEZFAzFqcQ1R4GKdrmImItGKTM9PZsLOABZv3eh1FalBr4e2c21S54Lu48gLn3H+dc2ucc/8FfgTc3lRBRURqU1HheHvpVsb2TSEpNtLrOCIinjlrSCdiI8OZpp7eISnQMdpJQFy1dXH+9SIinlqweQ9b84rUzUREWr2E6AjOHNKRWYtzKCpVT+9QE2jh/SzwgZldZ2Znmtl1wHv+9SIinpq1ZCvREWGcOrCD11FERDw3eWQ6+4vLeH/Fdq+jSDURAe73S2AtcCG+ft5bgceAfwYpl4hIQMr9w0xO6pdGQnSgH2kiIi3XMT3b06VtLNPmZ3HuMP0lMJQE9FPKOVcBPOlfRERCxrcbdrNjfzEThqmbiYgIVPb07sJjH69lW14RHZNivI4kfurDLSLN2qwlOcRGhnNy/zSvo4iIhIwLRqZT4WDGQl1kGUpUeItIs1VWXsHsZds4ZUAacVEaZiIiUikjJZ7RGclMV0/vkKLCW0Sara/W72JXQYm6mYiI1GByZjrrdhSwaIt6eocKFd4i0mzNWryVhOgIxvVL9TqKiEjIOWtIJ2Iiw9TTO4TU+rdZM7s/kAM45+5tvDgiIoEpKatg9vJtnDawAzGR4V7HEREJOYkxkYwf1JG3Fudwz4SB+qwMAYc74901wEVEpMnNWJBF3oFStcoSETmMyZld2VdUxgcr1dM7FNR6xts5d2VTBhERCVTegVL+8N5qRmcka5iJiMhhHNurPZ2TYpg2P0vXw4SAOo3xNrNEM+thZj0rl2AFExGpzV8//I7dhSX85pxBmJnXcUREQlZ4mDFxZBfmrNnB9n1FXsdp9QIqvM1soJktBPLwzWC5FvjOv4iINJm1uft59suNXDS6K4O7JHkdR0Qk5E3y9/SeuTDb6yitXqBnvP8OfAy0A/YBycA/gMuDlEtE5Aecc9w/ayWxUeIY1/kAACAASURBVOHcfno/r+OIiDQLPVMTyOyezDT19PZcoIX3MOBO59xewJxzecAdwANBSyYiUs2HK3OZs2YHt5zal/YJ0V7HERFpNiZnpvNdbj5LsvK8jtKqBVp4FwGR/vs7zayb/7ntA30hM7vBzOaZWbGZTa227RQzW2VmhWb2sZl1P8xxMvz7FPqfc2qgGUSk+SouK+eBt1fQOy2By46t9SNCRERqcPbQTkRHhDF9gXp6eynQwvsz4Mf++9OAd4FPgY/q8Fo5wIPA01VXmlkKMAO4B99QlnnAK4c5zkvAQnxF/6+BaWamtgYiLdzTn29k065C7p0wkMhwzf0lIlIXbWIiOWNQR95YlENxWbnXcVqtgH56Oed+7Jyb6n94F/Aw8E/g4kBfyDk3wzk3E9hVbdMFwHLn3GvOuSJgCjDMzPpXP4aZ9QVGAr9xzh1wzk0HlgKTAs0hIs1P7r4iHvvoO04d0IGxffV7tohIfUzOTCfvQCkfrsz1OkqrVefTRs65Cufc8865J5xzBY2QYRCwuMrxC4B1/vU17bveObe/yrrFteyLmV3nH94yb8eOHY0QVUS88PDsVZSWO+6ZMMDrKBJE+swWCa7je6fQsU2MppD30OGmjH/KOXed//7zQI2XwTrnLmtghgSg+idsHpBYy77VrwrIA7rUku0p4CmAUaNG6TJekWZo4eY9zFiQzc/G9aJ7+3iv40gQ6TNbJLgqe3o/NWc9ufuLSEuM8TpSq1Nr4Q1sqHJ/bRAz5ANtqq1rA+xv4L4NkrWnkLteX0a/Dgn069iGfh0S6dMhgZjI8MZ+KRGpRUWFY8qby0lLjOb6k3p7HUdEpNmbNDKdJz5ZxxsLc7h2rOZBbGqHmzL+d1Ue/sM5t636PmbWsREyLKdKP3Aziwd6+dfXtG9PM0usMtxkGPBiI+Q4xN7CUnblF/Ps+l2UlFUAEGbQvX08/Tok0rdjIv07JtK3QyIZ7eOI0MVeIo1u+oIsFmfl8ciFw0iIPtx5AhERCUTvtARGdGvLtPlZXDOmh2b/bWKB/iRbww/PNAOswNeJ5IjMLML/euFAuJnFAGXA68AfzGwS8DZwL7DEObeq+jGcc2vMbBHwGzO7GzgTGEoQLq4c3CWJt28aQ1l5BZt2F7J62/6Dy5rt+3l/xTYq/H8IjYoIo3dqgq8Q75hIv46JjOyWTFJs5OFfRERqtb+olP+bvZqR3dpy/vAaR5OJiEg9TBqZzt0zl7E8Z59mAG5igRbeP/h1yMzaABV1eK27gd9UeXwJcJ9zboq/6H4MeAH4Brioyus8CeCc+6l/1UXAVGAPsBmY7JwL2lU4EeFh9EpNoFdqAmcN6XRwfVFpOWtz81m1bT+rt+1j9fZ8vli3kxn+6Vg7JcUw68YTNMmHSD397aO17Coo5ukrRumMjIhIIzpnaGfun7WCafOzVHg3scMW3ma2Bd9FlbFmtrna5vb4emoHxDk3BV+rwJq2fQD8oH2gf9tPqz3eCIwL9HWDJSYynMFdkn7wH3ZvYQnzNu7h5y8u4JZXFvHslUcRFqaiQaQu1u/I55kvNvCjzHSGprf1Oo6ISIuSFBfJ6QM7MHNRNv/vrP5ER+j6taZypIHJlwCXASXApVWWS4CRzrlrghuv+WkbF8WpAztw37mD+Oy7nTz+cTCvSxVpmR6YtYKYiHDuOKPG38dFRKSBJmWms7ewlI9Xqad3UzrsGW/n3KdmFg68BXztnCtumljN30Wju/Ltht088sEaMrsnc1zvFK8jiTQLH6/K5ePVO/j1WQNITdRQLRGRYBjTO4W0xGimzc9m/OBOR36CNIojtuJwzpUDY6nbeO5Wz8x48PzB9ExN4KaXF5G7r8jrSCIhr6SsggdmraBnajyXH5fhdRwRkRYrIjyMiSO78PHqXHbs13nVphJoD7xHgPvMTG066iA+OoInLh5JQXEZN760kLJy/e4icjhTv9zA+p0F3DNhIFERatEpIhJMk0emU17heGNRttdRWo1Af7LdCNwB7DezLWa2uXIJYrYWoU+HRB6aOJhvNuzm0Q++8zqOSMjK3V/EXz9cy8n90zipX5rXcUREWrw+HRIZlp7E9AUqvJtKoO0ELwlqihbugpHpfLthN499vJbMjGQVFSI1+MPs1RSXlXPPhIFeRxERaTUmZ6ZzzxvLWZ6Tx6DOai0YbAGd8XbOfVrbEuyALcWUcwfRv2Mit72yiJy9B7yOIxJSFm/Zy2vzs7jq+B70SIn3Oo6ISKtxzrDORIWHMW1+ltdRWoWAB1Ga2XAzu9HM7jOz+yuXYIZrSWIiw/n7xSMpKavghhcXUKrx3iIAVFQ4pry1nJSEaG44ubfXcUREWpW2cVGcNrADbyzKoaRMtUmwBVR4m9l1wBfAycCdwBDgF4B+StZBz9QEHp40lAWb9/KH91Z7HUckJMxclM3CzXu5c3w/EmN0/baISFOblNmF3QUlfLJaPb2DLdAz3r8ExjvnJgIH/LeTgdKgJWuhzhnWmcuO7c5Tc9bz/vJtXscR8dSX63Zy98xlDO/alkkj072OIyLSKo3tk0pKQrSGmzSBQAvvNOfcZ/77FWYW5px7FzgnSLlatF+fPYAhXZK4/bXFbNld6HUcEU98sjqXK5+ZS3pyLE9dlklYmHkdSUSkVYoID+OCkV34aFUuu/LV0zuYAi28s8wsw39/DXCemY3BN5W81FF0hG+8twOuf3EBxWXlXkcSaVKzl23j2ufm0TstgZevO5a0xBivI4mItGqTRqZTVuF4c3GO11FatEAL798DA/z37wdeAD4C7gtGqNaga7s4/vSjYSzJyuO3b6/0Oo5Ik3ljUTbXv7iAwV2SePHaY2gXH+V1JBGRVq9fx0SGdEnScJMgC7Sd4FT/0BL8t8lAsnPuiWCGa+lOH9SRa8f04NmvNjFriX7DlJbv1blbuOWVRYzOSOb5q48mKVYXU4qIhIrJmeksz9nHipx9XkdpsQ5beJtZnJn91szeNLMpZhYN4Jwrcc7lN03Elu2X4/szsltbfjV9Ket36J9UWq7nvtrIL6cvYUyfVJ654igSogOdv0tERJrCucM6ExluTF+gs97BcqQz3o/hu4ByFb4uJn8MeqJWJjI8jMf+ZySR4cbP/7OAolKN95aW5x+fruPeN5Zz2sAO/POyTGKjwr2OJCIi1STHR3FK/w68sShb840EyZEK7zOB051zv/TfnxD8SK1P57ax/PnC4azatp/73lrudRyRRuOc49EP1vC7d1cxYWgn/n7xSKIjVHSLiISqyZnp7Mwv4dPVO7yO0iIdqfCOd85tBXDObQGSgh+pdTqpXxrXn9SLl77dwj0zl7G3UA1jpHlzzvHwu6t49IPvmJyZzl8uGkFkeMCT5YqIiAdO7JdKSkKULrIMkiMNsowws5MAq+UxzrmPghWutbn11L4UFJfz3FcbeXNxDree2oeLj+muYkWancpp4J/7ahOXHtOd+84dpD7dIiLNQGR4GOcN78JzX21kT0EJyeo81aiOVNHlAk8D//Yvu6o9/ldQ07UyEeFhTDl3EO/ePJYhXZKY8tYKzvzLZ5rCVZqV8grHr2Ys4bmvNnHtmB7cf56KbhGR5mRyZjql5erpHQyHLbydcxnOuR6HWXo2VdDWpF/HRJ6/+ij+ddkoyiscVzwzlyuf+Za1uep6IqGttLyCW19ZxKvzsrjplD7cddYAzFR0i4g0JwM6tWFQ5zYabhIEGsMQosyMUwd24L1bxnL32QOYt2kP4x+dw31vLdf4bwlJxWXl3PDiAt5cnMOd4/tz22l9VXSLiDRTkzPTWZqdx+pt+72O0qKo8A5xURFhXDOmJ5/cPo4LR3fl2S83Mu6Pn/DcVxspU6sfCRFFpeX87/PzeW/5dqacM5CfjevldSQREWmAc4d1JiJMPb0bmwrvZqJ9QjQPTRzC2zeNYWCnNtz7xnLO/MtnzFmjdj/ivd++s5JPVu/gdxcM4Yrje3gdR0REGqh9QjQn909jxoJsnehrRCq8m5kBndrwn2uO5qlLMykpr+Cyp7/l6qlzWadZL8Ujn6zO5bmvNnH1CT34yVHdvI4jIiKNxNfTu5g53+kkX2NR4d0MmRmnD+rI+7eO5a6z+vPtht2c8cgc7n1jGSty9nkdT1qR3QUl3DFtCf06JHLHGf28jiMiIo1oXL802sVHMX1+ttdRWowj9fGWEBYdEc51Y3txwch0/vT+al76djPPfbWJ/h0TmTiiC+cN70LHpBivY0oL5ZzjrhlLySss5dkrjyImUjNSioi0JFERYZw3vDP/+XozewtLaBunnt4NpTPeLUBKQjS/u2Ao3951Kg+cN4jYqHB+9+4qjn34Qy751zdMn59FQXGZ1zGlhZk2P4vZy7fxi9P7MrBzG6/jiIhIEEzOTKekvIK31NO7UajwbkGS46O49NgMXv/58Xx8+zhuPLkPm3YX8IvXFjPqwQ+45eWFfLpmB+UVzuuo0sxt2V3IfW+t4Oge7bhmjNr5i4i0VIM6JzGgk3p6NxYNNWmheqTEc9tpfbn11D7M37SHGQuzmbU4h5mLckhNjOa8YZ25YGS6zlRKnZVXOG59ZREG/OnHwwjXrJQiIi3apJFdePDtlXy3fT99OiR6HadZ0xnvFs7MGJXRjt9OHMLcu0/liYtHMrxrW579aiNn/fUzxj86h398uo6lWXkUlmg4ihzZk5+uY96mPdx//iDSk+O8jiMiIkF2/oguRIQZ09TTu8F0xrsViY4I58whnThzSCf2FJQwa0kOMxZm87t3Vx3cJz05lj5pCfTpkEjvtISDS5uYSA+TS6hYlp3HI/9dw9lDO3H+8C5exxERkSaQkhDNuH5pvL4gmztO70dEuM7b1pcK71aqcjz4pcdmsGV3Icuy81ibm893/uWLdbsoKfu+YX7HNjH06eArwvukJfrupyaQHK8rnFuLotJybnllESkJ0Tx0/mBNBy8i0opMzuzCByu38/nanYzrl+Z1nGZLhbfQtV0cXdsdOmSgvMKxZXehvxDfz9rcfNbm5vPK3C0UlpQf3K99fBTpybGkJ8fRJTmW9ORYurT9/nFCtP6LtRQPv7uKtbn5vHD10WopJSLSypzcvwPJcZFMm5+lwrsBVBVJjcLDjIyUeDJS4jltYIeD6ysqHDl5B/guN5+12/NZvzOfrD0HWLl1H/9duf2Qs+QAbeMiDy3G2/qL8+RYUhKiiYsKJy4qQhfohbg5a3Yw9cuNXHl8Bif0SfE6joiINDFfT+8uvPjtZvIKS0mK0xDU+lDhLXUSFmakJ8eRnhzHSdV+462ocOwsKCZrzwGy9xzw3e4tJGvPAdbtKGDOmp0cKC2v8bgxkWEkREcQFxVBXFQ48dH+26gI4qMjiI/2FejxUeHERUcQExlGdEQ40RFhxET6bg/er2WbxqTVz56CEm5/bTF90hK4c3x/r+OIiIhHJmemM/XLjby1JIdLjunudZxmKSQKbzMbADwOZAI7gDucc6/XsJ8BDwBXAgnAQuB659zyJowrtQgLM9ISY0hLjGFkt+QfbHfOsbughOy9vqJ8T2EJhcXlFJSUUVhSTkFxmW8pKaewpIz9RWVs31dEQeU+xeWUlFfU8MqBiQgz2sVHHbxgtE9aAr3891MTojVmuQbOOX49cyl7Ckt45srRmp1SRKQVG9S5Df06JDJ9QZYK73ryvPA2swjgDeBJ4DTgROAtMxvhnFtTbfcfAVcBJwCbgAeB54GRTZdY6svMaJ8QTfuEaIamt63XMUrKKjhQUk5xWTlFpRUUl5VTXFZx6OPSCorLKigqPXRbUWk5ufuLWZubz4wF2eRXmc0zKTbyYDHe21+Q90lLoHNSLGGteBjMjAXZvLN0G3eO78+gzklexxEREQ+ZGZMz03nonZWszc2nd1qC15GaHc8Lb6A/0Bl4xDnngI/M7AvgUuCeavv2AD53zq0HMLMXgFubMqx4KyoijKiIMKBhY8ucc2zfV3zIhaPf5ebz3xXbeXnuloP7xUaG0ystnj5piQzs1IZBXdowqHMSSbEtf2zblt2F/ObN5RyV0Y7rxmp2ShERgfNGdObh2auYviBLww/rIRQK75pOJxowuIb1LwMXmllfYANwOTC71gObXQdcB9CtW7eGJ5UWw8zomBRDx6QYxvRJPWTb7oKSg8X4Wn9Xl6/W7eL1hdkH9+nWLo4hXZIY1KUNgzsnMbhLEu1aUGvF8grHL15dDGh2Smk6+swWCX1piTGM65vK6wuyuf30fvr5UEehUHivAnKBO8zsEeAkfMNNPq5h363AZ8BqoBzYApxc24Gdc08BTwGMGjXKNW5saanaxUdxVI92HNWj3SHrd+UXszxnH0uz81iek8fS7DzeXrr14PbOSTEM6pLE4M5JDEn3FeRpbWKaOn6jeGrOer7duJs//WjYD1pNigSLPrNFmodJmel8uGoBX6zdydi+qUd+ghzkeeHtnCs1s/OBvwF3AvOAV4HiGnb/DTAa6ApsAy7BNzRlkHOusIkiSyvVPiGasX1TD/mQySssZfnWPJZn72OZvxj/YOV2nL9kSE2MZkzvFP7fWQNITYz2KHndLMvO48//Xc1ZQzpywUjNTikiIoc6ZUAaSbG+nt4qvOvG88IbwDm3BN9ZbgDM7Evg2Rp2HQa84pzL8j+eamaPAgPxFewiTSopLpLjeqVwXK/ve1vnF5excus+lmXnsTQrj1lLt/Lx6lwemjiEs4Z08jDtkRWVlnPrK4tIjoviofOHqNOLiIj8QHREOOcN78wrc7ewr6iUNjEt/7qnxhISjY3NbKiZxZhZnJndDnQCptaw61zgR2bWwczCzOxSfFfZrW3CuCKHlRAdweiMdlx5fA/+fOFw3r7xBNKT4/j5fxZw00sL2VtY4nXEGpWUVXDfW8v5LjefP/5oGMktaMy6iIg0rkkj0ykuq+DtJVuPvLMcFBKFN74OJlvxjfU+BTjNOVdsZt3MLN/MKq+y+T9gMbAI2Iuvo8kk59xeL0KLBKJPh0Rm/Pw4bjutL+8s3crpj8zho1XbvY51kHOOj1ZtZ/yjc3jp2y1cN7an/nQoIiKHNTQ9iT5pCUybn3XkneWgkCi8nXN3OOeSnXMJzrkznXNr/es3+9dt9j8ucs5d75zr5Jxr45wb6ZyrtauJSKiIDA/jplP6MPP640mOi+KqqfO4c9oS9heVepprbe5+rnhmLldNnQcGz1wxmrvOGuBpJhERCX2VPb3nb9rD+h35XsdpNkKi8BZpLQZ3SeLNG4/nZ+N68dr8LYx/9DO+XLuzyXPkFZZy31vLOePRz1iweQ93nz2A2TeP5aT+aU2eRUREmqeJI7oQZr7J1iQwKrxFmlh0RDh3ju/Paz89jqiIMP7nX98w5c3lHCgpD/prl5VX8PxXGxn3x4959suNXDi6K5/cPo5rxvT0T0wkIiISmLQ2MYztm8r0BVmUV6gDaCD0k1bEI5ndk3nnpjFccVwGU7/cyFl//Yz5m/YE7fW+WLuTs//6Ofe8sZx+HROZdeMYfjtxCO0TmkebQxERCT2TM9PZmlfEV+t2eR2lWVDhLeKh2Khwppw7iBevPZqSsgp+9OSXPPzuKorLGu/s96ZdBVz33Dwu/tc3FJSU8eQlI3np2mMY2LlNo72GiIi0TqcO6ECbmAimzd/idZRmIST6eIu0dsf1SmH2LWN46O2VPPnpOj5elcuffjyMwV2S6n3M/OIyHvtoLU9/voGIcOOOM/px9Qk9iIkMb8TkIiLSmsVEhnPOsM5MX5DF/qJSEtXT+7BUeIuEiMSYSB6eNJQzBnXkzulLOP/xL+iVmkB8dDjx0RHER0X4bv2PE6IjiIs69L7vNoJl2Xn8/r3V7MwvZtLIdH45vh8dmun09SIiEtomZ6bzn282887SrVw4utuRn9CKqfAWCTEn9U/j/VvH8thHa9myp5CC4nLyi8vYvq+IguJyCkrKKCguo7T88BeyjOjWln9dPorhXds2UXIREWmNhndtS6/UeKbNz1LhfQQqvEVCUNu4KO6eMPCw+xSXlVPoL8oLS3y3BcVlFJaUERcVwZg+KZryXUREgs7X07sr/zd7FRt3FpCREu91pJClwlukmYqOCCc6IlxTu4uIiOcmjujCH95bxYwFWdx2ej+v44QsdTURERERkQbpmBTDCX1Smb4gmwr19K6VCm8RERERabDJmelk7z3A1+vV07s2KrxFREREpMFOH9iBxJgIpi3I8jpKyFLhLSIiIiINFhMZzoShnXl36Tbyi8u8jhOSVHhLszJu3DjGjRvndYxWz4uvQyCvGWiuhuSv6blt27albdu2h91Hgs/MmmUnn+b6/6W55q7+/dpcNJd/78mZ6RwoLeedpVu9jhKSVHiLiIiISKMY2a0tPVPimT5fw01qosJbRERERBqFmTEpM51vNuxm865Cr+OEHBXeIiIiItJoJo7oghlM10WWP6DCW0REREQaTee2sZzQO4XpC7LU07saFd4iIiIi0qgmjUwna88Bvt242+soIUWFt4iIiIg0qjMGdSQhOoJpusjyECq8RURERKRRxUaFM2FoJ95ZupUC9fQ+SIW3iIiIiDS6yZnpFJaUM3vZNq+jhIwIrwOIiIiISMuT2T2ZjPZxvDx3M8O6JgGGGRgQZpX3fbcAYWGGwSHrfY+r3fc/H+OQ9WHVjld5nDD7/nnmv+8VFd4iIiIi0ujMjMmZ6fzx/TWc+uc5Xsc5RPVC/pBiv8r9l649hmFdG2+mU3OudbR5MbMdwKZ6PDUF2NnIcZpSc88Pzf89NPf80PzfQ3PP3905l+p1iKbUCj+zlbtpKXfTam25a/3MbjWFd32Z2Tzn3Civc9RXc88Pzf89NPf80PzfQ3PPL4Frrl9r5W5ayt20lPt7urhSRERERKQJqPAWEREREWkCKryP7CmvAzRQc88Pzf89NPf80PzfQ3PPL4Frrl9r5W5ayt20lNtPY7xFRERERJqAzniLiIiIiDQBFd4iIiIiIk1AhbeIiIiISBNoNYW3md1gZvPMrNjMplbbdo2ZrTWzfDObbWadq2ybYmal/m2VS88q24eb2XwzK/TfDg+l/P7tI81sjn/7djO7ucq2DDP72J9/lZmdGoz8DXkPZvZutX//EjNb2tTvoQH5o83sSf+//W4ze8vMulTZ3s7MXjezAjPbZGb/E4z8DXwPbc3sWTPL9S9Tqj23qb4G0Wb2b/+/034zW2hmZ1bZfor/9Qv9ebpXe+7TZrbPzLaZ2W3Vjl3rcyX0NeX3UWM53PdjKDvS92EoM7MXzGyr/3NgjZld43WmujCzPmZWZGYveJ0lEGb2iT9v5c/v1V5nqgszu8jMVvo/V9aZ2ZiGHrPVFN5ADvAg8HTVlWZ2IvBb4DygHbABeKnac19xziVUWdb7nxsFvAG8ACQDzwJv+NeHRH4zSwFmA/8A2gO9gferHOIlYKF/26+BaWYWrBny6vUenHNnVv33B74EXvPgPdT3/9DNwLHAUKAzsBf4W5XtjwMlQAfgYuAJMxsUhPxQ//fwCBAHZABHAZea2ZVVtjfV1yAC2AKcCCQB9wCv+gv/FGCGf107YB7wSpXnTgH6AN2Bk4Bfmtl4OPh9crjnSuhryu+jxlLj92MzUOv3oYeZAvU7IMM51wY4F3jQzDI9zlQXjwNzvQ5RRzdU+Rnez+swgTKz04D/A64EEoGxwPoGH9g516oWfB9yU6s8/iPweJXHnQEH9PI/ngK8UMuxTgey8XeH8a/bDIwPofy/BZ6v5Vh9gWIgscq6z4CfhtLXoNpzM4ByoIdX76EeX4MngN9X2X42sNp/Px5fsdC3yvbngYdD6WuAb8rc0VW23wV85uX/oyqvtQSYBFwHfFllfTxwAOjvf5wNnF5l+wPAy/77h32ultBevPo+asT8h3w/Nsel8vvQ6xx1zNwP2Ar82OssAea9CHiVw9QlobYAnwDXeJ2jntm/BK5u7OO2pjPetTH/UvUxwOAq687xDxFYbmY/q7J+ELDE+b9Cfkv865vKkfIfA+w2sy/9QwTeMrNu/m2DgPXOuf1Vnr+Yps0PgX0NKl2Gr+Db4H8cCu/hSPn/DRxvZp3NLA7f2bh3/dv6AuXOuTVVnh+qX4Pq2yu3efY1MLMO+P4Nl/tfb3HlNudcAbAOGGRmyfh+mVhc5elVM9b63GDml0YTKt9HrVK178OQZ2Z/N7NCYBW+wvsdjyMdkZm1Ae4HfuF1lnr4nZntNLMvzGyc12ECYWbhwCgg1T8EM8vMHjOz2IYeW4W37xvux2Y21P8Pei++M31x/u2vAgOAVOBa4F4z+4l/WwKQV+14efj+JNFUjpQ/Hbgc33CHbhw6hCAU8sOR30NVlwFTqzwOhfdwpPxr8P0lJBvYh+//0/3+baGQH478HmYDvzKzRDPrDVxVZZsn78HMIoH/AM8651YdIUdClcc1ZQyVr4PUj75+Hqnh+zDkOed+ju//xhh8Q8yKvU0UkAeAfzvntngdpI7uBHoCXfBNRvOWmfXyNlJAOgCRwGR8/0+GAyOAuxt64FZfeDvnPgR+A0wHNgEbgf1Aln/7CudcjnOu3Dn3JfD/27vbYKuqOo7j3x8XlQREUBlERcKYoSLRTF4oJaUODg2ooU5CaU6iYyKZ2jiiBTiiUy90FJ/GVMiHMBMRH8oySid8JBVTGqN85FkvDwIK+PTvxVpH9r3e6z3I5Zxz7/19Zvbcc/bae+3/PvvstddZe619ryYdCICNwG6Nstwtr18RLcVPul0+JyIWRMRmYCpwmKQe1ED8UNY+ACBpGNAHuKcwu+r7UEb8NwBdSP2fu5IK+lKLd9Xjh7L2YSLpu/Rf0riGWYW0iu+DpE6krgTvAxPKiGNj4X1TMdbEcbDPzcevCpo5D9uEfE2fT2qcOqul5atJ6aENR5HG2rQpEfF0RGyIiC0R8VvgcWBkteMqw6b8d3pErIiIeuBKz8ObDQAACW1JREFUWiH2Dl/xBoiI6yJiYET0JlU8OgMvNbc4W2+5LwIOlFS8BX8gFb7d1kL8/yLF/Mni+a9IcQ6QVGwVGkIVbheWeQxOBe6NiI2FeTWxDy3EP4TUf3NNRGwhDawcmgf0LQY6SxpYyK7mjkGOfVxE9ImIr5LKjmfyqhU9Bvl8u4XUIjEmIj4oxDGksFxX4ABgUUSsJd1SHlLIqhhjs+vuiH2wVlcz51FH8RnnYVvTmXSu17LhpPFNb0paCVwAjJH0XDWD+pyKdaiala8ZS2lYf2oVHabiLamzpC5AHVAnqUtpnqTBSvqRboVcnT90JB0rqWdOH0pq+Zubs32UNNBvYn68UukX/99qJX5gBnC80mMPdyKNPp8fEetyf8iFwOScz/GkHw6zWzv+7dwHcveHE2nYzYRK7sN2xL8AOEVSj3wMfgIsj4j63Jf4XuBSSV0lHU56ssjtrR3/9uyDpAMk7SGpTumxYWeQBoRV9BhkN5C664yKiE2F+XOAwZLG5H38JWkMRun2923AJfl8HkTqOjazzHWthlX6PGotzZ2P1Y6rTM2dhzVLUm+lx8N1y2XZCOBkdsA1u5XdRPpxcFCebgQeAkZUM6iWKD2GdkThOjOO9GSQP1c7tjLNAM7J35uewLnAg9uda7VHjVZqIo0CjkbTFGB3Uqvwu8BK0qOG6grrzQJWk25lvgxMbJTvwcCzpNsSzwEH11L8ed2zSP2L1wIPAPsV0vqTfkBsAv4DHFVrxyCvezKpC4SayLci+7Ad36E9SH0g3yI9SnA+MLSQ3gu4L6//JjC21o4BcBLp0WfvkSrZI6p0DPbPMW/O52RpGpfTjyKdp5tyPP0L6+5CemzbemAVcF6jvJtd11PtT5U8j1ox5ibPx2rHVUbcn3ke1upEGqv1WC6H1wMvAuOrHdfn/N7U/FNN8ue9gNTlax3wFHB0tePahvh3Aq7Psa8ErgG6bG++ypmbmZmZmdkO1GG6mpiZmZmZVZMr3mZmZmZmFeCKt5mZmZlZBbjibWZmZmZWAa54m5mZmZlVgCveZmZmZmYV4Iq3WY2RdIekKdWOw8ysrZA0XNLSKmx3kqSbK71da7tc8bZ2RdKdkm5tNO8ISasl7d3K27pDUkga2Wj+tXn+D1pze2ZmbZ2k1yVtkrRR0ipJMyR1q3ZcJZIelXR6o3nHSlooab2keknzJPUHiIjLI+L0pvIya4or3tbeTARGSjoaIP875t8A50fEitbaiKS6/HIxcGph/k7AGODV1tqWmVk7MyoiugFfBw4FLikmKqmJ+omkLwG3AecDPYAvkv6b4cfVjMvarpr4Ypu1lohYDZwD3CSpKzAZeCUiZkrqlG8LvpJbLe6S1BMgp90jaaWkdbnV48ulfHPr9nWSHpb0LvDNnHQfMFxSj/z+u8A/gbcL6w6U9Pfc6l4v6fbC8kg6JLembJA0i/SvzSmkj5b0Qo5rvqTBrf7BmZlVWEQsA/4EDM5l7jRJjwPvAQMk9ZV0v6Q1kv4naXxpXUlfkDRT0lpJ/yZV4CmkR640l97PlHRZ4X2xFfsVScdImkYq26/NLfLXAgcBr0XEvEg2RMTsiHgz5zNF0h35dWm90vRhqdtg3pfZkt6W9JqkiTvmU7Va54q3tTsR8QfgWWAWcAZwZk46j1Qx/hawL/AucE1h1QeBgUAf4CXg9kZZjwWmAt2BJ/O8TcBDwEn5/Smk1pEiAZcBewNfAQYAvwCQtAswF7gV6JVfH/fJitKhpBb704E98nJzJe1c3qdhZlabJO0HjASez7N+SCqzuwNvkMrwpUBf4ATgcklH5mUnAwfkaQSFO49lbHcoqZz+ObA76ZrwekRcDPwDmBAR3SJiAvAcMEjSVZK+/VndYiKitF43YBiwllRedwIeAF4A9gGOBM6VNKLcmK39cMXb2quzge8Al5ZaJkgV8EkRsSwiNgNTgJMkdYqIjyNiZm7NKKUdklvNS+ZExJN52S2F+bcBp0jqBRwG3F8MJCIW59aS9yPiLeAq4IicfDgQwPSI+CAi7mLrRQjSRej6iFgQER9FRKn/eoPWHTOzNuQ+SeuA+cBjwOV5/syIWBQRH5IaQIYBF0bE5ohYCNxMqpxDauyYFhFrImIJDRtRWvJj4NaIeCSX58si4uWmFoyIV4HhpArz3UB9bj1vtgIuaS/S3dBzIuJ5Unm9V0Rcmq8Dr5IaVL6/DTFbO9G52gGY7QgRsUpSPbCoMLsf8ICkYt+8AHpLehu4gtSqsidb++/tSWoZB1jSzOYeI7WgTwLmRsQWSZ8kSupDuigcTmrJ6cTWrih9gaUREYX83ii83h8YJ+lnhXk7ky4CZmZt0XER8dfijFxmFsvYvsCaiNhQmPcG8I1C+pJGaeXaD/hjuQtHxFPku5r5LuTvgYuBixovm8f53AP8LjekQCrH++YfGyV1pNZ162Dc4m0dyVLg6IjYvTB1iYiVpC4iI0mt5D2AUt9AFdYPmpArzXeSurI07mYC8CtgC/C1iNgN+FEh3xWkSntRv8LrJcDURjHvGhF3l7fLZmZtRrGMXQ70ktS9MK8fsCy/XkGqQBfTit4Ddi2871N4vYTURaWlGD6dGLEAuBdobqzNdGADDQeMLiH1Ey+W490jYmTTWVh75oq3dSQ3kvoI9gOQ1FvS6JzWnVQ5Xk0qrKdtY95XkSr1jzeR1p3Uav5O7tN4QSFtPtBJ0gRJnSWdSBrpX3ITcLakQ/NI/26SRjXqAmNm1q7k7iNPAFdI6iLpQFIXkTvzIncDF0nqKWlf0qD6ooXAWEl1ko5ha/c+gFuA0yQdmQfW7yNpUE5bRRqHA4CkYZLGS+qd3w8CRgNPNY5Z0pl5O2Mjonhn9RlgvaQL86DQOkmDc+u5dTCueFtHciXwMDBP0gZSoV4q+GaQWliWk7qnPLEtGUfE6oiY10zyZGAo8A6p//fswnpbgOOB8aSBON8j9Q0spT8NnAXckNMXA34+uJl1BCcD/Unl8hxgckQ8ktOmkrqXvAb8hU8Phv8pMApYB4yjYbn6DHAaqcHkHVJ3wf1z8tXACflpKdfk9UcDL0raSLqGzAF+3Uy8A4DlhSebTIqIj3IsB+V460n91Xs0kYe1c2rYtdTMzMzMzHYEt3ibmZmZmVWAK95mZmZmZhXgireZmZmZWQW44m1mZmZmVgGueJuZmZmZVYAr3mZmZmZmFeCKt5mZmZlZBbjibWZmZmZWAf8HquM7L8tZfpUAAAAASUVORK5CYII=\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"from sklearn.inspection import plot_partial_dependence\n",
"\n",
"fig,ax = plt.subplots(figsize=(12, 4))\n",
"plot_partial_dependence(m, valid_xs_final, ['YearMade','ProductSize'],\n",
" grid_resolution=20, ax=ax);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Looking first of all at the YearMade plot, and specifically at the section covering after 1990 (since as we noted this is where we have most of the data), we can see a nearly linear relationship between year and price. Remember that our dependent variable is after taking the logarithm, so this means that in practice there is a exponential increase in price. This is what we would expect: depreciation is generally recognised as being a multiplicative factor over time. So, for a given sale date, varying year made ought to show an exponential relationship with sale price.\n",
"\n",
"The `ProductSize` partial plot is a bit concerning. It shows that the final group, which we saw before is for missing values, has the lowest price. To use this insight in practice, we would want to find out *why* it's missing so often, and what that *means*. Missing values can sometimes be useful predictors--it entirely depends on what causes them to be missing. Sometimes, however, it can show *data leakage*."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Data leakage"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the paper [Leakage in Data Mining: Formulation, Detection, and Avoidance](https://dl.acm.org/doi/10.1145/2020408.2020496) the authors introduce leakage as \n",
"\n",
"> : \"the introduction of information about the target of a data mining problem, which should not be legitimately available to mine from. A trivial example of leakage would be a model that uses the target itself as an input, thus concluding for example that 'it rains on rainy days'. In practice, the introduction of this illegitimate information is unintentional, and facilitated by the data collection, aggregation and preparation process.\"\n",
"\n",
"They give as an example\n",
"\n",
"> : \"a real-life business intelligence project at IBM where potential customers for certain products were identified, among other things, based on keywords found on their websites. This turned out to be leakage since the website content used for training had been sampled at the point in time where the potential customer has already become a customer, and where the website contained traces of the IBM products purchased, such as the word 'Websphere' (e.g. in a press release about the purchase or a specific product feature the client uses).\"\n",
"\n",
"Data leakage is subtle and can take many forms. In particular, missing values often represent data leakage.\n",
"\n",
"For instance, Jeremy competed in a Kaggle competition designed to predict which researchers would end up receiving research grants. The information was provided by a university, and included thousands of examples of research projects, along with information about the researchers involved, along with whether or not the grant was eventually accepted. The University hopes that they would be able to use models developed in this competition to help him rank which grant applications were most likely to succeed, so that they could prioritise their processing .\n",
"\n",
"Jeremy used a random forest to model the data, and then used feature importance to find out which features were most predictive. He noticed three surprising things:\n",
"\n",
"- The model was able to correctly predict who would receive grants over 95% of the time\n",
"- Apparently meaningless identifier columns were the most important predictors\n",
"- The columns day of week and day of year were also highly predictive; for instance, the vast majority of grant applications dated on a Sunday were accepted, and many grant applications were dated on January 1 and were also accepted.\n",
"\n",
"For the identifier columns, a partial dependence plots showed that when the information was missing the grant was almost always rejected. It turned out that much of this information, in practice at the University, was only filled in *after* a grant application was accepted. Often, for applications that were not accepted, it was just left blank. Therefore, this information was not something that was actually available at the time that the application was received, and would therefor not be available for a predictive model — it was data leakage.\n",
"\n",
"In the same way, the final processing of successful applications was often done automatically as a batch at the end of the week, or the end of the year. It was this final processing date which ended up in the data, so again, this information, while predictive, was not actually available at the time that the application was received.\n",
"\n",
"This example shows the most practical and simple approaches to identifying data leakage, which are to build a model, and then:\n",
"\n",
"- Check whether the accuracy of the model is *too good to be true*\n",
"- Look for important predictors which don't make sense in practice\n",
"- Look for partial dependence plot results which don't make sense in practice.\n",
"\n",
"Thinking back to our bear detector, this mirrors the advice that we also provided there — it is often a good idea to build a model first, and then do your data cleaning, rather than vice versa. The model can help you identify potentially problematic data issues."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Tree interpreter"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#hide\n",
"import warnings\n",
"warnings.simplefilter('ignore', FutureWarning)\n",
"\n",
"from treeinterpreter import treeinterpreter\n",
"from waterfall_chart import plot as waterfall"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"At the start of this section, we said that we wanted to be able to answer five questions:\n",
"\n",
"- How confident are we in our projections using a particular row of data?\n",
"- For predicting with a particular row of data, what were the most important factors, and how did they influence that prediction?\n",
"- Which columns are the strongest predictors?\n",
"- Which columns are effectively redundant with each other, for purposes of prediction?\n",
"- How do predictions vary, as we vary these columns?\n",
"\n",
"We've handled three of these already--so just one to go, which is: \"For predicting with a particular row of data, what were the most important factors, and how did they influence that prediction?\" To answer this question, we need to use the `treeinterpreter` library. We'll also use the `waterfallcharts` library to draw the chart of the results.\n",
"\n",
" !pip install treeinterpreter\n",
" !pip install waterfallcharts"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have already seen how to compute feature importances across the entire random forest. The basic idea was to look at the contribution of each variable towards improving the model, at each branch of every tree, and then to add up all of these contributions per variable.\n",
"\n",
"We can do exactly the same thing, but for just a single row of data. For instance, let's say we are looking at some particular item at auction. Our model might predict that this item will be very expensive, and we want to know why. So we take that one row of data, and put it through the first decision tree, looking to see what split is used at each point throughout the tree. For each split, we see what the increase or decrease in the addiction is, compared to the parent node of the tree. We do this for every tree, and add up the total change in importance by split variable.\n",
"\n",
"For instance, let's pick the first few rows of our validation set:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"row = valid_xs_final.iloc[:5]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can pass these to `treeinterpreter`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"prediction,bias,contributions = treeinterpreter.predict(m, row.values)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`prediction` is simply the prediction that the random forest makes. `bias` is the prediction based on simply taking the mean of the dependent variable (i.e. the *model* that is the root of every tree). `contributions` is the most interesting bit--it tells us the total change in predicition due to each of the independent variables. Therefore, the sum of `contributions` plus `bias` must equal the `prediction`, for each row. Let's look just at the first row:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(array([9.98234598]), 10.104309759725059, -0.12196378442186026)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prediction[0], bias[0], contributions[0].sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The clearest way to display the contributions is with a *waterfall plot*. This shows how each positive and negative contribution from all the independent variables sum up to create the final prediction, which is the right-hand column labeled \"net\" here:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAaQAAAEUCAYAAABkhkJAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOydd5wURfbAv293WUCWzCqCR1Aw4QF6iyiIoBjwTk/MIgYExQhGDCeI8VQ4PMV4Kkg0gxhOvDOtghlU0L3zwMhPRUXAFZTM+/3xatjeYZbdHXZnBnjfz6c/011VXf26p7te1atXVaKqOI7jOE66yUq3AI7jOI4DrpAcx3GcDMEVkuM4jpMRuEJyHMdxMgJXSI7jOE5G4ArJcRzHyQhcITmO4zgZQVIKSUQaicjTIvKriHwtIqeUke4gEXlNRIpF5KsE8a1C/G8i8qmIHBIXf4mIfB/OHysiNZOR13Ecx8l8km0h3QOsBnYA+gL3iUi7BOl+BcYCQ8rI51HgQ6AxcA3wlIjkA4jI4cBVQE+gFbAzcH2S8jqO4zgZjlR2pgYRqQMsBfZS1XkhbCLwrapeVcY5hwAPqWqrSNiuwMdAE1VdFsJmAJNV9X4ReQT4SlX/EuJ6hrimlbxHx3EcZwsgJ4lzdgXWxZRRYA7QvZL5tAO+iCmjSD7tIvHPxMXtICKNVXVxWZk2adJEW7VqVUlRHMdxnFQwe/bsn1Q1P1FcMgopDyiOCysG6lZRPs3LiI/t1wVKKSQRGQgMBGjRogWzZs2qpCiO4zhOKhCRr8uKS6YPaTlQLy6sHrAsQdrNySc+Pra/0XVU9QFVLVDVgvz8hIrXcRzHyXCSUUjzgBwRaRsJ6wAUVTKfImBnEYm2rKL5FIXjaNwPmzLXOY7jOFsulVZIqvorMBW4QUTqiEhX4GhgYnxaEckSkVpADTuUWiKSG/KZB3wEDA/hxwDtgSnh9AnAABHZU0QaAkOBcZW+Q8dxHGeLIFm37/OB2sCPmOv2eapaJCLdRGR5JN2BwArgBaBF2P93JP5koADz2rsVOF5VFwGo6ovACOA14OuwDU9SXsdxHCfDqbTbd6ZTUFCg7tTgOI6TmYjIbFUtSBTnUwc5juM4GYErJMdxHCcjcIXkOI7jZASukBzHcZyMwBWS4ziOkxG4QnIcx3EyAldIjuM4TkbgCslxHMfJCFwhOY7jOBmBKyTHcRwnI3CF5DiO42QErpAcx3GcjMAVkuM4jpMRuEJyHMdxMgJXSI7jOE5G4ArJcRzHyQhcITmO4zgZQVIKSUQaicjTIvKriHwtIqeUkU5E5DYRWRy2ESIikfiOIjJbRH4Lvx0req7jOI6zdZFsC+keYDWwA9AXuE9E2iVINxDoDXQA2gNHAucAiEgu8AwwCWgIjAeeCeGbPNdxHMfZ+qi0QhKROsBxwDBVXa6qM4FngdMSJD8DGKWq36jqt8AooF+I6wHkAHeo6ipVHQ0IcHAFznUcx3G2MpJpIe0KrFPVeZGwOUCiFlK7EJcoXTtgrqpqJH5uXHxZ5zqO4zhbGTlJnJMHFMeFFQN1K5C2GMgLfUHl5VPmuXFKDBEZiJn4yM/PZ+DAgbRr145u3bpx//33U79+fYYMGcLQoUMBGDVqFFdffTWrV6/mkksuYdq0aXz55ZeceOKJLFy4kNeH3MT+9XagZc08Hlv0OS1q5nFCfmtGffMx2ZLFyNb7cukX7wAwvOU+3PVdEUvWrGJA0914b9kiPv51Cb0a7gTAi0u/oePQQXTu3JkHH3yQRo0acdFFFzF8+HAA7rjjDi677DLWrVvHkCFDePzxx1mwYAF9+vTh66+/5q233qJbt27suOOOPPHEE7Ru3ZrevXvz97//ndzcXG655RYuu+wyAG666SZGjhxJcXEx5557LjNmzKCoqIijjjqKVatW8e9//5u9996bvffem7Fjx5Kfn895553HDTfcAMBdd93FoEGDALjqqquYOHEi3377Laeeeirz58/n3XffpUePHjRu3JgpU6bQpk0b/vjHPzJ69Ghq167N9ddfzxVXXAHALbfcws0338zy5cu54IILeOWVV/j000/p3bs3y5Yt45VXXqGgoIB27doxfvx4mjZtyoABA7j55ps3kuWaa65hzJgxfP/995xxxhkUFRUxa9YsevbsSd26dZk2bRq77747PXv25J577iEvL49rrrmGq6++GoARI0YwfPhwVqxYweDBg3nhhRf47LPPOO6441i8eDGFhYV07tyZtm3bMmnSJJo3b85pp53GrbfeupEs1157Lffddx+LFi2if//+fPjhh3z44Yccdthh1KxZk+eee26z3r0ZM2bQpUsXWrZsyaOPPkqLFi046aSTGDlyJNnZ2YwaNYqLL74YgOuvv54777yTJUuWcPbZZ/Puu+8yd+5cjjjiCACmT59O+/bt/d3bht+9pw4ZwpL77d1rctEoFt97NbpmNQ36XMLy16ex9rsvyTvkRNb+tJCVH82gVvsu5DRtyfJ/P0rODi3IO/Qkfp40ErKyaXLxKH66/WJOyt/8d68sJK5sLxcR2Rt4U1W3i4RdBvRQ1aPi0hYDh6rqe+H4D0ChqtYVkUtC3B8j6Z8L8aM2de6m5CsoKNBZs2ZV6p7iWdttwGadH0/OjDFVmt+2yLiPxvHA7AcQEe464i722XGfDXFv/d9bnPP8OcxfPJ/PBn/GTvWsMnDikyfyzS/fsE7XcV7BefTr2I//LPoP5//zfABWrVvFvMXzWHzF4rTck+NUN23fr/o853favPNFZLaqFiSKS6aFNA/IEZG2qjo/hHUAihKkLQpx7yVIVwRcFtfiaY85TJR3rrMNsXTFUka/O5p3znqHb3/5ltOePo2Z/WduiG+X3463B7zNkY8cWeq8mw++mbaN27Jy7Ur2uncvTt7rZPbM35PCfoUAPFH0BK9++Woqb6VK2ZSSXrl2JQOeHcCC4gW0qN+CMX8eQ62cWgz59xDe++49VqxZQfeW3Rl52EgAbplxC1M/nYognLzXyVy6/6Xpui1nG6bSfUiq+iswFbhBROqISFfgaGBiguQTgEtFpLmINAMuA8aFuEJgHTBYRGqKyIUh/NUKnOtsQ7z77bt0a9GN3OxcWjdszfLVy1m1dtWG+Pq16pOXm7fReW0btwWgRlYNsiQLofSogUlzJ3Fq+1OrV/hqIqakC/sVMumYSQyePrhU/LiPxrF7492ZceYMdmu8G+M+GgfAzT1v5vV+r/Pe2e/x3nfvUfRjEctWLWPsR2N5Z8A7vDXgLe6fdT+/rv41DXflbOsk6/Z9PlAb+BF4FDhPVYtEpJuILI+k+wfwHPAx8AnwzxCGqq7G3LpPB34G+gO9Q/gmz3W2LZasWELD2g03HNevVZ8lK5ZU+Py/zvgrffbqQ82cmhvCFv+2mE9/+pSuv+tapbKmivKUdOFXhRy5q7UYj9r1KN74+g0AcrNtVMWadWuoU6MOzeo2o3aN2jSr24wVa1ewYs0KateoTY3sGqm/KWebJxmTHaq6BFMm8eEzMGeE2LECV4QtUT4fAn8oI26T5zrbDo1qN+LnlT9vOC5eWUyj2o0qdO6EORP4ZNEnPHrco6XCHy96nBP2PIEtdax1WUp6x7o7bhTfoFYDFq8o6Scb9MIgpv1vGoftfBj1a9UnS7L4Y5s/stvdu7Fe1zO029ANistxUolPHeRkPJ2bd2bmgpmsWbeGBcULyMvNK9XaKYtnPn2GRz5+hInHTCRLSr/qkz+evMWa66B8JR2NL15VOu6uP97Flxd9yU8rfuLFz15k3uJ5TPnvFL4Y/AVfDP6C8XPG8+0v36buZqqIcR+No8uYLnQd25UPFn5QKm7l2pX0ndqXbg93o+/UvqxcuxKA6fOn0+nBThvC165fu+GcJSuW0PC2hkyaOyml97Et4wrJyXga1m7I+Z3Op/u47vSZ0oc7et3BR99/xMg3rUN+3uJ5HDLhEOb8MIc+U/pw3/v3AdB3al9++u0nDpt4GD3G9dhQyH6x9AtWrV3FHvl7pO2eNpfylHT3lt15Yf4LALww/wW6t+wOsKEgzsnKoU6NOmxXYztUlbo161Izpya1a9SmZk5Nlq9evvFFM5hk+9SGvTaMp054ihlnzqBGVg1e+vylDefcMuOWLdaku6WSlMnOcVJN/73703/v/qXCOja1qQ93bbwrL5/+8kbnLP9L4kJ154Y7M2vg5g0NSDdRJS0i3NnrTj76/iNe+vwlhnQdQr+O/ej/bH+6PdyNnertxMNHPwyYkl7822LWrF/DAb87gB6tegCwb7N92e+h/VCUg1odxG5Ndkvj3VWesvrUYkq68KtCrux6JWB9aiPfGsm5BefSbvt2/LzyZ1poC4pXFZNfJx+ABcULWLh8IQXNEnonO9WEKyTH2ULZlJKuXaP2Rv1mAFNOnJIwr1sOuaXqBUwhyfapnd7+dHpN7kW9mvXosEOHDQro+sLruabbNTxe9HiK72Tbxk12juNs8STbp3bO8+fw3lnv8b8L/0ej2o14suhJPv7hY0Rkizbpbql4C8lxnC2ezs07M/TVoaxZt4aFyxeW2afWsWnHUn1q2VnZG1pO+dvls2TFEmYvnM3/Fv+PXpN68dmSz6iTW4ddG+/Kvs33Tcu9bUu4QnIcZ4sn2T61mw66iYPHH0ytnFo0qNWAKw+4krzcPPp17AfAdYXX0aZRG1dGKaLSc9llOj6X3dbLnov2r/I8/5P/dlLn+TvibAlsC3PZOY6TIVS1knYF7aQTd2pwHMdxMgJXSI7jOE5G4ArJcRzHyQi8D8lxnK2GTHF8qeo+Ndg2+tW8heQ4juNkBK6QHMdxnIzAFZLjOI6TEbhCchzHcTICV0iO4zhORpCUQhKRS0TkexEpFpGxIpJw+U4RyRWRp0TkKxFREekRFy8icpuILA7bCImsKS0iHUVktoj8Fn47JiOv4ziOk/lU2u1bRA4HrgIOBr4DngauD2GJmAncATyZIG4g0BvoACjwEvAFcL+I5ALPhHPvBc4BnhGRtqq6urJyO05VkikuuMlO9eM4mUgyLaQzgDGqWqSqS4EbgX6JEqrqalW9Q1VnAuvKyGuUqn6jqt8CoyJ59cAU5h2qukpVRwOCKULHcRxnKyMZhdQOmBM5ngPsICKNqyivdpG4uVp6OvK5kfgNiMhAEZklIrMWLVqUhBiO4zhOuklGIeUBxZHj2H7dKsorL/QjxcfF4je6jqo+oKoFqlqQn5+fhBiO4zhOuilXIYlIXxFZHrbpwHKgXiRJbH9ZEtdPlNfy0CqKj4vFJ3Mdx3EcJ8MpVyGp6mRVzQvbEUAR5oQQowPwg6ouTuL6ifIqisS1j3rdAe0j8Y6THj74ALp2hS5dYNy4xGn++ldLc/DB8NVXpeOGD4c2bUqOTzsNevSAggL4+9+rSWjHyXySMdlNAAaIyJ4i0hAYCowrK7GI1BSRWuEwV0RqRZTMBOBSEWkuIs2AyyJ5FWKOEINDHheG8FeTkNlxqo5Bg2DSJCgshNGjYenS0vGffgqvvgpvvgnXXQdXRRxQf/gB5s0rnX7MGMvrnXfg3nthmRsBnG2TSiskVX0RGAG8BnwdtuGxeBEpEpG+kVP+B6wAmgP/CvstQ9w/gOeAj4FPgH+GMIJrd2/gdOBnoD/Q212+nbSyahX8+iu0bg25udCtG7wft050YSH86U+2f+CBMCfit3PjjXD11aXT5+ba78qV0KIFbLddtYnvOJlMUstPqOrtwO1lxLWLO261iXwUuCJsieI/BP6QjIyOUy0sXgwNGpQcN2hgYVGWLIFmzUqO14URD/Pnw/Ll0L79xvmecAK8/jqcdx5kZ1e93E5KyZRxavM7pVuCyuHrITlORbj7bnjqKev7KY44fxYXQ6NGpdM2agQ//1xyHFMw110HN9yQOP8nn4TffrMW1UknwZ57Vqn4jrMl4HPZOZtk3Efj6DKmC13HduWDhR+Uilu5diV9p/al28Pd6Du1LyvXriwV331cd8569qwNx7O/m81hEw/joPEHccVLCRvFmcuFF5op7qGHzKS2YAGsWQMzZ8K++5ZO2707TJ9u+2+9BR2C384XX8AFF0CvXrBwIQweDKqwOliha9WC2rVtc5xtEG8hOWWydMVSRr87mnfOeodvf/mW054+jZn9Z26IH/fROHZvvDuTj53MDa/fwLiPxnFuwbkAPD/veerVLPHaX71uNVe9chVTT5xK3ZrJDFnLIO68E/r0MWVy/vnQsKGF9+0LkyfDHnvAAQeYl11urjktALwdmeanTRtziFizBg47zMJWrbLWUevWqb0fx8kQvIXklMm7375LtxbdyM3OpXXD1ixfvZxVa1dtiC/8qpAjdz0SgKN2PYo3vn4DgPW6nnvev4cLOl2wIe3b//c2ebl5nDL1FA4efzAzvp6R2pupSgoKzIPurbegf/+S8MmTS/aHDbM0r70GO++8cR6ffWa/NWpYy6uw0BTWxRdXp+SOk9G4QnLKZMmKJTSs3XDDcf1a9VmyYknC+Aa1GrB4hXXuj/9oPMfufiy1cmptSPvdsu+Y8/0cJh87mYnHTOTs586m9KxQjrOVkOw4tXHjrHXco4dt335r4f36wd57W9gJJ1StLMuWwf77m3POpEkl4SNGQOfOdu6gQWYNWLECDj3UWv/77Vdilq5CXCE5ZdKodiN+XlnSOV+8sphGtRsljC9eZXEr165k8seTOXPvMzfKq8vvulCvZj2a12tOk+2asOg3n3fQ2QrZnHFqAwaUtJibNy8Jv+suC3sy0aIJmyFL7drw9NMbt8yPOQbefddk/OEHkzcnBx580PpNn3++WlrzrpCcMuncvDMzF8xkzbo1LCheQF5uHjVzSpa+6t6yOy/MfwGAF+a/QPeW3fly6Zf8vPJnjnzkSK546Qr+9fm/eOiDh+i8U2fmLZ7H2vVrWbZqGT/++iONayczH6/jZDCbO05twgRrgQwbBuvXl4Rfeqnl9fjjVStLTg40bbrxuW3bluzn5lq6GjWgVSsLq1ULsqpefbhTg1MmDWs35PxO59N9XHdEhDt73clH33/ES5+/xJCuQ+jXsR/9n+1Pt4e7sVO9nXj46IeplVOLWQNnAdbHNGnuJM7axzztBu07iB7jerBm/RpuO+Q2srN8vM3WQqaMu0n7+lCbM07t6KNtGimAM8+0PsnTToO//Q2aNLHzevaETp0S90smI0t5FBaaR+iBB5YOv+giuKLqPWVdITmbpP/e/em/d/9SYR2b2sK9tWvU5tHjHi3z3B6tetCjVY8Nx6d1OI3TOpxWLXI6TlqpinFqDUv6azn5ZPjXv0whNWlSct6hh1qLalMKqTKybIq5c21Wkeeeg+iUojfeaLKeeWbZ5yaJm+wcx3E2l6oYpxZVUq++CrvtVjp89Wrr09l116qTpSw++8w8SB97rEQhgim7+fNh5MiK5VNJvIXkOI5TlSQ7Tm3kSHj5Zeuv2W03uOUWCz/pJJtyas0aOPVUaLfRGqXJywJw1FFQVGTKa+ZMuP9+c1j4+Wc44wxLM2SImQovusi88g46yMJfeaVKp7qSrc31tqCgQGfNmrVZeaztNqCKpDEyxb6+pbPnov2rPM+09zk4zjaGiMxW1YJEcW6ycxxn80h23M0PP9g0SgcdZDXxVWHQ9fTpVhvv1s1q8mvXpuIunAzATXZOuWRKy8RbMxlKbKxL8+Y2YPLoo0t30EfH3bzxho27eewxM0n162cd+LfdZi7PZ59tLs9TpkDLlhb/0ktwxBHpujsnhXgLKUPZ1KSmI94cQeeHOtN1bFcGvTBow4wHHyz8gK5ju9JlTBfGfTRuQ/rDJx1O/sh8bnrjplTegrMtsDnjbubNs2mYwDrbX3vN9tu1s/4LVfMMy89Pya046ccVUgYSm9S0sF8hk46ZxODpg0vFH7P7Mbx71ru82f9Nfvj1B1790hbRHTR9EJOOmURhv0JGvzuapStsVPaYP49h5KHV4xXjbONUdNxNtMUUG3fz+9/Diy/a/gsvWDqA0083U97uu9tgzIKE3Q3OVogrpAykvElN2zYuGUWdm51LTlYOq9au4tfVv9K6YWtys3Pp1qIb739nNdWd6u2U8ntwtnLuvtvmVrv22uTH3fzlLzY9zcEHWz9RbLDoOefAe+/B//5n51Z2uhxniyUphSQil4jI9yJSLCJjRaRmGen2E5GXRGSJiCwSkSdFZMdIvIjIbSKyOGwjREpGYIlIRxGZLSK/hd+Oyci7pVHepKYxCr8qZOHyhRzY8kAWr1hMg1olNdUGtRqw+LdKjsp2nIpSFeNu6teHiROtf6l2bTj+eAvPzi5pUeXnl7ScnK2eSiskETkcuAroCbQCdgauLyN5Q+CBkK4lsAx4OBI/EOgNdADaA0cC54Tr5ALPAJNCPuOBZ0L4Vk15k5oCzP1hLle/cjWPH/84IkKj2o0oXlVSU41Nduo41U5srEv37huPdYHS426uucY87sAU0UEH2XQ4eXnwxz9a+E03Wasp1t8Uy8fZ6knGy+4MYIyqFgGIyI3AZExJlUJVS81PLiJ3A6/H5TVKVb8J8aOAs4H7gR5BvjvUeu1Hi8jlwMHAi0nIvcXQuXlnhr46lDXr1rBw+cKNJjX9bMln9H+mP1NOnEKT7WwUda2cWmxXYzsWFC9gx7wdmblgJsO7D0/XLTjbErH1oeKJXx9q2LDS8QcfbFs8J5xQ+WUWnK2CZBRSO6zlEmMOsIOINFbV8mxEBwJFcXlFprplTgiLxc3V0iN354bwUgpJRAZirS1atGhRwdvIXMqb1PTiFy/m55U/c8Y0G0U9pMsQ/rTrn7iz1530mdIHVeX8TudvMPud/ezZvPXNW6xau4pZ381i2snT0nl7juM4Can0TA0i8jlwgaq+GI5rAKuB1qr61SbOaw8UAker6owQtg5op6qfhuO2wDzMlDg0xJ0cyWMyMF9VryvrOj5TQ9WTKeOQHMfZ8tmsmRpEpK+ILA/bdGA5UC+SJLa/bBN5tAGmAxfFlFEgUV7LQ6soPi4WX+Z1HMdxnC2XchWSqk5W1bywHYGZ3DpEknQAfijLXCciLYGXgRtVdWJcdKK8iiJx7aNed5jjQ9Tk5ziO42wlJOP2PQEYICJ7ikhDzLQ2LlFCEWkOvArco6r3l5HXpSLSXESaAZdF8ioE1gGDRaSmiFwYwl9NQmbHcRwnw6m0Qgp9RyOA14Cvw7bBnUtEikQk5qd5FuYWPjxi9lseye4fwHPAx8AnwD9DGKq6GnMJPx34GegP9A7hjuM4zlaGLz+RgExxanBnAsdxtjZ8+QnHcRwn43GF5DiO42QErpAcx3GcjMAVkuM4jpMRuEJyHMdxMgJXSI7jOE5G4ArJcRzHyQhcITmO4zgZgSskx3EcJyNwheQ4juNkBK6QHMdxnIzAFZLjOI6TEbhCchzHcTICV0iO4zhORuAKyXEcx8kIXCE5juM4GYErJMdxHCcjSEohicglIvK9iBSLyFgRqVlGuj1FZJaILA3byyKyZyReROQ2EVkcthEiIpH4jiIyW0R+C78dk5HXcRzHyXwqrZBE5HDgKqAn0ArYGbi+jOTfAccDjYAmwLPAY5H4gUBvoAPQHjgSOCdcJxd4BpgENATGA8+EcMdxHGcrI5kW0hnAGFUtUtWlwI1Av0QJVfVnVf1KVRUQYB3QJi6vUar6jap+C4yK5NUDyAHuUNVVqjo65HFwEjI7juM4GU4yCqkdMCdyPAfYQUQal3WCiPwMrATuAv5aTl7tInFzgzKLMTcSH81/YDANzlq0aFFl7sVxHMfJEJJRSHlAceQ4tl+3rBNUtQFQH7gQ+LCcvPJCP1J8XCx+o+uo6gOqWqCqBfn5+RW9D8dxHCeDKFchiUhfEVketunAcqBeJElsf9mm8lHVX4H7gQkisn0ITpTX8tAqio+LxW/yOo7jOM6WSbkKSVUnq2pe2I4AijAnhBgdgB9UdXEFr7cd0DwcJ8qrKBLXPup1hzk+FOE4juNsdSRjspsADAgu3Q2BocC4RAlF5FAR2VtEskWkHnA7sBT4bySvS0WkuYg0Ay6L5FWIOUEMFpGaInJhCH81CZkdx3GcDKfSCklVXwRGAK8BX4dteCxeRIpEpG84bAA8ivX9fI552PVS1ZUh/h/Ac8DHwCfAP0MYqroacwk/HfgZ6A/0DuGO4zjOVkZOMiep6u1YaydRXLvI/pPAk5vIR4ErwpYo/kPgD8nI6DiO42xZ+NRBjuM4TkbgCslxHMfJCFwhOY7jOBmBKyTHcRwnI3CF5DiO42QErpAcx3GcjMAVkuM4jpMRuEJyHMdxMgJXSI7jOE5G4ArJcRzHyQhcITmO4zgZgSskx3EcJyNwheQ4juNkBK6QHMdxnIzAFZLjOI6TEbhCchzHcTICV0iO4zhORpCUQhKRS0TkexEpFpGxIlKzAucMFxEVkUMiYTXD+b+E/C6NO6eniHwqIr+JyGsi0jIZeR3HcZzMp9IKSUQOB64CegKtgJ2B68s5ZxfgeGBhXNR1QFugJXAQcIWI9ArnNAGmAsOARsAs4PHKyus4juNsGSTTQjoDGKOqRaq6FLgR6FfOOXcDVwKr48JPB25U1aWq+l/gwUhexwJFqvqkqq7ElFcHEdk9CZkdx3GcDCcZhdQOmBM5ngPsICKNEyUWkROA1ar6Qlx4Q6BZgrzaJbqOqv4KfB6JdxzHcbYiklFIeUBx5Di2Xzc+oYjkAX8FLi4jn+j5sf26kfhiShONj15noIjMEpFZixYtKvcGHMdxnMyjXIUkIn1FZHnYpgPLgXqRJLH9ZQlOvx6YqKpfJohbHnd+bH9ZJL4epYnGb0BVH1DVAlUtyM/P3/QNOY7jOBlJuQpJVSeral7YjgCKgA6RJB2AH1R1cYLTewKDgwfd98DvgCdE5MrQ/7QwQV5FYb/UdUSkDrBLJN5xHMfZikjGZDcBGCAie4Z+oKHAuDLS9gT2AjqG7TvgHOCeSF5DRaRhcFY4O5LX08BeInKciNQCrgXmquqnScjsOI7jZDiVVkiq+iIwAngN+Dpsw2PxIlIkIn1D2sWq+n1sA9YBS1U1Zq4bjjkqfA28DowM+aOqi4DjgJuBpUBn4OSk7la/3nkAACAASURBVNJxHMfJeHKSOUlVbwduLyOuTC84VW0Vd7wK6B+2ROlfBtzN23EcZxsgKYW0tZMzY0y6RQDgP/lvp1sEx3GclOFz2TmO4zgZgSskx3EcJyNwheQ4juNkBK6QHMdxnIzAFVI8H3wAXbtCly4wblzZ6V57DUTgm2/s+OKLYb/9bLv11pJ0hxwCPXpAQQE8+mh1Su44jrNFI6qabhmqlIKCAp01a1byGXTtCpMmQfPmplxeeQUaNiydRhX+/Gf4/nt4+mnYaSeYPx/atoX160vy2GUXWL0acnPhl1+gQwf4MtEsSo7jONsGIjJbVQsSxXkLKcqqVfDrr9C6tSmRbt3g/fc3Tvfkk3D44VCnTklY27b2m5UF2dm2geUDlm87n6jccRynLFwhRVm8GBo0KDlu0MDCoqxZAw89BAMHJs5j4kRrGbVqZcfr1kH37vD738PRR1eL2I7jOFsDPjAW4O674amnoE0bKI6seFFcDI0alU77wANw6qklLZ8oL78M48fDc8+VhGVnw+uvm2Lr1AlOPBHq16+e+3Acx9mC8RYSwIUXQmGhtXy22w4WLLCW0MyZsO++pdN+8on1D/XqBXPnwmmnwcqV8O67MGyYKbbatS3tmjXWpwRm3qtVyzbHcRxnI7yFFM+dd0KfPua4cP75JQ4NffvC5Mlw330laXv0MBNdrVowYICF9e5tv6NGQdOmlld2tvVPDRsGNWum9HYcx3G2FNzLznEcx0kZ7mXnOI7jZDyukBzHcZyMwBWS4ziOkxG4QnIcx3EygqQUkohcIiLfi0ixiIwVkYSuYyLSSkRURJZHtmGR+Jrh/F9CfpfGnd9TRD4Vkd9E5DURaZmMvI7jOE7mU2mFJCKHA1cBPYFWwM7A9eWc1kBV88J2YyT8OqAt0BI4CLhCRHqF6zQBpgLDgEbALODxysrrOI7jbBkk00I6AxijqkWquhS4EeiX5PVPB25U1aWq+l/gwUhexwJFqvqkqq7ElFcHEdk9yWs5juM4GUwyCqkdMCdyPAfYQUQab+Kcr0XkGxF5OLR8EJGGQLMEecVmIC11HVX9Ffg8Eu84juNsRSSjkPKAyIRvG/brJkj7E9AJM8n9IaSZHMknen5sv24kPhoXH78BERkoIrNEZNaiRYsqeBuO4zhOJlGuQhKRvhGHhOnAcqBeJElsf1n8uaq6XFVnqepaVf0BuBA4TETqhXyi58f2Y/nEXyc+PnqdB1S1QFUL8vPzy7slx3EcJwMpVyGp6uSIQ8IRQBHQIZKkA/CDqi5OnEPp7MKvhP6nhQnyKgr7pa4jInWAXSLxjuM4zlZEMia7CcAAEdkz9AMNBcYlSiginUVkNxHJCn1Mo4FCVS2O5DVURBoGZ4WzI3k9DewlIseJSC3gWmCuqn6ahMyO4zhOhlNphaSqLwIjgNeAr8M2PBYvIkUi0jcc7gy8iJnZPgFWAX0i2Q3HHBW+Bl4HRob8UdVFwHHAzcBSoDNwcmXldRzHcbYMfLZvx3EcJ2VsarbvrU4hicgirMWVCppgnoTpxuXYmEyRxeUojctRmkyRA1InS0tVTeh9ttUppFQiIrPK0vQuR3rJFFlcDpdjS5ADMkMWn1zVcRzHyQhcITmO4zgZgSukzeOBdAsQcDk2JlNkcTlK43KUJlPkgAyQxfuQHMdxnIzAW0iO4zhORuAKyXEcx8kIXCE5zhaCiEj013G2NlwhOc6Wwx4AqqqulJytEVdIjrMFICLNgZdF5EZwpVQZRGSLK+dEJCdN15VE+6lii/ujtka8YEmOqipotpDnvwi4GDhRRK6FzFZKmSKXiGSr6noR2UVEjqyma+RE9jf7nRSRLFVdKyI1RaT35uZXievmxt4pEamhaXDBdoWUYhLVfNLxxydCRLLTLUNFiRQ0rUTklLDMSZ0KnhtfWNYpJz7tqOpqVX0C+AvQT0QuCeEZqZSCXNvHjtMho4iIqq4TkfbADKBLWAanKq+RHZRHlojcB3QOy+VsTn7rw+HrwFlVImj51x0G7Ckiudiq3vul4rrxpKVZuK0SfXmBOwABVgB/AxalUzEF2daFguNwYD3wmap+kS6ZNkWQdU/gTeBToBHwoIg8qaplTq4bCikN+5cDu2GrGE8H3lDVR2KFfCZVFCL3ewjwI3CLiNRS1VsyUN4s7N2eISJPqeo16ZAtPJd8bI2161S1ygd+hv8lC5gFfIUtp7NqM/MT4FTgA1U9H0q/t9XE3sCRQENs3bkZ1XitMvEWUooIzfDYy/s+0BIoBDoBU4Dfp1G86Ic1G7gIGAlcLyLXpFOueOJMIl2BYaq6P7ZGVxfgTBFpWca5UWU0BTgK+B9wJ9AM+EvUHFZ9d1E5wn/TAngZmIu1koYCA0VkaEiT9pZSpIUtqroOGAzsJyIHplGsuth6bFNCK6Y6PBXPwypvx6rqj1Xw7vTElOjRYguXVtv7GHsOqnossD3QGFsMtVR8qnCFVI2ISFMR6Q4QzEuC1XC/VtWjVXUKVquqCXycRjljL91o7MM6AtgXaz20T5dc8QSlvl5EWovICUA34FsAVR2DrTLcAThDRHaJPz+ijKZhH94hqvo3Vb0duBB4EDhFRM5JzR1VinbAV6p6n6q+CtwDXA1cICIXQPqVaFCcbYErRKStqv4Le6+7ikhOKgq3BGbnBkArIGYKqxnC24hIrySvEX8fzQGNxGeF30YVMd/Fy6yqLwPHYlaKXiJSNxk5K3DdrFCRyQrm1RuB54AxItJNRHIi30xKdIUrpGpCRGpgNY1BInIwbCgwmmEvMCIyASjAavp5qezADNfPjsgFZsJ9OOzfD9QC+opIAxHZLZWyxRNRRrsBnwH9MLNGdxFpCKCqE4GngEOB7mXkcyXwJ+BEVV0jIjXDuQuAJ4E3gCM3px+gmvgFyBaRnQFUdQVmrvwFuEtE+qVRtmghfTG2yvPTInIA8C72P7WLFX7VKEPMtLmHiFwlIjVV9QNgDvBcaCGvDMmHAacncY1oIR3rI/sUWC8iO4FVPkN4X6yCVBGZs0TkTyJyuog0VtVnsGd5Cdbqr1KlFO4jVkluA+Sr6lhV7YdZDf6GWW8QkT8Dx1fl9ctEVX2rpg3oiDW9HwZ6hbC2wKPAv4API2mHhLB6KZItJ/ZdAQeH/WmY+eoO4ANguxA+DDgXq2Wm83k2A04Czg/HvYD/A64BGkbSHQ5kxe4vLo+Tw/O/HfhdCMuKxB8E/Absksb7zA6/2wF5Yb8O8E54l2pH0j4AHJeu/yb2fCPvSh3gLmA8ZmI8Cyuw3wAapUCODsAPWIHaKIRtD0wHvsEqWs9ips8aSf4vWcBUYABWadsN+BC4Ddg3pDkPWFyR9yjk9z7wEvBKyGtAiOuDVcCujL0LVfissoC3gH+H72gCsFeImxDet0nhe9gtJe9TOl7ibWEDcsPvXqEQmYR5ruRg5pYFoXCsgfXZ/Ai0T5FsMWWUhXnyvBWO9w6FR3Ek7XnAd6l6ITf1PIFPgF+BEyLhvcKz/AvQOO6cqKI5M7J/HPAYcB+wYwirEX67h490uzTdZ0yRtg+F6NtYBeFQzMz4NfAEcCvwONbnFzsnXUppe+Bu4MJwfAFwDtYveiZW0VoPnFrNcjTBFM3ASNjumEUiBzgfq/hdGvkGcir7/2AtrqmY+S/27PcGXgjXfyN8R/tUMM9bgUcix18Cj1CiOM4M+TasjKwVuO7jwOSwXwtYA/w1En8ucBXWuk3Nu5SOF3hr3yIvUswENhVYDTwD/CG8yDeH4zeA14COKZIt9iFmY62gQuA/wO/CS3kS5jH0T6x19zmwdzqfY+S4AzAPeDwu/LDwMZ0RFx6rFOwYCsSHInHHU6KUmkfCr8HMfmlRSEGGVpjiGQTsgzkwfI+1EJsBN2CVmvsoUaRZKZZRIvu7ApeHQvgBrJU5l1BxwPpxbqOaFSamkJ4FemDK+w1gZnif/5wgfaXlwVpFkyLHfYH+wIGRZ7EPsH0l8hwPnBb2H8b63XKAppgpDarYcoK1vp+nxEpwT7huNtAk0f+ckvcqlRfb1jashvtUUEAHYZ3uj8QKeKy21QSoU81ytMRqOtGm+lzgiXD8AbBHJK5x+PB6Aa3S9Oxi5pGGQD2gWThuD/wEjI1Lv2+0gMFq7c8AncPxrsASylBK4fjMkHfKaoRl3PufKV1j/hi4P+zXTpC+UrX8KvxvGgP5wA7hOB8z//wd6xwvAv5QXbISp4SB2phjzttYJe+h8O09Btyc5DXiK0XnY4q3M9ZS/Q9m3loAdKros4sLGw7cEsqG2UDNED4C64fO2lzFkOi5Yyb6/sBYzExYK4TfDhydyndqg0zpuOi2sGE1jWnAfpGwdpg56J/AoSmUpR/QO3LcE7g1clwIHBQ5rnDtrprkjZqtZmEmn9kEUwxmBv0BeDDRcw+/e2AtvJdjBUVQSsUJlNIEzAxTTAXNLNVxv5Hj84Fnw/5sQosQa2mcS6QGmwZZY5Wa9uGZvQwsxMb5gJlWzwzKYD0wqJrkiP3PO2OVp9OwlmUOZj7rFEk7GRiZxDVyIvtR8++zmAPMQ1gf7HaYckpoScD61WpE88IcBpoGeXtilcIvgBYhzSDMg3T3qnq/wnXvw5R2X0wJvh6uG2tpXxCO09KHmpaXemvcKG3CiA0M/AC4Jy7dJcBSrNO12lpGcfJEO2NPT1AAvgH0D/uDMUVaP83Pc0esM/dCoD5wQijg9gvxe4XjqzeRx15Y30Yhm1ZKJ4d7TkkfXpyMsf9ml8i9NcNq4UuAuyJpHwuFa2rNKHHvAtYSmo8pzh2xoQzfxgr98J5lYQ4N1dZ6C//vQswkPg4bZP7nEJcTnuNUrKW2OQ4M48OzH4+ZjbOJtFqw/rKvCOavuHwaYy22EyNh74Xn9yZwRcjvFMya8na4l6+oAlN55D4Ec1J4Gus7moRVxJ7GFNRkTEEtrIrrJi1vui68NW2UrknlUOIZdTCmlC6LxF+A2Wt3qEZ5osqoAyUmgKOxKVSui5P5IeAI4ETMjfgP1SVbObJGZToIeDHsZ2M1uUnhePvwuwuJTSDR2mwHylZK/4ikqxIPpkree1ZExm8xE03dUHicEQqt67ABv5OxFkmsJpsSpQTsj9Wq946E7Q28GZduH2AlcFSCPKpcKWH9nY8DgyNhKyhpqTXGnBemRJ5ZpfqMIoX4k5iJbgxm2uoZef/uxeYZLLNljZnAPsZMsTdg491yg3xPYv3J2Zj5/sjwf2+k3DbjWQnQG7g3EnZ6kGMCcAzmWNUf2DXV30EpWdN58a1ho3RN6lHgRWAicHgIHxAKltexms8v1fmnU7qAn4h5/sT6rLYDBmK1xusjH+q1wHKs5ZYOk1VuZD/m9dYF64fYHlPqj4Tw2ljr8neRc6KKbEONMBLWNhSqhZQopbZYC2t0mt+flpgyOicSVgPrN+uKmSvvD/JvcEhJgVyx2n8vzMT8d4LjDeYA81/gyLhnPhk4r7re57jw7PBsDggF7oeRd6QT1veYF7mPSitFTMlOixyPw9yzs7C+qcaYO/aeZckY2b8RG9/zJMFcjymlflgr5a9U05APbPjJ+vAdxb6vLKzSMyG8W9Xaj11hWdMtwNawUTKO4DHMNnsN1uQ+JMS3xmpBF5f18laDTFOx1lAroG4kPKaUng0fSTbmBv15qmSLkzPW13Z7kHU15om4M9ZhvIhIXxFmaniaBF5llK4cXBYKi+Ox2vSOlCilgpCuDel3Zz8UeDLs18AqNdOxsTw1y7rH6v5Pwm9u2LqFgnQ00CHEPYS19Asi5/2bKlRIlLQgc0LhXxDe31hFamJ4l98huC9H3pH+keMyW5PlxPUG5of9sVgrpwamAE8J79Wmzt8h7vgibEzPlZQ4EGRj/V//Dt+jbCrPzXiWnTHHi2MoGTOWBZyNmSKrzWJTKTnTLcCWtFG61p0X2b8ceC5yPAmr9S4ggQkjBXIey8YmlfqYYowNFjwzFOw3h+O0dJSHwmZ/rCN1MaVbCgcCP2MegoOx2tyGAY2RAitqosvCastTMRPLREwJ1cDGxdwJfET6XNnj++86YQMiH8FmNZiEzbG3CDgmDfLFlFE9zKEk5rp9OKaU7gJaYOaqZyjp4H8UMylWiXkuIkddrOVVGJ7JV1i/S2PMBLYeKIycNxnroylXccd9zwdhyu184LCYDJjJ7z1KD2K/HHMnL3OgL6Z8bsUqVu8C54bw4Vj/4J8JHpPhG+gD7FTN/213bNjEsZFrZ5GiwfgV2Xy270qgsbdY5AHgWxF5UFW/w0war4a4h7DO1v2wQuYpETlJVaelUNR8zASH2JIM+wD/wF78n4LcD4fpSPYPU5X8lEL5CLKJ2uznn2Id0CswzyMAVPUNEfkTVhhuj5k++4dzckLYd2pToOSo6lrMCeIbtckiEZGnMZPgGuBjEfkHNmZpSQpvlSBLbJqYHbH7XKaq74vISOz+V6rqYyHtm1hrIJXyxSYAjimj11X1SQBV/ZeIKNaBfwXWAX4WZlo9APN6PC38N9lqk6smK0d2vBzYwOcvMe+z3pjJdRBmkbhCRF4E1mKmuq7h/DLliJtodzxmOo0NecgVkUJVPVtEXsM8Gx8NUwWdjrVwDlHVTb1DSzEl9ybwnqreD6Cq14d3dwQwREReVpsG6tEkH1eFUdXXReRsrN+rpohMC9f+pbqvXWHSrRG3tA2r0a7Hak0XUGKTzcZs/h8QahyYSWMo0KYa5dmoeY+1BD7AOn3/jk2ZMgIzD43EapsNsP6YBml6jrEa8PaYW2xLzNb9KTCqvHMx5foR8FJc3GDgprA/kdCiwgr3WGd0pTyuqvJ/wtylvwr/z3tEBvli5rGs8L99TIrHFwUZtsMG5Y6IC68ffg/DvMHuIoxdS/S/VoEceVify90J4vphrY7Lw3EHbBzafkTMfBW8zjTM5BcbS1Ufa638SHACwJTQE1il81mC2XJT73XYfw4b13ZNrJyIxN2AtfiOSMN/fAjWxVA31dcuV7Z0C7Albpip4gts8N0lQNMQXoCZX3pQMoAuJWN6iLgshw/qSKwP6Sbg2EjcfqEgb54KucqQNaaM2oeCuV8oCHMwb7//An8LaR4Fzi4jn7aYa/iUSNjlWN/TQ6GgqRPCrw6F6EYDS1NwvzFlVBczAQ3EKgOdsNr/2yF+d6z2OpMkPcOqQNY+WOsx6qbcFxtvFPMePQxTmuOpJgcdrCL1NbBzJCw6lufWEJ9ooHCFZq3AzHvzEryXOZhJ7f+AUyLxNUnQrxeJj86C0jJ8g7Fxbn+N3ktIdwHVWFkt597TNhPJJuVKtwCZvJX1YmN9HjdjpovZWAd6PjZn1stBKX1DijzWsPnmXsBMWx8D14bwRC7RF2JmhCqdFysJmffAao9DEsQdhk2VMzcol41aNJECu3V41hMicc9hrdhYS/U8rMa7Vxrvt2l4NwqBtnHhrwNnheNK1/KrQc7YO3IiVpv+ibipd0KBfVNFC/8k5NgVm9rmIWD/BP97S6yFUam+QKzikxW+16ewyUx3CnHRvsg8rI/17xXMNzoLyozwPX6L9X8+iE0JdCPWAjuVBC0/31whVewhWWvnEEo6AltjXjH7Y2aw2dikjVnYyqWtCK2mFMkX885aBrwaCY9+YC0wL78lpGjevARyRjuRrwSGh/1srLN3FNAnhOWHQm9DrTX+vrCa7EjMHXc9JTMaNMJMK3PC7/uVLbiq6f4nYl5Wp0YK1izMkWFYXNqUzk0Xd+3tMdPn3PBcY9Mv1UgkV1XKGvfONsX6Z8cQZtGOxB2ImdAqPIM45mk5lpKZ9/fAPC9fI0wXFf6P2Pv1EPBMJeV/Hmt55WCeeHdj8/v9DVOA72DOO2n5BjN9S7sAmb5h3jLrw8c5E+snqo+1RmaEwrRveNGGkcJpdyhtrx6OjUifjLnpxiYWFczJYkQooNP6IQA7BJmGYLN3nxKe7fOYW/EvxLVkSNzSk1CQTMSmZOqDmVii5rv9sJp0yqdCKquQDspzFjZoOtYv8xwl3o4pnYVhE/LHlNJHwHHl3VcVX7sJ1qqoi7WUprNxS+mR8Cwr/LxCfs9j/UE9QtgemMPPqxGlFDO9PQhcXIn8c7EK0O/j3sHHMdPdLqHcSNvSJpm+pV2ATN+wMTFPhZrOM6HQfDEogKcoWf9kIJWssW2mXNElJE7COnVjte0pWIsu1rrohPVPpHWsAWYumUxJK+gJzFvrynBcOzzjthXIq0Uo2HeKhO2KmXHGpPk+syPy/BVrDUaXzJiE9UHOCIXehhkYMmmLKKW3iCzpkILr7oW5J8daMrsFpfQPrN9wHKXd/8tVSpS0euoFZTYlgVJ6jZL1gC6iknO6hby/omQZjpgZ7++Y+S8jKhuZvKVdgEzdKG0iOjIUpCOwgWWdsL6AZUQmjyR1i+vFz9o9KfbhYG6vj2BjQ44IBcq/SXOfUZCtViiAN5oUNcRPwObySmQWip91uR7W93JGRAFkhzzWAw+n6R5jsvw+KMdY5/+HRGrbWAXne8yDKza1U6YqpQsxl/tqmQG6jP/7SszUGnMYaouZ777DWm2VdvqI/Df1MRP31ARK6VnMDPwTSZh5sel3PiEy/hBzWR/tCqkCzy/dAmTiRkltKhtzEjgc85x7PLy0rTEbcRtKr1Sa6kkv/0HpZQpiiqoeZnd/FuvwT9cg0ERT3rfFxqxEzUBdsA7/D0iwvg+lKwd1I2luw8xdHSNhV4ZCPmXeS5jbcbvIcT7Wr3hFOG6EeQMuBYZG0k3C3Jf3ZRPeW9Ukc2VMXTtiJqcq9/iLfGs7EFpE4fh3QWF0i4S1wbzrKrW4Hon7H+uH5z8topR2D4oq6VnfMWeIv2CKczzmNfkTaZi4d0vcYgWYkwARuRRrwvcPx/tjbt4/Y7X899MoWy6mIB9W1WdFpIaqrgmDG9eHwXf5wFpVXZQimWLX3jAgUUR+h5k/rgPWqOoqEbkBQFWvDWmysZkjxmkY9Ko2yDWaZ2yuwCZYP9NrqjpaRJ7AxjEtxgqBC7H/7KsU3K9gAymfx2rz96rqf0WkPtav+BCwDjMtfoLNqj4KuFNVh4U8xmIDS09V1feqW+ZwzegA3e1VdU4lz98wqHQz5Yj9t7tirfmrsJb9FLVBnHdgY50OT3DuhneknGvE7jUL83JriP0Xj2EDsR8krKMUrtkG+FVVF27GfdXAnC6Oxrw7p6rqf5LNb1siK90CZCoich02Ev2HcJylqm9j9uA84CIRKUihPKX+K1VdDawCeiRQRidiUwEtTKEyyg7XzgPGicjxIWo3zBHkZWCoiOyJucWfLiItw72sU9WH4pVRiFsfCv6XsdknhmHm0j4iMkpVT8T6opZgg333T5EyylLjJ8xBY09ggIjsoarFWIG3BhscPU9VT8NaSe8Au4UKBaGy8wpm3qt2IrMxtMdcuzuJSONIvMSnj+y3DTJXpTLaDRt3tgCb2qYhcLGIPIP1u7QSkV7x51dQGUlEGb2HmeWKMEUxEatMnA/8CgwTkW6q+tnmKKMg2xpVfUVVB6vqTa6MKkG6m2iZsrFxH0UvrCP1JeJMXljtZywpdO2OXHs4YalurP/kOWytoNigxcHYB56f6meHmQo/xVoGh8elORUzsX2PTcXyDdaPkltGnsMIZjfMOeBtgnkUcz8+AOtDOixyTkoGkVJ6vreDwn63IM8oIhO2Ym7FsRkFHgSuicQlvPcUyL8LZYwBi7u/qJv+BZjTTpX1RWIt+IspvTxLLczd+0GsEvIb1qKsaJ47srGLeG+slRI7fi78VzEzb/1wvWqdS863Cvx/6RYgEzbKsEVjrrlTsQ7JfeLiaqVBzsaY3ftR4E8h7Nbwgf0nfFTfx8uaItlqYC2X+AUJ68QdH4KNyZiLeTE1COHRwu9CzDHhHWxM106YU8AekTR5WIVhSCSs2vvw4pTRj0SWxo4oyb9R4q11F2E+M8Js0amSdRP3cEqskMdmH/gHVjmIrg8V7cM7nypeTTe8L29i5u+hiZ4J1t84MCilAyqQZ03MIeFFSruIXwT8M+xPoGTW7tqUePK5w0EGbGkXIN1bpIDJwlo947G+mVjBcQjmInoHcTWvFMi2UQ0am4TzbszWHltzaU9soskTSdfSw6Y43qD0Uhd/xAYFPojNvhDzJsvFzGuzSTBvHeY08noo8P+HjSWazsbz1t1FcBlP8btSD3NLTjTPWg9KWko7hrRHY+tipWw9ozLkzsOccQ7FFP4NWP/WM1jf6IeEJVMi556PmUOr3DEGM+e+F97l5gnkjbW8bwcuqGCeB4bv4wmCQwQ2Tu05rIX3fiTfoZhZMOWLM/pWxv+XbgEyYQvK6IPw0p5FicdXbH37gzHT3W1UszcUNl39eZHjHMwEtkMkrDnWN/FPIiarND/DfGwcx4VAT2x80Vqs9v0h1nraM6SNKftDMXf66ADfbKymewW2DMMErBO6Q8jnLazP5jrMkSGl6xlhDhQLiLSMQvjhsUIVaykVBqW0a1y6VCujmFdZe6zlcHA47h8K5DMjaV8kmCDD8dlYv91mt4wiSqYOkSUPsH6d/2CmzYTzK2KecPdX4l73C0psCra6bW74Xr6kZAzcJVi/3e835758q+L3Nd0CZMIWCo7oypD3YjXIzymZ56o7KbAxY4vlLQMuioR9gdXs8iNhLULB+A5wfAY8w9xQwM3A+pGep6SGWhvrMzo77pybMdPddtg0TFHFdGaIq4V1QP8HU9ajsJbspHQUJtiMEOsp7Y58bAg7IBJ2ANaBPijVMiaQeTfMOecKEre6hZLVUKNjde5jEzNbV+L6sTz3wip2/wr/aWyxv3bhWd1D6ZWAs7FW3cflKUVKWj2CVXKmYYs9Ph2u2xhbGXkWpnjfxqfvybgt7QJkwhYKw5gteRJWE6+Bjbr+ihTOjB0K5xOxmYyviIS/g7XaGkfC7glbyp0r4mSOFQY1Mbt/W+IW/MMmmzwuLv3NHJa0WAAAFvtJREFU2EwY40KB/gDmmhur6d5KUMxYa/BDSgYAp2UQaSioL8OmKeqKVVSWkKCliplSU90i2o/ImKgQdiVwa9iP9g0JpuSnBWVUarBpIuVVCTlK9ZVhLaEfsaUYTsDMrZ9ElNKemKPF5QnyqrAcmMKbhFWC+mAm+KcomRYoDxs6kDGL0vkW+f/SLUDKbziugIgUfrmYqek9SuaBuyO84CkZZElJH0MtzHYfv4LqO0G+Y7DWyEygWbqfaZAt0bpMscLoYaxmupH3VjjugpmG7sX6il7ETCojgDsi6V4hrBOU6Hqpur+glK7ETD7r2dgL825KTxWUimXHBXP+mMDGyxyMofS6S7H3rCU2aPeAyH+z2TOMY/1oAynxiqwVZLgtIuuHWEt6PqGli/UdJpy3sILXrYFVXKIODV3DO/U0FXCM8C29W9oFSOnNlnZgGI155IymZM2cw7FWSDNK+pLqpFjGrKB4HsBMcj9SuqX0WPjo5pIGb7oE8iYsLEIhdAxmx59FOVO9hILjY6xfaQDmDvxTKPAPjaRLuWtu5L2pFfuPwu/l4f+JmurGY30VKV06ghKHkZgSaEqJp9/xmKPOH+LOuQM4MP4+q0CWa7EZKPpF5OmMKcxsrFI1ORwvxDz4dq+sHGzc2msQ3qFLou8m1p/5A6YUU74elm+VeHfSLUDKb9g+iFmULJT2SPggmmMdv89jtbdvgU5pkO9e4Imwv3P4qL8mYsrAauhpWe0xUjhvcqVZzHx3PGbeqtBUL9iSzx8Dx4TjjiGPOqRpOYaI8mmHmX4mhf+oZQi/Eqs49MAqN9FJP1M1Lqo7ZlpuFo7zMDPoGKz/aMfwXt8b3qemlLg/V4uM4X9/Gatc1I0oh4FEvCUxh51LKysHG08n1SjsH4O5kkcXpRyMDWhvksy9+Ja6Le0CpOQmze04NjixN/B8JO4u4JPIcVPMDv+7VMoYuf79lB7bUgdb7fRnIgMI0yRbrFD5PebivXs56aOFRoX6fELBPp8McNSIyLQzVpO/GnPhvh9zPKkbtsuxltynEWWU6hbSW9ikozuG46OxVtFdQSHtgo2PmodNtvtidShOSlpqdbHK3lzMoy/mVXd6UI57Y84p/4y8V5VqGWHWhJdCHouCYmuPWTdWY5WHSVhLO6XemL4lt20rUwcti+xnA1a6ikzCapf7iEhtETkd+ElV31HV/6tuoeKnAwrHS4CdI1PL/IqN1/k/4AwRaRw/vUuqUFUVkSbAycCTqvppWWnD1DCx+ei2U9U1FbxGIVag3CAifWLPIVWIyOFhjrfoNDp/Al5U1VuwwrQj8LSqLsPmQ7sTeyZ7qU3hVKF51qpI3hwAVe2CvTuvikhTVX0GayU1wyb7XKuqlwMF2MDYIyKyrqsqedTmKtwHe2e/waZPGgocLyLbYZWNulhr8vdA7/BeSVlyxL/vqro+7P4bM5megVUWDsYU3jSsP/gLbLqm7qr6v6q6R6caSbdGrOqNxJ3rlwIzw/7h2Ajxpyjt5nop1vG5SVNUFcoZM2MJtpxFJ6zfZXtM+dxDyTios7Gabv00P9v6WB/bB4QaJ+V0QgPnYDXhyppkDgn/T0pNk5g78OdEPBexaXNGhv2PgEfDfnPMJFUjkjbVLaOsuONCrKUWW7bhiPCu30GcCTrRt1IF8mRjA11viIRdiymo/uE4H3NgKHep9rh3qXEsLLyLLxBZfDG8M+9gSi4t34hvm/n+pFuAarux0i/qnsArkePYOKMumAvohZhHW0qmiKfERJEVCt2XsE7XKVhtfEds0ba3MdPYj6RxzERE3nqYeeo3Igu2xRUa8fOfFScrO7Bdmu73Jax/JVaon4yZhP6DzegdSzeJBLM1pEHeFrHCPhzHK6Ve2CwFFV79dDNkqRGuNTAufAw2rdUgIhWreIW6iXwfx8ySMbf/7bEWYUzJxd7R+4ksyeLblrWlXYBquSmrkc3BBvZdh01h8wvQNZLmLqxp/wrW9E95gY+5Q8dq2x0xV+cZ2ESdDTETxLFA6zQ9x3jvstjxFaGAPjmSVuKUUcxtPe2egJW432if12vYYM3tw/GtWCVmn7A9TAas9Iq1SAZiE+qeGwkvDAo0ppT2q2jhn8w7Ehd2HTZ90s6RsEOxaaD+SiVbZlhF6L+Y48Z0StzEh4TjgyNpbyZuFg3ftpxtq1wPSURaYGMs9sPmtlqDfRB/wyb//Cmka4ot4bBOVX9JgVw5akssiKqqiDwMvKyqk0P8DpjpsJaqXlTd8pQja2x5gL2wwaqrMKX+N1WdJyIXY55Ul6vq43HnDsbmSTtIVT9MtezJEPlPNqz1IyKvYzXx7qr6Y1jH6Q9YC3E51gpYE13/KZWyRo6bYxWXU4CJqnpvCH8Fcx5oo6pLQliWlvTBbK4csXekDfadLcP62Bph02xlA7er6vsi8iBmCh0RzqnQmkqh/yi2ltEe2GDazliF5xes76gb5kpejPU/HqSqn1TFPTopJt0aMVUb8Gfsg7iS4CKa4utv8CTC+ocaYZ5OY+PS9cFMdSmfTTyBzK2xRe+uxFqZd2KKqQHm/TcIm6+uZ+ScNtjUMH9Il9xJ3Gd0upz6lJ4NYwbW0siPpKkR/T/TJPOOQJe448Hh3Tk7En53dcgYuf/dMNPZjHDtF7C+0PbYPIarsNbmfynpNy2zpfb/7Z170F1ldcZ/KxcggQYIt0oi2qYGgpoRBqyDihegFgIiIKiI1FQLJEGGggaIgiCg4JBAMEjkngQYI9ckxQTEYqIkwxgp1wkjNjQFyqAjF4GKkOTpH8/a+banARJzLt93vveZ2fN9Z593n/2ec/bZz7vWetZavIH1hGX3j+MF2yX529klz3UEzs+7gMy9Klvf3Do+gZa+uVqsJv8eglU359Di0iHr+2ElGc0CFuTjsTg+dGZtzETsSmwrIeHgdyWiqD63CcCs2tyXAbPz8WZ45XpU/YaX+9oiDGnS+67I6L248sUDOEfn87Uxi3FMaeeGYzvSsiA/41m4GHC9pt5fJwH9Gjhpfe+zyfMYjuOx4/Px2Lx2F5MJqFiss66E0obOA7veJud1ViWuT8aLoD2wgnARmXrQivdXtvZvHZ9A29+wV1MPUlsFt+Ac9VhKlZi4FVbLzacngXEAtjyez5vhrdgiaXqp/w2Y8z1Ypjuytu944OL8v64uexuOW2xeGzuwUzfoJrz33bCo5FRcMeKsJKAv1saswO6wTs2xsdzSSKyeuwGnLlQ3/H/Ka+n7NMT1mjyfYTipdRUZS8xr4F24ncXPaVj0seEChoNxvG4tLh81DVvrB+HY2BbYfTeT7MDb6WuobM3Z+kseErDO934LFjf8voXnqGIQc3BfloW4uvEEHNcaB86nkPRj7I6Yg1V2+6oDcRdJH8Ok8+8R8fbc/Tuc+/QYcK+kz+X+i/Aq9bXa8Wuq990XUOW2RMRgrKK7TtJUSffi4p8vAxMj4mgASWNwlYNOzHWgJGUO2oiIeIekp7CLbiiWnh+cw/fEbrNJzf4+6vlAcsx1MV5A7Z9zWoM9EKfmsO/Wj9ebxK4iYteI+GLm4i3D720VJrkqmXcHfN1NkLQCW0nLcC5YQRegK0UNb4UNDahu4jluw5Lyz2Gr7AD8o/oQXvldJunnrZzDhqKeyBkRi3BW/8ckPRURZ2OL4UM4XjQB3/T2kgP6Lf8sm41KhBARO+O2BENw7tdv8cr+19g9tARbApMk3VY/to1zrYQDY3Gc5AUcp/uKpLkpzLkAW+LD8c15L9XEM02aR/WZbYldaL/N/Ydgy/8h4GpJTyRxjQD+581IqPbaY/Fi6DhJV+W+IbjT6yTsal+DS0tNxsKaS3Lc5pL+1Iz3WNAL0GkTrRs37Mp4kj9PmFyClVDbYD/79fSC6sOsx42C83BW0tMLagqOV8zN99bWWm2teL843vEUtnp2yn2H8ef5anOAL3T6fQKjsSVyCq5yMAErR6scnGE4bWAcGxmr+Qs+s2VY0fYz7NIdhAljHlbCveutrq+G57fHVtUp+XgQtlCH5v9n52/po/n8tpTWEV27DdoI7irYcMzBUtT9gYUp594Jy8tfiIgJOA/qaxGxRtKyTkyytuodgS2FYZJ+IemAiFgILI6IfSV9OyKGAa8orYN2WwrNgmxt/A1O3rxA0nW1p4cCL0XE/rhB4A7ADXlMRyyjfFjVX5yWLsajcYHgK/L6mYUtjAfy2KbONd//SLwomZp/j8Vt6XeS9K2I2AoT5ZNYDbfu2A04xQrsKg4cLxqF5fZXY5ITMCcivizpzk6VzipoAzrNiN264UDzCnp+pFNyf7XaHIGtpI70M6LHXfseHNBfjAuI3kKPQmoRTmZ85/qO7asbcAx2L4Grkl+Fb7TzcJ7LPGwJVJZgpyqNv48eK2QPLFK4F/hhPr8Yu38PbsNc9qZWlDj3fSmvkcrC/MTGflbYzTgrv4OVWKRxMhb/nIgFDANxY79XqCXBlq37to5PoJs3nCz4cnUDyX0DaIFbZSPnVZHRdljZd3z+6EdgBeKC2tjl9fl3w4Y7BL+MJdL3JQlPyc/ik3mTfMs6a634XujJ1RmG859Oqj3/EeCe2uNzcCJo0+dYu0YrUt4P5xttWxszCLvbvtxw7MaS0mic1LqSHgXq6eshpck0uATL1l1bv1LZtRuSluBV49iIOCyrXq9VulPUIZeXJEXEDtgdMgr3p1kj6Wlc32/viDgjx1bVobsGsov0aGwRzpB0hKRvYyJaLek52U21rmJ5KxER74+I3WWsjojR2HV1i6RLa0NfBEZExEmp4DwQW3qrq6rfTZpP5cp9D3BJRIyS9FOcGnBdRGwBkJ/NI9jCXgdtfCWItXgx8DBwboocLsTW4Mdx7tFgSd+V9Pgbv0xBX0chpBZDlhFPAM4DDo02t1OoIyIG1h6+hgPjOwLHVH55ud3FtVh5Ru5bGw2tMvo6JM2XdL6kORExKCJm4VyxO2tjmlJi582Q8cWTgW9GxK65e3vcQ+gztXGBCfR6nI+zHU5fqModNY04k4zejYULz+J+QmCSWAPcHxGnRsStuFfUwk08328kjcfW0Ags3tgdy8Yfxu7KIW/8CgXdgn4p++4EMlD+HewDf+mtxrdwHqNzDjMzEP11/ONfiMsYvRYRdwEPyf1zuhZ5kx8NnIll0/uoM7XpDsKdcQcBF0p6NCL2weR4maTTa2MH57+rk4ya3nspz3E58BtJF9QFFkmgJ+ISSq8Dp6WF1pTPLCJGYcHPM7hU1cO41FdL8gYLehcKIbUR6bL73w7PYRxWSZ0s6dKI2BrHT/bDMYz/wFnwH1c/yO9Ii3Uv4L60DNrZXK9+o/8Utnw2x72E/jMiPoKTXC+VdEaOq+eMtSQHLIl6EW5KeHE+HpjEs10jOTSbwJOUZuN45smSXnuLQwq6BEX23UZ0gowa5MNIuiMiDgduzecuiYjzgFdx35xngIlpKXV90mHe7JbCuhtrW8ioOn2edwx26waO4UVEfEfS4rSeFkTE1pIm1ufXLDJazzWiiHgE2D7P+yKwOolpUkRcJ+m/a+Obak0mGR+DrcBCRv0IXRUXKPj/yPjPLhFxfG3f7bh6xLSI+Eq6EKfiKgW7AUdGxGbdTkaNaLfIJG/8O2JrZD5OMJ2Eq4mfERGjJS3G7rxdW5F/kyS8NiJGRsSBKbAYjuvkHYHji3vm8Fk5x6eaPY9GSHpC0pOtPk9B70Jx2XUx8gY2AFcbOA13O/1e7flTccO0KZKmZvLrGbg00CxJN3Zg2l2PuqstE05vAA6vXGHpVj0Ly74vkvTo+o5t1jwi4r3YNfgYjgsNwTlGe+Tf3bAkewiumPB6o1VVUNAMFJddF6K6WeSNa01E3I5jE+Pzuek59FdYWvupiJgu6Q8RcSHuXLukM7PvfiQJvA0r5Z7DMbs9gLuTJO6IiPG4fuBK4NGKPDaFjBqIsCKjrbGI4EJJMyJiN+CXmCAviogluK3FAODhtKbaFmcr6F8oFlKXoZZD8g7gozi7/TFJj0TERFylYKGkcyPiGpxbcn1dsdWqYHmBkXk807CKbVpEfBPHkMZJ+lWOuRxXyZjejO8ieoq0DsacWAkjhmL33GdxJ9z7gEclHRvOQ3qmLmIollFBK1EspD6ORvJQT0LjXfjmsg0wKiImS/p+RLyIY0dH4Oz345OM1uWyFDJqPurfk6RXI2IVcEJEXC7pnHB166UR8UPc+2cr4MT8bjaJBGqLlL/CFcPvj4izM2a2NfBO3Bb8TOBxScfmoScAtwE/rV6rkFFBK1EspD6M2qp3EK4C/ULewG4HfpYKum2AQ3HDtk9LWhgR2+Gb3v3FBdMa1Fxi6yTRjarFiFgALJZ0UT7+JP5eNgemqQn5PbV5DMNloO4GrgEelPR6jjkN58jdKenA3DcHJ73u226xR0H/RbGQ+igaVr1X4GZ6y3H+xmDgJQBJLwCzMrfjyIi4J10wVQC9LeVx+huSBAYDV0fEj4GbgBkRsUDS/Bx2F653WB0zv/4azVgoVKSIi5feLWli7fW3xdfJ93D5nq9HxBXAzriCxwfzGituuoK2oBBSH0SueisyWo4z+mcAT+eQ5/Dqtu4qWgWMkfRq/bXKjaalGI4rvU/CBV0fAa6KiJuwO/VK4KsRMV7StY0HbyoZ1YhkM+ya+0HuH4wl3V/K/TcAl2Exw464EeM8tTlRuKCgEFIfRCVAwHXNlko6qWHIlcAdGS+6Dfen2Qf4Q3tn2r8h6dmI+AEuinoIbnC4JyaDU3Bl8ZXAQRExF/hjEyXdRwJbRcRsnID7FHBIROyO3YInYMn/COA44GlJNze8RrsThQv6OUoMqY8iXK37Jlxa5YFGhVxEfBDXI3sBLzwGYRdMn2w73leQ7rEBVXwm9w3ATRv3BP5V0qLc/w3c8+hw3D14aZPm8C/YGnq/pOW57x9xd9x3446vV0i6L5+bB6xaz8KmoKCtKBZS38WWwC44XgSOAaxTyEm6NyLm42ZzQ4FfFBdMa5FkdCvwo4i4peYePRhbSDcDZ0XEcEk3SjovLd1zgG9ExKe1ieWlwq0bzseLj+UR8XbcAmUWVsttBbwu6eXaYWuxS7egoKMohNR38RL+/g4AfqmeFhED0lIag9VaD1XKruKCaS2S8GfgKtV/Am6OiMNwK+5DcYfX03E9uC0kXZPf1VzcubcZVusrOD71bES8D0u5h+I40mxJz4MFE/j6mYldeEc24dwFBZuEQkh9FJJ+HxHfwg3U/itX3GtJSwn4Ks6uf612TJHvthiSfhIRE4DLI+LvsXDgKEn3wDo59ZbAByLi2rRoP4wb0Q0F/riJU3gZx6wuxNXDp+Br4hPAoIi4Wu55dRQmyZHA3s2QmBcUbCoKIfVt3IiTGq+MiL/FVRdexa2exwB71pJeS8yoTZCrdB+HZd1TJd1de+7JiJgO/K72nTyG8302uedPCilm4xYjD+LY1fOYBPcD1kbETNxmBGBuceUW9BYUUUMfR5Z++Qx2zQh4Asu+P68ONJsr6EFEfBiLC84C/q1Rcp8uVjV7sRDu9joO+ABWVk7FkvOvYbXlUuDiWmJsuUYKegUKIXUJMslxS1yP7PlKGl5WvZ1FuMneTNzC/kd19V0bzv13OOn1WdwOfAVwLhbCnF6s5oLehkJIXYripus9iIh/AP5Z0mc7cO5RwKWYlKZLerCWGlCukYJehUJIBQVtQCdv/klKNwLzJZ3f6fkUFLwRCiEVFLQJHSalEbiVRCkVVdBrUQipoKAfoRRKLejNKIRUUFBQUNArMKDTEygoKCgoKIBCSAUFBQUFvQSFkAoKCgoKegUKIRUUFBQU9AoUQiooKCgo6BUohFRQUFBQ0CtQCKmgoKCgoFfg/wB/w0bk7H5f/gAAAABJRU5ErkJggg==\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"waterfall(valid_xs_final.columns, contributions[0], threshold=0.08, \n",
" rotation_value=45,formatting='{:,.3f}');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This kind of information is most useful in production, rather than during model development. You can use it to provide useful information to users of your data product about the underlying reasoning behind the predictions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Extrapolation and neural networks"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The extrapolation problem"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#hide\n",
"np.random.seed(42)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's consider the simple task of making predictions from 40 data points showing a slightly noisy linear relationship:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAD7CAYAAACYLnSTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAU+ElEQVR4nO3df6zdd13H8eebdaHXdvU6epm2ZC1M1pkxYe4SjNVhGNKMaJytJsAQSDQFzRL/ahjKoOKwhRH/0BiwCbCxLIiYrREXXCDdIgwh3qXZZrUbYUuVO8VOaG23sl++/eOc251ezs/7/Z7zPd/vfT6Sk/R+z/ec8+n3nPs63/v+vr/fT2QmkqR6e0nVA5AkFWeYS1IDGOaS1ACGuSQ1gGEuSQ2wpooX3bhxY27durWKl5ak2nrggQeezMy5bvdVEuZbt25lYWGhipeWpNqKiGO97rPMIkkNYJhLUgMY5pLUAIa5JDWAYS5JDVBJN4skVeHg4UVuuecRnjhxhk2zM+zZsY3rrtxc9bBKYZhLWhUOHl7kA3c+zJnnXgBg8cQZPnDnwwCNCPSBZZaIeGlEfDoijkXEqYg4HBHXdtx/TUQcjYinI+LeiNgy3iFL0uhuueeRs0G+5MxzL3DLPY9UNKJyDVMzXwP8B/BG4MeBm4C/iYitEbERuLO97EJgAfjCmMYqSSv2xIkzIy2vm4Fllsx8CtjbsejvI+Jx4CrgZcCRzPwiQETsBZ6MiMsy82j5w5Wkldk0O8Nil+DeNDtTwWjKN3I3S0RcBFwKHAEuBx5cuq8d/N9pL1/+uN0RsRARC8ePH1/5iCVpBfbs2MbM+eeds2zm/PPYs2NbRSMq10gHQCPifOAO4LbMPBoR64HlyXwSuGD5YzPzAHAAYH5+3rnqJE3U0kHOXt0sde90GTrMI+IlwO3As8AN7cWngQ3LVt0AnCpldJJUouuu3Nw1oJvQ6TJUmSUiAvg0cBGwKzOfa991BHhtx3rrgEvayyWpFprQ6TJszfyTwM8Av5aZnUcQ7gJeExG7ImIt8CHgIQ9+SqqTJnS6DNNnvgV4L/A64L8i4nT7dn1mHgd2AR8FfgC8AXjbOAcsSWXr1dFSp06XYVoTjwHR5/6vApeVOShJmqQ9O7adUzOH+nW6eDq/pFVvUKdLHRjmkkTvTpe68BK4ktQA7plL0gSM+6Qkw1yShlAkjCdxUpJlFkkaYCmMF0+cIXkxjA8eXhzq8ZM4Kckwl6QBiobxJE5KMswlaYCiYTyJk5IMc0kaYJgwPnh4ke37D/HKG+9m+/5D55RgJnH5XcNckgYYFMaDaurXXbmZfTuvYPPsDAFsnp1h384r7GaRpEkadIZov5r60jrjPinJMJdUG1VOINEvjKfhqouWWSTVQtH2wHGahqsuGuaSamGaJ5CYhvlFLbNIqoVhShlVlWGm4aqLhrmkWtg0O8Nil0BfKmVUPY9n1VddtMwiqRYGlTKmuQwzCe6ZS6qFQaWMaegoqZJhLqk2+pUyBpVhms4yi6RGmIaOkiq5Zy6pEaaho6RKhrmkxqi6o6RKllkkqQEMc0lqAMNckhrAMJekBjDMJakBDHNJagDDXJIawDCXpAYwzCWpAQxzSWoAT+eXdI4qJ03Wyhnmks4aZraeImHvF8X4WGaRdNag2XqWwn7xxBmSF8P+4OHFgc9d5LEazDCXdNag2XqKTM222qd1GzfDXNJZvWblWVpeZGq21T6t27gZ5pLOGjRbz6Cw76fIYzXYUGEeETdExEJEPBMRt3Ys3xoRGRGnO243jW20ksbquis3s2/nFWyenSGAzbMz7Nt5xdmDlEWmZhvmsQcPL7J9/yFeeePdbN9/yHr6CIbtZnkCuBnYAXT7Gp3NzOdLG5WkFSvaMdJvtp4iU7MNeuwwnTTqLTJz+JUjbgZekZnvaf+8FXgcOH+UMJ+fn8+FhYWRBippsOWBCK29386962m1ff8hFrvUzzfPznD/jW+qYETTJyIeyMz5bveVVTM/FhHfjYjPRsTGHoPY3S7VLBw/frykl5XUqc4dIx4gLaZomD8JvB7YAlwFXADc0W3FzDyQmfOZOT83N1fwZSV1U+dA9ABpMYXCPDNPZ+ZCZj6fmd8DbgDeEhEbyhmepFHUORCLHFxV+a2JSwX4KPl5JQ2hzoE4qJNG/Q3VzRIRa9rrngecFxFrgedplVZOAN8GfgL4c+C+zDw5nuFK6qdIt0lZinTT9OukUX/DtiZ+EPhwx8/vBP4YeAT4U+DlwP8CXwHeXuYAJY2mykC0vbA6Q4V5Zu4F9va4+/NlDUZSvfXrpjHMx8vT+SWVps7dNHVnmEsqTZ27aerOMJdUmjp309SdMw1JKs00dNOsVoa5VIEmT59me2E1DHNpwmzf0zhYM5cmrM4Xw9L0MsylCbN9T+NgmUWasE2zM12v2z1K+16Ta+5aGffMpQkr2r63VHNfPHGG5MWa+7BTrDk1WzMZ5tKEFb06YJGae9EvAk0vyyxSBYq07xWpuXvtlOZyz1yqmSKnzHvwtbkMc6lmitTcvXZKcxnmUs0Uqbl77ZTmsmYuTaFBrYcrrbl77ZTmMsylKTPu0/29dkozWWaRpoyn+2sl3DOXxqDIGZp2nGgl3DOXSlb0xBw7TrQShrm0Av1OiS9aJrHjRCthmUUa0aADlEXLJHacaCUMc2lEg06JL+OqiHacaFSWWaQRDdrztkyiKhjm0ogGHaAselVEaSUss0gj2rNj2zk1c/jRPW/LJJo0w1wakQcoNY0Mc2kF3PPWtLFmLkkNYJhLUgMY5pLUAIa5JDWAYS5JDWCYS1IDGOaS1ACGuSQ1gCcNadUqMhuQNG2G2jOPiBsiYiEinomIW5fdd01EHI2IpyPi3ojYMpaRSiUqOhuQNG2GLbM8AdwMfKZzYURsBO4EbgIuBBaAL5Q5QGkcnDRZTTNUmSUz7wSIiHngFR137QSOZOYX2/fvBZ6MiMsy82jJY9UqM84yiJMmq2mKHgC9HHhw6YfMfAr4Tnv5OSJid7tUs3D8+PGCL6umG3cZxEmT1TRFw3w9cHLZspPABctXzMwDmTmfmfNzc3MFX1ZNN+4yiLMBqWmKdrOcBjYsW7YBOFXwebXKjbsMMsw1ye12UZ0UDfMjwLuXfoiIdcAl7eXSipUxKfIg/a5JvlTmWfrrYKnMs/Q4adoM25q4JiLWAucB50XE2ohYA9wFvCYidrXv/xDwkAc/VVTVZRC7XVQ3w9bMPwicAW4E3tn+9wcz8ziwC/go8APgDcDbxjBOrTJVT4pst4vqZtjWxL3A3h73fRW4rLwhSS1VTs02iTKPVCavzSJ1UXWZRxqV12aRuhim20WaJoa5Gqtoa2GVZR5pVIa5GsnWQq02hrlqq9+ed7/WQsNcTWSYq1IrLYUM2vO2tVCrjd0sqkyRi2kNOqnHC2lptTHMVZkiZ1kO2vO2tVCrjWGuyhQphQza8676DFJp0qyZqzJFzrLcs2PbOTVz+NE9b1sLtZq4Z67KFCmFuOctncs9c1Wm6FmW7nlLLzLMVSkDWSqHZRZJagDDXJIawDCXpAYwzCWpAQxzSWoAw1ySGsAwl6QGMMwlqQEMc0lqAMNckhrAMJekBjDMJakBvNCWxmqlc3xKGo1h3gDTGpiDJl2WVB7DvOaqDsx+XyT95vg0zKVyWTOvuSKTIhe19EWyeOIMyYtfJAcPLwLF5viUNBrDvOaqDMxBXySDJl2WVB7DvOaqDMxBXyRF5viUNBrDvOaqDMxBXyROuixNjgdAa67opMhF7Nmx7ZyDr/CjXyTO8SlNhmHeAFUFZpVfJJLOZZirr0E97O55S9PBMFdPVfewSxqeB0DVU5U97JJGY5irJ0/6keqjlDCPiPsi4ocRcbp9c9etATzpR6qPMvfMb8jM9e2bZ4U0gCf9SPXhAdBVYKVXVbT1UKqPyMziTxJxH3A5EMAjwB9l5n3L1tkN7Aa4+OKLrzp27Fjh19VgyztSoLV37ZmYUv1ExAOZOd/tvrLKLO8HXgVsBg4AX4qISzpXyMwDmTmfmfNzc3MlvawGsSNFWh1KCfPM/FZmnsrMZzLzNuB+4K1lPLeKsSNFWh3G1ZqYtEouqpgdKdLqUDjMI2I2InZExNqIWBMR1wNXA/cUH56KGtSRcvDwItv3H+KVN97N9v2Hzk4sIaleyuhmOR+4GbgMeAE4ClyXmRZlp0C/jhRP15eao5RullHNz8/nwsLCxF9X59q+/xCLXWrnm2dnuP/GN1UwIkn9TKKbRTXkwVGpOQzzVcyDo1JzGOarmKfrS83h6fyrmKfrS81hmK9yzhQkNYNlFklqAPfMa2ClVz2UtHoY5lPOE3skDcMyy5TzqoeShuGe+QQUKZN4Yo+kYbhnPmZLZZLFE2dIXiyTDHtBK0/skTQMw3xIK726YNEyiSf2SBqGZZYhFDkIWbRM4ok9koZhmA+h3971oFDdNDvT9cqEo5RJPLFH0iCWWYZQZO/aMomkSTDMh1DkIOR1V25m384r2Dw7Q9C6Vvi+nVe4py2pVJZZhrBnx7ZzauYw2t61ZRJJ42aYD8GDkJKmXWPCfNzXL3HvWtI0q1WY9wrsYVoHp/liVdM8Nkn1UJsw7xfYg1oHp/liVdM8Nkn1UZtuln6BPah1cJovVjXNY5NUH7UJ836BPah1cJovVjXNY5NUH7UJ836BPejEnGm+WNU0j01SfdQmzPsF9qATc6b5LMxpHpuk+qjNAdBBvd79Wgcn0Se+0o4Ue9gllSEyc+IvOj8/nwsLCxN/3XFZ3pECrb1rT9uXVKaIeCAz57vdV5syyzSzI0VS1QzzEtiRIqlqhnkJ7EiRVDXDvAR2pEiqWm26WaaZHSmSqmaYl8SrKkqqkmUWSWoAw1ySGsAwl6QGMMwlqQFKCfOIuDAi7oqIpyLiWES8o4znlSQNp6xulr8EngUuAl4H3B0RD2bmkZKeX5LUR+E984hYB+wCbsrM05n5deDvgN8u+tySpOGUUWa5FHghMx/tWPYgcHnnShGxOyIWImLh+PHjJbysJGlJGWG+Hji5bNlJ4ILOBZl5IDPnM3N+bm6uhJeVJC0pI8xPAxuWLdsAnCrhuSVJQyjjAOijwJqIeHVmfru97LVArQ5+rnSmIEmaBoX3zDPzKeBO4CMRsS4itgO/Dtxe9LknZWmmoMUTZ0hg8cQZPnDnwxw8vFj10CRpKGWdNPT7wAzw38Dngd+rU1uiMwVJqrtS+swz8/vAdWU8VxWcKUhS3Xk6P84UJKn+DHOcKUhS/Tk5Bc4UJKn+DPM2ZwqSVGeWWSSpAQxzSWoAw1ySGsAwl6QGMMwlqQEiMyf/ohHHgWMFnmIj8GRJwymT4xqN4xqN4xpNE8e1JTO7XkO8kjAvKiIWMnO+6nEs57hG47hG47hGs9rGZZlFkhrAMJekBqhrmB+oegA9OK7ROK7ROK7RrKpx1bJmLkk6V133zCVJHQxzSWoAw1ySGmAqwzwiLoyIuyLiqYg4FhHv6LFeRMTHIuJ/2rePR0SMaUwvjYhPt8dzKiIOR8S1PdZ9T0S8EBGnO26/PI5xtV/vvoj4YcdrdZ28dMLb6/Sy2wsR8Rc91h3r9oqIGyJiISKeiYhbl913TUQcjYinI+LeiNjS53m2ttd5uv2YN49jXBHx8xHxlYj4fkQcj4gvRsRP9Xmeod7/Esa1NSJy2ft0U5/nmdT2un7ZmJ5uj/OqHs9T2vYalAsT/Xxl5tTdaE0K/QVgPfCLwEng8i7rvRd4BHgFsBn4V+B9YxrTOmAvsJXWl+CvAqeArV3WfQ/w9Qlur/uA3x1ivYltry7b7jRwdY/7x7q9gJ205qj9JHBrx/KN7c/WbwFrgVuAb/Z5nn8C/ozW5OW7gBPA3BjGdW17TBuAHwM+A/xD0fe/hHFtBRJYM+TzTGR79fg8fYd2g8c4t1e/XJj052ssvzwlbJxngUs7lt0O7O+y7jeA3R0//06/jTWGsT4E7OrxYZrGMK9kewHvBh7r88s1ke0F3LwsnHYD3+j4eR1wBrisy2MvBZ4BLuhY9jVK+DJcPq4u9/8ccKro+1/C9ho6zCveXvcCH5709up4/ofaYTzRz9c0llkuBV7IzEc7lj0IXN5l3cvb9w1ar3QRcRGtsR7pscqVEfFkRDwaETdFxLhnddrXfr37+5Qoqtpe7wY+l+1PaA+T3l6wbHtk5lO09uh6fdYey8xTHcsmtf2upvfnbMkw739ZjkXEdyPisxGxscc6lWyvdhnjauBzA1Ydy/ZalgsT/XxNY5ivp/WnSaeTwAVDrHsSWD+uOvCSiDgfuAO4LTOPdlnlH4HXAC+n9Q39dmDPGIf0fuBVtEonB4AvRcQlXdab+PaKiIuBNwK39Vlt0ttrSZHPWr91SxMRPwt8iP7bY9j3v6gngdcDW4CraP3f7+ixbiXbC3gX8LXMfLzPOmPZXl1yYaKfr2kM89O0aoWdNtCqQw1adwNwesAeYCER8RJaZZ9ngRu6rZOZj2Xm45n5f5n5MPAR4DfHNabM/FZmnsrMZzLzNuB+4K1dVp349qL1y/X1fr9ck95eHYp81vqtW4qI+Gngy8AfZObXeq03wvtfSGaezsyFzHw+M79H6/P/lohYvl2ggu3V9i767ziMZXv1yIWJfr6mMcwfBdZExKs7lr2W7n9mHmnfN2i9UrT3YD8NXESrVv7ckA9NYKx/LQz5ehPdXm0Df7m6mNT2Omd7RMQ64BJ6f9ZeFRGde0pj237tcsFXgT/JzNtHfPiktt/STkCvz9rEthdARGwHNgF/O+JDC22vPrkw2c/XuA4CFDyA8Ne0OlrWAdvp3c3yPuDfaP25tKn9Hx9bdwbwKeCbwPoB610LXNT+92XAv9DngEzBMc0CO2gdLV8DXA88BWybgu31C+2xXDBgvbFur/Z2WQvso7X3tLSt5tqfrV3tZR+jf7fBN4FPtNf9DYp3Z/Qa12ZatdU9Zb7/JYzrDcA2WjuBL6PVcXZv1dur4/4DtI7NTHp7dc2FSX++SvllKfsGXAgcbG/kfwfe0V7+S7TKAkvrBfBx4Pvt28fp0TFRwpi20PoG/yGtP4mWbtcDF7f/fXF73U8A32uP/zFaZYPzxzSuOeCfaf05dqL9gfiVqrdX+/X+Cri9y/KJbi9arWO57La3fd+bgaO0ugzuo6PVtP1L+qmOn7e21zlDq8XzzeMYF/Dh9r87P2ed7+MfAl8e9P6PYVxvBx5vv0//Sesg409Wvb3a961t//+v6fK4sW0v+uTCpD9fXmhLkhpgGmvmkqQRGeaS1ACGuSQ1gGEuSQ1gmEtSAxjmktQAhrkkNYBhLkkN8P+BPQdCqMhX4wAAAABJRU5ErkJggg==\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"x_lin = torch.linspace(0,20, steps=40)\n",
"y_lin = x_lin + torch.randn_like(x_lin)\n",
"plt.scatter(x_lin, y_lin);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Although we only have a single independent variable, sklearn expects a matrix of independent variables, not a single vector. So we have to turn our vector into a matrix with one column. In other words, we have to change the *shape* from `[40]` to `[40,1]`. One way to do that is with the `unsqueeze` method, which adds a new unit axis to a tensor at the requested dimension:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(torch.Size([40]), torch.Size([40, 1]))"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"xs_lin = x_lin.unsqueeze(1)\n",
"x_lin.shape,xs_lin.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A more flexible approach is to slice an array or tensor with the special value `None`, which introduces an additional unit axis at that location:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"torch.Size([40, 1])"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x_lin[:,None].shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now create a random forest for this data. We'll use only the first 30 rows to train the model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"m_lin = RandomForestRegressor().fit(xs_lin[:30],y_lin[:30])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"...and we test the model on the full dataset. The blue dots are the training data, and the red dots are the predictions."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAD7CAYAAACYLnSTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXDc5Z3n8fe3kazWZaktiUuyfA5mg0t4B6U4RBhIAgMM48zKkMlBEhanyCShZqu2iJ3NQsaVYTbgceWP3WJmlsIJIZtMwFgZMpmBLSjjCkdwrSCmjRnDru1YlrGDpehoHS1L7mf/+LWslqI+pL7bn1dVl1q/qx893frq0ff3HOacQ0REipsv3wUQEZH0KZiLiJQABXMRkRKgYC4iUgIUzEVESkBZPl60sbHRrVy5Mh8vLSJStN58880+51zTfPvyEsxXrlxJd3d3Pl5aRKRomdmxePuUZhERKQEK5iIiJUDBXESkBCiYi4iUAAVzEZESoGAuIiWvf2SCt48P0j8yke+iZE1euiaKiOTKc/tPsHV3kHKfj8lIhO2b2ti4oTnfxcq4pC1zM6sws51mdszMQmb2azO7LWb/J8zskJmNmdnLZrYiu0UWEUlN/8gEW3cHCU9GCE1MEZ6MsGV3sCRb6KmkWcqA48AfAXXAQ8AzZrbSzBqBrui2ZUA38HSWyioisiC9A+OU+2aHuXKfj96B8TyVKHuSplmcc6PAtphNvzCzo8BVQANw0Dm3C8DMtgF9Zna5c+5Q5osrIpK6lkAlk5HIrG2TkQgtgco8lSh7FnwD1MwuAi4DDgJXAG9P74sG/sPR7XPPu8/Mus2s+/Tp04svsYhIihpqKti+qQ1/uY/aijL85T62b2qjoaYi30XLuAXdADWzcuDHwA+dc4fMrAaYG5mHgNq55zrnHgceB2hvb9dadSKSExs3NNOxtpHegXFaApXzBvL+kYmE+4tBysHczHzAj4AzwP3RzSPA0jmHLgVCGSmdiEgGNNRUxA3SpdLbJaU0i5kZsBO4CNjknJuM7joIXBlzXDWwJrpdRKSglVJvl1Rz5n8P/DvgT51zsbeBfwasN7NNZuYHvg0EdfNTRIpBKfV2SaWf+QrgK8AG4JSZjUQfn3fOnQY2AX8DDABXA5/JZoFFRDKllHq7pNI18RhgCfa/BFyeyUKJiOTCdG+XLXNy5sV4E1TD+UXkvJZKb5dioGAuIue9RL1dioVmTRQRKQEK5iIiWZSr6XeVZhERSWKxI0RzOSBJwVxEJIHFBuTYAUlhvO6PW3YH6VjbmJX8vNIsIiJxpDNCNNcDkhTMRUTiSCcg53pAkoK5iEgcqQTkeDc4cz39rnLmIiJxJBshmiyfnssBSQrmIiIJxAvIqd7gzNWAJAVzESkK+VxAYr6APJ1Pnw7kMJNPz8doUgVzESl4hbiARKHNuKgboCJS0Ap1AYlCW19ULXMRKWippjPykYYppBkXFcxFpKClks7IZxqmUGZcVJpFRApasnRGoaZhck0tcxEpeInSGYXWqyRfFMxFpCjES2cUWq+SfFGaRUSKWqH1KskXtcxFpOgVUq+SfFEwF5GSUCi9SvJFaRYRkRKgYC4iUgIUzEVESoCCuYhICVAwFxEpAQrmIiIlQMFcRKQEKJiLiJQABXMRkRKgYC4iUgIUzEVklv6RCd4+PnjezQde7DQ3i4ics+eZFzn42JO0DJ3ml3VNrP/6Pdz06ZvP7U93abZ8LO12vlAwFxEABt/oZvjhR6heUk1P9TJqR0MMPfwIg60B6q9pTxroCQahqwt6eqC1FTo7oa3t3O6E5yc5N639hXztDFKaRUQACD/9LCF/LcP+Gpz5GPbXEPLXEn762ZlAPxqip3oZ1dOB/o1u7+RgEHbsgIEBaGnxvu7Y4W2HxOcnOTet/YV87QxTy1xEAKjvO8lwRSW4mW3DFZXU951kcDrQL6nytvtrvKXann4Wrmn3Wp+BgPeAma9dXdDWNvOHYr7z6/wJz0127YT7p78vxGtnmFrmIgKAf80qblteRZnPWHKBjzKfcdvyKvxrVs0E+hjTgR7w0gh1dbMvWFfnbYfE5yc5N639hXztDEspmJvZ/WbWbWYTZvZkzPaVZubMbCTm8VBWSioi2dXZyeqySTavD9C54VI2rw+wumwSOjsTBnrAywcPDc2+3tCQt53Efyimzx07M8WpoTBjZ6ZmnZvs2gn3p3Nutq+dYam2zD8AHga+H2d/vXOuJvr468wUTUQWY9FdC9va4IEHqLywkYtDfVRe2AgPPOBtTxDoAe/rwID3iERmnsfsj3t+Zyc9h0+w66UD/OzNHna9dIDjh08s6Npx96dzbravnWHmnEt+1PTBZg8DLc65e6LfrwSOAuXOualUr9Pe3u66u7sXVFARSe65/SfYujtIuc/HZCTC9k1tbNzQnJmLZ6lXSP/IBP/xP+/kpndfpXn4NCeWNvHyR67nB9/bPNN9Ub1ZADCzN51z7fPuy1Aw/wDvtsmLwDecc33znHsfcB9Aa2vrVceOHVvQDyEiifWPTNDx6B7Ck5Fz2/zlPl7b+vGC7tP99vFB7n5iH6GJmfZgbUUZ/+vLV3Pl8vo8lqzwJArm6d4A7QM+CqwArgJqgR/Pd6Bz7nHnXLtzrr2pqSnNlxWRuXoHxin3zf6VLvf56B0Yz1OJUtMSqGQyEpm1bTISoSVQGecMmU9awdw5N+Kc63bOTTnnfgvcD9xiZkszUzwRSVWxBsWGmgq2b2rDX+6jtqIMf7mP7ZvaCvq/iUKU6X7m0zkby/B1RSSJ6aC4ZU7OvBiC4sYNzXSsbdRQ/zSkFMzNrCx67AXABWbmB6bwUiuDwP8FAsB/B/Y654biXUtEsmejr59Pnn2d8fePULlmNVW+ZiBDN0BTkM7cKw01FQriaUg1zfIgMA58E7g7+vxBYDXwAhAC3gEmgM9mvpgiklR0+HjV6DAN69ZQNTqc1eHjcz23/wQdj+7h7if20fHoHn6+/0ROXlc8KbXMnXPbgG1xdv9jpgojImnI8fDxWP0jE2zdHSQ8GSGMl7ffsjtIx9pGtbZzRMP5RUpFjoePxyrWnjSlRMFcpFTkePh4rGLtSVNKFMxFSkWOh4/HUvfC/FvQCNBM0XB+kSzJ4WII89FKQtmVaASo5jMXyZOsBL62tsTBO8vBXt0L80dpFpE8yEs3vhyvfCO5pWAukmOx3fhCE1OEJyNs2R1c+JS1CxXbddHnm3k+vWqOFDUFc5Ecy1s3vjx2XZTsU85cJMdaApWsOnn49+bvbgncmPI1FpVvb231UivTg4kgZ10XJfvUMhfJsYYj7/HYB3tYdmaU/vomlp0Z5bEP9tBw5L2Uzt/zzIv85E/u5ciffpqf/Mm9vPzMi6m9cLTr4viHfZwaGGP8w76cdV2U7FPLXCTXurpYvqaZu9pqGR6fYmllGVWjoZSG3Q++0c3ww49QvaSanupl1I6GGHr4EQZbA9RfM2+PtRltbey54wscfOxJWobep7euifVfv4ebcth1UbJHwVwk13p6oKWFKp+PqiXRX8Gy1HLX4aefJeSvZXhJFQDD/hrKfT7CTz8LSYJ5/8gEXzswRfjaz53b5j8wxWu3T6g7YQlQmkUk19IYdl/fd5LhitlD5IcrKqnvO5n0XM2fUtoUzEVyLdmw+2AQtm2De+/1vsb0A/evWcVty6so8xlLLvBR5jNuW16Ff82qpC+r+VNKm4K5SK61tcEDD3i9Snp7va8PPOBtTzawp7OT1WWTbF4foHPDpWxeH2B12WRKNzE1f0pp09wsIoVk2zYYGGCses7N0UDA2wdpD8nX/CnFS3OziBSLnh4OlS3lpV8fxWdGxDluvryJdaGYm6PJ5l9JQvOnlCalWUQKyNjFzezbf5SpiOPM2QhTEccb+48ydnHu1vGU4qSWuUg2BIOM/XQX44ejCyt/5q6UWtPHb7yV+n/5FWcjjlBFFbUTY9RPjnH8xltZl4NiS/FSy1wk04JBer71HXa9FOSZk45dLwU5/q3vpDQ7YeN17ey8upMhfw2XhPoZ8tew8+pOGq9LMiBIzntqmYssUrwbiWM/3cXLH04ysKQaIjCxpJo9H45y1093UZWkdd5QU8GXv/YptuxeRbnPx2Qkoh4nkhIFc5FFeG7/CbbuDs4KuBs3eHnt8cNHGPN7gXzamL+a8cNHqErh2hs3NNOxtlE9TmRBlGYRWaBk85FXrllNVXh01jlV4VEq16xO+TUaaiq4cnm9ArmkTC1zkQXqHRjnIx/+ho8dfOXcFLavXPExegfGaaipoOozd/Hx4L+x58NRxvzVVIVH+fiF5d5NUJEsUTAXWaAVJ/4fX/zVs/xuSTUnaxuoC4/wxV89y4oTH4Xl7dDWxvL/9m3uOtebpS3l3iwii6VgLrJA9S/8grb1K/nX3jDlZoxdUMuNlzVR/8IvZmYubGujqq0tpRy5SCYomIssVE8Pq9e2cO/KyMyQ+zKfll+TvFIwF1mo6PJrVYHAzHzkAwNafk3ySr1ZRBYq2RS2InmgYC6yUImmsBXJE6VZRBYjzZkLRTJNLXMRkRKgYC4iUgIUzEVESoCCuYhICVAwFxEpAQrmcl7rH5ng7eOD52Y8FClWKQVzM7vfzLrNbMLMnpyz7xNmdsjMxszsZTNbkZWSimTYc/tP0PHoHu5+Yh8dj+7h5/tP5LtIIouWasv8A+Bh4PuxG82sEegCHgKWAd3A05ksoEg2JJuTXKTYpDRoyDnXBWBm7UBLzK5O4KBzbld0/zagz8wud84dynBZ5TwVb3m2dPQOjFPu8xGOWQ6o3Oc7Nyc54K3Z2dXlTaDV2uoN19dAISlQ6Y4AvQJ4e/ob59yomR2Obp8VzM3sPuA+gFZNSCQpSrQ8WzpaApWsOnmYm9599dwCEy9/5HpaAjd6BwSDsGOHN1S/pcWbe2XHDg3bl4KV7g3QGmBozrYhoHbugc65x51z7c659qampjRfVs4H2UyFNBx5j8c+2MOyM6P01zex7Mwoj32wh4Yj73kHdHV5gTwQAJ9v5nlXV9qvLZIN6bbMR4Clc7YtBUJpXlcktVTIYnV1sXxNM3e11c7MST4a8oJ1W5uXWmlpYezM1Mz+ujrNWS4FK91gfhD40vQ3ZlYNrIluF0lLS6CSyUhk1rbJSISWQGX6F48G6yqfb2ZO8rKYYN3ayuH3j/N8bxifGRHnuL3Fz+rLlCKUwpRq18QyM/MDFwAXmJnfzMqAnwHrzWxTdP+3gaBufkomNNRUsH1TG/5yH7UVZfjLfWzf1JaZm6CtrTA0J0M4NHRugYnBW+/gwDu/oWosxOTUFFVjIYLv/IbBW+9I/7VFsiDVnPmDwDjwTeDu6PMHnXOngU3A3wADwNXAZ7JQTjlPbfT189bZ19l76CneOvs6G339mblwkgUmjjWv5alr72TIX8MloX6G/DU8de2dHGtem5nXF8kwc87l/EXb29tdd3d3zl9Xikxsj5K6Oq/lPDCQeo+SZF0LE+zvH5mg49E9hCdn0jz+ch+vbf14xrpHiiyUmb3pnGufb58Wp5DCFdujBGa+Tt+kTCSVroUJFpiYTvFsmdMtUoFcCpWCuRSu6E3KWVLtUZLOH4KojRua6VjbmPEBSyLZoGAuhau11WtRTwdimHWTEoifKslQ18KGmgoFcSkKCuZSuDo7vdQIzM6Zb97sbUuUSlHXQjnPaApcKVxtbV5gDgSgt9f7GpvzjqZSxqprORU6w1h17blRmupaKOcbtcwl7xJOpJXgJiU9PRwqW8pLvz56rvV98+VNrAv1nOta+LGDr5ybe+UXV93Gqua11Gf/RxLJOQVzyat0JtIau7iZfS8FmVpSDXhdbN/Yf5Tln2yjJVDJuxeu5K3ATFrFX+7LzOhRkQKkNIvkTboTaR2/8VbqJ0ZZGh7BXISl4RHqJ0Y5fuOt2R09KlKA1DKXvEl3Iq3G69rZeXXnrGls/+kPb+UH13ljKtS1UM4nCuaSN0nnFE+ioaaCL3/tU2zZvSruwB51LZTzhYK55M30nOJ7zkx6c4qHp+cUv14De0QWSMFc8ifZnOIpUutbRMFc8inZnOIikjL1ZpH8STKnuIikTsFc8ifJnOIikjoFc8mfZMP1RSRlyplLfiUari8iKVPLXESkBCiYi4iUAAVzEZESoJy5ZF+yhZVFJG1qmUt2Ta8GNDAwezWgYDDfJRMpKQrmkl2xCyv7fDPPu7ryXTKRkqJgLtnV0+Ot3xlrEQsri0hiCuaSXRqyL5ITCuaSXdEh++Mf9nFqYIzxD/s0ZF8kC9SbpYQkXBg5X9ra2HPHFzj42JO0DL1Pb10T679+DzepN4tIRimYl4h0FkbOhHh/SPpHJvjagSnC137u3Db/gSleu32icP7giJQABfMSELsw8vR6mlt2B+lY25iTgJnoD0m663yKSGoUzEtAPgNm/8gEO//uOb4Ss47nE7+9no7vbaahpoKWQCWTkciscyYjEVoClVktl8j5RjdAS0A+A2bf691s3tdFXXiEk7UN1IVH2Lyvi77XuwFvSbftm9rwl/uorSjDX+77vUWXRSR9apmXgOmAuWVOqiMXAXP53hd4o6Ka4SXVAAz7a7jAZyzf+wLc0gFo0WWRXFAwLxH5CphVp05wzYZVvHjoND4zIs5xzYZVVJ06Mes4Lboskl0K5qUiGKShq4uGbExmlWiirNZW1g0MsPz6VQyPT7G0soyq0RAENChIJJeUMy8F2ZzMKnrt8Q/7OFXb6A36ib12dFBQ1WiIi2uXeIFcg4JEck7BvBRkczKrri4OT5Wz850BuvZ/wM53BjgyVT5zba3jKVIQlGYpBT09Xos8VoYmswofPsoLx88w5QxwADx/fIzNFxzFP32Q1vEUyTsF81LQ2uqlNgKBmW1zJ7Na5AIRg42XsPS9Q/QvqTq3benEOIONK7g4kz+DiKQlI2kWM9trZmEzG4k+3svEdSVF0bw1AwMQicw8n85bp5FT9//5ndSGQywNj2AuwtLwCLXhEP4/vzPLP5SILEQmc+b3O+dqoo91GbyuJJMsb51GTr3+mnbqHvwmo9W1tI7+jtHqWuoe/Cb117Rn+YcSkYVQmqVUJMpbR3PqY2emZroPLiCnftOnb6bt9hvoHRjnBg36ESlImQzm3zWzR4D3gP/qnNsbu9PM7gPuA2jVwgS51drK4feP83xv+NzAnttb/Ky+LPX3QYN+RApbptIsW4HVQDPwOPDPZrYm9gDn3OPOuXbnXHtTU1OGXlZSMXjrHRx45zdUjYWYnJqiaixE8J3fMHjrHTMHBYOwbRvce6/3VQsuixSVjARz59w+51zIOTfhnPsh8BpweyauLek71ryWp669kyF/DZeE+hny1/DUtXdyrHmtd0A2Bx2JSE5kK2fuAMvStWWBWgKVvHvhSt6KGWLvL/fNzKoYe4MUZr52dan/uEiRSLtlbmb1ZvbHZuY3szIz+zxwA/C/0y+eZELSaWh7eqCujrEzU5waCjN2Zipjg45EJDcy0TIvBx4GLgfOAoeAP3POqa95AUk4q2IGbpCKSH6lHcydc6eBj2agLJJl8XqkDN56Bwe6tlK1pJpQRRW1E2ME3znFsr/8KvV5KKeILJwm2pLkN0hFpOBp0JAkv0EqIgVPLXPROp0iJUAtcwG0TqdIsVMwl3M0ZF+keCnNIiJSAhTMi0z/yARvHx+kf2Qi30URkQKiNEsReW7/CbbuDlLu8zEZibB9UxsbNzTnu1giUgAUzItE/8gEO//uOb7y7qs0D5/mxNImnvjt9XR8b7Py3CKiNEuuLTZN0vd6N5v3dVEXHuFkbQN14RE27+ui7/XuLJVURIqJWuY5lE6aZPneF3ijoprhJdUADPtruMBnLN/7AtzSkc1ii0gRUMt8ERbTuu4fmWDr7iDhyQihiSnCkxG27A6mfI2qUye4ZsMqynzGkgt8lPmMazasourUicX+GCJSQtQyX6DFtq57B8Yp9/kIEzm3rdzno3dgPLWcd2sr6wYGWH79qpl1PEdDENDMhiKiYL4gsa3r6aC8ZXeQjrWNSQNyS6CSVScPc1PMDcyXP3I9LYEbU3vxzk7YsYMq8BZjHhryVgTavDm9H0pESoLSLAsw3bqONd26TqbhyHs89sEelp0Zpb++iWVnRnnsgz00HElx2ve2NnjgAW8VoN5e7+sDD2glIBEB1DJfkJZAJZORyKxtk5FIarMLdnWxfE0zd7XVzk6TLGRptrY2BW8RmZeC+QJMzy64ZU7OPKWcd08PtLRQ5fNRtSRa7WVzlmYLBr3g3tMDra1eakXBW0RSoDTLAm309fPW2dfZe+gp3jr7Oht9/amd2Nrq5bljDQ1528EL5Dt2eHnwlhbv644d3nYRkSRKNphnZQ6TaMCtGh2mYd0aqkaHUw+4nZ1egB4YgEhk5nlnp7e/q8vLgwcC4PPNPO/qylz5RaRkFW0wTxSsn9t/go5H93D3E/voeHQPP9//+32xFxXs0wm4yW5g9vRAXR1jZ6Y4NRRm7MwU1M1Jw4iIxFGUOfM9z7zIwceepGXoNL+sa2L91+/hpk/fDKQ2h8miR2JG896zLCTgJrqB2drK4feP83xvGJ8ZEee4vcXP6svUj1xEkiu6YD74RjfDDz9C9ZJqeqqXUTsaYujhRxhsDVB/Tfu5OUz6y6vmzGFyJQ23dKQ3YVVrq5caCQRmtsXmvWHRNzEHb72DA11bqVpSTaiiitqJMYLvnGLZX36V+kXWlYicP4ouzRJ++llC/lqG/TU48zHsryHkryX89LOAN4fJYEX1rP2DFdXeHCakOWFVsrx3GjcxjzWv5alr72TIX8MloX6G/DU8de2dHGteu+i6EpHzR9G1zOv7TjJcUQluZttwRSX1fSeBmTlMXjx0+ly6InYOk7QmrJrOe8e2vDdvnml5x+bUYeZrCn3JWwKVvHvhSt6KGZ7vL/el1oddRM57RRfM/WtWcdvZ4/xrTG75tpZK/GuWewckmcMkWbBPKlHeO42celp92EXkvFd0wZzOTlbv2MHm9QGGyquomxyjcmR4JtWRbA6TbE5YFc2pj1XPGeXZmtq1N25opmNtI70D47QEKhXIRSRlRZczn051VF7YyMWhPiovbJzdxS9ZF8Bo3rtqNMTFtUu8YBub905HZyc9h0+w66UD/OzNHna9dIDjh08s6NoNNRVcubxegVxEFqT4WuaQfI6SRPuT5b3T0L96Hfdf+nFuGnyV5kGvp8zXL72eH6xeR0PaVxcRia84g3m6sjRhVe/AOEcvWUNw2Ypz22orylKfs1xEZJGKL81SwNKaVVFEJA3nZ8s8kTRmLlSPFBHJF3POJT8qw9rb2113dwGuKj896CcQ8LoUTveEWeAiEP0jE+qRIiIZZ2ZvOufa59unlnmsNAb9xGqoqVAQF5GcUs48VnTmwlk0c6GIFAEF81jJFpAQESlQCuaxkk2kJSJSoBTMYyUbPSoiUqAycgPUzJYBO4FbgD7gvzjnfpKJa+dclgYUiYhkU6Z6szwGnAEuAjYA/2JmbzvnDmbo+iIikkDaaRYzqwY2AQ8550acc68CPwe+kO61RUQkNZnImV8GnHXOvR+z7W3gitiDzOw+M+s2s+7Tp09n4GVFRGRaJoJ5DTCnPx9DQG3sBufc4865dudce1NTUwZeVkREpmUimI8AS+dsWwqEMnBtERFJQSaC+ftAmZn9Qcy2K4GivfnZPzLB28cH6R+ZyHdRRERSknZvFufcqJl1Ad8xsy/j9Wb5FHBdutfOh+f2n2DrnFkPN25oznexREQSytSgoa8BlcCHwD8CXy3Gbon9IxNs3R0kPBkhNDFFeDLClt1BtdBFpOBlpJ+5c+53wJ9l4lr51DswTrnPR5iZBSbKfT6tFCQiBU/D+WNopSARKVYK5jGmVwryl/uorSjDX+7TSkEiUhS0OMUcGzc007G2USsFiUhRUTCfh1YKEpFiozSLiEgJUDAXESkBCuYiIiVAwVxEpAQomIuIlABzzuX+Rc1OA8cycKlGvGXqCk0hlktlSl0hlktlSl0hlitTZVrhnJt3DvG8BPNMMbNu51x7vssxVyGWS2VKXSGWS2VKXSGWKxdlUppFRKQEKJiLiJSAYg/mj+e7AHEUYrlUptQVYrlUptQVYrmyXqaizpmLiIin2FvmIiKCgrmISElQMBcRKQEFH8zNbJmZ/czMRs3smJl9Ls5xZmaPmll/9LHdzCwL5akws53RsoTM7NdmdlucY+8xs7NmNhLzuDHTZYp5vb1mFo55rffiHJeruhqZ8zhrZv8jzrFZqyszu9/Mus1swsyenLPvE2Z2yMzGzOxlM1uR4Doro8eMRc/5ZKbLZGbXmNmLZvY7MzttZrvM7JIE10npPU+zTCvNzM15bx5KcJ2M1VOScn1+TpnGouW8Ks51MllXCeNAXj5XzrmCfuAtEP00UANcDwwBV8xz3FeA94AWoBl4F/iLLJSnGtgGrMT7Y3gHEAJWznPsPcCrOayrvcCXUzguJ3U1T72NADfE2Z+1ugI68dao/XvgyZjtjdHP012AH/hb4I0E1/kV8D28xcs3AYNAU4bLdFu0PEuBKuD7wAvpvudplmkl4ICyFK+TsXpKVK44n6HDRDt2ZLmu4saBfH2uMv6Lk8lHtMLOAJfFbPsR8Mg8x74O3Bfz/eZEFZjhcgaBTXE+XIUYzHNeV8CXgCMJftGyXlfAw3OC1H3A63M+b+PA5fOcexkwAdTGbHuFNP8Izi3TPPv/EAil+56nWU8pB/Ns1VOKdfUy8Fe5rKs51w9Gg3FePleFnma5DDjrnHs/ZtvbwBXzHHtFdF+y4zLKzC7CK+fBOIf8ezPrM7P3zewhM8v26k7fjb7eawnSFPmoqy8BT7nopzWOXNfVrHpwzo3itezifb6OOOdCMdtyUW83EP+zNS2V9zwTjplZr5n9wMwa4xyTl3qKpjFuAJ5KcmhW6mpOHMjL56rQg3kN3r8rsYaA2hSOHQJqspELnmZm5cCPgR865w7Nc8gvgfXAhXh/sT8LfCNb5QG2AqvxUiePA/9sZmvmOS6ndWVmrcAfAT9McFiu6wrS+3wlOjYjzKwN+DaJ6yHV9zwdfcBHgRXAVR+v/SAAAAK4SURBVHg/84/jHJvzeor6IvCKc+5ogmOyUlfzxIG8fK4KPZiP4OUOYy3Fy00lO3YpMJKkJbhoZubDS/mcAe6f7xjn3BHn3FHnXMQ5dwD4DnBnNsoTfb19zrmQc27COfdD4DXg9nkOzWld4f2ivZroFy3XdRWVzucr0bFpM7O1wPPAf3LOvRLvuAW854vmnBtxznU756acc7/F+7zfYmZz6wNyXE8xvkjixkJW6ipOHMjL56rQg/n7QJmZ/UHMtiuZ/9/Og9F9yY5LW7QFuxO4CC9XPpniqQ7I2n8KC3i9nNVVVNJftHnkoq5m1YOZVQNriP/5Wm1msS2mrNRbNGXwEvDXzrkfLfD0XNTb9B/9eJ+tnNTTNDPrAC4Fnl3gqWnVVYI4kJ/PVbZuBmTwpsJP8Xq0VAMdxO/N8hfAv+H9C3VptDKy0kMD+AfgDaAmyXG3ARdFn18OvEOCGzRplqke+GO8u+dlwOeBUWBdnuvqumg5apMcl7W6itaHH/guXitquo6aop+nTdFtj5K418EbwI7osf+B9HqzxCtTM15+9RuZfM/TLNPVwDq8xl8DXu+yl3NRT4nKFbP/cbz7MTmrq+g1540D+fpcpf2Lku0HsAz4p2jF9wCfi27/GF5qYPo4A7YDv4s+thOn50Sa5VmB9xc9jPcv0vTj80Br9Hlr9NgdwG+jZT+Clzooz1I9NQH/B+/fs8HoB+TmfNZV9LX+J/CjebbnrK7wupC5OY9t0X2fBA7h9TbYS0wX0+gv6z/EfL8yesw4XtfOT2a6TMBfRZ/HfrZi37tvAc8ne88zXKbPAkej781JvJuMF+einlJ4//zRn/0T85yXzbqKGwfy9bnSRFsiIiWg0HPmIiKSAgVzEZESoGAuIlICFMxFREqAgrmISAlQMBcRKQEK5iIiJUDBXESkBPx/7haD9GritKIAAAAASUVORK5CYII=\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.scatter(x_lin, y_lin, 20)\n",
"plt.scatter(x_lin, m_lin.predict(xs_lin), color='red', alpha=0.5);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have a big problem! Our predictions outside of the domain that our training data covered are all too low. Have a think about why this is…\n",
"\n",
"Remember, a random forest is just the average of the predictions of a number of trees. And a tree simply predicts the average value of the rows in a leaf. Therefore, a tree and a random forest can never predict values outside of the range of the training data. This is particularly problematic for data where there is a trend over time, such as inflation, and you wish to make predictions for a future time.. Your predictions will be systematically to low.\n",
"\n",
"But the problem is actually more general than just time variables. Random forests are not able to extrapolate outside of the types of data you have seen, in a more general sense. It can only ever average previously seen observations."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Finding out of domain data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Sometimes it is hard to even know whether your test set is distributed in the same way as your training data or, if it is different, then what columns reflect that difference. There's actually a nice easy way to figure this out, which is to use a random forest!\n",
"\n",
"But in this case we don't use a random forest to predict our actual dependent variable. Instead we try to predict whether a row is in the validation set, or the training set. To see this in action, let's combine our training and validation sets together, create a dependent variable which represents which dataset each row comes from, build a random forest using that data, and get its feature importance:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" cols imp\n",
"5 saleElapsed 0.859446\n",
"9 SalesID 0.119325\n",
"13 MachineID 0.014259\n",
"0 YearMade 0.001793\n",
"8 fiModelDesc 0.001740\n",
"11 Enclosure 0.000657"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_dom = pd.concat([xs_final, valid_xs_final])\n",
"is_valid = np.array([0]*len(xs_final) + [1]*len(valid_xs_final))\n",
"\n",
"m = rf(df_dom, is_valid)\n",
"rf_feat_importance(m, df_dom)[:6]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This shows that there are three columns that are very different between training and validation set: `saleElapsed`, `SalesID`, and `MachineID`. `saleElapsed` is fairly obvious, since it's the number of days between the start of the dataset and each row, so it directly encodes the date. `SalesID` suggests that identifiers for auction sales might increment over time. `MachineID` suggests something similar might be happening for individual items sold in those auctions.\n",
"\n",
"We'll try training the original RF model, removing each of these in turn, and also checking the baseline model RMSE:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"orig 0.232795\n",
"SalesID 0.23109\n",
"saleElapsed 0.236221\n",
"MachineID 0.233492\n"
]
}
],
"source": [
"m = rf(xs_final, y)\n",
"print('orig', m_rmse(m, valid_xs_final, valid_y))\n",
"\n",
"for c in ('SalesID','saleElapsed','MachineID'):\n",
" m = rf(xs_final.drop(c,axis=1), y)\n",
" print(c, m_rmse(m, valid_xs_final.drop(c,axis=1), valid_y))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It looks like we should be able to remove `SalesID` and `MachineID` without losing any accuracy; let's check:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.231307"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"time_vars = ['SalesID','MachineID']\n",
"xs_final_time = xs_final.drop(time_vars, axis=1)\n",
"valid_xs_time = valid_xs_final.drop(time_vars, axis=1)\n",
"\n",
"m = rf(xs_final_time, y)\n",
"m_rmse(m, valid_xs_time, valid_y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Removing these variables has slightly improved the model's accuracy; but more importantly, it should make it more resilient over time, and easier to maintain and understand. We recommend that for all datasets you try building a model where your dependent variable is `is_valid`, like the above. It can often uncover subtle *domain shift* issues that you may otherwise miss.\n",
"\n",
"One thing that might help in our case is to simply avoid using old data. Often, old data shows relationships that just aren't valid any more. Let's try just using the most recent few years of the data:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAD7CAYAAACfQGjDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAZ/klEQVR4nO3df5Dc9X3f8ecLhCVV0hVk2ZuSaXQ1kaP0EJLH52EaD+EoxMZ2XKso9cgoxjIxZ8NgN+ESyrQIVDAFl5Bpm2DXx4AFNjiEqQA7eOQMCYtNHbcRIRK5IhMrlhIcBAKfhVZCwoff/eP72ZmvNvrqvne7+92VeD1mvsN+v+/v+/vjfcu+9f2x+1VEYGZmdjQn9XoDzMysf7lJmJlZITcJMzMr5CZhZmaF3CTMzKzQnF5vQKctWbIkBgcHK1vfgQMHWLBgQWXr62euRcZ1yLgOmeOlDk8++eRLEfGW1uknXJMYHBxk69atla2vXq8zMjJS2fr6mWuRcR0yrkPmeKmDpN1Hm+7TTWZmVshNwszMCrlJmJlZITcJMzMr5CZhZmaF3CTMzKyQm4SZmRVykzAzs0JuEmZmVuiE+8a1mRnA4DWP9GS9u275QE/W2y0+kjAzs0JuEmZmVshNwszMCrlJmJlZITcJMzMrVKpJSBqU9A1Jk5L2SPoDSXNSbJWkJyUdTP9dlcuTpM9JejkN/1WScvFZ55qZWfeVPZL4PPAi8M+AVcC5wBWS3gQ8DHwFOA24G3g4TQcYBVYDK4GzgF8FPgnQTq6ZmVWjbJP4F8AfRcShiNgDbAGGgBGy71r8t4g4HBH/AxDwr1Pex4DbIuK5iPghcBuwPsXayTUzswqU/TLdfwfWSqqT/av/fcAGskaxPSIiN+/2NL3ZSLblYtvSNNrMPYKkUbIjD2q1GvV6veRuta/RaFS6vn7mWmRch0yv6zC2Yqon623d517XoV1lm8TjwGXAK8DJZKeGHgKuBfa1zLsPWJReL2yJ7wMWpmsLrbHSuS2NhYgYB8YBhoeHo8rnyR4vz6+tgmuRcR0yva7D+l5943rdyBHjva5Du6Y93STpJOCbwGZgAbCE7Gjic0ADGGhJGQD2p9et8QGgkT7k28k1M7MKlLkmsRj458AfpGsHLwNfAt4PTABntdx1dFaaTvrvylxsZUtstrlmZlaBaZtERLwE/AC4XNIcSaeSXVTeBtSB14HPSJor6cqU9mfpv/cAV0n6WUmnA2PAphRrJ9fMzCpQ9u6mi4ALgb3A94Ep4Lci4jWy21QvAX4MXAqsTtMBvgh8HXga+GvgkTSNdnLNzKwapS5cR8Rfkd2yerTYU8A7C2IBXJ2GjuaamVn3+Wc5zMyskJuEmZkVcpMwM7NCbhJmZlbITcLMzAq5SZiZWSE3CTMzK+QmYWZmhdwkzMyskJuEmZkVcpMwM7NCbhJmZlao7JPpzMyshMGWJ+KNrZiq7Cl5u275QMeX6SMJMzMrVObxpY2W4XVJv5+Lny9ph6SDkh6TtDQXmyvpLkmvSNoj6aqWZc8618zMuq/Mk+kWNgegBrwKPAAgaQnZs683kD3mdCtwfy59I7AMWAqcB1wt6cJ2c83MrBozPd30a8CLwLfT+EXAREQ8EBGHyD7YV0panuKXADdGxGREPAPcAazvQK6ZmVVgpheuPwbck54aBzBE9qxrACLigKSdwJCkF4DT8/H0enUHco8gaRQYBajVatTr9Rnu1uw1Go1K19fPXIuM65DpdR3GVkz1bN15tfnVbUs36l26SUj6OeBc4DdykxeSPfc6bx+wKMWa462xdnOPEBHjwDjA8PBwjIyMHHtnOqher1Pl+vqZa5FxHTK9rkNVdxRNZ2zFFLc9Xc2NpLvWjXR8mTM53XQJ8ERE/CA3rQEMtMw3AOxPMVrizVi7uWZmVoGZNom7W6ZNACubI5IWAGeQXWuYBJ7Px9PriQ7kmplZBUo1CUm/BPws6a6mnAeBMyWtkTQPuA7YHhE7Uvwe4FpJp6UL0pcBmzqQa2ZmFSh7JPExYHNEHHG6JyL2AmuAm4BJ4GxgbW6W64GdwG7gceDWiNjSbq6ZmVWj1NWUiPjkMWKPAssLYoeBS9PQ0VwzM+s+/yyHmZkVcpMwM7NCbhJmZlbITcLMzAq5SZiZWSE3CTMzK+QmYWZmhdwkzMyskJuEmZkVcpMwM7NCbhJmZlbITcLMzAq5SZiZWSE3CTMzK+QmYWZmhUo3CUlrJT0j6YCknZLOSdPPl7RD0kFJj0lamsuZK+kuSa9I2iPpqpZlzjrXzMy6r+zjS38F+BzwcWAR8MvA30paAmwGNgCLga3A/bnUjcAyYClwHnC1pAvTMmeda2Zm1Sh7JPGfgRsi4rsR8dOI+GFE/BC4CJiIiAci4hDZB/vK9ExqgEuAGyNiMiKeAe4A1qdYO7lmZlaBaR9fKulkYBj4mqTvA/OAh4DfAYaAbc15I+KApJ3AkKQXgNPz8fR6dXrdTm7rNo4CowC1Wo16vT7dbnVMo9GodH39zLXIuA6ZXtdhbMVUz9adV5tf3bZ0o95lnnFdA04Bfg04B/gJ8DBwLbAQ2Nsy/z6yU1ILc+OtMdrMPUJEjAPjAMPDwzEyMjL9XnVIvV6nyvX1M9ci4zpkel2H9dc80rN1542tmOK2p8t81LZv17qRji+zzOmmV9N/fz8ino+Il4DfA94PNICBlvkHgP0pRku8GaPNXDMzq8C0TSIiJoHngDhKeAJY2RyRtAA4g+xawyTwfD6eXk90INfMzCpQ9sL1l4BPS3qrpNOA3wT+GHgQOFPSGknzgOuA7RGxI+XdA1wr6bR0QfoyYFOKtZNrZmYVKNskbgT+AngWeAZ4CrgpIvYCa4CbgEngbGBtLu96YCewG3gcuDUitgC0k2tmZtUodTUlIn4CXJGG1tijwPJ/lJTFDgOXpuFo8VnnmplZ9/lnOczMrJCbhJmZFXKTMDOzQm4SZmZWqJqvAZpZTw324NvHYyumGKl8rdZpPpIwM7NCbhJmZlbITcLMzAr5moSZdU0vroVYZ/lIwszMCrlJmJlZITcJMzMr5CZhZmaF3CTMzKyQm4SZmRUq1SQk1SUdktRIw/dysYsl7ZZ0QNJDkhbnYoslPZhiuyVd3LLcWeeamVn3zeRI4sqIWJiGXwCQNAR8EfgoUAMOAp/P5dwOvJZi64AvpJy2cs3MrBrtfpluHfD1iPgWgKQNwDOSFgE/JXs86ZkR0QCekPQ1sqZwTZu5ZmZWgZk0iZsl3QJ8D/hPEVEHhoDvNGeIiJ2SXgPeTvZB/3pEPJtbxjbg3PS6ndwjSBoFRgFqtRr1en0Gu9WeRqNR6fr6mWuR6cc6jK2Yqnydtfm9WW+/qbIO3XjflW0S/wH4f2Snf9YCX5e0ClgI7GuZdx+wCHj9GDHazD1CRIwD4wDDw8MxMjJSZp86ol6vU+X6+plrkenHOqzv0U+F3/a0f/mnyjrsWjfS8WWW2vKI+D+50bslfQR4P9AABlpmHwD2kx0NFMVoM9fMzCow21tgAxAwAaxsTpT0NmAu8Gwa5khalstbmXJoM9fMzCowbZOQdKqk90qaJ2mOpHXALwPfBO4FPijpHEkLgBuAzRGxPyIOAJuBGyQtkPRu4EPAl9Oi28k1M7MKlDmSOAX4LLAXeAn4NLA6Ir4XERPAp8g+8F8ku2ZwRS73CmB+in0VuDzl0E6umZlVY9prEhGxF3jXMeL3AfcVxH4ErO5GrpmZdZ9/lsPMzAq5SZiZWSE3CTMzK+QmYWZmhdwkzMyskJuEmZkVcpMwM7NCbhJmZlbITcLMzAq5SZiZWSE3CTMzK+QmYWZmhdwkzMyskJuEmZkVcpMwM7NCM2oSkpZJOiTpK7lpF0vaLemApIckLc7FFkt6MMV2S7q4ZXmzzjUzs+6b6ZHE7cBfNEckDQFfBD4K1ICDwOdb5n8txdYBX0g5beWamVk1pn0yXZOktcCPge8AP58mrwO+HhHfSvNsAJ6RtAj4KbAGODMiGsATkr5G1hSuaTPXzMwqUKpJSBoAbgDOB34jFxoiaxoARMROSa8Bbyf7oH89Ip7Nzb8NOLcDua3bNwqMAtRqNer1epnd6ohGo1Hp+vqZa5HpxzqMrZiqfJ21+b1Zb7+psg7deN+VPZK4EbgzIv5eUn76QmBfy7z7gEXA68eItZt7hIgYB8YBhoeHY2Rk5Nh700H1ep0q19fPXItMP9Zh/TWPVL7OsRVT3PZ06ZMVJ6wq67Br3UjHlzntlktaBVwAvOMo4QYw0DJtANhPdjRQFGs318zMKlCmvY0Ag8DfpaOIhcDJkv4lsAVY2ZxR0tuAucCzZB/0cyQti4i/SbOsBCbS64k2cs3MrAJlmsQ48Ie58d8maxqXA28F/lzSOcBfkl232BwR+wEkbQZukPQJYBXwIeCX0nLubSPXzMwqMO0tsBFxMCL2NAey00SHImJvREwAnyL7wH+R7JrBFbn0K4D5KfZV4PKUQzu5ZmZWjRlfTYmIjS3j9wH3Fcz7I2D1MZY161wzM+s+/yyHmZkVcpMwM7NCbhJmZlbITcLMzAq5SZiZWSE3CTMzK+QfVjGryGAPfj/JrF0+kjAzs0JuEmZmVshNwszMCrlJmJlZITcJMzMr5CZhZmaFfAus9UQvbwfddcsHerZus+NNqSMJSV+R9LykVyQ9mx4E1IydL2mHpIOSHpO0NBebK+mulLdH0lUty511rpmZdV/Z0003A4MRMQD8G+Czkt4paQmwGdgALAa2Avfn8jYCy4ClwHnA1ZIuBGgn18zMqlGqSUTEREQcbo6m4QzgImAiIh6IiENkH+wrJS1P814C3BgRkxHxDHAHsD7F2sk1M7MKlL4mIenzZB/S84GngG8ANwHbmvNExAFJO4EhSS8Ap+fj6XXzaXNDbeSazVq3r4eMrZhivX+Cw04QpZtERFwh6dPAvwJGgMPAQmBvy6z7yJ5XvTA33hqjzdwjSBoFRgFqtRr1er3MLnVEo9GodH39bCa1GFsx1d2N6aHa/BN7/8pyHTJV1qEbn0UzurspIl4HnpD068DlQAMYaJltANifYs3xQy0x2sxt3a5xYBxgeHg4RkZGZrJbbanX61S5vn42k1qcyP/SHlsxxW1P+8ZB1yFTZR12rRvp+DJn+z2JOWTXJCaAlc2JkhY0p0fEJPB8Pp5eT6TX7eSamVkFpm0Skt4qaa2khZJOlvRe4CPAnwEPAmdKWiNpHnAdsD0idqT0e4BrJZ2WLkhfBmxKsXZyzcysAmWOJILs1NJzwCTwu8BvRsTDEbEXWEN2AXsSOBtYm8u9HtgJ7AYeB26NiC0A7eSamVk1pj1Rlj7Mzz1G/FFgeUHsMHBpGjqaa2Zm3effbjIzs0JuEmZmVshNwszMCrlJmJlZITcJMzMr5CZhZmaF3CTMzKyQm4SZmRVykzAzs0JuEmZmVshNwszMCrlJmJlZITcJMzMr5CZhZmaF3CTMzKxQmSfTzZV0p6TdkvZLekrS+3Lx8yXtkHRQ0mOSlrbk3iXpFUl7JF3VsuxZ55qZWfeVOZKYA/w92YOH/imwAfgjSYOSlgCb07TFwFbg/lzuRmAZsBQ4D7ha0oUA7eSamVk1yjyZ7gDZB3bTH0v6AfBO4M3AREQ8ACBpI/CSpOXpWdWXAB+PiElgUtIdwHpgC3BRG7lmZlaBaZtEK0k14O3ABNmzr7c1YxFxQNJOYEjSC8Dp+Xh6vTq9Hmojt3WbRoFRgFqtRr1en+luzVqj0ah0ff1sJrUYWzHV3Y3podr8E3v/ynIdMlXWoRufRTNqEpJOAe4F7o6IHZIWAntbZtsHLAIW5sZbY6T4bHOPEBHjwDjA8PBwjIyMlNyj9tXrdapcXz+bSS3WX/NIdzemh8ZWTHHb0zP+99cJx3XIVFmHXetGOr7M0nc3SToJ+DLwGnBlmtwABlpmHQD2pxgt8Was3VwzM6tAqSYhScCdQA1YExE/SaEJYGVuvgXAGWTXGiaB5/Px9HqiA7lmZlaBskcSXwB+EfhgRLyam/4gcKakNZLmAdcB29OFZ4B7gGslnSZpOXAZsKkDuWZmVoEy35NYCnwSWAXskdRIw7qI2AusAW4CJoGzgbW59OuBncBu4HHg1ojYAtBOrpmZVaPMLbC7AR0j/iiwvCB2GLg0DR3NNTOz7vPPcpiZWSE3CTMzK+SbmN/gBjv4fYWxFVMn9PcfzN6IfCRhZmaF3CTMzKyQm4SZmRVykzAzs0K+cJ0zm4u4nbpYu+uWD7S9DDOzTvORhJmZFXKTMDOzQm4SZmZWyE3CzMwKuUmYmVkhNwkzMyvkW2D7RCd/Q8nMrFPKPr70SklbJR2WtKkldr6kHZIOSnosPaSoGZsr6S5Jr0jaI+mqTuWamVn3lT3d9A/AZ4G78hMlLQE2AxuAxcBW4P7cLBuBZcBS4DzgakkXtptrZmbVKNUkImJzRDwEvNwSugiYiIgHIuIQ2Qf7yvRMaoBLgBsjYjIingHuANZ3INfMzCrQ7jWJIWBbcyQiDkjaCQxJegE4PR9Pr1d3IPcIkkaBUYBarUa9Xp/VzoytmJpxTm3+7PJORK5FxnXIuA6ZKusw28++Y2m3SSwE9rZM2wcsSrHmeGus3dwjRMQ4MA4wPDwcIyMjpXcgbza/wTS2Yorbnvb1f3AtmlyHjOuQqbIOu9aNdHyZ7d4C2wAGWqYNAPtTjJZ4M9ZurpmZVaDdJjEBrGyOSFoAnEF2rWESeD4fT68nOpBrZmYVKHsL7BxJ84CTgZMlzZM0B3gQOFPSmhS/DtgeETtS6j3AtZJOSxekLwM2pVg7uWZmVoGyRxLXAq8C1wC/nl5fGxF7gTXATcAkcDawNpd3PbAT2A08DtwaEVsA2sk1M7NqlLqaEhEbyW5RPVrsUWB5QewwcGkaOpprZmbd599uMjOzQm4SZmZWyE3CzMwKuUmYmVkhNwkzMyvkJmFmZoXcJMzMrJCbhJmZFXKTMDOzQm4SZmZWyE3CzMwKuUmYmVkhNwkzMyvkJmFmZoXcJMzMrFBfNwlJiyU9KOmApN2SLu71NpmZvZGUeuhQD90OvAbUgFXAI5K2RYSfdW1mVoG+PZKQtIDs8aYbIqIREU8AXwM+2tstMzN741BE9HobjkrSO4DvRMT83LTfBs6NiA+2zDsKjKbRXwC+V9mGwhLgpQrX189ci4zrkHEdMsdLHZZGxFtaJ/bz6aaFwL6WafuARa0zRsQ4MF7FRrWStDUihnux7n7jWmRch4zrkDne69C3p5uABjDQMm0A2N+DbTEze0Pq5ybxLDBH0rLctJWAL1qbmVWkb5tERBwANgM3SFog6d3Ah4Av93bL/pGenObqU65FxnXIuA6Z47oOfXvhGrLvSQB3Ab8CvAxcExH39XarzMzeOPq6SZiZWW/17ekmMzPrPTcJMzMr5CaRI+lKSVslHZa0qSX2CUnfl9SQtEXS6bnYqZLulvRiGja25A5KekzSQUk7JF1QzR7NThfrsEvSqym3IelPqtmj2ZE0V9Kd6XfD9kt6StL7cvHz09/zYPr7Lm3JvUvSK5L2SLqqZdmFuf2mW3VI/19E7v3QkLSh6v0rq806fFjSd1KsfpRlr5L0ZIo/KWlVRbs1vYjwkAbgImA18AVgU276ucCLwBDwphR/PBf/EvAA8E+AQWAn8PFc/M+B3wPmk/3UyI+Bt/R6f3tQh13ABb3evxnUYQGwMe3LScCvkn1PZ5DsW7T7gH8HzANuBb6by70Z+DZwGvCLwB7gwhQ7Zm6/DV2swyAQwJxe72MFdbgA+DBwHVBvWe6bgN3AbwFzgc+k8Tf1ep8jwk2i4M3w2ZYPx98Fbs+Nn57e3Gek8ZeAd+Xi/xH4dnr9duAwsCgX/zbwqV7vZ5V1SOO7OI6aREFNtpM1+lGyn41pTl8AvAosT+M/BN6Ti98I/GF6fczc42HoUB2OqybRTh1y0z9xlCbxnlQn5ab9HamZ9nrw6aZylIb8OMCZR5nWfN2MDQF/GxH5b4pvS9OPN+3UoeleSXsl/YmklV3Yxq6RVCNr+hNkf79tzVhk3+vZCQxJOo2sgW7Lpef/5oW53dz+TulgHZp2S3pO0pckLenqxndQ2TqUWNQQsD1Sd0i2l8ztOjeJcr4BfFjSWZLmkx0yBtlpFYAtwDWSFkn6eeDSXKz0b1AdB9qpA8A6sn89LgUeA74p6dSqNr4dkk4B7gXujogdHPvvujA33hpjmty+1uE6vAS8i+z98M40/d7ubHlnzbAO0+nr94ObRAkR8afA9cD/IjtXuIvsXORzaZbPkB1a/g3wMPDVXOyE+Q2qNutARPzviHg1Ig5GxM1k12bOqWwHZknSSWTf9H8NuDJNPtbftZEbb41Nl9u3Ol2HyB4BsDUipiLihbTM90hqXV5fmUUdptPX7wc3iZIi4vaIWBYRbyX7kJwD/HWK/Sgi1kXEz0TEEFld/29KnQDeJin/r4Lj9jeo2qjDURfHkaen+o4kAXeSPfhqTUT8JIUmyP6OzfkWAGcAExExCTyfj3Pk37wwt0u70bYu1aFV83RL374nZlOHEoudAM5Ky246q2Ru9/X6okg/DWQfePPI7sj4cnrdnHYm2Zv354A68F9yeWcAbwZOBt5Hdhg9lIt/l+yi7zzg39L/dzd1vA5p/neT3ckxD/gdYC/w5l7v7zS1+J/p77ewZfpbyE4JrEn78zmOvJvlFuBxsrt6lpN9WF5YJrcfhy7V4Wyy57+clN439wOP9Xpfu1SHk9P0TwHfSq9PSbHm3U3/nuzupivx3U39OZDd3hYtw0bgVLILSQfIbuG7GTg5l/dh4B+Ag8BfAe9tWe5g+kB9leyBSH19h0836kC6OJdyXwb+FBju9b5OU4elad8PkZ0SaA7rUvwCYEf6u9aBwVzuXLLfHXsFeAG4qmXZhbn9NnSrDsBHgB+k98TzwD3Az/R6f7tUh/VH+X9qUy7+DuDJlPuXwDt6vb/Nwb/dZGZmhXxNwszMCrlJmJlZITcJMzMr5CZhZmaF3CTMzKyQm4SZmRVykzAzs0JuEmZmVuj/A4k3h7pF3zeIAAAAAElFTkSuQmCC\n",
"text/plain": [
"| \n", " | cols | \n", "imp | \n", "
|---|---|---|
| 5 | \n", "saleElapsed | \n", "0.859446 | \n", "
| 9 | \n", "SalesID | \n", "0.119325 | \n", "
| 13 | \n", "MachineID | \n", "0.014259 | \n", "
| 0 | \n", "YearMade | \n", "0.001793 | \n", "
| 8 | \n", "fiModelDesc | \n", "0.001740 | \n", "
| 11 | \n", "Enclosure | \n", "0.000657 | \n", "
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"xs['saleYear'].hist();"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"filt = xs['saleYear']>2004\n",
"xs_filt = xs_final_time[filt]\n",
"y_filt = y[filt]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here's the result of training on this subset:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(0.17768, 0.230631)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m = rf(xs_filt, y_filt)\n",
"m_rmse(m, xs_filt, y_filt), m_rmse(m, valid_xs_time, valid_y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It's a tiny bit better, which shows that you shouldn't always just use your entire dataset; sometimes a subset can be better."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Using a neural network"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can use the same approach to build a neural network model. Let's first replicate the steps we took to set up the `TabularPandas` object:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df_nn = pd.read_csv(path/'TrainAndValid.csv', low_memory=False)\n",
"df_nn['ProductSize'] = df_nn['ProductSize'].astype('category')\n",
"df_nn['ProductSize'].cat.set_categories(sizes, ordered=True, inplace=True)\n",
"df_nn[dep_var] = np.log(df_nn[dep_var])\n",
"df_nn = add_datepart(df_nn, 'saledate')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can leverage the work we did to trim unwanted column in the random forest, by using the same set of columns for our neural network."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df_nn_final = df_nn[list(xs_final_time.columns) + [dep_var]]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Categorical columns are handled very differently in neural networks, compared to decision tree approaches. As we have seen in <>, a great way to handle categorical variables is by using embeddings. In order to create embeddings, fastai needs to know which columns should be treated as categorical variables. It does this by comparing the number of distinct levels in the variable (this is known as the *cardinality* of the variable) to the `max_card` parameter. Anything lower than this is going to be treated as a categorical variable by fastai. Embedding sizes larger than 10,000 should generally only be used after you've tested whether there are better ways to group the variable, so we'll use 9000 as our `max_card`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cont_nn,cat_nn = cont_cat_split(df_nn_final, max_card=9000, dep_var=dep_var)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"However, one variable that we absolutely do not want to treat as categorical is the saleElapsed variable. A categorical variable cannot, by definition, extrapolate outside the range of values that it has seen. But we want to be able to predict auction sale prices in the future. Therefore, we need to make this a continuous variable:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cont_nn.append('saleElapsed')\n",
"cat_nn.remove('saleElapsed')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's take a look at the cardinality of each of our categorical variables that we have chosen so far:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"YearMade 73\n",
"ProductSize 6\n",
"Coupler_System 2\n",
"fiProductClassDesc 74\n",
"ModelID 5281\n",
"Hydraulics_Flow 3\n",
"fiSecondaryDesc 177\n",
"fiModelDesc 5059\n",
"ProductGroup 6\n",
"Enclosure 6\n",
"fiModelDescriptor 140\n",
"Drive_System 4\n",
"Hydraulics 12\n",
"Tire_Size 17\n",
"dtype: int64"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_nn_final[cat_nn].nunique()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The fact that there are two variables pertaining to the \"model\" of the equipment, both with similar very high cardinalities, suggests that they may contain similar, redundant information. Note that we would not necessarily see this in the dendrogram, since that relies on similar variables being sorted in the same order (that is, they need to have similarly named levels). Having a column with 5000 levels means needing a number 5000 columns in our embedding matrix, so this would be nice to avoid if possible. Let's see what the impact of removing one of these model columns has on the random forest:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(0.176706, 0.230642)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"xs_filt2 = xs_filt.drop('fiModelDescriptor', axis=1)\n",
"valid_xs_time2 = valid_xs_time.drop('fiModelDescriptor', axis=1)\n",
"m2 = rf(xs_filt2, y_filt)\n",
"m_rmse(m, xs_filt2, y_filt), m_rmse(m2, valid_xs_time2, valid_y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There's minimal impact, so we will remove it as a predictor for our neural network."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cat_nn.remove('fiModelDescriptor')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can create our `TabularPandas` object in the same way as when we created our random forest, with one very important addition: normalisation. A random forest does not need any normalisation--the tree building procedure cares only about the order of values in a variable, not at all about how they are scaled. But as we have seen, a neural network definitely does care about this. Therefore, we add the `Normalize` processor when we build our `TabularPandas` object."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"procs_nn = [Categorify, FillMissing, Normalize]\n",
"to_nn = TabularPandas(df_nn_final, procs_nn, cat_nn, cont_nn,\n",
" splits=splits, y_names=dep_var)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Tabular models and data don't generally require much GPU RAM, so we can use larger batch sizes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"dls = to_nn.dataloaders(1024)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we've discussed, it's a good idea to set `y_range` for regression models, so let's find the min and max of our dependent variable:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(8.465899897028686, 11.863582336583399)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y = to_nn.train.y\n",
"y.min(),y.max()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now create the `Learner` to create this tabular model. As usual, we use the application-specific learner function, to take advantage of its application-customized defaults. We set the loss function to MSE, since that's what this competition uses.\n",
"\n",
"By default, for tabular data fastai creates a neural network with two hidden layers, with 200 and 100 activations each, respectively. This works quite well for small datasets, but here we've got quite a large dataset, so we increase the layer sizes to 500 and 250."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from fastai2.tabular.all import *"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"learn = tabular_learner(dls, y_range=(8,12), layers=[500,250],\n",
" n_out=1, loss_func=F.mse_loss)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"(0.005754399299621582, 0.0002754228771664202)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAERCAYAAABhKjCtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZzVZd3/8ddn9o0BBoZhHXYERVFAFAG1NHPJFLXMNdPCtG7NrDvzzm7zl5WWLXeZe26VW6JprqWigIrigsoOw77NxjL7+vn9cc7UcTjDzMCcbeb9fDzmwTnfc53vec8Bzudc3+/1vS5zd0RERFpLinUAERGJTyoQIiISlgqEiIiEpQIhIiJhqUCIiEhYKhAiIhKWCoSIiISVEq0XMrNvA5cAhwKPuPslbbSbCNwGTAH6ubt1ZP/9+/f3ESNGdElWEZGe4r333it19/xwj0WtQABbgZ8Cnwcy99GuAXgc+CPwdEd3PmLECBYvXnxAAUVEehoz29DWY1ErEO4+NxhmKjB0H+1WAivNbEy0somIyN4S+hyEmc0xs8VmtrikpCTWcUREupWELhDufre7T3X3qfn5YQ+hiYjIfkroAiEiIpGjAiEiImFFc5hrSvD1koFkM8sAGt29sVU7A9KBtOD9DMDdvS5aWUVEJLo9iB8BNcB1wIXB2z8ys0IzqzSzwmC74cHHlgbv1wAro5hTRCRhbCyrpr6xOSL7jlqBcPcb3d1a/dzo7hvdPcfdNwbbrQ/TbkS0coqIJIrmZufk373Bz19YHpH96xyEiEiC2rKrhur6JsYO6BWR/atAiIgkqNXFFQCMK8iJyP5VIEREEtTqHZUAjC1QD0JEREKs2lFJQW46vTNTI7J/FQgRkQS1uriCcRHqPYAKhIhIQmpudlbvqIzYCWpQgRARSUhbdtVQ09AUsRPUoAIhIpKQVu0IjGAaqwIhIiKhVgVHMI3RISYREQm1ekcFA3MzIjaCCVQgREQS0qriiogeXgIVCBGRhNPc7KwprozoEFdQgRARSTibd9ZQ29Ac0RFMoAIhIpJw/jOCST0IEREJsSo4Sd/YAepBiIhIiNU7KhnUO4NeGZEbwQQqECIiCWfVjoqIH16CKBYIM/u2mS02szoze6CdtteY2XYz221mfzKz9CjFFBGJa00tI5gifHgJotuD2Ar8FPjTvhqZ2ecJrFt9AjACGAX8JNLhREQSwabyauoamyM+xBWiuyb1XHd/Gihrp+lXgfvcfam77wT+H3BJpPOJiCSCaMzB1CIez0EcAiwJub8EKDCzfjHKIyISN1YXR3YVuVDxWCBygN0h91tu7/VumNmc4HmNxSUlJVEJJyISS8u37WFYXiY56SkRf614LBCVQG7I/ZbbFa0buvvd7j7V3afm5+dHJZyISCyt3F7BQQW57TfsAvFYIJYCk0LuTwJ2uHt75y5ERLq1usYmikqrmDAo8oeXILrDXFPMLANIBpLNLMPMwvWRHgIuM7ODzawv8CPggWjlFBGJV2uKK2lqdg4a2M0KBIEP+hoCQ1gvDN7+kZkVmlmlmRUCuPuLwK3Aa8CG4M//RjGniEhcWrEtcKR9/MDoHGKK/FmOIHe/EbixjYc/NV7L3X8N/DrCkUREEsrKHRWkpSQxol9WVF4vHs9BiIhIGMu37WFcQQ4pydH56FaBEBFJECuiOIIJVCBERBJCWWUdJRV1URvBBCoQIiIJYeX2wAnqaI1gAhUIEZGEsGJ7dEcwgQqEiEhCWLF9D/2y08jvFb3VD1QgREQSwMrtFYyP4vkHUIEQEYl7Tc3Oyh3RHcEEKhAiInFvY3k1tQ3NjI/iCWpQgRARiXsrtu0B0CEmERH5tBXbKzCDsQNUIEREJMSK7XsY2S+bzLTkqL6uCoSISJxbub0iqhfItVCBEBGJY7UNTWwsr47KGtStqUCIiMSxopIqmh3GDshpv3EXU4EQEYlja0oqARijAiEiIqHW7KggyWBk/+yov7YKhIhIHFtTUklhXhYZqdEdwQRRLBBmlmdmT5lZlZltMLPz22jXx8weNLPi4M+N0cooIhJvVu+oZEyUr39oEc0exO1APVAAXADcYWaHhGn3GyALGAFMAy4ys69FK6SISLxoaGpmfVlVTM4/QJQKhJllA2cDN7h7pbsvAJ4BLgrT/HTgVnevdvf1wH3ApdHIKSISTzaUVdPQ5DEZwQTR60GMA5rcfVXItiVAuB4EgLW6PTFsI7M5ZrbYzBaXlJR0TVIRkTixpjh2I5ggegUiB9jdattuINyBtReB68ysl5mNIdB7yAq3U3e/292nuvvU/Pz8Lg0sIhJra4oDq8iN7uYFohJoPZF5LlARpu1VQA2wGvg78AiwOaLpRETi0JriSgb3ziAnPSUmrx+tArEKSDGzsSHbJgFLWzd093J3v8DdB7r7IcGM70Qpp4hI3FhdXMmYGEyx0SIqBcLdq4C5wE1mlm1mM4AzgIdbtzWz0WbWz8ySzewUYA7w02jkFBGJF83NztqSSsbkx+bwEkR3mOuVQCZQTOCw0RXuvtTMZplZZUi7KcDHBA4//Ry4wN336mmIiHRnW3bVUNvQzNiC2BWIqB3Ycvdy4Mww2+cTOIndcv9x4PFo5RIRiUexHsEEmmpDRCQurQ6OYOoph5hERKSD1hRX0j8njb7ZaTHLoAIhIhKHVhdXxvTwEqhAiIjEHXdnjQqEiIi0VlxRR0VtI2NjNItrCxUIEZE4Ew8jmEAFQkQk7rQUiNExHMEEKhAiInGnqKSSnPQUCnLTY5pDBUJEJM6sLalidH42ZtZ+4whSgRARiTNrSyoZFePDS6ACISISV6rqGtm2u5bR+dmxjqICISIST9aVVgGxP0ENKhAiInFlbUlwBFOMh7iCCoSISFxZW1xJksHwfmFXWo4qFQgRkTiytrSKYXlZpKckxzqKCoSISDxZW1wZF+cfQAVCRCRuNDc760qr4mIEE0SxQJhZnpk9ZWZVZrbBzM5vo126md1pZjvMrNzMnjWzIdHKKSISK1t21VDX2NwjexC3A/VAAXABcIeZHRKm3dXAdOAwYDCwC/h9tEKKiMRKywimeLhIDqJUIMwsGzgbuMHdK919AfAMcFGY5iOBl9x9h7vXAo8C4QqJiEi3srak5RqInnWIaRzQ5O6rQrYtIfwH/33ADDMbbGZZBHobL4TbqZnNMbPFZra4pKSky0OLiETT2pJK+mSlkhfDZUZDRatA5AC7W23bDYRbDWMVsBHYAuwBJgA3hdupu9/t7lPdfWp+fn4XxhURib6iksAIplhP0tciWgWiEshttS0XqAjT9g4gA+gHZANzaaMHISLSnawtqWJU//g4vATRKxCrgBQzGxuybRKwNEzbScAD7l7u7nUETlBPM7P+UcgpIhITe2obKKmoi4spNlpEpUC4exWBnsBNZpZtZjOAM4CHwzR/F7jYzHqbWSpwJbDV3UujkVVEJBaKSuJnkr4W0RzmeiWQCRQDjwBXuPtSM5tlZpUh7b4H1AKrgRLgVGB2FHOKiETd2n8vMxo/h5hSovVC7l4OnBlm+3wCJ7Fb7pcRGLkkItJjrC2pJCXJGJYX+0n6WmiqDRGROFBUUsXwflmkJsfPx3L8JBER6cHiZZnRUCoQIiIx1tTsbCirZlQcnX8AFQgRkZjbvLOa+qZmRvdXD0JEREIUBdehVg9CREQ+peUaiJFxdBU1qECIiMRcUUklvTPjZ5K+FioQIiIxtq60ilH52XEzSV+LDhcIM/uumR0evH20mW00syIzmx65eCIi3V9RSVXcHV6CzvUgrgHWBW//HPg1cDPw264OJSLSU1TVNbJ9T21czcHUojNTbfR2991m1ovAjKsnunuTmd0WoWwiIt3eupYRTHHYg+hMgdhkZscQWAXujWBxyAWaIhNNRKT7axniOjLOhrhC5wrE94G/AfUE1pcG+ALwTleHEhHpKYpKKjGDEf0SuEC4+/PA4Fabnwj+iIjIfigqqWJIn0wyUpNjHWUvnRnFdLCZFQRv55jZT4AfAqmRCici0t0VlcbfJH0tOjOK6a9An+DtXwHHAtOBu7o6lIhIT+DurIuzdahDdeYcxAh3X2mBKzlmEzhZXcN/hr6KiEgnFFfUUVXfFHdzMLXoTA+iLjjEdRqwKbhGdB2Q0ZEnm1memT1lZlVmtsHMzm+j3QtmVhnyU29mH3cip4hIQlhbElhmdFSczeLaojM9iL8CrwK9gD8Et02m4z2I2wmMgCoADgeeM7Ml7r40tJG7nxJ638zmBV9XRKRb+fckfXHag+jMKKZrzOwkoMHdXwtubiZwhfU+mVk2gaGxE929ElhgZs8AFwHX7eN5I4BZwNc6mlNEJFGsK60iIzWJQbkdOhATdZ3pQeDuL5tZYXD+pS3uvriDTx0HNLn7qpBtS4Dj2nnexcB8dw/bSzGzOcAcgMLCwg5GERGJD0UllYzsn0NSUnxN0teiM8NcB5nZ68BqYC6wxsxeN7PW10aEkwPsbrVtN4HDVftyMfBAWw+6+93uPtXdp+bn53cgxt4ampqprm+kqdn36/kiIvurqDR+RzBB53oQdxD41n+qu1cFDxv9DLgT+GI7z60EclttywUq2nqCmc0EBhK4ejti/rlsB1f+5X0AUpONjJRkzp4ylB9/4eC4reoikvjqG5vZVF7NFyd15Dt2bHSmQMwEBrl7A0CwSPw3sKUDz10FpJjZWHdfHdw2CVi6j+d8FZgbPGcRMeMKenHdKeOpa2imtrGJjeXVPPDmemobmvjZ7ENVJEQkIjaWV9Hs8bfMaKjOFIidwMEEehEtDgJ2tffEYDGZC9xkZl8nMIrpDOCYcO3NLBP4EnBWJ/LtlzEDchgz4D9DzNydkf2y+cNrazAzbj5zooqEiHS5tf9eZjQ+h7hC5wrErcC/zOw+YAMwnMDoohs6+PwrgT8BxUAZcIW7LzWzWcAL7h76Lp1J4BzFa3vvJrLMjGtPGofj3P7aWpqbnetPm0DvTM0oIiJdp+UaiNHdoQfh7veY2VrgfOAwYCuBYaozO/j8cgIf/K23zydwEjt02yPAIx3N1tXMjO+ddBAAt7+2lmc/2srZk4dyyYwRnV7Uo7iilo827aa+qRkDzGBPTSNFpVWsL61i255axuTnMGV4X6aO6MvgPpnUNTRR39RMQ6NjBinJRrIZa0uqeGddOYvWlbGutIrCvCzGFfRiXEEOw/tlM6RvJoN7Z5KeksT2PbWsL61iY3k12ekpDMvLYmjfTPplp3XpsoZNzU6SEXdLJYrEu6KSKvJ7pdMrI36/fJr7/o/eMbN0oNrdYz4N4dSpU33x4o6Ouu24T7bs5v6F63l2yVbqm5rJTE2m2R13SE9JYlR+NqMH5DA6P4f0lCTqGpupa2xm664aFq8vZ31Zddj9piYbhXlZFORmsHJ7BWVV9R3KYwYTBuYyriCHjeXVrN5RSUVd4177bmgK//eammzkpKeQk5FCdloK2ekpZKYmk5GaTHZ68r8fy0lLITUliZQkIyXJqKhtZH1ZNRvKqti8s4aq+kZqG5poaHJSkoy+2WnkZaXRLyeNIX0yGdo3UJAK+2UxvF8W+TnpKiIiIc7640LSUpJ4dE5sV202s/fcfWrYx7qgQNS4e2em7IiISBWIFiUVdcx9fzNlVfWYQZIZVXWNrC2pZE1xJTv21H2qfb/sNKYM78uRI/I4orAPvTJSaXan2Z1e6akM7pNBSnLgbXN31pdV896GnZRV1pGRmkx6ShIpyUmB5zQ7Dc3OkD4ZTBme96nDXe7O9j21bCqvYcuuarbuqmVPbQPD+mYxsn82hXlZVNU3srm8hk07q9mxp46qukYq6xqpqG2kpqGRmvomahqaqaprpKqukYq6Ruobm/d6Dwb1zmB4vyyG9c0iJyOFjNRkMlKSqW9qoryqnrLKekoq69i6q2av9yMrLZnBfTJJNvv3+zeodwYTBuUyflAvxhX0YkifTLLTO3VpjkhCcncOv+mfnHbYIH42+9CYZtlXgeiK/4094gKC/F7pXH7c6DYfr6prpNmd9JRkUpOtU9+WzYyR/bP3a9FyM2NQ70wG9c4E8tpsN35g61HG+1bf2ExjczONzU5Tk5OZltyp+eprG5rYuquGjeXVbCirZn1ZFdt31/6799XU7GzaWc28VSWfugald2Yqg/tkMrJ/FmPycxg9IIdxBb0YMyCH1OSYfw8R6RLlVfXsrmmIy3WoQ7VbIMzss/t4OK0LsyS07vbNNy0libROzeX4aRmpyYzKz2l3nvu6xibWFAd6YVt31bJtdw2bd9awfFsFL36ynZbakZaSxISBvZg4pDenHTqI6aP76ZCVJKyWEUzxPMQVOtaDuK+dxzd2RRDpmdJTkjlkcG8OGdx7r8dqG5pYX1bFyu0VLN26h0+27ObvH27lL4s2Mio/mwuPGs7ZU4ZqhJkknKLgCKYxid6DcPeR0Qgi0lpGajLjB+YyfmAuZxw+BAgUjec+2safF23gpn8s47aXV3LB0cO5bOZICuJ0wjOR1opKq0hLSWJwn8xYR9mn7nVcRLq9jNTAVChnTxnKJ1t2c8/8Iu6dX8QDC9dz9pQhXHHcGAr7ZcU6psg+rS2uZFT/bJLj/CJcnfWThDVxSG9+95UjmPe9z/ClqUN58v0tfOa2eXzviSWsK62KdTyRNhWVVsX9+QdQgZBuoLBfFjfPPpT5//0Zvjp9BP/4aCsn3DaPb/31fd4uKuNAhnKLdLX6xmY2llfH7SpyoXSISbqNgtwMfnz6wVxx/GjuXVDEI4s28txH2xgzIIcLjyrk3CMLyUyL+TWd0sNtLK+iqdkZPUA9CJGoy++Vzg9PmcCi60/k1nMOIzstmRufXcZxv3yNh95aT11jU6wjSg/27yGuCdCDUIGQbiszLZkvTx3G3789k8cvn86I/tn8+O9L+eyvXufJ9zbTrEWiJAZaJunTOQiRODFtZB6PzTmahy6dRr+cNK59Ygln3/kmSza1O1u9SJcqKqliQJxP0tdCBUJ6DDPj2HH5PH3lDH71pUlsKq/hzD8u5L//toTSyrr2dyDSBYpKKuN+io0WKhDS4yQlGedMGcpr3zuOb8waxdz3t/CZX83j/oXraGzae5JCka7i7qwtSYwhrqACIT1Yr4xUrj91Ai9+51gOH9aHnzy7jC/8fgEfbNwZ62jSTbVM0tfeHGXxQgVCerwxA3J46NJp3HnhFCpqGzn3rrd57F1NMSZdr2UEUzyvIhcqagXCzPLM7CkzqzKzDWZ2/j7aTjazN8ys0sx2mNnV0copPZOZcfLEgTx31UyOGpXHD578mP/9+yc06JCTdKGify8zqh5Ea7cD9UABcAFwh5kd0rqRmfUHXgTuAvoBY4CXo5hTerA+WWncf8mRfGPWSB58awMX3beInR1c7U8knPrG5sCiXPVNrNpRSXoCTNLXIipXUptZNnA2MNHdK4EFZvYMgTWtr2vV/LvAS+7+l+D9OmB5NHKKAKQkJ/E/px3MhEG5XPfkx5x1x5vcf8mRjNiPBZ2k5/pw0y7uX7iO5z/e9qklgMcP7BX3k/S1iNZUG+OAJndfFbJtCXBcmLZHAx+b2ZsEeg+LgG+5+14Hhc1sDjAHoLCwsMtDS8921uShDMvLYs5Di5n9x4Xcc/FUpo5oe9U+EXfn5WU7uOv1tby/cRc56SmcN62QwX0yaZkSbNrIxPk3dEBrUnf4RcxmAU+4+8CQbd8ALnD341u1XQUMAD4HfAzcCkxx9xn7eo1Ir0ktPde60ioufeBdtuys4bYvT+L0SYNjHUni0MI1pdz60kqWbNpFYV4WX5sxgnOmDI37C+IivSZ1R1QCrRdFzgUqwrStAZ5y93cBzOwnQKmZ9Xb33ZGNKbK3kf2zmXvFMcx5eDH/9cgHFFfUcdlMraMlAQ1NzXzz4fd4ZUUxg3tncOvZh3HW5CGkdIM11KNVIFYBKWY21t1XB7dNApaGafsRENqtabmdGAftpFvqm53Gw5cdxdWPfsD/+8cyduyp5bqTx5OUIMeSJXLe27CTV1YUc+Xxo7nqhLFkpHafGYOjUuLcvQqYC9xkZtlmNgM4A3g4TPP7gdlmdriZpQI3AAvcXZPmSExlpCbzxwumcNHRw7n7jSK++/iHGgYrvLmmlCSDbx4/ulsVB4juMNcrgUygGHgEuMLdl5rZLDOrbGnk7q8C1wPPBduOAdq8ZkIkmpKTjJvOOITvnTSOpz/cypyHFlNTr+nDe7IFa0o5bGgfcuP8XMP+iFqBcPdydz/T3bPdvdDd/xrcPt/dc1q1vcPdh7h7X3c/3d03RSunSHvMjG9/diw/m30o81aVcPGfFrG7piHWsSQGKmobWLJ5NzPG9It1lIhI/LMoIjFy/lGF/OG8yXy4aRfn3vUWxRW1sY4kUbaoqJymZmfGmP6xjhIRKhAiB+C0wwZx31ePZENZNefe9TZbdtXEOpJE0cK1paSnJDG5sG+so0SECoTIATp2XD5//vo0Sivr+PKdb7G+tCrWkSRK3lxTxpEj8rrdyekWKhAiXWDK8Dwe+cbR1DQ08aW73mLl9nCX+Eh3UlxRy8odFd328BKoQIh0mYlDevPYnKNJMvjK3W/xyRZd19mdvbW2DKDbnqAGFQiRLjW2oBePXz6dzNRkLrh3ER9vVpHorhasLiU3I4VDBveOdZSIUYEQ6WLD+2Xz2OXT6ZWRwvn3vs2Hm3SNZ3fj7ixcU8oxo/snzMys+0MFQiQChuVl8eico+mblcZF9y7ivQ1axrQ7WV9Wzdbdtd368BKoQIhEzNC+WTx2+dH0y0njovsW8XZRWawjSRdZuKYUgGO68QlqUIEQiahBvTN5/PLpDO6TySX3v8OC1aWxjiRdYN7KYob0yWRUN19ESgVCJMIG5Gbw6JyjGdEvm0sffJdXV+yIdSQ5ANX1jcxfXcrnDi7ArPuefwAVCJGo6J+TzqNzjuaggl7Meeg9nl2yNdaRZD+9saqUusZmTjqkINZRIk4FQiRK+mSl8ZdvHMXk4X256tEP+OuivVbRlQTw8rLt9M5MZVoPWH5WBUIkinIzUnno0ml85qABXP/Ux9wxb22sI0knNDY188ryYk6YMKBbrBjXnu7/G4rEmYzUZO66aApfnDSYW15cwY3PLKVRCw8lhHfWl7O7poGTDh4Y6yhREa0lR0UkRGpyEr8593AG9Ern3gXr2FBWxe/Pn0xOuv5LxrOXl+4gPSWJY8d17+GtLdSDEImR5CTjR184mJtnT+SN1aWcc8ebmi48jrk7/1y2g1lj88lK6xmFPGoFwszyzOwpM6sysw1mFnYZUTO70cwazKwy5GdUtHKKRNsFRw3nga8dyZadNZzxhwUs0gV1cWnZtj1s2VXTI0YvtYhmD+J2oB4oAC4A7jCzQ9po+5i754T8FEUtpUgMzBqbz1PfOobczFQuuHcRDyxch7vHOpaEeHnpDpIMThg/INZRoiYqBcLMsoGzgRvcvdLdFwDPABdF4/VFEsGYAb14+lszOP6gAdz47DKufWIJ1fWNsY4lQS8v28HUEXn0y0mPdZSoiVYPYhzQ5O6rQrYtAdrqQZxuZuVmttTMrmhrp2Y2x8wWm9nikpKSrswrEhO5GancfdEUrjlxHE99sIUv/H6B1pWIA2uKK1m+bQ+fP6RnjF5qEa0CkQO0/le+G+gVpu3jwAQgH/gG8GMzOy/cTt39bnef6u5T8/PzuzKvSMwkJRlXnziWv3z9KKrrmpj9x4Xc+fpampt1yClWnvpgM8lJxumTBsU6SlRFq0BUArmttuUCe63L6O7L3H2ruze5+5vA74BzopBRJK4cM7o/L35nFidOKOAXL6zgwvsWsW23RjlFW3Oz89T7Wzh2bH8G9MqIdZyoilaBWAWkmNnYkG2TgKUdeK4D3XtGLJE29MlK448XTOaWsw/lg427OPm383nxk22xjtWjvF1UxtbdtZw1eWiso0RdVAqEu1cBc4GbzCzbzGYAZwAPt25rZmeYWV8LmAZcBfw9GjlF4pGZce6RhTx31UwK87L45p/f57onP6KqTiewo+HJ97fQKyOFzx3cc4a3tojmMNcrgUygGHgEuMLdl5rZLDOrDGn3FWANgcNPDwG3uPuDUcwpEpdG5efw5BXH8M3jRvPY4k2c/Ls3dM1EhFXVNfLCJ9v4wmGDyEhNjnWcqIva5YDuXg6cGWb7fAInsVvuhz0hLSKQlpLEdaeM54QJA7j28SV85Z63uWzGSL73+YN65AdYpL20dDvV9U098vASaKoNkYR05Ig8Xrh6FudPK+TeBes45XfzeXOtVqvranPf30JhXhZTh/eNdZSYUIEQSVDZ6SncPPtQHr5sGk3Nzvn3LOL7TyxhZ1V9rKN1C1t31bBwbSlnTR7S7VeOa4sKhEiCmzU2n5e+cyxXHD+auR9s4YRfv84j72ykSddNHJCH396AO8w+Ykiso8SMCoRIN5CZlswPTh7PP/5rJqPzs/nh3I/54h8WsHh9eayjJaQ1xZXcO7+I2UcMYXi/7FjHiRkVCJFuZMKgXB6/fDr/d94RlFfVc86db/Gtv7zP+tKqWEdLGO7ODU9/QmZqMv9z2oRYx4mpnjGpuUgPYmZ8cdJgTpwwgLteL+Ke+UW8tHQ75x9VyFUnjKV/D5psbn88/eEW3ioq46dnTuzx75V6ECLdVFZaCtd8bhzzvn885x45jL8s2sjxv5zHna+vpa6xKdbx4tLu6gZufm45k4b14fxphbGOE3MqECLd3IBeGdw8+1BevuZYjh6Vxy9eWMFJv3mDl5Zu15oTrfzy5RWUV9Vz85kTSUrqmSOXQqlAiPQQo/NzuPerR/LQpdNIS07i8off42sPvMum8upYR4sLy7bu4S+LNnLx9BFMHNI71nHiggqESA9z7Lh8Xrh6Fjd84WDeWVfOSb95g3vnF9HY1BzraDHj7vzs+eX0zkzlmhPHxTpO3FCBEOmBUpKTuGzmSP753eM4ZnQ/fvrccr74h4W820OHxc5bWcKCNaVc9dmx9M5KjXWcuKECIdKDDemTyb1fncofL5jMzup6vnTnW3zn0Q/Yvrs21tGiprGpmZufX87I/tlcePTwWMeJKxrmKtLDmRmnHjqI4w/K50quT3AAAA9PSURBVI55a7nrjSJeXraDy2aO5OszR3X7b9SPvLuJNcWV3HXRFNJS9J05lN4NEQECw2KvPekg/nXNcXxm/AB+/+oaZt7yKr/55yp21zTEOl5E7Klt4Lf/XMVRI/M4qQeu99AeFQgR+ZTCflncfv5kXrh6FjPG9Od3r6zutoXimQ+3UlZVzw9PndBjJ+TbFxUIEQlrwqBc7rxoCv/4r5lMH9UvUCh+8Sq/fnlltykUy7btITcjhUlDNaw1HBUIEdmniUN6c/fFU3n+qlnMHNuf/3t1Dcfe+hp3zFtLTX1iX5G9cnsF4wfmqvfQhqgVCDPLM7OnzKzKzDaY2fnttE8zsxVmtjlaGUWkbQcPzuWOC6fw3FUzmVzYh1teXMGxv3yN+xeuo7o+8dbHdvdAgRjUK9ZR4lY0exC3A/VAAXABcIeZHbKP9t8nsH61iMSRQwb35v6vTeOJb05nZP9sfvLsMmb84lV++69VCbVY0eadNVTWNXLQQBWItkSlQJhZNnA2cIO7V7r7AuAZ4KI22o8ELgR+Ho18ItJ5R47I4/HLp/O3b05nyvC+/PZfq5n+i1e49vElvLOuPO7neVqxvQKA8QNzY5wkfkXrOohxQJO7rwrZtgQ4ro32vweuB2oiHUxEDszUEXncOyKPVTsquH/hep5dspUn39/MqP7ZnDetkC9PHRaX11Ks3L4HQD2IfYjWIaYcYHerbbuBvf5mzGw2kOLuT7W3UzObY2aLzWxxSUlJ1yQVkf0yrqAXPz/rUN75nxP41ZcmkZedxs3PL+eon/+LH879iOXb9sQ64qcs317BsLxMctJ1vXBbovXOVAKt+3G5QEXohuChqFuBUzuyU3e/G7gbYOrUqfHdnxXpIbLSUjhnylDOmTKUZVv38OCb65n7/hYeeWcTR47oy4VHD+fkiQNJT0mOac6WEUzStmj1IFYBKWY2NmTbJGBpq3ZjgRHAfDPbDswFBpnZdjMbEYWcItKFDh6cyy3nHMai60/g+lPHU1xRx9WPfsiMX7zKr15ayY49sZnzqbahiXWlVYzX4aV9ikoPwt2rzGwucJOZfR04HDgDOKZV00+AYSH3jwH+AEwGdAxJJEH1yUpjzrGj+frMUSxYU8pDb23g9nlruPP1tZxy6CAuOWY4kwv7Ru16hDXFlTQ1u3oQ7YjmwbcrgT8RGLpaBlzh7kvNbBbwgrvnuHsjsL3lCWZWDjS7+/awexSRhJKUZBw7Lp9jx+WzsayaB99az+PvbuLZJVsZV5DDuUcWMvuIIeRlp0U0R8sIJp2g3jeL96FoHTV16lRfvHhxrGOISCdV1jXy7JKtPPruJpZs2kVachInHjyAs44YynEH5ZOa3PVHwm9+bhkPvbWBpT/5PCkR2H8iMbP33H1quMd0+l5EYionPYXzphVy3rRCVmzfw2PvbuKZD7fy/Mfb6ZedxqmHDuLYcfkcPSqPXhldM1x2xfYKxhbk9Pji0B4VCBGJG+MH5vK/px/C9adO4PWVJTz5/mb+9t5mHn57AylJxuTCvpx++GDOOHwwuQdQLFZsr+C4cfldmLx7UoEQkbiTmpzEiQcXcOLBBdQ1NvHehp0sWF3KK8uLueHpT/jZc8s57bBBnDetkMmFfTp1crusso6SijqNYOoAFQgRiWvpKckcM7o/x4zuz/c/fxAfbd7No+9u5JkPt/K39zYzaVgfLp0xglMPHdSh8xUrNcVGh+kAnIgkDDNj0rA+/Pysw3jnf07kpjMOYU9NA1c/+iGzbnmN219bQ3k7EwYubykQmsW1XepBiEhCyk5P4eLpI7jwqOHMW1XMnxas55cvreR3r6zmjEmD+dLUYRxR2GevXsWKbXvon5NG/5z0GCVPHCoQIpLQkpKMz44v4LPjC1i9o4IHglN7PPHeZrLSkpk2Mo/Dh/VhZ1U9m3fW8M66ciYN6xPr2AlB10GISLezp7aBN9eUsXBNKQvXllJUUkWv9BSG5mUxpE8mXz1mOLPGahQT6DoIEelhcjNSOXniQE6eOBAIzL2UkRrbyQETkU5Si0i3p+Kwf1QgREQkLBUIEREJSwVCRETCUoEQEZGwVCBERCQsFQgREQlLBUJERMLqNldSm1kJsAHoDewOeSj0fsvt1n/2B0o7+ZKtX6cjj7e3bV9ZQ7d1dd62HuvIe9le7li9t23dT6Ss4TLGOmtb+cJlDd3Wk//NxnvW4e4e/rJyd+9WP8Ddbd1vuR3mz8UH+jodeby9bfvKGsm8bT3WkfeyA+9xTN7btu4nUtY2Murf7D4ei8d/s4mUtfVPdzzE9Ow+7j/bxp9d8Todeby9bfvK2pHX7Gye9h7ryHvZ1u1Yv7dt3U+krKG34yVr6236N9v+cxMp66d0m0NMB8LMFnsbk1XFo0TKq6yRkUhZIbHyKut/dMcexP64O9YBOimR8iprZCRSVkisvMoapB6EiIiEpR6EiIiEpQIhIiJhqUB0kJnNNLN5wZ9VZvabWGfaFzM73sxeMbPXzGx2rPO0xcxGmFlJyHubEMt8mdl5wWtv4paZFZjZm2b2upm9amaDYp2pLWY23czeCmZ9xMxSY52pLWbW28zeMbNKM5sY6zzhmNnNZjbfzP5mZln7ux8ViA5y9wXufry7Hw+8CTwd40htMrMM4FrgFHf/jLs/FetM7Xi95b1197j+0AUwsyTgHGBTrLO0oxSY6e7HAQ8Bl8U4z75sAD4bzFoEnBHjPPtSDZwG/C3WQcIJFq3R7j4L+Bdw6f7uSwWik4LfbKYB82OdZR+OAWqAZ83sKTMbGOtA7ZgR/LbzMzOzWIfpgPMJfDg0xzrIvrh7k7u3ZOwFLI1lnn1x963uXhO820gcv7fu3hDnX2RmAS8Eb78AzNzfHXXLAmFm3zazxWZWZ2YPtHosL/ihWWVmG8zs/E7u/nPAKyH/8eIxawEwBjgduAe4MY6zbgtmPRYYAJzVFVkjldfMkoEvA491Vc5IZQ0+93AzWwR8G3g/nrMGnz8SOAX4R7xnjbQDyN6X/0ylsRvI298MKfv7xDi3Ffgp8Hkgs9VjtwP1BD5EDweeM7Ml7r40+E07XLfxHHffHrz9JeD+eM4K7AIWunu9mb0CXBevWYPvax2Amc0FjgaejNe8wX097u7NXdzZich76+4fAkeZ2ZeBHwLfjNesZpYLPAhc5O71XZAzYlm7KFt79is7sJPAfEsE/yzf7wSdnccjkX6Cb+4DIfezg2/quJBtDwO/6OD+UoFPgKR4zgr0I3Ds0YCjgPvjOGtuyO2fAxfH+Xt7C/Ay8CKBb2f/F8dZ00Nufx74dRxnTQGeI3Aeokv//rs6a0j7B4CJkch7INmBQ4G/Bm/PAf5rf1+7u/Yg2jIOaHL3VSHblgDHdfD5JwKvehcdXmrHfmd19zIzewp4ncCx3P0+SdVBB/K+HmdmNxI48bcOuKHr4+3lQN7bH7TctsA0B1dFIF+oA3lvJ5vZLUATUEt8/zs4j8CXmR+b2Y+BO9y9Sw/jtXJAnwVm9jyBb+4Hmdld7v5A10ds0z6zu/vHwcNO84Fi4OL9faGeViBy2Hva3d0ETuC1y91f4D8nfyLtQLPeTqAbGg37ndXdn+XAJh7bHwf03rbw6MzXcyDv7VsEzu1Ey4FkfZjAt+BoOdD/X6d2eaKOaze7u/+wK16oW56k3odKILfVtlygIgZZ2qOskZNIeZU1MhIpa2tRy97TCsQqIMXMxoZsm0R8Dv9T1shJpLzKGhmJlLW1qGXvlgXCzFIscLFYMpBsZhlmluLuVcBc4CYzyzazGQQuyIlm11ZZlVdZlbVdcZE90mfgY/FDYNy/t/q5MfhYHoGroKuAjcD5ytr9siZaXmVV1njMrum+RUQkrG55iElERA6cCoSIiISlAiEiImGpQIiISFgqECIiEpYKhIiIhKUCISIiYalAiIhIWCoQIl3AzF4ws6/GOodIV1KBkIRmZuvN7MRY53D3U9z9wa7er5kdb2bNZlZpZhVmttLMvtaJ599oZn/u6lzSM6hAiLTDzGK9bspWd88hMKXzNcA9ZnZQjDNJD6ACId2WmX3BzD40s11m9qaZHRby2HVmtjb4rXyZmc0OeewSM1toZr8xs3LgxuC2BWb2KzPbaWbrzOyUkOfMM7Ovhzx/X21Hmtkbwdf+l5nd3pFv+R7wPIE1hkN/l9+Z2SYz22Nm75nZrOD2k4HrgXODPZAlwe29zew+M9tmZlvM7KdmlnwAb7V0UyoQ0i2Z2WTgT8DlBNbovgt4xszSg03WArMILOr+E+DPZjYoZBdHAUXAAODmkG0rgf7ArcB9ZmZtRNhX278C7wRz3Qhc1MHfKcnMvhjc55qQh94lsPxlXnDfT5hZhru/CPwMeMzdc9x9UrD9g0AjMAY4AjgJ+HpHMkjPogIh3dU3gLvcfZG7NwXPD9QBRwO4+xPuvtXdmz2w9vFqYFrI87e6++/dvdHda4LbNrj7Pe7eROBDdhBQ0Mbrh21rZoXAkcCP3b3e3RcAz7Tzuww2s11ADfAU8F13/6DlQXf/s7uXBbPeBqQDYQ9BmVkBcArwHXevcvdi4DfAV9rJID2QCoR0V8OBa4OHl3YFP2CHAYMBzOzikMNPu4CJBL6Zt9gUZp/bW264e3XwZk4br99W28FAeci2tl4r1FZ370PgHMT/AZ8NfdDMrjWz5Wa2O/i79G71u4QaDqQC20J+97sI9JREPiXWJ99EImUTcLO739z6ATMbDtwDnAC85e5NZvYhEHq4KFILpWwD8swsK6RIDOvIE929zsx+AKw0szPd/eng+YYfEPhdlrp7s5nt5D+/S+vfYxOBnlR/d2884N9GujX1IKQ7SA0ux9jyk0KgAHzTzI6ygGwzO83MegHZBD44SwCCw0YnRiOou28AFhM48Z1mZtOB0zvx/HrgNuDHwU29CJxPKCGwTvGP+fSC9juAEWaWFHz+NuBl4DYzyw2e1xhtZscd6O8m3Y8KhHQHzxM4Pt/yc6O7LyZwHuIPwE4CJ3UvAXD3ZQQ+ZN8i8AF6KLAwinkvAKYDZcBPgccIfKvvqD8BhWZ2OvAS8AKBhew3ALV8+pDVE8E/y8zs/eDti4E0YBmB9+ZvBM6RiHyKlhwViTEzewxY4e7/G+ssIqHUgxCJMjM7MnhYJyl4rcIZBBagF4krOkktEn0DgbkEroPYDFwROmxVJF7oEJOIiISlQ0wiIhKWCoSIiISlAiEiImGpQIiISFgqECIiEtb/ByB47SeblLNwAAAAAElFTkSuQmCC\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"learn.lr_find()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There's no need to use `fine_tune`, so we'll train with 1-cycle for a few epochs and see how it looks..."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
"
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"learn.fit_one_cycle(5, 1e-2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can use our `r_mse` function to compare to the random forest result we got earlier."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"0.2258"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"preds,targs = learn.get_preds()\n",
"r_mse(preds,targs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It's quite a bit better than the random forest (although it took longer to train, and it's more fussy about hyperparameter tuning).\n",
"\n",
"Before we move on, let's save our model in case we want to come back to it again later."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"learn.save('nn')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### fastai's Tabular classes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In fastai, a tabular model is simply a model which takes columns of continuous or categorical data, and predicts a category (a classification model) or a continuous value (a regression model). Categorical independent variables are passed through an embedding, and concatenated, as we saw in the neural net we used for collaborative filtering, and then continuous variables are concatenated as well.\n",
"\n",
"The model created in `tabular_learner` is an object of class `TabularModel`. Take a look at the source for `tabular_learner` now (remember, that's `tabular_learner??` in jupyter). You'll see that like `collab_learner`, it first calls `get_emb_sz` to calculate appropriate embedding sizes (which you can override by using the `emb_szs` parameter, which is a dictionary containing any column names you want to set sizes for manually), and it sets a few other defaults. Other than that, it just creates the `TabularModel`, and passes that to `TabularLearner` (and note that `TabularLearner` is identical to `Learner`, except for a customized `predict` method).\n",
"\n",
"That means that really all the work is happening in `TabularModel`, so take a look at the source for that now. With the exception of the `BatchNorm1d` and `Dropout` layers (which we'll be learning about shortly) you now have the knowledge required to understand this whole class. Take a look at the discussion of `EmbeddingNN` at the end of the last chapter. Recall that it passed `n_cont=0` to `TabularModel`. We now can see why that was: because there are zero continuous variables (in fastai the `n_` prefix means \"number of\", and `cont` is an abbreviation for \"continuous\")."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Ensembling"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Think back to the original reasoning behind why random forests work so well: each tree has errors, but those errors are not correlated with each other, so the average of those errors should tend towards zero once there are enough trees. Similar reasoning could be used to consider averaging the predictions of models trained using different algorithms.\n",
"\n",
"In our case, we have two very different models, trained using very different algorithms: a random forest, and a neural network. It would be reasonable to expect that the kinds of errors that each one makes would be quite different. Therefore, we might expect that the average of their predictions would be better than either one's individual predictions.\n",
"\n",
"As we mentioned earlier in this chapter, the approach of combining multiple models' predictions together is called *ensembling*. A random forest is itself an ensemble. But we can then include a random forest in *another* ensemble--an ensemble of the random forest and the neural network! Whilst it is not going to make the difference between a successful and unsuccessful modelling process, it can certainly add a nice little boost to any models that you have built.\n",
"\n",
"One minor issue we have to be aware of is that our PyTorch model and our sklearn model create data of different types--PyTorch gives us a rank 2 tensor (i.e a column matrix), whereas numpy gives us a rank 1 array (a vector). `squeeze()` removes any unit axes from a tensor, and `to_np` converts it into a numpy array."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rf_preds = m.predict(valid_xs_time)\n",
"ens_preds = (to_np(preds.squeeze()) + rf_preds) /2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This gives us a better result than either model achieved on its own:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.22291"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"r_mse(ens_preds,valid_y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In fact, this result is better than any score shown on the Kaggle leaderboard. This is not directly comparable, however, because the Kaggle leaderboard uses a separate dataset that we do not have access to. Kaggle does not allow us to submit to this old competition, to find out how we would have gone, so we have no way to directly compare. But our results certainly look very encouraging!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Boosting"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So far our approach to ensembling has been to use *bagging*, which involves combining many models together by averaging them, where each model is trained on a different data subset. When this is applied to decision trees, this is called a *random forest*.\n",
"\n",
"There is another important approach to ensembling, called *boosting*, where we add models, instead of averaging them. Here is how it works:\n",
"\n",
"- Train a small model which under fits your dataset\n",
"- Calculate the predictions in the training set for this model\n",
"- Subtract the predictions from the targets; these are called the \"residuals\", and represent the error for each point in the training set\n",
"- Go back to step one, but instead of using the original targets, use the residuals as the target for the training\n",
"- Continue doing this until you reach some stopping criterion, such as a maximum number of trees, or you observe your validation set error getting worse.\n",
"\n",
"Using this approach, each new tree will be attempting to fit the error of all of the previous trees combined. Because we are continually creating new residuals, by subtracting the predictions of each new tree from the residuals from the previous tree, the residuals will get smaller and smaller.\n",
"\n",
"To make predictions with an ensemble of boosted trees, we calculate the predictions from each tree, and then add them all together. There are many models following this basic approach, and many names for the same models! *Gradient boosting machines* (GBMs) and *gradient boosted decision trees* (GBDTs) are the terms you're most likely to come across, or you may see the names of specific libraries implementing these; at the time of writing, *XGBoost* is the most popular.\n",
"\n",
"Note that, unlike random forests, there is nothing to stop us from overfitting. Using more trees in a random forest does not lead to overfitting, because each tree is independent of the others. But in a boosted ensemble, the more trees you have, the better the training error becomes, and eventually you will see overfitting on the validation set.\n",
"\n",
"We are not going to go into details as to how to train a gradient boosted tree ensemble here, because the field is moving rapidly, and any guidance we give will almost certainly be outdated by the time you read this! As we write this, sklearn has just added a `HistGradientBoostingRegressor` class, which provides excellent performance. There are many hyperparameters to tweak for this class, and for all gradient boosted tree methods we have seen. Unlike random forests, gradient boosted trees are extremely sensitive to the choices of these hyperparameters. So in practice, most people will use a loop which tries a range of different hyperparameters, to find which works best."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Combining embeddings with other methods"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The abstract of the entity embedding paper states: \"*the embeddings obtained from the trained neural network boost the performance of all tested machine learning methods considerably when used as the input features instead*\". It includes this very interesting table:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![\"Embeddings](\"images/att_00054.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is showing the mean average percent error (MAPE) compared amongst four different modelling techniques, three of which we have already seen, along with \"KNN\" (K nearest neighbours), which is a very simple baseline method. The first numeric column contains the results using just the methods on the data provided in the competition; the second column shows what happens if you first train a neural network with categorical embeddings, and then use those categorical embeddings instead of the raw categorical columns in the model. As you see, in every case, the models are dramatically improved by using the embeddings, instead of the raw category.\n",
"\n",
"This is a really important result, because it shows that you can get much of the performance improvement of a neural network, without actually having to use a neural network at all at inference time. You could just use an embedding, which is literally just an array lookup, along with a small decision tree ensemble.\n",
"\n",
"These embeddings need not even be necessarily learned separately for each model or task in an organisation. Instead, once a set of embeddings are learned for some column for some task, they could be stored in a central place, and reused across multiple models. In fact, we know from private communication with other practitioners at large companies that this is already happening in many places."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Our advice for tabular modeling"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have two approaches to tabular modelling: decision tree ensembles, and neural networks. And we have mentioned two different decision tree ensembles: random forests, and gradient boosting machines. Each is very effective, but each also has compromises:\n",
"\n",
"**Random forests** are the easiest to train, because they are extremely resilient to hyperparameter choices, and require very little preprocessing. They are very fast to train, and should not overfit, if you have enough trees. But, they can be a little less accurate, especially if extrapolation is required, such as predicting future time periods\n",
"\n",
"**Gradient boosting machines** in theory are just as fast to train as random forests, but in practice you will have to try lots of different hyperparameters. They can overfit. But they are often a little bit more accurate than random forests.\n",
"\n",
"**Neural networks** take the longest time to train, and require extra preprocessing such as normalisation; this normalisation needs to be used at inference time as well. They can provide great results, and extrapolate well, but only if you are careful with your hyperparameters, and are careful to avoid overfitting.\n",
"\n",
"We suggest starting your analysis with a random forest. This will give you a strong baseline, and you can be confident that it's a reasonable starting point. You can then use that model for feature selection and partial dependence analysis, to get a better understanding of your data.\n",
"\n",
"From that foundation, you can try neural nets and GBMs, and if they give you significantly better results on your validation set in a reasonable amount of time, you can use them. If decision tree ensembles are working well for you, try adding the embeddings for the categorical variables to the data, and see if that helps your decision trees learn better."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Questionnaire"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. What is a continuous variable?\n",
"1. What is a categorical variable?\n",
"1. Provide 2 of the words that are used for the possible values of a categorical variable.\n",
"1. What is a \"dense layer\"?\n",
"1. How do entity embeddings reduce memory usage and speed up neural networks?\n",
"1. What kind of datasets are entity embeddings especially useful for?\n",
"1. What are the two main families of machine learning algorithms?\n",
"1. Why do some categorical columns need a special ordering in their classes? How do you do this in pandas?\n",
"1. Summarize what a decision tree algorithm does.\n",
"1. Why is a date different from a regular categorical or continuous variable, and how can you preprocess it to allow it to be used in a model?\n",
"1. Should you pick a random validation set in the bulldozer competition? If no, what kind of validation set should you pick?\n",
"1. What is pickle and what is it useful for?\n",
"1. How are `mse`, `samples`, and `values` calculated in the decision tree drawn in this chapter?\n",
"1. How do we deal with outliers, before building a decision tree?\n",
"1. How do we handle categorical variables in a decision tree?\n",
"1. What is bagging?\n",
"1. What is the difference between `max_samples` and `max_features` when creating a random forest?\n",
"1. If you increase `n_estimators` to a very high value, can that lead to overfitting? Why or why not?\n",
"1. What is *out of bag error*?\n",
"1. Make a list of reasons why a model's validation set error might be worse than the OOB error. How could you test your hypotheses?\n",
"1. How can you answer each of these things with a random forest? How do they work?:\n",
" - How confident are we in our projections using a particular row of data?\n",
" - For predicting with a particular row of data, what were the most important factors, and how did they influence that prediction?\n",
" - Which columns are the strongest predictors?\n",
" - How do predictions vary, as we vary these columns?\n",
"1. What's the purpose of removing unimportant variables?\n",
"1. What's a good type of plot for showing tree interpreter results?\n",
"1. What is the *extrapolation problem*?\n",
"1. How can you tell if your test or validation set is distributed in a different way to your training set?\n",
"1. Why do we make `saleElapsed` a continuous variable, even although it has less than 9000 distinct values?\n",
"1. What is boosting?\n",
"1. How could we use embeddings with a random forest? Would we expect this to help?\n",
"1. Why might we not always use a neural net for tabular modeling?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Further research"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Pick a competition on Kaggle with tabular data (current or past) and try to adapt the techniques seen in this chapter to get the best possible results. Compare yourself to the private leaderboard.\n",
"1. Implement the decision tree algorithm in this chapter from scratch yourself, and try it on this dataset.\n",
"1. Use the embeddings from the neural net in this chapter in a random forest, and see if you can improve on the random forest results we saw.\n",
"1. Explain what each line of the source of `TabularModel` does (with the exception of the `BatchNorm1d` and `Dropout` layers)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"jupytext": {
"split_at_heading": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
| epoch | \n", "train_loss | \n", "valid_loss | \n", "time | \n", "
|---|---|---|---|
| 0 | \n", "0.069705 | \n", "0.062389 | \n", "00:11 | \n", "
| 1 | \n", "0.056253 | \n", "0.058489 | \n", "00:11 | \n", "
| 2 | \n", "0.048385 | \n", "0.052256 | \n", "00:11 | \n", "
| 3 | \n", "0.043400 | \n", "0.050743 | \n", "00:11 | \n", "
| 4 | \n", "0.040358 | \n", "0.050986 | \n", "00:11 | \n", "
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is showing the mean average percent error (MAPE) compared amongst four different modelling techniques, three of which we have already seen, along with \"KNN\" (K nearest neighbours), which is a very simple baseline method. The first numeric column contains the results using just the methods on the data provided in the competition; the second column shows what happens if you first train a neural network with categorical embeddings, and then use those categorical embeddings instead of the raw categorical columns in the model. As you see, in every case, the models are dramatically improved by using the embeddings, instead of the raw category.\n",
"\n",
"This is a really important result, because it shows that you can get much of the performance improvement of a neural network, without actually having to use a neural network at all at inference time. You could just use an embedding, which is literally just an array lookup, along with a small decision tree ensemble.\n",
"\n",
"These embeddings need not even be necessarily learned separately for each model or task in an organisation. Instead, once a set of embeddings are learned for some column for some task, they could be stored in a central place, and reused across multiple models. In fact, we know from private communication with other practitioners at large companies that this is already happening in many places."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Our advice for tabular modeling"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have two approaches to tabular modelling: decision tree ensembles, and neural networks. And we have mentioned two different decision tree ensembles: random forests, and gradient boosting machines. Each is very effective, but each also has compromises:\n",
"\n",
"**Random forests** are the easiest to train, because they are extremely resilient to hyperparameter choices, and require very little preprocessing. They are very fast to train, and should not overfit, if you have enough trees. But, they can be a little less accurate, especially if extrapolation is required, such as predicting future time periods\n",
"\n",
"**Gradient boosting machines** in theory are just as fast to train as random forests, but in practice you will have to try lots of different hyperparameters. They can overfit. But they are often a little bit more accurate than random forests.\n",
"\n",
"**Neural networks** take the longest time to train, and require extra preprocessing such as normalisation; this normalisation needs to be used at inference time as well. They can provide great results, and extrapolate well, but only if you are careful with your hyperparameters, and are careful to avoid overfitting.\n",
"\n",
"We suggest starting your analysis with a random forest. This will give you a strong baseline, and you can be confident that it's a reasonable starting point. You can then use that model for feature selection and partial dependence analysis, to get a better understanding of your data.\n",
"\n",
"From that foundation, you can try neural nets and GBMs, and if they give you significantly better results on your validation set in a reasonable amount of time, you can use them. If decision tree ensembles are working well for you, try adding the embeddings for the categorical variables to the data, and see if that helps your decision trees learn better."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Questionnaire"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. What is a continuous variable?\n",
"1. What is a categorical variable?\n",
"1. Provide 2 of the words that are used for the possible values of a categorical variable.\n",
"1. What is a \"dense layer\"?\n",
"1. How do entity embeddings reduce memory usage and speed up neural networks?\n",
"1. What kind of datasets are entity embeddings especially useful for?\n",
"1. What are the two main families of machine learning algorithms?\n",
"1. Why do some categorical columns need a special ordering in their classes? How do you do this in pandas?\n",
"1. Summarize what a decision tree algorithm does.\n",
"1. Why is a date different from a regular categorical or continuous variable, and how can you preprocess it to allow it to be used in a model?\n",
"1. Should you pick a random validation set in the bulldozer competition? If no, what kind of validation set should you pick?\n",
"1. What is pickle and what is it useful for?\n",
"1. How are `mse`, `samples`, and `values` calculated in the decision tree drawn in this chapter?\n",
"1. How do we deal with outliers, before building a decision tree?\n",
"1. How do we handle categorical variables in a decision tree?\n",
"1. What is bagging?\n",
"1. What is the difference between `max_samples` and `max_features` when creating a random forest?\n",
"1. If you increase `n_estimators` to a very high value, can that lead to overfitting? Why or why not?\n",
"1. What is *out of bag error*?\n",
"1. Make a list of reasons why a model's validation set error might be worse than the OOB error. How could you test your hypotheses?\n",
"1. How can you answer each of these things with a random forest? How do they work?:\n",
" - How confident are we in our projections using a particular row of data?\n",
" - For predicting with a particular row of data, what were the most important factors, and how did they influence that prediction?\n",
" - Which columns are the strongest predictors?\n",
" - How do predictions vary, as we vary these columns?\n",
"1. What's the purpose of removing unimportant variables?\n",
"1. What's a good type of plot for showing tree interpreter results?\n",
"1. What is the *extrapolation problem*?\n",
"1. How can you tell if your test or validation set is distributed in a different way to your training set?\n",
"1. Why do we make `saleElapsed` a continuous variable, even although it has less than 9000 distinct values?\n",
"1. What is boosting?\n",
"1. How could we use embeddings with a random forest? Would we expect this to help?\n",
"1. Why might we not always use a neural net for tabular modeling?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Further research"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Pick a competition on Kaggle with tabular data (current or past) and try to adapt the techniques seen in this chapter to get the best possible results. Compare yourself to the private leaderboard.\n",
"1. Implement the decision tree algorithm in this chapter from scratch yourself, and try it on this dataset.\n",
"1. Use the embeddings from the neural net in this chapter in a random forest, and see if you can improve on the random forest results we saw.\n",
"1. Explain what each line of the source of `TabularModel` does (with the exception of the `BatchNorm1d` and `Dropout` layers)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"jupytext": {
"split_at_heading": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}