"Now that we understand what deep learning is, what it's for, and how to create and deploy a model, it's time for us to go deeper! In an ideal world deep learning practitioners wouldn't have to know every detail of how things work under the hood… But as yet, we don't live in an ideal world. The truth is, to make your model really work, and work reliably, there's a lot of details you have to get right. And a lot of details that you have to check. This process requires being able to look inside your neural network as it trains, and as it makes predictions, find possible problems, and know how to fix them.\n",
"\n",
"So, from here on in the book we are going to do a deep dive into the mechanics of deep learning. What is the architecture of a computer vision model, an NLP model, a tabular model, and so on. How do you create an architecture which matches the needs of your particular domain? How do you get the best possible results from the training process? How do you make things faster? What do you have to change as your datasets change?\n",
"\n",
"We will start by repeating the same basic applications that we looked at in the first chapter, but we are going to do two things:\n",
"\n",
"- make them better;\n",
"- apply them to a wider variety of types of data.\n",
"In order to do these two things, we will have to learn all of the pieces of the deep learning puzzle. This includes: different types of layers, regularisation methods, optimisers, putting layers together into architectures, labelling techniques, and much more. We are not just going to dump all of these things out, but we will introduce them progressively as needed, to solve an actual problem related to the project we are working on."
"In our very first model we learnt how to classify dogs versus cats. Just a few years ago this was considered a very challenging task. But today, it is far too easy! We will not be able to show you the nuances of training models with this problem, because we get the nearly perfect result without worrying about any of the details. But it turns out that the same dataset also allows us to work on a much more challenging problem: figuring out what breed of pet is shown in each image.\n",
"In the first chapter we presented the applications as already solved problems. But this is not how things work in real life. We start with some dataset which we know nothing about. We have to understand how it is put together, how to extract the data we need from it, and what that data looks like. For the rest of this book we will be showing you how to solve these problems in practice, including all of these intermediate steps necessary to understand the data that we are working with and test our modelling as we go.\n",
"Now if we are going to understand how to extract the breed of each pet from each image we're going to need to understand how this data is laid out. Such details of data layout are a vital piece of the deep learning puzzle. Data is usually provided in one of these two ways:\n",
"- Individual files representing items of data, such as text documents or images, possibly organised into folders or with filenames representing information about those items, or\n",
"- A table of data, such as in CSV format, where each row is an item, each row which may include filenames providing a connection between the data in the table and data in other formats such as text documents and images.\n",
"There are exceptions to these rules, particularly in domains such as genomics, where there can be binary database formats or even network streams, but overall the vast majority of the datasets you work with use some combination of the above two formats.\n",
"We can see that this dataset provides us with \"images\" and \"annotations\" directories. The website for this dataset tells us that the annotations directory contains information about where the pets are rather than what they are. In this chapter, we will be doing classification, not localization, which is to say that we care about what the pets are not where they are. Therefore we will ignore the annotations directory for now. So let's have a look inside the images directory:"
"Most functions and methods in fastai which return a collection use a class called `L`. `L` can be thought of as an enhanced version of the ordinary Python `list` type, with added conveniences for common operations. For instance, when we display an object of this class in a notebook it appears in the format you see above. The first thing that is shown is the number of items in the collection, prefixed with a `#`. You'll also see in the above output that the list is suffixed with a \"…\". This means that only the first few items are displayed — which is a good thing, because we would not want more than 7000 filenames on our screen!\n",
"By examining these filenames, we see how they appear to be structured. Each file name contains the pet breed, and then an _ character, a number, and finally the file extension. We need to create a piece of code that extracts the breed from a single `Path`. Jupyter notebook makes this easy, because we can gradually build up something that works, and then use it for the entire dataset. We do have to be careful to not make too many assumptions at this point. For instance, if you look carefully you may notice that some of the pet breeds contain multiple words, so we cannot simply break at the first `_` character that we find. To allow us to test our code, let's pick out one of these filenames:"
"The most powerful and flexible way to extract information from strings like this is to use a *regular expression*, also known as a *regex*. A regular expression is a special string, written in the regular expression language, which specifies a general rule for deciding if another string passes a test (i.e., \"matches\" the regular expression), and also possibly for plucking a particular part or parts out of that other string. \n",
"We do not have the space to give you a complete regular expression tutorial here, particularly because there are so many excellent ones online. And we know that many of you will already be familiar with this wonderful tool. If you're not, that is totally fine — this is a great opportunity for you to rectify that! We find that regular expressions are one of the most useful tools in our programming toolkit, and many of our students tell us that it is one of the things they are most excited to learn about. So head over to Google and search for *regular expressions tutorial* now, and then come back here after you've had a good look around. The book website also provides a list of our favorites.\n",
"> a: Not only are regular expressions dead handy, they also have interesting roots. They are \"regular\" because they were originally examples of a \"regular\" language, the lowest rung within the \"Chomsky hierarchy\", a grammar classification due to the same linguist Noam Chomsky who wrote _Syntactic Structures_, the pioneering work searching for the formal grammar underlying human language. This is one of the charms of computing: it may be that the hammer you reach for every day in fact came from a space ship.\n",
"When you are writing a regular expression, the best way to start is just to try it against one example at first. Let's use the `findall` method to try a regular expression against the filename of the `fname` object:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['great_pyrenees']"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"re.findall(r'(.+)_\\d+.jpg$', fname.name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This regular expression plucks out all the characters leading up to the last underscore character, as long as the subsequence characters are numerical digits and then the jpeg file extension.\n",
"\n",
"Now that we confirmed the regular expression works for the example, let's use it to label the whole dataset. Fastai comes with many classes to help you with your labelling. For labelling with regular expressions, we can use the `RegexLabeller` class. We can use this in the data block API that we saw in <<chapter_production>> (in fact, we nearly always use the data block API--it's so much more flexible than the simple factory methods we saw in <<chapter_intro>>):"
"These lines implement a fastai data augmentation strategy which we call *presizing*. Presizing is a particular way to do image augmentation, which is designed to minimize data destruction while maintaining good performance."
"We need our images to have the same dimensions, so that they can collate into tensors to be passed to the GPU. We also want to minimize the number of distinct augmentation computations we perform. So the performance requirement suggests that we should, where possible, compose our augmentation transforms into fewer transforms (to reduce the number of computations, and reduce the number of lossy operations) and transform the images into uniform sizes (to run compute efficiently on the GPU).\n",
"\n",
"The challenge is that, if performed after resizing down to the augmented size, various common data augmentation transforms might introduce spurious empty zones, degrade data, or both. For instance, rotating an image by 45 degrees fills corner regions of the new bounds with emptyness, which will not teach the model anything. Many rotation and zooming operations will require interpolating to create pixels. These interpolated pixels are derived from the original image data but are still of lower quality.\n",
"1. First, resizing images to relatively \"large dimensions\" that is, dimensions significantly larger than the target training dimensions. \n",
"1. Second, composing all of the common augmentation operations (including a resize to the final target size) into one, and performing the combined operation on the GPU only once at the end of processing, rather than performing them individually and interpolating multiple times.\n",
"The first step, the resize, creates images large enough that they have spare margin to allow further augmentation transforms on their inner regions without creating empty zones. This transformation works by resizing to a square, using a large crop size. On the training set, the crop area is chosen randomly, and the size of the crop is selected to cover the entire width or height of the image, whichever is smaller.\n",
"In the second step, the GPU is used for all data augmentation, and all of the potentially destructive operations are done together, with a single interpolation at the end."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img alt=\"Presizing on the training set\" width=\"600\" caption=\"Presizing on the training set\" id=\"presizing\" src=\"images/att_00060.png\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This picture shows the two steps:\n",
"\n",
"1. *Crop full width or height*: This is in `item_tfms`, so it's applied to each individual image before it is copied to the GPU. It's used to ensure all images are the same size. On the training set, the crop area is chosen randomly. On the validation set, the center square of the image is always chosen\n",
"2. *Random crop and augment*: This is in `batch_tfms`, so it's applied to a batch all at once on the GPU, which means it's fast. On the validation set, only the resize to the final size needed for the model is done here. On the training set, the random crop and any other augmentation is done first.\n",
"\n",
"To implement this process in fastai you use `Resize` as an item transform with a large size, and `RandomResizedCrop` as a batch transform with a smaller size. `RandomResizedCrop` will be added for you if you include the `min_scale` parameter in your `aug_transform` function, as you see in the `DataBlock` call above. Alternatively, you can use `pad` or `squish` instead of `crop` (the default) for the initial `Resize`.\n",
"\n",
"You can see in this example the difference between an image which has been zoomed, interpolated, rotated, and then interpolated again on the right (which is the approach used by all other deep learning libraries), compared to an image which has been zoomed and rotated as one operation, and then interpolated just once on the left (the fastai approach):"
"You can see here that the image on the right is less well defined, and has reflection padding artifacts in the bottom left, and the grass in the top left has disappeared entirely. We find that in practice using presizing significantly improves the accuracy of models, and often results in speedups too.\n",
"\n",
"Checking your data looks right is extremely important before training a model. There are simple ways to do this (and debug if needed) in the fastai library, let's look at them now."
"We can never just assume that our code is working perfectly. Writing a `DataBlock` is just like writing a blueprint. You will get an error message if you have a syntax error somewhere in your code but you have no guaranty that your template is going to work on your source of data as you intend. The first thing to do before we trying to train a model is to use the `show_batch` method and have a look at your data:"
"Have a look at each image, and check that each one seems to have the correct label for that breed of pet. Often, data scientists work with data with which they are not as familiar as domain experts may be: for instance, I actually don't know what a lot of these pet breeds are. Since I am not an expert on pet breeds, I would use Google images at this point to search for a few of these breeds, and make sure the images looks similar to what I see in this output.\n",
"If you made a mistake while building your `DataBlock` it is very likely you won't see it before this step. To debug this, we encourage you to use the `summary` method. It will attempt to create a batch from the source you give it, with a lot of details. Also, if it fails, you will see exactly at which point the error happens, and the library will try to give you some help. For instance, one common mistake is to forget to put a `Resize` transform, ending up with pictures of different sizes and not able to batch them. Here is what the summary would look like in that case (note that the exact text may have changed since the time of writing, but it will give you an idea):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Setting-up type transforms pipelines\n",
"Collecting items from /home/jhoward/.fastai/data/oxford-iiit-pet/images\n",
"evalue": "invalid argument 0: Sizes of tensors must match except in dimension 0. Got 414 and 375 in dimension 2 at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/TH/generic/THTensor.cpp:612",
"\u001b[0;31mRuntimeError\u001b[0m: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 414 and 375 in dimension 2 at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/TH/generic/THTensor.cpp:612"
"You can see exactly how we gathered the data and split it, how we went from a filename to a *sample* (the tuple image, category), then what item transforms were applied and how it failed to collate those samples in a batch (because of the different shapes). \n",
"\n",
"Once you think your data looks right, we generally recommend the next step should be creating a simple model. We often see people procrastinate the training of an actual model for far too long. As a result, they don't actually get to find out what their baseline results look like. Perhaps it doesn't need lots of fancy domain specific engineering. Or perhaps the data doesn't seem to train it all. These are things that you want to know as soon as possible. So we will use the same simple model that we used in <<chapter_intro>>:"
"As we've briefly discussed before, the table shown when we fit a model shows us the results after each epoch of training. Remember, an epoch is one complete pass through all of the images in the data. The columns shown are the average loss over the items of the training set, the loss on the validation set, and any metrics that you requested — in this case, the error rate.\n",
"\n",
"Remember that *loss* is whatever function we've decided to use to optimise the parameters of our model. But we haven't actually told fastai what loss function we want to use. So what is it doing? Fastai will generally try to select an appropriate loss function based on what kind of data and model you are using. In this case you have image data, and a categorical outcome, so fastai will default to using *cross entropy loss*."
"*Cross entropy loss* is a loss function which is similar to the loss function we used in the previous chapter, but (as we'll see) has two benefits:\n",
"\n",
"- It works even when our dependent variable has more than two categories\n",
"- It results in faster and more reliable training.\n",
"In order to understand how cross entropy loss works for dependent variables with more than two categories, we first have to understand what the actual data and activations that are seen by the loss function look like."
"As you see, this returns the dependent, and the independent variables, as a mini-batch. Let's see what is actually contained in our dependent variable:"
"Our batch size is 64, so we have 64 rows in this tensor. Each row is a single integer between zero and 36, representing our 37 possible pet breeds. We can view the predictions (that is, the activations of the final layer of our neural network) using `Learner.get_preds`. This function either takes a dataset index (0 for train and 1 for valid) or an iterator of batches. Thus, we can pass it a simple list with our batch to get our predictions. It returns predictions and targets by default, but since we already have the targets, we can effectively ignore them by assigning to the special variable `_`:"
"In our classification model, an activation function called *softmax* in the final layer is used to ensure that the activations are between zero and one, and that they sum to one.\n",
"We can apply this function to a single column of activations from a neural network, and get back a column of numbers between zero and one. So it's a very useful activation function for our final layer.\n",
"\n",
"Now think about what happens if we want to have more categories in our target (such as our 37 pet breeds). That means we'll need more activations than just a single column: we need an activation *per category*. We can create, for instance, a neural net that predicts \"3\"s and \"7\"s that returns two activations, one for each class--this will be a good first step towards creating the more general approach. Let's just use some random numbers with a standard deviation of 2 (so we multiply `randn` by 2) for this example, assuming we have six images and two possible categories (where the first columns represents \"3\"s and the second is \"7\"s):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#hide\n",
"torch.random.manual_seed(42);"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[ 0.6734, 0.2576],\n",
" [ 0.4689, 0.4607],\n",
" [-2.2457, -0.3727],\n",
" [ 4.4164, -1.2760],\n",
" [ 0.9233, 0.5347],\n",
" [ 1.0698, 1.6187]])"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"acts = torch.randn((6,2))*2\n",
"acts"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can't just take the sigmoid of this directly, since we don't get rows that add to one (i.e we want the probability of being a \"3\" plus the probability of being a \"7\" to add to one):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[0.6623, 0.5641],\n",
" [0.6151, 0.6132],\n",
" [0.0957, 0.4079],\n",
" [0.9881, 0.2182],\n",
" [0.7157, 0.6306],\n",
" [0.7446, 0.8346]])"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"acts.sigmoid()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In <<chapter_mnist_basics>>, the neural net created a single activation per image, which we passed through the sigmoid function. That single activation represented the confidence that the input was a \"3\". Binary problems are a special case of classification problems, because the target can be treated as a single boolean value, as we did in `mnist_loss`. Binary problems can also be thought of as part of the more general group of classifiers with any number of categories--where in this case we happen to have 2 categories. As we saw in the bear classifier, our neural net will return one activation per category.\n",
"So in the binary case, what do those activations really indicate? A single pair of activations simply indicates the *relative* confidence of being a \"3\" versus being a \"7\". The overall values, whether they are both high, or both low, don't matter--all that matters is which is higher, and by how much.\n",
"We would expect that since this is just another way of representing the same problem (in the binary case) that we would be able to use sigmoid directly on the two-activation version of our neural net. And indeed we can! We can just take the *difference* between the neural net activations, because that reflects how much more sure we are of being a \"3\" vs a \"7\", and then take the sigmoid of that:"
"The second column (the probability of being a \"7\") will then just be that subtracted from one. We need a way to do all this that also works for more than two columns. It turns out that this function, called `softmax`, is exactly that:\n",
"> jargon: Exponential function (exp): Literally defined as `e**x`, where `e` is a special number approximately equal to 2.718. It is the inverse of the natural logarithm function. Note that `exp` is always positive, and it increases *very* rapidly!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's check that `softmax` returns the same values as `sigmoid` for the first column, and that subtracted from one for the second column:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[0.6025, 0.3975],\n",
" [0.5021, 0.4979],\n",
" [0.1332, 0.8668],\n",
" [0.9966, 0.0034],\n",
" [0.5959, 0.4041],\n",
" [0.3661, 0.6339]])"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sm_acts = torch.softmax(acts, dim=1)\n",
"sm_acts"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Softmax is the multi-category equivalent of sigmoid--we have to use it any time we have more than two categories, and the probabilities of the categories must add to one. (We often use it even when there's just two categories, just to make things a bit more consistent.) We could create other functions that have the properties that all activations are between zero and one, and sum to one; however, no other function has the same relationship to the sigmoid function, which we've seen is smooth and symmetric. Also, we'll see shortly that the softmax function works well hand-in-hand with the loss function we will look at in the next section.\n",
"\n",
"If we have three output activations, such as in our bear classifier, calculating softmax for a single bear image would then look like something like this:"
"What does this function do in practice? Taking the exponential ensures all our numbers are positive, and then dividing by the sum ensures we are going to have a bunch of numbers that add up to one. The exponential also has a nice property: if one of the numbers in our activations `x` is slightly bigger than the others, the exponential will amplify this (since it grows, well... exponentially) which means that in the softmax, that number will be closer to 1. \n",
"Intuitively, the Softmax function *really* wants to pick one class among the others, so it's ideal for training a classifier when we know each picture has a definite label. (Note that it may be less ideal during inference, as you might want your model to sometimes tell you it doesn't recognize any of the classes that it has seen during training, and not pick a class because it has a slightly bigger activation score. In this case, it might be better to train a model using multiple binary output columns, each using a sigmoid activation.)\n",
"Just like we moved from sigmoid to softmax, we need to extend the loss function to work with more than just binary classification, to classifying any number of categories (in this case, we have 37 categories). Our activations, after softmax, are between zero and one, and sum to one for each row in the batch of predictions. Our targets are integers between 0 and 36.\n",
"\n",
"In the binary case, we used `torch.where` to select between `inputs` and `1-inputs`. When we treat a binary classification as a general classification problem with two categories, it actually becomes even easier, because (as we saw in the softmax section) we now have two columns, containing the equivalent of `inputs` and `1-inputs`. So all we need to do is select from the appropriate column. Let's try to implement this in PyTorch. For our synthetic \"3\"s and \"7\" example, let's say these are our labels:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"targ = tensor([0,1,0,1,1,0])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"...and these are the softmax activations:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[0.6025, 0.3975],\n",
" [0.5021, 0.4979],\n",
" [0.1332, 0.8668],\n",
" [0.9966, 0.0034],\n",
" [0.5959, 0.4041],\n",
" [0.3661, 0.6339]])"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sm_acts"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then for each item of `targ` we can use that to select that column of `sm_acts` using tensor indexing, like so:"
"To see exactly what's happening here, let's put all the columns together in a table. Here, the first two columns are our activations, then we have the targets, the row index, and finally the result shown immediately above:"
"Looking at this table, you can see that the final column can be calculated by taking the `targ` and `idx` columns as indices into the 2-column matrix containing the `3` and `7` columns. That's what `sm_acts[idx, targ]` is actually doing.\n",
"The really interesting thing here is that this actually works just as well with more than two columns. To see this, consider what would happen if we added an activation column above for every digit (zero through nine), and then `targ` contained a number from zero to nine. As long as the activation columns sum to one (as they will, if we use softmax), then we'll have a loss function that shows how well we're predicting each digit.\n",
"We're only picking the loss from the column containing the correct label. We don't need to consider the other columns, because by the definition of softmax, they add up to one minus the activation corresponding to the correct label. Therefore, making the activation for the correct label as high as possible, must mean we're also decreasing the activations of the remaining columns.\n",
"PyTorch provides a function that does exactly the same thing as `sm_acts[range(n), targ]` (except it takes the negative, because when applying the log afterward, we will have negative numbers), called `nll_loss` (*NLL* stands for *negative log likelihood*):"
"Despite the name being negative log likelihood, this PyTorch function does not take the log (we will see why in the next section). First, let's see why taking the logarithm can be useful."
"This does work quite well as a loss function, but we can make it a bit better. The problem is that we are using probabilities, and probabilities cannot be smaller than zero, or greater than one. But that means that our model will not care about whether it predicts 0.99 versus 0.999, because those numbers are so close together. But in another sense, 0.999 is 10 times more confident than 0.99. So we wish to transform our numbers between zero and one to instead be between negative infinity and infinity. There is a function available in maths which does exactly this: the logarithm (available as `torch.log`). It is not defined for numbers less than zero, and looks like this:"
"In this case, we're assuming that `log(y,b)` returns *log y base b*. However, PyTorch actually doesn't define `log` this way: `log` in Python uses the special number `e` (2.718...) as the base.\n",
"\n",
"Perhaps a logarithm is something that you have not thought about for the last 20 years or so. But it's a mathematical idea which is going to be really critical for many things in deep learning, so now would be a great time to refresh your memory. The key thing to know about logarithms is this relationship:\n",
"When we see it in that format, it looks a bit boring; but have a think about what this really means. It means that logarithms increase linearly when the underlying signal increases exponentially or multiplicatively. This is used for instance in the Richter scale of earthquake severity, and the dB scale of noise levels. It's also often used on financial charts, where we want to show compound growth rates more clearly. Computer scientists love using logarithms, because it means that modification, which can create really really large and really really small numbers, can be replaced by addition, which is much less likely to result in scales which are difficult for our computer to handle."
"> s: It's not just computer scientists that love logs! Until computers came along, engineers and scientists used a special ruler called a \"slide rule\" that did multiplication by adding logarithms. Logarithms are widely used in physics, for multiplying very big or very small numbers, and many other fields."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Taking the mean of the positive or negative log of our probabilities (depending on whether it's the correct or incorrect class) gives us the *negative log likelihood* loss. In PyTorch, `nll_loss` assumes that you already took the log of the softmax, so it doesn't actually do the logarithm for you."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> warning: The \"NLL\" in \"nll_loss\" stands for \"negative log likelihood\", but it doesn't actually take the log at all! It assumes you have _already_ taken the log. PyTorch has a function called \"log_softmax\" which combines \"log\" and \"softmax\" in a fast and accurate way."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When we first take the softmax, and then the log likelihood of that, that combination is called *cross entropy loss*. In PyTorch, this is available as `nn.CrossEntropyLoss` (which, in practice, actually does `log_softmax` and then `nll_loss`)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"loss_func = nn.CrossEntropyLoss()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you see, this is a class. Instantiating it gives you an object which behaves like a function:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor(1.8045)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"loss_func(acts, targ)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All PyTorch loss functions are provided in two forms: the class form seen above, and also a plain functional form, available in the `F` namespace:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor(1.8045)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"F.cross_entropy(acts, targ)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Either one works fine and can be used in any situation. We've noticed that most people tend to use the class version, and that's more often used in PyTorch official docs and examples, so we'll tend to use that too.\n",
"\n",
"By default PyTorch loss functions take the mean of the loss of all items. You can use `reduction='none'` to disable that:"
"> s: An interesting feature about cross entropy loss appears when we consider its gradient. The gradient of `cross_entropy(a,b)` is just `softmax(a)-b`. Since `softmax(a)` is just the final activation of the model, that means that the gradient is proportional to the difference between the prediction and the target. This is the same as mean squared error in regression (assuming there's no final activation function such as that added by `y_range`), since the gradient of `(a-b)**2` is `2*(a-b)`. Since the gradient is linear, that means that we won't see sudden jumps or exponential increases in gradients, which should lead to smoother training of models."
"We have now seen all the pieces hidden behind our loss function. While it gives us a number on how well (or bad) our model is doing, it does nothing to help us know if it's actually any good. Let's now see some ways to interpret our model predictions."
"It's very hard to interpret loss functions directly, because they are designed to be things which computers can differentiate and optimise, not things that people can understand. That's why we have metrics. These are not used in the optimisation process, but just used to help us poor humans understand what's going on. In this case, our accuracy is looking pretty good already! So where are we making mistakes?\n",
"\n",
"We saw in <<chapter_intro>> that we can use a confusion matrix to see where our model is doing well, and where it's doing badly:"
"Oh dear, in this case, a confusion matrix is very hard to read. We have 37 different breeds of pet, which means we have 37×37 entries in this giant matrix! Instead, we can use the `most_confused` method, which just shows us the cells of the confusion matrix with the most incorrect predictions (here with at least 5 or more):"
"Since we are not pet breed experts, it is hard for us to know whether these category errors reflect actual difficulties in recognising breeds. So again, we turn to Google. A little bit of googling tells us that the most common category errors shown here are actually breed differences which even expert breeders sometimes disagree about. So this gives us some comfort that we are on the right track.\n",
"We will now look at a range of techniques to improve the training of our model and make it better. While doing so, we will explain a little bit more about transfer learning and how to fine-tune our pretrained model as best as possible, without breaking the pretrained weights.\n",
"The first thing we need to set when training a model is the learning rate. We saw in the previous chapter that it needed to be just right to train as efficiently as possible, so how do we pick a good one? fastai provides something called the Learning rate finder for this."
"One of the most important things we can do when training a model is to make sure that we have the right learning rate. If our learning rate is too low, it can take many many epochs. Not only does this waste time, but it also means that we may have problems with overfitting, because every time we do a complete pass through the data, we give our model a chance to memorise it.\n",
"That did not look good. Here's what happened. The optimiser stepped in the correct direction, but it stepped so far that it totally overshot the minimum loss. Repeating that multiple times makes it get further and further away, not closer and closer!\n",
"What do we do to find the perfect learning rate, not too high, and not too low? In 2015 the researcher Leslie Smith came up with a brilliant idea, called the *learning rate finder*. His idea was to start with a very very small learning rate, something so small that we would never expect it to be too big to handle. We use that for one mini batch, find what the losses are afterwards, and then increase the learning rate by some percentage (e.g. doubling it each time). Then we do another mini batch, track the loss, and double the learning rate again. We keep doing this until the loss gets worse, instead of better. This is the point where we know we have gone too far. We then select a learning rate a bit lower than this point. Our advice is to pick either:\n",
"The Learning Rate Finder computes those points on the curve to help you. Both these rules usually give around the same value. In the first chapter, we didn't specify a learning rate, using the default value from the fastai library (which is 1e-3)."
"We can see on this plot that in the range 1e-6 to 1e-3, nothing really happens and the model doesn't train. Then the loss starts to decrease until it reaches a minimum and then increases again. We don't want a learning rate greater than 1e-1 as it will give a training that diverges (you can try for yourself) but 1e-1 is already too high: at this stage we left the period where the loss was decreasing steadily.\n",
"In this learning rate plot it appears that a learning rate around 3e-3 would be appropriate, so let's choose that."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> Note: The learning rate finder plot has a logarithmic scale, which is why the middle point between 1e-3 and 1e-2 is between 3e-3 and 4e-3. This is because we care mostly about the order of magnitude of the learning rate."
"Something really interesting about the learning rate finder is that it was only discovered in 2015. Neural networks have been under development since the 1950s. Throughout that time finding a good learning rate has been, perhaps, the most important and challenging issue for practitioners. The idea does not require any advanced maths, giant computing resources, huge datasets, or anything else that would make it inaccessible to any curious researcher. Furthermore, Leslie Smith, was not part of some exclusive Silicon Valley lab, but was working as a naval researcher. All of this is to say: breakthrough work in deep learning absolutely does not require access to vast resources, elite teams, or advanced mathematical ideas. There is lots of work still to be done which requires just a bit of common sense, creativity, and tenacity."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have a good learning rate to train our model, let's look at how we can finetune the weights of a pretrained model."
"We discussed briefly in <<chapter_intro>> how transfer learning works. We saw that the basic idea is that a pretrained model, trained potentially on millions of data points (such as ImageNet), is fine tuned for some other task. But what does this really mean?\n",
"We now know that a convolutional neural network consists of many layers with a non-linear activation function between each and one or more final linear layers, with an activation function such as softmax at the very end. The final linear layer uses a matrix with enough columns such that the output size is the same as the number of classes in our model (assuming that we are doing classification).\n",
"This final linear layer is unlikely to be of any use for us, when we are fine tuning in a transfer learning setting, because it is specifically designed to classify the categories in the original pretraining dataset. So when we do transfer learning we remove it, throw it away, and replace it with a new linear layer with the correct number of outputs for our desired task (in this case, there would be 37 activations).\n",
"This newly added linear layer will have entirely random weights. Therefore, our model prior to fine tuning has entirely random outputs. But that does not mean that it is an entirely random model! All of the layers prior to the last one have been carefully trained to be good at image classification tasks in general. As we saw in the images from the Zeiler and Fergus paper in <<chapter_intro>> (see <<img_layer1>> and followings), the first few layers encode very general concepts such as finding gradients and edges, and later layers encode concepts that are still very useful for us, such as finding eyeballs and fur.\n",
"We want to train a model in such a way that we allow it to remember all of these generally useful ideas from the pretrained model, use them to solve our particular task (classify pet breeds), and only adjust them as required for the specifics of our particular task.\n",
"Our challenge when fine tuning is to replace the random weights in our added linear layers with weights that correctly achieve our desired task (classifying pet breeds) without breaking the carefully pretrained weights and the other layers. There is actually a very simple trick to allow this to happen: tell the optimiser to only update the weights in those randomly added final layers. Don't change the weights in the rest of the neural network at all. This is called *freezing* those pretrained layers."
"When we create a model from a pretrained network fastai automatically freezes all of the pretrained layers for us. When we call the `fine_tune` method fastai does two things:\n",
"\n",
"- train the randomly added layers for one epoch, with all other layers frozen ;\n",
"- unfreeze all of the layers, and train them all for the number of epochs requested.\n",
"Although this is a reasonable default approach, it is likely that for your particular dataset you may get better results by doing things slightly differently. The `fine_tune` method has a number of parameters you can use to change its behaviour, but it might be easiest for you to just call the underlying methods directly if you want to get some custom behavior. Remember that you can see the source code for the method by using the following syntax:\n",
"So let's try doing this manually ourselves. First of all we will train the randomly added layers for three epochs, using `fit_one_cycle`. As mentioned in <<chapter_intro>>, `fit_one_cycle` is the suggested way to train models without using `fine_tune`. We'll see why later in the book; in short, what `fit_one_cycle` does is to start training at a low learning rate, gradually increase it for the first section of training, and then gradually decrease it again for the last section of training."
"...and run `lr_find` again, because having more layers to train, and weights that have already been trained for 3 epochs, means our previously found learning rate isn't appropriate any more:"
"Note that the graph is a little different from when we had random weights: we don't have that sharp descent that indicates the model is training. That's because our model has been trained already. Here we have a somewhat flat area before a sharp increase, and we should take a point well before that sharp increase, for instance 1e-5. The point with the maximum gradient isn't what we look for here and should be ignored.\n",
"This has improved our model a bit, but there's more we can do. The deepest layers of our pretrained model might not need as high a learning rate as the last ones, so we should probably use different learning rates for those, something called discriminative learning rates."
"Even after we unfreeze, we still care a lot about the quality of those pretrained weights. We would not expect that the best learning rate for those pretrained parameters would be as high as the randomly added parameters — even after we have tuned those randomly added parameters for a few epochs. Remember, the pretrained weights have been trained for hundreds of epochs, on millions of images.\n",
"\n",
"In addition, do you remember the images we saw in <<chapter_intro>>, showing what each layer learns? The first layer learns very simple foundations, like edge and gradient detectors; these are likely to be just as useful for nearly any task. The later layers learn much more complex concepts, like \"eye\" and \"sunset\", which might not be useful in your task at all (maybe you're classifying car models, for instance). So it makes sense to let the later layers fine-tune more quickly than earlier layers.\n",
"Therefore, fastai by default does something called *discriminative learning rates*. This was originally developed in the ULMFiT approach to NLP transfer learning that we introduced in <<chapter_intro>>. Like many good ideas in deep learning, it is extremely simple: use a lower learning rate for the early layers of the neural network, and a higher learning rate for the later layers (and especially the randomly added layers). The idea is based on insights developed by Jason Yosinski, who showed in 2014 that when transfer learning different layers of a neural network should train at different speeds, as seen in <<yosinski>>."
"<img alt=\"Impact of different layers and training methods on transfer learning (Yosinski)\" width=\"680\" caption=\"Impact of different layers and training methods on transfer learning (curtesy of Jason Yosinski)\" id=\"yosinski\" src=\"images/att_00039.png\">"
"Fastai lets you pass a Python *slice* object anywhere that a learning rate is expected. The first value past will be the learning rate in the earliest layer of the neural network, and the second value will be the learning rate in the final layer. The layers in between will have learning rates that are multiplicatively equidistant throughout that range. Let's use this approach to replicate the previous training, but this time we'll only set the *lowest* layer of our net to a learning rate of `1e-6`; the other layers will scale up to `1e-4`. Let's train for a while and see what happens."
"As you can see, the training loss keeps getting better and better. But notice that eventually the validation loss improvement slows, and sometimes even gets worse! This is the point at which the model is starting to over fit. In particular, the model is becoming overconfident of its predictions. But this does *not* mean that it is getting less accurate, necessarily. Have a look at the table of training results per epoch, and you will often see that the accuracy continues improving, even as the validation loss gets worse. In the end what matters is your accuracy, or more generally your chosen metrics, not the loss. The loss is just the function we've given the computer to help us to optimise."
"Often you will find that you are limited by time, rather than generalisation and accuracy, when choosing how many epochs to train for. So your first approach to training should be to simply pick a number of epochs that will train in the amount of time that you are happy to wait for. Have a look at the training and validation loss plots, like showed above, and in particular your metrics, and if you see that they are still getting better even in your final epochs, then you know that you have not trained for too long.\n",
"On the other hand, you may well see that the metrics you have chosen are really getting worse at the end of training. Remember, it's not just that we're looking for the validation loss to get worse, but your actual metrics. Your validation loss will first of all during training get worse because it gets overconfident, and only later will get worse because it is incorrectly memorising the data. We only care in practice about the latter issue. Our loss function is just something, remember, that we used to allow our optimiser to have something it could differentiate and optimise; it's not actually the thing we care about in practice.\n",
"Before the days of 1cycle training it was very common to save the model at the end of each epoch, and then select whichever model had the best accuracy, out of all of the models saved in each epoch. This is known as *early stopping*. However, with one cycle training, it is very unlikely to give you the best answer, because those epochs in the middle occur before the learning rate has had a chance to reach the small values, where it can really find the best result. Therefore, if you find that you have overfit, what you should actually do is to retrain your model from scratch, and this time select a total number of epochs based on where your previous best results were found.\n",
"\n",
"If we've got the time to train for more epochs, we may want to instead use that time to train more parameters, that is use a deeper architecture."
"In general, a model with more parameters can model your data more accurately. (There are lots and lots of caveats to this generalisation, and it depends on the specifics of the architectures you are using, but it is a reasonable rule of thumb for now.) For most of the architectures that we will be seeing in this book you can create larger versions of them by simply adding more layers. However, since we want to use pretrained models, we need to make sure that we choose a number of layers that has been already pretrained for us.\n",
"This is why, in practice, architectures tend to come in a small number of variants. For instance, the resnet architecture that we are using in this chapter comes in 18, 34, 50, 101, and 152 layer variants, pre-trained on ImageNet. A larger (more layers and parameters; sometimes described as the \"capacity\" of a model) version of a resnet will always be able to give us a better training loss, but it can suffer more from overfitting, because it has more parameters to over fit with.\n",
"\n",
"In general, a bigger model has the ability to better capture the real underlying relationships in your data, and also to capture and memorise the specific details of your individual images.\n",
"\n",
"However, using a deeper model is going to require more GPU RAM, so we may need to lower the size of our batches to avoid *out-of-memory errors*. This happens when you try to fit too much inside your GPU and looks like:\n",
"\n",
"```\n",
"Cuda runtime error: out of memory\n",
"```\n",
"\n",
"You may have to restart your notebook when this happens, and the way to solve it is to use a smaller *batch size*, which means we will pass smaller groups of images at any given time through our model. We can pass the batch size we want to the call creating our `DataLoaders` with `bs=`.\n",
"The other downside of deeper architectures is that they take quite a bit longer to train. One thing that can speed things up a lot is *mixed precision training*. This refers to using less precise numbers (*half precision floating point*, also called *fp16*) where possible during training. As we are writing these words (early 2020) nearly all current NVIDIA GPUs support a special feature called *tensor cores* which can dramatically (2x-3x) speed up neural network training. They also require a lot less GPU memory. To enable this feature in fastai, just add `to_fp16()` after your `Learner` creation (you also need to import the module).\n",
"You can't really know ahead of time what the best architecture for your particular problem is, until you try training some. So let's try a resnet 50 now with mixed precision:"
"You'll see here we've gone back to using `fine_tune`, since it's so handy! We can pass `freeze_epochs` to tell fastai how many epochs to train for while frozen. It will automatically change learning rates appropriately for most datasets.\n",
"\n",
"In this case, we're not seeing a clear win from the deeper model. This is useful to remember--bigger models aren't necessarily better models for your particular case! Make sure you try small models before you start scaling up."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this chapter we learned some important practical tips, both for getting our image data ready for modeling (presizing; data block summary) and for fitting the model (learning rate finder, unfreezing, discriminative learning rates, setting the number of epochs, and using deeper architectures). Using these tools will help you to build more accurate image models, more quickly.\n",
"We also learned about cross entropy loss. This part of the book is worth spending plenty of time on. You aren't likely to need to actually implement cross entropy loss from scratch yourself in practice, but it's really important you understand the inputs to and output from that function, because it (or a variant of it, as we'll see in the next chapter) is used in nearly every classification model. So when you want to debug a model, or put a model in production, or improve the accuracy of a model, you're going to need to be able to look at its activations and loss, and understand what's going on, and why. You can't do that properly if you don't understand your loss function.\n",
"If cross entropy loss hasn't \"clicked\" for you just yet, don't worry--you'll get there! First, go back to the last chapter and make sure you really understand `mnist_loss`. Then work gradually through the cells of the notebook for this chapter, where we step through each piece of cross entropy loss. Make sure you understand what each calculation is doing, and why. Try creating some small tensors yourself and pass them into the functions, to see what they return.\n",
"\n",
"Remember: the choices made in cross entropy loss are not the only possible choices that could have been made. Just like when we looked at regression, we could choose between mean squared error and mean absolute difference (L1), we could change the details inside cross entropy loss too. If you have other ideas for possible functions that you think might work, feel free to give them a try in this chapter's notebook! (Fair warning though: you'll probably find that the model will be slower to train, and less accurate. That's because the gradient of cross entropy loss is proportional to the difference between the activation and the target, so SGD always gets a nicely scaled step for the weights.)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Questionnaire"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Why do we first resize to a large size on the CPU, and then to a smaller size on the GPU?\n",
"1. If you are not familiar with regular expressions, find a regular expression tutorial, and some problem sets, and complete them. Have a look on the book website for suggestions.\n",
"1. What are the two ways in which data is most commonly provided, for most deep learning datasets?\n",
"1. Look up the documentation for `L` and try using a few of the new methods is that it adds.\n",
"1. Look up the documentation for the Python pathlib module and try using a few methods of the Path class.\n",
"1. Give two examples of ways that image transformations can degrade the quality of the data.\n",
"1. What method does fastai provide to view the data in a DataLoader?\n",
"1. What method does fastai provide to help you debug a DataBlock?\n",
"1. Should you hold off on training a model until you have thoroughly cleaned your data?\n",
"1. What are the two pieces that are combined into cross entropy loss in PyTorch?\n",
"1. What are the two properties of activations that softmax ensures? Why is this important?\n",
"1. When might you want your activations to not have these two properties?\n",
"1. Calculate the \"exp\" and \"softmax\" columns of <<bear_softmax>> yourself (i.e. in a spreadsheet, with a calculator, or in a notebook).\n",
"1. Why can't we use torch.where to create a loss function for datasets where our label can have more than two categories?\n",
"1. What is the value of log(-2)? Why?\n",
"1. What are two good rules of thumb for picking a learning rate from the learning rate finder?\n",
"1. What two steps does the fine_tune method do?\n",
"1. In Jupyter notebook, how do you get the source code for a method or function?\n",
"1. Why is early stopping a poor choice when using one cycle training?\n",
"1. What is the difference between resnet 50 and resnet101?\n",
"1. What does to_fp16 do?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Further research"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Find the paper by Leslie Smith that introduced the learning rate finder, and read it.\n",
"1. See if you can improve the accuracy of the classifier in this chapter. What's the best accuracy you can achieve? Have a look on the forums and book website to see what other students have achieved with this dataset, and how they did it."
