Yes, you should understand backprop!
The best way I found so far to really get the feeling of knowing the ins and outs of it is to follow AI guru master Andrej Karpathy explanations, and in particular follow his youtube lecture on building micrograd (find it on github).
This post is a trap. My goal is to distill Karpathy’s 2.5-hour lecture into a quicker read, but i will also attach additional resources that complement the lecture. So, fair warning: the rabbit hole of mathematical wonder could be quite consuming.
If you successfully manage to go through it, or if you already watched the lecture and are looking for a review and extra material, I published a simple github repo or google colab for you to practice your understanding of micrograd. Let’s dive into the material.
Unfortunately the original paper is behind a paywall and, unless you are in uni, there is no way to get it for free *cough cough sci-hub * What about then jump straight to see how it is implemented in libraries such as Pytorch and TensorFlow? Well, attacking their respective github repo might be a hard task.. we could indeed check out TensorFlow introduction to gradients documentation or Pytorch gentle introduction to torch.autograd, but the first snippet of code in this last one:
import torch
from torchvision.models import resnet18, ResNet18_Weights
model = resnet18(weights=ResNet18_Weights.DEFAULT)
data = torch.rand(1, 3, 64, 64)
labels = torch.rand(1, 1000)
is enough to scare our mind away to truly wrap our mind around the model.
Luckily, micrograd is a perfect tool in order to make us understand the topic… So whe
Here is some text and this is blue.
Useful material: The aim is to find a powerful synaptic modification rule that will allow an arbitrarily connected neural network to develop an internal structure that is appropriate for a particular task domain (from backprop paper).