As you know, Pytorch contains 3 major components:
tensorcan be seen as a replacement ofnumpyfor both GPU and CPU because it has unified API.autograd.Variableis an automatic differentiation tool given a forward formulation.nn: is a deep learning framework build based ontensorandautograd.
This project is aimed to re-implement the autograd part because:
- Pytorch's autograd functions are mostly implemented in
C/C++(for performance purposes) so it is much harder for create new autograd's function. - Instead of building the backward function of complex function, we can build just the forward function, the backward function will be automatically built based on autograd.
- Understand clearly how autograd works is very important for serious deep learning learner.
- You don't need to build the complex computational graph to do back-propagation in deep learning!
The reasons why we choose pytorch tensor over numpy array:
- Pytorch tensor supports GPU.
- We can easily validate your own autograd function with Pytorch's autograd with the same API.
- Getting familiar ourselves with Pytorch's API is a big plus for later deep learning project.
Requirement: pytorch=0.4 or later
Overview of autograd mechanism:
In variables.py:
datais the pytorch tensorgradis the Variable contains the gradient of current Variablegrad_fnis the Function creating the current Variable, ifgrad_fn = None, it is the leaf node of the graphbackward()receives thein_gradfrom the function that current Variable is used, and call the function that creates that Variable.- Other methods for overloading behavior of Variable
In base\base_function.py and base\functions_class.py:
forward()receives theinputand compute theoutputbackward()reveives thein_gradfrom the Variable it creates and compute theout_gradfor each itsinputVariable component
In functions.py:
- Interface for building functions
- Contains the most common of Pytorch tensor's function (not all), but enough for building any graph in Deep Learning
In eval.py:
check_example(): test case for checking your own autograd function whether it is working properly or not
If you want to define your own autograd functions, do the following step:
- If your autograd function are made up of existin
534A
g functions (in
functions.py) or classes (infunctions_class.py), just reuse them. - First, define your new function class in
functions_class.py, the main purpose of each function:forward: storeinputtoself.input, computeoutputand store it toself.output, and store other useful information for backward pass inself.contextdictionarybackward: storein_gradtoself.in_grad, computegradfor eachinput, multiplyin_gradwithgradto getout_grad(according to chain rule) and call eachinputvariable'sbackward()function to back-propagate the gradient until the leaf node.
- Second, define your interface in
functions.pysee other functions for more information - Third, if the
Variablesupports that function, define new method inVariableclass invariables.py - Finally, test the new function with your own test case in
eval.pywith commandpython eval.py