💡: The Latest Framework test case is under:
./example/.
Overview of Framework
- nn
- Model (Base Class for Nerual Networks, like nn.Module in torch)
- Conv
- Conv1d, Conv2d, Conv3d
- MaxPooling1d, MaxPooling2d, MaxPooling3d
- BatchNorm TODO
- RnnCell
- Basic rnn kernel
- LSTM kernel
- GRU kernel
- BiLSTM kernel
- BiGRU kernel
- Layer Norm TODO
- FC
- Dropout
- Linear
- Optimizer
- Algorithms
- Raw GD
- Momentum
- Nesterov(NAG)
- AdaGrad
- RMSProp
- AdaDelta
- Adam[1]
- Machanisms
- Lr Decay. TODO
- Weight Decay. TODO
- Freeze. TODO
- Algorithms
- Utils
- sigmoid
- one hot
- softmax
- cross_entropy_loss
- mean_square_error
- l1_regularization
- l2_regularization
Some small tests for debug during the development of this project:
- How to Use Mini-torch? A brief e.g. Doc TODO
- How to Use Jax Gradient, Ideas about how I manage parameters in this Framework.
- Some Jax Tips, About How to Use Jax Builtins & JIT to Optimize Loops & Matrix Operations.
- Kaiming Initialization[2] used in MLP & Conv, With math derivation.
- Difference between Conv2d Operation by python loop and by Jax.lax.
- Dropout mechanism impl, About Seed in Jax.
- Runge-Kuta solver for Neural ODE.
[1] Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization. Proceedings of the International Conference on Learning Representations (ICLR).
[2] He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) (pp. 1026–1034).