The first nonmonotone stochastic line search method training DL models faster than SGD and Adam.
The file PoNoS.py is a self contained class implementing our new algorithm PoNoS as torch.optim.Optimizer.
This optimizer can be easily integrated in any deep learning pipeline as shown in the example at the end of the file PoNoS.py. Instead, if you are interested in reproducing the results of the paper and/or in some other features of this repository, you can follow the steps below.
pip install git+https:github.com/leonardogalli91/PoNoS.git
or
git clone git@github.com:leonardogalli91/PoNoS.git
pip install -r requirements.txt
Set to True the options download in the file src/datasets.py
python trainval.py -e mnist_mlp -sb results/mnist_mlp -d data -r 1
where -e is the experiment group, -sb is the result directory, and -d is the dataset directory.
The experiment group is referring to the key of the dict EXP_GROUPS, that can be found in the file exp_configs.py,
from that file it is possible to customize thoroughly the experiment.
In the file plot.py set the variable savedir_base to point at the root directory where you saved the results, then run
python plot.py -p mnist_mlp
@inproceedings{galli2023don,
title={Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models},
author={Galli, Leonardo and Rauhut, Holger and Schmidt, Mark},
booktitle={Advances in Neural Information Processing Systems},
year={2023}
}