Abstract
The trend of Artificial Neural Networks becoming\bigger"and \deeper" persists. Training these networks using back-propagation is considered biologically implausible and a time-consuming task. Hence, we investigate how far we can go with fixed binary random projections (BRPs), an approach which reduces the number of trainable parameters using localized receptive fields and binary weights. Evaluating this approach on the MNIST dataset we discovered that contrary to models with fully-trained dense weights, models using fixed localized sparse BRPs yield equally good performance in terms of accuracy, saving 98% computations when generating the hidden representation for the input. Furthermore, we discovered that using BRPs leads to a more robust performance – up to 56% better compared to dense models – in terms of classifying noisy inputs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Chakraborty B, Shaw B, Aich J, et al. Does deeper network lead to better accuracy: a case study on handwritten Devanagari characters. Proc Int Anal Doc Syst (DAS). 2018 April; p. 411–417.
He K, Zhang X, Ren S, et al. Deep residual learning for image recognition. In: Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit; 2016. p. 770–778.
Geirhos R, Temme CRM, Rauber J, et al. Generalisation in humans and deep neural networks. In: Adv Neural Inf Process Syst. Curran Associates; 2018. p. 7538–7550.
LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015 May;521:436–444.
Dasgupta S, Stevens CF, Navlakha S. A neural algorithm for a fundamental computing problem. Science. 2017;358(6364):793–796.
Illing B, Gerstner W, Brea J. Biologically plausible deep learning - But how far can we go with shallow networks? Neural Netw. 2019;118:90–101.
He K, Zhang X, Ren S, et al. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. Proc IEEE ICCV. 2015; p. 1026–1034.
Hoffer E, Hubara I, Soudry D. Fix your classifieer: the marginal value of training the last weight layer. Proc Conf Learn Represent. 2018; p. 5145–5153.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Der/die Autor(en), exklusiv lizenziert durch Springer Fachmedien Wiesbaden GmbH, ein Teil von Springer Nature
About this paper
Cite this paper
Yang, Z., Schilling, A., Maier, A., Krauss, P. (2021). Neural Networks with Fixed Binary Random Projections Improve Accuracy in Classifying Noisy Data. In: Palm, C., Deserno, T.M., Handels, H., Maier, A., Maier-Hein, K., Tolxdorff, T. (eds) Bildverarbeitung für die Medizin 2021. Informatik aktuell. Springer Vieweg, Wiesbaden. https://doi.org/10.1007/978-3-658-33198-6_51
Download citation
DOI: https://doi.org/10.1007/978-3-658-33198-6_51
Published:
Publisher Name: Springer Vieweg, Wiesbaden
Print ISBN: 978-3-658-33197-9
Online ISBN: 978-3-658-33198-6
eBook Packages: Computer Science and Engineering (German Language)