[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

EuclidNets: An Alternative Operation for Efficient Inference of Deep Learning Models

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

With the advent of deep learning applications on edge devices, researchers actively try to optimize deep learning model deployment on low-power and restricted memory devices. There are established compression methods such as quantization, pruning, and architecture search that leverage commodity hardware. Apart from conventional compression algorithms, one may redesign the operations of deep learning models, leading to more efficient hardware implementation. To this end, we propose EuclidNet, an efficient computing method designed to be implemented on hardware that replaces multiplication, with squared difference. We show that EuclidNet is aligned with matrix multiplication and can be used as a measure of similarity in the case of convolutional layers. Furthermore, we show that under various transformations and noise scenarios, EuclidNet exhibits the same performance compared to the deep learning models designed with multiplication operations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Benmeziane H, Maghraoui KE, Ouarnoughi H, Niar S, Wistuba M, Wang N. A comprehensive survey on hardware-aware neural architecture search. 2021 arXiv preprint arXiv:2101.09336.

  2. Cheng Y, Wang D, Zhou P, Zhang T. A survey of model compression and acceleration for deep neural networks. 2017 arXiv preprint arXiv:1710.09282.

  3. Hennessy JL, Patterson DA. Computer architecture: a quantitative approach. London: Elsevier; 2011.

    MATH  Google Scholar 

  4. Chen H, Wang Y, Xu C, Shi B, Xu C, Tian Q, Xu C. Addernet: Do we really need multiplications in deep learning? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020 pp. 1468–1477.

  5. de Dinechin F, Pasca B. Large multipliers with fewer dsp blocks. In: 2009 International Conference on Field Programmable Logic and Applications, 2009 pp. 250–255. IEEE.

  6. Baluja S, Marwood D, Covell M, Johnston N. No multiplication? no floating point? no problem! training networks for efficient inference. 2018 arXiv preprint arXiv:1809.09244.

  7. Covell M, Marwood D, Baluja S, Johnston N. Table-based neural units: Fully quantizing networks for multiply-free inference. 2019 arXiv preprint arXiv:1906.04798.

  8. Razlighi MS, Imani M, Koushanfar F, Looknn Rosing T. Neural network with no multiplication. In: Design, Automation & Test in Europe Conference & Exhibition. IEEE; 2017. p. 1775–80.

    Google Scholar 

  9. Frankle J, Carbin M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 2018 arXiv preprint arXiv:1803.03635.

  10. Cheng Y, Wang D, Zhou P, Zhang T. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Sig Process Mag. 2018;35(1):126–36.

    Article  Google Scholar 

  11. Wu C-J, Brooks D, Chen K, Chen D, Choudhury S, Dukhan M, Hazelwood K, Isaac E, Jia Y, Jia B, et al. Machine learning at facebook: Understanding inference at the edge. In: 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA) 2019 , pp. 331–344. IEEE.

  12. Reed R. Pruning algorithms-a survey. IEEE Transact Neural Net. 1993;4(5):740–7.

    Article  Google Scholar 

  13. Guo, Y. A survey on methods and theories of quantized neural networks. 2018 arXiv preprint arXiv:1808.04752.

  14. Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network. 2015 arXiv preprint arXiv:1503.02531.

  15. Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. 2016 arXiv preprint arXiv:1602.07360

  16. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. 2017 arXiv preprint arXiv:1704.04861

  17. Zhang X, Zhou X, Lin M, Sun J. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018 pp. 6848–6856.

  18. Tan M, Le QV. Efficientnet: Rethinking model scaling for convolutional neural networks. 2019 arXiv preprint arXiv:1905.11946.

  19. Hubara I, Courbariaux M, Soudry D, El-Yaniv R, Bengio Y. Binarized neural networks. In: Lee D, Sugiyama M, Luxburg U, Guyon I, Garnett R, editors. Advances in neural information processing systems, vol. 29. Curran Associates Inc; 2016.

    MATH  Google Scholar 

  20. Xu Y, Xu C, Chen X, Zhang W, Xu C, Wang Y. Kernel based progressive distillation for adder neural networks. 2020 arXiv preprint arXiv:2009.13044

  21. Dogaru R, Chua LO. The comparative synapse: a multiplication free approach to neuro-fuzzy classifiers. IEEE Transact Circuits Syst. 1999;46(11):1366–71.

    Article  Google Scholar 

  22. Akbaş CE, Bozkurt A, çetin AE, çetin-Atalay R, Üner A. Multiplication-free neural networks. In: 2015 23nd Signal Processing and Communications Applications Conference (SIU), 2015 pp. 2416–2418. Doi: https://doi.org/10.1109/SIU.2015.7130369.

  23. Mallah M. Multiplication free neural networks. PhD thesis, Bilkent University, 2018.

  24. Afrasiyabi A, Badawi D, Nasir B, Yildi O, Vural FTY, Çetin AE. Non-euclidean vector product for neural networks. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2018, pp. 6862–6866. IEEE

  25. Badawi D, Akhan E, Mallah M, Üner A, Çetin-Atalay R, Çetin AE. Multiplication free neural network for cancer stem cell detection in h-and-e stained liver images. In: Compressive Sensing VI: From Diverse Modalities to Big Data Analytics, 2017, vol. 10211, pp. 102110C1–102110C9. International Society for Optics and Photonics. https://doi.org/10.1117/12.2262338

  26. Pan H, Badawi D, Zhang X, Cetin AE. Additive neural network for forest fire detection. Sig, Image Video Process. 2019;14:675–82.

    Article  Google Scholar 

  27. You H, Chen X, Zhang Y, Li C, Li S, Liu Z, Wang Z, Lin Y. Shiftaddnet: A hardware-inspired deep network. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual.

  28. Limonova E, Alfonso D, Nikolaev D, Arlazarov VV. Resnet-like architecture with low hardware requirements. 2020 arXiv preprint arXiv:2009.07190.

  29. Limonova E, Matveev D, Nikolaev D, Arlazarov VV. Bipolar morphological neural networks: convolution without multiplication. In: Twelfth International Conference on Machine Vision (ICMV 2019), 2020, vol. 11433, p. 114333. International Society for Optics and Photonics.

  30. Mondal R, Santra S, Mukherjee S, Chanda B. Morphological Network: How Far Can We Go with Morphological Neurons? 2022. arXiv:1901.00109

  31. Davidson JL, Ritter GX. Theory of morphological neural networks. In: Digital Optical Computing II. UK: International Society for Optics and Photonics; 1990. p. 709–17.

    Google Scholar 

  32. Ritter GX, Sussner P. An introduction to morphological neural networks. In: Proceedings of 13th International Conference on Pattern Recognition, 1996, vol. 4, pp. 709–717. IEEE.

  33. Weimerskirch A, Paar C. Generalizations of the karatsuba algorithm for efficient implementations. Cryptology ePrint Archive, Paper 2006/224. https://eprint.iacr.org/2006/224

  34. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234–241. Springer

  35. He K, Gkioxari G, Dollár P, Girshick R. Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2961–2969.

  36. Chen L-C, Papandreou G, Schroff F, Adam H. Rethinking atrous convolution for semantic image segmentation. 2017 arXiv preprint arXiv:1706.05587.

  37. NVIDIA Vingelmann P, Fitzek FHP. CUDA, Accessed: 10 Feb 89, 2020. https://developer.nvidia.com/cuda-toolkit.

  38. Allgower EL, Georg K. Introduct Num Cont Methods. New York: SIAM; 2003.

    Google Scholar 

  39. Chow J, Udpa L, Udpa S. Homotopy continuation methods for neural networks. In: 1991., IEEE International Sympoisum on Circuits and Systems, 1991, pp. 2483–2486. IEEE.

  40. Pathak HN, Paffenroth R. Parameter continuation methods for the optimization of deep neural networks. In: 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), 2019, pp. 1637–1643. IEEE.

  41. Cao Z, Long M, Wang J, Yu PS. Hashnet: Deep learning to hash by continuation. In: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 5608–5617.

  42. Mobahi H. Training recurrent neural networks by diffusion. 2016, arXiv preprint arXiv:1601.04114.

  43. Farhadi F, Partovi Nia V, Lodi A. Activation adaptation in neural networks. In: Proceedings of the 9th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, 2020.

  44. Gulcehre C, Moczulski M, Visin F, Bengio Y. Mollifying networks. 2016 arXiv preprint arXiv:1608.04980.

  45. Chen Q, Hao W. A homotopy training algorithm for fully connected neural networks. 2019. Proc R Soc A. https://doi.org/10.1098/rspa.2019.0662

  46. Bengio Y, Louradour J, Collobert R, Weston J. Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, 2009, pp. 41–48.

  47. Yim J, Joo D, Bae J, Kim J. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4133–4141.

  48. Krizhevsky A, Hinton G, et al. Learning multiple layers of features from tiny images. 2009. Technical Report. https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf

  49. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. 2015, CoRR abs/1512.03385 (2015).

  50. Wu H, Judd P, Zhang X, Isaev M, Micikevicius P. Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation, 2020. arXiv:2004.09602

  51. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L. ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 2009, pp. 248–255. https://doi.org/10.1109/CVPR.2009.5206848

Download references

Author information

Authors and Affiliations

  1. Huawei Noah’s Ark Lab, Montreal Research Centre, 7101 Park Avenue, Montreal, H3N 1X9, QC, Canada

    Xinlin Li, Alireza Ghaffari & Vahid Partovi Nia

  2. Department of Mathematics and Statistics, McGill University, 805 Sherbrooke Street West, Montreal, H3A 0B9, QC, Canada

    Mariana Parazeres, Adam Oberman & Masoud Asgharian

  3. Department of Mathematics and Industrial Engineering, Polytechnique Montreal, 2500 Chem. Polytechnique, Montreal, H3T 1J4, QC, Canada

    Vahid Partovi Nia

Authors
  1. Xinlin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mariana Parazeres
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adam Oberman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alireza Ghaffari
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Masoud Asgharian
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Vahid Partovi Nia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vahid Partovi Nia.

Ethics declarations

Conflict of interest

All authors declare that they have no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the topical collection “Advances on Pattern Recognition Applications and Methods 2022” guest edited by Ana Fred, Maria De Marsico and Gabriella Sanniti di Baja.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, X., Parazeres, M., Oberman, A. et al. EuclidNets: An Alternative Operation for Efficient Inference of Deep Learning Models. SN COMPUT. SCI. 4, 507 (2023). https://doi.org/10.1007/s42979-023-01921-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-023-01921-y

Keywords

Search

Navigation