More Web Proxy on the site http://driver.im/

research-article

Public Access

Examining the Robustness of Spiking Neural Networks on Non-ideal Memristive Crossbars

Authors:

Abhiroop Bhattacharjee,

Abhishek Moitra,

Priyadarshini PandaAuthors Info & Claims

ISLPED '22: Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design

Article No.: 1, Pages 1 - 6

https://doi.org/10.1145/3531437.3539729

Published: 01 August 2022 Publication History

All formats PDF

Abstract

Spiking Neural Networks (SNNs) have recently emerged as the low-power alternative to Artificial Neural Networks (ANNs) owing to their asynchronous, sparse, and binary information processing. To improve the energy-efficiency and throughput, SNNs can be implemented on memristive crossbars where Multiply-and-Accumulate (MAC) operations are realized in the analog domain using emerging Non-Volatile-Memory (NVM) devices. Despite the compatibility of SNNs with memristive crossbars, there is little attention to study on the effect of intrinsic crossbar non-idealities and stochasticity on the performance of SNNs. In this paper, we conduct a comprehensive analysis of the robustness of SNNs on non-ideal crossbars. We examine SNNs trained via learning algorithms such as, surrogate gradient and ANN-SNN conversion. Our results show that repetitive crossbar computations across multiple time-steps induce error accumulation, resulting in a huge performance drop during SNN inference. We further show that SNNs trained with a smaller number of time-steps achieve better accuracy when deployed on memristive crossbars.

References

[1]

Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-Icaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, 2015. Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip. IEEE TCAD (2015).

Digital Library

[2]

Aayush Ankit, Abhronil Sengupta, Priyadarshini Panda, and Kaushik Roy. 2017. Resparc: A reconfigurable and energy-efficient architecture with memristive crossbars for deep spiking neural networks. In Proceedings of the 54th Annual Design Automation Conference 2017. 1–6.

Digital Library

[3]

Adarsha Balaji 2019. Mapping spiking neural networks to neuromorphic hardware. IEEE TVLSI (2019).

[4]

Abhiroop Bhattacharjee, Lakshya Bhatnagar, Youngeun Kim, and Priyadarshini Panda. 2021. Neat: Non-linearity aware training for accurate, energy-efficient and robust implementation of neural networks on 1t-1r crossbars. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (2021).

[5]

Abhiroop Bhattacharjee, Lakshya Bhatnagar, and Priyadarshini Panda. 2022. Examining and Mitigating the Impact of Crossbar Non-idealities for Accurate Implementation of Sparse Deep Neural Networks. 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE) (2022).

[6]

Abhiroop Bhattacharjee, Abhishek Moitra, and Priyadarshini Panda. 2021. Efficiency-driven hardware optimization for adversarially robust neural networks. In 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 884–889.

[7]

Abhiroop Bhattacharjee and Priyadarshini Panda. 2020. Rethinking non-idealities in memristive crossbars for adversarial robustness in neural networks. arXiv preprint arXiv:2008.11298(2020).

[8]

Abhiroop Bhattacharjee and Priyadarshini Panda. 2021. SwitchX: Gmin-Gmax Switching for Energy-Efficient and Robust Implementation of Binary Neural Networks on ReRAM Xbars. arxiv:2011.14498 [cs.ET]

[9]

Indranil Chakraborty, Mustafa Fayez Ali, Dong Eun Kim, Aayush Ankit, and Kaushik Roy. 2020. Geniex: A generalized approach to emulating non-ideality in memristive xbars using neural networks. In 2020 57th ACM/IEEE Design Automation Conference (DAC). IEEE, 1–6.

[10]

I Chakraborty, A Jaiswal, AK Saha, SK Gupta, and K Roy. 2020. Pathways to efficient neuromorphic computing with non-volatile memory technologies. Applied Physics Reviews 7, 2 (2020), 021308.

[11]

Pai-Yu Chen 2018. NeuroSim: A circuit-level macro model for benchmarking neuro-inspired architectures in online learning. IEEE TCAD (2018).

[12]

Dennis Valbjørn Christensen, Regina Dittmann, Bernabé Linares-Barranco, Abu Sebastian, Manuel Le Gallo, Andrea Redaelli, Stefan Slesazeck, Thomas Mikolajick, Sabina Spiga, Stephan Menzel, 2022. 2022 roadmap on neuromorphic computing and engineering. Neuromorphic Computing and Engineering(2022).

[13]

Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, 2018. Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro 38, 1 (2018), 82–99.

[14]

Peter U Diehl 2015. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International joint conference on neural networks (IJCNN). ieee, 1–8.

[15]

Peter U Diehl and Matthew Cook. 2015. Unsupervised learning of digit recognition using spike-timing-dependent plasticity. Frontiers in computational neuroscience(2015).

[16]

Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. 2021. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In CVPR.

[17]

Steve B Furber, Francesco Galluppi, Steve Temple, and Luis A Plana. 2014. The spinnaker project. Proc. IEEE 102, 5 (2014), 652–665.

[18]

Bing Han and Kaushik Roy. 2020. Deep spiking neural network: Energy efficiency through time based coding. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16. Springer, 388–404.

[19]

Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167(2015).

[20]

Shubham Jain, Abhronil Sengupta, Kaushik Roy, and Anand Raghunathan. 2020. RxNN: A framework for evaluating deep neural networks on resistive crossbars. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 40, 2(2020), 326–338.

Digital Library

[21]

Youngeun Kim, Hyunsoo Kim, Seijoon Kim, Sang Joon Kim, and Priyadarshini Panda. 2022. Gradient-based Bit Encoding Optimization for Noise-Robust Binary Memristive Crossbar. arXiv preprint arXiv:2201.01479(2022).

[22]

Youngeun Kim and Priyadarshini Panda. 2021. Revisiting Batch Normalization for Training Low-Latency Deep Spiking Neural Networks From Scratch. Frontiers in Neuroscience(2021).

[23]

Alex Krizhevsky, Geoffrey Hinton, 2009. Learning multiple layers of features from tiny images. (2009).

[24]

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324.

[25]

Eimantas Ledinauskas 2020. Training deep spiking neural networks. arXiv preprint arXiv:2006.04436(2020).

[26]

Yuhang Li 2021. A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibration. arXiv preprint arXiv:2106.06984(2021).

[27]

Beiye Liu, Hai Li, Yiran Chen, Xin Li, Qing Wu, and Tingwen Huang. 2015. Vortex: Variation-aware training for memristor X-bar. In Proceedings of the 52nd Annual Design Automation Conference. 1–6.

Digital Library

[28]

Priyadarshini Panda 2020. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. (2020).

[29]

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017).

[30]

Steven J Plimpton, Sapan Agarwal, Richard Schiek, and Isaac Richter. 2016. CrossSim. Technical Report. Sandia National Lab.(SNL-NM), Albuquerque, NM (United States).

[31]

Nitin Rathi and Kaushik Roy. 2021. DIET-SNN: A Low-Latency Spiking Neural Network With Direct Input Encoding and Leakage and Threshold Optimization. IEEE TNNLS (2021).

[32]

Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. 2019. Towards spike-based machine intelligence with neuromorphic computing. Nature 575, 7784 (2019), 607–617.

[33]

Bodo Rueckauer 2017. Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Frontiers in neuroscience 11 (2017), 682.

[34]

Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. 2019. Going deeper in spiking neural networks: VGG and residual architectures. Frontiers in neuroscience 13 (2019), 95.

[35]

Saima Sharmin, Nitin Rathi, Priyadarshini Panda, and Kaushik Roy. 2020. Inherent adversarial robustness of deep spiking neural networks: Effects of discrete input encoding and non-linear activations. In European Conference on Computer Vision. Springer, 399–414.

Digital Library

[36]

Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, and Kay Chen Tan. 2019. A Tandem Learning Rule for Effective Training and Rapid Inference of Deep Spiking Neural Networks. arXiv e-prints (2019), arXiv–1907.

[37]

Jibin Wu, Chenglin Xu, Daquan Zhou, Haizhou Li, and Kay Chen Tan. 2020. Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks. arXiv preprint arXiv:2007.01204(2020).

[38]

Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. 2018. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience 12 (2018), 331.

[39]

Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. 2020. Going deeper with directly-trained larger spiking neural networks. arXiv preprint arXiv:2011.05280(2020).

Cited By

Bhattacharjee AYin RMoitra APanda P(2024)Are SNNs Truly Energy-efficient? — A Hardware PerspectiveICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10448269(13311-13315)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10448269
Saha AAmarnath CMa KChatterjee AKim T(2024)Signature Driven Post-Manufacture Testing and Tuning of RRAM Spiking Neural Networks for Yield RecoveryProceedings of the 29th Asia and South Pacific Design Automation Conference10.1109/ASP-DAC58780.2024.10473874(740-745)Online publication date: 22-Jan-2024
https://dl.acm.org/doi/10.1109/ASP-DAC58780.2024.10473874
Ahmad SSiddique AHoque K(2023)Security-Aware Approximate Spiking Neural Networks2023 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE56975.2023.10137269(1-6)Online publication date: Apr-2023
https://doi.org/10.23919/DATE56975.2023.10137269
Show More Cited By

Recommendations

Improved Izhikevich neurons for spiking neural networks

Spiking neural networks constitute a modern neural network paradigm that overlaps machine learning and computational neurosciences. Spiking neural networks use neuron models that possess a great degree of biological realism. The most realistic model of ...
Improved integrate-and-fire neuron models for inference acceleration of spiking neural networks
Abstract
We study the effects of different bio-synaptic membrane potential mechanisms on the inference speed of both spiking feed-forward neural networks and spiking convolutional neural networks. These mechanisms are inspired by biological neuron ...
Efficient STDP Micro-Architecture for Silicon Spiking Neural Networks
DSD '14: Proceedings of the 2014 17th Euromicro Conference on Digital System Design

Spiking neural networks (SNNs) are the closest approach to biological neurons in comparison with conventional artificial neural networks (ANN). SNNs are composed of neurons and synapses which are interconnected with a complex pattern. As communication ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ISLPED '22: Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design

August 2022

192 pages

ISBN:9781450393546

DOI:10.1145/3531437

Editors:
Helen Li
Duke University
,
Charles Augustine
Intel
,
Ayse Kivilcim Coskun
Boston University
,
Swaroop Ghosh
Penn State University

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGDA: ACM Special Interest Group on Design Automation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 August 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Best Paper

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Science Foundation

Conference

ISLPED '22

Sponsor:

SIGDA

ISLPED '22: ACM/IEEE International Symposium on Low Power Electronics and Design

August 1 - 3, 2022

MA, Boston, USA

Acceptance Rates

Overall Acceptance Rate 398 of 1,159 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
664
Total Downloads

Downloads (Last 12 months)324
Downloads (Last 6 weeks)23

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bhattacharjee AYin RMoitra APanda P(2024)Are SNNs Truly Energy-efficient? — A Hardware PerspectiveICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10448269(13311-13315)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10448269
Saha AAmarnath CMa KChatterjee AKim T(2024)Signature Driven Post-Manufacture Testing and Tuning of RRAM Spiking Neural Networks for Yield RecoveryProceedings of the 29th Asia and South Pacific Design Automation Conference10.1109/ASP-DAC58780.2024.10473874(740-745)Online publication date: 22-Jan-2024
https://dl.acm.org/doi/10.1109/ASP-DAC58780.2024.10473874
Ahmad SSiddique AHoque K(2023)Security-Aware Approximate Spiking Neural Networks2023 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE56975.2023.10137269(1-6)Online publication date: Apr-2023
https://doi.org/10.23919/DATE56975.2023.10137269
Bhattacharjee AMoitra APanda P(2023) XploreNAS: Explore Adversarially Robust and Hardware-efficient Neural Architectures for Non-ideal XbarsACM Transactions on Embedded Computing Systems10.1145/359304522:4(1-17)Online publication date: 24-Jul-2023
https://dl.acm.org/doi/10.1145/3593045
Moitra ABhattacharjee AKuang RKrishnan GCao YPanda P(2023)SpikeSim: An End-to-End Compute-in-Memory Hardware Evaluation Tool for Benchmarking Spiking Neural NetworksIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2023.327491842:11(3815-3828)Online publication date: 10-May-2023
https://dl.acm.org/doi/10.1109/TCAD.2023.3274918
Deng ZWang CLin HSun Y(2023)A Memristive Spiking Neural Network Circuit With Selective Supervised Attention AlgorithmIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2022.322889642:8(2604-2617)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1109/TCAD.2022.3228896
Dampfhoffer MLopez JMesquida TValentian AAnghel L(2023)Improving the Robustness of Neural Networks to Noisy Multi-Level Non-Volatile Memory-based Synapses2023 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN54540.2023.10191804(1-8)Online publication date: 18-Jun-2023
https://doi.org/10.1109/IJCNN54540.2023.10191804
Saha AAmarnath CChatterjee A(2023)A Resilience Framework for Synapse Weight Errors and Firing Threshold Perturbations in RRAM Spiking Neural Networks2023 IEEE European Test Symposium (ETS)10.1109/ETS56758.2023.10174229(1-4)Online publication date: 22-May-2023
https://doi.org/10.1109/ETS56758.2023.10174229

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents