[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

Gated filter pruning via sample manifold relationships

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Filter pruning is an essential method for compressing and accelerating deep neural networks on computationally restricted devices. Despite recognizing the high correlation between filter redundancy and samples, existing methods primarily focus on independently searching for optimal subnetworks from individual input while ignoring the relationships among different inputs. In this paper, we propose a novel approach called Gated Filter Pruning based on Sample Manifold Relationships, which exploits and aligns the manifold relationships of all samples during training to obtain an optimal subnetwork. Firstly, we introduce a Gated Filter Normalization Module (GFNM) that excavates the manifold information of each sample, applicable to the operator level without adding many additional parameters. GFNM incorporates explainable control variables jointly optimized with convolutional weights, explicitly determining the competition and cooperation among filters during training. Subsequently, Manifold Regularized Pruning Module (MRPM) measures the manifold relationships between samples and subnetworks, efficiently regularizing the solution space of sample-network pairs. The manifold relationships between samples and subnetworks are aligned in training to derive an effective subnetwork for all input samples. Extensive experimental results validate the effectiveness of our method, demonstrating competitive performance in terms of accuracy and computational cost compared to state-of-the-art (SOTA) methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The data and codes generated during and analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1106–1114

  2. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: Proceedings of the 3rd international conference on learning representations

  3. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  4. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp 91–99 (2015)

  5. Redmon J, Divvala SK, Girshick RB, Farhadi A (2016) You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788

  6. Alexander K, Eric M, Nikhila R, Hanzi M, Chloe R, Laura G, Tete X, Spencer WC, BA, Wan-Yen L, Piotr D, Ross G (2023) Segment anything. In: Proceedings of the IEEE international conference on computer vision, pp 4015–4026

  7. Han S, Pool J, Tran J, Dally WJ (2015) Learning both weights and connections for efficient neural network. In: Advances in neural information processing systems, pp 1135–1143

  8. Han S, Mao H, Dally WJ (2016) Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. In: Proceedings of the 4th international conference on learning representations

  9. Hinton GE, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network 2(7) . arXiv:1503.02531

  10. LeCun Y, Denker JS, Solla SA (1989) Optimal brain damage. In: Advances in neural information processing systems, pp 598–605

  11. Shi X, Ding J, Hao Z, Yu Z (2024) Towards energy efficient spiking neural networks: an unstructured pruning framework. In: Proceedings of the 12th international conference on learning representations

  12. Ding X, Hao T, Tan J, Liu J, Han J, Guo Y, Ding G (2021) Resrep: lossless cnn pruning via decoupling remembering and forgetting. In: Proceedings of the IEEE international conference on computer vision, pp 4490–4500

  13. Zhang G, Xu S, Li J, Guo AJX (2022) Group-based network pruning via nonlinear relationship between convolution filters. Appl Intell 52(8):9274–9288

    Article  Google Scholar 

  14. Gao X, Zhao Y, Dudziak L, Mullins RD, Xu C (2018) Dynamic channel pruning: feature boosting and suppression. In: Proceedings of the 7th international conference on learning representations

  15. Tang Y, Wang Y, Xu Y, Deng Y, Xu C, Tao D, Xu C (2021) Manifold regularized dynamic network pruning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5018–5028

  16. Li C, Wang G, Wang B, Liang X, Li Z, Chang X (2021) Dynamic slimmable network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8607–8617

  17. Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2017) Pruning filters for efficient convnets. In: Proceedings of the 5th international conference on learning representations

  18. Hu X, Fang H, Zhang L, Zhang X, Yang HH, Yang D, Peng B, Li Z, Hu H (2023) Dynamic connection pruning for densely connected convolutional neural networks. Applied Intelligence

  19. Huang C, Zhai S, Talbott W, Bautista MÁ, Sun S, Guestrin C, Susskind JM (2019) Addressing the loss-metric mismatch with adaptive loss alignment. In: Proceedings of the 36th international conference on machine learning, pp 2891–2900

  20. Weigend AS, Rumelhart DE, Huberman A (1990) Generalization by weight-elimination with application to forecasting. In: Advances in neural information processing systems, pp 875–882

  21. Dong X, Chen S, Pan SJ (2017) Learning to prune deep neural networks via layer-wise optimal brain surgeon. In: Advances in neural information processing systems, pp 4857–4867

  22. Frankle J, Carbin M (2018) The lottery ticket hypothesis: finding sparse, trainable neural networks. arXiv:1803.03635

  23. Frankle J, Dziugaite, GK, Roy DM, Carbin M (2019) Stabilizing the lottery ticket hypothesis. arXiv:1903.01611

  24. Zhu L, Victor Q, Van-Tam N, Enzo T (2023) Can unstructured pruning reduce the depth in deep neural networks? In: Proceedings of the ieee international conference on computer vision workshops, pp 1402–1406

  25. Zhong S, You Z, Zhang J, Zhao S, LeClaire Z, Liu Z, Zha D, Chaudhary V, Xu S, Hu X (2023) One less reason for filter pruning: gaining free adversarial robustness with structured grouped kernel pruning. In: Advances in neural information processing systems

  26. Park J, Kim Y, Kim J, Choi J, Lee S (2023) Dynamic structure pruning for compressing cnns. In: Proceedings of the AAAI conference on artificial intelligence, pp 9408–9416

  27. Liu Z, Mu H, Zhang X, Guo Z, Yang X, Cheng K, Sun J (2019) Metapruning: meta learning for automatic neural network channel pruning. In: Proceedings of the IEEE international conference on computer vision, pp 3296–3305

  28. Lin M, Ji R, Wang Y, Zhang Y, Zhang B, Tian Y, Shao L (2020) Hrank: filter pruning using high-rank feature map. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1529–1538

  29. Tang Y, Wang Y, Xu Y, Tao D, Xu C, Xu C, Xu C (2020) Scop: Scientific control for reliable neural network pruning. In: Advances in neural information processing systems, pp 10936–10947

  30. Sui Y, Yin M, Xie Y, Phan H, Zonouz SA, Yuan B (2021) Chip: channel independence-based pruning for compact neural networks. In: Advances in neural information processing systems, pp 24604–24616

  31. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, pp 448–456

  32. Ulyanov D, Vedaldi A, Lempitsky VS (2016) Instance normalization: the missing ingredient for fast stylization. arXiv:1607.08022

  33. You Z, Yan K, Ye J, Ma M, Wang P (2019) Gate decorator: global filter pruning method for accelerating deep convolutional neural networks. In: Advances in neural information processing systems, pp 2130–2141

  34. Wang J, Zhang Z, Zha H (2004) Adaptive manifold learning. In: Advances in neural information processing systems, pp 1473–1480

  35. Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. Technical report

  36. Deng J, Dong W, Socher R, Li L, Li K, Fei-fei L (2009) Imagenet: a large-scale hierarchical image database. In: Proceedings of the ieee conference on computer vision and pattern recognition, pp 248–255

  37. Lin S, Ji R, Yan C, Zhang B, Cao L, Ye Q, Huang F, Doermann DS (2019) Towards optimal structured CNN pruning via generative adversarial learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2790–2799

  38. He Y, Liu P, Wang Z, Hu Z, Yang Y (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4340–4349

  39. Mondal M, Das B, Roy SD, Singh P, Lall B, Joshi SD (2022) Adaptive cnn filter pruning using global importance metric. Comp Vision Image Underst 222:103511

    Article  Google Scholar 

  40. Wang Y, Zhang X, Xie L, Zhou J, Su H, Zhang B, Hu X (2020) Pruning from scratch. In: Proceedings of the AAAI conference on artificial intelligence, pp 12273–12280

  41. Fang G, Ma X, Song M, Mi MB, Wang X (2023) Depgraph: towards any structural pruning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 16091–16101

  42. Luo J, Wu J (2020) Autopruner: an end-to-end trainable filter pruning method for efficient deep model inference. Pattern Recognition 107:107461

  43. Sandler M, Howard AG, Zhu M, Zhmoginov A, Chen L (2018) Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520

Download references

Acknowledgements

The authors gratefully acknowledge support from the Natural Science Foundation of Jiangsu Province of China (BK20222012), Guangxi Science and Technology Project AB22080026/2021AB22167, National Natural Science Foundation of China (No. 61375021). Thanks to the editor and anonymous reviewers for their valuable comments and suggestions.

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China

    Pingfan Wu, Hengyi Huang, Han Sun & Ningzhong Liu

  2. MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, 211106, China

    Pingfan Wu, Hengyi Huang & Ningzhong Liu

  3. Collaborative Innovation Center of Novel Software Technology and Industrialization, Nanjing, 211106, China

    Pingfan Wu, Hengyi Huang & Ningzhong Liu

Authors
  1. Pingfan Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hengyi Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Han Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ningzhong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Pingfan Wu: Conceptualization, Methodology, Validation, Visualization, Formal analysis, Investigation, Writing - original draft. Hengyi Huang: Writing - review editing, Supervision. Ningzhong Liu: Writing - review editing, Project administration, Investigation. Han Sun: Writing - review editing, Supervision.

Corresponding author

Correspondence to Ningzhong Liu.

Ethics declarations

Competing of Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Compliance with Ethical Standards

The authors declared that they have no conflicts of interest to this work. We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted. The authors declared that the research do not involving Human Participants and Animals. Written informed consent was obtained from all the participants prior to the enrollment (or for the publication) of this study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, P., Huang, H., Sun, H. et al. Gated filter pruning via sample manifold relationships. Appl Intell 54, 9848–9863 (2024). https://doi.org/10.1007/s10489-024-05690-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05690-w

Keywords

Search

Navigation