Abstract
Pruning is a widely used method for compressing neural networks, reducing their computational requirements by removing unimportant connections. However, many existing pruning methods prune pre-trained models by using the same pruning rate for each layer, neglecting the protection of model trainability and damaging accuracy. Additionally, the number of redundant parameters per layer in complex models varies, necessitating adjustment of the pruning rate according to model structure and training data. To overcome these issues, we propose a trainability-preserving adaptive channel pruning method that prunes during training. Our approach utilizes a model weight-based similarity calculation module to eliminate unnecessary channels while protecting model trainability and correcting output feature maps. An adaptive sparsity control module assigns pruning rates for each layer according to a preset target and aids network training. We performed experiments on CIFAR-10 and Imagenet classification datasets using networks of various structures. Our technique outperformed comparison methods at different pruning rates. Additionally, we confirmed the effectiveness of our technique on the object detection datasets VOC and COCO.
This work is supported by the National Natural Science Foundation of China under Grant U22A2043, and the Unveiling the list of hanging (science and technology research special) of Liaoning province under Grant 2022JH1/10400030.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Chen, J., Zhu, Z., Li, C., Zhao, Y.: Self-adaptive network pruning. In: International Conference on Neural Information Processing, pp. 175–186 (2019)
Elkerdawy, S., Elhoushi, M., Zhang, H., Ray, N.: Fire together wire together: a dynamic pruning approach with self-supervised mask prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12454–12463 (2022)
Gao, X., Zhao, Y., Dudziak, Ł., Mullins, R., Xu, C.Z.: Dynamic channel pruning: feature boosting and suppression. arXiv preprint: arXiv:1810.05331 (2018)
Ge, Z., Liu, S., Wang, F., Li, Z., Sun, J.: YOLOX: exceeding yolo series in 2021. arXiv preprint: arXiv:2107.08430 (2021)
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, vol. 28 (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 770–778 (2016)
He, Y., Kang, G., Dong, X., Fu, Y., Yang, Y.: Soft filter pruning for accelerating deep convolutional neural networks. IJCAI, 2234–2240 (2018)
He, Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4335–4344 (2019)
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, P.H.: Pruning filters for efficient ConvNets. In: International Conference on Learning Representations (2017)
Liebenwein, L., Baykal, C., Lang, H., Feldman, D., Rus, D.: Provable filter pruning for efficient neural networks. ICLR (2020)
Lin, M., Cao, L., Zhang, Y., Shao, L., Lin, C.W., Ji, R.: Pruning networks with cross-layer ranking & k-reciprocal nearest filters. IEEE Trans. Neural Netw. Learn. Syst. (2022)
Lin, M., et al.: HRank: filter pruning using high-rank feature map. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1526–1535 (2020)
Lin, M., Ji, R., Zhang, Y., Zhang, B., Wu, Y., Tian, Y.: Channel pruning via automatic structure search. In: Bessiere, C. (ed.) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 673–679. International Joint Conferences on Artificial Intelligence Organization (2020). main track
Luo, J.H., Wu, J.: AutoPruner: an end-to-end trainable filter pruning method for efficient deep model inference. Pattern Recogn. 107, 107461 (2020)
Rachwan, J., Zügner, D., Charpentier, B., Geisler, S., Ayle, M., Günnemann, S.: Winning the lottery ahead of time: efficient early network pruning. In: International Conference on Machine Learning, pp. 18293–18309 (2022)
Romera, E., Alvarez, J.M., Bergasa, L.M., Arroyo, R.: ERFNet: efficient residual factorized ConvNet for real-time semantic segmentation. IEEE Trans. Intell. Transp. Syst. 19(1), 263–272 (2018)
Saxe, A., McClelland, J., Ganguli, S.: Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In: International Conference on Learning Represenatations (2014)
Su, X., et al.: Searching for network width with bilaterally coupled network. IEEE Trans. Pattern Anal. Mach. Intell. 45(7), 8936–8953 (2023)
Wang, H., Fu, Y.: Trainability preserving neural pruning. In: The Eleventh International Conference on Learning Representations (2023)
Ning, X., Zhao, T., Li, W., Lei, P., Wang, Yu., Yang, H.: DSA: more efficient budgeted pruning via differentiable sparsity allocation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12348, pp. 592–607. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58580-8_35
Zhang, Y., et al.: Learning best combination for efficient n:M sparsity. In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K. (eds.) Advances in Neural Information Processing Systems (2022)
Zhou, A., et al.: Learning n: M fine-grained structured sparse neural networks from scratch. ICLR (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Liu, J. et al. (2024). Adaptive Channel Pruning for Trainability Protection. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14434. Springer, Singapore. https://doi.org/10.1007/978-981-99-8549-4_12
Download citation
DOI: https://doi.org/10.1007/978-981-99-8549-4_12
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8548-7
Online ISBN: 978-981-99-8549-4
eBook Packages: Computer ScienceComputer Science (R0)