[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

Efficient feature transform module

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Deep neural networks have achieved impressive success in various applications, but they face challenges when deployed on mobile devices due to limited computational resources. One of the main reasons for this challenge is that these networks employ too many weight maps of convolution layers to generate redundant feature maps to ensure a thorough understanding of the input data, which leads to plenty of parameters and high computational costs. This paper proposes a method to exploit this redundancy efficiently and generate more lightweight convolutional neural networks. First, we employ the peak signal-to-noise ratio (PSNR) to quantitatively verify the similarity in feature maps of the convolution layer, which is also called redundancy. We also find the similarity correspondence between the feature maps and the weight maps. Second, inspired by the analysis, we propose a cost-efficient feature transform module (FTM) to generate redundant feature maps for replacing standard convolution layers and getting lightweight convolutional neural networks (CNNs). The FTM adopts a point-wise and a depth-wise convolution to obtain the prime weight maps which could extract distinguished feature maps based on the similarity correspondence. Then, these meaningful feature maps are broadcast by an efficient neighbor-1D convolution, ensuring the feature redundancy of the network information flow. Extensive experiments on image classification benchmark datasets (CIFAR-10, CIFAR-100, ImageNet) show that the FTM outperforms existing methods in terms of efficiency. In addition, if used as a drop-in replacement for standard architectures such as ResNet-50, the FTM variant could achieve a comparable score with only 21% parameters of the original layer. The code of the proposed method is available at: https://github.com/li-ju-bazhong/Efficient-feature-transform-module.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data Availability

The datasets analysed during the current study are available in the CIFAR (https://www.cs.toronto.edu/~kriz/cifar.html) and ImageNet (https://www.image-net.org).

References

  1. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. Advances in neural information processing systems 28

  2. He KM, Zhang XY, Ren SQ, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  3. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition

  4. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440

  5. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  6. Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1492–1500

  7. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1251–1258

  8. Real E, Aggarwal A, Huang Y, Le QV (2019) Regularized evolution for image classifier architecture search. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 4780–4789

  9. LeCun Y, Denker J, Solla S (1989) Optimal brain damage. Advances in neural information processing systems 2

  10. Liu Z, Li J, Shen Z, Huang G, Yan S, Zhang C (2017) Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE international conference on computer vision, pp 2736–2744

  11. Luo J.-H, Wu J, Lin W (2017) ThiNet: a filter level pruning method for deep neural network compression. In: Proceedings of the IEEE international conference on computer vision, pp 5058–5066

  12. Rastegari M, Ordonez V, Redmon J, Farhadi A (2016) Xnor-net: Imagenet classification using binary convolutional neural networks. In: European conference on computer vision. Springer, pp 525–542

  13. Hubara I, Courbariaux M, Soudry D, El-Yaniv R, Bengio Y (2016) Binarized neural networks. Advances in neural information processing systems 29

  14. Chen W, Wilson J, Tyree S, Weinberger K, Chen Y (2015) Compressing neural networks with the hashing trick. In: International conference on machine learning. PMLR, pp 2285–2294

  15. Jacob B, Kligys S, Chen B, Zhu M, Tang M, Howard A, Adam H, Kalenichenko D (2018) Quantization and training of neural networks for efficient integer-arithmetic-only inference. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2704–2713

  16. Chen H, Wang Y, Xu C, Yang Z, Liu C, Shi B, Xu C, Xu C, Tian Q (2019) Data-free learning of student networks. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 3514–3522

  17. Han B, Yao Q, Yu X, Niu G, Xu M, Hu W, Tsang I, Sugiyama M (2018) Co-teaching: robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems 31

  18. Hinton G, Vinyals O, Dean J, et al (2015) Distilling the knowledge in a neural network 2(7). arXiv:1503.02531

  19. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  20. Zhang X, Zhou X, Lin M, Sun J (2018) ShuffleNet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6848–6856

  21. Howard A.G, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861

  22. Howard A, Sandler M, Chu G, Chen L-C, Chen B, Tan M, Wang W, Zhu Y, Pang R, Vasudevan V et al (2019) Searching for MobileNetV3. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 1314–1324

  23. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L.-C (2018) MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520

  24. Ma N, Zhang X, Zheng H-T, Sun J (2018) ShuffleNet V2: practical guidelines for efficient CNN architecture design. In: Proceedings of the European conference on computer vision (ECCV), pp 116–131

  25. Han K, Wang Y, Tian Q, Guo J, Xu C, Xu C (2020) Ghostnet: more features from cheap operations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1580–1589

  26. Li Y, Liu D, Li H, Li L, Li Z, Wu F (2019) Learning a convolutional neural network for image compact-resolution. IEEE Trans Image Process 28(3):1092–1107. https://doi.org/10.1109/TIP.2018.2872876

    Article  MathSciNet  Google Scholar 

  27. Isogawa K, Ida T, Shiodera T, Takeguchi T (2018) Deep shrinkage convolutional neural network for adaptive noise reduction. IEEE Signal Process Lett 25(2):224–228. https://doi.org/10.1109/LSP.2017.2782270

    Article  Google Scholar 

  28. Dong C, Loy CC, He K, Tang X (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  29. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. Advances in neural information processing systems 25

  30. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-first AAAI conference on artificial intelligence

  31. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  32. Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size. arXiv:1602.07360

  33. Zhong H, Lv Y, Yuan R, Yang D (2022) Bearing fault diagnosis using transfer learning and self-attention ensemble lightweight convolutional neural network. Neurocomputing 501:765–777

    Article  Google Scholar 

  34. Krizhevsky A, Hinton G et al (2009) Learning multiple layers of features from tiny images

  35. Silva E.A, Panetta K, Agaian SS (2007) Quantifying image similarity using measure of enhancement by entropy. In: Mobile multimedia/image processing for military and security applications 2007, vol 6579, pp 219–230. SPIE

  36. Eckert MP, Bradley AP (1998) Perceptual quality metrics applied to still image compression. Signal Process 70(3):177–200

    Article  Google Scholar 

  37. Haase D, Amthor M (2020) Rethinking depthwise separable convolutions: how intra-kernel correlations lead to improved MobileNets. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14600–14609

  38. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, pp 448–456

  39. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) ImageNet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. IEEE, pp 248–255

  40. Wang Q, Wu B, Zhu P, Li P, Zuo W, Hu Q (2020) ECA-NET: efficient channel attention for deep convolutional neural networks, pp 11531–11539

  41. Tan M, Le QV (2019) EfficientNet: rethinking model scaling for convolutional neural networks. arXiv:1905.11946

  42. Nie C, Wang H (2018) Tensor neural networks via circulant convolution. Neurocomputing 483:22–31

    Article  Google Scholar 

  43. Yu R, Li A, Chen C-F, Lai J-H, Morariu VI, Han X, Gao M, Lin C-Y, Davis LS (2018) NISP: pruning networks using neuron importance score propagation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9194–9203

  44. Huang Z, Wang N (2018) Data-driven sparse structure selection for deep neural networks. In: Proceedings of the European conference on computer vision (ECCV), pp 304–320

  45. Yu J, Yang L, Xu N, Yang J, Huang T (2019) Slimmable neural networks

  46. Wu B, Wan A, Yue X, Jin P, Zhao S, Golmant N, Gholaminejad A, Gonzalez J, Keutzer K (2018) Shift: a zero flop, zero parameter alternative to spatial convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9127–9135

  47. Molchanov P, Mallya A, Tyree S, Frosio I, Kautz J (2019) Importance estimation for neural network pruning. In: ICLR, pp 11264–11272

  48. Luo J-H, Wu J (2020) Neural network pruning with residual-connections and limited-data. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1458–1467

  49. Liu Z, Mu H, Zhang X, Guo Z, Yang X, Cheng K-T, Sun J (2019) Metapruning: meta learning for automatic neural network channel pruning. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 3296–3305

Download references

Acknowledgements

This work was supported by the National Key Research and Development Program of China key special project under grant 2018YFB1700702, the Key Research and Development Projects of Sichuan Province under the grant (2022YFG0246, 22ZDZX0051, 2021YFS0021), and the 2022 Dazhou Cooperation Project of Sichuan University under grant 2022CDDZ-06.

Author information

Author notes

  1. Ju Li and Yang Wei contributed equally to this work.

Authors and Affiliations

  1. School of Mechanical Engineering and the Innovation Method and Creative Design Key Laboratory of Sichuan Province, Sichuan University, Chengdu, 610065, China

    Ju Li, Yang Wei, Kai Wang & Huiyang Xie

Authors
  1. Ju Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Huiyang Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kai Wang.

Ethics declarations

Conflicts of interest

The authors have declared that no confict of interest exists.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, J., Wei, Y., Wang, K. et al. Efficient feature transform module. Multimed Tools Appl 83, 76873–76889 (2024). https://doi.org/10.1007/s11042-024-18581-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-024-18581-6

Keywords

Search

Navigation