More Web Proxy on the site http://driver.im/

research-article

Finding the Secret of CNN Parameter Layout under Strict Size Constraint

Authors:

Ruoyu LiuAuthors Info & Claims

MM '17: Proceedings of the 25th ACM international conference on Multimedia

Pages 997 - 1005

https://doi.org/10.1145/3123266.3123346

Published: 19 October 2017 Publication History

Abstract

Although deep convolutional neural networks (CNNs) have significantly boosted the performance of many computer vision tasks, their complexities~(the size or the number of parameters) are also dramatically increased even with slight performance improvement. However, the larger network leads to more computation requirements, which are unfavorable to resource-constrained scenarios, such as the widely used embedded systems. In this paper, we tentatively explore the essential effect of CNN parameter layout, ıe, the allocation of parameters in the convolution layers, on the discriminative capability of CNN. Instead of enlarging the breadth or depth of networks, we attempt to improve the discriminative ability of CNN by changing its parameter layout under strict size constraint. Toward this end, a novel energy function is proposed to represent the CNN parameter layout, which makes it possible to model the relationship between the allocation of parameters in the convolution layers and the discriminative ability of CNN. According to extensive experimental results with plain CNN models and Residual Nets, we find that the higher the energy of a specific CNN parameter layout is, the better its discriminative ability is. Following this finding, we propose a novel approach to learn the better parameter layout. Experimental results on two public image classification datasets show that the CNN models with the learned parameter layouts achieve the better image classification results under strict size constraint.

References

[1]

Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. 2014. Neural Codes for Image Retrieval. In ECCV.

[2]

Yoshua Bengio and Olivier Delalleau. 2011. On the Expressive Power of Deep Architectures. In International Conference on Algorithmic Learning Theory.

Digital Library

[3]

Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. 2016. An Analysis of Deep Neural Network Models for Practical Applications. arXiv preprint arXiv:1605.07678 (2016).

[4]

George Cybenko. 1989. Approximation by Superpositions of A Sigmoidal Function. MCSS, Vol. 2, 4 (1989), 303--314.

[5]

Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. 2017. Deformable Convolutional Networks. arXiv preprint arXiv:1703.06211 (2017).

[6]

John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research Vol. 12, Jul (2011), 2121--2159.

Digital Library

[7]

Ronen Eldan and Ohad Shamir. 2016. The Power of Depth for Feedforward Neural Networks. arXiv preprint (2016).

[8]

Mark Everingham, L Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. 2008. The pascal visual object classes challenge 2007 (voc 2007) results (2007). (2008).

[9]

Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. 2016. EIE: Efficient Inference Engine on Compressed Deep Neural Network Proceedings of the 43rd International Symposium on Computer Architecture.

Digital Library

[10]

Song Han, Huizi Mao, and William J Dally. 2015. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv preprint arXiv:1510.00149 (2015).

[11]

Johan Hastad. 1986. Almost Optimal Lower Bounds for Small Depth Circuits Proceedings of the eighteenth annual ACM symposium on theory of computing.

Digital Library

[12]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR.

[13]

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989. Multilayer Feedforward Networks Are Universal Approximators. Neural networks, Vol. 2, 5 (1989), 359--366.

Digital Library

[14]

Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. 2016. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993 (2016).

[15]

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. arXiv preprint arXiv:1609.07061 (2016).

[16]

Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. 2016. SqueezeNet: AlexNet-level Accuracy with 50x Fewer Parameters and < 0.5 MB Model Size. arXiv preprint arXiv:1602.07360 (2016).

[17]

Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv preprint arXiv:1502.03167 (2015).

Digital Library

[18]

Hervé Jégou, Matthijs Douze, Cordelia Schmid, and Patrick Pérez. 2010. Aggregating local descriptors into a compact image representation CVPR.

[19]

Alex Krizhevsky and Geoffrey Hinton. 2009. Learning Multiple Layers of Features from Tiny Images. (2009).

[20]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet Classification with Deep Convolutional Neural Networks NIPS.

Digital Library

[21]

Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. 2016. Fractalnet: Ultra-deep Neural Networks without Residuals. arXiv preprint arXiv:1605.07648 (2016).

[22]

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based Learning Applied to Document Recognition. Proc. IEEE Vol. 86, 11 (1998), 2278--2324.

[23]

Min Lin, Qiang Chen, and Shuicheng Yan. 2013. Network in Network. arXiv preprint arXiv:1312.4400 (2013).

[24]

Ruoyu Liu, Yao Zhao, Shikui Wei, Zhenfeng Zhu, Lixin Liao, and Shuang Qiu. 2015. Indexing of CNN Features for Large Scale Image Search. arXiv preprint arXiv:1508.00217 (2015).

[25]

Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully Convolutional Networks for Semantic Segmentation CVPR.

[26]

David G Lowe. 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision Vol. 60, 2 (2004), 91--110.

Digital Library

[27]

Vinod Nair and Geoffrey E Hinton. 2010. Rectified Linear Units Improve Restricted Boltzmann Machines ICML.

Digital Library

[28]

Florent Perronnin and Christopher Dance. 2007. Fisher kernels on visual vocabularies for image categorization CVPR.

[29]

Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. 2016. Xnor-net: Imagenet Classification Using Binary Convolutional Neural Networks ECCV.

[30]

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You Only Look Once: Unified, Real-time Object Detection CVPR.

[31]

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards Real-time Object Detection with Region Proposal Networks NIPS.

Digital Library

[32]

Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-scale Image Recognition. arXiv preprint arXiv:1409.1556 (2014).

[33]

Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research Vol. 15, 1 (2014), 1929--1958.

Digital Library

[34]

Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway Networks. arXiv preprint arXiv:1505.00387 (2015).

[35]

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going Deeper with Convolutions. In CVPR.

[36]

Andreas Veit, Michael J Wilber, and Serge Belongie. 2016. Residual Networks Behave Like Ensembles of Relatively Shallow Networks NIPS.

[37]

Jingdong Wang, Zhen Wei, Ting Zhang, and Wenjun Zeng. 2016. Deeply-fused Nets. arXiv preprint arXiv:1605.07716 (2016).

[38]

Shikui Wei, Dong Xu, Xuelong Li, and Yao Zhao. 2013. Joint optimization toward effective and efficient image search. IEEE transactions on cybernetics Vol. 43, 6 (2013), 2216--2227.

[39]

Shikui Wei, Yao Zhao, Ce Zhu, Changsheng Xu, and Zhenfeng Zhu. 2011. Frame fusion for video copy detection. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 21, 1 (2011), 15--28.

Digital Library

[40]

Shikui Wei, Yao Zhao, Zhenfeng Zhu, and Nan Liu. 2010. Multimodal fusion for video search reranking. IEEE Transactions on Knowledge and Data Engineering, Vol. 22, 8 (2010), 1191--1199.

Digital Library

[41]

Yunchao Wei, Wei Xia, Min Lin, Junshi Huang, Bingbing Ni, Jian Dong, Yao Zhao, and Shuicheng Yan. 2016. HCP: A flexible CNN framework for multi-label image classification. IEEE transactions on pattern analysis and machine intelligence, Vol. 38, 9 (2016), 1901--1907.

[42]

Zifeng Wu, Chunhua Shen, and Anton van den Hengel. 2016. High-performance Semantic Segmentation Using Very Deep Fully Convolutional Networks. arXiv preprint arXiv:1604.04339 (2016).

[43]

Sergey Zagoruyko and Nikos Komodakis. 2016. Wide Residual Networks. arXiv preprint arXiv:1605.07146 (2016).

[44]

Matthew D Zeiler and Rob Fergus. 2014. Visualizing and Understanding Convolutional Networks ECCV.

[45]

Liming Zhao, Jingdong Wang, Xi Li, Zhuowen Tu, and Wenjun Zeng. 2016. On the Connection of Deep Fusion to Ensembling. arXiv preprint arXiv:1611.07718 (2016).

Cited By

Song WFang JWang RTan K(2021)Detection of pig based on improved RESNET model in natural sceneApplied Mathematics and Nonlinear Sciences10.2478/amns.2021.2.000406:2(215-226)Online publication date: 30-Oct-2021
https://doi.org/10.2478/amns.2021.2.00040
Salih SAbdulla HAhmed ZSurameery NRashid R(2020)Modified AlexNet Convolution Neural Network For Covid-19 Detection Using Chest X-ray ImagesKurdistan Journal of Applied Research10.24017/covid.14(119-130)Online publication date: 9-Jun-2020
https://doi.org/10.24017/covid.14
Liao LZhao YWei SWei YWang J(2020)Parameter Distribution Balanced CNNsIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2019.295639031:11(4600-4609)Online publication date: Nov-2020
https://doi.org/10.1109/TNNLS.2019.2956390

Finding the Secret of CNN Parameter Layout under Strict Size Constraint
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches

Recommendations

Deep CNN for Classification of Image Contents
IPMV '21: Proceedings of the 2021 3rd International Conference on Image Processing and Machine Vision

In recent years the classification of images has made great progress and has been used in many fields. However, it may not be possible to classify images perfectly through the CNN because of overfitting and gradient vanishing. Most existing CNNs have ...
RIC-CNN: Rotation-Invariant Coordinate Convolutional Neural Network
Abstract
Due to the lack of rotation invariance in traditional convolution operations, even acting a slight rotation on the input can severely degrade the performance of Convolutional Neural Networks (CNNs). To address this, we propose a Rotation-...
Highlights
- We propose RIC-C: a novel convolutional operation naturally invariant to all input center rotations, no extra parameters or data augmentation.
- Without data augmentation, RIC-CNN shows superior performance on MNIST compared to previous ...
Wavelet-Attention CNN for image classification
Abstract
The feature learning methods based on convolutional neural network (CNN) have successfully produced tremendous achievements in image classification tasks. However, the inherent noise and some other factors may weaken the effectiveness of the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '17: Proceedings of the 25th ACM international conference on Multimedia

October 2017

2028 pages

ISBN:9781450349062

DOI:10.1145/3123266

General Chairs:
Qiong Liu
FXPAL, USA
,
Rainer Lienhart
Universität Augsburg, Germany
,
Haohong Wang
TCL America, USA
,
Program Chairs:
Sheng-Wei "Kuan-Ta" Chen
Academia Sinica, Taiwan
,
Susanne Boll
University of Oldenburg, Germany
,
Phoebe Chen
La Trobe University, Australia
,
Gerald Friedland
Lawrence Livermore National Lab, USA
,
Jia Li
Google, USA
,
Shuicheng Yan
Qihoo 360, China

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

the Fundamental Research Funds for the Central Universities
by National Natural Science Foundation of China
National Key Research and Development of China
Joint Fund of Ministry of Education of China and China Mobile
National Natural Science Foundation of China

Conference

MM '17

Sponsor:

SIGMM

MM '17: ACM Multimedia Conference

October 23 - 27, 2017

California, Mountain View, USA

Acceptance Rates

MM '17 Paper Acceptance Rate 189 of 684 submissions, 28%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
165
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)4

Reflects downloads up to 27 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Song WFang JWang RTan K(2021)Detection of pig based on improved RESNET model in natural sceneApplied Mathematics and Nonlinear Sciences10.2478/amns.2021.2.000406:2(215-226)Online publication date: 30-Oct-2021
https://doi.org/10.2478/amns.2021.2.00040
Salih SAbdulla HAhmed ZSurameery NRashid R(2020)Modified AlexNet Convolution Neural Network For Covid-19 Detection Using Chest X-ray ImagesKurdistan Journal of Applied Research10.24017/covid.14(119-130)Online publication date: 9-Jun-2020
https://doi.org/10.24017/covid.14
Liao LZhao YWei SWei YWang J(2020)Parameter Distribution Balanced CNNsIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2019.295639031:11(4600-4609)Online publication date: Nov-2020
https://doi.org/10.1109/TNNLS.2019.2956390

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents