More Web Proxy on the site http://driver.im/

research-article

Combining deep residual neural network features with supervised machine learning algorithms to classify diverse food image datasets

Authors:

Patrick McAllister,

Anne MoorheadAuthors Info & Claims

Volume 95, Issue C

Pages 217 - 233

https://doi.org/10.1016/j.compbiomed.2018.02.008

Published: 01 April 2018 Publication History

Abstract

Obesity is increasing worldwide and can cause many chronic conditions such as type-2 diabetes, heart disease, sleep apnea, and some cancers. Monitoring dietary intake through food logging is a key method to maintain a healthy lifestyle to prevent and manage obesity. Computer vision methods have been applied to food logging to automate image classification for monitoring dietary intake. In this work we applied pretrained ResNet-152 and GoogleNet convolutional neural networks (CNNs), initially trained using ImageNet Large Scale Visual Recognition Challenge (ILSVRC) dataset with MatConvNet package, to extract features from food image datasets; Food 5K, Food-11, RawFooT-DB, and Food-101. Deep features were extracted from CNNs and used to train machine learning classifiers including artificial neural network (ANN), support vector machine (SVM), Random Forest, and Naive Bayes. Results show that using ResNet-152 deep features with SVM with RBF kernel can accurately detect food items with 99.4% accuracy using Food-5K validation food image dataset and 98.8% with Food-5K evaluation dataset using ANN, SVM-RBF, and Random Forest classifiers. Trained with ResNet-152 features, ANN can achieve 91.34%, 99.28% when applied to Food-11 and RawFooT-DB food image datasets respectively and SVM with RBF kernel can achieve 64.98% with Food-101 image dataset. From this research it is clear that using deep CNN features can be used efficiently for diverse food item image classification. The work presented in this research shows that pretrained ResNet-152 features provide sufficient generalisation power when applied to a range of food image classification tasks.

References

[1]

M. Di Cesare, et al., Trends in adult body-mass index in 200 countries from 1975 to 2014: a pooled analysis of 1698 population-based measure- ment studies with 19.2 million participants, Lancet 387 (10026) (2016) 1377–1396.

[2]

Y. He, C. Xu, N. Khanna, C.J. Boushey, E.J. Delp, Food image analysis: segmentation, identification and weight estimation, Proc. - IEEE Int. Conf. Multimed. Expo (2013),.

[3]

T. Lehnert, D. Sonntag, A. Konnopka, S. Riedel-Heller, H.-H. Knig, Economic costs of overweight and obesity, Best Pract. Res. Clin. En- docrinol. Metab 27 (2) (2013) 105–115.

[4]

C.M. Wharton, C.S. Johnston, B.K. Cunningham, D. Sterner, Dietary self-monitoring, but not dietary quality, improves with use of smartphone app technology in an 8-week weight loss trial, J. Nutr. Educ. Behav. 46 (5) (2014) 440–444.

[5]

M.M. Anthimopoulos, L. Gianola, L. Scarnato, P. Diem, S.G. Mougiakakou, A food recognition system for diabetic patients based on an optimized bag-of-features model, IEEE J. Biomed. Heal. Informatics 18 (4) (2014) 1261–1271.

[6]

G.M. Farinella, M. Moltisanti, S. Battiato, Food recognition using consensus vocabularies, Lecture Notes in Computer Science (Includ- Ing Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9281, 2015, p. 384392.

[7]

H. He, F. Kong, J. Tan, DietCam: multiview food recognition using a multikernel SVM, IEEE J. Biomed. Heal. Informatics 20 (3) (2016) 848–855.

[8]

N. Martinel, C. Piciarelli, C. Micheloni, G.L. Foresti, A structured committee for food recognition, Proceedings of the IEEE in- Ternational Conference on Computer Vision, 2015 February, 2015, pp. 484–492.

[9]

ImageNetLargeScaleVisualRecognitionCompetition(ILSVRC)”, Image- net.org, 2017, [Online]. Available: http://www.image-net.org/challenges/LSVRC/ [Accessed: 16- Sep- 2017].

[10]

A. Krizhevsky, I. Sutskever, G.E. Hinton, ImageNet classification with deep convolutional neural networks, Adv. Neural Inf. Process. Syst. (2012) 19.

[11]

N. Tajbakhsh, et al., Convolutional neural networks for medical image analysis: full training or fine tuning?, IEEE Trans. Med. Imag. 35 (5) (2016) 12991312.

[12]

A.S. Razavian, H. Azizpour, J. Sullivan, S. Carlsson, CNN features off-the-shelf: an astounding baseline for recognition, in: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Work- Shops, 2014, p. 512519.

[13]

A. Singla, L. Yuan, T. Ebrahimi, Food/Non-food image classification and food categorization using pre-trained GoogLeNet model, in: Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management - MADiMa 16, 2016, p. 311.

[14]

H, K. Aizawa, M. Ogawa, Food Detection and Recognition Using Convolutional Neural Network, vol. 2, ACM Multimed, 2014.

[15]

M. Farooq, E. Sazonov, Feature extraction using deep learning for food type recognition, in: I. Rojas, F. Ortuo (Eds.), Bioinformatics and Biomedical Engineering: 5th International Work-conference, IWBBIO 2017, Granada, Spain, April 26–28, 2017, Proceedings, Part I, Springer International Publishing, Cham, 2017, pp. 464–472.

[16]

Y. Kawano, K. Yanai, Food image recognition with deep convolutional features, ACM Int. Jt. Conf. Pervasive Ubiquitous Comput (2014) 589–593.

[17]

M. Bossard, Guillaumin, L. Van Gool, Food-101-Mining discriminative components with random forests, Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8694, 2014, pp. 446–461. LNCS, no. PART 6.

[18]

Y. Matsuda, H. Hoashi, K. Yanai, Recognition of multiple-food im- ages by detecting candidate regions, in: Proceedings - IEEE International Conference on Multimedia and Expo, 2012, p. 2530.

[19]

Y. Kawano, K. Yanai, Automatic expansion of a food image dataset leveraging existing categories with domain adaptation, Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intel- Ligence and Lecture Notes in Bioinformatics), vol. 8927, 2015, p. 317.

[20]

C. Cusano, P. Napoletano, R. Schettini, Local angular patterns for color texture classification, Lecture Notes in Computer Science (Includ- Ing Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9281, 2015, pp. 111–118.

[21]

A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, F.F. Li, Large-scale video classification with convolutional neural networks, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2014, pp. 1725–1732.

Digital Library

[22]

C. Szegedy, et al., Going deeper with convolutions, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2015, p. 19. vol. 7-12-2015.

[23]

A. Vedaldi, K. Lenc, MatConvNet, in: Proceedings of the 23rd ACM International Conference on Multimedia - MM 15, 2015, pp. 689–692.

[24]

Image Category Classification Using Deep Learning, Mathworks.com, 2017, [Online]. Available: https://www.mathworks.com/examples/matlab-computer-vision/mw/vision product- DeepLearningImageClassificationExample-image-category-classification-using- deep-learning [Accessed: 18- Sep- 2017].

[25]

K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.

[26]

K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, in: Int. Conf. Learn. Represent, 2015, p. 114.

[27]

R.G. Pontius, M. Millones, Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment, Int. J. Remote Sens. 32 (15) (2011) 4407–4429.

[28]

Weka 3-Data Mining with Open Source Machine Learning Software in Java, Cs.waikato.ac.nz, 2017, [Online]. Available: http://www.cs.waikato.ac.nz/ml/weka/index.html [Accessed: 18- Sep- 2017].

[29]

Waikato Environment for Knowledge Analysis(WEKA), Weka.sourceforge.net, 2017, [Online]. Available: http://weka.sourceforge.net/packageMetaData/wekaPython/Latest.html [Accessed: 18- Sep- 2017].

[30]

Java (Convolutional or Fully-connected) Neural Network Implementation, GitHub, 2017, [Online]. Available: https://github.com/amten/NeuralNetwork/releases/tag/v1.1 [Accessed: 18- Sep- 2017].

[31]

H. Zhang, The optimality of naive bayes, Proc. Seventeenth Int. Florida Artif. Intell. Res. Soc. Conf. FLAIRS 2004 1 (2) (2004) 16.

[32]

I.H. Witten, E. Frank, M. a Hall, Data Mining: Practical Machine Learning Tools and Techniques, 2011.

[33]

T. Malisiewicz, A. Gupta, A.A. Efros, Ensemble of exemplar-SVMs for object detection and beyond, in: Proceedings of the IEEE International Conference on Computer Vision, 2011, pp. 89–96.

[34]

C.-W. Hsu, C.-J. Lin, A comparison of methods for multiclass sup- port vector machines, IEEE Trans. Neural Network. 13 (2) (2002) 415–425.

[35]

G. James, D. Witten, T. Hastie, R. Tibishirani, An Introduction to Statistical Learning, 2013.

Digital Library

[36]

J.P. Mueller, et al., Hitting Complexity with Neural Networks,” Machine Learning for Dummies, Wiley, Hoboken, New Jersey, 2016, pp. 279–290. ch. 16.

[37]

L. Breiman, Random forests, Mach. Learn. 45 (1) (2001) 532.

[38]

H. Bay, A. Ess, T. Tuytelaars, L. Van Gool, Speeded-up robust features (SURF), Comput. Vis. Image Understand. 110 (3) (2008) 346359.

[39]

T. Ojala, M. Pietikinen, T. Menp, Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, IEEE Trans. Pattern Anal. Mach. Intell. 24 (7) (2002) 971987.

[40]

GPU vs CPU in Convolutional Neural Networks Using TensorFlow — Relink, Relink, 2017, [Online]. Available: https://relinklabs.com/gpu- vs-cpu-in-convolutional-neural-networks-using-tensorflow [Accessed: 19- Sep- 2017].

[41]

H. Hassannejad, G. Matrella, P. Ciampolini, I. De Munari, M. Mordonini, S. Cagnoni, Automatic diet monitoring: a review of computer vision and wearable sensor-based methods, Int. J. Food Sci. Nutr. (2017) 115.

[42]

RawFooT DB, Projects.ivl.disco.unimib.it, 2017, [Online]. Available: http://projects.ivl.disco.unimib.it/minisites/rawfoot// [Accessed: 19- Sep- 2017].

[43]

F. Ragusa, V. Tomaselli, A. Furnari, S. Battiato, G.M. Farinella, Food vs non-food classification, in: Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management - MADiMa 16, 2016, pp. 77–81.

[44]

E. Aguilar, et al., Exploring Food Detection using CNNs, arXiv, 2017, 1709.04800v1 [cs], Sept.

[45]

K. Yanai, Y. Kawano, Food image recognition using deep convolu- tional network with pre-training and fine-tuning, in: 2015 IEEE Interna- Tional Conference on Multimedia & Expo Workshops (ICMEW), 2015, pp. 1–6.

[46]

X. Wang, D. Kumar, N. Thome, M. Cord, F. Precioso, Recipe recog- nition with large multimodal food dataset, in: 2015 IEEE International Conference on Multimedia and Expo Workshops, ICMEW, 2015, p. 2015.

[47]

Kagaya, K. Aizawa, Highly Accurate Food/Non-food Image Classifi- Cation Based on a Deep Convolutional Neural Network BT - New Trends in Image Analysis and Processing – ICIAP 2015 Workshops: ICIAP 2015 International Workshops, BioFor, CTMR, RHEUMA, ISCA, MADiMa, SBMI, and QoEM, Genoa, Italy, September 7-8, 2015, Proceedings, V. Murino, E. Puppo, D. Sona, M. Cristani, and C. Sansone, Eds. Cham: Springer International Publishing, 2015, pp. 350–357.

[48]

C. Liu, Y. Cao, Y. Luo, G. Chen, V. Vokkarane, Y. Ma, Deepfood: deep learning-based food image recognition for computer-aided dietary assessment, Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformat- Ics), vol. 9677, 2016, pp. 37–48.

[49]

H. Hassannejad, G. Matrella, P. Ciampolini, I. De Munari, M. Mordonini, S. Cagnoni, Food image recognition using very deep convolutional networks, in: Proceedings of the 2nd International Workshop on Multime- Dia Assisted Dietary Management - MADiMa 16, 2016, pp. 41–49.

[50]

Y. Kawano, K. Yanai, FoodCam-256: a Large-scale Real-time Mobile Food Recognition System Employing High-dimensional Features and Compression of Classifier Weights, 2014, pp. 761–762. MM 14.

[51]

A.S. Razavian, H. Azizpour, J. Sullivan, S. Carlsson, CNN features off-the-shelf: an astounding baseline for recognition, in: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Work- Shops, 2014, pp. 512–519.

[52]

L. Fei-Fei, R. Fergus, P. Perona, Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories, Comput. Vis. Image Understand. 106 (1) (2007) 59–70.

[53]

G.M. Farinella, D. Allegra, F. Stanco, A benchmark dataset to study the representation of food images, Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8927, 2015, pp. 584–599.

[54]

Welcome to Python.org, Python.org, 2017, [Online]. Available: https://www.python.org/ [Accessed: 24- Nov- 2017].

[55]

Scikit-learn Machine Learning in Python, Scikit-learn.org, 2017, [On- line]. Available: http://scikit-learn.org/ [Accessed: 24- Nov- 2017].

[56]

S. Boseley, Global Cost of Obesity-related Illness to Hit $1.2tn a Year from 2025, the Guardian, 2017, [Online]. Available: https://www.theguardian.com/society/2017/oct/10/trea obesity-related-illness-will-cost-12tn-a-year-from-2025-experts-warn [Accessed: 24- Nov- 2017].

[57]

C. Cusano, P. Napoletano, R. Schettini, Evaluating color texture descriptors under large variations of controlled lighting conditions, J. Opt. Soc. Am. 33 (1) (2015) 17.

[58]

Pretrained CNNs - MatConvNet, 2018, [Online]. Available: Vlfeat.org http://www.vlfeat.org/matconvnet/pretrained/ [Accessed: 11- Feb- 2018].

[59]

Jia Deng, Wei Dong, R. Socher, Li-Jia Li, Kai Li, Li Fei-Fei, ImageNet: a large-scale hierarchical image database, in: 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255.

[60]

How Being Overweight Causes Cancer, Cancer Research UK, 2018, Available: http://www.cancerresearchuk.org/about-cancer/causes-of-cancer/obesity-weight-and-cancer/how-being-overweight-causes-cancer.

Cited By

Phiphitphatphaisit SSurinta O(2024)Multi-layer adaptive spatial-temporal feature fusion network for efficient food image recognitionExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.124834255:PDOnline publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1016/j.eswa.2024.124834
Zhang ROuyang DHe LKuang LBai H(2024)Recognize after early fusion: the Chinese food recognition based on the alignment of image and ingredientsMultimedia Systems10.1007/s00530-024-01297-w30:2Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1007/s00530-024-01297-w
Josephin Shermila PAhilan AJasmine Gnana Malar AJothin R(2023)MDEEPFICJournal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology10.3233/JIFS-23019345:2(3137-3148)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.3233/JIFS-230193
Show More Cited By

Index Terms

Combining deep residual neural network features with supervised machine learning algorithms to classify diverse food image datasets
1. Applied computing
  1. Life and medical sciences
2. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
  2. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches
      1. Neural networks

Index terms have been assigned to the content through auto-classification.

Recommendations

Image Retrieval Using Fused Deep Convolutional Features

This paper proposes an image retrieval using fused deep convolutional features to solve the semantic gap between low-level features and high-level semantic features of traditional contend-based image retrieval method. Firstly, the improved network ...
Content-based image retrieval with compact deep convolutional features

Convolutional neural networks (CNNs) with deep learning have recently achieved a remarkable success with a superior performance in computer vision applications. Most of CNN-based methods extract image features at the last layer using a single CNN ...
Food image recognition with deep convolutional features
UbiComp '14 Adjunct: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication

In this paper, we report the feature obtained from the Deep Convolutional Neural Network boosts food recognition accuracy greatly by integrating it with conventional hand-crafted image features, Fisher Vectors with HoG and Color patches. In the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Computers in Biology and Medicine

Computers in Biology and Medicine Volume 95, Issue C

Apr 2018

307 pages

ISSN:0010-4825

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 April 2018

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Phiphitphatphaisit SSurinta O(2024)Multi-layer adaptive spatial-temporal feature fusion network for efficient food image recognitionExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.124834255:PDOnline publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1016/j.eswa.2024.124834
Zhang ROuyang DHe LKuang LBai H(2024)Recognize after early fusion: the Chinese food recognition based on the alignment of image and ingredientsMultimedia Systems10.1007/s00530-024-01297-w30:2Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1007/s00530-024-01297-w
Josephin Shermila PAhilan AJasmine Gnana Malar AJothin R(2023)MDEEPFICJournal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology10.3233/JIFS-23019345:2(3137-3148)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.3233/JIFS-230193
Min WWang ZLiu YLuo MKang LWei XWei XJiang S(2023)Large Scale Visual Food RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.323787145:8(9932-9949)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1109/TPAMI.2023.3237871
Liu DZhao LWang YKato J(2023)Learn from each other to Classify betterPattern Recognition10.1016/j.patcog.2023.109550140:COnline publication date: 24-May-2023
https://dl.acm.org/doi/10.1016/j.patcog.2023.109550
Zhang YDeng LZhu HWang WRen ZZhou QLu SSun SZhu ZGorriz JWang S(2023)Deep learning in food category recognitionInformation Fusion10.1016/j.inffus.2023.10185998:COnline publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1016/j.inffus.2023.101859
Pan XHe JPeng AZhu FMougiakakou SFarinella GYanai KAllegra D(2022)Simulating Personal Food Consumption Patterns using a Modified Markov ChainProceedings of the 7th International Workshop on Multimedia Assisted Dietary Management10.1145/3552484.3555747(61-69)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3552484.3555747
Fei ZYang EYu LLi XZhou HZhou W(2022)A Novel deep neural network-based emotion analysis system for automatic detection of mild cognitive impairment in the elderlyNeurocomputing10.1016/j.neucom.2021.10.038468:C(306-316)Online publication date: 11-Jan-2022
https://dl.acm.org/doi/10.1016/j.neucom.2021.10.038
Aguilar ENagarajan BRadeva P(2022)Uncertainty-aware selecting for an ensemble of deep food recognition modelsComputers in Biology and Medicine10.1016/j.compbiomed.2022.105645146:COnline publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1016/j.compbiomed.2022.105645
Aktı ŞQaraqe MEkenel H(2022)A Mobile Food Recognition System for Dietary AssessmentImage Analysis and Processing. ICIAP 2022 Workshops10.1007/978-3-031-13321-3_7(71-81)Online publication date: 23-May-2022
https://dl.acm.org/doi/10.1007/978-3-031-13321-3_7
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents