Gradient-Boosted Based Structured and Unstructured Learning
Pages 439 - 451
Abstract
We propose two frameworks to deal with problem settings in which both structured and unstructured data are available. Structured data problems are best solved by traditional machine learning models such as boosting and tree-based algorithms, whereas deep learning has been widely applied to problems dealing with images, text, audio, and other unstructured data sources. However, for the setting in which both structured and unstructured data are accessible, it is not obvious what the best modeling approach is to enhance performance on both data sources simultaneously. Our proposed frameworks allow joint learning on both kinds of data by integrating the paradigms of boosting models and deep neural networks. The first framework, the boosted-feature-vector deep learning network, learns features from the structured data using gradient boosting and combines them with embeddings from unstructured data via a two-branch deep neural network. Secondly, the two-weak-learner boosting framework extends the boosting paradigm to the setting with two input data sources. We present and compare first- and second-order methods of this framework. Our experimental results on both public and real-world datasets show performance gains achieved by the frameworks over selected baselines by magnitudes of 0.1%–4.7%.
References
[1]
Balas VE, Roy SS, Sharma D, and Samui P Handbook of Deep Learning Applications 2019 Cham Springer
[2]
Blackard, J.A.: UCI machine learning repository (1999). https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/ covtype.info
[3]
Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., Kasneci, G.: Deep neural networks and tabular data: a survey. arXiv preprint arXiv:2110.01889 (2021)
[4]
Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification and Regression Trees (1984)
[5]
Caruana, R., Karampatziakis, N., Yessenalina, A.: An empirical evaluation of supervised learning in high dimensions. In: International Conference on Machine Learning (2008)
[6]
Chen M, Hao Y, Hwang K, Wang L, and Wang L Disease prediction by machine learning over big data from healthcare communities IEEE Access 2017 5 8869-8879
[7]
Chen, T., Guestrin, C.: XGBoost: a scalable tree boosting system. In: International Conference on Knowledge Discovery and Data Mining (2016)
[8]
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (2009)
[9]
Dua, D., Graff, C.: UCI machine learning repository (2017). https://archive.ics.uci.edu/ml/datasets/Census-Income+%28KDD%29
[10]
Freund Y and Schapire RE A decision-theoretic generalization of on-line learning and an application to boosting J. Comput. Syst. Sci. 1997 55 119-139
[11]
Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. Ann. Stat., 337–374 (2000)
[12]
Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Ann. Stat., 1189–1232 (2000)
[13]
Gorishniy, Y., Rubachev, I., Khrulkov, V., Babenko, A.: Revisiting deep learning models for tabular data. In: International Conference on Advances in Neural Information Processing Systems (2021)
[14]
Goyal, A., Morvant, E., Germain, P., Amini, M.: Multiview boosting by controlling the diversity and the accuracy of view-specific voters. arXiv preprint arXiv:1808.05784 (2018)
[15]
Graham, B.: Kaggle diabetic retinopathy detection competition report. Technical report, University of Warwick (2015)
[16]
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)
[17]
Ke, G., et al.: LightGBM: a highly efficient gradient boosting decision tree. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
[18]
Koço, S., Capponi, C., Béchet, F.: Applying multiview learning algorithms to human-human conversation classification. In: Conference of the International Speech Communication Association (2012)
[19]
Lahiri A, Paria B, and Biswas PK Forward stagewise additive model for collaborative multiview boosting IEEE Trans. Neural Netw. Learn. Syst. 2018 29 470-485
[20]
Li, P., Qin, Z., Wang, X., Metzler, D.: Combining decision trees and neural networks for learning-to-rank in personal search. In: International Conference on Knowledge Discovery and Data Mining (2019)
[21]
Lloyd JR GEFCom2012 hierarchical load forecasting: gradient boosting machines and Gaussian processes Int. J. Forecast. 2014 30 369-374
[22]
Mangal, A., Kumar, N.: Using big data to enhance the Bosch production line performance: a Kaggle challenge. arXiv preprint arXiv:1701.00705 (2017)
[23]
Mayr A, Binder H, Gefeller O, and Schmid M The evolution of boosting algorithms - from machine learning to statistical modelling Methods Inf. Med. 2014 53 419-27
[24]
Moghimi, M., Belongie, S., Saberian, M., Yang, J., Vasconcelos, N., Li, L.J.: Boosted convolutional neural networks. In: British Machine Vision Conference (2016)
[25]
Nguyen, C.V., et al.: Multimodal machine learning for credit modeling. In: IEEE Computer Society Signature Conference on Computers, Software and Applications (2021)
[26]
Peng J, Aved AJ, Seetharaman G, and Palaniappan K Multiview boosting with information propagation for classification IEEE Trans. Neural Netw. Learn. Syst. 2018 29 657-669
[27]
Redmon, J., Divvala, S.K., Girshick, R.B., Farhadi, A.: You only look once: unified, real-time object detection. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)
[28]
Ridgeway, G.: The state of boosting. In: Computing Science and Statistics (1999)
[29]
Saberian, M.J., Masnadi-Shirazi, H., Vasconcelos, N.: TaylorBoost: first and second-order boosting algorithms with explicit margin control. In: IEEE Conference on Computer Vision and Pattern Recognition (2011)
[30]
Saberian, M.J., Vasconcelos, N.: Multiclass boosting: theory and algorithms. In: Advances in Neural Information Processing Systems, vol. 24 (2011)
[31]
Sarawgi, U., Khincha, R., Zulfikar, W., Ghosh, S., Maes, P.: Uncertainty-aware boosted ensembling in multi-modal settings. arXiv preprint arXiv:2104.10715 (2021)
[32]
Schwenk, H., Bengio, Y.: Boosting neural networks. Neural Computation (2000)
[33]
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)
[34]
Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian optimization of machine learning algorithms. In: Advances in Neural Information Processing Systems, vol. 25 (2012)
[35]
Zou, H., Xu, K., Li, J.: The YouTube-8M Kaggle Competition: challenges and methods. arXiv preprint arXiv:1706.09274 (2017)
Recommendations
Boosted Embeddings for Time-Series Forecasting
Machine Learning, Optimization, and Data ScienceAbstractTime-series forecasting is a fundamental task emerging from diverse data-driven applications. Many advanced autoregressive methods such as ARIMA were used to develop forecasting models. Recently, deep learning based methods such as DeepAR, ...
Deep reinforcement learning boosted by external knowledge
SAC '18: Proceedings of the 33rd Annual ACM Symposium on Applied ComputingRecent improvements in deep reinforcement learning have allowed to solve problems in many 2D domains such as Atari games. However, in complex 3D environments, numerous learning episodes are required which may be too time consuming or even impossible ...
Understanding transfer learning and gradient-based meta-learning techniques
AbstractDeep neural networks can yield good performance on various tasks but often require large amounts of data to train them. Meta-learning received considerable attention as one approach to improve the generalization of these networks from a limited ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Sep 2023
623 pages
ISBN:978-3-031-44212-4
DOI:10.1007/978-3-031-44213-1
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 26 September 2023
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 13 Dec 2024