Abstract
A novel multi-objective algorithm denoted as MO-BayONet is proposed for the Neural Architecture Search (NAS) in this paper. The method based on Bayesian optimization encodes the candidate architectures directly as lists of layers and constructs an extra feature vector for the corresponding surrogate model. The general method allows to accompany the search for the optimal network by additional criteria besides the network performance. The NAS method is applied to combine classification accuracy with network size on two benchmark datasets here. The results indicate that MO-BayONet is able to outperform an available genetic algorithm based approach.
The work is supported by the project GA 22-02067S (“AppNeCo: Approximate Neurocomputing”) of the Czech Science Foundation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Archetti, F., Candelieri, A.: Bayesian Optimization and Data Science. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24494-1
Brochu, E., Cora, V.M., de Freitas, N.: A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning (2010)
Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002). https://doi.org/10.1109/4235.996017
Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. J. Mach. Learn. Res. 20(1), 1997–2017 (2019)
Eriksson, D., et al.: Latency-aware neural architecture search with multi-objective Bayesian optimization. CoRR abs/2106.11890 (2021). arxiv.org/abs/2106.11890
Fortin, F.A., De Rainville, F.M., Gardner, M.A., Parizeau, M., Gagné, C.: DEAP: evolutionary algorithms made easy. J. Mach. Learn. Res. 13, 2171–2175 (2012)
Galuzio, P.P., de Vasconcelos Segundo, E.H., dos Santos Coelho, L., Mariani, V.C.: MOBOpt - multi-objective Bayesian optimization. SoftwareX 12, 100520 (2020). https://doi.org/10.1016/j.softx.2020.100520. http://www.sciencedirect.com/science/article/pii/S2352711020300911
Goodfellow, I., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). Software available from tensorflow.org. https://www.tensorflow.org/
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016). http://www.deeplearningbook.org
Kandasamy, K., Krishnamurthy, A., Schneider, J., Póczos, B.: Parallelised Bayesian optimisation via Thompson sampling. In: AISTATS. Proceedings of Machine Learning Research, vol. 84, pp. 133–142. PMLR (2018)
Kandasamy, K., Neiswanger, W., Schneider, J., Póczos, B., Xing, E.P.: Neural architecture search with Bayesian optimisation and optimal transport. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS 2018, Red Hook, NY, USA, pp. 2020–2029. Curran Associates Inc. (2018)
Kitano, H.: Designing neural networks using genetic algorithms with graph generation system. Complex Syst. 4, 461–476 (1990)
Krizhevsky, A., Nair, V., Hinton, G.: The CIFAR-10 dataset. http://www.cs.toronto.edu/kriz/cifar.html
Lecun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015). https://doi.org/10.1038/nature14539
LeCun, Y., Cortes, C.: The MNIST database of handwritten digits (2012). http://research.microsoft.com/apps/pubs/default.aspx?id=204699
Miikkulainen, R., et al.: Evolving deep neural networks. CoRR abs/1703.00548 (2017). http://arxiv.org/abs/1703.00548
Mrazek, V., Sarwar, S.S., Sekanina, L., Vasicek, Z., Roy, K.: Design of power-efficient approximate multipliers for approximate artificial neural networks. In: 2016 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1–7 (2016). https://doi.org/10.1145/2966986.2967021
Rasmussen, C.E., Nickisch, H.: Gaussian processes for machine learning (GPML) toolbox. J. Mach. Learn. Res. 11, 3011–3015 (2010)
Real, E., Aggarwal, A., Huang, Y., Le, Q.: Regularized evolution for image classifier architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, February 2018. https://doi.org/10.1609/aaai.v33i01.33014780
Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian optimization of machine learning algorithms. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, NIPS 2012, Red Hook, NY, USA, vol. 2, pp. 2951–2959. Curran Associates Inc. (2012)
Vidnerová, P., Kalina, J.: Bayonet (2022). https://github.com/PetraVidnerova/BayONet
Vidnerova, P., Neruda, R.: Evolving keras architectures for sensor data analysis. In: 2017 Federated Conference on Computer Science and Information Systems (FedCSIS), pp. 109–112, September 2017. https://doi.org/10.15439/2017F241
White, C., Neiswanger, W., Nolen, S., Savani, Y.: A study on encodings for neural architecture search. In: Advances in Neural Information Processing Systems (2020)
White, C., Neiswanger, W., Savani, Y.: BANANAS: Bayesian optimization with neural architectures for neural architecture search. In: AAAI Conference on Artificial Intelligence (AAAI-2021) (2021)
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms (2017)
Xu, J., Zhou, W., Fu, Z., Zhou, H., Li, L.: A survey on green deep learning (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Vidnerová, P., Kalina, J. (2023). Multi-objective Bayesian Optimization for Neural Architecture Search. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2022. Lecture Notes in Computer Science(), vol 13588. Springer, Cham. https://doi.org/10.1007/978-3-031-23492-7_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-23492-7_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-23491-0
Online ISBN: 978-3-031-23492-7
eBook Packages: Computer ScienceComputer Science (R0)