Abstract
Neural architecture search (NAS) is an automated machine learning method that optimizes neural network architectures depending on the dataset or its purpose. With the advances in NAS, high-accuracy neural network architectures can be built for a specific dataset without any expert skills. However, NAS is an expensive, time-consuming, and resource-intensive technique. Therefore, searching for the optimal architecture from scratch for each new dataset is inefficient. To accommodate the expected future increase in datasets, a technique is required that directly predicts the optimized architecture for unknown datasets. Therefore, we propose a framework that generates architectures for unknown datasets by mapping adequate architectures for existing datasets into the latent feature space. A variational graph autoencoder (VGAE) is utilized for latent feature mapping. Our experimental results indicate that the architecture generated by the proposed method from the information of previously obtained high-accuracy architectures performs effectively for new datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Agiollo, A., Omicini, A.: Gnn2gnn: graph neural networks to generate neural networks. In: Uncertainty in Artificial Intelligence, pp. 32–42. PMLR (2022)
Alvarez-Melis, D., Fusi, N.: Geometric dataset distances via optimal transport. In: Advances in Neural Information Processing Systems, vol. 33, pp. 21428–21439. Curran Associates, Inc. (2020)
Chatzianastasis, M., et al.: Graph-based neural architecture search with operation embeddings. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 393–402 (2021). https://doi.org/10.1109/ICCVW54120.2021.00048
Chen, Y., et al.: Contrastive neural architecture search with neural architecture comparators. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9502–9511 (2021). https://doi.org/10.1109/CVPR46437.2021.00938
Dudziak, L., et al.: Brp-nas: prediction-based nas using gcns. Adv. Neural. Inf. Process. Syst. 33, 10480–10490 (2020)
Friede, D., et al.: A variational-sequential graph autoencoder for neural architecture performance prediction. arXiv preprint (2019). https://doi.org/10.48550/arXiv.1912.05317
Hull, J.J.: A database for handwritten text recognition research. IEEE Trans. Pattern Anal. Mach. Intell. 16(5), 550–554 (1994). https://doi.org/10.1109/34.291440
Jiang, D., et al.: Could graph neural networks learn better molecular representation for drug discovery? a comparison study of descriptor-based and graph-based models. J. Cheminf. 13(1), 1–23 (2021). https://doi.org/10.1186/s13321-020-00479-8
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint (2013). https://doi.org/10.48550/arXiv.1312.6114
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint (2016). https://doi.org/10.48550/arXiv.1609.02907
Kipf, T.N., Welling, M.: Variational graph auto-encoders. arXiv preprint (2016). https://doi.org/10.48550/arXiv.1611.07308
Krishnakumar, A., et al.: Nas-bench-suite-zero: accelerating research on zero cost proxies. Adv. Neural. Inf. Process. Syst. 35, 28037–28051 (2022)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of Toronto (2009)
LeCun, Y., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998). https://doi.org/10.1109/5.726791
Lee, H., Hyung, E., Hwang, S.J.: Rapid neural architecture search by learning to generate graphs from datasets. arXiv preprint (2021). https://doi.org/10.48550/arXiv.2107.00860
Li, J., et al.: Neural architecture optimization with graph vae. arXiv preprint (2020). 10.48550/arXiv.2006.10310
Li, Y., Peng, X.: Network architecture search for domain adaptation. arXiv preprint (2020). https://doi.org/10.48550/arXiv.2008.05706
Liu, H., et al.: Darts: differentiable architecture search. arXiv preprint (2018). https://doi.org/10.48550/arXiv.1806.09055
Lu, Z., et al.: Neural architecture transfer. arXiv preprint (2020). https://doi.org/10.48550/arXiv.2005.05859
Lukasik, J., et al.: Smooth variational graph embeddings for efficient neural architecture search. In: International Joint Conference on Neural Networks, pp. 1–8 (2020)
Lukasik, J., et al.: Learning where to look - generative NAS is surprisingly efficient. CoRR arxiv:2203.08734 (2022)
Luo, R., et al.: Neural architecture optimization. Adv. Neural Inf. Process. Syst. 31 (2018)
Mathisen, B.M., et al.: Learning similarity measures from data. Prog. Artif. Intelli. 9 (2019). https://doi.org/10.1007/s13748-019-00201-2
Mellor, J., et al.: Neural architecture search without training. In: International Conference on Machine Learning, vol. 139, pp. 7588–7598. PMLR (2021). https://doi.org/10.1109/ACCESS.2021.3052996
Muravev, A., et al.: Neural architecture search by estimation of network structure distributions. IEEE Access 9 (2021). https://doi.org/10.1109/ACCESS.2021.3052996
Ning, X., Zheng, Y., Zhao, T., Wang, Yu., Yang, H.: A generic graph-based neural architecture encoding scheme for predictor-based NAS. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12358, pp. 189–204. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58601-0_12
Pearson, K.: Liii. on lines and planes of closest fit to systems of points in space. Lond. Edinburgh Dublin Phil. Maga. J. Sci. 2(11) (1901). https://doi.org/10.1080/14786440109462720
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vision 115, 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
Sanchez-Gonzalez, A., et al.: Learning to simulate complex physics with graph networks. In: Proceedings of the International Conference on Machine Learning, pp. 8459–8468 (2020)
Scarselli, F., et al.: The graph neural network model. IEEE Trans. Neural Netw. 20(1), 61–80 (2008). https://doi.org/10.1109/TNN.2008.2005605
Schrod, S., et al.: FACT: federated adversarial cross training. arXiv preprint (2023). https://doi.org/10.48550/arXiv.2306.00607
Shi, H., et al.: Bridging the gap between sample-based and one-shot neural architecture search with bonas. Adv. Neural. Inf. Process. Syst. 33, 1808–1819 (2020)
Singamsetti, M., et al.: Conceptual expansion neural architecture search (cenas). arXiv preprint (2021). DOI: https://doi.org/10.48550/arXiv.2110.03144
Suchopárová, G., Neruda, R.: Graph embedding for neural architecture search with input-output information. In: AutoML Conference Workshop Track (2022)
Velickovic, P., et al.: Graph attention networks. In: International Conference on Learning Representations (2018). https://doi.org/10.17863/CAM.48429
Wen, W., et al.: Neural predictor for neural architecture search. In: Proceedings of European Conference on Computer Vision (2019)
White, C., et al.: Bananas: bayesian optimization with neural architectures for neural architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence (2021)
Wistuba, M.: XferNAS: transfer neural architecture search. In: Hutter, F., Kersting, K., Lijffijt, J., Valera, I. (eds.) ECML PKDD 2020. LNCS (LNAI), vol. 12459, pp. 247–262. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67664-3_15
Wong, C., et al.: Transfer learning with neural automl. Adv. Neural Inf. Process. Syst. (2018)
Xu, Y., et al.: PC-DARTS: partial channel connections for memory-efficient architecture search. In: International Conference on Learning Representations (2020)
Yan, S., et al.: Does unsupervised architecture representation learning help neural architecture search? Adv. Neural Inf. Process. Syst. (2020)
Zoph, B., et al.: Learning transferable architectures for scalable image recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8697–8710 (2018). DOI: https://doi.org/10.1109/CVPR.2018.00907
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hemmi, K., Tanigaki, Y., Onishi, M. (2024). NAVIGATOR-D3: Neural Architecture Search Using VarIational Graph Auto-encoder Toward Optimal aRchitecture Design for Diverse Datasets. In: Wand, M., Malinovská, K., Schmidhuber, J., Tetko, I.V. (eds) Artificial Neural Networks and Machine Learning – ICANN 2024. ICANN 2024. Lecture Notes in Computer Science, vol 15016. Springer, Cham. https://doi.org/10.1007/978-3-031-72332-2_20
Download citation
DOI: https://doi.org/10.1007/978-3-031-72332-2_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72331-5
Online ISBN: 978-3-031-72332-2
eBook Packages: Computer ScienceComputer Science (R0)