Deep functional factor models: forecasting high-dimensional functional time series via bayesian nonparametric factorization
Article No.: 1282, Pages 31709 - 31727
Abstract
This paper introduces the Deep Functional Factor Model (DF2M), a Bayesian nonparametric model designed for analysis of high-dimensional functional time series. DF2M is built upon the Indian Buffet Process and the multi-task Gaussian Process, incorporating a deep kernel function that captures non-Markovian and nonlinear temporal dynamics. Unlike many black-box deep learning models, DF2M offers an explainable approach to utilizing neural networks by constructing a factor model and integrating deep neural networks within the kernel function. Additionally, we develop a computationally efficient variational inference algorithm to infer DF2M. Empirical results from four real-world datasets demonstrate that DF2M provides better explainability and superior predictive accuracy compared to conventional deep learning models for high-dimensional functional time series.
References
[1]
Al-Shedivat, M., Wilson, A. G., Saatchi, Y., Hu, Z., and Xing, E. P. Learning scalable deep kernels with recurrent structure. The Journal of Machine Learning Research, 18 (1):2850-2886, 2017.
[2]
Bathia, N., Yao, Q., and Ziegelmann, F. Identifying the finite dimensionality of curve time series. The Annals of Statistics, 38:3352-3386, 2010.
[3]
Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. Variational inference: a review for statisticians. Journal of the American Statistical Association, 112(518):859-877, 2017.
[4]
Bonilla, E. V., Chai, K., and Williams, C. Multi-task gaussian process prediction. In Advances in Neural Information Processing Systems, volume 20, 2007.
[5]
Chang, J., Chen, C., Qiao, X., and Yao, Q. An autocovariance-based learning framework for high-dimensional functional time series. Journal of Econometrics, 2023a.
[6]
Chang, J., Fang, Q., Qiao, X., and Yao, Q. On the modelling and prediction of high-dimensional functional time series. Working Paper, 2023b.
[7]
Chen, C., Guo, S., and Qiao, X. Functional linear regression: dependence and error contamination. Journal of Business and Economic Statistics, 40:444-457, 2022.
[8]
Cho, K., Van Merrienboer, B., Bahdanau, D., and Bengio, Y. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.
[9]
Damianou, A. and Lawrence, N. D. Deep Gaussian processes. In Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, pp. 207-215, 2013.
[10]
Dawid, A. P. Some matrix-variate distribution theory: notational considerations and a Bayesian application. Biometrika, 68(1):265-274, 1981.
[11]
Fang, Q., Guo, S., and Qiao, X. Finite sample theory for high-dimensional functional/scalar time series with applications. Electronic Journal of Statistics, 16:527-591, 2022.
[12]
Fortuin, V. Priors in Bayesian deep learning: a review. International Statistical Review, pp. 12502, 2022.
[13]
Gao, Y., Shang, H. L., and Yang, Y. High-dimensional functional time series forecasting: An application to age-specific mortality rates. Journal of Multivariate Analysis, 170:232-243, 2019.
[14]
Griffiths, T. L. and Ghahramani, Z. The Indian buffet process: an introduction and review. Journal of Machine Learning Research, 12(32):1185-1224, 2011.
[15]
Guo, S. and Qiao, X. On consistency and sparsity for high-dimensional functional time series with application to autoregressions. Bernoulli, 29(1):451-472, 2023.
[16]
Guo, S., Qiao, X., and Wang, Q. Factor modelling for high-dimensional functional time series. arXiv:2112.13651, 2021.
[17]
Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., and Lew, M. S. Deep learning for visual understanding: A review. Neurocomputing, 187:27-48, 2016.
[18]
Hamelijnck, O., Wilkinson, W., Loppi, N., Solin, A., and Damoulas, T. Spatio-temporal variational Gaussian processes. In Advances in Neural Information Processing Systems, volume 34, pp. 23621-23633, 2021.
[19]
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, June 2016.
[20]
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(8):1735-1780, 1997. Publisher: MIT press.
[21]
Hofmann, T., Scholkopf, B., and Smola, A. J. Kernel methods in machine learning. The annals of statistics, 36(3): 1171-1220, 2008.
[22]
Hormann, S., Kidzinski, L., and Hallin, M. Dynamic functional principal components. Journal of the Royal Statistical Society: Series B, 77:319-348, 2015.
[23]
Horvath, L., Kokoszka, P., and Rice, G. Testing stationarity of functional time series. Journal of Econometrics, 179 (1):66-82, 2014.
[24]
Hughes, M. C. and Sudderth, E. Memoized online variational inference for Dirichlet process mixture models. In Advances in Neural Information Processing Systems 26, pp. 1133-1141, 2013.
[25]
Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., and Blei, D. M. Automatic differentiation variational inference. Journal of machine learning research, 2017.
[26]
Lawrence, N. Gaussian process latent variable models for visualisation of high dimensional data. Advances in Neural Information Processing Systems, 16, 2003.
[27]
Li, W., Sutherland, D. J., Strathmann, H., and Gretton, A. Learning deep kernels for exponential family densities. In International Conference on Machine Learning, pp. 6737-6746, 2019.
[28]
Lim, B. and Zohren, S. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A, 379(2194):20200209, 2021. Publisher: The Royal Society Publishing.
[29]
Liu, Y., Qiao, X., and Lam, J. CATVI: Conditional and adaptively truncated variational inference for hierarchical bayesian nonparametric models. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics, pp. 3647-3662. PMLR, 2022.
[30]
Liu, Y., Qiao, X., Wang, L., and Lam, J. EEGNN: Edge enhanced graph neural network with a Bayesian nonparametric graph model. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, pp. 2132-2146. PMLR, 2023.
[31]
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
[32]
Moreno-Munoz, P., Artes, A., and Alvarez, M. Heterogeneous multi-output gaussian process prediction. In Advances in Neural Information Processing Systems, volume 31, 2018.
[33]
Ramsay, J. O. and Silverman, B. W. Functional data analysis. Springer, New York, 2005.
[34]
Ranganath, R., Gerrish, S., and Blei, D. Black box variational inference. In Artificial Intelligence and Statistics, pp. 814-822, 2014.
[35]
Tang, C., Shang, H. L., and Yang, Y. Clustering and forecasting multiple functional time series. The Annals of Applied Statistics, 16(4):2523-2553, December 2022.
[36]
Teh, Y. W., Jordan, M. I., Beal, M. J., and Blei, D. M. Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101(476):1566-1581, 2006.
[37]
Titsias, M. Variational learning of inducing variables in sparse Gaussian processes. In Proceedings of the 12th International Conference on Artificial Intelligence and Statistics, pp. 567-574, April 2009. ISSN: 1938-7228.
[38]
Titsias, M. and Lawrence, N. D. Bayesian Gaussian process latent variable model. In Proceedings of the 13th international conference on artificial intelligence and statistics, pp. 844-851, 2010.
[39]
Torfi, A., Shirvani, R. A., Keneshloo, Y., Tavaf, N., and Fox, E. A. Natural language processing advancements by deep learning: A survey. arXiv preprint arXiv:2003.01200, 2020.
[40]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention Is All You Need. In Advances in Neural Information Processing Systems, volume 30, 2017.
[41]
Vu, V. Q. and Lei, J. Minimax sparse principal subspace estimation in high dimensions. The Annals of Statistics, 41(6):2905-2947, 2013.
[42]
Wang, J., Hertzmann, A., and Fleet, D. J. Gaussian process dynamical models. In Advances in Neural Information Processing Systems, volume 18, 2005.
[43]
Watson, J., Lin, J. A., Klink, P., Pajarinen, J., and Peters, J. Latent derivative Bayesian last layer networks. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pp. 1198-1206, 2021.
[44]
Williams, C. K. and Rasmussen, C. E. Gaussian Processes for Machine Learning. MIT Press Cambridge, 2006.
[45]
Wilson, A. G., Hu, Z., Salakhutdinov, R., and Xing, E. P. Deep kernel learning. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, pp. 370-378, 2016.
[46]
Xue, H., Wu, Z.-F., and Sun, W.-X. Deep spectral kernel learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 4019-4025, 2019.
[47]
Yao, J., Mueller, J., and Wang, J.-L. Deep learning for functional data analysis with adaptive basis layers. In International Conference on Machine Learning, pp. 11898-11908. PMLR, 2021.
[48]
Zhou, Z. and Dette, H. Statistical inference for highdimensional panel functional time series. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(2):523-549, 2023.
Index Terms
- Deep functional factor models: forecasting high-dimensional functional time series via bayesian nonparametric factorization
Index terms have been assigned to the content through auto-classification.
Recommendations
Deep Unfolding for Topic Models
Deep unfolding provides an approach to integrate the probabilistic generative models and the deterministic neural networks. Such an approach is benefited by deep representation, easy interpretation, flexible learning and stochastic modeling. This study ...
Deep Learning with Hierarchical Convolutional Factor Analysis
Unsupervised multilayered (“deep”) models are considered for imagery. The model is represented using a hierarchical convolutional factor-analysis construction, with sparse factor loadings and scores. The computation of layer-dependent model parameters ...
Auxiliary deep generative models
ICML'16: Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48Deep generative models parameterized by neural networks have recently achieved state-of-the-art performance in unsupervised and semi-supervised learning. We extend deep generative models with auxiliary variables which improves the variational ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
July 2024
63010 pages
Copyright © 2024.
Publisher
JMLR.org
Publication History
Published: 21 July 2024
Qualifiers
- Research-article
- Research
- Refereed limited
Acceptance Rates
Overall Acceptance Rate 140 of 548 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 05 Mar 2025