[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Log in

Semi-supervised regression with label-guided adaptive graph optimization

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

For the semi-supervised regression task, both the similarity of paired samples and the limited label information serve as core indicators. Nevertheless, most traditional semi-supervised regression methods cannot make full use of both simultaneously. To alleviate the above deficiency, this paper proposes a novel semi-supervised regression with label-guided adaptive graph optimization (LGAGO-SSR). Basically, LGAGO-SSR involves two phases: graph representation and label-guided adaptive graph construction. The first phase seeks two low-dimensional manifold spaces based on two similarity matrices. The second phase aims at adaptively learning these similarity matrices by integrating the data structure information in both the low-dimensional manifold spaces and the label spaces. Each phase has its optimization problems, and the final solution is obtained by iteratively solving problems in two phases. Additionally, the idea of decomposition optimization in twin support vector regression (TSVR) is used to accelerate the training of our LGAGO-SSR. Regression results on 12 benchmark datasets with different unlabeled rates demonstrate the effectiveness of LGAGO-SSR in semi-supervised regression tasks.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Fig. 1
Algorithm 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability and access

After obtaining a license to use the data, the data can be accessed by visiting the following websites: https://archive.ics.uci.edu/ml/index.php, https://hastie.su.domains/ElemStatLearn/data.html and https://tianchi.aliyun.com/dataset/159885. Users can use the data for study and research purposes, but not for commercial purposes.

Notes

  1. https://tianchi.aliyun.com/dataset/159885

References

  1. Smola AJ, Schölkopf B (2004) A tutorial on support vector regression. Statistics and Computing 14:199–222

    Article  MathSciNet  Google Scholar 

  2. Myles AJ, Feudale RN, Liu Y, Woody NA, Brown SD (2004) An introduction to decision tree modeling. Journal of Chemometrics: J Chemometr Soc 18(6):275–285

    Article  Google Scholar 

  3. Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc Ser B Stat Methodol 58(1):267–288

    Article  MathSciNet  Google Scholar 

  4. Czajkowski M, Jurczuk K, Kretowski M (20230) Steering the interpretability of decision trees using lasso regression - an evolutionary perspective. Inform Sci 638118944

  5. Jain N, Jana PK (2023) LRF: A logically randomized forest algorithm for classification and regression problems. Expert Syst Appl 213.(Part C) 119225

  6. Chen H, Wu L, Chen J, Lu W, Ding J (2022) A comparative study of automated legal text classification using random forests and deep learning. Inf Process Manag 59(2):102798

  7. Zhou Z, Li M (2005) Semi-supervised regression with co-training. In: Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, pp 908–913. Morgan Kaufmann, San Francisco, USA

  8. Zhou Z, Li M (2007) Semisupervised regression with cotraining-style algorithms. IEEE Trans Knowl Data Eng 19(11):1479–1493

    Article  Google Scholar 

  9. Wang X, Ma L, Wang X (2010) Apply semi-supervised support vector regression for remote sensing water quality retrieving. IEEE Int Geosci Remote Sens Symp. IEEE, Piscataway, USA, pp 2757–2760

    Google Scholar 

  10. Emadi M, Tanha J, Shiri ME, Aghdam MH (2021) A selection metric for semi-supervised learning based on neighborhood construction. Inf Process Manag 58(2):102444

    Article  Google Scholar 

  11. Lin K, Pai P, Lu Y, Chang P (2013) Revenue forecasting using a least-squares support vector regression model in a fuzzy environment. Inf Sci 220196–209

  12. Yue Y, Wang G, Hu J, Li Y (2023) An improved label propagation algorithm based on community core node and label importance for community detection in sparse network. Appl Intell 5317935–17951

  13. Hua Z, Yang Y (2022) Robust and sparse label propagation for graph-based semi-supervised classification. Appl Intell 523337–3351

  14. Hua Z, Yang Y, Qiu H (2021) Node influence-based label propagation algorithm for semi-supervised learning. Neural Comput & Applic 332753–2768

  15. Wang B, Tsotsos J (2016) Dynamic label propagation for semi-supervised multi-class multi-label classification. Pattern Recogn 5275–84

  16. Yoo J, Kim HJ (2014) Semisupervised location awareness in wireless sensor networks using Laplacian support vector regression. Int J Distrib Sensor Netw 10265801

  17. Belkin M, Niyogi P, Sindhwani V (2006) Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. J Mach Learn Res 72399–2434

  18. Yu J, Son Y (2021) Weighted co-association rate-based Laplacian regularized label description for semi-supervised regression. Inf Sci 545688–712

  19. Yu Z, Ye F, Yang K, Cao W, Chen CLP, Cheng L, You J, Wong H (2022) Semisupervised classification with novel graph construction for high-dimensional data. IEEE Trans Neural Netw Learn Syst 33(1):75–88

    Article  MathSciNet  Google Scholar 

  20. Zhou B, Liu W, Zhang W, Lu Z, Tan Q (2022) Multi-kernel graph fusion for spectral clustering. Inf Process Manag 59(5):103003

  21. Nie F, Dong X, Li X (2021) Unsupervised and semisupervised projection with graph optimization. IEEE Trans Neural Netw Learn Syst 32(4):1547–1559

  22. Wang S, Chen Y, Yi S, Chao G (2022) Frobenius norm-regularized robust graph learning for multi-view subspace clustering. Appl Intell 52(13):14935–14948

    Article  Google Scholar 

  23. Zhang R, Nie F, Li X (2019) Semisupervised learning with parameter-free similarity of label and side information. IEEE Trans Neural Netw Learn Syst 30(2):405–414

    Article  MathSciNet  Google Scholar 

  24. Zhang L, Liu Z, Pu J, Song B (2020) Adaptive graph regularized nonnegative matrix factorization for data representation. Appl Intell 50438–447

  25. Li D, Madden AD (2019) Cascade embedding model for knowledge graph inference and retrieval. Inf Process Manag 56(6):102093

  26. Chen L, Zhong Z (2022) Adaptive and structured graph learning for semi-supervised clustering. Inf Process Manag 59(4):102949

    Article  Google Scholar 

  27. Liu J, Lin M, Zhao M, Zhan C, Li B, Chui JKT (2023) Person re-identification via semi-supervised adaptive graph embedding. Appl Intell 53(3):2656–2672

    Article  Google Scholar 

  28. Zhang L, Zhou W, Chang P, Liu J, Yan Z, Wang T, Li FZ (2012) Kernel sparse representation-based classifier. IEEE Trans Signal Process 60(4):1684–1695

    Article  MathSciNet  Google Scholar 

  29. Nie F, Wang X, Huang H (2014) Clustering and projected clustering with adaptive neighbors. International Conference on Knowledge Discovery and Data Mining. ACM, New York, USA, pp 977–986

    Google Scholar 

  30. Peng X (2010) TSVR: A n efficient twin support vector machine for regression. Neural Netw 23(3):365–372

    Article  Google Scholar 

  31. Zhuang L, Zhou Z, Gao S, Yin J, Lin Z, Ma Y (2017) Label information guided graph construction for semi-supervised learning. IEEE Trans Image Process 26(9):4182–4192

    Article  MathSciNet  Google Scholar 

  32. Peng X, Chen D, Xu D (2019) Hyperplane-based nonnegative matrix factorization with label information. Inf Sc 4931–19

  33. Liu Z, Wang T, Zhu F, Chen X, Pelusi D, Vasilakos AV (2024) Domain adaptive learning based on equilibrium distribution and dynamic subspace approximation. Expert Syst Appl 249123673

  34. Kiyadeh APH, Zamiri A, Yazdi HS, Ghaemi H (2015) Discernible visualization of high dimensional data using label information. Appl Soft Comput 27474–486

  35. Zhu X, Ghahramani Z (2002)Learning from labeled and unlabeled data with label propagation. Technical Report, Technical Report CMU-CALD-02–107, Carnegie Mellon University

  36. Quinlan JR (1993) Combining instance-based and model-based learning. In: Machine Learning, Proceedings of the Tenth International Conference, University of Massachusetts, pp 236–243. Morgan Kaufmann, San Francisco, USA

  37. Breiman L, Friedman JH (1985) Estimating optimal transformations for multiple regression and correlation. J Am Stat Assoc 80(391):580–598

    Article  MathSciNet  Google Scholar 

  38. Zhou F, Q C, King RD (2014) Predicting the geographical origin of music. 2014 IEEE Int Conf Data Min. IEEE Computer Society, Los Alamitos, USA, pp 1115–1120

  39. Yeh IC, Hsu TK (2018) Building real estate valuation models with comparative approach through case-based reasoning. Appl Soft Comput 65260–271

  40. Yeh IC (1998) Modeling of strength of high-performance concrete using artificial neural networks. Cem Concr Res 28(12):1797–1808

    Article  Google Scholar 

  41. Yeh IC (2007) Modeling slump flow of concrete using second-order regressions and artificial neural networks. Cem Concr Res 29(6):474–480

    Article  Google Scholar 

  42. Grisoni F, Consonni V, Vighi M, Villa S, Todeschini R (2016) Investigating the mechanisms of bioconcentration through QSAR classification trees. Environ Int 88198–205

  43. Akbilgic O, Bozdogan H, Balaban ME (2014) A novel hybrid RBF neural networks model as a forecaster. Stat Comput 24(3):365–375

    Article  MathSciNet  Google Scholar 

  44. Owen AB (1999) Tubular neighbors for regression and classification. Citeseer

  45. Nash W, Sellers T, Talbot S, Cawthorn A, Ford W (1995) Abalone. UCI Machine Learning Repository. https://doi.org/10.24432/C55C7W

  46. Cortez P, Cerdeira A, Almeida F, Matos T, Reis J (2009) Modeling wine preferences by data mining from physicochemical properties. Decision support systems 47(4):547–553

    Article  Google Scholar 

  47. Timilsina M, Figueroa A, d’Aquin M, Yang H (2021) Semi-supervised regression using diffusion on graphs. Appl Soft Comput 104107188

  48. Liu L, Huang P, Yu H, Min F (2023) Safe co-training for semi-supervised regression. Intelligent Data Analysis 27:959–975

    Article  Google Scholar 

  49. Demsar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res Learning Research 7(1):1–30

    MathSciNet  Google Scholar 

  50. Dunn OJ (1961) Multiple comparisons among means. J Am Stat Assoc 56(293):52–64

Download references

Acknowledgements

We would like to thank five anonymous reviewers and Editor for their valuable comments and suggestions, which have significantly improved this paper. This work was supported in part by the Natural Science Foundation of the Jiangsu Higher Education Institutions of China under Grant No. 19KJA550002, by the Six Talent Peak Project of Jiangsu Province of China under Grant No. XYDXX-054, by the Priority Academic Program Development of Jiangsu Higher Education Institutions, and by the Collaborative Innovation Center of Novel Software Technology and Industrialization.

Author information

Authors and Affiliations

Authors

Contributions

Xiaohan Zheng: Methodology, Software, Validation, Writing - original draft, Writing - review & editing. Li Zhang: Conceptualization, Methodology, Software, Writing - review & editing, Validation, Project administration, Funding acquisition. Leilei Yan: Investigation, Software, Visualization. Lei Zhao: Investigation, Software, Visualization.

Corresponding author

Correspondence to Li Zhang.

Ethics declarations

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethical and informed consent for data used

This article does not contain any studies with human participants or animals performed by any of the authors

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendices

1.1 Appendix A: Derivation from (13) to (17)

Let \(v^{low}_{ij}={\Vert \varvec{w}_1^T \varvec{x}_i-\varvec{w}_1^T \varvec{x}_j\Vert }_2^2+\Vert y^{low}_i-y^{low}_j\Vert _2^2\), then the objective function in (13) can be rewritten as

$$\begin{aligned}&\sum _{i=1}^{n}\sum _{j=1}^{n}\left( v^{low}_{ij}s^{low}_{ij}+\lambda ^{low}_i \left( {s^{low}_{ij}}\right) ^2\right) \nonumber \\ =&\sum _{i=1}^{n}\sum _{j=1}^{n}\lambda ^{low}_i\left( \left( \frac{v^{low}_{ij}s^{low}_{ij}}{\lambda ^{low}_i}+\left( {s^{low}_{ij}}\right) ^2+\frac{({v^{low}_{ij}})^2}{4({\lambda _i^{low}})^2}\right) -\frac{({v^{low}_{ij}})^2}{4({\lambda _i^{low}})^2}\right) \nonumber \\ =&\sum _{i=1}^{n}\lambda ^{low}_i\sum _{j=1}^{n}\left( \left( \frac{v^{low}_{ij}s^{low}_{ij}}{\lambda ^{low}_i}+\left( {s^{low}_{ij}}\right) ^2+\frac{({v^{low}_{ij}})^2}{4({\lambda _i^{low}})^2}\right) -\frac{({v^{low}_{ij}})^2}{4({\lambda _i^{low}})^2}\right) \nonumber \\ =&\sum _{i=1}^{n}\lambda ^{low}_i\!\left( \left( \frac{(\varvec{v}^{low}_{i})^T{\varvec{s}^{low}_{i}}}{\lambda ^{low}_i}\!+\!\left( \varvec{s}^{low}_{i}\right) ^T\varvec{s}^{low}_{i}\!+\!\frac{({\varvec{v}^{low}_{i}})^T{\varvec{v}^{low}_{i}}}{4{(\lambda ^{low}_i)}^2}\right) \!-\frac{({\varvec{v}^{low}_{i}})^T{\varvec{v}^{low}_{i}}}{4({\lambda ^{low}_i})^2}\right) \nonumber \\ =&\sum _{i=1}^{n}\lambda ^{low}_i\left\| \varvec{s}^{low}_{i}+\frac{\varvec{v}^{low}_{i}}{2\lambda ^{low}_i}\right\| ^2_2-\frac{(\varvec{v}^{low}_{i})^T{\varvec{v}^{low}_{i}}}{4\lambda ^{low}_i}, \end{aligned}$$
(57)

where \(\varvec{v}^{low}_i=[v^{low}_{i1},v^{low}_{i2},\cdots ,v^{low}_{in}]^T\).

Because \(\lambda ^{low}_i\) and \(\varvec{v}^{low}_{i}\) are unrelated to \(\varvec{s}_i^{low}\), the last term in (57) is a constant. Therefore, the objective function in (13) is equivalent to

$$\begin{aligned} \sum _{i=1}^{n}\left\| \varvec{s}^{low}_{i}+\frac{\varvec{v}^{low}_{i}}{2\lambda ^{low}_i}\right\| ^2_2. \end{aligned}$$

In a summary, we have completed the derivation from (13) to (17).

$$\begin{aligned} \min _{\varvec{w}_1,b_1,\varvec{\xi }_1,\varvec{S}^{low},\varvec{y}^{low}}~~~~&G_1\left( \varvec{w}_1,b_1,\varvec{\xi }_1,\varvec{S}^{low},\varvec{y}^{low}\right) \nonumber \\ =&\frac{1}{2}\Vert \varvec{w}_1\Vert ^2_2+\sum _{i=1}^{n}\Big ({C_1}\Vert y^{low}_i-\epsilon _1-f_1(\varvec{x}_i)\Vert _2^2\nonumber \\&+{C_2}{\xi _1}_i\Big )\frac{1}{2}\sum _{i=1}^{n}\sum _{j=1}^{n}\Big (\Vert f_1(\varvec{x}_i)-f_1(\varvec{x}_j)\Vert _2^2s^{low}_{ij}\nonumber \\&+\Vert y^{low}_i-y^{low}_j\Vert ^2_2s^{low}_{ij}+\lambda ^{low}_i \left( {s^{low}_{ij}}\Big )^2\right) \nonumber \\ s.t.~~~~~~~~~~~&y^{low}_i-\epsilon _1-f_1(\varvec{x}_i)+{\xi _1}_i\ge 0,\nonumber \\&{\varvec{y}}^{low}_u ={\left( \varvec{I}- {\varvec{S}}^{low}_{uu}\right) }^{-1} {\varvec{S}}^{low}_{ul}\varvec{y}_l,\nonumber \\&{\xi _1}_i\ge 0,~s^{low}_{ij}\ge 0,~ \varvec{1}^T{{\varvec{s}}^{low}_{i}} = 1,\nonumber \\&i,j=1,\cdots ,n, \end{aligned}$$
(58)

1.2 Appendix B: Proof of Theorem 1

The sequences \(\left\{ \left( \varvec{w}_1^{(p)},b_1^{(p)},\varvec{\xi }_1^{(p)},{\varvec{S}^{low}}^{(p)},{\varvec{y}^{low}}^{(p)}\right) \right\} \) and \(\left\{ \!(\varvec{w}_2^{(p)},b_2^{(p)},{\varvec{\xi }_2}^{(p)},{\varvec{S}^{up}}^{(p)},{\varvec{y}^{up}}^{(p)})\!\right\} \) generated by Algorithm 1 can guarantee that \(\left\{ G_1(\varvec{w}_1^{(p)},b_1^{(p)},\varvec{\xi }_1^{(p)},{\varvec{S}^{low}}^{(p)},{\varvec{y}^{low}}^{(p)})\right\} \) and \(\left\{ G_2(\varvec{w}_2^{(p)},b_2^{(p)},{\varvec{\xi }_2}^{(p)},{\varvec{S}^{up}}^{(p)},{\varvec{y}^{up}}^{(p)})\right\} \) are monotonic decreasing and bounded, respectively, where p refers to the current number of iterations.

Assume that p is the current iteration. First, we prove that the sequence \(\left\{ G_1\left( \varvec{w}_1^{(p)},b_1^{(p)},\varvec{\xi }_1^{(p)},{\varvec{S}^{low}}^{(p)},{\varvec{y}^{low}}^{(p)}\right) \right\} \) decreases monotonically. Given \(\left( \varvec{w}_1^{(p)},b_1^{(p)},\varvec{\xi }_1^{(p)}\right) \), we optimize (13) to obtain \(\left( {\varvec{S}^{low}}^{(p+1)},{\varvec{y}^{low}}^{(p+1)}\right) \), thus we have

$$\begin{aligned}&G_1\left( \varvec{w}_1^{(p)},b_1^{(p)},\varvec{\xi }_1^{(p)},{\varvec{S}^{low}}^{(p)},{\varvec{y}^{low}}^{(p)}\right) \ge G_1\Big (\varvec{w}_1^{(p)},b_1^{(p)},\nonumber \\&\quad \varvec{\xi }_1^{(p)},{\varvec{S}^{low}}^{(p+1)},{\varvec{y}^{low}}^{(p+1)}\Big ). \end{aligned}$$
(59)

Given \(\left( {\varvec{S}^{low}}^{(p+1)},{\varvec{y}^{low}}^{(p+1)}\right) \), we optimize (9) to obtain \(\left( \varvec{w}_1^{(p+1)},b_1^{(p+1)},\varvec{\xi }_1^{(p+1)}\right) \), thus we have

$$\begin{aligned}&G_1\left( \varvec{w}_1^{(p)},b_1^{(p)},\varvec{\xi }_1^{(p)},{\varvec{S}^{low}}^{(p+1)},{\varvec{y}^{low}}^{(p+1)}\right) \ge G_1\Big (\varvec{w}_1^{(p+1)},\nonumber \\&\quad b_1^{(p+1)},\varvec{\xi }_1^{(p+1)},{\varvec{S}^{low}}^{(p+1)},{\varvec{y}^{low}}^{(p+1)}\Big ). \end{aligned}$$
(60)

By combining (59) and (60), we have

$$\begin{aligned}&G_1\left( \varvec{w}_1^{(p)},b_1^{(p)},\varvec{\xi }_1^{(p)},{\varvec{S}^{low}}^{(p)},{\varvec{y}^{low}}^{(p)}\right) \ge G_1\Big (\varvec{w}_1^{(p+1)},\nonumber \\&\quad b_1^{(p+1)},\varvec{\xi }_1^{(p+1)},{\varvec{S}^{low}}^{(p+1)},{\varvec{y}^{low}}^{(p+1)}\Big ), \end{aligned}$$
(61)

which indicates that the sequence \(\Big \{G_1\Big (\varvec{w}_1^{(p)},b_1^{(p)},\varvec{\xi }_1^{(p)},\) \({\varvec{S}^{low}}^{(p)},{\varvec{y}^{low}}^{(p)}\Big )\Big \}\) for \(p=1, 2, \cdots \) is monotonic decreasing. In the same way, we can prove that the sequence \(\left\{ G_2\left( \varvec{w}_2^{(p)},b_2^{(p)},{\varvec{\xi }_2}^{(p)},{\varvec{S}^{up}}^{(p)},{\varvec{y}^{up}}^{(p)}\right) \right\} \) is monotonic dec-reasing when \(p=1,2,\cdots \).

Next, we prove that both the sequences \(\Big \{G_1\Big (\varvec{w}_1^{(p)},b_1^{(p)},\) \(\varvec{\xi }_1^{(p)},{\varvec{S}^{low}}^{(p)},{\varvec{y}^{low}}^{(p)}\Big )\Big \}\) and \(\Big \{G_2\Big (\varvec{w}_2^{(p)},b_2^{(p)},{\varvec{\xi }_2}^{(p)},{\varvec{S}^{up}}^{(p)},\) \({\varvec{y}^{up}}^{(p)}\Big )\Big \}\) have the infimum. In (15) and (16), we know that \(s^{low}_{ij}\), \(s^{up}_{ij}\), \(\lambda ^{low}_i\), \(\lambda ^{up}_i\), and \(C_i~(i=1,2,3,4)\) are all great than zero. Thus, it is easy to infer that

$$\begin{aligned} G_1\left( \varvec{w}_1^{(p)},b_1^{(p)},\varvec{\xi }_1^{(p)},{\varvec{S}^{low}}^{(p)},{\varvec{y}^{low}}^{(p)}\right) \ge 0, \end{aligned}$$
(62)

and

$$\begin{aligned} G_2\left( \varvec{w}_2^{(p)},b_2^{(p)},{\varvec{\xi }_2}^{(p)},{\varvec{S}^{up}}^{(p)},{\varvec{y}^{up}}^{(p)}\right) \ge 0. \end{aligned}$$
(63)

In the other word, \(\left\{ G_1\!\left( \varvec{w}_1^{(p)},b_1^{(p)},\varvec{\xi }_1^{(p)},\!{\varvec{S}^{low}}^{(p)},{\varvec{y}^{low}}^{(p)}\right) \!\right\} \) and \(\left\{ G_2\left( \varvec{w}_2^{(p)},b_2^{(p)},{\varvec{\xi }_2}^{(p)},{\varvec{S}^{up}}^{(p)},{\varvec{y}^{up}}^{(p)}\right) \right\} \) have the infimum that equals 0.

This completes the proof of Theorem 1.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zheng, X., Zhang, L., Yan, L. et al. Semi-supervised regression with label-guided adaptive graph optimization. Appl Intell 54, 10671–10694 (2024). https://doi.org/10.1007/s10489-024-05766-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05766-7

Keywords

Navigation