Evidence Transfer: Learning Improved Representations According to External Heterogeneous Task Outcomes
Authors: 
Athanasios Davvetas, Iraklis A. Klampanos, Spiros Skiadopoulos, Vangelis KarkaletsisAuthors Info & Claims
Article No.: 85, Pages 1 - 22
Abstract
Unsupervised representation learning tends to produce generic and reusable latent representations. However, these representations can often miss high-level features or semantic information, since they only observe the implicit properties of the dataset. On the other hand, supervised learning frameworks learn task-oriented latent representations that may not generalise in other tasks or domains. In this article, we introduce evidence transfer, a deep learning method that incorporates the outcomes of external tasks in the unsupervised learning process of an autoencoder. External task outcomes also referred to as categorical evidence, are represented by categorical variables, and are either directly or indirectly related to the primary dataset—in the most straightforward case they are the outcome of another task on the same dataset. Evidence transfer allows the manipulation of generic latent representations in order to include domain or task-specific knowledge that will aid their effectiveness in downstream tasks. Evidence transfer is robust against evidence of low quality and effective when introduced with related, corresponding, or meaningful evidence.
References
[1]
Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. 2013. Generalized denoising auto-encoders as generative models. In Proceedings of the Advances in Neural Information Processing Systems, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 899–907.
[2]
G. Bhatt, P. Jha, and B. Raman. 2017. Common representation learning using step-based correlation multi-modal CNN. In Proceedings of the 2017 4th IAPR Asian Conference on Pattern Recognition. IEEE, 864–869.
[3]
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. 2018. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, the 30th innovative Applications of Artificial Intelligence, and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, Louisiana,Sheila A. McIlraith and Kilian Q. Weinberger (Eds.). AAAI Press, 2095–2102.
[4]
Marc-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. 2018. Multiple instance learning: A survey of problem characteristics and applications. Pattern Recognition 77 (2018), 329–353.
[5]
Shiyu Chang, Wei Han, Jiliang Tang, Guo-Jun Qi, Charu C. Aggarwal, and Thomas S. Huang. 2015. Heterogeneous network embedding via deep architectures. In Proceedings of the 21stACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 119–128.
[6]
Jin Chen, Xinxiao Wu, Lixin Duan, and Lin Chen. 2021. Sequential instance refinement for cross-domain object detection in images. IEEE Transactions on Image Processing 30 (2021), 3970–3984.
[7]
Athanasios Davvetas and Iraklis A. Klampanos. 2020. Unsupervised severe weather detection via joint representation learning over textual and weather data. In Proceedings of the CEUR Workshop. 83–87. Retrieved 11 January, 2022 from http://ceur-ws.org/Vol-2844/ainst7.pdf.
[8]
A. Davvetas, I. A. Klampanos, and V. Karkaletsis. 2019. Evidence transfer for improving clustering tasks using external categorical evidence. In Proceedings of the 2019 International Joint Conference on Neural Networks. 1–8.
[9]
Athanasios Davvetas, Iraklis A. Klampanos, Spiros Skiadopoulos, and Vangelis Karkaletsis. 2019. The effect of evidence transfer on latent feature relevance for clustering. Informatics 6, 2 (2019), 17.
[10]
J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 248–255.
[11]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.
[12]
C. Doersch, A. Gupta, and A. A. Efros. 2015. Unsupervised visual representation learning by context prediction. In Proceedings of the 2015 IEEE International Conference on Computer Vision. 1422–1430.
[13]
J. A. Figueroa and A. R. Rivera. 2017. Learning to cluster with auxiliary tasks: A semi-supervised approach. In Proceedings of the 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images. IEEE, 141–148.
[14]
Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference on International Conference on Machine Learning. JMLR.org, 1180–1189.
[15]
Zan Gao, Leming Guo, Weili Guan, An-An Liu, Tongwei Ren, and Shengyong Chen. 2021. A pairwise attentive adversarial spatiotemporal network for cross-domain few-shot action recognition-R2. IEEE Transactions on Image Processing 30 (2021), 767–782.
[16]
Zan Gao, Leming Guo, Tongwei Ren, An-An Liu, Zhi-Yong Cheng, and Shengyong Chen. 2020. Pairwise two-stream ConvNets for cross-domain action recognition with small data. IEEE Transactions on Neural Networks and Learning Systems (2020), 1–15.
[17]
Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2014. An empirical investigation of catastrophic forgeting in gradientbased neural networks. In Proceedings of International Conference on Learning Representations.
[18]
Jonathan Gordon and José Miguel Hernández-Lobato. 2020. Combining deep generative and discriminative models for Bayesian semi-supervised learning. Pattern Recognition 100 (2020), 107156.
[19]
Xifeng Guo, Xinwang Liu, En Zhu, and Jianping Yin. 2017. Deep clustering with convolutional autoencoders. In Proceedings of the Neural Information Processing, Derong Liu, Shengli Xie, Yuanqing Li, Dongbin Zhao, and El-Sayed M. El-Alfy (Eds.), Springer International Publishing, Cham, 373–382.
[20]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778.
[21]
C. Huang, C. C. Loy, and X. Tang. 2016. Unsupervised learning of discriminative attributes and visual representations. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. 5175–5184.
[22]
M. H. Jafari, H. Girgis, A. H. Abdi, Z. Liao, M. Pesteie, R. Rohling, K. Gin, T. Tsang, and P. Abolmaesumi. 2019. Semi-supervised learning for cardiac left ventricle segmentation using conditional deep generative models as prior. In Proceedings of the 2019 IEEE 16th International Symposium on Biomedical Imaging. 649–652.
[23]
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. 2014. Semi-supervised learning with deep generative models. In Proceedings of the International Conference on Neural Information Processing Systems. MIT Press, Cambridge, MA, 3581–3589.
[24]
Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. Technical Report.
[25]
G. Larsson, M. Maire, and G. Shakhnarovich. 2017. Colorization as a proxy task for visual understanding. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. 840–849.
[26]
Julia A. Lasserre, Christopher M. Bishop, and Thomas P. Minka. 2006. Principled hybrids of generative and discriminative models. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, Washington, DC, 87–94.
[27]
David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. 2004. RCV1: A new benchmark collection for text categorization research. Journal of Machine Learning Research 5 (2004), 361–397.
[28]
Linchao Li, Bowen Du, Yonggang Wang, Lingqiao Qin, and Huachun Tan. 2020. Estimation of missing values in heterogeneous traffic data: Application of multimodal deep learning model. Knowledge-Based Systems 194 (2020), 105592.
[29]
Wenyuan Li, Zichen Wang, Yuguang Yue, Jiayun Li, William Speier, Mingyuan Zhou, and Corey Arnold. 2020. Semi-supervised learning using adversarial training with good and bad samples. Machine Vision and Applications 31, 6 (2020), 49.
[30]
Yuanpeng Li, Liang Zhao, Joel Hestness, Ka Yee Lun, Kenneth Church, and Mohamed Elhoseiny. 2020. Transferability of Compositionality. (2020). Retrieved from https://openreview.net/forum?id=GHCu1utcBvX.
[31]
Zhizhong Li and Derek Hoiem. 2018. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 12 (2018), 2935–2947.
[32]
Hongfu Liu and Yun Fu. 2018. Consensus guided multi-view clustering. ACM Transactions on Knowledge Discovery from Data 12, 4, Article 42 (2018), 21 pages.
[33]
Jixiong Liu, Weike Pan, and Zhong Ming. 2020. CoFiGAN: Collaborative filtering by generative and discriminative training for one-class recommendation. Knowledge-Based Systems 191 (2020), 105255.
[34]
Q. Liu, Q. Dou, L. Yu, and P. A. Heng. 2020. MS-Net: Multi-site network for improving prostate segmentation with heterogeneous MRI data. IEEE Transactions on Medical Imaging 39, 9 (2020), 2713–2724.
[35]
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. 2019. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the International Conference on Machine Learning.
[36]
Arjun Magotra and Juntae Kim. 2020. Improvement of heterogeneous transfer learning efficiency by using hebbian learning principle. Applied Sciences 10, 16 (2020), 5631.
[37]
Tong Meng, Xuyang Jing, Zheng Yan, and Witold Pedrycz. 2020. A survey on machine learning for data fusion. Information Fusion 57 (2020), 115–129.
[38]
Gabriel Michau and Olga Fink. 2021. Unsupervised transfer learning for anomaly detection: Application to complementary operating condition transfer. Knowledge-Based Systems 216 (2021), 106816.
[39]
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of the Workshop Poster in The International Conference on Learning Representations (2013).
[40]
Seungwhan Moon and Jaime G. Carbonell. 2017. Completely heterogeneous transfer learning with attention-what and what not to transfer. In Proceedings of the Joint Conference on Artificial Intelligence. 1–2.
[41]
Mehdi Noroozi and Paolo Favaro. 2016. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proceedings of the Computer Vision – ECCV 2016, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.), Springer International Publishing, Cham, 69–84.
[42]
Augustus Odena. 2016. Semi-Supervised Learning with Generative Adversarial Networks. (2016). arXiv:1606.01583. Retrieved from https://arxiv.org/abs/1606.01583.
[43]
S. J. Pan and Q. Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering 22, 10 (2010), 1345–1359.
[44]
Jiajie Peng, Hansheng Xue, Zhongyu Wei, Idil Tuncali, Jianye Hao, and Xuequn Shang. 2020. Integrating multi-network topology for gene function prediction using deep neural networks. Briefings in Bioinformatics (2020). bbaa036.
[45]
Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence. 2009. Dataset Shift in Machine Learning. The MIT Press.
[46]
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems. MIT Press, Cambridge, MA, 91–99.
[47]
M. Rosenstein, Zvika Marx, and L. Kaelbling. 2005. To transfer or not to transfer. In Proceedings of the International Conference on Neural Information Processing Systems Workshop Inductive Transfer: 10 Years Later.
[48]
Xiangbo Shu, Guo-Jun Qi, Jinhui Tang, and Jingdong Wang. 2015. Weakly-shared deep transfer networks for heterogeneous-domain knowledge propagation. In Proceedings of the 23rd ACM International Conference on Multimedia. Association for Computing Machinery, New York, NY, 35–44.
[49]
Karen Simonyan and Andrew Zisserman. 2014. Two-stream convolutional networks for action recognition in videos. In Proceedings of the Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 568–576.
[50]
Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations. IEEE.
[51]
Xiao Song, Xu Zhao, Liangji Fang, Hanwen Hu, and Yizhou Yu. 2020. EdgeStereo: An effective multi-task learning network for stereo matching and edge detection. International Journal of Computer Vision 128, 4 (2020).
[52]
Jost Tobias Springenberg. 2015. Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks. (2015). arXiv:1511.06390. Retrieved from https://arxiv.org/abs/1511.06390.
[53]
C. Sun, A. Shrivastava, S. Singh, and A. Gupta. 2017. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the 2017 IEEE International Conference on Computer Vision. IEEE, 843–852.
[54]
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2818–2826.
[55]
Jinhui Tang, Xiangbo Shu, Zechao Li, Guo-Jun Qi, and Jingdong Wang. 2016. Generalized deep transfer networks for knowledge propagation in heterogeneous domains. ACM Transactions on Multimedia Computing, Communications, and Applications 12, 4s (2016), 22 pages.
[56]
Y. Tu, Yun Lin, J. Wang, and J.-U. Kim. 2018. Semi-supervised learning with generative adversarial networks on digital signal modulation classification. Computers, Materials and Continua 55, 2 (2018), 243–254.
[57]
Haifeng Xia and Zhengming Ding. 2020. Structure preserving generative cross-domain learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
[58]
Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analysis. In Proceedings of The 33rd International Conference on Machine Learning. 478–487.
[59]
Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on International Conference on Machine Learning. JMLR.org, 2048–2057.
[60]
Xiaoqiang Yan, Zhengzheng Lou, Shizhe Hu, and Yangdong Ye. 2020. Multi-task information bottleneck co-clustering for unsupervised cross-view human action categorization. ACM Transactions on Knowledge Discovery from Data 14, 2 (2020), 1–23.
[61]
Zhiyu Yao, Yunbo Wang, Mingsheng Long, and Jianmin Wang. 2020. Unsupervised transfer learning for spatiotemporal predictive networks. In Proceedings of the International Conference on Machine Learning. PMLR, 10778–10788.
[62]
N. Ye, X. Li, H. Yu, L. Zhao, W. Liu, and X. Hou. 2020. DeepNOMA: A unified framework for NOMA using deep multi-task learning. IEEE Transactions on Wireless Communications 19, 4 (2020), 2208–2225.
[63]
D. Yun-Mei, A. Maalla, L. Hui-Ying, H. Shuai, L. Dong, L. Long, and L. Hongsheng. 2020. The abnormal detection of electroencephalogram with three-dimensional deep convolutional neural networks. IEEE Access 8 (2020), 64646–64652.
[64]
D. Zhang, S. Li, Q. Zhu, and G. Zhou. 2020. Multi-modal sentiment classification with independent and interactive knowledge via semi-supervised learning. IEEE Access 8 (2020), 22945–22954.
[65]
Zhi-Hua Zhou. 2017. A brief introduction to weakly supervised learning. National Science Review 5, 1 (2017), 44–53.
[66]
Xiaojin Zhu. 2005. Semi-Supervised Learning Literature Survey. Technical Report 1530. Computer Sciences, University of Wisconsin-Madison.
Index Terms
- Evidence Transfer: Learning Improved Representations According to External Heterogeneous Task Outcomes
Recommendations
Deep representation-based transfer learning for deep neural networks
AbstractIn recent years, deep neural networks (DNNs) have become the de facto models for practically all visual tasks and most temporal analysis tasks due to the abundance of available labeled data and advances in computational resources. Deep ...
Highlights- A deep representation-based transfer learning method is proposed for knowledge transfer between deep neural networks.
Deep learning of representations for unsupervised and transfer learning
UTLW'11: Proceedings of the 2011 International Conference on Unsupervised and Transfer Learning workshop - Volume 27Deep learning algorithms seek to exploit the unknown structure in the input distribution in order to discover good representations, often at multiple levels, with higher-level learned features defined in terms of lower-level features. The objective is ...
Transfer metric learning by learning task relationships
KDD '10: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data miningDistance metric learning plays a very crucial role in many data mining algorithms because the performance of an algorithm relies heavily on choosing a good metric. However, the labeled data available in many applications is scarce and hence the metrics ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2022
532 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 09 March 2022
Accepted: 01 November 2021
Revised: 01 August 2021
Received: 01 January 2021
Published in TKDD Volume 16, Issue 5
Check for updates
Author Tags
Qualifiers
- Research-article
- Refereed
Funding Sources
- Industrial Scholarships program of Stavros Niarchos Foundation
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 323Total Downloads
- Downloads (Last 12 months)44
- Downloads (Last 6 weeks)4
Reflects downloads up to 12 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format