More Web Proxy on the site http://driver.im/

research-article

Free access

Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation

AUTHORs:

Jiashi FengAuthors Info & Claims

ICML'20: Proceedings of the 37th International Conference on Machine Learning

Article No.: 560, Pages 6028 - 6039

Published: 13 July 2020 Publication History

PDF eReader Publisher Site

Abstract

Unsupervised domain adaptation (UDA) aims to leverage the knowledge learned from a labeled source dataset to solve similar tasks in a new unlabeled domain. Prior UDA methods typically require to access the source data when learning to adapt the model, making them risky and inefficient for decentralized private data. This work tackles a practical setting where only a trained source model is available and investigates how we can effectively utilize such a model without source data to solve UDA problems. We propose a simple yet generic representation learning framework, named Source HypOthesis Transfer (SHOT). SHOT freezes the classifier module (hypothesis) of the source model and learns the target-specific feature extraction module by exploiting both information maximization and self-supervised pseudo-labeling to implicitly align representations from the target domains to the source hypothesis. To verify its versatility, we evaluate SHOT in a variety of adaptation cases including closed-set, partial-set, and open-set domain adaptation. Experiments indicate that SHOT yields state-of-the-art results among multiple domain adaptation benchmarks.

References

[1]

Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. Springer MLJ, 79(1-2):151-175, 2010.

[2]

Bendale, A. and Boult, T. E. Towards open set deep networks. In CVPR, 2016.

[3]

Bhushan Damodaran, B., Kellenberger, B., Flamary, R., Tuia, D., and Courty, N. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In ECCV, 2018.

[4]

Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., and Seth, K. Practical secure aggregation for privacy-preserving machine learning. In ACM CCS, 2017.

Digital Library

[5]

Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., Kiddon, C., Konecny, J., Mazzocchi, S., McMahan, H. B., et al. Towards federated learning at scale: System design. arXiv preprint arXiv:1902.01046, 2019.

[6]

Bousmalis, K., Trigeorgis, G., Silberman, N., Krishnan, D., and Erhan, D. Domain separation networks. In NeurIPS, 2016.

[7]

Cao, Z., Long, M., Wang, J., and Jordan, M. I. Partial transfer learning with selective adversarial networks. In CVPR, 2018.

[8]

Cao, Z., You, K., Long, M.,Wang, J., and Yang, Q. Learning to transfer examples for partial domain adaptation. In CVPR, 2019.

[9]

Cariucci, F. M., Porzi, L., Caputo, B., Ricci, E., and Bulo, S. R. Autodial: Automatic domain alignment layers. In ICCV, 2017.

[10]

Caron, M., Bojanowski, P., Joulin, A., and Douze, M. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.

Digital Library

[11]

Chen, X., Wang, S., Long, M., and Wang, J. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In ICML, 2019.

[12]

Chen, Y., Li, W., Sakaridis, C., Dai, D., and Van Gool, L. Domain adaptive faster r-cnn for object detection in the wild. In CVPR, 2018.

[13]

Chidlovskii, B., Clinchant, S., and Csurka, G. Domain adaptation in the absence of source domain data. In KDD, 2016.

Digital Library

[14]

Choi, J., Jeong, M., Kim, T., and Kim, C. Pseudolabeling curriculum for unsupervised domain adaptation. In BMVC, 2019.

[15]

Csurka, G. A comprehensive survey on domain adaptation for visual applications. In Domain Adaptation in Computer Vision Applications, pp. 1-35. Springer, 2017.

[16]

Deng, Z., Luo, Y., and Zhu, J. Cluster alignment with a teacher for unsupervised domain adaptation. In ICCV, 2019.

[17]

French, G., Mackiewicz, M., and Fisher, M. Self-ensembling for visual domain adaptation. In ICLR, 2018.

[18]

Ganin, Y. and Lempitsky, V. Unsupervised domain adaptation by backpropagation. In ICML, 2015.

Digital Library

[19]

Ghifary, M., Kleijn, W. B., Zhang, M., Balduzzi, D., and Li, W. Deep reconstruction-classification networks for unsupervised domain adaptation. In ECCV, 2016.

[20]

Glorot, X., Bordes, A., and Bengio, Y. Domain adaptation for large-scale sentiment classification: a deep learning approach. In ICML, 2011.

Digital Library

[21]

Gong, B., Shi, Y., Sha, F., and Grauman, K. Geodesic flow kernel for unsupervised domain adaptation. In CVPR, 2012.

[22]

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In NeurIPS, 2014.

Digital Library

[23]

Grandvalet, Y. and Bengio, Y. Semi-supervised learning by entropy minimization. In NeurIPS, 2005.

[24]

Gretton, A., Borgwardt, K., Rasch, M., Schölkopf, B., and Smola, A. J. A kernel method for the two-sampleproblem. In NeurIPS, 2007.

[25]

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.

[26]

Hoffman, J., Tzeng, E., Park, T., Zhu, J.-Y., Isola, P., Saenko, K., Efros, A., and Darrell, T. Cycada: Cycle-consistent adversarial domain adaptation. In ICML, 2018.

[27]

Hu, W., Miyato, T., Tokui, S., Matsumoto, E., and Sugiyama, M. Learning discrete representations via information maximizing self-augmented training. In ICML, 2017.

[28]

Huang, J., Gretton, A., Borgwardt, K., Schölkopf, B., and Smola, A. J. Correcting sample selection bias by unlabeled data. In NeurIPS, 2007.

[29]

Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.

Digital Library

[30]

Jiang, J. and Zhai, C. Instance weighting for domain adaptation in nlp. In ACL, 2007.

[31]

Krause, A., Perona, P., and Gomes, R. G. Discriminative clustering by regularized information maximization. In NeurIPS, 2010.

[32]

Kuzborskij, I. and Orabona, F. Stability and hypothesis transfer learning. In ICML, 2013.

Digital Library

[33]

LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.

[34]

Lee, C.-Y., Batra, T., Baig, M. H., and Ulbricht, D. Sliced wasserstein discrepancy for unsupervised domain adaptation. In CVPR, 2019a.

[35]

Lee, D.-H. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, 2013.

[36]

Lee, S., Kim, D., Kim, N., and Jeong, S.-G. Drop to adapt: Learning discriminative features for unsupervised domain adaptation. In ICCV, 2019b.

[37]

Liang, J., He, R., Sun, Z., and Tan, T. Aggregating randomized clustering-promoting invariant projections for domain adaptation. IEEE TPAMI, 41(5):1027-1042, 2018.

[38]

Liang, J., He, R., Sun, Z., and Tan, T. Distant supervised centroid shift: A simple and efficient approach to visual domain adaptation. In CVPR, 2019a.

[39]

Liang, J., He, R., Sun, Z., and Tan, T. Exploring uncertainty in pseudo-label guided unsupervised domain adaptation. Elsevier PRJ, 96:106996, 2019b.

[40]

Liu, H., Cao, Z., Long, M., Wang, J., and Yang, Q. Separate to adapt: Open set domain adaptation via progressive separation. In CVPR, 2019.

[41]

Long, M., Wang, J., Ding, G., Sun, J., and Yu, P. S. Transfer feature learning with joint distribution adaptation. In ICCV, 2013.

Digital Library

[42]

Long, M., Cao, Y., Wang, J., and Jordan, M. Learning transferable features with deep adaptation networks. In ICML, 2015.

Digital Library

[43]

Long, M., Zhu, H., Wang, J., and Jordan, M. I. Deep transfer learning with joint adaptation networks. In ICML, 2017.

Digital Library

[44]

Long, M., Cao, Z., Wang, J., and Jordan, M. I. Conditional adversarial domain adaptation. In NeurIPS, 2018.

[45]

Mansour, Y., Mohri, M., and Rostamizadeh, A. Domain adaptation with multiple sources. In NeurIPS, 2009.

[46]

McMahan, H. B., Ramage, D., Talwar, K., and Zhang, L. Learning differentially private recurrent language models. In ICLR, 2018.

[47]

Müller, R., Kornblith, S., and Hinton, G. E. When does label smoothing help? In NeurIPS, 2019.

[48]

Nelakurthi, A. R., Maciejewski, R., and He, J. Source free domain adaptation using an off-the-shelf classifier. In IEEE BigData, 2018.

[49]

Panareda Busto, P. and Gall, J. Open set domain adaptation. In ICCV, 2017.

[50]

Peng, X., Usman, B., Kaushik, N., Hoffman, J., Wang, D., and Saenko, K. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924, 2017.

[51]

Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., and Wang, B. Moment matching for multi-source domain adaptation. In ICCV, 2019a.

[52]

Peng, X., Huang, Z., Sun, X., and Saenko, K. Domain agnostic learning with disentangled representations. In ICML, 2019b.

[53]

Peng, X., Huang, Z., Zhu, Y., and Saenko, K. Federated adversarial domain adaptation. In ICLR, 2020.

[54]

Saenko, K., Kulis, B., Fritz, M., and Darrell, T. Adapting visual category models to new domains. In ECCV, 2010.

Digital Library

[55]

Saito, K., Ushiku, Y., Harada, T., and Saenko, K. Adversarial dropout regularization. In ICLR, 2018a.

[56]

Saito, K., Watanabe, K., Ushiku, Y., and Harada, T. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR, 2018b.

[57]

Saito, K., Yamamoto, S., Ushiku, Y., and Harada, T. Open set domain adaptation by backpropagation. In ECCV, 2018c.

[58]

Saito, K., Kim, D., Sclaroff, S., Darrell, T., and Saenko, K. Semi-supervised domain adaptation via minimax entropy. In ICCV, 2019.

[59]

Salimans, T. and Kingma, D. P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In NeurIPS, 2016.

[60]

Shi, Y. and Sha, F. Information-theoretical learning of discriminative clusters for unsupervised domain adaptation. In ICML, 2012.

Digital Library

[61]

Shu, R., Bui, H. H., Narui, H., and Ermon, S. A dirt-t approach to unsupervised domain adaptation. In ICLR, 2018.

[62]

Sun, B., Feng, J., and Saenko, K. Return of frustratingly easy domain adaptation. In AAAI, 2016.

[63]

Tommasi, T., Orabona, F., and Caputo, B. Learning categories from few examples with multi model knowledge transfer. IEEE TPAMI, 36(5):928-941, 2013.

[64]

Tsai, Y.-H., Hung, W.-C., Schulter, S., Sohn, K., Yang, M.- H., and Chandraker, M. Learning to adapt structured output space for semantic segmentation. In CVPR, 2018.

[65]

Tzeng, E., Hoffman, J., Saenko, K., and Darrell, T. Adversarial discriminative domain adaptation. In CVPR, 2017.

[66]

Venkateswara, H., Eusebio, J., Chakraborty, S., and Panchanathan, S. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.

[67]

Vu, T.-H., Jain, H., Bucher, M., Cord, M., and Pérez, P. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR, 2019.

[68]

Wang, X., Jin, Y., Long, M., Wang, J., and Jordan, M. I. Transferable normalization: Towards improving transferability of deep neural networks. In NeurIPS, 2019.

[69]

Xie, S., Zheng, Z., Chen, L., and Chen, C. Learning semantic representations for unsupervised domain adaptation. In ICML, 2018.

[70]

Xu, R., Chen, Z., Zuo, W., Yan, J., and Lin, L. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In CVPR, 2018.

[71]

Xu, R., Li, G., Yang, J., and Lin, L. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In ICCV, 2019.

[72]

Yang, J., Yan, R., and Hauptmann, A. G. Cross-domain video concept detection using adaptive svms. In ACMMM, 2007.

Digital Library

[73]

Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. How transferable are features in deep neural networks? In NeurIPS, 2014.

[74]

Zellinger, W., Grubinger, T., Lughofer, E., Natschläger, T., and Saminger-Platz, S. Central moment discrepancy (cmd) for domain-invariant representation learning. In ICLR, 2017.

[75]

Zhang, J., Ding, Z., Li, W., and Ogunbona, P. Importance weighted adversarial nets for partial domain adaptation. In CVPR, 2018a.

[76]

Zhang, W., Ouyang, W., Li, W., and Xu, D. Collaborative and adversarial network for unsupervised domain adaptation. In CVPR, 2018b.

[77]

Zhang, Y., David, P., and Gong, B. Curriculum domain adaptation for semantic segmentation of urban scenes. In ICCV, 2017.

[78]

Zou, Y., Yu, Z., Vijaya Kumar, B., and Wang, J. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In ECCV, 2018.

Cited By

Li ZZhao SChen CChen Q(2024)Reducing the Impact of Time Evolution on Source Code Authorship Attribution via Domain AdaptationACM Transactions on Software Engineering and Methodology10.1145/365215133:6(1-27)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3652151
Zhang LWang YSong RZhang MLi XZhang W(2024)Neighborhood-Aware Mutual Information Maximization for Source-Free Domain AdaptationIEEE Transactions on Multimedia10.1109/TMM.2024.339497126(9564-9574)Online publication date: 30-Apr-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3394971
Zhao XHuang LNie JWei Z(2024)Towards Adaptive Multi-Scale Intermediate Domain via Progressive Training for Unsupervised Domain AdaptationIEEE Transactions on Multimedia10.1109/TMM.2023.333008826(5054-5064)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3330088
Show More Cited By

Index Terms

Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
  2. Machine learning
    1. Learning paradigms
    2. Machine learning approaches

Index terms have been assigned to the content through auto-classification.

Recommendations

Source Hypothesis Transfer for Zero-Shot Domain Adaptation
Machine Learning and Knowledge Discovery in Databases. Research Track
Abstract
Making predictions in target unseen domains without training samples is frequent in real-world applications, such as new products’ sales predictions. Zero-shot domain adaptation (ZSDA) has been studied to achieve this important but difficult task. ...
Source Free Graph Unsupervised Domain Adaptation
WSDM '24: Proceedings of the 17th ACM International Conference on Web Search and Data Mining

Graph Neural Networks (GNNs) have achieved great success on a variety of tasks with graph-structural data, among which node classification is an essential one. Unsupervised Graph Domain Adaptation (UGDA) shows its practical value of reducing the labeling ...
Source-free domain adaptation with unrestricted source hypothesis
Abstract
Domain adaptation aims to bridge the distribution discrepancy across different domains and improve the generalization ability of learning models on the target domain. The existing domain adaptation approaches align the distribution shift via ...
Highlights
- We derive the learning bound for the source-free domain adaptation and propose source-free adversarial domain adaptation method.
- Our method does not require specific restrictions to source hypothesis.
- Experiments on three image-...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

ICML'20: Proceedings of the 37th International Conference on Machine Learning

July 2020

11702 pages

Editors:
Hal Daumé,
Aarti Singh

Copyright © 2020.

Publisher

JMLR.org

Publication History

Published: 13 July 2020

Qualifiers

Research-article
Research
Refereed limited

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
167
Total Downloads

Downloads (Last 12 months)98
Downloads (Last 6 weeks)39

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li ZZhao SChen CChen Q(2024)Reducing the Impact of Time Evolution on Source Code Authorship Attribution via Domain AdaptationACM Transactions on Software Engineering and Methodology10.1145/365215133:6(1-27)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3652151
Zhang LWang YSong RZhang MLi XZhang W(2024)Neighborhood-Aware Mutual Information Maximization for Source-Free Domain AdaptationIEEE Transactions on Multimedia10.1109/TMM.2024.339497126(9564-9574)Online publication date: 30-Apr-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3394971
Zhao XHuang LNie JWei Z(2024)Towards Adaptive Multi-Scale Intermediate Domain via Progressive Training for Unsupervised Domain AdaptationIEEE Transactions on Multimedia10.1109/TMM.2023.333008826(5054-5064)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3330088
Wu TJia FQi XWang JSehwag VMahloujifar SMittal PKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Uncovering adversarial risks of test-time adaptationProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619967(37456-37495)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3619967
Guo LZhou ZLi YZhou ZKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Identifying useful learnwares for heterogeneous label spacesProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3618896(12122-12131)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3618896
Goyal SSun MRaghunathan AKolter ZKoyejo SMohamed SAgarwal ABelgrave DCho KOh A(2022)Test-time adaptation via conjugate pseudo-labelsProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3600719(6204-6218)Online publication date: 28-Nov-2022
https://dl.acm.org/doi/10.5555/3600270.3600719
Yu YZhai YZhang YMagalhães Jdel Bimbo ASatoh SSebe NAlameda-Pineda XJin QOria VToni L(2022)Align and Adapt: A Two-stage Adaptation Framework for Unsupervised Domain AdaptationProceedings of the 30th ACM International Conference on Multimedia10.1145/3503161.3547973(4723-4732)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3503161.3547973
Zhao YWang MCai LRanzato MBeygelzimer ADauphin YLiang PVaughan J(2021)Reducing the covariate shift by mirror samples in cross domain alignmentProceedings of the 35th International Conference on Neural Information Processing Systems10.5555/3540261.3540992(9546-9558)Online publication date: 6-Dec-2021
https://dl.acm.org/doi/10.5555/3540261.3540992

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents