More Web Proxy on the site http://driver.im/

research-article

Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation

Authors:

Yi YangAuthors Info & Claims

International Journal of Computer Vision, Volume 129, Issue 4

Pages 1106 - 1120

https://doi.org/10.1007/s11263-020-01395-y

Published: 01 April 2021 Publication History

Abstract

This paper focuses on the unsupervised domain adaptation of transferring the knowledge from the source domain to the target domain in the context of semantic segmentation. Existing approaches usually regard the pseudo label as the ground truth to fully exploit the unlabeled target-domain data. Yet the pseudo labels of the target-domain data are usually predicted by the model trained on the source domain. Thus, the generated labels inevitably contain the incorrect prediction due to the discrepancy between the training domain and the test domain, which could be transferred to the final adapted model and largely compromises the training process. To overcome the problem, this paper proposes to explicitly estimate the prediction uncertainty during training to rectify the pseudo label learning for unsupervised semantic segmentation adaptation. Given the input image, the model outputs the semantic segmentation prediction as well as the uncertainty of the prediction. Specifically, we model the uncertainty via the prediction variance and involve the uncertainty into the optimization objective. To verify the effectiveness of the proposed method, we evaluate the proposed method on two prevalent synthetic-to-real semantic segmentation benchmarks, i.e., GTA5

\to

Cityscapes and SYNTHIA

\to

Cityscapes, as well as one cross-city benchmark, i.e., Cityscapes

\to

Oxford RobotCar. We demonstrate through extensive experiments that the proposed approach (1) dynamically sets different confidence thresholds according to the prediction variance, (2) rectifies the learning from noisy pseudo labels, and (3) achieves significant improvements over the conventional pseudo label learning and yields competitive performance on all three benchmarks.

References

[1]

Blum, A., & Mitchell, T. (1998). Combining labeled and unlabeled data with co-training. In COLT.

[2]

Chen L-C, Papandreou G, Kokkinos I, Murphy K, and Yuille AL Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs IEEE Transactions on Pattern Analysis and Machine Intelligence 2017 40 4 834-848

[3]

Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016). The cityscapes dataset for semantic urban scene understanding. In CVPR.

[4]

Fu Y, Hospedales TM, Xiang T, and Gong S Transductive multi-view zero-shot learning IEEE Transactions on Pattern Analysis and Machine Intelligence 2015 37 11 2332-2345

[5]

Gal, Y., & Ghahramani, Z. (2016). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML.

[6]

Ganin, Y., & Lempitsky, V. (2015). Unsupervised domain adaptation by backpropagation. In ICML.

[7]

Grandvalet, Y., & Bengio, Y. (2005). Semi-supervised learning by entropy minimization. In NeurIPS.

[8]

Han, L., Zou, Y., Gao, R., Wang, L., & Metaxas, D. (2019). Unsupervised domain adaptation via calibrating uncertainties. In CVPR Workshops.

[9]

He, K., Zhang, X., Ren, S., & Sun, J. (2016a). Deep residual learning for image recognition. In CVPR.

[10]

He, K., Zhang, X., Ren, S., & Sun, J. (2016b). Identity mappings in deep residual networks. In ECCV.

[11]

Hendrycks, D., & Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. In ICLR.

[12]

Hoffman, J., Tzeng, E., Park, T., Zhu, J.-Y., Isola, P., Saenko, K., Efros, A. A., & Darrell, T. (2018). Cycada: Cycle-consistent adversarial domain adaptation. In ICML.

[13]

Huang, H., Huang, Q., & Krahenbuhl, P. (2018). Domain transfer through deep activation matching. In ECCV.

[14]

Kang, G., Wei, Y., Yang, Y., & Hauptmann, A. (2020). Pixel-level cycle association: A new perspective for domain adaptive semantic segmentation. In NeurIPS.

[15]

Kendall, A., & Gal, Y. (2017). What uncertainties do we need in bayesian deep learning for computer vision? In NeurIPS.

[16]

Lee, D.-H. (2013). Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML: In Workshop on challenges in representation learning.

[17]

Li, G., Kang, G., Liu, W., Wei, Y., & Yang, Y. (2020a). Content-consistent matching for domain adaptive semantic segmentation. In ECCV.

[18]

Li, P., Wei, Y., & Yang, Y. (2020). Consistent structural relation learning for zero-shot segmentationg. In NeurIPS.

[19]

Li, P., Wei, Y., & Yang, Y. (2020). Meta parsing networks: Towards generalized few-shot scene parsing with adaptive metric learning. In ACM Multimedia.

[20]

Liang X, Lin L, Wei Y, Shen X, Yang J, and Yan S Proposal-free network for instance-level object segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 2017 40 12 2978-2991

[21]

Luo, Y., Liu, P., Guan, T., Yu, J., & Yang, Y. (2019a). Significance-aware information bottleneck for domain adaptive semantic segmentation. In ICCV.

[22]

Luo, Y., Liu, P., Guan, T., Yu, J., & Yang, Y. (2020). Adversarial style mining for one-shot unsupervised domain adaptation. In NeurIPS.

[23]

Luo, Y., Zheng, L., Guan, T., Yu, J., & Yang, Y. (2019b). Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. In CVPR.

[24]

Maddern W, Pascoe G, Linegar C, and Newman P 1 Year, 1000 km: The Oxford RobotCar Dataset The International Journal of Robotics Research (IJRR) 2017 36 1 3-15

[25]

Nielsen TD and Jensen FV Bayesian networks and decision graphs 2009 New York Springer

[26]

Pan, Y., Yao, T., Li, Y., Wang, Y., Ngo, C.-W., & Mei, T. (2019). Transferrable prototypical networks for unsupervised domain adaptation. In CVPR.

[27]

Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., & Lerer, A. (2017). Automatic differentiation in pytorch.

[28]

Reed, S., Lee, H., Anguelov, D., Szegedy, C., Erhan, D., & Rabinovich, A. (2014). Training deep neural networks on noisy labels with bootstrapping. arXiv:1412.6596.

[29]

Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016). Playing for data: Ground truth from computer games. In ECCV.

[30]

Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016). The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR.

[31]

Saito, K., Watanabe, K., Ushiku, Y., & Harada, T. (2018). Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR.

[32]

Srivastava N, Hinton G, Krizhevsky A, Sutskever I, and Salakhutdinov R Dropout: a simple way to prevent neural networks from overfitting The journal of machine learning research 2014 15 1 1929-1958

[33]

Tsai, Y.-H., Hung, W.-C., Schulter, S., Sohn, K., Yang, M.-H., & Chandraker, M. (2018). Learning to adapt structured output space for semantic segmentation. In CVPR.

[34]

Tsai, Y.-H., Sohn, K., Schulter, S., & Chandraker, M. (2019). Domain adaptation for structured output via discriminative representations. In ICCV.

[35]

Tzeng, E., Hoffman, J., Darrell, T., & Saenko, K. (2015). Simultaneous deep transfer across domains and tasks. In ICCV.

[36]

Vu, T.-H., Jain, H., Bucher, M., Cord, M., & Pérez, P. (2019). Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR.

[37]

Wang, J., Zhu, X., Gong, S., & Li, W. (2018). Transferable joint attribute-identity deep learning for unsupervised person re-identification. In CVPR.

[38]

Wang S, Zhang L, Zuo W, and Zhang B Class-specific reconstruction transfer learning for visual recognition across domains IEEE Transactions on Image Processing 2019 29 2424-2438

[39]

Wei, Y., Xiao, H., Shi, H., Jie, Z., Feng, J., & Huang, T. S. (2018). Revisiting dilated convolution: A simple approach for weakly-and semi-supervised semantic segmentation. In CVPR.

[40]

Wu, Z., Han, X., Lin, Y.-L., Gokhan Uzunbas, M., Goldstein, T., Nam Lim, S., & Davis, L. S. (2018). Dcan: Dual channel-wise alignment networks for unsupervised scene adaptation. In ECCV.

[41]

Wu, Z., Wang, X., Gonzalez, J. E., Goldstein, T., & Davis, L. S. (2019). Ace: adapting to changing environments for semantic segmentation. In ICCV.

[42]

Yang, J., Xu, R., Li, R., Qi, X., Shen, X., Li, G., & Lin, L. (2020). An adversarial perturbation oriented domain adaptation approach for semantic segmentation. In AAAI.

[43]

Yu, T., Li, D., Yang, Y., Hospedales, T. M., & Xiang, T. (2019). Robust person re-identification by modelling feature uncertainty. In ICCV.

[44]

Yue, X., Zhang, Y., Zhao, S., Sangiovanni-Vincentelli, A., Keutzer, K., & Gong, B. (2019). Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. In ICCV.

[45]

Zhang, L., Li, X., Arnab, A., Yang, K., Tong, Y., & Torr, P. H. (2019a). Dual graph convolutional network for semantic segmentation. In BMVC.

[46]

Zhang L, Wang S, Huang G-B, Zuo W, Yang J, and Zhang D Manifold criterion guided transfer learning via intermediate domain generation IEEE Transactions on Neural Networks and Learning Systems 2019 30 12 3759-3773

[47]

Zhang, L., Xu, D., Arnab, A., and Torr, P. H. (2020). Dynamic graph message passing network. In CVPR.

[48]

Zhang, Y., Qiu, Z., Yao, T., Liu, D., & Mei, T. (2018). Fully convolutional adaptation networks for semantic segmentation. In CVPR.

[49]

Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017). Pyramid scene parsing network. In CVPR.

[50]

Zheng, Z., & Yang, Y. (2020). Unsupervised scene adaptation with memory regularization in vivo. In IJCAI.

[51]

Zou, Y., Yu, Z., Liu, X., Kumar, B., & Wang, J. (2019). Confidence regularized self-training. In ICCV.

[52]

Zou, Y., Yu, Z., Vijaya Kumar, B., & Wang, J. (2018). Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In ECCV.

Cited By

Wang ZSuganuma MOkatani T(2025)Rethinking unsupervised domain adaptation for semantic segmentationPattern Recognition Letters10.1016/j.patrec.2024.09.022186:C(119-125)Online publication date: 30-Jan-2025
https://dl.acm.org/doi/10.1016/j.patrec.2024.09.022
Zhao JLi S(2025)Uncertainty-guided and cross-modality attention network for liver tumor segmentation and quantification via integrating dynamic MRIKnowledge-Based Systems10.1016/j.knosys.2025.113021310:COnline publication date: 15-Feb-2025
https://dl.acm.org/doi/10.1016/j.knosys.2025.113021
Dong MYang AWang ZLi DYang JZhao R(2025)Uncertainty-aware consistency learning for semi-supervised medical image segmentationKnowledge-Based Systems10.1016/j.knosys.2024.112890309:COnline publication date: 18-Feb-2025
https://dl.acm.org/doi/10.1016/j.knosys.2024.112890
Show More Cited By

Recommendations

Semantic contrast with uncertainty-aware pseudo label for lumbar semi-supervised classification
Abstract Background
Lumbar disc herniation (LDH) is a prevalent spinal disease that can result in severe pain, with Magnetic resonance imaging (MRI) serving as a commonly diagnostic tool. However, annotating numerous MRI images, necessary for deep ...
Graphical abstract

Display Omitted
Highlights
- Semi-supervised learning is applied to address the challenge of MRI images annotation for lumbar disc herniation diagnosis.
- Semi-supervised learning integrated with semantic contrast maintain the semantic consistency of labeled and ...
Domain Adaptive Video Segmentation via Temporal Pseudo Supervision
Computer Vision – ECCV 2022
Abstract
Video semantic segmentation has achieved great progress under the supervision of large amounts of labelled training data. However, domain adaptive video segmentation, which can mitigate data labelling constraints by adapting from a labelled source ...
Pseudo-label refinement via hierarchical contrastive learning for source-free unsupervised domain adaptation
Abstract
Source-free unsupervised domain adaptation aims to adapt a source model to an unlabeled target domain without accessing the source data due to privacy considerations. Existing works mainly solve the problem by self-training methods and ...
Highlights
- Propose a hierarchical self-supervised learning method for source-free UDA.
- Devise an adaptive pseudo-labeling method to obtain much more reliable labels.
- Improve the pseudo-label quality with hierarchical contrastive learning.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image International Journal of Computer Vision

International Journal of Computer Vision Volume 129, Issue 4

Apr 2021

535 pages

ISSN:0920-5691

Issue’s Table of Contents

© Springer Science+Business Media, LLC, part of Springer Nature 2021.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 April 2021

Accepted: 15 October 2020

Received: 07 March 2020

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

151
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang ZSuganuma MOkatani T(2025)Rethinking unsupervised domain adaptation for semantic segmentationPattern Recognition Letters10.1016/j.patrec.2024.09.022186:C(119-125)Online publication date: 30-Jan-2025
https://dl.acm.org/doi/10.1016/j.patrec.2024.09.022
Zhao JLi S(2025)Uncertainty-guided and cross-modality attention network for liver tumor segmentation and quantification via integrating dynamic MRIKnowledge-Based Systems10.1016/j.knosys.2025.113021310:COnline publication date: 15-Feb-2025
https://dl.acm.org/doi/10.1016/j.knosys.2025.113021
Dong MYang AWang ZLi DYang JZhao R(2025)Uncertainty-aware consistency learning for semi-supervised medical image segmentationKnowledge-Based Systems10.1016/j.knosys.2024.112890309:COnline publication date: 18-Feb-2025
https://dl.acm.org/doi/10.1016/j.knosys.2024.112890
Jiang JZhao SZhu JTang WXu ZYang JLiu GXing TXu PYao H(2025)Multi-source domain adaptation for panoramic semantic segmentationInformation Fusion10.1016/j.inffus.2024.102909117:COnline publication date: 1-May-2025
https://dl.acm.org/doi/10.1016/j.inffus.2024.102909
Zhao SJiang JTang WZhu JChen HXu PSchuller BTao JYao HDing G(2025)Multi-source multi-modal domain adaptationInformation Fusion10.1016/j.inffus.2024.102862117:COnline publication date: 1-May-2025
https://dl.acm.org/doi/10.1016/j.inffus.2024.102862
Yu XTeng LZhang DZheng JChen H(2025)Attention correction feature and boundary constraint knowledge distillation for efficient 3D medical image segmentationExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.125670262:COnline publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1016/j.eswa.2024.125670
Luo YLiu PYang Y(2025)Kill Two Birds with One Stone: Domain Generalization for Semantic Segmentation via Network PruningInternational Journal of Computer Vision10.1007/s11263-024-02194-5133:1(335-352)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1007/s11263-024-02194-5
Mai HSun RWang YZhang TWu FWooldridge MDy JNatarajan S(2024)Pay attention to targetProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i5.28211(4162-4170)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i5.28211
Li SHe CXu XShen FYang YShen HWooldridge MDy JNatarajan S(2024)Adaptive uncertainty-based learning for text-based person retrievalProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i4.28101(3172-3180)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i4.28101
Ge LHu CMa GLiu JZhang HWooldridge MDy JNatarajan S(2024)Discrepancy and uncertainty aware denoising knowledge distillation for zero-shot cross-lingual named entity recognitionProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i16.29762(18056-18064)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i16.29762
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents