More Web Proxy on the site http://driver.im/

research-article

A Boundary-Aware Network for Shadow Removal

Authors:

Guanyu XingAuthors Info & Claims

IEEE Transactions on Multimedia, Volume 25

Pages 6782 - 6793

https://doi.org/10.1109/TMM.2022.3214422

Published: 13 October 2022 Publication History

Abstract

Shadow removal is a challenging computer vision and multimedia task that aims to restore image content in shadow regions. The state-of-the-art shadow removal methods introduce artifacts near shadow boundaries or inconsistencies between shadow and nonshadow areas, which can be easily noticed by the human eye at first glance. In this paper, we design a boundary-aware shadow removal network (BA-ShadowNet) that improves shadow removal accuracy by increasing the removal performance at shadow boundaries. In contrast with previously developed methods, which usually consider shadow boundary optimization to be a postprocessing technique, our method performs shadow removal and shadow boundary optimization simultaneously. For this purpose, the proposed BA-ShadowNet is designed as a multiscale encoder-decoder structure, where the decoder consists of a shadow removal branch and a shadow optimization branch. An interaction module is then introduced to fuse and exchange the features of the two branches. This module facilitates the removal branch in perceiving the locations and colors of shadow boundaries. Additionally, it optimizes the boundary branch according to the image context extracted from the removal branch. A three-term loss function is further developed to supervise the shadow removal results and to address the issue of imbalanced supervision between shadow boundary pixels and pixels inside shadows. Extensive experiments conducted on the ISTD+ and SRD datasets demonstrate that the proposed BA-ShadowNet greatly outperforms the state-of-the-art methods with respect to shadow removal.

References

[1]

S. Nadimi and B. Bhanu, “Physical models for moving shadow and object detection in video,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 8, pp. 1079–1087, Aug. 2004.

Digital Library

[2]

M. Sultana, A. Mahmood, and S. K. Jung, “Unsupervised moving object detection in complex scenes using adversarial regularizations,” IEEE Trans. Multimedia, vol. 23, pp. 2005–2018, 2021.

[3]

Z. Shao, Y. Pu, J. Zhou, B. Wen, and Y. Zhang, “Hyper RPCA: Joint maximum correntropy criterion and laplacian scale mixture modeling on-the-fly for moving object detection,” IEEE Trans. Multimedia, early access, Oct. 20, 2021.

[4]

S. Saravanakumar, A. Vadivel, and C. S. Ahmed, “Multiple human object tracking using background subtraction and shadow removal techniques,” in Proc. Int. Conf. Signal Image Process., 2010, pp. 79–84.

[5]

C. Liu, P. Liu, W. Zhao, and X. Tang, “Robust tracking and redetection: Collaboratively modeling the target and its context,” IEEE Trans. Multimedia, vol. 20, no. 4, pp. 889–902, Apr. 2018.

Digital Library

[6]

N. Liang, G. Wu, W. Kang, Z. Wang, and D. D. Feng, “Real-time long-term tracking with prediction-detection-correction,” IEEE Trans. Multimedia, vol. 20, no. 9, pp. 2289–2302, Sep. 2018.

Digital Library

[7]

T. Zhang et al., “Decoupled spatial neural attention for weakly supervised semantic segmentation,” IEEE Trans. Multimedia, vol. 21, no. 11, pp. 2930–2941, Nov. 2019.

[8]

C. Yu et al., “BiSeNet: Bilateral segmentation network for real-time semantic segmentation,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 325–341.

[9]

B. Kang, Y. Lee, and T. Q. Nguyen, “Depth-adaptive deep neural network for semantic segmentation,” IEEE Trans. Multimedia, vol. 20, no. 9, pp. 2478–2490, Sep. 2018.

Digital Library

[10]

H. Barrow, J. Tenenbaum, A. Hanson, and E. Riseman, “Recovering intrinsic scene characteristics from images,” Comput. Vis. Syst., vol. 2, no. 2, pp. 3–26, 1978.

[11]

Y.-Y. Chuang, D. B. Goldman, B. Curless, D. H. Salesin, and R. Szeliski, “Shadow matting and compositing,” in Proc. ACM SIGGRAPH, 2003, pp. 494–500.

[12]

Q. Yang, K.-H. Tan, and N. Ahuja, “Shadow removal using bilateral filtering,” IEEE Trans. Image Process., vol. 21, no. 10, pp. 4361–4368, Oct. 2012.

Digital Library

[13]

R. Guo, Q. Dai, and D. Hoiem, “Paired regions for shadow detection and removal,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 12, pp. 2956–2967, Dec. 2013.

Digital Library

[14]

L. Qu, J. Tian, S. He, Y. Tang, and R. W. Lau, “DeshadowNet: A multi-context embedding deep network for shadow removal,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2017, pp. 2308–2316.

[15]

J. Wang, X. Li, and J. Yang, “Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018, pp. 1788–1797.

[16]

H. Le and D. Samaras, “Shadow removal via shadow image decomposition,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 8578–8587.

[17]

X. Hu, Y. Jiang, C.-W. Fu, and P.-A. Heng, “Mask-Shadowgan: Learning to remove shadows from unpaired data,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 2472–2481.

[18]

X. Cun, C.-M. Pun, and C. Shi, “Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting GAN,” in Proc. AAAI Conf. Artif. Intell., 2020, vol. 34, no. 7, pp. 10680–10687.

[19]

Y. Jin, A. Sharma, and R. T. Tan, “DC-shadowNet: Single-image hard and soft shadow removal using unsupervised domain-classifier guided network,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2021, pp. 5027–5036.

[20]

Z. Liu et al., “From shadow generation to shadow removal,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, pp. 4927–4936.

[21]

L. Fu et al., “Auto-exposure fusion for single-image shadow removal,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, pp. 10571–10580.

[22]

G. D. Finlayson, S. D. Hordley, C. Lu, and M. S. Drew, “On the removal of shadows from images,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 1, pp. 59–68, Jan. 2006.

Digital Library

[23]

M. Gryka, M. Terry, and G. J. Brostow, “Learning to remove soft shadows,” ACM Trans. Graph., vol. 34, no. 5, pp. 1–15, 2015.

Digital Library

[24]

Y. Shor and D. Lischinski, “The shadow meets the mask: Pyramid-based shadow removal,” Comput. Graph. Forum, vol. 27, no. 2, pp. 577–586, 2008.

[25]

C. Xiao, R. She, D. Xiao, and K.-L. Ma, “Fast shadow removal using adaptive multi-scale illumination transfer,” Comput. Graph. Forum, vol. 32, no. 8, pp. 207–218, 2013.

[26]

L. Zhang, Q. Zhang, and C. Xiao, “Shadow remover: Image shadow removal based on illumination recovering optimization,” IEEE Trans. Image Process., vol. 24, no. 11, pp. 4623–4636, Nov. 2015.

Digital Library

[27]

T. F. Y. Vicente, M. Hoai, and D. Samaras, “Leave-one-out kernel optimization for shadow detection and removal,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 3, pp. 682–695, Mar. 2018.

[28]

X. Hu, C.-W. Fu, L. Zhu, J. Qin, and P.-A. Heng, “Direction-aware spatial context features for shadow detection and removal,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 42, no. 11, pp. 2795–2808, Nov. 2020.

[29]

H. Le and D. Samaras, “Physics-based shadow image decomposition for shadow removal,” IEEE Trans. Pattern Anal. Mach. Intell., early access, Nov. 04, 2021.

[30]

W. Wu et al., “Look at boundary: A boundary-aware face alignment algorithm,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018, pp. 2129–2138.

[31]

J. Su, J. Li, Y. Zhang, C. Xia, and Y. Tian, “Selectivity or invariance: Boundary-aware salient object detection,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 3799–3808.

[32]

K. Fu, Q. Zhao, and I. Y. -H. Gu, “Refinet: A deep segmentation assisted refinement network for salient object detection,” IEEE Trans. Multimedia, vol. 21, no. 2, pp. 457–469, Feb. 2019.

Digital Library

[33]

H. Ding, X. Jiang, A. Q. Liu, N. M. Thalmann, and G. Wang, “Boundary-aware feature propagation for scene segmentation,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 6819–6829.

[34]

Z. Chen et al., “A multi-task mean teacher for semi-supervised shadow detection,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 5611–5620.

[35]

J. Wei et al., “Label decoupling framework for salient object detection,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 13022–13031.

[36]

R. Qian, R. T. Tan, W. Yang, J. Su, and J. Liu, “Attentive generative adversarial network for raindrop removal from a single image,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018, pp. 2482–2491.

[37]

H. Gong and D. Cosker, “Interactive shadow removal and ground truth for variable scene categories,” in Proc. Brit. Mach. Vis. Conf., 2014, pp. 1–11.

[38]

Z. Liu, H. Yin, Y. Mi, M. Pu, and S. Wang, “Shadow removal by a lightness-guided network with training on unpaired data,” IEEE Trans. Image Process., vol. 30, pp. 1853–1865, 2021.

Digital Library

[39]

R. Gadde, V. Jampani, M. Kiefel, D. Kappler, and P. V. Gehler, “Superpixel convolutional networks using bilateral inceptions,” in Proc. Eur. Conf. Comput. Vis., 2016, pp. 597–613.

[40]

S. Ghosh, R. G. Gavaskar, and K. N. Chaudhury, “Saliency guided image detail enhancement,” in Proc. Nat. Conf. Commun., 2019, pp. 1–6.

Cited By

Wang YLiu SLi LZhou WLi H(2024)SwinShadow: Shifted Window for Ambiguous Adjacent Shadow DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/368880320:11(1-20)Online publication date: 27-Aug-2024
https://dl.acm.org/doi/10.1145/3688803

Index Terms

A Boundary-Aware Network for Shadow Removal
1. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

Simple shadow removal using shadow depth map and illumination-invariant feature
Abstract
Shadows included in images provide useful information for visual scene analysis, but are also factors that negatively affect digital image analysis. Therefore, shadow detection and removal must be considered essential in the preprocessing of the ...
From Shadow Segmentation to Shadow Removal
Computer Vision – ECCV 2020
Abstract
The requirement for paired shadow and shadow-free images limits the size and diversity of shadow removal datasets and hinders the possibility of training large-scale, robust shadow removal algorithms. We propose a shadow removal method that can be ...
Shadow Remover: Image Shadow Removal Based on Illumination Recovering Optimization
In this paper, we present a novel shadow removal system for single natural images as well as color aerial images using an illumination recovering optimization method. We first adaptively decompose the input image into overlapped patches according to the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Multimedia

IEEE Transactions on Multimedia Volume 25, Issue

2023

8932 pages

ISSN:1520-9210

Issue’s Table of Contents

1520-9210 © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 13 October 2022

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang YLiu SLi LZhou WLi H(2024)SwinShadow: Shifted Window for Ambiguous Adjacent Shadow DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/368880320:11(1-20)Online publication date: 27-Aug-2024
https://dl.acm.org/doi/10.1145/3688803

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents