[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content
Springer Nature Link
Log in

From Shadow Segmentation to Shadow Removal

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12356))

Included in the following conference series:

Abstract

The requirement for paired shadow and shadow-free images limits the size and diversity of shadow removal datasets and hinders the possibility of training large-scale, robust shadow removal algorithms. We propose a shadow removal method that can be trained using only shadow and non-shadow patches cropped from the shadow images themselves. Our method is trained via an adversarial framework, following a physical model of shadow formation. Our central contribution is a set of physics-based constraints that enables this adversarial training. Our method achieves competitive shadow removal results compared to state-of-the-art methods that are trained with fully paired shadow and shadow-free images. The advantages of our training regime are even more pronounced in shadow removal for videos. Our method can be fine-tuned on a testing video with only the shadow masks generated by a pre-trained shadow detector and outperforms state-of-the-art methods on this challenging test. We illustrate the advantages of our method on our proposed video shadow removal dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
GBP 19.95
Price includes VAT (United Kingdom)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
GBP 71.50
Price includes VAT (United Kingdom)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
GBP 89.99
Price includes VAT (United Kingdom)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    All code, trained models, and data are available at: .

References

  1. Arbel, E., Hel-Or, H.: Shadow removal using intensity surfaces and texture anchor points. IEEE Trans. Pattern Anal. Mach. Intell. 33, 1202–1216 (2011)

    Article  Google Scholar 

  2. Choi, Y., Choi, M.J., Kim, M., Ha, J.W., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8789–8797 (2017)

    Google Scholar 

  3. Chuang, Y.Y., Goldman, D.B., Curless, B., Salesin, D.H., Szeliski, R.: Shadow matting and compositing. ACM Trans. Graph. 22(3), 494–500 (2003). Special Issue of SIGGRAPH 2003 Proceeding

    Article  Google Scholar 

  4. Dare, P.: Shadow analysis in high-resolution satellite imagery of urban areas. Photogram. Eng. Remote Sens. 71, 169–177 (2005). https://doi.org/10.14358/PERS.71.2.169

    Article  Google Scholar 

  5. Ding, B., Long, C., Zhang, L., Xiao, C.: ARGAN: attentive recurrent generative adversarial network for shadow detection and removal. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10212–10221 (2019)

    Google Scholar 

  6. Drew, M.S.: Recovery of chromaticity image free from shadows via illumination invariance. In: IEEE Workshop on Color and Photometric Methods in Computer Vision, ICCV’03, pp. 32–39 (2003)

    Google Scholar 

  7. Finlayson, G., Drew, M.S.: 4-sensor camera calibration for image representation invariant to shading, shadows, lighting, and specularities. In: Proceedings of the International Conference on Computer Vision, vol. 2, pp. 473–480 (2001). https://doi.org/10.1109/ICCV.2001.937663

  8. Finlayson, G., Hordley, S., Lu, C., Drew, M.: On the removal of shadows from images. IEEE Trans. Pattern Anal. Mach. Intell. 28(1), 59–68 (2006)

    Article  Google Scholar 

  9. Finlayson, C., Hordley, S.D., Drew, M.S.: Removing shadows from images. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) Computer Vision – ECCV 2002. Lecture Notes in Computer Science, vol. 2353, pp. 823–836. Springer, Berlin, Heidelberg (2002). https://doi.org/10.1007/3-540-47979-1_55

    Chapter  Google Scholar 

  10. Fredembach, C., Finlayson, G.D.: Hamiltonian path based shadow removal. In: BMVC (2005)

    Google Scholar 

  11. Gong, H., Cosker, D.: Interactive removal and ground truth for difficult shadow scenes. J. Opt. Soc. Am. A 33(9), 1798–1811 (2016). https://doi.org/10.1364/JOSAA.33.001798. http://josaa.osa.org/abstract.cfm?URI=josaa-33-9-1798

  12. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein GANs. In: Advances in Neural Information Processing Systems, pp. 5767–5777 (2017)

    Google Scholar 

  13. Guo, R., Dai, Q., Hoiem, D.: Single-image shadow detection and removal using paired regions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2011)

    Google Scholar 

  14. Guo, R., Dai, Q., Hoiem, D.: Paired regions for shadow detection and removal. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2956–2967 (2012)

    Article  Google Scholar 

  15. Hu, X., Jiang, Y., Fu, C.W., Heng, P.A.: Mask-ShadowGAN: learning to remove shadows from unpaired data. In: ICCV (2019)

    Google Scholar 

  16. Hu, X., Wang, T., Fu, C.W., Jiang, Y., Wang, Q., Heng, P.A.: Revisiting shadow detection: a new benchmark dataset for complex world. arXiv:abs/1911.06998 (2019)

  17. Hu, X., Zhu, L., Fu, C.W., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

    Google Scholar 

  18. KaewTrakulPong, P., Bowden, R.: An improved adaptive background mixture model for real- time tracking with shadow detection (2002)

    Google Scholar 

  19. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the International Conference on Learning Representations (2015)

    Google Scholar 

  20. Le, H., Goncalves, B., Samaras, D., Lynch, H.: Weakly labeling the antarctic: the penguin colony case. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)

    Google Scholar 

  21. Le, H., Nguyen, V., Yu, C.P., Samaras, D.: Geodesic distance histogram feature for video segmentation. In: ACCV (2016)

    Google Scholar 

  22. Le, H., Samaras, D.: Shadow removal via shadow image decomposition. In: Proceedings of the International Conference on Computer Vision (2019)

    Google Scholar 

  23. Le, H., Vicente, T.F.Y., Nguyen, V., Hoai, M., Samaras, D.: A+D Net: training a shadow detector with adversarial shadow attenuation. In: Proceedings of the European Conference on Computer Vision (2018)

    Google Scholar 

  24. Le, H., Yu, C.P., Zelinsky, G., Samaras, D.: Co-localization with category-consistent features and geodesic distance propagation. In: ICCV 2017 Workshop on CEFRL: Compact and Efficient Feature Representation and Learning in Computer Vision (2017)

    Google Scholar 

  25. Li, Y., Tang, S., Zhang, R., Zhang, Y., Li, J., Yan, S.: Asymmetric gan for unpaired image-to-image translation. IEEE Trans. Image Process. 28, 5881–5896 (2019)

    Article  MathSciNet  Google Scholar 

  26. Liu, F., Gleicher, M.: Texture-consistent shadow removal. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) Computer Vision – ECCV 2008. Lecture Notes in Computer Science, vol. 5305, pp. 437–450. Springer, Berlin, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88693-8_32

    Chapter  Google Scholar 

  27. Liu, H., Gu, X., Samaras, D.: Wasserstein GAN with quadratic transport cost. In: The IEEE International Conference on Computer Vision (ICCV) (2019)

    Google Scholar 

  28. Liu, H., Xianfeng, G., Samaras, D.: A two-step computation of the exact GAN Wasserstein distance. In: International Conference on Machine Learning, pp. 3165–3174 (2018)

    Google Scholar 

  29. Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. arXiv:abs/1703.00848 (2017)

  30. Mescheder, L., Nowozin, S., Geiger, A.: Which training methods for GANs do actually converge? In: International Conference on Machine Learning (2018)

    Google Scholar 

  31. Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. In: International Conference on Machine Learning (2018)

    Google Scholar 

  32. Müller, T., Erdnüeß, B.: Brightness correction and shadow removal for video change detection with UAVs. In: Defense + Commercial Sensing (2019)

    Google Scholar 

  33. Panagopoulos, A., Wang, C., Samaras, D., Paragios, N.: Estimating shadows with the bright channel cue. In: Kutulakos, K.N. (ed.) Trends and Topics in Computer Vision – ECCV 2010. Lecture Notes in Computer Science, vol. 6554, pp. 1–12. Springer, Berlin, Heidelberg (2010). https://doi.org/10.1007/978-3-642-35740-4_1

    Chapter  Google Scholar 

  34. Panagopoulos, A., Wang, C., Samaras, D., Paragios, N.: Simultaneous cast shadows, illumination and geometry inference using hypergraphs. IEEE Trans. Pattern Anal. Mach. Intell. 35(2), 437–449 (2013). https://doi.org/10.1109/TPAMI.2012.110

    Article  Google Scholar 

  35. Porter, T., Duff, T.: Compositing digital images. Proc. ACM SIGGRAPH Conf. Comput. Graph. 18(3), 1–12 (1984)

    Article  Google Scholar 

  36. Prati, A., Mikic, I., Trivedi, M.M., Cucchiara, R.: Detecting moving shadows: algorithms and evaluation. IEEE Trans. Pattern Anal. Mach. Intell. 25, 918–923 (2003)

    Article  Google Scholar 

  37. Qu, L., Tian, J., He, S., Tang, Y., Lau, R.W.H.: DeshadowNet: a multi-context embedding deep network for shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)

    Google Scholar 

  38. Shiting, W., Hong, Z.: Clustering-based shadow edge detection in a single color image. In: International Conference on Mechatronic Sciences, Electric Engineering and Computer, pp. 1038–1041 (2013). https://doi.org/10.1109/MEC.2013.6885215

  39. Shor, Y., Lischinski, D.: The shadow meets the mask: pyramid-based shadow removal. Comput. Graph. Forum 27(2), 577–586 (2008)

    Article  Google Scholar 

  40. Smith, A.R., Blinn, J.F.: Blue screen matting. In: Proceedings of the ACM SIGGRAPH Conference on Computer Graphics (1996)

    Google Scholar 

  41. Su, N., Zhang, Y., Tian, S., Yan, Y., Miao, X.: Shadow detection and removal for occluded object information recovery in urban high-resolution panchromatic satellite images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 9, 2568–2582 (2016)

    Article  Google Scholar 

  42. Thanh-Tung, H., Tran, T., Venkatesh, S.: Improving generalization and stability of generative adversarial networks. In: International Conference on Learning Representations (2019)

    Google Scholar 

  43. Vicente, T.F.Y., Hoai, M., Samaras, D.: Noisy label recovery for shadow detection in unfamiliar domains. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)

    Google Scholar 

  44. Vicente, T.F.Y., Hoai, M., Samaras, D.: Leave-one-out kernel optimization for shadow detection and removal. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(3), 682–695 (2018)

    Article  Google Scholar 

  45. Vicente, T.F.Y., Hou, L., Yu, C.P., Hoai, M., Samaras, D.: Large-scale training of shadow detectors with noisily-annotated shadow examples. In: Proceedings of the European Conference on Computer Vision (2016)

    Google Scholar 

  46. Vicente, T.F.Y., Samaras, D.: Single image shadow removal via neighbor-based region relighting. In: Proceedings of the European Conference on Computer Vision Workshops (2014)

    Google Scholar 

  47. Wang, J., Li, X., Yang, J.: Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

    Google Scholar 

  48. Wang, T., Hu, X., Wang, Q., Heng, P.A., Fu, C.W.: Instance shadow detection. In: CVPR (2020)

    Google Scholar 

  49. Wright, S.: Digital Compositing for Film and Video. Focal Press (2001)

    Google Scholar 

  50. Wu, Q., Zhang, W.,Vijay Kumar, B.V.K.: Strong shadow removal via patch-based shadow edge detection. In: 2012 IEEE International Conference on Robotics and Automation, pp. 2177–2182 (2012)

    Google Scholar 

  51. Wu, T.P., Tang, C.K.: A Bayesian approach for shadow extraction from a single image. In: Tenth IEEE International Conference on Computer Vision (ICCV’05), vol. 1 1, pp. 480–487 (2005)

    Google Scholar 

  52. Wu, T.P., Tang, C.K., Brown, M.S., Shum, H.Y.: Natural shadow matting. ACM Trans. Graph. 26, 2 (2007). https://doi.org/10.1145/1243980.1243982. http://doi.acm.org/10.1145/1243980.1243982

  53. Yang, Q., Tan, K., Ahuja, N.: Shadow removal using bilateral filtering. IEEE Trans. Image Process. 21, 4361–4368 (2012)

    Article  MathSciNet  Google Scholar 

  54. Yi, Z., Zhang, H., Tan, P., Gong, M.: DualGAN: unsupervised dual learning for image-to-image translation. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2868–2876 (2017)

    Google Scholar 

  55. Zhang, L., Long, C., Zhang, X., Xiao, C.: RIS-GAN: explore residual and illumination with generative adversarial networks for shadow removal. In: AAAI Conference on Artificial Intelligence (AAAI) (2020)

    Google Scholar 

  56. Zhang, W., Zhao, X., Morvan, J.M., Chen, L.: Improving shadow suppression for illumination robust face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 41, 611–624 (2019)

    Article  Google Scholar 

  57. Zheng, Q., Qiao, X., Cao, Y., Lau, R.W.H.: Distraction-aware shadow detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5162–5171 (2019)

    Google Scholar 

  58. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV) (2017)

    Google Scholar 

  59. Zhu, L., et al.: Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In: Proceedings of the European Conference on Computer Vision (2018)

    Google Scholar 

Download references

Acknowledgements

This work was partially supported by the Partner University Fund, the SUNY2020 ITSC, and a gift from Adobe. Computational support provided by IACS and a GPU donation from NVIDIA. We thank Kumara Kahatapitiya and Cristina Mata for assistance with the manuscript.

Author information

Authors and Affiliations

  1. Stony Brook University, Stony Brook, NY, 11794, USA

    Hieu Le & Dimitris Samaras

Authors
  1. Hieu Le
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dimitris Samaras
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hieu Le .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 17621 KB)

Supplementary material 2 (mp4 19388 KB)

Supplementary material 3 (mp4 19313 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Le, H., Samaras, D. (2020). From Shadow Segmentation to Shadow Removal. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12356. Springer, Cham. https://doi.org/10.1007/978-3-030-58621-8_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58621-8_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58620-1

  • Online ISBN: 978-3-030-58621-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation