More Web Proxy on the site http://driver.im/

research-article

Simulating Photo-realistic Snow and Fog on Existing Images for Enhanced CNN Training and Evaluation

Authors:

Alexander von Bernuth,

Oliver BringmannAuthors Info & Claims

2019 IEEE Intelligent Transportation Systems Conference (ITSC)

Pages 41 - 46

https://doi.org/10.1109/ITSC.2019.8917367

Published: 01 October 2019 Publication History

Abstract

Verification and robustness testing of machine learning algorithms for autonomous driving is crucial. Due to the increasing complexity and quantity of those systems in a single vehicle, just driving the required distance with a newly developed vehicle is not feasible anymore: billions of hours on the street without failure are necessary to qualify for industry standards like ISO 26262. That is where simulation comes into play: machine learning algorithms are trained and evaluated on well known image data sets like KITTI or Cityscapes. But today’s data sets mostly contain images taken under perfect weather conditions and therefore do not harden optical object detection algorithms against various weather conditions. This paper focuses on reusing these established and labeled data sets by augmenting them with adverse weather effects like snow and fog. Those effects are rendered physically correct and life like while being added to existing real world images. Thanks to easy parametrization the weather influences may be varied as necessary and allow for finely tuned learning and optimization processes. The weather effects are evaluated with regard to realism and impact on an established object detection algorithm. These newly created weather-influenced images may be used to validate or train new object detection algorithms.

References

[1]

A. J. Hawkins, “We spoke to a Waymo One customer about how robot taxis get confused by rainstorms,” https://www.theverge.com/2019/1/20/18175563/waymo-one-customer-interview-self-driving-arizona, Jan. 2019.

[2]

A. Weitzel, H. Winner, C. Peng, S. Geyer, F. Lotz, and M. Sefati, Absicherungsstrategien Fuer Fahrerassistenzsysteme Mit Umfeld-wahrnehmung. Fachverlag NW in der Carl Schuenemann Verlag GmbH, 2014, https://trid.trb.org/view/1339882.

[3]

H. Winner, “Quo vadis, FAS?” in Handbuch Fahrerassistenzsysteme, H. Winner, S. Hakuli, F. Lotz, and C. Singer, Eds. Wiesbaden: Springer Fachmedien Wiesbaden, 2015, pp. 1167–1186, http://link.springer.com/10.1007/978-3-658-05734-3_62.

[4]

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Be-nenson, U. Franke, S. Roth, and B. Schiele, “The Cityscapes Dataset for Semantic Urban Scene Understanding,” in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

[5]

A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the KITTI vision benchmark suite,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2012, pp. 3354–3361.

[6]

W. Maddern, G. Pascoe, C. Linegar, and P. Newman, “1 year, 1000 km: The Oxford RobotCar dataset,” The International Journal of Robotics Research, vol. 36, no. 1, pp. 3–15, Jan. 2017.

Digital Library

[7]

D. Hospach, S. Mueller, O. Bringmann, J. Gerlach, and W. Rosenstiel, “Simulation and evaluation of sensor characteristics in vision based advanced driver assistance systems,” in 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), 2014, pp. 2610–2615.

[8]

S. Mueller, D. Hospach, O. Bringmann, and W. Rosenstiel, “Framework for Varied Sensor Perception in Virtual Prototypes,” in Methoden Und Beschreibungssprachen Zur Modellierung Und Verifikation von Schaltungen Und Systemen (MBMV), 2015, pp. 145–154.

[9]

D. Hospach, S. Mueller, W. Rosenstiel, and O. Bringmann, “Simulation of Falling Rain for Robustness Testing of Video-Based Surround Sensing Systems,” in Proceedings of the 2016 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2016, pp. 233–236.

[10]

N. Wang and B. Wade, “Rendering falling rain and snow,” in ACM SIGGRAPH 2004 Sketches on - SIGGRAPH ’04, 2004, p. 14.

[11]

T. Zhuo and Libaicheng, “A Simulation System of Snow Based on Particle System,” in International Conference on Intelligent Systems Research and Mechatronics Engineering (ISRME 2015), 2015, pp. 740–743.

[12]

V. Aleshin, V. Afanasiev, A. Bobkov, S. Klimenko, V. Kuliev, and D. Novgorodtsev, “Visual 3D Perception of Motion Environment and Visibility Factors in Virtual Space,” in Transactions on Computational Science XVI, M. L. Gavrilova and C. J. K. Tan, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, vol. 7380, pp. 17–33, http://link.springer.com/10.1007/978-3-642-32663-9_2.

[13]

H. Y. Lv and F. Liu, “Real-Times Snowfall Simulation Based on Particle System and Pulverization,” in Applied Mechanics and Materials, vol. 373-375, 2013, pp. 1168–1171.

[14]

I. Saltvik, A. C. Elster, and H. R. Nagel, “Parallel methods for real-time visualization of snow,” Ph.D. dissertation, 2006, http://dl.acm.org/citation.cfm?id=1775059.1775091.

[15]

K. Muraoka and N. Chiba, “Visual simulation of snowfall, snow cover and snowmelt,” in Proceedings - 7th International Conference on Parallel and Distributed Systems: Workshops, 2000, pp. 187–194.

[16]

C. Wang, Z. Wang, T. Xia, and Q. Peng, “Real-time snowing simulation,” Visual Computer, vol. 22, no. 5, pp. 315–323, 2006.

Digital Library

[17]

N. V. Festenberg and S. Gumhold, “A geometric algorithm for snow distribution in virtual scenes,” Natural Phenomena, pp. 17–25, 2009.

[18]

M. S. Langer and L. Zhang, “Rendering falling snow using an inverse Fourier transform,” ACM SIGGRAPH technical sketches program, 2003.

[19]

M. Jaeger, “Enhancing virtual natural scenes using quick and dirty image based recipes,” Proceedings - 2012 IEEE 4th International Symposium on Plant Growth Modeling, Simulation, Visualization and Applications, PMA 2012, pp. 164–171, 2012.

[20]

G. Sellers, R. S. Wright, Jr., and N. Haemel, “Atmospheric Effects,” in OpenGL SuperBible: Comprehensive Tutorial and Reference, Seventh Edition, 7th ed. Addison-Wesley Professional, July 2015.

[21]

C. Sakaridis, D. Dai, and L. Van Gool, “Semantic Foggy Scene Understanding with Synthetic Data,” International Journal of Computer Vision, vol. 126, no. 9, pp. 973–992, Sept. 2018.

Digital Library

[22]

D. Zdrojewska, “Real time rendering of heterogenous fog based on the graphics hardware acceleration,” Proceedings of CESCG, vol. 4, pp. 95–101, 2004.

[23]

V. Biri and S. Michelin, “Real Time Animation of Realistic Fog,” in Poster Session of 13th Eurographic Workshop on Rendering, June 2002, p. 9.

[24]

S. Raudsepp, “Volumetric Fog Rendering,” Bachelor Thesis, University of Tartu, Tartu, 2018.

[25]

R. V. Klassen, “Modeling the effect of the atmosphere on light,” ACM Transactions on Graphics, vol. 6, no. 3, pp. 215–237, July 1987.

Digital Library

[26]

E. Dumont, “Semi-Monte Carlo Light Tracing Applied to the Study of Road Visibility in Fog,” in Monte-Carlo and Quasi-Monte Carlo Methods 1998, H. Niederreiter and J. Spanier, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2000, pp. 177–187, http://link.springer.com/10.1007/978-3-642-59657-5_11.

[27]

H. W. Jensen and P. H. Christensen, “Efficient simulation of light transport in scenes with participating media using photon maps,” in Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques. ACM, July 1998, pp. 311–320.

[28]

M. Colomb, K. Hirech, P. André, J. Boreux, P. Lacôte, and J. Dufour, “An innovative artificial fog production device improved in the Eu-ropean project "FOG",” Atmospheric Research, vol. 87, no. 3-4, pp. 242–251, Mar. 2008.

[29]

M. Dupuis and W. Karl, “VTD - VIRES Virtual Test Drive,” https://vires.com/vtd-vires-virtual-test-drive/, 2017.

[30]

A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An Open Urban Driving Simulator,” in Proceedings of the 1st Annual Conference on Robot Learning, 2017, pp. 1–16.

[31]

G. Koh, J. Lacombe, and D. L. Hutt, “Snow mass concentration and precipitation rate,” Cold Regions Science and Technology, vol. 15, no. 1, pp. 89–92, Feb. 1988.

[32]

M. P. Langleben, “Terminal Velocity of Snowflakes,” no. January, 1954.

[33]

K. L. S. Gunn and J. S. Marshall, “The Distribution With Size of Aggregate Snowflakes,” Journal of Meteorology, vol. 15, no. 5, pp. 452–461, 1958.

[34]

R. S. Sekhon and R. C. Srivastava, “Snow Size Spectra and Radar Reflectivity,” Journal of the Atmospheric Sciences, vol. 27, pp. 299–307, 1970.

[35]

C. F. Bohren and D. R. Huffman, Absorption and Scattering of Light by Small Particles. Weinheim: Wiley-VCH, 2004, oCLC: 254937169.

[36]

R. M. Rasmussen, J. Vivekanandan, J. Cole, B. Myers, and C. Masters, “The Estimation of Snowfall Rate Using Visibility,” Journal of Applied Meteorology, vol. 38, no. 10, pp. 1542–1563, Oct. 1999.

[37]

S. T. Surjikov, “Mie scattering,” Feb. 2011.

[38]

J. Pomeroy and D. Male, “Optical Properties of Blowing Snow,” Journal of Glaciology, vol. 34, no. 116, pp. 3–10, 1988.

[39]

L. Wang and S. L. Jacques, “Sphere Mie Scattering Program,” University of Texas, Texas, Sept. 1995, https://omlc.org/software/mie/miesphr.c.

[40]

J. S. J. Ren, X. Chen, J. Liu, W. Sun, J. Pang, Q. Yan, Y.-W. Tai, and L. Xu, “Accurate Single Stage Detector Using Recurrent Rolling Convolution,” CoRR, vol. abs/1704.0, 2017, http://arxiv.org/abs/1704.05776.

Cited By

Khosravian AAmirkhani AKashiani HMasih-Tehrani M(2022)Generalizing state-of-the-art object detectors for autonomous vehicles in unseen environmentsExpert Systems with Applications: An International Journal10.1016/j.eswa.2021.115417183:COnline publication date: 3-Jan-2022
https://dl.acm.org/doi/10.1016/j.eswa.2021.115417

Index Terms

Simulating Photo-realistic Snow and Fog on Existing Images for Enhanced CNN Training and Evaluation
1. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

Enhanced aerial vehicle system techniques for detection and tracking in fog, sandstorm, and snow conditions
Abstract
Unmanned aerial vehicles are rapidly being utilized in surveillance and traffic monitoring because of their great mobility and capacity to cover regions at various elevations and positions. It is a challenging task to detect vehicles due to their ...
Modeling and evaluation of cooperative Vulnerable Road User protection schemes in realistic C-ITS environments
Abstract
Vulnerable Road Users (VRUs) comprising pedestrians, cyclists, and motorcyclists are more susceptible to road traffic accidents and fatalities than other road users. Improving safety for vulnerable road users is a priority for many cities and ...
Comparative Evaluation Study of GLOSA Approaches Under Realistic Scenario Conditions
Ad-Hoc, Mobile, and Wireless Networks
Abstract
Cooperative Intelligent Transport Systems (C-ITS) are a promising solution to enhance road management, traffic efficiency, fuel consumption and road safety. Green Light Optimal Speed Advisory (GLOSA) is one of the traffic efficiency ITS services ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

2019 IEEE Intelligent Transportation Systems Conference (ITSC)

October 2019

4550 pages

Copyright © 2019.

Publisher

IEEE Press

Publication History

Published: 01 October 2019

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Khosravian AAmirkhani AKashiani HMasih-Tehrani M(2022)Generalizing state-of-the-art object detectors for autonomous vehicles in unseen environmentsExpert Systems with Applications: An International Journal10.1016/j.eswa.2021.115417183:COnline publication date: 3-Jan-2022
https://dl.acm.org/doi/10.1016/j.eswa.2021.115417

View Options

View options

Media

Figures

Other

Tables

View Table of Contents