Distill Drops into Data: Event-based Rain-Background Decomposition Network
Authors: 
Ciyu Ruan, Chenyu Zhao, Chenxin Liang, Xinyu Luo, Jingao Xu, Xinlei ChenAuthors Info & Claims
Pages 2072 - 2077
Abstract
Event cameras excel in high-speed and high-dynamic-range scenarios but are highly sensitive to rain, which introduces significant noise while also revealing detailed rain features. This paper introduces a novel Event-based Rain-Background Decomposition Network that integrates Spiking Neural Networks (SNNs) and Convolutional Neural Networks (CNNs). By "Distilling Rain," we reconstruct a rain-free background for downstream tasks, and by "Collecting Rain," we extract the physical characteristics of rain. Experimental evaluations demonstrate the network's effectiveness in both background reconstruction and rain modeling. This work extends the capabilities of event cameras by mitigating the adverse effects of rain while also leveraging rain-induced noise to extract valuable environmental data, enhancing their utility in both challenging weather conditions and detailed environmental analysis.
References
[1]
Ao Wang, Hui Chen, Lihao Liu, Kai Chen, Zijia Lin, Jungong Han, and Guiguang Ding. Yolov10: Real-time end-to-end object detection. arXiv preprint arXiv:2405.14458, 2024.
[2]
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Processings of IEEE CVPR, pages 3354--3361, 2012.
[3]
Daniel Gehrig, Mathias Gehrig, Javier Hidalgo-Carrió, and Davide Scaramuzza. Video to events: Recycling video datasets for event cameras. In Processings of IEEE CVPR, June 2020.
[4]
Maxime Tremblay, Shirsendu S. Halder, Raoul de Charette, and Jean-François Lalonde. Rain rendering for evaluating and improving robustness to bad weather. International Journal of Computer Vision, 2020.
[5]
Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, et al. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1):154--180, 2020.
[6]
Hao Cao, Jingao Xu, Danyang Li, Zheng Yang, and Yunhao Liu. Eventboost: Event-based acceleration platform for real-time drone localization and tracking. In IEEE INFOCOM, pages 1851--1859. IEEE, 2024.
[7]
Xinlei Chen, Aveek Purohit, Carlos Ruiz Dominguez, Stefano Carpin, and Pei Zhang. Drunkwalk: Collaborative and adaptive planning for navigation of micro-aerial sensor swarms. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems, pages 295--308, 2015.
[8]
Haoyang Wang, Xuecheng Chen, Yuhan Cheng, Chenye Wu, Fan Dang, and Xinlei Chen. H-swarmloc: efficient scheduling for localization of heterogeneous mav swarm with deep reinforcement learning. In Proceedings of the ACM SenSys, pages 1148--1154, 2022.
[9]
Guang Chen, Peigen Liu, Zhengfa Liu, Huajin Tang, Lin Hong, Jinhu Dong, Jörg Conradt, and Alois Knoll. Neuroaed: Towards efficient abnormal event detection in visual surveillance with neuromorphic vision sensor. IEEE Transactions on Information Forensics and Security, 16:923--936, 2020.
[10]
Jingao Xu, Guoxuan Chi, Zheng Yang, Danyang Li, Qian Zhang, Qiang Ma, and Xin Miao. Followupar: Enabling follow-up effects in mobile ar applications. In Proceedings of the ACM MobiSys, pages 1--13, 2021.
[11]
Xuecheng Chen, Haoyang Wang, Yuhan Cheng, Haohao Fu, Yuxuan Liu, Fan Dang, Yunhao Liu, Jinqiang Cui, and Xinlei Chen. Ddl: Empowering delivery drones with large-scale urban sensing capability. IEEE Journal of Selected Topics in Signal Processing, 2024.
[12]
Jiyuan Ren, Yanggang Xu, Zuxin Li, Chaopeng Hong, Xiao-Ping Zhang, and Xinlei Chen. Scheduling uav swarm with attention-based graph reinforcement learning for ground-to-air heterogeneous data communication. In Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing, pages 670--675, 2023.
[13]
Jingao Xu, Danyang Li, Zheng Yang, Yishujie Zhao, Hao Cao, Yunhao Liu, and Longfei Shangguan. Taming event cameras with bio-inspired architecture and algorithm: A case for drone obstacle avoidance. In Proceedings of the ACM MobiCom, pages 1--16, 2023.
[14]
Haoyang Wang, Jingao Xu, Chenyu Zhao, Zihong Lu, Yuhan Cheng, Xuecheng Chen, Xiao-Ping Zhang, Yunhao Liu, and Xinlei Chen. Transformloc: Transforming mavs into mobile localization infrastructures in heterogeneous swarms. In IEEE INFOCOM, pages 1101--1110. IEEE, 2024.
[15]
Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. Unsupervised event-based learning of optical flow, depth, and egomotion. In Proceedings of the IEEE CVPR, pages 989--997, 2019.
[16]
Haoyang Wang, Yuxuan Liu, Chenyu Zhao, Jiayou He, Wenbo Ding, and Xinlei Chen. Califormer: Leveraging unlabeled measurements to calibrate sensors with self-supervised learning. In Proceedings of the UbiComp, pages 743--748, 2023.
[17]
Xinlei Chen, Susu Xu, Haohao Fu, Carlee Joe-Wong, Lin Zhang, Hae Young Noh, and Pei Zhang. Asc: Actuation system for city-wide crowdsensing with ride-sharing vehicular platform. In Proceedings of the Fourth Workshop on International Science of Smart City Operations and Platforms Engineering, pages 19--24, 2019.
[18]
Zuxin Li, Fanhang Man, Xuecheng Chen, Baining Zhao, Chenye Wu, and Xinlei Chen. Tract: Towards large-scale crowdsensing with high-efficiency swarm path planning. In Adjunct Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2022 ACM International Symposium on Wearable Computers, pages 409--414, 2022.
[19]
Xuecheng Chen, Zijian Xiao, Yuhan Cheng, ChenChun Hsia, Haoyang Wang, Jingao Xu, Susu Xu, Fan Dang, Xiao-Ping Zhang, Yunhao Liu, et al. Soscheduler: Toward proactive and adaptive wildfire suppression via multiuav collaborative scheduling. IEEE Internet of Things Journal, 2024.
[20]
Shangquan Sun, Wenqi Ren, Jingzhi Li, Kaihao Zhang, Meiyu Liang, and Xiaochun Cao. Event-Aware Video Deraining via Multi-Patch Progressive Learning. IEEE Transactions on Image Processing, 32:3040--3053, 2023.
[21]
Jin Wang, Wenming Weng, Yueyi Zhang, and Zhiwei Xiong. Unsupervised Video Deraining with An Event Camera. In Proceedings of the IEEE CVPR, pages 10831--10840, 2023.
[22]
Haoyang Wang, Xinyu Luo, Ciyu Ruan, Xuecheng Chen, Wenhua Ding, Yuxuan Liu, and Xinlei Chen. Poster: Fusing event and depth sensing for dynamic objects localization and tracking. In Proceedings of the IEEE IPSN, pages 141--141, 2024.
[23]
Junyu Yang, Yan Dong, Zhennan Long, Xin Yang, and Bin Han. Rain detection algorithm based on event camera. Journal of Computer Applications, 43(9):2904, September 2023.
[24]
Gopinath Kathiravelu, Terry Lucke, and Peter Nichols. Rain Drop Measurement Techniques: A Review. Water, 8(1):29, January 2016.
[25]
Rong Dong, Juan Liao, Bo Li, Huiyu Zhou, and Danny Crookes. Measurements of rainfall rates from videos. In Processings of IEEE CISP-BMEI, pages 1--9, 2017.
[26]
Yang Hong, Kaixuan Wei, Linwei Chen, and Ying Fu. Crafting object detection in very low light. In BMVC, volume 1, pages 1--15, 2021.
[27]
Mohd Farid Mohd Ariff, Mohammad Ehsan Kosnan, Zulkepli Majid, Albert K Chong, and Khairulnizam M Idris. A study of near-infrared (nir) filter for surveillance application. Jurnal Teknologi, 77(26), 2015.
[28]
Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck. A 240× 180 130 db 3 μs latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, 49(10):2333--2341, 2014.
[29]
Xuecheng Chen, Haoyang Wang, Zuxin Li, Wenbo Ding, Fan Dang, Chengye Wu, and Xinlei Chen. Deliversense: Efficient delivery drone scheduling for crowdsensing with deep reinforcement learning. In Proceedings of the UbiComp, pages 403--408, 2022.
[30]
Xinlei Chen, Aveek Purohit, Shijia Pan, Carlos Ruiz, Jun Han, Zheng Sun, Frank Mokaya, Patric Tague, and Pei Zhang. Design experiences in minimalistic flying sensor node platform through sensorfly. ACM TOSN, 13(4):1--37, 2017.
[31]
Zhuozhu Jian, Zejia Liu, Haoyu Shao, Xueqian Wang, Xinlei Chen, and Bin Liang. Path generation for wheeled robots autonomous navigation on vegetated terrain. IEEE Robotics and Automation Letters, 2023.
[32]
Yuxuan Liu, Haoyang Wang, Fanhang Man, Jingao Xu, Fan Dang, Yunhao Liu, Xiao-Ping Zhang, and Xinlei Chen. Mobiair: Unleashing sensor mobility for city-scale and fine-grained air-quality monitoring with airbert. In Proceedings of MobiSys, pages 223--236, 2024.
[33]
Xinlei Chen, Susu Xu, Jun Han, Haohao Fu, Xidong Pi, Carlee Joe-Wong, Yong Li, Lin Zhang, Hae Young Noh, and Pei Zhang. Pas: Prediction-based actuation system for city-scale ridesharing vehicular mobile crowdsensing. IEEE Internet of Things Journal, 7(5):3719--3734, 2020.
[34]
Xiang Zhang, Wei Liao, Lei Yu, Wen Yang, and Gui-Song Xia. Event-based synthetic aperture imaging with a hybrid network. In Proceedings of the IEEE CVPR, pages 14235--14244, 2021.
[35]
Priyadarshini Panda, Sai Aparna Aketi, and Kaushik Roy. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience, 14:653, 2020.
[36]
Ana I Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso García, and Davide Scaramuzza. Event-based vision meets deep learning on steering prediction for self-driving cars. In Proceedings of the IEEE CVPR, pages 5419--5427, 2018.
[37]
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Processings of NeurIPS, 28, 2015.
[38]
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI, pages 1311--1318, 2019.
[39]
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In IEEE CVPR, pages 14821--14831, 2021.
[40]
Ugur Demir and Gozde Unal. Patch-based image inpainting with generative adversarial networks. arXiv preprint arXiv:1803.07422, 2018.
[41]
Sahil Yadav, Aryan Mehra, Honnesh Rohmetra, Rahul Ratnakumar, and Pratik Narang. DerainGAN: Single image deraining using wasserstein GAN. Multimedia Tools and Applications, 80(30):36491--36507, December 2021.
[42]
Long Cheng, Ni Liu, Xusen Guo, Yuhao Shen, Zijun Meng, Kai Huang, and Xiaoqin Zhang. A Novel Rain Removal Approach for Outdoor Dynamic Vision Sensor Event Videos. Frontiers in Neurorobotics, 16, August 2022.
[43]
Christian E. Willert. Event-based imaging velocimetry using pulsed illumination. Experiments in Fluids, 64(5):98, May 2023.
Index Terms
- Distill Drops into Data: Event-based Rain-Background Decomposition Network
Recommendations
Methodology to simulate rain and attenuation fields spatially correlated at mid and large scale
Propagation Modelling for Space Radio Frequency LinksThis paper is related to the modelling of rain fields and attenuation fields at small (size of a rain cell), mid (∼150 km 2) and large scale (∼1000 km 2). The methodology lies on a cellular description of rain fields by the HYCELL model. The latter ...
Importance of satellite-retrieved rain/no-rain information on short-range weather predictions
The impact of assimilating rain satellite-retrieved rainfall is greater than zero and no-rain satellite-retrieved rainfall is equal to zero information retrieved from the Tropical Rainfall Measuring Mission TRMM 3B42 precipitation is assessed during ...
Event-Driven Heterogeneous Network for Video Deraining
AbstractRestoring clear frames from rainy videos poses a significant challenge due to the swift motion of rain streaks. Traditional frame-based visual sensors, which record dense scene content synchronously, struggle to capture the fast-moving rain ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2024
2476 pages
ISBN:9798400704895
DOI:10.1145/3636534
- Chair:
- Weisong Shi,
- Program Chairs:
- Deepak Ganesan,
- Nicholas Lane
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 December 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- the National Key R&D program of China
- Natural Science Foundation of China
- Guangdong Innovative and Entrepreneurial Research Team Program
- Shenzhen 2022 Stabilization Support Program
Conference
ACM MobiCom '24
Sponsor:
ACM MobiCom '24: 30th Annual International Conference on Mobile Computing and Networking
November 18 - 22, 2024
DC, Washington D.C., USA
Acceptance Rates
Overall Acceptance Rate 440 of 2,972 submissions, 15%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 83Total Downloads
- Downloads (Last 12 months)83
- Downloads (Last 6 weeks)83
Reflects downloads up to 01 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in