Abstract
An accurate model of building interiors with detailed annotations is critical to protecting first responders and building occupants during emergencies. First responders and building occupants can use these 3D building models to navigate indoor environments or vacate the building safely. In collaboration with the City of Memphis, we have collected extensive LiDAR and video data from seven buildings in Memphis. We apply machine learning techniques to the video frames to detect and classify objects of interest to first responders. We then utilize data fusion methods on the LiDAR and image data to create a comprehensive colored 3D indoor point cloud dataset with labeled safety-related objects. This paper documents the challenges we encountered in data collection and processing, and it presents a complete 3D mapping and labeling system for the environments inside and adjacent to buildings. Moreover, we used two of the scanned buildings as a case study to illustrate our process and show detailed evaluation results. Our results show that the deep neural network Mask R-CNN with transfer learning and hard-negative mining performs well in labeling public-safety objects in our image dataset, especially for large objects.
Supported by the financial assistance award 70NANB18H247 from U.S. Department of Commerce, National Institute of Standards and Technology.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ahrens, M., Evarts, B.: NFPA report: fire loss in the United States during 2019 (2020). https://www.nfpa.org/News-and-Research/Data-research-and-tools/US-Fire-Problem/Fire-loss-in-the-United-States
Andriluka, M., Uijlings, J.R., Ferrari, V.: Fluid annotation: a human-machine collaboration interface for full image annotation. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 1957–1966 (2018)
Armeni, I., et al.: 3D scene graph: a structure for unified semantics, 3D space, and camera. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5664–5673 (2019)
Armeni, I., Sax, S., Zamir, A.R., Savarese, S.: Joint 2D–3D-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01105 (2017)
Armeni, I., et al.: 3D semantic parsing of large-scale indoor spaces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1534–1543 (2016)
Bashiri, F.S., LaRose, E., Peissig, P., Tafti, A.P.: Mcindoor20000: a fully-labeled image dataset to advance indoor objects detection. Data Brief 17, 71–75 (2018). https://doi.org/10.1016/j.dib.2017.12.047. https://www.sciencedirect.com/science/article/pii/S2352340917307424
CDC/National Institute for Occupational Safety and Health (NIOSH) website: Fire fighter fatality investigation and prevention program (2020). https://www.cdc.gov/niosh/fire/default.html
Chang, A., et al.: ShapeNet: an information-rich 3D model repository. arXiv preprint arXiv:1512.03012 (2015)
CloudCompare (version 2.10.0) [GPL software] (2019). http://www.cloudcompare.org/
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848
ESRI: ArcGIS Pro (v2.5.0): Next-generation desktop GIS (2020). https://www.esri.com/en-us/arcgis/products/arcgis-pro/overview
Ester, M., Kriegel, H.P., Sander, J., Xu, X., et al.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: KDD, pp. 226–231 (1996)
Fahy, R.F., Petrillo, J.T., Molis, J.L.: NFPA report: firefighter fatalities in the US - 2019 (2020). https://www.nfpa.org/News-and-Research/Data-research-and-tools/Emergency-Responders/Firefighter-fatalities-in-the-United-States
GVI: GreenValley International LiBackpack 50 (2018). https://greenvalleyintl.com/hardware/libackpack/
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
Hossain, M., et al.: Building indoor point cloud datasets with object annotation for public safety. In: Proceedings of the 10th International Conference on Smart Cities and Green ICT Systems (SMARTGREENS 2021) (2021)
Huang, J., et al.: Speed/accuracy trade-offs for modern convolutional object detectors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7310–7311 (2017)
Insta360 Pro 2 Camera (2018). https://www.insta360.com/product/insta360-pro2
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660 (2017)
Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017). https://proceedings.neurips.cc/paper/2017/file/d8bf84be3800d12f74d8b05e9b89836f-Paper.pdf
Quigley, M., et al.: Ros: an open-source robot operating system. In: ICRA Workshop on Open Source Software, p. 5 (2009)
REACH RS+: Single-band RTK GNSS receiver with centimeter precision (2018). https://emlid.com/reachrs/
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2016)
Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823 (2015)
Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.: Inception-v4, Inception-ResNet and the impact of residual connections on learning. In: Proceedings of the AAAI Conference on Artificial Intelligence (2017)
Wada, K.: LabelMe: image polygonal annotation with python (2016). https://github.com/wkentaro/labelme
Yi, L., et al.: A scalable active framework for region annotation in 3D shape collections. ACM Trans. Graph. (ToG) 35(6), 1–12 (2016)
Zhou, Q.Y., Park, J., Koltun, V.: Open3D: a modern library for 3D data processing. arXiv preprint arXiv:1801.09847 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Hossain, M. et al. (2022). Building Rich Interior Hazard Maps for Public Safety. In: Klein, C., Jarke, M., Helfert, M., Berns, K., Gusikhin, O. (eds) Smart Cities, Green Technologies, and Intelligent Transport Systems. VEHITS SMARTGREENS 2021 2021. Communications in Computer and Information Science, vol 1612. Springer, Cham. https://doi.org/10.1007/978-3-031-17098-0_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-17098-0_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-17097-3
Online ISBN: 978-3-031-17098-0
eBook Packages: Computer ScienceComputer Science (R0)