Abstract
Robots that perform chemical experiments autonomously have been implemented, using the same chemical apparatus as human chemists and capable of performing complex chemical experiments unmanaged. However, most robots in chemistry are still programmed and cannot adapt to diverse environments or to changes in displacement and angle of the object. To resolve this issue, we have conceived a computer vision method for identifying and detecting chemical apparatus automatically. Identifying and localizing such apparatus accurately from chemistry lab images is the most important task. We acquired 2246 images from real chemistry laboratories, with a total of 33,108 apparatus instances containing 21 classes. We demonstrate a Chemical Apparatus Benchmark Dataset (CABD) containing a chemical apparatus image recognition dataset and a chemical apparatus object detection dataset. We evaluated five excellent image recognition models: AlexNet, VGG16, GoogLeNet, ResNet50, MobileNetV2 and four state-of-the-art object detection methods: Faster R-CNN (3 backbones), Single Shot MultiBox Detector (SSD), YOLOv3-SPP and YOLOv5, respectively, on the CABD dataset. The results can serve as a baseline for future research. Experiments show that ResNet50 has the highest accuracy (99.9%) in the chemical apparatus image recognition dataset; Faster R-CNN (ResNet50-fpn) and YOLOv5 performed the best in terms of mAP (99.0%) and AR (94.5%) in the chemical apparatus object detection dataset.
Similar content being viewed by others
Data availability
The data that support the findings of this study are openly available in Zenodo,reference number [49].
Notes
An earlier version of this paper was presented at the International conference on Artificial Intelligence and Big Data in Digital Era.
References
Ding Z, Ran S, Wu Z et al (2022) A new benchmark data set for Chemical laboratory apparatus detection[C]. Artificial Intelligence in Data and Big Data Processing: Proceedings of ICABDE, pp 201–210
Kelley EW (2021) Sample plan for easy, inexpensive, safe, and relevant hands-on, aat-home wet organic Chemistry laboratory activities[J]. J Chem Educ 98(5):1622–1635
Willey RJ, Carter T, Price J et al (2020) Instruction of hazard analysis of methods for chemical process safety at the university level[J]. J Loss Prev Process Ind 63:103961
Christensen H, Amato N, Yanco H et al (2021) A roadmap for us robotics–from internet to robotics 2020 edition[J]. Foundations and Trends®. Robotics 8(4):307–424
Althoff M, Giusti A, Liu SB et al (2019) Effortless creation of safe robots from modules through self-programming and self-verification[J]. Science. Robotics 4(31):eaaw1924
Zoph B, Vasudevan V, Shlens J, et al. (2018) Learning transferable architectures for scalable image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697–8710
Liu L, Ouyang W, Wang X (2020) Deep learning for generic object detection: a survey)[J]. Int J Comput Vis 128(2):261–318
Kumar A, Zhou Y, Gandhi CP et al (2020) Bearing defect size assessment using wavelet transform based Deep Convolutional Neural Network (DCNN)[J]. Alexandria Eng J 59(2):999–1012
Frizzi S, Kaabi R, Bouchouicha M et al (2016) Convolutional neural network for video fire and smoke detection[C]//IECON 2016-42nd Annual Conference of the IEEE Industrial Electronics Society. IEEE:877–882
Knickmeyer D (2020) Social factors influencing household waste separation: A literature review on good practices to improve the recycling performance of urban areas[J]. J Clean Prod 245:118605
Cao W, Liu Q, He Z (2020) Review of pavement defect detection methods[J]. IEEE Access 8:14531–14544
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks[J]. Adv Neural Inf Proces Syst 25
Simonyan K, Zisserman A. (2014) Very deep convolutional networks for large-scale image recognition[J]. ArXiv Preprint ArXiv:1409.1556
Khan RU, Zhang X, Kumar R (2019) Analysis of ResNet and GoogleNet models for malware detection[J]. J Comput Virolog Hacking Techniq 15(1):29–37
Mukti IZ, Biswas D (2019) Transfer learning based plant diseases detection using ResNet50[C]//2019 4th International Conference on Electrical Information and Communication Technology (EICT). IEEE:1–6
Sandler M, Howard A, Zhu M et al (2018) Mobilenetv2: Inverted residuals and linear bottlenecks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4510–4520
Ren S, He K, Girshick R (2015) Faster R-cnn: towards real-time object detection with region proposal networks[J]. Adv Neural Inf Proces Syst 28:91–99
Huang Z, Wang J, Fu X (2020) DC-SPP-YOLO: Dense connection and spatial pyramid pooling based YOLO for object detection[J]. Inf Sci 522:241–258
Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg A C (2016) SSD: Single shot multibox detector[C]. In: European Conference on Computer Vision. Springer, Cham, pp 21–37
Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation[C]. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 580–587
Wang H, Yuan H, Hong SS et al (2015) Physical and chemical tuning of two-dimensional transition metal dichalcogenides[J]. Chem Soc Rev 44(9):2664–2680
Zanchettin AM, Ceriani NM, Rocco P et al (2015) Safety in human-robot collaborative manufacturing environments: Metrics and control[J]. IEEE Trans Autom Sci Eng 13(2):882–893
Henson AB, Gromski PS, Cronin L (2018) Designing algorithms to aid discovery by chemical robots[J]. ACS Centr Sci 4(7):793–804
Zhavoronkov A (2018) Artificial intelligence for drug discovery, biomarker development, and generation of novel chemistry[J]. Mol Pharm 15(10):4311–4313
Burger B, Maffettone PM, Gusev VV et al (2020) A mobile robotic chemist[J]. Nature 583(7815):237–241
Patrício DI, Rieder R (2018) Computer vision and artificial intelligence in precision agriculture for grain crops: A systematic review[J]. Comput Electron Agric 153:69–81
Chai J, Li A (2019) Deep learning in natural language processing: A state-of-the-art survey[C]//2019 International Conference on Machine Learning and Cybernetics (ICMLC). IEEE:1–6
Vrancken C, Longhurst P, Wagland S (2019) Deep learning in material recovery: Development of method to create training database[J]. Expert Syst Appl 125:268–280
Jadon A, Omama M, Varshney A, et al. (2019) FireNet: a specialized lightweight fire & smoke detection model for real-time IoT applications[J]. arXiv preprint arXiv:1905.11922
Nguyen A, Yosinski J, Clune J (2015) Deep neural networks are easily fooled: High confidence predictions for unrecognizable images[C]. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 427–436
Savadjiev P, Chong J, Dohan A et al (2019) Demystification of AI-driven medical image interpretation: past, present and future[J]. Eur Radiol 29(3):1616–1624
Fujiyoshi H, Hirakawa T, Yamashita T (2019) Deep learning-based image recognition for autonomous driving[J]. IATSS Res 43(4):244–242
Parmar DN, Mehta BB (2014) Face recognition methods & applications[J]. arXiv preprint arXiv:1403.0485
Pan X, Shi J, Luo P et al (2018) Spatial as deep: Spatial CNN for traffic scene understanding[C]. Proceedings of the AAAI Conference on Artificial Intelligence 32(1):7276–7283
Lu ZM, Li SZ, Burkhardt H (2006) A content-based image retrieval scheme in JPEG compressed domain[J]. Int J Innov Comput, Inform Contr 2(4):831–839
Li LF, Wang X, Hu WJ et al (2020) Deep learning in skin disease image recognition: A review[J]. IEEE Access 8:208264–208280
Ma N, Zhang X, Zheng HT et al (2018) Shufflenet v2: Practical guidelines for efficient CNN architecture design[C]. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 116–131
Tan M, Le Q (2021) Efficientnetv2: Smaller models and faster training[C]//International Conference on Machine Learning. PMLR:10096–10106
Hossain S, Lee D (2019) Deep learning-based real-time multiple-object detection and tracking from aerial imagery via a flying robot with GPU-based embedded devices[J]. Sensors 19(15):3371
Du L, Zhang R, Wang X (2020) Overview of two-stage object detection algorithms[C]//Journal of Physics: Conference Series. IOP Publishing 1544(1):012033
Girshick R, Donahue J, Darrell T, et al. (2014) Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580–587
Girshick R (2015) Fast R-cnn[C]. In: Proceedings of the IEEE International Conference on Computer Vision, pp 1440–1448
He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-cnn[C]. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2961–2969
Lin TY, Goyal P, Girshick R (2017) Focal loss for dense object detection[C]. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2980–2988
Batth RS, Nayyar A, Nagpal A (2018) Internet of robotic things: driving intelligent robotics of future-concept, architecture, applications and technologies[C]//2018 4th International Conference on Computing Sciences (ICCS). IEEE:151–160
Wan S, Gu Z, Ni Q (2020) Cognitive computing and wireless communications on the edge for healthcare service robots[J]. Comput Commun 149:99–106
Rostianingsih S, Setiawan A, Halim CI (2020) COCO (creating common object in context) dataset for chemistry apparatus[J]. Procedia Comput Sci 171:2445–2452
Deng J, Dong W, Socher R et al (2009) Imagenet: A large-scale hierarchical image database[C]. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp 248–255
Ding Z (2023) Chemical Apparatus Benchmark Dataset: CABD, Zenodo.org, https://doi.org/10.5281/zenodo.7919476.
Shetty S (2016) Application of convolutional neural network for image classification on Pascal VOC challenge 2012 dataset[J]. arXiv preprint arXiv:1607.03785
Zhang Q, Zhang X, Mu X et al (2021) Recyclable waste image recognition based on deep learning[J]. Resour Conserv Recycl 171:105636
Varga D (2020) A combined full-reference image quality assessment method based on convolutional activation maps[J]. Algorithms 13(12):313
Acknowledgements
This work was supported by the grant of Anhui Provincial Natural Science Foundation, Nos. 1908085MF184, 1908085QF285, the grant of Scientific Research and Talent Development Foundation of the Hefei University, No.21-22RC15, the Key Research Plan of Anhui Province, Nos. 202104d07020006, 2022 k07020011, the grant of the Hefei University Postgraduate Innovation and Entrepreneurship Project, Nos. 21YCXL16,21YCXL14, in part by the grant of Key Generic Technology Research and Development Project of Hefei, No. 2021GJ030, the grant of Program for Scientific Research Innovation Team in Colleges and Universities of Anhui Province 2022AH010095, as well as the AI General Computing Platform of Hefei University.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
All authors declare that there are no conflicts of interests.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zou, L., Ding, ZS., Ran, SY. et al. A benchmark dataset in chemical apparatus: recognition and detection. Multimed Tools Appl 83, 26419–26437 (2024). https://doi.org/10.1007/s11042-023-16563-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-16563-8