Abstract
This article describes an approach to automatic recognition of charts images using neural networks with hybrid deep learning model, which allows to extract data from an image and use this data to quickly find information, as well as to describe charts for visually impaired people. The key feature of this approach is the model of the recognition process, which includes classical algorithms for image analysis and deep learning models with flexible model tuning to improve the key quality indicators of recognition software.
Currently, the problem of chart recognition is usually solved in an interactive mode, which makes it possible to recognize in a semi-automatic way with a gradual refinement of the recognized data: “end-to-end” models of neural networks or pure computer vision algorithms cannot be used for complete recognition. This article describes an approach and models that use both deep learning models with attention and computer vision algorithms to accurately extract data from charts. This article describes an approach to recognizing only function charts with continuous lines, not pie or histograms. The resulting accuracy of using a deep learning network for localizing parts of charts is 72%, this is enough for recognition since post-processing algorithms significantly improve the final recognition accuracy.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Prasad, V.S.N., Siddiquie, B., Golbeck, J., Davis, L.S.: Classifying computer generated charts. In: Content-Based Multimedia Indexing Workshop, pp. 85–92 (2007)
Savva, M., Kong, N., Chhajta, A., Fei-Fei, L., Agrawala, M., Heer, J.: Revision: automated classification, analysis and redesign of chart images. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 393–402 (2011)
Zhou, Y., Tan, C.L.: Hough-based model for recognizing bar charts in document images. In: SPIE, pp. 333–341 (2000)
Huang, W., Tan, C.L.: A system for understanding imaged infographics and its applications. In: ACM Symposium on Document Engineering, pp. 9–18 (2007)
Cliche, M., Rosenberg, D. Madeka, D., Yee, C.: Scatter-act: automated extraction of data from scatter plots. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 135–150 (2017)
Poco, J., Heer, J.: Reverse-engineering visualizations: recovering visual encodings from chart images. In: Computer Graphics Forum, pp. 353–363 (2017)
Siegel, N., Horvitz, Z., Levin, R., Divvala, S., Farhadi, A.: FigureSeer: parsing result-figures in research papers. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 664–680. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_41
Gao, J., Zhou, Y., Barner, K.E.:. View: visual information extraction widget for improving chart images accessibility. In: Proceedings of the 19th IEEE International Conference on Image Processing (ICIP 2012), 2865–2868 (2012)
Huang, W., Tan, C.L., Leow, W.K.: Model-based chart image recognition. In: Lladós, J., Kwon, Y.-B. (eds.) GREC 2003. LNCS, vol. 3088, pp. 87–99. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-25977-0_8
Huang, W., Liu, R., Tan, C.L.: Extraction of vectorized graphical information from scientific chart images. In: Proceedings of the 9th International Conference on Document Analysis and Recognition (ICDAR 2007), pp. 521–525 (2007)
Shao, M., Futrelle, R.P.: Recognition and classification of figures in PDF documents. In: Liu, W., Lladós, J. (eds.) GREC 2005. LNCS, vol. 3926, pp. 231–242. Springer, Heidelberg (2006). https://doi.org/10.1007/11767978_21
Savva, M., Kong, N., Chhajta, A., Fei-Fei, L., Agrawala, M., Heer, J.: ReVision: automated classification, analysis and redesign of chart images (2011). http://vis.stanford.edu/papers/revision
Rohatgi, A.: WebPlotDigitizer, Version 3.8 (2015). http://arohatgi.info/WebPlotDigitizer. Accessed 22 Sept 2015
Méndez, G.G., Nacenta, M.A., Vandenheste, S.: iVoLVER: interactive visual language for visualization extraction and reconstruction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2016), pp. 4073–4085 (2016)
Tummers, B.: DataTheif III (2015). http://www.datathief.org/. Accessed 22 Sept 2015
Gross, A., Schirm, S., Scholz, M.: Ycasd–a tool for capturing and scaling data from graphical representations. BMC Bioinform. 15(1), 219 (2014)
Liu, X., Klabjan, D., Bless, P.N.: Data extraction from charts via single deep neural network. https://arxiv.org/abs/1906.11906
Girshick, R.: Fast R-CNN. https://arxiv.org/pdf/1504.08083.pdf
Zhao, Z.-Q., Zheng, P., Xu, S., Wu, X.: Object detection with deep learning: a review. https://arxiv.org/pdf/1807.05511.pdf
Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. https://arxiv.org/pdf/1502.03044.pdf
Wang, W., et al.: Learning unsupervised video object segmentation through visual attention. http://openaccess.thecvf.com/content_CVPR_2019/papers/Wang_Learning_Unsupervised_Video_Object_Segmentation_Through_Visual_Attention_CVPR_2019_paper.pdf
Sun, J., Darbehani, F., Zaidi, M., Wang, B.: SAUNet: shape attentive U-net for interpretable medical image segmentation. https://arxiv.org/pdf/2001.07645v3.pdf
Sviatov, K., Miheev, A., Kanin, D., Sukhov, S., Tronin, V.: Scenes segmentation in self-driving car navigation system using neural network models with attention. In: Misra, S., et al. (eds.) ICCSA 2019. LNCS, vol. 11623, pp. 278–289. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24308-1_23
Papers with code. https://paperswithcode.com/
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf
Graph-and-Chart-Recognition. https://github.com/Grigorii-24/Graph-and-Chart-Recognition
Oktay, O., et al.: Attention U-net: learning where to look for the pancreas. https://arxiv.org/abs/1804.03999
Behera, R.K., Shukla, S., Rath, S.K., Misra, S.: Software reliability assessment using machine learning technique. In: Gervasi, O., et al. (eds.) ICCSA 2018. LNCS, vol. 10964, pp. 403–411. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-95174-4_32
Abayomi-Alli, A., et al.: Facial image quality assessment using an ensemble of pre-trained deep learning models (EFQnet). In: 2020 20th International Conference on Computational Science and Its Applications (ICCSA). IEEE (2020)
Acknowledgments
This study was supported Ministry of Education and Science of Russia in framework of project № 075-00233-20-05 from 03.11.2020 «Research of intelligent predictive multimodal analysis of big data, and the extraction of knowledge from different sources» and RFBR grant 18-47-732004 p_мк.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Sviatov, K., Yarushkina, N., Sukhov, S. (2021). Data Extraction of Charts with Hybrid Deep Learning Model. In: Gervasi, O., et al. Computational Science and Its Applications – ICCSA 2021. ICCSA 2021. Lecture Notes in Computer Science(), vol 12957. Springer, Cham. https://doi.org/10.1007/978-3-030-87013-3_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-87013-3_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87012-6
Online ISBN: 978-3-030-87013-3
eBook Packages: Computer ScienceComputer Science (R0)