[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content

Data Extraction of Charts with Hybrid Deep Learning Model

  • Conference paper
  • First Online:
Computational Science and Its Applications – ICCSA 2021 (ICCSA 2021)

Abstract

This article describes an approach to automatic recognition of charts images using neural networks with hybrid deep learning model, which allows to extract data from an image and use this data to quickly find information, as well as to describe charts for visually impaired people. The key feature of this approach is the model of the recognition process, which includes classical algorithms for image analysis and deep learning models with flexible model tuning to improve the key quality indicators of recognition software.

Currently, the problem of chart recognition is usually solved in an interactive mode, which makes it possible to recognize in a semi-automatic way with a gradual refinement of the recognized data: “end-to-end” models of neural networks or pure computer vision algorithms cannot be used for complete recognition. This article describes an approach and models that use both deep learning models with attention and computer vision algorithms to accurately extract data from charts. This article describes an approach to recognizing only function charts with continuous lines, not pie or histograms. The resulting accuracy of using a deep learning network for localizing parts of charts is 72%, this is enough for recognition since post-processing algorithms significantly improve the final recognition accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
GBP 19.95
Price includes VAT (United Kingdom)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
GBP 87.50
Price includes VAT (United Kingdom)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
GBP 109.99
Price includes VAT (United Kingdom)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Prasad, V.S.N., Siddiquie, B., Golbeck, J., Davis, L.S.: Classifying computer generated charts. In: Content-Based Multimedia Indexing Workshop, pp. 85–92 (2007)

    Google Scholar 

  2. Savva, M., Kong, N., Chhajta, A., Fei-Fei, L., Agrawala, M., Heer, J.: Revision: automated classification, analysis and redesign of chart images. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 393–402 (2011)

    Google Scholar 

  3. Zhou, Y., Tan, C.L.: Hough-based model for recognizing bar charts in document images. In: SPIE, pp. 333–341 (2000)

    Google Scholar 

  4. Huang, W., Tan, C.L.: A system for understanding imaged infographics and its applications. In: ACM Symposium on Document Engineering, pp. 9–18 (2007)

    Google Scholar 

  5. Cliche, M., Rosenberg, D. Madeka, D., Yee, C.: Scatter-act: automated extraction of data from scatter plots. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 135–150 (2017)

    Google Scholar 

  6. Poco, J., Heer, J.: Reverse-engineering visualizations: recovering visual encodings from chart images. In: Computer Graphics Forum, pp. 353–363 (2017)

    Google Scholar 

  7. Siegel, N., Horvitz, Z., Levin, R., Divvala, S., Farhadi, A.: FigureSeer: parsing result-figures in research papers. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 664–680. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_41

    Chapter  Google Scholar 

  8. Gao, J., Zhou, Y., Barner, K.E.:. View: visual information extraction widget for improving chart images accessibility. In: Proceedings of the 19th IEEE International Conference on Image Processing (ICIP 2012), 2865–2868 (2012)

    Google Scholar 

  9. Huang, W., Tan, C.L., Leow, W.K.: Model-based chart image recognition. In: Lladós, J., Kwon, Y.-B. (eds.) GREC 2003. LNCS, vol. 3088, pp. 87–99. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-25977-0_8

    Chapter  Google Scholar 

  10. Huang, W., Liu, R., Tan, C.L.: Extraction of vectorized graphical information from scientific chart images. In: Proceedings of the 9th International Conference on Document Analysis and Recognition (ICDAR 2007), pp. 521–525 (2007)

    Google Scholar 

  11. Shao, M., Futrelle, R.P.: Recognition and classification of figures in PDF documents. In: Liu, W., Lladós, J. (eds.) GREC 2005. LNCS, vol. 3926, pp. 231–242. Springer, Heidelberg (2006). https://doi.org/10.1007/11767978_21

    Chapter  Google Scholar 

  12. Savva, M., Kong, N., Chhajta, A., Fei-Fei, L., Agrawala, M., Heer, J.: ReVision: automated classification, analysis and redesign of chart images (2011). http://vis.stanford.edu/papers/revision

  13. Rohatgi, A.: WebPlotDigitizer, Version 3.8 (2015). http://arohatgi.info/WebPlotDigitizer. Accessed 22 Sept 2015

  14. Méndez, G.G., Nacenta, M.A., Vandenheste, S.: iVoLVER: interactive visual language for visualization extraction and reconstruction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2016), pp. 4073–4085 (2016)

    Google Scholar 

  15. Tummers, B.: DataTheif III (2015). http://www.datathief.org/. Accessed 22 Sept 2015

  16. Gross, A., Schirm, S., Scholz, M.: Ycasd–a tool for capturing and scaling data from graphical representations. BMC Bioinform. 15(1), 219 (2014)

    Article  Google Scholar 

  17. Liu, X., Klabjan, D., Bless, P.N.: Data extraction from charts via single deep neural network. https://arxiv.org/abs/1906.11906

  18. Girshick, R.: Fast R-CNN. https://arxiv.org/pdf/1504.08083.pdf

  19. Zhao, Z.-Q., Zheng, P., Xu, S., Wu, X.: Object detection with deep learning: a review. https://arxiv.org/pdf/1807.05511.pdf

  20. Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. https://arxiv.org/pdf/1502.03044.pdf

  21. Wang, W., et al.: Learning unsupervised video object segmentation through visual attention. http://openaccess.thecvf.com/content_CVPR_2019/papers/Wang_Learning_Unsupervised_Video_Object_Segmentation_Through_Visual_Attention_CVPR_2019_paper.pdf

  22. Sun, J., Darbehani, F., Zaidi, M., Wang, B.: SAUNet: shape attentive U-net for interpretable medical image segmentation. https://arxiv.org/pdf/2001.07645v3.pdf

  23. Sviatov, K., Miheev, A., Kanin, D., Sukhov, S., Tronin, V.: Scenes segmentation in self-driving car navigation system using neural network models with attention. In: Misra, S., et al. (eds.) ICCSA 2019. LNCS, vol. 11623, pp. 278–289. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24308-1_23

    Chapter  Google Scholar 

  24. Papers with code. https://paperswithcode.com/

  25. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf

  26. Graph-and-Chart-Recognition. https://github.com/Grigorii-24/Graph-and-Chart-Recognition

  27. Oktay, O., et al.: Attention U-net: learning where to look for the pancreas. https://arxiv.org/abs/1804.03999

  28. Behera, R.K., Shukla, S., Rath, S.K., Misra, S.: Software reliability assessment using machine learning technique. In: Gervasi, O., et al. (eds.) ICCSA 2018. LNCS, vol. 10964, pp. 403–411. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-95174-4_32

    Chapter  Google Scholar 

  29. Abayomi-Alli, A., et al.: Facial image quality assessment using an ensemble of pre-trained deep learning models (EFQnet). In: 2020 20th International Conference on Computational Science and Its Applications (ICCSA). IEEE (2020)

    Google Scholar 

Download references

Acknowledgments

This study was supported Ministry of Education and Science of Russia in framework of project № 075-00233-20-05 from 03.11.2020 «Research of intelligent predictive multimodal analysis of big data, and the extraction of knowledge from different sources» and RFBR grant 18-47-732004 p_мк.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Kirill Sviatov .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sviatov, K., Yarushkina, N., Sukhov, S. (2021). Data Extraction of Charts with Hybrid Deep Learning Model. In: Gervasi, O., et al. Computational Science and Its Applications – ICCSA 2021. ICCSA 2021. Lecture Notes in Computer Science(), vol 12957. Springer, Cham. https://doi.org/10.1007/978-3-030-87013-3_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-87013-3_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87012-6

  • Online ISBN: 978-3-030-87013-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics