Semantic-Aware Non-local Network for Handwritten Mathematical Expression Recognition
Pages 361 - 371
Abstract
Handwritten mathematical expression recognition (HMER) is a challenging task due to its complex two-dimensional structure of mathematical expressions and the high similarity between handwritten texts. Most existing encoder-decoder approaches for HMER mainly depend on local visual features but are seldom studied in explicit global semantic information. Besides, existing works for HMER primarily focus on local information. However, this obtained information is difficult to transmit between distant locations. In this paper, we propose a semantic-aware non-local network to tackle the above problems for HMER. Specifically, we propose to adopt the non-local network to capture long-term dependencies while integrating local and non-local features. Moreover, we customized the FastText language model to our backbone to learn the semantic-aware information. The experimental results illustrate that our design consistently outperforms the state-of-the-art methods on the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014 and 2016 datasets.
References
[1]
He F, Tan J, and Bi N Lu Y, Vincent N, Yuen PC, Zheng W-S, Cheriet F, and Suen CY Handwritten mathematical expression recognition: a survey Pattern Recognition and Artificial Intelligence 2020 Cham Springer 55-66
[2]
Mouchere, H., et al.: ICFHR 2014 competition on recognition of on-line handwritten mathematical expressions (CROHME 2014). In: International Conference on Frontiers in Handwriting Recognition. IEEE (2014)
[3]
Mouchère, H., et al.: ICFHR2016 CROHME: competition on recognition of online handwritten mathematical expressions. In: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2016)
[4]
Wang, D.-H., et al.: ICFHR 2020 competition on offline recognition and spotting of handwritten mathematical expressions-OffRaSHME. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2020)
[5]
Cheng, Z., Bai, F., Xu, Y., Zheng, G., Pu, S., Zhou, S.: Focusing attention: towards accurate text recognition in natural images. In: ICCV 2017, pp. 5086–5094 (2017)
[6]
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q.V., Salakhutdinov, R.: Transformer-XL: attentive language models beyond a fixed-length context. In: ACL (1), pp. 2978–2988 (2019)
[7]
Cheng, J., Dong, L., Lapata, M.: Long short-term memory-networks for machine reading. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016)
[8]
Gordo, A., et al.: LEWIS: latent embeddings for word images and their semantics. In: IEEE International Conference on Computer Vision. IEEE (2015)
[9]
Wilkinson, T., Brun, A.: Semantic and verbatim word spotting using deep neural networks. In: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2016)
[10]
Wang Q-F, Yin F, and Liu C-L Handwritten Chinese text recognition by integrating multiple contexts IEEE Trans. Pattern Anal. Mach. Intell. 2011 34 8 1469-1481
[11]
Wang, X., et al.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)
[12]
Bojanowski P et al. Enriching word vectors with subword information Trans. Assoc. Comput. Linguist. 2017 5 135-146
[13]
Anderson, R.H.: Syntax-directed recognition of hand-printed two-dimensional mathematics. In: Symposium on Interactive Systems for Experimental Applied Mathematics: Proceedings of the Association for Computing Machinery Inc. Symposium, pp. 436–459. ACM (1967)
[14]
Chan KF and Yeung DY Error detection, error correction and performance evaluation in on-line mathematical expression recognition Pattern Recogn. 2001 34 8 1671-1684
[15]
Lavirotte S and Pottier L Mathematical formula recognition using graph grammar Proc. SPIE Int. Soc. Opt. Eng. 2016 3305 44-52
[16]
Yamamoto, R., et al.: On-line recognition of handwritten mathematical expressions based on stroke-based stochastic context-free grammar. In: Proceedings of International Workshop on Frontiers in Handwriting Recognition, pp. 249–254, October 2006
[17]
Maclean S and Labahn G A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets Int. J. Doc. Anal. Recogn. 2013 16 2 139-163
[18]
Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. Computer Science (2014)
[19]
Bahdanau, D., et al.: End-to-end attention-based large vocabulary speech recognition. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2016)
[20]
Zhang, J., et al.: Radical analysis network for zero-shot learning in printed Chinese character recognition. In: 2018 IEEE International Conference on Multimedia and Expo (ICME). IEEE (2018)
[21]
Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. Computer Science, pp. 2048–2057 (2015)
[22]
Zhang, J., Du, J., Dai, L.: A GRU-based encoder-decoder approach with attention for online handwritten mathematical expression recognition. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). IEEE (2018)
[23]
Zhang J et al. Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition Pattern Recogn. 2017 71 196-206
[24]
Zhang, J., Du, J., Dai, L.: Multi-scale attention with dense encoder for handwritten mathematical expression recognition. In: 2018 24th International Conference on Pattern Recognition (ICPR) (2018)
[25]
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Computer Science (2014)
[26]
Huang, G., et al.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)
[27]
Zhang, J., et al.: A tree-structured decoder for image-to-markup generation. In: International Conference on Machine Learning. PMLR (2020)
[28]
Wu, J.-W., Yin, F., Zhang, YM., Zhang, X.-Y., Liu, C.-L.: Image-to-markup generation via paired adversarial learning. In: Berlingerio, M., Bonchi, F., Gärtner, T., Hurley, N., Ifrim, G. (eds.) ECML PKDD 2018. LNCS, vol. 11051, pp. 18–34. Springer, Cham (2019).
[29]
Wu J-W et al. Handwritten mathematical expression recognition via paired adversarial learning Int. J. Comput. Vis. 2020 128 10 2386-2401
[30]
Li, Z., et al.: Improving attention-based handwritten mathematical expression recognition with scale augmentation and drop attention. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2020)
[31]
Zhao, W., Gao, L., Yan, Z., Peng, S., Du, L., Zhang, Z.: Handwritten mathematical expression recognition with bidirectionally trained transformer. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12822, pp. 570–584. Springer, Cham (2021).
[32]
Truong, T.-N., et al.: Improvement of end-to-end offline handwritten mathematical expression recognition by weakly supervised learning. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2020)
[33]
Truong T-N, Nguyen CT, and Nakagawa M Syntactic data generation for handwritten mathematical expression recognition Pattern Recogn. Lett. 2022 153 83-91
[34]
Bian, X., et al.: Handwritten mathematical expression recognition via attention aggregation based bi-directional mutual learning. arXiv e-prints (2021)
[35]
Yuan, Y., et al.: Syntax-aware network for handwritten mathematical expression recognition. arXiv preprint arXiv:2203.01601 (2022)
[36]
Liu Y-L et al. A robust and fast non-local means algorithm for image denoising J. Comput. Sci. Technol. 2008 23 2 270-279
Index Terms
- Semantic-Aware Non-local Network for Handwritten Mathematical Expression Recognition
Index terms have been assigned to the content through auto-classification.
Recommendations
Semantic Graph Representation Learning for Handwritten Mathematical Expression Recognition
Document Analysis and Recognition - ICDAR 2023AbstractHandwritten mathematical expression recognition (HMER) has attracted extensive attention recently. However, current methods cannot explicitly study the interactions between different symbols, which may fail when faced similar symbols. To alleviate ...
When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition
Computer Vision – ECCV 2022AbstractRecently, most handwritten mathematical expression recognition (HMER) methods adopt the encoder-decoder networks, which directly predict the markup sequences from formula images with the attention mechanism. However, such methods may fail to ...
ICAL: Implicit Character-Aided Learning for Enhanced Handwritten Mathematical Expression Recognition
Document Analysis and Recognition - ICDAR 2024AbstractSignificant progress has been made in the field of handwritten mathematical expression recognition, while existing encoder-decoder methods are usually difficult to model global information in [inline-graphic not available: see fulltext]. Therefore,...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Oct 2022
788 pages
ISBN:978-3-031-18912-8
DOI:10.1007/978-3-031-18913-5
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2022.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 14 October 2022
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 04 Jan 2025