Abstract
Building a system for automatic handwritten mathematical expressions recognition (HMER) has received considerable attention for its extensive applications. However, HMER remains challenging due to its own many characteristics such as the ambiguity of handwritten symbol, the two-dimensional characteristics of expression structure, and a large amount of context information. Inspired by research on machine translation and image caption, we proposed an Encoder-Decoder structure to recognize the handwritten mathematical expression. Encoder based dual attention is used to extract the features of the expression image and attention-decoder achieves symbol recognition and structural analysis. The mask information is added to the input data allows the model to better focus on the region of interest. In order to verify the effectiveness of our method, we train the model on the CROHME-2016 train set and use the CROHME-2014 test set as the validation set, the CROHME-2016 test set as the test set. The experimental results show that our method is greatly improved compared with other recognition methods, achieved respectively 47.49% and 45.10% ExpRate in the two test sets.
Supported by Transformation and industrialization demonstration of marine scientific and technological achievements in Xiamen Marine and Fisheries Bureau (No. 18CZB033HJ11).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Zhang, J., Du, J., Dai, L.: Multi-scale attention with dense encoder for handwritten mathematical. In: Conference on Pattern Recognition (ICPR), pp. 2245–2250. IEEE, Beijing (2018)
Mouchere, H., Viard-Gaudin, C., Zanibbi, R.: ICFHR 2014 competition on recognition of on-line handwritten mathematical expressions. In: 2014 14th International Conference on Frontiers in Handwriting Recognition, pp. 791–796. IEEE, Hersonissos, Greece (2014)
Long, C., Zhang, H., Xiao, J.: SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6298–6306. IEEE, Honolulu (2017)
Le, A.D., Nakagawa, M.: Training an end-to-end system for handwritten mathematical expression recognition by generated patterns. In: 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), pp. 1056–1061. IEEE, Kyoto (2017)
Vinyals, O., Toshev, A., Bengio, S.: Show and tell: a neural image caption generator. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3156–3164. IEEE, Boston (2015)
Xu, K., Ba, J., Kiros, R.: Show, attend and tell: neural image caption generation with visual attention. In: Proceedings of the 32nd International Conference on Machine Learning, pp. 2048–2057(2015)
Zhang, J., Du, J., Zhang, S.: Watch, attend and parse: an end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recognit. 71, 196–206 (2017)
Jie, H., Li, S., Gang, S.: Squeeze-and-excitation networks. In: 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7132–7141. IEEE, Salt Lake City (2018)
Tu, Z., Lu, Z., Liu, Y.: Modeling coverage for neural machine translation. In: 54th Annual Meeting of the Association-for-Computational-Linguistics (ACL), pp. 76–85. Berlin (2016)
Mouchère, H., Viard-Gaudin, C., Zanibbi, R.: ICFHR2016 CROHME: competition on recognition of online handwritten mathematical expressions. In: 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), Shenzhen, pp. 607–612 (2016)
He, K., Zhang, X., Ren, S.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. IEEE, Seattle (2016)
Huang, G., Liu, Z., Laurens, V.D.M.: Densely connected convolutional networks. In: 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4700–4708. IEEE, Honolulu (2017)
Bhunia, A.K., Bhowmick, A., Bhunia, A.K.: Handwriting trajectory recovery using end-to-end deep encoder-decoder network. In: 24th International Conference on Pattern Recognition (ICPR), pp. 3639–3644. IEEE, Beijing (2018)
Chan, K.F., Yeung, D.Y.: Mathematical expression recognition: a survey. Int. J. Document Anal. Recognit. (ICDAR), 3–15 (2000). https://doi.org/10.1007/PL00013549
Chen, X., Ma, L., Jiang, W.: Regularizing RNNs for caption generation by reconstructing the past with the present. In: 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7995–8003. IEEE, Salt Lake City (2018)
Le, A.D., Indurkhya, B., Nakagawa, M.: Pattern generation strategies for improving recognition of handwritten mathematical expressions. Pattern Recognit. Lett.128, 255–262 (2019)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks, pp. 249–256 (2010)
Woo, S., Park, J., Lee, J.Y.: CBAM: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19(2018)
Alvaro, F., Sánchez, J., Benedí, J.: Recognition of on-line handwritten mathematical expressions using 2D stochastic context free grammars and Hidden Markov models. Pattern Recognit. Lett. 35, 58–67 (2014)
Alvaro, F., Sánchez, J., Benedí, J.: An integrated grammar based approach for mathematical expression recognition. Pattern Recognit. 51, 135–147 (2016)
Zanibbi, R., Blostein, D., Cordy, J.R.: Recognizing mathematical expressions using tree transformation. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1455–1467 (2002)
MacLean, S., Labahn, G.: A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets. Int. J. Document Anal. Recognit. 139–163 (2013). https://doi.org/10.1007/s10032-012-0184-x
Mouchere, H., Zanibbi, R., Garain, U.: Advancing the state-of-the-art for handwritten math recognition. Int. J. Document Anal. Recognit. 173–189 (2016)
Zanibbi, R., Blostein, D.: Recognition and retrieval of mathematical expressions. Int. J. Document Anal. Recognit. 15(4), 331–357 (2012). https://doi.org/10.1007/s10032-011-0174-4
Chan, K.F., Yeung, D.Y.: Error detection, error correction and performance evaluation in on-line mathematical expression recognition. Pattern Recognit. 34(8), 1671–1684 (2001)
Dauphin, Y.N., Fan, A., Auli, M.: Language modeling with gated convolutional networks. In: Proceedings of the 34th International Conference on Machine Learning, pp. 70:933–941(2017)
Mouchere, H., Zanibbi, R., Garain, U.: Advancing the state-of-the-art for handwritten math recognition. Inte. J. Document Anal. Recognit. 19(2), 173–189 (2016)
Zhang, J., Du, J., Dai, L.: A GRU-based encoder-decoder approach with attention for online handwritten mathematical expression recognition. In: 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), pp. 902–907. IEEE, Kyoto (2017)
Anderson, R.H.: Syntax-directed recognition of hand-printed two-dimensional mathematics. In: Symposium on Interactive Systems for Experimental Applied Mathematics: Proceedings of the Association for Computing Machinery Inc., Symposium, pp. 436–459 (1967)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Hu, J., Zhou, P., Liao, S., Li, S., Su, S., Su, S. (2021). MDA-Network: Mask and Dual Attention Network for Handwritten Mathematical Expression Recognition. In: Sun, Y., Liu, D., Liao, H., Fan, H., Gao, L. (eds) Computer Supported Cooperative Work and Social Computing. ChineseCSCW 2020. Communications in Computer and Information Science, vol 1330. Springer, Singapore. https://doi.org/10.1007/978-981-16-2540-4_15
Download citation
DOI: https://doi.org/10.1007/978-981-16-2540-4_15
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-2539-8
Online ISBN: 978-981-16-2540-4
eBook Packages: Computer ScienceComputer Science (R0)