More Web Proxy on the site http://driver.im/

research-article

Chinese Character Inpainting with Contextual Semantic Constraints

Authors:

Jiawan ZhangAuthors Info & Claims

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

Pages 1829 - 1837

https://doi.org/10.1145/3474085.3475333

Published: 17 October 2021 Publication History

Abstract

Chinese character inpainting is a challenging task where large missing regions have to be filled with both visually and semantic realistic contents. Existing methods generally produce pseudo or ambiguous characters due to lack of semantic information. Given the key observation that Chinese characters contain visually glyph representation and intrinsic contextual semantics, we tackle the challenge of similar Chinese characters by modeling the underlying regularities among glyph and semantic information. We propose a semantics enhanced generative framework for Chinese character inpainting, where a global semantic supervising module (GSSM) is introduced to constrain contextual semantics. In particular, sentence embedding is used to guide the encoding of continuous contextual characters. The method can not only generate realistic Chinese character, but also explicitly utilize context as reference during network training to eliminate ambiguity. The proposed method is evaluated on both handwritten and printed Chinese characters with various masks. The experiments show that the method successfully predicts missing character information without any mask input, and achieves significant sentence-level results benefiting from global semantic supervising in a wide variety of scenes.

References

[1]

Coloma Ballester, Marcelo Bertalmio, Vicent Caselles, Guillermo Sapiro, and Joan Verdera. 2001. Filling-in by joint interpolation of vector fields and gray levels. TIP, Vol. 10, 8 (2001), 1200--1211.

Digital Library

[2]

Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ToG, Vol. 28, 3 (2009), 24.

Digital Library

[3]

Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. 2000. Image inpainting. In Siggraph. ACM, USA, 417--424.

Digital Library

[4]

Jie Chang, Yujun Gu, Ya Zhang, and Yan-Feng Wang. 2018. Chinese Handwriting Imitation with Hierarchical Generative Adversarial Network. In BMVC. BMVA Press, UK, 290.

[5]

Huizhong Chen, Sam S Tsai, Georg Schroth, David M Chen, Radek Grzeszczuk, and Bernd Girod. 2011. Robust text detection in natural images with edge-enhanced maximally stable extremal regions. In ICIP. IEEE Computer Society, USA, 2609--2612.

[6]

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2016. Fast and accurate deep network learning by exponential linear units (ELUs). In ICLR. JMLR.org, USA, 1--14.

[7]

Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. 2004. Region filling and object removal by exemplar-based image inpainting. TIP, Vol. 13, 9 (2004), 1200--1212.

Digital Library

[8]

Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-training with whole word masking for chinese bert. arXiv preprint arXiv:1906.08101 (2019).

[9]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. Association for Computational Linguistics, USA, 4171--4186.

[10]

Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 (2012).

[11]

Gao Huang, Zhuang Liu, and Laurens Van Der Maaten. 2017. Densely connected convolutional networks. In CVPR. IEEE Computer Society, USA, 4700--4708.

[12]

Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML. JMLR.org, USA, 448--456.

Digital Library

[13]

Shunsuke Kitada, Ryunosuke Kotani, and Hitoshi Iyatomi. 2018. End-to-End Text Classification via Image-based Embedding using Character-level Networks. In AIPR. IEEE Computer Society, USA, 1--4.

[14]

Praveen Krishnan, Kartik Dutta, and C. V. Jawahar. 2018. Word Spotting and Recognition Using Deep Embedding. In IAPRW. IEEE, USA, 1--6,.

[15]

Jianwu Li, Ge Song, and Minhua Zhang. 2020. Occluded offline handwritten Chinese character recognition using deep convolutional generative adversarial network and improved GoogLeNet. Neural Computing and Applications, Vol. 32, 9 (2020), 4805--4819.

[16]

Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. 2018. Image inpainting for irregular holes using partial convolutions. In ECCV. Springer, USA, 85--100.

[17]

Yuxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, and Qinghong Han. 2019. Glyce: Glyph-vectors for Chinese character representations. In NIPS. Curran Associates, USA, 2746--2757.

Digital Library

[18]

Fang Miao and Li Feng. 2020. Research on Character Image Inpainting based on Generative Adversarial Network. In ICCST. IEEE Computer Society, USA, 137--140.

[19]

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral normalization for generative adversarial networks. In ICLR. JMLR.org, USA, 1--26.

[20]

Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. 2016. Context encoders: Feature learning by inpainting. In CVPR. IEEE Computer Society, USA, 2536--2544.

[21]

Zhi Qiao, Yu Zhou, Dongbao Yang, Yucan Zhou, and Weiping Wang. 2020. SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition. In CVPR. IEEE Computer Society, USA, 13528--13537.

[22]

Alec Radford, Luke Metz, and Soumith Chintala. 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR. JMLR.org, USA, 1--16.

[23]

Jose A. Rodriguez-Serrano, Albert Gordo, and Florent Perronnin. 2015. Label Embedding: A Frugal Baseline for Text Recognition. IJCV, Vol. 113, 3 (2015), 193--207. https://doi.org/10.1007/s11263-014-0793--6

Digital Library

[24]

Prasun Roy, Saumik Bhattacharya, Subhankar Ghosh, and Umapada Pal. 2020. STEFANN: scene text editor using font adaptive neural network. In CVPR. IEEE Computer Society, USA, 13228--13237.

[25]

Kazuma Sasaki, Satoshi Iizuka, and Edgar Simo-Serra. 2017. Joint gap detection and inpainting of line drawings. In CVPR. IEEE Computer Society, USA, 5725--5733.

[26]

Kazuma Sasaki, Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2018. Learning to restore deteriorated line drawing. The Visual Computer, Vol. 34, 6--8 (2018), 1077--1085.

Digital Library

[27]

Baoguang Shi, Xiang Bai, and Cong Yao. 2016. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. PAMI, Vol. 39, 11 (2016), 2298--2304.

Digital Library

[28]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. In ILSVRC Workshop. JMLR.org, USA, 1--14.

[29]

Tzu-Ray Su and Hung-Yi Lee. 2017. Learning chinese word representations from glyphs of characters. In EMNLP. ACL, USA, 1--10.

[30]

M Sun, J Li, Z Guo, Z Yu, Y Zheng, X Si, and Z Liu. 2016. Thuctc: an efficient chinese text classifier. GitHub Repository (2016).

[31]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. Curran Associates, USA, 1--11.

Digital Library

[32]

Tomas Wilkinson and Anders Brun. 2016. Semantic and Verbatim Word Spotting Using Deep Neural Networks. In ICFHR. IEEE Computer Society, USA, 307--312.

[33]

Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. 2017. High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis. In CVPR. IEEE Computer Society, USA, 4076--4084.

[34]

Deli Yu, Xuan Li, Chengquan Zhang, Tao Liu, Junyu Han, Jingtuo Liu, and Errui Ding. 2020. Towards accurate scene text recognition with semantic reasoning networks. In CVPR. IEEE Computer Society, USA, 12113--12122.

[35]

Xiang Zhang and Yann LeCun. 2017. Which encoding is the best for text classification in Chinese, English, Japanese and Korean? arXiv preprint arXiv:1708.02657 (2017), 1--24.

[36]

Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. 2017. East: an efficient and accurate scene text detector. In CVPR. IEEE Computer Society, USA, 5551--5560.

Cited By

Zhu SXue HNie NZhu CLiu HFang PCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Reproducing the Past: A Dataset for Benchmarking Inscription RestorationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680587(7714-7723)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680587
Zhao LYuan ZLou Y(2024)Cross Auto-Encoder for Inscription Character Inpainting2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10649951(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10649951
Lakshya KKamal AAnjali ARai H(2024)Dual-Stage Inpainting Approach for Character Reconstruction in Ancient Hindi Texts2024 IEEE International Conference on Computer Vision and Machine Intelligence (CVMI)10.1109/CVMI61877.2024.10782411(1-6)Online publication date: 19-Oct-2024
https://doi.org/10.1109/CVMI61877.2024.10782411
Show More Cited By

Index Terms

Chinese Character Inpainting with Contextual Semantic Constraints
1. Computing methodologies

Recommendations

Semantic-Based Handwritten Chinese Character Recognition Model
ICCMS '10: Proceedings of the 2010 Second International Conference on Computer Modeling and Simulation - Volume 01

There have been many different literals discussing algorithms for handwritten Chinese character recognition, but most algorithms aim at recognizing isolated Chinese character one by one. Therefore, their recognition accuracy isn’t good enough for the ...
Generative character inpainting guided by structural information
Abstract
Character inpainting is an attractive and challenging task, especially for Chinese calligraphy characters with complex structures and styles. The diversity of Chinese calligraphy styles has created its unique artistic beauty, but specific style ...
Improving Off-Line Handwritten Chinese Character Recognition with Semantic Information
Neural Information Processing
Abstract
Off-line handwritten Chinese character recognition (HCCR) is a well-developed area in computer vision. However, existing methods only discuss the image-level information. Chinese character is a kind of ideograph, which means it is not only a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

October 2021

5796 pages

ISBN:9781450386517

DOI:10.1145/3474085

General Chairs:
Heng Tao Shen
University of Electronic Science&Technology of China, China
,
Yueting Zhuang
Zhejiang University, China
,
John R. Smith
IBM, USA
,
Program Chairs:
Yang Yang
University of Electronic Science and Technology of China, China
,
Pablo Cesar
CWI&TU Delft, The Netherlands
,
Florian Metze
FACEBOOK, Inc., USA
,
Balakrishnan Prabhakaran
University of Texas at Dallas, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Key Research and Development Program of China under Grant

Conference

MM '21

Sponsor:

SIGMM

MM '21: ACM Multimedia Conference

October 20 - 24, 2021

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
346
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)8

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhu SXue HNie NZhu CLiu HFang PCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Reproducing the Past: A Dataset for Benchmarking Inscription RestorationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680587(7714-7723)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680587
Zhao LYuan ZLou Y(2024)Cross Auto-Encoder for Inscription Character Inpainting2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10649951(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10649951
Lakshya KKamal AAnjali ARai H(2024)Dual-Stage Inpainting Approach for Character Reconstruction in Ancient Hindi Texts2024 IEEE International Conference on Computer Vision and Machine Intelligence (CVMI)10.1109/CVMI61877.2024.10782411(1-6)Online publication date: 19-Oct-2024
https://doi.org/10.1109/CVMI61877.2024.10782411
Han KYou WDeng HSun LSong JHu ZYi H(2024)LanT: finding experts for digital calligraphy character restorationMultimedia Tools and Applications10.1007/s11042-023-17844-y83:24(64963-64986)Online publication date: 18-Jan-2024
https://doi.org/10.1007/s11042-023-17844-y
Sun DYang TPan XWang JPan G(2024)Chinese Character Image Inpainting with Skeleton Extraction and Adversarial LearningAdvanced Intelligent Computing Technology and Applications10.1007/978-981-97-5600-1_21(246-256)Online publication date: 30-Jul-2024
https://doi.org/10.1007/978-981-97-5600-1_21
Xing CRen Z(2023)Binary Inscription Character Inpainting Based on Improved Context EncodersIEEE Access10.1109/ACCESS.2023.328244211(55834-55843)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3282442
Yu HChen JLi BXue X(2023)Chinese character recognition with radical-structured stroke treesMachine Learning10.1007/s10994-023-06450-6113:6(3807-3827)Online publication date: 22-Dec-2023
https://doi.org/10.1007/s10994-023-06450-6

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents