JP7570843B2 - IMAGE PROCESSING APPARATUS, IMAGE FORMING SYSTEM, IMAGE PROCESSING METHOD, AND PROGRAM - Google Patents

Description

This disclosure relates to a technique for extracting character strings of specified items from a scanned image.

There is a method of extracting character strings in a scanned image as digital data by performing OCR (Optical Character Recognition) processing on the scanned image obtained by scanning a document. There is also a method of extracting character strings in a specified item from the results of OCR processing by detecting the type of document indicated by the scanned image and identifying the position of a specified item according to the document type.

Patent document 1 describes a method for registering information about the characteristics of ruled lines on forms in a database, and then identifying forms from the database that have matching line characteristics in an input image.

JP 2009-230498 A

For example, in the case of a form with multiple writing areas for handwriting as shown in FIG. 3, a form with handwritten characters written in either writing area 301 or writing area 302 is the subject of OCR processing. In such forms, the same item such as "phone number" and "email address" may be written in each writing area. For this reason, when attempting to extract a character string for a specific item from the scanned image, it is not possible to determine from which writing area the character string should be extracted, and there is a risk that the desired character string will not be extracted.

The image processing device according to the technology disclosed herein includes an image acquisition unit that acquires a read image obtained by reading a document, a separation unit that separates an image of handwritten characters and an image of an area other than the handwritten characters from the read image, and a detection unit that detects character string areas in each image obtained by the separation, and detects character string areas of handwritten characters and character string areas of printed characters included in the read image, respectively .
the form has a plurality of areas for writing handwritten characters, and the form has handwritten characters written in any of the plurality of areas, the form having the first position information of a character string area of handwritten characters and the second position information of a character string area of printed characters ; an identification means for identifying, when a degree of coincidence between pixel information of the character string area of the form based on the position information included in the form data and pixel information of the character string area of the read image exceeds a threshold value, that the character string area of the form and the character string area of the read image are similar in arrangement ; and an extraction means for extracting, when it is identified that the character string area of printed characters of the form based on the second position information included in the form data and the character string area of printed characters detected from the read image are similar, a character string of a predetermined item associated with the character string area of handwritten characters of the form data from the character string area of handwritten characters detected from the read image that is further identified as being similar to the character string area of handwritten characters of the form based on the first position information included in the form data.

The technology disclosed herein makes it possible to extract a character string written by hand from a scanned image obtained by scanning a document having multiple areas for writing characters by hand in any of the areas.

FIG. 1 is a block diagram showing a configuration of an image forming system. FIG. 2 is a diagram showing the functional configuration of the image forming apparatus. FIG. 13 is a diagram showing an example of a blank form. A diagram showing form data. 11 is a flowchart showing a process of extracting character information from metadata items. 5A and 5B are diagrams showing an example of read image data and a processing result. 5A and 5B are diagrams showing an example of read image data and a processing result. FIG. 13 is a diagram showing an example of a content confirmation screen. FIG. 13 is a diagram showing an example of a content confirmation screen. FIG. 13 is a diagram showing an example of a correction confirmation screen. A diagram showing form data. 11 is a flowchart showing a process of extracting character information from metadata items.

Below, the embodiments of the technology disclosed herein are described in detail with reference to the drawings. Note that the following embodiments do not limit the technology disclosed herein according to the claims, and not all of the combinations of features described in the following embodiments are necessarily essential to the solutions of the technology disclosed herein. In addition, the same components are given the same reference numbers, and descriptions are omitted.

<Embodiment 1>
[Overview of Image Forming System]
FIG. 1 is a diagram showing the configuration of an image forming system. As shown in FIG. 1, the image forming system includes an image forming apparatus 100, a host computer 170, and a server 191 (which may be a cloud server). In this embodiment, the image forming apparatus 100 will be described as being a multifunction printer (MFP) that integrates multiple functions such as a print function, a read function, and a FAX function. The server 191 will be described as having a document management function. The image forming apparatus 100, the host computer 170, and the server 191 are connected to a network 190 such as a LAN (Local Area Network) so as to be able to communicate with each other. A plurality of image forming apparatuses, host computers, and servers may be connected, or other devices may be connected. The network 190 may be a wired network, a wireless network, or a combination thereof.

The image forming apparatus 100 includes a control device 110, a reader device 120, a printer device 130, an operation unit 140, and a storage device 150. The control device 110 is a control board (controller) that controls the image forming apparatus 100 as a whole. The control device 110 includes a CPU 111, a ROM 112, a RAM 113, and an image processing unit 114. The CPU 111 controls each block in the control device 110 via a system bus (not shown). For example, the CPU 111 executes the functions of the image forming apparatus 100 by reading and executing a program stored in the ROM 112, the RAM 113, the storage device 150, or another storage medium. The ROM 112 stores, for example, a control program, and tables and setting data necessary for executing the functions of the image forming apparatus 100. The RAM 113 is used, for example, as a work memory for the CPU 111.

The image processing unit 114 performs various image processing such as conversion, correction, editing, compression/decompression, etc. on the scanned image data generated by the reader device 120 reading a document and on image data received from the outside. The image processing unit 114 may be configured as hardware or may be realized as software. In this way, the image forming device 100 of this embodiment also functions as an image processing device.

The reader device 120 has a scanner engine configuration, and optically reads an original document set on a document table (not shown) or fed from an automatic document feeder (ADF) (not shown), and generates read image data. The printer device 130 has a printer engine configuration compatible with various recording methods such as inkjet recording method and electrophotographic method, and forms an image on a recording medium.

The storage device 150 stores, for example, image data, device information such as modes and licenses, and user information such as an address book and customization. The operation unit 140 has operation keys for accepting user operations and an LCD panel for displaying various settings and user interface screens, and outputs information such as accepted user operations to the control device 110. The image forming device 100 is not limited to the configuration shown in FIG. 1, and may include other configurations depending on the functions that the image forming device 100 can execute. For example, it may include a configuration necessary for executing a FAX function or a configuration that enables short-range wireless communication.

The server 191 includes a control device 198, an operation unit 195, a storage device 196, and a display unit 197. The control device 198 is a control board (controller) that controls the server 191 in an integrated manner. The control device 198 includes a CPU 192, a ROM 193, and a RAM 194. The CPU 192 controls each block in the control device 198 via a system bus (not shown). For example, the CPU 192 executes the functions of the server 191 by reading and executing a program stored in the ROM 193, the RAM 194, the storage device 196, or another storage medium. The ROM 193 stores, for example, various control programs such as an operating system program (OS), and tables and setting data necessary for executing the functions of the server 191. The RAM 194 is used, for example, as a work memory for the CPU 192. The storage device 196 stores, for example, various application programs, data, user information, device information, and the like. The operation unit 195 includes a keyboard, a pointing device, and the like for accepting user operations, and outputs information on accepted user operations, etc., to the control device 198. The display unit 197 is, for example, a liquid crystal display, and displays various user interface screens and information.

[Functional Configuration of Image Forming Apparatus]
2 is a diagram for explaining a functional configuration related to the extraction process of metadata items of the image forming apparatus 100. The image forming apparatus 100 of this embodiment has an image data acquisition unit 201, a character string area detection unit 202, a character recognition unit 203, a handwritten character separation unit 204, a character information extraction unit 205, and a display control unit 206.

The image data acquisition unit 201 functions as an image acquisition unit that acquires read image data generated by the reader device 120 reading a document. The character string area detection unit 202 performs processing to detect character string areas present in the read image. The character string area detection method may be a known method, such as a method of extracting a rectangular area that is assumed to be a character from an image that has been binarized using a certain threshold value.

The character recognition unit 203 performs optical character recognition (OCR) processing on the character string region. The character recognition unit 203 of this embodiment can perform OCR processing in an appropriate manner according to the classification of the character string region. For example, handwritten characters have a lower density than printed characters, and there is a large difference in the character shape depending on the writer. For this reason, when performing OCR processing on handwritten characters, the character recognition unit 203 performs OCR processing in a manner different from that used for OCR processing on printed characters. As a method for character recognition of handwritten characters, OCR processing is performed by increasing the contrast during binarization processing and strengthening noise removal. Alternatively, the character recognition unit 203 may perform OCR processing dedicated to handwritten characters using a recognition engine trained by a neural network including the characters before and after the characters, rather than judging each character based on the image feature points. OCR processing dedicated to handwritten characters can improve the recognition accuracy of both handwritten characters and printed characters.

The handwritten character separation unit 204 performs a process to separate handwritten characters from the read image, and generates a handwritten character image representing the handwritten characters in the read image and an image other than handwritten characters. One method of separating handwritten character images from a read image is to register an image of a form template, which is an image other than handwritten characters, in advance, and derive the difference between that image and the image of the read image data. The method of separating handwritten characters is not limited to this method. Another method that can be used is, for example, to have a neural network learn about handwritten character string areas and other background areas in the read image data, and generate a handwritten character image and an image other than handwritten characters by determining whether each pixel or area is a handwritten character.

The character information extraction unit 205 acquires form data, which will be described later, and identifies character string areas registered in the form data that have a similar layout to the character string areas in the scanned image. The character information extraction unit 205 then performs processing to extract character strings corresponding to metadata items such as document name, name, and telephone number from the character strings obtained as a result of the character recognition processing by the character recognition unit 203. The processing of the character information extraction unit 205 will be described in detail later.

The display control unit 206 controls the display of the confirmation screen and the correction screen on the liquid crystal panel of the operation unit 140. The confirmation screen and the correction screen will be described later.

The functions of each part in FIG. 2 are realized by the CPU 111 expanding the program code stored in the ROM 112 or the storage device 150 into the RAM 113 and executing it. Alternatively, some or all of the functions of each part in FIG. 2 may be realized by hardware such as an ASIC or electronic circuit.

[Regarding forms that are subject to character information extraction processing]
The image forming apparatus 100 performs a character information extraction process to extract character strings of specified items such as the document name, name, and telephone number from the image data of the scanned image obtained by reading a document that contains handwritten and printed characters, such as an application form.

In the character information extraction process, character string areas that are similar in layout to the character string areas in the scanned image are identified from the positional information of the character string areas for each form registered in the database. Then, a character string in the scanned image that has been recognized from the same area as the character string area of the item to be extracted that is associated with the character string area of the form data is extracted as the character string of the specified item. If no similar character string area is registered, the form indicated by the scanned image is registered as a new form.

Figure 3 shows a form 300 with no handwritten characters. As form 300 shows, the manuscript to be read is based on a form that is handwritten in either the "individual member" writing area 301 or the "corporate member" writing area 302. The handwritten content differs between the "individual member" and "corporate member" writing areas of the form, but there are also common items such as "telephone number" and "email address."

The manuscript handwritten in the "individual member" entry area 301 of the form 300 and the manuscript handwritten in the "corporate member" entry area 302 have multiple areas for the same item, such as "phone number" and "email address." For this reason, it is not possible to determine whether the character string the user wishes to extract is a character string for an item included in entry area 301 or a character string for an item included in entry area 302, and there is a risk that the desired character string will not be extracted.

In this embodiment, the information on the character string areas of printed characters and the information on the character string areas of handwritten characters contained in the scanned image are separated and each piece of information is registered in a database. When extracting a specific item from the scanned image, the document type is determined based on the information on the character string areas of printed characters contained in the scanned image and the information on the character string areas of handwritten characters contained in the scanned image, and the character string of the specific item in the scanned image is extracted.

[Regarding form data registration]
4 is a diagram showing an example of form data indicating information of a registered form (document) stored in the server 191. In the image forming apparatus 100, character strings of metadata items included in a scanned image are extracted using the form data.

The form data includes registered image information and metadata item information. Specifically, the image information includes image data, handwritten character BS, and printed character BS information. The scanned image 400 shows the data of the scanned image of the registered document. The handwritten character BS information and printed character BS information are positional information that show the arrangement of the character string area of the registered document. BS refers to block selection.

The handwritten character BS image 401 is an image that represents handwritten character BS information, which is position information of a character string area (handwritten character string area) that contains handwritten characters of a registered document. The solid-line rectangular area in the handwritten character BS image 401 indicates the position and size of the handwritten character string area. The printed character BS image 402 is an image that represents printed character BS information, which is position information of a character string area (printed character string area) that contains printed characters of a registered document. The solid-line rectangular area in the printed character BS image 402 indicates the position and size of the printed character string area.

In this way, in this embodiment, the position information of the printed character string area and the handwritten character string area are registered as information on the character string area for the registered document. Hereinafter, the handwritten character BS image refers to the position information of the character string area of handwritten characters, and the printed character BS image refers to the position information of the character string area of printed characters.

The dashed rectangles in each of the handwritten character BS image 401 and the printed character BS image 402 indicate the location information of the character string area containing the character string of the metadata item. The metadata items in this embodiment are "document name," "application date," "name," "corporate name," and "telephone number," and location information corresponding to each item is registered as metadata item information. Location information 403 indicates the location corresponding to each of the items "document name," location information 404 indicates the location corresponding to "application date," location information 405 indicates the location corresponding to "name," and location information 406 indicates the location corresponding to each of the items "telephone number." In addition, information is registered to the effect that the location information for "corporate name" is unknown.

In this embodiment, the image forming device 100 is configured to obtain the registered form data from the server 191 each time a scan process is performed, thereby allowing registered information to be shared among multiple image forming devices. Alternatively, the registered form data may be stored in the storage device 150 of each image forming device 100.

[Text information extraction process flow]
Fig. 5 is a flowchart for explaining the details of the character information extraction process. The series of processes shown in the flowchart in Fig. 5 are performed by CPU 111 of image forming apparatus 100 expanding program code stored in ROM 112 into RAM 113 and executing it. In addition, some or all of the functions of the steps in Fig. 5 may be realized by hardware such as an ASIC or electronic circuit. Note that the symbol "S" in the explanation of each process indicates a step in the flowchart, and the same applies to the following flowcharts.

In S501, when the CPU 111 receives an instruction to scan a document from a user via the operation unit 140, it instructs the reader device 120 to scan the document. The reader device 120 scans the document and generates read image data corresponding to the document. The image data acquisition unit 201 then acquires the read image data of the document from the reader device 120.

Figure 6 is a diagram showing an example of data generated as a result of the processing of this flowchart. The scanned image 600 in Figure 6 is a scanned image obtained by scanning the document entered in the "Individual Member" entry area 301 of the form 300.

Figure 7 is a diagram showing an example of data generated in the process of this flowchart for a manuscript different from the manuscript used to obtain the data shown in Figure 6. The scanned image 700 in Figure 7 is a scanned image obtained by scanning a manuscript filled in the "Corporate Member" field 302 of the form 300.

In S502, the handwritten character separation unit 204 performs a process of separating handwritten characters from the read image data generated in S501, and generates a handwritten character image and an image other than handwritten characters. Note that before the process of S502 is performed, a position adjustment process of the read image may be performed by determining skew that occurs during the scanning process, image misalignment when the document is set, and differences in image orientation. By performing a position adjustment process of the read image, the accuracy of extracting the difference image can be improved. By performing the process of this step on the read image 600 in FIG. 6, a handwritten character image 601 from which handwritten characters are extracted and an image other than handwritten characters 602 are obtained. In addition, by performing the process of this step on the read image 700 in FIG. 7, a handwritten character image 701 and an image other than handwritten characters 702 are obtained. Since the read image 600 and the read image 700 are images obtained by scanning a document on which handwritten writing is written on the same form of document, the image other than handwritten characters 602 and the image other than handwritten characters 702 are essentially the same image.

In S503, processing for character recognition is performed on the images 602, 702 other than the handwritten characters generated in S502. In this step, the character string area detection unit 202 performs block division processing before OCR processing, analyzes the form structure of the image, and performs BS processing to divide the image into a background area and rectangular areas (blocks) that indicate the character string areas.

Printed character BS images 604, 704 are images that, for the purpose of explanation, show the coordinate information (position information) of the character string area obtained as a result of performing BS processing on images 602, 702 other than handwritten characters. The character string area represented by a rectangle in the printed character BS images 604, 704 is the printed character string area. Since the images 602, 702 other than handwritten characters are the same image, the printed character BS image 604 and the printed character BS image 704 are the same image.

The character recognition unit 203 then performs OCR processing on each character string area of the printed character BS images 604 and 704. By obtaining the character code of the character string in the character string area, there is no need to perform OCR processing on areas other than the character string area. This reduces the processing load in OCR processing and improves the accuracy of character recognition.

In S504, processing for character recognition is performed on the handwritten character images 601, 701 generated in S502. As in S503, the character string area detection unit 202 performs BS processing on the handwritten character images 601, 701 before OCR processing. Handwritten character BS images 603, 703 are diagrams that, for the purpose of explanation, represent as images the coordinate information of the handwritten character string area obtained as a result of performing BS processing on the handwritten character images 601, 701. The rectangles in the diagrams represent the handwritten character string areas.

The character recognition unit 203 performs OCR processing on each character string area to obtain the character code of the character string in the character string area. The OCR processing method in this step may be the same as the OCR processing method in S503, or may be an OCR processing method different from S503 that is specialized for character recognition of handwritten characters.

In S505, the character information extraction unit 205 acquires the form data of the registered document described with reference to FIG. 4, which is stored in the storage device 196 of the server 191, via the I/F of the document management function of the server 191.

In S506, the character information extraction unit 205 determines whether a print character string area with a similar layout to the print character string area of the scanned image is registered in the form data. If a print character string area with a similar layout is registered in the form data, the process proceeds to S507; if not, the process proceeds to S510.

In S507, the character information extraction unit 205 determines whether a handwritten character string area with a similar layout to the handwritten character string area in the scanned image is registered in the form data. If a handwritten character string area with a similar layout is registered in the form data (YES in S507), the process proceeds to S508.

In the determinations at S506 and S507, character string areas with similar layouts are identified based on the degree of similarity of the layouts of the character string areas. For example, at S506, the pixels of the print character string area based on the coordinate information of the print character string area in the scanned image are compared with the pixels of the print character string area based on the coordinate information of each print character string area registered in the form data. If the comparison shows that there is a print character string area registered in the form data whose degree of similarity exceeds a threshold, the layout of the print character string area in the scanned image is identified as similar to the layout of that print character string area registered in the form data.

One way to identify character string areas with similar layouts is to compare the image data themselves, but by using the coordinate information of the character string areas, the processing load for similarity determination can be reduced. Also, by registering coordinate information as position information of the character string areas in the form data, the data size of the information registered as form data can be reduced.

In S508, the character information extraction unit 205 acquires coordinate information indicating the position of each metadata item that is stored in association with the position information of the character string area registered in the form data and that was identified as similar in S506 and S507. Then, a character string recognized from the character string area of the scanned image that corresponds to the acquired coordinates is extracted as the character string for that item. The extracted character string is associated with the item and output to the display control unit 206 to display a screen for the user to confirm.

Figure 8 is a diagram showing an example of a confirmation screen 800 that allows the user to confirm the character strings for each metadata item extracted by the character information extraction unit 205. The confirmation screen is a screen that is displayed on the liquid crystal panel of the operation unit 140. The confirmation screen 800 in Figure 8 is an image that is displayed as a result of performing the processes of S502 to S508 on the scanned image 600 in Figure 6. The scanned image 600 is displayed in a preview display area 801. The character strings for each item extracted by the character information extraction unit 205 are displayed in metadata item display areas 803 to 807.

At the start of this flowchart, the contents registered in the form data are as shown in FIG. 4. In this case, the layout of the printed character string area shown by the printed character BS image 604 in the scanned image 600 is determined to be similar to the layout of the printed character string area shown by the registered printed character BS image 402 in FIG. 4. Also, the layout of the handwritten character string area shown by the handwritten character BS image 603 in the scanned image 600 is determined to be similar to the layout of the handwritten character string area shown by the registered handwritten character BS image 401 in FIG. 4.

Therefore, coordinates indicating the positions of the metadata items surrounded by dashed lines in the registered printed character BS image 402 and handwritten character BS image 401 in Figure 4 are obtained. Then, among the character strings obtained as a result of OCR processing of the scanned image, the character string recognized in the character string area at the position of the metadata item is determined as the character string of the metadata item. As a result, the confirmation screen 800 in Figure 8 is displayed.

Therefore, in the metadata item display area 803 on the confirmation screen 800, the character string in the character string area indicated by the location information 403 of the scanned image 600 is extracted and displayed as the document name character string "Membership application form". Similarly, "January 31, 2020" is extracted from the location information 404 as the application date character string, and "Shimomaruko Taro" is extracted from the location information 405 as the name character string, and these are displayed on the confirmation screen. In addition, "03-3758-xxxx" is extracted from the location information 406 as the telephone number character string, and these are displayed on the confirmation screen. In this way, the extracted metadata item character strings can be displayed on the confirmation screen.

If a character string area similar in layout to the printed character string area has been registered, but a character string area similar in layout to the handwritten character string area has not been registered (NO in S507), the process proceeds to S509. In S509, the character information extraction unit 205 acquires coordinate information indicating the position of each metadata item stored in association with the printed character string area identified as similar. Then, the character string recognized from the character string area of the scanned image indicated by the acquired coordinates is extracted as the character string for that item. The extracted character string is associated with the item and output to the display control unit 206 to display a screen for the user to confirm.

Figure 9 is a diagram showing an example of a confirmation screen that allows the user to confirm the character strings for each item extracted by the character information extraction unit 205 in S509. The confirmation screen 900 in Figure 9 is an image that is displayed as a result of performing the processes of S502 to S507 and S509 on the scanned image 700 in Figure 7. The scanned image 700 is displayed in a preview display area 901, and the character strings of the metadata items extracted in S509 are displayed in metadata item display areas 903 to 907.

At the start of this flowchart, the contents registered in the form data are as shown in FIG. 4. In this case, when the scanned image from which character information is extracted is scanned image 700, it is determined that the layout of the printed character string area indicated by printed character BS image 704 of scanned image 700 is similar to the layout of the printed character string area indicated by registered printed character BS image 402 in FIG. 4. However, the layout of the handwritten character string area indicated by handwritten character BS image 703 of scanned image 700 is not similar to the layout of the handwritten character string area indicated by registered handwritten character BS image 401. For this reason, in the case of scanned image 700, in S507 it is determined that the handwritten character string area similar in layout to the handwritten character string area of the scanned image is not registered in the form data. In this case, unlike the case of scanned image 600, the character strings of some metadata items are not extracted.

In the metadata item display area 903 on the confirmation screen 900, the character string in the print character string area indicated by the position information 403 in the scanned image 700 is extracted and displayed as the document name character string "Membership application". Similarly, "January 31, 2020" is extracted and displayed as the application date character string from the print character string area indicated by the position information 403. However, in the registered printed character BS image 402, there are no character string areas in the position information 405, 406 for the name and phone number items. For this reason, the character information extraction unit 205 extracts only the character strings of some of the metadata items from the scanned image 700.

At S510, the display control unit 206 displays the confirmation screen described above on the liquid crystal panel of the operation unit 140 and accepts correction or registration operations from the user.

In the file name display areas 802, 902 of the confirmation screens 800, 900 in Figures 8 and 9, file names using metadata items are displayed based on a preset naming rule. For example, assume that the naming rule is set to "Document name_Application date_Name.pdf" or "Document name_Application date_Corporate name.pdf". In the confirmation screen 900 in Figure 9, neither the character strings "Name" nor "Corporate name", which are metadata items required for naming a file name, have been extracted and are blank. For this reason, an "*" is displayed in the file name to indicate that metadata has not been obtained.

Confirmation screens 800 and 900 display an Item Modification button 508 and a Registration Complete button 509. When the user presses the Item Modification button 508, a modification screen is displayed for modifying the character strings displayed in each metadata item display area. The modification screen can accept modifications to metadata items from the user. Also, in the confirmation screen 900 of Figure 9, some of the metadata required for the file name are blank, so the Registration Complete button 509 is grayed out and is set so that the user cannot press it.

Figure 10 is a diagram showing a correction screen displayed on the liquid crystal panel of the operation unit 140 after a metadata item correction operation is accepted. The correction screen 1000 in Figure 10 is the correction screen when the metadata item display area 906 for "Corporate Name" on the confirmation screen 900 in Figure 9 is selected and the item correction button 508 is pressed.

On the correction screen 1000, character string areas included in the scanned image that can be selected by the user are highlighted. In this embodiment, character string areas that can be selected by the user as character strings for metadata items are displayed as dashed frames superimposed on the scanned image 700. When the user selects the frame 1010 indicating "Corporate Name", the character string "XXX Co., Ltd." is displayed in the metadata item display area 1006 as a result of character recognition from the character string area indicated by the frame 1010. In this way, the character string for an item that the character information extraction unit 205 was unable to extract can be obtained by accepting an instruction from the user.

When the character string for the metadata item of the corporate name is accepted, a file name is determined based on the naming rules. This allows the user to press the registration completion button 509. Similarly, in the metadata item display area 1007, the selection of box 1011 by the user is accepted, and so the character string for the telephone number "03-3756-xxxx" is displayed.

In S511, when the display control unit 206 accepts the user's pressing of the registration completion button 509, information indicating the content of the correction instruction is sent to the storage device 196 of the server 191 via the I/F of the document management function of the server 191. Furthermore, if the printed characters BS indicating the printed character string area of the read image have not been registered, the image data of the read image and information on the printed characters BS and handwritten characters BS are sent as newly registered information. Furthermore, if the handwritten characters BS indicating the handwritten character string area of the read image have not been registered, the image data of the read image and information on the handwritten characters BS are sent as newly registered information.

The server 191 registers the corrections or new information in the form data stored in the storage device 196. If the form data is stored in the image forming device 100, the image forming device 100 registers the corrections or new information in the form data.

Figure 11 is a diagram showing updated form data. Figure 11 shows an example of form data that is updated as a result of performing the processing of this flowchart on the scanned image 700, from the form data shown in Figure 4 at the start of this flowchart.

As can be seen by comparing FIG. 11 with FIG. 4, the image data of scanned image 700 has been registered as new information. Scanned image 700 is a document of the same form as registered scanned image 400, but the writing area is different, so it has been registered as new information. Furthermore, as handwritten character BS image 1101 shows, the coordinate information of the handwritten character string area of scanned image 700, which is different from handwritten character BS image 401, is newly registered.

Also, as can be seen by comparing FIG. 11 with FIG. 4, although new information indicated by scanned image 700 has been newly registered, the position information of the print character string area of scanned image 700 has not been added to FIG. 11. Scanned image 700 is an image based on the same form as scanned image 400. In such a case, printed character BS image 402 indicating the same position information as the position information of the print character string area of the newly registered scanned image has already been registered. For this reason, it is not necessary to register the position information of the print character string area of scanned image 700.

In this manner, in this embodiment, information common to multiple images, such as information about the print character string area indicated by the print character BS image, is registered as shared information. This makes it possible to reduce the data size of the form data. In addition, the data in the search range is also reduced, making it possible to reduce the processing load during searches.

If the determination in S506 is NO, the handwritten character string area and the printed character string area that are similar in layout to the handwritten character string area and the printed character string area of the scanned image are not registered in the form data. Therefore, after this flowchart ends, the position information of the handwritten character string area and the printed character string area of the scanned image are registered as form data for a new document.

In addition, location information 1107 for "corporate name" and location information 1108 for "phone number", which were not registered in the form data of FIG. 4, are registered as location information for metadata items in FIG. 11. Location information for metadata items may be registered for each registered scanned image, but in this embodiment, location information for common metadata items is registered in association with multiple scanned images that share information on the print character string area. In other words, location information 403 to 406 and location information 1107 and 1108 for metadata items are associated with handwritten character BS image 401 of scanned image 400 and handwritten character BS image 1101 of scanned image 700 in FIG. 11. In this way, by associating location information for common metadata items with multiple scanned images, the data size of the form data to be registered can be reduced.

As shown in FIG. 11, by registering scanned image 700 as new information, two pieces of location information, location information 406 and location information 1108, are registered for the metadata item "phone number". In this way, when multiple character string areas of the same item are registered, the character string of that item is extracted from the one in which the character string area is detected in the scanned image to be processed. Alternatively, whether location information 406 or location information 1108 is used as the location information for "phone number" may be switched depending on whether the checkboxes located at "individual member" and "corporate member" are checked. Alternatively, it may be switched depending on whether the location information of the handwritten character string area of the scanned image to be processed is similar to the registered handwritten character BS image 401 or handwritten character BS image 1101.

When newly registering scanned image data, instead of a normal PDF incorporating the scanned image data, the data obtained by converting the text information resulting from OCR processing into a document format such as a full-text searchable PDF may be registered as the scanned image data. Alternatively, the scanned image data may be registered in its original image format such as JPEG or PNG.

As described above, according to this embodiment, the position information of the printed character string area and the position information of the handwritten character string area of the scanned image are used to extract character strings of metadata items. This makes it possible to extract desired character strings from scanned images of documents that were written in different writing areas of the same form.

In the above description, the processing of the flowchart in FIG. 5 is performed on the image forming apparatus 100. Alternatively, in order to distribute the processing load, the scanned image data generated in S501 may be transmitted to the server 191 via the network 190, and the server 191 may perform processing other than accepting operations from the user.

In the above description, OCR processing is performed following BS processing in S503 and S504, but OCR processing may be performed after the character string area of the metadata item is extracted. In this case, OCR processing may be performed only on the character string area of the extracted metadata item. If a similar character string area cannot be identified from the form data, or if there is a metadata item from which a character string cannot be extracted, OCR processing may be performed on the entire detected character string area.

<Embodiment 2>
In this embodiment, a character information extraction process using a method different from that of the first embodiment will be described. The present embodiment will be described focusing on the differences from the first embodiment. The configuration and process are the same as those of the first embodiment unless otherwise specified.

Figure 12 is a flowchart for explaining the details of the character information extraction process of this embodiment. S1201 is the same process as S501, so the explanation will be omitted.

In S1202, the character string area detection unit 202 performs BS processing on the scanned image data acquired in S1201 to detect the position of the character string area in the scanned image.

In S1203, the character information extraction unit 205 acquires the form data of the registered document stored in the storage device 196 of the server 191 via the I/F of the document management function of the server 191.

In S1204, the character information extraction unit 205 compares the position information of the character string area in the scanned image obtained as a result of the BS processing in S1202 with the position information of the print character string area registered in the form data. Then, it is determined whether a print character string area whose layout is similar to that of the character string area in the scanned image is registered in the form data.

The method of determining similarity in this step may be the same as the method described in embodiment 1. Note that, in order to improve the accuracy of the determination, the determination process in this step may be performed after excluding pixels of handwritten characters from among the pixels based on the coordinate information of the character string area of the scanned image.

If a print character string area similar to the character string area of the scanned image has not been registered (NO in S1204), the processes in S1207 to S1209 are performed. The processes in S1207 to S1209 are similar to the processes in S502 to S504, and therefore will not be described. Note that the same processes as S506 to S509 may be performed between S1209 and S1210.

In this manner, in this embodiment, processing similar to the processing in S502 to S504 is performed only if the form data does not contain a printed character string area whose layout is similar to that of a character string area contained in the scanned image. Processing time can be reduced by limiting the cases in which the process separates handwritten character images from the scanned image and the OCR process is performed on each image.

On the other hand, if a printed character string area similar to the character string area of the scanned image is registered in the form data (YES in S1204), the process proceeds to S1205. In S1205, the character recognition unit 203 acquires position information of the metadata item associated with the character string area registered in the form data that has been identified as similar. The character recognition unit 203 then performs OCR processing on the character string area of the scanned image that corresponds to the position of the metadata item. If it is possible to determine whether the character string area in the scanned image is a handwritten character string area or a printed character string area, the OCR processing method may be switched for each character string area.

In S1206, the character information extraction unit 205 extracts character strings obtained as a result of OCR processing from the character string area of each metadata item as character strings for each item. The extracted character strings are associated with the items and output to the display control unit 206 to display a screen for the user to confirm. If there is a metadata item from which a character string cannot be extracted, the processes of S1207 to S1209 may be performed to make it possible to register a character string in the handwritten character string area. When the process of S1206 or S1209 is completed, the processes of S1210 to S1211 are performed. The processes of S1210 to S1211 are similar to the processes of S510 to S511, and therefore will not be described.

As described above, according to this embodiment, processing time can be reduced by limiting the target of OCR processing to character string areas of metadata items.

<Other embodiments>
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

100 Image forming device 111 CPU
201 Image data acquisition unit 202 Character string area detection unit 205 Character information extraction unit

Claims (11)

an image acquisition means for acquiring a scanned image obtained by scanning a document;
a separation means for separating an image of handwritten characters and an image of an area other than the handwritten characters included in the read image from the read image;
a detection means for detecting character string regions in each image obtained by the separation, and detecting a character string region of handwritten characters included in the read image and a character string region of printed characters included in the read image, respectively ;
a data acquisition means for acquiring form data including first position information of a character string area of handwritten characters for each of a plurality of areas for writing handwritten characters and second position information of a character string area of printed characters for each of the areas of a document on which handwritten characters are written;
a specifying means for specifying that an arrangement of the character string area of the document and an arrangement of the character string area of the scanned image are similar when a degree of coincidence between pixel information of the character string area of the document based on position information included in the form data and pixel information of the character string area of the scanned image exceeds a threshold value ;
an extraction means for extracting, when a character string region of printed characters on the form based on the second position information included in the form data and a character string region of printed characters detected from the read image are identified as being similar, a character string of a predetermined item associated with the character string region of handwritten characters in the form data from a character string region of handwritten characters detected from the read image which has been further identified as being similar to the character string region of handwritten characters on the form based on the first position information included in the form data;
13. An image processing device comprising: The extraction means includes:
2. The image processing device according to claim 1, wherein a character string included in a character string area of the scanned image that corresponds to the character string area of the specified item among the character string areas of printed characters and handwritten characters of the form data is extracted as the character string of the specified item . character recognition means for performing character recognition processing on a character string area included in the read image;
The extraction means includes:
3. The image processing device according to claim 1, wherein a character string obtained as a result of character recognition by the character recognition means for a character string area of handwritten characters detected from the scanned image that corresponds to a character string area of the specified item among character string areas of the form data is extracted as the character string of the specified item . 4. The image processing apparatus according to claim 3 , wherein the character recognition means performs character recognition processing on a character string area of handwritten characters by a method different from the method used for character recognition processing on a character string area of printed characters. The character recognition means
5. The image processing device according to claim 3, wherein when the identification means is unable to identify the specified items, or when there is an item among the specified items from which the extraction means cannot extract a character string , character recognition processing is performed on all character string areas detected by the detection means. a receiving means for receiving a character string area corresponding to the predetermined item included in the scanned image based on an instruction from a user;
6. The image processing device according to claim 1, further comprising a transmission means for transmitting data for registering information indicating an arrangement of character string areas of printed characters and character string areas of handwritten characters included in the scanned image and information indicating positions of the character string areas corresponding to the specified items in the form data. The display control means further includes a display control unit that controls displaying the read image on a display unit while emphasizing a character string area of the read image,
7. The image processing device according to claim 6, wherein the accepting unit accepts a character string area selected by a user from among the character string areas displayed in a highlighted manner on the display unit as the character string area of the scanned image corresponding to the predetermined item. The method further includes registering the data transmitted from the transmitting means into the form data,
The registration means includes:
8. The image processing device according to claim 6, wherein, when position information of a character string area of printed characters detected from the read image is registered in the form data and position information of a character string area of handwritten characters detected from the read image is not registered in the form data, the position information of the character string area of handwritten characters detected from the read image is registered as new information. The image processing device according to claim 6 or 7 ,
a server having a registration means for registering data transmitted from the transmission means of the image processing device in the form data;
An image forming system comprising: an image acquisition step of acquiring a scanned image obtained by scanning the document;
a separation step of separating an image of handwritten characters and an image of an area other than handwritten characters included in the read image from the read image;
a detection step of detecting character string regions in each image obtained by the separation, and detecting a character string region of handwritten characters included in the read image and a character string region of printed characters included in the read image, respectively;
a data acquisition step of acquiring form data including first position information of a character string area of handwritten characters for each of a plurality of areas for writing handwritten characters and second position information of a character string area of printed characters for each of the areas of a document on which handwritten characters are written;
a specifying step of specifying that an arrangement of the character string area of the form and an arrangement of the character string area of the read image are similar when a degree of coincidence between pixel information of the character string area of the form based on position information included in the form data and pixel information of the character string area of the read image exceeds a threshold value ;
an extraction step of extracting a character string of a predetermined item associated with the character string area of handwritten characters of the form data from a character string area of handwritten characters detected from the read image which has been further specified as being similar to a character string area of handwritten characters of the form data based on the first position information included in the form data when the character string area of printed characters of the form based on the second position information included in the form data and the character string area of printed characters detected from the read image have been specified as being similar;
13. An image processing method comprising: A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 8 .

Images