JP2024079918A - Image processing system, image processing method and image processing program - Google Patents

Abstract

To provide an image processing system, an image processing method and an image processing program that appropriately extract an item value of an extraction object from image data.SOLUTION: In an image processing system, an extraction processing unit of an image processing apparatus extracts a plurality of item names and a plurality of item values corresponding to each of the plurality of item names from image data P1. An identification processing unit identifies a plurality of related item names associated with an object item name of a plurality of item names B1 through B3 extracted by the extraction processing unit, and identifies a plurality of related item values corresponding to each of the plurality of related item names, of the plurality of item values A2, A3 and A6 extracted by the extraction processing unit. An output processing unit outputs a calculated item value that is calculated on the basis of the plurality of related item values identified by the identification processing unit as an object item value.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to an image processing system, an image processing method, and an image processing program that perform image processing such as character recognition on image data.

Conventionally, there are known techniques for extracting character strings from image data of documents such as forms. For example, there is known a technique for extracting item values and item names from character strings obtained by character recognition of image data, determining an item value corresponding to an item name based at least on the positional relationship between the item value and the item name, and generating a combination of one item name and one or more item values such that the determined item value cannot be associated with multiple item names (see, for example, Patent Document 1).

JP 2021-99688 A

However, in conventional technology, combinations of item values and item names are generated based on the positional relationship between the item values and item names, so if the positional relationship between the item values and item names does not match specified rules, a problem occurs in that the combination of item values and item names to be extracted cannot be correctly identified.

The objective of the present disclosure is to provide an image processing system, an image processing method, and an image processing program that are capable of appropriately extracting item values to be extracted from image data.

An image processing system according to one aspect of the present disclosure is an image processing system that extracts target item values corresponding to target item names to be extracted from image data. The image processing system includes an extraction processing unit, the identification processing unit, and the output processing unit. The extraction processing unit extracts a plurality of item names and a plurality of item values corresponding to each of the plurality of item names from the image data. The identification processing unit identifies a plurality of related item names related to the target item name from the plurality of item names extracted by the extraction processing unit, and identifies a plurality of related item values corresponding to each of the plurality of related item names from the plurality of item values extracted by the extraction processing unit. The output processing unit outputs a calculated item value calculated based on the plurality of related item values identified by the identification processing unit as the target item value.

An image processing method according to another aspect of the present disclosure is an image processing method for extracting target item values corresponding to target item names to be extracted from image data. In the image processing method, one or more processors execute the following steps: extracting from the image data a plurality of item names and a plurality of item values corresponding to each of the plurality of item names; identifying a plurality of related item names related to the target item name from the plurality of item names; identifying a plurality of related item values from the plurality of item values corresponding to each of the plurality of related item names; and outputting a calculated item value calculated based on the plurality of related item values as the target item value.

An image processing program according to another aspect of the present disclosure is an image processing program that extracts target item values corresponding to target item names to be extracted from image data. The image processing program is a program for causing one or more processors to execute the following operations: extracting multiple item names and multiple item values corresponding to each of the multiple item names from the image data; identifying multiple related item names related to the target item name from the multiple item names; identifying multiple related item values from the multiple item values corresponding to each of the multiple related item names; and outputting a calculated item value calculated based on the multiple related item values as the target item value.

According to the present disclosure, it is possible to provide an image processing system, an image processing method, and an image processing program that are capable of appropriately extracting item values to be extracted from image data.

FIG. 1 is a functional block diagram showing a configuration of an image processing system according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of an estimate included in a form according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of character recognition processing on image data according to an embodiment of the present disclosure. FIG. 4 is a diagram illustrating an example of character recognition processing on image data according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating an example of an extraction result page displayed on the operation terminal according to the embodiment of the present disclosure. FIG. 6 is a diagram illustrating an example of an extraction result page displayed on the operation terminal according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating an example of an extraction result page displayed on the operation terminal according to the embodiment of the present disclosure. FIG. 8 is a flowchart for explaining an example of the procedure of an amount extraction process executed in the image processing system according to the embodiment of the present disclosure. FIG. 9 is a flowchart for explaining an example of the procedure of a price determination process executed in the image processing system according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example of a method for extracting price candidates executed in the image processing system according to an embodiment of the present disclosure.

Embodiments of the present disclosure will be described below with reference to the attached drawings. Note that the following embodiments are examples of the present disclosure and are not intended to limit the technical scope of the present disclosure.

[Image processing system 10]
1 is a block diagram showing a configuration of an image processing system 10 according to an embodiment of the present disclosure. The image processing system 10 includes an image processing device 1 and an operation terminal 2. The image processing device 1 and the operation terminal 2 are connected to each other via a network N1 (e.g., the Internet, a LAN, etc.). The image processing system 10 may include a plurality of operation terminals 2.

In the image processing system 10, the image processing device 1 acquires image data of documents such as forms sent from the operation terminal 2, and extracts target item values corresponding to the target item names to be extracted from the image data. For example, the operation terminal 2 scans paper forms such as estimates, order forms, invoices, and delivery notes, and sends the generated image data (e.g., PDF data) to the image processing device 1. The operation terminal 2 also creates a document file of the form based on a user's operation, for example, using a word processing application, and sends the document file to the image processing device 1 as image data (e.g., searchable PDF data (image + text data)). When the image processing device 1 receives the image data sent from the operation terminal 2, it executes various processes to be described later on the image data to extract target item values corresponding to the target item names to be extracted contained in the form. For example, the image processing device 1 extracts the estimated amount of the estimate (estimated amount including tax, estimated amount excluding tax, etc.), the order amount of the order (order amount including tax, order amount excluding tax, etc.), the billing amount of the invoice (billing amount including tax, billing amount excluding tax, etc.), the total amount of the delivery note (total amount including tax, total amount excluding tax, etc.), etc. Furthermore, the image processing device 1 registers the target item values corresponding to the extracted target item names in a specified database. For example, each time the image processing device 1 acquires image data of a quote, it registers the quote details (e.g., product name, quote issue date, company name, etc.) and the estimated amount of the quote in a database that manages quotes. This allows paper forms to be saved as electronic data.

The image processing system 10 is an example of an image processing system according to the present disclosure. Note that the image processing system according to the present disclosure may be configured with the image processing device 1 alone.

[Image processing device 1]
1, the image processing device 1 includes a control unit 11, a storage unit 12, an operation display unit 13, and a communication unit 14. The image processing device 1 may be one or more cloud servers, or one or more physical servers.

The communication unit 14 is a communication interface for connecting the image processing device 1 to the network N1 by wire or wirelessly and for executing data communication with the operation terminal 2 via the network N1 in accordance with a predetermined communication protocol. The network N1 is composed of, for example, the Internet, a LAN, etc.

The operation display unit 13 is a user interface that includes a display unit such as a liquid crystal display or an organic EL display that displays various information, and an operation unit such as a mouse, keyboard, or touch panel that accepts operations.

The storage unit 12 is a non-volatile storage unit such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory that stores various information. The storage unit 12 stores a control program such as an amount extraction program (an example of an image processing program of the present disclosure) for causing the control unit 11 to execute the amount extraction process (see Figures 8 and 9) described below. For example, the amount extraction program is non-temporarily recorded on a computer-readable recording medium such as a CD or DVD, and is read by a reading device (not shown) such as a CD drive or DVD drive provided in the image processing device 1 and stored in the storage unit 12. The amount extraction program may be distributed from a cloud server and stored in the storage unit 12.

The memory unit 12 also stores image data (e.g., scanned data) of documents such as forms acquired from the operation terminal 2.

Figure 2 shows an estimate as an example of a document. As shown in Figure 2, an estimate includes multiple items such as the document type ("estimate"), issue date, contact information for the estimator (address, telephone number, fax number, person in charge), estimated amount, product name, quantity, standard price, discount amount, subtotal, consumption tax, and total amount. The user scans the estimate on the operation terminal 2 and uploads image data P1 to the image processing device 1. When the control unit 11 acquires the image data P1 of the estimate, it stores it in the memory unit 12. In another embodiment, the control unit 11 may acquire a document file of the estimate created on the operation terminal 2 and store the document file in the memory unit 12.

The control unit 11 has control devices such as a CPU, a ROM, and a RAM. The CPU is a processor that executes various types of arithmetic processing. The ROM pre-stores control programs such as a BIOS and an OS for causing the CPU to execute various types of processing. The RAM stores various types of information and is used as a temporary storage memory (work area) for the various types of processing executed by the CPU. The control unit 11 controls the image processing device 1 by having the CPU execute various control programs pre-stored in the ROM or the memory unit 12.

Specifically, as shown in FIG. 1, the control unit 11 includes various processing units such as an acquisition processing unit 111, an extraction processing unit 112, a specific processing unit 113, a determination processing unit 114, and an output processing unit 115. The control unit 11 functions as the various processing units by executing various processes according to the amount extraction program. Some or all of the processing units included in the control unit 11 may be configured with electronic circuits. The amount extraction program may be a program for causing multiple processors to function as the various processing units.

The acquisition processing unit 111 acquires image data P1 of the document to be processed. Specifically, when a user scans an estimate (see FIG. 2) on the operation terminal 2 and performs an upload operation, the acquisition processing unit 111 acquires image data P1 of the estimate. When the acquisition processing unit 111 acquires the image data P1, it stores it in the memory unit 12.

The extraction processing unit 112 extracts predetermined item names and item values from the image data P1. Specifically, the extraction processing unit 112 performs preprocessing such as resolution conversion processing, skew correction processing, and background removal processing on the image data P1, and then performs document part recognition processing (object recognition processing). For example, the extraction processing unit 112 recognizes each part, such as character strings, tables, illustrations, and seals, in the image data P1 on which the preprocessing has been performed as a rectangular area of an object. In addition, after performing the object recognition processing, the extraction processing unit 112 performs character recognition processing (OCR processing) on the entire image data P1 (entire image).

The extraction processing unit 112 also uses the results of the OCR processing to integrate the character data into a string (convert it into words). The extraction processing unit 112 also performs processing such as separating the item names and item values in the string and associating the item names with the item values.

For example, in the image data P1 of the quotation shown in FIG. 3, the extraction processing unit 112 determines the classification of the document. Specifically, the extraction processing unit 112 extracts character strings consisting of relatively large characters (for example, characters with a font size of 16 points or more) in the area above the vertical center position of the document, including "quotation," "order form," "invoice," and "delivery note." Here, the extraction processing unit 112 extracts the character string T1 of "quotation." In this way, the extraction processing unit 112 extracts document information from the image data P1.

The extraction processing unit 112 also extracts dates from the image data P1. Specifically, the extraction processing unit 112 extracts a character string that exists to the right of the character string "Issuance date" in the area above the vertical center of the document, where the characters "(Japanese calendar or nothing) (Gen- or 19xx-20xx) year (1-12) month (1-31) day" exist in this order (or numbers are separated by symbols such as "/", "-", or "." instead of the year, month, and date). Here, the extraction processing unit 112 extracts the character string D1 of "November 1, 2022". In this way, the extraction processing unit 112 extracts dates (transaction date, issue date) from the image data P1.

The extraction processing unit 112 also extracts monetary amount information from the image data P1. Specifically, the extraction processing unit 112 extracts numeric character strings as monetary amount candidates in the central area Ar1 (see FIG. 4), excluding the top and bottom of the document, according to a predetermined extraction rule. For example, the extraction processing unit 112 excludes character strings that include symbols other than numbers (such as "-", "/", and "〒"), and extracts character strings consisting only of numbers without including such symbols as monetary amount candidates. The extraction processing unit 112 may also extract character strings consisting of numbers and characters that represent monetary amounts, such as "￥", ","", "円", "JPY", and "金", as monetary amount candidates. In the example shown in FIG. 4, the extraction processing unit 112 excludes the character string E1 of the postal code, the character string E2 of the telephone number, and the character string E3 of the fax number, and extracts the character string A1 of "¥5,489,000", the character string A2 of "¥5,000,000", the character string A3 of "¥10,000", the character string A4 of "¥4,990,000", the character string A5 of "¥499,000", the character string A6 of "¥5,489,000", and the character string A7 of "¥5,489,000" as candidate amounts. In this way, the extraction processing unit 112 extracts candidate amounts from the image data P1.

In this way, the extraction processing unit 112 extracts multiple item names and multiple item values (e.g., price candidates) corresponding to each of the multiple item names from the image data P1.

The identification processing unit 113 identifies multiple item names (keywords) necessary to calculate the target item value from among the multiple items (character strings) extracted by the extraction processing unit 112. For example, when the estimated amount in the quotation (estimated amount including tax, estimated amount excluding tax) is set as the amount to be extracted (target item value), the identification processing unit 113 determines "standard price", "discount amount", and "consumption tax" as keywords for the item names necessary to calculate the estimated amount. The identification processing unit 113 then searches for character strings matching the keywords from among the multiple items extracted from the image data P1 by the extraction processing unit 112. In the example shown in FIG. 4, the identification processing unit 113 identifies the character string B1 for "standard price", the character string B2 for "discount amount", and the character string B3 for "consumption tax" in the image data P1.

In addition, when the identification processing unit 113 identifies the keyword (related item name), it identifies the item value (amount candidate) corresponding to the keyword based on the positional relationship with the keyword. Specifically, the identification processing unit 113 identifies the amount candidate that exists to the right or below the keyword as the amount candidate corresponding to the keyword. In the example shown in FIG. 4, since the character string A2 of "￥5,000,000" exists below the character string B1 of "standard price", the identification processing unit 113 identifies "￥5,000,000" as the amount candidate of "standard price". Furthermore, since the character string A3 of "￥10,000" exists below the character string B2 of "discount amount", the identification processing unit 113 identifies "￥10,000" as the amount candidate of "discount amount". Furthermore, since the character string A6 of "￥499,000" exists to the right of the character string B3 of "consumption tax", the identification processing unit 113 identifies "￥499,000" as the amount candidate of "consumption tax".

The identification processing unit 113 also identifies candidates for the target item value from among the multiple items (character strings) extracted by the extraction processing unit 112. Here, the identification processing unit 113 identifies the character string A7 of the estimated amount (total amount) "￥5,489,000" as a candidate for the estimated amount.

In this way, the identification processing unit 113 identifies multiple related item names related to the target item name (estimated amount in the above example) from among the multiple item names extracted by the extraction processing unit 112, and identifies multiple related item values corresponding to each of the multiple related item names (standard price, discount amount, and consumption tax in the above example) from among the multiple item values (e.g., potential amounts) extracted by the extraction processing unit 112.

The determination processing unit 114 determines the amount to be extracted. For example, the determination processing unit 114 determines the estimated amount (estimated amount including tax and estimated amount excluding tax) based on the multiple amount candidates identified. Specifically, the determination processing unit 114 calculates the estimated amount including tax and estimated amount excluding tax (calculation item value) based on the amount candidates (character strings A2, A3, A6) corresponding to the standard price, discount amount, and consumption tax, respectively. For example, the determination processing unit 114 subtracts the second largest amount from the largest amount among the amount candidates identified by the identification processing unit 113, and if the difference does not match the consumption tax amount, the determination processing unit 114 subtracts the next (third) largest amount from the largest amount to calculate the difference. If the calculated difference matches the consumption tax amount, the determination processing unit 114 determines the largest amount to be the estimated amount including tax and the second (third) largest amount to be the estimated amount excluding tax. In the example shown in FIG. 2, the difference between the largest amount "¥5,489,000" and the second largest amount "¥5,000,000" does not match the consumption tax amount "¥499,000", and the difference between the largest amount "¥5,489,000" and the next (third) largest amount "¥4,990,000" matches the consumption tax amount "¥499,000", so the determination processing unit 114 determines that the largest amount "¥5,489,000" is the estimated amount including tax, and excludes the second largest amount "¥5,000,000" and determines that the next largest amount "¥4,990,000" is the estimated amount excluding tax. The method of determination by the determination processing unit 114 will be described in detail later.

The output processing unit 115 outputs the calculated item value calculated based on the multiple related item values identified by the identification processing unit 113 as the target item value. For example, as shown in FIG. 5, the output processing unit 115 displays an extraction result page P2 on the operation terminal 2. The output processing unit 115 also displays the calculated item value determined by the determination processing unit 114 on the extraction result page P2, in this case the estimated amount including tax ("¥5,489,000") and the estimated amount excluding tax ("¥4,990,000")

In this way, the output processing unit 115 outputs the calculated item value calculated based on the multiple related item values identified by the identification processing unit 113 as the target item value (the estimated amount in the above example).

The output processing unit 115 also displays the image data P1 on the extraction result page P2, and identifiably displays character strings corresponding to the estimated amount (estimated amount including tax, estimated amount excluding tax) extracted from the image data P1. For example, as shown in FIG. 5, the output processing unit 115 displays rectangular frames superimposed on "￥4,990,000" (character string A5) and "￥5,489,000" (character string A7).

In addition, the output processing unit 115 may display image data P1 on the extraction result page P2, and may also display character strings corresponding to the standard price, discount amount, and consumption tax (related item values) used to calculate the estimated amount in an identifiable manner using a rectangular frame or the like.

The output processing unit 115 also displays an "OK" button K1 on the extraction result page P2. On the extraction result page P2, the user checks whether the estimated amount (estimated amount including tax, estimated amount excluding tax) calculated from the image data P1 is correct, and presses the "OK" button K1 if they determine that it is correct. When the user presses the "OK" button K1, the output processing unit 115 registers the determined (calculated) estimated amount (estimated amount including tax, estimated amount excluding tax) and the estimate details (e.g., product name, date of issue of the estimate, company name, etc.) in the database.

If the user determines that the calculated estimated amount (tax-inclusive estimated amount, tax-exclusive estimated amount) is incorrect, the user corrects the estimated amount and presses the "OK" button K1.

Furthermore, when the user checks the "Show all character string candidates" checkbox on the extraction result page P2, the output processing unit 115 causes all character strings identified by the identification processing unit 113 to be displayed in an identifiable manner, as shown in FIG. 6. This allows the user to check each of the identified character strings. The user can also correct specific parts of the character strings (character recognition results) on the extraction result page P2 shown in FIG. 6. For example, the user can change the size and position of the rectangular area to correct parts that were not recognized as character strings or parts where character recognition errors occurred. FIG. 7 shows the rectangular area after it has been corrected (character strings C1 to C7).

Here, the control unit 11 may have a function of learning the correction content when the rectangular area (character string rectangular area) is corrected. For example, the control unit 11 displays the "Learn" button K2 as selectable when the user corrects the size and position of a rectangular area or adds a new rectangular area on the extraction result page P2. Note that the "Learn" button K2 cannot be selected before the user corrects the rectangular area (see Figures 5 and 6). When the user presses the "Learn" button K2 on the extraction result page P2, the control unit 11 learns the correction content and corrects (relearns) the learning model. This makes it possible to improve the accuracy of character string recognition (object recognition) for the image data P1.

As described above, the image processing device 1 extracts character strings that resemble amounts from the image data P1, and from the extracted character strings, identifies related item values (in the above example, standard price, discount amount, and consumption tax) that correspond to the target item name (in the above example, "estimated amount"), and outputs the target item values (in the above example, estimated amount including tax and estimated amount excluding tax) based on the related item values.

[Operation Terminal 2]
1, the operation terminal 2 includes a control unit 21, a storage unit 22, an operation display unit 23, and a communication unit 24. The operation terminal 2 is, for example, an image forming device or a multifunction peripheral equipped with an image forming function (e.g., a scanning function). Note that the operation terminal 2 may also be, for example, a personal computer, a smartphone, a tablet terminal, or the like.

The communication unit 24 is a communication interface that connects the operation terminal 2 to the network N1 by wire or wirelessly and performs data communication with an external device such as the image processing device 1 via the network N1 in accordance with a predetermined communication protocol.

The operation display unit 23 is a user interface that includes a display unit such as a liquid crystal display or an organic EL display that displays information such as various web pages, and an operation unit such as a mouse, keyboard, or touch panel that accepts operations.

The storage unit 22 is a non-volatile storage unit such as an HDD, SSD, or flash memory that stores various types of information. The storage unit 22 stores a control program that causes the control unit 21 to execute various processes. For example, the control program is non-temporarily recorded on a computer-readable recording medium such as a CD or DVD, and is read by a reading device (not shown) such as a CD drive or DVD drive provided in the operation terminal 2 and stored in the storage unit 22. The control program may be distributed from a cloud server or the image processing device 1 and stored in the storage unit 22.

The control unit 21 has control devices such as a CPU, a ROM, and a RAM. The CPU is a processor that executes various arithmetic processes. The ROM is a non-volatile storage unit in which control programs such as a BIOS and an OS for causing the CPU to execute various processes are stored in advance. The RAM is a volatile or non-volatile storage unit that stores various information, and is used as a temporary storage memory (work area) for various processes executed by the CPU. The control unit 21 controls the operation terminal 2 by executing the various control programs stored in advance in the ROM or the storage unit 22 with the CPU. Note that some or all of the processing units included in the control unit 21 may be composed of electronic circuits.

Specifically, the control unit 21 executes various processes according to the control program stored in the memory unit 22. For example, the control unit 21 scans paper documents such as estimates, purchase orders, invoices, and delivery notes, and transmits the generated image data (e.g., PDF data) to the image processing device 1. The control unit 21 also creates a document file of the document based on a user's operation using a word processing application or the like, and transmits the document file to the image processing device 1 as image data (e.g., searchable PDF data (image + text data)).

In another embodiment, the control unit 21 may execute a browser process in which a web page provided from the image processing device 1 via the network N1 is displayed on the operation display unit 23 and an operation on the operation display unit 23 is input to the image processing device 1. In this case, the control unit 21 executes an upload process of the image data on the web page.

The control unit 21 also displays the results of the amount extraction process performed by the image processing device 1. Specifically, the control unit 21 displays the extraction result page P2 shown in Figures 5 to 7 on the operation display unit 23. The control unit 21 also accepts various operations by the user on the extraction result page P2. For example, the control unit 21 accepts an operation to press the "OK" button K1, an operation to correct the extraction result (the estimated amount in the above example), an operation to correct the character string recognition result (rectangular area), an operation to press the "Learn" button K2, and the like. The control unit 21 transmits information corresponding to the user's operations to the image processing device 1.

As described above, the image processing device 1 executes the amount extraction process based on the image data P1 acquired from the operation terminal 2, and displays the result of the amount extraction process on the operation terminal 2. The image processing device 1 is capable of acquiring image data P1 from each of the multiple operation terminals 2, and executes the amount extraction process each time image data P1 is acquired.

[Amount extraction process]
An example of the procedure of the amount extraction process executed in the image processing system 10 will be described below with reference to FIGS.

The present disclosure can be understood as an amount extraction method (image processing method of the present disclosure) that executes one or more steps included in the amount extraction process. One or more steps included in the amount extraction process described here may be omitted as appropriate. The steps in the amount extraction process may be executed in a different order as long as the same effect is achieved. Furthermore, although an example is described here in which the control unit 11 of the image processing device 1 executes each step in the amount extraction process, in other embodiments, one or more processors may execute each step in the amount extraction process in a distributed manner. Furthermore, when the control unit 11 acquires image data P1 from each of the multiple operation terminals 2, it is possible for the control unit 11 to execute the amount extraction process in parallel for each image data P1.

First, in step S1, the control unit 11 determines whether or not image data P1 of the document to be processed has been acquired from the operation terminal 2. When the control unit 11 acquires the image data P1 from the operation terminal 2 (S1: Yes), the control unit 11 transitions the process to step S2. The control unit 11 waits until it acquires the image data P1 from the operation terminal 2 (S1: No). Here, it is assumed that the control unit 11 acquires image data P1 of a quotation (see FIG. 2) from the operation terminal 2.

In step S2, the control unit 11 executes object recognition processing (character string area recognition processing) and character recognition processing. Specifically, the control unit 11 executes preprocessing such as resolution conversion processing, skew correction processing, and background removal processing on the image data P1, and then executes document part recognition processing (object recognition processing). For example, the control unit 11 recognizes each part such as character strings, tables, illustrations, and seals in the image data P1 that has been preprocessed as a rectangular area of an object. After executing the object recognition processing, the control unit 11 also executes character recognition processing (OCR processing) on the entire image data P1. The control unit 11 also uses the results of the OCR processing to integrate the character data into character strings (wordization).

Next, in step S3, the control unit 11 executes an amount determination process. Figure 9 shows a specific example of the amount determination process.

In the amount determination process, first, in step S31, the control unit 11 extracts a string of numbers that resembles an amount (amount candidate). Here, an example of a method for extracting amount candidates will be described with reference to FIG. 10. FIG. 10 shows an example of the string "Total: ￥1,280" contained in the image data.

The control unit 11 uses the object recognition process to identify the top left coordinates (OBJ_X1[N], OBJ_Y1[N]) and bottom right coordinates (OBJ_X2[N], OBJ_Y2[N]) of the rectangular area (dotted frame in FIG. 10) of the character string ("Total ￥1,280").

The control unit 11 also identifies the upper left coordinates (OCR_X1[M], OCR_Y1[M]) and the lower right coordinates (OCR_X2[M], OCR_Y2[M]) of the rectangular area (solid line frame in FIG. 10) of the character ("￥") and the character ("1,280") from the results of the OCR processing.

The control unit 11 determines the following conditions for each of N×M combinations of "n: 0 to N-1, m: 0 to M-1".
・X1 = OBJ_X1[n] > OCR_X1[m]?
OBJ_X1[n]: OCR_X1[m] (get the larger coordinate value)
・Y1 = OBJ_Y1[n] > OCR_Y1[m]?
OBJ_Y1[n]: OCR_Y1[m] (gets the larger coordinate value)
・X2 = OBJ_X2[n] < OCR_X2[m]?
OBJ_X2[n]: OCR_X2[m] (gets the smaller coordinate value)
・Y2 = OBJ_Y2[n] < OCR_Y2[m]?
OBJ_Y2[n]: OCR_Y2[m] (obtains the smaller coordinate value)
If X1<X2 and Y1<Y2, it is determined that the rectangles overlap.

In addition, the control unit 11 calculates the overlap rate A of the rectangular areas using the following formula.
A = {(X2-X1) x (Y2-Y1)} / {(OCR_X2[m]-OCR_X1[m]) x (OCR_Y2[m]-OCR_Y1[m])}

When A ≧ Th1 (for example, Th1 = 0.8), the control unit 11 determines that the rectangular area obtained by the object recognition process and the rectangular area obtained by the OCR result are the same area.

The control unit 11 determines whether the OCR results of the rectangular area that became the same area contain any numeric characters (0 to 9), and then determines whether it is likely to be an amount based on various conditions such as "there is a '￥' at the beginning," "there is no '- (hyphen)' in the middle," "there is the presence of characters that are commonly used in amounts," and "there is no presence of characters that represent a postal code, telephone number, fax number, document number, product model number, etc." If it is determined to be likely to be an amount, it extracts it as a candidate amount string. In the above example, the control unit 11 extracts "￥1,280" as a candidate amount string. This makes it possible to eliminate misrecognition of places other than characters in the OCR process by narrowing down the area to those that were recognized as characters in the object recognition process.

For example, the control unit 11 searches for character strings corresponding to "standard price", "discount amount", "consumption tax", "account number (only for invoices)", and "various codes" (customer code, store code, etc.) from the results of the OCR processing, and extracts their position information. The control unit 11 then searches for numeric character strings (amount, account number, various codes, etc.) that exist to the right or below the position information of the character string, and extracts the nearest number as a candidate character string for the amount. The control unit 11 also excludes the account number and various codes from the candidate character strings for the amount. In the quotation shown in FIG. 2, if "standard price", "discount amount", and "consumption tax" are set as keywords, the control unit 11 extracts "￥5,000,000" that exists under the character string for "standard price", "￥10,000" that exists under the character string for "discount amount", and "￥499,000" that exists to the right of the character string for "consumption tax".

In this way, the control unit 11 uses the OCR results to extract character strings containing numbers from the character string position information obtained by the object recognition process, and determines whether the character string is likely to represent an amount (amount candidate).

Next, in step S32, the control unit 11 sets the item name (keyword) required to calculate the target item value. For example, if the estimated amount in the quotation (estimated amount including tax, estimated amount excluding tax) is the amount to be extracted (target item value), the control unit 11 sets "consumption tax" as the keyword.

Next, in step S33, the control unit 11 extracts a candidate amount corresponding to the keyword ("consumption tax"). For example, the control unit 11 searches for a character string containing "tax" among the multiple candidate amount amounts in the image data P1, and determines whether there is a candidate amount to the right or below the searched character string. If there is a candidate amount to the right or below the character string, the control unit 11 extracts it as the candidate amount corresponding to "consumption tax". Here, the control unit 11 extracts "￥499,000" that is to the right of the character string "tax" as the candidate amount.

Next, in step S34, the control unit 11 sets "discount amount" as the item name (keyword) required to calculate the target item value.

Next, in step S35, the control unit 11 extracts a candidate amount corresponding to the keyword ("discount amount"). For example, the control unit 11 searches for a character string containing "discount" among the multiple candidate amount values in the image data P1, and determines whether there is a candidate amount to the right or below the searched character string. If there is a candidate amount to the right or below the character string, the control unit 11 extracts it as the candidate amount corresponding to "discount amount". Here, the control unit 11 extracts "￥10,000" present below the character string "discount" as the candidate amount.

Next, in step S36, the control unit 11 sets "standard price" as the item name (keyword) required to calculate the target item value.

Next, in step S37, the control unit 11 extracts a price candidate corresponding to the keyword ("standard price"). For example, the control unit 11 searches for character strings including "standard" or "reference" among the multiple price candidates in the image data P1, and determines whether there is a price candidate to the right or below the searched character string. If there is a price candidate to the right or below the character string, the control unit 11 extracts it as the price candidate corresponding to "standard price". Here, the control unit 11 extracts "￥5,000,000" present below the character string "standard" as the price candidate.

Next, in step S38, the control unit 11 identifies the largest and second largest amounts from among the multiple amount candidates corresponding to the keyword. For example, if the control unit 11 finds a price candidate for "standard price," it excludes that price candidate, and if the control unit 11 finds a price candidate for "discount amount," it excludes the largest amount as the amount before discount if the difference between the largest and second largest amounts corresponds to the discount amount.

In the quotation shown in FIG. 2, first, the control unit 11 extracts the largest amount ("¥5,489,000"), the second largest amount ("¥5,000,000"), and the third largest amount ("¥4,990,000") from the price candidates extracted in step S31. Next, the control unit 11 determines whether or not there is a price candidate ("¥5,000,000") that corresponds to the "standard price" among these three amounts. In the example of FIG. 2, since the second largest amount ("¥5,000,000") matches the standard price candidate ("¥5,000,000"), the control unit 11 determines that the second amount ("¥5,000,000") is likely to be the standard price. Next, the control unit 11 subtracts the second largest amount ("¥5,000,000") from the largest amount ("¥5,489,000"), the third largest amount ("¥4,990,000") from the first largest amount ("¥5,489,000"), and the third largest amount ("¥4,990,000") from the second largest amount ("¥5,000,000"). If any of the differences obtained by each subtraction match the discount amount, the control unit 11 determines that the amount to be subtracted is the amount before the discount. If none of the differences obtained by each subtraction match the discount amount, the control unit 11 excludes only the amount that is determined to be the standard price if there is one. If there is no amount candidate that is determined to be the standard price, the control unit 11 leaves it as it is. In the example shown in FIG. 2, first, the difference ("¥489,000") obtained by subtracting the second largest amount from the first largest amount does not match the discount amount ("¥10,000"). Next, the difference ("¥499,000") obtained by subtracting the third largest amount from the first largest amount does not match the discount amount ("¥10,000"). Finally, the difference ("¥10,000") obtained by subtracting the third largest amount from the second largest amount matches the discount amount ("¥10,000"). Therefore, the control unit 11 determines that the second largest amount is the amount before the discount. The control unit 11 excludes the second largest amount ("¥5,000,000"), which may be the standard price and which has been determined to be the amount before the discount. The control unit 11 extracts the largest amount ("¥5,489,000") and the second largest amount ("¥4,990,000"), excluding the excluded amount ("¥5,000,000"). The control unit 11 then identifies "¥5,489,000" as the largest amount, and "¥4,990,000" as the second largest amount.

Next, in step S39, the control unit 11 determines the estimated amount (estimated amount including tax and estimated amount excluding tax). Here, if the amount obtained by subtracting the second largest amount ("¥4,990,000") from the largest amount ("¥5,489,000") matches the amount corresponding to "consumption tax" ("¥499,000"), the control unit 11 determines the largest amount ("¥5,489,000") as the estimated amount including tax, and determines the second largest amount ("¥4,990,000") as the estimated amount excluding tax.

If "consumption tax" is not present as a keyword, the control unit 11 finds the ratio between the largest amount and the second largest amount, and determines whether the found ratio corresponds to the consumption tax rate. In this case, if the control unit 11 has been able to extract the issue date from the top of the document from the OCR results in advance, it will determine the largest amount as the estimated amount including tax if the tax rate corresponding to that date matches the calculated tax rate according to the issue date, and will determine the second largest amount as the estimated amount excluding tax.

In addition, if the control unit 11 is able to extract a character string such as "10%" along with the keyword "consumption tax," and if the tax rate (percentage) in the character string matches the calculated tax rate, it will determine the largest amount as the estimated amount including tax, and the second largest amount as the estimated amount excluding tax.

Here, the control unit 11 may determine the amount according to the classification of the document. For example, if the document is a quotation or order form, if the keyword "consumption tax" is not present and the ratio between the largest amount and the second largest amount does not correspond to the consumption tax rate, the control unit 11 determines whether the character strings "excluding tax" and "including tax" are present around the position of the largest amount, and if "excluding tax" is present or neither is present, it determines that the amount including tax is not displayed, and determines the largest amount as the amount including tax. Also, if the character string "including tax" is present, the control unit 11 determines that the amount including tax is not displayed, and determines the largest amount as the amount including tax.

For example, if the document is an invoice or delivery note, and the keyword "consumption tax" is not present and the ratio between the largest amount and the second largest amount does not correspond to the consumption tax rate, the control unit 11 determines whether the character strings "excluding tax" and "including tax" are present around the position of the largest amount, and if "including tax" is present or neither is present, it determines that the amount excluding tax is not displayed and determines the largest amount as the amount including tax. Also, if the character string "excluding tax" is present, the control unit 11 determines that the amount including tax is not displayed and determines the largest amount as the amount excluding tax.

If "consumption tax" is set as the name of the item to be extracted, the control unit 11 determines the amount of consumption tax.

The control unit 11 executes the amount determination process in the above manner. When the control unit 11 determines the estimated amount (estimated amount including tax, estimated amount excluding tax), which is the target item value, in step S4 (see FIG. 8), it outputs the result of the amount determination process. Here, the control unit 11 causes the operation terminal 2 to display the extraction result page P2 shown in FIG. 5, and also causes the extraction result page P2 to display the estimated amount including tax ("¥5,489,000") and the estimated amount excluding tax ("¥4,990,000") of the estimated amount.

Next, in step S5, the control unit 11 determines whether or not it has received an operation from the user to press the "OK" button K1. When the user presses the "OK" button K1 on the extraction result page P2 (S5: Yes), the control unit 11 transitions the process to step S6. The user checks whether the estimated amount (estimated amount including tax, estimated amount excluding tax) is correct on the extraction result page P2, and if it is determined to be correct, presses the "OK" button K1. If the user does not press the "OK" button K1 on the extraction result page P2 (S5: No), the control unit 11 transitions the process to step S51.

In step S51, the control unit 11 determines whether or not a correction operation has been received from the user. If the user determines that the estimated amount (estimated amount including tax, estimated amount excluding tax) is incorrect, the user corrects the estimated amount to the correct amount. If the control unit 11 receives a correction operation from the user (S51: Yes), the control unit 11 corrects the estimated amount and redisplays it in step S52. Thereafter, the control unit 11 transitions the process to step S5.

In step S6, the control unit 11 registers the results of the amount determination process. Here, the control unit 11 registers the determined estimated amount (estimated amount including tax, estimated amount excluding tax) and the estimate contents (e.g., product name, date of issue of estimate, company name, etc.) in the database. Note that if the extraction target includes "consumption tax," the control unit 11 displays the consumption tax amount on the extraction result page P2 and also registers it in the database.

In this manner, the control unit 11 executes the amount extraction process. The control unit 11 executes the amount extraction process each time it acquires image data P1 from each operation terminal 2.

As described above, the image processing system 10 according to this embodiment is a system that extracts target item values corresponding to target item names to be extracted from image data P1. Furthermore, the image processing system 10 extracts multiple item names and multiple item values corresponding to each of the multiple item names from the image data P1, identifies multiple related item names (e.g., "standard price," "discount amount," "consumption tax") related to the target item name from the extracted multiple item names, and identifies multiple related item values corresponding to each of the multiple related item names from the extracted multiple item values. Then, the image processing system 10 outputs a calculated item value calculated based on the identified multiple related item values as the target item value.

For example, the control unit 11 extracts keywords such as "standard price", "discount amount", and "consumption tax" from the recognized character strings for the character string information (PDF character string information) of the image data P1 recognized as an estimate or order form, and further extracts the numeric character strings present to the right or below the keyword as the value (amount) corresponding to the keyword, and extracts all numeric character strings resembling amounts as candidate amounts from all recognized character strings. Then, when the keyword "standard price" or "discount amount" exists and the value is found, the control unit 11 excludes the amount related to the amount of the value from the candidate amounts, and then extracts the largest and second largest amounts from the candidate amounts, and determines the tax-inclusive amount and tax-exclusive amount from these two amounts and the amount of consumption tax or the consumption tax rate. Also, when the keyword "consumption tax" does not exist and the ratio between the first and second amounts does not correspond to the amount of consumption tax, the control unit 11 determines that the tax-inclusive amount is not displayed if the keywords "tax-inclusive" and "tax-exclusive" do not exist, and determines the first amount as the tax-exclusive amount.

For example, the control unit 11 extracts keywords such as "standard price", "discount amount", and "consumption tax" from the recognized character strings of the character strings (PDF character string information) of the image data P1 recognized as an invoice or delivery note, and further extracts the numeric character strings present to the right or below the keyword as the value (amount) corresponding to the keyword, and extracts all numeric character strings resembling amounts as candidate amounts from all recognized character strings. Then, when the keyword "standard price" or "discount amount" exists and the value is found, the control unit 11 excludes the amount related to the amount of the value from the candidate amounts, and then extracts the largest and second largest amounts from the candidate amounts, and determines the tax-inclusive amount and tax-exclusive amount from these two amounts and the consumption tax amount or consumption tax rate. Also, when the keyword "consumption tax" does not exist and the ratio between the first and second amounts does not correspond to the consumption tax amount, the control unit 11 determines that the tax-exclusive amount is not displayed if the keywords "tax included" and "tax excluded" do not exist, and determines the first amount as the tax-inclusive amount.

According to the above configuration, even if the positional relationship between the item value and the item name does not match the specified rules, it is possible to appropriately extract the item value to be extracted from the image data P1.

The present disclosure is not limited to the above-described embodiments. The present disclosure may include the following embodiments:

In another embodiment of the present disclosure, the control unit 11 may extract only a character string of numbers from the image data P1. For example, the control unit 11 learns character strings containing numbers such as amounts, telephone numbers, postal codes, company numbers, and document numbers as character strings of numbers in advance using a learning model. Then, when the control unit 11 acquires the image data P1, it executes an AI inference process to extract character strings of numbers and an OCR process for the entire image. Next, the control unit 11 extracts the amount of money using the result of the OCR process (with delimiter information for each character string) and the result of the extraction of the character string of numbers (the coordinate positions of the upper left and lower right of the character string rectangle) as inputs. Since the AI infers and outputs a rectangular area that resembles a character string of numbers from the characteristics of the image, if the learning accuracy improves, it will be possible to trust the output of the AI, perform word correction, and return to the correct number even if the numbers are misrecognized as alphabets or other symbols in the OCR process. Furthermore, if it becomes possible to infer only character strings that resemble amounts with even greater accuracy, part of the amount extraction process can be simplified, and the processing performance will improve along with the accuracy.

For example, the control unit 11 extracts keywords such as "standard price," "discount amount," and "consumption tax" from the recognized character strings of image data P1 recognized as an estimate, order form, delivery note, or invoice, and further extracts numeric character strings to the right or below the keyword as a value (amount) corresponding to the keyword, and extracts all numeric character strings that resemble amounts from the recognized numeric character string area as amount candidates. Then, when the keyword "standard price" or "discount amount" is present and a value is found, the control unit 11 excludes the amount associated with the value from the amount candidates, and then extracts the largest and second largest amounts from the amount candidates, and determines the tax-inclusive amount and tax-exclusive amount from these two amounts and the consumption tax amount or consumption tax rate. Furthermore, if the keyword "consumption tax" does not exist and the ratio between the first and second amounts does not correspond to the amount of consumption tax, and if the keywords "tax included" and "tax excluded" do not exist either, the control unit 11 determines that the amount including tax is not displayed in the case of a quotation or order form and determines the first amount to be the amount including tax, and determines that the amount excluding tax is not displayed in the case of an invoice or delivery note and determines that the amount excluding tax is not displayed in the case of an invoice or delivery note and determines the first amount to be the amount including tax.

In the above embodiment, the image processing device 1 alone corresponds to the image processing system according to the present disclosure, but the image processing system according to the present disclosure may be composed of the image processing device 1 and the operation terminal 2. For example, when the components of the image processing device 1 and the operation terminal 2 cooperate to share and execute the amount extraction process, a system including the multiple components that execute the process corresponds to the image processing system according to the present disclosure. Also, when the operation terminal 2 executes the amount extraction process, the operation terminal 2 alone may constitute the image processing system according to the present disclosure.

[Disclosure Addendum]
The following will provide an overview of the disclosure extracted from the above-described embodiment. Note that the configurations and processing functions described in the following supplementary notes can be selected and combined as desired.

<Appendix 1>
An image processing system for extracting a target item value corresponding to a target item name to be extracted from image data, comprising:
an extraction processing unit that extracts a plurality of item names and a plurality of item values corresponding to each of the plurality of item names from the image data;
an identification processing unit that identifies a plurality of related item names related to the target item name among the plurality of item names extracted by the extraction processing unit, and identifies a plurality of related item values corresponding to each of the plurality of related item names among the plurality of item values extracted by the extraction processing unit;
an output processing unit that outputs a calculated item value calculated based on the plurality of related item values identified by the identification processing unit as the target item value;
An image processing system comprising:

<Appendix 2>
the identification processing unit identifies, as the plurality of related item names, a plurality of item names necessary for calculating the target item value among the plurality of item names;
2. The image processing system of claim 1.

<Appendix 3>
the identification processing unit identifies an item value corresponding to the related item name based on a positional relationship between the item name and the item value;
3. The image processing system of claim 2.

<Appendix 4>
The identification processing unit identifies an item value present to the right or below the related item name as an item value corresponding to the related item name.
4. The image processing system of claim 3.

<Appendix 5>
The identification processing unit identifies a first item name and a second item name related to an amount, and identifies a first amount corresponding to the first item name and a second amount corresponding to the second item name;
The image processing system according to any one of appendices 1 to 4, wherein the output processing unit outputs a third amount calculated based on the first amount and the second amount as the target item value.

<Appendix 6>
The output processing unit displays the target item value on a display terminal.
6. An image processing system according to any one of claims 1 to 5.

<Appendix 7>
the output processing unit causes the display terminal to display the image data and to identifiably display an item value corresponding to the target item value in the image data.
7. The image processing system of claim 6.

<Appendix 8>
When an operation to modify the target item value is received from a user on the display terminal, the output processing unit re-displays the modified target item value.
8. The image processing system according to claim 6 or 7.

<Appendix 9>
the output processing unit causes the display terminal to display the image data and to identifiably display the plurality of related item values in the image data.
7. The image processing system of claim 6.

<Appendix 10>
displaying a result of character recognition processing on the image data on an operation terminal, and learning the correction content when an operation to correct the result of the character recognition processing is received from a user;
10. An image processing system according to any one of claims 6 to 9.

1: Image processing device 2: Operation terminal 10: Image processing system 11: Control unit 12: Storage unit 13: Operation display unit 14: Communication unit 21: Control unit 22: Storage unit 23: Operation display unit 24: Communication unit 111: Acquisition processing unit 112: Extraction processing unit 113: Identification processing unit 114: Determination processing unit 115: Output processing unit P1: Image data P2: Extraction result page

Claims (12)

An image processing system for extracting a target item value corresponding to a target item name to be extracted from image data, comprising:
an identification processing unit that identifies, in the image data including a plurality of item names and a plurality of item values corresponding to each of the plurality of item names, a plurality of related item names related to the target item name among the plurality of item names, and identifies a plurality of related item values corresponding to each of the plurality of related item names among the plurality of item values;
an output processing unit that outputs a calculated item value calculated based on the plurality of related item values identified by the identification processing unit as the target item value;
An image processing system comprising: the identification processing unit identifies, as the plurality of related item names, a plurality of item names necessary for calculating the target item value among the plurality of item names;
The image processing system according to claim 1 . the identification processing unit identifies an item value corresponding to the related item name based on a positional relationship between the item name and the item value;
The image processing system according to claim 2 . The identification processing unit identifies an item value present to the right or below the related item name as an item value corresponding to the related item name.
The image processing system according to claim 3 . The identification processing unit identifies a first item name and a second item name related to an amount, and identifies a first amount corresponding to the first item name and a second amount corresponding to the second item name;
The image processing system according to claim 1 , wherein the output processing unit outputs a third amount calculated based on the first amount and the second amount as the target item value. The output processing unit displays the target item value on a display terminal.
The image processing system according to claim 1 . the output processing unit causes the display terminal to display the image data and to identifiably display an item value corresponding to the target item value in the image data.
The image processing system according to claim 6. When an operation to modify the target item value is received from a user on the display terminal, the output processing unit re-displays the modified target item value.
The image processing system according to claim 6. the output processing unit causes the display terminal to display the image data and to identifiably display the plurality of related item values in the image data.
The image processing system according to claim 6. displaying a result of the object recognition processing on the image data on an operation terminal, and when an operation to correct a character string rectangular area that is the result of the object recognition processing is received from a user, learning the correction content;
The image processing system according to claim 6. An image processing method for extracting a target item value corresponding to a target item name to be extracted from image data, comprising:
extracting a plurality of item names and a plurality of item values corresponding to each of the plurality of item names from the image data;
Identifying a plurality of related item names related to the target item name among the plurality of item names, and identifying a plurality of related item values corresponding to each of the plurality of related item names among the plurality of item values;
outputting a calculated item value calculated based on the plurality of related item values as the target item value;
An image processing method executed by one or more processors. An image processing program for extracting a target item value corresponding to a target item name to be extracted from image data,
extracting a plurality of item names and a plurality of item values corresponding to each of the plurality of item names from the image data;
Identifying a plurality of related item names related to the target item name among the plurality of item names, and identifying a plurality of related item values corresponding to each of the plurality of related item names among the plurality of item values;
outputting a calculated item value calculated based on the plurality of related item values as the target item value;
An image processing program for causing one or more processors to execute the above.

Images