JP6785930B2 - Mobile devices, image processing methods, and programs - Google Patents

Description

The present invention relates to mobile terminals, image processing methods, and programs.

Conventionally, a technique for performing image collation has been disclosed.

Here, a technique is disclosed in which a plurality of feature points of an input form are matched with a plurality of feature points of a registered form to perform rotation correction (see Patent Document 1).

Further, a technique is disclosed in which document features are extracted from a document image whose size is matched with a registered image, and the features of the extracted document are sequentially matched with features of a registered image whose size is matched with the document image (Patent Document). See 2).

In addition, a technology that determines whether or not the driver's license image is a processable driver's license image and outputs that the driver's license image cannot be processed in the case of an unprocessable driver's license image. It is disclosed (see Patent Document 3).

Further, a technique for determining which type of document is a digital image by comparing a digital image with one or more known dimensions of a plurality of known document types is disclosed (see Patent Document 4). ..

Japanese Unexamined Patent Publication No. 2012-98984 Japanese Unexamined Patent Publication No. 2001-325563 Japanese Unexamined Patent Publication No. 7-65118 Special Table 2015-503813

However, the conventional photographing apparatus (Patent Document 1 and the like) has a problem that matching takes time and high-precision form type determination cannot be performed at high speed.

The disclosed technology matches one partial feature point with the image to be judged, and if the image quality of the image to be judged is poor, it automatically re-shoots and retries the matching. The purpose is to carry out highly accurate form type judgment at high speed.

The mobile terminal according to the disclosure aspect includes a frame acquisition means for acquiring a frame taken by a photographing unit, a manuscript acquisition means for acquiring manuscript image data of a manuscript from the frame, and one partial feature of a registered form. The partial feature determining means for determining whether or not the document partial feature in the manuscript at the position corresponding to the position of the partial feature in the form matches, and the partial feature and the said by the partial feature determining means. It is characterized in that it is provided with a re-acquisition means for acquiring a frame by re-imaging of the photographing unit when it is determined that the features of the original portion do not match.

Further, the image processing method according to the disclosure aspect includes a frame acquisition step of acquiring a frame taken by the photographing unit, a manuscript acquisition step of acquiring the manuscript image data of the manuscript from the frame, and one part of the registered form. The partial feature determination step for determining whether or not the feature and the original partial feature in the manuscript at a position corresponding to the position of the partial feature in the form match, and the partial feature determination step described above. When it is determined that the partial feature and the document partial feature do not match, a re-acquisition step of acquiring a frame by re-imaging of the photographing unit is included.

Further, the program according to the mode of disclosure includes a frame acquisition step of acquiring a frame taken by the photographing unit, a manuscript acquisition step of acquiring the manuscript image data of the manuscript from the frame, and one partial feature of the registered form. , The partial feature determination step for determining whether or not the original partial feature in the manuscript at the position corresponding to the position of the partial feature in the form matches, and the partial feature determination step in the partial feature determination step. When it is determined that the features of the original document do not match the features of the original, the computer is made to perform a re-acquisition step of acquiring a frame by re-imaging of the photographing unit.

According to the aspect of disclosure, the target form is automatically determined from the captured image, and the image is acquired only when the image quality is such that OCR recognition is possible, so that the user does not have to take the trouble after the start of shooting. You can shoot the desired manuscript.

FIG. 1 is a block diagram showing an example of the configuration of a mobile terminal according to the present embodiment. FIG. 2 is a flowchart showing an example of processing in the mobile terminal of the present embodiment. FIG. 3 is a diagram showing an example of screen display in the present embodiment. FIG. 4 is a flowchart showing an example of processing in the mobile terminal of the present embodiment. FIG. 5 is a diagram showing an example of partial feature determination in the present embodiment. FIG. 6 is a diagram showing an example of overall feature determination in the present embodiment. FIG. 7 is a flowchart showing an example of processing in the mobile terminal of the present embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The present invention is not limited to this embodiment.

[Structure of the present embodiment]
Hereinafter, an example of the configuration of the mobile terminal 100 according to the embodiment of the present invention will be described with reference to FIG. 1, and then the processing and the like of the present embodiment will be described in detail. FIG. 1 is a block diagram showing an example of the configuration of the mobile terminal 100 according to the present embodiment.

However, the embodiments shown below exemplify the mobile terminal 100 for embodying the technical idea of the present invention, and are not intended to specify the present invention to the mobile terminal 100. It can be equally applied to the mobile terminal 100 of other embodiments included in the above scope.

Further, the form of functional distribution in the mobile terminal 100 illustrated in the present embodiment is not limited to the following, and is functionally or physically distributed / integrated in any unit within the range in which similar effects and functions can be obtained. be able to.

Here, the mobile terminal 100 is a portable information processing device such as a tablet terminal, a mobile phone, a smartphone, a PHS, a PDA, a notebook personal computer, or a wearable computer such as a glasses type or a clock type. It may be a device.

First, as shown in FIG. 1, the mobile terminal 100 is roughly configured to include a control unit 102, a storage unit 106, a photographing unit 110, an input / output unit 112, a sensor unit 114, and a communication unit 116.

Here, although omitted in FIG. 1, in the present embodiment, an input / output interface unit (not shown) that connects the input / output unit 112 and the control unit 102 may be further provided. Each part of these mobile terminals 100 is communicably connected via an arbitrary communication path.

Here, the communication unit 116 is a network interface (NIC (Network Interface Controller) or the like) for transmitting and receiving IP data by wired communication and / or wireless communication (WiFi or the like), Bluetooth (registered trademark), infrared communication or the like. It may be an interface for wireless communication.

Here, the mobile terminal 100 may be communicably connected to an external device via a network by using the communication unit 116.

Further, the sensor unit 114 detects a physical quantity and converts it into a signal (digital signal) of another medium. Here, the sensor unit 114 includes a proximity sensor, a direction sensor, a magnetic field sensor, a linear acceleration sensor, a brightness sensor, a gyro sensor, a pressure sensor, a gravity sensor, an acceleration sensor, a pressure sensor, and / or a temperature sensor and the like. May be good.

Further, the input / output unit 112 performs input / output (I / O) of data. Here, the input / output unit 112 may be, for example, a key input unit, a touch panel, a control pad (for example, a touch pad, a game pad, etc.), a mouse, a keyboard, and / or a microphone.

Further, the input / output unit 112 may be a display unit (for example, a display composed of a liquid crystal or an organic EL or the like, a monitor, a touch panel, or the like) that displays a display screen of an application or the like.

Further, the input / output unit 112 may be an audio output unit (for example, a speaker or the like) that outputs audio information as audio. Further, the input / output unit (touch panel) 112 may include a sensor unit 114 that detects physical contact and converts it into a signal (digital signal).

Further, the photographing unit 110 acquires continuous (moving image) image data (frames) by continuously photographing (moving a moving image) a subject (for example, a document or the like). For example, the photographing unit 110 may acquire video data. Further, the photographing unit 110 may acquire ancillary data.

Here, the photographing unit 110 may be a camera or the like provided with an image sensor such as a CCD (Charge Coupled Device) and / or a CMOS (Complementary Metal Oxide Semiconductor).

Further, the photographing unit 110 may acquire the image data of the still image by photographing the subject as a still image. Here, the frame may be uncompressed image data. Further, the frame may be high-resolution image data.

Here, the high resolution may be full high-definition, 4K resolution, super high-definition (8K resolution), or the like. Further, the photographing unit 110 may shoot a moving image at 24 fps, 30 fps or the like. Further, the photographing unit 110 may be a camera.

The storage unit 106 is a storage means, and for example, a memory such as a RAM / ROM, a fixed disk device such as a hard disk, an SSD (Solid State Drive), and / or an optical disk or the like can be used.

In addition, the storage unit 106 stores various databases, tables, buffers, and / or files (feature data file 106a, image data file 106b, etc.). Here, the storage unit 106 may record a computer program or the like for giving instructions to the CPU (Central Processing Unit) to perform various processes.

Among the constituent elements of the storage unit 106, the feature data file 106a stores (registers) feature data related to the features of a plurality of forms. Here, the features may be partial features and / or overall features of the form.

Here, the partial feature may be composed of a ruled line of a form, characters, and / or a characteristic image or the like. Here, the characteristic image may be a picture image, a character image, a face image, a background image, or the like. Moreover, the whole feature may show a plurality of partial features.

Further, the image data file 106b stores image data (frames and the like). Here, the image data file 106b may store the original image data, the area image data of a part of the original, and / or the divided area image data of the divided area of the original.

Further, the image data file 106b may store the original and / or the position data such as the divided area. Further, the image data file 106b may store character data corresponding to the image data.

For example, the image data file 106b may store the form type identified from the image data and / or the character data relating to the characters described in the form. Further, the image data file 106b may store video data. Further, the image data file 106b may store ancillary data.

Further, the control unit 102 is a CPU, GPU (Graphics Processing Unit), DSP (Digital Signal Processor), LSI (Large Scale Integration), ASIC (Application Special Circuit), and ASIC (Application Special Circuit) , FPGA (Field-Programming Gate Array) and the like may be composed of a tangible controller.

Further, the control unit 102 has an internal memory for storing a control program, a program defining various processing procedures, and required data, and performs information processing for executing various processes based on these programs. Do.

Here, the control unit 102 functionally conceptually includes a frame acquisition unit 102a, a document acquisition unit 102b, a partial feature determination unit 102c, an overall feature determination unit 102d, a reacquisition unit 102e, an image display unit 102f, and an OCR unit 102g. To be equipped with.

The frame acquisition unit 102a acquires image data (frame). Here, the frame acquisition unit 102a may acquire image data taken by the photographing unit 110. Further, the frame acquisition unit 102a may acquire uncompressed and high-resolution image data.

Further, the frame acquisition unit 102a may control continuous image shooting or moving image shooting by the shooting unit 110 to acquire image data (frame) corresponding to one frame. Further, the frame acquisition unit 102a may control the still image photography by the photographing unit 110 to acquire image data. Further, the frame acquisition unit 102a may acquire ancillary data.

The document acquisition unit 102b acquires the document image data of the document. Here, the document acquisition unit 102b may acquire the document image data of the document from the frame. Further, the document acquisition unit 102b may acquire area image data of a part of the document from the frame. Here, the original may be a rectangular original.

Further, the document acquisition unit 102b may acquire the divided area image data of the divided area of the document from the frame. Here, the document acquisition unit 102b may detect the position data of the document from the frame. Here, the document acquisition unit 102b may detect the corner coordinates (4 points) of the document from the frame.

Further, the document acquisition unit 102b may detect the layout of the document from the frame. Further, the document acquisition unit 102b may detect the position data of the document from the frame by using the edge detection method and / or the feature point matching method, and acquire the document image data based on the position data of the document. Good.

The partial feature determination unit 102c determines whether or not the partial feature of the form registered in the feature data file 106a matches the document partial feature in the manuscript at the position corresponding to the position of the partial feature in the form. To judge.

Here, the partial feature determination unit 102c has one partial feature of the form registered in the feature data file 106a, and a document partial feature in the manuscript at a position corresponding to the position of the partial feature in the form. It may be determined whether or not they match.

Further, the partial feature determination unit 102c matches one partial feature of the form registered in the feature data file 106a with the document partial feature in the manuscript at a position corresponding to the position of the partial feature in the form. By comparing the degree with the threshold value, it may be determined whether or not the partial feature and the original partial feature match.

Further, the partial feature determination unit 102c uses the position (coordinates), color gradation value, number and / or of the ruled lines, characters and / or characteristic images constituting the partial features of the form registered in the feature data file 106a. Whether or not the partial feature and the original partial feature match by comparing the degree of matching between the partial feature and the original partial feature in the manuscript at the position corresponding to the position of the partial feature with the threshold value based on the size. May be determined.

Here, the size may be a length, a size, an angle, and / or a width, or a concept represented by a combination thereof (for example, a shape, etc.).

When the partial feature is a ruled line, the partial feature determination unit 102c is based on the position of the intersection coordinates of the ruled line, the gradation value of the color of the ruled line, the number of ruled lines, and / or the length and / or width of the ruled line. Therefore, the degree of coincidence between the partial feature and the partial feature of the original may be compared with the threshold value.

When the partial feature is a character, the partial feature determination unit 102c determines the position of the character, the gradation value of the color of the character, the number of characters, and / or the size, length, and / or width of the area of the character. The degree of agreement between the partial features and the original partial features may be compared with the threshold value based on the above.

When the partial feature is a characteristic image (for example, a picture image), the position of the characteristic image, the color gradation value of the characteristic image, the number of characteristic images, and / or the size of the characteristic image. , Length, width and / or shape, etc., and the degree of agreement between the partial feature and the original partial feature may be compared with the threshold value.

The overall feature determination unit 102d determines whether or not the overall feature of the form registered in the feature data file 106a matches the overall feature of the original manuscript.

Here, when the partial feature determination unit 102d determines that the partial feature and the original partial feature match, the overall feature determination unit 102d sets the overall feature of the form registered in the feature data file 106a and the overall feature of the original document. And may be determined whether or not they match.

Here, when the partial feature determination unit 102d determines that the partial feature and the original partial feature match, the overall feature determination unit 102d includes a plurality of partial features of the form registered in the feature data file 106a and the form. It may be determined whether or not the original partial features in the original at the positions corresponding to the positions of the respective partial features of the above match.

The reacquisition unit 102e acquires the frame used for the feature determination, or acquires the image data (frame) obtained by the re-imaging of the photographing unit 110. Here, the reacquisition unit 102e may further output pseudo shutter operation information.

Here, when the partial feature determination unit 102c determines that the partial feature and the original partial feature do not match, the reacquisition unit 102e may acquire the frame by re-imaging of the photographing unit 110.

Further, the reacquisition unit 102e may acquire a frame by re-imaging of the photographing unit 110 when the overall feature determining unit 102d determines that the overall feature and the entire document feature do not match.

Further, when the reacquisition unit 102e determines that the plurality of partial features and the original partial features do not match by the overall feature determination unit 102d, the reacquisition unit 102e may acquire the frame by re-imaging of the photographing unit 110.

Further, the reacquisition unit 102e is in the manuscript at a position corresponding to one partial feature of all the forms registered in the feature data file 106a by the partial feature determination unit 102c and the position of the partial feature in the form. If it is determined that the features of the original portion do not match, the frame may be acquired by re-imaging of the photographing unit 110.

Further, when the partial feature determination unit 102c determines that the partial feature and the original partial feature match, the reacquisition unit 102e acquires the frame used for the feature determination or re-photographs the frame of the photographing unit 110. It may be acquired, and at that time, pseudo shutter operation information may be output.

Here, when the form identified by the feature determination is a small form such as a driver's license or a business card, the reacquisition unit 102e may acquire the frame used for the feature determination as it is.

On the other hand, when the form identified by the feature determination is a form that requires high resolution such as A4, the reacquisition unit 102e acquires a new high resolution and uncompressed frame by re-imaging the photographing unit 110. You may.

Further, the reacquisition unit 102e may acquire a new frame by controlling the shooting (continuous image shooting, moving image shooting, still image shooting, etc.) by the shooting unit 110. Further, the reacquisition unit 102e may further store the acquired frame in the image data file 106b.

The image display unit 102f displays image data. Here, the image display unit 102f may display the frame and / or the original image data.

Further, the image display unit 102f may display the image data on the input / output unit 112. Further, the image display unit 102f may display character data.

The OCR unit 102g performs OCR processing on the image data and acquires character data. Here, the OCR unit 102g may perform OCR processing on the original image data or the frame to acquire character data. Further, the OCR unit 102g may further store the acquired character data in the image data file 106b.

[Processing of this embodiment]
An example of the form feature determination process in the present embodiment will be described with reference to FIGS. 2 to 7 with respect to an example of the process executed by the mobile terminal 100 having the above-described configuration.

[Form feature judgment processing (1)]
First, with reference to FIGS. 2 and 3, an example of the form feature determination process when there is one type of registered form in the present embodiment will be described. FIG. 2 is a flowchart showing an example of processing in the mobile terminal 100 of the present embodiment.

As shown in FIG. 2, first, the frame acquisition unit 102a activates the shooting unit (camera) 110 to control continuous image shooting or moving image shooting by the shooting unit 110 (step SA-1).

Then, the frame acquisition unit 102a inputs (acquires) a frame of the image captured by the camera 110 (step SA-2).

Then, the document acquisition unit 102b cuts out the document image data of the rectangular document from the frame acquired by the frame acquisition unit 102a by using the edge detection method and / or the feature point matching method (step SA-3).

Then, the partial feature determination unit 102c detects the document partial feature in the document at the position corresponding to the position of one partial feature of the form registered in the feature data file 106a (step SA-4).

Here, in the feature data file 106a, position information and feature information (ruled lines, character images, texts, picture images, face images, etc.) may be registered in advance for each form. Here, the color may be a gradation value such as RGB, HSV, Lab, or YUV.

That is, in the present embodiment, unique features and area information that cannot be found elsewhere may be registered as form data in advance.

Then, the partial feature determination unit 102c compares the degree of agreement between one partial feature of the form registered in the feature data file 106a and the detected original partial feature with the threshold value to obtain the partial feature and the original partial feature. (Step SA-5), it is determined whether or not the items match (the form for which the rectangular document is the target).

As described above, in the present embodiment, the cut-out image data is feature-detected, and the consistency with the feature of the form registered in the feature data file 106a in advance is determined.

Here, in the present embodiment, in the determination of consistency, it is determined whether or not the position of the intersection coordinates, the gradation value of the color, the number of ruled lines, the length or width, the number of characters, etc. satisfy the threshold value. Alternatively, it may be determined whether or not the images match.

That is, in the present embodiment, the partial feature determination may be made based on the position (coordinates, etc.), color, number, and / or size (length, size, and / or width, etc.) of the partial feature.

Then, when the partial feature determination unit 102c determines that the rectangular original is not the target form (step SA-5: No), the partial feature determination unit 102c makes an image reacquisition request and shifts the process to step SA-1.

On the other hand, when the partial feature determination unit 102c determines that the rectangular original is the target form (step SA-5: Yes), the process shifts to step SA-6.

That is, in the present embodiment, if the registered form is consistent, the matched form type information may be set and the process may be shifted to the image processing (OCR analysis or the like) in the subsequent stage.

Then, the reacquisition unit 102e acquires the frame used for the feature determination, or controls the shooting by the camera 110 to acquire a new high-resolution and uncompressed frame and stores it in the image data file 106b. At that time, the pseudo shutter operation information is output to the input / output unit 112 (step SA-6).

That is, in the present embodiment, if it is determined that the form is the target form, the pseudo shutter may be automatically released and the image data may be acquired.

Then, the OCR unit 102g acquires the character data by performing character recognition on the original image data included in the frame acquired by the reacquisition unit 102e or the frame by image processing (OCR processing) (step SA-). 7).

Then, the image display unit 102f sends the form name of the form identified by the feature determination, the character data acquired by the OCR unit 102g, and the manuscript image data included in the frame acquired by the reacquisition unit 102e to the input / output unit 112. Output (display) is performed (step SA-8), and the process ends.

Here, an example of screen display in the present embodiment will be described with reference to FIG. FIG. 3 is a diagram showing an example of screen display in the present embodiment.

As shown in FIG. 3, in the present embodiment, the form name (driver's license) of the form identified by the feature determination, the icon of the form, and the character information (name, date of birth) written on the form. , Address, expiration date, and / or license number, etc.) and manuscript image data may be displayed on the input / output unit (display) 112.

Then, in the present embodiment, the read information is passed (data transmission) to the service (system) or the like according to the form by the user operation on the screen after the display, thereby performing the desired data processing of the user. realizable.

As described above, in the present embodiment, high-speed retry processing is realized by performing partial feature determination.

[Form feature judgment processing (2)]
Next, with reference to FIGS. 4 to 6, an example of the form feature determination process when there are a plurality of types of registered forms in the present embodiment will be described. FIG. 4 is a flowchart showing an example of processing in the mobile terminal 100 of the present embodiment.

As shown in FIG. 4, first, the frame acquisition unit 102a activates the shooting unit (camera) 110 to control continuous image shooting or moving image shooting by the shooting unit 110 (step SB-1).

Then, the frame acquisition unit 102a inputs (acquires) a frame of the image captured by the camera 110 (step SB-2).

Then, the document acquisition unit 102b cuts out the document image data of the rectangular document from the frame acquired by the frame acquisition unit 102a by using the edge detection method and / or the feature point matching method (step SB-3).

Then, the partial feature determination unit 102c detects the document partial feature in the document at the position corresponding to the position of one partial feature of the form registered in the feature data file 106a (step SB-4).

Then, the partial feature determination unit 102c compares the degree of agreement between one partial feature of the form registered in the feature data file 106a and the detected original partial feature with the threshold value to obtain the partial feature and the original partial feature. (Step SB-5), it is determined whether or not the items match (the form for which the rectangular document is the target).

As described above, in the present embodiment, since the feature amount of the form is detected and determined, even the form of the same size can be identified with high accuracy.

Then, when the partial feature determination unit 102c determines that the rectangular document is not the target form (step SB-5: No), the process shifts to step SB-6.

Then, the partial feature determination unit 102c determines whether or not the determination of whether or not one partial feature of all the forms registered in the feature data file 106a and the detected original partial feature match is completed. (Step SB-6).

Then, when the partial feature determination unit 102c determines that the determination of whether or not one partial feature of all the registered forms and the detected original partial feature match is completed (step SB-6: Yes), the image reacquisition request is made, and the process shifts to step SB-1.

On the other hand, when the partial feature determination unit 102c determines that the determination of whether or not one partial feature of all the registered forms and the detected original partial feature match has not been completed (step SB-). 6: No), the type of the form registered in the feature data file 106a to be compared is changed, and the process is shifted to step SB-4.

Here, an example of partial feature determination in the present embodiment will be described with reference to FIG. FIG. 5 is a diagram showing an example of partial feature determination in the present embodiment.

As shown in FIG. 5, in the present embodiment, the consistency of the cut-out original image data with the partial features of the form registered in the feature data file 106a in advance may be determined.

As described above, in the present embodiment, since the consistency determination is a partial feature determination, the determination can be performed at high speed even if there are a plurality of registered forms, and the retry process at the time of error can be performed at high speed. it can.

Returning to FIG. 4, when the partial feature determination unit 102c determines that the rectangular document is the target form (step SB-5: Yes), the process shifts to step SB-7. That is, in the present embodiment, if the registered form is consistent, the process is shifted to the overall feature detection.

Then, the overall feature determination unit 102d detects the entire document feature in the manuscript (for example, the manuscript partial feature in the manuscript at a position corresponding to the position of the plurality of partial features of the form registered in the feature data file 106a). (Step SB-7).

Then, the overall feature determination unit 102d matches the overall feature of the form registered in the feature data file 106a and determined by the partial feature determination unit 102c to be the target form with the overall feature of the original manuscript ( It is determined whether or not the rectangular original is the target form (step SB-8).

Here, an example of overall feature determination in the present embodiment will be described with reference to FIG. FIG. 6 is a diagram showing an example of overall feature determination in the present embodiment.

In the present embodiment, in the overall feature determination, the overall feature of the entire image may be compared with the overall feature of the registered form. Here, as in the partial feature determination, comparison may be performed based on the feature data registered in the feature data file 106a.

As shown in FIG. 6, for example, in the case of a driver's license, in the case of the driver's license, layout determination such as name, date of birth, address, expiration date, and license number may be performed.

Returning to FIG. 4, when the overall feature determination unit 102d determines that the rectangular document is not the target form (step SB-8: No), the image reacquisition request is made, and the process shifts to step SB-1.

That is, in the present embodiment, if the partial feature determination is consistent with some partial features of the specific form, but the overall feature determination is not consistent, the image data of the first sequence is acquired (re-). Let's start over from (shooting).

On the other hand, when the overall feature determination unit 102d determines that the rectangular original is the target form (step SB-8: Yes), the process shifts to step SB-9.

That is, in the present embodiment, when consistency is obtained in the overall feature determination, the consistent form type information is set and the processing is shifted to image processing (OCR analysis or the like). Here, the overall feature determination unit 102d may acquire information necessary for image processing for each of the various forms.

Then, the reacquisition unit 102e acquires the frame used for the feature determination, or controls the shooting by the camera 110 to acquire a new high-resolution and uncompressed frame and stores it in the image data file 106b. At that time, the pseudo shutter operation information is output to the input / output unit 112 (step SB-9).

Then, the OCR unit 102g acquires the character data by performing character recognition on the original image data included in the frame acquired by the reacquisition unit 102e or the frame by image processing (OCR processing) (step SB-). 10).

Then, the image display unit 102f outputs the form name identified by the feature determination, the character data acquired by the OCR unit 102g, and the manuscript image data included in the frame acquired by the reacquisition unit 102e to the input / output unit 112 ( (Display) (step SB-11), and the process ends.

As described above, in the present embodiment, the form identification can be performed accurately and at high speed by performing the feature determination in two stages.

[Form feature judgment processing (3)]
Then, with reference to FIG. 7, an example of the form feature determination process when there are a plurality of types of registered forms in the present embodiment will be described. FIG. 7 is a flowchart showing an example of processing in the mobile terminal 100 of the present embodiment.

First, since the processing of steps SC-1 to SC-7 shown in FIG. 7 is the same as the processing of steps SB-1 to SB-7 shown in FIG. 4, the description thereof will be omitted.

Then, the overall feature determination unit 102d matches the overall feature of the form registered in the feature data file 106a and determined by the partial feature determination unit 102c to be the target form with the overall feature of the original manuscript ( It is determined whether or not the rectangular original is the target form (step SC-8).

Then, when the overall feature determination unit 102d determines that the rectangular document is the target form (step SC-8: Yes), the process shifts to step SC-9.

Here, since the processing of steps SC-9 to SC-11 shown in FIG. 7 is the same as the processing of steps SB-9 to SB-11 shown in FIG. 4, the description thereof will be omitted.

On the other hand, when the overall feature determination unit 102d determines that the rectangular original is not the target form (step SC-8: No), the overall feature determination unit 102d requests image reacquisition and shifts the process to step SC-12.

Then, the reacquisition unit 102e activates the shooting unit (camera) 110 to control continuous image shooting or moving image shooting by the shooting unit 110 (step SC-12).

Then, the reacquisition unit 102e inputs (acquires) a frame of the image captured by the camera 110 (step SC-13).

Then, the document acquisition unit 102b cuts out the document image data of the rectangular document from the frame acquired by the frame acquisition unit 102a by using the edge detection method and / or the feature point matching method (step SC-14).

Then, the overall feature determination unit 102d detects the overall feature of the document in the document (step SC-15).

Then, the overall feature determination unit 102d matches the overall feature of the form registered in the feature data file 106a and determined by the partial feature determination unit 102c to be the target form with the overall feature of the original manuscript ( It is determined whether or not the rectangular original is the target form (step SC-16).

Then, when the overall feature determination unit 102d determines that the rectangular document is not the target form (step SC-16: No), the overall feature determination unit 102d requests image reacquisition and shifts the process to step SC-12.

On the other hand, when the overall feature determination unit 102d determines that the rectangular document is the target form (step SC-16: Yes), the process shifts to step SC-9.

As described above, in the present embodiment, once the medium to be photographed is clarified by the partial feature determination, it is possible to pass the partial feature determination process in the second and subsequent processes.

As a result, in the present embodiment, the time required for the partial feature determination process can be shortened from the second time, and a faster retry process can be performed.

As described above, in the present embodiment, a rectangular cutout of the original area is performed from the high-resolution image data extracted by only one frame from the moving image data of the camera, feature detection is performed on the cut-out document area, and the form type. If there is no match, the image data acquisition may be restarted.

Here, in the present embodiment, in order to make a determination at high speed and with high accuracy, a two-step feature determination may be performed. In the first stage, comparison with the characteristics of a specific area set for each form may be performed.

Here, the specific areas and features set for each form may be unique to each form type. Further, in the second stage, the characteristics of the entire form may be compared. As a result, in the present embodiment, high-speed processing and highly accurate form identification can be performed.

In recent years, with the spread of mobile terminals such as smartphones and tablets, the demand for document scanning with cameras possessed by mobile terminals has increased. This is because the mobile terminal is easy to use, and unlike the conventional scanner, it does not require a dedicated terminal.

Conventionally, there has been a mechanism for automatically shooting a document using a mobile terminal, and when personal authentication is performed using a camera of a mobile terminal, it is necessary to shoot only the document for the purpose of personal authentication.

However, in the past, there have been problems such as shooting an unnecessary document or shooting an image that cannot be recognized by OCR.

Therefore, in the present embodiment, we propose a method of determining whether the target document taken by the camera is a target document and automatically taking a picture when it is determined that the document can be recognized by OCR. There is.

[Other Embodiments]
Although the embodiments of the present invention have been described above, the present invention may be implemented in various different embodiments within the scope of the technical idea described in the claims, in addition to the above-described embodiments. It is a thing.

For example, the mobile terminal 100 may perform processing in a stand-alone form, performs processing in response to a request from a client terminal (which is a separate housing from the mobile terminal 100), and outputs the processing result to the client terminal. You may return it to.

Further, among the processes described in the embodiment, all or a part of the processes described as being automatically performed can be manually performed, or all the processes described as being performed manually. Alternatively, a part thereof can be automatically performed by a known method.

In addition, information including parameters such as processing procedures, control procedures, specific names, registered data of each processing or search conditions, screen examples, or database configurations shown in the specification and drawings are to be specified. Can be changed arbitrarily except for.

Further, with respect to the mobile terminal 100, each component shown in the figure is a functional concept and does not necessarily have to be physically configured as shown in the figure.

For example, with respect to the processing functions included in each device of the mobile terminal 100, particularly each processing function performed by the control unit 102, all or any part thereof is realized by the CPU and a program interpreted and executed by the CPU. It may be realized as hardware by wired logic.

The program is recorded on a non-temporary computer-readable recording medium including a programmed instruction for causing the computer to execute the method according to the present invention, which will be described later, and the mobile terminal 100 is required. Is read mechanically. That is, a computer program for giving instructions to the CPU in cooperation with the OS (Operating System) and performing various processes is recorded in a storage unit 106 such as a ROM or an HDD. This computer program is executed by being loaded into RAM, and constitutes a control unit in cooperation with a CPU.

Further, this computer program may be stored in an application program server connected to the mobile terminal 100 via an arbitrary network, and all or a part thereof can be downloaded as needed. ..

Further, the program according to the present invention may be stored in a computer-readable recording medium, or may be configured as a program product. Here, the "recording medium" includes a memory card, a USB memory, an SD card, a flexible disk, a magneto-optical disk, a ROM, an EPROM, an EEPROM, a CD-ROM, an MO, a DVD, and a Blu-ray (registered trademark). It shall include any "portable physical medium" such as a disk.

Further, the "program" is a data processing method described in an arbitrary language or description method, regardless of the format such as source code or binary code. The "program" is not necessarily limited to a single program, but is distributed as a plurality of modules or libraries, or cooperates with a separate program represented by the OS to achieve its function. Including things. A well-known configuration or procedure can be used for a specific configuration, a reading procedure, an installation procedure after reading, or the like for reading the recording medium in each device shown in the embodiment.

Various databases and the like stored in the storage unit 106 are memory devices such as RAM or ROM, fixed disk devices such as hard disks, flexible disks, and / or storage means such as optical disks, and are used for various processes and provision of websites. It may store various programs, tables, databases, and / or files for web pages to be used.

Further, the mobile terminal 100 may be configured as an information processing device such as a known personal computer, or may be configured by connecting an arbitrary peripheral device to the information processing device. Further, the mobile terminal 100 may be realized by mounting software (including a program, data, etc.) that realizes the method of the present invention on the information processing device.

Further, the specific form of distribution / integration of the device is not limited to the one shown in the figure, and all or a part thereof may be functionally or physically in an arbitrary unit according to various additions or functional loads. It can be configured in a distributed and integrated manner. That is, the above-described embodiments may be arbitrarily combined and implemented, or the embodiments may be selectively implemented.

As described above, the mobile terminal, the image processing method, and the program can be implemented in many industrial fields, particularly in the image processing field for handling the image read by the camera, and are extremely useful.

100 Mobile terminal 102 Control unit 102a Frame acquisition unit 102b Manuscript acquisition unit 102c Partial feature determination unit 102d Overall feature determination unit 102e Reacquisition unit 102f Image display unit 102g OCR unit 106 Storage unit 106a Feature data file 106b Image data file 110 Imaging unit 112 Input / output unit 114 Sensor unit 116 Communication unit

Claims (9)

A frame acquisition means for acquiring a frame taken by the shooting unit,
A manuscript acquisition means for acquiring manuscript image data of a manuscript from the frame, and
A partial feature determining means for determining whether or not a partial feature of a registered form and a document partial feature in the manuscript at a position corresponding to the position of the partial feature in the form match.
When the partial feature determining means determines that the partial feature matches the original partial feature, the overall feature determination determines whether or not the overall feature of the form and the entire document feature of the manuscript match. Means and
When it is determined by the partial feature determining means that the partial feature and the original partial feature do not match, or when the overall feature determining means determines that the overall feature and the original document overall feature do not match. A re-acquisition means for acquiring a frame by re-imaging of the photographing unit, and
Mobile terminal equipped with. The partial feature is a feature of a part of the plurality of items of the form, and the overall feature is a feature of the layout of the plurality of items.
The mobile terminal according to claim 1. The partial features consist of ruled lines, characters, and / or characteristic images.
The mobile terminal according to claim 1. The partial feature determining means compares the degree of agreement between the partial feature of the registered form and the document partial feature in the manuscript at a position corresponding to the position of the partial feature in the form with a threshold value. By doing so, it is determined whether or not the partial feature and the document partial feature match.
The mobile terminal according to claim 1. The partial feature determining means has the degree of matching based on the position, color gradation value, number and / or size of the ruled lines, characters and / or characteristic images constituting the partial feature of the registered form. Is compared with the threshold value to determine whether or not the partial feature and the original partial feature match.
The mobile terminal according to claim 4. The reacquisition means outputs pseudo-shutter operation information when a frame is acquired by re-imaging of the photographing unit.
The mobile terminal according to claim 1. An OCR means for performing OCR processing on the original image data and acquiring character data is further provided.
The mobile terminal according to claim 1. A frame acquisition step to acquire a frame by shooting in the shooting section,
A manuscript acquisition step of acquiring manuscript image data of a manuscript from the frame, and
A partial feature determination step for determining whether or not a partial feature of a registered form and a document partial feature in the manuscript at a position corresponding to the position of the partial feature in the form match.
When it is determined in the partial feature determination step that the partial feature and the original partial feature match, the overall feature that determines whether or not the overall feature of the form and the entire document feature of the manuscript match. Judgment step and
When it is determined in the partial feature determination step that the partial feature and the original partial feature do not match, or in the overall feature determination step, it is determined that the overall feature and the original document overall feature do not match. In the case, the reacquisition step of acquiring the frame by re-imaging of the photographing unit, and
Image processing method including. A frame acquisition step to acquire a frame by shooting in the shooting section,
A manuscript acquisition step of acquiring manuscript image data of a manuscript from the frame, and
A partial feature determination step for determining whether or not a partial feature of a registered form and a document partial feature in the manuscript at a position corresponding to the position of the partial feature in the form match.
When it is determined in the partial feature determination step that the partial feature and the original partial feature match, the overall feature that determines whether or not the overall feature of the form and the entire document feature of the manuscript match. Judgment step and
When it is determined in the partial feature determination step that the partial feature and the original partial feature do not match, or in the overall feature determination step, it is determined that the overall feature and the original document overall feature do not match. In the case, the reacquisition step of acquiring the frame by re-imaging of the photographing unit, and
A program that lets your computer run.

Images