KR101831783B1 - Apparatus for image and text recognition included in output printout and method thereof - Google Patents

Abstract

The present invention relates to a device and a method thereof to recognize an image and a text included in a printout. The image and text recognizing device includes: an input part receiving a photographed image of a printout to be recognized; an image analyzing part dividing an image and a text from the photographed image through deep-learning, and generating pattern data by analyzing arrangement patterns and paragraphs of the text and the image; a database storing pattern data about a plurality of different images and texts; a search part searching for pattern data matched with the generated pattern data by comparing the generated pattern data to pre-stored pattern data; and a control part displaying a screen or image corresponding to the matched pattern data if the pattern data exist. According to the present invention, the device is capable of reducing a necessary amount of initial data for analysis on an image or text by correcting a photographed image with a preset reference value for a plurality of photographing environments, thereby improving the performance and accuracy of analysis. Moreover, the present invention is capable of providing data corresponding to a marker function in accordance with a result of text or image analysis without an extra marker such as a QR code, thereby conveniently providing a variety of information about a text or image.

Description

[0001] APPARATUS FOR IMAGE AND TEXT RECOGNITION INCLUDED IN OUTPUT PRINTOUT AND METHOD THEREOF [0002]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image and text recognition apparatus and method in an output printed matter, and more particularly to an image and text recognition apparatus and method in an output printed matter which recognize an image and text in an output printed matter and provide corresponding information, And a method thereof.

Conventionally, character recognition has been commercialized by focusing on image generation through a scanner and document processing. However, in recent years, the user's mobility has become more active due to the development of the mobile terminal environment. In recent years, the user acquires the information through the image acquisition through the camera mounted on the user terminal and the image and text recognition of the captured image, Service is required.

Accordingly, technologies for providing various services by recognizing and utilizing images and texts through photographed images are being developed by mobile communication companies, portal companies, solution providers, and content providers capable of building services as well as mobile terminal and camera manufacturers.

However, because the photographed image is greatly influenced by the photographing environment, even if the same image or text is photographed in different photographing environments, it can be recognized as a different image or text.

For example, when the shooting environment is poor, such as light reflection, poor lighting, wrinkling of prints, etc., there is a problem that the recognition of colors depending on the brightness of illumination may be changed and the images or text may be incorrectly recognized in the shot image .

In particular, the abstract image is a result of combining various elements such as line, surface, shape, color, and size. Therefore, it is not enough to distinguish the linearity by simple conditions because the portion recognized according to the shooting environment may be different.

That is, it is possible to solve a problem of being misrecognized in various shooting environments, and a technique of accurately extracting and recognizing images and texts is required.

The technology that provides the background of the present invention is disclosed in Korean Patent Laid-Open No. 10-2014-0068302 (published on November 26, 2012).

An object of the present invention is to provide an apparatus and method for recognizing images and texts included in an output printed matter that recognize images and text in an output printed matter and provide corresponding information.

According to an embodiment of the present invention, there is provided an apparatus for recognizing images and texts included in an output printed matter, the apparatus comprising: an input unit for inputting a captured image of an output printed material to be recognized; An image analyzing unit that separates an image and a text from an image and generates pattern data by analyzing a paragraph and an arrangement pattern of the image and text, a database storing pattern data for a plurality of images and texts of different types, A search unit for comparing the pattern data and pre-stored pattern data to search for pattern data that matches the generated pattern data; and a display unit for displaying a screen or an image corresponding to the corresponding pattern data, And a control unit.

The image analyzing unit may estimate the photographing environment through at least one of color, brightness, and wrinkling degree of the photographed image, and may correct the photographed image with a preset reference value corresponding to the estimated photographed environment.

Wherein the image analyzing unit comprises: The method includes the steps of: dividing an image and a text by analyzing at least one of an outline, a shape, a color, and a size of the image by using deep running in the captured image; separating paragraphs using indentation and spacing of the text; It is possible to extract the arrangement pattern through the arrangement position.

The search unit may search, If there is a plurality of matching pattern data in the database, matching pattern data is primarily searched from the database using at least one of the type, size, thickness, Can,

The search unit may search, If there is a plurality of matching pattern data in the database after the first search, The matching pattern data can be secondarily retrieved from the database by using the keyword repeatedly created in the text above the threshold value.

The controller may display or execute at least one of a matching moving image, an image, a text, a diagram, a related URL, and related apps in association with each other if pattern data corresponding to the captured image exists.

According to another embodiment of the present invention, there is provided a method of recognizing an image and a text using an image and text recognition apparatus, the method comprising: receiving a captured image of an output printed material to be recognized; Generating pattern data by analyzing paragraphs and arrangement patterns of images and texts, generating pattern data by using a database storing pattern data for a plurality of different types of images and texts, Comparing the stored pattern data to search for pattern data that coincides with the generated pattern data, and displaying a screen or an image corresponding to the corresponding pattern data if there is matching pattern data.

According to the present invention, it is possible to reduce the amount of initial data necessary for analyzing an image or text by correcting the photographed image by using a predetermined reference value for a plurality of photographing environments, thereby improving analysis performance and accuracy .

In addition, according to the present invention, it is possible to conveniently provide various information related to a text or an image by providing data corresponding to a marker function according to a recognized text or image analysis result without using a marker such as a separate QR code can do.

1 is a diagram for explaining an image and text recognition system included in an output print according to an embodiment of the present invention.
2 is a block diagram of an image and text recognition apparatus according to an embodiment of the present invention.
3 is a flowchart illustrating an image and text recognition method of an image and text recognition apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when an element is referred to as "comprising ", it means that it can include other elements as well, without excluding other elements unless specifically stated otherwise.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention.

Hereinafter, an image and text recognition system included in an output printed matter will be described in detail with reference to FIG.

1 is a diagram for explaining an image and text recognition system included in an output print according to an embodiment of the present invention.

As shown in FIG. 1, an image and text recognition system according to an embodiment of the present invention includes an image and text recognition apparatus 200 and a camera 300 for recognizing an output printed matter 100.

First, the output printed matter 100 includes a pamphlet, a leaflet, an information sheet, a promotional article, a catalog, a booklet, a business card, and the like, as an object to be recognized.

Next, the image and text recognizing apparatus 200 analyzes the output printed material 100, separates the image and the text included in the output printed material 100, and generates pattern data analyzing an arrangement pattern, a margin size, and the like. Then, the image and text recognition apparatus 200 compares the generated pattern data with pre-stored pattern data, and displays the corresponding screen or image when searching for previously stored pattern data.

Meanwhile, the image and text recognition apparatus 200 stores pattern images of a plurality of images and texts of different types in a database, and stores a plurality of images, texts, images, texts, charts, related URLs, At least one of them is linked and stored.

In addition, the image and text recognition apparatus 200 stores reference values of color, brightness, and wrinkle degree values of the photographed image for various shooting environments in the database.

Next, the camera 300 may be configured as a separate device or mounted in the image and text recognition apparatus 200, and is not limited to a specific type of camera.

Then, the camera 300 photographs the output printed matter 100 and transmits the photographed image to the image and text recognition apparatus 200 connected to the network.

That is, the image and text recognizing apparatus 200 and the camera 300 are connected to each other via a network so that information can be exchanged. In addition to the wired communication network, an Internet, a LAN (Local Area Network) But are not limited to, wireless communication networks such as a LAN (Wireless Local Area Network), a WAN (Wide Area Network), a PAN (Personal Area Network), 3G, 4G, Wi-Fi.

Hereinafter, the image and text recognition apparatus 200 for recognizing images and text included in an output printed matter will be described in detail with reference to FIG.

2 is a configuration diagram of an image and text recognition apparatus according to an embodiment of the present invention. 2, an image and text recognition apparatus 200 according to an exemplary embodiment of the present invention includes an input unit 210, an image analysis unit 220, a database 230, a search unit 240, and a control unit 250 ).

First, the input unit 210 receives a photographed image of an output printed material to be recognized. That is, the input unit 210 may receive the captured image in real time from the interlocked camera or download the captured image through the network.

Here, the printed matter includes pamphlets, flyers, information papers, promotional materials, catalogs, brochures, and the like.

Next, the image analyzer 220 separates an image or a text from the photographed image by using deep running, extracts each paragraph and an array pattern, and generates pattern data.

Here, the image analyzing unit 220 can separate the image and the text through a repetitive process of extracting and distorting a line included in the image from the captured image using Deep Learning, which is a technique of clustering or classifying objects or data. have.

In addition, the image analysis unit 220 may generate pattern data using the sizes of the top, bottom, and left and right margins according to the ratio of the size of the output printed matter in the photographed image, the position of the image or text in the output printed matter .

Next, the database 230 stores pattern data for a plurality of images and texts of different types. Also, the database 230 stores moving pictures, images, texts, graphics, related URLs, related apps, etc. highly related to each pattern data.

Next, the search unit 240 compares the generated pattern data with pre-stored pattern data, and searches for pattern data that matches the generated pattern data. If there is a plurality of matching pattern data, the search unit 240 searches for the pattern data that coincides most with the size of the margin based on the font type, size, thickness, and spacing between characters of the text of the photographed image .

Next, the control unit 250 displays a screen or an image corresponding to the corresponding pattern data if there is matching pattern data. At this time, the control unit 250 can display and execute moving images, images, texts, graphics, related URLs, and related apps in cooperation with the screen or image.

Hereinafter, a process of recognizing images and texts of the image and text recognition apparatus 200 included in an output printed matter according to an embodiment of the present invention will be described with reference to FIG.

3 is a flowchart illustrating an image and text recognition method of an image and text recognition apparatus according to an embodiment of the present invention.

First, the image and text recognition apparatus 200 receives a captured image of an output printed material to be recognized (S310).

At this time, the image and text recognition apparatus 200 can receive the photographed image of the output printed material 100 at the same time as the photographed image or receive the photographed image of the output printed material 100 photographed at the previous time.

Next, the image and text recognition apparatus 200 separates the image or text from the photographed image by using deep running, extracts each of the paragraphs and the array pattern, and generates pattern data (S320).

First, the image and text recognition apparatus 200 can estimate the photographing environment through at least one of color, brightness, and wrinkle degree of the photographed image. Then, the image and text recognition apparatus 200 can correct the photographed image to a preset reference value corresponding to the estimated photographic environment.

The image and text recognition apparatus 200 according to the embodiment of the present invention assumes various shooting environments and extracts values of the color, brightness and wrinkle degree of the photographed output print 100 in each of various shooting environments. Then, the image and text recognition apparatus 200 recognizes the recognition rate of the image and the text through the simulation of recognizing the image and text of the output printed matter 100 while adjusting the values of the color, brightness and wrinkle degree of the photographed image in each photographing environment The value of the color, brightness, and wrinkle degree of the highest captured image can be extracted.

Then, the image and text recognition apparatus 200 can store the values of the color, brightness, and wrinkle degree of the extracted photographic image as reference values corresponding to the respective photographing environments.

When the image and text recognition apparatus 200 receives the captured image, the image and text recognition apparatus 200 extracts the values of the color, brightness, and wrinkle degree of the captured image to estimate the shooting environment, Value.

On the other hand, when the wrinkled degree value of the output printed matter 100 in the photographed image is larger than the reference value, the image and text recognition apparatus 200 spreads the wrinkled region by using the ratio of the estimated output printed matter 100 in the horizontal and vertical directions, It can be realized through the augmented reality as the photographed image of the output printed material 100 that has been removed.

Next, the image and text recognition apparatus 200 can distinguish between the image and the text by analyzing at least one of the outline, shape, color, and size of the image by using deep running in the captured image.

Then, the image and text recognition apparatus 200 can distinguish paragraphs by using indentation and spacing of text in the photographed image, and extract an arrangement pattern through arrangement positions of images and text.

In the case of the photographed image of the wrinkled output printed material 100 implemented as an augmented reality, the image and text recognition apparatus 200 estimates whether the obscured area is an image area or a text area due to wrinkling using deep running .

For example, the image and text recognition apparatus 200 analyzes the color of the surrounding region of the obscured region due to wrinkling in the photographed image, classifies it into an image if a plurality of colors are detected, If it is detected above the reference value, it can be classified as an image. The image and text recognition apparatus 200 can estimate the size of the image or text paragraph of the obscured region due to wrinkles corresponding to the size of the estimated image or text paragraph in the other normal region of the output printed matter 100 have.

Next, the image and text recognition apparatus 200 compares pattern data and pre-stored pattern data of the database to search for matching pattern data (S330).

Here, the database stores pattern data for a plurality of images and texts of different types, and each pattern data is at least one of a highly related video, image, text, diagram, related URL, have.

If there is a plurality of matching pattern data in the database, the image and text recognition apparatus 200 can extract the font type, size, thickness, and size of margins between characters in the text of the text segment in the captured image .

Then, the image and text recognition apparatus 200 can primarily search for matching pattern data from the database using at least one of the font type, size, thickness, and the size of the space between characters of the extracted text.

That is, the image and text recognition apparatus 200 can retrieve pattern data having the same text characteristics as the font type, size, thickness, and size of the space between characters extracted from the photographed image.

On the other hand, if a plurality of matching pattern data exist in the database even after the first search, the image and text recognition apparatus 200 can extract the keywords repeatedly created in the text above the threshold value. Then, the image and text recognition apparatus 200 can secondarily search for matching pattern data from the database using the extracted keywords.

That is, the image and text recognition apparatus 200 can retrieve pattern data including or extracted from a plurality of matching pattern data after the primary search.

As described above, the image and text recognition apparatus 200 can search pattern data previously stored in a database that best matches the pattern data generated by changing the search condition in stages.

Next, the image and text recognition apparatus 200 displays a screen or an image corresponding to the corresponding pattern data if there is matching pattern data (S340).

That is, if there is pattern data corresponding to the photographed image, the image and text recognition apparatus 200 may display or execute at least one of the matched moving image, image, text, diagram, related URL, .

At this time, a matching moving image can provide a three-dimensional image using an augmented reality.

As described above, according to the embodiment of the present invention, it is possible to reduce the initial amount of data necessary for analyzing an image or text by correcting the photographed image using preset reference values for a plurality of photographing environments, Can be improved.

According to the embodiment of the present invention, even if a marker such as a separate QR code is not used, data corresponding to the marker function is provided according to the recognized text or image analysis result, Can be provided.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

100: output print 200: image and text recognition device
210: Input unit 220: Image analysis unit
230: database 240: search unit
250: control unit 300: camera

Claims (12)

An apparatus for recognizing images and text contained in an output print,
An input unit for inputting a photographed image of an output print to be recognized,
An image analyzing unit for separating an image and text from the photographed image by using deep running and analyzing a short circuit and an arrangement pattern of an image and a text to generate pattern data,
A database storing pattern data for a plurality of images and texts of different types,
A search unit for comparing the generated pattern data with pre-stored pattern data and searching for pattern data matching the generated pattern data; and
And a control unit for displaying a screen or an image corresponding to the corresponding pattern data if there is matching pattern data. The method according to claim 1,
Wherein the image analyzing unit comprises:
And estimates the photographing environment through at least one of color, brightness, and wrinkling degree of the photographed image, and corrects the photographed image with a predetermined reference value corresponding to the estimated photographing environment. The method according to claim 1,
Wherein the image analyzing unit comprises:
The method includes at least one of an outline, a shape, a color, and a size of an image by using deep running in the captured image,
The paragraphs are separated using indentation and spacing of the text,
And extracts an arrangement pattern through an arrangement position of the image and the text. The method of claim 3,
The search unit may search,
If there is a plurality of matching pattern data in the database, matching pattern data is primarily searched from the database using at least one of the font type, size, thickness, and size of the space between characters of the text Image and text recognition device. 5. The method of claim 4,
The search unit may search,
If there is a plurality of matching pattern data in the database after the first search,
And secondarily searching for matching pattern data from the database by using a keyword repeatedly created in the text above a threshold value. The method according to claim 1,
Wherein,
And displaying at least one of a matching moving image, an image, a text, a diagram, a related URL, and related apps in association with each other if pattern data corresponding to the captured image exists. A method of recognizing an image and a text using an image and text recognition apparatus,
Receiving a photographed image of an output printed matter to be recognized,
Separating an image and a text from the photographed image using deep running, analyzing a paragraph and an arrangement pattern of an image and a text to generate pattern data,
Comparing the generated pattern data with pre-stored pattern data using a database storing pattern data for a plurality of images and texts of different types, and searching for pattern data corresponding to the generated pattern data; And
And displaying a screen or an image corresponding to the corresponding pattern data if there is matching pattern data. 8. The method of claim 7,
Wherein the generating the pattern data comprises:
And estimating the photographing environment through at least one of color, brightness, and wrinkling degree of the photographed image, and correcting the photographed image with a preset reference value corresponding to the estimated photographing environment. 8. The method of claim 7,
Wherein the generating the pattern data comprises:
The method includes at least one of an outline, a shape, a color, and a size of an image by using deep running in the captured image,
The paragraphs are separated using indentation and spacing of the text,
And extracting an arrangement pattern through an arrangement position of the image and the text. 10. The method of claim 9,
Wherein the searching comprises:
If there is a plurality of matching pattern data in the database, matching pattern data is primarily searched from the database using at least one of the font type, size, thickness, and size of the space between characters of the text Image and text recognition method. 11. The method of claim 10,
Wherein the searching comprises:
If there is a plurality of matching pattern data in the database after the first search,
Wherein the matching pattern data is secondarily searched from the database by using a keyword repeatedly created in the text above a threshold value. 8. The method of claim 7,
Wherein the displaying comprises:
And displaying and executing at least one of a matching moving image, an image, a text, a diagram, an associated URL, and an associated app when the matching pattern data for the captured image exists.

Images