KR101957417B1 - Motion recognition control system and method for separately displaying shot image - Google Patents

Abstract

Information extracting means for extracting a plurality of frames from a plurality of frames, and camera means for generating a visible light image that is a real image of a plurality of frames by capturing a subject; content storing means for storing contents controlled according to an operation of the subject; An operation recognition control means for recognizing an operation of the subject by information and a display means having a general display region for displaying the visible light image in the background image of the content, a system, in said display means, and that the common display area in which the visible light image and the background image to be displayed together is separately provided, a clear display area, the predetermined background image is displayed excluding only the live-action images And a gong.

Description

TECHNICAL FIELD [0001] The present invention relates to a motion recognition control system and a motion recognition image display method,

The present invention relates to an operation recognition control system and method for a real image display system.

In the control of an information processing device such as an augmented reality (AR) device or a virtual reality (VR) device, an operation of a photographed subject is recognized by an operation recognition control engine, and an event corresponding to the operation is transmitted or generated to an application The configuration is known. At this time, in a display such as a goggle, an image of an operating subject is configured to be displayed together with a background image.

For example, when a hand is extended to touch an object such as an apple or a gun displayed on a display of an augmented reality or a virtual reality device, an image is generated by imaging the hand as an object in the image capture device, And may be configured to display on the display, tracking the shape and position of the hand in the images captured over the frame. That is, the image of the subject moves on the background image in accordance with the motion of the user, so that the operability in the augmented reality or the virtual reality is guaranteed and improved. However, the present invention is not limited to this example, and the present invention can be applied to a technique in which the entire human body is applied as a subject, for example.

At this time, the image of the subject displayed on the display may be displayed as a modeled model image in which the position, shape, and posture of the subject are reflected and displayed in a simple manner by continuous combination of simple figures such as a square, a triangle, , It may be displayed simply as a real image as it is captured by the image pickup apparatus. The display by the model image is advantageous in that the display processing speed can be improved by remarkably reducing the data to be displayed on the display by simplifying or omitting the wrinkles or the bending. However, since the modeling must be performed by the preprocessing, And the displayed object does not reflect the details of the actual object, which is disadvantageous in that it disturbs reality. On the contrary, the display by the real image is disadvantageous in that the displayed data is an actual image as it is when the resolution is high. However, there is a disadvantage in that a load may be generated in the display processing. However, And there is an advantage that it produces a very intimate sense of unity that reflects the actual subject as it is.

The method of controlling a virtual training system includes the steps of: receiving, from a motion detection sensor, authenticated motion information for logging into a virtual training system, and logging into the virtual training system; And outputting virtual training selection request information on the screen through the output unit when the virtual training system is logged in. The virtual training selection motion information sensed based on the output virtual training selection request information is output from the motion sensing sensor , The control unit controls the virtual training execution unit to execute the virtual training content corresponding to the inputted virtual training selection request information. .

Patent registration 10-1401656

3A, 3B, 6 and 7, a conventional control display implementation method for a camera output image used in these augmented reality or virtual reality includes a method of displaying a part of a user's body It is a way to take a picture. Therefore, the image in which the image of the object to be captured is displayed clearly becomes clear.

However, in such a system, if the aperture value (F) of the camera is not very high, all or part of the real image may be blurred, and FIGS. 3A, 3B, 6, Respectively. For example, this phenomenon becomes particularly serious when the illuminance is low. That is, the blurred display of the conventional technique is caused by the fact that a camera used for photographing a subject has adopted a non-high aperture value for visible light.

In addition, since the background image and the real image are displayed together, if the background image and the real image are similar in texture, color, brightness, etc., there is a problem that it is difficult to identify the real image. This is because, for example, in a case where an operation, an attitude, and a shape of a subtle subject should be accurately distinguished in augmented reality or virtual reality content, for example, in the case of scissors rocking against a user of another user, The motion recognition control engine in the unit recognizes the motion from the frame images of the subject. The fact that the user can not identify the form of the real image, in particular, the form of the real image of the other user, It causes a situation. Therefore, in addition to the display of a blurry real image by the prior art, this real image is displayed together with the background of the content.

The present invention has been made in order to solve the above problems of the prior art, and it is an object of the present invention to provide an image processing apparatus and a method thereof, in which an actual image of a subject together with a content background image of an augmented reality or a virtual reality is displayed by a motion recognition technique, It is an object of the present invention to provide an operation recognition control system and method of a real image display system in which only a real image is displayed and visibility of a real image of a user is improved.

In addition, since the real image displayed in the separate area includes the visible light information and the infrared information extracted at the same time, the visibility is improved even under low illumination, and even if the background image is similar in texture, color, And an object of the present invention is to provide an operation recognition control system and method of a real image display method.

In addition, the real image displayed in the separate area extracts visible light information and infrared light information at the same time. By displaying these visible light information and infrared light information together, visibility is improved even under low illumination, And an object of the present invention is to provide an operation recognition control system and method of a real image display method in which visibility is improved even when brightness and the like are similar.

The simultaneous extraction of the visible light information and the infrared information is performed by an information extracting apparatus including a dual lens (DA) dual lens (Single Lens) 4-color sensor camera, Control system and method.

The real image displayed together with the background image is processed by the visible light information, the real image displayed in the separate area is processed by the mixture of the visible light information and the infrared information, and the motion recognition is performed from the visible light information and the infrared information And to provide an operation recognition control system and method of real image display method that is processed by depth information extracted by a map module.

According to an aspect of the present invention, there is provided an operation recognition control system of a real image display system, comprising: information extracting means including camera means for generating a visible light image which is a real image of a plurality of frames by picking up an image of a subject; content storage means for storing the content to be controlled according to the operation and the operation recognition control means by the information extracted from the inspection image recognizing operation of the subject and the normal to display the visible light image within the background image of the content in the real picture image display mode operation recognition control system which comprises a display means having a display area in the display means, independent of the normal display area in which the visible light image and the background image that is displayed with the background And a predetermined clear display area in which only the real image is displayed is provided.

Here, as the real image, an infrared image having infrared information as an image different from the visible light image is further captured, and the visible light image and the infrared image are generated simultaneously .

Here, the camera means may include visible light ray imaging means for generating the visible ray image and infrared ray imaging means for generating the infrared ray image as imaging means separate from the visible light ray imaging means.

Alternatively, the camera means is a double diaphragm (DA, Dual Aperture) can comprise a single lens (Single Lens) 4- color sensor camera.

At this time, the information extracting means extracts the depth information from the visible light ray image obtained from the double-aperture (DA, Dual Aperture) single-lens four-color sensor camera and the depth information the yirueojim including means.

At this time, the motion recognition control means can be controlled to detect and track the subject in the real image by the depth information, and recognize the motion.

Preferably, the real image displayed in the clear display area includes at least the infrared image .

Meanwhile, the motion recognition control method of the live-action image display system of the present invention for solving the aforementioned problems is the information extracting step including a camera process by imaging a subject produce photorealistic images of visible light image of the multiple frames, the display and a content storage step of storing the content to be controlled in accordance with the motion of a subject, and a gesture recognition control method comprising: by the information extracted from the inspection image recognizing operation of the subject, the visible light image within the background image of the content in the general display process display step inspection operation recognition control method of the image display system comprising a comprising a that, in said display step, apart from the above general display process that the visible light image and the background image to be displayed together, The ship It characterized in that the image of the rule is provided with a sharp display process which displays only the predetermined photorealistic images.

According to the present invention, a real image of a subject together with a content background image of an augmented reality or a virtual reality is displayed by an operation recognition technology, so that only a real image of the subject is displayed in a separate area of a part of the display, A motion recognition control system and method for real image display are provided.

In addition, since the real image displayed in the separate area includes the visible light information and the infrared information extracted at the same time, the visibility is improved even under low illumination, and even if the background image is similar in texture, color, There is provided an operation recognition control system and method of a real image display method.

In addition, the real image displayed in the separate area extracts visible light information and infrared light information at the same time. By displaying these visible light information and infrared light information together, visibility is improved even under low illumination, An operation recognition control system and method for real image display system in which visibility is improved even when brightness and the like are similar are provided.

The simultaneous extraction of the visible light information and the infrared information is performed by an information extracting apparatus including a dual lens (DA) dual lens (Single Lens) 4-color sensor camera, A control system and method are provided.

The real image displayed together with the background image is processed by the visible light information, the real image displayed in the separate area is processed by the mixture of the visible light information and the infrared information, and the motion recognition is performed from the visible light information and the infrared information There is provided an operation recognition control system and method of a real image display method which is processed by depth information extracted by a map module.

1 is a block diagram showing a schematic configuration and data of an operation recognition control system of a real image display system according to an embodiment of the present invention.
Fig. 2 is an exemplary picture for explaining an example of a normal display in which a real image is displayed together with a content background image, and a clear display in which only a real image is displayed without a background image, with the user's hand as a subject.
3 is an enlarged photograph of the display portion of Fig.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, for elements having the same function by the same structure, the same reference numerals are retained even if the drawings are different, and detailed description thereof may be omitted. Also, to be connected includes the concept of passing through the medium in the middle. And a certain functional block may be configured to be absorbed by other functional blocks or may be divided into a plurality of different blocks and implemented.

≪ First Embodiment - General Display and Sharp Display Area >

The operation recognition control system of the real image display system according to the present invention comprises: And a clear display 14b area including the information extracting means 11 , the content storing means (not shown ) , the motion recognition control means 14c and the display means 13 , do.

The information extracting means 11 is a means including a camera means 11a for picking up an image of a subject 10 and generating a visible light image 12a which is a real image of a plurality of frames. The information extracting means 11 may be a photographing and image extracting apparatus, for example, a dual aperture single lens depth information extracting apparatus for simultaneously extracting visible light, infrared rays, depth information, and the like.

The content storage means (not shown ) is a means for storing contents controlled according to the operation of the subject 10. The content is composed of, for example, the content of augmented reality or virtual reality, and interaction of the user is an important factor.

The operation recognition control means 14c is means for recognizing the operation of the subject 10 based on the information extracted from the real image. The motion recognition is performed in such a manner that an object is identified from a subject and its shape, posture, and position are tracked.

The display means 13 is means having an ordinary display 14a region for displaying the visible light image 12a in the background image of the content. As a display of augmented reality or virtual reality, it includes a goggle type display.

The area of the clear display 14b is arranged so that in the display means 13 the background image is excluded in addition to the area of the general display 14a in which the background image and the visible light image 12a are displayed together, Only a real image is displayed. Since only the real image is displayed without the background image, the decrease in visibility due to the interference with the background image is eliminated.

<Operation and Effect>

In an augmented reality or a virtual reality device, in order to perform an interaction between the user and the content of the device, the progress of the content is controlled by receiving the user's operation. The operation of the user is recognized by processing of the real image generated by the imaging means in the information extraction means 11, that is, the camera means 11a.

A visible light (RGB) image can be captured as a real image. In a dual aperture (DA) camera or a conventional visible light camera, the visible light channel consists of a filter with a medium aperture (F) value. Therefore, the photographed real image is sharp only in the focused area, and the non-focused area appears blurred in the display.

On the other hand, an infrared (IR) image as a real image can be picked up. In the DA camera, the infrared channel consists of a filter with a high aperture (F) value. Therefore, the sharpness of the entire image is high.

According to the present invention, by using this point, a part of the display means can be separately allocated to the crisp display area, and a real image can be displayed here for a target, such as a hand operation of the user, in which the user desires clarity.

As illustrated in FIG. 2, when the technique shown in FIG. 1 is configured, for example, as an augmented reality (AR) or virtual reality (VR) system, a real image of a clear infrared channel can be displayed in a separate area.

Fig. 3 is an enlarged view of the example of Fig. Here, the general control display 14a, which is the entire display, is composed of a combination of a real image and a background image graphic. However, since the hand-operated actual image included in the synthesized image is displayed in a visible light channel, Able to know. Therefore, the output image of the infrared channel is scattered on the sharpness enhancement display 14b (inside the black circle), which is a separate display area, so that the user can see a clear and real image.

This allows the user to view an accurate real image.

<Simultaneous infrared imaging>

On the other hand, as the real image, an infrared image 12b having infrared information as an image different from the visible light image 12a may be further picked up. Here, the visible light image 12a and the infrared image 12b are preferably generated simultaneously .

Since the real image is used for controlling the information processing device, the visible light image and the infrared image must be generated at the same time. And it is preferable to be generated at the same point of view as possible.

<Separate Means for Simultaneous Imaging>

For generating an infrared image, the camera means 11a includes a visible light image pickup means (not shown ) for generating the visible light image 12a, and an infrared image (not shown ) as an image pickup means separate from the visible light ray image pickup means And infrared imaging means (not shown ) for generating infrared rays 12a and 12b.

Since the visible ray image and the infrared ray image are simultaneously obtained by a separate means, the infrared ray imaging member can be added to a general camera for visible light ray imaging so that the technique of the present invention can be modified.

&Lt; Single means for simultaneous imaging >

For the generation of the infrared image, said camera means (11a) is, may be formed by a double diaphragm (DA, Dual Aperture) comprises a single lens (Single Lens) 4- color sensor camera.

As described above, since the visible ray image and the infrared ray image are simultaneously obtained by one means, simultaneous imaging by a single sight point is possible. Particularly, since the light rays of the object are received by the single lens through the same optical path into the visible light region and the infrared light region of the image pickup element and divided into the visible light region and the infrared light region of the image pickup element by the double aperture, Infrared images provide the best possible precise matching.

The technology of the present invention, which requires both visible light and infrared images, includes a new Dual Aperture Single Lens camera solution Dual Aperture Single Lens 4-color sensor camera The depth information extracting apparatus 11 can be used. In this apparatus, visible light, infrared light, and depth information are simultaneously output. According to the present invention, a clear infrared image is separately applied to the clear display 14b, which is a part of the existing display area 13, without complicated hardware equipment by using this advantage.

The present invention is for providing a clear image separately using an all-in-focus infrared image. In the double diaphragm (DA) method, infrared rays are already implemented as a filter having a large F value (deep depth) , It is possible to implement the present invention with little additional cost without significantly increasing the form factor of the hardware. However, as mentioned above, a separate infrared ray apparatus may be added without using the DA system, and the present invention is not excluded to realize a sharpness improvement region by the separate infrared ray apparatus.

<Specific Configuration by DA Camera>

DA camera, the information extracting means 11 extracts the visible light image obtained from the DA (Dual Aperture) Single Lens 4-color sensor camera, which is the camera means 11a, And a depth map means (11b) for extracting depth information (12c ) from the infrared image (12a) and the infrared image (12b).

Thus, a multi-image output device based on a Dual Aperture Single Lens shown in the flowchart of FIG. 1 can be used to simultaneously transmit visible light, infrared light, and depth information to an information processing device such as an input of an AR / VR device And output.

The visible light image and the infrared image by the double-aperture single-lens four-color sensor camera have extremely high matching accuracy, so that the depth information can be obtained with high accuracy. That is, in the information extracting means 11, the depth information is extracted together with the visible light image from the DA camera and the infrared image, and can be used as an input of the 3D hand gesture recognition program.

A technology for extracting depth with a double aperture (DA) single lens camera has developed a solution with WB (SoC) of the present applicant, wherein visible light, infrared rays, and depth information are all output. Therefore, all of the necessary channels can be utilized as input of the motion recognition control device for the augmented reality AR / virtual reality VR. Therefore, the technique proposed by the present invention is to reuse an infrared (IR) channel among these, and to utilize it in a portion where a clear image is required as described above.

At this time, it is preferable that the motion recognition control means 14c is controlled so as to detect and track the subject 10 within the real image by the depth information 12c and recognize the operation.

Due to the high matching of the visible light image and the infrared image by the DA camera, the extracted depth information is obtained with high accuracy, so that the object detection and tracking using the depth information can be obtained with high precision.

Further, since an infrared image is used, a high recognition rate can be achieved even for a subject with low illumination.

<Sharp display with infrared image>

It is preferable that the real image displayed in the area of the clear display 14b includes at least the infrared image 12b .

That is, Fig. 2 or as shown in the block diagram of a picture, and Fig. 1 of Figure 3, the live-action image is already being printed on the normal display (14a) in a separate clear display (14b) region of the sharp infrared image corresponding to the scene (12b) can be applied and output simultaneously. The real image displayed on the sharp display 14b may be composed of only the infrared image 12b or may be displayed by overlapping the visible light image 12a and the infrared image 12b, Can be adjusted accordingly. As the ratio of the infrared image 12b increases, the sharpness improves, and as the ratio of the visible light image 12a increases, the saturation can be improved.

Thereby, even in the case of contents requiring accurate motion, posture, and shape of the object, such as scissors rock, it is possible to precisely confirm the operation of the opponent's subject between the users interacting with each other, improve the visibility even at low illumination, The image is interfered with, and visibility is improved even when it is similar in texture, color, lightness, and the like.

A configuration in which a sharp infrared image is separately displayed by outputting a clear infrared image by overlapping with a visible light ray on the basis of this principle is effective even if the camera means 11a is not a device based on a dual aperture single lens If it is possible to obtain an infrared image, it can be applied to the present invention.

<Invention of Method>

Meanwhile, the motion recognition control method of the live-action image display system of the present invention, an information extraction step (11), a content storing step (not shown), and, comprising: a motion recognition control step (14c), the display step (13) And a clear display 14b process is provided.

The information extracting step 11 is a step including a camera process 11a for picking up a subject 10 and generating a visible light image 12a which is a real image of a plurality of frames.

The contents storage step (not shown ) stores contents controlled according to the operation of the subject 10. [

The operation recognition control step 14c is a step of recognizing the operation of the subject 10 based on the information extracted from the real image.

The display step 13 includes a general display 14a for displaying the visible light image 12a in the background image of the content.

The clear display 14b may be configured to display the background image and the visible light image 12a in the display step 13 separately from the normal display 14a process in which the background image and the visible light image 12a are displayed together, It is a predetermined process that displays only the real image.

Although the present invention has been described with reference to specific embodiments, the present invention is not limited thereto. Modifications, modifications and variations made by those skilled in the art within the scope of the claims are to be construed as being within the scope of the present invention.

For example, FIG. 2 or in a separate clear display (14b) region of the shown in Figure 3, if not the specific forms, such as circular, by separating the display area if increasing the sharpness of the specific portion using the infrared to the present invention any other type Belongs.

A display technology that realizes a display in which the sharpness of a real image of a specific object is enhanced by using an infrared channel in a separate area corresponds to the present patent even if the object is not a part of the user's body .

INDUSTRIAL APPLICABILITY The present invention can be used in the industry of an operation recognition control system and method of a real image display system.

10: Subject
11: Information Extraction Apparatus
11a: DA camera module
11b: DA depth map module
12a: visible ray image
12b: Infrared image
12c: Depth information
13: AR / VR display
14a: General Dislay (control)
14b: Sharp display (sharpness enhancement)
14c: Motion recognition control engine

Claims (8)

Information extracting means including camera means for picking up a subject and generating a visible light image which is a real image of a plurality of frames;
Content storing means for storing contents controlled according to an operation of the subject;
An operation recognition control means for recognizing an operation of the subject by information extracted from the real image;
Display means having a common display area for displaying the synthesized image in the background image inspection of the contents
A motion recognition control system of a real image display system, comprising:
The camera means comprises a Dual Aperture (DA) Single Lens 4-color sensor camera ,
An infrared image having infrared information as an image different from the visible light image is further imaged as the real image,
Wherein the visible light image and the infrared image are generated simultaneously ,
Wherein the information extracting means includes a depth map means for extracting depth information from the visible light image and the infrared image obtained from the double-aperture (DA, Dual Aperture) single lens 4-color sensor camera as the camera means , &Lt; / RTI &gt;
Wherein the motion recognition control means is controlled to detect and track the subject in the real image by the depth information,
The display means is provided with a predetermined clear display region in which, apart from the normal display region, the background image of the content is excluded and only the real image is displayed,
The real image displayed in the clear display area is made up of the infrared image
Wherein the image recognition system comprises: And the information extracting step comprising the process of the camera by imaging a subject produce photorealistic images of visible light image of the plurality of frames,
A contents storing step of storing contents controlled according to an operation of the subject;
An operation recognition control step of recognizing an operation of the subject by information extracted from the real image;
A display step of having a normal display process for displaying the live-action images in the background image of the content
A method for controlling an operation recognition of a real image display system,
Wherein the camera means comprises a dual aperture (DA) Dual Lens 4-color sensor camera , wherein the real image includes an infrared image having infrared information as an image separate from the visible light image Wherein the visible light image and the infrared image are generated simultaneously ,
Wherein the information extracting means includes a depth map means for extracting depth information from the visible light image and the infrared image obtained from the double-aperture (DA) Dual Lens 4-color sensor camera as the camera means Wherein the motion recognition control means is controlled to detect and track the subject in the real image by the depth information and to recognize the motion,
Wherein the display step includes a predetermined clear display process in which the background image is excluded and only the real image is displayed separately from the normal display process, and the real image displayed in the clear display area is the infrared image &Lt; / RTI &gt;
The method of claim 1, further comprising: delete delete delete delete delete delete

Images