JP4711885B2 - Remote control device and method - Google Patents

Description

  The present invention relates to a remote operation method for performing remote operation on a device to be operated from a remote position by a gesture, and a remote operation device that realizes the remote operation method.

  In general, a remote controller (remote controller) using infrared rays is used to remotely operate an operation target device. However, the conventional remote control has the following problems.

  First, there is a problem that the user easily loses the remote control. In recent years, there are many problems that a user often owns a plurality of remote controllers corresponding to a plurality of devices, so that the remote controllers can be easily mistaken.

  Further, since the operation buttons of the remote control are generally small, there is a problem that the user can easily press the wrong operation button. Furthermore, since the remote control is generally driven by a battery, there is a problem that the battery must be replaced periodically.

  In order to solve these problems, a method of performing a remote operation by reading a user's gesture with a video camera has been proposed (for example, see Patent Document 1). As a technique for recognizing a user's gesture, it has been proposed to extract a skin color portion from a captured image, specify a face region by template matching, and further recognize an operation by pattern matching (for example, patents). Reference 2).

JP 2003-186596 (Page 6, FIG. 1) JP 2003-216955 A (page 11, FIG. 1)

  However, Patent Document 1 does not disclose details of a method for reading a user's gesture. Further, in Patent Document 2, assuming that a personal computer with high processing capability is used, a complicated process with high accuracy such as template matching is performed, so that it is difficult to apply to, for example, household appliances. There is.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to realize a remote operation by a user's gesture with a simple process that can be applied to household appliances and the like.

The remote control device according to the present invention is:
Image capturing means for acquiring a captured image including a user image, feature point detecting means for detecting a feature point from the captured image, gesture estimation means for estimating a user's gesture based on the detected feature point, and corresponding to the gesture A feature point existence range estimating means for estimating a feature point existence range as a region where a feature point may exist in a captured image. Is further provided. In the captured image, the feature point detection unit performs the first detection process in a region where a plurality of feature point existence ranges overlap , and the plurality of feature point existence ranges do not overlap, and only one feature point existence range exists. Then, the second detection process, which is lower in accuracy than the first detection process but has a smaller calculation amount, is performed, and feature points are detected by integrating the results of both detection processes.

According to the remote operation method of the present invention, an imaging step for acquiring a captured image including an image of a user, a feature point detecting step for detecting a feature point from the captured image, and estimating a user's gesture based on the detected feature point gesture estimation step of, generating an operation signal corresponding to the gesture, saw including an operation signal transmitting step of transmitting to the remote operation target apparatus, there feature points in the captured image, as an area where there may be minutiae A feature point existence range estimation step for estimating the range is further included. The feature point detection step, Oite the captured image, in a region where a plurality of feature points existing range overlap, performing a first detection process, do not overlap the plurality of feature points existing range, only one of the feature points existing range In a certain area, a second detection process is performed that is less accurate than the first detection process but requires a smaller amount of calculation, and feature points are detected by integrating the results of both detection processes.

  According to the present invention, the entire load of the remote operation processing is reduced by limiting the region for performing detection processing with high accuracy and a large amount of calculation (and thus a large processing load) to the first region in the captured image. be able to. As a result, remote operation processing by gesture input can be realized even in a device having a control device with relatively low processing capacity, such as a household appliance.

  The present invention limits a region for performing detection processing with high accuracy and a large amount of calculation, compared to the conventional technique in which high-precision and complicated detection processing is performed over the entire captured image (that is, a plurality of feature points). In other words, it is possible to reduce the processing load required for gesture estimation by performing detection processing with a high accuracy and a large amount of calculation only in an area that may include the. That is, the remote operation processing can be realized by a device (for example, a household appliance) provided with a control device having a relatively high processing capacity.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a configuration of a remote control device using gesture input according to Embodiment 1 of the present invention. The remote control device includes an imaging unit 10 such as a camera that acquires a captured image including a user's image, and a gesture recognition unit 1 that reads a gesture from the captured image acquired by the imaging unit 10. The gesture recognition unit 1 estimates an gesture from an image processing unit 11 that detects a feature point from an image captured by the imaging unit 10, and a change in the position of the feature point detected by the image processing unit 11, and receives an operation signal corresponding to the gesture. And a recognition processing unit 12 that transmits to the remote operation target device 13. The device control unit 13a of the remote operation target device 13 receives an operation signal from the remote operation device and performs an operation according to the received operation signal.

  The image processing unit 11 of the remote control device includes a feature point existence range estimation unit 14 that estimates a range where a feature point may exist in a captured image, and a feature point detection unit that detects a feature point in the feature point existence range. 15. The recognition processing unit 12 of the remote operation device generates a gesture estimation unit 16 that estimates a gesture based on a feature point, an operation signal corresponding to the estimated gesture, and transmits the operation signal to the remote operation target device 13. And a transmission unit 17.

  FIG. 2 is a diagram illustrating an example of a captured image according to Embodiment 1 of the present invention. The feature point existence range estimation unit 14 (FIG. 1) changes, from the captured image, an area in which there is a high possibility that a feature point necessary for gesture estimation exists, changes in the image between several past frames, and This is determined based on the position of the feature point in the previous frame. The feature point refers to a “part having a certain feature” existing in the captured image, and here refers to a user's hand or head.

  The region where the feature point may exist is a region including the user in the captured image. For example, in the captured image illustrated in FIG. 2, if a region boundary 20 that includes the user image is defined inside, the region boundary 20 is a region 21 in which a feature point may exist, and the region boundary 20 Outside 20 is a region 22 where there is a low possibility of feature points. The region boundary 20 shown in FIG. 2 has a rectangular shape, but may have another shape such as an ellipse.

  When detecting feature points, the entire captured image is not the target of detection, but the region in which the captured image may change among the regions where the captured image has changed during the past several frames Is estimated as “feature point existence range”, and only this feature point existence range is set as a detection target. The area where the captured image has not changed is excluded from the detection target because there is a low possibility that a feature point exists.

  The feature point existence range estimation unit 14 divides a region where a plurality of feature point existence ranges overlap from a region where the feature points do not overlap so that only one feature point may be included. The captured image is divided such that there is a region where there is a possibility that a plurality of feature points may be included.

  FIG. 3 is a diagram illustrating an example in which a captured image is divided as described above. The existence range 30 of the feature point 1 (for example, the user's head) is an area where the feature point 1 may exist inside. The existence range 31 of the feature point 2 (for example, the user's right hand) is an area where the feature point 2 may exist inside. Depending on the overlapping state of these two regions (feature point existence ranges 30, 31), a region 32 containing only feature point 1 at most, a region 33 containing only feature point 2 at most, and feature points 1, Can be divided into regions 34 that may include both.

  FIG. 4 is a diagram schematically illustrating an example of a method for dividing a captured image. As shown in FIG. 4, the feature point existence range estimation unit 14 estimates the feature point existence range 30 so as to include the feature point 1 that is the user's head, and includes the feature point 2 that is the user's right hand. When the feature point existence range 31 is estimated, a portion where both feature point existence regions 30 and 31 overlap is generated. This overlapping portion is referred to as “region 34 where the feature point existence ranges overlap”. In FIG. 4, the feature point existence range is greatly illustrated in order to easily understand the region where the feature point existence ranges overlap.

  The region 34 where the feature point existence ranges 30 and 31 overlap may include feature points as many as the number of overlapping regions. For example, in the region 34 where the feature point existence range 30 of the feature point 1 and the feature point existence range 31 of the feature point 2 overlap, there is a possibility that both the feature point 1 and the feature point 2 are included. . The feature point detection unit 15 (FIG. 1) performs a feature point detection process with high accuracy and a large amount of calculation (and therefore a large load) in the overlapping region 34, thereby collecting a plurality of pixel clusters (clusters). These are detected separately, and these are set as a plurality of “feature point candidate points”.

  On the other hand, in the regions 32 and 33 where the feature point existence ranges 30 and 31 do not overlap, there is no feature point close to each other, and there is a high possibility that only one feature point is included at most. There is little need to distinguish and detect. Therefore, in these areas 32 and 33, feature point detection processing with low accuracy and low calculation amount (and thus light load) is performed to detect only one pixel block, which is set as a “feature point candidate point”. . If a plurality of pixel clusters are detected in these areas 32 and 33, the most likely feature point candidate point, for example, the one with the largest area (the maximum likelihood feature point candidate point) ).

  FIG. 5 is a diagram showing a case where feature points exist so as to straddle the region 34 where the feature point existence ranges 30 and 31 overlap and the region 32 where they do not overlap. The pixel block C1 detected by the high-precision detection process in the region 34 where the feature point existence ranges overlap and the pixel block C2 detected by the low-precision detection process in the non-overlapping region 32 sandwich the boundary line. If they are continuous, they are combined and determined as one feature point (feature point 1 in FIG. 5). Further, the pixel block C3 which is not continuous with the other is determined as one feature point (feature point 2 in FIG. 5). Determining the continuity of pixel clusters detected in the areas 34 and 32 is referred to as “integration of detection results”.

  The gesture estimation unit 16 (FIG. 1) detects the current motion vector of the feature point based on the position change between several frames of each feature point detected by the feature point detection unit 15, and previously stores the motion vector as a sequence of motion vectors. A user's gesture is estimated by comparing with the registered gesture pattern sequence.

  The operation signal transmission unit 17 (FIG. 1) generates an operation signal corresponding to the user's gesture estimated by the gesture estimation unit 16, and transmits the operation signal to the device control unit 13a of the remote operation target device 13 (for example, home appliance). Then, the remote operation target device 13 is made to perform an operation corresponding to the operation signal, thereby realizing the remote operation by the gesture.

  FIG. 6 is a flowchart showing an overall flow of the remote operation processing in the first embodiment. In the remote operation processing shown in FIG. 6, the processing from the feature point existence range estimation processing (step S <b> 11) to the detection result integration processing (step S <b> 18) is performed by the image processing unit 11. Further, the process from the gesture estimation process (step S19) to the operation signal transmission process (step S21) is performed by the recognition processing unit 12.

  In this remote operation process, first, an image is captured by the imaging unit 10 of the camera, and a captured image including a user image is acquired (step S10).

  Next, the existence range estimation unit 14 of the image processing unit 11 selects the feature point existence range from among the changed portions of the captured image based on the history of several frames of the captured image captured by the imaging unit 10. Is estimated (step S11). When there is no change in the captured image and the feature point existence range cannot be estimated (step S12), the process returns to step S10 and the image capturing unit 10 captures an image.

  Next, based on the feature point existence range estimated in step S11, it is determined whether there is a region where a plurality of feature point existence ranges overlap (step S13).

  When there are regions where the feature point existence ranges overlap, the feature point detection unit 15 performs detection processing with a high accuracy and a large amount of calculation in the overlapping regions, A “feature point candidate point” is obtained (step S14). This detection process is, for example, a labeling process.

  If there is a region where the feature point existence ranges overlap, there is always a region that does not overlap (see FIG. 3). Therefore, in the non-overlapping region, detection processing with low accuracy and a small amount of calculation is performed, and one pixel Is detected, and one “feature point candidate point” is obtained (step S15). Since this detection process is a simple process such as a detection process using a color histogram (color histogram method), it can be performed at a relatively high speed. Note that the order of steps S14 and S15 may be reversed.

  Further, the detection results of the detection process (step S14) with high accuracy and a large calculation amount and the detection process (step S15) with low accuracy and a small calculation amount are integrated (step S16). In other words, it is determined whether feature point candidate points detected by a detection process with high accuracy and a large amount of calculation and feature point candidate points detected by a detection process with low accuracy and a small amount of calculation are consecutive. If they are, these are collectively referred to as one “feature point” (see FIG. 5). When feature point candidate points are not continuous, each of them is set as a separate “feature point”.

  On the other hand, in step S13 described above, if there is no overlapping region of the feature point existence ranges, only detection processing with low accuracy and a small amount of calculation is performed to obtain one “feature point candidate point” (step S17). This detection process with low accuracy and a small amount of calculation is a detection process using, for example, a color histogram, as in step S15.

  Subsequently, it is determined whether or not a feature point has been detected (step S18). If no feature point has been detected, the process returns to step S10 and an image is captured by the imaging unit 10.

  When the feature point is detected, the gesture estimation unit 16 estimates the user's gesture based on the position information of the feature point (step S19). Here, for example, based on the change of the captured image over a plurality of frames, it is determined that the position of the feature point 2 (user's hand) is changing up and down with respect to the position of the feature point 1 (user's head). In such a case, a gesture such as the user moving his / her right hand up and down is estimated.

  It should be noted that the “head” is a portion above a human neck and includes a face. Further, as long as it is a part that can be used as a reference for determining the movement of the user's hand, another part may be used as a feature point instead of the user's head.

  Next, it is determined whether or not a gesture has been estimated (step S20). If the gesture has been estimated, the operation signal transmission unit 17 generates an operation signal corresponding to the gesture and sends it to the remote operation target device 13. Transmit (step S21). When the gesture has not been estimated yet and when the transmission of the operation information has been completed, the process returns to step S10 to capture an image by the imaging unit 10, and the above-described processes (steps S10 to S21) are repeated.

  The device control unit 13a of the remote operation target device 13 controls the remote operation target device 13 so as to execute an operation corresponding to the received operation information. In this way, remote operation of the remote operation target device 13 is realized.

  As described above, in the first embodiment, the region 34 where the feature point existence ranges 30 and 31 overlap is detected with a high accuracy and a large amount of calculation (a large load), and the feature point existence range 30 and 31 are detected. For the non-overlapping areas 32 and 33, detection processing is performed with low accuracy and a small amount of calculation (load is small). In this way, by limiting the area where detection processing with high accuracy and large amount of calculation is performed, remote operation can be realized with simple processing.

  As described above, since the remote operation by the gesture can be performed with a simple process, the remote operation device can be realized even with a device having relatively few hardware resources such as a home appliance.

  In addition, in the region 34 where the feature point existence ranges 30 and 31 overlap, detection processing with high accuracy and a large amount of calculation is performed, so even if a plurality of feature points exist close to each other, Therefore, the user's gesture can be read accurately.

  Further, by using the user's hand and head as the feature points, it becomes easy to estimate a gesture for moving the user's hand based on the position of the head.

Embodiment 2. FIG.
FIG. 7 is a flowchart showing the flow of the feature point existence range estimation process according to Embodiment 2 of the present invention. The feature point existence range estimation process in the second embodiment is performed by the feature point existence range estimation unit 14 (FIG. 1) described in the first embodiment (step S11 in FIG. 6). ).

  In the estimation process of the feature point existence range, first, a difference between the captured image and the background image or an image several frames before is determined to identify a changed portion of the captured image (step S30). Furthermore, the vicinity of the position where the feature point existed in the captured image of the previous frame is searched from among them (step S31). The time interval of the frames is as short as about 60 ms, for example, and even if the gesture is being performed, the change in the position of the feature point between the frames is slight. Therefore, it can be considered that the feature point exists in the vicinity of the position where the feature point existed in the captured image of the previous frame even in the current frame. That is, it is possible to narrow down a portion where a feature point may exist from a portion where the captured image has changed. Therefore, “the part where the captured image has changed and the vicinity of the position of the feature point in the previous frame” is estimated as the feature point existence range (step S32).

  As a target for obtaining a difference from the captured image in step S30, a background image captured in advance or a captured image several frames before can be used. When obtaining the difference, the difference between the values of the RGB space in each pixel may be obtained, or the difference obtained by converting the RGB space into a value of another color space, for example, the HSV space may be obtained.

  When a difference from the background image (hereinafter referred to as background difference) is used, the captured image can be distinguished into a background portion and a foreground portion. Further, since the background portion is a portion in which the image has hardly changed, and it is unlikely that the feature point is included in this portion, the feature point existence range is determined when estimating the feature point existence range in step S32. Only the foreground part in the captured image can be narrowed down.

  The background image may be a background image automatically captured by the image capturing unit 10 when the remote control device is activated, or may be a background image captured by the image capturing unit 10 by a user operating a button or the like of the apparatus main body. May be. However, it is necessary to shoot so that the user who is a foreground image candidate is not included in the background image.

  The background image is sequentially updated in order to reduce the influence of illumination changes over time (step S33). The update may be performed at a constant cycle, or may be performed when a sudden illumination change is detected. Note that when the lighting change is small, instead of updating the entire background image at once, it is divided into several frames and updated locally (for example, the entire image is divided into 1 block 4 × 4 blocks). Processing can be reduced by updating only one pixel in each block per frame).

  On the other hand, in step S30, when a difference from a captured image several frames before (hereinafter referred to as inter-frame difference) is used, a change occurring during the several frames, mainly a movement of the photographing target, can be obtained. Therefore, in step S32, the feature point existence range can be narrowed down to the vicinity of the portion where the movement has occurred. Note that when the inter-frame difference is used, the process for the first frame cannot be performed, but in that case, the background difference can be used only for the first frame, and the inter-frame difference can be performed from the second frame. .

  In the second and subsequent frames, both the background difference and the interframe difference may be used. In this way, it is possible to further narrow down the feature point existence range as a particularly moving part in the foreground part.

  In step S30, when obtaining the background difference and the inter-frame difference, the difference may be obtained for each pixel, or the differences of a plurality of pixels, for example, 2 × 2 rectangular areas may be obtained together. When obtaining a difference for each pixel, a foreground portion and a portion with motion can be cut out by performing threshold processing on the difference value for each pixel. When obtaining the difference of a plurality of pixels at once, the similarity between a plurality of pixels, for example, 2 × 2 rectangular areas, is obtained, and threshold processing is performed on the similarity of the rectangular areas, so that the foreground portion and the motion It is possible to cut out a certain part. As the similarity of a plurality of pixels, an average value of differences for each pixel may be used, or a correlation value obtained by performing pattern matching such as normalized correlation may be used.

  The shape of the feature point existence range may be a rectangle or a shape defined according to the shape feature of the feature point to be detected, for example, an ellipse. If no feature point is detected in the first frame or the previous frame, the feature point existence range cannot be estimated in step S32. Therefore, feature point detection processing is performed on all pixels of the captured image. Although it is necessary to carry out detection, it is possible to deal with a detection target that enters from outside the field of view of the imaging unit 10 by performing detection processing only in a region from the outer periphery of the captured image to several pixels, for example. .

  As described above, according to the second embodiment, by detecting the feature point existence range in the captured image, in the feature point detection process in the subsequent process, the detection process is performed on the region where the feature point existence range overlaps with the region where the feature point existence range does not overlap. Therefore, it is possible to realize remote operation by gestures while reducing the processing load.

  In particular, in the second embodiment, the changed part of the captured image is obtained from the difference between the captured image acquired by the imaging unit 10 and the background image or the image several frames before, and further the feature point position in the previous frame is obtained. Since the feature point existence range is estimated based on this, it is possible to reduce the influence of noise included in the captured image. Further, when obtaining the difference, the processing time can be shortened by processing several pixels together.

Embodiment 3 FIG.
FIG. 8 is a diagram illustrating an example of a gesture to be read in the remote operation device. Here, for example, a gesture (A) in which the user swings the right arm left and right, a gesture (B) in which the right arm is swung up and down, and a gesture (C) in which the right arm is rotated can be considered. These gestures can be detected by using both hands 2 and 3 of the user as feature points. However, if only both hands 2 and 3 are actually used as feature points, the swing of the arm and the movement of the whole body May be indistinguishable. Therefore, in order to estimate the position of the user's body, the user's head (including face) 1 is further used as a feature point.

  Since each of the feature points 1, 2, and 3 has the feature of “skin color”, when detecting the feature point, first, the skin color portion is detected from the captured image within the above-described feature point existence range. To do. For the detection of skin color, threshold processing may be performed on the RGB space, or threshold processing may be performed on another color space obtained by converting the RGB space, for example, a color space called a normalized color space. Furthermore, feature point candidate points are detected by recognizing a cluster of skin color pixels using the result of skin color detection.

  FIG. 9 is a flowchart showing a flow of feature point detection processing according to Embodiment 3 of the present invention. This flowchart is a specific example of the processing of steps S13 to S17 (FIG. 6) executed by the feature point detection unit 15 (FIG. 1) described in the first embodiment.

  In the feature point detection process, first, as described in the first embodiment, presence / absence of overlap of feature point existence ranges is detected (step S13). When there is an area where the feature point existence ranges overlap, a detection process with high accuracy and a large amount of calculation is performed on the overlapping areas by using a labeling method, and a plurality of pixel clusters are distinguished and detected (step S50). “Feature point candidate points” are obtained (step S51). Note that the detection process by the labeling method is a process of assigning the same label to pixels belonging to the same connected figure and assigning different labels to pixels belonging to different connected figures. Since there are many, processing load is large.

  On the other hand, in the region where the feature point existence ranges do not overlap, detection processing by the color histogram method with low accuracy and a small amount of calculation is performed (step S52), and one pixel block is detected. When only one pixel block is detected, the pixel block is set as a “feature point candidate point”. When two or more pixel clusters are detected, for example, the one having the largest area is selected and set as one “maximum likelihood feature point candidate point” (step S53). The detection process using the color histogram method is a process for identifying the feature point existence range from the skin color pixel distribution in the captured image, and the calculation load is small, so the processing load is small, but the detection accuracy is inferior to the labeling method. It is.

  Furthermore, it is determined whether or not the feature point candidate points obtained in step S51 and the feature point candidate points (maximum likelihood feature point candidate points) obtained in step S53 are continuous. Are combined into one “feature point”, and if they are not continuous, separate “feature points” (step S54).

  If there is no overlapping region of the feature point existence ranges in step S13, only the detection process by the color histogram method with low accuracy and a small amount of calculation is performed to detect one pixel block, thereby A “feature point candidate point” is obtained (step S56). When two or more pixel clusters are detected, for example, the one having the largest area is selected to obtain the “maximum likelihood feature point candidate point” (step S57).

  When only the detection process by the color histogram method is performed, the integration of the feature point candidate points (step S54) is unnecessary, and thus the feature point candidate points (maximum likelihood feature point candidate points) acquired in steps S56 to S57. Are directly used as “feature points”.

  Here, feature point detection processing using skin color information has been described, but feature point detection processing using shape information (such as contour lines) in addition to skin color information is also possible. That is, for example, by extracting a contour line in the foreground region and applying a simple figure such as an ellipse from the contour line, it is possible to estimate the feature point existence range as a simple figure.

  In the detection process using the contour line, a plurality of “feature point candidate points” each having an outer periphery defined by the contour line are obtained by detecting the contour line in the overlapping region of the feature point existence ranges. In the area where the feature point existence ranges do not overlap, one “feature point candidate point” whose outer periphery is defined by the outline is obtained, and when multiple pixel clusters are detected, the area with the largest area is obtained. For example, one “maximum likelihood feature point candidate point” is obtained. As a result, a portion whose outer periphery is defined by a simple figure outline can be approximated as a feature point candidate point. Also in this case, feature point candidate points obtained by detection processing with high accuracy and a large amount of calculation, and feature point candidate points (maximum likelihood feature point candidate points) obtained by detection processing with a low accuracy and a small amount of computation The “feature point” is obtained by integrating.

  Further, in the feature point detection processing according to the third embodiment, information on the gravity center position of the feature points such as the head and both hands can be used for the feature point detection. That is, information about the center of gravity of the feature points can be stored in advance and used to distinguish feature points such as the head, right hand, and left hand from each other. If the feature points are correctly distinguished in the feature point existence range in the previous frame, the feature point candidate points after that are correctly distinguished. On the other hand, for the feature points not detected in the previous frame, for example, the head is often present on both hands, the right hand is often present on the right side of the head and left hand, and the left hand is located on the head or Information such as being often present on the left side with respect to the right hand is stored in advance, and the head, right hand, and left hand can be determined based on this information.

  In addition to information on the center of gravity of the feature point, a figure (for example, an ellipse, a rectangle, etc.) that touches the feature point candidate point is used, and the center position, shape, inclination, etc. of this figure are detected for the position and orientation of the feature point. Can be used. For example, when a rectangle in contact with the feature point candidate area is used, it is possible to detect not only the position of the feature point but also whether the hand is spread or closed by changing the shape of the rectangle. In this way, the types of gesture input can be diversified. Further, in the case where a reading error occurs only with the position information of the feature point, information on the shape and inclination of the figure can be used for reducing the error.

  As described above, according to the third embodiment, by using color information and shape information (for example, contour lines), both hands and heads as feature points can be obtained without performing complicated processing such as template matching. Can be detected.

  In addition, the type of gesture input can be diversified by using the position of the center of gravity and the figure in contact with the feature point to detect the feature point, and the error included in the position information of the feature point can be reduced. it can.

Embodiment 4 FIG.
FIG. 10 is a flowchart showing a flow of gesture estimation processing according to the fourth embodiment. The fourth embodiment relates to a specific example of the gesture estimation process (step S19 in FIG. 6) by the gesture estimation unit 16 (FIG. 1) described in the first embodiment. FIG. 11 is a diagram illustrating an example of a motion vector used in the processing illustrated in FIG.

  In the gesture estimation process shown in FIG. 10, first, the movement of the feature point is detected based on the position information of the feature point detected in the feature point detection process (step S18 in FIG. 6) (step S60). If no feature point motion is detected (step S61), the process returns to the feature point motion detection (step S60). When the movement of the feature point is detected, a gesture corresponding to the movement is estimated (step S62). Subsequent processing is as described with reference to FIG. 6 (steps S20 and S21).

  In the above-described step S62, when a gesture corresponding to the motion of the feature point is estimated, as shown in FIG. 11, the motion vector 80 for each frame of the feature point is represented by up (1), top left (2), left ( 3) Using “motion ID 81” defined by unit vectors in 8 directions of lower left (4), lower (5), lower right (6), right (7), and upper right (8), and using this motion ID To approximate the motion of the feature points. The gesture estimation unit 16 (FIG. 1) registers a combination of these motion IDs 81 for each frame in association with the user's gesture pattern.

  12A and 12B are diagrams illustrating examples of gesture pattern sequences that are combined with the motion ID 81 and are stored in the gesture estimation unit 16. For example, in the gesture A in which the user repeatedly moves his / her hand up and down twice, the movements of the feature points are “up, down, up, down”, and thus gestures 1, 5, 1, and 5 shown in FIG. Corresponding to the pattern row 82. In addition, the gesture B in which the user moves his / her hand up / down / left / right has the movements of the feature points “up, down, left, right”, and therefore the gesture pattern strings 1, 5, 3, and 7 shown in FIG. 83 is associated.

  In this gesture estimation process, the motion vector of a feature point is compared with previously registered gesture pattern sequences 82 and 83 to estimate the user's gesture. In this gesture estimation process, it is preferable to compare the motion vector of the feature point for each frame and the gesture pattern sequence and sequentially delete inappropriate ones from the gesture pattern options.

  For example, if the motion vector of the feature point in the first frame is 1 (upper) and the motion vector of the feature point in the second frame is 5 (lower), the possibility of both gestures A and B However, if the motion vector of the feature point in the third frame is 1 (upper), there is no possibility of gesture B (3 in the third frame). Exclude from the choices. When the motion vector of the feature point is 5 in the fourth frame, since it is confirmed that the feature point matches the pattern (1, 5, 1, 5) of the gesture A, it is determined that the user's gesture is the gesture A. . In this way, since the options gradually narrow, the gesture estimation process can be performed easily.

  As shown in FIG. 12C, information on the speed of gesture movement can be added to the gesture pattern sequence. The speed of movement of the gesture is determined, for example, by detecting the magnitude of displacement between the previous frame and the current frame and comparing it with a preset reference value. In this case, for example, symbols H and L indicating low speed and high speed can be added following the motion ID of the gesture pattern sequence. As an example, a gesture pattern string of 1, H, 5, H, 1, H, 5, H can be associated with a gesture in which the user quickly shakes his / her hand up and down. In this way, the types of gesture patterns that can be defined can be increased.

  In addition, since the movement of the user's hand may be ambiguous, it is preferable to set a priority for each gesture pattern sequence. For example, if the user's hand is halfway and the movement of the feature point is ambiguous “upper, lower, upper, lower” or “upper right, lower, upper right, lower”, the gesture pattern string “ It is preferable to compare the priorities of “1, 5, 1, 5” and “1, 2, 1, 2” and select the higher priority.

  In order to compare the motion vector of the feature point with the gesture pattern as needed, it is necessary to determine the start time of the user's gesture. Therefore, it is assumed that the gesture is estimated after confirming that the user's hand has once stopped.

  As described above, according to the fourth embodiment, it is possible to determine the motion of a feature point only by detecting the direction of motion of the feature point by using a unit vector (motion vector) directed in a plurality of directions. Therefore, it is possible to reduce the calculation amount and memory consumption required for gesture estimation.

  Further, by comparing the magnitude of the displacement from the previous frame with a reference value and performing gesture estimation based on the speed of movement, remote control corresponding to a wider variety of gesture patterns can be realized.

  As an application example of the present invention, by applying it to an existing home digital appliance (digital home appliance), the operation can be performed without using an existing infrared remote controller.

It is a block diagram which shows the structure of the remote control apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows an example of the captured image in Embodiment 1 of this invention. It is a figure which shows the example of a division | segmentation of the captured image in Embodiment 1 of this invention. It is a schematic diagram for demonstrating the division method of the captured image in Embodiment 1 of this invention. It is a schematic diagram for demonstrating integration of the feature point presence range which concerns on Embodiment 1 of this invention. It is a flowchart which shows the whole flow of the remote control process which concerns on Embodiment 1 of this invention. It is a flowchart which shows the estimation process of the feature point presence range which concerns on Embodiment 2 of this invention. It is a figure which shows an example of the gesture which concerns on Embodiment 3 of this invention. It is a flowchart which shows the flow of the feature point detection process which concerns on Embodiment 3 of this invention. It is a flowchart which shows the flow of the gesture estimation process which concerns on Embodiment 4 of this invention. It is a figure which shows the motion vector used by the gesture estimation process which concerns on Embodiment 4 of this invention. It is a figure which shows an example of the gesture pattern row | line | column used by the gesture estimation process which concerns on Embodiment 4 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Gesture recognition part, 10 Imaging part, 11 Image processing part, 12 Recognition processing part, 13 Target apparatus control part, 14 Feature point existence range estimation part, 15 Feature point detection part, 16 Gesture estimation part, 17 Operation signal transmission part, 20 region boundary, 21 region where feature point may exist, 22 region where feature point may not exist, 30 region where feature point 1 exists, 31 region where feature point 2 exists, 32 only feature point 1 An area that may exist, 33 An area where only feature point 2 may exist, 34 An area where feature points 1 and 2 may exist, 80 Motion vector, 81 Motion ID, 82, 83 Gesture pattern Column.

Claims (22)

Imaging means for acquiring a captured image including an image of the user;
Feature point detecting means for detecting a feature point from the captured image;
Gesture estimation means for estimating the user's gesture based on the detected feature points;
An operation signal transmitting means for generating an operation signal corresponding to the gesture and transmitting the operation signal to a remote operation target device;
In the captured image, further comprising a feature point existence range estimation means for estimating a feature point existence range as a region where the feature point may exist,
The feature point detection means performs a first detection process in a region where a plurality of the feature point existence ranges overlap in the captured image, and a plurality of the feature point existence ranges do not overlap, and one feature point exists. In a region that is only a range, a second detection process that is less accurate than the first detection process but has a smaller amount of calculation is performed, and the feature points are detected by integrating the results of both detection processes. Features remote control device. The remote control device according to claim 1 , wherein the feature point detection unit uses a user's head and hand as the feature point. The feature point detection unit, the remote control device according to claim 1 or 2, characterized in that the detection of the feature point based on color information. The feature point detection unit, the remote control device according to claim 1 or 2, characterized in that the detection of the feature point based on information of the contour line. The feature point detecting means utilizes the information of the center of gravity of the feature point, the remote operation device according to any one of claims 1 to 4, characterized in that the detection of the feature point. The feature point detecting means utilizes the shape in contact with the feature point, the remote operation device according to any one of claims 1 to 4, characterized in that the detection of the feature point. The gesture estimating means, remotely according to the movement of the feature point, in any one of claims 1, characterized in that expressed using the motion vector is a unit vector oriented in the direction of the plurality of types up to 6 Operating device. The remote operation device according to claim 7 , wherein the gesture estimation unit defines a gesture to be estimated using a pattern configured by the sequence of motion vectors. The gesture estimation means represents the motion of the feature point using a motion vector that is a unit vector directed in a plurality of directions, and represents the speed of motion of the feature point by a comparison result with a reference value. The remote control device according to any one of claims 1 to 6, wherein: The gesture estimating means, wherein the sequence of motion vectors, by combining the result of comparison between the reference value of the speed of movement of the feature point, claim 9, characterized in that to define the gesture estimation target The remote control device described in 1. The gesture estimating means from the gesture choices, by sequentially excluding incompatible choices based on the feature point, in any one of claims 1 to 10, characterized in that an estimate of the gesture The remote control device described. An imaging step of acquiring a captured image including an image of the user;
A feature point detecting step of detecting a feature point from the captured image;
A gesture estimation step of estimating the user's gesture based on the detected feature points;
An operation signal transmission step of generating an operation signal corresponding to the gesture and transmitting the operation signal to a remote operation target device,
A feature point existence range estimation step for estimating a feature point existence range as an area where the feature points may exist in the captured image;
In the feature point detection step,
The Oite the captured image, the plurality of the regions feature point existing range overlap, performing a first detection process, do not overlap the plurality of the feature points existing range, only the feature points existing range of one region Then, the second detection process, which is less accurate than the first detection process but has a smaller calculation amount, is performed, and the feature points are detected by integrating the results of both detection processes. Method. The remote operation method according to claim 12 , wherein in the feature point detection step, a user's head and hand are used as the feature points. The remote operation method according to claim 12 or 13 , wherein, in the feature point detection step, the feature point is detected based on color information. The remote operation method according to claim 12 or 13 , wherein, in the feature point detection step, the feature points are detected based on contour information. The remote operation method according to any one of claims 12 to 15, wherein, in the feature point detection step, the feature point is detected using information on a centroid position of the feature point. 16. The remote operation method according to claim 12, wherein, in the feature point detection step, the feature points are detected using a figure in contact with the feature points. In the gesture estimation step, the remote according to movement of the feature point, in any one of claims 12, characterized in that expressed using the motion vector is a unit vector oriented in the direction of the plurality of types up to 17 Method of operation. 19. The remote operation method according to claim 18 , wherein in the gesture estimation step, a gesture to be estimated is defined using a pattern configured by the sequence of motion vectors. In the gesture estimation step, the motion of the feature point is represented by using a motion vector that is a unit vector facing a plurality of directions, and the speed of the feature point motion is represented by a comparison result with a reference value. The remote control method according to any one of claims 12 to 17 , characterized by: In the gesture estimation step, by combining the string of the motion vector, and a comparison result between the reference value of the speed of movement of the feature point, claims, characterized in that to define the gesture estimation target 20 The remote control method described in 1. The gesture estimation step estimates the gesture by sequentially excluding from the gesture options non-conforming options based on the feature points, according to any one of claims 12 to 21. Remote control method.

Images