JP6170696B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for recognizing movement of a human body in order to operate a screen of a device or a display device.

  2. Description of the Related Art Conventionally, there is known an image processing technique used for recognizing a human action (gesture) and operating a device such as a robot based on the recognition result.

  An image processing apparatus described in Non-Patent Document 1 below performs hand shape recognition by obtaining information on the center of gravity and fingertip position of a hand region from an image captured by a fixed camera. Furthermore, the robot moves remotely by adjusting the behavior of the robot moving by two wheels and the rotation speed of the wheels according to the shape of the left hand and the distance between the fingers of the right hand.

Marutani Seikei et al., “Real-time hand shape recognition based on fingertip position detection from two-dimensional images and its application to mobile robot remote control tasks” Information Processing Society of Japan [Computer Vision and Image Media (2002.5.9)] P145-152

  However, in the above apparatus, when the hand is put in the imaging region of the camera from an oblique direction, there is a possibility that the shape of the hand may not be accurately recognized. Will move to.

  The present invention has been made in view of such circumstances, and the shape of the human body part can be accurately determined no matter which direction the human hand or other part of the human body moves relative to the imaging region of the camera. An object of the present invention is to provide an image processing apparatus and an image processing method capable of recognizing and matching the direction of movement intended by a person and the direction recognized by a computer.

An image processing apparatus according to the present invention uses an imaging unit that images a human body part in an imaging region and acquires it as image information, and recognizes the operation of the human body part based on the image information acquired by the imaging unit. A human body region extracting unit for extracting a region of a human body region from the image information, and detecting a pixel around the image information from the human body region extracted by the human body region extracting unit , Part coordinate calculation means for calculating a first part coordinate that is a coordinate of the central part and a second part coordinate that is a coordinate of the center of gravity of the palm or the back of the hand; a first part coordinate calculated by the part coordinate calculation part; Based on the inclination of the straight line connecting the part coordinates, the insertion angle calculation means for calculating the insertion angle to the imaging region of the human body part, and the image information based on the insertion angle calculated by the insertion angle calculation means. Image times for rotation correction To; and a correction means.

  According to the image processing apparatus of the present invention, when a human body part such as a hand or a foot is inserted into the imaging region, the imaging unit acquires this as image information. Since the human body part region extracting unit extracts the region of the human body part from the image information, and the part coordinate calculating unit calculates the coordinates of the first part and the coordinates of the second part among the extracted human body parts. The shape and position of the human body part in the image information is recognized.

  Further, the insertion angle calculation means calculates the angle (insertion angle) of the human body part inserted into the imaging region from the inclination with respect to the coordinate axis of the straight line connecting the first part coordinate and the second part coordinate. Then, the image rotation correction means corrects the rotation of the image information based on the insertion angle. Thereby, even when a human body part is inserted from an oblique direction with respect to the imaging region, it can be corrected in a certain direction. Therefore, when a human body part such as a hand or a foot moves, the image information is corrected in a certain direction by the above-described series of processing, so that the direction of movement intended by the person can be matched with the direction recognized by the computer. .

  In the present invention, the part coordinate calculation means detects pixels around the image information (for example, four sides), and determines, for example, whether or not a human body part is recorded in the pixel to determine the position of the human body part. To do. Thereby, it is recognized correctly from which direction of the imaging region the human body part was inserted.

  According to the present invention, the shape of a human hand is recognized by using the first part as the wrist and the second part as the palm or the back of the hand. At this time, the part coordinate calculating means calculates the coordinates so that the center part of the wrist is the first part coordinate and the center of gravity of the palm or the back of the hand is the second part coordinate. Therefore, the angle of the hand inserted into the imaging region is It is recognized correctly.

The image processing method of the present invention is an image processing method for capturing an image of a human body part in an imaging region and acquiring it as image information, and recognizing the action of the human body part based on the image information. A human body part region extracting step for extracting a region of the part, and a pixel surrounding the image information is detected from the human body parts extracted in the human body part region extracting step, and the first coordinates which are coordinates of the center part of the wrist A part coordinate calculating step of calculating a part coordinate and a second part coordinate which is a coordinate of the center of gravity of the palm or the back of the hand; and a straight line connecting the first part coordinate and the second part coordinate calculated in the part coordinate calculating step An insertion angle calculation step of calculating an insertion angle of the human body part into the imaging region from the inclination, and an image rotation correction for correcting the rotation of the image information based on the insertion angle calculated in the insertion angle calculation step Specially equipped with a process To.

  According to the image processing method of the present invention, when a human body part such as a hand or a foot is inserted into the imaging region, it is first acquired as image information. In the human body part region extracting step, the human body part region is extracted from the image information, and in the part coordinate calculating step, the coordinates of the first part and the second part of the extracted human body parts are calculated. Therefore, the shape and position of the human body part in the image information are recognized.

  In the insertion angle calculation step, the angle (insertion angle) of the human body part inserted into the imaging region is calculated from the inclination with respect to the coordinate axis of the straight line connecting the first part coordinate and the second part coordinate. . In the image rotation correction step, the image information is rotationally corrected based on the insertion angle. Thereby, even when a human body part is inserted from an oblique direction with respect to the imaging region, it can be corrected in a certain direction. Therefore, when a human body part such as a hand or a foot moves, the image information is corrected in a certain direction by the above-described series of processing steps, so that the direction of movement intended by the person can be matched with the direction recognized by the computer. .

  As described above, the shape of the part of the human body can be accurately recognized, and the direction of the motion intended by the person can be matched with the direction recognized by the computer.

The figure explaining the positional relationship of a person and a camera. The flowchart of image processing (whole). The figure explaining hand area extraction. The figure explaining wrist coordinate calculation. The flowchart of a wrist coordinate calculation process. Chart of wrist coordinate calculation table. The figure explaining palm coordinate calculation. The figure explaining manual insertion angle calculation. The figure explaining image rotation correction | amendment. The figure explaining the movement of a hand and the direction which a computer recognizes.

  First, the positional relationship between a person and a camera will be described with reference to FIG.

  As shown in FIG. 1 (a), a camera 2 corresponding to the “imaging means” of the present invention is disposed in front of or below the person 1. Regardless of the position of the camera 2, the imageable range is expanded in proportion to the distance from the camera 2, and an object within the range can be imaged.

  The camera 2 is connected to the computer 3 by a cable. Since the image information captured by the camera 2 is transmitted to the computer 3, the image information can be confirmed on a monitor (not shown) of the computer 3, for example. Further, the computer 3 can move the pointer (cursor) in the monitor by analyzing the image information, for example, by the movement of the hand of the person 1. The computer 3 includes an image processing unit (an “image processing apparatus” of the present invention) inside.

  Next, FIG.1 (b) is a figure when the state of Fig.1 (a) is seen from upper direction. The area S is an area in which the camera 2 can image the hand of the person 1 (hereinafter referred to as an imaging area or simply an area). In the area S, for example, the person 1 moves the hand to the right (in the positive x-axis direction). The computer 3 analyzes the image and recognizes that the hand has moved to the right. Thereby, in the above example, the pointer moves rightward on the monitor.

  Similarly, when the person 1 moves his hand in the forward direction (y-axis positive direction), the computer 3 recognizes that the hand has moved in that direction, and the pointer moves downward on the monitor. This technology makes it possible to operate equipment switches and control robots.

  In addition, the person 1 is not always in the same direction with respect to the camera 2, but sometimes moves to change the standing position. Therefore, the hand of the person 1 may be inserted from the lateral direction or the oblique direction of the region S.

  For example, the person 1 may insert his hand from the left lateral direction of the imaging region S, move the pointer to the right, and move the hand to the right. Since this direction is the positive y-axis direction, the pointer is considered to move downward, but in the present invention, the direction in which the hand is inserted is detected to correct the image information. It transforms so that it moves in the right direction that person 1 intends. Hereinafter, image processing performed by the computer 3 to realize this will be described.

  FIG. 2 is a flowchart of image processing (overall) according to the embodiment. Hereinafter, the details of each process will be described by taking as an example a case where a human hand is inserted into the imaging region S. This image processing is executed by an image processing unit constituted by a CPU or the like of the computer 3. The image processing unit corresponds to “human body region extraction unit”, “part coordinate calculation unit”, “insertion angle calculation unit”, and “image rotation correction unit” of the present invention.

  First, hand region extraction processing is performed (step S10). This is processing for extracting a hand region from an image captured by the camera 2. In this embodiment, a hand is imaged using a range image camera. When the distance image camera is used, the hand region can be easily extracted by setting the distance between the hand inserted into the imaging region S and the distance image camera in a predetermined range. Hereinafter, extraction of a hand region will be described with reference to FIG.

  FIG. 3 is an image when the hand region 4 is extracted by the hand region extraction process. As shown in the drawing, the wrist is in the lower left part of the image region R, and the palm is in the center part. The image in the image region R corresponds to “image information” of the present invention.

  There are several methods for extracting a hand region other than the above method. For example, when an ordinary camera is used, a hand region can be extracted by extracting information corresponding to skin color from RGB information. In addition, when there is no possibility that an object other than the hand appears in the camera, the hand region can be easily extracted by the background subtraction method. In addition, a stereo camera that can calculate the distance to the object by arranging the two lenses so as to generate parallax may be used.

  Returning to FIG. 2, after the hand region extraction process is completed, the process proceeds to the wrist coordinate calculation process (step S20). For calculating the wrist coordinates, information on the outer peripheral pixel value of the image (image region R) obtained by the distance image camera is used.

  FIG. 4 is a diagram illustrating an area necessary for calculating wrist coordinates. First, a pixel value acquisition region P for acquiring pixel values around the image region R including the hand region 4 extracted by the hand region extraction process (step S10) (hatched portion in the figure, hereinafter referred to as region P). ).

  Next, the region P is divided into four, with the left side portion of the region P as the left region (A), the right side portion as the right region (B), the upper side portion as the upper region (C), and the lower side portion as the lower region (D). And the pixel value which is the value of a pixel is detected about these four area | regions (A)-(D).

  Here, when an object such as a hand is present in each region, a predetermined pixel value (luminance value) is detected, but when there is no object, a pixel value “0” is detected. Use. Further, in the case of an object having a certain size, pixels having luminance values are continuously present, and each region is examined in consideration of this.

  Next, wrist coordinate calculation processing will be described with reference to FIG. The image processing unit of the embodiment examines the pixel values in the above-described regions (A) to (D) in the following order, and calculates wrist coordinates.

  First, it is determined whether or not an arbitrary number of pixels having a pixel value ≠ 0 are continuous in the left region (A) (step S201). If an arbitrary number of portions where pixel value ≠ 0 (with luminance value) exist continuously, it means that there is a hand in the left area (A). At this time, the process proceeds to step S202. On the other hand, if an arbitrary number of pixel value ≠ 0 portions do not exist continuously, this means that there is no hand in this area. At this time, the process proceeds to step S203.

  If “YES” is determined in the step S201, “1” is set to the flag F_left (step S202). On the other hand, if “NO” is determined in the step S201, the flag F_left is set to “0” (step S203). Although details will be described later, the flag F_left is used to calculate the y coordinate of the wrist coordinates.

  Next, it is determined whether or not an arbitrary number of pixels having a pixel value ≠ 0 are continuous in the right region (B) (step S204). This is a determination as to whether or not there is a hand in the right region (B). If there is an arbitrary number of portions where the pixel value is not equal to 0, the process proceeds to step S205. On the other hand, if an arbitrary number of portions where the pixel value ≠ 0 does not exist continuously, the process proceeds to step S206.

  If “YES” is determined in the step S204, the flag F_right is set to “1” (step S205). On the other hand, if “NO” is determined in the step S204, the flag F_right is set to “0” (step S206). The flag F_right is also used to calculate the y coordinate of the wrist coordinates.

  Next, it is determined whether or not an arbitrary number of pixels having a pixel value ≠ 0 are continuous in the upper region (C) (step S207). This is a determination as to whether or not there is a hand in the upper region (C). If there is an arbitrary number of consecutive pixel values ≠ 0, the process proceeds to step S208. On the other hand, if an arbitrary number of portions where the pixel value ≠ 0 does not exist continuously, the process proceeds to step S209.

  If “YES” is determined in the step S207, “1” is set to the flag F_top (step S208). On the other hand, if “NO” is determined in the step S207, the flag F_top is set to “0” (step S209). Although details will be described later, the flag F_top is used to calculate the x coordinate of the wrist coordinates.

  Next, it is determined whether or not an arbitrary number of pixels having a pixel value ≠ 0 are continuous in the lower region (D) (step S210). This is a determination as to whether or not there is a hand in the lower region (D). If an arbitrary number of portions where the pixel value ≠ 0 exists continuously, the process proceeds to step S211. On the other hand, if an arbitrary number of portions where the pixel value ≠ 0 does not exist continuously, the process proceeds to step S212.

  If “YES” is determined in the step S210, “1” is set to the flag F_bottom (step S211). On the other hand, if “NO” is determined in the step S210, “0” is set to the flag F_bottom (step S212). The flag F_bottom is also used to calculate the x coordinate of the wrist coordinates.

  Finally, wrist coordinates are calculated from the setting of each flag (step S213). In calculating the wrist coordinates, it is necessary to confirm the establishment of each flag set above. Hereinafter, a method for calculating wrist coordinates will be described with reference to FIG. For the image region R, the x axis and the y axis are taken in the same direction as the imaging region S, and the origin is at the upper left of the image region R.

  First, the x coordinate of the wrist coordinates is determined using the upper and lower flags (F_top, F_bottom) of the image (see FIG. 6A).

  (I) When F_top = 1 and F_bottom = 1, it is determined that the hand region is not in a state where the hand region can be extracted because the hand exists in a state of straddling the upper side and the lower side of the image region R. In this case, the x coordinate is not calculated.

  (Ii) When F_top = 1 and F_bottom = 0, the hand is in the upper side of the image region R. Therefore, the average x coordinate of the pixels in the upper region (C) (pixel value ≠ 0) is defined as “wrist x coordinate”. Further, the x coordinate final determination flag X_Flag is set to “0”. Details of the final determination by the x-coordinate final determination flag will be described later.

  (Iii) When F_top = 0 and F_bottom = 1, the hand is in the lower side of the image region R. Therefore, the average x coordinate of the pixels (pixel value ≠ 0) in the lower region (D) is defined as “wrist x coordinate”. Again, the x-coordinate final determination flag X_Flag is set to “0”.

  (Iv) When F_top = 0 and F_bottom = 0, the hand is not on the upper side or the lower side of the image region R, but may be present on the left or right side of the image region R. Here, the “x coordinate of the wrist” is not determined, but the x coordinate final determination flag X_Flag is set to “1”, and the “x coordinate of the wrist” is determined by the final determination described later.

  Next, the y coordinate of the wrist coordinate is determined by the left and right flags (F_left, F_right) of the image (see FIG. 6B).

  (I) When F_left = 1 and F_right = 1, it is determined that the hand region is not in a state where the hand region can be extracted because the hand exists in a state of straddling the left and right sides of the image region R. In this case, the y coordinate is not calculated.

  (Ii) When F_left = 1 and F_right = 0, the hand is on the left side of the image region R. Accordingly, the average y coordinate of the pixels in the left area (A) (pixel value ≠ 0) is defined as “wrist y coordinate”. Further, the y coordinate final determination flag Y_Flag is set to “0”. Details of the final determination by the y-coordinate final determination flag will be described later.

  (Iii) When F_left = 0 and F_right = 1, the hand is on the right side of the image region R. Accordingly, the average y coordinate of the pixels in the right region (B) (pixel value ≠ 0) is defined as the “y coordinate of the wrist”. Again, the y-coordinate final determination flag Y_Flag is set to “0”.

  (Iv) When F_left = 0 and F_right = 0, the hand is not on the left side or the right side of the image region R, but may be present on the upper side or the lower side of the image region R. Here, the “y coordinate of the wrist” is not determined, but the y coordinate final determination flag Y_Flag is set to “1”, and the “y coordinate of the wrist” is determined by the final determination described later.

  Finally, final “wrist x-coordinate” and “wrist y-coordinate” are determined based on the above result and final determination flags (X_Flag, Y_Flag) (see FIG. 6C). In the following, “pixel” means a pixel having a pixel value ≠ 0.

  (I) When X_Flag = 1 and Y_Flag = 1, it means that there is no hand on any side around the image region R. In this case, since the hand region cannot be extracted, the x coordinate and the y coordinate are not calculated.

  (Ii) When X_Flag = 1 and Y_Flag = 0, it means that the hand exists on the left side or the right side of the image region R and does not straddle the upper side and the lower side. Here, wrist coordinates are calculated according to the following calculation method α.

(Α-1) When F_left = 1,
Wrist coordinates (x, y) = (0, average y coordinate of pixels in left area (A))
(Α-2) When F_right = 1,
Wrist coordinates (x, y) = (average y coordinate of pixels in the right end of the image area R, right area (B))
Thereby, wrist coordinates in this case are determined.

  (Iii) When X_Flag = 0 and Y_Flag = 1, it means that the hand exists on the upper side or the lower side of the image region R and does not straddle the left side and the right side. Here, wrist coordinates are calculated according to the following calculation method β.

(Β-1) When F_top = 1
Wrist coordinates (x, y) = (average x coordinate of pixels in upper region (C), 0)
(Β-2) When F_bottom = 1
Wrist coordinates (x, y) = (average x coordinate of lower region (D) pixels, lower end of image region R)
Thereby, wrist coordinates in this case are determined.

  (Iv) When X_Flag = 0 and Y_Flag = 0, it means that the hand exists over the upper side or the lower side of the image region R and the left side or the right side (any combination). Here, wrist coordinates are calculated according to the following calculation method γ.

(Γ−1) When F_left = 1 and F_top = 1 (upper left corner of the image region R),
Wrist coordinates (x, y) = (average x coordinate of pixels in upper area (C), average y coordinate of pixels in left area (A))
(Γ-2) When F_left = 1 and F_bottom = 1 (lower left corner of the image area R),
Wrist coordinates (x, y) = (average x coordinate of pixels in lower region (D), average y coordinate of pixels in left region (A))
(Γ-3) When F_right = 1 and F_top = 1 (upper right corner of the image region R),
Wrist coordinates (x, y) = (average x coordinate of pixels in upper area (C), average y coordinate of pixels in right area (B))
(Γ-4) When F_right = 1 and F_bottom = 1 (lower right corner of the image region R),
Wrist coordinates (x, y) = (average x coordinate of pixels in lower region (D), average y coordinate of pixels in right region (B))
Thereby, the wrist coordinates when the hand is obliquely inserted from the corner (any one of the four corners) with respect to the imaging region S are determined. As described above, the wrist coordinates can be calculated regardless of the direction in which the hand is inserted into the imaging region S.

  Returning to FIG. 2 again, after the wrist coordinate calculation process is completed, the process proceeds to the palm coordinate calculation process (step S30). This is a process of calculating palm coordinates from the image of the hand area extracted in the above-described hand area extraction process (step S10). Hereinafter, a palm coordinate calculation method will be described with reference to FIG.

  FIG. 7A is an image of the hand region 4 extracted by the hand region extraction process. There are several methods for extracting a palm region from this image. In the embodiment, a method based on Opening processing is used.

  The opening process is a process for removing fine patterns and small patterns by repeatedly performing the erosion process (shrinkage process) and the dilation process (expansion process) for the same number of images. When the opening process is performed on the image region R in FIG. 7A, the finger and wrist patterns are removed from the hand region 4, and only the palm region 5 remains (see FIG. 7B).

  Further, the center of gravity of the palm region 5 is calculated and used as palm coordinates. If the extracted hand region 4 is on the back side of the hand, the back hand coordinates are calculated instead of the palm coordinates.

  As another method of cutting out the palm region from the image of the hand region 4, there is a method of matching a palm shape mask with the image of the hand region and identifying a region having a high matching rate as a palm. In this method, the shape mask needs to correspond to the area of the hand region that changes according to the distance, or to rotate according to the insertion angle of the hand, but can be used instead of the opening process.

  Returning to FIG. 2 again, after the palm coordinate calculation process is completed, the process proceeds to the manual insertion angle calculation process (step S40). This is based on the wrist coordinates (referred to as coordinate A) calculated in the wrist coordinate calculation processing (step S20) and the palm coordinates (referred to as coordinate B) calculated in the palm coordinate calculation processing (step S30). This is a process for calculating the insertion angle (denoted as angle φ). Hereinafter, a method for calculating the manual insertion angle will be described with reference to FIG.

  In FIG. 8, the x-axis and y-axis are taken as shown, and wrist coordinates A (x1, y1) and palm coordinates B (x2, y2) in the hand region 4 are shown, and these two points are connected by a straight line. Yes. At this time, the angle formed by this straight line and the x axis is the insertion angle φ. The angle φ is given by the following formula (1).

  In an image rotation correction process to be described later, the image is rotationally converted by the insertion angle φ calculated in this process to obtain a new image.

  Returning to FIG. 2 again, after the manual insertion angle calculation process is completed, the process proceeds to the image rotation correction process (step S50). This is because the image region R is rotated by the angle φ around the center coordinate (coordinate C) using the insertion angle φ calculated in the manual insertion angle calculation process (step S40) (counterclockwise). This is a process for creating a new image. Hereinafter, the image rotation correction will be described with reference to FIG.

  First, FIG. 9A shows an image (image region R) before correction. When the palm coordinate is point B (x, y) and the center coordinate is point C (cx, cy), the vector v going from point C to point B is vector v = (x-cx, y-cy). .

  Since the hand insertion angle is φ when the wrist coordinates are point A, the rotation of the vector v by the angle φ around the coordinate C is always converted into the imaging region S of the hand. It is corrected in a certain direction. At this time, a matrix T represented by the following equation (2) is used for rotational conversion of the angle φ of the vector v.

  FIG. 9B shows a corrected image (image region R ′) that has been subjected to the rotation conversion described above. By the rotation conversion, the coordinate B moves to the coordinate B ′ (x ′, y ′). Of course, since all the pixels except the coordinate C move in the same manner, the hand region 4 becomes the hand region 4 '.

  At this time, the vector v ′ traveling from the coordinate C to the coordinate B ′ becomes the vector v ′ = (x′−cx, y′−cy), and thus the coordinate B ′ (x ′, y ′) after the rotation conversion is Is given by the following equation (3).

This rotation conversion is performed on each image (every frame) captured by the camera 2.

  As described above, the rotation conversion is sequentially performed regardless of the direction of the imaging region S from which the hand is inserted. Therefore, the hand area 4 always faces a certain direction, and when the person moves his / her hand, for example, when trying to move the pointer on the monitor, the direction of movement intended by the person and the moving direction of the pointer are always the same. To do.

  Finally, with reference to FIG. 10, various processes when a person actually moves his / her hand within the imaging region S will be described. The following is an example when the left hand is inserted into the imaging area S and a person moves the pointer in the right direction.

  First, when the left hand is inserted obliquely upward from the left side of the imaging region S, the hand region 6 is extracted in the image region R, and wrist coordinates A (x1, y1) and back of the hand coordinates B (x2, y2) are calculated. (See FIG. 10A). The insertion angle φ1 is also calculated with the horizontal direction of the image region R as the x axis.

  Since the hand region 6 is rotationally corrected based on the insertion angle φ1, the coordinates A (x1, y1) and the coordinates B (x2, y2) are converted into coordinates A ′ (x1 ′, y2 ′) and coordinates B ′, respectively. Moving to (x2 ′, y2 ′), the hand region 6 becomes the hand region 6 ′ (see FIG. 10B).

  Next, when a person moves the left hand within the imaging region S, the hand region 7 is extracted from the image region R (see FIG. 10A). Actually, the hand regions 6 and 7 are in different frames, but are shown in one image region R for explanation.

  When the wrist coordinate A (x3, y3) and the back coordinate B (x4, y4) of the hand region 7 are calculated, the vector m and the coordinate B (from the coordinate A (x1, y1) to the coordinate A (x3, y3) Since the vector n from x2, y2) to the coordinate B (x4, y4) is determined, the image processing unit recognizes that the left hand has moved.

  Further, since the insertion angle φ2 is calculated from the coordinates A (x3, y3) and the coordinates B (x4, y4), the rotation is corrected based on the insertion angle φ2, and the coordinates A (x3, y3), coordinates B ( x4, y4) move to coordinates A ′ (x3 ′, y3 ′) and coordinates B ′ (x4 ′, y4 ′), respectively. As a result, the hand region 7 becomes the hand region 7 '(see FIG. 10B).

  Accordingly, in the corrected image (image region R ′), the vector m ′ and the coordinate B ′ (x2 ′, y2) from the coordinate A ′ (x1 ′, y2 ′) to the coordinate A ′ (x3 ′, y3 ′) are displayed. A vector n ′ from “) to the coordinate B ′ (x4 ′, y4 ′) is determined.

  The image processing unit recognizes the direction of the motion intended by the person based on the vector m ′ and the vector n ′. Here, since the magnitudes and directions of the vectors m ′ and n ′ are almost the same, the positive direction of the x axis is set as the recognition direction according to the vectors m ′ and n ′, and the pointer moves to the right.

  Actually, three or more images are created while moving the hand by several centimeters, so the image processing unit can connect the vector of the number of images to obtain a trajectory and draw a curve by hand. In some cases, the direction can be accurately recognized.

  Even when the shape of the hand changes, the image processing unit can recognize the change in the shape from a plurality of images. Therefore, the present invention can also be used when a device is operated by a gesture, and even in this case, the shape of the hand can be accurately determined from any direction of the imaging region S by the above-described series of processing. Can be recognized.

1 person 2 camera (imaging means)
3 Computer (image processing device)
4, 4 ', 6, 6', 7, 7 'Hand area 5 Palm area P Pixel value acquisition area R Image area S Imaging area

Claims (2)

An image processing apparatus for recognizing an action of a human body part based on image information acquired by the imaging unit using an imaging unit that images a human body part in an imaging region and acquires the image information.
A human body part region extracting means for extracting a human body part region from the image information;
Among the human body parts extracted by the human body part region extracting means , pixels around the image information are detected, and the first part coordinates which are the coordinates of the center part of the wrist and the coordinates of the center of gravity of the palm or the back of the hand . A part coordinate calculating means for calculating two part coordinates;
An insertion angle calculation means for calculating an insertion angle to the imaging region of the human body part from an inclination of a straight line connecting the first part coordinate and the second part coordinate calculated by the part coordinate calculation unit;
An image processing apparatus comprising: an image rotation correction unit that rotationally corrects the image information based on the insertion angle calculated by the insertion angle calculation unit. An image processing method for imaging a human body part in an imaging region and acquiring it as image information, and recognizing the operation of the human body part based on the image information,
A human body part region extracting step of extracting a region of a human body part from the image information;
Among the human body parts extracted in the human body part region extraction step, pixels around the image information are detected, and the first part coordinates that are the coordinates of the center part of the wrist and the coordinates of the center of gravity of the palm or the back of the hand A part coordinate calculation step of calculating a second part coordinate;
An insertion angle calculation step of calculating an insertion angle to the imaging region of the human body part from the inclination of the straight line connecting the first part coordinate and the second part coordinate calculated in the part coordinate calculation step;
An image rotation correction step of rotating and correcting the image information based on the insertion angle calculated in the insertion angle calculation step.