JP5233977B2 - Crew attitude estimation device - Google Patents

Description

  The present invention relates to an occupant posture estimation device that estimates the posture of a vehicle occupant.

  Conventionally, a method of estimating the posture of a human body by detecting human body feature points such as a shoulder, an elbow, and a hand from an image of the human body (see, for example, Patent Document 1) is known. There is also known a method for estimating the behavior of a vehicle occupant by detecting human body feature points of an occupant (hereinafter referred to as a vehicle occupant) in a vehicle cabin (see, for example, Patent Document 2).

  Here, since the movement space of the vehicle occupant is three-dimensional, when the human occupant feature point of the vehicle occupant is detected from the monocular image captured in two dimensions, the vehicle occupant's arm is in front of the upper body of the vehicle occupant If it is behind the head, so-called self-shielding occurs where the arm part is hidden behind the head or torso part. When this self-occlusion occurs, the estimation accuracy of the posture of the vehicle occupant decreases.

  For such a problem, a method of installing a plurality of cameras in the vehicle interior, acquiring a three-dimensional image of the vehicle occupant, and detecting an arm portion from the acquired three-dimensional image (for example, see Patent Document 3) ) And a method of detecting an arm portion from a three-dimensional image of a vehicle occupant by installing a 3D camera in the vehicle interior (see, for example, Patent Document 4).

  Conventionally, in the technique of markerless motion capture, a method for taking measures against self-shielding using a posture dictionary with shielding information (see, for example, Patent Document 5) is known.

JP 2007-66094 A JP 2008-140268 A JP 2008-84141 A JP 2008-2838 A JP 2007-310707 A

  However, in the detection methods described in Patent Literatures 3 and 4, in order to enable posture estimation even when self-occlusion occurs when estimating the posture of a vehicle occupant, a plurality of cameras are arranged in the vehicle interior. Or, a special 3D camera that is not usually mounted on the vehicle is required. Furthermore, when a special camera is used, there is a possibility that specifications such as a photographed image size and a photographing frame rate do not match the specifications required for posture estimation. That is, the above detection method has a problem that the cost of the posture estimation device increases.

  In addition, the method described in Patent Document 5 requires creation of a posture dictionary including various posture images, which increases costs and limits the number of posture estimation target persons.

  The present invention has been made in view of these problems, and an object of the present invention is to provide an occupant posture estimation device that can improve posture estimation accuracy when self-occlusion occurs with a simple configuration.

In the occupant posture estimation apparatus according to claims 1 and 2 made to achieve the above object, the photographing means repeatedly photographs a predetermined photographing region in the vehicle and acquires an in-vehicle image that is a photographed image. Then, the human body feature point detecting means uses the in-vehicle image acquired by the photographing means to specify a plurality of predetermined joint positions of the vehicle occupant as human body feature points in order to specify the posture of the vehicle occupant included in the in-vehicle image. Further, the human body feature point position predicting means detects the positions of the plurality of human body feature points after a predetermined time has elapsed based on the change in the position of the human body feature point detected by the human body feature point detecting means. Predict. Further, the behavior predicting unit predicts the occupant behavior that is the behavior of the vehicle occupant based on the human body feature point predicted position that is the position of the human body feature point predicted by the human body feature point position predicting unit. Further, the self-shielding judging means judges whether or not self-shielding, which is a phenomenon in which the human body feature points are hidden by the vehicle occupant, based on the prediction result by the behavior predicting means. Then, when the self-shielding countermeasure means determines that self-shielding occurs, the self-shielding judgment means determines that the human body feature point hidden by the self-shielding is a self-shielding feature point based on the prediction result by the behavior prediction means. Implement self-shielding measures, which are measures to minimize the detection of human body feature points other than points by the body feature point detection means, and prevent self-shielding from occurring. When it is determined by the occlusion determination means, the human body feature point detection means is made to detect the human body feature point by a method that does not consider the occurrence of self-occlusion.

  According to the occupant posture estimation device configured as described above, the change of the human body feature point of the vehicle occupant is predicted, and thereby the behavior of the vehicle occupant is predicted. If self-shielding does not occur, human body feature points are detected by a method that does not consider self-shielding. If self-shielding occurs, self-shielding measures are taken based on the prediction results of vehicle occupant behavior. To implement. That is, when self-occlusion occurs, human body feature points can be detected in consideration of occupant behavior when self-occlusion occurs.

  For this reason, when the self-occlusion does not occur, the human body feature point can be detected using a general-purpose detection method that does not consider the occurrence of the self-occlusion. That is, it is not necessary to use a special photographing means like a 3D camera. Furthermore, even when self-occlusion occurs, occupant behavior can be taken into account, so that human body feature points can be detected using a method improved from the above general-purpose detection method based on occupant behavior. it can. That is, it is not necessary to use a special photographing means like a 3D camera. And when self-occlusion occurs, measures are taken to minimize the influence of the self-occlusion feature points, so that self-occlusion is less than when using a general-purpose detection method. It is possible to improve the detection accuracy of human body feature points when shielding occurs.

From the above, according to the occupant posture estimation device according to claims 1 and 2 , without using a special imaging means like a 3D camera, using a general-purpose imaging means like a 2D camera, that is, The estimation accuracy for estimating the posture of the vehicle occupant can be improved with a simple configuration.

Further, in the passenger posture estimating device according to claim 1 and 2, behavior predicting means, the position and the body feature points of the body part which is specified based on the detected human body feature point position by the body feature point detecting means positional relationship between the predicted position, and, based on at least one of the positional relationship between the installed operation target position and the body feature point predicted position in the vehicle, predicts the occupant behavior.

  That is, when a human body feature point prediction position is approaching a certain body part, it is predicted that the vehicle occupant is going to contact this body part, and the human body feature point prediction position is close to a certain operation target position. In this case, it can be predicted that the vehicle occupant is going to contact the operation target. That is, occupant behavior can be easily predicted.

Further, in the passenger posture estimating device according to claim 1, behavior predicting means, the position of the head of the vehicle occupant is specified based on the human body feature points of the shoulders of the vehicle occupant, human hand of the vehicle occupant Based on the positional relationship with the predicted feature point position, the vehicle occupant predicts whether or not the head contact action, which is a contact action on the head, is performed as the occupant action. When the behavior prediction means predicts that the partial contact behavior is performed, it is determined that self-occlusion occurs.

  That is, when the human body feature point prediction position of the vehicle occupant's hand is close to the position of the vehicle occupant's head, it can be predicted that the vehicle occupant is about to contact the head. That is, the head contact behavior can be easily predicted. Further, since it is determined that self-shielding occurs when it is predicted that the head contact action will be performed, it is possible to easily determine whether or not self-shielding has occurred.

Further, the occupant posture estimating device according to claim 2, behavior predicting means, connecting the body feature point predicted position of the hand of the right arm of a vehicle occupant, the body feature point predicted position of the right arm of the elbow of the vehicle occupant line The right arm prediction line segment, the line segment connecting the human body feature point prediction position of the left hand of the vehicle occupant and the human body feature point prediction position of the left arm elbow of the vehicle occupant as the left arm prediction line segment And the left arm prediction line segment, the vehicle occupant predicts that the arm crossing action, which is the action of crossing the right arm and the left arm, is performed as the occupant action, and the self-occlusion judging means If it does, it will be judged that self-occlusion occurs when the behavior prediction means predicts.

  According to the occupant posture estimation device configured as described above, it is possible to easily predict the behavior of the vehicle occupant to cross the right arm and the left arm, and to easily determine whether or not self-occlusion has occurred. .

1 is a block diagram illustrating a schematic configuration of an occupant posture estimation device 1. FIG. It is a flowchart which shows a driver | operator operation assistance process. It is a figure which shows the example of the image image | photographed with the camera 11, and the example of the detected human body feature point. It is a figure which shows the 1st-3rd method of detecting a human body feature point. It is a figure which shows the 4th-6th method of detecting a human body feature point. It is a figure explaining the method of detecting a human body feature point at the time of self-occlusion occurrence. It is a figure which shows the calculation method of a human body feature point estimated position. It is a figure which shows the 1st-2nd method of estimating a driver | operator's action. It is a figure which shows the 3rd-4th method of estimating a driver | operator's action.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram illustrating a schematic configuration of an occupant posture estimation device 1.
The occupant posture estimation device 1 according to the present embodiment is mounted on a vehicle, and as shown in FIG. 1, a camera 11 that continuously captures a driver's seat and its surroundings, and data of images captured by the camera 11 are temporarily stored. An image capture board 12 that stores data, a data storage device 13 that stores various data, an in-vehicle LAN communication unit 14 that exchanges various vehicle information and the like with other devices via an in-vehicle LAN (not shown), and various guides An audio output unit 15 for outputting sound, an illumination device 16 for illuminating the vehicle interior with visible light or infrared light for photographing by the camera 11, the image capture board 12, the data storage device 13, and the in-vehicle LAN described above Various processes are executed in response to an input from the communication unit 14, and the in-vehicle LAN communication unit 14 and the control unit 17 that controls the audio output unit 15 are provided.

  The control unit 17 is configured around a known microcomputer including a CPU, a ROM, a RAM, an I / O, a bus line for connecting these components, and the like, and various types are performed based on programs stored in the ROM and the RAM. Execute the process.

  In the occupant posture estimation device 1 configured as described above, the control unit 17 performs a driver operation assist process that assists an operation performed by the driver. FIG. 2 is a flowchart showing driver operation assistance processing. This driver operation assisting process is a process that is repeatedly executed when the occupant posture estimation device 1 is activated.

  When the driver operation assisting process is executed, the CPU of the control unit 17 first acquires image data photographed by the camera 11 in S10. An example of the acquired image is shown in FIG.

Thereafter, in S20, it is set or cleared in the processing of S110 and 130 described later, and it is determined whether or not a self-shielding flag F indicating whether or not self-shielding occurs is set.
If the self-occlusion flag F is not set (S20: NO), the normal human body feature point detection algorithm that does not consider the occurrence of self-occlusion is set in S30, and the process proceeds to S50.

  As a human body feature point detection algorithm, as shown in FIG. 4A, a joint model formed by human body feature points of the upper body part of the human body and hands, elbows and shoulders is used. It is common to use a method of matching the above and a joint model as a detection constraint.

  Then, as a method of detecting the human body feature point by analyzing the pixel value information of the input image on the joint model and extracting the edge from the image and analyzing the curvature as shown in FIG. The method of detecting the human body feature point position, as shown in FIG. 4C, the method of detecting the human body feature point position by dividing the silhouette of the person, and the HOG and shape context as shown in FIG. 5A. There is a method of calculating a local feature amount and detecting a human body feature point position.

  FIG. 5B shows a method for detecting the human body feature point position by using the joint model and successively estimating and tracking the movement amount of the human body feature point position between the current frame and the previous frame. As described above, there are a method of calculating the optical flow of the body part region of the joint model and detecting the human body feature point position, and a method of detecting the human body feature point position by the particle filter as shown in FIG. In addition, when using these methods, it is necessary to execute after calculating the initial value of the human body feature point position in advance.

  On the other hand, if the self-occlusion flag F is set (S20: YES), in S40, the self-occlusion human body feature point detection algorithm is set in consideration of the occurrence of self-occlusion, and the process proceeds to S50. In addition, the human body feature point detection algorithm for self-occlusion considers the operation characteristics of each feature point when performing the action predicted in the processing of S80 described later. For example, as shown in FIG. Using the characteristic that the length LA of the shoulder and the elbow does not change when the driver operates the mobile phone, the human body feature point is detected by an algorithm to which the condition that the length of the shoulder and the elbow does not change. If the optical flow is used as the normal human body feature point detection algorithm without adding the condition that the length LA of the shoulder and the elbow does not change when operating the mobile phone, as shown in FIG. The human body feature point of the arm on the side where the mobile phone is placed on the head cannot be detected properly.

  In S50, the human body feature points of the shoulder, elbow and hand are detected using the human body feature point detection algorithm set in S30 or S40. An example of detected human body feature points is shown in FIG. The points P1, P2, P3, P4, P5, and P6 in FIG. 3B indicate human body feature points on the right shoulder, right elbow, right hand, left shoulder, left elbow, and left hand, respectively. In S50, information indicating the detected human body feature point is stored in the RAM of the control unit 17.

  Next, in S60, the moving speed V of the human body feature point detected in S50 is calculated. Specifically, the position of the human feature point detected last time is Pa, the position of the human feature point detected this time is Pb, the time when the human feature point was detected last time is Ta, and the time when the human feature point is detected this time is Tb, The moving speed V is calculated by equation (1).

V = (Pb−Pa) / (Tb−Ta) (1)
The calculation of the moving speed V of the human body feature point is executed for each of the human body feature points of the right shoulder, the left shoulder, the right elbow, the left elbow, the right hand, and the left hand.

  In S70, the predicted position of the human body feature point is calculated. Specifically, the direction from the position Pa of the previously detected human body feature point to the position Pb of the human body feature point detected this time is set as Dp, and the predicted time Tp at the moving speed V from the position Pb as the starting point to the direction Dp. The point moved (for example, 0.5 seconds in this embodiment) is set as a human body feature point predicted position Q.

The calculation of the human body feature point predicted position Q is executed for each of the human body feature points of the right shoulder, left shoulder, right elbow, left elbow, right hand, and left hand.
A calculation example of the human body feature point predicted position Q is shown in FIG. In FIG. 7, when the human body feature point on the left shoulder moves from the current position P4b at the moving speed V4 in the direction Dp4, the human body feature point predicted position Q4 when the human body feature point on the left elbow moves in the direction Dp5 at the moving speed V5. The human body feature point predicted position Q5 when moving from the current position P5b and the human body feature point predicted position Q6 when the left hand human body feature point moves from the current position P6b at the moving speed V6 in the direction Dp6 are shown.

  Thereafter, in S80, the driver's behavior is predicted based on the human body feature point prediction positions Q1 to Q6 calculated in S70 on the right shoulder, right elbow, right hand, left elbow, left shoulder, and left hand. Specifically, for example, as shown in FIG. 8A, the human body feature point predicted position Q6 of the left hand is in an area R1 (hereinafter referred to as a glove box area R1) indicating the position of the glove box in the captured image. If it is included, the driver is expected to open the glove box. That is, when the human body feature point predicted position is included in the region indicating the position of the in-vehicle device in the captured image, it is predicted that the driver is going to operate the in-vehicle device.

  For example, as shown in FIG. 8B, when the human body feature point predicted position Q6 of the left hand is included in the body part region R2 of the driver's right shoulder, the driver brings the left hand into contact with the right shoulder. Predict that it is going. In other words, if the human body feature point prediction position is included in the driver's body part region, the driver's operation related to the body part (scramble head, hold shoulders, put the mobile phone on the ear, remove the juice can) Predict that you ’re trying to do it.

  For example, as shown in FIG. 9A, the human body feature point predicted position Q6 in the left hand is within the head region R3 set by using the human body feature point positions P1b and P4b on both shoulders and the statistical data of the body part. If it is included, the driver predicts that he / she wants to touch the head (such as putting his head, putting a mobile phone on his ear, putting a can of juice on his mouth). For example, an AIST / HQL human body size / shape database 2003 may be used as the statistical data of the body part. That is, first, a ratio between the head length and the shoulder width (hereinafter referred to as the head length / shoulder width ratio) is set in advance based on the data in this database. Then, the length of the lateral side H3 of the rectangular head region R3 is determined using the human body feature point positions on both shoulders, and the lateral side length and the head length / shoulder width of the head region R3 are determined. From the ratio, the length of the side V3 in the vertical direction of the rectangular head region R3 is determined.

  For example, as shown in FIG. 9B, the human body feature point predicted positions Q1, Q2, Q3 of the right shoulder, right elbow, and right hand and the human body feature point predicted positions Q4, Q4 of the left shoulder, left elbow, and left hand. When the line segment connecting Q5 and Q6 intersects, it is predicted that the driver is about to cross both arms.

  When the processing of S80 is completed, it is determined in S90 whether or not the self-shielding flag F is set. If the self-shielding flag F is not set (S90: NO), it is determined in S100 whether self-shielding occurs based on the driver's behavior predicted in S80. For example, if it is predicted in S80 that the driver is going to perform a touch operation on the head, it is determined that self-shielding occurs, and if it is predicted in S80 that the driver is going to operate the in-vehicle device Judge that self-shielding does not occur. In this embodiment, whether or not self-occlusion occurs is determined according to each of the actions that can be predicted in S80 (hereinafter referred to as self-occlusion). A judgment table) is provided in advance, and it is judged whether or not self-shielding occurs by referring to this self-shielding judgment table.

  If it is determined that self-shielding occurs (S100: YES), the self-shielding flag F is set in S110, and the process proceeds to S140. On the other hand, when it is determined that self-shielding does not occur (S100: NO), the process proceeds to S140.

  If it is determined in S90 that the self-shielding flag F is set (S90: YES), in S120, whether or not the self-shielding is canceled based on the driver's behavior predicted in S80. to decide. This determination method is the same as the method used in S100. If it is determined that self-shielding is eliminated (S120: YES), the self-shielding flag F is cleared in S130, and the process proceeds to S140. On the other hand, when it is determined that the self-shielding is not resolved (S130: NO), the process proceeds to S140.

  When the process proceeds to S140, the in-vehicle LAN communication unit 14 and the voice output unit 15 are controlled to assist the operation performed by the driver based on the driver's behavior predicted in S80. For example, if it is predicted in S80 that the driver is going to open the glove box, the ECU that controls the opening / closing device (not shown) of the glove box via the in-vehicle LAN by controlling the in-vehicle LAN communication unit 14 When an opening command is transmitted to (not shown) and the glove box is thereby automatically opened or when it is predicted in S80 that the driver is going to operate the mobile phone while driving, the voice output unit 15 Is controlled so that a warning sound is generated from the voice output unit 15 and a display command is transmitted to the meter ECU (not shown) via the in-vehicle LAN by controlling the in-vehicle LAN communication unit 14. Warning display (not shown). Then, when the process of S140 ends, the driver operation assistance process is temporarily ended.

  In the occupant posture estimation device 1 configured as described above, the camera 11 repeatedly captures the driver's seat and the vicinity thereof, and acquires captured image data (S10). Then, using the acquired image data, the human body feature points of the driver's right shoulder, left shoulder, right elbow, left elbow, right hand, and left hand included in the image data are detected (S50), and further detected. Based on the change in the position of the human body feature point, the position of the human body feature point (human body feature point predicted position Q) after the predicted time Tp has elapsed, the right shoulder, left shoulder, right elbow, left elbow, right hand, and left hand. Prediction is performed (S60, S70). Further, based on the human body feature point prediction positions Q of the right shoulder, left shoulder, right elbow, left elbow, right hand, and left hand, the driver's behavior is predicted (S80). Further, it is determined whether self-shielding occurs based on the predicted driver behavior (S100, S120).

  If it is determined that self-occlusion occurs (S100: YES, S20: YES), the self-occlusion human body feature point detection algorithm is set (S40). This human body feature point detection algorithm for self-occlusion takes into consideration the operation characteristics of each feature point when performing predicted driver behavior. Thereby, a human body feature point is detected using the human body feature point detection algorithm for self-shielding (S50). On the other hand, when it is determined that self-occlusion does not occur (S120: YES, S20: NO), the normal human body feature point detection algorithm that does not consider the occurrence of self-occlusion is set (S30). Thereby, a human body feature point is detected using the normal human body feature point detection algorithm (S50).

  According to the occupant posture estimation device 1 configured as described above, a change in the human body feature point of the driver is predicted, and thereby the behavior of the driver is predicted. If self-shielding does not occur, human body feature points are detected by a method that does not consider self-shielding. If self-shielding occurs, self-shielding measures are taken based on the driver's behavior prediction results. To implement. That is, when self-occlusion occurs, the human body feature point can be detected in consideration of driver behavior when self-occlusion occurs.

  For this reason, when the self-occlusion does not occur, the human body feature point can be detected using a general-purpose detection method that does not consider the occurrence of the self-occlusion. That is, it is not necessary to use a special photographing means like a 3D camera. Furthermore, even when self-occlusion occurs, driver behavior can be taken into account, and human body feature points are detected using a method improved from the general-purpose detection method based on driver behavior. be able to. That is, it is not necessary to use a special photographing means like a 3D camera. When self-shielding occurs, measures are taken to minimize the detection of human body feature points due to human body feature points hidden by self-shielding (hereinafter referred to as self-shielding feature points). Therefore, the detection accuracy of the human body feature point when self-occlusion occurs can be improved as compared with the case where a general-purpose detection method is used.

  As described above, according to the occupant posture estimation apparatus 1, a general imaging unit such as a 2D camera is used without using a special imaging unit such as a 3D camera, that is, driving with a simple configuration. The estimation accuracy for estimating the posture of the person can be improved.

  Further, the human body feature point detection algorithm for self-occlusion takes into account the operation characteristics of each feature point when performing the predicted driver action. In other words, since the general-purpose detection that does not consider the occurrence of self-shielding can be implemented by adding improvements that take into account the predicted driver behavior, it is possible to implement self-shielding measures. It is not necessary to use the human body feature point detection method.

  Further, the positional relationship between the position of the body part (in this embodiment, the body part region R2 and the head region R3 on the right shoulder) identified based on the position of the detected human body feature point and the predicted body feature point Q And a driver | operator's action is estimated based on the positional relationship of the position (in this embodiment, glove box area | region R1) of the vehicle equipment installed in the vehicle, and the human body feature point prediction position Q (S80).

  That is, when the human body feature point predicted position Q is approaching a certain body part, it is predicted that the driver is going to contact this body part, and the human body feature point predicted position Q is close to the position of a certain in-vehicle device. It can be predicted that the driver is going to contact this in-vehicle device. That is, the driver's behavior can be easily predicted.

  Further, based on the positional relationship between the driver's head region R3 specified based on the human body feature points P1 and P4 on both shoulders of the driver and the human body feature point predicted positions Q3 and Q6 of the driver's hand, Whether or not the driver performs a contact action on the head is predicted (S80), and when it is predicted that the driver will perform a contact action on the head, it is determined that self-occlusion occurs (S100).

  That is, when the human body feature point prediction position of the driver's hand is close to the position of the driver's head, it can be predicted that the driver is going to contact the head. That is, the head contact behavior can be easily predicted. Further, since it is determined that self-shielding occurs when it is predicted that the head contact action will be performed, it is possible to easily determine whether or not self-shielding has occurred.

  In addition, a line segment connecting the human body feature point predicted position Q3 of the driver's right arm hand and the human body feature point predicted position Q2 of the driver's right arm elbow, and the human body feature point predicted position Q6 of the driver's left arm hand When the line connecting the human body feature point predicted position Q5 of the elbow of the driver's left arm intersects, it is predicted that the driver will perform an action (hereinafter referred to as an arm crossing action) that intersects the right arm and the left arm ( In this case, it is determined that self-occlusion occurs (S100).

That is, it is possible to easily predict the action of the driver crossing the right arm and the left arm, and to easily determine whether or not self-occlusion has occurred.
In addition, since only the human body feature points of the driver's right shoulder, left shoulder, right elbow, left elbow, right hand, and left hand are detected, lower body features that are not necessary for estimating the posture of the driver are not used. The processing load when estimating the driver's posture can be reduced.

  Moreover, the prediction time Tp when calculating the human body feature point prediction position Q is a preset constant. For this reason, it is not necessary to set the human body feature point predicted position Q to be predicted every second from the present time every time the human body feature point predicted position Q is calculated. Processing load can be reduced.

  In addition, the human body feature point predicted position Q is calculated using only two consecutive image data before and after the plurality of image data captured by the camera 11. The two front and rear image data are the minimum necessary image data for calculating the human body feature point predicted position Q, and the processing for calculating the human body feature point predicted position Q by using only the two front and rear image data. The load can be reduced.

  In the embodiment described above, the camera 11 is the photographing means in the present invention, the process in S50 is the human body feature point detecting means in the present invention, the processes in S60 and S70 are the human body feature point position predicting means in the present invention, and the process in S80 is the present process. The behavior predicting means in the invention, the processes of S100 and S120 are the self-shielding determining means in the present invention, the processes of S20 to S40 are the self-shielding countermeasure means in the present invention, and the predicted time Tp is a predetermined time in the present invention.

As mentioned above, although one Example of this invention was described, this invention is not limited to the said Example, As long as it belongs to the technical scope of this invention, a various form can be taken.
For example, in the above embodiment, when it is determined that self-occlusion occurs, a change from the normal human body feature point detection algorithm to the self-occlusion human body feature point detection algorithm is shown, but when it is determined that self-occlusion occurs Alternatively, detection of human body feature points may be interrupted. Thereby, when self-occlusion has occurred, it is possible to suppress the occurrence of a situation in which the detection accuracy of the human body feature point is deteriorated by using the normal human body feature point detection algorithm.

  When it is determined that self-occlusion occurs, the human body feature point may be detected by fixing the human body feature point based on the driver's behavior predicted in S80. For example, when it is predicted that an action of placing a mobile phone on the ear is predicted, the position of the elbow on the side holding the mobile phone is fixed and the human body feature point is detected. As a result, at least for fixed human body feature points, when self-shielding occurs, it is possible to suppress the occurrence of a situation where the detection accuracy deteriorates by using a general-purpose detection method that does not consider the occurrence of self-shielding can do.

  Further, when it is determined that self-occlusion occurs, the human body feature point may be detected by ignoring the human body feature point hidden by the self-occlusion. In addition, the position of the hidden human body feature point may be estimated by a time series filter (Kalman filter or the like) from the operation history of the human body feature point before being hidden by self-occlusion.

  Moreover, in the said embodiment, when it estimated that the driver | operator is going to open a glove box, what supported the operation which a driver | operator performs by opening a glove box automatically was shown. In addition, for example, when it is predicted that the driver's hand is approaching the navigation device, the power of the navigation device may be automatically turned on.

  In the above embodiment, the right shoulder, the left shoulder, the right elbow, the left elbow, the right hand, and the left hand are detected. However, the number of human feature points to be detected may be reduced. The human body feature points (for example, neck and waist) may be detected.

Moreover, although the said embodiment showed what calculates the predicted position of a human body feature point using two image data before and behind, you may make it use three or more image data.
In the above embodiment, the human body feature point predicted position is calculated by a simple method of calculating the moving speed V of the human body feature point based on the moving distance between the two front and rear human body feature points. The moving speed V may be calculated by a method to calculate the human body feature point predicted position.

  In the above embodiment, the camera 11 is used only for occupant posture estimation. However, the camera 11 may be used for both security functions and occupant posture estimation.

  In the above-described embodiment, the image data captured by the camera 11 is acquired via the image capture board 12. However, the image data may be acquired directly from the camera 11 without using the image capture board 12. Good.

  Moreover, in the said embodiment, what showed the operation which a driver | operator performs by transmitting various instructions to other apparatuses via in-vehicle LAN in S140 was shown. However, after the driver behavior is predicted in S80, the prediction result may be transmitted to another device via the in-vehicle LAN. Thereby, the apparatus which received the prediction result of driver | operator action can perform control which assists operation which a driver | operator performs uniquely using this prediction result.

  DESCRIPTION OF SYMBOLS 1 ... Passenger attitude | position estimation apparatus, 11 ... Camera, 12 ... Image capture board, 13 ... Data storage device, 14 ... In-vehicle LAN communication part, 15 ... Audio | voice output part, 16 ... Illumination device, 17 ... Control part

Claims (2)

An occupant posture estimation device for estimating the posture of an occupant in a vehicle,
Photographing means for repeatedly photographing a predetermined photographing region in the vehicle and obtaining an in-vehicle image that is the photographed image;
A human body feature that detects a plurality of predetermined joint positions of the vehicle occupant as human body feature points in order to identify the posture of the vehicle occupant included in the in-vehicle image, using the in-vehicle image acquired by the photographing unit. Point detection means;
Human body feature point position predicting means for predicting the position of each of the plurality of human body feature points after a predetermined time has elapsed based on a change in the position of the human body feature points detected by the human body feature point detecting means; ,
Action predicting means for predicting an occupant action that is an action of the vehicle occupant based on a human body feature point predicted position that is a position of the human body feature point predicted by the human body feature point position predicting means;
Self-shielding judging means for judging whether self-shielding, which is a phenomenon in which the human body feature point is hidden by the vehicle occupant, based on the prediction result by the behavior predicting means;
When the self-shielding is determined by the self-shielding determining means, based on the prediction result by the behavior predicting means, the human body feature point hidden by the self-shielding is defined as the self-shielding feature point. Self-shielding is performed by implementing a self-shielding measure that is a measure for minimizing the detection of the human body feature points by the human body feature point detecting means other than being affected by the self-shielding feature points. If it is determined by the self-occlusion determination means, the human body feature point detection means to detect the human body feature point in a method that does not consider the occurrence of self-shielding ,
The behavior prediction means includes
The positional relationship between the position of the body part specified based on the position of the human body feature point detected by the human body feature point detection means and the predicted position of the human body feature point, and the operation target installed in the vehicle Predicting the occupant behavior based on at least one of a positional relationship between a position and the human body feature point predicted position;
Based on the positional relationship between the position of the head of the vehicle occupant specified based on the human body feature point on both shoulders of the vehicle occupant and the predicted position of the human body feature point of the hand of the vehicle occupant, the vehicle occupant Predicting whether or not to perform the head contact behavior that is a head contact behavior as the occupant behavior,
The self-shielding judging means is
The occupant posture estimation device according to claim 1 , wherein when the behavior predictor predicts that the vehicle occupant performs the head contact behavior, the self-occlusion is determined to occur . An occupant posture estimation device for estimating the posture of an occupant in a vehicle,
Photographing means for repeatedly photographing a predetermined photographing region in the vehicle and obtaining an in-vehicle image that is the photographed image;
A human body feature that detects a plurality of predetermined joint positions of the vehicle occupant as human body feature points in order to identify the posture of the vehicle occupant included in the in-vehicle image, using the in-vehicle image acquired by the photographing unit. Point detection means;
Human body feature point position predicting means for predicting the position of each of the plurality of human body feature points after a predetermined time has elapsed based on a change in the position of the human body feature points detected by the human body feature point detecting means; ,
Action predicting means for predicting an occupant action that is an action of the vehicle occupant based on a human body feature point predicted position that is a position of the human body feature point predicted by the human body feature point position predicting means;
Self-shielding judging means for judging whether self-shielding, which is a phenomenon in which the human body feature point is hidden by the vehicle occupant, based on the prediction result by the behavior predicting means;
When the self-shielding is determined by the self-shielding determining means, based on the prediction result by the behavior predicting means, the human body feature point hidden by the self-shielding is defined as the self-shielding feature point. Self-shielding is performed by implementing a self-shielding measure that is a measure for minimizing the detection of the human body feature points by the human body feature point detecting means other than being affected by the self-shielding feature points. If it is determined by the self-occlusion determination means, self-occlusion countermeasure means for causing the human body feature point detection means to detect the human body feature point by a method not considering the occurrence of the self-occlusion;
With
The behavior prediction means includes
The positional relationship between the position of the body part specified based on the position of the human body feature point detected by the human body feature point detection means and the predicted position of the human body feature point, and the operation target installed in the vehicle Predicting the occupant behavior based on at least one of a positional relationship between a position and the human body feature point predicted position;
A line segment connecting the human body feature point predicted position of the right hand of the vehicle occupant and the human body feature point predicted position of the elbow of the right arm of the vehicle occupant is a right arm predicted line segment, and the left arm hand of the vehicle occupant When the line segment connecting the human body feature point prediction position and the human body feature point prediction position of the left arm elbow of the vehicle occupant is a left arm prediction line segment, and the right arm prediction line segment and the left arm prediction line segment intersect Predicting that the vehicle occupant performs an arm crossing action as an action of crossing a right arm and a left arm as the occupant action,
The self-shielding judging means is
When the behavior predicting means predicts that the vehicle occupant performs the arm crossing behavior, it is determined that the self-occlusion occurs.
An occupant posture estimation device.