JP6444573B2 - Work recognition device and work recognition method - Google Patents

Description

  The present invention relates to a work recognition device and a work recognition method for recognizing work contents by an operator.

2. Description of the Related Art Conventionally, there is known a technique for recognizing and analyzing a human motion by acquiring information related to a human body part or an object position using a camera image or the like.
For example, Patent Document 1 captures the motion of a worker who performs a work with both hands as a subject, and in particular, the motion of both hands of the worker is based on the motion of a reference worker that is a reference subject. A motion analysis device for analysis is disclosed.
In the motion analysis apparatus disclosed in Patent Document 1, the coordinates of both hands are acquired for each frame of the video, and the locus of both hands is acquired by tracking the coordinates. When the coordinates cannot be taken, the left hand or right hand coordinates where the coordinates cannot be taken are estimated from the coordinates where the coordinates have disappeared.

JP 2011-34234 A

  In the motion analysis device as disclosed in Patent Document 1, for example, when the movement of the object is included in the work, and the coordinates of the position of the hand or the object cannot be acquired, the coordinates are simply Since only the coordinates of the hand or the object are estimated from the coordinates of the disappeared place, there is a problem that the locus representing the work content cannot be obtained with high accuracy.

  The present invention has been made to solve the above-described problems, and an object thereof is to provide a work recognition device and a work recognition method for recognizing work contents by an operator with high accuracy.

  The work recognition apparatus according to the present invention detects a part of a worker's body based on the sensor data acquisition unit that acquires sensor data and the sensor data acquired by the sensor data acquisition unit. A body part information acquisition unit that acquires body part information about the body, an object information acquisition unit that acquires object information about the object based on the sensor data acquired by the sensor data acquisition unit, and body part information acquisition An association unit for associating an object with a body part of an operator who performed an operation based on the body part information acquired by the unit and the object information acquired by the object information acquisition unit; And a recognition result analysis unit for recognizing work performed by the worker based on association information related to the association result associated by the association unit.

  According to this invention, the work performed by the worker is recognized by associating the object with the body part of the worker who performed the work using the object. It can be recognized with high accuracy.

It is a figure explaining an example of the whole structure of the work recognition system provided with the work recognition apparatus which concerns on Embodiment 1 of this invention. It is a block diagram of the work recognition apparatus which concerns on Embodiment 1 of this invention. It is a figure explaining an example of the calculation method of the association score by the score calculation part in this Embodiment 1. FIG. It is a figure explaining an example of the correlation score calculation operation | movement by interpolation of the locus | trajectory of a position coordinate by the score calculation part in this Embodiment 1, and interpolation. It is a figure explaining an example of the correction | amendment operation | movement of the position coordinate by the position correction part in this Embodiment 1. FIG. 6A and 6B are diagrams showing an example of a hardware configuration of the work recognition apparatus according to Embodiment 1 of the present invention. 3 is a diagram illustrating an example of a hardware configuration of a control device according to Embodiment 1. FIG. It is a flowchart explaining operation | movement of the work recognition apparatus which concerns on Embodiment 1 of this invention. It is a flowchart explaining the detail of operation | movement of the body part information acquisition part in step ST802 of FIG. It is a flowchart explaining the detail of operation | movement of the object information acquisition part in step ST803 of FIG. FIG. 9 is a flowchart illustrating details of an operation of an associating unit in step ST805 of FIG. In this Embodiment 1, it is a figure explaining the output example of the recognition result of the whole operation | work by the output control part, or a work analysis result. In this Embodiment 1, it is a figure explaining the output example of the recognition result of the whole operation | work by the output control part, or a work analysis result. In this Embodiment 1, it is a figure explaining the output example of the recognition result of the whole operation | work by the output control part, or a work analysis result.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
Embodiment 1 FIG.
FIG. 1 is a diagram illustrating an example of the overall configuration of a work recognition system including a work recognition device 100 according to Embodiment 1 of the present invention.
In this work recognition system, the work recognizer 100 is based on, for example, video information received from the visible light camera 111 or the like, or sensing information received from the acceleration sensor 114 or the like. Then, the work performed using the object 131 is recognized, and the work recognition result and the work analysis result based on the work recognition result are output to an external device or the like.
The number of body parts 121 used when the worker 120 performs work is not limited to one. The worker 120 may perform work using a plurality of body parts 121 such as a right hand and a left hand, for example. Also, the number of objects 131 used when the worker 120 performs the work is not limited to one, and the worker 120 may perform work using a plurality of objects 131.
That is, in the first embodiment, the worker 120 performs work using one or more body parts 121 and one or more objects 131.

  As shown in FIG. 1, the work recognition system includes a work recognition device 100, a visible light camera 111, an infrared camera 112, a depth sensor 113, an acceleration sensor 114, a gyro sensor 115, a display 151, and a speaker 152. And a storage device 153 and a control device 154.

The visible light camera 111, the infrared camera 112, and the depth sensor 113 are imaging devices that capture the periphery of the worker 120, and transmit visible light images, infrared images, and depth images to the work recognition device 100, respectively. .
In the first embodiment, the vicinity of the worker 120 is a preset range, and at least the body part 121 and the object 131 used when the worker 120 works are mainly moved during the work. A range is included.

In the first embodiment, as shown in FIG. 1, the work recognition system includes the visible light camera 111, the infrared camera 112, and the depth sensor 113 as the imaging device. The work recognition system only needs to include at least one of these.
In the work recognition system of the first embodiment, a sensor in which two or more of the visible light camera 111, the infrared camera 112, and the depth sensor 113 are integrated may be used.
In addition to the visible light camera 111, the infrared camera 112, and the depth sensor 113, an imaging device that can capture an image around the worker 120 may be provided.
Moreover, there is no restriction | limiting in the installation number of each of the visible light camera 111, the infrared camera 112, and the depth sensor 113, for example, in order to image | photograph the periphery of the operator 120 from a different viewpoint, the visible light camera 111, the infrared camera 112, or A plurality of depth sensors 113 may be installed.

In the work recognition system according to the first embodiment, a marker may be attached to the body part 121 or the object 131 of the worker 120. As the marker, a bar code or a marker printed with a characteristic figure, a reflective marker that shines when reflected by visible light or infrared light, a color marker using a characteristic color, an infrared marker that emits infrared light, or the like is used. be able to. The work recognition apparatus 100 can detect, track, or recognize the body part 121 or the object 131 of the worker 120 with high accuracy by detecting, tracking, or recognizing the marker. Details of the work recognition device 100 will be described later.
When using a marker, all the markers to be used may be the same marker, or may be different for each body part 121 or object 131 of the worker 120.

The acceleration sensor 114 is attached to one or both of the body part 121 and the object 131 of the worker 120, and transmits time series information of acceleration to the work recognition device 100.
The gyro sensor 115 is attached to one or both of the body part 121 and the object 131 of the worker 120, and transmits time-series information on angular acceleration to the work recognition device 100.
As shown in FIG. 1, the work recognition system according to the first embodiment includes an acceleration sensor 114 and a gyro sensor 115 as sensors capable of sensing the movement of the body part 121 or the object 131 of the worker 120. However, the present invention is not limited to this, and the work recognition system only needs to include at least one of these.
In addition to the acceleration sensor 114 or the gyro sensor 115, the work recognition system may include a sensor that can sense the movement of the body part 121 and the object 131 of the worker 120.

  Hereinafter, the visible light camera 111, the infrared camera 112, the depth sensor 113, the acceleration sensor 114, and the gyro sensor 115 are collectively referred to as sensors, and data obtained from these sensors are collectively referred to as sensor data. . That is, image data such as a visible light image obtained by the visible light camera 111, an infrared image obtained by the infrared camera 112, and a depth image obtained by the depth sensor 113 are also included in the sensor data.

  The work recognition device 100 recognizes the work by capturing the movement of the body part 121 and the object 131 of the worker 120 based on at least one of the sensor data received from the sensor, and recognizes the work recognition result or the work recognition. The work analysis result using the result is output to at least one of the display 151, the speaker 152, the storage device 153, or the control device 154.

The display 151 outputs the work recognition result output from the work recognition device 100 or the work analysis result using the work recognition result as a video or the like.
The speaker 152 outputs the work recognition result output from the work recognition apparatus 100 or the work analysis result using the work recognition result by voice or the like .
The storage device 153 stores the work recognition result output from the work recognition device 100 or the work analysis result using the work recognition result.
For example, the work recognition device 100 causes the storage device 153 to store the work recognition result for a long time. The work recognition apparatus 100 may store a plurality of work recognition results in the storage device 153. The manager who manages the work of the worker 120 refers to the work recognition result stored in the storage device 153 and performs work analysis or creates a work analysis report. As described above, by storing the work recognition result or the like in the storage device 153, the manager or the like can perform the work analysis later instead of the real-time work recognition. This is only an example, and the work recognition result stored in the storage device 153 may be used by other methods.

In addition, the control device 154 performs various controls based on a work recognition result output from the work recognition device 100 or a control signal corresponding to a work analysis result using the work recognition result.
Specifically, for example, the control device 154 controls the robot to assist the work of the worker 120 based on the work recognition result and the like. This makes it possible to supply necessary parts or tools to the worker 120 according to the work situation, or to assist the work so that the work delay can be recovered when the work of the worker 120 is slow. This is only an example, and the control device 154 may perform other controls.
Hereinafter, the display 151, the speaker 152, the storage device 153, and the control device 154 are collectively referred to as an output device.
The work recognition system according to the first embodiment includes a display 151, a speaker 152, a storage device 153, and a control device 154 as shown in FIG. The system may include at least one of these.

FIG. 2 is a configuration diagram of the work recognition apparatus 100 according to Embodiment 1 of the present invention.
As shown in FIG. 2, the work recognition apparatus 100 includes a sensor data acquisition unit 210, a body part information acquisition unit 220, an object information acquisition unit 230, an association unit 240, a recognition result analysis unit 250, and an output control unit. 260.

The sensor data acquisition unit 210 acquires sensor data from at least one of the visible light camera 111, the infrared camera 112, the depth sensor 113, the acceleration sensor 114, and the gyro sensor 115.
The sensor data acquisition unit 210 outputs the acquired sensor data to the body part information acquisition unit 220 and the object information acquisition unit 230.

  The body part information acquisition unit 220 detects the body part 121 of the worker 120 based on the sensor data output from the sensor data acquisition unit 210 and acquires body part information regarding the body part 121 of the worker 120. The body part information acquisition unit 220 causes the body part information storage unit 270 to store the acquired body part information related to the body part 121 of the worker 120.

As shown in FIG. 2, the body part information acquisition unit 220 includes a body part detection unit 221, a body part tracking unit 222, and a body part recognition unit 223.
The body part detection unit 221 detects the body part 121 of the worker 120 based on the sensor data output from the sensor data acquisition part 210 and acquires the position coordinates of the part 121. The position coordinates of the part 121 may be, for example, the coordinates of an arbitrary point of the detected body part 121, or may be the coordinates of the upper right and lower left corners of a rectangle enclosed so as to include the body part 121. It is possible to appropriately set which point the position coordinate of the body part 121 of the person 120 is set to.
In addition, the body part detection unit 221 may use the position coordinates of the body part 121 of the worker 120 as two-dimensional coordinates on the acquired image or three-dimensional coordinates estimated using the depth information.
The origin and the coordinate system of the position coordinates of the part 121 detected by the body part detection unit 221 are arbitrary. For example, the body part detection unit 221 may have a two-dimensional coordinate system in which the upper left of the acquired video is the origin, the right direction is the X axis, and the lower direction is the Y axis, or a specific point in the work area is the origin, and the vertical A three-dimensional coordinate system with the upward direction as the Z-axis may be used.

Moreover, the body part detection part 221 shall acquire the position coordinate of the body part 121 of the operator 120 by arbitrary methods. The body part detection unit 221 calculates the position coordinates of the body part 121 of the worker 120 by an existing feature point based detection method such as SURF (Speed Up Robust Features) or HOG (Hisgrams of Oriented Gradients). Alternatively, the position coordinates of the body part 121 of the worker 120 may be acquired by a model-based detection method such as a neutral network. The body part detection unit 221 may employ a detection method using only a single frame image acquired from the sensor data acquisition unit 210, or may employ a detection method using a plurality of frames of video. Good. The body part detection unit 221 may acquire the position coordinates of the body part 121 of the worker 120 by any detection method using any one of an optical image, an infrared image, a depth image, and the like. The position coordinates of the body part 121 of the worker 120 may be acquired by a detection method using a combination of infrared video or depth video.
When a plurality of body parts 121 of the worker 120 are detected, the body part detection unit 221 acquires position coordinates for each detected part 121.

  The body part detection unit 221 stores the acquired position coordinates of the body part 121 of the worker 120 in the body part information storage unit 270 in association with the acquisition date information. The acquisition date / time information may be information on the acquisition date / time associated with the video frame acquired from the sensor data acquisition unit 210. When the body part detection unit 221 detects a plurality of body parts 121 of the worker 120, the body part information storage unit 270 stores the position coordinates and the acquisition date information in association with each part 121. Let

The body part tracking unit 222 acquires the locus of the body part 121 of the worker 120 detected by the body part detection unit 221. Specifically, the body part tracking unit 222 tracks the body part 121 of the worker 120 from which the body part detection unit 221 has acquired the position coordinates based on the sensor data acquired from the sensor, and moves the part 121. Get the later position coordinates.
The body part tracking unit 222 tracks the position coordinates of the body part 121 of the worker 120 based on the information about the part 121 stored in the body part information storage unit 270 by the body part detection unit 221. When there are a plurality of parts 121 detected by the body part detection unit 221, the body part tracking unit 222 acquires the position coordinates after movement for each of the plurality of parts 121.

The body part tracking unit 222 may track the part 121 by any tracking method. For example, the body part tracking unit 222 may perform tracking of the part 121 by using an existing region-based tracking method using update template matching, active search, Mean-shift method, particle filter, or the like. The part 121 may be tracked using a KLT method (Kanade-Lucas-Tomasi Feature Tracker) or a feature point-based tracking method such as SURF Tracking.
Further, the body part tracking unit 222 may track the part 121 by a tracking method using any one of an optical image, an infrared image, a depth image, and the like, or an optical image, an infrared image, a depth image, etc. The part 121 may be tracked by a detection method using a combination of the above.
The body part tracking unit 222 acquires the acquired position coordinates after the movement of the body part 121 of the worker 120, that is, the information on the trajectory of the body part 121 of the worker 120, along with information on the acquisition date and time of the position coordinates. The body part information storage unit 270 stores the information in association with the part 121 information.

The body part recognition unit 223 recognizes the type, shape, or state of the body part 121 of the worker 120 that the body part detection unit 221 acquires the position coordinates and stores in the body part information storage unit 270. In the first embodiment, the type of body part 121 of the worker 120 refers to, for example, a right hand, a left hand, an elbow, a face, and the like. Note that this is only an example, as long as the body part 121 of the worker 120 is specified. The shape or state of the body part 121 of the worker 120 is, for example, the shape of the right hand holding the object 131 when the worker 120 is holding the object 131 with the palm of the right hand, or the worker A state in which 120 is holding the object 131 or the like. Note that this is only an example, and the same applies to, for example, the case where the worker 120 pinches the object 131 with the fingertip, the worker 120 grasps the fist, and the worker 120 opens the right hand. .
The body part recognition unit 223 may recognize the type, shape, or state of the part 121 by an arbitrary recognition method. For example, the type, shape, or state of the part 121 may be recognized by an existing feature point-based detection method such as SURF or HOG, or the part 121 may be recognized by a model-based detection method such as a neutral network. The type, shape, or state may be recognized.
The body part recognizing unit 223 may recognize the type, shape, or state of the part 121 by any recognition method using an optical image, an infrared image, or a depth image. The type, shape, or state of the part 121 may be recognized by a recognition method using a combination of video and depth video.
The body part recognition unit 223 stores the information on the recognition result of the part 121 in the body part information storage unit 270 in association with the information on the part 121.
Information relating to the part 121 stored in the body part information storage unit 270 by the body part detection unit 221, the body part tracking unit 222, and the body part recognition unit 223 becomes the body part information.
The body part information by the body part detection unit 221, the body part tracking unit 222, and the body part recognition unit 223 is stored for each series of operations.

  The object information acquisition unit 230 detects the object 131 based on the sensor data output from the sensor data acquisition unit 210 and acquires object information related to the object 131. The object information acquisition unit 230 causes the object information storage unit 280 to store the acquired object information regarding the object 131.

As illustrated in FIG. 2, the object information acquisition unit 230 includes an object detection unit 231, an object tracking unit 232, and an object recognition unit 233.
The object detection unit 231 detects the object 131 based on the sensor data output from the sensor data acquisition unit 210 and acquires the position coordinates of the object 131. The position coordinates of the object 131 may be, for example, the coordinates of an arbitrary point of the detected object 131, or may be the coordinates of the upper right and lower left of a rectangle enclosed so as to include the object 131. The coordinates of which point can be set as appropriate.
The object detection unit 231 may use the position coordinates of the object 131 as two-dimensional coordinates on the acquired video or three-dimensional coordinates estimated using depth information.
The origin and coordinate system of the position coordinates of the object 131 detected by the object detection unit 231 are arbitrary. For example, the object detection unit 231 may be a two-dimensional coordinate system in which the upper left of the acquired image is the origin, the right direction is the X axis, and the lower direction is the Y axis. A three-dimensional coordinate system with the direction as the Z-axis may be used.

The object detection unit 231 acquires the position coordinates of the object 131 by an arbitrary method. The object detection unit 231 may acquire the position coordinates of the object 131 by an existing feature point-based detection method such as SURF (Speed Up Robust Features) or HOG (Hisgrams of Oriented Gradients), for example. The position coordinates of the object 131 may be acquired by a model-based detection method such as a neutral network. The object detection unit 231 may employ a detection method using only a single frame image acquired from the sensor data acquisition unit 210, or may employ a detection method using a plurality of frames of video. . Further, the object detection unit 231 may acquire the position coordinates of the object 131 by a detection method using any one of an optical image, an infrared image, a depth image, and the like, or an optical image, an infrared image, or a depth image. The position coordinates of the object 131 may be acquired by a detection method using a combination of the above.
When a plurality of objects 131 are detected, the object detection unit 231 acquires position coordinates for each detected object 131.

  The object detection unit 231 stores the acquired position coordinates of the object 131 in the object information storage unit 280 in association with the acquisition date information. The acquisition date / time information may be information on the acquisition date / time associated with the video frame acquired from the sensor data acquisition unit 210. When a plurality of objects 131 are detected, the object detection unit 231 stores the position coordinates and the acquisition date information in the object information storage unit 280 in association with each object 131.

The object tracking unit 232 acquires the trajectory of the object 131 detected by the object detection unit 231. Specifically, the object tracking unit 232 tracks the object 131 from which the object detection unit 231 has acquired the position coordinates based on the sensor data acquired from the sensor, and acquires the position coordinates after the movement of the object 131.
The object tracking unit 232 tracks the position coordinates of the object 131 based on the information regarding the object 131 stored in the object information storage unit 280 by the object detection unit 231. When there are a plurality of objects 131 detected by the object detection unit 231, the object tracking unit 232 acquires the position coordinates after movement for each of the plurality of objects 131.

The object tracking unit 232 may track the object 131 by any tracking method. For example, the object tracking unit 232 may track the object 131 using an update template matching, an active search, a Mean-shift method, or an existing region-based tracking method using a particle filter or the like. The object 131 may be tracked using a feature point-based tracking method such as a method (Kanade-Lucas-Tomasi Feature Tracker) or SURF Tracking.
In addition, the object tracking unit 232 may track the object 131 by a tracking method using any one of an optical image, an infrared image, a depth image, and the like, and an optical image, an infrared image, a depth image, or the like. The object 131 may be tracked by a detection method used in combination.
The object tracking unit 232 stores the acquired position coordinates after the movement of the object 131, that is, information on the trajectory of the object 131, together with information on the acquisition date and time of the position coordinates, in the object information storage unit 280. The information is stored in association with the information.

The object recognition unit 233 recognizes the type, shape, or state of the object 131 that the object detection unit 231 acquires the position coordinates and stores in the object information storage unit 280. In the first embodiment, the type of the object 131 is, for example, information for specifying the object 131 used when the worker 120 performs the assembly work, and refers to each part to be assembled or an assembly tool. In addition, the shape or state of the object 131 means, for example, when the worker 120 performs an assembly operation using a certain component, the direction of the component, a state where the component is incorporated in the board, or the like. Note that this is only an example, and the same applies to the case where the worker 120 performs a work using a tool, for example.
The object recognition unit 233 may recognize the type, shape, or state of the object 131 by any recognition method. For example, the type, shape, or state of the object 131 may be recognized by an existing feature point-based detection method such as SURF or HOG, or the object 131 may be detected by a model-based detection method such as a neutral network. The type, shape, or state may be recognized.
Further, the object recognition unit 233 may recognize the type, shape, or state of the object 131 by a recognition method using any one of an optical image, an infrared image, and a depth image, and the optical image, the infrared image Alternatively, the type, shape, or state of the object 131 may be recognized by a recognition method using a combination of depth images.
The object recognition unit 233 stores information on the recognition result of the object 131 in the object information storage unit 280 in association with the information on the object 131.
Information relating to the object 131 stored in the object information storage unit 280 by the object detection unit 231, the object tracking unit 232, and the object recognition unit 233 becomes object information.
The object information by the object detection unit 231, the object tracking unit 232, and the object recognition unit 233 is stored for each series of operations.

  The associating unit 240 stores the body part information related to the body part 121 of the worker 120 stored in the body part information storage unit 270 by the body part information acquisition unit 220 and the object information acquisition unit 230 stores it in the object information storage unit 280. Based on the object information regarding the object 131, the association between the object 131 and the body part 121 of the worker 120 who performed the work using the object 131 is performed. The associating unit 240 outputs the associating result to the recognition result analyzing unit 250.

The associating unit 240 includes a score calculating unit 241, a matching unit 242, and a position correcting unit 243.
The score calculation unit 241 includes the body part information related to the body part 121 of the worker 120 stored in the body part information storage unit 270 by the body part information acquisition unit 220 and the object information acquisition unit 230 stored in the object information storage unit 280. Using the stored object information regarding the object 131, an association score representing the degree of association between the body part 121 of the worker 120 and the object 131 is calculated. The degree of association between the body part 121 of the worker 120 and the object 131 indicates which worker body part 121 is likely to be associated with the movement of the object 131. The higher the association score, the more the worker It is assumed that there is a high possibility that 120 body parts 121 and the object 131 are related.
The score calculation unit 241 may calculate the association score by any method. For example, the score calculation unit 241 may increase the association score if the position coordinate between the body part 121 of the worker 120 and the object 131 is close, or the movement direction of the body part 121 of the worker 120 If the moving direction of the object 131 is close, the association score may be increased. In addition, for example, the score calculation unit 241 may increase the association score if the body part 121 of the worker 120 has a shape that holds the object 131.

Here, FIG. 3 is a diagram for explaining an example of an association score calculation method by the score calculation unit 241 in the first embodiment.
In FIG. 3, the body part information acquisition unit 220 detects the part X and the part Y as the body part 121 and acquires body part information, and the object information acquisition unit 230 acquires the objects A to D as the object 131. It is assumed that object information has been detected. Also, in FIG. 3, it is assumed that information on the locus of the position coordinates for 10 seconds of the parts X and Y and the objects A to D is obtained, and the position for 10 seconds in FIG. Show.

For example, with respect to the object A for 2 seconds from 0 to 2 seconds later, the part X and the part Y exist around the object A, but the position coordinates of the object A and the part X are the object A and the part X. It is closer than the position coordinate of Y. That is, the distance between the object A and the part X is closer than the distance between the object A and the part Y.
Therefore, the score calculation unit 241 sets the object A higher in the association score with the part X that is closer to the object A than the association score with the part Y.
Similarly, for 2 seconds from 8 to 10 seconds later, the score calculation unit 241 causes the association score between the object A and the part X closer to the object A to be more than the association score with the part Y. Is also set high.

Further, for example, if the part Y after 3 seconds is in the shape of holding the object B, the score calculation unit 241 sets the association score between the object B and the part Y high.
In this way, the score calculation unit 241 determines which body part 121 is likely to be related to the movement of the object 131 with respect to the combination of the body part 121 and the object 131 of the worker 120 by any method. Calculate the association score to represent.
Note that, as described above, the calculation method of the association score by the score calculation unit 241 described with reference to FIG. 3 is merely an example, and the score calculation unit 241 may calculate the association score by other methods. The strength of the association with respect to the combination of the working object 131 and the body part 121 of the worker 120 may be set in accordance with a preset criterion.

  In addition, when the object information acquisition unit 230 determines that the detection or tracking of the object 131 has failed and the locus of the position coordinates of the object 131 is interrupted, the score calculation unit 241 determines the position coordinates of the object 131. The association score may be calculated by combining the vanishing point and the detection point and interpolating the locus of the position coordinates.

Here, FIG. 4 is a diagram for explaining an example of position coordinate locus interpolation and post-interpolation association score calculation operations by the score calculation unit 241 in the first embodiment.
In FIG. 3, for the object A, after the position coordinates are detected after 2 seconds, the locus of the position coordinates is interrupted until 8 seconds later. Therefore, the score calculation unit 241 interpolates the locus of the position coordinates for the object A in which the locus shown in FIG. 3 is interrupted using FIG. 4, and the association score with the parts X and Y for the object A obtained by interpolating the locus. An example of the operation for calculating the will be described.
First, the score calculation unit 241 interpolates the locus of the position coordinates of the object A by combining the vanishing point and the detection point of the object A as indicated by the dotted line in FIG.
Next, the score calculation unit 241 calculates an association score between the object A and the parts X and Y in the interpolated position coordinate trajectory.

For example, the part Y is closer to the part Y than the part X with respect to the position coordinates of the object A three seconds after the interpolation. However, the position X disappearance point and the position coordinate detection point, that is, the part X for which a higher association score is calculated with the object A after 2 seconds and after 8 seconds. Therefore, for example, the score calculation unit 241 determines that there is a high possibility that the part X is used for the work accompanying the movement of the object A even while the locus of the position coordinates is interrupted. , The association score with the part X is set higher than the association score with the part Y.
The score calculation unit 241 not only detects the positional relationship between the interpolated position coordinate and the surrounding part 121 while the position coordinate of the object 131 is interrupted, but also the vanishing point and the detection point of the object 131, that is, the position The association score can be calculated in consideration of the association with the body part 121 of the worker 120 before and after the coordinates are interrupted.

  Note that, as described above, the interpolation of the position coordinate trajectory by the score calculation unit 241 and the correlation score calculation method after the interpolation described with reference to FIG. 4 are merely examples, and the score calculation unit 241 uses other methods. You may make it perform the calculation of the correlation score after interpolation of the locus | trajectory of a position coordinate, and interpolation.

For example, the score calculation unit 241 may increase the association score if the body part 121 of the worker 120 moves in the same direction.
Specifically, for example, it is assumed that the operator 120 moves a part V with the right hand from point a to point d, from point a → point b → point c → point d. At that time, since the object detection unit 231 cannot detect the component V at the point b, the object tracking unit 232 cannot track the component V halfway, and the object detection unit 231 detects the component V again at the point c. The object tracking unit 232 can resume tracking of the part V. In this case, the point b is a vanishing point and the point c is a detection point. The object tracking unit 232 can acquire the locus of position coordinates from point a to point b and the locus of position coordinates from point c to point d, but obtains the locus of position coordinates from point b to point c. It is assumed that it is not possible to do so. Further, it is assumed that the object recognition unit 233 cannot recognize whether the object 131 from the point a → the point b and the object 131 from the point c → the point d are the same object 131 or different objects 131.
In such a case, the score calculation unit 241 tries to interpolate the trajectory of the object 131 from the point b → the point c by, for example, linear interpolation, and the object 131 of the point a → the point b → the point c → the point d. If the movement direction matches the movement direction of the right hand, it is determined that the part V has been moved from point a → point b → point c → point d with the right hand, and the association score between the part V and the right hand is increased. . As the association score increases, the matching unit 242 can easily select the association when the matching unit 242 determines the combination of the body part 121 and the object 131 of the worker 120, and as a result, the point b → The discontinuity of the locus of the point c can be interpolated well. The matching unit 242 will be described later.

If the score calculation unit 241 determines that the vanishing point and the object 131 at the detection point are the same, the score calculation unit 241 increases the association score between the vanishing point and the part 121 associated with the object 131 at the detection point. Also good.
For example, in the example described above, the object recognition unit 233 recognizes that the object 131 from the point a → the point b and the object 131 from the point c → the point d are both components V. In this case, the score calculation unit 241 interpolates the locus of position coordinates from point b to point c, and increases the association score between the component V and the right hand.

In addition, the score calculation unit 241 increases the score associated with the fact that the object 131 is not moved in the body part 121 of the worker 120 if the difference between the position coordinates of the vanishing point and the detection point is equal to or less than a certain value. It may be.
For example, it is assumed that the object detection unit 231 cannot detect the component W for a moment because the left hand passes over the stationary component W for a moment. If the position of the part W is not changed before and after the object detection unit 231 can no longer detect the part W, the score calculation unit 241 determines that the part W is not moved with the left hand, and the part W If the is not moved with the left hand, increase the associated score. The score calculation unit 241 may lower the association score between the component W and the left hand.

The score calculation unit 241 is only required to perform the interpolation of the locus of the position coordinates of the object 131 and the setting of the association score after the interpolation in accordance with a preset criterion.
The score calculation unit 241 outputs information on the calculated association score to the matching unit 242.

The matching unit 242 uses the association score calculated by the score calculation unit 241 to combine the body part 121 of the worker 120 and the object 131 so as to maximize the association score within a range where the matching as the work can be achieved. To decide. That is, based on the association result in which the body part 121 of the worker 120 and the object 131 are associated with each other, the object 131 is combined with the body part 121 of the worker 120 that is determined to be most related to the object 131. Will be.
Conditions for achieving consistency as work are arbitrary. For example, the matching unit 242 prevents the same body part 121 from being present at a position coordinate separated by a certain value or more at the same time, or prevents the same object 131 from being associated with a plurality of conflicting operations at the same time. Is a condition for achieving consistency. The matching unit 242 determines a combination of the body part 121 and the object 131 of the worker 120 that satisfies the condition and maximizes the sum of the association scores.

Here, an example of an operation in which the matching unit 242 determines a combination with the parts X and Y for the objects A to D detected in 10 seconds after 0 to 10 seconds as illustrated in FIGS. 3 and 4. Will be explained.
First, regarding the object A, as described above with reference to FIGS. 3 and 4, the score calculation unit 241 sets a high association score with the part X until 0 to 10 seconds later. Therefore, the matching unit 242 determines that the object A has been moved according to the part X, and combines the object A and the part X.
Further, the position coordinates of the object B from 0 to 3 seconds later do not change, and the score of the association with the region Y is set high by the score calculation unit 241 until 3 to 10 seconds later. Therefore, the matching unit 242 combines the object B and the part Y on the assumption that the object B starts to move after 3 seconds by the part Y and has moved to the position coordinates detected after 10 seconds.

Note that, in the position coordinate locus interpolation and association score calculation operation by the score calculation unit 241 described with reference to FIG. 4, the score calculation unit 241 prioritizes the proximity of the position coordinates for the object A after 3 seconds. Assume that the association score with the part Y is calculated higher than the association score with the part X. Since the interpolation position of the object A after 3 seconds based on the data of only the object A is closer to the part Y than the part X, as shown in FIG. Score calculation can occur. In this case, a high association score is set for the part Y in both the object A and the object B after 3 seconds.
Therefore, the matching unit 242 determines whether the part Y should be combined with the object A or B within a range where matching as work can be performed.
Specifically, for example, if the part Y has a shape that holds the object B after 3 seconds, the object B is likely to be operated using the part Y. The part Y is determined that the combination of the object B and the object B is more consistent as work, and the object B and the part Y are combined.
Alternatively, for example, the matching unit 242 may determine whether the combination of the object B and the parts X and Y is matched from the combination of the front and rear objects B and the parts X and Y. Here, between 0 and 2 seconds and between 8 and 10 seconds, the part X has a high association score with the object A. That is, it can be said that the object A is more consistent as a work if the work is performed as a series of operations using the part X. Therefore, the matching unit 242 combines the object B and the part Y after 3 seconds.

  Further, the objects C and D have no movement in position coordinates for 10 seconds. In addition, the parts X and Y are detected around 8 to 10 seconds later, and the parts X and Y are set to have an association score higher than that of the object C, respectively, and the work involving movement is performed. There are objects A and B that can be determined. If the objects C and D that do not move are combined with the parts X and Y after 8 to 10 seconds, a contradiction occurs as work. Therefore, the matching unit 242 determines that the work using the parts X and Y is not performed for the objects C and D, and does not perform the combination with the parts X and Y.

Note that the above-described method is merely an example, and the matching unit 242 determines a range in which the matching as the work can be obtained by an appropriate method set in advance, and the combination of the body part 121 and the object 131 of the worker 120. As long as it comes to decide.
The matching unit 242 outputs the determined combination of the body part 121 and the object 131 of the worker 120 to the position correction unit 243.

The position correction unit 243 uses the combination of the body part 121 and the object 131 of the worker 120 determined by the matching unit 242 to determine the position coordinates of the body part 121 and / or the object 131 of the worker 120 or both. to correct.
The position correction unit 243 may correct the position coordinates of the body part 121 and / or the object 131 of the worker 120 using any correction method. For example, when the position coordinates of the object 131 are interrupted, the position correction unit 243 may interpolate the position coordinates of the object 131 in association with the locus of the part 121 combined with the object 131. On the contrary, when the locus of the position coordinate of the part 121 is interrupted, the position coordinate of the part 121 may be interpolated in association with the locus of the object 131 combined with the part 121.

Here, FIG. 5 is a diagram for explaining an example of the position coordinate correction operation by the position correction unit 243 in the first embodiment.
In FIG. 5, as an example, the position correction unit 243 has the position coordinates interpolated by the score calculation unit 241 and combined with the part X by the matching unit 242 as illustrated in FIG. An operation for correcting the position coordinates until after will be described.
In FIG. 4, with respect to the trajectory of the object A in which the position coordinates are interpolated as indicated by the dotted line, the position correction unit 243 adds the position coordinates to the locus of the position coordinates of the part X combined with the object A. to correct.
As a result, as shown in FIG. 5, the position coordinate of the object A becomes a locus similar to that of the part X until 2 to 8 seconds later.

As described above, the operation of the position correction unit 243 described with reference to FIG. 5 is merely an example, and the position correction unit 243 may correct the position coordinates by other methods.
For example, when interpolating, the position correction unit 243 causes the position correction unit 243 to copy the locus as it is, and associates the locus with the start point and the end point so that the start point and the end point of the locus are not discontinuous. The position coordinates of the part 121 or the object 131 may be corrected.
Specifically, for example, the part Z is moved from the point e → the point f → the point g → the point h with the right hand, the locus of the part Z is interrupted at the point f → the point g, and the position coordinates between the interruptions are interpolated. At this time, for example, the position correction unit 243 interpolates the locus of the position coordinates of the component Z at the point f → the point g with the locus of the position coordinates of the right hand at the same time. In this case, since the position coordinate of the part Z and the position coordinate of the right hand may not coincide with each other, it is assumed that the locus of the position coordinate of the part Z becomes unnaturally discontinuous from the point f to the point g. . Therefore, the position correction unit 243 adjusts the part Z so that the part Z is natural, that is, continuous, in accordance with the positions of the start point and the end point so that the start point and end point of the path are not discontinuous. The position coordinates may be corrected.

  Thus, for example, even when the locus of the position coordinates of the object 131 is interrupted, considering the association between the body part 121 of the worker 120 and the object 131, the body part of the worker 120, or The trajectory of the object 131 can be interpolated. As a result, for example, the left hand or right hand coordinates that cannot be taken are estimated from the coordinates of the location where the coordinates have disappeared, and the correspondence between the object and the body part of the worker is not taken into consideration. Compared with the conventional technique in which the locus is interpolated by the body part or the object alone, the locus can be interpolated with higher accuracy.

  The position correcting unit 243 outputs the corrected position coordinates of the body part 121 and the object 131 of the worker 120 and information on the combination result to the recognition result analyzing unit 250 as association information. In addition, the position correction | amendment part 243 outputs the position coordinate before correction | amendment to the recognition result analysis part 250, when not correct | amending.

The recognition result analysis unit 250 recognizes the recognition result of the entire work through a series of work based on the association information output by the association unit 240. Further, the recognition result analysis unit 250 analyzes the work based on the recognition result of the whole work. Here, the recognition result of the entire work refers to association information in which the object 131 and the part 121 that performed the work using the object 131 are associated with each other, and is information indicating a series of work flows.
Then, the recognition result analysis unit 250 outputs to the output control unit 260 information regarding the recognition result of the entire work or the analysis result obtained by analyzing the recognition result.

The recognition result analysis unit 250 includes a success / failure determination unit 251, a type identification unit 252, a content comparison unit 253, and a condition determination unit 254.
The success / failure determination unit 251 determines whether or not the prescribed work has been performed using the association information between the body part 121 of the worker 120 and the object 131 determined by the association unit 240. Specifically, for example, for each object 131, a work list in which work to be performed by the worker 120 using the object 131 is registered in advance, and the stored work list and the association information are stored. Based on this, the success / failure determination unit 251 determines whether the worker 120 has performed an operation to be performed. This is merely an example, and the success / failure determination unit 251 may determine whether or not the prescribed work has been performed by another method.
The success / failure determination unit 251 outputs a determination result on whether or not the prescribed work has been performed to the output control unit 260.

The type identifying unit 252 identifies which of the prescribed work types corresponds to using the association information between the body part 121 of the worker 120 and the object 131 determined by the associating unit 240. Specifically, for example, for each work, work type information in which a procedure of which object 131 is used to perform what operation is linked and classified is stored in advance, and the stored work is stored. Based on the type information and the association information, the type identification unit 252 identifies which of the prescribed work types the work performed by the worker 120 corresponds. Note that this is merely an example, and the type identification unit 252 may identify which of the prescribed work types corresponds to other methods.
The type identifying unit 252 outputs to the output control unit 260 an identification result indicating which of the prescribed work types corresponds.

The content comparison unit 253 uses the association information determined by the association unit 240 between the body part 121 of the worker 120 and the object 131 to compare the work performed by the worker 120 with a work different from the work performed by the worker 120. The contents of work performed by the person 120 are analyzed. Specifically, for example, work by another worker who serves as a model is stored in advance as a work history, and based on the stored work history and the association information, the content comparison unit 253 The contents are compared, and, for example, it is determined whether there is a missing work, there is a useless work, the work time is slow, or the work is faster than the work of other workers. This is merely an example, and the content comparison unit 253 may compare the work content by other methods.
The content comparison unit 253 outputs the work content comparison result to the output control unit 260.

  The condition determination unit 254 determines whether or not the work content satisfies a prescribed condition, using the association information between the body part 121 of the worker 120 and the object 131 determined by the association unit 240. Specifically, based on the work type information as described above and the association information stored in advance, the condition determination unit 254 determines whether the time taken by the worker 120 for the work is too long. Determine. The conditions can be set as appropriate, and the condition determination unit 254 may determine whether or not other conditions are satisfied.

  The recognition result analysis unit 250 according to the first embodiment includes a success / failure determination unit 251, a type identification unit 252, a content comparison unit 253, and a condition determination unit 254 as shown in FIG. Not limited to this, the recognition result analysis unit 250 only needs to include at least one of the success / failure determination unit 251, the type identification unit 252, the content comparison unit 253, and the condition determination unit 254.

  The recognition result analysis unit 250 outputs the success / failure determination unit 251, the type identification unit 252, the content comparison unit 253, or the work analysis result using the overall work recognition result by the condition determination unit 254, and the association unit 240 outputs the result. The association information, that is, information on the recognition result of the entire work is also output to the output control unit 260.

The output control unit 260 outputs the recognition result of the entire work output by the recognition result analysis unit 250 to at least one of the output devices.
In addition, when receiving an analysis result output instruction, the output control unit 260 outputs a work analysis result using the recognition result of the entire work output by the recognition result analysis unit 250 to at least one of the output devices.

The user can specify an analysis result to be output by using an input device (not shown), and an instruction receiving unit (not shown) receives the designation from the user as an analysis result output instruction, and performs recognition result analysis. An output instruction is given to the unit 250.
Then, the output control unit 260 outputs the work analysis result output from the recognition result analysis unit 250 to at least one of the output devices.
Specifically, for example, when information on a determination result indicating whether or not a prescribed work has been performed is output from the success / failure determination unit 251 of the recognition result analysis unit 250, the output control unit 260 determines whether or not the determination is made on the display 151. The result is displayed with ○ and ×. Alternatively, the output control unit 260 may cause the speaker 152 to output a sound corresponding to the determination result, or may cause the storage device 153 to store the determination result. The output control unit 260 may transmit different control signals to the control device 154 according to the determination result.

  Further, for example, when information on the work type is output from the type identification unit 252 of the recognition result analysis unit 250, the output control unit 260 causes the display 151 to display a video indicating the work type. Alternatively, the output control unit 260 may cause the speaker 152 to output a sound corresponding to the work type, or may cause the storage device 153 to store the work type. The output control unit 260 may transmit different control signals to the control device 154 according to the work type.

  Further, for example, when information on the comparison result of the work is output from the content comparison unit 253 of the recognition result analysis unit 250, the output control unit 260 causes the display 151 to display a video indicating the information on the comparison result of the work. Alternatively, the output control unit 260 may output a sound or sound for work that is lacking in the speaker 152, or may store the comparison result in the storage device 153. The output control unit 260 may transmit different control signals to the control device 154 according to the comparison result.

  Further, for example, when information on a determination result indicating whether the work content satisfies a specified condition is output from the condition determination unit 254 of the recognition result analysis unit 250, the output control unit 260 delays the operation on the display 151. A message indicating that it is too much is displayed. Alternatively, the output control unit 260 may cause the speaker 152 to output a sound or sound indicating that the work is too slow, or may cause the storage device 153 to store the determination result. The output control unit 260 may transmit different control signals to the control device 154 according to the comparison result.

The body part information storage unit 270 stores body part information regarding the body part 121 of the worker 120 acquired by the body part information acquisition unit 220.
The object information storage unit 280 stores object information related to the object 131 acquired by the object information acquisition unit 230.
Here, as shown in FIG. 2, the body part information storage unit 270 and the object information storage unit 280 are provided in the work recognition device 100, but not limited thereto, the body part information storage unit 270 and The object information storage unit 280 may be provided outside the work recognition apparatus 100 in a place where the work recognition apparatus 100 can refer to it.
The body part information storage unit 270 and the object information storage unit 280 may be a single storage medium.

6A and 6B are diagrams showing an example of a hardware configuration of the work recognition apparatus 100 according to Embodiment 1 of the present invention.
In the first embodiment of the present invention, the functions of the body part information acquisition unit 220, the object information acquisition unit 230, the association unit 240, the recognition result analysis unit 250, and the output control unit 260 are realized by the processing circuit 601. Is done. That is, the work recognition apparatus 100 includes a processing circuit 601 for performing processing for recognizing work performed by the worker 120 on the acquired sensor data and performing control for outputting a recognition result or the like.
The processing circuit 601 may be dedicated hardware as shown in FIG. 6A or may be a CPU (Central Processing Unit) 606 that executes a program stored in the memory 607 as shown in FIG. 6B.

  When the processing circuit 601 is dedicated hardware, the processing circuit 601 includes, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), and an FPGA (Field-Programmable). Gate Array) or a combination thereof.

  When the processing circuit 601 is the CPU 606, each function of the body part information acquisition unit 220, the object information acquisition unit 230, the association unit 240, the recognition result analysis unit 250, and the output control unit 260 is software, firmware, or Realized by a combination of software and firmware. That is, the body part information acquisition unit 220, the object information acquisition unit 230, the association unit 240, the recognition result analysis unit 250, and the output control unit 260 are stored in an HDD (Hard Disk Drive) 602, a memory 607, and the like. The processing is realized by a processing circuit such as a CPU 606 that executes a program and a system LSI (Large-Scale Integration). The programs stored in the HDD 602, the memory 607, and the like are the procedures and methods of the body part information acquisition unit 220, the object information acquisition unit 230, the association unit 240, the recognition result analysis unit 250, and the output control unit 260. It can be said that it is what is executed by a computer. Here, the memory 607 includes, for example, RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory, etc.) Or a volatile semiconductor memory, a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versatile Disc), or the like.

Note that some of the functions of the body part information acquisition unit 220, the object information acquisition unit 230, the association unit 240, the recognition result analysis unit 250, and the output control unit 260 are realized by dedicated hardware. The unit may be realized by software or firmware. For example, the function of the body part information acquisition unit 220 is realized by a processing circuit 601 as dedicated hardware, and the object information acquisition unit 230, the association unit 240, the recognition result analysis unit 250, and the output control unit 260 are realized. The processing circuit can realize the function by reading and executing the program stored in the memory 607.
The work recognition apparatus 100 also includes a communication device 605 that communicates with external devices such as a visible light camera 111, an infrared camera 112, a depth sensor 113, an acceleration sensor 114, a gyro sensor 115, a display 151, and a speaker 152.
The sensor data acquisition unit 210 configures an input interface device 603 that acquires sensor data from the visible light camera 111, the infrared camera 112, the depth sensor 113, the acceleration sensor 114, the gyro sensor 115, and the like using the communication device 605.
In addition, the work recognition device 100 includes an output interface device 604 for outputting work recognition results and the like. The output control unit 260 communicates with the display 151, the speaker 152, and the like using the communication device 605, and outputs work recognition results and the like to the display 151, the speaker 152, and the like using the output interface device 604.
The body part information storage unit 270 and the object information storage unit 280 use, for example, the HDD 602. This is only an example, and the body part information storage unit 270 and the object information storage unit 280 may be configured by a DVD, a memory 607, and the like.
Further, the work recognition device 100 may be configured to have an auxiliary storage device (not shown) for storing work recognition results and the like.

FIG. 7 is a diagram illustrating an example of a hardware configuration of the control device 154 according to the first embodiment.
The control device 154 includes a communication device 703 that communicates with the work recognition device 100, and receives the work recognition result and the like transmitted by the work recognition device 100 via the communication device 605 using the communication device 703. .
The control device 154 includes a CPU 701 that generates a command signal for operating a driving device (not shown) such as a servo.
In addition, the control device 154 includes a memory 702 that temporarily stores a work recognition result received using the communication device 703 and a command signal for operating a driving device such as a servo.
The communication device 703 receives a work recognition result from the work recognition device 100 and transmits a command signal for operating a drive device such as a servo.

The operation of the work recognition apparatus 100 according to the first embodiment will be described.
FIG. 8 is a flowchart for explaining the operation of the work recognition apparatus 100 according to Embodiment 1 of the present invention.
Sensor data acquisition section 210 acquires sensor data from at least one of visible light camera 111, infrared camera 112, depth sensor 113, acceleration sensor 114, and gyro sensor 115 (step ST801).
The sensor data acquisition unit 210 outputs the acquired sensor data to the body part information acquisition unit 220 and the object information acquisition unit 230.
Body part information acquisition section 220 acquires body part information related to body part 121 of worker 120 based on the sensor data output from sensor data acquisition section 210 (step ST802).

FIG. 9 is a flowchart illustrating details of the operation of body part information acquisition section 220 in step ST802 of FIG.
The body part detection unit 221 of the body part information acquisition unit 220 acquires the already acquired body part information from the body part information storage unit 270 (step ST901).

  The body part detection unit 221 is based on the body part information acquired from the body part information storage unit 270, in the optical image, infrared image, or depth image before frame included in the sensor data output from the sensor data acquisition unit 210. It is determined whether each part 121 of the worker's 120 body has been detected (step ST902). The body part detection unit 221 may determine the previous frame from the position information acquisition date / time information included in the body part information and the current date / time information.

In step ST902, when the part 121 has not been detected in the previous frame (in the case of “NO” in step ST902), the body part detection unit 221 uses the sensor data output by the sensor data acquisition unit 210 to perform the operator 120. When the body part 121 of the worker 120 is detected ("YES" in step ST904), the position coordinates of the part 121 are acquired (step ST907). .
Then, the body part detection unit 221 stores the acquired position coordinate information of the body part 121 of the worker 120 in the body part information storage unit 270 in association with the acquisition date information.
When the body part 121 of the worker 120 cannot be detected (in the case of “NO” in step ST904), the process proceeds to step ST803 in FIG.

In step ST902, when the part 121 has been detected in the previous frame (in the case of “YES” in step ST902), the body part tracking unit 222 of the body part information acquisition unit 220 indicates that the body part detection unit 221 is in the previous frame. The body part 121 of the worker 120 that has acquired the position coordinates is tracked (step ST905), and when the body part 121 of the worker 120 can be tracked (in the case of “YES” in step ST906), The position coordinates after the movement are acquired (step ST907).
Then, the body part tracking unit 222 stores the acquired position coordinate information in the body part information storage unit 270 in association with the acquisition date information.
If the body part 121 of the worker 120 cannot be tracked (in the case of “NO” in step ST906), the process proceeds to step ST803 in FIG.

  The body part recognition unit 223 recognizes the type, shape, or state of the body part 121 of the worker 120, which the body part detection unit 221 acquires the position coordinates and stores in the body part information storage unit 270 ( Step ST908).

  As described above, the series of processing from step ST901 to step ST908 shown in FIG. 9 is performed by the body part information acquisition unit 220, such as an optical image, an infrared image, or an infrared image included in the sensor data output from the sensor data acquisition unit 210. This is a process of acquiring body part information for one frame of a depth image.

Returning to the flowchart of FIG.
Object information acquisition section 230 acquires object information related to object 131 used by operator 120 for work based on the sensor data output from sensor data acquisition section 210 (step ST803).

FIG. 10 is a flowchart illustrating details of the operation of the object information acquisition unit 230 in Step ST803 of FIG.
The object detection unit 231 of the object information acquisition unit 230 acquires already acquired object information from the object information storage unit 280 (step ST1001).

  Based on the object information acquired from the object information storage unit 280, the object detection unit 231 includes, for each object 131, an optical image, an infrared image, or a depth image of the sensor data output by the sensor data acquisition unit 210. It is determined whether or not it has been detected (step ST1002). The object detection unit 231 may determine the previous frame from the position information acquisition date / time information included in the object information and the current date / time information.

In step ST1002, when the object 131 is not detected in the previous frame (in the case of “NO” in step ST1002), the object detection unit 231 detects the object 131 based on the sensor data output from the sensor data acquisition unit 210. If the object 131 can be detected (in the case of “YES” in step ST1004), the position coordinates of the object 131 are acquired (step ST1007).
Then, the object detection unit 231 stores the acquired position coordinate information of the object 131 in the object information storage unit 280 in association with the acquisition date information.
When the object 131 cannot be detected (in the case of “NO” in step ST1004), the process proceeds to step ST804 in FIG.

In step ST1002, when the object 131 has been detected in the previous frame (in the case of “YES” in step ST1002), the object tracking unit 232 of the object information acquisition unit 230 causes the object detection unit 231 to detect the position coordinates in the previous frame. The acquired object 131 is tracked (step ST1005), and when the object 131 can be tracked (in the case of “YES” in step ST1006), the position coordinates after the movement of the object 131 are acquired (step ST1007).
The object tracking unit 232 stores the acquired position coordinate information in the object information storage unit 280 in association with the acquisition date information.
If the object 131 can not be tracked (if "NO" in step ST 1006), the process proceeds to step ST804 in FIG.

The object recognition unit 233 recognizes the type, shape, or state of the object 131 that the object detection unit 231 acquires the position coordinates and stores in the object information storage unit 280 (step ST1008).
As described above, a series of processing from step ST1001 to step ST1008 shown in FIG. 10 is performed by the object information acquisition unit 230, such as an optical image, an infrared image, or a depth included in the sensor data output from the sensor data acquisition unit 210. This is object information acquisition processing for one frame of video.

Returning to the flowchart of FIG.
The control part (illustration omitted) of the work recognition apparatus 100 determines whether the work by the worker 120 is finished (step ST804). For example, the control unit determines whether or not the work has been completed based on the fact that the worker 120 cannot be detected from the video acquired from the sensor, and determines that the work has been completed if the worker 120 cannot be detected. Also good. Alternatively, the control unit may determine that the work is finished when the worker 120 receives a work end notification input from an input device (not shown). This is merely an example, and it is sufficient that the control unit can determine whether or not the work of the worker 120 has been completed by any means.

  If it is determined in step ST804 that the work has not ended (in the case of “NO” in step ST804), the process returns to step ST801, and the processes in steps ST801 to ST803 are repeated.

If it is determined in step ST804 that the work has been completed (in the case of “YES” in step ST804), the associating unit 240 performs the optical image, infrared image, or depth of the sensor data acquired in step ST801 until the work is completed. For all the frames of the video, the object 131 is associated with the body part 121 of the worker 120 who performed the work using the object 131 (step ST805). Specifically, the association unit 240, the body part information acquisition unit 220 in step ST 802 is to be stored in the body part information storing unit 270, and the body parts information on parts 121 of the body of the operator 120, in step ST 803 Based on the object information related to the object 131 stored in the object information storage unit 280 by the object information acquisition unit 230, the object information acquisition unit 230 associates with the body part 121 of the worker 120.
The associating unit 240 outputs the result of associating the object 131 with the body part 121 of the worker 120 who performed the work using the object 131 to the recognition result analyzing unit 250.

FIG. 11 is a flowchart illustrating details of the operation of the associating unit 240 in step ST805 of FIG.
The score calculation unit 241 of the association unit 240 calculates an association score indicating which body part 121 is likely to be related to the movement of the object 131 with respect to the combination of the body part 121 and the object 131 of the worker 120. (Step ST1101). Specifically, the score calculation unit 241 includes the body part information related to the body part 121 of the worker 120 and the object information acquisition unit 230 that are stored in the body part information storage unit 270 by the body part information acquisition unit 220. The association score is calculated using the object information regarding the object 131 stored in the information storage unit 280. That is, the score calculation unit 241 calculates an association score representing the degree of association between the body part 121 of the worker 120 and the object 131 with respect to the combination of the body part 121 and the object 131 of the worker 120.
The score calculation unit 241 outputs the calculated association score information to the matching unit 242 of the association unit 240.

The matching unit 242 uses the association score calculated by the score calculation unit 241 to combine the body part 121 of the worker 120 and the object 131 so as to maximize the association score within a range where the matching as the work can be achieved. Is determined (step ST1102).
The matching unit 242 outputs the determined combination of the body part 121 of the worker 120 and the object to the position correction unit 243 of the association unit 240.
The position correction unit 243 uses the combination of the body part 121 and the object 131 of the worker 120 determined by the matching unit 242 to determine the position coordinates of the body part 121 and / or the object 131 of the worker 120 or both. Correction is performed (step ST1103).

Returning to the flowchart of FIG.
The recognition result analysis unit 250 analyzes the recognition result of the entire work based on the association result output by the association unit 240 (step ST806).
The recognition result analysis unit 250 outputs to the output control unit 260 information regarding the recognition result of the entire work, which is the association information output from the association unit 240, or the analysis result obtained by analyzing the recognition result.

The output control unit 260 outputs the recognition result of the entire work output by the recognition result analysis unit 250 to at least one of the output devices.
When the output control unit 260 receives an analysis result output instruction, the output control unit 260 outputs the work analysis result to at least one of the output devices using the recognition result of the entire work output from the recognition result analysis unit 250 (step) ST807).

  12-14 is a figure explaining the example of an output of the recognition result of the whole operation | work by the output control part 260, or a work analysis result in this Embodiment 1. FIG. 12 to 14 illustrate an example in which the output control unit 260 causes the display 151 to output the recognition result of the entire work or the work analysis result as a video.

FIG. 12 is a diagram illustrating the position coordinates of the object 131 and the left and right hands based on the result of the recognition result analysis unit 250 recognizing the assembly work by moving the object 131 with the left and right hands in the first embodiment. , And an output example in which the timing at which the object is moved is displayed on the display 151.
On the left side of the drawing of FIG. 12, as a work recognition result, the three-dimensional position coordinates of each object 131 in a certain frame are converted into two-dimensional position coordinates viewed from the vertical direction, and an icon of each object 131 is displayed. ing. A certain frame is an arbitrary frame, and FIG. 12 shows, as an example, a frame at the end of the work in which the object 131 is moved with the left hand.

On the right side of the drawing of FIG. 12, the timing at which the object 131 is associated with the left and right hands, that is, the timing at which the object 131 is moved is displayed.
In FIG. 12, as an example, the time axis is taken in the vertical direction of the drawing, with 0 as the work start time. As described above, the output control unit 260, for example, regarding the work performed by the worker 120, the recognition result at a certain time (see the left side of the drawing in FIG. 12) and the recognition result of the flow of the entire work in accordance with the passage of time. (See the right side of the drawing in FIG. 12) can be displayed.
The output control unit 260 receives information related to the recognition result of the work of the worker 120 from the recognition result analysis unit 250, and displays the time interval in which the object 131 is moved together with a color rectangle indicating the object 131. In FIG. 12, the object 131 is represented not by color coding but by the difference in dot density.

The left side of the drawing of FIG. 12 displays the work recognition result in a certain frame, but the output control unit 260 displays the display of the work recognition result after the time displayed on the right side of the drawing of FIG. It can be displayed in synchronization with the display of the flow of the combined work.
Specifically, for example, when the user designates a certain time in the time axis direction of the work analysis result displayed on the right side of the drawing of FIG. 12, the output control unit 260 performs the work recognition result in the frame of the designated time. The display on the left side of the drawing of FIG. 12 can be updated so as to be displayed. Thereby, the work content can be visualized with higher accuracy. In FIG. 12, it is assumed that the operation recognition result about 12 seconds after the start of the operation is designated to be displayed.
At this time, the output control unit 260 displays the corresponding icon of the object 131 displayed on the left side of the drawing of FIG. 12 and the rectangle indicating the object 131 displayed on the right side of the drawing of FIG. The relationship may be shown by connecting with a line.

The work recognition result shown in FIG. 12 will be described in detail.
FIG. 12 shows two objects m (m ₁ and m _{2 in} FIG. 12), _two objects n (n ₁ and n _{2 in} FIG. 12), one object o, and one object p on the desk. This shows a situation where there are six objects 131 in total.
The trajectory x indicates that the operator 120 is holding the object m ₁ with the left hand (L in FIG. 12) and moving it to the right of the screen. The object o in the middle of the movement of the object m ₁ is in a situation where it can be accidentally placed on the trajectory x of the object m ₁ regardless of the work of the worker 120. The reason why the object o is not shown in the display displayed on the right side of the drawing is that the object o is not related to the work of the worker 120, and the worker 120 holds the object o in the right hand (see FIG. 12). This is because neither the left hand nor the left hand is moved.
12 on the locus y is the operator 120 slightly before the time indicated in the drawing on the left is a locus obtained by moving the object m _2. Looking at the right side of the drawing, it can be seen that the object m ₂ is moved with the right hand by 10 seconds after the start of the work.
In FIG. 12, it is assumed that the trajectories x and y are displayed for several seconds even after the movement of the object m is completed and disappear after a few seconds. Note that the time for which the trajectories x and y remain displayed can be set as appropriate.

As described above, when the recognition result analysis unit 250 recognizes that a certain object 131 is associated with the left hand in the frame, that is, when it is recognized that the certain object 131 is moved with the left hand, the output control unit 260 In response to the recognition result, the left hand icon is displayed over the object icon. This is only an example, and the same applies when an object 131 is associated with the right hand. The same applies when using body parts other than the left and right hands.
When the object 131 moves, the output control unit 260 displays the locus of the position coordinates of the object 131 with a line. The output control unit 260 may continue to display the trace line to be displayed, or may delete the line after a certain period of time, that is, not display it.
Further, the output control unit 260 may display the object 131 and the three-dimensional position coordinates of the body part 121 of the worker 120 as they are three-dimensionally.

  In FIG. 12, at which timing the object 131 is associated with the left and right hands (left side of the drawing in FIG. 12) and at what timing the object 131 is associated with the left and right hands (FIG. 12). 12, the right side of the drawing) is displayed together. However, the present invention is not limited to this, and the timing at which the object 131 is associated with the left and right hands (left side of the drawing in FIG. 12) Only one of the timings when the object 131 is associated with the hand (right side in the drawing of FIG. 12) may be displayed.

FIG. 13 shows an analysis of work contents based on the recognition result analysis unit 250 recognizes the work of moving and assembling the object 131 with the left and right hands in the first embodiment, and the output control unit 260 analyzes the analysis result. It is an output example displayed on the display 151.
In FIG. 13, in addition to the display of the object movement timing as shown on the right side of the drawing of FIG. 12, the output control unit 260 is the analysis result that the recognition result analysis unit 250 performs using the information of the work recognition result. The information about is displayed. Instead of displaying the object movement timing as shown on the right side of the drawing in FIG. 12, information regarding the analysis result as shown in FIG. 13 may be displayed.

In FIG. 13, as an example, the worker 120 places two objects n ₃ and n ₄ on the base using the right hand (R in FIG. 13) or the left hand (L in FIG. 13), respectively, to n _3, on _{n 4,} respectively, with the right hand or left hand, after placing the object _m 3, _{m 4,} with the right hand, an object q to bridge the the object _{m 3} and _{m 4,} It is assumed that the work of placing on the objects m ₃ and m ₄ has been performed.
13, the recognition result analysis unit 250 calculates the total time during which the objects n ₃ , n ₄ , m ₃ , m ₄ , and q are moved by the left and right hands in the above-described operation. It is calculated as 0 seconds, the time from the start to the end of the work is calculated as 17.3 seconds, and the output control unit 260 displays the calculation result.
The total time for moving the object 131 with the left and right hands and the calculation of the time from the start to the end of the work are calculated by the success / failure determination unit 251, the type identification unit 252, and the content comparison unit 253 of the recognition result analysis unit 250. Any one of the condition determination units 254 may perform the recognition result analysis unit 250. The recognition result analysis unit 250 determines the total time from the time associated with the video frame, the total time when the object 131 is moved by the left and right hands, and The time from the start to the end of the work may be calculated.

In FIG. 13, as an example, the recognition result analysis unit 250 calculates the sum of the distances of moving the objects n ₃ , n ₄ , m ₃ , m ₄ , and q with the left and right hands to calculate 254 cm, In response to the calculation result, the output control unit 260 displays the total distance calculated by the recognition result analysis unit 250.
The total distance may be determined by any of the success / failure determination unit 251, the type identification unit 252, the content comparison unit 253, and the condition determination unit 254 of the recognition result analysis unit 250. The total distance may be calculated from the position coordinates.

In FIG. 13, as an example, the success / failure determination unit 251 determines that the prescribed work has been performed in the prescribed order, and the content comparison unit 253 performs unnecessary work without using the wrong object 131. It is determined that it is not included, and the condition determination unit 254 receives the result that the work is determined not to be slower than the specified time, and causes the output control unit 260 to display “◎ Good!” ing.
This is only an example, and depending on the determination results of the success / failure determination unit 251, the type identification unit 252, the content comparison unit 253, and the condition determination unit 254 included in the recognition result analysis unit 250, what kind of display is appropriately displayed The output control unit 260 may display the analysis result in accordance with the appropriately set conditions.

In addition, the determination results of the success / failure determination unit 251, the type identification unit 252, the content comparison unit 253, and the condition determination unit 254 may be individually displayed.
For example, when the type identification unit 252 determines from the work recognition result that the work corresponds to “assembly work W1” of the prescribed work types, the output control unit 260 receives the analysis result, The heading “Assembly work W1” may be displayed.
Further, for example, when the success / failure determination unit 251 determines that the prescribed work has been performed, the output control unit 260 may display “◯ (success)” in response to the analysis result.
Further, for example, when the content comparison unit 253 compares with other work content and determines that the assembly of a certain part is not performed, the output control unit 260 receives “the component Z assembly not performed” in response to the analysis result. May be displayed.

  In this way, by displaying the work analysis result, the worker 120 or the like cannot make any point regarding the work contents if the work is performed correctly or the work is not performed accurately. By confirming the display, it can be easily grasped.

14, as in FIG. 13, in the first embodiment, the recognition result analysis unit 250 analyzes the work content based on the result of recognizing the work of moving and assembling the object 131 with the left and right hands, and the output control unit 260 It is an output example in which the analysis result is displayed on the display 151.
Although FIG. 13 shows an example in which the recognition result analysis unit 250 determines that the work has been performed as specified, FIG. 14 illustrates the result of the analysis performed by the recognition result analysis unit 250 as a result of the work being performed as specified. An example in the case where it is determined that it has not been performed is shown. Specifically, an example is shown in which the content comparison unit 253 of the recognition result analysis unit 250 compares with other reference work and determines that the work is performed using the wrong object 131 having a different color. Yes.
Further, an example is shown in which the condition determination unit 254 of the recognition result analysis unit 250 determines that the work is too late for the work using a certain object 131.

Specifically, in FIG. 14, as an example, the operator 120 moves the object n ₅ with the right hand (R in FIG. 14) → moves the object r with the left hand (L in FIG. 14) → moves the object m ₅ with the right hand. moving → the object m ₆ in the order of movement in the left hand, it is assumed that the work was done. In this work, the work for moving the object r with the left hand should have moved the object n ₆ with the left hand, but the operator 120 mistakenly moved the object r of a different color. Shall be.
Then, the content comparison unit 253 of the recognition result analysis unit 250 determines that the operation is performed using the wrong object r having a different color as compared with other reference operations.
The condition determining unit 254 of the recognition result analysis unit 250, the work using the object m _5, it is assumed that it is determined that the operation is too slow.

The output control unit 260 receives the analysis result that the content comparison unit 253 has performed the work using the object r having a different color, and “x” so that the wrong work can be recognized at the position indicating the work. "The color is different".
In addition, the output control unit 260 receives the analysis result by the condition determination unit 254 that the work using the object m5 is too late, so that the work that is determined to be too late can be found at a location indicating the corresponding work. In addition, “△ work is slow” is displayed.
Note that the display example as shown in FIG. 14 is merely an example, and the output control unit 260 is based on a preset method, and the wrong work and the content of the mistake, or the work time is longer than the reference time. It is only necessary to display the image on the display 151 so that it can be seen.

As described above, the output control unit 260 displays the work recognition result or the analysis result on the display 151 based on the work recognition result output from the recognition result analysis unit 250 or the analysis result information obtained by analyzing the work recognition result. Is displayed as appropriate.
Thereby, the operator 120 etc. can grasp | ascertain in detail about the work content by confirming the display content of the display 151. FIG.

In the above description, an example in which the output control unit 260 displays the work recognition result or the analysis result information output from the recognition result analysis unit 250 on the display 151 has been described. The output of the result or the information on the analysis result is not limited to this.
For example, when the success / failure determination unit 251 of the recognition result analysis unit 250 determines that the specified work is not performed, the output control unit 260 is set in advance to indicate that the specified work is not performed. The work recognition result or the analysis result information output from the recognition result analysis unit 250, such as outputting sound from the speaker 152, can be output from the speaker 152 by sound or voice.

In addition, the output control unit 260 can store, in the storage device 153, information related to the work that is determined to have not been performed as a prescribed work, along with the determination result.
By storing work result information in the storage device 153, the worker 120 or the like can analyze the cause determined that the prescribed work is not performed from the stored data.
Further, the output control unit 260 may transmit a control signal to the control device 154 to the effect that it is determined that the prescribed work has not been performed.
The control device 154 receives the control signal, and performs control that helps to perform the prescribed work, for example, downloads a manual and provides it to the work recognition device 100.
As described above, the output control unit 260 can transmit a control signal corresponding to the analysis result of the recognition result analysis unit 250 to the control device 154 so that the control device 154 performs control according to the analysis result. .

  As described above, according to the first embodiment, the sensor data acquisition unit 210 that acquires sensor data and the body part 121 of the worker 120 are detected based on the sensor data acquired by the sensor data acquisition unit 210. Based on the sensor data acquired by the body part information acquisition unit 220 that acquires the body part information related to the body part 121 of the worker 120 and the sensor data acquisition unit 210, the object 131 is detected. Based on the body part information acquired by the object information acquisition unit 230, the body part information acquisition unit 220, and the object information acquired by the object information acquisition unit 230, the object 131 and the object 131 are used. The association unit 240 that performs association with the body part 121 of the worker 120 that performed the work, and the association that is associated with the association unit 240 Since the recognition result analysis unit 250 for recognizing the work performed by the worker 120 based on the association information regarding the result is provided, the correspondence between the object 131 and the body part 121 of the worker 120 should be considered. The position of the body part 121 or the object 131 of the worker 120 cannot be estimated in part or compared with the conventional technique for obtaining the movement locus of the body part 121 or the object 131 alone. Even when the estimation error between the movement locus of the object 121 or the object 131 and the movement locus of the actual part 121 or the object 131 is large, the work of the operator 120 can be recognized with high accuracy.

  In the present invention, any constituent element of the embodiment can be modified or any constituent element of the embodiment can be omitted within the scope of the invention.

  The work recognition apparatus according to the present invention is configured to recognize the work content by the worker in a series of flows related to the body part and the object of the worker with high accuracy. The present invention can be applied to a work recognition device that recognizes contents.

  100 Work Recognition Device, 111 Visible Light Camera, 112 Infrared Camera, 113 Depth Sensor, 114 Acceleration Sensor, 115 Gyro Sensor, 120 Worker, 121 Site, 131 Object, 151 Display, 152 Speaker, 153 Storage Device, 154 Control Device, 210 sensor data acquisition unit, 220 body part information acquisition unit, 221 body part detection unit, 222 body part tracking unit, 223 body part recognition unit, 230 object information acquisition unit, 231 object detection unit, 232 object tracking unit, 233 object recognition Unit, 240 association unit, 241 score calculation unit, 242 matching unit, 243 position correction unit, 250 recognition result analysis unit, 251 success / failure determination unit, 252 type identification unit, 253 content comparison unit, 254 condition determination unit, 260 output control unit 270 body part information storage unit, 28 Object information storage unit, 601 processing circuits, 602 HDD, 603 input interface device 604 output interface device, 605,703 communication device, 606,701 CPU, 607,702 memory.

Claims (17)

A sensor data acquisition unit for acquiring sensor data;
Based on the sensor data acquired by the sensor data acquisition unit, a body part information acquisition unit that detects a part of the worker's body and acquires body part information regarding the part of the worker's body;
An object information acquisition unit that detects an object based on the sensor data acquired by the sensor data acquisition unit and acquires object information related to the object;
Based on the body part information acquired by the body part information acquisition unit and the object information acquired by the object information acquisition unit, the object, and a part of the body of the worker who performed the work using the object An association unit for associating
A work recognition apparatus comprising: a recognition result analysis unit for recognizing work performed by the worker based on association information related to an association result associated by the association unit. The work recognition apparatus according to claim 1, further comprising an output control unit that outputs information based on the association information recognized by the recognition result analysis unit. The body part information acquisition unit
The work recognition apparatus according to claim 1, further comprising a body part detection unit that detects a body part of the worker and acquires position coordinates of the worker's body part. The body part information acquisition unit
The body part tracking part which tracks the part of the worker's body detected by the body part detection part and acquires the position coordinates of the part of the worker's body after movement is provided. Item 4. The work recognition apparatus according to Item 3. The body part information acquisition unit
The work recognition apparatus according to claim 3, further comprising a body part recognition unit that recognizes the type, shape, or state of the body part of the worker detected by the body part detection unit. The object information acquisition unit
The work recognition apparatus according to claim 1, further comprising an object detection unit that detects the object and obtains position coordinates of the object. The object information acquisition unit
The work recognition apparatus according to claim 6, further comprising an object tracking unit that tracks the object detected by the object detection unit and acquires the position coordinates of the object after movement. The object information acquisition unit
The work recognition apparatus according to claim 6, further comprising: an object recognition unit that recognizes a type, shape, or state of the object detected by the object detection unit. The association unit includes
Using the body part information acquired by the body part information acquisition unit and the object information acquired by the object information acquisition unit, an association score representing a degree of association between the body part of the worker and the object is calculated. The work recognition apparatus according to claim 1, further comprising a score calculation unit. The association unit includes
The matching unit that determines a combination of the body part of the worker and the object according to the matching of work using the association score calculated by the score calculation unit. Work recognition device. The association unit includes
A position correction unit that corrects the position of the body part of the worker or the position of the object based on a combination of the body part of the worker and the object determined by the matching unit is provided. The work recognition apparatus according to claim 10. The recognition result analysis unit
The success / failure determination unit that determines whether or not a prescribed work has been performed based on association information between the body part of the worker and the object associated by the association unit. Work recognition device. The recognition result analysis unit
2. A type identification unit for identifying which of the prescribed work types corresponds based on association information between the body part of the worker and the object associated by the association unit. The work recognition apparatus described. The recognition result analysis unit
The work performed by the worker by comparing the work performed by the worker with a work different from the work performed by the worker based on the association information between the body part of the worker and the object associated by the associating unit. The work recognition apparatus according to claim 1, further comprising a content comparison unit that analyzes the content. The recognition result analysis unit
A condition determining unit that determines whether or not the work content performed by the worker satisfies a predetermined condition based on association information between the body part of the worker and the object associated by the associating unit; The work recognition apparatus according to claim 1. The output control unit
The work recognition apparatus according to claim 2, wherein information based on the recognition result recognized by the recognition result analysis unit is displayed as an image. A step in which a sensor data acquisition unit acquires sensor data;
A body part information acquiring unit detecting a body part of the worker based on the sensor data acquired by the sensor data acquiring unit, and acquiring body part information relating to the body part of the worker;
An object information acquisition unit detecting an object based on the sensor data acquired by the sensor data acquisition unit and acquiring object information related to the object;
Based on the body part information acquired by the body part information acquisition unit and the object information acquired by the object information acquisition unit, the associating unit performs the work using the object and the object. Associating with a body part;
And a step of recognizing the work performed by the worker based on association information related to the association result associated by the association unit.

Images