JP2016149678A - Camera calibration unit, camera calibration method and camera calibration program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for saving the trouble and time for camera calibration.SOLUTION: A first feature point extracting function part 24 extracts a plurality of feature points of an object person imaged in a first frame image that a first camera 10 captures. A second feature point extracting function part 25 extracts a plurality of feature points of an object person imaged in a second frame image that a second camera 11 captures at a different angle from the first camera 10. A feature point associating function part 26 associates the feature points of the object person that the first feature point extracting function part 24 and the feature points of the object person that the second feature point extracting function part 25 with each other. An external parameter calculating function part 27 calculates external parameters of the first and second cameras associated with relative position relation between the cameras using positions of the feature points of the object person in the first frame image and positions of the feature points of the object person in the second frame image.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a technique for calibrating external parameters of each camera in a multi-camera system.

  2. Description of the Related Art Conventionally, as a multi-camera system using a plurality of cameras with different angles, one that generates a three-dimensional image (three-dimensional image) of a target area captured by a plurality of cameras (for example, see Patent Document 1) There is one that generates an all-round image (see, for example, Patent Document 2). In these techniques, a frame image captured by each camera is projected onto a common coordinate system to generate a three-dimensional image or an all-round image.

  In generating a stereoscopic image and an all-round image, a positional relationship between frame images captured by each camera is necessary. For this reason, as described in Patent Documents 1 and 2, etc., camera calibration (camera calibration) is performed, and external parameters of each camera are acquired in advance. The external parameters of the camera are parameters related to the position and orientation of the camera.

JP2013-93787A JP2011-86111A

  However, camera calibration has been performed by a method in which an object having a feature point is set in a target area, and a feature point of the object is extracted from a frame image obtained by capturing the target area. For example, when a chess board is used as an object having a feature point, a method in which a point where two adjacent black squares contact each other is extracted as a feature point.

  As described above, the conventional method for camera calibration has a problem that it is necessary to set an object having a feature point in a target region, which takes time and effort.

  In addition, the mounting state (position and posture) of the camera body changes depending on external factors such as external force. That is, the external parameters of the camera change depending on external factors. For this reason, the external parameters of the camera, unlike the internal parameters of the camera (parameters related to focal length, lens distortion, etc.), need to be updated regularly, and it is desired to reduce the time and labor required for camera calibration. ing.

  An object of the present invention is to provide a technique capable of reducing labor and time for camera calibration.

  In order to achieve the above object, the camera calibration unit of the present invention is configured as follows.

  The imaging area of the first camera and the imaging area of the second camera partially overlap each other. The first feature point extraction unit extracts a plurality of feature points for the subject imaged in the first frame image captured by the first camera. In addition, the second feature point extraction unit extracts a plurality of feature points for the subject imaged in the second frame image captured by the second camera at an angle different from that of the first camera. For example, the first feature point extraction unit and the second feature point extraction unit extract a plurality of feature points of the face of the subject captured in the frame image.

  The feature point association unit associates the feature points of the subject extracted by the first feature point extraction unit with the feature points of the subject extracted by the second feature point extraction unit. The external parameter calculation unit is configured to associate the feature points of the target person with the feature point association unit, the position of the feature point of the target person on the first frame image, and the feature of the target person on the second frame image. Using the position of the point, an external parameter of the camera related to the relative positional relationship between the first camera and the second camera is calculated. The external parameter calculation unit calculates an external parameter related to the relative positional relationship of the second camera with respect to the first camera, for example, based on the first camera.

  In this configuration, the first camera and the second camera are utilized by using the characteristics of the subject positioned in the imaging region (hereinafter referred to as the target region) that overlaps between the first camera and the second camera. The external parameters of the camera related to the relative positional relationship between are calculated. That is, camera calibration for calculating the external parameters of the camera related to the relative positional relationship between the first camera and the second camera can be performed without setting an object having a feature point in the target area. Therefore, it is possible to reduce the labor and time required for camera calibration.

  Further, the second feature point extraction unit selects the subject imaged in the second frame image by face authentication using the face image of the subject imaged in the first frame image. Also good.

  If comprised in this way, it can prevent that a 1st feature point extraction part and a 2nd feature point extraction part extract a feature point from a different subject.

  Further, when the first camera and the second camera are video cameras that capture moving images, the following configuration may be used.

  The first moving image input unit receives the first moving image captured by the first camera. The second moving image input unit receives the second moving image captured by the second camera. Then, the first feature point extraction unit applies a first frame image for detecting a plurality of feature points of the subject being imaged to the first moving image input to the first moving image input unit. Select from frame images. For example, the first feature point extraction unit selects, as the first frame image, a frame image in which a person's face in the target region is captured from substantially the front. In addition, the second feature point extraction unit uses the second frame image for detecting a plurality of feature points of the subject being imaged as the imaging timing of the first frame image selected by the first feature point extraction unit. In response, the frame image corresponding to the second moving image input to the second moving image input unit is selected.

  Thereby, since the camera calibration can be performed using the first frame image and the second frame image captured at substantially the same timing, a decrease in accuracy of the calculated external parameter of the camera can be suppressed.

  According to the present invention, it is possible to reduce the labor and time required for camera calibration.

It is a figure which shows the structure of the principal part of the camera calibration unit concerning this example. It is a block diagram which shows the function structure of the image processor concerning a camera calibration process. It is a flowchart which shows a camera calibration process. It is a figure which shows a 1st process target frame image and a 2nd process target frame image. It is a figure which shows the matching result of the feature point of the extracted subject's face, and the feature point of the subject's face.

  Hereinafter, a camera configuration unit according to an embodiment of the present invention will be described.

  FIG. 1 is a diagram showing the configuration of the main part of the camera calibration unit according to this example. The camera calibration unit 1 includes an image processor 2, a first moving image input unit 3, a second moving image input unit 4, a storage unit 5, an output unit 6, and an input unit 7. Yes.

  The camera calibration unit 1 according to this example performs camera calibration (camera calibration) for calculating external parameters of the camera related to the relative positional relationship between the first camera 10 and the second camera 11, and A stereoscopic image (three-dimensional image) is generated using the external parameters of the camera calculated by calibration. The external parameters of the camera are the camera rotation R and translation T as is well known. In this example, camera calibration is performed to calculate the rotation R and translation T of the second camera 11 with the first camera 10 as a reference as external parameters of the camera.

  The image processor 2 includes a computer that executes the camera calibration method according to the present invention. The camera calibration program according to the present invention is installed in a computer provided in the image processor 2. The functional configuration of the image processor 2 will be described later. The computer included in the image processor generates a stereoscopic image using the frame image captured by the first camera 10 and the frame image captured by the second camera 11.

  A first camera 10 is connected to the first moving image input unit 3. A second camera 11 is connected to the second moving image input unit 4. The first camera 10 and the second camera 11 are video cameras that capture moving images. The frame rate of the first camera 10 and the second camera 11 is several tens of frames / sec (for example, 30 frames / sec). The first camera 10 and the second camera 11 are at different angles. The imaging area of the first camera 10 and the imaging area of the second camera 11 partially overlap. This overlapping imaging region is referred to as a target region here.

  A frame image related to a moving image captured by the first camera 10 is input to the first moving image input unit 3. A frame image related to a moving image captured by the second camera 11 is input to the second moving image input unit 4. The first moving image input unit 3 and the second moving image input unit 4 have a storage medium (not shown) such as an image memory or a hard disk for storing a frame image related to the input moving image.

  The storage unit 5 stores external parameters of the camera calculated by camera calibration processing described later.

  The output unit 6 outputs a three-dimensional image of the target area captured by the first camera 10 and the second camera 11. This stereoscopic image is generated by the image processor 2. The output unit 6 is connected to a stereoscopic image processing apparatus that processes a stereoscopic image. The stereoscopic image processing device is, for example, a display device that displays a stereoscopic image or an analysis device that analyzes the stereoscopic shape of an object located in a target region.

  An input device for inputting a command to the camera calibration unit 1 main body is connected to the input unit 7. The input device inputs a command related to an execution instruction of a camera calibration process, which will be described later, to the input unit 7.

  FIG. 2 is a block diagram showing a functional configuration of an image processor related to execution of camera calibration processing to be described later. The image processor 2 also has a function of generating a stereoscopic image as described above, but the functional configuration relating to the processing of generating the stereoscopic image is not shown. The image processing processor 2 includes a first processing target frame image selection function unit 21, a second processing target frame image selection function unit 22, a face authentication function unit 23, a first feature point extraction function unit 24, A second feature point extraction function unit 25, a feature point association function unit 26, and an external parameter calculation function unit 27 are provided.

  The first processing target frame image selection function unit 21 selects the first image from the frame images (the frame image input to the first moving image input unit 3) related to the moving image captured by the first camera 10. Select a frame image to be processed. The second processing target frame image selection function unit 22 selects the second image from the frame images (the frame images input to the second moving image input unit 4) related to the moving images captured by the second camera 11. Select a frame image to be processed. In particular, the second processing target frame image selection function unit 22 selects the first processing target frame image selection function unit 21 selected from the frame images related to the moving image captured by the second camera 11. A frame image having an imaging timing closer in time to the imaging timing of the processing target frame image is selected as the second processing target frame image. That is, the imaging timing of the first processing target frame image selected by the first processing target frame image selection function unit 21 and the second processing target frame image selected by the second processing target frame image selection function unit 22. The imaging timing is substantially the same.

  The first feature point extraction function unit 24 extracts a plurality of facial feature points for the subject imaged in the target region on the first processing target frame image. Further, a position on the first processing target frame image is detected for each extracted facial feature point.

  The face authentication function unit 23 performs face authentication on the target person from which the first feature point extraction function unit 24 has extracted the facial feature quantity from among the persons imaged in the target region on the second processing target frame image. To detect.

  The second feature point extraction function unit 25 extracts a plurality of face feature points for the subject detected by the face authentication function unit 23. Further, the position on the second processing target frame image is detected for each extracted facial feature point.

  The feature point association function unit 26 includes feature points of the subject's face extracted by the first feature point extraction function unit 24, and feature points of the subject's face extracted by the second feature point extraction function unit 25. , Are associated.

  The external parameter calculation function unit 27 includes the position of the feature point of the face of the subject associated with the feature point association function unit 26 on the first processing target frame image, and the feature point on the second processing target frame image. An external parameter related to the relative positional relationship between the first camera 10 and the second camera 11 is calculated using the position of the feature point of the subject's face associated with the association function unit 26. .

  Next, the camera calibration operation in the camera calibration unit 1 will be described. FIG. 3 is a flowchart showing camera calibration processing. The camera calibration unit 1 performs the camera calibration process shown in FIG. 3 when a command related to the execution instruction of the camera calibration process is input to the input unit 7.

  In this example, the camera calibration process is executed at the timing when the command related to the camera calibration process execution instruction is input to the input unit 7. However, the camera calibration process may be periodically executed. . For example, the camera calibration unit 1 may be configured to perform scheduling so that camera calibration processing is executed at midnight every Sunday or at midnight on the first day of every month.

  In the camera calibration unit 1, a frame image related to a moving image captured by the first camera 10 is input to the first moving image input unit 3, and a frame image related to a moving image captured by the second camera 11. Is input to the second moving image input unit 4.

  The image processing processor 2 uses a first image used for a camera calibration process to be described later from frame images (frame images input to the first moving image input unit 3) related to the moving images captured by the first camera 10. A processing target frame image is selected (s1). In s1, a frame image of a predetermined number of frames (for example, 10 frames) input immediately after the start timing of the camera calibration processing (timing at which a command related to the camera calibration processing execution instruction is input to the input unit 7). Then, the first processing target frame image is selected. For example, a frame image in which a person whose face is imaged from substantially the front in the target region is selected as the first processing target frame image. Also, this person (a person in the target area whose face is imaged from substantially the front) is selected as the target person. The processing related to s1 is performed by the first processing target frame image selection function unit 21 described above.

  The image processor 2 uses a second image used for camera calibration processing to be described later from frame images (frame images input to the second moving image input unit 4) of the moving images captured by the second camera 11. A processing target frame image is selected (s2). In s2, a frame image having an imaging timing that is closer in time to the imaging timing of the first processing target frame image selected in s1 is selected as the second processing target frame image. The processing related to s2 is performed by the second processing target frame image selection function unit 22 described above.

  Therefore, the imaging timing of the first processing target frame image selected in s1 and the imaging timing of the second processing target frame image selected in s2 are substantially the same.

  FIG. 4 is a diagram illustrating the first processing target frame image selected in s1 and the second processing target frame image selected in s2. In FIG. 4, illustrations of people other than the subject being imaged, the background, and the like are omitted. Depending on the angle difference between the first camera 10 and the second camera 11, the imaging position of the subject, the orientation of the subject, etc. in the first processing target frame image and the second processing target frame image are different. ing.

  The image processor 2 detects the target person selected in s1 by face authentication from the persons captured in the target area on the second processing target frame image (s3). The processing related to s3 is performed by the face authentication function unit 23 described above.

  The image processor 2 extracts a plurality of feature points of the face of the target person (target person selected in s1) captured in the target area on the first processing target frame image (s4). Further, the image processor 2 extracts a plurality of facial feature points of the target person (target person detected by face authentication in s3) imaged in the target area on the second processing target frame image (s5). The number of feature points of the subject's face extracted in s4 and the number of feature points of the subject's face extracted in s5 may or may not be the same. The processing relating to s4 is performed by the first feature point extraction function unit 24 described above. Further, the processing relating to s5 is performed by the second feature point extraction function unit 25 described above.

  The first feature point extraction unit referred to in the present invention includes a first processing target frame image selection function unit 21 and a first feature point extraction function unit 24. The second feature point extraction unit referred to in the present invention includes a second processing target frame image selection function unit 22, a face authentication function unit 23, and a second feature point extraction function unit 25.

  The image processor 2 associates the feature point of the subject's face extracted in s4 with the feature point of the subject's face extracted in s5 (s6). FIG. 5 is a diagram illustrating a correspondence result of the feature points of the subject's face extracted in s4 and s5 and the feature points of the subject's face in s6. The hatched small circles shown in FIG. 5 are the facial feature points of the subject extracted at s4 and s5. In s6, there may be a feature point that is not associated with the feature point extracted in s5 among the feature points of the subject's face extracted in s4, and conversely, Among the feature points, there may be a feature point that is not associated with the feature point extracted in s4. A straight line with double-ended arrows shown in FIG. 5 indicates that feature points are associated with each other. Also, feature points that are not marked with a straight line with double-ended arrows are feature points that could not be associated. The process related to s6 is performed by the above-described feature point association function unit 26.

  The image processor 2 determines whether or not the number of sets of feature points that can be associated in s6 is equal to or greater than a predetermined number (s7). In s7, the number of sets of feature points that can be associated in s6 is equal to or greater than the number of sets that can calculate the external parameters related to the relative positional relationship between the first camera 10 and the second camera 11. Judging whether or not. That is, in s7, it is determined whether or not an external parameter related to the relative positional relationship between the first camera 10 and the second camera 11 can be calculated.

  The camera calibration unit 1 determines the position (coordinate) on the first processing target frame image and the position (coordinate) on the second processing target frame image of the feature point associated with the external parameters of the camera in s6. Use to calculate. At this time, based on the first camera 10, the rotation R and translation T of the second camera 11 are calculated as external parameters of the camera. Various calculation methods are known for calculating the camera external parameters. For example, it is described in Patent Document 1 mentioned above. The camera calibration unit 1 may calculate the external parameters of the camera by any known calculation method.

  If the image processor 2 determines that the number of feature point pairs that can be associated in s6 is less than a predetermined number, the image processor 2 updates the start timing (s10), returns to s1, and repeats the above processing. In s10, the current time is set as the start timing.

  If the image processor 2 determines that the number of sets of feature points that can be associated in s6 is equal to or greater than a predetermined number, the relative position relationship between the first camera 10 and the second camera 11 is determined. Such external parameters are calculated (s8).

  The camera calibration unit 1 stores the external parameter calculated in s8 in the storage unit 5 (s9), and ends this process.

  Thus, the camera calibration unit 1 according to this example has a relative positional relationship between the first camera 10 and the second camera 11 without setting an object such as a chess board having a feature point in the target area. It is possible to calculate an external parameter of the camera according to the above. Therefore, it is possible to reduce the labor and time required for camera calibration.

  Further, since the target person imaged in the first process target frame image and the target person imaged in the second process control frame image are associated by face authentication, feature points are extracted from different target persons. Can be prevented. That is, processing after s5 is not performed wastefully. Further, in the first processing target frame image, a person whose face is imaged from substantially the front is selected as the target person, so that more feature points of the target person's face can be extracted.

  Moreover, since the imaging timings of the first processing target frame image and the second processing target frame image are substantially the same, a decrease in the accuracy of the external parameters of the camera to be calculated can be suppressed.

  In addition, the camera calibration unit 1 uses the external parameters stored in the storage unit 5 and applies the frame image related to the moving image captured by the first camera 10 and the moving image captured by the second camera 11. A stereoscopic image of the target area is generated using the frame image, and the generated stereoscopic image is output from the output unit 6.

  Further, the camera calibration unit 1 may be configured such that three cameras are connected and external parameters of each camera are calculated. In this case, by using one arbitrary camera as a reference, external parameters relating to the relative positional relationship with respect to the reference camera are calculated for the other two cameras.

  In the above example, the first processing target frame image and the second processing target frame image having substantially the same imaging timing used in camera calibration are set as one set, but the imaging timing is substantially the same. Camera calibration may be performed using a plurality of sets of one processing target frame image and second processing target frame images. If a plurality of sets of first processing target frame images and second processing target frame images having substantially the same imaging timing are used, the number of sets of associated feature points used for calculating external parameters in s8 can be increased. it can. Therefore, the accuracy of camera calibration can be improved.

  In addition, here, an example has been described in which a stereoscopic image of a target area is generated. However, even with a plurality of cameras that generate a full-circumference image described in Patent Document 2 described above, Parameters can be calculated.

  Further, the camera calibration unit 1 may be configured to be incorporated in any camera, or may be configured to be incorporated in an information processing apparatus such as a personal computer.

DESCRIPTION OF SYMBOLS 1 ... Camera calibration unit 2 ... Image processor 3 ... 1st moving image input part 4 ... 2nd moving image input part 5 ... Memory | storage part 6 ... Output part 7 ... Input part 10 ... 1st camera 11 ... 2nd Camera 21 ... first processing target frame image selection function unit 22 ... second processing target frame image selection function unit 23 ... face authentication function unit 24 ... first feature point extraction function unit 25 ... second feature point extraction Function unit 26: Feature point correspondence function unit 27: External parameter calculation function unit

Claims (7)

A first feature point extraction unit that extracts a plurality of feature points for the subject imaged in the first frame image captured by the first camera;
A second feature point extracting unit that extracts a plurality of feature points for the subject imaged in the second frame image captured by the second camera at an angle different from that of the first camera;
A feature point association unit that associates the feature points of the subject extracted by the first feature point extraction unit with the feature points of the subject extracted by the second feature point extraction unit;
Results of associating feature points of the subject by the feature point association unit, positions of the feature points of the subject on the first frame image, and features of the subject on the second frame image A camera calibration unit comprising: an external parameter calculation unit that calculates an external parameter of the camera related to a relative positional relationship between the first camera and the second camera using a point position. The first feature point extraction unit extracts a plurality of feature points of the face of the subject captured in the first frame image captured by the first camera,
The camera calibration according to claim 1, wherein the second feature point extraction unit extracts a plurality of feature points of the face of the subject captured in the second frame image captured by the second camera. unit.   The second feature point extraction unit selects the subject imaged in the second frame image by face authentication using the face image of the subject imaged in the first frame image. The camera calibration unit according to claim 1 or 2. A first moving image input unit for inputting a first moving image captured by the first camera;
A second moving image input unit for inputting a second moving image captured by the second camera,
The first feature point extracting unit is configured to input the first moving image obtained by inputting, to the first moving image input unit, the first frame image for detecting a plurality of feature points of the subject being imaged. Select from the frame images
The second feature point extraction unit is configured to select the second frame image for detecting a plurality of feature points of the subject being imaged, the first frame image selected by the first feature point extraction unit. The camera calibration unit according to any one of claims 1 to 3, wherein the camera calibration unit is selected from frame images related to the second moving image input to the second moving image input unit in accordance with an imaging timing.   The external parameter calculation unit calculates an external parameter related to a relative positional relationship of the second camera with respect to the first camera with the first camera as a reference. The camera calibration unit described. A first feature point extracting step of extracting a plurality of feature points for the subject imaged in the first frame image captured by the first camera;
A second feature point extracting step of extracting a plurality of feature points for the subject imaged in the second frame image captured by the second camera at an angle different from that of the first camera;
A feature point associating step for associating the feature point of the subject extracted in the first feature point extraction step with the feature point of the subject extracted in the second feature point extraction step;
Results of associating feature points of the subject in the feature point association step, positions of the feature points of the subject on the first frame image, and features of the subject on the second frame image A camera calibration method in which a computer executes an external parameter calculation step of calculating an external parameter of a camera related to a relative positional relationship between the first camera and the second camera using a point position. A first feature point extracting step of extracting a plurality of feature points for the subject imaged in the first frame image captured by the first camera;
A second feature point extracting step of extracting a plurality of feature points for the subject imaged in the second frame image captured by the second camera at an angle different from that of the first camera;
A feature point associating step for associating the feature point of the subject extracted in the first feature point extraction step with the feature point of the subject extracted in the second feature point extraction step;
Results of associating feature points of the subject in the feature point association step, positions of the feature points of the subject on the first frame image, and features of the subject on the second frame image A camera calibration program for causing a computer to execute an external parameter calculation step of calculating an external parameter of a camera related to a relative positional relationship between the first camera and the second camera using a point position.

Images