JP2019062279A - Imaging device, imaging system, and control method of imaging system - Google Patents

Abstract

To provide an imaging system that photographs a subject from a plurality of angles using a plurality of imaging devices and is not likely to lose sight of the subject, a control method of the imaging system, the imaging devices constituting the imaging system.SOLUTION: An imaging device 100A included in an imaging system in which a plurality of imaging devices are connected and track and photograph a main subject includes: photographing means 101 for tracking and photographing the main subject; reliability acquisition means 103 for acquiring the reliability of tracking; and first communication means 105 capable of receiving the reliability of tracking of other imaging devices. The imaging device further includes: selection means 106 for selecting an imaging device that operates in a master mode on the basis of the reliability of tracking; and switching means 117 for switching, according to a selection result, between a master mode of detecting the position of the main subject and tracking and photographing the main subject and a slave mode of tracking and photographing the main subject on the basis of positional information on the main subject acquired from another imaging device.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an imaging system, an imaging apparatus, and a control method of an imaging system, which capture an object using a plurality of imaging apparatuses.

  2. Description of the Related Art There has been developed an imaging system which captures an object from a plurality of directions using a plurality of cameras capable of pan and tilt control (hereinafter, pan and tilt cameras). By using this system, it becomes possible to shoot multi-view images in which the subject can be viewed from any viewpoint.

  According to Patent Document 1, in an imaging system in which a plurality of pan / tilt cameras are arranged, a cameraman operates one of them as a master camera, and direction control is performed so that the other slave cameras face the gazing point of the master camera. There is a description of the imaging system being

  On the other hand, there is known a technique for automatically tracking and photographing a specified subject by using a subject detection technique from a photographed image or using a subject marker for transmitting a radio wave.

JP, 2012-114593, A

  When the automatic tracking technology is applied to the imaging system described in Patent Document 1, the imaging system is controlled such that the subject specified by the master camera is automatically tracked, and the slave camera is directed to the fixation point of the master camera. However, when the master camera automatically tracks the subject, a scene is assumed in which the master camera loses sight of the subject due to the positional relationship between the subject and the master camera, and the subject can not be brought into view. The slave camera is controlled to point at the gazing point of the master camera assuming that the subject is at the gazing point of the master camera. Therefore, when the subject can not be contained within the angle of view of the master camera, all cameras are desired. A subject different from the subject is photographed.

  The present invention provides an imaging system that captures a subject from a plurality of angles using a plurality of imaging devices, and provides an imaging system that does not easily lose sight of the subject, a control method of the imaging system, and an imaging apparatus configuring the imaging system. The purpose is

  An imaging apparatus according to one aspect of the present invention is an imaging apparatus connected to another imaging apparatus and configured to track and capture a main subject, the imaging unit configured to track and capture the main subject and output an imaging signal. A reliability acquisition unit for acquiring the reliability of tracking by the imaging unit; a first communication unit capable of receiving the reliability of tracking of the other imaging apparatus from the other imaging apparatus; According to the reliability of the tracking of the imaging device and the reliability of the tracking acquired by the reliability acquisition device, according to the selection device for selecting the imaging device operating in the master mode and the selection result of the selection device The master mode for detecting the position of the subject and tracking and photographing the main subject based on the detection result, and the position information of the main subject acquired from the other imaging device are received, and the received main subject Based on location information There are characterized by comprising a switching means for switching a slave mode for photographing by tracking the main subject.

  Other aspects of the present invention will be clarified in the embodiments described below.

  According to the present invention, an imaging system for imaging a subject from a plurality of angles using a plurality of imaging devices, wherein the imaging system hardly loses sight of the subject, a control method of the imaging system, and an imaging device constituting the imaging system Can be provided.

It is a figure which shows the structural example of the camera which concerns on 1st and 2nd embodiment. It is a figure which shows the camera imaging which concerns on 1st and 2nd embodiment. It is a flowchart explaining the process of the master camera and slave camera in 1st Embodiment. It is a figure which shows the camera photography after the direction of the to-be-photographed object changed. It is a figure which shows the subject detection reliability of each camera before and behind the direction change of a subject. It is a flow chart explaining operation of a master camera in a 2nd embodiment. It is a figure which shows the time change of the output value of an acceleration sensor. It is a flowchart explaining operation | movement of the slave camera in 2nd Embodiment. It is a figure which shows the position of the reference | standard subject in the local coordinate before and behind camera movement. It is a figure which shows the captured image before and behind a camera movement, and the captured image after reference | standard object search.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In each embodiment, a multi-viewpoint camera system having a plurality of lens integrated cameras with pan (horizontal direction (right and left) swing) and tilt (vertical direction (upper and lower) swing) functions will be described. The present invention is applicable to various imaging devices such as interchangeable lens cameras. An imaging apparatus having a pan-tilt function may be realized by combining an apparatus for automatically adjusting pan-tilt with an imaging apparatus, such as a camera platform that moves automatically, or the pan-tilt adjustment mechanism is integrated It may be an imaging device.

First Embodiment
[Camera system configuration]
As shown in FIG. 2, the camera system according to the present embodiment is arranged so as to surround the subject 201 by the cameras 100A to 100D connected to each other via a communication unit and performs shooting. There are two types of roles of each camera constituting the camera system of this embodiment: a master camera (master imaging device) and a slave camera (slave imaging device). The master camera transmits, to the slave camera, positional information of a subject necessary to center the subject same as the master camera in the angle of view. The slave camera performs pan / tilt operation based on the position information of the subject received from the master camera such that the subject of the master camera fits within the angle of view of the slave camera. The camera system of the present embodiment is a mechanism in which roles of the master camera and the slave camera are switched according to the detection state of the subject of each camera, and a camera suitable for tracking the subject from a plurality of cameras constituting the camera system is master camera It can be selected as Each camera has a master mode and a slave mode as operation modes, and by switching the operation mode, the camera can operate as a master camera or can operate as a slave camera.

  FIG. 1 is a functional block diagram showing a configuration example in which the camera 100A operates as a master camera, which is one of a plurality of cameras constituting the camera system of the present embodiment. Each configuration will be described.

  The imaging unit 101 includes an imaging optical system and an imaging element that photoelectrically converts an optical image formed by passing through the imaging optical system into an electrical signal, and outputs an imaging signal. The imager optical system is composed of a plurality of lenses and stops. In addition, as the imaging device, a CCD or CMOS image sensor in which a plurality of pixels are two-dimensionally arranged can be used. The optical axis of the imaging optical system can be changed by a pan / tilt control unit 115 described later, and the pan / tilt is controlled according to the acquired subject position, so that the subject can be tracked and photographed.

  The imaging processing unit 102 obtains an imaging signal output from the imaging unit 101, and generates an image signal by applying processing such as noise reduction, AD conversion, white balance adjustment, and color interpolation.

  The reliability acquisition unit 103 acquires the reliability of subject detection by the camera 100A. In the present embodiment, the reliability is acquired using the imaging signal imaged by the imaging unit.

  A method of acquiring the reliability using an imaging signal will be described. The registered image 202 stored in the subject image storage unit 104, which will be described later, is compared with the image corresponding to the image pickup signal picked up by the image pickup unit 101 every predetermined cycle (for example, every frame rate). Get confidence based on. The similarity is determined by image processing techniques such as feature point comparison, block matching, pattern recognition techniques and the like. If the degree of similarity is high, it is considered that the subject of the registered image 202 exists within the angle of view of the imaging device, and a high degree of reliability is given. The similarity may be used directly as the reliability, or the similarity may be converted to the reliability. FIG. 2 is a view showing an example of the positional relationship between each camera 100A to 100D and the subject 201, and FIG. 5 (A) is an image of a registered subject taken with each camera in the state of FIG. It shows the relationship of the reliability of existing registered images. When the positional relationship between each of the cameras 100A to 100D and the subject 201 is the positional relationship shown in FIG. 2, the reliability between the image corresponding to the imaging signal captured by each of the cameras 100A to 100D and the registered image 202 is camera 100A is the highest, and camera 100C is the lowest. Therefore, as shown in FIG. 5A, the reliability acquired by the reliability acquisition unit 103 is also the highest for the camera 100A and the lowest for the camera 100C. When a plurality of subjects exist in the image corresponding to the imaging signal, the similarity with the registered image is acquired for each subject, and the reliability is acquired based on the similarity of the subject with the highest similarity. The reliability may be acquired not only for one subject but also for each of a plurality of subjects, and may be stored in association with position information of the subjects. Thus, by acquiring the reliability based on each of a plurality of subjects and associating it with the position of each subject, it is possible to continue tracking without erroneously detecting a registered subject even when there is a similar subject. It can be enhanced. The reliability acquisition by the reliability acquisition unit 103 is performed whether the imaging apparatus operates as a master camera or operates as a slave camera.

  The subject image storage unit 104 is a storage unit that stores information of an image of a subject to be tracked as registered image information, and can be configured by various memories such as a hard disk and a flash memory. By storing the registered image information before photographing, the reliability acquisition using the above-described imaging signal can be performed. In addition, when acquiring a reliability using a signal from a subject, it is also possible to omit this configuration.

  The communication unit 105 communicates with other cameras (100B to 100D) constituting the system. The master camera (camera 100A) performs wireless communication with all slave cameras (cameras 100B to 100D) of the camera system via the communication unit 105. On the other hand, each slave camera communicates with only the master camera. The master camera transmits the position information of the subject to be tracked to each slave camera at predetermined time intervals. Each slave camera transmits the reliability acquired by the reliability acquisition unit of each slave camera to the master camera at predetermined time intervals.

  The master camera selection unit 106 selects a master camera based on the reliability acquired by each slave camera and the reliability acquired by the master camera itself. The selection result including information specifying the camera operating in the master mode is transmitted to the mode switching unit 117, and switching between the slave mode and the master mode is performed according to the selection result. Basically, the master camera selection unit 106 may select a camera with a high degree of reliability as a master camera, and a method of selecting a camera with the highest degree of reliability as a master camera can be considered. However, in order to prevent the master camera from switching frequently, the master camera is switched only when the slave camera has higher reliability and the difference between the reliability of the slave camera and the reliability of the master camera is equal to or greater than the threshold. May be With such a configuration, even if the reliability of the slave camera is slightly higher than the reliability of the master camera, the master camera is not switched, so frequent switching of the master camera can be prevented. Alternatively, a camera that operates in the master mode may be selected after a predetermined time has elapsed since it started operating as a master camera. Further, the master camera may not be switched unless the state in which the reliability of the slave camera is higher than the reliability of the master camera continues for a predetermined time or more. With such a form, the same effect can be obtained.

  The reference subject storage unit 107 is a storage unit that stores an image of a subject used as a reference when detecting the movement amount of the camera, and can be configured by various memories such as a hard disk and a flash memory. The movement amount detection of the camera using the image stored in the reference subject storage unit 107 will be described in detail in a second embodiment.

  The subject position detection unit 108 detects the distance to the subject based on the radio wave signal detected by the radio wave detection unit 112. The radio wave detection unit 112 receives a radio wave emitted by an object to be photographed to notify the camera of its position by beacon or the like, and outputs this signal to the object position detection unit 108. The subject position detection unit 108 acquires the position of the subject on the local coordinates based on the distance from the camera 100 detected by the radio wave signal to the subject and the direction information of the subject acquired based on a known tracking technique. As a known tracking technique, a method of acquiring a subject position based on an image signal acquired from the imaging processing unit 102 (considered as a kind of imaging signal for an imaging signal subjected to video processing) can be used. Simply, the position of the subject to be tracked in the imaging range may be detected, or the motion vector may be acquired to predict the position of the subject. Further, the subject position detection unit 108 converts the subject position information on the local coordinates into the subject position information on the world coordinates, using the information acquired at the time of calibration.

  The photographing parameter holding unit 109 is for storing photographing conditions such as the pan and tilt angles of the camera and the subject distance, and can be configured by various memories. The acceleration sensor 110 is for detecting the acceleration of the camera, and can be configured by various gyro sensors or the like.

  The camera movement amount detection unit 111 detects the acceleration acquired by the acceleration sensor 110, the position information of the reference subject stored in the reference subject storage unit 107, and shooting parameters such as the pan / tilt angle and the subject distance when the reference subject is registered. Calculate the movement amount of the camera based on.

  The rotation matrix generation unit 113 generates a rotation matrix of pan / tilt target positions based on subject position information received by the subject position detection unit 108 or the communication unit 105. The pan / tilt position detection unit 114 detects a pan / tilt position based on the encoder position. The pan / tilt control unit 115 uses the comparison result between the target position set by the rotation matrix generation unit 113 and the pan / tilt position detected by the pan / tilt position detection unit 114 so that both are matched. Control.

  The camera position detection unit 116 acquires the position of the camera on world coordinates based on the camera position on world coordinates at the time of calibration and the camera movement amount from the camera movement amount detection unit 111. When the camera position detection unit 116 detects movement of the camera, based on the camera movement amount acquired by the camera movement amount detection unit 111 and the camera position on the world coordinates before movement, the world coordinates of the camera after movement Detect the position above. In addition, the camera position detection unit 116 notifies the subject position detection unit 108 and the communication unit 105 of the detected position of the camera on world coordinates.

  A method of switching the master camera and the slave camera in accordance with the reliability of subject detection in the present embodiment will be described. In this case, the subject is tracked based on the radio wave signal from the beacon acquired by the radio wave detection unit 112, and the reliability is acquired based on the similarity to the registered image stored in the subject image storage unit 104 in advance. Explain about

  First, before starting photographing by the camera system, calibration of the arrangement of the cameras 100A to 100D and the gaze point (camera gaze point alignment processing) is performed. When arranging each of the cameras 100A to 100D of the multi-viewpoint camera system, the cameras are arranged so as to surround the subject. After placing the cameras, perform calibration for each camera. Calibration is performed using the same calibration chart (eg, dot pattern or checkered pattern) in each camera.

  The calibration is performed in the order of setting of the reference subject, search of the reference subject, and detection of the position on the local coordinates of the camera and the position on the world coordinates. In setting the reference subject, first, a calibration chart is set as the reference subject, and the image is stored in the reference subject storage unit 107. The chart used for calibration has a clear pattern of the origin (a position at an angle of 0 °) and has a pattern that changes with each shooting angle. By using a chart that changes according to the angle at which the image is taken, when the chart is placed at the center of the camera image, it is possible to detect from the image the angle at which the camera is based on the origin of the chart. A process of obtaining an angle of a camera based on an origin of a chart from an image is performed by the imaging processing unit 102. After setting the reference subject, a search for the reference subject is performed. The search pans and tilts the camera 100A until the chart comes to the center of the camera image. During pan and tilt driving, the pan and tilt position detection unit 114 detects pan and tilt positions at predetermined time intervals, and the pan and tilt control unit 115 performs speed control so that the driving speed becomes constant. When the chart as the reference subject comes to the center of the camera image, the pan and tilt driving is stopped. After the search for the reference subject is completed, the position of the camera on the local coordinates and the position on the world coordinates are respectively detected. The distance from the chart to the camera is detected based on radio wave information from the beacon by setting a beacon for distance measurement on the chart before calibration starts. The radio wave detection unit 112 detects radio waves from the beacon. The position of the camera on world coordinates is acquired from the angle at which the camera is based on the chart origin and the distance from the chart to the camera. The angle at which the camera is present is detected from the pan and tilt angles detected by the pan and tilt position detection unit 114. The position of the camera in local coordinates is detected from the distance from the chart to the camera and the pan and tilt angles when the chart is placed at the center of the camera image. The pan and tilt angles are detected by the pan / tilt position detector 114. The detected position on the world coordinate and the position on the local coordinate are stored in the camera position detection unit 116. When exchanging positional information with another camera (master or slave), coordinate conversion is performed using the relationship between the position on the local coordinates at the time of this calibration and the position on the world coordinates.

In calibration, a target subject (pattern for calibration) is set as a reference point of world coordinates, and the amount of deviation of camera coordinates (local coordinates) with respect to world coordinates is converted into a rotation vector R _m and a translation vector T _m Remember. The relationship between the world coordinate system (X, Y, Z) and the local coordinates (x, y, z) is expressed by the following equation (1).

  Such calibration is performed for all the cameras constituting the camera system. After the calibration is completed, a camera to be a master camera is selected. There are two methods for determining the master camera, which can be selected by the user as appropriate. One is a method in which a camera system automatically determines a master camera. The other is a method in which the user designates a master camera. As described above, in the method of automatically determining the master camera by the camera system, the reliability of detection of the subject during shooting is acquired for all the cameras 100A to 100D that constitute the camera system, and the camera with the highest reliability is selected. This is a method of using a master camera. In a method in which the user specifies the master camera, which is the other method, the setting using wireless communication of an external device by a mobile phone or the like, or the master camera or the slave camera is set by directly operating the camera.

  Next, a process of switching the master camera and the slave camera during shooting according to the similarity between the registered subject and the subject photographed by each camera will be described. FIG. 3 is a flowchart of processing for switching the master camera and the slave camera in accordance with the similarity of the subject during tracking of the subject to be photographed. The processing on the master camera side on the left side in the drawing will be described. In step S301, first, an object to be tracked is specified. In the present embodiment, a subject is specified by storing image information of a subject to be tracked (hereinafter, referred to as a main subject) in the subject image storage unit 104. The subject image storage unit 104 may store a plurality of images of candidates for the main subject in advance, and may select the present main subject from among the candidates.

  After the designation of the main subject is completed, the process proceeds to step S302. In step S302, the pan / tilt control unit 115 pan / tilt drives the master camera so that the main subject is at the center of the angle of view based on the position of the subject acquired by the subject position detection unit 108. Perform tracking. When the main subject is captured near the center of the angle of view by pan / tilt driving, the process proceeds to step S303, and position detection of the subject in camera coordinates (local coordinates) of the master camera is performed. As described above, the position of the master camera on the local coordinates is calculated using the subject distance estimated from the radio wave intensity of the beacon and the pan and tilt angle information. It should be noted that instead of using a beacon, the subject distance is detected using a distance sensor other than a beacon such as the position of the focus lens of the camera or the subject image captured by the image sensor or each pixel of the subject image You can also After the position of the registered subject in the local coordinates is obtained, the process proceeds to step S304, where the position of the main subject in the local coordinates is converted to the position in the world coordinates (S304). When converting to a position on world coordinates, it is converted to world coordinates based on a matrix (rotation matrix R and translation vector T) representing the relationship between local coordinates and world coordinates stored at the time of calibration. When the conversion from the local coordinates of the object position to the world coordinates is completed, the process proceeds to step S305, and the position information of the main object on the world coordinates is transmitted to each slave camera via the communication unit 105. Next, in step S306, the reliability of subject detection is acquired. As described above, the reliability of subject detection is obtained from the similarity between the comparison result of the subject image registered in the subject image storage unit 104 and the image acquired by the imaging unit 101.

  On the other hand, in each of the slave cameras 100B to 100D, in steps S601 to S605 described later, it is assumed that the main subject is within the angle of view, assuming that the main subject is present at the main subject position on the world coordinates sent from the master camera. Pan and tilt at the same time. Then, the reliability is acquired based on the captured image and transmitted to the master camera. When the master camera receives the degree of reliability sent from each slave camera in step S307, the process advances to step S308, and the master camera selection unit 106 selects a master camera. When the master camera is selected, selection information is transmitted to each slave camera in step S309. After transmission, the process proceeds to step S310, and the mode switching unit 117 of each camera performs master switching processing. FIG. 4 shows the positional relationship between the subject 201 and the cameras 100A to 100D when the direction of the subject is changed from the state of FIG. 2, and FIG. 5B shows the registered subject as each camera in the state of FIG. The relationship between the reliability of the image photographed in the above and the registered image stored in the subject image storage unit is shown. In the case of the positional relationship of FIG. 4, the reliability of the camera 100D which has been the slave camera up to now is the highest. If shooting with the camera system is continued with the camera 100A as the master camera in such a state, there is a possibility that the detection accuracy of the main subject position may be degraded, or the main subject may be erroneously detected when there are multiple similar subjects. Get higher. As a result, a subject other than the main subject may be photographed, or the slave camera may not be able to track the main subject. As described above, when the camera with the highest reliability is changed to the camera 100D, the master camera selection unit 106 transmits master selection information to each camera so as to change the camera 100D to the master camera.

  Next, processing on the slave camera side of switching processing between the master camera and the slave camera will be described. The right side of FIG. 3 shows a flowchart on the slave camera side in which the master camera and the slave camera are switched according to the reliability of the subject while tracking the main subject. First, in step S601, subject position information on world coordinates transmitted from the master camera at predetermined time intervals by wireless communication is received. Next, in step S602, the received subject position based on the world coordinate is converted to the position on the slave camera's own local coordinate. The conversion method will be described.

Let R _{s0 be} a rotation matrix indicating the deviation between the world coordinates stored at the time of calibration and the local coordinates of the camera at the time of calibration, and let the translation vector be T _s0 . At this time, the equation for converting the world coordinate position (X _s , Y _s , Z _s ) ^T of the object to the position (x _s , y _s , z _s ) ^T on the local coordinates of the slave camera is the following equation (2) Is represented by

  Further, when the position of the subject on the local coordinates of the slave camera is converted into the position on the world coordinates, it is converted by Expression (3).

  When the object position on the world coordinates sent from the master camera is converted to the position on the local coordinates of the slave camera, next, in step S603, the pan and tilt target angles are acquired. A method of acquiring the pan and tilt target angles of the camera from the target position on the local coordinates will be described.

Calibration was performed with the direction of rotation around the Z axis as the pan direction, the direction of rotation around the X axis as the tilt direction, and the position on the local coordinates where the object exists as (x _s , y _s , z _s ) It is assumed that θ is rotated in the pan direction and φ in the tilt direction from the camera position at that time. At this time, the subject distance l is expressed by the following equation (4).

  The target angle θ of pan and the target angle φ of tilt are expressed by Equations (5) and (6).

  In step S604, pan / tilt drive control is performed such that the gaze point of the camera is directed to the target angle obtained by calculation using equations (4) to (6).

  Next, in step S605, the reliability of subject detection is acquired. The reliability is acquired as in step S306. If the reliability is acquired, the process advances to step S606 to transmit reliability information to the master camera. Next, in step S607, the master select information from the master camera is received, and the process proceeds to step S608. In step S608, it is determined based on the master select information whether to change to the master or to remain as the slave. When switching to the master camera, processing to switch the operation mode to the master camera is performed.

  Thus, according to the multi-viewpoint camera system of the present embodiment, the camera with the highest degree of similarity between the photographed subject and the registered subject is switched to the master camera. This reduces the possibility of erroneous detection of the subject even when the subject's orientation is changed, or when an obstacle appears between the camera and the subject and the master camera makes it difficult to catch the subject, while shooting with the multi-viewpoint camera system Can continue.

Second Embodiment
As described above, the viewpoints of the cameras constituting the multi-viewpoint camera system are adjusted for each camera so as to face the fixation point of the master camera. Therefore, if the position of the master camera is moved during shooting, the slave camera shoots an object different from the master camera. In addition, even if the position of the slave camera is moved, that camera captures an object different from the master camera. These problems can be solved by recalibration of the multi-viewpoint camera system, but it takes time to be able to shoot again because calibration needs to adjust the gazing points of all cameras. .

  Therefore, in the present embodiment, a method of determining the positional relationship between the camera coordinates and the world coordinates by detecting the movement amount of the camera using the image information and continuing the multi-view camera shooting will be described. The amount of movement of the camera sets the reference subject at the time of calibration, stores the position of the reference subject on camera coordinates, and searches for the reference subject when the camera moves from the position at the time of calibration. It can detect by doing.

  A general multi-view camera system assumes that the relationship between the local coordinates of the camera obtained by calibration and the world coordinates is known. Therefore, when the camera moves from the calibrated position, the relationship between the camera coordinates and the world coordinates changes, and conventionally, it has been difficult to correctly track the subject unless calibration is performed again. Instead of performing calibration, there is also a method of detecting the amount of movement of the camera and correcting the relationship between the local coordinates of the camera and the world coordinates. As a method of detecting the movement amount of the camera, there is a method using GPS (Global Positioning System). However, in order to detect the movement of the camera accurately using the GPS, a high precision GPS receiver is required, resulting in an increase in cost. In addition, GPS has a problem that the accuracy is significantly reduced when there is an obstacle indoors or around. As a method of solving these problems, a method (PDR: Pedestrian Dead Reckoning) which detects a movement amount using an acceleration sensor or a geomagnetic sensor can be considered. However, even with this method, the acceleration sensor may not be able to accurately estimate the amount of movement of the camera because there is a limit that can detect acceleration. For example, in the case of applying an abrupt impact such as an impact to the acceleration sensor or moving the camera at a high speed, the detection limit of the sensor is exceeded, so it is often impossible to obtain an accurate amount of movement. In the present embodiment, in order to solve such a problem, a method of detecting the movement amount of a camera using image information and determining the positional relationship between camera coordinates and world coordinates will be described.

  FIG. 6 is a flowchart of the master camera in the processing of detecting the amount of movement of the camera using image information and correcting the relationship between world coordinates and local coordinates. First, in step S701, a reference subject is registered. Registration of a reference subject is performed at the time of calibration operation. First, the user automatically selects a plurality of reference subjects from among the images taken by each camera, and stores the images of the subjects in the subject image storage unit 104. The reference subject to be registered is one in which the subject itself does not move, the color and shape are prominent, and the proportion of the angle of view at the time of calibration is a predetermined amount or more (the number of pixels recognizable as a reference object) Is preferred. When the reference subject is determined, the subject position detection unit 108 acquires image information from the imaging processing unit 102, and the local coordinates from the position of the reference object in the captured image, the pan / tilt position of the camera at shooting and the subject distance Detect the position of the reference subject above. Then, the position of the reference object on the detected local coordinates is converted into the position on the world coordinates, and the position of the reference object on the world coordinates is stored, thereby completing the registration of the reference object.

  When registration of the reference subject is completed, shooting with the multi-viewpoint camera system is started. During shooting, the presence or absence of movement of the camera is constantly detected in step S702. The presence or absence of movement of the camera is determined in accordance with the output value of the acceleration sensor. If the value of the detected acceleration is equal to or less than a predetermined threshold value, it is determined that the camera has not moved, and the process waits until it is determined that the camera has moved. If it is the threshold value or more, it is determined that the camera has moved, and the process proceeds to step S703, and a new master camera is selected from slave cameras. The camera selected as the master camera is a slave camera with the highest degree of reliability acquired immediately before detecting the movement of the camera. For example, as shown in FIG. 2, when the camera 100A as the master camera moves in a situation where the camera A is used as a master camera, the camera 100B having the highest similarity among the slave cameras is regarded as the master camera. To be elected. After the master camera is selected, the process advances to step S704 to transmit master select information to each slave camera. After the transmission, the process advances to step S705 to switch the operation mode from the master camera mode to the slave camera mode. Next, in step S706, it is determined whether the detection result of the acceleration sensor 110 is valid. Whether the detection result of the acceleration sensor 110 is valid or invalid is determined by the magnitude of the detected value.

  FIG. 7 is a diagram showing temporal changes as a result of applying a low pass filter to the output of the acceleration sensor 110 when the camera is moved. In FIG. 7, ath is an upper limit value of acceleration that can be detected by the acceleration sensor 110, and −ath is a lower limit value of acceleration that can be detected by the acceleration sensor 110. In FIG. 7, the period acceleration sensor at tc indicates the upper limit value of the detection range. If the output of the acceleration sensor 110 indicates the upper limit value, the camera may actually move at an acceleration higher than that, and the acceleration may not be detected correctly. Therefore, when the camera movement amount is acquired from the output value of the acceleration sensor 110 when the output value of the acceleration sensor 110 is the upper limit value, the error between the actual camera movement amount and the acquired movement amount of the camera becomes large. There is. Therefore, in the present embodiment, when the output value of the acceleration sensor 110 indicates the upper limit value or the lower limit value as described above, it is assumed that the acceleration due to the movement of the camera exceeds the detection limit of the acceleration sensor 110 and the output of the acceleration sensor The result is determined to be invalid. Then, the process proceeds to step S 708, where the movement amount of the camera using the registered reference subject is acquired. If the output value of the acceleration sensor is between the lower limit value and the upper limit value, it is determined that the output result of the acceleration sensor is valid, and the process proceeds to step S 707. After acquiring the movement amount in step S707 or S708 of acquiring the movement amount based on the information of the acceleration sensor, the process proceeds to step S709, and the transformation matrix of the world coordinates and the local coordinates is updated.

A method of acquiring the movement amount of the camera using the reference subject in step S708 will be described. FIG. 9 shows the relationship between camera coordinates before and after camera movement. Usually, the representation is performed in three-dimensional coordinates, but in the present embodiment, each camera is disposed at the same height for simplicity and is represented in a two-dimensional coordinate system as a search is performed only in the pan direction. . In FIG. 9, reference numeral 1001 denotes a camera coordinate axis before camera movement detection, and reference numeral 1002 denotes a camera coordinate axis after camera movement detection. P ₁ and P ₂ in FIG. 9 is the center position of the reference subject existing in the image registered during calibration. D 9 is the distance of the distance _{P 1} and _{P 2,} from l 1, _{l 2} is the front camera moving each camera to the reference object position _P 1, _{P 2.} First, at the time of calibration, the position (P ₁ , P ₂ ) of the reference subject in the local coordinates and the distance between the two (d in FIG. 9) are calculated and stored (S 901). FIG. 10A shows an example of an image acquired when the reference subject is registered. In the image of FIG. 10, the position of the reference object flowers 1101 corresponds to _{P 1,} 2 present position of the tree 1102 of the reference object corresponds to _{P 2.} As shown in FIG. 10A, two stationary objects existing at the same angle of view are stored in the subject image storage unit 104 as a reference subject. When the camera automatically determines the reference subject, it is necessary to determine whether the subject in the image is a stationary object. The determination as to whether or not the object is a stationary object detects the amount of movement (moving speed) on the image in a predetermined time of each subject in the image, and determines an object having a predetermined velocity or less as the stationary object. When detecting the amount of movement on the image, optical flow or template matching can be used. After registration of the reference subject is completed, detection of the movement amount and direction of the camera is started. FIG. 10B is an image acquired after camera movement detection. In the image of FIG. 10B, only the reference subject 1101 exists, and the arrangement on the captured image is shifted to the far left. Therefore, the camera has moved to the right with respect to the subject. The moving direction of the camera may be determined not by image processing such as optical flow or template matching but by the sign of the output value of the acceleration sensor. From these pieces of information, the pan / tilt drive direction is determined, and a search for reference subjects 1101 and 1102 is performed. FIG. 10C shows a camera image when the two reference subjects 1101 and 1102 are accommodated in the angle of view by panning the camera. The positions P ₁ and P ₂ of the reference objects 1101 and 1102 in camera coordinates at this time are acquired and stored. From the reference object position P ₁ and P ₂ before and after the movement of the cameras to obtain a movement amount of the camera. From FIG. 9, after the camera moves the reference subject positions P ₁ and P ₂ after rotational movement by ∠E around the z axis perpendicular to the xy plane as compared with the reference subject positions P ₁ and P ₂ before the camera movement. It is equal to the position translated by the translation vector T _s1 . The amount of movement of the center of the camera coordinates before and after the movement of the camera is expressed by the following equations (7) to (11) using the vector T _s1 .

Angle D of formula (10) is obtained from the pan encoder position when matches the angle of the reference subject P ₂ after the camera movement. The rotation matrix R _s1 for converting the local coordinates before the camera movement to the local coordinates after the camera movement is expressed by the following equation.

  A transformation matrix for transforming local coordinates after camera movement into world coordinates is expressed by the following equations (13) and (14).

  As described above, even if calibration is not performed for all the cameras as described in the first embodiment, the relationship between the local coordinates after the movement of the camera and the world coordinates can be acquired, so that the fixation point can be matched. .

Next, processing for correcting the relationship between world coordinates and local coordinates in the slave camera will be described using the flowchart in FIG. Steps S901 to S902 are the same as steps S701 to S702, and therefore the description thereof is omitted. If it is determined that the slave camera has moved, the process advances to step S 903 to notify the master camera of the movement. When the photographer intentionally moves the slave camera to a position where it is easy to detect the subject, there is a high possibility that the order of the degree of reliability of each camera will change when it is moved.
In such a case, it is better to communicate immediately without waiting for communication between the slave and the master, which are performed at regular intervals. If you notify the master that there has been a move, you can quickly switch between the slave and the master. When the notification to the star camera is finished, the process proceeds to step S904. Since steps S904 to S907 are the same as steps S706 to S709, the description will be omitted.

  As described above, even when the information of the sensor for detecting the camera movement can not be used, the positions of the two objects serving as the reference on the local coordinates are detected and stored, and the local coordinates of the reference object before and after the camera movement are stored. The amount of movement of the camera can be estimated from the difference in position at. As a result, even when moving a camera while performing multi-view camera shooting by cooperation of a plurality of cameras, multi-view camera shooting can be continued without performing position adjustment (calibration) of the camera.

  Although an example using an acceleration sensor is shown in this embodiment, the same effect can be expected even when using other sensors such as a geomagnetic sensor, a millimeter wave radar, a gyro sensor, etc. without being limited to the acceleration sensor. The sensor for detection may be other than the acceleration sensor.

[Modification]
In the first embodiment, the method of acquiring the reliability based on the similarity between the registered subject image and the subject included in the captured image has been described as the method of acquiring the detection reliability. However, another method can be used as long as it is possible to acquire an index indicating whether the subject included in the captured image is the main subject specified in step S301.

  As an example, a method of acquiring the reliability using a signal from a subject will be described. Specific examples of the signal from the subject include a radio wave signal from a radio wave transmitter (such as a beacon) held by the subject, and an audio signal from an audio generator (such as a speaker) held by the subject itself or the subject. When obtaining the reliability from the radio wave signal or the audio signal, a value indicating the strength of the signal can be used as the reliability. For example, it is possible to use a value obtained by standardizing RSSI (Received Signal Strength Indication) (a value obtained by multiplying the received signal strength by a coefficient so that the maximum signal strength that can be received by the camera is 1).

Other Embodiments
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

104 subject image storage unit 106 master camera selection unit 107 reference object storage unit 115 pan / tilt control unit

Claims (15)

Another imaging apparatus is connected, and is an imaging apparatus that tracks and shoots a main subject,
Shooting means for tracking and shooting the main subject and outputting a shooting signal;
Reliability acquisition means for acquiring the reliability of tracking by the imaging means;
First communication means capable of receiving the tracking reliability of said another imaging device from said another imaging device;
Selection means for selecting an imaging device operating in a master mode based on the tracking reliability of the other imaging device and the tracking reliability obtained by the reliability obtaining means;
Depending on the selection result of the selection means
The master mode in which the position of the main subject is detected, and the main subject is tracked and photographed based on the detection result;
Switching means for receiving the position information of the main subject acquired from the other imaging device, and switching to a slave mode for tracking and photographing the main subject based on the received position information of the main subject An imaging device characterized by   The imaging apparatus according to claim 1, wherein the communication unit transmits the selection result by the selection unit to the another imaging apparatus. When the reliability of the tracking acquired by the reliability acquiring unit is lower than the reliability of the tracking of the other imaging device received by the first communication unit, the selection unit is another imaging device. Select as an imaging device operating in master mode,
The imaging device according to claim 1, wherein the switching unit switches an operation mode from the master mode to the slave mode. The selection unit is configured such that the reliability of tracking of another imaging device received by the first communication unit is higher than the reliability of tracking acquired by the reliability acquisition unit, and the reliability acquisition unit An imaging apparatus that operates another imaging apparatus in the master mode when the difference between the tracking reliability acquired by the first communication unit and the tracking reliability of another imaging apparatus received by the first communication unit is equal to or greater than a threshold Choose as
The imaging apparatus according to any one of claims 1 to 3, wherein the switching unit switches an operation mode from the master mode to the slave mode.   5. The image pickup apparatus according to claim 1, wherein the selection unit selects an image pickup apparatus operating in the master mode based on the reliability acquired after a predetermined time has elapsed since the operation in the master mode starts. An imaging device according to any one of the items. Object position detection means for detecting the position of the main object;
The photographing means tracks the main subject based on the detection result of the position of the main subject by the subject position detection means,
The imaging apparatus according to any one of claims 1 to 5, wherein the communication unit transmits the detection result of the subject position detection unit to the another imaging apparatus. The subject position detection means
7. The image pickup apparatus according to claim 6, wherein the position of the main subject is detected based on an image pickup signal output from the image pickup means. The photographing means is
The imaging apparatus according to claim 6, wherein the position of the main subject is tracked based on the position of the main subject acquired by the subject position detection unit. A storage unit for storing image information of the main subject;
The reliability degree acquiring means acquires the reliability degree based on the similarity between the image of the main subject stored in the storage unit and the image corresponding to the imaging signal output from the photographing means. The imaging device according to any one of 8. A second communication unit configured to receive a signal from the main subject;
9. The reliability degree acquiring unit according to claim 1, wherein the reliability degree acquiring unit acquires the reliability degree based on the strength of the signal from the main subject received by the second communication unit. 10. Imaging device. Another imaging apparatus is connected, and is an imaging apparatus that tracks and shoots a main subject,
Shooting means for tracking and shooting the main subject and outputting a shooting signal;
Reliability acquisition means for acquiring the reliability of tracking by the imaging means;
The first communication capable of transmitting the reliability of the tracking acquired by the reliability acquiring unit to the other imaging device and capable of receiving the position information of the main subject from the other imaging device Means, and
The photographing means is
An imaging apparatus characterized by tracking the main subject based on position information of the main subject received from the other imaging apparatus. The first communication means receives information specifying an imaging device operating in a master mode,
Based on the information
A master mode for detecting the position of the main subject and tracking and photographing the main subject based on the detection result;
It is characterized by comprising switching means for receiving the position information of the main subject acquired from the other imaging device, and switching to a slave mode for tracking and photographing the main subject based on the received position information of the main subject. The imaging device according to claim 11. An imaging system comprising a plurality of imaging devices, the imaging system comprising:
Each of the plurality of imaging devices is
Shooting means for tracking and shooting the main subject;
Reliability acquisition means for acquiring the reliability of tracking by the imaging means;
A first communication unit capable of transmitting or receiving the tracking reliability acquired by the reliability acquiring unit;
A master mode for detecting the position of the main subject and tracking and photographing the main subject based on the detection result;
Switching means for receiving the position information of the main subject acquired from another imaging device, and switching to a slave mode for tracking and photographing the main subject based on the received position information of the main subject,
Based on the tracking reliability of each of a plurality of imaging devices,
An imaging system comprising: selection means for selecting an imaging device operating in the master mode.   The imaging system according to claim 13, wherein the selection unit is provided with an imaging device operating in the master mode among the plurality of imaging devices.   15. The imaging system according to claim 13, wherein the imaging device including the selection unit switches in response to switching of the imaging device operating in the master mode.

Images