JP3412945B2 - Image synthesis device and photographing device - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing apparatus and a photographing apparatus for obtaining a panoramic image having an arbitrary aspect ratio by synthesizing a plurality of images. The present invention relates to an image synthesizing apparatus and a photographing apparatus which obtain a high-quality synthesized image by performing coordinate transformation processing using distance information of a subject when is distributed over a wide range from a short distance to a long distance. is there. [0002] Conventionally, a method of changing the aspect ratio of a television or video image (for example, NTSC 4 to 3 and H
D or ED2 conversion to 16: 9)
There is known a method of trimming the top and bottom or left and right of an image at the time of output. However, since these methods use a part of a picked-up image, there is a disadvantage that the field of view is reduced. For example, image information captured by an HD camera using a 16: 9 image sensor is converted into a 4: 3 NTSC image.
When output is performed on a monitor for use, there is no problem in terms of image quality, but there is a disadvantage that the horizontal field is reduced by 1/3. In order to solve the above-mentioned drawbacks, the applicant of the present invention disclosed in Japanese Patent Application No. Hei 5-223544 a plurality of image pickup systems, each of which picks up an image with a part of the field of view overlapped, and performs coordinate transformation processing on each image. A compound eye imaging apparatus has been proposed in which an image with a wide field of view is obtained by combining the images with each other. According to the proposal, in a device that uses a plurality of image pickup systems to image a common subject while partially overlapping a field of view, the amount of positional shift of a viewpoint from the plurality of image pickup systems and the convergence of the optical axis. Assuming an imaging system in which the viewpoint position and the direction of the optical axis are defined by the angles, such as when photographed by the virtual imaging system,
By combining and converting one image signal output in a state defined by an arbitrary object distance and an imaging magnification, it is possible to obtain an image in which the image quality is less deteriorated and the distortion generated at the time of convergence is corrected. I have to. FIG. 10 is an explanatory view of an image pickup system of the compound eye image pickup apparatus, and shows an arrangement when two image pickup systems 10L and 10R are used. In FIG. 1, reference numeral 1 denotes a plane of a subject to be photographed by the imaging systems 10L and 10R, and 11L and 11R denote a left imaging optical system and a right imaging optical system having equivalent specifications, respectively. Have been. Reference numerals 12L and 12R denote a left image sensor and a right image sensor having equivalent specifications, respectively, and use an image pickup tube such as a saticon or a solid-state image pickup device such as a CCD. LL and LR are imaging optical systems 11L and 11L, respectively.
The optical axis 11R, 2L, and 2R are each an image sensor 12R.
It is an object plane of the imaging systems 10L and 10R conjugate to L and 12R. The imaging systems 10L and 10R adjust the optical axes LL and LR within the same plane so as to overlap the respective fields by a required amount according to the selected aspect ratio.
Are arranged symmetrically about θ with respect to the normal line OO ′. (However, the point O is a point on the object plane 1) At this time, 2θ is referred to as a convergence angle. Further, the object planes 2L and 2R are inclined by θ with respect to the subject plane 1, respectively. The points OL and OR are the optical axes LL and LR, respectively.
The points CL and CR are the principal points of the imaging optical systems 11L and 11R, respectively (specifically, the principal points on the object side, hereinafter also referred to as viewpoints). Each of the imaging optical systems 11L and 11R has
There are a variable power lens group and a focusing lens group, a driving system for driving them, an encoder for obtaining position information in the optical axis direction, a mechanism system for rotating the imaging system in a plane including the optical axis of the imaging system, A drive system, an encoder for detecting a rotation angle, and the like are provided respectively. The convergence angle control system sets a convergence angle control target value according to the output signal of each encoder so as to obtain an image having a desired aspect ratio, and performs convergence control. Next, a method of image synthesis will be described with reference to FIGS.
This will be described with reference to FIG. As shown in FIG. 10, the imaging magnification of the imaging optical systems 11L and 11R is β, and the object distances (OL-CL and O
The distance between R-CR) is z, and the principal points CL and CR are separated by a distance 2d (referred to as base line length or viewpoint displacement). Then, the viewpoint is taken at a position z 'away from the object plane 1 toward the O' side in the direction of the normal line OO ',
When a virtual image plane (that is, the distance between the viewpoint and the image plane is β′z ′) is taken so that the virtual imaging magnification at that viewpoint becomes β ′, the image plane of the first image sensor 12L LI, image plane RI and virtual image plane ILR on second image sensor 11R
Is shown in FIG. (However, in FIG. 3, β <0, β ′>
The state at the time of 0 is illustrated. In FIG. 3, points AL, BL, CL, and DL are the first points, respectively.
Are the points on the diagonal of the image plane LI of the image sensor 12L, and AR, BR, CR, and DR are the points on the diagonal of the image plane RI of the second image sensor 12R. These points correspond to points AL ', BL', CL ', DL', AR ', BR', and C on the virtual image plane ILR, respectively.
R 'and DR'. As shown in FIG. 3, points EL, FL, ER, and FR are points on the upper and lower sides at the center of the overlap between the image planes LI and RI, and point E 'on the virtual image plane ILR. , F ', and coincide with each other. At this time, when the center of each of the image planes LI, RI, and ILR is defined as the origin, the horizontal direction is defined as the x-axis, and the vertical direction is defined as the y-axis, the coordinates of the image points on each of the image planes LI, RI, and ILR are defined. The image point (x1, y1) on the plane LI corresponds to the image point (x1 ′, y1 ′) represented by the equation on the virtual image plane ILR. [Equation 2] Similarly, the image point (x2, y2) on the image plane RI is the virtual image plane IL.
This corresponds to the image point (x2 ', y2') represented by the equation on R. (Equation 3) Therefore, by performing a geometric transformation process as shown in the equation and synthesizing the plurality of images in the overlapped area, a plurality of image sensors 12L, 12
The image on R is obtained as an image on one virtual image plane ILR. In addition, Japanese Patent Application Laid-Open No. Hei 5-110926 and the like disclose a plurality of images taken in an overlapping manner, respectively.
By performing coordinate transformation processing and combining the transformed images,
2. Description of the Related Art An image synthesizing apparatus for obtaining a high-precision image with corrected distortion has been proposed. Problems to be Solved by the Invention The above-mentioned Japanese Patent Application No. 5-223.
No. 544, when subjects are distributed over a wide range, for example, when the distance to a main subject such as a person is significantly different from the distance to the background (hereinafter referred to as having a distance distribution), the following problem occurs. Spots sometimes occurred. For example, assuming that the object A is located at a distance different from the object distance z shown in FIG. 4, when the geometric transformation processing of the equation is performed, the difference between the object distance z and the distance to the object A is calculated. The position of the object A on the virtual image plane ILR is shifted by the distance. When the distance to a certain point a1 on the object A is zA, and the coordinates of the point on the image sensors 12L and 12R are (x1, y1) and (x2, y2), the position Virtual image plane I of an ideal image point where no displacement occurs
Coordinates on LR (x1 ", y1"), (x2 ", y2")
Are obtained as shown in the following equations. (Equation 4) Therefore, an image shift occurs due to the difference between the object distance z and the distance to each point in the subject, and the shift amounts are expressed by the following equations, respectively. Δx1 = x1 ″ −x1 ′, Δy1 = y1 ″ −y1 ′ Δx2 = x2 ″ −x1 ′, Δy2 = y2 ″ −y2 ′ Therefore, the point a1 of the object A is, for example, as shown in FIG. Δx1, Δx2, y in the x direction with respect to the ideal position a
There has been a problem that a shift may occur in the directions by Δy1 and Δy2, and the image quality of the output image may be degraded. An object of the present invention is to provide an image synthesizing apparatus and a photographing apparatus in which a plurality of images are coordinate-converted and synthesized when coordinate conversion is performed based on distance information to thereby obtain a highly accurate image with little image quality deterioration. I have. According to a first aspect of the present invention, there is provided a photographing apparatus comprising: a first image pickup system for forming a subject image on an image pickup element S1 by an optical system L1 for photographing; and an optical system L2. A subject is obtained from a photographed image G1 obtained by the first image pickup system and from the second image pickup system by a photographing means having a second image pickup system for forming and photographing a subject image on the image pickup device S2. When a part of the photographed image G2 is photographed so as to be overlapped, and the photographed image G1 and the photographed image G2 are combined and converted by the image combining / conversion processing means to obtain one output image, according to set a magnification β'the object distance z ', the relative inclination angle between the shooting direction and the shooting <br/> shading direction of the second imaging system of the first imaging system and 2 [Theta], the A principal point Co is determined on a bisector of the inclination angle 2θ, and the coordinates of the output image are defined by the output image. When the center of the image is the origin, the predetermined direction is the X 'axis, and the direction orthogonal to the X' axis is the Y 'axis, the coordinates (X',
The distance Z from the point on the subject corresponding to the point on the output image indicated by Y ′) to the principal point Co is detected by the distance detection unit, and the coordinates of the photographed image G1 are set to the origin of the center of the photographed image G1. The direction corresponding to the X 'axis is denoted by XL , the direction corresponding to the Y' axis is denoted by YL , and the coordinates of the captured image G2 are
The direction corresponding to the X 'axis is represented by XR , the direction corresponding to the Y' axis is represented by XR , and the direction corresponding to the Y 'axis is represented by YR , and the subject-side principal point of the optical system L1 and the subject-side principal point of the optical system L2 are represented by YR . Distance 2
d, the imaging magnification of the optical system L1 and the optical system L2 beta, when the object distance of the optical system L1 and the optical system L2 and z, and, in the image combining conversion processing unit coordinates (X ', Y ')of
Image data (XL, YL) and (XR, YR)
Is obtained by interpolating from the image data . FIG. 1 is a block diagram showing a main part of a first embodiment of the present invention.
FIG. 2 is an explanatory diagram showing a basic arrangement of the compound-eye photographing means in the present embodiment. In FIG. 1, reference numeral 20 denotes a compound-eye photographing means, which has two photographing systems 10L and 10R, and performs photographing by overlapping a part of the field so as to obtain an output image having an arbitrary aspect ratio. I have. Numeral 40 denotes an image synthesizing and converting means for synthesizing and converting the photographic images from the photographic systems 10L and 10R as described later, and outputting the synthesized image. Reference numeral 1 denotes a virtual object plane assuming an imaging system for photographing the output image. The photographing systems 10L and 10R are equivalent to the left photographing optical system 11L and the right photographing optical system 11R having equivalent specifications, respectively.
Left image sensor 12L, right image sensor 12R
And a zoom lens is used for the imaging optical systems 11L and 11R. LL and LR are respectively an image pickup optical system 11L,
An optical axis 11R (imaging direction), 2L, and 2R are object planes of the imaging systems 10L and 10R, respectively. CL and CR are the principal points (specifically, the principal points on the subject side) of the imaging optical systems 11L and 11R, respectively, and the interval (base line length) between the principal points CL and CR.
Is 2d. The compound eye photographing means 20 includes an image pickup optical system 11
A driving system for driving the L and 11R zoom lens groups and focusing lens groups, an encoder for obtaining positional information of the zoom lens groups and focusing lens groups in the optical axis direction, and an imaging system 10
A mechanical system for rotating the L and 10R in a plane including the optical axes LL and LR, a drive system, and an encoder for detecting a rotation angle and the like are provided. The compound-eye photographing means 20 includes a photographing direction of the photographing system 10L and a photographing direction of the photographing system 10R (in FIG. 2, the optical axes LL,
LR) are arranged so as to have a relative inclination in substantially the same plane (horizontal plane), and a convergence angle (∠CLO ′) when each photographing direction converges on a point O ′ on the image side. C
R) is controlled by setting a control target value such that an output image having a desired aspect ratio is obtained by a convergence angle control system based on output signals from the encoders. Assuming an imaging system for photographing the output image, the virtual object plane 1 has a bisector O− of the convergence angle 2θ.
A plane having O 'as a normal line (where point O is the center point of the object plane 1). The distance from the point O to the virtual principal point Co is z ', and the imaging magnification at this time is β'. Reference numeral 21 denotes an image memory L, and an image pickup system 10L
A storage unit for storing the left image from the camera as digital image data. Similarly, reference numeral 22 denotes an image memory R;
This is a storage unit for storing the right image from R as digital image data. Reference numeral 23 denotes a parameter storage unit, which is a parameter of the compound-eye photographing means 20; a base line length 2d of the imaging systems 10L and 10R, a convergence angle θ, an imaging magnification β, an object distance z, and a virtual imaging system of an output image. The imaging magnification β ′ and the object distance z ′ are stored. Numeral 24 designates a three-dimensional measuring means for measuring the distance to the subject, which is, for example, an optical radar type as shown in FIG. The three-dimensional measuring means 24 includes a pulse laser 241
Is transmitted through the beam splitters 242 and 243 to irradiate a target object (subject), and the reflected light reflected by the target object is deflected by the beam splitter 243 to be guided to a timing circuit 244 for detection. The conversion circuit 245 calculates the distance Z from the principal point Co to the target object from the time difference between the time and the time when the light beam from the pulse laser 241 is deflected by the beam splitter 242 and guided to the timing circuit 244 and detected. I have. Then, the pulse laser 241
By scanning the laser light two-dimensionally, a distance Z (distance information) over the entire subject is obtained. A distance information memory 25 stores the distance information obtained by the three-dimensional measuring means 24. Reference numeral 26 denotes an address generator, which is an address of an image to be output (FIG.
(Coordinates defined by the x and y axes shown in FIG. 3C) are sequentially generated and sent to the coordinate conversion unit 27. In the coordinate conversion unit 27, the parameter storage unit 23
And the distance information memory 25
The two-dimensional coordinates of the left image and the right image are obtained from the address generated by the address generation unit 26 on the basis of the distance data Z stored in. Reference numeral 28 denotes an image data interpolation unit which stores the two-dimensional coordinate data obtained by the coordinate conversion unit 27 in an image memory L2.
1 and the data of the left eye image and the right image stored in the image memory R22 are obtained by interpolation processing and output to the output image memory 29. Each of these elements 21, 22, 23, 24,
25, 26, 27, 28, 29 constitute one element of the image synthesis conversion processing means 40. The control of the entire system including the compound eye photographing means 20 is performed by a system control unit (not shown). In this embodiment, the aspect ratio of the output image and the virtual object plane (subject plane 1) assuming an imaging system for photographing the output image are set in advance, and an output image having the aspect ratio can be obtained. The object distances z 'and the imaging magnification β' of the imaging system are set such that the object planes of the imaging systems 10L and 10R overlap each other and the output image can be obtained. In the processing means 40, the imaging system 1
By synthesizing the captured images from 0L and 10R while performing coordinate conversion processing based on the distance information, it is possible to obtain an image with little image deterioration and a particularly wide field of view. Next, the synthesis conversion process in the coordinate conversion unit 27 will be described. The coordinate converter 27 converts the coordinates in the output image plane sent from the address generator 26 into (x ′,
y ′), the coordinates (xL, y) of the left image and the right image
L) and (xR, yR) are inverse transformations of the equation (θ → −θ,
d → −d), that is, by the above-described equation. Accordingly, the base line length 2d, the convergence angle θ, the imaging magnification β and the object distance z, which are parameters of the compound-eye photographing means 20, and the imaging magnification β ', which is a parameter of the virtual imaging system.
And the object distance z 'is read from the parameter storage unit 23, and the distance information Z is read from the distance information memory 25 in accordance with the coordinates (x', y ') in the output image plane. And the coordinates (xL, yL) of the right image, (x
R, yR) to the image data interpolation unit 28. Next, a second embodiment of the present invention will be described.
FIG. 7 is a block diagram showing the configuration of the present embodiment.
Components that are substantially the same as those in FIG. 1 are denoted by the same reference numerals, and a description thereof is partially omitted. This embodiment is different from the first embodiment shown in FIG. 1 in that distance information is calculated by extracting corresponding points from the overlap region of the input image. Reference numeral 30 denotes a corresponding point extracting unit, which stores an image memory L
21, corresponding points are extracted from the overlap area of the image stored in the image memory R22. Reference numeral 31 denotes a distance information calculation unit that calculates distance information based on the coordinates of the corresponding point on each image and each parameter stored in the parameter storage unit 23. Reference numeral 32 denotes a distance information interpolating unit which obtains distance information of non-overlapping areas by interpolation processing. In the corresponding point extracting section 30, the image memory L21
Is set as a template, a template offset is applied to the template, and the template is moved in parallel to perform template matching with the image data in the image memory R22 (collation between the template and the image data in the image memory R22).
And the coordinates (xR, yR) of the right image corresponding to the coordinates (xL, yL) of the left image are detected. In the distance information calculating section 31, the corresponding point extracting section 3
The coordinates of the left image and the right image (xL,
yL), (xR, yR) and the base line length 2d stored in the parameter storage unit 23, the convergence angle θ, the imaging magnification β, the object distance z, the imaging magnification β ′ of the virtual imaging system of the output image, and the object distance z ′, distance data Z and coordinates (X ′,
Y ′) is calculated and stored in the distance information memory 25.
In this embodiment, the value of 1 / Z is stored in the distance information memory 25 as distance information. The distance information data stored in the distance information memory 25 is data of a partial area in the output image plane, and the data of the shaded area on the virtual image plane shown in FIG. Therefore, the corresponding points cannot be detected by the corresponding point extraction unit 30 and the distance information is not obtained. The distance information interpolator 32 interpolates the distance information of such an area from the distance information of the overlap portion calculated in advance. The area in which the distance information is not obtained as shown in FIG.
The left part outside the overlap part (the area shown by OL in the figure), the right part outside the overlap part (the area shown by OR in the figure, the left part inside the overlap part (I in the figure)
L), and a right portion (a region indicated by IR in the drawing) inside the overlap portion. The interpolation by the distance information interpolating unit 32 is based on the distance information data of the right overlapped part in the OL area.
In the area, the determination is performed from the distance information data of the left overlap portion. In this case, in the IL area, it is considered that the distance information corresponds to the background portion. Therefore, interpolation is performed from the distance information data of the distant overlap portion of the distance information (ie, the distance information data of the right overlap portion). In the IR area, the distance information data of the farther overlap portion of the distance information (that is, the distance information data of the left overlap portion)
Is interpolated. FIG. 9 is a diagram showing a state of the interpolation processing.
9 shows a cross section taken along a broken line PP 'in FIG. In FIG. 9, a solid line portion is data for which distance information has been calculated in advance, and a dotted line portion is an interpolation unit. In addition, as an interpolation method, the distance information of the nearest overlap portion in the horizontal direction may be assigned as it is. At this time, if the distance information is significantly different at the boundary of the overlap portion, it is preferable to assign the average value or the median of the distance information at the boundary (that is, in the y direction). Further, the distribution of the distance Z may be approximated by a function such as a spline function from the distance information data of the overlap portion, and interpolation may be performed. In this case, although the interpolation process of the distance information becomes complicated, the distance information can be smoothly interpolated, and a better output image can be obtained. In the present embodiment, the coordinate conversion is performed by the coordinate conversion unit 27 based on the equation using the distance information thus obtained. The imaging magnification β 'and the object distance z' are arbitrary coefficients. For example, the imaging magnification β 'is -1 and the object distance z' is
If is assumed to be 1, It becomes. Further, the image synthesizing apparatus according to the present invention obtains each parameter of the above-described compound-eye photographing means from the encoder or the like of the compound-eye photographing means at any time, performs a synthesis conversion process, and obtains a photographed image which has been photographed in advance. An image to be subjected to a synthesis conversion process when reproducing the photographed image and each parameter from a storage medium storing the image together with each parameter; and photographing each parameter by a predetermined method, and synthesizing according to the method. Conversion processing may be performed. According to the present invention, when a plurality of images are coordinate-converted and synthesized, coordinate conversion based on distance information is performed, so that an image with little deterioration in image quality and a high-precision image can be obtained. A synthesizing device and a photographing device can be achieved.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram of a main part of a first embodiment of the present invention. FIG. 2 is an explanatory diagram of a compound-eye photographing unit according to the first embodiment of the present invention. FIG. FIG. 4 is an explanatory diagram of a virtual image plane. FIG. 4 is an explanatory diagram of a case where a subject has a wide distance distribution in a conventional compound-eye photographing unit. FIG. 5 is an explanatory diagram of a positional shift of an image generated in a conventional image synthesis conversion process. FIG. 7 is an explanatory diagram of a three-dimensional measuring unit. FIG. 7 is a block diagram of a main part of a second embodiment of the present invention. FIG. 8 is an explanatory diagram of interpolation processing of distance information on a virtual image plane. FIG. 10 is an explanatory view of a conventional compound eye photographing means. [Description of References] LL, LR Optical axis 10L, 10R Imaging system 11L, 11R Imaging optical system 12L, 12R Image sensor 2L, 2R Object surface 241 Pulse laser 242, 243 Beam splitter 244 timing circuit 2 5 Conversion circuit 20 Compound eye photographing means 21, 22 Image memory 23 Parameter storage unit 24 Three-dimensional measuring means 25 Distance information memory 26 Address generation unit 27 Coordinate conversion unit 28 Image data interpolation unit 29 Output memory 30 Corresponding point extraction unit 31 Distance information Calculation unit 32 Distance information interpolation unit

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-8-18857 (JP, A) JP-A-7-79379 (JP, A) JP-A-7-135605 (JP, A) 344422 (JP, A) JP-A-5-110926 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) H04N 5/225-5/232 H04N 5/262-5/265

Claims (1)

(57) [Claim 1] A first imaging system for forming a subject image on the image sensor S1 by the optical system L1 and photographing the same, and a subject image on the image sensor S2 by the optical system L2. The second to form and shoot
The subject is moved by the photographing means having the first imaging system.
The photographed image G1 obtained by the image pickup system and the photographed image G2 obtained by the second image pickup system are photographed so as to partially overlap each other, and the photographed image G1 and the photographed image G2 are obtained by the image synthesis conversion processing means. When a single output image is obtained by performing composite conversion processing on the output image, a magnification β ′ and an object distance z ′ of the output image are set, and the imaging direction of the first imaging system and the imaging direction of the second imaging system are set. The inclination angle relative to the photographing direction is 2θ, the principal point Co is determined on the bisector of the inclination angle 2θ, the coordinates of the output image are set to the origin of the center of the output image, and the predetermined direction is the X ′ axis. When the direction orthogonal to the X 'axis is the Y' axis, the distance Z from the point on the subject corresponding to the point on the output image indicated by the coordinates (X ', Y') to the principal point Co is The coordinates of the photographed image G1 are detected by the distance detection unit and correspond to the X 'axis with the center of the photographed image G1 as the origin. Directions XL, 'indicates the direction corresponding to the axis in YL, the coordinates of the photographed image G2, the center of the photographed image G2 as the origin X' said Y corresponds to the direction corresponding to the axis XR, to the Y 'axis directions are shown in YR, 2d the distance between the object side principal point of the object side principal point and the optical system L2 of the optical system L1, the optical system L1 and the optical system L
The second imaging magnification beta, when is z, the distance of the object optical system L1 and the optical system L2, in the image combining conversion processing unit coordinates (X ', Y') image
The data is (XL, YL) and (XR, YR)
An image capturing apparatus characterized in that the image data is obtained by interpolating from the image data .