JP2002063572A - Image compositing system having camera parameter detecting means - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image compositing system in which processing is simple and is performed at a low cost and which has a camera parameter detecting means in which the delay, etc., of image processing does not occur in an image compositing system by a chromakey means. SOLUTION: This system is provided with a chromakey back whose color is the same as that of its front surface and on which a plurality of markers having different various densities are provided, a studio camera on a camera mounting tool, a video system for controlling studio camera, a mechanical detector for detecting and recording a camera parameter based on a camera angle at the time when a part of a plurality of marker images is sequentially imaged at a prescribed position of the image pickup picture of the studio camera, a CG generating device for generating a background image on the basis of the camera parameter output of the device and a compositing device for combining the respective output signals of the CG generating device and the mechanical detector. The mechanical detector can estimate the camera parameter showing a camera position on the basis of the detected and recorded camera angle.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to background image generation in image synthesis in a virtual studio, and to a means for estimating a camera position using a marker provided on a chroma keyback.

[0002]

2. Description of the Related Art In a virtual studio of a television or the like, an image region of a subject such as a person is extracted from a photographed image of the subject by a chroma key means with a chroma key back (blue back) as a background, and the background image is subjected to CG. 2. Description of the Related Art An image synthesizing system for generating and synthesizing (computer graphics) is used. In order for the background image to adapt in real time without any discomfort in response to changes due to camera work, the camera position (x, y, z) and camera angle (pan,
The background image needs to be generated based on camera parameters indicating tilt, roll) and focal length (zoom) (hereinafter simply referred to as camera position, camera angle, and focal length). For example, a mechanical detection device detects a camera angle by providing a sensor such as a rotary encoder on a camera installation tool such as a tripod of a studio camera, and detects a focal length by providing a lens sensor on a lens. In addition, the present applicant has a method of estimating a camera parameter by detecting means for photographing a subject against a chroma key-back in which a lattice pattern is drawn by applying dark / light blue separately and detecting the lattice pattern image in a pattern. To
This is proposed in Japanese Patent Application No. 11-129426, "Method for Estimating Camera Parameters in Image Synthesis".

[0003]

The detection means has advantages / disadvantages. The mechanical detection device can easily and quickly detect a camera angle and a focal length, but the camera position can be detected by installing a rail on a studio. The camera is moved by a trolley, and the rotation of the wheel is detected. There is a restriction on the movement position and an accumulation of errors on a large scale. The pattern recognition means can detect all camera parameters, but has disadvantages such as complicated processing, high cost, and delay in image processing. Here, in connection with the above, the studio camera is described from the viewpoint of shooting while moving the camera position. However, in practice, photography is often performed at a fixed camera position, and even in a system using a plurality of studio cameras, usually only one set of images is taken while moving the position, and the rest is a fixed camera position. Even if the detecting means is provided, the camera position detecting means such as the installation of the rail is not provided. Even if the camera position is constant, there is no need for real time, but it is necessary to detect the camera position once at the beginning. Conventionally, a camera position based on a three-dimensional coordinate system of a virtual studio is measured by a measure or the like at the time of setup or the like. However, there were frequent changes in camera position due to standing position adjustment during rehearsals, and measurement with a measure or the like was troublesome and had many errors. Furthermore, due to limited staffing and recording time, it was difficult to exchange the background image between the virtual studio and the control room, or to correct the background image.

The present invention has been made in view of such a background, and an image synthesizing system in which a camera position of a studio camera is constant or sometimes changes frequently.
The camera angle is easily and quickly detected using a mechanical detection device. When changing the camera position, the camera parameters that indicate the camera position can be estimated simply by capturing each of the markers provided on the chroma key back at the specified positions on the screen. It is an object of the present invention to provide an image composition system having a camera parameter detecting means capable of eliminating the above-mentioned disadvantages of measuring means, measuring rails, laying rails or detecting patterns, and easily aligning a background image. And

[0005]

Means for Solving the Problems The present inventor has solved the problem by the following means as a result of intensive studies in view of the above. (1) In an image synthesizing system using chroma key means, a chroma key back in which a plurality of markers of the same color as the surface and having different densities / shades are arranged, a studio camera on a camera installation tool, and a video device for controlling the studio camera ,
A mechanical detection device that detects and records a camera parameter based on a camera angle when a part of the plurality of marker images is sequentially captured at a predetermined position on an imaging screen of the studio camera, based on a camera parameter output of the device. A CG generation device that generates a background image; and a synthesis device that synthesizes respective output signals of the CG generation device and the video device.
An image synthesizing system comprising a camera parameter detecting means, wherein the mechanical detecting device can estimate a camera parameter indicating a camera position based on the detected and recorded camera angle.

(2) The image synthesizing apparatus having the camera parameter detecting means according to the above (1), wherein the markers arranged on the chroma key back are constituted by at least two or more markers. system. (3) A camera indicating the camera position based on the detected and recorded camera angle when the mechanical detection device sequentially captures the center of each of the plurality of marker images at the center position of the imaging screen of the studio camera. An image synthesizing system comprising the camera parameter detecting means according to the above (1) or (2), wherein a parameter can be estimated. (4) The mechanical detection device includes a switch at a predetermined position on an imaging screen of the studio camera, the switch being capable of setting a camera angle when sequentially capturing a part of the plurality of marker images. An image composition system comprising the camera parameter detection means according to any one of the above items (1) to (3). (5) The mechanical detection device includes a camera parameter adjuster that can add and adjust an offset value for each of the camera parameters of the output of the mechanical detection device. An image composition system comprising the camera parameter detection means according to any one of the above. (6) The image synthesizing system according to any one of (1) to (5), wherein the mechanical detection device is mounted on the camera installation tool.

[0007]

Embodiments of the present invention will be described with reference to the accompanying drawings. FIG. 1 is a configuration diagram of a main part of an image composition system according to an embodiment of the present invention. FIG. 2 is a conceptual diagram of camera position estimation using a marker according to the embodiment. FIG. 3 is an explanatory diagram of a pinhole camera model of the embodiment. FIG. 4 is a diagram in which the image plane of FIG. 3 is brought to a focal plane. FIG. 5 is an explanatory diagram of the movement of the coordinate system of the embodiment. In the figure, 1 is a chroma key back,
N is a marker, 3 is a subject, 4 is a studio camera, 5
Is a lens sensor, 6 is a camera angle sensor, 7 is a camera installation tool, 8 is a mechanical detection device, 9 is a video device, 10 is CG
A generation device, 11 is a synthesis device, 12 is a terminal, 13 is a lens, and 14 is a screen.

In FIG. 1, a chroma key back 1 in which a plurality of markers of the same color as the surface and having different darkness / lightness is provided.
A studio camera 4 on a camera installation tool 7, a video device 9 for controlling the studio camera 4, and the plurality of markers at predetermined positions on an imaging screen of the studio camera 4.
a mechanical detection device 8 for detecting and recording camera parameters based on a camera angle when each central portion of the n images is sequentially imaged, and a CG generation device 10 for generating a background image based on the camera parameter output of the device. And a synthesizing device for synthesizing output signals of the CG generating device 10 and the video device 9, wherein the mechanical detection device estimates a camera parameter indicating a camera position based on the detected and recorded camera angle. Further, this is an image synthesizing system having a camera parameter detecting means for photographing with the photographing position of the studio camera 4 being fixed or changed with time. here,
A feature of the present invention resides in that a plurality of markers to n having the same color but different in density / lightness and having an appropriate mutual interval are provided at a set position on the surface of the chroma key back 1 in the virtual studio. Marker ~ n
Is based on, for example, pasting a tape or the like in a cross shape so that the center of the virtual studio can be identified, and these three-dimensional coordinates based on the space of the virtual studio are known. The camera parameter indicating the camera position is based on a camera angle detected by the mechanical detection device 8 including an arithmetic circuit in a state where the marker image displayed by the density difference is captured at a predetermined position on an imaging screen. presume.

In FIG. 1 and FIG. 2, first, for example, at a position where the studio camera 4 is in a horizontal position and an image is taken in a direction near the center of the chroma key back 1, a reset button (not shown) of the camera angle sensor 6 is used. Reset the sensor value. After setting this as the zero point of the sensor, one marker image of the markers to n is set at the center position of the imaging screen 14 of the studio camera 4 based on the optical axis of the lens 13 of the screen 14 by the studio camera 4. Of the center of the image. In FIG. 1, when the central part of the marker image is imaged, the detected camera angle is recorded in the memory at a timing such as when a marker capture switch button (not shown) provided on the mechanical detection device 8 is pressed. Similarly, images are sequentially taken for each of the markers to n, and the respective camera angles are recorded. In this case, the focal length of the lens 13 may be any constant value. As described above, the camera position among the camera parameters output from the mechanical detection device 8 is calculated by the camera position estimation algorithm as follows based on the detected and recorded camera angle.

[0010] 1. Pinhole Camera and Center Projection FIG. 3 shows a pinhole camera. Distance f from image plane I
And place a plane F parallel to the image plane I, and make a pinhole at point C on that plane. Light coming from the object passes through a pinhole, ie, point C, and forms an image on an image plane. The point of the object, the pinhole, and the image on the image plane are on a straight line. Such a projection is a central projection (perspectiv).
e projection). The point C is located at the lens center (focal point) or the focal point (foc).
us), the plane F is referred to as a focal plane, and the distance f from the point C (the center of the lens) to the image plane is referred to as a focal length. A line passing through the point C and perpendicular to the image plane is called an optical axis, and an intersection c between the optical axis and the image plane is defined as an image center (prin).
chip point). Of course, the optical axis is also orthogonal to the focal plane. First, a coordinate system that describes the equation of central projection is defined. In the image coordinate system, the origin is the image center c, and the x-axis and the y-axis are respectively aligned with the arrangement axes of the camera elements. When the optical axis is the third axis, the order is determined so as to be a right-handed system. Further, in the coordinate system of the three-dimensional space, the focal point C is set as the origin, the optical axis is set as the Z axis, and the X axis and the Y axis are set in directions opposite to the x axis and the y axis, respectively. In this way, the right-handed system is maintained. The right-handed system is convenient when using a rotation matrix. Using the above coordinate system, three points in a three-dimensional space
The following relationship holds between the dimensional coordinates and the two-dimensional image.

[0011]

(Equation 1)

As shown in FIG. 4, the above relationship holds even if the image plane behind the focal plane is brought forward of the focal plane and inverted.

2. Projection matrix and external variables Non-linear equation (1) can be linearly expressed as follows.

[0014]

(Equation 2)

[0015]

(Equation 3)

[0016]

(Equation 4)

3. When the derivation equation (9) is written by an element,

[0018]

(Equation 5)

Here, R = rij (i, j = 1,2,3) ri = (ri1, ri2, ri3) T (i = 1,2,2)
3). For any vectors a and b, the inner product of the vectors is represented as (a, b). Substituting this into equation (3) gives

[0020]

(Equation 6)

That is, two equations are obtained from one marker point. The unknown to be determined is tx, ty, and tz.
Therefore, it is sufficient that at least two three-dimensional positions can observe a known marker point. Now, the first marker point M = (X1,
Y, Z1) The rotation matrix when observing T around the image is R, and the second marker point M ′ = (X2, Y2, Z2)
Let R 'be the rotation matrix when T is observed around the center of the image.
Putting this together, we can write:

[0022]

(Equation 7)

There are four equations for the unknowns tx, ty and tz, which can be solved by the least squares method. In general, solving the equation Aχ = b in the sense of least squares for χ is to minimize ‖Aχ−b‖2. This is simply to solve the normal equation ATAχ = ATb.

4. Algorithm n points of marker M1 = (X1, Y
1, Z1) T,..., Mn = (Xn, Yn, Zn)
T is set, and these three-dimensional positions are assumed to be known. Each marker point is imaged at the center in the viewfinder of the camera, and each rotation matrix from the sensor at that time is represented by R1 = (rij (1)),..., Rn = (rij
(N)) (ij = 1, 2, 3). At this time, the focal length may be arbitrary. Using the rotation matrix from the sensor at the time of observing each marker point, the following equations are solved by the least-squares method to obtain camera positions tx, ty, and tz.

[0025]

(Equation 8)

[0026]

Embodiment An embodiment of the present invention will be described with reference to FIG. Chroma key back 1 in virtual studio
Markers -n of the same color but different in darkness / lightness are provided at set positions having appropriate mutual intervals on the surface of. A studio camera 4 installed on an installation tool 7 such as a tripod captures a subject 3 as a foreground with a chroma key back 1 as a background, and outputs an image signal via a video device 9 having a camera control function and has a chroma key unit. The image area of the subject 3 is extracted by input to the synthesizing device 11. In the above-described extraction of the image area, the markers to n indicate the chroma key back 1.
Since it is the same color as, extraction is possible. Mechanical detection device 8
Are the lens sensor 5 provided on the lens 13 of the studio camera 4 and the camera angle sensor 6 provided on the camera
Are connected, and the camera angle and the focal length corresponding to the camera work are detected in real time. The mechanical detection device 8 including the arithmetic circuit outputs the camera parameters indicating the camera angle, the focal length, and the camera position estimated and calculated on the basis of the data of the camera angle as described later, and performs CG.
The CG image is input to the generation device 10, and the CG image generation device 10 generates and outputs a CG image as a background of the extracted three subject images based on the camera parameters.

The background CG image generated and output is input to the synthesizing device 11 and synthesized with the extracted subject image as a background image adapted to the motion of the camera work. Output to terminal 12. That is, for example, when setting up the apparatus system or changing the camera position, for example, the lens 13 of the studio camera 4 is used.
An image of the central part of one of the markers to n is captured at the center position of the imaging screen of the studio camera 4 based on the optical axis of (1). In the above state, the mechanical detection device 8 records the detected camera angle in the memory at a timing such as when a marker capture switch button (not shown) provided on the mechanical detection device 8 is pressed. Similarly, each of the markers to n is sequentially imaged, and the detected camera angle is recorded. The camera parameters indicating the camera position are calculated by the above-described camera position estimation algorithm using the recorded camera angle. Further, since the mechanical detection device 8 includes a camera parameter adjuster (not shown), each of the camera parameter values output from the mechanical detection device 8 can be output with an offset value added and adjusted. Further, if the mechanical detection device 8 is used by attaching it to a leg of the camera installation tool 7 such as a tripod or a pan stick for controlling the camera angle with a fastener or the like, the cameraman can operate the studio camera 4 while operating the studio camera 4. The camera parameter adjuster of the type detection device 8 can be operated. Here, the marker may be a chroma key back 1 having a large area, or at least two or more markers. For example, even with two markers, the camera position can be estimated, and an accurate camera position can be estimated by adding and adjusting the offset value by the camera parameter adjuster.

[0028]

According to the present invention, the following effects can be obtained. 1. According to the invention of claim 1 of the present application, since the mechanical detection device can estimate the camera parameter indicating the camera position based on the detected and recorded camera angle, the camera position of the studio camera is constant or more frequently. Even when there is a change, the camera angle and focal length can be detected by a quick and simple mechanical detection device, and the camera position can be detected simply by capturing the respective marker images at predetermined positions on the screen. In addition, it is possible to easily, accurately and at low cost detect camera parameters and synthesize images by eliminating the disadvantages of the above-mentioned means for measuring the measurement, which is troublesome and has a lot of errors, the above-mentioned rail laying, or the pattern recognition means.

2. According to the invention of claim 2 of the present application, since the camera position can be estimated only by configuring at least two or more markers disposed on the chroma key back, even a chroma key back having a large area is simple. Thus, accurate image synthesis can be performed.

3. According to the invention of claim 3 of the present application, a camera that detects and records when the mechanical detection device sequentially captures the center of each of the plurality of marker images at the center position of the imaging screen of the studio camera. Since camera parameters indicating the camera position can be estimated based on the angle,
Image composition can be performed with a more accurate camera position.

4. According to the invention of claim 4 of the present application, the mechanical detection device is provided with a switch that can set a camera angle when sequentially capturing a part of the plurality of marker images at a predetermined position on an imaging screen of the studio camera. Since the camera is provided, the camera angle can be easily and accurately reset even when the camera is moved up and down, left and right, or returned after diverting the camera to another program or the like.

5. According to the invention of claim 5 of the present application, the mechanical detection device is provided with a camera parameter adjuster that can additionally adjust an offset value of the output of the mechanical detection device with respect to each of the camera parameters. Even in staff personnel and recording time, delicate alignment with the background image between the photographer and the adjustment room,
Or, exchange of corrections can be easily performed.

6. According to the invention of claim 6 of the present application, since the mechanical detection device is mounted on the camera installation tool, the cameraman can operate the studio camera at hand, and at the same time, operate the camera of the mechanical detection device at hand. By operating the parameter adjuster, the position of the background CG image can be adjusted.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a main part of an image composition system according to an embodiment of the present invention.

FIG. 2 is a conceptual diagram of camera position estimation using a marker according to the embodiment.

FIG. 3 is an explanatory diagram of a pinhole camera model of the embodiment.

FIG. 4 shows the image plane of FIG. 3 before the focal point.

FIG. 5 is an explanatory diagram of movement of a coordinate system according to the embodiment.

[Explanation of symbols]

1: Chroma key back ~ n: Marker 3: Object 4: Studio camera 5: Lens sensor 6: Camera angle sensor 7: Camera installation tool 8: Mechanical detection device 9: Video device 10: CG generation device 11: Synthesizing device 12: Terminal 13: Lens 14: Screen

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference)

Claims (6)

[Claims] 1. An image synthesizing system using chroma key means, a chroma key back in which a plurality of markers of the same color as the surface and having different densities are provided, a studio camera on a camera installation tool, and an image for controlling the studio camera. A device, a mechanical detection device that detects and records a camera parameter based on a camera angle when a part of the plurality of marker images is sequentially captured at a predetermined position on an imaging screen of the studio camera, and a camera parameter of the device. A CG generation device for generating a background image based on the output, and a synthesis device for synthesizing output signals of the CG generation device and the video device, wherein the mechanical detection device is configured to detect a background image based on the detected and recorded camera angle. An image synthesizing system having a camera parameter detecting means, wherein a camera parameter indicating a camera position can be estimated. Tem. 2. The image synthesizing system according to claim 1, wherein the number of markers provided on the chroma key back is at least two or more. 3. The camera position based on the detected and recorded camera angle when the mechanical detection device sequentially captures the center of each of the plurality of marker images at the center position of the imaging screen of the studio camera. An image composition system having a camera parameter detection unit according to claim 1, wherein a camera parameter shown can be estimated. 4. The system according to claim 1, wherein the mechanical detection device includes a switch at a predetermined position on an imaging screen of the studio camera, the switch being capable of setting a camera angle when sequentially capturing a part of the plurality of marker images. An image synthesizing system comprising the camera parameter detecting means according to claim 1. 5. The mechanical detection device according to claim 1, wherein the mechanical detection device includes a camera parameter adjuster capable of additionally adjusting an offset value for each of the camera parameters of the output of the mechanical detection device. An image composition system comprising the camera parameter detection means according to any one of the preceding claims. 6. The image synthesizing system according to claim 1, wherein the mechanical detection device is mounted on the camera installation tool.