JP2024064155A - Video synthesizing frame generation apparatus, video synthesizing apparatus, and program - Google Patents

Abstract

To generate information by which an orientation and an angle of view of a camera image can be easily recognized in a perspective manner, the camera image being obtained by imaging the real world.SOLUTION: A camera parameter receiving unit 11 of a video synthesizing frame generation apparatus 2 converts a zoom value and a focus value included in a camera parameter into a shooting angle θ and a focal distance f. A camera parameter interpolation unit 12 interpolates data of the camera parameter so that a data rate of the camera parameter is coincident with a frame rate of a camera video. A coordinate conversion unit 13 converts, for each display element, world coordinate value of the display element to an image coordinate value, using the camera parameter. A synthesizing frame generation unit 14 arranges, for each display element, CG of the display element in a position of the image coordinate value, determines an imaging region S in a map image including a position of a camera, generates a PinP image representing the imaging region S in the map image, and arranges the PinP image in a position of a predetermined image coordinate value to generate a synthesizing frame.SELECTED DRAWING: Figure 3

Description

The present invention relates to a frame generation device for image synthesis that generates frames used when synthesizing camera images, an image synthesis device that synthesizes frames into camera images, and a program.

Conventionally, there is known a device that uses a sensor to detect the shooting direction of a camera and synthesizes a virtual image using a computer in accordance with the camera movement (see, for example, Patent Document 1).

When combining a real image captured by a camera with a virtual image captured by a virtual camera, this device maintains a constant display positional relationship between the two images based on angle information that reflects changes in the camera's shooting direction due to manual operation.

There is also a system known that displays map information related to a television program on the program image (see, for example, Patent Document 2).

This system transmits location information to a map server in response to a viewer's remote control operation for a program in which location information is embedded, receives map information related to the location information from the map server, and displays the map information on the television screen.

JP 2004-227332 A JP 2008-295001 A

The device of the aforementioned Patent Document 1 can provide the cameraman and viewers with information about the direction in which the camera is shooting by, for example, synthesizing an image of the landmark's name or other information with the camera image that includes the landmark. In other words, the cameraman and viewers can recognize the direction in which the camera is facing from information such as the landmark's name displayed in the camera image.

However, with this device, it is not possible to recognize the direction and angle of view of the camera image from a bird's-eye view in the real world where the camera image was captured. It is particularly difficult for cameramen and viewers who are unfamiliar with the area to recognize the direction and angle of view.

The system in Patent Document 2 mentioned above obtains map information related to location information embedded in a program and presents a map related to the program to the viewer.

However, the map information used in this system does not indicate the direction and angle of view of the camera images for the program, so the map information cannot indicate the camera's shooting area.

In such a device that performs image synthesis, it is desirable to present not only the synthesized image with names of landmarks and the like, but also information about the shooting together with the camera image to the cameraman operating the camera that captures the real world and the viewers of the camera image. In particular, it is necessary to present further information about the shooting to cameramen and viewers who are unfamiliar with the location where the camera image was shot.

The present invention has been made to solve the above problems, and its purpose is to provide an image synthesis frame generation device, an image synthesis device, and a program that generate information that allows the direction and angle of view of a camera image captured in the real world to be easily recognized from a bird's-eye view.

In order to solve the above problem, the image synthesis frame generating device of claim 1 generates a frame including information related to the shooting of a camera image as a synthesis frame to be used when synthesizing the camera image, and includes a coordinate information storage unit in which, for each display element of the information related to the shooting of the camera image, the world coordinate values of the display element are stored, a map image storage unit in which a map image including the position of the camera that shoots the camera image is stored, a CG storage unit in which CG for each of the display elements is stored, a camera parameter receiving unit that receives the camera parameters of the camera, sets the zoom value and focus value included in the camera parameters to the shooting angle of view and the focus distance, respectively, and outputs the camera parameters including the shooting angle of view and the focus distance, and an interpolation unit that outputs the camera parameters so that the data rate of the camera parameters output by the camera parameter receiving unit matches the frame rate of the camera image. a camera parameter interpolation unit that converts the world coordinate values from the coordinate information storage unit to image coordinate values based on the camera parameters interpolated by the camera parameter interpolation unit for each of the display elements; a synthesis frame generation unit that reads the map image from the map image storage unit, extracts a pan angle and a shooting angle of view from the camera parameters interpolated by the camera parameter interpolation unit, determines a shooting area based on the position of the camera in the map image based on the pan angle and the shooting angle of view, generates a PinP (Picture in Picture) image that shows the shooting area on the map image, reads the CG from the CG storage unit for each of the display elements, arranges the CG at the location of the image coordinate values converted by the coordinate conversion unit, and arranges the PinP image at the location of the image coordinate values that have been set in advance, thereby generating the synthesis frame.

The video synthesis frame generating device of claim 2 is the video synthesis frame generating device of claim 1, further comprising a new camera parameter receiving unit replacing the camera parameter receiving unit, and further comprising a conversion information storage unit in which a shooting angle of view corresponding to the zoom value and a focus distance corresponding to the focus value are stored, the new camera parameter receiving unit receives the camera parameters of the camera, extracts the zoom value and the focus value from the camera parameters, reads out the shooting angle of view corresponding to the zoom value and the focus distance corresponding to the focus value from the conversion information storage unit, and outputs the camera parameters including the shooting angle of view and the focus distance, and the camera parameter interpolation unit interpolates the camera parameters so that the data rate of the camera parameters output by the new camera parameter receiving unit matches the frame rate of the camera image.

により姿勢Ｒを算出し、前記表示要素毎に、前記フォーカス距離ｆ、前記姿勢Ｒ、前記カメラ位置ｔ_w及び前記ピクセル数の半値μ₀，ν₀を用いて、以下の式：

により、前記世界座標値（Ｘ_w，Ｙ_w，Ｚ_w）を前記画像座標値（ｘ_i，ｙ_i）に変換する、ことを特徴とする。 Further, in the video synthesis frame generation device of claim 3, in the video synthesis frame generation device of claim 1 or 2, the coordinate conversion unit extracts the pan angle, tilt angle, roll angle, the focus distance and camera position from the camera parameters, and calculates the following equation using the pan angle α, the tilt angle δ and the roll angle φ, where α is the pan angle, δ is the tilt angle, φ is the roll angle, f is the focus distance, t _w = [t _wx , t _wy , t _wz ], (X _w , Y _w , Z _w ), (x _i , y _i ), and half the number of pixels in the horizontal and vertical directions are μ ₀ and ν ₀ :

The orientation R is calculated by the following formula, using the focus distance f, the orientation R, the camera position t _w and the half values μ ₀ and ν ₀ of the number of pixels for each display element:

The world coordinate values (X _w , Y _w , Z _w ) are converted into the image coordinate values (x _i , y _i ) by:

The image synthesis frame generating device of claim 4 is the image synthesis frame generating device of claim 1 or 2, characterized in that the synthesis frame generating unit extracts the pan angle and the shooting angle of view from the camera parameters, determines the camera orientation from the pan angle, determines the angle of the shooting area based on the camera position from the shooting angle of view, and determines the shooting area in the map image from the camera orientation and the angle of the shooting area.

The image synthesis frame generating device of claim 5 is the image synthesis frame generating device of claim 1 or 2, characterized in that the CG storage unit stores, for each display element, a plurality of CGs corresponding to the shooting angle of view or the focus distance of the display element, and the synthesis frame generating unit extracts the shooting angle of view or the focus distance from the camera parameters, reads out the CG corresponding to the shooting angle of view or the focus distance from the CG storage unit for each display element, places the CG at the location of the image coordinate values, and places the PinP image at the preset location, thereby generating the synthesis frame.

The video synthesis frame generating device of claim 6 is the video synthesis frame generating device of claim 1 or 2, characterized in that the synthesis frame generating unit extracts the shooting angle of view or the focus distance from the camera parameters, changes the scale of the PinP image based on the shooting angle of view or the focus distance, and generates the synthesis frame by placing the PinP image with the changed scale in the preset location.

Furthermore, the image synthesizing device of claim 7 generates a frame including information regarding the shooting of a camera image as a synthesis frame, and synthesizes the synthesis frame with the frame of the camera image to generate a synthetic image, and the image synthesizing device includes a coordinate information storage unit in which, for each display element of the information regarding the shooting of the camera image, the world coordinate value of the display element is stored, a map image storage unit in which a map image including the position of the camera that shoots the camera image is stored, a CG storage unit in which CG for each display element is stored, a camera parameter receiving unit that receives the camera parameters of the camera, sets the zoom value and focus value included in the camera parameters as the shooting angle of view and the focus distance, respectively, and outputs the camera parameters including the shooting angle of view and the focus distance, a camera parameter interpolation unit that interpolates the camera parameters so that the data rate of the camera parameters output by the camera parameter receiving unit matches the frame rate of the camera image, and a camera parameter interpolation unit that reads out the world coordinate value from the coordinate information storage unit for each display element, and calculates the world coordinate value based on the camera parameters interpolated by the camera parameter interpolation unit. a coordinate conversion unit that converts the world coordinate values into image coordinate values based on the world coordinates; a synthesis frame generation unit that reads the map image from the map image storage unit, extracts the pan angle and the shooting angle of view from the camera parameters interpolated by the camera parameter interpolation unit, obtains a shooting area based on the position of the camera in the map image based on the pan angle and the shooting angle of view, generates a PinP (Picture in Picture) image that shows the shooting area on the map image, reads the CG from the CG storage unit for each display element, arranges the CG at the location of the image coordinate values converted by the coordinate conversion unit, and arranges the PinP image at the location of a preset image coordinate value to generate the synthesis frame; a video delay unit that delays the camera video frame so that the camera video frame is synchronized with the synthesis frame generated by the synthesis frame generation unit; and a frame synthesis unit that synthesizes the synthesis frame generated by the synthesis frame generation unit with the camera video frame delayed by the video delay unit, and outputs the synthesized video.

Furthermore, the program of claim 8 includes a computer constituting an image synthesis frame generating device that generates a frame including information regarding the shooting of a camera image as a synthesis frame to be used when synthesizing the camera image, the computer comprising: a coordinate information storage unit in which, for each display element of the information regarding the shooting of the camera image, the world coordinate values of the display element are stored; a map image storage unit in which a map image including the position of the camera that shoots the camera image is stored; a CG storage unit in which CG for each display element is stored; a camera parameter receiving unit that receives the camera parameters of the camera, sets the zoom value and focus value included in the camera parameters as the shooting angle of view and the focus distance, respectively, and outputs the camera parameters including the shooting angle of view and the focus distance; and a camera parameter output unit that interpolates the camera parameters so that the data rate of the camera parameters output by the camera parameter receiving unit matches the frame rate of the camera image. a data interpolation unit, a coordinate conversion unit that reads the world coordinate values from the coordinate information storage unit for each of the display elements, and converts the world coordinate values into image coordinate values based on the camera parameters interpolated by the camera parameter interpolation unit, and a synthesis frame generation unit that reads the map image from the map image storage unit, extracts a pan angle and the shooting angle of view from the camera parameters interpolated by the camera parameter interpolation unit, determines a shooting area based on the position of the camera in the map image based on the pan angle and the shooting angle of view, generates a PinP (Picture in Picture) image that shows the shooting area on the map image, reads the CG from the CG storage unit for each of the display elements, arranges the CG at the location of the image coordinate values converted by the coordinate conversion unit, and arranges the PinP image at the location of the image coordinate values that have been set in advance, thereby functioning as a synthesis frame generation unit.

The program of claim 9 further comprises a computer constituting an image synthesizing device that generates a composite image by generating a frame including information regarding the shooting of a camera image as a synthesis frame and synthesizing the synthesis frame with the frame of the camera image, the computer comprising: a coordinate information storage unit in which, for each display element of the information regarding the shooting of the camera image, the world coordinate values of the display element are stored; a map image storage unit in which a map image including the position of the camera that shoots the camera image is stored; a CG storage unit in which CG for each display element is stored; a camera parameter receiving unit that receives the camera parameters of the camera, sets the zoom value and focus value included in the camera parameters as the shooting angle of view and the focus distance, respectively, and outputs the camera parameters including the shooting angle of view and the focus distance; a camera parameter interpolation unit that interpolates the camera parameters so that the data rate of the camera parameters output by the camera parameter receiving unit matches the frame rate of the camera image; and a computer program that reads out the world coordinate values from the coordinate information storage unit for each display element, and outputs the world coordinate values based on the camera parameters interpolated by the camera parameter interpolation unit. a coordinate conversion unit that converts the world coordinate values into image coordinate values based on the world coordinates; a frame synthesis unit that reads the map image from the map image storage unit, extracts the pan angle and the shooting angle from the camera parameters interpolated by the camera parameter interpolation unit, obtains a shooting area based on the position of the camera in the map image based on the pan angle and the shooting angle, generates a PinP (Picture in Picture) image that shows the shooting area on the map image, reads the CG from the CG storage unit for each display element, arranges the CG at the location of the image coordinate values converted by the coordinate conversion unit, and arranges the PinP image at the location of a preset image coordinate value to generate the synthesis frame; a video delay unit that delays the camera video frame so that the camera video frame is synchronized with the synthesis frame generated by the synthesis frame generation unit; and a frame synthesis unit that synthesizes the synthesis frame generated by the synthesis frame generation unit with the camera video frame delayed by the video delay unit and outputs the synthesized video.

As described above, according to the present invention, it is possible to generate information that allows the direction and angle of view of a camera image captured in the real world to be easily recognized from a bird's-eye view.

1 is a schematic diagram illustrating an example of the overall configuration of a video synthesis system according to a first embodiment. FIG. 2 is a schematic diagram showing an outline of the operation of the image synthesis system according to the first embodiment; 1 is a block diagram showing an example of the configuration of a video synthesis frame generating device according to a first embodiment; 4 is a flowchart showing an example of processing performed by the video synthesis frame generating device in the first embodiment. 13 is a flowchart showing an example of processing by a camera parameter receiving unit. 13 is a flowchart illustrating an example of processing by a coordinate conversion unit. 11A and 11B are diagrams illustrating an example of processing by a coordinate conversion unit; 13 is a flowchart illustrating an example of processing by a synthesis frame generating unit. FIG. 13 is a diagram showing an example of a synthesis frame. FIG. 1 is a diagram showing an example of a PinP image. 13 is a diagram illustrating an example of a data configuration of a conversion information storage unit; 5 is a diagram illustrating an example of a data configuration of a coordinate information storage unit; FIG. 4 is a diagram illustrating an example of a data structure of a CG storage unit. 13A and 13B are diagrams illustrating another example of the data structure of the CG storage unit. 10 is a flowchart showing another example of processing performed by the video synthesis frame generating device in the first embodiment. 1 is a block diagram showing an example of the configuration of a video frame synthesizing device according to a first embodiment. 11A and 11B are diagrams illustrating an example of a composite image before and after a panning operation is performed. 11A and 11B are diagrams illustrating an example of a composite image before and after a tilt operation and a zoom operation are performed. 11A and 11B are diagrams showing examples of composite images before and after the display of landmarks is switched in accordance with a zoom operation. FIG. 11 is a block diagram showing a configuration example of an image synthesizing device according to a second embodiment.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
Example 1
First, a description will be given of Example 1. Fig. 1 is a schematic diagram showing an example of the overall configuration of a video synthesis system according to Example 1, and Fig. 2 is a schematic diagram showing an outline of the operation thereof.

This video synthesis system 1 is comprised of a video synthesis frame generation device 2 and a video frame synthesis device 3. The video synthesis system 1 inputs camera images captured by a camera operated by a photographer, as well as camera parameters (camera parameters CP). The video synthesis system 1 then generates a synthesis frame including CG for each display element such as direction and landmark, and a PinP image that shows the shooting area S on a map image, synthesizes the synthesis frame with the camera image frame, and outputs the synthesized image in real time. The shooting area S is an area that indicates which direction the camera is capturing images at and with what angle of view.

The video synthesis frame generation device 2 inputs the camera parameters CP, and based on the camera parameters CP, generates a frame containing information related to the shooting of the camera video as a synthesis frame to be used when synthesizing the camera video in the video frame synthesis device 3 described below, and outputs the synthesis frame to the video frame synthesis device 3.

Specifically, the image synthesis frame generating device 2 converts the world coordinate values of each display element, such as directions and landmarks, into image coordinate values (camera coordinate values). Then, the image synthesis frame generating device 2 determines the shooting area S in the map image that includes the camera position, and generates a PinP image that shows the shooting area S on the map image. The world coordinate values indicate coordinate values in the world coordinate system, and the image coordinate values indicate coordinate values in the image coordinate system (camera coordinate system).

The video compositing frame generator 2 generates a compositing frame by placing the CG and PinP images for each display element at a location with a specified image coordinate value, and outputs the compositing frame to the video frame compositing device 3.

As shown in the lower part of Figure 2, the synthesis frame is composed of, for example, a landmark display element "Asahikawa Station" placed at the image coordinate value, and two PinP images placed at the specified image coordinate value.

Returning to FIG. 1, the video frame synthesizer 3 inputs the camera video and a synthesis frame from the video synthesis frame generator 2, delays the camera video by a predetermined time, synthesizes the synthesis frame with the delayed camera video frame, and outputs the synthesized video.

The composite image (frame) is an image in which the camera image is combined with a composite frame consisting of the landmark display element "Asahikawa Station" and two PinP images, as shown on the right side of Figure 2.

This allows the cameraman and viewers to recognize from the PinP image included in the composite image which direction the camera is shooting at and what angle of view it is shooting at, and they can specifically recognize the direction and angle of view of the camera image from the display of the landmark display element "Asahikawa Station" included in the composite image. In other words, by using the image synthesis system 1 shown in Figure 1, it is possible to generate information that allows the direction and angle of view of camera images captured in the real world to be easily recognized from a bird's-eye view.

In addition, this composite image includes a CG image of a specific landmark ("Asahikawa Station" in this example), but it may also include a CG image showing the names of simple directions such as north, south, east and west. The CG image of the display elements to be composited can be freely set by the user operating the image composition frame generation device 2.

(Video Composition Frame Generator 2/Example 1)
Next, a detailed description will be given of the video synthesis frame generating device 2 in the first embodiment shown in Fig. 1. Fig. 3 is a block diagram showing an example of the configuration of the video synthesis frame generating device 2 in the first embodiment, and Fig. 4 is a flowchart showing an example of the processing thereof.

This video synthesis frame generation device 2 includes a frame processing unit 10 and a storage unit 20. The frame processing unit 10 includes a camera parameter receiving unit 11, a camera parameter interpolation unit 12, a coordinate conversion unit 13, and a synthesis frame generation unit 14, and the storage unit 20 includes a conversion information storage unit 21, a coordinate information storage unit 22, a CG storage unit 23, and a map image storage unit 24.

The camera parameter receiving unit 11 receives the camera parameters CP from the camera, and converts the zoom value and focus value included in the camera parameters CP into the shooting angle of view θ and the focus distance f, respectively, using the conversion information stored in the conversion information storage unit 21, to generate camera parameters CP1 including the shooting angle of view θ and the focus distance f (step S401).

The camera parameter receiving unit 11 may generate the camera parameters CP1 by setting the zoom value and focus value included in the camera parameters CP as the shooting angle of view θ and focus distance f, respectively.

The camera parameter receiving unit 11 outputs the camera parameters CP1 to the camera parameter interpolation unit 12. Here, the camera parameters CP input by the camera parameter receiving unit 11 are composed of a pan angle α, a tilt angle δ, a roll angle φ, a zoom value, a focus value, a camera position _tw , etc. The camera parameters CP1 output by the camera parameter receiving unit 11 are composed of a pan angle α, a tilt angle δ, a roll angle φ, a shooting angle of view θ, a focus distance f, a camera position _tw , etc. The camera parameters CP2 described later are the same as the camera parameters CP1. Details of the processing by the camera parameter receiving unit 11 will be described later.

The camera parameter interpolation unit 12 inputs the camera parameters CP1 from the camera parameter receiving unit 11. Then, when the data rate of the data sequence constituting the camera parameters CP1 (the rate at which the camera parameters CP are updated in the camera) is insufficient for the frame rate of the camera image, the camera parameter interpolation unit 12 generates camera parameters CP2 whose values change smoothly by interpolating the data of the camera parameters CP1 so that the data rate of the camera parameters CP1 matches the frame rate of the camera image (step S402). The camera parameter interpolation unit 12 outputs the camera parameters CP2 to the coordinate conversion unit 13.

The interpolation method is preset and selectable by the user. The basic method is a zero-order hold, which holds the same value after the data string of the camera parameters CP1 is input, until the next data string is input. For smoother interpolation, spline interpolation is performed, approximating each section of the input data string using different low-order polynomials.

Furthermore, commonly used cubic spline interpolation is expressed by the following equation, where a _j , b _j , and c _j are the coefficients of a cubic function in an interval to be interpolated, and d _j is the data at the start point of the interval.

When cubic spline interpolation is performed sequentially in real time in an infinite interval, the coefficients _aj , bj, and _cj can be obtained by determining the influence constants in the intervals before and after the interpolation target and the number of intervals before and after the _{interpolation} target in addition to the boundary conditions that smoothly connect the control points, and the interpolation is performed.In addition, a learned inference model that predicts time series data by deep learning such as LSTM (Long Short Term Memory) may be used.

The coordinate conversion unit 13 inputs the camera parameters CP2 from the camera parameter interpolation unit 12. Then, the coordinate conversion unit 13 converts the world coordinate values stored in the coordinate information storage unit 22 into image coordinate values using the camera parameters CP2 for each display element such as a direction or a landmark (step S403).

The coordinate conversion unit 13 outputs the image coordinate values and camera parameters CP2 for each display element to the synthesis frame generation unit 14. The processing of the coordinate conversion unit 13 will be described in detail later.

The synthesis frame generation unit 14 inputs the image coordinate values and camera parameters CP2 for each display element from the coordinate conversion unit 13. Then, for each display element, the synthesis frame generation unit 14 places the CG of that display element stored in the CG storage unit 23 at the location of the input image coordinate values. In addition, the synthesis frame generation unit 14 uses the map image stored in the map image storage unit 24 and the camera parameters CP2 to determine the shooting area S based on the position of the camera in the map image, generates a PinP image that shows the shooting area S on the map image, and places the PinP image at a location of a specified image coordinate value.

The synthesis frame generation unit 14 generates a synthesis frame including a PinP image that represents the CG for each display element and the shooting area S on a map image, and outputs the synthesis frame to the video frame synthesis device 3 (step S404). The processing of the synthesis frame generation unit 14 will be described in detail later.

The video synthesis frame generating device 2 determines whether the processing by the video synthesis frame generating device 2 has been completed (step S405). If the video synthesis frame generating device 2 determines in step S405 that the processing has not been completed (step S405: N), it proceeds to step S401 and continues the processing of the above-mentioned steps S401 to S404, such as receiving the next camera parameters CP. On the other hand, if the video synthesis frame generating device 2 determines that the processing has been completed (step S405: Y), it ends the processing.

(Camera parameter receiving unit 11)
Next, a detailed description will be given of the process of the camera parameter receiving unit 11 shown in Fig. 3. Fig. 5 is a flowchart showing an example of the process of the camera parameter receiving unit 11, which corresponds to step S401 shown in Fig. 4.

The camera parameter receiving unit 11 receives the camera parameters CP from the camera (step S501) and extracts the zoom value and focus value from the camera parameters CP (step S502).

The camera parameter receiving unit 11 reads out the shooting angle of view θ corresponding to the zoom value from the conversion information storage unit 21 (step S503), and also reads out the focus distance f corresponding to the focus value (step S504).

Figure 11 is a diagram showing an example of the data configuration of the conversion information storage unit 21. The conversion information storage unit 21 stores multiple sets of data, each of which is a pair of a zoom value and a shooting angle of view θ corresponding to that zoom value. The conversion information storage unit 21 also stores multiple sets of data, each of which is a pair of a focus value and a focus distance f corresponding to that focus value. These data are set in advance and can be changed by the user.

This converts the zoom value and focus value into the shooting angle of view θ and focus distance f, respectively.

5, the camera parameter receiving unit 11 generates camera parameters CP1 including the imaging angle of view θ and the focus distance f instead of the zoom value and the focus value, i.e., the pan angle α, the tilt angle δ, the roll angle φ, the imaging angle of view θ, the focus distance f, and the camera position _tw (step S505).Then, the camera parameter receiving unit 11 outputs the camera parameters CP1 to the camera parameter interpolation unit 12 (step S506).

As a result, if the zoom value and focus value included in the camera parameters CP are not values that directly indicate the angle of view of the zoom lens and the focused distance, respectively, they are converted into the shooting angle of view θ and the focus distance f using pre-set conversion information stored in the conversion information storage unit 21. In this case, if the camera position is fixed and does not change, the known fixed camera parameters CP are input, and the known fixed camera parameters CP1 are output.

When the zoom value and focus value included in the camera parameter CP directly indicate the angle of view of the zoom lens and the focused distance, respectively, the camera parameter receiving unit 11 generates the camera parameter CP1 by setting the zoom value and focus value to the shooting angle of view θ and the focus distance f, respectively.

(Coordinate conversion unit 13)
Next, a detailed description will be given of the process of the coordinate conversion unit 13 shown in Fig. 3. Fig. 6 is a flow chart showing an example of the process of the coordinate conversion unit 13, which corresponds to step S403 shown in Fig. 4.

The coordinate conversion unit 13 inputs the camera parameters CP2 from the camera parameter interpolation unit 12 (step S601). The coordinate conversion unit 13 then reads out the world coordinate values for each display element from the coordinate information storage unit 22 (step S602).

12 is a diagram showing an example of the data configuration of the coordinate information storage unit 22. The coordinate information storage unit 22 stores a plurality of sets of data, each set consisting of a display element ID that is an identifier of a display element, a name of the display element, and world coordinate values ( _Xw , _Yw , _Zw ) that indicate the position of the display element in the world coordinate system. These display elements are elements such as landmarks that exist within an area that can be photographed by the camera.

For example, the name of the landmark display element "Asahikawa Station" and its world coordinate values ( _Xw1 , _Yw1 , _Zw1 ) are stored corresponding to the display element ID = 1. These data are set in advance and can be changed by the user.

6, in steps S603 to S605 described later, the coordinate conversion unit 13 converts the world coordinate values ( _Xw , _Yw , Zw) of each display element into image coordinate values (x _i , y _i ) using the camera parameter CP2. Here, in order to convert the world coordinate values ( _Xw , _Yw , _Zw ) into image coordinate values (x _{i , y i )} , it is necessary to define a world coordinate system and an image coordinate system.

7 is a diagram for explaining an example of processing by the coordinate conversion unit 13. (1) shows the world geodetic system in which coordinates are expressed by longitude, latitude, and altitude, (2) shows the world coordinate system in which coordinates are expressed by world coordinate values ( _Xw , _Yw , _Zw ), and (3) shows the image coordinate system in which coordinates are expressed by image coordinate values (x _i , y _i ).

For example, the world coordinate system is defined with the camera's viewpoint position as its origin, the eastward direction of the meridian from the origin as the +X axis, the northward direction of the latitude as the +Y axis, and the axis perpendicular to the horizon and pointing upward as the +Z axis, with the order X, Y, Z forming a right-handed system.

The camera coordinate system is also defined. That is, the camera coordinate system is defined so that the optical axis direction is the +z axis, the origin of the camera coordinate system coincides with the origin of the world coordinate system, and when there is no rotation of the axes, the +x axis of the camera coordinate system coincides with the +X axis of the world coordinate system, the +y axis of the camera coordinate system coincides with the -Z axis of the world coordinate system, and the +z axis of the camera coordinate system coincides with the +Y axis of the world coordinate system, forming a right-handed system in the order of x, y, z.

Then, for example, if the position of the landmark display element "Asahikawa Station" is shown by a mark consisting of an X surrounded by a circle, as shown in Figure 7, "Asahikawa Station" is located at the position of the mark in the center of the world geodetic system (1), and the corresponding world coordinate value in the world coordinate system (2) is ( _Xw1 , _Yw1 , _Zw1 ), and the image coordinate value in the image coordinate system (3) is ( _xi1 , _yi1 ).

For the display element "Asahikawa Station" of this landmark, the coordinate conversion unit 13 converts the world coordinate values ( _Xw1 , _Yw1 , _Zw1 ) into image coordinate values ( _xi1 , _yi1 ).

Returning to FIG. 6, after step S602, the coordinate conversion unit 13 extracts the pan angle α, tilt angle δ, roll angle φ, shooting angle of view θ, focus distance f, and camera position _tw from the camera parameters CP2 (step S603).

The coordinate conversion unit 13 defines the rotation angle about the y-axis as the pan angle α [rad], the rotation angle about the x-axis as the tilt angle δ [rad], and the rotation angle about the z-axis as the roll angle φ [rad], and calculates the attitude R using the pan angle α, tilt angle δ, and roll angle φ according to the following equation (step S604).

Here, the camera position t _w , which is the position of the origin of the camera coordinate system in the world coordinate system (the viewpoint position of the camera), is expressed by the following equation.

画像座標系は、画像左上を原点として水平右方向を＋ｘ軸、 垂直下方向を＋ｙ軸とし、横・縦方向のピクセル数の半値をμ₀，ν₀とし、フォーカス距離ｆはピクセル単位の値とする。このように、座標変換部１３により、フォーカス距離ｆ、姿勢Ｒ、カメラ位置ｔ_w及び世界座標値（Ｘ_w，Ｙ_w，Ｚ_w）から画像座標値（ｘ_i，ｙ_i）が算出される。 The coordinate conversion unit 13 converts the world coordinate values ( _Xw , _Yw , Zw) into image coordinate values (x _i , y _i ) for each display element using the focus distance f, orientation R and _camera position _tw (step S605).

The image coordinate system has the upper left corner of the image as the origin, the horizontal right direction as the +x axis, the vertical down direction as the +y axis, half the number of pixels in the horizontal and vertical directions as _μ0 and _ν0 , and the focus distance f is a value in pixels. In this way, the coordinate conversion unit 13 calculates the image coordinate values (x _i , y _i ) from the focus distance f, the attitude R, the camera position t _w and the world coordinate values (X _w , Y _w , Z _w ).

The coordinate conversion unit 13 outputs the image coordinate values (x _i , y _i ) and the camera parameters CP2 for each display element to the synthesis frame generation unit 14 (step S606).

As a result, the world coordinate values ( _Xw , _Yw , _Zw ) for each display element stored in the coordinate information storage unit 22 are converted into image coordinate values (x _i , y _i ).

(Synthesis Frame Generator 14)
Next, a detailed description will be given of the processing of the synthesis frame generating unit 14 shown in Fig. 3. Fig. 8 is a flow chart showing an example of the processing of the synthesis frame generating unit 14, which corresponds to step S404 shown in Fig. 4.

The compositing frame generation unit 14 inputs the image coordinate values and camera parameters CP2 for each display element from the coordinate conversion unit 13 (step S801). The compositing frame generation unit 14 then reads out the CG for each display element from the CG storage unit 23 (step S802).

Figure 13 is a diagram showing an example of the data structure of the CG storage unit 23. The CG storage unit 23 stores multiple sets of data, each set consisting of a display element ID, which is an identifier of a display element, and the CG of that display element.

For example, a CG that graphically represents the name of the landmark display element "Asahikawa Station" (see Figure 12) is stored in correspondence with display element ID = 1. This data is preset and can be changed by the user.

Returning to FIG. 8, the synthesis frame generation unit 14 reads a map image from the map image storage unit 24 (step S803). The map image is, for example, an image of a map with the camera position positioned in the center.

The synthesis frame generation unit 14 extracts the pan angle α and the shooting angle of view θ from the camera parameters CP2 (step S804). The synthesis frame generation unit 14 then determines the camera direction α1 from the pan angle α, and determines the angle θ1 between two sides of the shooting area S based on the camera position from the shooting angle of view θ (see the right side of Figure 9 described later) (step S805).

The synthesis frame generation unit 14 determines the shooting area S in the map image from the camera position in the map image, the camera direction α1, and the angle θ1 of the shooting area S, and generates a PinP image by arranging the shooting area S in the map image (step S806). Since the camera position in the map image is set in advance (for example, because it is set to the center position in the map image), the shooting area S can be identified in the map image from the camera direction α1 and the angle θ1.

Here, a PinP image is an image that shows the shooting area S projected from the viewpoint position of the camera on a map image. In three dimensions, the shooting area S is a quadrangular pyramid that changes depending on the camera's attitude, sensor shape, and angle of view, but when shown on a map, the quadrangular pyramid may be a perspective or parallel projection onto a two-dimensional map, or it may be a triangle calculated from the horizontal angle of view of the lens.

If the map of the map image is not a projection of a three-dimensional spherical model such as the Mercator projection, the distortion due to the projection may be reflected in the shooting area S. The shooting area S may be displayed by alpha blending the interior with a color preset by the user, or by showing the lines that form its boundaries.

The camera position (viewpoint position) in the shooting area S may be the center of the map in the PinP image, or may be set at any position, and a camera icon image indicating the shooting direction may be displayed superimposed. Also, in the PinP image, CG may be displayed at the position of the image coordinate values of display elements such as directions and landmarks, as in step S807 described below.

The compositing frame generating unit 14 generates a compositing frame by placing each CG read out in step S802 at the image coordinate values for each display element input in step S801, and placing the PinP image generated in step S806 at the image coordinate values set in advance (step S807). It is assumed that the user sets in advance whether or not to place the PinP image in the compositing frame.

9 is a diagram showing an example of a compositing frame, in which a CG image of a landmark display element "Asahikawa Station" is placed at the location of image coordinate values (x _i1 , y _i1 ), and two PinP images with different scales showing the shooting area S on a map image are placed at the locations of preset image coordinate values.

In this example, a camera is installed at the vertex of the shooting area S, and the shooting area S is set based on the position of this camera, with the angle θ1 between two sides of the camera direction (optical axis) α1, with the dotted line (see the right side of Figure 9) at the center.

In this example, two PinP images with different scales are placed, but one PinP image may be placed, or three or more PinP images with different scales may be placed. Also, the size and position of the PinP image within the composition frame can be freely set and changed by the user.

Figure 10 shows an example of a PinP image, where the scale can be freely changed. The map image used to generate the PinP image is enlarged or reduced according to a scale preset by the user, for example. Then, as shown in Figure 10, a PinP image is generated in which the shooting area S is displayed on the enlarged or reduced map image. The preset scale can be changed by the user.

In other words, the synthesis frame generation unit 14 enlarges or reduces the map image read from the map image storage unit 24 in step S803 of FIG. 8 according to a preset scale (or reads a map image of a preset scale from the map image storage unit 24), and uses the map image of that scale when generating the PinP image in step S806.

The scale of the map image may be changed based on the shooting angle of view θ or the focus distance f included in the camera parameters CP2. For example, the synthesis frame generation unit 14 sets a scale that is smaller the shooting angle of view θ, and a scale that is larger the shooting angle of view θ, based on a threshold judgment of the shooting angle of view θ extracted from the camera parameters CP2. The synthesis frame generation unit 14 also sets a scale that is smaller the longer the focus distance f, and a scale that is larger the shorter the focus distance f, based on a threshold judgment of the focus distance f extracted from the camera parameters CP2.

Returning to FIG. 8, after step S807, the synthesis frame generation unit 14 outputs the synthesis frame to the video frame synthesis device 3 (step S808).

As a result, using the image coordinate values for each display element, the camera parameters CP2, the CG for each display element stored in the CG storage unit 23, and the map image stored in the map image storage unit 24, the CG for each display element and a PinP image showing the shooting area S on the map image are placed at a location with a specified image coordinate value, and a synthesis frame is generated.

In this way, the image synthesis frame generating device 2 shown in FIG. 3 can generate synthesis frames in succession for each frame that are synchronized with the camera image frames, thereby generating synthesis frames in which the shooting area S in the CG and PinP images for each display element follows movements such as panning, tilting, or zooming of the camera.

In this case, the processing of the video synthesis frame generating device 2 is performed in series by step S401 by the camera parameter receiving unit 11, step S402 by the camera parameter interpolation unit 12, step S403 by the coordinate conversion unit 13, and step S404 by the synthesis frame generating unit 14.

In other words, the video synthesis frame generating device 2 inputs the camera parameters CP, then increases the camera parameters CP2 that are synchronized with the frame rate of the camera video through interpolation processing, and performs the processing of steps S401 to S404 shown in FIG. 4 until generating a synthesis frame for each frame in an infinite loop, and continues to output synthesis frames. In other words, the video synthesis frame generating device 2 serially processes each camera parameter CP2 that corresponds to the input camera parameters CP, i.e., each frame of the camera video. Alternatively, the video synthesis frame generating device 2 may perform parallel processing instead of the serial processing shown in FIG. 4.

FIG. 15 is a flowchart showing another example of processing by the video synthesis frame generating device 2 in the first embodiment, and shows an example of parallel processing instead of the serial processing shown in FIG. 4. Steps S1501 to S1505 shown in FIG. 15 correspond to steps S401 to S405 shown in FIG. 4, respectively.

The video synthesis frame generating device 2 performs the processes of steps S1501 to S1505 in parallel, and performs the process of the relevant step immediately after the process of the previous step is completed. In other words, each component of the video synthesis frame generating device 2, from the camera parameter receiving unit 11 to the synthesis frame generating unit 14, executes processes in parallel, and after completing the process and outputting the data, inputs data from the previous component and continues the process.

(Video Frame Synthesizer 3/Example 1)
Next, a detailed description will be given of the video frame synthesizer 3 in the embodiment 1 shown in Fig. 1. Fig. 16 is a block diagram showing an example of the configuration of the video frame synthesizer 3 in the embodiment 1. The video frame synthesizer 3 includes a video delay unit 30 and a frame synthesizer 31.

The video delay unit 30 inputs the camera video, delays the camera video by a preset delay amount so that the camera video frames are synchronized with the synthesis frames in the frame synthesis unit 31 described below, and outputs the delayed camera video to the frame synthesis unit 31. This adjusts the delay amount of the camera video so that the timing of the camera video frames matches the timing of the synthesis frames generated by the video synthesis frame generation device 2.

When synchronized camera image frames and camera parameters CP are input to the image frame synthesizer 3 and the image synthesis frame generator 2, respectively, at a certain time, if it is necessary to sample future camera parameters CP2 in the camera parameter interpolation unit 12 provided in the image synthesis frame generator 2, a delay corresponding to the amount of sampling occurs in the camera image frames since the input time is treated as future.

Therefore, the video delay unit 30 delays the camera video according to the delay amount of the synthesis frame determined by the processing of the frame processing unit 10 shown in Figure 3, which includes this delay amount.

The amount of delay can be freely set by the user, and may be set to 0. For example, the user measures the processing time from when the camera parameters CP are input by the frame processing unit 10 shown in FIG. 3 until the synthesis frame is output, and sets this processing time as the amount of delay.

The frame synthesis unit 31 inputs the delayed camera video from the video delay unit 30, and inputs the synthesis frames from the video synthesis frame generation device 2. As described above, the delayed camera video frames and synthesis frames input by the frame synthesis unit 31 are synchronized, and the timing of the delayed camera video frames matches the timing of the synthesis frames.

The frame synthesis unit 31 synthesizes a synthesis frame with a delayed camera image frame to generate a synthetic image, for example, as shown in FIG. 2, and outputs the synthetic image.

(Example of composite images before and after camera operation)
Next, an example of a composite image output by the image frame synthesizer 3 in the image synthesis system 1 shown in Fig. 1 when the camera parameter CP is changed by the cameraman operating the camera will be described. The examples of composite images shown in Figs. 17 to 19 described later are also applicable to Example 2 described later.

Figure 17 shows an example of a composite image before and after a panning operation. The image on the left shows the composite image before a panning operation is performed, and the image on the right shows the composite image after a panning operation is performed to the left on the landmark display element "Asahikawa Station."

From Figure 17, we can see that panning to the left causes the CG position of the landmark display element "Asahikawa Station" placed in the composite image to move from left to right within the composite image.

It can also be seen that panning to the left causes the orientation of the shooting area S (camera orientation α1 (see Figure 9)) in the two PinP images located at the bottom right of the composite video to move from the bottom left to the bottom within the PinP images.

In this way, the shooting area S in the CG and PinP images of the landmark display element "Asahikawa Station" can be made to follow the panning operation of the camera.

Figure 18 shows an example of a composite image before and after a tilt operation and a zoom operation. The image on the left shows the composite image before a tilt operation and a zoom operation are performed, and the image on the right shows the composite image after a tilt operation and a zoom operation are performed on the landmark display element "Tokachi-dake."

From Figure 18, it can be seen that by using tilt and zoom operations, the CG of the landmark display elements "Asahikawa Airport" and "Tokachi Mountain" displayed in the composite image are reduced to only the CG of "Tokachi Mountain," and the image of "Tokachi Mountain" is displayed in an enlarged scale.

It can also be seen that the tilt operation and zoom operation do not change the orientation (camera orientation α1) of the shooting area S in the two PinP images located at the bottom right of the composite video. It can also be seen that the zoom operation narrows the width (angle θ1 (see FIG. 9)) of the shooting area S in the two PinP images located at the bottom right of the composite video (angle θ1 becomes smaller).

In this way, the CG of the landmark display elements "Asahikawa Airport" and "Tokachi Mountain", as well as the shooting area S in the PinP image, can be made to follow the tilt and zoom operations of the camera.

Figure 19 shows an example of a composite image before and after the landmark display is switched in response to a zoom operation. The image on the left shows the composite image before the zoom operation is performed, and the image on the right shows the composite image after the zoom operation is performed.

From Figure 19, we can see that the zoom operation has switched the CG of the landmark display element "Asahiyama" displayed in the composite image to the CG of "Asahiyama Zoo."

It can also be seen that the orientation (camera orientation α1) of the shooting area S in the two PinP images located at the bottom right of the composite video does not change due to the zoom operation. It can also be seen that the width (angle θ1) of the shooting area S in the two PinP images located at the bottom right of the composite video becomes narrower (the angle θ1 becomes smaller) due to the zoom operation.

In this way, the CG of the landmark display elements "Asahiyama" and "Asahiyama Zoo", as well as the shooting area S in the PinP image, can be made to follow the zoom operation of the camera. Furthermore, the process described below makes it possible to change the CG of the display elements in response to the zoom operation of the camera.

Figure 14 shows another example of the data structure of the CG storage unit 23 shown in Figure 3, and is applicable to cases where the display of landmarks is switched in response to a zoom operation, as in the example of the composite image shown in Figure 19.

The CG storage unit 23 stores a plurality of sets of data each including a display element ID and a CG of the display element when the shooting angle of view θ is equal to or larger _than a preset threshold value θa (θ≧ _θa ), and also stores a plurality of sets of data each including a display element ID and a CG of the display element when the shooting angle of view θ is smaller than the threshold value _θa (θ< _θa ).

For example, when the shooting angle of view θ is equal to or greater than the threshold value _θa (θ≧ _θa ), a CG that graphically represents the name of the landmark's display element, "Asahiyama," is stored in correspondence with the display element ID=n. When the shooting angle of view θ is smaller than the threshold value _θa (θ< _θa ), a CG that graphically represents the name of the landmark's display element, "Asahiyama Zoo," is stored in correspondence with the display element ID=n. These data are preset and can be changed by the user.

In this case, the synthesis frame generation unit 14 shown in FIG. 3 extracts the shooting angle of view θ from the camera parameter CP2 in step S802 shown in FIG. 8, and determines whether the shooting angle of view θ is equal to or larger than a preset threshold value _θa or whether the shooting angle of view θ is smaller than the threshold _{value θa} .

When the synthesis frame generation unit 14 determines that the shooting angle of view θ is equal to or greater than the threshold _{value θa} , it reads out CG of display elements for which the shooting angle of view θ is equal to or greater than the threshold value _θa (θ≧ _θa ) from the CG storage unit 23 shown in Fig. 14. On the other hand, when the synthesis frame generation unit 14 determines that the shooting angle of view θ is smaller than the threshold value _θa , it reads out CG of display elements for which the shooting angle of view θ is smaller than the threshold value _θa (θ< _θa ) from the CG storage unit 23 shown in Fig. 14.

The compositing frame generating unit 14 generates a compositing frame by, for example, arranging each CG read from the CG storage unit 23 at the location of the image coordinate value for each display element. In the examples of Fig. 14 and Fig. 19, for a display element ID=n, when the shooting angle of view θ is equal to or greater than the threshold value _θa (θ≧ _θa ), a composite image is generated in which the CG of the display element "Asahiyama" is arranged, as shown on the left side of Fig. 19. Then, when the shooting angle of view θ becomes smaller than the threshold value _θa (θ< _θa ) due to a zoom operation of the camera, a composite image is generated in which the CG of the display element "Asahiyama Zoo" is arranged, as shown on the right side of Fig. 19.

As a result, a compositing frame containing CG of display elements corresponding to the shooting angle of view θ is generated in response to the camera zoom operation, and as a result, a composite image containing CG of display elements corresponding to the shooting angle of view θ is generated.

In other words, the CG of the display element changes according to the shooting angle of view θ included in the camera parameter CP2. For example, when the camera image is zoomed in and the display element of the landmark is enlarged, the CG showing the outline of the landmark (for example, the display element "Tokachi Mountain Range" before the zoom operation) is switched to the CG showing the details of the landmark (for example, the display elements "Biei Mountain", "Tokachi Mountain", "Furano Mountain" after the zoom operation).

Note that while the example in FIG. 14 shows a case where there is one threshold, display element IDs and CGs of the display elements distinguished by multiple thresholds may be stored in the CG storage unit 23.

In addition, in the above example, the CG of the display element included in the synthesis frame is changed according to the shooting angle of view θ included in the camera parameter CP2. That is, in the example of the shooting angle of view θ shown in FIG. 14, the larger the shooting angle of view θ is, the more CG showing an outline of the landmark is read from the CG storage unit 23, and the smaller the shooting angle of view θ is, the more CG showing the details of the landmark is read.

In contrast, the CG of the display element may be changed according to the focus distance f included in the camera parameter CP2. In this case, the shorter the focus distance f, the more CG showing an outline of the landmark (the CG shown on the left side of FIG. 14) is read from the CG storage unit 23, and the longer the focus distance f, the more CG showing the details of the landmark (the CG shown on the right side of FIG. 14) is read.

As described above, according to the video synthesis system 1 of the first embodiment, the camera parameter receiving unit 11 of the video synthesis frame generating device 2 receives the camera parameters CP from the camera, and uses the conversion information stored in the conversion information storage unit 21 to convert the zoom value and focus value included in the camera parameters CP into the shooting angle of view θ and the focus distance f, respectively, to generate the camera parameters CP1 including the shooting angle of view θ and the focus distance f.

The camera parameter interpolation unit 12 generates camera parameters CP2 by interpolating the data of the camera parameters CP1 so that the data rate of the camera parameters CP1 matches the frame rate of the camera image.

For each display element, the coordinate conversion unit 13 converts the world coordinate values into image coordinate values using the camera parameters CP2 and the world coordinate values of the display element stored in the coordinate information storage unit 22.

For each display element, the synthesis frame generation unit 14 places the CG of that display element stored in the CG storage unit 23 at the location of the image coordinate value, and uses the map image stored in the map image storage unit 24 to determine the shooting area S in the map image including the camera position, generates a PinP image that shows the shooting area S on the map image, and places the PinP image at the location of the specified image coordinate value. Then, the synthesis frame generation unit 14 generates a synthesis frame that includes the CG of each display element and the PinP image that shows the shooting area S on the map image.

As a result, a synthesis frame synchronized with the camera image frame is generated continuously for each frame. In other words, a synthesis frame is generated in which the shooting area S in the CG and PinP images for each display element follows the camera's movements such as panning, tilting, or zooming.

In addition, the video delay unit 30 of the video frame synthesis device 3 delays the camera video by a preset delay amount so that the camera video frame is synchronized with the synthesis frame generated by the video synthesis frame generation device 2.

The frame synthesis unit 31 generates a synthetic image by synthesizing a synthesis frame with a delayed camera image frame.

As a result, a composite image synchronized with the camera image frames is generated continuously for each frame. In other words, a composite image is generated in which the CG for each display element and the shooting area S in the PinP image follow the camera's movements such as panning, tilting, or zooming.

This makes it possible to generate information that allows the direction and angle of view of a camera image capturing the real world to be easily recognized from a bird's-eye view.

In this way, in camera footage of the real world, the shooting area S on the map image in the PinP image can be displayed in real time, along with CG for each display element such as direction and landmarks, by following the camera's movement. Cameramen and viewers who receive this display can easily recognize which direction and what angle of view the image currently being shot is taking. This is particularly useful for cameramen and viewers who are unfamiliar with the area.

Example 2
Next, a description will be given of a second embodiment of the present invention. Fig. 20 is a block diagram showing the configuration of a video synthesizing apparatus according to the second embodiment of the present invention.

This image synthesis device 4 includes a frame processing unit 10, a memory unit 20, an image delay unit 30, and a frame synthesis unit 31. The frame processing unit 10 includes a camera parameter receiving unit 11, a camera parameter interpolation unit 12, a coordinate conversion unit 13, and a synthesis frame generation unit 14, and the memory unit 20 includes a conversion information storage unit 21, a coordinate information storage unit 22, a CG storage unit 23, and a map image storage unit 24.

The image synthesizer 4 inputs the camera image captured by the camera as well as the camera parameters CP. The image synthesizer 4 then generates a synthesis frame including CG for each display element and a PinP image that shows the shooting area S on a map image, synthesizes the synthesis frame with the camera image frame, and outputs the synthesized image in real time.

Comparing the video synthesis system 1 of Example 1 shown in FIG. 1 (video synthesis frame generation device 2 shown in FIG. 3 and video frame synthesis device 3 shown in FIG. 16) with the video synthesis device 4 of Example 2, both video synthesis systems 1 and video synthesis device 4 have the same components. In other words, the video synthesis device 4 is a single device configured by combining the video synthesis frame generation device 2 and the video frame synthesis device 3 that constitute the video synthesis system 1 of Example 1 shown in FIG. 1.

The processing of the frame processing unit 10 (camera parameter receiving unit 11, camera parameter interpolation unit 12, coordinate conversion unit 13, and synthesis frame generation unit 14), memory unit 20 (conversion information memory unit 21, coordinate information memory unit 22, CG memory unit 23, and map image memory unit 24), video delay unit 30, and frame synthesis unit 31 provided in the image synthesis device 4 is the same as in Example 1, so a description thereof will be omitted here.

As described above, the image synthesis device 4 of the second embodiment provides the same effect as the image synthesis system 1 of the first embodiment. In other words, a synthetic image in which the CG for each display element and the shooting area S in the PinP image follow the camera's panning, tilting, zooming, or other movements is generated continuously for each frame.

This makes it possible to generate information that allows the direction and angle of view of a camera image capturing the real world to be easily recognized from a bird's-eye view.

The present invention has been described above using Examples 1 and 2, but the present invention is not limited to the above Examples 1 and 2, and various modifications are possible without departing from the technical concept thereof.

For example, the roll angle φ of the camera is included in the camera parameters CP, CP1, and CP2, but if the cameraman does not roll the camera, the roll angle φ may be treated as a fixed angle. In other words, the coordinate conversion unit 13 does not extract the roll angle φ from the camera parameter CP2 in step S603 shown in FIG. 6, but calculates the attitude R using the preset roll angle φ in step S604.

In addition, for example, in step S802 shown in FIG. 8, the synthesis frame generation unit 14 reads out CG of a display element, such as a direction or a landmark, from the CG storage unit 23, and in step S807 generates a synthesis frame by, for example, placing the CG at the location of the image coordinate values.

In response to this, the compositing frame generating unit 14 may read out the CG of the display element of the distance to the object from the CG storage unit 23. In this case, the CG storage unit 23 stores CG that graphically represents the actual distance corresponding to the display element ID. The compositing frame generating unit 14 reads out the CG of the distance display element from the CG storage unit 23, and generates a compositing frame by, for example, placing the CG at the image coordinate value. In this case, the compositing frame generating unit 14 may assign a color corresponding to the distance to the CG of the read out distance display element, and generate a compositing frame. This allows the cameraman and the viewer to easily grasp the distance to the object according to the color of the CG.

In addition, a normal computer can be used as the hardware configuration of the video synthesis frame generation device 2 and video frame synthesis device 3 of Example 1 of the present invention, and the video synthesis device 4 of Example 2. The video synthesis frame generation device 2, the video frame synthesis device 3, and the video synthesis device 4 are configured by a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, an interface, etc.

The functions of the frame processing unit 10 (camera parameter receiving unit 11, camera parameter interpolation unit 12, coordinate conversion unit 13, and synthesis frame generation unit 14) and memory unit 20 (conversion information memory unit 21, coordinate information memory unit 22, CG memory unit 23, and map image memory unit 24) provided in the video synthesis frame generation device 2 are each realized by having the CPU execute a program that describes these functions.

Furthermore, the functions of the video delay unit 30 and the frame synthesis unit 31 provided in the video frame synthesis device 3 are each realized by having the CPU execute a program that describes these functions.

Furthermore, the functions of the frame processing unit 10, memory unit 20, video delay unit 30, and frame synthesis unit 31 provided in the video synthesis device 4 are each realized by having the CPU execute a program that describes these functions.

These programs are stored in the storage medium and are read and executed by the CPU. In addition, these programs can be distributed by storing them on storage media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, etc.), and semiconductor memories, and can also be transmitted and received via a network.

1 Video synthesis system 2 Video synthesis frame generation device 3 Video frame synthesis device 4 Video synthesis device 10 Frame processing unit 11 Camera parameter receiving unit 12 Camera parameter interpolation unit 13 Coordinate conversion unit 14 Synthesis frame generation unit 20 Memory unit 21 Conversion information memory unit 22 Coordinate information memory unit 23 CG memory unit 24 Map image memory unit 30 Video delay unit 31 Frame synthesis unit CP, CP1, CP2 Camera parameters S Shooting area α Pan angle δ Tilt angle φ Roll angle θ Shooting angle of view f Focus distance t _w Camera position R Attitude μ ₀ , ν ₀ Half value of number of pixels in horizontal and vertical directions (X _w , Y _w , Z _w ) World coordinate values (x _i , y _i ) Image coordinate values α 1 Camera direction θ 1 Angle θ _a Threshold value

Claims (9)

1. A video synthesis frame generating device that generates a frame including information about a camera image as a synthesis frame used when synthesizing the camera images, comprising:
a coordinate information storage unit in which, for each display element of information relating to the shooting of the camera image, a world coordinate value of the display element is stored;
a map image storage unit in which a map image including the positions of the cameras that capture the camera images is stored;
a CG storage unit in which CG for each of the display elements is stored;
a camera parameter receiving unit that receives camera parameters of the camera, sets a zoom value and a focus value included in the camera parameters as a shooting angle of view and a focus distance, respectively, and outputs camera parameters including the shooting angle of view and the focus distance;
a camera parameter interpolation unit that interpolates the camera parameters so that a data rate of the camera parameters output by the camera parameter receiving unit matches a frame rate of the camera image;
a coordinate conversion unit that reads out the world coordinate values from the coordinate information storage unit for each of the display elements, and converts the world coordinate values into image coordinate values based on the camera parameters interpolated by the camera parameter interpolation unit;
reading out the map image from the map image storage unit, extracting a pan angle and the shooting angle of view from the camera parameters interpolated by the camera parameter interpolation unit, determining a shooting area based on the pan angle and the shooting angle of view in the map image and using the position of the camera as a reference, and generating a PinP (Picture-in-Picture) image in which the shooting area is represented on the map image,
a synthesis frame generating unit that generates the synthesis frame by reading the CG from the CG storage unit for each of the display elements, arranging the CG at the location of the image coordinate values converted by the coordinate conversion unit, and arranging the PinP image at a location of a preset image coordinate value;
A video synthesis frame generating device comprising: 2. The video synthesis frame generating device according to claim 1,
a new camera parameter receiving unit replacing the camera parameter receiving unit, and further comprising a conversion information storage unit storing a photographing angle of view corresponding to the zoom value and a focus distance corresponding to the focus value;
The new camera parameter receiving unit
receiving camera parameters of the camera, extracting a zoom value and a focus value from the camera parameters, reading out the shooting angle of view corresponding to the zoom value and the focus distance corresponding to the focus value from the conversion information storage unit, and outputting the camera parameters including the shooting angle of view and the focus distance;
The camera parameter interpolation unit
a frame generating unit for generating frames for video synthesis, the frame generating unit interpolating the camera parameters so that a data rate of the camera parameters output by the new camera parameter receiving unit matches a frame rate of the camera video; 3. The video synthesis frame generating device according to claim 1,
The coordinate conversion unit is
The pan angle, tilt angle, roll angle, focus distance and camera position are extracted from the camera parameters, and the pan angle is α, the tilt angle is δ, the roll angle is φ, the focus distance is f, the camera position is t _w = [t _wx , t _wy , t _wz ], the world coordinate values are (X _w , Y _w , Z _w ), the image coordinate values are (x _i , y _i ), and half the number of pixels in the horizontal and vertical directions are μ ₀ and ν ₀ ,
Using the pan angle α, the tilt angle δ, and the roll angle φ, the following equation is calculated:

The attitude R is calculated by
For each of the display elements, the focus distance f, the orientation R, the camera position t _w and the half-values μ ₀ and ν ₀ of the number of pixels are calculated using the following equation:

and converting the world coordinate values (X _w , Y _w , Z _w ) into the image coordinate values (x _i , y _i ) by 3. The video synthesis frame generating device according to claim 1,
The synthesis frame generating unit
extracting the pan angle and the shooting angle from the camera parameters, determining the orientation of the camera from the pan angle, determining an angle of the shooting area based on the position of the camera from the shooting angle of view, and determining the shooting area in the map image from the orientation of the camera and the angle of the shooting area. 3. The video synthesis frame generating device according to claim 1,
the CG storage unit stores a plurality of CGs for each of the display elements, the CGs corresponding to the shooting angle of view or the focus distance of the display element;
The synthesis frame generating unit
Extracting the photographing angle of view or the focus distance from the camera parameters;
a CG corresponding to the shooting angle of view or the focus distance is read from the CG storage unit for each of the display elements, the CG is placed at the location of the image coordinate values, and the PinP image is placed at the predetermined location, thereby generating the synthesis frame. 3. The video synthesis frame generating device according to claim 1,
The synthesis frame generating unit
Extracting the photographing angle of view or the focus distance from the camera parameters;
A video synthesis frame generating device characterized in that the device changes the scale of the PinP image based on the shooting angle of view or the focus distance, and generates the synthesis frame by placing the PinP image with the changed scale in the predetermined location. 1. An image synthesizing device that generates a frame including information related to shooting of a camera image as a synthesis frame, and synthesizes the synthesis frame with a frame of the camera image to generate a synthetic image, comprising:
a coordinate information storage unit in which, for each display element of information relating to the shooting of the camera image, a world coordinate value of the display element is stored;
a map image storage unit in which a map image including the positions of the cameras that capture the camera images is stored;
a CG storage unit in which CG for each of the display elements is stored;
a camera parameter receiving unit that receives camera parameters of the camera, sets a zoom value and a focus value included in the camera parameters as a shooting angle of view and a focus distance, respectively, and outputs camera parameters including the shooting angle of view and the focus distance;
a camera parameter interpolation unit that interpolates the camera parameters so that a data rate of the camera parameters output by the camera parameter receiving unit matches a frame rate of the camera image;
a coordinate conversion unit that reads out the world coordinate values from the coordinate information storage unit for each of the display elements, and converts the world coordinate values into image coordinate values based on the camera parameters interpolated by the camera parameter interpolation unit;
reading out the map image from the map image storage unit, extracting a pan angle and the shooting angle of view from the camera parameters interpolated by the camera parameter interpolation unit, determining a shooting area based on the pan angle and the shooting angle of view in the map image with the position of the camera as a reference, and generating a PinP (Picture-in-Picture) image in which the shooting area is represented on the map image,
a synthesis frame generating unit that generates the synthesis frame by reading the CG from the CG storage unit for each of the display elements, arranging the CG at the location of the image coordinate values converted by the coordinate conversion unit, and arranging the PinP image at a location of a preset image coordinate value;
a video delay unit that delays the camera video frame so that the camera video frame is synchronized with the synthesis frame generated by the synthesis frame generation unit;
a frame synthesis unit that synthesizes the camera video frame delayed by the video delay unit with the synthesis frame generated by the synthesis frame generation unit and outputs the synthesized video;
An image synthesizing device comprising: A computer constituting an image synthesis frame generating device that generates a frame including information regarding shooting of a camera image as a synthesis frame to be used when synthesizing the camera image,
a coordinate information storage unit in which, for each display element of information relating to the shooting of the camera image, the world coordinate values of the display element are stored;
a map image storage unit in which a map image including the positions of the cameras capturing the camera images is stored;
a CG storage unit in which CG for each of the display elements is stored;
a camera parameter receiving unit that receives camera parameters of the camera, determines a zoom value and a focus value included in the camera parameters as a photographing angle of view and a focus distance, respectively, and outputs camera parameters including the photographing angle of view and the focus distance;
a camera parameter interpolation unit that interpolates the camera parameters so that a data rate of the camera parameters output by the camera parameter receiving unit matches a frame rate of the camera image;
a coordinate conversion unit that reads out the world coordinate values from the coordinate information storage unit for each of the display elements, and converts the world coordinate values into image coordinate values based on the camera parameters interpolated by the camera parameter interpolation unit; and
reading out the map image from the map image storage unit, extracting a pan angle and the shooting angle of view from the camera parameters interpolated by the camera parameter interpolation unit, determining a shooting area based on the pan angle and the shooting angle of view in the map image and using the position of the camera as a reference, and generating a PinP (Picture-in-Picture) image in which the shooting area is represented on the map image,
A program for functioning as a synthesis frame generation unit that generates the synthesis frame by reading the CG from the CG storage unit for each display element, placing the CG at the location of the image coordinate values converted by the coordinate conversion unit, and placing the PinP image at a location of predetermined image coordinate values. a computer that constitutes an image synthesizing device that generates a frame including information regarding shooting of a camera image as a synthesis frame, and synthesizes the synthesis frame with a frame of the camera image to generate a synthetic image;
a coordinate information storage unit in which, for each display element of information relating to the shooting of the camera image, the world coordinate values of the display element are stored;
a map image storage unit in which a map image including the positions of the cameras capturing the camera images is stored;
a CG storage unit in which CG for each of the display elements is stored;
a camera parameter receiving unit that receives camera parameters of the camera, sets a zoom value and a focus value included in the camera parameters as a shooting angle of view and a focus distance, respectively, and outputs camera parameters including the shooting angle of view and the focus distance;
a camera parameter interpolation unit that interpolates the camera parameters so that a data rate of the camera parameters output by the camera parameter receiving unit matches a frame rate of the camera image;
a coordinate conversion unit that reads out the world coordinate values from the coordinate information storage unit for each of the display elements, and converts the world coordinate values into image coordinate values based on the camera parameters interpolated by the camera parameter interpolation unit;
reading out the map image from the map image storage unit, extracting a pan angle and the shooting angle of view from the camera parameters interpolated by the camera parameter interpolation unit, determining a shooting area based on the pan angle and the shooting angle of view in the map image with the position of the camera as a reference, and generating a PinP (Picture-in-Picture) image in which the shooting area is represented on the map image,
a synthesis frame generating unit which generates the synthesis frame by reading out the CG from the CG storage unit for each of the display elements, arranging the CG at the location of the image coordinate values converted by the coordinate conversion unit, and arranging the PinP image at a location of a preset image coordinate value;
a video delay unit that delays the camera video frame so that the camera video frame is synchronized with the synthesis frame generated by the synthesis frame generation unit; and
A program for functioning as a frame synthesis unit that synthesizes the camera image frame delayed by the image delay unit with the synthesis frame generated by the synthesis frame generation unit and outputs the synthesized image.

Images