JP6200316B2 - Image generation method, image generation apparatus, and image generation program - Google Patents

Description

  The present invention relates to an image generation method, an image generation apparatus, and an image generation program for generating an image at a desired viewpoint position from an existing image.

  2. Description of the Related Art Conventionally, a technique for displaying an image corresponding to a change in viewpoint on a two-dimensional display is known (see, for example, Patent Document 1). FIG. 10 (cited from FIG. 1 of Patent Document 1) is a conceptual diagram showing an image display image according to the prior art. In FIG. 10, when the user 1 changes the viewpoint (corresponding to the three-axis movement), the depth position of the conversation partner 2 in the video (the conversation partner 2 and the background wall 3) is displayed in two dimensions according to the change of the viewpoint. By displaying on the display 4, the appearance of the conversation partner 2 and the background wall 3 can be reproduced as if they were real.

  When the user 1 faces the two-dimensional display 4 at the position P1, the conversation partner 2 is displayed as if viewed from the front, and when the user 1 is viewed from the left position P2, When the dialog partner 2 is displayed as if viewed from the left side, reflecting the positional relationship (depth) between the dialog partner 2 and the wall 3 in the background, and the user 1 is viewing from the right position P3 Reflecting the positional relationship (depth) between the conversation partner 2 and the wall 3 behind it, the conversation partner 2 is displayed as if viewed from the right side.

  In order to realize this conventional technique, two techniques of detecting the viewpoint position / attitude of the user 1 with respect to the display 4 and generating / displaying the video of the conversation partner 2 in accordance with the viewpoint position of the user 1 are important. Proposed to separate a person and background from a captured 2D video, generate a 3D video with depth by multilayering, and project and display it on the display surface according to the user's viewpoint position. .

  However, in the video display method described in Patent Literature 1, although the positional relationship and size between the user and the background can be expressed correctly, if the background has a three-dimensional structure, the viewpoint of the user 1 is Since the background does not change even if it changes, there is a problem that the background state cannot be expressed correctly. Such a problem can be solved by generating a background image corresponding to the line-of-sight direction. As a technique for solving such a problem, a virtual viewpoint image generation method for generating a video from a viewpoint at a desired position from a plurality of camera images is known (for example, see Patent Document 2).

Japanese Patent No. 5237234 Japanese Patent No. 4052331

  However, in the virtual viewpoint image generation method described in Patent Document 2, since the depth position is estimated by matching a plurality of images and the image is calculated, the depth cannot be acquired depending on the texture state of the image, There is a problem that the error is too large. In the virtual viewpoint image generation method described in Patent Document 2, the image quality is improved in consideration of the likelihood of the depth position, but an image from a specific viewpoint with a quality that gives a high sense of reality is created. There is a problem that it is difficult to do.

  The present invention has been made in view of such circumstances, and an image generation method, an image generation apparatus, and an image generation program capable of generating an image viewed from a desired viewpoint position with a quality that gives a high sense of presence. The purpose is to provide.

  The present invention provides a plurality of imaging means for imaging a subject from different positions, an image selection means for selecting and outputting images taken by the plurality of imaging means, and a viewpoint position setting means for setting a viewpoint position for the subject. And a selection step of selecting a plurality of imaging means having a close positional relationship between the subject and the viewpoint position and a positional relationship between the subject and the imaging means. A plurality of images output by the selected image pickup means are selected by the image selection means, and a transmission step of transmitting the selected plurality of images via a transmission path; An image generation step of generating an image of the subject viewed from the viewpoint position set by the viewpoint position setting means based on the plurality of images. The features.

  According to the present invention, in the image generation step, the luminance value of the target pixel of the image to be generated is a value obtained by weighted averaging from the luminance values of the target pixel of the plurality of images captured by the plurality of selected imaging units. It is characterized by doing.

  According to the present invention, in the image generation step, a pixel at a shifted position is selected as the target pixel according to a shift between a pixel position of an image to be generated and pixel positions of a plurality of images captured by the imaging unit. A weighted average of luminance values of the selected target pixels is obtained.

  The present invention provides a plurality of imaging means for imaging a subject from different positions, a viewpoint position setting means for setting a viewpoint position with respect to the subject, a positional relationship between the subject and the viewpoint position, and a relationship between the subject and the imaging means. Selection means for selecting a plurality of the imaging means having a close positional relationship, image selection means for selecting and outputting images picked up by the plurality of imaging means, and image selection for output of the selected imaging means A plurality of selections by means, and transmission means for transmitting the selected plurality of images via a transmission line, and setting by the viewpoint position setting means based on the plurality of images transmitted via the transmission line And an image generation means for generating an image of the subject viewed from the viewpoint position.

  The present invention is an image generation program for causing a computer to execute the image generation method.

  According to the present invention, there is an effect that an image viewed from a desired viewpoint position can be generated with a quality that gives a high presence. In addition, when the shooting location and the display location are separated, a transmission path is provided between the imaging means and the image generation means, and only necessary images are transmitted among the images taken by the plurality of imaging means. Since transmission is performed by a channel, it is possible to perform communication with high image quality even when a low-capacity transmission channel is used.

It is a block diagram which shows the structure of the 1st Embodiment of this invention. It is explanatory drawing which shows the outline of the image generation process which the image generation part 5 shown in FIG. 1 performs. It is a flowchart which shows the processing operation of the image generation part 5 shown in FIG. It is a figure which shows the relationship between the weighted average of two images, and an outline position. It is explanatory drawing which shows the outline of an image generation process in case there exists a viewpoint position behind a camera position. It is a figure which shows the relationship between the weighted average of two images, and an outline position. It is explanatory drawing which shows the outline of the image generation process which the image generation part 5 shown in FIG. 1 performs. It is a block diagram which shows the structure of the 4th Embodiment of this invention. It is a figure which shows the relationship between the weighted average of two images in case a weight is negative, and an outline position. It is a conceptual diagram which shows the image of the video display by a prior art.

<First Embodiment>
Hereinafter, an image generation apparatus according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, reference numeral 1 denotes a subject for image generation. Reference numeral 2 denotes a virtual screen that displays an image to be generated. Reference numerals 3-1 and 3-2 denote a plurality of cameras that capture an image of the subject 1. Although only two cameras are illustrated in FIG. 1, three or more cameras may be provided as necessary. In order to improve the quality of an image to be generated, it is desirable to have a large number of cameras. It is desirable that the plurality of cameras be arranged in a plane parallel to the screen 2. Reference numeral 4 denotes a viewpoint position setting unit that detects the viewpoint position of a person who views the generated image and sets the viewpoint position of the image to be generated. Reference numeral 5 denotes an image generation unit that generates an image of the subject 1 viewed from the viewpoint position set by the viewpoint position setting unit 4. Reference numeral 6 denotes a display device that displays a generated image generated by the image generation unit 5.

Next, the principle of image generation of the image generation unit 5 shown in FIG. 1 will be described with reference to FIG.
FIG. 2 is an explanatory diagram showing an outline of image generation processing performed by the image generation unit 5 shown in FIG. FIG. 2 shows a cross section perpendicular to the screen 2 and shows an outline of a method of calculating the luminance of a certain pixel on the screen 2. A method for generating an image when viewed from the viewpoint position shown in FIG. 2 will be described. The luminance of the target pixel shown in FIG. 2 is calculated from an image captured from a camera position in the vicinity of the light beam from which the light from the target pixel reaches the viewpoint position. In the example illustrated in FIG. 2, an image at a viewpoint position set from images captured by two cameras, a first camera position and a second camera position, is generated. The luminance L of the target pixel is set such that the luminance (pixel value) of the pixel of the image captured from the first camera position and the second camera position is L1 and L2, and light rays are emitted from the first camera position and the second camera position, respectively. If the distances up to d1 and d2 are determined by the equation (1). Here, the values of d1 and d2 are calculated by calculating as coordinates X1 and X2 of the camera position and coordinates X0 of the viewpoint position as d1 = X0−X1 and d2 = X2−X0.

In the case of a color image, the same calculation may be performed for each primary color. Further, in order to reduce the calculation amount, only the green color having the highest resolution among the three primary colors may be calculated, and the pixel values of one camera image may be used for the other primary colors. Further, when an image is provided with a luminance signal and a color difference signal as in YPbPr, the same calculation may be performed only for the luminance signal, and information of one camera may be used for the color difference signal. The distance between the camera and the light beam is the length of the perpendicular line drawn from the camera position to the light beam, but when the camera is arranged in a lattice shape, the distance between the intersection of the surface forming the lattice and the light beam and the camera position is used. You may approximate. Although the number of nearby cameras is two, it may be more than two. In this case, assuming that the number of nearby cameras is n, the luminance L of the target pixel is obtained by equation (2).

  If the viewpoint position is the same as the camera position, equations (1) and (2) become unstable. In this case, the luminance value of the camera image arranged at the viewpoint position is used as the luminance L of the target pixel as it is. Use it.

  Next, the processing operation of the image generation unit 5 shown in FIG. 1 will be described with reference to FIG. FIG. 3 is a flowchart showing the processing operation of the image generation unit 5 shown in FIG. The processing operation shown in FIG. 3 is a processing operation for generating one still image, but it is also possible to deal with a moving image by continuously repeating the processing operation shown in FIG.

  First, the image generation unit 5 selects one target pixel in the image on the screen to be generated (step S1). Then, the image generation unit 5 calculates a ray (straight line) connecting the selected target pixel and the viewpoint position set by the viewpoint position setting unit 4 (step S2).

  Next, the image generation unit 5 selects a camera located in the vicinity of the obtained light beam (step S3). Here, the selection of the cameras located in the vicinity is performed by (1) calculating the distance from the light ray for each camera position and selecting a predetermined number in order of closeness. It is possible to apply techniques such as selecting the number, (3) arranging the cameras in a grid, and selecting the camera position corresponding to the vertex of the polygon formed by the unit cells of the grid.

  Next, the image generation unit 5 performs a weighted average of the luminance of the target pixel on the screen of the selected camera image with a predetermined weight using the distance from the light ray as a parameter, and obtains the luminance value of the target pixel (step). S4). Then, the image generation unit 5 determines whether or not an uncalculated pixel remains (step S5). If it remains, the process returns to step S1 and repeats the calculation of other target pixels. finish.

  An image created by this processing operation is an image captured from a plurality of camera positions. This is a weighted average of images shifted due to differences in camera positions. When the width of the shift is small, the contour position continuously changes depending on the weighting ratio as shown in FIG. 4, so that an image of an appropriate contour position that matches the viewpoint position is generated. FIG. 4 is a diagram illustrating the relationship between the weighted average of two images and the contour position.

  Although the case where the camera position and the viewpoint are aligned on the same straight line has been described here, they may be different. FIG. 5 is an explanatory diagram when the viewpoint position is behind the camera position. In such a case, image generation may be performed according to the processing operation shown in FIG. According to the processing operation shown in FIG. 3, the weighted average image changes depending on the pixel. The first target pixel shown in FIG. 5 is an image at the first and second camera positions, the second target pixel is the second, the third camera position image, the third target pixel is the third, Although the image is at the fourth camera position, the image can be generated by the same processing operation even if the camera position changes back and forth.

  Assume that the images captured from each camera position used in the present embodiment are obtained by projecting the images captured at the respective camera positions onto the screen surface. Such an image can be easily obtained by capturing an image from a predetermined position with a general camera and performing keystone correction corresponding to a deviation of the optical axis of the camera from the normal line of the screen. Alternatively, the image may be taken with the optical axis of the camera parallel to the screen normal, and a portion corresponding to the display area of the screen may be cut out. In addition, the camera optical axis is parallel to the screen normal, and a shiftable lens is used to capture and capture the offset between the center of the screen display area and the optical axis, and the portion corresponding to the display area is cut out. May be.

  As described above, in generating an image viewed from a desired viewpoint position, when generating an image of a specific viewpoint from images of a plurality of cameras, depth information that causes image deterioration is not used. It is possible to generate an image viewed from a desired viewpoint position with a quality that is given.

<Second Embodiment>
Next, an image generation apparatus according to a second embodiment of the present invention will be described. Since the apparatus configuration in the second embodiment is the same as the apparatus configuration shown in FIG. 1, detailed description thereof is omitted here. The difference between the image generation apparatus in the second embodiment and the image generation apparatus in the first embodiment is that the weighting is suitable when the amount of image shift due to the camera position is large. When the image shift is large as shown in FIG. 6, the contour position is not proportional to the weight as shown in FIG. 4, but the person perceives that it changes sensitively as shown in FIG. That is, the weight of the image at the camera position closest to the viewpoint position is mainly used as 0.5 or more, and the images at other camera positions in the vicinity are averaged with a smaller weight. As a result, an image having an appropriate contour position can be generated without conspicuous multiple images.

<Third Embodiment>
Next, an image generating apparatus according to a third embodiment of the present invention will be described. Since the apparatus configuration in the third embodiment is the same as the apparatus configuration shown in FIG. 1, detailed description thereof is omitted here. The image generation processing operation is the same as the processing operation shown in FIG. The image generation apparatus according to the third embodiment is an image generation method suitable for a case where the screen position and the object (subject 1) position of the image are different. In this case, if the images at different camera positions are superimposed as they are, the shift of the image on the screen is superimposed as it is, and the multiplexed image becomes an unsightly image. In the weighted average process (step S4) in the processing operation shown in FIG. 3, the position on the object (subject 1) corresponding to the pixel of interest is calculated, and the image object (first camera position, second camera position) Multiple images are suppressed by weighted averaging of the luminance of pixels corresponding to positions on the subject 1). That is, as shown in FIG. 7, in the weighted average processing (step S4) in the image generation processing shown in FIG. 3, the first camera position and the second camera position are calculated at the first pixel position. Calculation is performed by adding an offset corresponding to the shift amount and the second shift amount. FIG. 7 is an explanatory diagram showing an outline of image generation processing performed by the image generation unit 5. Thereby, it is possible to easily perform luminance calculation for suppressing multiple images.

  Note that the calculation method is not limited to the calculation method described above, and another calculation method that can obtain an equivalent result may be used. For example, the screen position in the first to third embodiments is calculated assuming the object (subject) position of the present embodiment, and the obtained image is shifted by a parallax corresponding to the difference between the subject position and the screen position. It may be.

  Next, the result of displaying an image at a desired viewpoint position generated by the image generation apparatus according to the first to third embodiments described above on the display device 6 will be described. The viewpoint position of the user (the person who views the image) is detected, the viewpoint position is set by the viewpoint position setting unit 4, and an image corresponding to the viewpoint position is displayed on the display device 6 by the image generation unit 5. Since the image displayed on the display device 6 is an image corresponding to the viewpoint position, it has a high sense of orientation and a highly realistic display. Further, by making the display image displayed on the display device 6 the actual size of the subject 1, it was possible to obtain a higher sense of realism. Also, when the image is generated according to the position of the left and right eyes of the user and the images are presented to each of the left and right eyes using 3D glasses, the image can be viewed with a high sense of presence as if there is a real thing. I was able to. Furthermore, when a naked eye 3D display device is used as a display device and an image corresponding to each viewpoint position is generated and displayed, it is as if there is a real thing without wearing special equipment such as 3D glasses. I was able to see images with such a high sense of reality.

<Fourth Embodiment>
Next, an image generating apparatus according to a fourth embodiment of the present invention will be described. FIG. 8 is a block diagram showing a configuration of an image generation apparatus according to the fourth embodiment. The image generation apparatus shown in FIG. 8 assumes a case where a place where shooting is performed and a place where display is performed are far apart. 8, parts that are the same as the parts shown in FIG. 1 are given the same reference numerals, and descriptions thereof are omitted. The apparatus shown in this figure is different from the apparatus shown in FIG. 1 in that a video switch 7 is newly provided, and a first point (a point where a camera is installed) and a second point (a point where a display device is installed). Is connected by a transmission line 8. The transmission line 8 may be a public network. With this configuration, the image generation unit 51 sends video selection signals for selecting videos from a plurality of cameras (here, six cameras 3-1 to 3-6) via the transmission path 8 to the video switch 7. Is output. Based on the viewpoint position information output from the viewpoint position setting unit 4, the image generation unit 51 specifies a camera that outputs a video to be acquired, and outputs a video selection signal so as to select a video output by the camera. .

  A plurality of cameras 3-1 to 3-6 and a video switch 7 are provided at a first point where photographing is performed, and a viewpoint position setting unit 4, an image generation unit 51, and a display device 6 are provided at a second point. It is done. A video selection signal is transmitted from the image generation unit 51 at the second point to the video switch 7 at the first point via the transmission path 7 to select a video from a desired camera. From the first point to the second point, only the image selected by the image switch 7 among the camera images output by the six cameras 3-1 to 3-6 is transmitted via the transmission path 8. For example, when the viewpoint position set by the viewpoint position setting unit 4 is between the camera 3-3 and the camera 3-4, the camera images of the camera 3-3 and the camera 3-4 are selected and transmitted. As the video switch, a commercially available video matrix switch having a remote control function can be used.

  The configuration shown in FIG. 8 is an example in which the number of videos is reduced most and the two videos closest to the viewpoint position are sent, but four systems may be sent to the viewpoint positions of both eyes, or more camera videos may be sent. . Further, when the viewpoint is behind the camera, a wide range of cameras is required. However, since a narrow camera interval is not necessary, the interval may be selected from a wide range of cameras and may be thinned and transmitted.

  With this configuration, image generation by the image generation unit 51 is performed on the second point side where the observer is present, so that there is no delay in transmitting the observer's position information, and the response to changes in the observer position is fast. Display can be realized. In addition, since the camera video is selected on the first point side, it can be transmitted at high speed even if the bandwidth of the transmission line 8 is narrower than when all the video is sent.

  In the configuration shown in FIG. 8, a time delay may occur until the input image is switched due to the transmission delay of the transmission path 8 or the delay of the video switch 7, but the extrapolation is generally performed as shown in FIG. FIG. 9 is a diagram illustrating a relationship between a weighted average of two images and a contour position when the weight is negative. For this reason, even when the viewpoint is outside the camera, it is possible to provide a video with a substantially high sense of realism by the same processing.

  As described above, in the generation of an image at a desired viewpoint position, an image is generated by weighted average of images captured at positions near a straight line (light ray) connecting the viewpoint position and the position on the screen at a predetermined ratio. I did it. As the predetermined ratio, an amount that is inversely proportional to the distance between the straight line (light ray) and the imaging position is used as a weight. In addition, as a predetermined ratio, the weight for an image having the closest distance between the straight line (light ray) and the imaging position is increased. Further, the plurality of captured images are shifted by a predetermined amount determined from the relationship between the light beam and the imaging position.

  With this configuration, since depth information is not used in generating an image at a desired viewpoint position, a high-quality image can be generated. Further, when the shift between the target images to be weighted average is not so large, the edge of the object can be expressed at the correct position by human perceptual characteristics. In addition, when the difference between the images of the target to be weighted average is relatively large, the edge of the object is represented at the correct position by human perceptual characteristics, and it is possible to suppress the edge from becoming an unsightly double image. Can do. In addition, even when the screen position and the depth direction processing of the subject are different, it is possible to prevent multiple images. Thereby, it is possible to realize generation and display of a highly realistic expression image.

  You may make it implement | achieve the image generation apparatus in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  The present invention can be applied to applications where it is indispensable to generate an image at a desired viewpoint position from an existing image.

  DESCRIPTION OF SYMBOLS 1 ... Subject, 2 ... Screen, 3-1 to 3-6 ... Camera (imaging means), 4 ... Viewpoint position setting part, 5, 51 ... Image generation part, 6 ...・ Display device, 7 ... Video switch, 8 ... Transmission path

Claims (5)

A plurality of imaging means for imaging the subject from different positions; an image selection means for selecting and outputting images captured by the plurality of imaging means; and a viewpoint position setting means for setting a viewpoint position for the subject. An image generation method performed by an image generation apparatus,
A selection step of selecting a plurality of the imaging means having a positional relationship between the subject and the viewpoint position and a positional relationship between the subject and the imaging means;
A transmission step of selecting a plurality of images output by the selected imaging means by the image selection means and transmitting the selected plurality of images via a transmission path;
An image generated by extrapolation using a negative weight, based on the plurality of images transmitted through the transmission path, from the viewpoint position set by the viewpoint position setting unit and viewed from the viewpoint position. An image generation method comprising: a generation step. In the image generation step, a value obtained by weighted averaging from luminance values of target pixels of a plurality of images captured by the selected plurality of imaging means is set as a luminance value of the target pixels of the image to be generated. The image generation method according to claim 1 . In the image generation step, the pixel at the shifted position is selected as the target pixel in accordance with the shift between the pixel position of the image to be generated and the pixel positions of the plurality of images captured by the imaging unit, and the selected pixel The image generation method according to claim 2 , wherein a weighted average of luminance values of the target pixel is obtained. A plurality of imaging means for imaging the subject from different positions;
Viewpoint position setting means for setting a viewpoint position with respect to the subject;
A selection means for selecting a plurality of the imaging means having a positional relationship between the subject and the viewpoint position and a positional relationship between the subject and the imaging means;
Image selection means for selecting and outputting images picked up by a plurality of the image pickup means;
A transmission unit that selects a plurality of images output by the selected imaging unit by the image selection unit, and transmits the selected plurality of the images via a transmission path;
An image generated by extrapolation using a negative weight, based on the plurality of images transmitted through the transmission path, from the viewpoint position set by the viewpoint position setting unit and viewed from the viewpoint position. An image generation apparatus comprising: generation means. The computer, the image generation program for executing the image generating method according to any one of claims 1 to 3.

Images