JP4018273B2 - Video processing apparatus, control method therefor, and storage medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a video processing apparatus, a control method thereof, and a storage medium, and more particularly to a video processing apparatus that transmits a captured video to a remote terminal via a predetermined network line, a control method thereof, and a storage medium.
[0002]
[Prior art]
There is a technology that enables video information captured by a video camera or the like to be transmitted to a remote terminal via a so-called network typified by the Internet, and that the captured video can be displayed live on the remote terminal, that is, a video can be displayed. It has been put into practical use.
[0003]
Considering this technology, a technique for transmitting two images taken by two video cameras arranged in parallel to a remote location and displaying a stereo image having a stereoscopic effect at a remote location will be considered.
[0004]
In general, in order to construct this type of photographing system, it may be considered to use a camera module built in a home video camera supplied at low cost.
[0005]
[Problems to be solved by the invention]
However, a camera module or the like built in a home video camera often cannot release the AE (automatic exposure) function. Therefore, there is no problem when transmitting a pair of stereo images obtained by two video cameras, which is not a problem at all when transmitting images obtained by one video camera. There is. The reason is that, since the AE function of the video camera cannot be canceled, images shot with different AE values are transmitted.
[0006]
Even if photographing can be performed with the same AE value or the AE function can be canceled, the same luminance signal is not always obtained due to individual differences among the camera modules. When a pair of stereo images obtained by such a video camera is transmitted as it is, when reproducing the stereo image on a remote terminal, present images with different brightness to the left and right eyes of the observer. It is inconvenient because it does not wipe off the feeling of incongruity.
[0007]
[Means for Solving the Problems]
The present invention has been made in view of the above points, and by unifying the brightness of stereo images obtained by two systems of imaging means and transmitting the same to a remote terminal, the remote terminal operator feels comfortable. It is an object of the present invention to provide an image processing apparatus, a control method thereof, and a storage medium that make it possible to provide an image.
[0008]
A video processing apparatus for distributing two video information constituting a stereo image obtained from two systems of imaging means via a predetermined network,
Input means for inputting each of the two videos to be stereo images obtained by the imaging means as video data and storing them in a memory;
Detecting means for detecting a positional deviation amount in units of pixels in the entire video data of the other video data with respect to one video data, based on a cross-correlation value between the two input video data;
In order to reduce the brightness difference between the one of the picture data constituting a stereo image and the other image data, of the other image data and the luminance signal value of the pixel at the coordinate position of the coordinate system of the video data of the one A difference value with a luminance signal value of a pixel at a coordinate position shifted from the coordinate system by the amount of the position shift is obtained for each pixel of the one video data, and the whole of the one video data is obtained from the obtained plurality of difference values. A correction means for obtaining an average value of the difference values as a luminance value correction amount, and correcting brightness of all pixels of the other video data based on the luminance value correction amount ;
Comprising the a hand of video data, by combining the video data corrected the other hand by the correction means, to generate a single video data, and transmitting means for transmitting via the network.
[0009]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.
[0010]
The basic concept of this embodiment will be briefly described as follows.
[0011]
In a video distribution device (camera server) capable of capturing in real time two video signals (stereo video signals) obtained by two video cameras arranged in parallel, the two captured images are compared, The brightness is corrected so that the corresponding parts of the image have the same brightness value, and the two corrected images are recombined as a single image by reducing and combining them. A stereo image is transmitted to a terminal (client).
[0012]
Hereinafter, a configuration for realizing this and its operation will be described.
[0013]
FIG. 1 is a block diagram of a video distribution apparatus according to this embodiment.
[0014]
The hardware includes imaging devices 1-1 and 1-4 and a video processing device 1-8. The video processing device 1-8 has hardware for capturing video from a video camera, but can be realized by a general-purpose information processing device (for example, a personal computer) and software.
[0015]
The image capturing device 1-1 (the same applies to the image capturing device 1-4) includes an image capturing unit 1-2 (1-5) that inputs an image and an exposure control unit 1-3 (1-) that automatically determines an exposure state during photographing. 6). Here, as the parameters for determining the exposure state, shutter speed, iris, gain, white balance, and the like can be considered, but are not particularly limited here. In the following description, these are simply referred to as exposure parameters for the sake of simplicity.
[0016]
The video output signal of the imaging unit 1-2 (1-5) has an NTSC format, a YC separation method, and the like, and the video is transmitted through the I / O unit 1-13 included in the video processing device 1-8. The data are sequentially stored in the RAM 1-10 in 1-8. At this time, the I / O 1-13 is assumed to have two systems of suitable interfaces that can capture a video called a capture board or the like in real time.
[0017]
The exposure control unit 1-3 (1-6) has a function of automatically adjusting the brightness balance with respect to the entire screen being shot. The exposure control unit 1-3 (1-6) refers to the luminance signal obtained from the imaging unit 1-2 (1-5) through the control signal line, or sets an optimal exposure parameter calculated based on the information. It is possible to instruct the imaging unit 1-2 (1-5) to use and shoot.
[0018]
An imaging device 1-1 (1-4) including an imaging unit 1-2 (1-5) and an exposure control unit 1-3 (1-6) uses a camera module built in a home video camera. This can be easily realized.
[0019]
The video processing device 1-8 uses a RAM 1-10 for loading and executing the CPU 1-9 that controls the entire device, a work area for the CPU 1-9, an OS, an application that functions as a camera server, and the like. First, the following configuration is provided.
[0020]
Reference numeral 1-11 denotes a ROM that stores a boot program and BIOS, and reference numeral 1-12 denotes a secondary storage device (hard disk device or the like) that stores an OS, a program that functions as a camera server, and the like. Reference numeral 1-13 denotes an I / O that performs communication with the photographing apparatus 1-1 (1-14), and reference numeral 1-14 denotes a communication control unit that transmits a stereo image to a remote place through the network 1-19. Reference numeral 1-18 denotes a bus for connecting the above processing units. The video processing device 1-8 can be realized by a general-purpose computer as described above. The bus 1-18 is connected to a keyboard 1-15, a pointing device 1-16 such as a mouse, and a video monitor 1-17.
[0021]
The video processing device 1-8 performs processing as described later on the two systems of video obtained from the imaging devices 1-1 and 1-4 to generate a stereo video, and through a network 1-19, a remote terminal ( A process for transmitting a stereo image to a client.
[0022]
The basic principle of this embodiment will be described with reference to FIG.
[0023]
FIG. 2A shows a situation where two video cameras arranged in parallel are shooting a scene at the same time. Reference numerals 2-1 and 2-2 denote video cameras, which correspond to the photographing apparatuses 1-1 and 1-4 in FIG. 2-3 represents a scene shot by two video cameras, and 2-4 and 2-5 represent video signals for scenes shot by the video cameras 2-1 and 2-2, respectively. Represents a video.
[0024]
Since the two video cameras 2-1 and 2-2 have different shooting positions, the appearance of the scene 2-3 is slightly different in the images 2-4 and 2-5. Furthermore, the appropriate exposure determination results of the exposure control units 1-3 and 1-6 included in the video cameras 2-1 and 2-2 are different, or due to individual differences between the video cameras 2-1 and 2-2 themselves, In the images 2-4 and 2-5, the brightness of the entire screen is not the same.
[0025]
FIG. 2B shows the principle of generating a stereo image by inputting the two video signals captured in FIG. 1A to the video processing device 1-8.
[0026]
2-6 shown corresponds to the video processing apparatus 1-8, and schematically shows how the two systems of video input into the apparatus are processed and output. Reference numerals 2-7 and 2-8 are two images captured by the two video cameras 2-1 and 2-2, captured by the I / O 1-13, and stored in the RAM 1-10. A video obtained by examining the correspondence between videos using a known image processing technique such as obtaining a correlation between which the cross-correlation value is maximum between the two systems of videos, and correcting the corresponding locations to the same brightness. Are 2-9 and 2-10.
[0027]
Next, 2-11 and 2-12 are obtained by compressing these videos to a half size only in the vertical direction by a known image processing technique. Finally, the two compressed images 2-11 and 2-12 are integrated into one image 2-13, and this finally generated image 2-13 is transmitted to the remote site via the network. Called to the terminal.
[0028]
The program that realizes the above processing is in the secondary storage 1-12, and is loaded into the RAM 1-10 and executed by the CPU 1-9. Of course it is good.
[0029]
Now, in the remote terminal (which can be configured by a general-purpose information processing apparatus) that has received the video 2-13, the processing as described above is reversed. That is, the video 2-13 is separated in the vertical direction, and the obtained two systems of video are respectively expanded in the vertical direction. The two systems of videos thus obtained are presented as a pair of stereo videos. Various methods such as a head-mounted display, a lenticular screen, a liquid crystal shutter, and deflecting glasses are conceivable as a method of presentation, but this is not an essential part of the present invention and is not particularly limited.
[0030]
As described above, by providing a means for comparing and correcting the luminance signals before transmitting the stereo video obtained by the two video cameras over the network, a pair of signals obtained from the two video cameras is provided. Since the brightness of the stereo signal can be unified, it is possible to provide a stereo image with less sense of incongruity to the remote location of the transmission destination.
[0031]
FIG. 3 is a flowchart showing the basic processing procedure of the video processing apparatus of this embodiment. The overall flow is to unify the brightness of the two video images taken by the two video cameras and recombine and transmit one video by combining the two videos.
[0032]
First, two systems of video captured using the two video cameras 1-1 and 1-4 are input to the video processing device 1-8 (step S301). This is performed via the I / O 1-1 provided in the video processing device 1-8. The two input videos are immediately stored in the RAM 1-10 (or the secondary storage 1-12) (step S302). Next, in step S303, the correlation is obtained by using the cross-correlation values for the two systems of videos stored in the RAM 1-10 (or the secondary storage 1-12). Here, the correlation between the two videos is extracted by calculating the cross-correlation value between the two videos and obtaining the maximum correlation. Next, in step S304, brightness correction is performed so that the two images obtained have the same brightness using the correspondence relationship. In this method, the brightness correction amount of the video is calculated by comparing the luminance information of the corresponding part in the video, and the brightness correction of one (or both) video is performed using the correction amount.
[0033]
Next, the two video images with the same brightness are reduced to half the size in the vertical direction (step S305), and the reduced two video images are combined in the vertical direction and recombined as one video image (step S305). S306). These processes can also be realized by a known image processing technique. Thus, the generated stereo image is transmitted to a remote place via the network 1-19 (step SS307).
[0034]
Note that the purpose of performing the processes in steps S305 and S306 is that the two systems of video are originally a pair of stereo videos, so if these are transmitted separately, it is necessary to synchronize the two systems of video on the receiving side. This is to reduce the occurrence of an unfavorable situation for stereo video, such as when it occurs or one video is lost in the network being transmitted.
[0035]
In the process of step S304, there are several methods for correcting the brightness of both videos based on the correspondence between the two systems of videos. A simple example will be described below.
[0036]
When the correspondence that maximizes the cross-correlation value of two images is obtained, how much the pixel position of one image (hereinafter referred to as image A) is deviated from the other image (hereinafter referred to as image B) Only one piece of information is required for the entire image. The amount of this displacement is represented as (δx, δy). If the pixel of the image A is represented as A (x, y) and the pixel of the image B is represented as B (x, y), A (x, y) corresponds to B (x + δx, y + δy) (of course. Therefore, the image A is raster-scanned, and the difference between the luminance signals of A (x, y) and B (x + δx, y + δy) for each pixel is e (x, y). ), And the average value of e (x, y) in the entire video is obtained, whereby the luminance value correction amount for all the pixels of the video B can be calculated.
[0037]
The above is an example of a method for correcting the brightness of a video when a correspondence relationship that maximizes the cross-correlation value between videos is used. Of course, this is an example of an extremely simple process and a more appropriate method. Therefore, improvement in correction accuracy can be expected.
[0038]
<Second Embodiment>
A second embodiment will be described. In the second embodiment, in addition to the first embodiment, a DSP (Digital Signal Processor) is added in order to execute the processing of the video signal by hardware. As described above, by executing image processing with relatively high calculation cost by hardware, the processing speed of the video signal is increased, and real-time performance can be improved.
[0039]
FIG. 4 is a block diagram showing a hardware configuration of the second embodiment. Only the parts different from FIG. 1 will be described below. In FIG. 4, a DSP 1-7 is provided between the photographing apparatuses 1-1 and 1-4 and the video processing apparatus 1-8. The DSP 1-7 is connected to the image capturing apparatuses 1-1 and 1-4 via video signal lines, captures two systems of images captured by the image capturing apparatuses 1-1 and 1-4, and is described in the first embodiment. It has a function to execute such video processing at high speed by hardware.
[0040]
The basic principle and processing procedure of the second embodiment are the same as those of the first embodiment shown in FIGS. However, since it is possible to perform high-speed video processing using the DSP 1-7, it is possible to implement an image processing method having a higher calculation cost than that of the first embodiment. In the first embodiment, the cross-correlation value is used as a method for obtaining the association between videos. However, in the second embodiment, a method for obtaining the association using a stereo correspondence search is adopted.
[0041]
In the first embodiment, in the processing procedure (S303), the cross-correlation value is used as a method for obtaining correspondence between videos. This method has a relatively low calculation cost and can be executed at high speed, but has the disadvantage that only one correspondence can be extracted for the entire video. That is, since the photographing positions of the two photographing apparatuses 4-1 and 4-4 are different, the appearance of the scene is also slightly changed, and it is impossible that the entire image can be accurately handled without any contradiction. Therefore, in the second embodiment, in the processing procedure (S303), a stereo correspondence search capable of obtaining a correspondence relationship in units of one pixel (or a block of a certain size) instead of the entire video is used. This is because the present invention is originally intended to transmit a pair of stereo images obtained from two imaging devices arranged in parallel to a remote place. The effectiveness can also be expected from the fact that several conditions that are advantageous for realizing a high stereo correspondence search are already satisfied. Since the stereo correspondence search is a known image processing technique as well as the cross-correlation, a description of a specific algorithm will be omitted.
[0042]
As described above, the introduction of DSP and stereo correspondence search can be expected to realize lightness correction with higher accuracy at higher speed.
[0043]
<Third Embodiment>
A third embodiment will be described. In order to correct the brightness of the images obtained from the two video cameras, the cross correlation value is used in the first embodiment, and the stereo correspondence search is used in the second embodiment. In the present embodiment, in order to reduce the processing cost, the brightness is corrected using only the histogram conversion of the video without obtaining the correspondence between the videos.
[0044]
The hardware configuration and basic principle of the third embodiment are the same as those of the first embodiment shown in FIGS. Further, since the processing procedure of the third embodiment shown in FIG. 5 is similar to that of the first embodiment shown in FIG. 3, only the different parts will be described below.
[0045]
Steps S503 and S504 of the processing procedure of the third embodiment are portions corresponding to steps SS303 and S304 in the processing procedure of the first embodiment. In the first and second embodiments, the correspondence between the images is shown. In the third embodiment, as described above, a histogram is obtained (step SS503), and one (or both) image histogram conversion is performed. Is performed to correct the brightness (step S504). Since the histogram conversion is also a known image processing technique, detailed description thereof is omitted.
[0046]
As described above, the calculation cost can be reduced by correcting the brightness difference between the images using a simple method without obtaining the correspondence between the images, so that it is not necessary to introduce expensive equipment such as a DSP. A certain amount of brightness correction can be realized.
[0047]
In the embodiment, the correction target video is described as being performed on a video obtained by one predetermined photographing device (one video camera). However, this may be determined dynamically. . For example, if there are two videos A and B and the overall luminance (total value) of the video A is smaller than the video B, it means that the dynamic range of the video A is narrow. In other words, video B has a wider dynamic range. When correction is performed, the wide range includes the characteristics of the narrow range. In this case, the video B is the correction target.
[0048]
A part of the first, third, and second embodiments is also realized by software. As described above, since the video processing apparatus described in the embodiment can be realized by a general-purpose information processing apparatus, the present invention records a program code of software that realizes the functions of the above-described embodiment. Needless to say, this can also be achieved by supplying the storage medium to the system or apparatus, and the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium.
[0049]
In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.
[0050]
As a storage medium for supplying the program code, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.
[0051]
Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.
[0052]
Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.
[0053]
As described above, according to the present embodiment, the brightness of a pair of stereo signals is obtained by comparing and correcting the luminance signals before transmitting the stereo image obtained by the two video cameras to the network. Therefore, it is possible to provide a stereo image with less sense of incongruity to a remote place of the transmission destination.
[0054]
【The invention's effect】
As described above, according to the present invention, the brightness of the images obtained by the two systems of imaging means is unified and transmitted to the remote terminal, so that the remote terminal operator does not feel uncomfortable. It becomes possible to provide.
[Brief description of the drawings]
FIG. 1 is a block diagram of a schematic configuration of an embodiment.
FIG. 2 is an explanatory diagram of a basic principle of the embodiment.
FIG. 3 is a flowchart showing an operation processing procedure of the apparatus in the embodiment.
FIG. 4 is a schematic block diagram of a second embodiment.
FIG. 5 is a flowchart showing an operation processing procedure in the third embodiment.

Claims (4)

A video processing apparatus for distributing two video information constituting a stereo image obtained from two systems of imaging means via a predetermined network,
Input means for inputting each of the two videos to be stereo images obtained by the imaging means as video data and storing them in a memory;
Detecting means for detecting a positional deviation amount in units of pixels in the entire video data of the other video data with respect to one video data, based on a cross-correlation value between the two input video data;
In order to reduce the brightness difference between the one of the picture data constituting a stereo image and the other image data, of the other image data and the luminance signal value of the pixel at the coordinate position of the coordinate system of the video data of the one A difference value with a luminance signal value of a pixel at a coordinate position shifted from the coordinate system by the amount of the position shift is obtained for each pixel of the one video data, and the whole of the one video data is obtained from the obtained plurality of difference values. A correction means for obtaining an average value of the difference values as a luminance value correction amount, and correcting brightness of all pixels of the other video data based on the luminance value correction amount ;
Characterized in that it comprises the a hand of video data, by combining the video data corrected the other hand by the correction means, to generate a single video data, and transmitting means for transmitting via the network A video processing device.   The video processing apparatus according to claim 1, wherein in the combining process in the transmitting unit, both of the two systems of images obtained by the imaging unit are reduced to a half size and combined. A control method for a video processing apparatus for distributing two video information constituting a stereo image obtained from two systems of imaging means via a predetermined network,
An input step of inputting each of the two videos to be a stereo image obtained by the imaging means as video data and storing it in a memory;
A detection step of detecting a positional deviation amount in units of pixels in the entire video data of the other video data with respect to one video data based on a cross-correlation value between the two input video data;
In order to reduce the brightness difference between the one of the picture data constituting a stereo image and the other image data, of the other image data and the luminance signal value of the pixel at the coordinate position of the coordinate system of the video data of the one A difference value with a luminance signal value of a pixel at a coordinate position shifted from the coordinate system by the amount of the position shift is obtained for each pixel of the one video data, and the whole of the one video data is obtained from the obtained plurality of difference values. A correction step of obtaining an average value of the difference values as a luminance value correction amount, and correcting brightness of all pixels of the other video data based on the luminance value correction amount ;
Comprising: the image data of the hand, it said to combine the video data corrected the other hand in correcting step, to generate a single video data, and a transmission step of transmitting over the network A method for controlling the video processing apparatus. A storage medium storing a program code that functions as a video processing device that distributes, via a predetermined network, two pieces of video information constituting a stereo image obtained from two systems of imaging means when read and executed by a computer. And
Input means for inputting each of the two videos to be stereo images obtained by the imaging means as video data and storing them in a memory;
Detecting means for detecting a positional deviation amount in units of pixels in the entire video data of the other video data with respect to one video data, based on a cross-correlation value between the two input video data;
In order to reduce the brightness difference between the one of the picture data constituting a stereo image and the other image data, of the other image data and the luminance signal value of the pixel at the coordinate position of the coordinate system of the video data of the one A difference value with a luminance signal value of a pixel at a coordinate position shifted from the coordinate system by the amount of the position shift is obtained for each pixel of the one video data, and the whole of the one video data is obtained from the obtained plurality of difference values. A correction means for obtaining an average value of the difference values as a luminance value correction amount, and correcting all pixels of the other video data based on the luminance value correction amount ;
Wherein the hand of the video data, by combining the video data corrected the other hand by the correction means, to generate a single video data, the program code functions as a transmitter for transmitting via the network A storage medium characterized by storing.

Images