JP5742886B2 - Stereoscopic video encoding apparatus, stereoscopic video decoding apparatus, stereoscopic video encoding system, stereoscopic video encoding program, and stereoscopic video decoding program - Google Patents

Description

  The present invention relates to a stereoscopic video encoding device, a stereoscopic video decoding device, a stereoscopic video encoding system, a stereoscopic video encoding program, and a stereoscopic video decoding program. For example, the left-eye image and the right-eye image (in this specification, The image data may be simply referred to as an image), and can be applied to transmission / reception of moving image data in which a stereoscopic effect can be obtained by projecting each eye.

  As a method for recognizing a three-dimensional moving image to human vision, a right eye image (moving image) and a left eye image (moving image) taken from different directions are prepared, and the right eye image is used as the right eye and the left eye. A method of causing an observer to recognize a stereoscopic effect using a display device that projects a commercial image to the left eye is widely known (see, for example, Patent Document 1). As a method of transmitting a moving image having such parallax, there is a method of generating and transmitting data obtained by encoding and multiplexing a right-eye image and a left-eye image.

  As a conventional method of multiplexing and encoding a right-eye image and a left-eye image, the right-eye image and the left-eye image are each compressed in half in the horizontal direction (or vertical direction) and then in the horizontal direction (or vertical direction). There is a method in which the images are combined and combined into one image, and the combined image is encoded. In this conventional method, information on the right-eye image and the left-eye image at the same time is included in one piece of encoded data, so even if a packet loss occurs due to an error during transmission, the left-right viewpoint (for the right eye) Synchronization between the image and the left-eye image) can be ensured.

JP 2006-135783 A

  However, in the conventional method described above, the distance between corresponding points in the composite image is far away, so that when a packet loss occurs, the difference between corresponding regions in the decoded image increases due to error propagation. Can happen. Here, the corresponding point or region refers to a point or region that is perceived at the same position when viewed stereoscopically (when the image for the right eye and the image for the left eye are viewed with such a display device, respectively). . Since the composite image combines the right-eye image and the left-eye image, there are corresponding points and regions in the composite image.

With reference to FIG. 13, it will be described that a large difference may occur between corresponding regions due to error propagation. Assume that the region Y is lost due to packet loss in the composite image of the nth frame shown in FIG. 13A, in which the reduced image of the right-eye image and the reduced image of the left-eye image are combined in parallel. Note that regions XL and XR in FIGS. 13A and 13B are regions corresponding to the left-eye image and the right-eye image.

  In the nth frame, due to the lack of the area Y due to packet loss, the data in this area Y is subjected to interpolation processing, data diversion of the previous frame, etc. As a result, it differs from the original data obtained by decoding Become. That is, noise is mixed. In the (n + 1) th frame, when decoding, if the region XL of the left-eye image refers to the region Y and the region XR of the right-eye image does not refer to the region Y, the decoded data in the region XL has noise. Although mixed, noise is not mixed in the decoded data in the region XR. In this case, a difference occurs between the left and right viewpoints (between the right-eye image and the left-eye image) regardless of the corresponding region.

  In a stereoscopic image, when there is a large difference between corresponding regions between the left and right viewpoints, the stereoscopic effect is not felt. Furthermore, when the image difference between the viewpoints is large, a phenomenon called visual field struggle occurs in the observer's brain. This is a phenomenon in which, when there is a large difference between the video entering the right eye and the video entering the left eye, the brain recognizes only the video from either viewpoint and perceives it by switching in a short time. When such a phenomenon occurs, flickering is felt, resulting in an unpleasant stereoscopic image.

  Therefore, even if a part of information obtained by multiplexing and encoding the right-eye image and the left-eye image is lost, the decoded right-eye image and left-eye image can feel a three-dimensional effect and do not cause a visual field conflict. A stereoscopic video encoding apparatus, a stereoscopic video decoding apparatus, a stereoscopic video encoding system, a stereoscopic video encoding program, and a stereoscopic video decoding program are desired.

The stereoscopic video encoding apparatus according to the first aspect of the present invention includes (1) image multiplexing means for generating a multiplexed image by multiplexing a left-eye image and a right-eye image having the same number of vertical and horizontal pixels, and (2) generated multiplexing And (3) the image multiplexing unit is configured to store information on a predetermined range of horizontal lines of the left-eye image and the right-eye image in the same vertical direction with a predetermined number of pixels. By alternately mixing in the horizontal direction in units, information on the horizontal lines of the multiple images having the same vertical direction is generated . (4) The predetermined range of the horizontal lines is the same for all the horizontal lines. It is determined according to the amount of parallax between the left-eye image and the right-eye image, and is a range in which there are areas corresponding to the left-eye image and the right-eye image .
The stereoscopic video encoding apparatus according to the second aspect of the present invention includes (1) image multiplexing means for generating a multiplexed image by multiplexing a left-eye image and a right-eye image having the same number of vertical and horizontal pixels, and (2) generated multiplexing And (3) the image multiplexing means stores information on a predetermined range of horizontal lines of the left-eye image and the right-eye image having the same vertical direction in a predetermined number of pixels. The horizontal line information of the multiple images having the same vertical direction is generated by alternately mixing in the horizontal direction in units. (4) The predetermined range of the horizontal line is set for each horizontal line or several lines. The left-eye image and the right-eye image, which are different for each set of horizontal lines, are determined according to the amount of parallax between the left-eye image and the right-eye image of the horizontal line or the horizontal line set to be processed. Is a range where there is an area corresponding to ( ) A search is performed for a region having a high correlation between the left-eye image and the right-eye image, and among the plurality of parallax amounts obtained by the search, the maximum parallax amount or the highest-frequency parallax amount is determined as the left-eye image and Disparity parameter calculation means for setting the amount of parallax for the right-eye image is provided.

The stereoscopic video decoding apparatus according to the third aspect of the present invention receives the encoded data of the multiplexed image formed by the stereoscopic video encoding apparatus according to the first aspect of the present invention and receives the parallax from the stereoscopic video encoding apparatus of the first aspect of the present invention. be one amount of information is input, (1) decoding means for decoding the encoded data of the input multiplexed image, (2) for multiple image obtained by decoding, parallax amount inputted The image demultiplexing means for separating and extracting the left-eye image and the right-eye image by performing demultiplexing of the inverse processing of the image multiplexing means in the stereoscopic video encoding device of the first aspect of the present invention by applying the above information It is characterized by providing.
The stereoscopic video decoding apparatus according to the fourth aspect of the present invention receives the encoded data of the multiplexed image formed by the stereoscopic video encoding apparatus according to the second aspect of the present invention and receives the parallax from the stereoscopic video encoding apparatus of the second aspect of the present invention. (1) decoding means for decoding the encoded data of the input multiple image, and (2) the amount of parallax input to the multiple image obtained by decoding. The image demultiplexing means for separating and extracting the left-eye image and the right-eye image by performing demultiplexing of the inverse processing of the image multiplexing means in the stereoscopic video encoding device of the second aspect of the present invention by applying the above information It is characterized by providing.

Fifth stereoscopic video encoding system of the present invention is characterized by having a three-dimensional video encoding apparatus of the first aspect of the present invention, a stereoscopic video decoding apparatus of the third invention.
The stereoscopic video encoding system of the sixth aspect of the present invention includes the stereoscopic video encoding apparatus of the second aspect of the present invention and the stereoscopic video decoding apparatus of the fourth aspect of the present invention.

According to a seventh aspect of the present invention, there is provided a three-dimensional video encoding program for generating a multiple image by multiplexing a left-eye image and a right-eye image having the same number of vertical and horizontal pixels, and having the same vertical direction. The information of the predetermined range of the horizontal line of the left eye image and the right eye image of the left eye image and the right eye image is alternately mixed in the horizontal direction in units of a predetermined number of pixels, and the horizontal line information of the multiple images having the same vertical direction is obtained. And (2) a stereoscopic video encoding program that functions as an encoding unit that encodes the generated multiplexed image , and (3) a predetermined range of the horizontal line includes all horizontal directions It is the same in the line, and is determined according to the amount of parallax between the left-eye image and the right-eye image, and the range corresponding to the left-eye image and the right-eye image exists regardless of the parallax. Features.
According to an eighth aspect of the present invention, there is provided a stereoscopic video encoding program for generating a multiple image by multiplexing a left-eye image and a right-eye image having the same number of vertical and horizontal pixels and having the same vertical direction. The information of the predetermined range of the horizontal line of the left eye image and the right eye image of the left eye image and the right eye image is alternately mixed in the horizontal direction in units of a predetermined number of pixels, and the horizontal line information of the multiple images having the same vertical direction is obtained. (2) encoding means for encoding the generated multiplexed image; and (3) searching for a region having a high correlation between the left-eye image and the right-eye image, and obtaining by the search. A stereoscopic video encoding program that functions as a parallax parameter calculation unit that uses a parallax amount of the left-eye image and the right-eye image as a parallax amount of the left-eye image and the right-eye image among a plurality of parallax amounts obtained. , (4) The predetermined range of horizontal lines is different for each horizontal line or for each set of horizontal lines consisting of several lines, and the left eye image and the right eye of the horizontal line or horizontal line set to be processed It is determined according to the amount of parallax of the image for use, and is a range in which areas corresponding to the image for the left eye and the image for the right eye exist regardless of the parallax.

The stereoscopic video decoding program of the ninth aspect of the present invention receives the encoded data of the multiplexed image formed by the stereoscopic video encoding apparatus of the first aspect of the present invention, and receives the parallax from the stereoscopic video encoding apparatus of the first aspect of the present invention. A computer mounted on a device to which the quantity information is input is input to (1) decoding means for decoding encoded data of the input multiple image, and (2) input to the multiple image obtained by decoding. Image demultiplexing that separates and extracts a left-eye image and a right-eye image by performing demultiplexing by inverse processing of the image multiplexing means in the stereoscopic video encoding device of the first aspect of the present invention by applying the parallax information It is made to function as a means.
The stereoscopic video decoding program according to the tenth aspect of the present invention receives encoded data of a multiplexed image formed by the stereoscopic video encoding apparatus according to the second aspect of the present invention and receives parallax from the stereoscopic video encoding apparatus according to the second aspect of the present invention. A computer mounted on a device to which the quantity information is input is input to (1) decoding means for decoding encoded data of the input multiple image, and (2) input to the multiple image obtained by decoding. Image demultiplexing that separates and extracts a left-eye image and a right-eye image by performing demultiplexing of inverse processing of the image multiplexing means in the stereoscopic video encoding device of the second aspect of the present invention by applying the information on the amount of parallax It is made to function as a means.

  According to the present invention, even if a part of information obtained by multiplexing and encoding the right-eye image and the left-eye image is lost, the decoded right-eye image and left-eye image can feel a stereoscopic effect, and the field of view A stereoscopic video encoding device, a stereoscopic video decoding device, a stereoscopic video encoding system, a stereoscopic video encoding program, and a stereoscopic video decoding program that do not cause a conflict can be provided.

1 is a block diagram illustrating a configuration of a stereoscopic video communication system including a stereoscopic video encoding system according to a first embodiment. It is explanatory drawing of the function of the multiplexing part of FIG. It is a flowchart which shows operation | movement of the stereo image transmission apparatus of FIG. It is a flowchart which shows operation | movement of the three-dimensional video receiver of FIG. It is explanatory drawing of the reason which introduced the parallax parameter in the 2nd-4th embodiment, and the function of the multiplexing part using a parallax parameter. It is a block diagram which shows the structure of the stereoscopic video communication system containing the stereoscopic video encoding system of 2nd Embodiment. It is a flowchart which shows operation | movement of the stereoscopic video transmission apparatus of FIG. It is a flowchart which shows operation | movement of the three-dimensional video receiver of FIG. It is a block diagram which shows the structure of the stereoscopic video communication system containing the stereoscopic video encoding system of 3rd Embodiment. It is a block diagram which shows the structure of the stereoscopic video communication system containing the stereoscopic video encoding system of 4th Embodiment. It is a flowchart which shows operation | movement of the stereo image transmission apparatus of FIG. It is a flowchart which shows operation | movement of the three-dimensional video receiver of FIG. It is explanatory drawing of the subject of the conventional method which multiplexes and encodes the image for right eyes, and the image for left eyes.

(A) First Embodiment Hereinafter, a first embodiment of a stereoscopic video encoding device, a stereoscopic video decoding device, a stereoscopic video encoding system, a stereoscopic video encoding program, and a stereoscopic video decoding program according to the present invention will be described. This will be described with reference to the drawings.

  The stereoscopic video encoding system of the first embodiment is incorporated in a stereoscopic video communication system.

(A-1) Configuration of First Embodiment FIG. 1 is a block diagram showing a configuration of a stereoscopic video communication system including the stereoscopic video encoding system of the first embodiment.

  In FIG. 1, the stereoscopic video communication system 1 includes a stereoscopic video transmission device 10 and a stereoscopic video reception device 20.

  The stereoscopic video transmission device 10 includes an image multiplexing unit 11, an encoding unit 12, and a transmission unit 13. Here, the image multiplexing unit 11 and the encoding unit 12 are components of the stereoscopic video encoding apparatus of the first embodiment. At least the image multiplexing unit 11 and the encoding unit 12 may be constructed by connecting various circuits in hardware, and a general-purpose device or unit having a CPU, a ROM, a RAM, and the like is used as a predetermined program. It may be constructed to realize the corresponding function by executing.

  The image multiplexing unit 11 multiplexes (combines) the input left-eye image PICL and right-eye image PICR, and outputs the multiplexed image to the encoding unit 12. Here, the pair of left-eye images PICL and right-eye images PICR are images for stereoscopic viewing that have different viewpoints at the same time (positions at which shooting was performed).

  The left-eye image PICL and the right-eye image PICR may be those just obtained by the imaging device, may be read from a recording medium or the like, and are transmitted and received from other devices. Also good. That is, the input method of the left-eye image PICL and the right-eye image PICR is not limited.

  Further, the pair of left-eye image PICL and right-eye image PICR may be images of luminance signals (Y) as long as they are the same type of image, and may be images of one of the color difference signals (Cb, Cr). It may be an image of any color signal (R, G, B). The sizes of the left-eye image PICL and the right-eye image PICR are not limited. For example, vertical and horizontal 1080 lines × 1920 pixels, vertical and horizontal 540 lines × 960 pixels, vertical and horizontal 480 lines × 720 pixels, vertical and horizontal 288 lines × 360 pixels, vertical and horizontal 240 lines × 360 pixels, vertical and horizontal 144 lines × 180 pixels, vertical and horizontal 120 lines × 180 pixels Any size such as vertical and horizontal 72 lines × 90 pixels may be used. Also, the number of bits per pixel is not limited.

  As shown in FIGS. 2A1 and 2A, the image multiplexing unit 11 converts the left-eye image PICL and the right-eye image PICR into a plurality of vertical stripe image portions SL-1 to SL, respectively, along a vertical dividing line. After dividing into SL-N, SR-1 to SR-N (N is an integer of 2 or more), as shown in FIG. 2B, the vertical stripe image portions SL-1 to SL-N of the left eye image PICL A multiple image CPIC in which the vertical stripe image portions SR-1 to SR-N of the right eye image PICR are alternately arranged in the horizontal direction is formed and output.

  The multiple image CPIC is an image in which the vertical stripe image portions SL-1 to SL-N of the left eye image PICL and the vertical stripe image portions SR-1 to SR-N of the right eye image PICR are alternately arranged in the horizontal direction. Therefore, the distance between the pixels in the corresponding region of the left and right viewpoints is short (the pixels in the corresponding region of the left and right viewpoints are close to each other). For this reason, even when packet loss or the like occurs during communication, there is a high possibility that the corresponding areas of the left and right viewpoints are lost.

  The left-eye image PICL and the right-eye image PICR are divided into a plurality of vertical stripe image portions SL-1 to SL-N and SR-1 to SR-N along the vertical dividing line as follows. According to the idea. The two human eyes are separated in the horizontal direction (left and right direction), and the left and right eyes feel a three-dimensional effect. Therefore, in a stereoscopic image, the horizontal direction has a higher weight than the vertical direction, and the horizontal stripe image portion is If there is an alternating arrangement, the missing areas corresponding to the left and right viewpoints exist in the vertical direction, which occurs when packet loss or the like occurs during communication. In the case where they are alternately arranged, there is a corresponding missing region in the left and right viewpoints that occurs when packet loss or the like during communication occurs, and the effects described later are effective.

  Here, the widths of the vertical stripe image portions SL-1 to SL-N and SR-1 to SR-N are not limited. For example, the vertical stripe image portions SL-1 to SL-N and SR-1 to SR-N may have a width of one pixel or a plurality of pixels. In the case of a plurality of images, the specific number of pixels may be determined in consideration of the block size determined by interpolation processing or encoding of the missing region. For example, half the number of pixels on one side of the block used for motion vector search may be selected as the width of the vertical stripe image portion. In the section of the operation described later, the description will be made assuming that the vertical stripe image portions SL-1 to SL-N and SR-1 to SR-N have a width of one pixel.

  The encoding unit 12 encodes the multiplexed image CPIC of the left-eye image PICL and the right-eye image PICR output from the image multiplexing unit 11 according to a predetermined encoding method. Here, the encoding method is not limited. An encoding scheme such as H.264 / MPEG-4 AVC can be applied.

  The transmission unit 13 forms transmission data (for example, a packet) from the encoded data output from the encoding unit 12 and transmits the transmission data to the stereoscopic video reception device 20.

  Here, the transmission path between the stereoscopic video transmitting apparatus 10 and the stereoscopic video receiving apparatus 20 may be a dedicated line or a communication network that can be used by many people. The communication method (for example, presence / absence of encryption, communication speed, transmission path coding method, wired / wireless communication, etc.) between the stereoscopic video transmission device 10 and the stereoscopic video reception device 20 is not limited. Further, the transmission unit 13 may form transmission data from, for example, encoded data for luminance signals and two encoded data for color difference signals. Further, encoded data related to a plurality of frames may be inserted into transmission data.

  The stereoscopic video reception device 20 includes a reception unit 21, a decoding unit 22, and an image demultiplexing unit 23. Here, the decoding unit 22 and the image demultiplexing unit 23 are components of the stereoscopic video decoding device according to the first embodiment. At least the decoding unit 22 and the image demultiplexing unit 23 may be constructed by connecting various circuits in hardware, and a general-purpose device or unit having a CPU, a ROM, a RAM, or the like may be a predetermined program. It may be constructed to realize the corresponding function by executing.

  The receiving unit 21 receives transmission data from the stereoscopic video transmission device 10, extracts encoded data, and provides the encoded data to the decoding unit 22.

  The decoding unit 22 performs decoding on the encoded data, obtains a multiplexed image CPIC shown in FIG.

  The image demultiplexing unit 23 decomposes the multiplexed image CPIC into vertical stripe portions SL-1, SR-1,..., SL-N, SR-N, and converts the odd-numbered vertical stripe portions SL-1,. The left eye image PICL is reproduced by juxtaposing in the horizontal direction, and the right eye image PICR is reproduced by juxtaposing even-numbered vertical stripe portions SR-1,..., SR-N in the horizontal direction.

  The reproduced left-eye image PICL and right-eye image PICR are provided to the display device 30 for display of stereoscopic video, for example.

(A-2) Operation of the First Embodiment Next, the operation of the stereoscopic video communication system 1 including the stereoscopic video encoding system of the first embodiment will be described.

  First, the operation of the stereoscopic video transmission apparatus 10 will be described with reference to the flowchart of FIG. FIG. 3 shows the operation of multiplexing and transmitting the left-eye image PICL and the right-eye image PICR for one frame in synchronization, and the operation shown in FIG. 3 constitutes a stereoscopic moving image. Repeated for the number of frames.

  First, the image multiplexing unit 11 acquires a pair of synchronized left-eye image PICL and right-eye image PICR (left and right viewpoint images) from a video source such as a photographing camera or a recording file (step S101). Here, the acquisition of the left eye image PICL and the right eye image PICR may be performed in parallel or sequentially.

  Next, as illustrated in FIG. 2, the image multiplexing unit 11 includes vertical pixel columns (vertical stripe image portions) SL-1 to SL-N and SR-1 to SR of the left-eye image PICL and the right-eye image PICR. The left-eye image PICL and the right-eye image PICR are multiplexed in a format in which −N is alternately arranged in the horizontal direction (step S102). That is, the vertical pixel rows SL-1 to SL-N obtained from the left-eye image PICL are arranged in odd-numbered rows of the multiple image CPIC, and the vertical pixel rows SR-1 to SR-N obtained from the right-eye image PICR. Are multiplexed so that the images are arranged in even-numbered columns of the multiple image CPIC. As described above, the multiple image CPIC is such that the corresponding areas of the left-eye image PICL and the right-eye image PICR are close to each other.

  The image multiplexing unit 11 gives the obtained multiplexed image CPIC to the encoding unit 12 (step S103). At this time, the encoding unit 12 encodes the multiple image CPIC according to a predetermined encoding method (step S104), and gives the obtained encoded data to the transmission unit 13 (step S105).

  The transmission unit 13 assembles transmission data from the given encoded data (step S106), and transmits the transmission data to the stereoscopic video reception device 20 (step S107). For example, in step S106, the transmission unit 13 divides the given encoded data into sizes that can be transmitted to the transmission path, packetizes them according to RTP (Real-time Transport Protocol), and forms packets that become transmission data. In such a case, a plurality of packets are formed and transmitted from the left-eye image PICL and the right-eye image PICR for one frame.

  Next, the operation of the stereoscopic video receiving device 20 will be described with reference to the flowchart of FIG. FIG. 4 shows the operation of demultiplexing the left-eye image PICL and the right-eye image PICR for one frame in synchronization based on the received data, and the operation shown in FIG. 4 forms a stereoscopic moving image. It is repeated as many times as the number of frames. In the following description, it is assumed that the transmission data is a packet.

The receiving unit 21 receives the packet sent from the transmitting side (step S201),
Header data and the like are removed from the received packet to obtain encoded data (step S202), and the obtained encoded data is provided to the decoding unit 22 (step S203).

  After decoding the input encoded data (step S204), the decoding unit 22 determines whether a decoded image for one frame including the current decoding information is obtained (step S205). When the decoded image for one frame is not obtained, the process returns to step S201, and when the decoded image for one frame is obtained, the decoding unit 22 converts the obtained decoded image into the image demultiplexing unit 23. (Step S206). Here, when the encoded data for one frame is distributed and inserted into a plurality of packets on the transmission side, the process from step S205 to step S201 is executed.

  The image demultiplexing unit 23 demultiplexes the left-eye image and the right-eye image from the decoded image (decoded multiple image) (step S207). This demultiplexing process is the reverse of the multiplexing process performed by the image multiplexing unit 11 (see FIG. 2). That is, only the odd-numbered vertical pixel columns of the decoded multiple image CPIC are collected to obtain the left-eye image PICL, and only the even-numbered vertical pixel columns of the multiplexed image CPIC are collected to obtain the right-eye image PICR.

  The image demultiplexing unit 23 gives the obtained left-eye image and right-eye image to the display device 30 (step S208), and causes stereoscopic display.

(A-3) Effects of the First Embodiment In the first embodiment, since the multiple images are formed by alternately arranging the vertical columns of the left-eye image and the right-eye image, the left-eye image and the right-eye image Corresponding regions between images are located in the vicinity of the multiple images.

  Therefore, it is in a close position in the encoded data obtained by encoding this multiplexed image. Even if a packet loss occurs in the middle of transmission, it is less likely that only one viewpoint video is correctly decoded and the other cannot be decoded correctly.

  Also, with respect to noise propagation, since a corresponding region exists in the vicinity, a region where a point with one viewpoint exists (for example, a macroblock) referred to a decoded image having an error at the time of decoding. However, if the corresponding point of the other viewpoint exists in the same region (macroblock), an error is referred to in the same manner, so that the corresponding points have the same noise. For this reason, when the video output from the stereoscopic video receiver is stereoscopically displayed, although the corresponding area is noisy, the difference between the viewpoints is small, resulting in an uncomfortable stereoscopic video due to a visual field conflict. Can be avoided.

(B) Second Embodiment Next, a stereoscopic video encoding device, a stereoscopic video decoding device, a stereoscopic video encoding system, and a stereoscopic video encoding program and a stereoscopic video decoding program according to a second embodiment of the present invention will be described. This will be described with reference to the drawings.

  The stereoscopic video encoding system of the second embodiment is incorporated in a stereoscopic video communication system.

(B-1) Configuration of the Second Embodiment In the stereoscopic video encoding system of the first embodiment, the predetermined width (for example, one pixel) of each of the left-eye image and the right-eye image in the multiplex processing before encoding. By alternately arranging the vertical stripe image portions in the horizontal direction, corresponding regions between the left-eye image and the right-eye image (left and right viewpoints) are arranged in the vicinity.

  However, depending on the distance of the object being photographed from the camera, the corresponding regions may not exist in the vicinity in the multiplex processing as in the first embodiment. For example, as the distance from the photographing camera to the object gets closer, the lateral distance between the article region in the left-eye image and the article region in the right-eye image increases, and the amount of parallax increases. The left-eye image PICL and the right-eye image PICR shown in FIGS. 5A1 and 5A2 are a pair of images having such a large parallax region. In the left-eye image PICL, the isosceles triangle-shaped article is at a position away from the left end by a distance LL, and in the right-eye image PICR, the isosceles triangle-shaped article is at a position away from the left end by a distance LR. The difference n = LL−LR between the distances LL and LR is quite large. As described above, regarding the regions in the left-eye image PICL and the right-eye image PICR where an object that is close to the photographing camera is located, even if the multiple processing of the first embodiment is performed, in the multiple image CPIC The distances between the corresponding regions in the left-eye image PICL and the right-eye image PICR are increased. In the example of FIG. 5, it is separated by a distance n.

  Therefore, it is possible that the influence of noise due to packet loss is different at each viewpoint (left-eye image PICL and right-eye image PICR). Even if the vertical stripe image portions of the left and right viewpoints (left eye image PICL and right eye image PICR) are alternately arranged in the horizontal direction, if the distance from the photographing camera is short, the vertical stripe image portions of the non-corresponding region are alternately arranged. Therefore, the coding efficiency is deteriorated.

  In view of the above points, the stereoscopic video encoding system according to the second embodiment to the fourth embodiment described later takes the image PICL for the left eye and the image for the right eye even when the article to be photographed is close to the photographing camera. This is an attempt to multiplex so that corresponding areas in the PICR are adjacent to each other in the multiplex image CPIC.

  FIG. 6 is a block diagram illustrating a configuration of a stereoscopic video communication system including the stereoscopic video encoding system according to the second embodiment. The configuration is the same as that of FIG. 1 according to the first embodiment, and the corresponding parts are the same. A reference numeral is attached.

  In FIG. 6, the stereoscopic video communication system 1A includes a stereoscopic video transmission device 10A and a stereoscopic video reception device 20A.

  The stereoscopic video transmission apparatus 10A includes an image multiplexing unit 11A, an encoding unit 12, a transmission unit 13A, and a multiplexing setting information unit 14. The encoding unit 12 is the same as that of the first embodiment. Here, the image multiplexing unit 11A, the encoding unit 12, and the multiplexing setting information unit 14 are components of the stereoscopic video encoding device of the second embodiment.

  The multiple setting information unit 14 newly provided in the second embodiment holds a parallax parameter designated in the form of user designation or the like. The parallax parameter is, for example, a lateral displacement amount n between corresponding regions in the left-eye image PICL and the right-eye image PICR (see FIG. 5). The parallax parameter may be set for each frame, may be set for each same scene of the moving image, or may be set to be common to the entire frame of the moving image. good. In the description of the operation described later, it is assumed that the parallax parameter is set for each frame.

  The image multiplexing unit 11A according to the second embodiment refers to the parallax parameter held in the multiplexing setting information unit 14, and in accordance with the parallax parameter, in the lateral direction with respect to the input left-eye image PICL. The non-alternating multi-region ARL1 and the alternating multi-region ARL2 are divided, and the input right-eye image PICR is divided into the alternating multi-region ARR2 and the non-alternating multi-region ARR1 in the horizontal direction, and further, FIG. As shown in (A1) and (A2), a plurality of vertical stripe image portions ARL2-1 to ARL2-M and ARR2-1 to ARR2-M (M is an integer of 2 or more) along the vertical dividing line. After that, as shown in FIG. 5B, the non-alternate multiple region ARL1 of the left eye image PICL, the vertical stripe image portions ARL2-1 to ARL2-M of the left eye image PICL, and the right eye image PICR. By arranging transverse interleaved region of the fringe image portion ARR2-1~ARR2-M, and a non-alternating multiple regions ARR1 of the right eye image PICR from the left, and outputs to form a multi-image CPICA.

  Note that the non-alternate multiple region ARL1 of the left-eye image PICL corresponds to a region that is captured in the left-eye image PICL but not in the right-eye image PICR, and the non-alternate multiple region ARR1 of the right-eye image PICR is the right-eye image. It corresponds to a region that is captured in the image PICR for the image but is not captured in the image PICL for the left eye. This corresponds to a region that is also shown in the image PICR.

  13 A of transmission parts which concern on 2nd Embodiment form transmission data (for example, packet), after multiplexing the encoding data output from the encoding part 12, and the parallax parameter currently hold | maintained at the multiplexing setting information part 14 Then, it is transmitted toward the stereoscopic video receiving device 20A. In the above description, the parallax parameter information is described as multiplexed on the transmission data of all frames. However, depending on the set validity period of the parallax parameter, the parallax parameter information may be transferred to the stereoscopic video reception device 20A by another method. You may make it give. For example, the parallax parameter may be multiplexed only on the encoded data of the first frame of a series of moving image data and provided to the stereoscopic video reception device 20A. Further, for example, a parallax parameter is given from the stereoscopic video transmitting apparatus 10A to the stereoscopic video receiving apparatus 20A in a negotiation period performed before the start of the operation of exchanging image data between the stereoscopic video transmitting apparatus 10A and the stereoscopic video receiving apparatus 20A. May be.

  The stereoscopic video reception device 20A includes a reception unit 21A, a decoding unit 22, an image demultiplexing unit 23A, and a demultiplexing setting information unit 24. The decoding unit 22 is the same as that of the first embodiment. Here, the decoding unit 22, the image demultiplexing unit 23A, and the demultiplexing setting information unit 24 are components of the stereoscopic video decoding device according to the second embodiment.

  The receiving unit 21A according to the second embodiment receives the data transmitted by the stereoscopic video transmitting apparatus 10A, demultiplexes the parallax parameter into encoded data, and gives the parallax parameter to the demultiplexing setting information unit 24. The converted data is given to the decoding unit 22.

  The demultiplexing setting information unit 24 newly provided in the second embodiment holds the parallax parameter given from the receiving unit 21A.

  The image demultiplexing unit 23A according to the second embodiment refers to the parallax parameter set in the demultiplexing setting information unit 24 and reverses the multiplexing process by the image multiplexing unit 11A described above with reference to FIG. Processing (demultiplexing processing) is performed to reproduce the left-eye image PICL and the right-eye image PICR (that is, the left and right viewpoint images).

(B-2) Operation of the Second Embodiment Next, the operation of the stereoscopic video communication system 1A including the stereoscopic video encoding system of the second embodiment will be described.

  First, the operation of the stereoscopic video transmission apparatus 10A will be described with reference to the flowchart of FIG. In FIG. 7, the same steps as those in FIG. 3 according to the first embodiment are denoted by the same reference numerals. FIG. 7 shows an operation of multiplexing and transmitting the synchronized left-eye image PICL and right-eye image PICR for one frame, and the operation shown in FIG. 7 constitutes a stereoscopic moving image. Repeated for the number of frames.

  First, the parallax parameter information held in the multiple setting information unit 14 is updated by external designation such as an instruction from the user (step S301).

  Then, the image multiplexing unit 11 obtains a pair of synchronized left-eye image PICL and right-eye image PICR (left and right viewpoint images) from a video source such as a photographing camera or a recording file (step S101), and FIG. As shown, according to the parallax parameter, the input left-eye image PICL and right-eye image PICR are divided into non-alternating multiple regions ARL1 and ARR1 and alternating multiple regions ARL2 and ARR2 in the horizontal direction. , Non-alternate multiple regions ARL1 of the left eye image PICL, vertical stripe image portions ARL2-1 to ARL2-M of the left eye image PICL, and vertical stripe image portions ARR2-1 to ARR2-M of the right eye image PICR in the horizontal direction The left eye image PICL and the right eye image are arranged so that the region and the non-alternate multiple region ARR1 of the right eye image PICR are arranged from the left side. The ICR multiplexing (step S302).

  The image multiplexing unit 11 gives the obtained multiple image CPICA to the encoding unit 12 (step S103), and the encoding unit 12 encodes the multiple image CPICA according to a predetermined encoding method (step S104), and obtained The encoded data is given to the transmission unit 13A (step S105).

  After acquiring the disparity parameter from the multiplex setting information unit 14 (step S303), the transmission unit 13A adds disparity parameter information to the end of the header portion of the encoded data output from the encoding unit 12, for example. Thus, the encoded data and the parallax parameter are multiplexed (step S304). As the header of the encoded data, the encoding information (sequence header) of the entire sequence, the encoding information (frame header) related to each frame, and the code related to each slice obtained by combining several macroblocks that are encoding units of moving images In the following description, it is assumed that a disparity parameter is included at the end of the frame header, although it may be added to the end of any header. Note that the encoded data may be multiplexed in another format such as adding a parallax parameter with a fixed-length code at the beginning of the encoded data.

  The transmission unit 13A assembles transmission data from the encoded data to which the given parallax parameter is added (step S106), and transmits the transmission data to the stereoscopic video reception device 20A (step S107).

  Next, the operation of the stereoscopic video reception device 20A will be described with reference to the flowchart of FIG. In FIG. 8, the same steps as those in FIG. 4 according to the first embodiment are denoted by the same reference numerals. FIG. 8 shows the operation of demultiplexing the left-eye image PICL and the right-eye image PICR for one frame in synchronization based on the received data, and the operation shown in FIG. 8 forms a stereoscopic moving image. It is repeated as many times as the number of frames. In the following description, it is assumed that the transmission data is a packet.

  The receiving unit 21A receives the packet sent from the transmitting side (step S201), demultiplexes the parallax parameter and the encoded data from the received packet (step S401), and is held in the demultiplexing setting information unit 24. The information on the existing parallax parameter is updated to the value of the received parallax parameter (step S402), and the obtained encoded data is given to the decoding unit 22 (step S203).

  After decoding the input encoded data (step S204), the decoding unit 22 determines whether a decoded image for one frame including the current decoding information is obtained (step S205). When the decoded image for one frame is not obtained, the process returns to step S201, and when the decoded image for one frame is obtained, the decoding unit 22 converts the obtained decoded image into the image demultiplexing unit 23A. (Step S206).

  The image demultiplexing unit 23A refers to the parallax parameters held in the demultiplexing setting information unit 24, and performs demultiplexing processing corresponding to the multiplexing processing of the image multiplexing unit 11A on the decoded image (decoded multiple image) (See FIG. 5) to obtain a left-eye image and a right-eye image (step S403).

  The image demultiplexing unit 23A gives the obtained left-eye image and right-eye image to the display device 30 (step S208), and stereoscopically displays them.

(B-3) Effects of the Second Embodiment According to the second embodiment, the left eye image and the corresponding points having the designated parallax are adjacent to each other when the user designates the parallax parameter. Since the right-eye image is multiplexed, an area having the designated parallax is adjacent to the area, so that no difference between noise viewpoints due to packet loss occurs in this area. For this reason, when the amount of parallax of a region that a person is interested in is specified as a parallax parameter, there is no difference between viewpoints with respect to the region of interest, so that it is possible to suppress a deterioration in subjective image quality. In addition, since the spatial correlation of the area having the designated parallax becomes high, it can be expected that the coding efficiency of this part is improved and the subjective quality is improved. As described above, there is an effect of suppressing the field-of-view struggle in the left-right difference and improving the coding efficiency with respect to the region having an arbitrary amount of parallax.

(C) Third Embodiment Next, a third embodiment of a stereoscopic video encoding device, a stereoscopic video decoding device, a stereoscopic video encoding system, a stereoscopic video encoding program, and a stereoscopic video decoding program according to the present invention will be described. This will be described with reference to the drawings.

  The stereoscopic video encoding system of the third embodiment is incorporated in a stereoscopic video communication system.

  In the second embodiment described above, by specifying a parallax parameter, the influence of noise due to packet loss in an area having the designated parallax is reduced. However, since it is necessary to specify a parallax parameter and the video changes with time, an appropriate parallax parameter also changes. In the third embodiment, a function for estimating a parallax parameter from an input image is added.

  FIG. 9 is a block diagram illustrating a configuration of a stereoscopic video communication system including the stereoscopic video encoding system according to the third embodiment. The configuration is the same as that of FIG. 1 according to the second embodiment, and the corresponding parts are the same. A reference numeral is attached.

  In FIG. 9, the stereoscopic video communication system 1B includes a stereoscopic video transmission device 10B and a stereoscopic video reception device 20A. The stereoscopic video receiving device 20A is the same as that of the second embodiment, and the functional description thereof is omitted.

  The stereoscopic video transmission device 10B includes a parallax parameter calculation unit 15 in addition to the image multiplexing unit 11A, the encoding unit 12, the transmission unit 13A, and the multiple setting information unit 14. The image multiplexing unit 11A, the encoding unit 12, the transmission unit 13A, and the multiplex setting information unit 14 are the same as those in the second embodiment, and description of their functions is omitted.

The parallax parameter calculation unit 15, based on the input left-eye image PICL and the right-eye image PICR, is shall give the multiplexing setting information unit 14 calculates the parallax parameters. For example, the parallax parameter calculation unit 15 estimates corresponding points of the left-eye image PICL and the right-eye image PICR by a method such as a block matching method. The parallax amount of each point is estimated by comparing pixel positions of corresponding points in the left-eye image PICL and the right-eye image PICR. Then, from the plurality of obtained parallax amounts, for example, the parallax amount (parallax parameter) to be given to the multiple setting information unit 14 is determined as follows. First, the largest amount of parallax is determined as the amount of parallax (parallax parameter) to be given to the multiple setting information unit 14. Second, the parallax amount with the highest frequency is determined as the parallax amount (parallax parameter) to be given to the multiple setting information unit 14.

  In the above, the case where the parallax amount (parallax parameter) to be given to the multiple setting information unit 14 is determined from the input left-eye image PICL and right-eye image PICR has been described. In addition to the left-eye image PICL and the right-eye image PICR Other information (hereinafter referred to as auxiliary information) may also be used for determining the amount of parallax (parallax parameter). Depth information can be used as auxiliary information. For example, the focal length of the imaging camera, the magnification at the time of zooming, and the like can be captured and used as the depth information. By using depth information in addition to the left-eye image PICL and the right-eye image PICR, the parallax amount search process can be speeded up. For example, the point with the largest amount of parallax according to the first method for determining the parallax parameter is the point with the shortest depth, so the other corresponding to one point of the left-eye image PICL and the right-eye image PICR. These image points are shifted by the amount of parallax determined by the depth information or a value in the vicinity thereof. Accordingly, the search range may be limited to a narrow range determined according to the depth information, and the parallax amount may be calculated.

  According to the third embodiment, since the parallax parameter is estimated from the input image for each frame, the parallax parameter is dynamically set even for a stereoscopic image in which a change in depth occurs with a change in time. can do.

  Here, when the parallax parameter to be set is the maximum parallax in the left and right viewpoint images, there is an effect of reducing the influence of noise and improving the coding efficiency for the region closest to rejection. Moreover, since such a region with a large amount of parallax is a region that appears in the foreground, an improvement in subjective image quality can be expected by improving the coding efficiency of this region and suppressing the binocular rivalry. Further, when the parallax parameter to be set is the largest in the amount of parallax in the left and right viewpoint images, the overall correlation between the left and right viewpoint images is increased, and the coding efficiency is improved.

(D) Fourth Embodiment Next, a fourth embodiment of a stereoscopic video encoding device, a stereoscopic video decoding device, a stereoscopic video encoding system, a stereoscopic video encoding program, and a stereoscopic video decoding program according to the present invention will be described. This will be described with reference to the drawings.

  The stereoscopic video encoding system of the fourth embodiment is incorporated in a stereoscopic video communication system.

(D-1) Configuration of Fourth Embodiment In the first to third embodiments, the same multiple processing is performed on the entire image. However, the amount of parallax in the left and right viewpoint images varies depending on the region. For example, a region with a small amount of parallax, that is, a region with a deep depth as a background is widely shown in the upper part of the image, and a region with a large amount of parallax, that is, an object existing in the foreground at the lower part of the image. There are times when the area of is widely reflected.

  In view of such a point, the stereoscopic video encoding system according to the fourth embodiment sets a parallax parameter for each line in the horizontal direction (not limited to one line but may be a plurality of lines). Perform multiple processing. Hereinafter, a case where the parallax parameter is set for each line will be described.

  FIG. 10 is a block diagram illustrating a configuration of a stereoscopic video communication system including the stereoscopic video encoding system according to the fourth embodiment. The same reference numerals are assigned to the same or corresponding parts as those in the above-described drawings. Show.

  In FIG. 10, the stereoscopic video communication system 1C includes a stereoscopic video transmission device 10C and a stereoscopic video reception device 20C.

  The stereoscopic video transmission apparatus 10C includes an image multiplexing unit 11C, an encoding unit 12, a transmission unit 13C, a multiple setting information unit 14C, and a parallax parameter calculation unit 15C. The encoding unit 12 is the same as that of the first embodiment. Here, the image multiplexing unit 11C, the encoding unit 12, the multiplexing setting information unit 14C, and the parallax parameter calculation unit 15C are constituent elements of the stereoscopic video encoding apparatus according to the fourth embodiment.

  The parallax parameter calculation unit 15C in the fourth embodiment may use auxiliary information based on the input left-eye image PICL and right-eye image PICR (as described in the third embodiment). ), The parallax parameter for each line is calculated, and the parallax parameter for each line is given to the multiplex setting information unit 14C. The parallax parameter calculation method for each line is almost the same as the parallax parameter calculation method for the entire image described in the third embodiment. That is, the points corresponding to the left eye image PICL and the right eye image PICR are estimated by a method such as a block matching method, and the pixel positions of the corresponding points in the left eye image PICL and the right eye image PICR are compared. Is estimated. Then, from the plurality of parallax amounts obtained for one line, for example, the largest parallax amount is determined as the parallax amount (parallax parameter) for each line to be given to the multiplex setting information unit 14, The parallax amount (parallax parameter) for each line given to the multiplex setting information unit 14 is determined.

  The multiplex setting information unit 14C in the fourth embodiment holds the parallax parameter for each line given from the parallax parameter calculation unit 15C.

  The image multiplexing unit 11C according to the fourth embodiment displays the left-eye image PICL and the right-eye image PICR while switching the multiplexing method for each line based on the parallax parameter for each line held in the multiplexing setting information unit 14C. Multiplexing is performed for each line, and a multiplexed image CPICCC is given to the encoding unit 12. The image multiplexing unit 11C refers to the parallax parameter of the line held in the multiplexing setting information unit 14 when a certain line becomes the target line of the multiplexing process, and inputs the left-eye image according to the parallax parameter. The PICL processing target line is divided into a non-alternating multiple region and an alternating multiple region in the horizontal direction, and the alternating multiple region in the horizontal direction with respect to the processing target line of the input right eye image PICR. After dividing into non-alternating multiple regions, the non-alternating multiple regions of the processing target line of the left-eye image PICL, the alternating multi-region pixels of the processing target line of the left-eye image PICL, and the processing target line of the right-eye image PICR alternately A multiple image is obtained by arranging a region where pixels of a multiple region are alternately arranged and a non-alternate multiple region of the processing target line of the right-eye image PICR from the left side. Forming the processing object line in PICA.

  The transmission unit 13C according to the fourth embodiment multiplexes the encoded data output from the encoding unit 12 and the parallax parameters for each line held in the multiplexing setting information unit 14C, and then transmits transmission data (for example, a packet). ) And transmitted to the stereoscopic video receiving device 20C.

  The stereoscopic video receiving device 20C includes a receiving unit 21C, a decoding unit 22, an image demultiplexing unit 23C, and a demultiplexing setting information unit 24C. The decoding unit 22 is the same as that of the first embodiment. Here, the decoding unit 22, the image demultiplexing unit 23C, and the demultiplexing setting information unit 24C are constituent elements of the stereoscopic video decoding apparatus according to the fourth embodiment.

  The receiving unit 21C according to the fourth embodiment receives the data transmitted by the stereoscopic video transmission device 10C, demultiplexes the parallax parameter for each line and the encoded data, and demultiplexes the parallax parameter for each line. This is given to the unit 24C, and the coded data is given to the decoding unit 22.

  The demultiplexing setting information unit 24C according to the fourth embodiment holds a parallax parameter for each line given from the receiving unit 21C.

  The image demultiplexing unit 23C according to the fourth embodiment refers to the parallax parameter for each line set in the demultiplexing setting information unit 24C, and reverses the multiplexing process for each line by the image multiplexing unit 11C described above ( The left-eye image PICL and the right-eye image PICR (that is, the left and right viewpoint images) are reproduced.

(D-2) Operation of the Fourth Embodiment Next, the operation of the stereoscopic video communication system 1C including the stereoscopic video encoding system of the fourth embodiment will be described.

  First, the operation of the stereoscopic video transmission apparatus 10C will be described with reference to the flowchart of FIG. In FIG. 11, the same steps as those in FIG. 7 according to the second embodiment are denoted by the same reference numerals. FIG. 11 shows an operation of multiplexing and transmitting the synchronized left-eye image PICL and right-eye image PICR for one frame, and the operation shown in FIG. 11 forms a stereoscopic moving image. Repeated for the number of frames.

  First, based on the input left-eye image PICL and right-eye image PICR (based on auxiliary information when auxiliary information is used), the parallax parameter calculation unit 15C calculates a parallax parameter for each line and performs multiplexing. The parallax parameter information for each line held in the setting information unit 14C is updated to the parallax parameters for each line obtained by calculation (step S501).

  Then, the image multiplexing unit 11C acquires a pair of left-eye images PICL and right-eye images PICR (left and right viewpoint images) (step S101), and the input left-eye image PICL according to the parallax parameter for each line N Then, the above-mentioned multiplexing processing for each line is performed on the image PICR for the right eye and the multiple image CPICCC of the image PICL for the left eye and the image PICR for the right eye is formed (step S502).

  The image multiplexing unit 11C gives the obtained multiple image CPICCC to the encoding unit 12 (step S103), and the encoding unit 12 encodes the multiple image CPICCC according to a predetermined encoding method (step S104) and obtains it. The encoded data is given to the transmission unit 13C (step S105).

  After acquiring the disparity parameter for each line from the multiplex setting information unit 14C (step S503), the transmission unit 13C, for example, the disparity parameter for each line to the end of the header portion of the encoded data output from the encoding unit 12 Is added, the encoded data and the parallax parameter for each line are multiplexed (step S504).

  The transmission unit 13C assembles transmission data from the encoded data to which the given parallax parameter for each line is added (step S106), and transmits the transmission data to the stereoscopic video reception device 20C (step S107).

  Next, the operation of the stereoscopic video reception device 20C will be described with reference to the flowchart of FIG. In FIG. 12, the same steps as those in FIG. 8 according to the second embodiment are denoted by the same reference numerals. FIG. 12 shows an operation of demultiplexing the left-eye image PICL and the right-eye image PICR for one frame in synchronization based on the received data, and the operation shown in FIG. 12 forms a stereoscopic moving image. It is repeated as many times as the number of frames. In the following description, it is assumed that the transmission data is a packet.

  The receiving unit 21C receives the packet sent from the transmitting side (step S201), demultiplexes the parallax parameter and the encoded data for each line from the received packet (step S601), and sends the demultiplexing setting information unit 24C to the demultiplexing setting information unit 24C. The stored parallax parameter information for each line is updated to the received parallax parameter value for each line (step S602), and the obtained encoded data is provided to the decoding unit 22 (step S203).

  After decoding the input encoded data (step S204), the decoding unit 22 determines whether a decoded image for one frame including the current decoding information is obtained (step S205). When the decoded image for one frame is not obtained, the process returns to step S201, and when the decoded image for one frame is obtained, the decoding unit 22 converts the obtained decoded image into the image demultiplexing unit 23C. (Step S206).

  The image demultiplexing unit 23C refers to the parallax parameter for each line held in the demultiplexing setting information unit 24C, and performs the multiplexing process for each line by the image multiplexing unit 11C on the decoded image (decoded multiple image). The left-eye image and the right-eye image are obtained by performing the demultiplexing process for each line corresponding to (step S603).

  The image demultiplexing unit 23C gives the obtained left-eye image and right-eye image to the display device 30 (step S208), and causes stereoscopic display.

(D-3) Effect of the fourth embodiment According to the fourth embodiment, the parallax parameter is set for each line, and the multiplexing process is switched for each line according to the parallax parameter for each line. It is possible to suppress the occurrence of a field-of-view struggle due to the influence of noise due to packet loss on left and right viewpoint images having various depths. Moreover, since an appropriate parallax parameter is set for each line, the correlation within the line is increased, and an improvement in coding efficiency can be expected.

(E) Other Embodiments In the description of each of the above embodiments, various modified embodiments have been referred to, but further modified embodiments as exemplified below can be given.

  In each of the above embodiments, the input left-eye image and right-eye image are multiplexed as they are, and the horizontal length of the multiplexed image is twice the horizontal length of the left-eye image and right-eye image. However, after the input left-eye image and right-eye image are each reduced by half in the horizontal direction, the multiple processing described in the above embodiments is performed on the two reduced images. Multiple images having the same horizontal length as the left-eye image and the right-eye image may be formed.

  In the fourth embodiment, one or a plurality of pixel line units is subjected to multiple processing according to the parallax parameter for each unit. However, the vertical width of the macroblock (for example, 8 pixels × 8 pixels) In the case of a macroblock, the parallax parameter may be calculated in units of 8 pixels in the vertical direction), and multiple processing may be performed in units of the width.

In each of the above embodiments, the encoded data of the multiplexed image of the left and right viewpoint images is transmitted and received between the two devices . However, the encoded data of the multiplexed image of the left and right viewpoint images is recorded on the recording medium, and the recording is performed. The technical idea of the present invention can also be applied when decoding encoded data reproduced from a medium.

  After the left and right viewpoint images of the luminance signal are multiplexed, encoding is performed to obtain luminance encoded data, and the left and right viewpoint images of the color difference signal are multiplexed and then encoded to obtain the color difference encoded data. If the technical idea of the fourth embodiment is applied, for example, the parallax parameter obtained for the luminance signal and the parallax parameter for each line may be used as they are in the color difference signal processing system.

1, 1A, 1B, 1C ... stereoscopic video communication system,
10, 10A, 10B, 10C ... stereoscopic video transmission device,
11, 11A, 11C ... image multiplexing unit, 12 ... encoding unit, 13, 13A, 13C ... transmission unit, 14, 14C ... multiple setting information unit, 15, 15C ... parallax parameter calculation unit,
20, 20A, 20C ... stereoscopic video receiver,
21, 21 </ b> A, 21 </ b> C... Receiving unit, 22... Decoding unit, 23, 23 </ b> A, 23 </ b> C image demultiplexing unit, 24, 24C.

Claims (12)

Image multiplexing means for generating a multiple image by multiplexing a left-eye image and a right-eye image having the same number of vertical and horizontal pixels;
Encoding means for encoding the generated multiple images,
The image multiplexing means mixes the information of the predetermined range of the horizontal line of the left-eye image and the right-eye image having the same vertical direction alternately in the horizontal direction in a predetermined number of pixels, and the vertical direction is the same. Generate information on the horizontal lines of the multiple images ,
The predetermined range of the horizontal line is the same for all horizontal lines, and is determined according to the amount of parallax between the left-eye image and the right-eye image, and corresponds to the left-eye image and the right-eye image. A stereoscopic video encoding apparatus characterized by being in a range in which there exists . The stereoscopic video encoding apparatus according to claim 1 , wherein information on a parallax amount of the left-eye image and the right-eye image is given from outside. There row the search for high correlation region of the left eye image and the right-eye image, among the plurality of parallax amounts obtained by the search, the parallax amount of the maximum parallax amount or maximum frequency, the image and for the left eye The stereoscopic video encoding apparatus according to claim 1 , further comprising a parallax parameter calculating unit that calculates the parallax amount information of the right-eye image. Image multiplexing means for generating a multiple image by multiplexing a left-eye image and a right-eye image having the same number of vertical and horizontal pixels;
Encoding means for encoding the generated multiple images,
The image multiplexing means mixes the information of the predetermined range of the horizontal line of the left-eye image and the right-eye image having the same vertical direction alternately in the horizontal direction in a predetermined number of pixels, and the vertical direction is the same. Generate information on the horizontal lines of the multiple images,
The predetermined range of the horizontal line is different for each horizontal line or for each set of horizontal lines consisting of several lines, and the left eye image and the right eye of the horizontal line or horizontal line set to be processed Is determined according to the amount of parallax of the image for use, and is a range where there are regions corresponding to the image for the left eye and the image for the right eye,
There row the search for high correlation region of the left eye image and the right-eye image, among the plurality of parallax amounts obtained by the search, the parallax amount of the maximum parallax amount or maximum frequency, the image and for the left eye A stereoscopic video encoding apparatus, comprising: a parallax parameter calculating unit that sets a parallax amount of the right-eye image . A stereoscopic video decoding device to which encoded data of a multiplexed image formed by the stereoscopic video encoding device according to claim 1 is input and parallax information is input from the stereoscopic video encoding device according to claim 1 . ,
Decoding means for decoding the encoded data of the input multiple images;
The left eye is demultiplexed by performing reverse processing of the image multiplexing means in the stereoscopic video encoding device according to claim 1 by applying the input parallax information to the multiplexed image obtained by decoding. A stereoscopic video decoding apparatus comprising: an image demultiplexing unit that separates and extracts the image for use and the image for the right eye. A stereoscopic video decoding device to which encoded data of a multiplex image formed by the stereoscopic video encoding device according to claim 4 is input and parallax amount information is input from the stereoscopic video encoding device according to claim 4. ,
Decoding means for decoding the encoded data of the input multiple images;
The left eye is demultiplexed by performing reverse processing of the image multiplexing means in the stereoscopic video encoding device according to claim 4 by applying the input parallax information to the multiplexed image obtained by decoding. And image demultiplexing means for separating and extracting the image for the right eye and the image for the right eye
A stereoscopic video decoding apparatus characterized by that. A stereoscopic video encoding system comprising the stereoscopic video encoding device according to claim 1 and the stereoscopic video decoding device according to claim 5 . A stereoscopic video encoding system comprising the stereoscopic video encoding device according to claim 4 and the stereoscopic video decoding device according to claim 6. Computer
A left-eye image and a right-eye image having the same number of vertical and horizontal pixels are multiplexed to generate a multiplexed image, and information on a predetermined range of horizontal lines of the left-eye image and the right-eye image having the same vertical direction Are multiplexed alternately in the horizontal direction in units of a predetermined number of pixels, and image multiplexing means for generating information on the horizontal lines of the multiple images having the same vertical direction;
A stereoscopic video encoding program for functioning as an encoding means for encoding a generated multiple image ,
The predetermined range of the horizontal line is the same for all the horizontal lines, and is determined according to the amount of parallax of the left-eye image and the right-eye image. Regardless of the parallax, the left-eye image and the right-eye line A stereoscopic video encoding program characterized in that a region corresponding to an image is present . Computer
A left-eye image and a right-eye image having the same number of vertical and horizontal pixels are multiplexed to generate a multiplexed image, and information on a predetermined range of horizontal lines of the left-eye image and the right-eye image having the same vertical direction Are multiplexed alternately in the horizontal direction in units of a predetermined number of pixels, and image multiplexing means for generating information on the horizontal lines of the multiple images having the same vertical direction;
Encoding means for encoding the generated multiple images;
The left eye image and the right eye image are searched for a region having a high correlation, and among the plurality of parallax amounts obtained by the search, the maximum parallax amount or the highest frequency parallax amount is determined as the left eye image and the above-described parallax amount. A stereoscopic video encoding program that functions as a parallax parameter calculation unit that sets a parallax amount of a right-eye image,
The predetermined range of the horizontal line is different for each horizontal line or for each set of horizontal lines consisting of several lines, and the left eye image and the right eye of the horizontal line or horizontal line set to be processed It is determined in accordance with the amount of parallax of the image for use, and is a range in which areas corresponding to the image for the left eye and the image for the right eye exist regardless of the parallax.
A stereoscopic video encoding program characterized by the above. The encoded data of the multiplexed image formed by the stereoscopic video encoding device according to claim 1 is input and mounted on a device to which information on the amount of parallax is input from the stereoscopic video encoding device according to claim 1. Computer
Decoding means for decoding the encoded data of the input multiple images;
The left eye is demultiplexed by performing reverse processing of the image multiplexing means in the stereoscopic video encoding device according to claim 1 by applying the input parallax information to the multiplexed image obtained by decoding. A stereoscopic video decoding program that functions as an image demultiplexing unit that separates and extracts a for-image and a right-eye image. The encoded data of the multiplexed image formed by the stereoscopic video encoding device according to claim 4 is input and mounted on a device to which information on the amount of parallax is input from the stereoscopic video encoding device according to claim 4. Computer
Decoding means for decoding the encoded data of the input multiple images;
The left eye is demultiplexed by performing reverse processing of the image multiplexing means in the stereoscopic video encoding device according to claim 4 by applying the input parallax information to the multiplexed image obtained by decoding. And image demultiplexing means for separating and extracting the image for the right eye and the image for the right eye
3D video decoding program characterized by being made to function.

Images