CN100563340C - Multichannel video stream encoder and decoder based on deep image rendering - Google Patents
Multichannel video stream encoder and decoder based on deep image rendering Download PDFInfo
- Publication number
- CN100563340C CN100563340C CN 200810062865 CN200810062865A CN100563340C CN 100563340 C CN100563340 C CN 100563340C CN 200810062865 CN200810062865 CN 200810062865 CN 200810062865 A CN200810062865 A CN 200810062865A CN 100563340 C CN100563340 C CN 100563340C
- Authority
- CN
- China
- Prior art keywords
- channel
- image frame
- unit
- video
- map
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000009877 rendering Methods 0.000 title claims abstract description 22
- 238000010561 standard procedure Methods 0.000 claims abstract description 29
- 238000005516 engineering process Methods 0.000 claims abstract description 19
- 230000006835 compression Effects 0.000 claims abstract description 17
- 238000007906 compression Methods 0.000 claims abstract description 17
- 238000000034 method Methods 0.000 claims description 30
- 230000003287 optical effect Effects 0.000 claims description 9
- 238000012937 correction Methods 0.000 claims description 6
- 238000003702 image correction Methods 0.000 claims description 5
- 230000000007 visual effect Effects 0.000 abstract description 8
- 230000008878 coupling Effects 0.000 description 6
- 238000010168 coupling process Methods 0.000 description 6
- 238000005859 coupling reaction Methods 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 5
- 239000007787 solid Substances 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 208000018375 cerebral sinovenous thrombosis Diseases 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 101100391181 Dictyostelium discoideum forH gene Proteins 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000005284 excitation Effects 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000001766 physiological effect Effects 0.000 description 1
- 210000000162 simple eye Anatomy 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Landscapes
- Compression Or Coding Systems Of Tv Signals (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
本发明公开了一种基于深度图像渲染的多通道视频流编码器和解码器。本发明在对多通道视频流进行编码时，对中心通道视频流图像帧和深度图按照视频编码标准方法进行编码；根据通道重建后的图像帧和深度图，采用深度图像渲染技术得到邻近的待编码的辅助通道的预测图，对遮挡信息进行变换、量化、熵编码。本发明在对多通道视频压缩码流进行解码时，利用人类视觉系统的生理特点，采用深度图像渲染技术，根据两个相邻通道的图像帧和其深度图，得到位于该两个相邻通道中心位置的一个虚拟通道。在采用本发明解码器输出视频流的显示端，每个立体视点由一个高质量通道和一个虚拟通道构成，观众可以获得良好的立体视觉体验。The invention discloses a multi-channel video stream encoder and decoder based on depth image rendering. When encoding the multi-channel video stream, the present invention encodes the image frame and depth map of the central channel video stream according to the video coding standard method; according to the reconstructed image frame and depth map of the channel, the depth image rendering technology is used to obtain the adjacent waiting The prediction map of the encoded auxiliary channel, which transforms, quantizes, and entropy encodes the occlusion information. When the present invention decodes the multi-channel video compression code stream, it utilizes the physiological characteristics of the human visual system, adopts the depth image rendering technology, and obtains the image frames located in the two adjacent channels according to the image frames and their depth maps of the two adjacent channels. A virtual passage in a central location. At the display end where the video stream is output by the decoder of the present invention, each stereoscopic view point is composed of a high-quality channel and a virtual channel, and the audience can obtain a good stereoscopic visual experience.
Description
Technical field
The present invention relates to the moving image treatment technology, relate in particular to a kind of multichannel video stream encoder and decoder based on deep image rendering.
Background technology
Television system has experienced from the black and white to the colour, from the evolution of analog to digital.What so far two-dimentional television system of development offered spectators is the image on plane, and three-dimensional television system can offer spectators more near the viewing experience of natural vision.Therefore will be a nature from the two-dimentional system to the three dimension system, the evolution that can expect be the development to present two-dimensional digital television system.
To studies show that of human visual system (HVS, Human Visual System), when eyes were observed same object, there was parallax in two width of cloth images of formation.There are two theories in formation to human stereoscopic vision: blending theory (Fusion Theory) is thought if eyes are observed respectively be there are differences, and difference limitation image within the specific limits, merges by vision, and the mankind will form stereoscopic vision.Suppress theoretical (SuppressionTheory) and think the human visual system in the process that forms stereoscopic vision, third dimension and stereo-picture total quality depend on the simple eye preferably image of quality.The continuous quality yardstick of double excitation (DSCQS, Double-Stimulus Continuous-Quality Scale) the subjective testing experiment of people such as Lew Stelmach design has confirmed this theory to a certain extent.
Digital video technology is along with the fast development of Internet and mobile communication has obtained increasingly extensive application, but containing much information of digital video information, bandwidth requirement height to transmission network, so generally digital video signal is carried out compressed encoding earlier before storage or transmission, so that save the memory space and the network bandwidth.
Forming stereoscopic vision needs the digital video of two passages at least, present auto-stereoscopic display supports a plurality of spectators to watch simultaneously, its a plurality of stereos copic viewing points (three-dimensional viewpoint) require the digital video of a plurality of passages of input, therefore a kind of good decoding method need be considered the picture quality behind compression ratio, the decoding and rebuilding, a plurality of factors such as stereoscopic vision experience of spectators, under band-limited restriction, obtain the balance of compression ratio and three-dimensional view-point image quality.
At present many (two) passage digital video is carried out Methods for Coding and be broadly divided into four classes, the first kind is based on the MPEG video encoding standard, second class is based on deep image rendering (DIBR, Depth-Image-BasedRendering) technology, the 3rd class is based on object coding, and the 4th class is based on three-dimensional grid technology (3D mesh).
First kind method is based on the MPEG video encoding standard.
The MVP of MPEG-2 (Multi-View Profile) uses time domain telescopic tool (TS, TemporalScalability tool), and the support to two-channel digital video (three-dimensional video-frequency) coding is provided.MVP uses a kind of double-deck coding structure, and as basic layer, right viewpoint passage is as enhancement layer with left viewpoint passage.Referring to: X.Chen and A.Luthra, MPEG-2 Multi-View Profile and its application in 3DTV, in proceedings of SPIE, vol.3021, pp.212-223,1997.Adopt MVP to carry out the coding of multi-channel digital video, its picture frame predict is similar to the multi-vision-point encoding standard (MVC that is studying in the world at present, Muiti-view Video Coding) picture frame predict, but because MVP adopts Moving Picture Experts Group-2 as coding tools, its code efficiency less than present video coding international standard H.264/AVC.
In May, 2003, formulated the video coding international standard H.264/AVC by the common joint video team JVT (JointVideo Team) that forms of the expert of ITU-T and ISO/IEC.H.264 adopted the hybrid encoding frame structure, adopted binary arithmetic coding or the like the advanced technology of minimum 4 * 4 variable-block motion prediction, a plurality of reference image frame, context-adaptive, compare with MPEG-2, under the situation of same picture quality, can obtain higher compression efficiency.
JVT is is studying and defining multi-vision-point encoding (MVC, Muiti-view Video Coding) international standard at present.MVC has utilized the picture frame correlation between viewpoint inside and the different points of view, utilize and H.264/AVC carry out encoding compression, because the associated prediction in employing time and space coding, radio hookup (Simulcast) with each viewpoint absolute coding is compared, experiment at present shows, under different video contents, the space-time unite coding can improve the gain of 0.5dB to 3dB.Referring to: P.Merkle, A.Smolic and K.Muller, Efficientprediction structures for multiview video coding, IEEE Trans.CSVT, vol.17, no.11, pp.1461-1473,2007.
MVC uses parallax to predict the correlation of excavating between viewpoint.But because of the non-identity of the installation site of video camera, camera site, illumination condition, the same area of the picture frame of a plurality of viewpoints of picked-up, its brightness and colourity exist inconsistent.This inconsistent efficient that can influence parallax accuracy for predicting and coding, a kind of method of solution are to add brightness and colourity compensation term in the coupling cost function.Referring to: J.H.Hur, S.Cho andY.L.Lee, Adaptive local illumination change compensation method forH.264/AVC-based multiview video coding, IEEE Trans.CSVT, vol.17, no.11, pp.1496-1505,2007.
The coding structure more complicated of MVC needs big amount of calculation, the encoding time delay of length and big reference frame storing space.MVC each viewpoint passage of need encoding, when the viewpoint number increases, the also corresponding increase of code check.MVC all viewpoints of encoding, transmit, decode link together the size of photographic images and video camera distance picture size and the viewing distance with display end, have limited the flexibility of display end viewing location like this.
2006, AVS (Advanced Video Coding Standard) is confirmed as the video coding national standard.AVS adopts the hybrid encoding frame structure equally, has adopted integer transform, arithmetic coding or the like the advanced technology of variable-block structure, a plurality of reference image frame, pre-convergent-divergent.Also can adopt AVS that multichannel video stream is carried out encoding and decoding.
Second class methods are based on deep image rendering (DIBR) technology.
The advanced three-dimensional television system of European information technology project (IST, Information Society Technologies) (ATTEST, Advanced Three-Dimensional Television System Technology) has adopted the DIBR method.Referring to: C.Fehn, Depth-Image-Based Rendering (DIBR), compressionand transmission for a new approach on 3D-TV, in Proceedings of SPIE, StereoscopicDisplays and Virtual Reality Systems XI, USA, pp.93-104,2004.
The ATTEST system is at coding side only the encode two-dimensional video of a passage (central passage) and the depth map of this passage, adopt the method for DIBR in decoding end, according to depth information and camera parameters, the central passage picture frame that decoding is recovered projects to three dimensions, project to the imaging plane of virtual video camera again, reconstruct a plurality of virtual two-dimensional video passages thus.
DIBR utilizes the depth information of a passage to play up a plurality of video channels, compares with MVC, can obtain higher compression ratio, and can not produce owing to the different brightness that cause with parameter of camera position, colourity do not match.But owing to block, play up the inner meeting of synthetic virtual view channel image frame and the cavity occurs, and since the decline of virtual visual point image quality, the viewing location in the off-center position, and spectators' stereoscopic vision impression is with influenced.
Alleviate the approach of playing up the inner appearance of synthetic picture frame cavity and have three at present, the one, come filling cavity with the texture around the cavity, the 2nd, it is level and smooth that depth map is carried out filtering, the 3rd, the depth map of a plurality of passages of coding transmission, utilize a plurality of channel image frames and depth map to play up the image to be synthesized of same virtual view, the 4th, adopt comparatively complicated multi-level depth map (LDI, Layered Depth Image) technology, referring to: S.U.Yoonand Y.S.Ho, Multiple color and depth video coding using a hierarchicalrepresentation, IEEE Trans.CSVT, vol.17, no.11,2007.
The 3rd class methods are based on object coding.In the middle of MPEG-4, an object video can be represented with shape (shape), motion (motion), three kinds of features of texture (texture), can use auxiliary element (AC, Auxiliary Component) to deposit disparity map.When adopting MPEG-4MAC that the binary channels video is encoded, usually,, deposit parallax information with MAC (Multiple AuxiliaryComponent) with the standard MPEG-4 left paths video of encoding.Referring to: S.Cho, K.Yun, C.Ahn and S.Lee, Disparity-Compensated stereoscopic video coding using the MAC in MPEG-4, RTRIJournal, vol.27, no.3, pp.326-329,2005.When adopting object-based coding techniques that natural scene is encoded, need cut apart a plurality of objects that extract in the scene, its algorithm complexity.
The 4th class methods are based on three-dimensional grid (3D mesh) technology.Adopt triangular mesh (Triangle mesh) to come the piece wire approximation body surface, this approximate error of bringing is closely related with the triangular mesh number, and grid number is many more, and error is more little, brings problem but the grid of enormous quantity also is storage and transmission.Referring to: J.L.Peng, C.S.Kim and C.C.J.Kuo, Technologies for 3D mesh compression:A survey, Journal of Visual Communication and Image Representation, vol.16, no.6, pp.688-733,2005.
Summary of the invention
The objective of the invention is to overcome the deficiencies in the prior art, a kind of multichannel video stream encoder and decoder based on deep image rendering is provided.
Multichannel video stream encoder comprises:
Image correction unit is used for a plurality of passage video streaming image frames of input are proofreaied and correct, so that corresponding points are positioned on the horizontal scanning line;
The channel selecting unit is used for selecting central passage and accessory channel from a plurality of video channels of input;
Degree of depth generation unit is used to generate the depth map of each picture frame in central passage and the accessory channel video flowing;
The accessory channel predicting unit is used for producing the prognostic chart of accessory channel picture frame according to the reconstruction frames of passage reconstruction unit generation and the depth map of degree of depth generation unit generation;
The central passage coding unit, be used for deep stream, encode according to the video encoding standard method to central passage video flowing and depth map composition, generating the central passage code stream, the video encoding standard method comprise video coding international standard MPEG-X, H.26X with video coding national standard AVS;
The accessory channel coding unit is used for the Occlusion Map of accessory channel picture frame is encoded according to the video encoding standard method, to generate the accessory channel code stream;
The passage reconstruction unit, be used for center channel bit stream and accessory channel code stream, decode according to the video encoding standard method, to generate central passage reconstructed image frame, to rebuild depth map and accessory channel reconstruction Occlusion Map, prognostic chart according to the accessory channel picture frame of rebuilding the generation of Occlusion Map and accessory channel predicting unit produces the accessory channel reconstruction frames;
Multiplexer is used for camera parameters, central passage code stream and accessory channel code stream, according to time division multiplexing mode, generates the multi-channel video compressed bit stream.
Described degree of depth generation unit produces the depth map of this time chart picture frame of central passage according to the picture frame of the synchronization of central passage picture frame and any one accessory channel adjacent thereto; According to the passage reconstruction unit produce when the reconstruction frames of prepass with when the reconstruction frames of the synchronization of the adjacent channel of prepass, produce the depth map of reconstruction frames that should be constantly when prepass.
The reconstruction frames that described accessory channel predicting unit produces according to the passage reconstruction unit, the depth map of this reconstruction frames that produces with degree of depth generation unit, according to method, synthesize the prognostic chart of picture frame of synchronization of the adjacent channel of this reconstruction frames place passage based on deep image rendering.
Described accessory channel coding unit, the prognostic chart of this picture frame that accessory channel picture frame and accessory channel predicting unit are produced is poor, and generation accessory channel Occlusion Map, Occlusion Map have reflected owing to blocking does not have the information that occurs on prognostic chart.
The multi-channel video stream decoder comprises:
Demodulation multiplexer is used for the multi-channel video compressed bit stream is decomposed into camera parameters, central passage code stream and accessory channel code stream;
The central passage decoding unit, be used for the center channel bit stream, decode according to the video encoding standard method, generating central passage reconstructed image frame and to rebuild depth map, the video encoding standard method comprise video coding international standard MPEG-X, H.26X with video coding national standard AVS;
Degree of depth generation unit is used to produce the depth map of accessory channel reconstructed image frame;
Channel prediction unit is used to produce the prognostic chart of accessory channel picture frame and the prognostic chart of tunnel picture frame;
The accessory channel decoding unit, be used for the accessory channel code stream is decoded according to the video encoding standard method, to generate the reconstruction Occlusion Map of accessory channel picture frame, this is rebuild the prognostic chart addition of Occlusion Map and channel prediction unit generation, to generate the reconstructed image frame of accessory channel;
The anti-correcting unit of image is used for central passage reconstructed image frame, accessory channel reconstructed image frame and anti-correction of tunnel prognostic chart picture frame to the decoding generation, so that each channel image frame returns to the position of shooting.
Described degree of depth generation unit, the reconstructed image frame of the synchronization of the reconstructed image frame of the current accessory channel that produces according to the accessory channel decoding unit and the adjacent channel of current accessory channel produces the depth map of this reconstructed image frame of current accessory channel.
Described channel prediction unit, the reconstructed image frame of the central passage that produces according to the central passage decoding unit and the depth map of this picture frame, according to method, produce the prognostic chart of picture frame of the synchronization of the contiguous accessory channel of concentricity passage based on deep image rendering; The reconstructed image frame of the current accessory channel that produces according to the accessory channel decoding unit, the depth map of this picture frame that produces with degree of depth generation unit, according to method, produce with the contiguous not prognostic chart of the picture frame of the synchronization of the accessory channel of reconstruction of current accessory channel based on deep image rendering.
Described channel prediction unit, center at two adjacent channels, reconstructed image frame and depth map according to the synchronization of these two passages, produce the prognostic chart of tunnel picture frame, the photocentre of the virtual video camera of this tunnel is in the mid point of line of photocentre of the video camera of two adjacent channel, and the optical axis of this virtual video camera is parallel with the optical axis of the video camera of central passage.
The anti-correcting unit of described image, compressing video frequency flow for N passage of input decoder, the non-compression video stream of 2N-1 passage of output, the reconstruction non-compression video stream of N the passage that recovers comprising decoding and adopt the non-compression video of N-1 the tunnel that the deep image rendering technology synthesizes to flow.
The anti-correcting unit of described image, 2N-1 passage of output is divided into rebuilds passage and tunnel, to N reconstruction passage, by the relative position sequence arrangement of its real camera, to tunnel, be inserted into two centers that rebuild passage contiguous with this tunnel; 2N-1 passage of output can produce 2N-2 three-dimensional viewpoint viewing location altogether, and each three-dimensional viewpoint viewing location rebuilds passage by one and a tunnel is formed.
In multichannel video stream encoder of the present invention, the deep stream that video flowing and its depth map of central passage are formed is encoded according to the video encoding standard method, excavated central passage interior view picture frame and picture frame correlation, depth map and depth map correlation in time in time; To the picture frame of accessory channel, adopt the method for DIBR to synthesize prognostic chart, the Occlusion Map of accessory channel picture frame is encoded according to the video encoding standard method, excavated the picture frame correlation spatially of adjacent channel synchronization.
In multi-channel video stream decoder of the present invention, adopt the DIBR method, synthesize the prognostic chart of a tunnel in the center of two adjacent channels.The prognostic chart of this tunnel is synthetic according to the picture frame and the depth map of two adjacent channels, and the prognostic chart quality greatly improves; The compressed bit stream of N passage of decoder input synthesizes N-1 tunnel predicted flows, can export the non-compressed bit stream of 2N-1 passage altogether, because per two passages can form a three-dimensional viewpoint, has therefore increased the number of the three-dimensional viewpoint of display end; When display end was supported a plurality of three-dimensional viewpoint, each three-dimensional viewpoint comprised central passage or an accessory channel and the tunnel that the picture frame quality is low slightly that picture quality is higher, and spectators will produce the constant visual effect of scene third dimension.
In multi-channel video stream decoder of the present invention, when display end only supports that the plane shows, central passage or arbitrary accessory channel dynamic image distribution to display unit can be shown; When display end is supported binary channels stereo display, any two adjacent channels can be delivered to display unit and show; When display end is supported a plurality of three-dimensional viewpoint, can at the most 2N-1 passage be delivered to display unit and show that N is the passage number of input decoder.
Adopt the three-dimensional television system of encoder of the present invention,, utilize the video encoding standard method to excavate the correlation of viewpoint inside, utilize the degree of depth to play up (DIBR) method and excavate correlation between the viewpoint at coding side; In decoding end, utilize the physiological property of DIBR method and HVS to obtain more three-dimensional viewpoint.Compare with MVC, this system can obtain lower code check, compares with ATTEST, and spectators can obtain better stereoscopic vision and experience.
Description of drawings
Fig. 1 is according to multichannel video stream encoder schematic diagram of the present invention;
Fig. 2 is according to multi-channel video stream decoder schematic diagram of the present invention.
Embodiment
Multichannel video stream encoder comprises:
Image correction unit is used for a plurality of passage video streaming image frames of input are proofreaied and correct, so that corresponding points are positioned on the horizontal scanning line;
The channel selecting unit is used for selecting central passage and accessory channel from a plurality of video channels of input;
Degree of depth generation unit is used to generate the depth map of each picture frame in central passage and the accessory channel video flowing;
The accessory channel predicting unit is used for producing the prognostic chart of accessory channel picture frame according to the reconstruction frames of passage reconstruction unit generation and the depth map of degree of depth generation unit generation;
The central passage coding unit, be used for deep stream, encode according to the video encoding standard method to central passage video flowing and depth map composition, generating the central passage code stream, the video encoding standard method comprise video coding international standard MPEG-X, H.26X with video coding national standard AVS;
The accessory channel coding unit is used for the Occlusion Map of accessory channel picture frame is encoded according to the video encoding standard method, to generate the accessory channel code stream;
The passage reconstruction unit, be used for center channel bit stream and accessory channel code stream, decode according to the video encoding standard method, to generate central passage reconstructed image frame, to rebuild depth map and accessory channel reconstruction Occlusion Map, prognostic chart according to the accessory channel picture frame of rebuilding the generation of Occlusion Map and accessory channel predicting unit produces the accessory channel reconstruction frames;
Multiplexer is used for camera parameters, central passage code stream and accessory channel code stream, according to time division multiplexing mode, generates the multi-channel video compressed bit stream.
Described degree of depth generation unit produces the depth map of this time chart picture frame of central passage according to the picture frame of the synchronization of central passage picture frame and any one accessory channel adjacent thereto; According to the passage reconstruction unit produce when the reconstruction frames of prepass with when the reconstruction frames of the synchronization of the adjacent channel of prepass, produce the depth map of reconstruction frames that should be constantly when prepass.
The reconstruction frames that described accessory channel predicting unit produces according to the passage reconstruction unit, the depth map of this reconstruction frames that produces with degree of depth generation unit, according to method, synthesize the prognostic chart of picture frame of synchronization of the adjacent channel of this reconstruction frames place passage based on deep image rendering.
Described accessory channel coding unit, the prognostic chart of this picture frame that accessory channel picture frame and accessory channel predicting unit are produced is poor, and generation accessory channel Occlusion Map, Occlusion Map have reflected owing to blocking does not have the information that occurs on prognostic chart.
The multi-channel video stream decoder comprises:
Demodulation multiplexer is used for the multi-channel video compressed bit stream is decomposed into camera parameters, central passage code stream and accessory channel code stream;
The central passage decoding unit, be used for the center channel bit stream, decode according to the video encoding standard method, generating central passage reconstructed image frame and to rebuild depth map, the video encoding standard method comprise video coding international standard MPEG-X, H.26X with video coding national standard AVS;
Degree of depth generation unit is used to produce the depth map of accessory channel reconstructed image frame;
Channel prediction unit is used to produce the prognostic chart of accessory channel picture frame and the prognostic chart of tunnel picture frame;
The accessory channel decoding unit, be used for the accessory channel code stream is decoded according to the video encoding standard method, to generate the reconstruction Occlusion Map of accessory channel picture frame, this is rebuild the prognostic chart addition of Occlusion Map and channel prediction unit generation, to generate the reconstructed image frame of accessory channel;
The anti-correcting unit of image is used for central passage reconstructed image frame, accessory channel reconstructed image frame and anti-correction of tunnel prognostic chart picture frame to the decoding generation, so that each channel image frame returns to the position of shooting.
Described degree of depth generation unit, the reconstructed image frame of the synchronization of the reconstructed image frame of the current accessory channel that produces according to the accessory channel decoding unit and the adjacent channel of current accessory channel produces the depth map of this reconstructed image frame of current accessory channel.
Described channel prediction unit, the reconstructed image frame of the central passage that produces according to the central passage decoding unit and the depth map of this picture frame, according to method, produce the prognostic chart of picture frame of the synchronization of the contiguous accessory channel of concentricity passage based on deep image rendering; The reconstructed image frame of the current accessory channel that produces according to the accessory channel decoding unit, the depth map of this picture frame that produces with degree of depth generation unit, according to method, produce with the contiguous not prognostic chart of the picture frame of the synchronization of the accessory channel of reconstruction of current accessory channel based on deep image rendering.
Described channel prediction unit, center at two adjacent channels, reconstructed image frame and depth map according to the synchronization of these two passages, produce the prognostic chart of tunnel picture frame, the photocentre of the virtual video camera of this tunnel is in the mid point of line of photocentre of the video camera of two adjacent channel, and the optical axis of this virtual video camera is parallel with the optical axis of the video camera of central passage.
The anti-correcting unit of described image, compressing video frequency flow for N passage of input decoder, the non-compression video stream of 2N-1 passage of output, the reconstruction non-compression video stream of N the passage that recovers comprising decoding and adopt the non-compression video of N-1 the tunnel that the deep image rendering technology synthesizes to flow.
The anti-correcting unit of described image, 2N-1 passage of output is divided into rebuilds passage and tunnel, to N reconstruction passage, by the relative position sequence arrangement of its real camera, to tunnel, be inserted into two centers that rebuild passage contiguous with this tunnel; 2N-1 passage of output can produce 2N-2 three-dimensional viewpoint viewing location altogether, and each three-dimensional viewpoint viewing location rebuilds passage by one and a tunnel is formed.
Embodiment
Fig. 1 is the schematic diagram according to multichannel video stream encoder of the present invention.Multichannel video stream encoder carries out compressed encoding, the compressed bit stream behind the output encoder to the multichannel video stream and the camera parameters of input.Encoder comprises image correction unit 11, channel selecting unit 12, degree of depth generation unit 13, accessory channel predicting unit 14, central passage coding unit 15, accessory channel coding unit 16, passage reconstruction unit 17 and multiplexer 18.
Referring to Fig. 1, the encoder encodes multichannel video stream comprises following 11 steps:
Step 1: image correction unit 11 is accepted the multichannel video stream and the camera parameters of input, according to the correcting algorithm of standard, picture frame is proofreaied and correct.The result who proofreaies and correct is at synchronization, and for different channel image frames, its corresponding points are positioned on the horizontal line.
Step 2: a plurality of passages of the 12 pairs of inputs in channel selecting unit are classified, and select 1 central passage, remaining passage as accessory channel.Select the algorithm of central passage as follows: be designated as in order No. 1 taking N the passage that obtains, No. 2 ..., N number, N is a positive integer, N 〉=2; Choose the c passage as central passage:
Symbol
Round under the expression.N-1 remaining passage as accessory channel.
Referring to Fig. 1, note central passage video streaming image frame is I
c, accessory channel video streaming image frame is I
a, the depth map of central passage picture frame is Z
c, the depth map of accessory channel picture frame is Z
a, subscript a is a positive integer, satisfies 1≤a≤N, and a ≠ c.
Step 3: the I of 13 pairs of synchronizations of degree of depth generation unit
cAnd I
aCarry out the solid coupling, be somebody's turn to do depth map Z constantly to generate central passage
c, a can get a=c-1 or a=c+1.
Step 4: central passage coding unit 15 is encoded to the deep stream of central passage video flowing and depth map composition according to the video encoding standard method, generates the central passage code stream.
Step 5: 17 pairs of center channel bit streams of passage reconstruction unit are decoded, and generate central passage reconstructed video stream picture frame and rebuild depth map, are designated as I ' respectively
cAnd Z '
c
Step 6: accessory channel predicting unit 14, according to I '
cAnd Z '
c, adopt method based on deep image rendering (DIBR), synthesize the prognostic chart of the contiguous accessory channel of concentricity passage, be designated as P
a, a can get a=c-1 or a=c+1.
For convenience of description, get a=c-1 earlier, carry out step 7, get a=c+1 again, carry out step 7 to 10 to step 10.The a accessory channel that satisfies a＜c is called the left side accessory channel, a accessory channel that satisfies a＞c is called the right side accessory channel.Coded sequence described below is earlier the left side accessory channel to be encoded, and the right side accessory channel is encoded again.But it is limitation of the invention that following description should not be construed as, and in fact also can encode to the right side accessory channel earlier, the left side accessory channel is encoded, perhaps to left side, the right side accessory channel coding that hockets again.
Step 7: accessory channel coding unit 16 is at first to the I of a secondary channels synchronization
aAnd P
aDiffer from, produce the Occlusion Map of a passage, be designated as R
a, then to R
aEncode according to the video encoding standard method, produce a channel bit stream.If a equals 1, get back to step 6; If a equals N, jump to step 11; If 1＜a＜N carries out next step.
Step 8: 17 pairs of a channel bit streams of passage reconstruction unit are decoded, and produce the reconstruction Occlusion Map of a passage, are designated as R '
a, R '
aAnd P
aDo and, produce the picture frame of the reconstructed video stream of a passage, be designated as I '
a
Step 9: degree of depth generation unit 13, according to the reconstructed image frame I ' of a passage of importing
aAnd with the reconstructed image frame of its adjacent channel, generate the depth map of a passage reconstructed image frame, be designated as Z '
aIf a＜c, then the a+1 passage is rebuilt prior to a passage, the 13 couples of I ' in unit
aAnd I '
A+1Carry out the solid coupling, to generate Z '
aIf a＞c, then the a-1 passage is rebuilt prior to a passage, the 13 couples of I ' in unit
aAnd I '
A-1Carry out the solid coupling, to generate Z ' a.
Step 10: accessory channel predicting unit 14, according to I '
aAnd Z '
a, the method for employing DIBR synthesizes the prognostic chart with the contiguous uncoded accessory channel of a passage, remembers that this accessory channel is j number.If a＜c, j equals a-1, makes a equal a-1; If a＞c, j equals a+1, makes a equal a+1.Get back to step 7.
Step 11: 18 pairs of camera parameters of multiplexer, central passage code stream and accessory channel code stream, take mode according to the time-division, generate the compressed bit stream of N passage.
By above 11 steps, encoder finally generates the compressed bit stream of N passage of input.In above-mentioned step 6 and step 10, accessory channel predicting unit 14 adopts reconstructed image frame I '
c(perhaps I '
a) and rebuild depth map Z '
c(Z '
a), rather than I
c(perhaps I
a) and Z
c(perhaps Z
a), synthesize the prognostic chart P of a passage
a, be in order to keep consistency with decoder.
In above-mentioned step 7, the Occlusion Map of 16 pairs of accessory channels of accessory channel coding unit is encoded and is transferred to decoding end, can compensate owing to the cavity of adopting the synthetic prognostic chart of DIBR method to produce, reconstruct high-quality accessory channel picture frame in decoding end.
Fig. 2 is the schematic diagram according to multi-channel video stream decoder of the present invention.The multi-channel video stream decoder is decoded to the compressed bit stream and the camera parameters of input, non-compression video stream behind the output decoder and the non-compression video stream that synthesizes through tunnel.Decoder comprises demodulation multiplexer 21, central passage decoding unit 22, degree of depth generation unit 23, channel prediction unit 24, accessory channel decoding unit 25, the anti-correcting unit 26 of image.
Referring to Fig. 2, the decoder decode compressed bit stream comprises following 8 steps:
Step 1: demodulation multiplexer 21 resolves into camera parameters, central passage code stream and accessory channel code stream to the compressed bit stream of input.
Step 2: the central passage code stream of 22 pairs of inputs of central passage decoding unit, according to the video encoding standard method, decoding generates the picture frame I ' of central passage reconstructed video stream
cWith reconstruction depth map Z '
c
Step 3: channel prediction unit 24 is according to the I ' of input
cAnd Z '
c, adopt the DIBR method, synthesize the prognostic chart P of the contiguous a accessory channel of concentricity passage
a, a can get a=c-1 or a=c+1.
For convenience of description, get a=c-1 earlier, carry out step 4, get a=c+1 again, carry out step 4 to step 7 to step 7.The accessory channel of a＜c is called the left side accessory channel, the accessory channel of a＞c is called the right side accessory channel.Decoding order described below is earlier the left side accessory channel to be decoded, and the right side accessory channel is decoded again.But it is limitation of the invention that following description should not be construed as, and in fact also can decode to the right side accessory channel earlier, the left side accessory channel is decoded, perhaps to the decoding that hockets of left side, right side accessory channel again.
Step 4: 25 pairs of accessory channel code streams of accessory channel decoding unit, decode according to the video encoding standard method, generate the reconstruction Occlusion Map R ' of a accessory channel
a, to the prognostic chart P of a accessory channel
aWith reconstruction Occlusion Map R '
aDo and, generate the reconstructed image frame I ' of a accessory channel
a
Step 5: degree of depth generation unit 23, according to a passage reconstructed image frame I ' of input
aAnd the reconstructed image frame of the synchronization of passage adjacent thereto, generate a passage depth map Z ' of reconstructed image frame constantly
aIf a＜c, then the a+1 passage is rebuilt prior to a passage, 23 couples of I ' of degree of depth generation unit
aAnd I '
A+1Carry out the solid coupling, to generate Z '
aIf a＞c, then the a-1 passage is rebuilt prior to a passage, 23 couples of I ' of degree of depth generation unit
aAnd I '
A-1Carry out the solid coupling, to generate Z '
a
Step 6: channel prediction unit 24 is according to the reconstructed image frame I ' of a passage
aWith depth map Z '
aAnd the reconstructed image frame I ' of k accessory channel synchronization adjacent thereto
kWith depth map Z '
k, the method for employing DIBR synthesizes the tunnel prognostic chart that is positioned at a passage and k channel center position, remembers that this tunnel prognostic chart is V
a, the corresponding virtual video camera in this tunnel position, its photocentre is positioned at the point midway of the line of a passage video camera photocentre and k passage video camera photocentre, and its optical axis is parallel to central passage.If a＜c, k equals a+1; If a＞c, k equals a-1.Synthesize V
aAfter, if a equals 1, get back to step 3; If a equals N, jump to step 8; If 1＜a＜N carries out next step.
Step 7: channel prediction unit 24, according to I '
aAnd Z '
a, the method for employing DIBR synthesizes the prognostic chart with the contiguous q accessory channel of a passage.If a＜c, q equals a-1, makes a equal a-1; If a＞c, q equals a+1, makes a equal a+1.Get back to step 4.
Step 8: the anti-correcting unit 26 of image, according to the camera parameters of input, to central passage reconstructed image frame, accessory channel reconstructed image frame, tunnel prognostic chart, the anti-correction.The virtual video camera parameter that tunnel needs according to the camera parameters of two passages that are adjacent, is carried out the linear weighted function interpolation and is obtained.
Claims (4)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 200810062865 CN100563340C (en) | 2008-07-07 | 2008-07-07 | Multichannel video stream encoder and decoder based on deep image rendering |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 200810062865 CN100563340C (en) | 2008-07-07 | 2008-07-07 | Multichannel video stream encoder and decoder based on deep image rendering |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101309412A CN101309412A (en) | 2008-11-19 |
| CN100563340C true CN100563340C (en) | 2009-11-25 |
Family
ID=40125591
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 200810062865 Active CN100563340C (en) | 2008-07-07 | 2008-07-07 | Multichannel video stream encoder and decoder based on deep image rendering |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN100563340C (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI448145B (en) * | 2010-03-26 | 2014-08-01 | Sony Corp | Image processing apparatus and method, and program |
Families Citing this family (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2197217A1 (en) * | 2008-12-15 | 2010-06-16 | Koninklijke Philips Electronics N.V. | Image based 3D video format |
| CN104811685B (en) * | 2008-12-18 | 2017-04-12 | Lg电子株式会社 | Method for 3D image signal processing and image display for implementing the same |
| WO2010093351A1 (en) | 2009-02-13 | 2010-08-19 | Thomson Licensing | Depth map coding to reduce rendered distortion |
| US9148673B2 (en) | 2009-06-25 | 2015-09-29 | Thomson Licensing | Depth map coding |
| WO2011120228A1 (en) * | 2010-04-01 | 2011-10-06 | Intel Corporation | A multi-core processor supporting real-time 3d image rendering on an autostereoscopic display |
| WO2012128241A1 (en) * | 2011-03-18 | 2012-09-27 | ソニー株式会社 | Image-processing device, image-processing method, and program |
| CN102325259A (en) * | 2011-09-09 | 2012-01-18 | 青岛海信数字多媒体技术国家重点实验室有限公司 | Method and device for synthesizing virtual viewpoints in multi-viewpoint video |
| WO2013100790A1 (en) * | 2011-12-30 | 2013-07-04 | Intel Corporation | Coarse-to-fine multple disparity candidate stereo matching |
| CN103379340B (en) | 2012-04-19 | 2017-09-01 | 乐金电子(中国)研究开发中心有限公司 | A kind of residual error prediction method and device |
| CN111031302A (en) * | 2012-04-25 | 2020-04-17 | 浙江大学 | Decoding method, encoding method and device for auxiliary information of three-dimensional video sequence |
| CN102724525B (en) * | 2012-06-01 | 2014-11-05 | 宁波大学 | Depth video coding method on basis of foveal JND (just noticeable distortion) model |
| WO2017080420A1 (en) * | 2015-11-09 | 2017-05-18 | Versitech Limited | Auxiliary data for artifacts –aware view synthesis |
| CN119094796A (en) * | 2017-10-27 | 2024-12-06 | 松下电器（美国）知识产权公司 | Three-dimensional model encoding device, three-dimensional model decoding device, three-dimensional model encoding method, and three-dimensional model decoding method |
| CN109708655A (en) * | 2018-12-29 | 2019-05-03 | 百度在线网络技术（北京）有限公司 | Air navigation aid, device, vehicle and computer readable storage medium |
| CN111667438B (en) * | 2019-03-07 | 2023-05-26 | 阿里巴巴集团控股有限公司 | Video reconstruction method, system, device and computer readable storage medium |
| CN109842811B (en) * | 2019-04-03 | 2021-01-19 | 腾讯科技（深圳）有限公司 | Method and device for implanting push information into video and electronic equipment |
| CN113840171B (en) * | 2021-09-16 | 2023-06-13 | 星宸科技股份有限公司 | Video data processing method and device |
| CN115908156B (en) * | 2021-09-30 | 2024-08-13 | 想象技术有限公司 | Method for rendering an image of a 3D scene, graphics processing unit and storage medium |
| CN115941914B (en) * | 2023-01-06 | 2023-05-23 | 湖南马栏山视频先进技术研究院有限公司 | Video rendering system based on video frame analysis |
-
2008
- 2008-07-07 CN CN 200810062865 patent/CN100563340C/en active Active
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI448145B (en) * | 2010-03-26 | 2014-08-01 | Sony Corp | Image processing apparatus and method, and program |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101309412A (en) | 2008-11-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100563340C (en) | Multichannel video stream encoder and decoder based on deep image rendering | |
| CN100563339C (en) | A Multi-channel Video Stream Coding Method Using Depth Information | |
| Yea et al. | View synthesis prediction for multiview video coding | |
| Shao et al. | Asymmetric coding of multi-view video plus depth based 3-D video for view rendering | |
| DK2594077T3 (en) | Hybrid video coding that supports the intermediate-view synthesis | |
| CN104838648B (en) | For creating the apparatus and method with reconstructed video stream | |
| CN101466038B (en) | Method for encoding stereo video | |
| Smolic et al. | Development of a new MPEG standard for advanced 3D video applications | |
| CN107277550A (en) | Multi-view signal codec | |
| CN102055982A (en) | Coding and decoding methods and devices for three-dimensional video | |
| Lim et al. | A multiview sequence CODEC with view scalability | |
| JP2016537869A (en) | View synthesis prediction method and merge candidate list construction method using the same in multi-view video coding | |
| CN104243966A (en) | method and device for generating, storing, transmitting, receiving and reproducing depth maps | |
| MX2008003375A (en) | Method of estimating disparity vector, and method and apparatus for encoding and decoding multi-view moving picture using the disparity vector estimation method. | |
| CN101312542A (en) | Natural three-dimensional television system | |
| CN101867816A (en) | Stereoscopic video asymmetric compression coding method based on human-eye visual characteristic | |
| CN106210722A (en) | The coded method of depth of seam division video residual error layer data based on HEVC | |
| Chen et al. | Coding techniques in multiview video coding and joint multiview video model | |
| Puri et al. | Stereoscopic video compression using temporal scalability | |
| Sanchez et al. | Performance analysis of depth intra-coding in 3D-HEVC | |
| Lee et al. | A framework of 3D video coding using view synthesis prediction | |
| WO2013159702A1 (en) | Method and device for decoding and encoding auxiliary information of three-dimensional video sequence | |
| Vetro et al. | Analysis of 3D and multiview extensions of the emerging HEVC standard | |
| Müller et al. | 3D video formats and coding methods | |
| CN102625097A (en) | Three-dimensional video intra-frame prediction method and encoding and decoding method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20160620 Address after: 518000 new energy building, Nanhai Road, Shenzhen, Guangdong, Nanshan District A838 Patentee after: Meng Qi media (Shenzhen) Co. Ltd. Address before: 310027 Hangzhou, Zhejiang Province, Zhejiang Road, No. 38 Patentee before: Zhejiang University |
|
| C41 | Transfer of patent application or patent right or utility model | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20160831 Address after: 518000, 101, 2, Fengyun technology building, Fifth Industrial Zone, North Ring Road, Shenzhen, Guangdong, Nanshan District Patentee after: World wide technology (Shenzhen) Limited Address before: 518000 new energy building, Nanhai Road, Shenzhen, Guangdong, Nanshan District A838 Patentee before: Meng Qi media (Shenzhen) Co. Ltd. |
|
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20180903 Address after: 518000 B unit 101, Fengyun mansion 5, Xili street, Nanshan District, Shenzhen, Guangdong. Patentee after: Wan D display technology (Shenzhen) Co., Ltd. Address before: 518000 2 of Fengyun tower, Fifth Industrial Zone, Nanshan District North Ring Road, Shenzhen, Guangdong, 101 Patentee before: World wide technology (Shenzhen) Limited |