TWI653882B - Video device and encoding/decoding method for 3d objects thereof - Google Patents

Abstract

The present invention provides a video device, comprising: a memory; a processor for acquiring a three-dimensional object, and encoding a plurality of viewing angle images corresponding to the plurality of viewing angles of the three-dimensional object to generate a video stream; a graphics processing unit, configured to decode the video stream to obtain a plurality of viewing angle images corresponding to the plurality of viewing angles, and store the plurality of viewing angle images in the memory; wherein the drawing processing unit is based on Sensing information of one of the display devices of a user determines a relative viewing angle between the user and the three-dimensional object, and calculates corresponding to the relative image from the plurality of viewing angle images stored in the memory One of the viewing angles outputs an image; wherein the drawing processing unit further transmits the output image to the display device for playing.

Description

Video device and its three-dimensional object codec method

The present invention relates to a display device, and more particularly to a video device and a method for encoding and decoding the three-dimensional object thereof.

In recent years, with the development of science and technology, devices and related applications of virtual reality (VR) and Augmented Reality (AR) have become more and more popular. In general, a virtual reality device typically includes two displays, and each display separately plays an image to the left and right eyes of the user to form a 3D image. The video content of the virtual reality may have a 360 degree field of view (FOV) and has a picture update rate of 60 Hz to 90 Hz. Further, the video content of the virtual reality mentioned above usually includes a three-dimensional object, so the required file space is quite large. Therefore, for the traditional virtual reality device, the bottleneck of its design lies in the transmission bandwidth and the storage space of the video file. In other words, the traditional virtual reality device often has too large a video file after the video encoding of the three-dimensional object, and thus the transmission bandwidth is insufficient, so that the image quality needs to be sacrificed, so that the user experience is poor.

Therefore, there is a need for a video device and its three-dimensional object encoding and decoding method to reduce encoding.

The present invention provides a video device, comprising: a memory; a processor for acquiring a three-dimensional object, and encoding a plurality of viewing angle images corresponding to the plurality of viewing angles of the three-dimensional object to generate a video stream; a graphics processing unit, configured to decode the video stream to obtain a plurality of viewing angle images corresponding to the plurality of viewing angles, and store the plurality of viewing angle images in the memory; wherein the drawing processing unit is based on Sensing information of one of the display devices of a user determines a relative viewing angle between the user and the three-dimensional object, and calculates corresponding to the relative image from the plurality of viewing angle images stored in the memory One of the viewing angles outputs an image; wherein the drawing processing unit further transmits the output image to the display device for playing.

In an embodiment of the invention, the plurality of viewing angles of the three-dimensional object are equidistantly arranged at a vertical angle and a horizontal angle.

In an embodiment of the present invention, the processor arranges the plurality of views of the three-dimensional object into a two-dimensional image and performs a spatial encoding to generate the video stream, and the video stream is a meta-data. It records the image of each view corresponding to each angle of view.

In an embodiment of the present invention, after the drawing processing unit obtains the plurality of viewing angle images, the drawing processing unit further performs background removal and image capturing processing on the plurality of viewing angle images, and then processes the complex number The viewing angle image is stored in the memory.

In an embodiment of the present invention, the drawing processing unit determines, by the plurality of viewing angle images, a first viewing angle image that is closest to the relative viewing angle and obtains one or more viewing angle images adjacent thereto, and is configured by the first One-view image and its The output image is calculated by the adjacent view image.

The present invention further provides a three-dimensional object encoding and decoding method for a video device, wherein the video device includes a processor and a graphics processing unit, the method includes: using the processor to obtain a three-dimensional object, and the three-dimensional object The plurality of viewing angle images corresponding to the plurality of viewing angles are encoded to generate a video stream; the graphics processing unit is used to decode the video stream to obtain a plurality of viewing angle images corresponding to the plurality of viewing angles; and the drawing processing unit is utilized Determining a relative viewing angle between the user and the three-dimensional object based on sensing information from one of the display devices of a user, and calculating, by the drawing processing unit, the relative viewing angle from the plurality of viewing angle images And outputting the image; and transmitting the output image to the display device for playing by using the drawing processing unit.

10‧‧‧Video System

100‧‧‧ video equipment

110‧‧‧ processor

120‧‧‧Drawing processing unit

130‧‧‧ memory

140‧‧‧Storage device

150‧‧‧Transport interface

160‧‧‧ transmission line

200‧‧‧ display device

210‧‧‧ processor

220‧‧‧ memory

230‧‧‧Transport interface

240‧‧‧Left eye display panel

241‧‧‧left eye lens

250‧‧‧right eye display panel

251‧‧‧right eye lens

260‧‧‧Sensing device

270‧‧‧shell

20‧‧‧Users

30‧‧‧Three-dimensional objects

31‧‧‧ center point

32‧‧‧ points

300‧‧‧ sphere area

50A, 50B‧‧‧ output image

S410-S470‧‧‧Steps

1 is a block diagram showing a video system in accordance with an embodiment of the present invention.

2A-2F are schematic views showing a view image of a three-dimensional object according to an embodiment of the present invention.

3A and 3B are schematic views showing the relative viewing angles of the user and the three-dimensional object in accordance with an embodiment of the present invention.

Figure 4 is a flow chart showing a method for encoding and decoding a three-dimensional object in accordance with an embodiment of the present invention.

The above described objects, features and advantages of the present invention will become more apparent from the description of the appended claims.

1 is a block diagram showing a video system in accordance with an embodiment of the present invention. In one embodiment, the video system 10 includes a video device 100 and a display device 200. The video device 100 and the display device 200 are connected through a transmission line 160.

In an embodiment, the video device 100 can be an computing device, such as a personal computer, a smart phone, a tablet, etc., but the invention is not limited thereto. The video device 100 is used to calculate a 3D image to be displayed (for example, including a three-dimensional object). However, the calculated three-dimensional object cannot be directly displayed. Therefore, the 3D image needs to be encoded into a video stream, and the output image to be played is generated by drawing processing. The output image is transmitted to the display device 200 via the transmission line 160 for playback, so that the user can view the virtual reality and the 3D scene through the display device 200.

The video device 100 includes a processor 110, a graphics processing unit 120, a memory 130, a storage device 140, and a transmission interface 150. The processor 110 is, for example, a central processing unit (CPU), and the memory 130 is, for example, a volatile memory of a static random access memory (SRAM) or a dynamic random access memory (DRAM). body. For example, the processor 110 is configured to calculate a three-dimensional object and encode the corresponding view image of the three-dimensional object into a video stream (or video file).

The graphics processing unit 120 performs corresponding graphics processing and video decoding according to instructions of the application executed by the processor 110 to generate an output image. Like, the details will be detailed later.

The storage device 140 is used to store operating systems, various applications, display drivers, and the like executed on the mobile device. The storage device 140 is, for example, a non-volatile memory of a hard disk drive, a solid state hard disk, a flash memory, or the like, but the present invention is not limited thereto. The processor 110 can read and execute the operating system and application stored in the storage device 140 to the memory 130.

The transmission interface 150 can support high-speed multimedia data transmission, for example, can be a multimedia transmission interface, such as a high-definition multimedia interface or a display port interface, but the invention is not limited thereto. .

In an embodiment, the drawing processing unit 120 includes a drawing processing pipeline (not shown) that can tessellation the vertex data and the primitive data, and after subdivision The data is sequentially subjected to vertex processing, geometry processing, and pixel processing, in which a texture is added during pixel processing, and then pixel rendering is performed.

In addition, the graphics processing unit 120 can also perform video decoding, for example, decoding the video stream of the three-dimensional object generated by the processor 110, thereby obtaining a plurality of viewing angle images corresponding to the plurality of viewing angles of the three-dimensional object.

The display device 200 can be, for example, a virtual reality display device, for example, a head-mounted display (HMD) for playing video content from the video device 100.

In some embodiments, the display device 200 further has video decoding capability, for example, can receive video from the video device 100 through its transmission interface 230. The video stream is decoded into a video material (for example, a stereoscopic image, a multi-view image, etc.), and the video data is played on the left-eye display panel 240 and the right-eye display panel 250. In other words, in this embodiment, the display device 200 can be used to perform video decoding processing in the video device 100.

In one embodiment, the display device 200 includes a processor 210, a memory 220, a transmission interface 230, a left-eye display panel 240, and a corresponding left-eye lens 241, right-eye display panel 250, and corresponding right lens. 251, a sensing device 260, a housing 270. The processor 210 can be, for example, a central processing unit (CPU), or a microprocessor, etc., but the invention is not limited thereto. The memory 220 can be, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM) volatile memory for storing temporary data during video decoding and as an image. Buffer (image buffer).

The transmission interface 230 can support high-speed multimedia data transmission, for example, can be a multimedia transmission interface, such as a high-definition multimedia interface or a display port interface, but the invention is not limited thereto. .

The left-eye display panel 240 and the right-eye display panel 250 can respectively receive and play video content from the processor 210, such as a left-eye image and a right-eye image, respectively. For example, the left-eye display panel 240 and the right-eye display panel 250 can be, for example, a liquid crystal display (LCD), a light-emitting diode (LED) screen, or an organic light-emitting diode (OLED) screen, but the invention is not limited thereto. this. Therefore, the left-eye display panel 240 and the right-eye display panel 250 and the corresponding left-eye lens 241 and right-eye lens 251 respectively project the left-eye image and the right-eye image to the left and right eyes of the user, The user can view the corresponding stereoscopic image and feel the virtual reality (VR).

The sensing device 260 is configured to detect the movement and direction of the user's head and generate a sensing data. The sensing data is returned to the processor 210, so the processor 210 can calculate the perspective (FOV) of the user's eyes based on the sensing data, and can determine the corresponding viewing angle image of the three-dimensional object from the viewing angle. For example, the sensing device 260 can include a gyroscope and an accelerometer, but the invention is not limited thereto.

The components 210-260 in the display device 200 are disposed in the housing 270, and the housing 270 can include a strap or other auxiliary device (not shown) for the user to wear on the head to view through the display device 200. Picture.

For example, the processor 210 can receive the video content from the video device 100 through the transmission interface 230 and play it on the left-eye display panel 240 and the right-eye display panel 250.

In the present invention, the encoding method for the three-dimensional object in the above video stream is different from the conventional method. Furthermore, the three-dimensional objects performed by the drawing processing unit 120 are composed of a plurality of polygons and textures. Traditional image/video compression methods are used directly for material images and object shapes. However, for virtual reality applications, the amount of computation required is much higher than that of normal video compression, mainly in virtual reality devices where the user's eyes are fairly close to the display panel, so the image details can be felt. The more the image quality is sacrificed due to the consideration of the transmission bandwidth, the user can easily feel it, resulting in a poor user experience.

In addition, another limitation of the conventional method is the usability. User-generated image content requires specific software and processing methods, although it can be used Existing images simplify the process, but additional intermediate steps are still required to generate a 3D model, and the steps of image rendering are still used to produce a virtual reality view image.

In the present invention, the video device 100 simplifies the process of drawing a three-dimensional object in a virtual reality image, for example, using a three-dimensional object to replace a three-dimensional object in a plurality of viewing angle images (two-dimensional images) at different viewing angles. It should be understood that at each specific time point, the user only sees one of the view images of the three-dimensional object when viewing the three-dimensional object. As the relative viewing angle between the user and the three-dimensional object is different, the viewing angle image of the three-dimensional object seen by the user also varies.

Furthermore, the video device 100 is, for example, both an encoding end and a decoding end. When the processor 110 draws a three-dimensional object, a plurality of perspective images of the three-dimensional object (for example, a two-dimensional image in a bitmap or JPEG format) may be encoded in advance, for example, the plurality of viewing angles may be first used. The images are first arranged into an integrated image and then spatially encoded for the integrated image, or each of the two-dimensional images can be separately. Therefore, the video stream generated after the encoding (which can be stored as a video compressed file) includes the view image of each view of the three-dimensional object, and the information of each view of the three-dimensional object and the corresponding view image are stored in the image. Metadata or header information in the generated video stream. Therefore, the drawing processing unit 120 in the video device 100 does not need to calculate the three-dimensional model and then draw the three-dimensional model on the output image as in the conventional method.

Furthermore, the size of the integrated image is related to the number of views of the three-dimensional object and the resolution of the original view image. When the number of views is increased or The higher the resolution of the perspective image, the larger the integrated image. However, for a smaller number of views, the video stream produced by the integrated image is much smaller than the video stream produced by encoding the three-dimensional object.

According to the above embodiment, the graphics processing unit 120 in the video device 100 is, for example, a decoding terminal, and reads the metadata of the video stream generated by the processor 110, and can analyze that the current video stream is a plurality of viewing angle images. To replace the encoding of three-dimensional objects. Then, the graphics processing unit 120 can decode the video stream to obtain a plurality of perspective images related to the three-dimensional object, and store the plurality of perspective images in the memory 130.

In some embodiments, after the plurality of view images related to the three-dimensional object are obtained, the drawing processing unit 120 may perform additional image processing on the plurality of view images, such as background removal or image capture (image). Cropping).

Then, the mapping processing unit 120 obtains the sensing data of the display device 200 from the sensing device 260, thereby determining a relative viewing angle between the user and the three-dimensional object, and selecting from the plurality of viewing angle images according to the relative viewing angle. The corresponding view image is used as the output image. In some embodiments, the graphics processing unit 120 then images the selected perspective image and the corresponding background and material to generate an output image (eg, including a left eye image and a right eye image).

In some embodiments, the drawing processing unit 120 does not need to obtain the sensing data of the display device 200 from the sensing device 260, but performs a stereoscopic projection method to recognize the directions of the respective viewing angles and calculate the relative directions. Perspective, but in this method the graphics processing unit 120 requires a higher computing power to read the associated image data.

It should be noted that when the processor 110 performs encoding of a plurality of view images of a three-dimensional object, a reasonable number of views is set to save the length of the video stream generated by the encoding. When the drawing processing unit 120 selects the corresponding view image according to a relative viewing angle between the user and the three-dimensional object, the relative viewing angle does not necessarily correspond to one of the three-dimensional objects. Therefore, in consideration of the memory bandwidth, the drawing processing unit 120 reads a view image closest to the relative view angle and its adjacent one or more view images, and interpolates to generate an output image.

2A-2F are schematic views showing a view image of a three-dimensional object according to an embodiment of the present invention.

In the 2A-2F drawings, the perspective images corresponding to different viewing angles of the three-dimensional object 280 are shown. For example, the three-dimensional object 280 is, for example, a cube, and the six surfaces have different colors. The top view and the bottom view of the three-dimensional object 280 are shown in FIGS. 2A and 2B, respectively.

The 2C~2F diagrams respectively show oblique views in three different directions of the three-dimensional object 280. For example, for the user, different viewing angle images are viewed at different viewing angles obliquely above the three-dimensional object 280, that is, for example, as shown in FIGS. 2C-2F, and different viewing angle images are two-dimensional images.

It should be noted that, for convenience of description, only six viewing angle images are used in this embodiment to represent different viewing angles of the three-dimensional object. Those of ordinary skill in the art will be able to understand the number of viewing angle images that can be adjusted as a function of the actual situation.

3A and 3B are schematic views showing the relative viewing angles of the user and the three-dimensional object in accordance with an embodiment of the present invention.

As shown in FIG. 3A, the user 20 wears the display device 200 and aligns the three-dimensional object 30 with the user's line of sight. For example, if a center point 31 of the three-dimensional object 30 is used, a sphere region 300 is defined, and the point 32 is defined as a 0 degree angle in the horizontal plane. The angle between the line of sight of the user and the horizontal plane of the center point 31 (i.e., the angle of inclination) is φ, and the horizontal angle is θ. Therefore, the relative viewing angle between the user and the three-dimensional object at this time can be defined as (θ, φ). At this time, the user 20 can view the output image 50A corresponding to the relative angle of view (θ, φ) through the display device 200.

As shown in FIG. 3B, when the head of the user 20 is moved upward (for example, the user stands up) and the line of sight is still aligned with the three-dimensional object 30, the angle between the user's line of sight and the horizontal plane of the center point 31 ( That is, the inclination angle is increased to φ', but the horizontal angle is still maintained at θ. Therefore, the relative viewing angle between the user and the three-dimensional object at this time can be defined as (θ, φ'). At this time, the user 20 can view the output image 50B corresponding to the relative angle of view (θ, φ') through the display device 200.

It should be noted that the output image 50A or 50B seen by the user 20 may be calculated by interpolating the view image closest to the relative view angle and its adjacent view image.

Table 1-1 and Table 1-2 show different combinations of horizontal angles and inclinations in an embodiment of the present invention.

In the embodiments of Table 1-1 and Table 1-2, the viewing angle of the three-dimensional object can be set at a 45-degree angle on the horizontal axis and the vertical axis, and the horizontal angle θ ranges from 0 to 360 degrees, and the inclination angle is The range of φ is -90 degrees to 90 degrees.

Furthermore, when the inclination angle φ is 90 degrees or -90 degrees, it means that the user views the three-dimensional object from directly above and below the three-dimensional object. The angle of inclination of the user can be set to 45 degrees, 0 degrees, and -45 degrees, and the horizontal angle can be set every 45 degrees. Therefore, a combination of different relative viewing angles of Table 1-1 and Table 1-2 can be obtained. For example, the set of relative viewing angles C1 represents a horizontal angle of 0 degrees, the set C2 represents a horizontal angle of 45 degrees, and so on. The information of the relative viewing angle described above is recorded in the metadata of the video stream or the image file when the corresponding view image is encoded by the processor 110.

In some embodiments, the present invention can set different numbers of perspective images for different viewing angles. For example, the angles directly above and below the three-dimensional object are not viewed by the user, but the angles above and below the oblique angle (for example, the angle of inclination is 45 degrees and -45 degrees) are more easily used by the user. Watch it. For example, the viewing angle images directly above and below only need to set a single viewing angle image, but the viewing angles above and below the oblique angle are easily viewed by the user, so the angle of view can be divided into obliquely upward and obliquely below. The number of more angles of view, for example, can be divided into 12 angles at a horizontal angle of 45 degrees, that is, a horizontal angle every 30 degrees. Similarly, at an inclination angle of -45 degrees, the horizontal angle is equally divided into 12 angles, that is, a horizontal angle every 30 degrees.

In some embodiments, the three-dimensional object may have a specific standard shape, for example, a cube may be represented by a view angle of 6 angles of view, and an icosahedron may be represented by a view angle of 20 angles, wherein Each viewing angle is equidistant at intervals of vertical angles and equally spaced at horizontal angles.

Figure 4 is a flow chart showing a method for encoding and decoding a three-dimensional object in accordance with an embodiment of the present invention.

In step S410, the processor 110 is used to obtain a three-dimensional object. For example, the three-dimensional object may be a three-dimensional model calculated by the video device 100 or an actual three-dimensional object.

In step S420, the processor 110 is used to create a plurality of view images of the three-dimensional object, wherein each view image corresponds to viewing a view angle of the three-dimensional object.

In step S430, the processor 110 is used to encode the plurality of view images to generate a video stream, wherein the metadata of the video stream records the view angle corresponding to each view image.

In step S440, the video stream is decoded by the graphics processing unit 120 to obtain the plurality of view images. For example, the graphics processing unit 120 decodes the video stream according to the metadata of the video stream to obtain the plurality of view images. In addition, the graphics processing unit 120 can selectively perform image processing on the plurality of perspective images, such as background removal or image capture, and store the processed perspective images in the memory 130.

In step S450, the drawing processing unit 120 is used to calculate a relative viewing angle between a user and the three-dimensional object. For example, the drawing processing unit 120 The relative viewing angle may be calculated based on the sensing information from the sensing device 260, and may also be calculated using a stereoscopic projection method. In some embodiments, the calculation of the relative viewing angle is performed by the processor 110, and the processor 110 notifies the graphics processing unit 120 of the relative viewing angle information.

In step S460, the first processing unit image corresponding to the relative viewing angle is determined from the plurality of viewing angle images by the drawing processing unit 120.

In step S470, the drawing processing unit 120 generates an output image according to the first view image, and transmits the output image to the display device 200 for playing. For example, the output image may include a left eye image and a right eye image, which are respectively played by the left eye display panel 240 and the right eye display panel 250, and are focused by the left lens 214 and the right lens 251. The left eye and the right eye of the user can view the virtual reality image or the stereo image through the display device 200.

In summary, the present invention provides a video system and a three-dimensional object encoding and decoding method thereof, which can represent a three-dimensional object in a virtual reality by using a perspective image at different viewing angles, and at the encoding end is a three-dimensional object. The view image is encoded instead of directly encoding the three-dimensional object, so that the size of the generated video stream can be greatly encoded, and the memory bandwidth required for the operation of the encoding end and the decoding end is saved. In addition, at the decoding end, only the video stream is decoded according to the metadata of the video stream to obtain the respective view images of the three-dimensional object. The decoding end may also select a corresponding one of the three-dimensional object view images as the output image according to the relative viewing angle of the user and the three-dimensional object. Therefore, for the virtual reality device, the operations required at the encoding end and the decoding end are also greatly reduced, and the memory bandwidth required for the operation can also be saved.

The present invention has been disclosed in the above preferred embodiments, and is not intended to limit the scope of the present invention. Any one of ordinary skill in the art can make a few changes without departing from the spirit and scope of the invention. The scope of protection of the present invention is therefore defined by the scope of the appended claims.

Claims (8)

A video device includes: a memory; a processor for acquiring a three-dimensional object, and encoding a plurality of viewing angle images corresponding to the plurality of viewing angles of the three-dimensional object to generate a video stream; and a graphics processing unit Decoding the video stream to obtain a plurality of viewing angle images corresponding to the plurality of viewing angles of the three-dimensional object, and storing the plurality of viewing angle images in the memory; wherein the processor is the plural number of the three-dimensional object The image of the viewing angle is arranged as a two-dimensional image and spatially encoded to generate the video stream, and the metadata of the video stream records each view image corresponding to each view; wherein the drawing processing unit is based on one Sensing information of one of the display devices of the user determines a relative viewing angle between the user and the three-dimensional object, and calculates a corresponding viewing angle from the plurality of viewing angle images stored in the memory. One of the output images; wherein the drawing processing unit further transmits the output image to the display device for playing. The video device of claim 1, wherein the plurality of viewing angles of the three-dimensional object are equidistantly arranged at a vertical angle and a horizontal angle. The video processing device of claim 1, wherein after the drawing processing unit obtains the plurality of viewing angle images, the graphics processing unit further performs background removal and image capturing processing on the plurality of viewing angle images, and then After processing The plurality of viewing angle images are stored in the memory. The video processing device of claim 1, wherein the mapping processing unit determines, by the plurality of viewing angle images, a first viewing angle image that is closest to the relative viewing angle and obtains one or more viewing angle images adjacent thereto. And calculating the output image from the first view image and the adjacent view image. A three-dimensional object encoding and decoding method for a video device, wherein the video device comprises a processor and a graphics processing unit, the method comprising: using the processor to obtain a three-dimensional object, and the plurality of viewing angles of the three-dimensional object Corresponding multiple viewing angle images are encoded to generate a video stream; the graphics processing unit is used to decode the video stream to obtain a plurality of viewing angle images corresponding to the plurality of viewing angles; and the processor is used to select the three-dimensional object The plurality of viewing angle images are arranged as a two-dimensional image and spatially encoded to generate the video stream, wherein the video stream metadata records the respective viewing angle images corresponding to the respective viewing angles; Sensing information from one of the display devices of a user determines a relative viewing angle between the user and the three-dimensional object; and the drawing processing unit calculates the relative viewing angle from the plurality of viewing angle images An output image; and transmitting the output image to the display device by using the graphics processing unit Play. The three-dimensional object encoding and decoding method according to claim 5, wherein the plurality of viewing angles of the three-dimensional object are equidistantly arranged at a vertical angle and a horizontal angle. The method for encoding and decoding a three-dimensional object according to claim 5, further comprising: after the plurality of viewing angle images are acquired by the drawing processing unit, performing a background removal and image detection on the plurality of viewing angle images by using the drawing processing unit. Take the processing. The method for encoding and decoding a three-dimensional object according to claim 5, further comprising: determining, by the drawing processing unit, one of the plurality of viewing angle images that is closest to the first viewing angle image and obtaining an adjacent one of the plurality of viewing angle images Or a plurality of viewing angle images; and the drawing processing unit calculates the output image from the first viewing angle image and the adjacent viewing angle image.