CN115841539A - Three-dimensional light field generation method and device based on visual shell - Google Patents

Abstract

The present invention provides a method and device for generating a three-dimensional light field based on a visual shell, including: collecting continuous synchronous video streams of a target model to be reconstructed based on multiple RGB cameras; determining the parameters of a virtual camera array based on parameters of a three-dimensional light field display device Arrangement method; use the light field multi-view encoding method to determine all the sub-pixels and their corresponding virtual camera information required to generate the composite image; based on all the sub-pixels and their corresponding virtual camera information required to generate the composite image, control the corresponding multiple A virtual camera emits rendering light; based on the visible shell algorithm and multiple synchronous video frames, determine the three-dimensional intersection and non-intersecting points of the rendering light and the target model to be reconstructed, and determine the color, coordinates and non-intersecting points of the intersection points. Background color; generate a composite image based on the color and coordinates of the three-dimensional intersection point and the background color of the non-intersecting point; generate a three-dimensional light field based on the composite image. In this way, the rapid generation of three-dimensional light field can be realized.

Description

A method and device for generating a three-dimensional light field based on a visible shell

technical field

The invention relates to the technical field of three-dimensional imaging, in particular to a method and device for generating a three-dimensional light field based on a visible shell.

Background technique

In recent years, with the development of three-dimensional light field display technology, because it does not need to wear additional equipment, it can provide viewers with large viewing angles, high resolution, and detailed display effects. It has been widely used and attracted extensive attention of researchers at home and abroad.

However, for the method of 3D light field display, it is still difficult to realize real-time dynamic display of real scenes. Specifically, traditional light field content generation methods include: 3D reconstruction and multi-view rendering of real scenes, multi-view rendering of simulation models, and dense viewpoint generation. For the first type of method, it performs 3D reconstruction, multi-view rendering and synthetic image calculation process on the collected multi-view images, which can realize the 3D display of the real scene, but the modeling, rendering and other operations take a lot of time and cannot meet the requirements of 3D. Requirements for fast generation of light fields. The second type of method can realize real-time 3D display of static scenes, but its simulation model is pre-manufactured, which is not realistic enough, and for dynamic scenes, each frame needs to be modeled, which is costly and takes a lot of time, and cannot meet the requirements of 3D. Requirements for fast generation of light fields. The third type of method is based on deep learning for dense viewpoint generation, which often requires a training process, and is very sensitive to changes in the environment and characters, and lacks robustness and stability, and cannot meet the needs of rapid generation of 3D light fields.

Contents of the invention

The present invention provides a method and device for generating a three-dimensional light field based on a visible shell, which is used to solve the defect in the prior art that it takes a long time to generate a three-dimensional light field in a real scene, and realize rapid generation of a three-dimensional light field.

The present invention provides a method for generating a three-dimensional light field based on a visual shell, comprising: acquiring a continuous synchronous video stream of a target model to be reconstructed in a three-dimensional scene based on multi-channel RGB cameras, and the continuous synchronous video stream includes multiple synchronous video streams frame; determine the arrangement of the virtual camera array based on the parameters of the three-dimensional light field display device, and the virtual camera array includes a plurality of virtual cameras; based on the arrangement of the virtual camera array and the parameters of the three-dimensional light field display device, use The light field multi-viewpoint encoding method determines all sub-pixels and corresponding virtual camera information required for generating a composite image; based on all sub-pixels and corresponding virtual camera information required for generating a composite image, control the corresponding A virtual camera emits rendering light; based on the visible shell algorithm and the multi-channel synchronous video frame, determine the three-dimensional intersection point and non-intersection point between the rendering light and the target model to be reconstructed, and determine the color, coordinates and non-intersection point of the intersection point The background color of the point; generating a composite image based on the color and coordinates of the three-dimensional intersection point and the background color of the non-intersecting point; generating a three-dimensional light field based on the composite image.

In some embodiments, the acquisition of the continuous synchronous video stream of the target model to be reconstructed in the 3D scene based on multi-channel RGB cameras includes: synchronously collecting and saving the target model to be reconstructed in the 3D scene based on the multi-channel RGB cameras Continuously moving video stream, and after performing distortion correction, color correction and instance segmentation, a continuous video stream of the target model to be reconstructed is obtained, wherein the multiple RGB cameras are calibrated in the 3D scene before synchronous acquisition.

In some embodiments, the arrangement of the virtual camera array includes the number, position and orientation of the virtual cameras.

In some embodiments, based on the arrangement of the virtual camera array and the parameters of the three-dimensional light field display device, all sub-pixels and their corresponding virtual pixels required to generate the composite image are determined using a light field multi-viewpoint encoding method. Camera information, including: based on the maximum viewing angle range of the three-dimensional light field display device, determining the viewing angle range of the composite image; based on the arrangement of the virtual camera array and the viewing angle range of the composite image, determining the The sub-pixels and their corresponding virtual camera information.

In some embodiments, the controlling the corresponding plurality of virtual cameras to emit rendering rays based on all the sub-pixels required for generating the composite image and their corresponding virtual camera information includes: according to the information required for generating the composite image The virtual camera number corresponding to the sub-pixel is used to control the virtual camera corresponding to the virtual camera number to emit corresponding rendering rays.

In some embodiments, the visual hull algorithm and the multi-channel synchronous video frames determine the three-dimensional intersection points and non-intersection points of the rendering light and the target model to be reconstructed, and determine the color, coordinates, and non-intersection points of the intersection points. The background color of the intersection point, including: the coordinate point where the front end of the rendering ray is located in the process of searching the surface of the target model to be reconstructed in the forward direction, perspective projection into multiple synchronous video frames, and calculating all 2D projection points Coordinates; determine whether the coordinates of all the two-dimensional projected points are within the contour map of the target model to be reconstructed during the forward search process; if all the coordinates of the two-dimensional projected points are in the contour map of the target model to be reconstructed Within, it is determined that the coordinate point at the front end of the ray arrives at the surface of the target model to be reconstructed and intersects, and the coordinate point at the front end of the ray is determined as the coordinate of the three-dimensional intersection point, and according to the corresponding For a single video frame, the color of the three-dimensional intersection point is obtained; if the coordinates of the two-dimensional projection point are unevenly within the contour map of the target model to be reconstructed, it is determined that the coordinate point where the front of the light is located has not yet reached the surface of the target model to be reconstructed, Then continue to make the forward search of the rendering ray. If the coordinates of the coordinate point where the front end of the ray is located exceeds the range of the 3D scene where the target model to be reconstructed is located, then stop the forward search and determine that the rendering ray does not match the target model to be reconstructed. Intersect, and determine the coordinate point where the front end of the light is located as a non-intersecting point, and determine the color of the non-intersecting point as the background color.

The present invention also provides a three-dimensional light field generation device based on a visible shell, including: multiple RGB cameras, used to synchronously acquire continuous synchronous video streams of the target model to be reconstructed in the three-dimensional scene, and the continuous synchronous video streams include Multi-channel synchronous video frames; an optical computing device, used to determine the arrangement of the virtual camera array based on the parameters of the three-dimensional light field display device, and the virtual camera array includes a plurality of virtual cameras; and also used for the arrangement based on the virtual camera array method, the parameters of the three-dimensional light field display device, using the light field multi-viewpoint encoding method to determine all the sub-pixels required to generate the composite image and the corresponding virtual camera information; it is also used to generate all the sub-pixels required to generate the composite image based on the and its corresponding virtual camera information, controlling the corresponding plurality of virtual cameras to emit rendering rays; it is also used to determine the three-dimensional intersection between the rendering rays and the target model to be reconstructed based on the visible shell algorithm and the multi-channel synchronous video frame points and non-intersecting points, and determine the color of the intersecting point, the coordinates and the background color of the non-intersecting point; it is also used to generate a composite map based on the color, coordinates and the background color of the non-intersecting point of the three-dimensional intersection point; three-dimensional light A field display device for generating a three-dimensional light field based on the composite map.

The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and operable on the processor. When the processor executes the program, the visual-based Steps of a method for generating a 3D light field for a shell.

The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the methods for generating a three-dimensional light field based on a visible shell as described above can be realized. .

The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of any method for generating a three-dimensional light field based on a visible shell as described above are implemented.

The present invention provides a method and device for generating a three-dimensional light field based on a visible shell. The necessary sub-pixel information in the composite image is determined by the light field multi-viewpoint coding method, and then based on the visible shell algorithm, the sub-pixel information in the composite image is determined. Correspondingly, the necessary 3D model points and colors are reconstructed, that is, the calculation of the necessary sub-pixel information in the composite image is completed, the calculation speed of the composite image is improved, and the effect of real-time display is achieved. The displayed content is closer to reality, and has Good mobility and robustness improve the perception and utilization of 3D information for 3D light field display, making the light field display of dynamic scenes more flexible, saving time and computing costs.

Description of drawings

In order to more clearly illustrate the present invention or the technical solutions in the prior art, the accompanying drawings that need to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are the present invention. For some embodiments of the invention, those skilled in the art can also obtain other drawings based on these drawings without creative effort.

Fig. 1 is a schematic flow chart of a method for generating a three-dimensional light field based on a visible shell provided by the present invention;

Fig. 2 is a schematic diagram of the arrangement of the virtual camera array provided by some embodiments of the present invention;

Fig. 3 is a schematic diagram of the GPU multi-thread calculation process provided by the present invention

Fig. 4 is a schematic diagram of a three-dimensional light field generating device based on a visible shell provided by the present invention

Fig. 5 is a schematic structural diagram of an electronic device provided by the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention , but not all examples. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

A method for generating a three-dimensional light field based on a visible shell of the present invention will be described below with reference to FIGS. 1-3 .

Fig. 1 is one of the schematic flow charts of a method for generating a three-dimensional light field based on a visible shell provided by the present invention, including steps S101 to S107:

In step S101, a continuous synchronous video stream of a target model to be reconstructed in a 3D scene is acquired based on multiple RGB cameras, and the continuous synchronous video stream includes multiple synchronous video frames.

Build a multi-channel RGB camera acquisition device in the real-world 3D scene, calibrate the multi-channel RGB camera in the 3D scene, and the multi-channel RGB camera synchronously collect and save the video stream of the continuous motion of the target model to be reconstructed in the 3D scene, and perform distortion After rectification, color correction and instance segmentation, a continuous synchronized video stream is obtained. The target model to be reconstructed is a three-dimensional model.

In some embodiments, synchronous acquisition of large viewing angles and multi-viewpoint light fields can be realized in three-dimensional space by building a multi-channel RGB camera synchronous acquisition array. The multi-channel RGB camera array does not force the adjustment of the position and orientation of each camera. Follow the Arbitrary distribution.

In some embodiments, based on the OBS (Open Broadcaster Software) open source software, the acquisition and local storage of multiple synchronous video streams can be performed on the target model to be reconstructed in continuous motion, and multiple channels can also be performed through an incremental camera calibration method. Camera internal and external reference calibration and distortion correction.

In some embodiments, color correction may be performed on the captured video stream based on a color calibration panel;

In some embodiments, the target model to be reconstructed in each channel and frame of video can be instance-segmented based on an instance segmentation algorithm; the instance segmentation algorithm includes a deep learning algorithm, a green screen matting method, and the like.

Step S102, determining an arrangement mode of a virtual camera array based on parameters of a three-dimensional light field display device, where the virtual camera array includes a plurality of virtual cameras.

A 3D light field display device is used to display a 3D light field. Different 3D light field display devices have different parameters. For example, different 3D light field display devices have different display angles, the shortest viewing distance, the longest viewing viewing distance etc. In some embodiments, the arrangement of the virtual camera array can be determined according to parameters such as the number of viewpoints of the three-dimensional light field display device and the optimal viewing distance.

In some embodiments, the arrangement of the virtual camera array includes the number, position and orientation of the virtual cameras. For example, the orientation of the virtual camera is parallel, converging or off-axis. As an example, FIG. 2 is a schematic diagram of an arrangement manner of a virtual camera array provided by some embodiments of the present invention. As shown in Figure 2, for the position of each virtual camera, the vertical distance between the virtual camera array and the center of the target model to be reconstructed can be set according to the optimal viewing distance of the 3D light field display device and the artificially selected viewing angle d and the distance l between every two virtual cameras.

In some embodiments, the conditions met when the virtual cameras are arranged include that the overall viewing angle range of the virtual camera array is smaller than the overall collection viewing angle range of the RGB camera array, and the total rendering viewing angle range of the virtual camera array must be within the viewing angle range provided by the three-dimensional light field display device. , the position of the virtual camera array should be between the shortest viewing distance and the longest viewing distance provided by the three-dimensional light field display device.

Step S103, based on the arrangement of the virtual camera array and the parameters of the 3D light field display device, use a light field multi-viewpoint encoding method to determine all sub-pixels and their corresponding virtual camera information required to generate a composite image.

The sub-pixels required for the composite image and the corresponding virtual camera information represent each sub-pixel required for the composite image, and which virtual camera obtains each sub-pixel. For example, the corresponding virtual camera information may index the sub-image. As an example, for a synthetic image with a resolution of 4K, it needs 3840x 2160x 3 sub-pixels, and the sub-image index of a certain sub-pixel is 10, which means that the color of this sub-pixel comes from virtual camera No. 10, and the synthetic image requires Each sub-pixel of and its corresponding virtual camera information can be determined using a light field multi-view encoding method.

Multiple sub-pixels and their corresponding virtual camera information required by the composite image are determined by the light field multi-view encoding method, thereby avoiding the complete target model reconstruction task, and only the necessary multi-view information in the composite image can be rendered. The synthetic graph obtained by this method avoids the calculation of redundant information. As an example, in order to generate a composite image with a resolution of 4k, in the traditional method, after the 3D reconstruction of the target model is completed, 4k sub-images from multiple viewpoints are rendered, and then a 4k composite image is synthesized. The 3D reconstruction of a complete object model contains more redundant information. In addition, the resolution of the composite image is the same as the sub-image resolution of multiple viewpoints, both are 4k, so the sub-pixel information with a resolution of 4k for each viewpoint only uses a small part of it. For example, if there are 60 virtual Viewpoints, the subpixels of the 4k subimages of each viewpoint only use 1/60, based on the 1/60 subpixel information corresponding to 60 viewpoints to generate a composite image, other redundant 59/60 subpixel information cannot It is used to generate a composite graph, which also slows down the calculation speed when computing the composite graph.

However, from the perspective of generating a composite image, the present invention first judges all the sub-pixels required for generating the composite image and their corresponding virtual camera information, and then only calculates the required 3D model point information to obtain the sub-pixels required for generating the composite image. Therefore, the synthesis efficiency of the composite image is greatly improved, and the waste of other redundant pixel information is avoided when the composite image is calculated.

In some embodiments, based on the maximum viewing angle range of the three-dimensional light field display device, the viewing angle range of the composite image is determined; based on the arrangement of the virtual camera array and the viewing angle range of the composite image, determine the All required sub-pixels and their corresponding virtual camera information. For example, if the maximum viewing angle range of a three-dimensional light field display device is 60 degrees, the present invention only displays the information required to calculate the 60-degree viewing angle, that is, calculates the multiple sub-pixel indexes and corresponding virtual camera information required to generate the composite image, By only calculating the target model to be reconstructed within the effective viewing angle range, and not calculating the complete three-dimensional model, the calculation efficiency is improved, and the real-time performance of the overall calculation process can be guaranteed.

Step S104, based on all the sub-pixels required for generating the composite image and the corresponding virtual camera information, control the corresponding multiple virtual cameras to emit rendering rays.

In some embodiments, according to the virtual camera number corresponding to the sub-pixel required to generate the composite image, the virtual camera corresponding to the virtual camera number is controlled to emit a corresponding rendering light. For example, according to the viewpoint index number corresponding to the sub-pixel required to generate the composite image, the corresponding virtual camera can be selected to emit the corresponding rendering light, so as to detect the coordinates and color of the 3D model point corresponding to the sub-pixel, that is, the color of the sub-pixel. For example, the sub-image index of the No. 1 sub-pixel is 10, which means that the color of the No. 1 sub-pixel comes from the No. 10 virtual camera, and the No. 1 sub-pixel and its corresponding virtual camera information required for the composite image can be expressed as No. 1 The sub-pixel and its corresponding virtual camera No. 10 control the No. 10 virtual camera to emit rendering light to detect the coordinates and color of the 3D model point corresponding to the No. 1 sub-pixel, and this color is the color of the No. 1 sub-pixel.

The sub-image index indicates the viewpoint number of the sub-image displayed by the 3D light field display device at a certain viewing position, and also indicates the virtual camera number corresponding to the sub-pixel. The sub-image index is calculated by the light field multi-view coding method.

Controlling the corresponding multiple virtual cameras to emit rendering rays can mean performing multi-viewpoint rendering on the target model to be reconstructed from multiple viewpoints. Multi-viewpoint rendering is a real-time rendering method for 3D models, which can generate images of target objects under any observation conditions. virtual viewpoint.

Step S105, based on the visible shell algorithm and the multi-channel synchronous video frames, determine the three-dimensional intersection and non-intersection points of the rendering light and the target model to be reconstructed, and determine the color of the intersection point, the coordinates and the background color of the non-intersection point .

Visual hull algorithm (visual hull) is a real-time 3D model reconstruction method. It obtains the multi-viewpoint image and the extracted contour map information about the target model to be reconstructed, and determines the intersection convex hull of the target object in the three-dimensional space through the mathematical calculation of perspective projection, so as to realize the reconstruction of the target model.

In some embodiments, the coordinate points where the front end of the rendering light is located in the process of forward searching the surface of the target model to be reconstructed are perspective-projected into multiple synchronous video frames, and the coordinates of all two-dimensional projection points are calculated;

Judging during the forward search process, whether the coordinates of all the two-dimensional projected points are within the contour map of the target model to be reconstructed;

If the coordinates of all the two-dimensional projection points are within the contour map of the target model to be reconstructed, then determine that the coordinate point at the front end of the light reaches the surface of the target model and intersects, and determines the coordinate point at the front end of the rendering light as a three-dimensional The coordinates of the intersection point, and according to the single video frame corresponding to the two-dimensional projection point, obtain the color of the three-dimensional intersection point;

If the coordinates of the two-dimensional projection point are unevenly within the contour map of the target model to be reconstructed, it is determined that the coordinate point where the front end of the rendering ray is located has not yet reached the surface of the target model to be reconstructed, and then continue to search forward for the rendering ray, if When the coordinates of the front-end point of the ray exceed the range of the three-dimensional scene where the target model to be reconstructed is located, stop the forward search, determine that the rendering ray does not intersect the target model, and determine the front-end point of the ray as a non-intersecting point, and The color of the non-intersecting points is determined as the background color.

Step S106, generating a composite image based on the color and coordinates of the three-dimensional intersection point and the background color of the non-intersection point.

After the color of each sub-pixel in the composite image is solved, the composite image can be generated based on the color and coordinates of the three-dimensional intersection point and the background color of the non-intersection point.

Step S107, generating a three-dimensional light field based on the composite image.

A three-dimensional light field display device generates a three-dimensional light field based on a synthetic image, so as to display a frame of image. Through the continuous calculation of the composite image, the three-dimensional light field can generate and display multiple frames of images in real time, so as to achieve the purpose of real-time video display.

In order to improve computing efficiency, this 3D light field generation method is implemented using GPU parallel computing, each GPU thread is responsible for the computing task of a virtual ray, and the computing process is also accelerated using CUDA (Compute Unified Device Architecture, unified computing device architecture), so as to achieve Process the continuous video stream captured by the RGB real camera in real time to generate a real-time 3D light field. FIG. 3 is a schematic diagram of the GPU multi-thread calculation process provided by the present invention. As shown in Figure 3, after the GPU starts computing, it first reads data and initializes parameters, then starts the GPU kernel function for parallel computing, and calculates the coordinates of all model points required to generate the composite map through synchronous computing of GPU multi-threads And the corresponding texture, that is, the calculation of the color of each sub-pixel in the composite image is completed, and then the composite image is sent to a three-dimensional light field display device for display. Each GPU thread is responsible for the calculation task of a virtual ray, and the calculation task of each virtual ray includes: projecting a rendering ray through the virtual camera in the virtual camera array, and then calculating the rendering ray and the 3D model to be reconstructed based on the visible shell algorithm Intersecting 3D point coordinates, and then calculate the texture corresponding to the 3D model point, if the rendering light does not intersect with the 3D model to be reconstructed, assign the corresponding non-intersecting point as the background color, that is, the reconstruction of the 3D model point corresponding to this virtual light is realized And the calculation of the corresponding sub-pixel color in the composite image. Finally, the calculation tasks of multiple virtual rays are calculated synchronously through GPU multithreading, and the coordinates and textures of multiple 3D model points are obtained, and a composite image is generated.

Based on the same inventive concept, an embodiment of the present invention provides a three-dimensional light field generation device based on a visual shell, as shown in Figure 4, the following describes the three-dimensional light field generation device provided by the present invention, and the following description is based on a visual The three-dimensional light field generation device of the housing and the three-dimensional light field generation method described above can be referred to each other correspondingly:

A three-dimensional light field generation device based on a visual shell, comprising: multiple RGB cameras 41, used to synchronously acquire a continuous synchronous video stream of a target model to be reconstructed in a three-dimensional scene, and the continuous synchronous video stream includes multiple synchronous Video frame; optical computing device 42, used to determine the arrangement of the virtual camera array based on the parameters of the three-dimensional light field display device, the virtual camera array includes a plurality of virtual cameras; it is also used to determine the arrangement based on the virtual camera array, The parameters of the three-dimensional light field display device, using the light field multi-viewpoint encoding method to determine the multiple sub-pixels required to generate the composite image and their corresponding virtual camera information; The corresponding virtual camera information is used to control the corresponding plurality of virtual cameras to emit rendering light; it is also used to determine the three-dimensional intersection point and non-intersecting point, and determine the color, coordinates and background color of the non-intersecting point of the intersecting point; it is also used to generate a composite map based on the color, coordinates and background color of the non-intersecting point of the three-dimensional intersection point; three-dimensional light field display A device 43, configured to generate a three-dimensional light field based on the composite image.

FIG. 5 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG. 5, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, Wherein, the processor 510 , the communication interface 520 , and the memory 530 communicate with each other through the communication bus 540 . The processor 510 can call the logic instructions in the memory 530 to execute a method for generating a three-dimensional light field, the method comprising: acquiring a continuous synchronous video stream of a target model to be reconstructed in a three-dimensional scene based on multi-channel RGB cameras, and the continuous synchronous video stream The stream includes multiple synchronous video frames; the arrangement of the virtual camera array is determined based on the parameters of the three-dimensional light field display device, and the virtual camera array includes a plurality of virtual cameras; based on the arrangement of the virtual camera array, the three-dimensional light field For the parameters of the display device, use the light field multi-viewpoint encoding method to determine all the sub-pixels required to generate the composite image and their corresponding virtual camera information; based on all the sub-pixels required to generate the composite image and their corresponding virtual camera information, Control the corresponding plurality of virtual cameras to emit rendering rays; determine the three-dimensional intersection points and non-intersecting points of the rendering rays and the target model to be reconstructed based on the visible shell algorithm and the multi-channel synchronous video frame, and determine the intersection point Color, coordinates, and background color of the non-intersecting point; generating a composite image based on the color, coordinates, and background color of the three-dimensional intersection point; generating a three-dimensional light field based on the composite image.

In addition, the above logic instructions in the memory 530 may be implemented in the form of software function units and be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes. .

On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can Executing the three-dimensional light field generation method provided by the above-mentioned methods, the method includes: collecting and obtaining a continuous synchronous video stream of a target model to be reconstructed in a three-dimensional scene based on multi-channel RGB cameras, and the continuous synchronous video stream includes multiple synchronous video frames ; Determine the arrangement of the virtual camera array based on the parameters of the three-dimensional light field display device, the virtual camera array includes a plurality of virtual cameras; based on the arrangement of the virtual camera array and the parameters of the three-dimensional light field display device, using light The field multi-viewpoint encoding method determines all sub-pixels required for generating a composite image and corresponding virtual camera information; based on all sub-pixels required for generating a composite image and corresponding virtual camera information, controls the corresponding multiple The virtual camera emits rendering light; based on the visible shell algorithm and the multi-channel synchronous video frame, determine the three-dimensional intersection and non-intersection points of the rendering light and the target model to be reconstructed, and determine the color, coordinates and non-intersection points of the intersection points the background color of the 3D intersection point; generate a composite image based on the color and coordinates of the three-dimensional intersection point and the background color of the non-intersection point; generate a three-dimensional light field based on the composite image.

In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it is implemented to perform the three-dimensional light field generation method provided by the above methods, the method Including: obtaining the continuous synchronous video stream of the target model to be reconstructed in the 3D scene based on multi-channel RGB camera acquisition, the continuous synchronous video stream includes multiple synchronous video frames; determining the arrangement of the virtual camera array based on the parameters of the three-dimensional light field display device In this way, the virtual camera array includes a plurality of virtual cameras; based on the arrangement of the virtual camera array and the parameters of the three-dimensional light field display device, a light field multi-viewpoint encoding method is used to determine all the sub-pixels required to generate the composite image and its corresponding virtual camera information; based on all sub-pixels and corresponding virtual camera information required for generating the composite image, control the corresponding multiple virtual cameras to emit rendering light; based on the visible shell algorithm and the multiple synchronous video frame, determine the three-dimensional intersection point and non-intersection point of the rendering light and the target model to be reconstructed, and determine the color, coordinates and background color of the non-intersecting point; based on the color, coordinates and non-intersecting point of the three-dimensional intersection point The background colors of the non-intersecting points generate a composite map; a three-dimensional light field is generated based on the composite map.

The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without any creative efforts.

Through the above description of the implementations, those skilled in the art can clearly understand that each implementation can be implemented by means of software plus a necessary general hardware platform, and of course also by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic discs, optical discs, etc., including several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (10)

1. A three-dimensional light field generation method based on a visual shell is characterized by comprising the following steps:

acquiring a continuous synchronous video stream of a target model to be reconstructed in a three-dimensional scene based on multiple RGB cameras, wherein the continuous synchronous video stream comprises multiple synchronous video frames;

determining an arrangement of a virtual camera array based on parameters of a three-dimensional light field display device, the virtual camera array comprising a plurality of virtual cameras;

determining all sub-pixels required for generating a synthetic image and corresponding virtual camera information thereof by using a light field multi-view coding method based on the arrangement mode of the virtual camera array and the parameters of the three-dimensional light field display equipment;

controlling the corresponding virtual cameras to emit rendering rays based on all sub-pixels and corresponding virtual camera information required for generating the composite image;

determining a three-dimensional intersection point and a non-intersection point of a rendering ray and a target model to be reconstructed based on a visual shell algorithm and the multi-path synchronous video frame, and determining the color, the coordinate and the background color of the non-intersection point of the rendering ray and the target model to be reconstructed;

generating a composite map based on the color and the coordinates of the three-dimensional intersection points and the background color of the non-intersection points;

generating a three-dimensional light field based on the composite map.

2. The method for generating a three-dimensional light field based on a visual shell according to claim 1, wherein the acquiring of the continuous synchronous video stream of the object model to be reconstructed in the three-dimensional scene based on the multiple RGB cameras comprises: and synchronously acquiring and storing a video stream of continuous motion of a target model to be reconstructed in the three-dimensional scene based on the multiple RGB cameras, and obtaining the continuous video stream of the target model to be reconstructed after distortion correction, color correction and instance segmentation, wherein the multiple RGB cameras are calibrated in the three-dimensional scene before synchronous acquisition.

3. The visual shell-based three-dimensional light field generation method of claim 1 wherein the arrangement of the array of virtual cameras includes the number, position and orientation of virtual cameras.

4. The method of claim 1, wherein the determining all sub-pixels and their corresponding virtual camera information required for generating the composite image using a light field multi-view coding method based on the arrangement of the virtual camera array and the parameters of the three-dimensional light field display device comprises: determining a view angle range of a composite map based on a maximum view angle range of the three-dimensional light field display device; and determining the sub-pixels required for generating the composite image and the corresponding virtual camera information based on the arrangement mode of the virtual camera array and the visual angle range of the composite image.

5. The method of claim 1, wherein the controlling the corresponding plurality of virtual cameras to emit rendering rays based on all sub-pixels required for the generating of the composite map and their corresponding virtual camera information comprises: and controlling the virtual camera corresponding to the virtual camera number to emit corresponding rendering rays according to the virtual camera number corresponding to the sub-pixel required by the generation of the composite image.

6. The method for generating a three-dimensional light field based on a visual shell according to claim 1, wherein the determining the three-dimensional intersection points and non-intersection points of the rendering rays and the object model to be reconstructed based on the visual shell algorithm and the multi-path synchronous video frame, and determining the color, coordinates and background color of the intersection points and the non-intersection points comprises: the coordinate point where the front end of the rendering light ray is located in the process of searching the surface of a target model to be reconstructed in the forward direction is projected to a multi-path synchronous video frame in a perspective mode, and the coordinates of all two-dimensional projection points are calculated;

judging whether all the two-dimensional projection point coordinates are in a contour map of the target model to be reconstructed or not in the forward searching process;

if all the coordinates of the two-dimensional projection points are in the contour map of the target model to be reconstructed, determining that the coordinate point where the front end of the light ray is located reaches the surface of the target model to be reconstructed and intersects, determining the coordinate point where the front end of the light ray is located as the coordinate of the three-dimensional intersection point, and obtaining the color of the three-dimensional intersection point according to a single video frame corresponding to the two-dimensional projection points;

if the coordinates of the two-dimensional projection points are not all in the contour map of the target model to be reconstructed, determining that the coordinate point of the front end of the light ray does not reach the surface of the target model to be reconstructed, continuing to perform forward search on the rendering light ray, if the coordinates of the coordinate point of the front end of the light ray exceed the three-dimensional scene range of the target model to be reconstructed, stopping the forward search, determining that the rendering light ray is not intersected with the target model to be reconstructed, determining the coordinate point of the front end of the light ray as a non-intersection point, and determining the color of the non-intersection point as the background color.

7. A three-dimensional light field generating apparatus based on a visual shell, comprising:

the system comprises a plurality of paths of RGB cameras, a reconstruction module and a reconstruction module, wherein the RGB cameras are used for synchronously acquiring continuous synchronous video streams of a target model to be reconstructed in a three-dimensional scene, and the continuous synchronous video streams comprise a plurality of paths of synchronous video frames;

an optical computing device to determine an arrangement of a virtual camera array based on parameters of a three-dimensional light field display device, the virtual camera array including a plurality of virtual cameras;

determining all sub-pixels required for generating a synthetic image and corresponding virtual camera information thereof by using a light field multi-viewpoint coding method based on the arrangement mode of the virtual camera array and the parameters of the three-dimensional light field display equipment;

controlling the corresponding virtual cameras to emit rendering rays based on all the sub-pixels required for generating the composite image and the corresponding virtual camera information;

determining a three-dimensional intersection point and a non-intersection point of a rendering ray and a target model to be reconstructed based on a visual shell algorithm and the multi-path synchronous video frame, and determining the color, the coordinate and the background color of the non-intersection point of the rendering ray and the target model to be reconstructed;

generating a composite map based on the color and the coordinates of the three-dimensional intersection points and the background color of the non-intersection points;

and the three-dimensional light field display device is used for generating a three-dimensional light field based on the synthetic graph.

8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method for generating a three-dimensional light field based on a visual shell according to any one of claims 1 to 6 when executing the program.

9. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program, when being executed by a processor, implements the steps of the visual shell-based three-dimensional light field generation method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, wherein the computer program when executed by a processor implements the steps of the visual shell-based three-dimensional light field generation method of any one of claims 1 to 6.

Images