RU2825722C1 - Visualization of reconstruction of 3d scene using semantic regularization of normals tsdf when training neural network - Google Patents

Abstract

FIELD: image processing.

SUBSTANCE: method for reconstruction of 3D scene and its visualization using a neural network consisting of a basic neural network, including a skeleton, which is the main part of the basic neural network, which calculates features, a TSDF head, which performs prediction of TSDF values in each voxel, and a segmentation head, which is 3D sparse convolutional segmentation module, which predicts the segmentation marks in each voxel, comprising steps of: basic neural network is trained to obtain TSDF volume for scene voxels as follows, training data is input, TSDF loss between TSDF prediction is calculated, calculating the segmentation loss between the segmentation prediction and the segmentation reference, calculating the coordinates of the normals, calculating the total loss function as the total loss function TSDF, minimizing the total loss function, using a trained basic neural network to obtain a TSDF volume of an input sequence of RGB frames of a real scene, applying an algorithm to the TSDF volume, which calculates reconstruction of 3D scene, 3D scene reconstruction is rendered to obtain 3D scene visualization.

EFFECT: high accuracy of reconstructing 3D scene.

11 cl, 2 dwg, 3 tbl

Description

Field of technology to which the invention relates

The present invention can find application in computer vision for creating a 3D scene reconstruction and visualizing the results of this 3D scene reconstruction, which can be used for visual localization, visual analysis of scene contents, in particular, for detecting and segmenting a scene in 3D space, assessing a room plan, as well as for visualizing the interior space of rooms based on the created 3D model, and for solving other problems.

Description of the prior art

3D reconstruction is a core task in computer vision, applied in areas such as robotics and AR/VR. These application scenarios require finely detailed, plausible, and dense reconstructions.

However, obtaining such reconstructions still poses a challenge for modern reconstruction methods due to the imperfection of the input data: incomplete observations in the form of occluded and invisible regions, interframe inconsistency, and inevitable measurement errors. Therefore, reconstruction artifacts may occur. Global artifacts include corrupted scene structure and incorrect partitioning; such serious flaws make the reconstructed scene practically useless.

In addition, many local artifacts can occur: for example, duplicated surfaces that could not be superimposed after loop closure, holes in closed and invisible areas, non-flat walls and floors covered with depressions and elevations. Although such artifacts are not so dramatic, they still impose limitations, for example, on navigation scenarios that require precise estimation of room boundaries.

Dense 3D reconstruction from RGB images traditionally involves estimating depth maps for RGB images and merging them, which is another potential source of error. Recent 3D reconstruction methods attempt to minimize this undesirable effect by directly predicting the Truncated Signed distance function (TSDF), which conveniently fits the scene in 3D space; using this function with the Marching Cubes algorithm allows reconstructing a mesh or point cloud corresponding to the given scene, see e.g., https://en.wikipedia.org/wiki/Signed_distance_function). Such methods extract image features using a 2D CNN, project them back to 3D space, aggregate them, and predict the final TSDF volume using a 3D CNN. Although this is an effective way to consider all input images together, using TSDF does not address the issues of global scene structure and flat surfaces.

In the vast majority of rooms, the walls are flat and vertical, and the floor is flat and horizontal. This basic, non-limiting knowledge of the geometry of the indoor scene can be used in the reconstruction process.

Recently, methods have been presented that can reconstruct a 3D scene by estimating the TSDF volume directly from images. The TSDF volume is an effective way to consider all input images together, but combining features from multiple views remains a challenging task. Accordingly, progress in this area of TSDF reconstruction methods is mainly related to the development of feature aggregation strategies.

Atlas [2] first implemented an end-to-end 3D reconstruction model in which image features are back-projected and accumulated in a voxel-based volume, which is then fed to a 3D CNN predicting a TSDF (Truncated Signed Distance Function) volume. A signed distance function (SDF) is the orthogonal distance of a given point x to the boundary of a set Ω in a metric space, where the sign is determined by whether x is inside Ω or not. SDF is a standard way of encoding 3D space (the distance between points in space and the object in question). TSDF is a modification of SDF in which large distance values are truncated to speed up the conversion of the 3D reconstruction algorithm. Signed distance functions are commonly used for 3D reconstruction and have appeared several times in the prior art, for example in "A volumetric method for building complex models from range images", SIGGRAPH, 1996.

Due to its simplicity, the TSDF volume is a widely used representation for 3D reconstructions and is also used in modern reconstruction approaches. Atlas uses a clearly suboptimal averaging strategy for image feature fusion, which is revisited in subsequent approaches. NeuralRecon [3] uses a hierarchical fusion strategy: it averages features from neighboring views and fusions them across view clusters using an RNN network.

NeuralRecon demonstrates real-time performance on serial inputs. VoRTX [1] overcomes the limitations of serial processing by combining all views together.

This method introduces a transformer model for TSDF reconstruction and incorporates a 20-feature attention mechanism to fuse multiple views. Another transformer-based model, TransformerFusion [4], uses voxel-level attention to pay attention to the most informative features in images from different views.

Surface normals represent the scene geometry in a way that they are in some sense complementary to depth maps. Therefore, they can be used to constrain depth estimates or considered as a separate source of spatial information. Therefore, the use of normals for 3D reconstruction has been actively studied in recent years. VolSDF [6] and NeUS [7] minimize the photometric loss and further constrain the SDF using an eikonal loss, forcing the normals estimated as the gradients of the SDF to have norm equal to 1. NeuRIS [12] regularizes the predicted normals using a priori normals predicted by a trained method and exploits the multi-view photometric consistency between normals and depths to filter out invalid constraints. NeuralRoom [18] also uses a normal estimation network and applies a normal loss for textureless regions that cannot be effectively constrained by the photometric consistency loss due to shape and brightness ambiguities. Unlike NeurlS and NeuralRoom, in the proposed invention, normals are obtained directly from the predicted TSDF itself.

In addition, various ways of applying spatial segmentation for 3D reconstruction were discussed.

SceneCode [19] obtains a VAE segmentation representation given an RGB image and solves the segmentation label fusion problem by jointly optimizing spatially-aware low-dimensional codes of overlapping images.

A number of methods [13-17] use object recognition and their replacement with CAD models. Although the results obtained are visually plausible, they are unlikely to be able to reconstruct the real scene; rather, a 3D model is created that is more or less similar to it. The proposed invention, on the contrary, is aimed at reconstructing the real scene.

Manhattan-SDF [5], which is the closest analogue of the proposed normal segmentation regularization, NSR (Normal-Segmentation Regularization), focuses on improving the reconstruction of low-texture regions. Manhattan-SDF identifies floor and wall regions using a pre-trained segmentation network and forces the surface normals of floors and walls to be collinear with the three dominant directions so that the resulting reconstruction satisfies the Manhattan world assumption. However, indoor benchmarks such as ScanNet contain non-Manhattan scenes with more than three dominant directions, for which Manhattan-SDF is not applicable.

Recent 3D volumetric reconstruction methods have an undesirable trade-off when relying on input data to reconstruct the correct scene geometry. Even minor errors or inconsistencies in the input data can lead to corruption of the reconstructed scene. Global artifacts can manifest themselves as distorted room shapes, and reconstructed surfaces can contain local artifacts such as depressions, holes, and elevations. Fortunately, some of these problems can be addressed using prior knowledge of the scene, since most rooms are surrounded by flat vertical walls and flat horizontal floors.

The essence of the invention

A method is proposed for reconstructing a 3D scene and visualizing it using a neural network consisting of a base neural network including a TSDF skeleton and head, and a segmentation head, containing the following stages:

train a basic neural network to obtain volume TSDFs for scene voxels, for which the following steps are performed:

- input training data, including a training sequence of RGB frames with corresponding camera position data, into the skeleton;

- calculate the TSDF loss between the TSDF prediction obtained by the TSDF head from the skeleton data outputs and the reference scan;

- obtain, using the segmentation head, from the output of the skeleton data a segmentation prediction defining the "floor", "walls", "other" regions for each voxel;

- calculate the segmentation loss between the segmentation forecast and the segmentation standard;

- calculate normal coordinates as gradients of the TSDF prediction for the TSDF values in each voxel over all voxels in the scene;

- calculate the normal loss for wall region normals and floor region normals based on the TSDF reference results and TSDF prediction gradients;

- calculate the overall loss function as the sum of the TSDF loss, segmentation loss, and regular normal loss;

- minimize the overall loss function;

use a trained base neural network to obtain the TSDF of the volume of the input sequence of RGB frames of a real scene;

apply an algorithm to the TSDF volume that computes the reconstruction of the 3D scene;

render the reconstruction of the 3D scene to obtain a visualization of the 3D scene.

In this case, the algorithm that calculates the reconstruction of the 3D scene is the marching cubes algorithm. The segmentation head and the TSDF head are used in parallel during training. The method contains additional stages implemented after the stage of calculating the normal coordinates, in which:

select the normal in the wall areas and calculate their vertical components;

select the normal in the floor areas and calculate their horizontal components.

The normal loss calculation step may be as follows:

at each point in the areas where the segmentation head predicts a "wall", the vertical components of the normal vectors are considered, and the length of the z-component of the normal vector is added to the usual normal losses,

and at each point in the regions where the segmentation head predicts "floor", consider the horizontal components of the normal vectors, and also consider the x- and y-components of the normal vector, and add the norm of the two-dimensional vector consisting of these two components to the usual normal loss.

The overall loss function can be calculated as the sum of the TSDF loss, segmentation loss, and normal loss. Minimization of the overall loss function can be implemented by calculating the gradient of the overall loss function over all parameters of the base neural network. The method can additionally provide backpropagation of the error of the minimized overall loss function and updating the parameters of the base neural network in accordance with the minimized loss function, while updating the parameters of the base neural network. The training stages are repeated until the overall loss function stops decreasing. In another embodiment, the training stages are repeated until the overall loss function reaches a specified threshold value.

A computing device is also proposed, containing a processor and memory, in which instructions for performing the steps of the proposed method are stored.

At least one of the plurality of modules may be implemented as a neural network. The function associated with the neural network may be performed by means of non-volatile memory, volatile memory, and a processor.

The processor may include one or more processors. In this case, the one or more processors may be a general-purpose processor, such as a central processing unit (CPU), an application processor (AP) or the like, a graphics-only processing unit, such as a graphics processing unit (GPU), a visual processing unit (VPU), and/or a special processor for artificial intelligence, such as a neural processing unit (NPU).

These one or more processors control the processing of input data in accordance with a predetermined operating rule or artificial intelligence (AI) network stored in non-volatile memory or volatile memory and implemented by hardware. The hardware is a machine-readable medium on which software is stored.

In this context, provision by training means that by applying a training algorithm to a set of training data, a predetermined operating rule or neural network with the desired characteristics is created. The training may be performed in the device itself, in which the neural network processing is performed according to the embodiment, and/or may be implemented on a separate server/system.

A neural network (basic neural network) may consist of multiple neural network layers. Each layer has multiple weight values and performs the layer operation by computing the previous layer and operating on the multiple weights. Examples of basic neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

A learning algorithm is a method for training a predetermined target device (e.g., a robot) using a set of training data to induce, enable, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

In addition, the proposed method, performed by an electronic device, can be implemented using an artificial intelligence model.

Visual understanding is a method of recognizing and processing things similar to human vision, and it includes, for example, object recognition, object tracking, image retrieval, people recognition, scene recognition, 3D scene visualization, or image enhancement.

Brief description of the drawings

The above and/or other aspects will become more apparent from the description of exemplary embodiments with reference to the accompanying drawings, in which:

Fig. 1 illustrates the proposed procedure for training modified 3D reconstruction methods.

Fig. 2 illustrates the visualization of 3D scenes reconstructed using the original VoRTX and VoRTX+NSR obtained using the original VoRTX (a) and the proposed VoRTX+NSR (b).

Detailed description of the invention

The present invention can find application in indoor visual analytics systems, mobile and terrestrial 3D scanning systems for various indoor scenes, such as apartments, offices, retail spaces, etc. The proposed invention is aimed at detecting and aligning structural elements, such as vertical and horizontal surfaces (e.g., floors and walls) in rooms and can be used in 3D reconstruction and visualization of a 3D scene.

The following terms are used in this description.

The 3D scene reconstruction is a point cloud or mesh, i.e. a mathematical representation of the 3D object obtained by applying known algorithms (e.g. marching cubes) to the TSBF volume.

A 3D scene rendering is the rendering, for example on a display device, of the said 3D scene reconstruction, i.e. a two-dimensional image that shows what an object represented by a scene point cloud or scene mesh looks like.

Rendering is a method of obtaining the mentioned visualization of a 3D scene from a reconstruction of a 3D scene.

The base neural network is any known neural network implementing the TSDF volume derivation for scene voxels from a sequence of RGB frames with camera positions. The base neural network includes the TSBF skeleton and head.

NSR (Normal Segmentation Regularization), an automatic modification of methods that perform scene reconstruction by predicting a Truncated Signed Bistance Function (TSBF), is proposed. The proposed regularization can be integrated into any known reconstruction method (base neural network) that predicts TSBF from a sequence of frames with camera positions. According to the present invention, NSR includes the operation of several components, one of which is a 3D sparse convolutional segmentation module (segmentation head), in addition, NSR includes normal calculation, normal detection in wall and floor areas and calculation of their deviations from horizontal/vertical, as will be described below.

The proposed modification considers the scene structure during training by detecting walls and floors in the scene and penalizing the corresponding surface normals for deviation from the horizontal and vertical directions, respectively (while taking into account deviation of normals from the required directions), as will be described below.

The proposed invention allows eliminating global artifacts during the reconstruction of a 3D scene; thanks to the proposed invention, distortions of the shape of a room are eliminated, in addition, the invention allows getting rid of some local artifacts, such as depressions, holes and elevations, in the reconstruction of a 3D scene and its visualization.

Currently, the proposed invention can be applied to data obtained by smartphones with tracking systems (e.g., ARCore for Android smartphones), or any video data with a known camera trajectory (e.g., obtained using RealSense T265). The user can shoot video of a room, for example, with a smartphone, and with the help of the proposed invention obtain a reconstruction of a 3D scene in the form of a scene mesh or a scene point cloud, which after rendering is transformed into a visualization of a 3D scene that the user can see on the screen.

It is possible to render a grid or a point cloud, then the visualization (image) of the scene will be presented in the form required by the user. It can be used, for example, for VR/AR applications (games, interior design, real estate applications or navigation). The minimum list of components: a camera, a tracking module that evaluates the position and angle of the camera, a memory device with a processor and the ability to use neural networks.

The main technical result provided by the present invention is as follows:

- NSR is proposed, which is a modification of 3D scene reconstruction methods, which includes a new trainable module and an associated training procedure with normal losses and segmentation of the base neural network on a dataset for which there is a video with camera positions and angles, reference reconstructions and floor and wall segmentation markings. As a result of training, a base neural network is obtained that accepts a video with camera positions and angles as input and predicts TSBF and segmentation by 3 classes (floor, walls, other). Moreover, when the base neural network is already trained, the segmentation prediction is no longer required for the main purpose (determining walls, floors, etc.), and only the TSDF prediction is used, from which the grid can be calculated. NSR can be implemented in an arbitrary trainable model (neural network) that outputs TSDF and is trained end-to-end on semantically annotated 3D data. There are publicly available sets of such training data (training sequences of RGB frames with corresponding camera position data), such as ScanNet [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes, CVPR, 2017] (which was used to train the base neural network in the invention). The concept of "supervised learning" is implemented, i.e., in order for the base neural network to correctly predict TSDF and segmentation classes, it needs to receive examples of such data during the training process.

- NSR is applied to a number of state-of-the-art reconstruction approaches and demonstrates performance gains and reconstruction quality improvements over state-of-the-art reconstruction methods on several datasets. The proposed development is a regularization that can be applied to an arbitrary room scene reconstruction method that predicts TSDFs from videos of camera positions.

The main contribution of the present invention is a new regularization of geometric segmentation, consisting of the introduction of auxiliary losses of normals optimized together with an additional segmentation head. The training blocks for the base neural network proposed in the invention improve the quality of scene reconstruction: coverage, accuracy, recall, F1-measure, and allow obtaining smoother and more even reconstructions with fewer "deepenings" and "elevations".

During training, the 3D reconstruction results are not calculated. During training, the TSBF prediction, segmentations, and normal prediction are calculated, the loss function values are calculated, and they are minimized. However, if the TSDF is predicted for the scene, then the 3D reconstruction itself can be calculated based on it (using the marching cubes algorithm). This is not necessary during training, but once the network is trained, it is done to obtain the reconstruction.

The trained basic neural network considered in the invention is characterized by the fact that as a result of its operation (if the marching cubes algorithm is also used), it is possible to obtain a point cloud or a triangular grid (triangle grid) corresponding to a given scene. That is, if it is a point cloud, then the result is a set of scene points (their coordinates and colors), and if it is a grid, then the result is triangular faces connecting some triplets of these points (i.e., a surface of triangles).

In brief, the procedure for obtaining a visualization of a 3D scene is as follows:

- use a basic neural network model trained, as described below, to obtain the TSDF of the volume of the input sequence of RGB frames of a real scene.

- apply an algorithm to the TSDF volume that calculates the reconstruction of the 3D scene (representation in the form of a triangular grid);

- perform rendering of the 3D scene reconstruction to obtain a visualization of the 3D scene.

Implemented as a sparse 3D convolutional module, NSR can be embedded in an arbitrary trainable model that outputs a TSDF, and can be trained end-to-end on 3D segmentation point clouds.

During the inference process, when applying an already trained base neural network to new (never seen before) data, the modules used in training the base neural network are not required, so the presence of NSR during training does not impose any restrictions on application scenarios.

Also proposed is a hardware device, which is a machine-readable medium, on which a software product is stored, implementing a method for reconstructing a 3D scene using the TSDF normal segmentation regularization during training of a base neural network, wherein the proposed training stages, according to the method, are suitable for any existing base neural networks for 3D reconstruction. More specifically, the proposed training stages can train any base neural network implementing the reconstruction of a 3D scene with TSDF prediction based on RGB video with camera positions. In this case, a color video of the scene and camera positions for each frame of this video are fed to the input of the trained base neural network, and the result in the form of a fixed 3D reconstruction of this scene is obtained after applying the algorithm to the TSDF volume at the output of the trained base neural network.

The present invention can be used with any device capable of filming video and estimating the camera trajectory (camera positions and angles for each video frame). This can be a surveillance camera (e.g., RealSense T265), a smartphone (e.g., an Android smartphone with the Google ArCore system), a robot vacuum cleaner equipped with a video camera and a tracking system, or even just a video camera, to the data of which the trajectory estimation methods are applicable.

Fig. 1 shows the proposed training procedure for the modified 3D reconstruction methods. Briefly, the key concept of the proposed invention is a method for training an artificial intelligence (AI) network (base neural network) to perform 3D scene reconstruction using NSR (normal segmentation regularization), comprising the following steps:

Step 1: extract the image feature values and predict the 3D space representation using the neural network. It should be noted that the image feature values are the data elements of maximum interest that will be fed and analyzed by the neural network, i.e. the image feature values are internal (hidden, intermediate) representations for the neural network. This step is performed by blocks 2, 3, 5, 6, 9 shown in Fig. 1. As will be described in detail below, blocks 2, 3, 5, 6, 9 perform the input of training data to the skeleton; use the TSDF head to calculate the TSDF loss.

Step 2 is used only when training the base neural network: segment the space using segmentation information (e.g. floor/wall). This step is performed by blocks 4, 7, 8, 11, 12 shown in Fig. 1. As will be described in detail below, blocks 4, 7, 8, 11, 12 apply the segmentation head; calculate the segmentation loss; determine the floor and wall areas.

Step 3 is used only when training the base neural network: the base neural network is trained such that the specified floor and wall are flat. This step is performed by blocks 10, 11, 13, 14, 15, 16 shown in Fig. 1. As will be described in detail below, blocks 10, 11, 13, 14, 15, 16 calculate the coordinates of the normals as TSDF gradients; select the normals in the wall regions and calculate their vertical components; select the normals in the floor regions and calculate their horizontal components; calculate the usual loss of the normals; provide backpropagation of the error by updating the parameters of the neural network according to the calculated gradient to reduce the overall loss function.

The above steps are performed repeatedly during the training of the base neural network. However, when the base neural network is already trained, only the first step is performed. The input training data is used only for training, and when the base neural network is trained, the input data are RGB frames and the camera position in space. Thus, blocks 4, 7, 10, 11, 12, 13, 14, 15, 16 are not used in the operation of the base neural network.

Therefore, the basic neural network is trained in a new way. When the basic neural network is already trained in this way, it is possible to obtain a 3D scene reconstruction without artifacts in the floor and wall areas during the operation of the basic neural network. When the basic neural network operates after the proposed training, the output of the 3D scene reconstruction occurs in the usual way, as in the basic version of the basic neural network.

Thus, NSR is a normal segmentation regularization that is applied specifically during training. When deriving a 3D scene reconstruction from an already trained base neural network, the base neural network is used, which is better trained thanks to NSR. NSR is a modification of 3D scene reconstruction methods, the operation of which is based on the assumption of the traditional scene structure.

In the process, the trained base neural network simply determines the TSBF value in all voxels, but thanks to the NSR training, the output of the 3D reconstruction is such that the floors and walls in it are smooth.

The walls and floor are in the scene being reconstructed, and during training, NSR imposes constraints on their normals in steps 2 and 3. These constraints are that:

- normals (i.e. perpendiculars) to the surface of the walls should be as horizontal as possible so that the walls are vertical, and

- normals to the floor surface should be as vertical as possible so that the floor is horizontal.

The assumption "should be maximally horizontal/vertical" means that deviation of normals from the required direction is penalized, i.e. a term corresponding to deviation of normals from the required directions is added to the minimized overall loss function. Deviation is the difference between the current value and the ideal value. This difference can be zero, which corresponds to no deviation, but even for a high-quality 3D model it will be a non-zero number, i.e. from a mathematical point of view it is an additional term. Minimization of the overall loss function can be performed by any known method, however, in the present invention gradient methods are used, for example, the adaptive gradient descent method, namely the Adam method [Adam: A Method for Stochastic Optimization, https://arxiv.org/abs/1412.6980].

Individual loss functions consist of values corresponding to the degree of difference between the predicted TSDF and the reference (TSDF loss), the degree of difference between the predicted semantic labels and the reference semantics (semantic loss), the degree of difference between surface normals and reference normals (normal loss), the degree of non-verticality of the walls and non-horizontality of the floor (added normal deviations). From such individual loss functions (penalties) corresponding to different quality criteria of the 3D model, a common loss function is formed, which is minimized during the training of the base neural network.

General loss functions describe some mapping that takes a prediction as input and returns a value, where the better the prediction, the smaller this value. In other words, the general loss function is a mathematical interpretation of how much the current base neural network differs from the "ideal" one. Moreover, the smaller the value of the general loss function, the better, so it is minimized, that is, the values of such basic parameters of the neural network as weights are selected for which the value of the general loss function will be as small as possible (for example, the closer the projection of the normal to 0, the closer the normal is to the ideal). The general loss function is used only for training the base neural network. The general loss function contains all the constraints imposed on the 3D model. Namely, as will be detailed below: constraints on the TSDF (these are taken from the underlying neural network, this could be the degree of difference between the predicted TSDF and the reference, the degree of difference between the prediction whether a given voxel will contain a scene point or not based on the reference, etc.) In addition, the overall loss function represents the terms of the difference between the predicted semantic labels and the reference, the difference between the predicted normals and the reference, the difference of the length of the normals from one, the deviation of the normals to the walls/floors from the horizontal/vertical directions.

The deviation of wall normals from the horizontal direction is equal to the length of their vertical component (i.e. the projection of the wall normal onto the z axis), i.e. if the wall normal has an ideal horizontal direction, then the vertical component is equal to 0. The deviation of floor normals from the vertical direction is equal to the length of their horizontal component (i.e. the projection of the floor normal onto the x-y plane), i.e. if the floor normal has an ideal horizontal direction, then the horizontal component is equal to 0.

The TSDF function is calculated in the voxel volume for the scene, the distance from each voxel to the nearest surface is stored as the TSDF value for each voxel. The surface normals in the voxel volume of the scene are calculated as the gradients of the TSDF function (since the reconstructed surface is a level surface for the TSDF, i.e., the solution to the equation TSDF=0), which are also calculated and stored for each voxel.

Normals are perpendicular to the surface by definition, but the surface may not be perfect, for example, the floor may not be horizontal, but inclined or have depressions/elevations, and the walls may not be vertical. It should be noted that artifacts in scene reconstruction arise due to many factors: inaccurate camera trajectory estimation, insufficiently trained base neural network, or insufficient scene coverage with frames, due to which part of the scene may be "invisible". This invention uses the natural assumption that in a real scene, the walls in the rooms are vertical and have no artifacts, the floors are horizontal and have no artifacts, and training is based on this assumption.

The 3D model of the scene is a point cloud or a mesh. The 3D reconstruction of the scene is performed based on a sequence of frames representing images of the scene and the corresponding positions and camera angles for each frame of the sequence based on the 3D model of the scene. The image frames are captured by a camera, and the camera parameters are estimated by a tracking system or one of the corresponding localization methods applied to video in this field of technology.

The basic neural network for which the proposed invention can be used consists of a "skeleton" - the main part that calculates features, and a TSDF "head". The two "heads" are the calculated predictions based on these features, namely, the TSDF prediction is calculated by the TSDF head, and the segmentations are calculated by the segmentation head. In other words, if we take an arbitrary basic neural network for applying the proposed method, then this basic neural network initially has a skeleton and a TSDF head, and a segmentation head is added for training. For example, in Atlas [2], the skeleton is a 2D CNN that predicts features in images, a projection in 3D and the encoding part of the 3D network, and the remaining decoding part is reconstruction. VoRTX [1] has a similar structure, but is also an intermediate transformer network for merging features.

The proposed invention is a method for visualizing a 3D scene using a neural network consisting of a base neural network including a TSDF skeleton and head, and a segmentation head based on learning using TSDF normal segmentation regularization (i.e. using the definition of semantic structural elements and their alignment), which contains the following steps:

Training the basic neural network (as shown in Fig. 1):

- The skeleton (block 2 in Fig. 1) is fed with training data, which are training RGB frames with corresponding camera data in the form of input tensors (block 1 in Fig. 1). The skeleton processes the input tensors and calculates intermediate features, which will be further used for prediction and segmentation of the TSDF. The skeleton outputs are intermediate representations of the input, which can be considered as internal features of this sequence.

- The TSDF head (block 3 in Fig. 1) is used for the intermediate features produced by the skeleton. The TSDF head predicts the TSDF volume to obtain a TSDF prediction of the TSDF values in each voxel of the corresponding scene, with each voxel having a corresponding TSDF value at that location. For each voxel in space, the distance from that voxel to the surface of the reconstructed object must be determined. The TSDF sign prediction characterizes on which side of the surface the given voxel is located, if it is inside that object, the distance is taken with a minus sign, otherwise with a plus sign. Block 6 represents the result of block 3, i.e. the TSDF prediction for the entire voxel volume.

- The TSDF loss is calculated between the TSDF prediction data and the reference scan data (from block 5 in Fig. 1). The calculation is performed after the skeleton in a separate module "TSDF Loss" (block 9 in Fig. 1). This module takes the TSDF and reference scan predictions as input and calculates the TSDF loss. The reference scan is a "correct" scan obtained using a laser scanner, depth sensor or other known method. The reference scan is an ideal reconstruction that any trained base neural network strives to reconstruct during training. The loss is an indicator of how much the current reconstruction differs from the ideal (reference). The base neural network is trained to reconstruct the scene as accurately as possible, the loss is calculated and the gradient optimization method is applied to minimize the overall loss function. In this way, the parameters of the base neural network are iteratively updated to reduce the loss and bring the predictions as close to the "ideal" as possible.

The segmentation head (block 4 in Fig. 1) is applied to the intermediate features derived from the skeleton to obtain a segmentation prediction (block 7 in Fig. 1). The segmentation head consists of several layers that are added to the base neural network and predict segmentation labels at each voxel. A segmentation label is the class label to which a given voxel belongs, in this case the class label "floor", "walls", "other".

- The segmentation loss (block 12 in Fig. 1) is calculated between the segmentation prediction (block 7 in Fig. 1) and the segmentation standard (block 8 in Fig. 1). The segmentation standard is known for the training set.

The TSDF head and the segmentation head can work in parallel. The TSDF loss is calculated based on (i.e. after) the TSDF prediction. Similarly, the segmentation loss is calculated based on (i.e. after) the segmentation prediction.

All computed losses (TSDF loss, segmentation loss, and normal-dependent loss (which will be described below) are added to a common loss function, which is optimized during the training of the neural network. The computed losses are used for backpropagation, i.e. updating the parameters of the underlying neural network according to the gradient of the common loss function with respect to these parameters.

Determine the floor and wall areas from the segmentation forecast (block 11 in Fig. 1)

This means that during segmentation, floor and wall areas are determined, i.e. those voxels for which the label "floor" or "wall" is predicted. For each point in space, it is determined whether it is a wall, a floor, or something else. Thus, wall and floor areas are determined in one pass.

The coordinates of the normals are calculated as TSDF gradients (a normal is a 3D vector) (block 10 in Fig. 1). The normals are calculated for all voxels in space (the scene) as gradients of the predicted TSDF. If the surface is an isosurface of some function, then the gradient of this function will be perpendicular to the surface. First, the normals are calculated for all voxels, i.e. the direction of the normal to the surface in each voxel is calculated as the gradient of the TSDF function. If the surface is a level surface of some function (if this is a level surface 0 for TSDF, then the solution TSDF=0), then the gradients of this function are perpendicular to this surface, i.e. they are normals.

Then all normals are calculated, those corresponding to wall voxels are selected (the segmentation head predicts which voxels are walls, which are floors, and which are all the rest). For normals in wall voxels, the vertical component is calculated. Similarly, for normals in floor voxels, the horizontal component is calculated. Namely:

The normals in the wall regions (block 15 in Fig. 1) are selected and their vertical components, i.e. the z-components of the normal (block 14 in Fig. 1), are calculated based on the data obtained from blocks 10 and 11), i.e. the projections of the normal onto the vertical plane, while selecting the normals corresponding to the voxels segmented as "wall". If the wall is perfectly flat, the normals at all points of the wall will be the same, if not, they may differ. The normal in the floor regions (block 15 in Fig. 1) is selected and their horizontal components (block 16 in Fig. 1) are calculated, i.e. the projection of the normal onto the horizontal plane Oxy, while selecting the normals corresponding to the voxels segmented as "floor".

The usual normal losses (block 13 in Fig. 1) (loss functions for normals), which are traditionally used for reconstruction, are calculated from blocks 10 (prediction) and 5 (reference).

Common normal losses include:

1. Difference between predicted normals (gradients of predicted TSDF) and reference normals (gradients of TSDF for reference scan). To calculate this loss, predicted and real normals are needed, predicted normals are calculated from the underlying neural network, and real normals are calculated from scans available in the training data.

2. Eikonal loss is the difference between the lengths of the normal vectors and unity (since the TSDF corresponds to the distance to the surface, the gradient of this function should ideally have unity length. And the difference in length from unity is also penalized). To calculate these losses, only predicted normals are needed.

Both of these common normal losses are often found in prior art, such as [MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction, NeurlPS, 2022]).

- Calculate the overall loss function, which is equal to the sum of the TSBF loss, segmentation loss, and conditional normal loss;

- The gradient of the general loss function is calculated relative to the parameters of the base neural network. The parameters of the base neural network are updated in accordance with this gradient to reduce (minimize) the general loss function from epoch to epoch of training. That is, the backpropagation of the error, known from the state of the art, is carried out. Minimization is performed in order to find such basic parameters of the neural network at which the forecasts will be most accurate (the principles of gradient-based optimization used in neural networks were described above). The process of training the neural network consists of choosing such parameters of the neural network at which the TSDF forecast will be as close to the ideal as possible. The parameters of the base neural network are chosen (trained) in such a way that there are no irregularities. And the finally trained base neural network immediately reconstructs the scene so that the floors and walls become smooth. The basic neural network takes into account all deviations of walls and floors from the "ideal" planes (vertical/horizontal) during training and penalizes (adds to the general loss function) them all, regardless of whether this deviation occurs on a large area or on a small one. Each point, segmented as walls or floors, contributes to the optimization of the basic neural network. And this contribution is equal to the deviation of the normal at this point from the horizontal/vertical direction. During training, there are such parameters of the basic neural network for which the general loss function is minimal, when finding these parameters, the general loss function itself is no longer used, but these "optimal" parameters of the basic neural network are used.

- Backpropagation of the error ensures that the parameters of the underlying neural network are updated according to the minimized loss function. The overall loss function is reduced after each training epoch. The training steps are repeated until the overall loss function stops reducing or until the overall loss function reaches a specified threshold value specified by the developer. The training process can also be performed for a fixed number of epochs, or, as in the basic versions of this neural network, other criteria for stopping the training process can also be applied, for example, until the overall loss function in the next training epoch is below a specified threshold.

At the stage of the trained base neural network operation, only blocks 1, 2, 3, 6 are used. The operation of the trained base neural network contains the following stages: a sequence of RGB frames of the scene is shot by a camera from different angles and camera positions in space. The camera data contains the camera position in space in a fixed coordinate system, which is determined by a three-dimensional vector, and the camera angle, which is a 3×3 rotation matrix (direction cosine matrix). The 3×3 rotation matrix can be obtained by any method, such as SfM (e.g., COLMAP (Structure-from-Motion Revisited, CVPR, 2016) or some SLAM (e.g., ORB-SLAM: A Versatile and Accurate Monocular SLAM System, IEEE Transactions on Robotics, 2015). The input RGB frames with the corresponding camera data are taken from the camera that acquired them or from the memory of the device that implements the proposed method. If the camera is capable of calculating the camera position (e.g., an Android smartphone with an ArCore tracker), then this data can be immediately transferred to the device that will perform the calculations. Otherwise, the corresponding camera data must first be obtained (possibly on the same device that will calculate everything else) of the TSDF head.

- The RGB frames and camera data are input into the skeleton of the trained base neural network as input tensors. The trained base neural network receives the TSDF volume of the input sequence of RGB frames of the real scene. The 3D surface is defined as the isosurface of the zero level of the TSDF. This is done using known algorithms, for example, the marching cubes algorithm (presented in the paper "Marching cubes: A high resolution 3D surface construction algorithm", SIGGRAPH, 1987). Thus, for an RGB video scene with a known camera trajectory, the 3D scene reconstruction is specified as a point cloud or a triangular mesh, as described above. In this case, the floor and walls in the resulting 3D scene reconstruction do not have local artifacts such as depressions, holes and elevations, etc. The floor and walls are segmented, i.e. each voxel is marked with a label of the class to which the given point corresponds. In the present invention, these classes are "floor", "wall", "other".

- Next, the 3D scene reconstruction is rendered to produce a 3D scene visualization. The resulting 3D scene visualization is then displayed to the user.

The proposed trained base neural network is formed such that the regions corresponding to the floor are flat and horizontal, and the regions corresponding to the walls are flat and vertical.

Unlike traditional 3D reconstruction methods that optimize the geometric correctness of scans, in the proposed regularization method, structural elements of the scene (floor and walls) are additionally segmented, and the underlying neural network is optimized so that these elements are flat, the floor surface of the scene is horizontal, and the walls are vertical.

The auxiliary module of wall and floor segmentation (segmentation head), namely additional layers that are added after the skeleton to predict segmentation labels, is trained together with the basic neural network that predicts the truncated signed distance function (TSDF) of the scene, and in the areas corresponding to the floor and walls, the directions of the normals to the scene surface are estimated, i.e. all surfaces in the scene are estimated. In the wall areas, the vertical component of this normal is penalized, and in the floor areas, the horizontal component is penalized, which means that:

at each point in the regions where the segmentation head predicts the class "wall", the vertical components of the normal vectors are considered, and the length of this vertical component (i.e. the z-component of the normal vector) is added to the loss function,

and in the areas where the class "floor" is predicted, the horizontal component (x- and y-components of the normal vectors) is considered and the loss function for this component (the norm of the two-dimensional vector composed of these two components) is added.

If the loss function has terms x, y, z, then the parameters of the underlying neural network are updated to make these terms smaller. Smaller values of these terms result in a more level scene surface in the floor and wall areas. If the floor normals become more vertical, then the floors themselves become more horizontal, and the same goes for the walls.

That is, the optimization is carried out by gradient descent methods (namely, the stochastic optimization method [Adam: A Method for Stochastic Optimization, https://arxiv.org/abs/1412.6980], which are generally accepted for training neural networks. Optimization allows for the alignment of wall and floor areas, since as a result of this optimization process, the values of the general loss function are reduced, including the terms corresponding to the normals to the walls and floor. The smaller the mentioned terms, the less the normals deviate from the "correct" directions and, accordingly, the more aligned the floor and wall surfaces themselves become. Such optimization allows for the alignment of wall and floor areas and prevents the occurrence of reconstruction errors, such as wall roundings, depressions, holes, and elevations on the surfaces.

A. New segmentation head (block 4 in Fig. 1).

The segmentation head is a sparse 3D convolutional network built on the sparse 3D U-Net network. Since the TSDF prediction model architectures also have a U-Net architecture, the segmentation head branches off from its encoder and consists of two sparse 3D convolution modules. Each module includes a sparse 3D convolution followed by batch normalization and two residual 3D convolution blocks. The proposed segmentation head outputs the 3D segmentation as a spatial map with three channels, each corresponding to one segmentation category: wall, floor, and other.

B. Training

A base neural network is trained that predicts the TSDF volume of a scene. This base neural network is trained such that the floor and wall regions tend to be horizontal and vertical, respectively. To achieve this, the base neural network is trained to predict the segmentation label together with the TSDF. When the TSDF and segmentation labels are predicted, the surface normals are computed as the gradients of the TSDF. The floor and wall regions are considered. For points segmented as floors, the deviation of the corresponding normals from the vertical direction is added to the loss function. Similarly, for points segmented as walls, the deviation of the corresponding normals from the horizontal direction is added to the loss function. These loss terms aim to make the floor surface more horizontal and the wall surfaces more vertical.

We propose training a 3D segmentation head on the ScanNet dataset with point clouds annotated with segmentation labels for each point, i.e. the dataset contains point clouds in which each point is annotated (labeled) with a label of the corresponding segmentation class. The original ScanNet categories floor and carpet are transformed into the "floor" category, and wall, window, door, painting are included in the "walls" category. All points considered to belong neither to the wall nor to the floor category fall under the "other" category. A separate ceiling category is not considered, since there are too few ceiling points in the ScanNet scans due to the shooting process.

The training is carried out in two stages. In the first stage, the base neural network that predicts the TSDF is trained together with the 3D segmentation head that learns to divide points into three categories (floor, walls, and other) based on the segmentation loss. The scene geometry is not penalized during the first epochs, so as not to disrupt the training procedure with early erroneous estimates of the segmentation classes. This means that in the initial stages, the still untrained base neural network will most likely incorrectly predict membership in the classes "floor", "walls", "other".

During the first epochs of training, the proposed penalty is not applied to the floor and wall normals. This means that in these first epochs, the terms corresponding to the vertical/horizontal components of the wall/floor normals are not added to the overall loss function. The neural network is primarily trained to reconstruct the full scene (using the TSDF head) and to detect walls, floors, etc. (using the segmentation head).

Once the base neural network is able to determine where the walls and floor are, normal terms are added to the overall loss function and the overall loss function is optimized. The TSDF and segmentation losses also continue to participate in the optimization.

In general, the term "penalization" means that a corresponding term is added to the overall loss function. The fact that no normal terms are added at the beginning of training is dictated by common sense: while the base neural network is not yet trained, it will most likely incorrectly detect floors and walls, so the wrong normals are corrected. Nothing happens to the penalized data. The difference between the predictions and the "ideal" is "penalized", and this term means that this difference is added to the minimized overall loss function. In subsequent epochs, the loss on surface normals is added and the process continues with the combined prediction of geometry and scene segmentations (blocks 6, 7 in Fig. 1). That is, the terms corresponding to the vertical/horizontal components of wall/floor normals are added to the overall loss function. The optimization process continues. From the TSBF estimation representation, surface normals are derived: being first-order derivatives of the surface function, they are easily computed by a single convolution. Normal estimation is implemented as a special untrainable 3D convolutional layer, so this operation can be easily incorporated into a trainable model.

S. Losses

Two sets of losses are used depending on the surface normals. First, a regular normal loss is exploited. The reference normals are extracted from the reference TSDF representation and the discrepancy between the predicted and reference normals is penalized (the difference between the prediction and the reference is added as a term to the loss function, which is minimized) using both the cosine distance (the distance that is the inverse of the normalized dot product of vectors) and the Euclidean distance. This reference loss is complemented by a reference-free eikonal loss that regularizes the L2-norm of the normal, forcing it to converge to 1. The reference data is used only during the training phase. After training, the basic neural network can be applied to data for which there is no reference.

In addition to the usual losses, a normal segmentation regularization is introduced, which considers both the surface normals and the 3D segmentation (the segmentation markup (annotation with a metaclass) of each 3D point). In general, the incentive is for the walls to be vertical and the floor to be horizontal. For convenience, we denote the set of 3D points classified as the floor by the symbol F, and the walls by the symbol W.

W. Non-vertical components of the normal n _x , n _y are regularized at each point :

For walls, the L1 loss is applied for the vertical component ^:

The ablation study examined how each loss impacted the quality of reconstruction and showed that they complemented each other, affecting different aspects of reconstruction.

D. Conclusion

During inference, 3D segmentation is not required. Therefore, the 3D segmentation annotation serves only for training purposes: guided by additional segmentation information, the method learns to predict the TSDF more accurately.

Experiments

A. Basic methods

Several modern TSDF reconstruction methods were considered as basic methods. Methods based on different principles of operation were selected to prove the applicability of the proposed modification to a wide range of approaches to merging several views. That is, many different TSDF reconstruction methods for a scene taken from different angles are known in this field of technology. The regularization proposed in the invention can be applied to any of them, regardless of the principles they are based on. To confirm this, several existing methods based on different principles were considered (described below).

In particular, the VoRTX [1] converter-based and the Atlas [2] averaging-based were used (blocks 2, 3 in Fig. 1 (TSDF skeleton and head).

Atlas [2] combines objects into a single feature volume. It does not involve depth estimation and performs 3D reconstruction with direct prediction of the TSDF of the volume. This scheme allows considering input images jointly and efficiently. Moreover, Atlas is equipped with a mechanism that allows reconstructing invisible regions of the scene using 3D prior images.

VoRTX [1] is an end-to-end 3D volumetric reconstruction method that combines multiple images using transforms.

VoRTX preserves fine details by improving fusion based on camera position, and handles occlusions by estimating the original scene geometry to avoid projecting image elements into occluded areas. Occluded areas occur when one object occludes another from the camera's perspective, in which case some areas may not be visible (specifically in video frames). The combination of these techniques allows for the highest quality reconstruction.

B. Datasets

The proposed method was trained on ScanNet [8], which contains 1613 indoor scenes with reference camera positions, 3D reconstructions, and segmentation labels. In total, it contains 2.5 million RGB-D frames in 707 different spaces. Standard splits and reporting results for the test subset were adopted. The transfer quality of the baseline neural networks trained on ScanNet to other datasets was also assessed: TUM RGB-D, a long-established RGB-D SLAM benchmark; ICL-NUIM, a small RGBD reconstruction benchmark with eight scenes rendered in a synthetic structure; and 7-Scenes [9], a small but challenging indoor RGB-D dataset containing 7 real indoor spaces.

C. Evaluation Protocol

For each frame, a reference depth map was rendered from the reference grid with respect to the corresponding camera position. Similarly, an estimated depth map was rendered from the estimated grid and masked in areas where the reference depth was invalid. All estimated masked depth maps were integrated into a single TSDF volume.

In all experiments, reconstruction quality was assessed using standard reconstruction benchmark metrics: accuracy, recall, precision, rejection, and F-score with a threshold of 5 cm [2]; the grid obtained from the masked estimated depths was compared with the benchmark grid.

Table I: Reconstruction quality of the baseline and modified methods on ScanNet. The best values are in bold.

Acc - accuracy - the average distance from the points of the predicted 3D model to the nearest points of the reference model.

Comp - completeness - the average distance from the main points of the 3D model standard to the nearest points of the predicted model.

Prec - precision - the percentage of points in the predicted 3D model for which the distance to the nearest point from the reference model is less than 5 cm.

Recall - the percentage of reference model points for which the distance to the nearest point from the predicted 3D model is less than 5 cm.

F-score is the harmonic mean of precision and rejection.

The arrows indicate which value is better (down arrow - the smaller the better, up arrow - the larger the better)

Conclusions: For both methods (Atlas, Vortx), adding NSR regularization improves the quality for all metrics.

Table II: Reconstruction quality of the baseline VoRTX and the modified method trained on ScanNet on the ICL-NUIM, TUM RGB-D, and 7Scenes datasets. The best performance for each dataset is highlighted in bold.

Conclusions: When transferring a basic neural network trained on one dataset to other data, adding NSR regularization also improves the quality for almost all metrics (tests were conducted on 3 datasets: ICL-NUIM, TUM RGB-B, 7Scenes)

D. Conclusion

During the inference process, K=60 key frames were sampled arbitrarily so that for each pair of consecutive key frames the relative rotation was at least 15° and the translation was at least 10 cm. These sampled key frames were used to estimate the TSDF of the entire scene; the resulting scene mesh was obtained from the TSBF using the marching cubes algorithm.

The selection of key frames was made at the very beginning, before entering data into the neural network. The purpose of this is to reduce the time and memory costs of the algorithm (not all frames in a row are processed, but only key ones, since if the frames are located very close to each other (taken from the same camera positions), then the information in them is duplicated and redundant).

The marching cubes algorithm was applied in the final step to the results of the base neural network (the base neural network returns a TSDF for the entire scene, and the marching cubes algorithm builds a point cloud or mesh from this TSDF).

E. Implementation Details

The proposed segmentation head (block 4 in Fig. 1) was trained from scratch for the same number of epochs as the base neural network. The Adam optimizer was used with a default initial learning rate of 0.001.

The proposed method was implemented on the PyTorch platform, and the proposed network was trained on a single NVIDIA Tesla P40 GPU.

The intrinsic and extrinsic parameters presented in the datasets [8-11] were used and adjusted along with the image scaling. According to [5], the camera positions were initialized to units and represented the original SDF as a unit sphere with surface normals directed inward.

V. Results

A. Comparison with a known level

The reconstruction metrics in ScanNet are summarized in Table I, and the 3D model transfer quality in ICL-NUIM, TUM RGB-D, and 7Scenes benchmarks is shown in Table II.

Table III illustrates the reconstruction quality obtained using different combinations of losses in ScanNet. "C" stands for the normal loss, "N" for the normal loss, and "NS" for the normal segmentation loss.

Conclusions: Adding individual terms to the overall loss function improves the quality of the 3D model, and the best results can be obtained by adding all the modifications proposed in the invention (loss functions for both normals and segmentation, and classical normal losses).

Fig. 2 shows the visualization of the 3D reconstruction results. Fig. 2 shows the visualization of the 3D reconstructions obtained using the basic versions of VoRTX (a) and VoRTX+NSR (b). The table shows that adding NSR to the basic method significantly improves the quality. The problem areas are enlarged to highlight the benefits of using NSR. As can be seen, the proposed modification allowed filling holes in flat surfaces, while the basic versions created an incomplete reconstruction with missing points in these areas. Moreover, unlike the basic versions, NSR helps to ensure that the walls and floor are smooth and flat.

The impact of using each NSR component on the reconstruction quality was analyzed. All ablation studies were performed using VoRTX [1] as the baseline and the same training/evaluation protocols described above unless otherwise stated. First, the quality of the estimated segmentation was investigated by using reference segmentation annotations instead of predicted ones.

Then, baseline neural networks trained using different combinations of regular normal losses and normal segmentation losses were examined to verify that the reconstruction quality was improved when using the proposed losses.

In summary, we propose NSR, a modification of scene reconstruction methods that allows considering the typical structure of a scene. In particular, the proposed modification identifies walls and floors in a point cloud and penalizes the corresponding surface normals for deviation from the horizontal and vertical directions, respectively. Implemented as a sparse 3D convolutional module, NSR can be included in an arbitrary trainable model that outputs a TSBF and trained end-to-end on 3D segmentation point clouds. No 3D segmentation is required in the inference process, so the use of NSR does not impose any restrictions on user scenarios. The proposed modification has been applied to several state-of-the-art TSDF reconstruction methods and has demonstrated significant performance gains on standard datasets: ScanNet, ICL-NUIM, and TUM RGB-D.

The illustrative embodiments described above are examples and should not be considered as limiting. Furthermore, the description of these embodiments is intended to illustrate and not to limit the scope of the claims, and many alternatives, modifications and variations will be obvious to those skilled in the art.

LITERATURE

[1] N. Stier, A. Rich, P. Sen, and T. H"ollerer, "VoRTX: Volumetric 3D Reconstruction With Transformers for Voxelwise View Selection and Fusion," in 3DV, 2021, pp.320-330.

[2] Z. Murez, T. van As, J. Bartolozzi, A. Sinha, V. Badrinarayanan, and A. Rabinovich, "Atlas: End-to-end 3D scene reconstruction from posed images," in ECCV, Glasgow, UK, 2020, pp.414-431.

[3] J. Sun, Y. Xie, L. Chen, X. Zhou, and H. Bao, "NeuralRecon: Real-time coherent 3D reconstruction from monocular video," in CVPR, 2021, pp.15598-15607.

[4] A. Bovzivc, P. Palafox, J. Thies, A. Dai, and Matthias Niesner, “TransformerFusion: Monocular RGB scene reconstruction using transformers,” in NeurlPS, 2021.

[5] H. Guo, S. Peng, H. Lin, Q. Wang, G. Zhang, H. Bao, and X. Zhou, “Neural 3D Scene Reconstruction with the Manhattan-world Assumption,” in CVPR, 2022.

[6] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman, “Volume rendering of neural implicit surfaces,” in NeurlPS, 2021.

[7] P. Wang, L. Liu, Y. Liu, Ch. Theobalt, T. Komura, and W. Wang, "NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction," in NeurlPS, 2021.

[8] A. Dai, A. X. Chang, M. Savva, M. Halber, Th. Funkhouser, and M. Niesner, "ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes," in CVPR, 2017.

[9] J. Shotton, B. Glocker, Ch. Zach, Sh. Izadi, A. Criminisi, and A. Fitzgibbon, "Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images," in CVPR, 2013.

[10] A. Handa, T. Whelan, J.B. McDonald, and A.J. Davison, "A Benchmark for RGB-B Visual Odometry, 3D Reconstruction and SLAM," in ICRA, Hong Kong, China, 2014.

[11] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and B. Cremers, “A Benchmark for the Evaluation of RGB-B SLAM Systems,” in IROS, 2012.

[12] J. Wang, P. Wang, X. Long, C. Theobalt, T. Komura, L. Liu, W. Wang, “NeuRIS: Neural Reconstruction of Indoor Scenes Using Normal Priors,” in ECCV, 2022.

[13] A. Avetisyan, M. Bahnert, A. Dai, M. Savva, A. X. Chang, and M. Niesner, “Scan2cad: Learning CAB Model Alignment in RGB-B Scans,” in CVPR, 2019.

[14] A. Avetisyan, A. Dai, and M. Niesner, “End-To-End CAD Model Retrieval and 9dof Alignment in 3D Scans,” in ICCV, 2019.

[15] M. Dahnert, A. Dai, L. Guibas, and M. Niesner, “Joint Embedding of 3D Scan and CAD Objects,” in ICCV, 2019.

[16] Sh. Hampali, S. Stekovic, S. Deb Sarkar, Ch. S. Kumar, F. Fraundorfer, and V. Lepetit, "Monte Carlo Scene Search for 3D Scene Understanding," in CVPR, 2021.

[17] S. Ainetter, S. Stekovic, F. Fraundorfer, V. Lepetit, “Automatically Annotating Indoor Images With CAD Models via RGB-D Scans,” in WACV, 2023, pp. 3156-3164.

[18] Y. Wang, Z. Li, Y. Jiang, K. Zhou, T. Cao, Y. Fu, C. Xiao, "Neural-Room: Geometry-Constrained Neural Implicit Surfaces for Indoor Scene Reconstruction," in ACM Transactions on Graphics (TOG), 2022.

[19] Sh. Zhi, M. Bloesch, S. Leutenegger, and A. J. Davison, "SceneCode: Monocular Dense Segmentation Reconstruction Using Learned Encoded Scene Representations," in CVPR, 2019, pp. 11768-11777.

Claims (27)

1. A method for reconstructing a 3D scene and visualizing it using a neural network consisting of a base neural network including a skeleton, which is the main part of the base neural network that calculates features, a TSDF (Truncated Signed distance function) head that predicts TSDF values in each voxel, and a segmentation head, which is a 3D sparse convolutional segmentation module that predicts segmentation labels in each voxel, comprising the steps of: train a basic neural network to obtain volume TSDFs for scene voxels as follows: - input training data, including a training sequence of RGB frames with corresponding camera position data, into the skeleton; - calculate the TSDF loss between the TSDF prediction obtained by the TSDF head from the skeleton output and the reference scan; - obtain, using the segmentation head, from the output of the skeleton data a segmentation prediction defining the "floor", "walls", "other" regions for each voxel; - calculate the segmentation loss between the segmentation forecast and the segmentation standard; - calculate normal coordinates as gradients of the TSDF prediction for the TSDF values in each voxel over all voxels in the scene; - calculate normal losses for normals for wall regions and floor regions based on the TSDF reference results and TSDF prediction gradients; - calculate the overall loss function as the sum of the TSDF loss, segmentation loss, and normal loss; - minimize the overall loss function; use a trained base neural network to obtain the TSDF of the volume of the input sequence of RGB frames of a real scene; apply an algorithm to the TSDF volume that computes the reconstruction of the 3D scene; render the reconstruction of the 3D scene to obtain a visualization of the 3D scene. 2. The method according to claim 1, wherein the algorithm that calculates the reconstruction of the 3D scene is the marching cubes algorithm. 3. The method according to any one of claims 1, 2, wherein the segmentation head and the TSDF head are used in parallel during training. 4. The method according to any of paragraphs 1-3, containing additional stages implemented after the stage of calculating the coordinates of the normals, in which: select the normal in the wall areas and calculate their vertical components; select the normal in the floor areas and calculate their horizontal components. 5. The method according to any of paragraphs 1-4, in which at the stage of calculating the usual losses of normals the following is performed: at each point in the areas where the segmentation head predicts a "wall", the vertical components of the normal vectors are considered, and the length of the z-component of the normal vector is added to the usual normal losses, and at each point in the regions where the segmentation head predicts "floor", consider the horizontal components of the normal vectors, and also consider the x- and y-components of the normal vector and add the norm of the two-dimensional vector consisting of these two components to the usual normal loss. 6. The method according to any one of paragraphs 1-5, in which the overall loss function is calculated as the sum of the TSDF loss, the segmentation loss and the usual normal loss. 7. The method according to any one of paragraphs 1-6, in which the minimization of the general loss function is implemented by calculating the gradient of the general loss function for all parameters of the base neural network. 8. The method according to any one of claims 1-7, further comprising providing backpropagation of the error of the minimized overall loss function and updating the parameters of the base neural network in accordance with the minimized loss function, while updating the parameters of the base neural network. 9. The method according to any one of paragraphs 1-8, in which the training steps are repeated until the overall loss function no longer decreases. 10. The method according to any one of claims 1-8, in which the training steps are repeated until the overall loss function reaches a given threshold value. 11. A computing device comprising a processor and memory in which instructions are stored that, when executed in the processor, implement the method according to paragraph 1.