KR20220082260A - Terminal for rendering 3d content and operaing method of thereof - Google Patents

Abstract

The embodiment relates to a method of operating a terminal for rendering 3D content. may include A method of operating a terminal, comprising: transmitting a first photographed image including a target object to a server; obtaining an object image in which the target object is extracted from the first captured image; generating 3D content using an object image; acquiring a second captured image including a display; determining a position in the display where the 3D content is to be rendered based on the second captured image; and transmitting a second captured image including a location and three-dimensional content to a computing device controlling the display, wherein the three-dimensional content is rendered to be displayed at the location of the display by the computing device.

Description

A terminal for rendering three-dimensional content and an operating method thereof

The embodiment relates to a terminal for rendering 3D content and an operating method thereof.

In general, when designing a specific structure such as a ship, plant, aircraft, or building, after designing using a 2D CAD program such as 'AUTOCAD', the overall three-dimensional shape of the structure and the modification or change of the structure In order to easily identify problems, etc., the work of converting to 3D CAD drawings is performed.

In addition, a method of extracting an object by removing the background of an image obtained to generate and render 3D content has been proposed, but various methods of using the method do not exist.

It takes a lot of time for a designer to create a complex 3D model during design work, and it is difficult to change the model when the 3D model is brought to another workspace.

In this regard, Korean Patent Registration No. 1473956 discloses "a 3d spatial information modeling method based on a 2d plan view and an image control system using the 3d spatial model".

The invention according to an embodiment is to provide a method of generating 3D content by using an image obtained by photographing a 3D object and rendering the 3D content.

In detail, in the process of creating 3D content, various processes (concept design, volume control, testing, iteration) are simplified by using photography, and the 3D content generated in the terminal is rendered to the display of a computing device such as a PC. We want to provide a way.

The invention according to an embodiment may provide a method of generating 3D content using an image obtained by photographing a 3D object and rendering the 3D content.

In detail, in the process of creating 3D content, various processes (concept design, volume control, testing, iteration) are simplified by using photography, and the 3D content generated in the terminal is rendered to the display of a computing device such as a PC. method can be provided.

1 is a flowchart illustrating a method of rendering 3D content according to an embodiment.
2 is a diagram for explaining a method of rendering 3D content in a computing device according to an embodiment.
3 is a diagram for explaining a method of determining a position for rendering 3D content according to an embodiment.
4 is an example of a 3D content rendering method in the embodiment.
5 is a block diagram for explaining the configuration of a terminal in the embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

The terms used in the examples are used for the purpose of description only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that a feature, number, step, operation, component, part, or a combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

In addition, in describing the components of the embodiment, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. When it is described that a component is "connected", "coupled" or "connected" to another component, the component may be directly connected or connected to the other component, but another component is between each component. It will be understood that may also be "connected", "coupled" or "connected".

Components included in one embodiment and components having a common function will be described using the same names in other embodiments. Unless otherwise stated, descriptions described in one embodiment may be applied to other embodiments as well, and detailed descriptions within the overlapping range will be omitted.

1 is a flowchart illustrating a method of rendering 3D content according to an embodiment.

According to the method according to the embodiment, it is possible to provide a method of generating 3D content using interworking based on communication between a terminal, a server, and a computing device, and rendering the 3D content by the computing device.

In step 101, the terminal acquires a first captured image. In an embodiment, the first captured image is an image captured by the camera of the terminal, and the terminal may acquire the first captured image by capturing the target object to be produced as 3D content.

In step 102, the terminal transmits the first captured image to the server.

In step 103 , the server removes the background from the received first captured image and estimates the object depth.

In an embodiment, a target object may be recognized in the first captured image, and all parts of the first captured image except for the target object may be recognized as a background. The server may extract the target object by removing all regions recognized as the background from the first captured image. For this, the server can use a deep learning model such as BASNet.

The server may estimate the depth of the extracted target object by removing the background. For this, VGGNet, CNN model, etc. can be used.

In step 104 , the server may transmit the extracted target object and information on the target object to the terminal. In an embodiment, the image of the object from which the background is removed may be transmitted. The information about the object may include information (eg, masking data) necessary to implement the target object in three dimensions, such as depth information of the object.

In step 105, the terminal generates 3D content using the object image. In an embodiment, the terminal may execute an app for generating 3D content. The terminal may create a 3D skeleton for the received target object, and may generate 3D content by applying a texture.

In an embodiment, the terminal may convert the viewpoint of the 3D content to be rendered under the control of the user. For example, the user may change or control the viewpoint at which the 3D content is exposed on the display of the terminal, such as rotating the generated 3D content or changing the viewpoint, by touching the display of the terminal.

In step 106, the terminal acquires a second captured image. In an embodiment, the second captured image may be used to determine an object and a location to paste the generated 3D content.

The second captured image may correspond to an image input through a camera provided in the terminal, and the image may include a display for pasting the generated 3D content. For example, the second captured image may include a display screen such as a monitor of a PC or a monitor of a laptop computer.

In step 107, the terminal determines a location to render the 3D content. In an embodiment, a position at which the 3D content is to be rendered may be selected in a display included in the second captured image. The display may contain the window being worked on. The terminal may recognize the display area from the second captured image, and may deactivate the area other than the display area so that a position to be rendered within the corresponding area can be selected. A method of recognizing the display from the second captured image will be described later in detail.

In an embodiment, the terminal may detect and display a display area from an image exposed to the display of the terminal through a camera, and determine a position where 3D content is to be rendered in the corresponding area.

In step 108, the terminal may transmit the second captured image and the 3D content to the computing device connected to the display. Alternatively, the computing device may include a display. In an embodiment, the second captured image may include information on a location where 3D content is to be rendered in the second captured image.

In step 109, the computing device renders 3D content based on the received information. In an embodiment, based on the rendering position determined in the terminal, 3D content may be rendered to the corresponding position.

The computing device is executing at least one program, and the execution of the corresponding program may be confirmed through a display connected to the computing device.

In an embodiment, the computing device may render the 3D content received from the terminal on the display workspace of the running program. Depending on the program running on the computing device, 3D content may be rendered as it is or converted to 2D and rendered. Referring to FIG. 2, the embodiment will be described in detail.

2 is a diagram for explaining a method of rendering 3D content in a computing device according to an embodiment.

In step 201, the terminal may generate 3D content. An embodiment of generating 3D content may refer to the description of FIG. 1 .

In step 202, the terminal may change the viewpoint of the 3D content. For example, by freely rotating the generated content in the left-right or front-back direction, the viewpoint in space in which the 3D content is expressed on the display may be changed.

In step 203, the terminal may target a location to render the 3D content in the image input through the camera of the terminal. The terminal may detect and display the display area from the image exposed to the display of the terminal through the camera, and determine a position where the 3D content is to be rendered in the corresponding area. In an embodiment, a location where 3D content is to be rendered may be targeted to the captured image, or a location where 3D content will be rendered may be targeted with respect to an image inputted in real time through a camera.

In an embodiment, the terminal transmits the 3D content to the computing device, and the computing device may render the 3D content as it is or convert it into 2D based on information about the program being executed.

In another embodiment, the 3D content may be transmitted as it is, or the 3D content may be converted into 2D content and transmitted based on information about a program running in the computing device to which the 3D content is to be rendered.

When the program being executed in step 204 is a 3D program, for example, a program such as 3D Max or Maya, the computing device may render the 3D content to the workspace of the corresponding program. In an embodiment, based on the second captured image, 3D content may be rendered to a determined position of the second captured image.

When the running program is a 2D program, for example, a program such as Power Point or Word, the computing device may convert 3D content into 2D content in step 205 . According to an embodiment, step 205 may be performed in the terminal.

Subsequently, in step 206 , the computing device may render the 2D content converted into 2D into the workspace of the running program.

In an embodiment, when the computing device renders the 3D content, the 3D content may be rendered in the form of an image based on the converted viewpoint of the 3D content.

3 is a diagram for explaining a method of determining a position for rendering 3D content according to an embodiment.

In an embodiment, after 3D content is generated, a method of extracting a display area from a terminal to determine a position to render 3D content and determining a rendering position in the display area will be described. To this end, the terminal and the computing device may interact with each other.

In step 310, the terminal may acquire a second captured image.

In an embodiment, by using an app in a terminal such as a smartphone, an image may be captured such that the image captured by the camera includes a display area connected to the computing device. In an embodiment, a display area may be detected within an image detected in real time through a camera through an app, and the corresponding area may be guided.

In step 320 , the terminal may recognize the display area from the second captured image.

In an embodiment, the terminal may extract the outline of the display from the second captured image. In addition, a screen output through the display may be recognized based on the outline of the display. Even if a part of the display of the second captured image is recognized, the outline may be extracted. The terminal may transmit the second captured image to the computing device.

In operation 330, the computing device may acquire an image of the entire display.

In an embodiment, upon receiving the second captured image from the computing device, the computing device executes a screen capture on the display area to obtain a captured image for the entire display corresponding to the image output to the display area included in the second captured image. can be obtained

In operation 340, the computing device may extract image feature points for the entire image of the display.

In an embodiment, the computing device may extract a feature point of a display area included in the second captured image transmitted from the terminal, and extract a feature point of a captured image that captures the entire display.

In operation 350, the computing device may match the terminal and the display.

In step 360, the terminal may obtain coordinates to be rendered on the display.

In an embodiment, the display area may be matched by matching the feature points included in the captured image with the feature points of the display area included in the second captured image transmitted from the terminal based on the previously extracted feature points.

For example, according to the image 351 , the left image may correspond to a captured image, and the right image may correspond to an image in which the display area is recognized from the second captured image. The computing device may match the two images (ORB, object request broker). In this process, the midpoint captured on the second captured image may be calculated based on the center point of the camera and matched to the corresponding position.

In an embodiment, coordinate information at which the second captured image captured by the terminal is located on the display may be obtained through homograpy matching. Based on the coordinate information on the display, the terminal may determine coordinates for rendering the 3D content, and obtain information on the coordinates.

Thereafter, when the coordinates for rendering the 3D content on the display are determined, the terminal hooks the event being executed in the computing device connected to the display, and transmits a command for executing the event so that the 3D content is added to the corresponding program. have.

In an embodiment, the computing device may render the 3D content transmitted to the running program based on the coordinate information.

4 is an example of a 3D content rendering method in the embodiment.

In an embodiment, the terminal may extract the object object 411 from the first captured image 410 . For example, the terminal may extract a plurality of object objects from the first photographed image 410 and extract one object object 411 from among the extracted object objects, or the first photographed image 410 . It is possible to detect the closest object target 411 based on the depth information of .

In an embodiment, the server may perform masking on the received first captured image, detect the background 421 in the masked image 420 , and remove the background 421 portion.

The server may obtain the first captured image 430 in which only the target object 431 is expressed by alpha (Alpha=0) processing of the background 421 part removed with respect to the received first captured image 410 original. have. In an embodiment, the server may estimate the depth with respect to the target object 431 extracted by removing the background. For this, VGGNet, CNN model, etc. can be used. The server may transmit the extracted target object and information on the target object to the terminal.

In an embodiment, the terminal may generate 3D content for the target object based on information received from the server.

The terminal may acquire the second captured image 440 . In an embodiment, a location 442 to render the generated 3D content may be displayed in the display 441 included in the second captured image 440 . For example, after the terminal determines the rendering position 442 of the 3D content in the second captured image 440 , an image including information on the corresponding position 442 may be captured. The rendering position 442 based on the second captured image 440 may be predetermined as the center of the image input to the camera, and the rendering position 442 is located at a specific point of the display 441 while moving and rotating the terminal. can be adjusted to For example, the location to be rendered may be determined based on a point in time at which the user captures the second captured image 440 , and a size as well as a location of the 3D content may be determined based on the point of capture.

The terminal may generate the 3D content 451 as a rendering position corresponding to the center of the second captured image 450 .

In an embodiment, the terminal may transmit the 3D content to the rendering location 442 so that the 3D content is rendered to a computing device connected to a display on which the 3D content is to be rendered. For transmission of 3D content, the terminal and the computing device may be connected through short-range communication. The computing device may render the 3D content 451 on the display based on the program being executed.

5 is a block diagram for explaining the configuration of a terminal in the embodiment.

In an embodiment, the terminal 500 may include one or more memories 510 , one or more processors 520 , and a communication unit 530 for communicating with the server 501 and the computing device 502 .

According to the method according to the embodiment, the 3D content is generated using the interworking based on communication between the terminal 500, the server 501, and the computing device 502, and the 3D content is rendered to the computing device 502. method can be provided.

The app (web app) of the terminal 500 and the server 501 transmit/receive data by https communication, and the computing device 502 is not affected by the platform.

The terminal 500 acquires a first photographed image. In an embodiment, the first photographed image is an image photographed through the camera of the terminal, and the terminal 500 may acquire the first photographed image by including a target object to be produced as 3D content.

The terminal 500 transmits the first captured image to the server.

In the server 501 , the background may be removed from the received first captured image and the object depth may be estimated.

In an embodiment, a target object may be recognized in the first captured image, and all parts of the first captured image except for the target object may be recognized as a background. The server 501 may extract the target object by removing all regions recognized as the background from the first captured image. A deep learning model such as BASNet may be used. BASNet is an image recognition server and consists of Python language. When NVIdia GPU is installed, fast calculation is possible by linking CUDA lib, and when there is no GPU, it operates as a CPU.

A depth may be estimated for the extracted target object by removing the background from the server 501 . For this, VGGNet, CNN model, etc. can be used.

The terminal 500 may receive the extracted target object and information on the target object from the server 501 . In an embodiment, an image of an object from which a background is removed may be transmitted. The information about the object may include information (eg, masking data) necessary to implement the target object in three dimensions, such as depth information of the object.

The terminal 500 generates 3D content using the object image. In an embodiment, the terminal may execute an app for generating 3D content. The terminal may create a 3D skeleton for the received target object, and may generate 3D content by applying a texture.

In an embodiment, the terminal 500 may convert the viewpoint of the 3D content to be rendered under the control of the user. For example, the user may change or control the viewpoint at which the 3D content is exposed on the display of the terminal, such as rotating the generated 3D content or changing the viewpoint, by touching the display of the terminal.

The terminal 500 acquires a second captured image. In an embodiment, the second captured image may be used to determine an object and a location to paste the generated 3D content.

The second captured image may correspond to an image input through a camera provided in the terminal, and the image may include a display for pasting the generated 3D content. For example, the second captured image may include a display screen such as a monitor of a PC or a monitor of a laptop computer.

The terminal 500 determines a position to render the 3D content. In an embodiment, a position at which the 3D content is to be rendered may be selected in a display included in the second captured image. The display may contain the window being worked on. The terminal may recognize the display area from the second captured image, and may deactivate the area other than the display area so that a position to be rendered within the corresponding area can be selected. A method of recognizing the display from the second captured image will be described later in detail.

In an embodiment, the terminal 500 may detect and display a display area from an image exposed to the display of the terminal through a camera, and determine a position where 3D content is to be rendered in the corresponding area. The position to be rendered in the image exposed on the display of the terminal may be fixed as a central point, and the position to be rendered in the display may be determined by moving the camera.

The terminal 500 may transmit the second captured image and the 3D content to the computing device 502 connected to the display. Alternatively, the computing device 502 may include a display. In an embodiment, the second captured image may include information on a location where 3D content is to be rendered in the second captured image.

3D content may be rendered based on the information received by the computing device 502 . In an embodiment, based on the rendering position determined by the terminal 500, 3D content may be rendered to the corresponding position.

The computing device 502 is executing at least one program, and it is possible to check whether the program is being executed through a display connected to the computing device 502 .

In an embodiment, the 3D content received from the terminal in the computing device 502 may be rendered on the workspace of the display of the running program. Depending on the program being executed in the computing device 502 , 3D content may be rendered as it is or converted into 2D and rendered.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (19)

In the operating method of a terminal for rendering three-dimensional content,
transmitting a first captured image including a target object to a server;
obtaining an object image from which the target object is extracted from the first captured image;
generating 3D content using the object image;
acquiring a second captured image including a display;
determining a position where the 3D content is to be rendered in the display based on the second captured image; and
Transmitting the second captured image and the 3D content including the location to a computing device controlling the display
including,
wherein the three-dimensional content is rendered by the computing device to be displayed at the location of the display;
How the terminal operates.
According to claim 1,
The step of acquiring a second photographed image including the display comprises:
extracting an outline of the display from the second captured image; and
Recognizing an area corresponding to the display
containing,
How the terminal operates.
According to claim 1,
Determining a position where the 3D content is to be rendered in the display based on the second captured image includes:
extracting feature points in the display; and
Matching the display to the screen in the terminal
containing,
How the terminal operates.
According to claim 1,
The step of determining a position where the 3D content is to be rendered in the monitor from the second captured image,
obtaining coordinate information of the display;
determining the coordinates at which the three-dimensional content is to be rendered within the coordinates of the display;
containing,
How the terminal operates.
5. The method of claim 4,
The three-dimensional content is rendered based on the program running on the computing device,
How the terminal operates.
6. The method of claim 5,
When the program is a three-dimensional program, the three-dimensional content is rendered in the three-dimensional program,
How the terminal operates.
6. The method of claim 5,
If the program is not a three-dimensional program, the three-dimensional content is imaged and rendered in the program,
How the terminal operates.
According to claim 1,
The step of generating 3D content using the object image includes:
Transforming the viewpoint of the 3D content based on a control command input from the user
containing,
How the terminal operates.
According to claim 1,
The step of generating 3D content using the object image includes:
Generating the 3D content by using the depth information of the object image obtained from the server
containing,
How the terminal operates.
A computer program stored on a computer-readable medium in combination with hardware to execute the method of any one of claims 1 to 9.
In the terminal for rendering 3D content,
one or more processors;
Memory; and
one or more programs stored in the memory and configured to be executed by the one or more processors;
The program is
transmitting a first captured image including a target object to a server;
obtaining an object image from which the target object is extracted from the first captured image;
generating 3D content using the object image;
acquiring a second captured image including a display;
determining a position where the 3D content is to be rendered in the display based on the second captured image; and
Transmitting the second captured image and the 3D content including the location to a computing device controlling the display
including,
wherein the three-dimensional content is rendered by the computing device to be displayed at the location of the display;
terminal.
12. The method of claim 11,
The step of acquiring a second photographed image including the display comprises:
extracting an outline of the display from the second captured image; and
Recognizing an area corresponding to the display
containing,
terminal.
12. The method of claim 11,
Determining a position where the 3D content is to be rendered in the display based on the second captured image includes:
extracting feature points in the display; and
Matching the display to the screen in the terminal
containing,
terminal.
12. The method of claim 11,
The step of determining a position where the 3D content is to be rendered in the monitor from the second captured image,
obtaining coordinate information of the display;
determining the coordinates at which the three-dimensional content is to be rendered within the coordinates of the display;
containing,
terminal.
15. The method of claim 14,
The three-dimensional content is rendered based on the program running on the computing device,
terminal.
16. The method of claim 15,
When the program is a three-dimensional program, the three-dimensional content is rendered in the three-dimensional program,
terminal.
16. The method of claim 15,
If the program is not a three-dimensional program, the three-dimensional content is imaged and rendered in the program,
terminal.
12. The method of claim 11,
The step of generating 3D content using the object image includes:
Transforming the viewpoint of the 3D content based on a control command input from the user
containing,
terminal.
12. The method of claim 11,
The step of generating 3D content using the object image includes:
Generating the 3D content by using the depth information of the object image obtained from the server
containing,
terminal.

Images