KR102116673B1 - System and Method of Determining Blend Shape Weight Using Markerless Image and Computer Readable Storage Medium - Google Patents

Abstract

According to one embodiment of the present invention, a method for determining a blend shape weight using a markerless image may comprise the steps of: generating learning data by combining facial landmark data and blend shape weight information; learning a learning model using the learning data; and receiving real-time facial landmark data extracted from three-dimensional capture data obtained by three-dimensionally capturing a face of a person in real time while a marker is not attached to automatically determine a blend shape weight corresponding to the real-time facial landmark data.

Description

System and Method of Determining Blend Shape Weight Using Markerless Image and Computer Readable Storage Medium}

The present application relates to a system and method for determining a blend shape weight using a markerless image, and a computer-readable storage medium.

In order to implement facial animation, a technique for tracking a person's fine facial expressions by a motion capture system and applying the tracked facial expressions to a virtual character has been widely used.

To this end, in general, a marker is attached to a person's face, and an image of the marker is acquired by a camera, and then a blend shape weight is determined by analyzing it to use a method of tracking a person's facial expression.

However, when a person's facial expression is tracked using the marker as described above, inaccurate results may occur depending on the alignment of the marker. have.

Accordingly, there is a need in the art for a method for determining a blend shape weight using a markerless image that does not use a marker in realizing facial animation.

In order to solve the above problems, an embodiment of the present invention provides a method for determining a blend shape weight using a markerless image.

The method for determining a blend shape weight using the markerless image includes: generating learning data by combining facial landmark data and blend shape weight information; Learning a learning model using the learning data; And using the learned learning model, input real-time facial landmark data extracted from 3D capture data obtained by real-time 3D capture of a human face without attaching a marker to automatically receive the corresponding blend shape weight. And determining.

Meanwhile, another embodiment of the present invention provides a blend shape weight determination system using a markerless image.

The blend shape weight determination system using the markerless image includes: a 3D capture unit for 3D capturing a face of a person without a marker attached in real time and outputting 3D capture data; A facial landmark extracting unit extracting facial landmark data from the 3D capture data; A learning model that is learned in advance using learning data, receives the facial landmark data, and automatically determines a corresponding blend shape weight; And a blend shape data output unit that outputs a blend shape weight automatically determined by the learning model.

In addition, the solution means of the above-mentioned subject does not list all the characteristics of this invention. Various features of the present invention and advantages and effects thereof may be understood in more detail with reference to specific embodiments below.

According to an embodiment of the present invention, in implementing the face animation, a blend shape weight may be determined using a markerless image without a marker.

In addition, artificial intelligence technology can be used to automate the process of determining the blend shape weight, thereby enabling faster and more accurate face animation.

1 is a conceptual diagram illustrating a method for determining a blend shape weight using a markerless image according to an embodiment of the present invention.
2 is a block diagram of a blend shape weight determination system using a markerless image according to another embodiment of the present invention.

Hereinafter, preferred embodiments will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains can easily implement the present invention. However, in the detailed description of a preferred embodiment of the present invention, when it is determined that a detailed description of related known functions or configurations may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, the same reference numerals are used throughout the drawings for parts having similar functions and functions.

In addition, throughout the specification, when a part is said to be 'connected' to another part, it is not only 'directly connected', but also 'indirectly connected' with another element in between. Includes. In addition, "including" a component means that other components may be further included instead of excluding other components, unless otherwise stated.

1 is a conceptual diagram illustrating a method for determining a blend shape weight using a markerless image according to an embodiment of the present invention.

Referring to FIG. 1, as preparation process for determining a blend shape weight based on AI, first, learning data 30 for training the learning model 100 may be generated.

Specifically, a plurality of facial landmark data 15 may be extracted from a plurality of 3D capture data 10 for a human face. Here, the facial landmark data 15 may include information on a plurality of points for expressing the main features of the face (eg, eyes, nose, mouth, chin, and eyebrows). The method of extracting the facial landmark data 15 from the 3D capture data 10 may employ various methods known to those skilled in the art.

Also, as described above, by receiving the blend shape data 20 for the captured human face, a plurality of blend shape weight information 25 may be extracted for each part. Here, the blend shape data 20 may include blend shape weight information generated by existing software or the like.

Thereafter, the learning data 30 may be generated by combining the facial landmark data 15 of the human face and the blend shape weight information 25. For example, facial landmark data 15 is n + 1 (0, 1, 2, ..., n), and blend shape data 20 is m + 1 (0, 1) for k sites. ..., m) Assuming that a weight is included, learning data 30 may be generated by combining each facial landmark data 15 and blend shape weight information 25.

Thereafter, the learning model 100 may be trained using the generated learning data 30. Here, the learning model 100 may utilize, for example, CNN (Convolutional Nerual Network), but the present invention is not necessarily limited thereto. In other words, the learning model 100 may use a model selected from various machine learning or deep learning models known to those skilled in the art.

As described above, according to an embodiment of the present invention, in the learning model 100, facial shape data extracted from 3D capture data obtained by 3D capturing a human face is used together with blend shape weight information generated according to an existing method. By performing the learning can be implemented to learn the blend shape weight corresponding to the 3D capture data.

Accordingly, the learning model 100, in which learning is completed, receives real-time facial landmark data 40 extracted from 3D capture data obtained by real-time 3D capture of a human face without attaching a marker, and receives a corresponding blend. The shape data 50, that is, the automatically determined blend shape weight may be output.

According to the above-described embodiment of the present invention, it is possible to automatically determine a blend shape weight based on AI using a markerless image.

The method for determining the blend shape weight using the markerless image described above with reference to FIG. 1 may be performed by a processing device capable of 3D image processing and learning of an artificial intelligence model.

In addition, according to another embodiment of the present invention, a computer-readable storage medium in which instructions executable by a processor for executing each step of a method for determining a blend shape weight using a markerless image described above with reference to FIG. 1 is recorded. Can be provided.

2 is a block diagram of a blend shape weight determination system using a markerless image according to another embodiment of the present invention.

Referring to FIG. 2, a blend shape weight determination system 200 using a markerless image according to another embodiment of the present invention includes a 3D capture unit 210, a facial landmark extraction unit 220, a learning model 230, and It may be configured to include a blend shape data output unit 240.

The 3D capture unit 210 is for capturing a human face in 3D, and may be embodied as, for example, a stereo camera capable of acquiring a 3D image. The 3D capture unit 210 may output a 3D capture data by real-time 3D capturing the face of a person without a marker attached.

The facial landmark extracting unit 220 is for extracting the facial landmark data from the 3D capture data captured by the 3D capturing unit 210. For example, the facial landmark extracting unit 220 may extract facial landmark data from 3D capture data by employing various methods known to those skilled in the art.

The learning model 230 is for receiving facial landmark data extracted by the facial landmark extracting unit 220 and automatically determining a corresponding blend shape weight. To this end, the learning model 230 may be pre-learned using learning data combining facial landmark data and blend shape weight information for a human face. Here, since the learning data and the learning model are the same as described above with reference to FIG. 1, redundant description thereof will be omitted.

The blend shape data output unit 240 is for outputting a blend shape weight automatically determined by the learning model 230.

As described above, the face animation may be implemented such that the virtual character tracks the face target of the person by using the blend shape weight output by the blend shape data output unit 240.

The present invention is not limited by the above-described embodiments and accompanying drawings. For those skilled in the art to which the present invention pertains, it will be apparent that components, according to the present invention, may be substituted, modified and changed without departing from the technical spirit of the present invention.

200: blend shape weight determination system
210: 3D capture
220: facial landmark extraction unit
230: learning model
240: blend shape data output unit

Claims (5)

Generating learning data by combining facial landmark data and blend shape weight information;
Learning a learning model using the learning data; And
Using the learned learning model, real-time facial landmark data extracted from 3D capture data obtained by real-time 3D capture of a human face without a marker is input, and a corresponding blend shape weight is automatically determined. The steps include,
The step of generating the learning data,
Extracting a plurality of facial landmark data from a plurality of 3D capture data of the human face;
Extracting a plurality of blend shape weight information for each part by receiving blend shape data for the person's face; And
And combining the plurality of facial landmark data and the plurality of blend shape weight information to generate learning data.
delete A 3D capture unit that 3D captures the face of a person without a marker attached in real time and outputs 3D capture data;
A facial landmark extracting unit extracting facial landmark data from the 3D capture data;
A learning model that is learned in advance using learning data and receives the facial landmark data and automatically determines a corresponding blend shape weight; And
It includes a blend shape data output unit for outputting a blend shape weight determined automatically by the learning model,
The learning data extracts a plurality of facial landmark data from a plurality of 3D capture data on a person's face, receives blend shape data for the person's face, and extracts a plurality of blend shape weight information for each part. , A blend shape weight determination system using a markerless image, characterized in that it is generated by combining the plurality of facial landmark data and the plurality of blend shape weight information.
delete A computer-readable storage medium in which instructions executable by a processor for executing a method for determining a blend shape weight using a markerless image according to claim 1 are recorded.

Images