KR101902553B1 - Terminal for providing storytelling contents tool and Method for providing storytelling - Google Patents

Abstract

According to an aspect of the present invention, there is provided a terminal for providing a storytelling content tool, the terminal which includes a storytelling content storage unit having storytelling content; a user face image generation unit for generating a user face image by photographing a face of a user enjoying the storytelling content; a face reference value storage model in which a face model composed of a face identity value obtained by representing a face shape of each reference face as successive numerical values and a face expression value obtained by representing a face expression of each reference face as successive numerical values is stored; a user face analysis unit for estimating a face identity value and a face expression value matching with the user face image generated through the user face image generation unit from the face reference value storage model to determine a user face identity value and a user face expression value; and a storytelling content progressing unit for processing the user face image by reflecting the user face identity value and the user face expression value determined through the user face analysis unit to output the user face image as the storytelling content.

Description

Technical Field [0001] The present invention relates to a terminal providing storytelling contents tool and method for providing storytelling contents,

The present invention relates to a storytelling content tool providing terminal and a storytelling content providing method. The present invention relates to a storytelling content tool providing terminal and a storytelling content providing method, in which a user (child) And a system that allows the system to analyze user's content usage information and recommend and proceed with the content.

Many people want to be the protagonists of media content while using trick art, course play, face conversion and modification applications. This behavior gives people pleasure and allows users to experience more immersive content.

In particular, young children enjoy children's storybooks, children's videos, and follow a lot of characters in them.

In order to provide a more realistic experience to users, more accurate user capture algorithms and corresponding realistic image presentation technologies are required. There is also a need for a personalized content delivery strategy based on user usage information.

Conventional technologies and systems are based on a paper assortment using mirrors, a post-production of a facial animation image based on a user-submitted facial photograph, and a facial photograph input by the user to satisfy the needs of such users It provides simple real-time image content creation, system that captures and analyzes face images of users in real time, and adds and edits the enhancement data.

In such a case, there is a problem that the user does not meet the following requirements.

In the case of a paper assortment using a mirror, only the contents indicated by the assortment can be used by the user. The user does not directly view and view the result image, because the face animation image production and delivery system based on the face photographs submitted by the user. In the case of a simple image content real-time creation system based on a user's input face image, the existing system allows the face information to be modified according to a predetermined frame without real-time transfer to the facial expression and emotion of the current user. In the system for photographing and analyzing the user's face image in real time and adding and editing the augmentation data, it is natural that the augmentation data is applied to the face information, but it provides short-term enjoyment based on the face information and does not provide long- .

Above all, existing technologies are unnatural because they can not naturally synthesize user's face information into content. It also ends with short-term enjoyment and is not being used as an educational system through interaction with long-term users. In addition, the existing technology does not provide an accurate methodology in which the system recognizes the user's facial information and naturally inserts it into a storybook, and extracts the facial image based on the manual facial feature point and pastes it into the storybook.

Korean Patent Publication No. 10-2015-0011651

The technical problem of the present invention is to naturally transmit user's face information and user's name information to storytelling contents to provide a feeling that the user becomes real and feel like a main character. Another problem to be solved by the present invention is to provide a system for analyzing information experienced by a user in storytelling contents and recommending and proceeding educational contents suitable for a user.

An embodiment of the present invention relates to a storytelling content storage unit having a storytelling content used in storytelling; A user facial image generating unit for capturing a face of a user enjoying the storytelling contents to generate a user facial image; A face reference value storage model in which a face identity representing a face shape of each reference face is represented by successive numerical values and a face expression value representing a facial expression value expressed by successive numerical values of each reference face is stored; A user facial analysis unit for estimating a face identity value and a facial expression value matching with a user facial image generated through the user facial image generation unit based on the face reference value storage model and determining the user facial identity value and the user facial expression value; And a storytelling content advancing unit for processing the user's face image by reflecting the user's face identity value and the user's facial expression value determined through the user's facial analysis unit and outputting the user's facial image as a storytelling content.

The storytelling content progressing unit may exaggerate and transform the user's face identity value and the user's facial expression value to the face model of the storytelling content, and output the same.

Wherein the user facial analysis unit grasps a face skin color distribution of a user using a pixel value of a user facial image generated through the user facial image generation unit and determines a color distribution of the storytelling contents output from the storytelling content progress unit, And the storytelling content progress unit may output the user's facial skin color distribution to the 3D facial model or the user's facial skin color distribution corresponding to the content to reflect the user facial skin color distribution.

The storytelling contents progress section may modify and output the subtitle contents in accordance with the user's personal information.

The storytelling contents tool providing terminal includes a face information model for a person who receives face information of a person other than the user who enjoys the storytelling contents from the storytelling contents server and stores the face information as the face information of the other person, The controller may output the tethered 3D face model reflecting the tether face information together with the 3D face model of the user to the storytelling contents.

The storytelling contents tool providing terminal may further include a conversion output unit for converting the output storytelling contents into a printable document file.

Further, an embodiment of the present invention is a user face image generating process for capturing a face of a user enjoying a storytelling content to generate a user face image; Determining the user's face identity value and the facial expression value for determining the user's facial expression value representing the user's facial expression as a continuous numerical value by using the captured user's facial image, process; And outputting the user's face identity value and the user's facial expression value to the storytelling contents to output the storytelling contents.

There is provided a face reference value storage model in which a face expression value expressed by successive numerical values of each facial expression of each reference face and a face identity value representing a face shape of each reference face by continuous numerical values are assigned for each reference face,

The face reference value storage model includes at least one of Principal Component Analysis (PCA), Singular Value Decomposition (SVD), and High Order Singular Value Decomposition (HOSVD) from face image data of various people photographed for each face shape and facial expression The facial information of the data can be constructed by factoring the facial identity value and the facial expression value into consecutive numerical values through a data analysis method.

The storytelling content output process may reflect the user's face identity value and the user's facial expression value in the 3D facial model or reflect the user facial identity value and the user facial expression value in the 2D facial model to output to the storytelling content can do.

The user facial identity value and the user facial expression value are reflected in the 3D facial model, where V is a 3D facial model, C _r is a facial feature distribution core structure model learned from the facial reference value storage model, w _id is a user facial identity Value, and w _exp is the user facial expression value, it can be reflected by using the function V = g (C, w _id , w _exp ).

The reflection of the user's face identity value and the user's facial expression value on the 3D facial model can be reflected on the 3D facial model by emphasizing the characteristic values of the facial identity value (w _id ) and the facial expression value (w _exp ).

에 의해 산출되는 모델을 화면에 출력할 수 있다.When outputting the 3D facial model (V) reflecting the function V = g (C, w _id , w _exp ) on the screen, the 3D facial model is projected on the two-dimensional screen _(F) is a camera function that maps the 3D position to the 2D position on the screen, R is face rotation information for rotating the face, T is the position of the face When the face position information is moved,

Can be output to the screen.

Reflecting the user facial identity value and the user facial expression value into the 2D facial model, the facial identity value and the facial expression value analyzed in the user facial image can be reflected in the two-dimensionally projected face reference value storage model.

The method may further comprise the step of exaggerating the characteristics of the user's facial information derived from the facial identity value and the facial expression value determination process before the storytelling content output process after the process of determining the user's facial identity value and facial expression value, And a step of re-processing the user's face information that re-processes the texture information including the skin information to match the texture of the story telling contents.

The method may further include a scorer content conversion step of converting the output storytelling content into a printable document file.

는 입력 정보로부터 구별되는 특징(feature)를 추출하는 특징 추출 연산자라고 할 때,

에 의해 얼굴 정체성값과 얼굴 표정값이 결정되는 것을 특징으로 하는 스토리텔링 콘텐츠 제공 방법.Wherein the step of determining the user's face identity value and the facial expression value in the user facial image comprises the steps of: determining a facial identity value (w _id ), a facial expression value (w _exp ), a size value (s) rotation of the translational value (t _x, t _y), the face rotation information of the information _{(r x, r y, z} ), and the set of components including a camera focal length (f), f is a factor of m I is an input image, V is a 3D face model that is _elementized by a facial identity value (w _id ) and a facial expression value (w _exp ), and x _2d and x _3d are input to a _two- A function that extracts the location of the feature points in the model, A is a texture learning model of the 3D face model,

Is a feature extraction operator that extracts features distinguished from input information,

Wherein the face identification value and the facial expression value are determined by the face identification value and the facial expression value.

The present invention can analyze the face of the user in real time and re-process the face to naturally enter the storytelling contents to provide contents that make the user feel like the main character. In addition, the present invention analyzes a user's facial expressions and emotions in real time, and incorporates emotions of the user into the storytelling contents, thereby providing a function of allowing the user to experience the storytelling contents more realistically. In addition, the present invention provides a function of automatically analyzing a learning level through a user's storytelling content, recommending content suitable for a learning level to a user, and allowing a user to enjoy contents more conveniently.

1 is a configuration block diagram of a storytelling contents tool providing terminal according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating facial identities and facial expressions stored in a face reference value storage model according to an embodiment of the present invention. FIG.
3 is a diagram illustrating storytelling contents before and after reflection of a user's 3D face model according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating a storytelling content server according to an embodiment of the present invention.
5 is a flowchart illustrating a story telling contents providing process according to an embodiment of the present invention.
FIG. 6 is a view showing a 3D face model in which a face value of a user and a user's face characteristic value are reflected in a user face image according to an embodiment of the present invention.
7 is a view illustrating a 3D face model rotated according to an embodiment of the present invention.
8 is a diagram illustrating an example of projecting or inserting a user's facial model analyzed according to an embodiment of the present invention on a content by modifying the facial model of the user according to the environment such as color and style of the content.
9 is a diagram illustrating rotation or deformation of a face reference value storage model and a 3D face model.
FIG. 10 is a diagram illustrating creation of a three-dimensional character face that has been animated according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and how to achieve them, will be apparent from the following detailed description of embodiments thereof taken in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the exemplary embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concept of the invention to those skilled in the art. And the present invention is only defined by the scope of the claims. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

FIG. 1 is a block diagram of a terminal for providing a storytelling contents tool according to an embodiment of the present invention. FIG. 2 is a diagram illustrating face identities and facial expressions stored in a face reference value storage model according to an embodiment of the present invention. 3 is a view showing a storytelling content before and after reflection of a 3D face model of a user according to an embodiment of the present invention, and FIG. 4 is a diagram illustrating a state in which a storytelling content server is provided according to an embodiment of the present invention to be.

Hereinafter, the storytelling contents tool providing terminal 100 refers to a terminal for outputting storytelling contents. The storytelling contents tool providing terminal 100 will be described as an example of a smart phone. In addition to a smart phone, If a camera capable of recognizing a face in the system or a sensor capable of recognizing facial expressions is incorporated in a player (PMP, Portable Multimedia Player), a personal digital assistant (PDA, Personal Digital Assistant) Of course, it is needless to say that the terminal to which the present invention is applicable is not limited to the above-described kind, but may include all the terminals provided with the calculation means.

For reference, story telling is a compound word of "story + telling", which literally means talking about persuasively what you want to tell the other person with interesting and vivid stories. Accordingly, the storytelling contents may include various types of contents such as images, pictures, and animations output as a storytelling form.

The storytelling content tool providing terminal 100 of the present invention analyzes the user's face in real time and reprocesses the face to naturally enter the storytelling contents to provide a content that makes the user feel as if they are the main character.

1, a storytelling contents tool providing terminal 100 according to the present invention includes an input unit 160, a display unit 170, a storytelling contents storage unit 120, A user facial image generation unit 130, a face reference value storage model 110, a user facial analysis unit 140, and a storytelling content progress unit 150. In addition, at least one of the other person's face information model 180 and the conversion output unit 190 may be additionally provided.

The input unit 160 is a module that receives a user's selection event at the time of storytelling output. Page selection, page transfer, and the like can be input from the user. Also, the user can be guided to the selection by presenting the user with a storytelling content to recommend to the user. Also, it is possible to input the user's personal information including the name to the user. In addition, it is possible to take a picture of a face of a person other than the user and input it in advance.

The display unit 170 is a display window on which the storytelling contents are outputted as video images. The display unit 170 naturally displays the storytelling content experienced by the current user and the natural user's face information reproduced in the storytelling content progressing unit 150 to the user in the storytelling content to make the user feel the main character of the storytelling content .

The input unit 160 and the display unit 170 may be implemented as a single type of touch screen panel. A touch screen panel is provided on a front surface of a terminal to provide a touch screen screen capable of simultaneously performing input and display, and includes a graphic user interface (GUI) for communicating with a user enjoying storytelling contents. Graphic User Interface).

The storytelling content storage unit 120 is a memory having a 3D face model used in storytelling, a 2D face model, and a storytelling content including caption content. The 2D face model refers to a character modeling a face of a person used for story telling as a 2D information value, and the 3D face model is a character modeling a face of a person used for storytelling as a 3D information value. Also, the term " caption content " refers to a caption provided together with the output of the storytelling content.

The user face image generating unit 130 photographs a face of a user enjoying the storytelling contents to generate a user face image. That is, the face image analyzing unit 140 provides a user face image that captures an image of a user through a camera or a sensor for recognizing various facial expressions. Accordingly, the user face image may refer to a photographing device including a camera, a file of a user's photographed through a face recognition sensor, or a photographed image of a user. The user face image generating unit 130 may generate at least one or more user face images for reprocessing the face information suitable for incorporating the storytelling contents and provide the user face images to the user face analyzing unit 140, Or may be generated by separately extracting the face portion from the previously captured image.

The face reference value storage model 110 is a reference value model data used when analyzing a user's face image. The face reference value storage model 110 stores a face identity value representing a face shape of each reference face as a continuous numerical value, And the face expression values expressed by numerical values are assigned to the reference faces. The face reference value storage model is a model data constructed by machine learning of 3D facial images of various facial expressions of various people and is expressed as an aggregate data model in which face shapes and facial expressions of various people are expressed by successive numerical values Or can be built in database form. When w _id is a user face identity value and w _exp is a user face expression value, a user face identity value and a user facial expression value may be assigned differently for each reference face in the form of a matrix as shown in FIG. For reference, the concrete face value and the specific numerical value of the face standard value for each reference face are not shown in Fig.

Here, the face identity value is a value obtained by ellifying the type of the face shape as a continuous numerical value. For example, if the forehead has a wide face, a cheekbone protruding face, a small or large eye, a long face with a cone or chin, eyes, forehead, nose, lips, jaws, cheeks, Various face shapes may exist depending on the shape and size, and different face identity values may be set according to each face shape. Even if you practice the same expression, the shape of your face may vary from person to person. Therefore, when user face information is generated, a face identity value having a different value can be considered for each user. Thus, the face identity value is a value obtained by ellating the constituent parts of the face into consecutive numerical values. That is, values obtained by digitizing the eye size of the face, the nose size, the mouth size, the width of the face, and the height of the face may be applicable.

The facial expression value means a value obtained by factoring facial expressions into consecutive numerical values. That is, it can have a different value depending on the degree of change of expression of a face model such as an exercise face model or a user's face model to be performed. The facial expression value can be expressed by various variables. For example, the facial expression value associated with the eye shape of the face model may have a different value depending on the degree of change of the eye shape, and the facial expression value associated with the mouth shape of the face model may be different depending on the degree of change of the mouth shape Lt; / RTI > In addition, various facial expression values can be defined in consideration of the positions and orientations of facial parts and facial muscles. At this time, the facial expression value may have the same facial expression value if the facial expression represented by the facial model, that is, the facial identity value, is different from that of the same facial expression.

Facial image data of various people photographed by face shapes and facial expressions are arranged through Procrustes Analysis, and at least one of Principal Component Analysis (PCA), Singular Value Decomposition (SVD) and High Order Singular Value Decomposition (HOSVD) The facial reference value storage model 110 can be constructed by making the facial information of the data into elements by successive numerical values of the facial identity value and the facial expression value through the data analysis method.

The user facial analysis unit 140 analyzes the identity value and the facial expression value of the face image of the user in the provided photograph information. To this end, the user facial analysis unit 140 estimates a face identity value and a facial expression value matching with the user facial image generated through the user facial image generation unit 130 based on the face reference value storage model 110, The facial identity value and the user facial expression value are determined.

That is, the face image of the user is analyzed to match the face value of the user, which is a value obtained by factoring the eye size, the nose size, the mouth size, the face width, and the face height of the face into successive numerical values, The reference face matching with the facial expression value considering the position and direction of the facial muscle is estimated from the face reference value storage model 110. Here, the term " matching " may mean completely identical, but actually refers to matching the face identity of the user face image and the reference face having the highest similarity with the face expression value. Accordingly, the skin color information and the texture map of the user analyzed from the pixel values corresponding to the face region of the user can be modified to match the color, texture, and texture of the storytelling contents to give a stereoscopic effect.

The storytelling content advancement unit 150 exaggerates and transforms the user's face identity value and the user's facial expression value determined through the user's facial analysis unit 140 and outputs the result to the display unit 170 as the storytelling content . As described above, it is needless to say that the process of determining the value and the facial expression value by comparing with the reference value, and the process of exaggerating and deforming the facial expression value and the user's facial identity value can be performed in real time. The user facial-identity storytelling content progressing section 150 may transform the user facial identity value and the user facial expression value into a 3D facial model or a 2D facial model of the storytelling contents, or a user facial skin color distribution, The user's facial skin color distribution can be reflected and output. That is, the storytelling content progressing unit 150 analyzes the emotion from the user face image photographed in real time, emphasizes the facial expression value corresponding to the emotion, reflects it on the user 3D face model, and displays it. For example, as shown in Fig. 3 (a), as a 3D face model reflecting user face identity values and user facial expression values in Fig. 3 (b) in the storytelling contents of the Mona Lisa pictorial information, And the output can be displayed. For reference, a 3D face model refers to a model that is displayed in a three-dimensional three-dimensional face shape.

In addition, the storytelling content progressing unit 150 can analyze the information interacted with the storytelling contents by the user, and recommend and process the next storytelling contents.

Furthermore, the storytelling content progressing unit 150 may reflect the user's facial skin color distribution to the 3D facial model and output it. To this end, the user facial analysis unit 140 grasps the user's facial skin color distribution using the pixel values of the user facial image generated in real time through the user facial image generation unit 130, ) Transforms the skin color distribution of the user's face analyzed according to the color distribution and style of the storytelling contents being output.

In addition, the storytelling content advancement unit 150 can modify and output the subtitle contents according to the user's personal information.

In addition, the storytelling content progressing unit 150 may be configured to output a third-person 3D face model reflecting the third-person face information together with the 3D face model of the user together with the storytelling content. To this end, as shown in FIG. 4, the storytelling contents tool providing terminal is provided with a storytelling content server, and receives the face information of a person other than the user enjoying the storytelling contents from the storytelling contents server, And a face information model 180 for storing the face information. Therefore, the storytelling content advancement unit 150 can output a different 3D face model reflecting the other person's face information together with the storytelling content, thereby doubling the interest.

Therefore, the present invention discloses a storytelling content platform in which a user who operates on a platform including a smart device (tablet) including a camera or a sensor capable of photographing a user's face is a main character, The telling contents platform acquires the face of the user who has photographed the face of the user in real time or photographed the user, interacts with the storytelling contents by the user, displays the storytelling contents in which the face and the personal information of the user are reproduced and inputted, But also a user who is using the present invention in another person or another device that has been photographed and analyzed in advance can also appear in the content used by the user.

On the other hand, the storytelling content progressing unit 150 reflects and outputs the user's face identity value and the user's facial expression value determined in real time through the user's facial analysis unit 140. The facial expression value reflects the 3D face Model or 2D facial model. In this case, it can be reflected through the calculation of formulas and can be reflected by various reprocessing. Hereinafter, an example of such reflection will be described in detail with reference to FIG.

Meanwhile, the conversion output unit 190 converts the storytelling contents reflected on the display unit 170 into a printable document file through various processes, or generates a printing file and outputs the printing file. After the storytelling contents are created as a document file, they can easily be output to paper through a printer or easily exchanged and modified between users.

FIG. 5 is a flowchart illustrating a story-telling content providing process according to an embodiment of the present invention. FIG. 6 is a flowchart illustrating a process of providing a storytelling content according to an exemplary embodiment of the present invention. FIG. 7 is a view illustrating a 3D face model rotated according to an embodiment of the present invention.

The storytelling contents providing process of the present invention includes a user face image generation process S610, a user face identity value and a facial expression value determination process S620, a user face information reprocessing process S630, a storytelling content output process S640, And a storytelling content conversion process (S650).

The user face image generation process (S610) is a process of generating a user face image by photographing the face of a user enjoying the storytelling contents in real time. The face of the user can be photographed in real time through various photographing means such as a camera and a sensor or a face image of a user that has been photographed before.

The user's facial identity value and facial expression value determination process S620 is a process for determining a user's facial identity and a facial expression value using a user facial image obtained by parameterizing a user's facial shape to a continuous numerical value, And determining the user facial expression value that is elementized by the numerical value.

The facial expression value obtained by elementizing the facial expression of each reference face by continuous numerical values and the facial identity value obtained by factoring the facial shape of each reference face into consecutive numerical values are assigned to the reference face, The user's face identity value and facial expression value determination process S620 may be performed by comparing the user face image obtained in S610 with the face reference value storage model by comparing and matching the face identification value and the facial expression value Can be extracted and determined.

The process of re-processing the user's face information (S630) may include a process of exaggerating the characteristics of the user's facial information derived from the facial identity value and facial expression determination process, or re-processing the texture information including skin information of the user's face information, .

The storytelling content output process S640 is a process of displaying the user's face identity value and the user's facial expression value on the storytelling contents through the display unit. Such a storytelling content output process (S640) may reflect the user facial identity value and the user facial expression value in the 3D facial model, or reflect the user facial identity value and the user facial expression value in the 2D facial model to be displayed in the storytelling content Can be output. That is, as shown in FIG. 7, the 3D facial model can be reflected, and the rotated 3D facial model can be reflected.

The storytelling contents conversion process S650 is a process of transforming the storytelling contents reflected on the display unit 170 into a printable document file through various processes, or outputting it as paper. After the storytelling contents are created as a document file, they can easily be output to paper through a printer or easily exchanged and modified between users.

As described above, the face reference value storage model 110 stores the 3D facial image data of various people photographed by face shape and facial expression through the Procrustes Analysis, and then performs PCA (Principal Component Analysis), SVD (Singular Value Decomposition) , And HOSVD (High Order Singular Value Decomposition), the facial identity value and the facial expression value can be constituted into a continuous set of numerical values and constructed as a set of various facial models.

이다. 여기서,

은 실수 공간을 의미하고, p는 3D 얼굴 모델인 V의 vertex의 수를 의미한다. 즉, 3p는 vertex 수의 3차원 공간을 의미한다.At this time, if the 3D face model in the face reference value storage model 110 is V,

to be. here,

Denotes the real space, and p denotes the number of vertices of the 3D face model, V. That is, 3p denotes a three-dimensional space of the number of vertices.

On the other hand, the 3D face model V in the face reference value storage model 110 can be expressed as [Equation 1].

[Formula 1]

V = g (C, w _id , w _exp )

Where w _id is the face identity value of the user, w _exp is the face facial expression value of the user, C is the learning model of the facial feature distribution learned through at least one of PCA, SVD, and HOSVD in the system, The function g () can be expanded in various ways. In other words, the 3D face model can generate various face models by changing the values of w _id and w _exp in C.

On the other hand, when the face reference value storage model is learned and configured through HOSVD, the 3D face model V can be expressed by a tensor operation as shown in the following [Equation 2].

[Formula 2]

Here, C is the core tensor calculated through HOSVD.

Here, the tensor operation is an expression for parameterizing the face identity value, the facial expression value, and the face structure information (vertex) with each other. The facial identity value, facial expression value, face size information, face position information, face rotation information, and camera information analyzed from the user facial image can be used in the rendering process as follows.

On the other hand, when the 3D face model V is expressed by the mean and eigenvector of the data through the PCA, it can be expressed as the following [Equation 3].

[Formula 3]

,

,

는 각각 PCA를 통해 학습된 데이터의 평균값, 얼굴의 정체성과 관련된 고유벡터, 얼굴의 표정값과 관련된 고유벡터이다.At this time

,

,

Is an eigenvector related to the average value of the data learned through the PCA, the eigenvector associated with the facial identity, and the facial expression value.

On the other hand, in the storytelling content outputting process (S640), the 3D face model (V) reflecting the function V = g (C, w _id , w _exp ) is projected onto the two- .

[Formula 4]

를 이용해 3D 위치를 화면상의 2D 위치로 매핑하는 카메라 함수이고, R은 얼굴을 회전시키는 얼굴 회전 정보이며, T는 얼굴의 위치를 이동시키는 얼굴 위치 정보이다. 여기서,

이고,

이며,

이다.Here, F is a 2D face model, a 3D face model is projected on a two-dimensional screen, s is face size information for adjusting the face size, and M _{P (f)} is a perspective projection procession

R is a face rotation information for rotating a face, and T is a face position information for moving a position of a face. here,

ego,

Lt;

to be.

[Equation 4] corresponds to a mathematical expression when a 3D face model calculated by [Equation 1] is displayed to a user on an actual screen and is modeled.

That is, the 3D facial model can be rotated at various angles and the size can be adjusted and output by the above-described [Expression 4], and the face identity value and the facial expression value can be changed and output.

9 is a diagram illustrating rotation or deformation of a face reference value storage model and a 3D face model. That is, the 3D facial model can be rotated at various angles and the size can be adjusted and output by the above-described [Expression 4], and the face identity value and the facial expression value can be changed and output.

라고 기술하도록 한다.To simplify the expression of variables afterwards

.

Here, m is a set of each component, and the constituent elements of the set include a facial identity value (w _id ), a facial expression value (w _exp ), a size value (s) in the face size information, a value (t _x, t _y), rotation of the face in the rotation information _{(r x, r y, r} z), the camera focal length (f).

[Formula 5]

는 2차원 화면에 투영된 3D 얼굴 모델, 즉 2D 얼굴 모델로부터 기정의된 vertex의 위치만을 추출하는 함수이고,

는 w_id와 w_exp로부터 생성된 3D 얼굴 모델인 V로부터 기정의된 vertex의 위치만을 추출하는 함수이다. 이 때,

은 기정의된 vertex의 수이며

이다.

Is a function for extracting only a position of a vertex determined from a 3D face model projected on a two-dimensional screen, that is, a 2D face model,

Is a function that extracts only the positions of the vertices that have been defined from the 3D face model V generated from w _id and w _exp . At this time,

Is the number of vertices predefined

to be.

M represents a face value (w _id ), a face expression value (w _exp ), a size value s in the face size information, a translation value (t _x , t _y ) in the face position information, rotation of the rotation information _{(r x, r y, r} z), the set of components including a camera focal length (f).

를 후술하겠지만 입력 정보로부터 구별이 분명한(distinctive) 특징(feature)를 추출하는 특징 추출 연산자라고 할 때, 상기 [식 5]에 의해 얼굴 정체성값(w_id)과 얼굴 표정값(w_exp)이 결정될 수 있다.F is a model projected on a two-dimensional image generated from elements of m, I is an input image, V is a 3D face model that is _elementized by facial identity value (w _id ) and facial expression value (w _exp ), x _2d and x _3d is a function for extracting the positions of predetermined feature points in the input model, A is a texture learning model of the 3D face model,

A face identity value w _id and a facial expression value w _exp are determined by Expression 5 as a feature extraction operator that extracts distinctive features from input information as will be described later .

[Equation 5] is an objective function used when restoring the 3D face model of [Equation 4] from the input image I, which is photographing information received from the camera, or when learning the model. That is, it means an objective function used when restoring the face shown in the image to the 3D face model. In this case, A is a texture learning model of a 3D face model. That is, the aim is to find? M that updates the parameter m of the current 3D face model that depicts the face in the image.

는 합성곱 신경망(Convolutional Neural Network), 심층 신경망(Deep Neural Network), SIFT 중 적어도 하나를 사용하여 데이터의 특징(feature)을 추출(extraction)하는 특징 추출 연산자이다. 이때,

가 합성곱 신경망 혹은 심층 신경망인 경우, 비록 두 입력 데이터의 종류가 다르더라도(

) 특징 추출 연산자(

)를 통과한 두 데이터가 유사한 값을 출력하도록 상기 특징 추출 연산자(

)를 학습시킬 수 있다. 이 때,

과

는 상기 설명한

의 형태를 갖는 특징 추출 연산자의 예를 나타내며, 둘은 내부 상수 혹은 변수만 다를 뿐 서로 동일한 연산자일 수 있으며, 혹은 서로 다른 형태의 입력을 받지만 같은 값을 출력하도록 디자인된 서로 다른 연산자일 수 있다. here,

Is a feature extraction operator that extracts features of data using at least one of Convolutional Neural Network, Deep Neural Network, and SIFT. At this time,

Is a composite neural network or a neural network, even though the types of the two input data are different

) Feature Extraction Operator (

) To output a similar value to the feature extraction operator (

Can be learned. At this time,

and

Quot;

And two of them may be the same operator with different internal constants or variables or they may be different operators designed to receive different types of input but output the same value.

The feature extraction operator means to display the data more informally with information other than the image information. In the neural network, the neural network extends the existing neural network, such as Convolution Layer, Pooling Layer, Normalization Layer and Recurrent Layer. It can mean a learning technique that uses a network of combinations.

And dist [] represents the distance between two vectors, and the similarity can be judged as the distance between two vectors. Meanwhile, the distance between the two vectors can be measured using at least one of cosine distance, Euclidean distance, and Hamming distance.

Here, the cosine distance means a method of finding the similarity between two data by a cosine operation, and the Euclidean distance means a method of finding the similarity between two data by L2 norm. Difference between elements means a method of measuring by the number of parts.

Meanwhile, when analyzing the user facial identity value and the user facial expression value in the image, the 2D facial feature point (X ^2d ) is obtained through the objective function of the following [Formula 6] rather than the method using the 3D facial model using [Formula 5] The user's face identity value and the user facial expression value can be analyzed.

[Formula 6]

는 상기 [식 5]에서의 설명과 같다. In this case, I is the input image from which the facial feature points are to be extracted, dist [

Is the same as that described in the above-mentioned [Formula 5].

는 2D 얼굴 영상이나 얼굴 기준값 저장 모델의 3D 얼굴 영상에서 기정의된 특징점, D는 특징점이 마킹되어 있는 2D 얼굴 영상 혹은 3D 얼굴 영상을 의미한다. 이때, D가 3D 얼굴 모델 데이터베이스의 경우 [식 1]의 얼굴 기준값 저장 모델을 만들기 위한 3D얼굴 모델 데이터베이스와 동일할 수 있다. 즉, [식 6]은 입력 영상 I에서 현재 추측(estimating)중인

로부터 나온 Feature가 데이터베이스로부터 학습한 Feature,

와 가까워지도록

를 업데이트 하는

를 추측하는 것이다. 이와 같은 방법으로

를 추측함으로써 영상 속 얼굴의 특징점의 위치를 알아낼 수 있다.

Is a predetermined feature point in a 3D face image of a 2D face image or face reference value storage model, and D is a 2D face image or a 3D face image in which feature points are marked. In this case, D may be the same as the 3D face model database for creating the face reference value storage model of [Expression 1] in the 3D face model database. In other words, [Equation 6] is obtained by dividing the input image I currently estimated

Feature extracted from the database,

To be close to

To update

. In this way

The position of the feature point of the face in the image can be found.

과

는 상기 설명한

의 형태를 갖는 특징 추출 연산자의 예를 나타내며, 둘은 내부 상수 혹은 변수만 다를 뿐 서로 동일한 연산자일 수 있으며, 혹은 서로 다른 형태의 입력을 받지만 같은 값을 출력하도록 디자인된 서로 다른 연산자일 수 있다. 또한,

는 특정 고차원의 특징을 추출하는 것이 아닌, 입력된 값, 혹은 입력된 값의 요소값을 그대로 출력하도록 디자인할 수 있음은 물론이다.At this time,

and

Quot;

And two of them may be the same operator with different internal constants or variables or they may be different operators designed to receive different types of input but output the same value. Also,

It is needless to say that the input value or the element value of the input value can be designed to be output without directly extracting the specific high dimensional feature.

[Equation 7]

[Equation 8]

는 데이터베이스에서의 평균 얼굴 형태이다.Thereafter, the method comprises one of [Formula 7] and [Expression 8] and the face position (X ^2d) the PCA (Principle Component Analysis), SVD (Singular Value Decomposition), (High Order SVD) HOSVD of the feature points as The user's facial identity value or user's facial expression value can be inferred by projecting the facial identity or the facial expression shape space (C _id , C _exp ) of the analyzed face reference value storage model. At this time, ,

Is the average face shape in the database.

For calculation through [Equation 5] and [Equation 6], a gradient regression method using a gradient descent method, a linear regression method using a continuous regressor, a recurrent neural network ) May be used.

The slope descent method is a method using the differential concept, which means an asymptotic algorithm that approaches the minimum or maximum of the function step by step in proportion to the gradient at the current position. That is, it means searching along the gradient starting from the initial estimate of the solution of the problem to find the minimum or maximum value of the given problem.

In addition, the regression method through the continuous regressor means an algorithm that continuously searches for a series of regressors by learning a solution to an existing problem in order to obtain a minimum or maximum value of a function. At this time, as a regressor, there are a continuous linear regressor which expresses a problem as a feature space of another dimension and obtains a problem solution by repetitive linear combination in the feature space, And a tree bases regressor for finding a solution by a series of combinations of feature vectors of a FERN structure.

In addition, the iterative neural network is a neural network that is characterized by a repetitive cyclic structure that can proceed both forward and backward (forward) rather than forward (forward) data direction when constructing a neural network do.

Finally, in order to utilize the user's face information from the facial image, the present invention analyzes the facial information through the steps of 1) face detection, 2) facial feature point detection, and 3) facial feature recognition including emotion and facial expression. In this case, 1) a template-based detection method using Adaboost (Adaptive Boosting) can be used for fast and accurate processing in the face detection step. 2) In the facial feature point detection step, a method using [Equation 6] can be used and a Convolutional Neural Network can be used. You can also use Convolutional Neural Network at all stages. 3) Convolutional Neural Network can be used at the stage of face recognition including emotion and facial expression.

Here, both Adaboost and Spiral Neural Network are algorithms used for Machine Learning. Adaboost means to create a stronger classifier by combining weak classifiers successively, and it can be increased by using it with other kinds of learning algorithms. In addition, the articulated neural network is a type of artificial neural network that allows a certain pattern to be grasped in a large amount of data.

The user's facial characteristics (eye size, nose size, face height, etc.) are analyzed from the analyzed facial identity value (w _id ) or the analyzed facial shape (X _i ^2d ). We emphasize facial characteristics of 3D face model by emphasizing user 's face identity value based on analyzed facial characteristics. In addition, the user's face information is received in real time and the user's [Formula 5] or [Formula 6] is solved in real time to update the user's face model. When updating the face model of the user, when the change of w _id converges to some extent, fix w _id and solve [Expression 5] or [Expression 6].

Similarly, while updating the face model of the user, if the change of W _exp converges to some extent, fix w _id and solve [Eq. 5] or [Eq. 6].

In addition, the texture of the analyzed face model is mapped to have a distribution similar to the texture of the pre-defined current storytelling contents, The texture of the user's face model is matched with the texture of the current storytelling contents. In this case, a clustering algorithm such as k-means clustering can be used. Or a Convolutional Neural Network technique in which styles of image contents are applied to other image contents.

The real-time rotation information of the user's face analyzed through [Expression 5], [Expression 6], and [Expression 7] can be added to influence the reference face motion of the main character on the current storytelling content by giving a predetermined weight.

In addition, the real-time facial expression information of the user's face analyzed through [Formula 5] or [Formula 7] can be added to affect the reference expression of the main character on the current storytelling content by giving a predetermined weight.

In addition, real-time emotion information of the analyzed user's face can be added to influence the reference emotion of the main character on the current storytelling content by giving a certain weight. In addition, the perceived emotion information may affect changes in the sub-contents constituting the storytelling contents.

Also, it is possible to recommend the next contents by analyzing the stored story information contents interaction information of the user. At this time, the analysis index can be a comprehension of the contents of the user's storytelling contents and general knowledge. The information analyzing the user's storytelling content interaction information is stored in a user database (not shown).

8 is a diagram illustrating an example of projecting or inserting a user's facial model analyzed according to an embodiment of the present invention on a content by modifying the facial model of the user according to an environment such as color and style of the content.

8A, the system analyzes the face characteristics of the user in three dimensions or two dimensions according to an embodiment of the present invention, as shown in Fig. 8B, As shown in FIG. 8 (c), the analyzed facial characteristics can be projected or inserted onto the contents by modifying the facial characteristics corresponding to the environment of the contents such as the color and style of the contents.

On the other hand, restoring the face shown in the image to the 3D face model is to estimate the face identity value (w _id ) and the facial expression value (w _exp ). The estimated facial identity value (w _id ) is represented by a consecutive numerical value of the characteristics of the face (wide face, long face, face with high nose height, low face, long face with long face, etc.) . At this time, it is possible to emphasize the characteristics of the reconstructed face model by increasing or decreasing the conspicuous value (for example, very large or very small value) among the numerical values (for example, People can be shorter). Accordingly, the user's facial characteristics can be emphasized and modified to match the color and style of the contents according to the above process. That is, the feature values of the estimated face identity value (w _id ) and the facial expression value (w _exp ) can be emphasized and reflected in the 3D face model.

Meanwhile, in another embodiment of the present invention, facial characteristics that are already drawn (existing) on the content may be analyzed and the facial characteristics of the user may be changed to reflect the facial characteristics already drawn on the content. Therefore, in the restored face information of the user, it is possible not to emphasize the facial characteristic element of the prominent user, but to emphasize the facial characteristic element of the face that is already drawn (existing) on the contents to the restored face information of the user. For example, if the face of the user is a round face but the face existing on the fairytale content is a long face, the face shape of the restored user is reflected for a long time, This is because it is better to transfer to the top.

FIG. 10 is a diagram illustrating creation of a three-dimensional character face that has been animated according to an embodiment of the present invention.

FIG. 10 (a) illustrates an input face image of a user, FIG. 10 (b) illustrates a reconstructed three-dimensional face image from a face image of an input user, FIG. 10 (d) illustrates an example in which the face characteristic is emphasized in the analysis restored face image, and FIG. 10 (e) This is an illustration that the facial expression is processed and transformed in the image and reflected on the content.

It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. It is intended that the present invention cover the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents.

110: Face value storage model
120: Storytelling content storage unit
130: User face image generation unit
140: user facial analysis unit
150: Storytelling Content Progressive Department

Claims (16)

A storytelling contents storage unit having storytelling contents used in storytelling;
A user facial image generating unit for capturing a face of a user enjoying the storytelling contents to generate a user facial image;
A face reference value storage model in which a face identity representing a face shape of each reference face is represented by successive numerical values and a face expression value representing a facial expression value expressed by successive numerical values of each reference face is stored;
A user facial analysis unit for estimating a face identity value and a facial expression value matching with the user facial image based on the face reference value storage model and determining the user facial identity value and the user facial expression value; And
And a storytelling content advancing unit for processing the user's face image by reflecting the user's face identity value and the user's facial expression value, and transmitting the processed user's face image as a storytelling content,
The user facial analysis unit analyzes the skin color distribution of the user's face using the pixel values of the user facial image and transforms the skin color distribution and texture of the user's face in accordance with the color, ,
Wherein the storytelling content progress unit reflects the skin color distribution and texture of the modified user's face on the processed user face image to output the storytelling content.
The contents processing system according to claim 1,
And outputting the user facial identity value and the user facial expression value to the facial model of the storytelling contents, and reflecting the result.
delete The contents processing system according to claim 1,
And outputting the modified subtitle contents according to the user's personal information.
The storytelling contents tool providing terminal of claim 1,
And a third person's face information model for receiving face information of a person other than the user who enjoys the storytelling contents from the storytelling contents server and storing the received face information as third person's face information,
Wherein the storytelling content progressing unit outputs the other's face model reflecting the other's face information to the storytelling contents together with the processed user's face image.
The storytelling contents tool providing terminal of claim 1,
And a conversion output unit for converting the outputted storytelling contents into a printable document file.
(a) a step of photographing a face of a user enjoying the storytelling contents to generate a user face image;
(b) determining a face facial expression value of a user expressing the user facial expression as a continuous numerical value by matching the user facial image and the face reference value storage model to represent the user facial shape as a continuous numerical value, ; And
(c) processing the user's face image reflecting the user's face identity value and the user's facial expression value, and transmitting the processed user's face image as a storytelling content,
The step (c)
(c1) determining a skin color distribution of the user's face using the pixel value of the user's face image, and transforming the skin color distribution and texture of the user's face according to the color, style, and texture of the storytelling content being output; And
(c2) reflecting the skin color distribution and texture of the deformed user's face on the processed user face image to output the story telling contents,
The face reference value storage model stores a face identity value obtained by expressing the face shape of each reference face as a continuous numerical value and a facial expression value representing a facial expression value expressed by successive numerical values of the facial expression of each reference face Wherein the story-telling content providing method comprises the steps of:
The method of claim 7,
The face reference value storage model includes at least one of Principal Component Analysis (PCA), Singular Value Decomposition (SVD), and High Order Singular Value Decomposition (HOSVD) from face image data of various people photographed for each face shape and facial expression Wherein the facial information of the data is constructed by elementizing the facial information of the data by the continuous numerical value of the facial identity value and the facial expression value through the data analysis method
The method of claim 7, wherein the step (c)
Reflecting the user facial identity value and the user facial expression value to the 3D facial model or reflecting the user facial identity value and the user facial expression value to the 2D facial model to output to the storytelling contents.
The method of claim 9, wherein reflecting the user facial identity value and the user facial expression value to the 3D facial model comprises:
V is a 3D face model, C is when the core structure model learning facial feature distribution from the face reference value storage model, w _id is user face identity value, w _exp user facial expression value, V = g (C, w id , w _exp ) function.
11. The method of claim 10, wherein reflecting the user facial identity value and the user facial expression value to the 3D facial model comprises:
_Wherein the characteristic values of the face identity value (w _id ) and the face expression value (w _exp ) are emphasized and reflected in the 3D face model.
11. The method of claim 10, wherein the step (c)
In the case of outputting a 3D face model (V) reflecting the function V = g (C, w _id , w _exp ) on the screen, F is a model in which a 3D face model is projected on a two- _(F) is a camera function for mapping the 3D position to the 2D position on the screen, R is the face rotation information for rotating the face, and T is the face position information for moving the position of the face time,

And outputting the model calculated by the modeling unit to the screen.
The method of claim 9, wherein reflecting the user facial identity value and the user facial expression value to the 2D facial model comprises:
Wherein the facial identity value and the facial expression value analyzed in the user facial image are reflected in the two-dimensionally projected face reference value storage model.
8. The method of claim 7, wherein, prior to step (c)
A step of re-processing the user's face information which exaggerates the characteristics of the user's facial information derived from the facial identity value and the facial expression determination process or re-processes the texture information including the skin information of the user's face information to match the texture of the story telling contents. How to provide teled content.
The method according to claim 7,
And converting the outputted storytelling contents into a printable document file.
The method of claim 7, wherein the step (c)
m represents the face identity value (w _id ), the facial expression value (w _exp ), the size value (s) in the face size information, the translation value (t _x , t _y ) Is a set of components including values r _x , r _y , r _z , and camera focal length f,
F is a model projected on a two-dimensional image generated from the elements of m, I is an input image, V is a 3D face model _elementized by a face identity value (w _id ) and a facial expression value (w _exp ), x _2d and x _3d is a function for extracting the positions of predetermined feature points in the input model, A is a texture learning model of the 3D face model,

Is a feature extraction operator that extracts features distinguished from input information,

Wherein the user's face identification value and the user's facial expression value are determined by the user's face identification value.

Images