KR100229538B1 - Apparatus and method for encoding a facial movement - Google Patents

Abstract

The present invention relates to a method for encoding a facial motion of a new face image based on a speech signal and a two-dimensional (2D) image signal in a three-dimensional (3D) model-based coding system. The adaptive 3D model is generated from the initial data of the new face based on the basic 3D model of the human common face, and the basic pattern for the 2D image signal is based on the rotational correlation between the 2D image signal and the adaptive 3D model. Is generated. One or more feature areas of the new face from the 2D image signal are compared with the base pattern to detect a number of conversion parameters. The conversion parameters are changed based on the speech signal.

Description

Method and apparatus for encoding facial movements {APPARATUS AND METHOD FOR ENCODING A FACIAL MOVEMENT}

The present invention relates to a method and apparatus for encoding a moving object, and more particularly, to a method and apparatus for encoding and decoding facial motion using a three-dimensional face model.

In digital broadcasting systems such as video telephony, teleconferencing and high definition television systems, the video line signal within the video frame signal includes a sequence of digital data referred to as pixel values, so that each video Defining a frame signal requires a large amount of digital data. However, the available frequency bandwidth of a typical transmission channel is limited, so that the transmission of large amounts of digital data can lead to various data compression techniques, especially video telephony and teleconference systems employed to transmit human shapes. It is essential to compress or reduce the amount of data through the low bit rate video signal encoder used.

Typically, a video coding system transmits images that consist of continuously changing pixels. However, in a 3D model-based coding system, a specific motion parameter is extracted from an image and the extracted motion parameter is transmitted to a receiver. At the receiving end, in order to reconstruct the images, for example facial images, the received motion parameter is combined with data such as a general three-dimensional model of the basic facial shape and head of the person previously sent to the receiving end.

In video telephony and teleconferencing systems, video images are mainly comprised of head-and-shoulder shots, i.e., the upper body of the person. In addition, the most interesting object for the viewer is the person's face, so that the viewer includes moving parts such as lips, chins, heads, etc., especially if the person is talking on the video screen, which is not a wallpaper or other details. Note the mouth area of the person. Therefore, if only general information about the shape of the face is to be transmitted, the amount of digital data can be significantly reduced.

It is therefore an object of the present invention to provide a method and apparatus for reducing the amount of transmitted data by encoding and decoding facial motion using a three-dimensional face model.

According to the present invention for achieving the above object, there is provided a method for encoding a face motion of a new face image based on a speech signal and a two-dimensional (2D) image signal in a three-dimensional (3D) model-based coding system . The method of the present invention described above comprises the steps of (a) generating an adaptive 3D model from initial data representing one or more 2D face images of a new face image based on a base 3D model representing a 3D model of a human common face; b) generating a basic pattern for the 2D image signal based on the adaptive 3D model, representing a 2D picture obtained by rotational correlation between the 2D image signal and the adaptive 3D model; c) extracting one or more feature regions from which the multiple transformations of the new face image occur from the 2D image signal; and (d) detecting the plurality of transformation parameters indicative of a comparison result. Comparing basic patterns, and (e) converting the conversion based on the speech signal to generate modified conversion parameters. Changing the parameters, and (f) encoding the initial data and the modified transformation parameters.

1 is a block diagram of an apparatus for encoding facial motion according to the present invention;

2 is a block diagram of an apparatus for decoding facial motion according to the present invention;

3A shows a number of eye and eyebrow parameters in accordance with the present invention;

3b illustrates a plurality of mouth parameters in accordance with the present invention;

3c shows three jaw parameters in accordance with the present invention;

3d shows three head parameters according to the invention.

<Explanation of symbols for main parts of drawing>

10: adaptive three-dimensional model unit 12: encoder

14: basic 3D model section 16: head parameters

18: basic pattern generator 20: feature extraction unit

30: speech analyzer 36: formatter

38: buffer

DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail as follows with reference to the accompanying drawings, in which an object of the input image is a human face and a predetermined feature portion of the face image to be encoded. Are assumed to be parts of the head, mouth, chin and eyes.

Referring to FIG. 1, shown is a block diagram of an apparatus 100 for encoding face motion classified based on a method that will be described in detail below in accordance with an embodiment of the present invention.

For convenience of explanation, it is assumed that the human body is divided into an upper body including a waist and a lower body below the waist. The upper body is again divided into a head, a torso and an arm. The head is separated into eyes, nose, mouth and ears. If eye, nose, mouth, and ear are considered as the base pattern, the tissue system for the human body may be constructed based on transformation parameters indicating that the base patterns and the transformation of the base patterns can be extracted. The conversion parameters and data structure for the head are described below.

The basic patterns of the head are divided into two categories. The first category corresponds to areas in which conversion of the basic patterns occurs frequently, and the second category corresponds to areas in which conversion of the basic patterns occurs very rarely. The first category corresponds to areas of the eye, eyebrow mouth, chin, cheeks, bangs, and the second category corresponds to areas of hair, nose, ear, and the like. The basic patterns used to extract the transform parameters correspond to active motion areas. Thus, the selected conversion parameters include parameters for the eye, eyebrows, mouth, chin and head, which are described in detail below. The wrinkles of the bangs and cheeks are passively moved according to the movement of the eyebrows and chin respectively.

1) Eyebrow: As shown in FIG. 3A, the eyebrow is divided into left and right eyebrows, and the left and right eyebrow conversion parameters are inner eyebrow up and down motion parameters EB1 and EB2 and eyebrow left and right motion parameters EB3, respectively. , EB4) and outer eyebrow up and down motion parameters EB5 and EB6.

2) Eye: As shown in FIG. 3A, the left and right eye conversion parameters are eyelid up and down motion parameters EL1 and EL2, and pupil up and down motion parameters E1 and E2 and pupil left and right motion parameters E3, respectively. , E4).

3) Mouth: As shown in FIG. 3B, mouth movement depends on lip movement. The mouth transformation parameters include the left and right motion parameters L1 and L2 of both end points of the lips, the up and down motion parameters L3 and L4 at the top and bottom of the center area of the lips, and the top and bottom of the center area of the lips. Forward and backward motion parameters L5 and L6 and up and down motion parameters L7 and L8 of the endpoints of the lip.

4) Jaw: As shown in FIG. 3C, the jaw conversion parameters include an up and down motion parameter C1, a left and right motion parameter C2, and a forward and backward motion parameter C3.

5) Head: As shown in FIG. 3D, the reference of the coordinate system is a three-dimensional coordinate system located near the first cervical spine, a face plane which is an imaginary plane perpendicular to the x axis and parallel to the face, and z passing through the center of the head of the head. An axis and a y axis perpendicular to both the x and z axes. In other words, each of the x, y, z axes is parallel to the direction of movement of the parameters L5, L1, L3 shown in FIG. 3B. The head transformation parameters are a yawing parameter H1 representing left and right rotation around the z axis, a pitching parameter H2 representing up and down rotation around the y axis, and a rolling representing left and right tilt around the x axis. rolling) parameter H3.

These parameters have a value of zero at individual base locations on the base pattern.

The conversion parameters of each basic pattern, marked as independent items, are stored and transmitted in the data format given below.

Item (TITLE) CODE Number of bits START CODEHEAD ORIENTATION BITHEAD ORIENTATION ITEMShead_orientation_items [0] head_orientation_items [1] head_orientation_items [2] EYEBROW TRANSFORMATION BITLEFT-RIGHT EYEBROW ITEMSLEFT EYEBROW ITEMSlefteyebrow_eyems_browser_browser_browser_items_items_items ] EYE TRANSFORMATION BIT headhead_orientation_bithead_orientation_itemsH1H2H3eyebrow_biteyebrowslefteyebrow_itemsEB1EB3EB5righteyebrow_itemsEB2EB4EB6eye_bit 31387512333333331

Item (TITLE) CODE Number of bits MOUTH TRANSFORMATION BITTRANSFORMATION SELECTION BITCHIN TRANSFORMATION BITFACE TEXTURE BIT E1E2E3E4EL1EL2mouth_bitspeech_bitL1L2L3L4L5L6L7L8soundpaceaccentchin_bitC1C2C3face_texture_bitface_data 333333113333333384314331VLB

Hereinafter, each item of Table 1 is demonstrated.

1. START CODE (head): 3-bit code indicating the start of head data. For example, it is set to "001". If the start code is not "001", no head data follows.

2. HEAD ORIENTATION BIT (head_orientation_bit): Indicates whether the head is rotated by 1 bit code. Bit value 1 indicates that the head is to be rotated and followed by the head rotation parameter. Bit value 0 indicates that the head is not rotated.

1) HEAD ORIENTATION ITEMS (head_orientation_items): 3-bit code indicates the direction in which the head is rotated. Each three bit code indicates the presence of a corresponding direction item. Bit value 1 indicates that the head is rotated in its corresponding direction. For example, the bit value " 110 " indicates that yawing of the head, i.e., left and right rotation and pitching, i.e., vertical rotation, occurs.

a) head_orientation_items [0]: The 8-bit head yawing parameter H1 represents an integer value given in 181 steps from -90 degrees to 90 degrees.

b) head_orientation_items [1]: The 7-bit head pitching parameter H2 represents an integer value given in 121 steps from -60 degrees to 60 degrees.

c) head_orientation_items [2]: The 5-bit head rolling parameter H3 represents an integer value given in 31 steps from -15 degrees to 15 degrees.

3. EYEBROW TRANSFORMATION BIT (eyebrow_bit): This 1-bit code indicates whether the eyebrows are moving. Bit value 1 indicates that the eyebrows are moving and bit value 0 means that the eyebrows are not moving.

1) LEFT-RIGHT EYEBROW ITEMS (eyebrows): It shows the movement of the eye with 2 bit code.

a) 00: Both eyebrows do not move.

b) 01: The left eyebrow moves.

c) 10: Right eyebrow movement.

d) 11: both eyebrows move.

2) LEFT EYEBROW ITEMS (lefteyebrow_items): 3-bit code that indicates the direction of left eyebrow movement. A three bit code indicates the presence of three motion parameters described below.

a) lefteyebrow_items [0]: Code value 1 of lefteyebrow_items indicates that the inner left eyebrow moves up and down. The 3-bit inner left eyebrow up-and-down motion parameter EB1 is given in seven steps from -1.0 to 1.0.

step One 2 3 4 5 6 7 Weight (W) -1.0 -0.6 -0.3 0.0 0.3 0.6 1.0

Typically, the fourth stage with little motion and the two end stages, the first and seventh stages, are given in predetermined three-dimensional absolute coordinates, and the positions of the remaining stages are preset weights as illustrated in Table 2 above. Computed by applying factors. Coordinates of the second, third, fifth, and sixth steps may be calculated as in the following equation.

First, the second and third steps are calculated as in Equation 1 below.

The fifth and sixth steps are calculated as in Equation 2 below.

In the above equation, x (j), y (j), z (j) represent the x, y, z coordinates for the jth step, w (j) is the preset weight factor for the jth step, x (Step i), y (Step i) and z (Step i) are the x, y, z coordinates in the i-th step.

b) lefteyebrow_items [1]: Bit value 1 indicates that the left eyebrow moves left or right. The 3-bit left eyebrow left and right movement parameter (EB3) is given in seven steps from -1.0 to 1.0. The location of the steps is determined in the same way as in EB1.

c) lefteyebrow_items [2]: Bit value 1 indicates that the outer left eyebrow moves up and down. The left eyebrow up and down motion parameter EB5 of 3 bits is given in seven steps from -1.0 to 1.0. The weight used in (EB1) is applied to (EB5) and the position of the steps is determined in the same way as in EB1.

3) RIGHT EYEBROW ITEMS (righteyebrow_items): A 1-bit code indicates the direction of movement of the right eyebrow. The functions EB2, EB4, EB6 of the right eyebrow transformation parameters are the same as the functions EB1, EB3, EB5 of the left eyebrow transformation parameters.

4. EYE TRANSFORMATION BIT (eye_bit): Indicates whether the eye moves with a 1-bit code. Bit value 1 indicates that the eye is moving and bit value 0 indicates that the eye is not moving.

1) PUPIL UP-DOWN MOVEMENT PARAMETERS (E1, E2): (E1) and (E2) represent the up and down movement of the left and right eyes, respectively. (E1) and (E2) each have seven steps. Typically, the fourth stage with almost no movement and the first and seventh stages, both end stages, are given in predetermined three-dimensional absolute coordinates, and the positions of the remaining stages are calculated as in the case of EB1.

2) PUPIL LEFT-RIGHT MOVEMENT PARAMETERS (E3, E4): (E3) and (E4) represent left and right eye movements, respectively. (E3) and (E4) each have seven steps, and the positions of the steps are calculated as in the case of E1 and E2.

3) OUTER EYELID UP-DOWN MOVEMENT PARAMETERS (EL1, EL2): (EL1) and (EL2) represent the vertical movement of the left and right eyelids, respectively. (EL1) and (EL2) each have seven steps, and the positions of the steps are calculated as in the case of E1 and E2.

5. MOUTH TRANSFORMATION BIT (mouth_bit): Indicates whether the mouth shape changes with a 1-bit code. Bit value 1 indicates that the mouth shape changes, and bit value 0 indicates that the mouth shape does not change.

1) TRANSFORMATION SELECTION BIT (speech_bit): Indicates that the transformation parameter of the mouth is selected as a 1-bit code. The shape of the lip is classified into two types: a person speaking and a person expressing emotion. Typically, the shape of the lips uses features of spoken sound, pace, and accents, since the shape of the lips depends heavily on a pronounced sound when a person is speaking. Can be configured. However, when a person expresses emotions, the shape of the lips does not have any feature. Thus, the shape of the lips is determined by all the transformation parameters, i.e., the left and right motion parameters L1 and L2 of both endpoints of the lips shown in FIG. 3B, and the up and down motion parameters L3 at the top and bottom of the central area of the lips. , L4), and the forward and backward motion parameters L5 and L6 at the top and bottom of the central region of the lip and the up and down motion parameters L7 and L8 of the endpoints of the lip. Bit value 1 represents the case in which the person is speaking, followed by pronunciation, speed, and accent code. If the code value is zero, the L1 through L8 codes follow. Each of the three bits, L1 to L8 motion parameters, has seven steps each. The position of the steps is calculated based on the scheme described above for the motion parameters EB1. The uttered sound, speed and accent characteristics of the voice are represented by 8 bits, 4 bits and 3 bits, respectively.

6. CHIN TRANSFORMATION BIT (chin_bit): It indicates whether the jaw moves by 1 bit code. Bit value 1 indicates that the jaw is moving and bit value 0 indicates that the jaw is not moving.

1) CHIN UP-DOWN MOVEMENT PARAMETER (C1): 4-bit Jaw Up / Down Movement Parameter (C1) shows the displacement of jaw from the time the mouth is closed and is given in 16 steps. That is, the 0th stage represents the multilingual mouth and the 15th stage represents the largest open mouth. The jaw position for the 16 steps is calculated in the same way as in EB1.

2) CHIN LEFT-RIGHT MOVEMENT PARAMETER (C2): 3-bit jaw left and right movement parameter (C2) represents the left and right movement of the jaw, which is given in three steps in the left direction and three steps in the right direction from the reference in the center area. . The position of the steps is calculated as in the case of EB1.

3) CHIN FORWARD-BACKWARD MOVEMENT PARAMETER (C3): The 3-bit jaw forward and backward movement parameter (C3) indicates the jaw moved forward and backward, which is given three steps forward and three steps backward from the reference in the center area. The position of the steps is calculated as in the case of EB1.

7. FACE TEXTURE BIT (face_texture_bit): When a new face image is sent, the code is set to one.

1) FACE DATA (face_data): This represents the compressed base face image of the new face image and its length is variable.

Referring again to FIG. 1, initial data is applied to adaptive three-dimensional (3D) model block 10 and encoder 12, where the initial data is one or more faces that are expressionless and silent face images of new face images displayed on the screen. Represent one or more still pictures of the images, ie, the new face image. Encoder 12 encodes the initial data of the 2D face images according to conventional encoding principles and provides the encoded face image as face_data to formatter 36.

On the other hand, the basic 3D model representing the 3D model of the human common face stored in the basic 3D model block 14 is applied to the adaptive 3D model block 10. The adaptive 3D model block 10 generates an adaptive 3D model similar to the new face image by changing the 2D initial data based on the basic 3D model, so that the adaptive 3D model is converted into the head parameter block 16 and the basic pattern generation block 18. To provide.

On the other hand, the image signal of the new face is provided to the head parameter block 16 and the feature extraction block 20 from the camera (not shown), and the voice signals of the new face image are received from the microphone (not shown) and the voice analyzer 30. ), Where the input image and voice signal of the new face are continuously input in one of frame units or field units.

Initially, the head parameter block 16 applies conventional affine transform discipline to the adaptive 3D model of the new face image, so that the head yawing, pitching and Rolling parameters H1 to H3 are detected. Head yawing, pitching and rolling parameters H1 to H3 are provided to the basic pattern generation block 18 and the formatter 36. The base pattern generation block 18 generates a base pattern of the new face image, where the base pattern is a 2D of the new face projected onto the screen obtained by rotating an adaptive 3D model for yawing, pitching and rolling parameters. Represents an adaptive image and indexes the left and right brows and the left and right eyes in the basic pattern to generate basic eyebrow, eye, mouth and chin patterns. The basic pattern generation block 18 provides indexed eyebrows, eyes, mouths, and jaws to the eyebrow extraction block 22, eye extraction block 24, mouth 1 extraction block 26, and jaw 1 extraction block 28, respectively. do.

On the other hand, feature extraction block 20 detects the edges of the predetermined feature areas from image signals of the new face using a conventional edge detector such as a sobel operator. The feature areas here include the left and right eyebrows, the left and right eyebrows, the mouth and the chin of the new face, and outline information indicating the shape and location of each feature area, namely the left and right eyebrows, the left and right eye, the mouth and The jaws are provided to the eyebrow extraction block 22, the eye extraction block 24, the mouth 1 extraction block 26 and the jaw 1 extraction block 28, respectively.

The eyebrow extraction block 22 detects the movement of the left and right eyebrows based on the basic eyebrow pattern input from the basic pattern generation block 18. If there is movement of the left and right eyebrows, the left and right eyebrow transformation parameters E1 to E6 are each computed by 3 bits. In the eyebrow extraction block 22, a 3-bit lefteyebrow_items signal is generated indicating that the left eyebrow transformation parameters E1, E3, E5 will be encoded, and a 3-bit righteyebrow_items signal indicating that the right eyebrow transformation parameters E2, E4, E6 will be encoded. do. The 2-bit eyebrow signal, which indicates that the eyebrows are moving, is generated based on the lefteyebrow_items and righteyebrow_items signals. The eyebrow data is provided continuously to the formatter 36 in the data format given above, where the eyebrow data, if present, is the eyebrow signal, the lefteyebrow_items signal, the left eyebrow conversion parameters EB1, EB3, EB5, the righteyebrow_items signal, and the right eyebrow. Conversion parameters EB2, EB4, EB6.

The eye extraction block 24 detects the movement of the left and right eyes based on the basic eye input from the basic pattern generation block 18, and based on the movement of the left and right eyes, the pupil up and down motion parameters E1 and E2, Generates pupil left and right motion parameters E3 and E4 and outer eyelid up and down motion parameters EL1 and EL2. The eye extraction block 24 provides the formatter 36 with eye data including eye up and down motion parameters E1 and E2, eye left and down motion parameters E3 and E4 and outer eyelid up and down motion parameters EL1 and EL2. do.

The mouth 1 extraction block 26 detects the movement of the mouth during the emotional expression of the new face based on the basic mouth pattern input from the basic pattern generation block 18 and generates mouth conversion parameters L1 to L8. The input conversion parameters L1 through L8 are provided to the formatter 36.

The jaw 1 extraction block 28 detects the movement of the jaw during the expression of the new face based on the basic jaw pattern input from the base pattern generation block 18 and generates the jaw transformation parameters C1 to C3. Tuck conversion parameters C1 to C3 are provided to the formatter 36.

On the other hand, the speech analyzer 30 compares the speech signal with a preset threshold to determine whether a new face image is speaking or expressing emotions. A speech_bit signal is provided to the formatter 36 indicating whether a new face image is speaking for communication.

If a new face image is talking for communication, then the spoken sounds, pace and accents are sent to the mouth 2 extraction block 32 and the jaw 2 extraction block 34 to provide speech signals. Is extracted from.

The mouth 2 extraction block 32 generates an 8-bit sound parameter, a 4-bit speed parameter and a 3-bit accent parameter for each of the sound, speed and accent spoken to determine the shape of the mouth, the sound parameter, velocity parameter and The accent parameter is provided to the formatter 36. If necessary, three tuck conversion parameters C1 to C3 are generated in the tuck 2 extraction block 24 to be provided to the formatter 36.

Each time a new face image is transmitted, the formatter 36 generates one bit of face_texture_bit indicating that the face_data of the new face is followed. The formatter 36 also generates a three bit start code signal, one bit head_orientation_bit signal, one bit eyebrow_bit signal, one bit eye_bit signal, one bit mouth_bit signal and one bit chin_bit signal. Here, the 1-bit eyebrow_bit signal generated based on the eyebrow signal indicates whether there is any movement in the left and right eyebrows, and the 1-bit eye_bit signal generated based on the parameters E1 to E4, EL1, and EL2 is used for the left or right eye. Indicates whether there is any movement, and the mouth_bit signal generated based on the mouth conversion parameters L1 to L8 or the sound, speed and accent parameters indicates whether there is any movement in the mouth, and the jaw conversion parameters C1 to C3. The 1-bit chin_bit signal generated based on indicates whether there is any movement in the jaw. The formatter 36 multiplexes all signals and parameters according to the data format given above and provides them to a buffer 38 that stores the multiplexed signals and data. Data stored in the buffer 38 is transmitted through a transmission transmitter (not shown).

2, a block diagram of an apparatus 200 for decoding facial movements in accordance with the present invention is shown, wherein the transmitted data is temporarily stored in a buffer 50 to provide an initial data decoder 52. And to the parameter decoder 54.

The initial data decoder 52 decodes face_data from the transmitted data and provides 2D initial data of the new face image to the adaptive 3D model block in the adaptive 3D model generation block 57.

The adaptive 3D model block 58 generates an adaptive 3D model similar to the new face image by changing the 2D initial data based on the basic 3D model input from the basic 3D model block 60 in the adaptive 3D model generation block 57. Here, the basic 3D model is the same as the basic 3D model of the encoding apparatus 100. The adaptive 3D model generated in adaptive 3D model block 58 is provided to pattern generation block 62.

Meanwhile, the parameter decoder 54 decodes all transmitted data except for face_data and generates all conversion parameters. The conversion parameters generated at this time are head transformation parameters H1 to H3, left and right eyebrow transformation parameters EB1 to EB6, left and right eye transformation parameters E1 to E4, EL1 and EL2, mouth transformation parameters L1 to L8 or sound, Velocity and accent parameters, tuck transformation parameters C1 to C3. The transform parameters generated at the parameter decoder 54 are input to a reconstruction block 63 consisting of an eyebrow reconstruction block 64, an eye reconstruction block 66, a mouth reconstruction block 68, and a jaw reconstruction block 70.

More specifically, hair transformation parameters H1 to H3 are provided to pattern generation block 62 via line L62, and left and right eyebrow transformation parameters EB1 to EB6 are provided to eyebrow reconstruction block 64, left and right. Eye conversion parameters E1 to E4, EL1 and EL2 are provided to the eye reconstruction block 66, and mouth conversion parameters L1 to L8 or one of the sound, speed and accent parameters are provided to the mouth reconstruction block 68, Jaw conversion parameters C1 through C3 are provided to the jaw reconstruction block 70.

Initially, pattern generation block 62 provides the basic pattern of the new face to image reconstruction block 56, where the basic pattern is adaptable to the head transformation parameters, i.e. yawing, pitching and rolling parameters H1 to H3. Represents a 2D adaptive image obtained by rotating and projecting a 3D model onto a screen, and the left and right eyebrows, the left and right eyes, the mouth and the chin in the basic pattern are indexed, resulting in basic eyebrow, eye, mouth and chin patterns . The pattern generation block 62 provides the indexed eyebrows, eyes, mouth and chin to the eyebrow reconstruction block 64, the eye reconstruction block 66, the mouth reconstruction block 68 and the jaw reconstruction block 70, respectively.

The eyebrow reconstruction block 64 reconstructs the left and right brows from the indexed eyebrows based on the left and right eyebrow transformation parameters EB1 through EB6 to provide this reconstructed left and right eyebrows to the image reconstruction block 56.

The eye reconstruction block 66 reconstructs the left and right eyes from the indexed eye based on the left and right eye transformation parameters E1 to E4, EL1 and EL2, thereby reconstructing the reconstructed left and right eyes into an image reconstruction block 56. To provide.

The mouth reconstruction block 68 reconstructs the mouth from the indexed mouth based on the mouth transformation parameters L1 to L8 or one of the sound, speed and accent parameters, and provides the reconstructed mouth to the image reconstruction block 56.

The jaw reconstruction block 70 reconstructs the jaw from the jaws indexed based on the jaw transformation parameters C1 through C3 to provide this reconstructed jaw to the image reconstruction block 56.

The image reconstruction block 56 replaces the eyebrows, eyes, mouths, and jaws in the basic pattern input from the pattern generation block 62 with the eyebrows, eyes, mouths, and jaws input from the reconstruction block 63, and then frames or fields. Reconstruct a new image of a new face with one of them.

While the invention has been described with respect to particular embodiments, it will be apparent to those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention as defined in the following claims.

Therefore, according to the present invention, it is possible to reduce the amount of transmission data by transmitting general information about a face shape including a sender's mouth area using a three-dimensional face model in a videophone and a teleconference system.

Claims (10)

Apparatus for encoding facial movement of a new face image based on a speech signal and a two-dimensional (2D) image signal of a new face image selectively provided in units of frames or fields in a 3D model-based coding system To When initial data represents one or more 2D face images of a new face and a basic 3D model represents a 3D model of a common face of a human, an initial 3D model for generating an adaptive 3D model from the initial data of the new face based on the basic 3D model Adaptive 3D model generator; A basic pattern generator for generating a basic pattern representing a 2D picture obtained by a rotation correlation between the 2D image signal and the adaptive 3D model, for the 2D image based on the adaptive 3D model; A feature extractor for extracting from the 2D image signal one or more feature regions of the new face representing one or more regions where many transformations occur; A parameter generator for comparing the feature areas with the basic pattern to detect a plurality of conversion parameters representing a comparison result; A speech analyzer for generating a sound_bit signal representing one of said new face speaking or expressing emotion based on said speech signal; A parameter modulator for modulating conversion parameters based on the speech signal to generate modified conversion parameters in response to the sound bit; And a formatter for encoding the initial data and the transformation parameters or the modified transformation parameters. The method of claim 1, wherein the basic pattern generator, Means for determining, for the 2D image signal based on the adaptive 3D model, head parameters indicative of rotational conditions for the projected image of the adaptive 3D model to be similar to the 2D image signal; Means for replacing the basic pattern with the projection image corresponding to the head parameters. The method of claim 1, wherein the parameter generator, Means for matching the feature areas with the basic pattern to calculate an amount of motion for each of the feature areas; Means for storing the respective motion amounts in corresponding conversion parameters. 4. The method of claim 3, wherein the feature regions comprise left and right eyebrows, left and right eyes, mouth and chin, the transformation parameters comprising eyebrow transformation parameters, eye transformation parameters, mouth transformation parameters and jaw transformation parameters. Facial motion encoding device comprising. The method of claim 4, wherein the voice analyzer, Means for obtaining sound, speed, and accent spoken from the speech signal; Means for comparing each of said spoken sound, velocity, and accents with a predetermined threshold to generate said sound_bit signal. 6. The apparatus of claim 5, wherein in response to the sound_bit signal, the mouth and jaw transformation parameters are changed based on the speech signal to generate a changed mouth transformation parameter and a changed jaw transformation parameter, respectively. A method for encoding facial motion of a new face image based on a speech signal and a two-dimensional (2D) image signal of a new face image selectively provided on a frame or field basis in a 3D model-based coding system To (a) generating an adaptive 3D model from initial data representing one or more 2D face images of a new face image based on a base 3D model representing a 3D model of a human common face; (b) generating a basic pattern representing the 2D picture obtained by the rotation correlation between the 2D image signal and the adaptive 3D model for the 2D image signal based on the adaptive 3D model; (c) extracting one or more feature regions from which the multiple transformations of the new face image occur from the 2D image signal; (d) comparing the feature areas with the basic pattern to detect a plurality of conversion parameters indicative of a comparison result; (e) changing the conversion parameters based on the speech signal to generate modified conversion parameters; (f) encoding said initial data and said modified transformation parameters. The method of claim 7, wherein step (b) (b1) determining head parameters indicative of a rotation condition for an image of the adaptive 3D model that is similar to a 2D image signal based on the adaptive 3D model; (b2) replacing the basic pattern with the projection image corresponding to the head parameters. The method of claim 7, wherein step (d) is (d1) matching the feature areas with the basic pattern to calculate an amount of motion for each of the feature areas; (d2) storing the respective motion amounts in corresponding conversion parameters. The method of claim 7, wherein step (e) (e1) obtaining sound, speed and accent spoken from the speech signal; (e2) comparing a spoken sound, speed and accent with a preset threshold to generate a sound_bit signal indicative of either said new face image speaking or expressing emotion; (e3) if the new face image is determined to speak, modulating mouth conversion parameters based on the spoken sound, speed, and accent to generate modified mouth conversion parameters. Motion encoding method.