JP2008015713A - Motion deformation system and method for it - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a deformation data extraction method for extracting deformation data allowing deformation of an input motion style for general use from a plurality of teacher motions. <P>SOLUTION: The motion deformation system comprises an extraction means 21 extracting deformation data serving as a differential from a reference motion and a teacher motion, a means receiving input of a motion style of the teacher motion from a user, a means 22 storing the deformation data extracted by the extraction means and the motion style of the teacher motion inputted by the user in association with each other, a means receiving input of a motion style, which is applied to the input motion, from the user, a retrieval means 31 retrieving the deformation data based on the motion style inputted by the user according to the stored motion style of the teacher motion and the deformation data, an application means 32 applying the deformation data retrieved by the retrieval means to the input motion, and an output means outputting the output data generated by application. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a deformed data generation method for generating deformed data for applying an input operation to an output operation in which a style is deformed, and the method.

  There is a great demand for transformation of human motion data. If the operation data can be modified to match the situation, the existing operation data can be effectively reused.

An animation system generally used at present is provided with a motion editing interface for changing motion data in a direct manner. By using such an interface, the animator performs direct deformation such as changing the joint angle in the key frame, changing the position of the hand or toe, or adjusting the speed of movement. be able to.
Rose C, Cohen MF, Bodenheimer B. Verbs and Adverbs: Multidimensional motion interpolation.IEEE Computer Graphics and Applications. 18 (5), 32-40, 1998.

  However, when the animator tries to transform the motion data, in general, it is often desired to change the motion style. For example, when there is a certain walking motion, there is a case where it is desired to make the motion look like an enjoyable walking motion, a tired walking motion, or an excited walking motion. It is difficult to realize such a modification of the operation style with an existing system as described in the background art. This is because an animator needs to repeatedly perform detailed direct corrections such as correction of joint angles, positions of limbs, and movement speeds in order to realize deformation of movement styles using an existing system. Furthermore, in order to realize such deformation, it is necessary to have knowledge about human movements, such as how to change the movements so that the movements are fun, tired, and excited. It becomes. Therefore, such deformation of the operation style is extremely difficult and requires time.

  Therefore, if it is possible to extract from the existing motion data knowledge about how to change the motion data to change the motion data style, it is possible to easily realize the motion deformation. It is considered to be. However, with the existing technology, such deformation of operation is difficult.

  As a representative method for generating new motion based on existing motion data, motion blending [Rose C, Cohen MF, Bodenheimer B. Verbs and Adverbs: Multidimensional motion interpolation. IEEE Computer Graphics and Applications. 18 (5 ), 32-40, 1998.]. However, this motion blending simply creates a new motion by simply mixing the joint angles of the motion data, so it cannot handle cases where the motion direction or posture direction differs between motion data. It cannot be solved. For example, it is not possible to generate a strong upper-cut operation by mixing a whirling operation, a strong whirling operation, and an upper-cut operation. This is because the direction of the hand movement differs between the scolding movement and the upper cut. In this case, the main difference between the scolding motion and the scolding motion is that the horizontal motion of the hand is large in the strong scolding motion. However, such a change cannot be applied to the upper cut operation using the operation blending method. Since the hand moves in the vertical direction in the arper cut operation, it is necessary to increase the vertical movement of the hand in order to make it appear that the upper cut is strongly performed. If blending a strong scooping action and an upper cut action, the horizontal movement of the hand becomes large, resulting in an unnatural output action. As another example, it is not possible to generate a joyful running motion by blending a walking motion, a happy walking motion, and a running motion. By blending a walking motion that seems to be fun, the output motion is close to a walking motion, not a running motion. This is because not only the deformation related to the posture but also the global movement is blended at the same time.

  The present invention has been made to solve the above-described problem, and an object of the present invention is to provide a modified data extraction method for extracting modified data capable of transforming a wide range of input motion styles from a plurality of motion data. And It is another object of the present invention to provide a deformation data application method for applying extracted deformation data to an input operation.

(Outline of the present invention)
The present invention solves these problems by expressing information representing changes between two pieces of motion data as abstract deformation data. By expressing the deformed data in an abstract format, the deformed data can be applied to a wide range of input operations that differ from the original motion data in the direction of operation and global movement. The deformation data includes posture deformation data related to the posture between motion data and time deformation data related to time. In order to express posture deformation data, an expression based on a relative position of a body part serving as a reference is used instead of an expression based on a general joint angle. Also, the movement of the waist is expressed by dividing it into relative and absolute positions and directions. Also, changes in the position and orientation of the hand and foot are expressed by local coordinates calculated from the direction of movement.

  According to the present invention, for example, deformation data relating to “fatigue” can be extracted from two motion data of a walking motion and a tired walking motion (FIG. 1). Here, a new operation can be generated by applying the extracted deformation data to another operation. As an example, by applying deformation data relating to “fatigue” to a running motion, a tired running motion can be generated. At this time, by adjusting the degree of application of the deformation data, it is possible to change how much the “fatigue” operation is shown. That is, in the motion deformation method of the present invention, one new motion is generated from the three motion data. In addition, by storing a large number of pre-extracted deformation data in the database, selecting appropriate deformation data from the database according to the input operation and the style specified by the user, and applying it to the input operation, any input The action style of action can be transformed.

In the following, means for solving each item in more detail will be shown.
(1)
The motion deformation system according to the present invention includes an extraction unit that extracts deformation data that is a difference from a reference motion and a teacher motion, a unit that receives an input of a motion style of a teacher motion from a user, the deformation data extracted by the extraction unit, and a user Means for recording the action style of the teacher movement input from the user, means for receiving the action style input to be applied to the input action from the user, and from the action style and deformation data of the recorded teacher action from the user A search unit that searches for deformation data based on an input operation style; an application unit that applies the deformation data searched by the search unit to an input operation; and an output unit that outputs output data generated by the application. .

  It is desirable to construct a database by the above-mentioned “means for recording the deformed data extracted by the extracting means in association with the action style of the teacher action input by the user”. At this time, it is desirable to receive an input or selection of the motion type of the reference motion or the teacher motion from the user, and record the received motion type of the reference motion or the teacher motion in association with the deformation data and the motion style of the teacher motion. By accepting not only the operation style but also the operation type of the input operation at the time of the search, it is possible to search for deformation data matching the operation style and the operation type of the input operation, and apply the matching deformation data to the input operation. And can produce the desired natural motion.

  The search means does not search for deformation data related to an operation style and / or operation type that completely matches the input search key, but can also search for deformation data related to a similar operation style and / or operation type. . It is also possible to use a thesaurus to search for deformation data that is semantically close to the search key.

  The search means can also search for deformation data of a reference action or teacher action similar to the input action. Since there is already a well-known technique regarding the similarity of motion, it is not described in detail here (for example, the type of motion motion (periodic or non-periodic), the body part mainly used for motion, the direction of motion, etc.) Can be used as a parameter to determine similarity). If the deformation data is extracted from the reference motion / teacher motion similar to the input motion, a natural motion can be generated when the deformation data is applied. In this case, it is necessary to record the reference action or the teacher action when recording the deformation data. Here, data necessary for at least a similar process to the input action may be recorded instead of the reference action or the teacher action. This data may be part of the reference action or teacher action or converted.

(2)
In the motion deformation system according to the present invention, as necessary, the deformation data includes time deformation data and posture deformation data, the time deformation data indicates a temporal correspondence between the reference motion and the teacher motion, and the posture deformation data is the reference It shows the posture and / or motion difference at each time corresponding to the motion and the teacher motion.

(3)
The motion modification system according to the present invention includes means for accepting a key time set representing an operation milestone for the reference motion from the user and means for accepting a key time set representing the motion milestone for the teacher motion from the user, as necessary. A set of key times that are newly included and set for the reference action and the teacher action are supported, and the extraction unit extracts the deformation data based on the mutually corresponding key time sets received from the user. .

  As described above, according to the present invention, a set of key times is set so as to correspond to each of the reference action and the teacher action from which the deformed data is based, and the deformed data is generated based on the set key time set. Therefore, there is an effect that it is possible to generate highly accurate deformation data that appropriately associates the reference action and the teacher action and shows the action style of the teacher action well.

(4)
The motion modification system according to the present invention newly includes means for searching for a key time that represents a milestone in the motion from the motion data as needed, and a set of key times retrieved from the reference motion and the teacher motion is supported. The extraction means extracts the deformation data based on the set key times corresponding to each other.

  The means for automatically searching for the key time representing the knot of the action from the action data eliminates the need for the user to manually set the key time set. For example, the key time can be searched by using a known technique, for example, by applying a high-dimensional trajectory including the positions, angles, velocities, and angular velocity trajectories of each joint to a low-dimensional Euclidean distance space. In this case, there are times when the key time is searched more than necessary. In this case, the user is prompted to select or the time distance between the key times is taken into consideration (between the key frames). Take keyframes with a high degree of feature from the top (minimum required time).

  When the key time is set automatically or manually for only the reference action or the teacher action, the key time that has already been set using the similar action search or the similar attitude search that is already known in the art. The key time corresponding to is automatically detected and set for an operation in which the key time is not set. The detected key time can be presented to the user and selected from a plurality of key times.

(5)
In the motion deformation system according to the present invention, the application means applies deformation data corresponding to the degree of applicability to the input operation as required.

  As described above, according to the present invention, deformation data is variably applied to an input operation. Therefore, when the applicability is reduced, the influence of the operation style of the deformation data on the input operation can be controlled. Generation of an appropriate operation can be prevented, an appropriate operation can be generated, and conversely, when the degree of application is increased, an operation that emphasizes the operation style can be generated.

(6)
In the motion deformation system according to the present invention, the deformation data includes a relative position and / or orientation of each main part of the character's body with respect to a specific body part, as necessary.

  As described above, according to the present invention, since the deformation data is represented by relative posture expression, the deformation data that is the relative posture expression is applied even to an input motion whose absolute position and orientation are different from the reference motion. Therefore, it is possible to cope with a wide range of input operations.

  In the embodiment to be described later, the hand, the tip of the foot, and the waist are described as the main parts of the body. In addition, shoulders and hip joints are illustrated as reference parts for representing the relative positions and / or orientations of the hands and toes, respectively. In addition, as a reference coordinate system for representing the relative position and / or orientation of the waist, a standard waist position and / or orientation obtained from the positions of both feet is shown. The position and / or orientation of the main part other than these may be included in the deformation data, and the relative position and / or orientation of the main part with reference to the coordinate system of other reference parts or the coordinate system obtained by another method It can also represent the orientation.

  Further, by including the position and / or orientation of the main part calculated based on a plurality of reference coordinate systems in the deformation data, the most appropriate deformation can be applied according to the input operation.

  Specifically, instead of using the absolute coordinate system of the virtual space, a relative coordinate system is provided for each reference part, and this relative coordinate system is used to represent the difference in position and / or orientation of the main part. The position of the main part between the reference motion and the posture of the teacher data is provided with a relative coordinate system based on the difference vector between the position at the key time posture with the reference motion reference position and the position at the previous key time posture. And / or the difference in orientation is represented on this relative coordinate system. At the time of applying the deformation data, a relative coordinate system is provided based on a difference vector between the position of the reference part of the input operation at the key time posture and the position at the previous key time posture, and the relative coordinates of the reference operation are provided. After the deformation data on the system is converted into the deformation data on the relative coordinate system of the input operation, the output operation is generated by adding the deformation data on the relative coordinate system of the input operation to the input operation. By doing so, it is possible to generate a natural motion by applying the deformation data, even if the teaching motion and the input motion have opposite directions in the absolute coordinate system of the virtual space. Moreover, it is possible to apply not only the deformation data of the same body part but also the deformation data of other body parts by applying processing such as left-right reversal of the deformation data as necessary, and generate a natural motion. .

(7)
The motion deformation system according to the present invention includes an extraction unit that extracts deformation data that is a difference from a reference motion and a teacher motion, and an application unit that applies the extracted deformation data to an input motion. Extract and apply deformation data that represents the difference between the position and / or orientation of the hand and foot using the relative coordinate system based on the difference between the position of the hand and foot of the posture and the position of the hand and foot of the previous posture. The deformation data extracted by the extraction means by means of the relative coordinate system based on the difference between the position of the hand and foot of the target posture of the input operation and the position of the hand and foot of the previous posture is used. Applies to position and / or orientation.

(8)
According to the modified data generation method of the present invention, the speed change in the interval between each key time of the reference action and the teacher action from the reference action to which the key time set is set and the teacher action to which the key time set is set. Obtaining time deformation data for each section, and obtaining posture deformation data at each key time by obtaining a difference in posture from the posture at each key time of the reference motion and the posture at each key time of the teacher motion And obtaining posture deformation data at each time by obtaining a difference in posture from the posture at each time of the reference motion and the posture at the time of the teacher motion corresponding to the time of the reference motion obtained from the extracted time deformation data And the step of recording the obtained time deformation data, the posture deformation data at the key time, and the posture deformation data at each time as deformation data.

  As described above, in the present invention, first, the time deformation data of the reference motion and the teacher motion is obtained, and the posture deformation data at each time is obtained from the mutually corresponding postures using the obtained time deformation data. Along with the posture deformation data at the key time, the difference data is obtained by appropriately matching the posture at the time with the reference motion and the posture at the corresponding time of the teacher motion, and the deformation data showing the motion style of the teacher motion with high accuracy It has the effect that can be acquired.

  Each step is performed by a computer comprising a processor and memory, as will be apparent to those skilled in the art.

(9)
The modified data generation method according to the present invention newly includes a step of accepting an action style of a teacher action from a user as necessary, and records the action style received from the user in association with the obtained modification data.

  As described above, according to the present invention, the deformation data and the operation style are stored in association with each other, so that one deformation data is not used once immediately after generation, but the deformation data of the operation style desired by the user is required. Accordingly, there is an effect that it can be read out and applied accordingly and can be used effectively.

(10)
The deformation data application method according to the present invention includes a step of temporally deforming an input operation in which a set of key times is set from time deformation data of the deformation data, and posture deformation data at the key time of the deformation data. The method includes applying to the posture at the key time, and applying to the posture at each time of the input operation obtained by temporally deforming the posture deformation data at each time of the deformation data.

(11)
In the method according to the present invention, if necessary, the deformation data includes a relative position and / or orientation relative to a specific body part of each main part of the character body.

  These outlines of the invention do not enumerate the features essential to the present invention, and a sub-combination of these features can also be an invention.

(First embodiment of the present invention)
[1. Deformation data]
As shown in FIG. 1, deformation data is extracted from the reference motion and the teacher motion. Here, the reference motion has no special style, while the teacher motion has the same motion as the reference motion and some style. By applying the deformation data to a certain motion using the degree of applicability, a styled motion can be generated. Here, the input operation is a simple operation and is desirably the same type of operation as the reference operation. When deforming a long and complicated motion, it can be divided into a plurality of simple motions.

  In the present invention, it is necessary to set a key time representing an operation timing in advance in all operation data. In general, the operation is divided into three sections. These are the initial section, main section, and end section shown in FIG. Key time is set at the transition point of the preceding and following sections. In addition, the key time can be additionally set to the effective point of the operation in the main section. The effective point corresponds to, for example, when the highest position is reached by a jumping action, or when the person just touches the opponent by a scolding action. Any additional key time can be set as long as it corresponds to the operation data used for the deformation. In principle, the key time must be set manually by the user. However, since the time at which the key time should be set is usually apparent from the operation data, this setting operation is not so difficult and can be set in a few minutes. If the corresponding key time does not exist in the operation, it is not necessary to set the key time. Only the section in which the key time is set is used in the motion transformation process.

  As shown in FIG. 3, the deformation data SF includes time-deformation data related to time and posture deformation data related to posture as in the following equation.

ここで、Ｔ_i（ｔ）はｉ番目の区間に関する時間変形データ、ΔＰ_jはｊ番目のキー時間の姿勢に関する姿勢変形データ、ΔＱ_iはｉ番目の区間の姿勢に関する姿勢変形データである。Ｔ_i（ｔ）は各区間における基準動作の時間と教師動作の時間の対応関係を表すタイムワープ関数である。ΔＰ_jは３９次元のベクトルで、各キー時刻における基準動作と教師動作の姿勢の差分を抽象的な形式で表現している。ΔＱ_iは３９次元ベクトルの時間関数で、各区間における基準動作と教師動作の姿勢の差分の変化を示す。

Here, T _i (t) is time deformation data related to the i-th interval, ΔP _j is posture deformation data related to the posture of the j-th key time, and ΔQ _i is posture deformation data related to the posture of the i-th interval. T _i (t) is a time warp function representing the correspondence between the time of the reference action and the time of the teacher action in each section. ΔP _j is a 39-dimensional vector that expresses the difference between the posture of the reference motion and the teacher motion at each key time in an abstract format. ΔQ _i is a time function of a 39-dimensional vector and indicates a change in the difference between the posture of the reference motion and the teacher motion in each section.

ここで、ｔ_bは基準動作のｉ番目の区間における正規化時刻、ｔ_rは教師動作のｉ番目の区間における正規化時刻である。なお、正規化時刻とは、区間の開始時に０、終了時に１となるように、動作時刻を線形に変換したものである。タイムワープ関数は、２つの動作の同じ区間における対応する時刻同士の組によって表現することができる。 [1.1 Time deformation data]
The time deformation data is a time warp function for each interval of the reference motion and the teacher motion.

Here, t _b is the normalized time in the i th interval of the reference operation, t _r is the normalized time in the i th interval teacher operation. The normalization time is obtained by linearly converting the operation time so that it becomes 0 at the start of the section and 1 at the end. A time warp function can be represented by a set of corresponding times in the same interval of two actions.

  In order to obtain a time warp function, it is necessary to search corresponding postures in the same section of two motions and specify a set of corresponding frames. Traditionally, methods for retrieving similar poses have been used (Lee J, Chai J, Reistma P, Hodgins J, Pollard N. Interactive Control of Avatars Animated with Human Motion Data. ACM Transactions of Graphics (Proc. of SIGGRAPH 2002), 22 (3), 491-500, 2002.) (Kovar L, Gleicher M. Automated Extraction and Parameterization of Motions in Large Data Sets. ACM Transactions on Graphics (Proc. of SIGGRAPH 2004), 23 (3 ), 559-568, 2004.). However, this previous method only works if the two styles of action are very close. Accordingly, in the present invention, a normalized posture change amount indicating how much the current posture is located between the start posture and the end posture of the section is used. In the present invention, since a key time representing each section is given in advance as a precondition, such an approach is used. Since each section is clear in advance, it is only necessary to obtain the corresponding posture in the corresponding section. Although the same approach is used in the existing motion blending method, the corresponding posture is not searched as in the present invention, and the time between sections is simply linearly interpolated (Rose C, Cohen MF, Bodenheimer B. Verbs and Adverbs: Multidimensional motion interpolation. IEEE Computer Graphics and Applications. 18 (5), 32-40, 1998. (Park SL, Shin HJ, Kim TH, Shin S Y. On-Line Motion Blending for Real-time Locomotion Generation. Computer Animation and Virtual Worlds, 15 (4), 125-128, 2004.) (Ashraf G, Wong KC. Dynamic Time Warp Based Framespace Interpolation for Motion Editing, Proc. Of Graphics Interface 2000. 45-52, 2000. ), Parameter curve interpolation (Menardais S, Kulpa R, Multon F, Arnaldi B. Synchronization for Dynamic Blending of Motions. In Proceedings of the 2004 ACM Siggraph / Eurographics Symposium on Computer Animation 2004, 2004.) Yes. Therefore, the conventional method cannot extract a fine speed change between two operations.

ここで、ｘ_j（ｔ）はｊ番目の関節の位置、Ｍはその区間でのフレーム数を示す。姿勢に関する進度を得るために関節位置の差分の累積を計算する。Ｓ（ｔ）は動作区間での正規化時間ｔにおける正規化姿勢変化量を示す。Ｓ（ｔ）はｔが０から１に変化するのに従って、０から１まで単調に増加する。２つの動作の対応する区間の各フレームにおけるＳ（ｔ）を用いることで、その区間のタイム時間ワープ関数を表すことができる。 As shown in the following equation, a normalized posture change amount related to the posture of each motion section is calculated.

Here, x _j (t) is the position of the j-th joint, and M is the number of frames in that section. Calculate the cumulative joint position difference to get progress on posture. S (t) indicates the normalized posture change amount at the normalization time t in the motion section. S (t) monotonously increases from 0 to 1 as t changes from 0 to 1. By using S (t) in each frame of the sections corresponding to the two operations, the time-time warp function of the section can be expressed.

[1.2 Posture deformation data]
Hereinafter, the posture expression used to acquire posture deformation data in an abstract format will be described first. In addition, a method for expressing the position and direction of the end effector (for example, hands and feet) and the waist will be described in detail.

[1.2.1 Posture expression]
Information on the posture is expressed by a 39-dimensional vector P (see FIG. 4). Posture deformation data ΔP _i , ΔQ _i (t) (formula (1)) is expressed in this format by calculating the difference between the corresponding postures of the two motions. In this expression, the position and direction of the main body part are used instead of the posture expression by the joint angle generally used.

ローカルな方向ｒ_iを表現するために四元数(quaternions)を用いる。変形データの抽出処理過程及び適用処理過程において動作の方向的相違を取り扱うため、動作データの動作方向から求まるローカル座標を用いて、手先及び足先の空間的及び方向的変形量を表現する（後説１．２．３を参照）。 In order to express the posture of each limb, the positions and directions e _i and r _i of the hand and foot are used. Here, e _i and r _i are expressed in the local coordinate system of the body part serving as a reference for each hand and toe, such as a hip joint in the case of a foot and a shoulder in the case of a hand. e _{global, i,} r _{global, i} is a global position and orientation of the i-th hand and feet, when R _i is to show the local coordinate system of the i-th hand feet of the reference body part, global The local positions and directions e _i and r _i are obtained by converting the positions and directions e _{global, i} and r _{global, i} into the local coordinate system of the reference body part as in the following equation.

Use quaternions to represent local directions r _i . In order to deal with directional differences in motion in the deformation data extraction process and application process, the spatial and directional deformation amounts of the hand and toes are expressed using local coordinates determined from the motion direction of the motion data (later (See Theory 1.2.3).

In order to separate the absolute movement and relative movement of the waist, the absolute position, the relative position and the relative direction, respectively, p _abs , p _rel and q _rel are expressed independently ( _explained in 1.2 below). .4).

Back and neck rotations J _back and J _neck are also represented by quaternions, and indicate average rotations at a plurality of joints in the back and neck, respectively.

[1.2.2 Extracting posture deformation data]
As posture deformation data, a posture change ΔPi at each key frame and a posture change function ΔQi (t) of each motion section are used.

ｔはｉ番目の区間における正規化された時間、ｓは適用度、Ｐ_i（ｔ）、Ｐ_i（ｔ）´はそれぞれ入力動作、出力動作の姿勢を示す。入力動作の各フレームは抽象的な表現Ｐ_i（ｔ）で示され、ΔＰ_i、ΔＱ_i（ｔ）はＰ_i（ｔ）と共にスケールパラメータｓを用いて出力動作Ｐ_i（ｔ）´を求める。 In each section of the input motion, the posture deformation data is applied to the motion as follows.

t is the normalized time in the i-th section, s is the applicability, and P _i (t) and P _i (t) ′ are the postures of the input operation and the output operation, respectively. Each frame of the input operation is shown in an abstract representation P _i (t), determining the ΔP _i, ΔQ _i (t) is the output operation using the scale parameter s with _{P i (t) P i (} t) ' .

ここで、Ｐ_reference,j、Ｐ_base,jはｊ番目のキー時刻の基準動作、教師動作の姿勢である。 The postures of all frames of the reference motion and the teacher motion are converted into the above-described posture expressions P _base and P _reference . The difference in key time is calculated as follows:

Here, P _{reference, j} and P _{base, j} are the postures of the reference motion and teacher motion at the j-th key time.

以上により、全てのキー時刻におけるΔＰ_iと全ての区間におけるΔＱ_i（ｔ）が算出され、姿勢変形データとして記録される。 The difference ΔQ _i (t) related to the postures of all the frames in each section is obtained from the key time posture deformation data ΔP 1,..., ΔPn and the difference between the teacher motion and the reference motion as shown in the following equation.

As described above, ΔP _i at all key times and ΔQ _i (t) in all sections are calculated and recorded as posture deformation data.

[1.2.3 Expression of changes in hand and toe]
As shown in FIG. 5, the amount of change in the direction and position of the hand and foot is expressed using a local coordinate system calculated from the movement of the hand and foot. For example, when the movement of the hand increases in the forward direction between the reference action “straight punch” and the teacher action “strongly straight punch”, the amount of change in the Z-axis direction of the local coordinate system calculated from the movement direction It is expressed as a change amount (upper part of FIG. 5). When this amount of change is applied to the input operation “upper cut”, the movement in the Z-axis direction calculated from the direction of movement of the hand in the upper cut increases, and as a result, the vertical movement of the upper cut increases ( FIG. 5 bottom).

First, from the hand or foot position e _{base, i} of the key time and the hand or foot position e _{base, i + 1} of the next key time, for each hand and foot of each section as follows: Calculate the direction of motion.

ｅはＰの一部であることから、式（１０）は式（７）（８）に含まれる。そして、式（１１）はΔＰ_j、ΔＱ_iに適用される。 When a 3 × 3 matrix local coordinate system calculated based on the movement direction of the hand or foot is M _base , the direction d _{base, i} is used as the Z-axis component of M _base . Next, the X-axis component is calculated from the outer product of the Z-axis component and the upward vector. Finally, the Y-axis component is calculated from the outer product of the X-axis component and the Z-axis component. After setting the local coordinates, the amount of change in the position and direction of the hand or foot between the reference motion and the teacher motion is converted into the local coordinate system.

Since e is a part of P, equation (10) is included in equations (7) and (8). Expression (11) is applied to ΔP _j and ΔQ _i .

その結果、手先及び足先の運動方向が基準動作と入力動作の間で異なってても、適切な変形を実現できる。 When these deformation data are applied to an input operation, the local coordinates M _input of the hand and foot of the input operation are similarly calculated, and the deformation data is applied in the local coordinate system.

As a result, appropriate deformation can be realized even if the movement directions of the hand and toe are different between the reference action and the input action.

[1.2.4 Absolute position, relative position, absolute orientation, and relative orientation of the waist]
In order to apply deformation, the absolute position, relative position, absolute orientation, and relative orientation of the waist in each frame are handled separately.

The absolute position p _abs and the absolute direction q _abs are assumed to be neutral positions and orientations when it is assumed that the human body model stands upright without changing the posture of the waist (see FIG. 6). The relative position p _rel and the relative direction q _rel are the difference between the actual waist position p and direction q and the absolute position p _abs and absolute direction q _abs , respectively.

First, the absolute position _pabs at each key time is calculated from the positions of both feet, assuming that the human body model is upright without relative movement or rotation of the waist. At the key time when both feet are in contact with the ground, the center position of both feet is the horizontal coordinate ( _pabs.X , _pabs.Z ) of the absolute position. Further, the height of the waist in an upright posture is _defined as a vertical coordinate _pabs.V. At the key time when both feet are not in contact with the ground, the absolute position of the waist at the key time is obtained by linear interpolation of the absolute position of the waist before and after the time when both legs are in contact with the ground. Ask for.

Next, the absolute direction q _abs of each key time is obtained from the absolute position p _abs , _{j + 1} of the key time and the absolute position p _abs , _{j of the} previous key time. Here, since q _abs handles only the rotation in the horizontal direction, q _abs can be obtained from the horizontal elements of ( _pabs , _{j + 1} , _pabs , _j ). In order to calculate the absolute direction of the motion interval, the spherical liner interpolation (SLERP) (Pletinckx D. Quaternion calculus as a basic tool in computer graphics. The Visual Computer, 5 (2), 1989.) is used.

When the absolute position p _abs and the absolute orientation q _abs are obtained, the waist local coordinate system M _base is obtained from the waist absolute orientation q _{abs in} the same manner as the calculation of the hand and foot local coordinate systems.
The relative positions and orientations p _rel and q _rel are obtained by the following equations, respectively.

以上の結果、各キー時刻において、腰の状態を表す（ｐ_abs、ｐ_rel、ｑ_rel）が記録され、姿勢変形データの計算に用いられる。ｑ_absは、ローカル座標系Ｍ_baseを計算するために使用されるだけであるため、記録する必要はない。

As a result, at each key time, (p _abs , p _rel , q _rel ) representing the waist state is recorded and used for calculation of posture deformation data. Since q _abs is only used to calculate the local coordinate system M _base , it need not be recorded.

[2. Extraction and application of deformation data]
When extracting deformation data, the system first obtains time deformation data T _i (t), then obtains posture deformation data ΔP _j at the key time, and finally obtains posture deformation data ΔQ _i (t) of the section. . The reason for performing this procedure is that it is necessary to apply time deformation data to the teacher motion in order to calculate the posture deformation data from the corresponding frame between the reference motion and the teacher motion. In addition, the posture deformation data at the key time is first applied to the reference motion, and then the posture deformation data in the motion section is obtained using equation (8).

The deformation data is applied in the same order. When the applicability is designated by the user, the applicability is multiplied after all the posture deformation data are applied. At the time of deformation, the posture of each frame of the input motion is converted into an abstract posture representation P (t). Then, the posture deformation data ΔP _j and ΔQ _i (t) are applied using the equation (6). Based on the deformed posture expression P ′ (t), the posture of the output operation is calculated.

  However, when deformation data is simply applied to an input operation, an unnatural posture may be generated. For example, if a deformation that lowers the waist position is applied to an input operation in which the waist position is originally low, the waist position may become too low. In order to avoid such an unnatural posture, the relative position and relative orientation of the waist and the positions and orientations of the hands and toes are limited within a range that can be realized as a human posture. The rotation of each joint of the limb during the input motion is determined from the position and orientation of the hand and toes, an analytical inverse kinematics method (Lee J, Shin S Y. A Hierarchical Approach to Interactive Motion Editing). for Human-like Characters. Proc. of SIGGRAPH 99, 39-48, 1999.).

[3. System configuration]
A system configuration diagram is shown in FIG. This system is divided into two processes, that is, an extraction process of deformation data representing a feature amount of change and an application process. In the extraction process, two motion data of a certain standard motion and a teacher motion are input and received, and a change in feature is extracted as deformation data from the difference between the motions. In the application process, an output operation is generated by applying the input data to the deformed data with a degree of applicability.

  In this system, the deformed data is reused by storing the extracted deformed data in the database as an operation style such as “pleasant” or “fatigue” together with the type of the reference action. When the user inputs operation data as an input operation and inputs or selects an operation style such as “fun” or “tired”, optimum deformation data is retrieved from the database and applied to the input operation. By using this system, it is possible to change the operation style of operation data in various ways.

[4. Hardware configuration]
The motion deformation system according to the present embodiment is constructed in a computer having the following configuration. However, a system can be constructed using a plurality of computers. For example, it is constructed so that the process from generation of deformation data to storage in the database is processed by the first computer, and subsequent search processing and application processing are processed by the second computer. The database itself can be constructed with a third computer.

  FIG. 8 is a hardware configuration diagram of a computer that constructs an operation modification system according to the present embodiment.

  The computer includes a CPU (Central Processing Unit) 11, a DRAM (Dynamic Random Access Memory) 12 and other main memory, an external storage device HD (hard disk) 13, an input device keyboard 14 and mouse 15, and a display 16a display. It comprises a video card 16 for controlling the operation, a sound card 17 for controlling the sound operation of the speaker 17a, a LAN card 18 as an expansion card for connection to a network, a CD-ROM drive 19, and the like. Although the LAN card 18 is not used in the present embodiment, as is apparent to those skilled in the art, the present invention can be realized by constructing a LAN depending on the system configuration. For example, connect to another computer with an action creation system built in via LAN, and input the action data created by the designer to the computer as a reference action, teacher action, and input action via the LAN, and reversely output action Can be returned to the computer used by the designer from another computer on which the motion creation system is constructed.

  For example, the operation modification program stored in the CD-ROM is duplicated (installed) on the HD 13, and the operation modification program is read into the main memory 12 as necessary, and the CPU 11 executes the program to modify the operation. Configure the system.

[5. System block configuration]
FIG. 9 is a system configuration block diagram of the motion modification system according to the present embodiment.

  The motion deformation system according to the present embodiment generates deformation data representing a style difference between two motions from between a standard motion that is a normal motion style and a teacher motion having some motion style. An extraction unit 21, a storage unit 22 that stores the deformation data generated by the extraction unit 21 in the database 23, a search unit 31 that searches the database 23 for deformation data to be applied to the input operation, and the search unit 31 An application unit 32 that generates the output operation by applying the read deformation data to the input operation.

  The extraction unit 21 calculates the reference motion and the teacher motion from [1. The deformation data described above in [Deformation data] is extracted by the method described in [2 Extraction and application of deformation data].

  The storage unit 22 does not sequentially store the deformation data generated from the extraction unit 21 in the database 23, but instead generates the deformation data generated in the database 23 in association with the operation style (“fatigue”, “fun”, etc.) of the teacher operation. Is stored.

  As the action style of the teacher action, input from the user may be received in a text box, or a plurality of action styles may be displayed in a list and received from the user. Of course, this combination is also possible. Initially, text input can be performed and a list of already input operation styles can be displayed. Further, it is desirable that the list display is performed by classifying and arranging the operation styles according to meaning as in a thesaurus.

  Note that the deformation data and the motion style of the teacher motion and the motion type of the reference motion (“walk”, “run”, etc.) can also be stored in the database 23 in association with each other. At this time, the action type of the reference action is designated by the user in the same manner as the action style of the teacher action. Although all deformation data can be applied to any input motion, an output motion can be generated, but by specifying the motion style of the teacher motion, the output motion unintended by the designer who creates the motion, or Unnatural output behavior may be generated. One of the main reasons is that the operation type of the reference operation and the operation type of the input operation may be greatly different. Therefore, in addition to the motion style of the teacher motion, the motion type of the reference motion at the time of creating the deformation data is also associated and stored in the database 23 so that it can be referred to and specified when searching for the deformation data. As a result, generation of an unexpected output operation can be prevented, and a desired output operation can be generated. Note that the action type can use information input or selected by the user in the same manner as the action style, or can calculate a feature amount representing the action type by analyzing the reference action.

  The search unit 31 searches the database 23 for deformation data to be applied to the input operation using the search key (keyword) for the operation style received from the user. The input from the user is the same as the operation style of the teacher operation when storing the deformation data.

  The application unit 32 generates the output operation by applying the deformation data searched by the search unit 31 to the input operation by the method described in [2 Extraction and application of deformation data]. The application unit 32 can change the degree of deformation data given to the input operation according to the degree of application received from the user. This degree of applicability can also be applied to each part.For example, when applying deformation data, it can be set so that the foot is not applied much, but the arm is applied as it is, and the user can be flexibly assumed. An output action can be generated. The ability to change the applicability for each part in this way is also one of the advantages of this operation system compared to the conventional method of directly changing the joint angle.

[6. Operation]
[6.1 Modified data generation and storage]
FIG. 10 is a flowchart of the deformation data extraction operation and the storage operation of the operation deformation system according to the first embodiment.

  First, the motion deformation system inputs a reference motion (step 101). The motion can be created using a motion capture device or motion design device, and the created motion is taken into the motion deformation system as a reference motion.

The motion transformation system inputs a teacher motion (step 111).
The extraction unit 21 generates deformation data from the input reference motion and teacher motion (step 200).
Accepts input or selection of action style from the user.
The storage unit 22 stores the received operation style in association with the generated deformation data (step 301). In the case of storing in the database, a table having an operation style and modified data attributes is set in the database. In this database, a table to which attributes of reference actions and teacher actions are added can be set as necessary.

FIG. 11 is a flowchart of the detailed operation of extracting deformation data of the operation deformation system according to the first embodiment.
The extraction unit 21 performs association based on the normalized posture change amounts of the reference motion and the teacher motion in each section (step 210).
The extraction unit 21 calculates posture deformation data at the key time from the posture at the key time of the standard motion and the posture at the key time of the teacher motion (step 220).
Using the temporal deformation data obtained in step 210 and the deformation data obtained in step 220, the extraction unit 21 calculates motion deformation data from the difference in posture between corresponding frames in each section (step 230).

[6.2 Modified data generation and storage]
FIG. 12 is a flowchart of the modification data application operation of the operation modification system according to the first embodiment.
First, the motion deformation system inputs an input motion for making a desired change to the motion (step 401).
Accepts an input or selection of an action style to be applied to the input action from the user.
An input or selection of the applicability indicating the degree of deformation applied to the input operation is received from the user.
The search unit 31 searches for deformation data using the operation style as a search key (step 500).
The application unit 32 applies the retrieved deformation data to the input operation with the application degree specified by the user to generate an output operation (step 600).

FIG. 13 is a flowchart of the detailed modification data application operation of the behavior modification system according to the first embodiment.
The application unit 32 modifies the speed of the input operation using the time deformation data of the deformation data (step 610).
The applying unit 32 multiplies the posture deformation data at the key time by the applicability, and adds it to the key time posture corresponding to the input operation, thereby deforming the key time posture (step 620).
The applying unit 32 multiplies the posture deformation data of each section by the applicability, and adds it to the corresponding frame posture of the input operation, thereby deforming the posture of each frame (step 630).

(Other embodiments)
[Similar search of operation style]
When searching for deformation data, not only an operation style input and selected by the user, but also an operation style similar to the input and selected operation style can be presented to the user. In particular, it is desirable that the similar operation style to be presented is the operation style of the deformation data already stored in the database 23. Further, when there are a plurality of similar operation styles, it is preferable to present the operation styles to the user in the order of similarity. The presented similar action style can also be selected by the user, and a similar action style can be presented to the user with respect to the selected similar action style. More specifically, similar behavior styles can be detected using a thesaurus, which is similar to the behavior style input and selected by the user. Similar behavior styles include synonyms, related terms, broad terms, narrow terms, and the like.

  In this case, when the action style already stored in the database 23 is presented to the user, it is desirable to present not only the action style but also the action type of the reference action stored in association therewith.

[Select action style]
In the selection of an action style by the user when searching for deformation data, for example, it is assumed that an already registered action style can be selected and displayed, but with this display, deformation data associated with each action style is generated. It is also possible to display the reference action and / or the teacher action used for the purpose. By doing so, it is difficult for the user to understand only the operation style, but the operation style can be grasped by referring to the actual operation.

[Display applicable action styles based on input actions]
The operation style that can be applied to the input operation is not necessarily the operation style of all deformation data stored in the database 23. If the operation types of the reference operation and the input operation are completely different, the deformation data may not be applied. Therefore, it is desirable to display a list by removing operation styles that cannot be applied to input operations. The deformation data that cannot be applied to the input operation can be determined based on the difference between the reference operation and the input operation. Specifically, it can be determined that the application is difficult when there is no certain similarity using the motion similarity evaluation. Alternatively, a combination of operation types that cannot be applied in advance may be recorded, and an operation style that cannot be applied may be determined from the operation types of the reference operation and the operation types of the input operation. In addition, information that cannot be applied by the user (operation type of reference action, action type of input action) is recorded as history information, and if it is a combination of history information, natural output before generating the output action A warning message indicating that there is a high possibility that the action cannot be generated is output, or the action style that is presented by the user is not displayed or cannot be selected. Can be.

[Display multiple output operations]
When a plurality of operation styles are selected by the user, each output operation can be output by applying deformation data corresponding to each operation style to the input operation, and the plurality of output operations can be simultaneously displayed on the display. As a result of the comparison display, the user can use a desired output operation. At this time, by applying only the posture deformation data without applying the time deformation data, the difference between the postures of the input operation and the output operation can be compared in more detail. In addition, for parts with a large amount of deformation in the deformation data to be applied, the user can better recognize the difference in posture between the input action and the output action by changing the color of the part in the output action and displaying it. Can do.

  In addition to displaying the output behavior of the application results of multiple behavior styles at the same time, the user can determine the appropriate applicability by displaying the output behavior of the application results at various applicability at the same time. You can also.

[Composite action consisting of multiple actions]
Even for a combined action in which a plurality of actions such as three-step jump and dance are combined, it is possible to create deformation data if the user can appropriately set the key time. However, when the created deformation data is applied, input operations that are natural operations are limited. Therefore, in the case of a composite operation, the operation breaks to be configured are set automatically or manually. If it is a repetitive motion such as a walking motion, the periodicity of the motion (one motion consists of the initial posture to the final posture, and the break can be detected by detecting the initial posture or the final posture of the subsequent motion) It is possible to set automatically by paying attention to the analysis. If a break between motions is designated, the motion between the breaks can be recognized as a single motion, and deformation data can be created. When the number of operations constituting the reference operation that is a composite operation is the same as the number of operations constituting the input operation that is also a composite operation, an output operation can be obtained by applying the corresponding modified data. If the number of operations is different and the number of input operations is small, it can be handled by not using the deformation data of the number of operations of the reference operation that is larger than the number of operations of the input operation. When the number is large, it can be dealt with by repeatedly using the deformation data of the reference motion for the number of motions that are insufficient.

[Suppression of unnatural motion output due to changes in each part]
For a body part whose amount of change is less than or equal to a predetermined threshold during input operation, it is possible to adopt a configuration in which deformation data is not applied to that part, and the movement of the part that does not change greatly during input operation is changed by the deformation data By doing so, it is possible to prevent an unnatural operation from being output.

[Transformation of transformation data]
For example, when the deformation data is generated from the movement with the right hand, the deformation data may be applied to the movement with the left hand. If the deformation data is simply applied, the deformation data of the right hand that moves with the right hand is applied to the right hand of the movement that moves with the left hand, resulting in an unnatural movement. In this case, it is necessary to convert the deformation amount of the right hand in the deformation data into the deformation amount of the left hand, and it is necessary to replace the data of the left and right body parts in the deformation data. However, simply changing the deformation amount of each part will cause the left hand position to change inward when the position of the right hand changes to the outside. Needs to be converted.

[Use with different characters]
In the embodiment, deformation data generated from the same character can be applied to characters having different physiques. At this time, it is desirable that the deformation amount in the deformation data is appropriately scaled according to the length of the limb.

  Although the present invention has been described with the above embodiments, the technical scope of the present invention is not limited to the scope described in the embodiments, and various modifications or improvements can be added to these embodiments. . And embodiment which added such a change or improvement is also contained in the technical scope of the present invention. This is apparent from the claims and the means for solving the problems.

It is a conceptual explanatory view of the present invention. It is explanatory drawing of the setting of the area of operation which concerns on the 1st Embodiment of this invention, and key time. It is explanatory drawing of the deformation | transformation data which concerns on the 1st Embodiment of this invention. It is explanatory drawing of the attitude | position expression which concerns on the 1st Embodiment of this invention. It is explanatory drawing of the expression of the deformation | transformation amount of the hand and foot tip in the local coordinate system based on the operation | movement direction which concerns on the 1st Embodiment of this invention. It is explanatory drawing of the absolute position / relative position / absolute direction / relative direction of the waist according to the first embodiment of the present invention. It is a system configuration figure concerning a 1st embodiment of the present invention. It is a hardware block diagram of the computer which builds the operation | movement deformation system which concerns on the 1st Embodiment of this invention. It is a system configuration block diagram of an operation modification system concerning a 1st embodiment of the present invention. It is a flowchart of the deformation | transformation data extraction storage operation | movement of the operation | movement deformation system which concerns on the 1st Embodiment of this invention. It is a flowchart of the deformation | transformation data extraction detailed operation | movement of the operation | movement deformation system which concerns on the 1st Embodiment of this invention. It is a flowchart of the deformation | transformation data application operation | movement of the operation | movement deformation system which concerns on the 1st Embodiment of this invention. It is a flowchart of the deformation | transformation data application detailed operation | movement of the motion deformation | transformation system which concerns on the 1st Embodiment of this invention.

Explanation of symbols

11 CPU
12 RAM
13 Hard Disk 14 Keyboard 15 Mouse 16 Video Card 16a Display 17 Sound Card 17a Speaker 18 LAN Card 19 CD-ROM Drive 21 Extraction Unit 22 Storage Unit 23 Database 31 Search Unit 32 Application Unit

Claims (11)

Extraction means for extracting deformation data that is a difference from the reference motion and the teacher motion;
Means for accepting an input of a teacher action style from a user;
Means for associating and recording the deformation data extracted by the extracting means and the action style of the teacher action input by the user;
Means for accepting an input of an action style to be applied to an input action from a user;
Search means for searching for deformation data based on the action style input from the user from the action style and deformation data of the recorded teacher action;
Application means for applying the deformation data searched by the search means to the input operation;
An operation deformation system including output means for outputting output data generated by application. The deformation data is composed of time deformation data and posture deformation data,
Temporal deformation data shows the temporal correspondence between the standard action and the teacher action,
The motion deformation system according to claim 1, wherein the posture deformation data indicates a difference in posture and / or motion at a time corresponding to a reference motion and a teacher motion. Means for accepting a set of key times representing a milestone in the action relative to the reference action from the user;
Newly includes means for accepting a set of key times from the user that represents the milestones of motion for teacher motion,
A set of key times corresponding to the standard action and teacher action is supported,
The motion deformation system according to claim 1 or 2, wherein the extraction unit extracts deformation data based on a set of key times corresponding to each other received from a user. Newly includes means for searching for key times representing movement milestones from movement data,
A set of key times retrieved from the reference action and the teacher action is supported,
4. The motion deformation system according to claim 1, wherein the extraction unit extracts deformation data based on a set of key times corresponding to each other. The motion deformation system according to claim 1, wherein the applying unit applies deformation data corresponding to a degree of application to an input operation. The motion deformation system according to any one of claims 1 to 5, wherein the deformation data includes a relative position and / or orientation of each main part of the body of the character with reference to a specific body part. An extraction means for extracting deformation data that is a difference from the reference action and the teacher action; and an application means for applying the extracted deformation data to the input action,
Deformation data in which the position and / or orientation of the hand and foot are expressed by a relative coordinate system based on the difference between the hand and foot position of the target posture of the reference motion and the hand and foot position of the previous posture by the extraction means Extract
The deformation data extracted by the extraction means by the applying means is applied to the input action by a relative coordinate system based on the difference between the hand and foot position of the target posture of the input action and the hand and foot position of the previous posture. Motion deformation system. Temporal deformation data indicating the speed change in the interval between each key time of the reference action and the teacher action from the reference action to which the key time set is set and the teacher action to which the key time set is set for each interval Seeking steps,
Obtaining posture difference data at each key time by obtaining a difference in posture from the posture at each key time of the reference motion and the posture at each key time of the teacher motion; and
Obtaining posture deformation data at each time by obtaining a difference in posture from the posture at each time of the reference motion and the posture at the time of the teacher motion corresponding to the time of the reference motion obtained from the extracted time deformation data; ,
And a step of recording the obtained time deformation data, the posture deformation data at the key time, and the posture deformation data at each time as deformation data. A step of accepting a teacher action style from the user,
The deformation data generation method according to claim 8, wherein the operation style received from the user is recorded in association with the acquired deformation data. Time-deforming an input operation in which a set of key times is set from the time-deformed data of the deformed data of claim 8 or 9,
Applying the posture deformation data at the key time of the deformation data to the posture at the key time of the input operation;
Applying the deformation data at each time of the deformation data to the posture at each time of the input operation obtained by temporally deforming the deformation data. The method according to any one of claims 8 to 10, wherein the deformation data includes a relative position and / or orientation of each main part of the body of the character with respect to a specific body part.

Images