JP2005063463A - Dynamic image processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a dynamic image processing device that divides off a region with different motion by the use of a direction of translational motion. <P>SOLUTION: The dynamic image processing device comprises an image inputting part 21 for inputting image information corresponding to a plurality of images, a feature point extracting part 22 for performing feature point extraction processing on the image information to extract a feature point from each of the plurality of images, a translational flow computing part 23 for computing a direction of translational motion from motion of an adjacent feature point, and an independent motion dividing part 24 for dividing off a region with different motion by the use of the direction of translational motion computed by the translational flow computing part. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a moving image processing apparatus that inputs a plurality of pieces of image information and detects the movement and structure of a target object from a change in the position of a feature point.

  Several methods have already been proposed for detecting the structure of an object photographed in a plurality of images.

  For example, S. Ullman is the interpretaion of visual motion. MIT Press Cambridge, USA, 1919 is an image of three or more parallel projections, and the correspondence of four points that are not on the same plane of a rigid object is determined. In this case, a method for completely determining the structure and movement of the four points is introduced.

  In addition, HCLonguest-Higgins is a computer algorithm for reconstructing a scene from two projections Nature, 293: 133-135, 1981, where there are eight corresponding points on two perspective transformed images. A linear calculation method for detecting is disclosed.

  In addition, ODFaugeras and SJMaybank have proposed that Motion from point matches: multiplicity of solutions, IEEE Workshop on Motion 248-255 1989, if there are five corresponding points in two centrally projected images, It is described that the motion becomes finite.

  In Japanese Patent Laid-Open No. 3-6780, first, a three-dimensional rotational motion is obtained from corresponding points on two images, and then, the three-dimensional rotational motion information based on one of the corresponding points is obtained from the rotational motion information. A method for obtaining a positional relationship is disclosed.

  In all of these methods, an equation is established between the three-dimensional coordinates of the object and the coordinates on the image on which the object is projected by central projection, and the answer is obtained by solving the equation.

  Also, as disclosed in Jan J Koenderink and Andrea J. van Doolrn's Affine structure from motion, Journal of Optiical Society of America pp. 377-385 vol.8, No.2 1991, the motion of an object is affine ( (affine) transformation (primary transformation), and a method for detecting the structure of an object from there is also calculated. In this method, an approximate structure of an object can be calculated from two frames of a moving image. The structure of the object calculated by this method is a structure obtained by multiplying information in the depth direction by an unknown coefficient proportional to the distance from the camera to the object.

  The above-described method for solving the central projection equation can efficiently calculate the motion and structure of an object when the object to be imaged is very close to the image capturing apparatus and is reflected in a large image. As occurs in the image, when the area of the object to be photographed is small in the image or when the distance from the photographing device to the target object is long, the deformation of the image by the center projection is small, and the deformation Since the motion of the object is calculated based on the above, the calculation result becomes unstable. For example, it becomes difficult to distinguish parallel translation in the direction perpendicular to the line-of-sight direction and rotation around an axis perpendicular to the movement direction. Directional ambiguity occurs, making it difficult to distinguish between a nearby carved shallow object or a distantly carved object, whether a small object is moving near the observer, or a large object far away It happened that it was difficult to determine whether the person was exercising.

  Also, Koendoerink's method includes an unknown coefficient in the structure of the detected object, so it was difficult to calculate the motion of the object from here.

  The present invention has been made to solve such a problem, and represents the deformation of an image due to the movement of the observer by approximating it with an affine transformation (primary transformation), and the actual feature point. By calculating the virtual parallax that is the difference between the position moved by the movement of the object and the position moved by the affine deformation due to the movement of the surrounding features, and calculating the motion of the object directly from the virtual parallax information, the ambiguity of the central projection It is an object of the present invention to provide a moving image processing apparatus that accurately calculates a motion parameter of an object without being affected.

  The present invention provides image input means for inputting image information corresponding to a plurality of images, feature point extraction means for performing feature point extraction processing on the image information to extract feature points from each of the plurality of images, A moving image comprising a translational flow calculation unit for calculating a translational motion direction from the motion of a feature point, and an independent motion splitting unit for dividing a region performing different motions using the translational motion direction calculated by the translational flow calculation unit. An image processing apparatus is provided.

  The present invention also includes an image input unit that inputs image information corresponding to a plurality of images, a motion information estimation unit that estimates motion information on the observation side that captures images between the plurality of images, and an estimated motion information. There is provided a moving image processing apparatus including a pointing information generation unit that generates a pointing event, and pointer display means that displays a 3D model of a previously stored object by translation and rotation according to the pointing information.

  Furthermore, the present invention provides an image input means for inputting image information corresponding to a plurality of images, and a plurality of pieces of image information obtained by photographing an object whose shape is unknown. There is provided an image processing apparatus including a structure extracting unit for extracting and an image synthesizing unit for synthesizing an image when an object is viewed from a new viewpoint by deforming an input image.

  According to the present invention, it is possible to extract a region indicating an object that is moving differently from a moving background image, remove the region that is moving differently, and translate from the remaining region indicating the background image. Motion parameters other than projection θA of motion onto the image plane can also be calculated to determine camera motion.

  Embodiments of the present invention will be described below with reference to the drawings.

  According to a simple embodiment used to indicate the motion posture of a numerically represented object model shown in FIG. 1, the image input unit 1 is connected to a pointing information generation unit 4 via a feature point extraction unit 2. . An affine flow analysis unit 3 and a gesture pattern matching unit 6 are connected to the pointing information generation unit 4. The gesture pattern matching unit 6 is connected to the gesture pattern storage unit 7 and further connected to the status switching unit 5 together with the pointer display unit 8. The pointer display unit 8 is connected to a 3D (three-dimensional) model storage unit 9, an image composition unit 10, and a pointer model storage unit 12. The image composition unit 10 is connected to the image input unit 1 and to the image display unit 11.

  The image input unit 1 inputs time-series image information obtained by photographing a moving object with a television camera or the like, and transfers this to the feature point extraction unit 2. This image is an image obtained by photographing a person sitting in front of the image display unit 11 from the ceiling so as to easily photograph a moving object, an image photographed by a frame-shaped camera of the display of the image display unit 11, and a central projection. For example, an image obtained by arranging a plurality of cameras having a long focal distance and a narrow imaging range so that the effect is not large and connecting images input from these cameras is connected. Also, here, an object to be input as an image is easily distinguished from a part of a body such as a human hand or another part so that it can be easily processed by the feature point extraction unit 2 as shown in FIG. For example, a part of the body such as a gloved hand that has been colored by attaching a colored part or a colored object, etc., or a person can move it by hand An instrument having a feature that can be distinguished from other parts by image processing. FIG. 3 shows an example of such an instrument, in which four spheres of the same size are painted in different colors so that they can be distinguished from other spheres, and connected in a positional relationship that is not on the same plane in a three-dimensional space. It is. FIG. 4 is an example of another device, which is a device in which a light emitting element such as an LED is embedded on the surface of a box, and emits light by electric power from a built-in battery. There is a switch at a position to be pushed with a finger when grasping the instrument, and the instrument can be moved without instructing movement during that period while the switch is depressed. The features on the glove in FIG. 2 and the light emitting elements in FIG. 4 are arranged so that four or more features that are not in the vicinity can be seen when viewed from all three-dimensional viewing directions.

  The feature point extraction unit 2 receives time-series image information from the image input unit 1 and can easily identify it from other regions by image processing and can identify that the same point on the object has been projected. Is extracted and tracked, and the coordinate value is output to the pointing information generation unit 4. If the photographed object is a hand or the like, a point having a large brightness difference with a neighboring pixel in the image is extracted as a feature point. 2 or 3, if the feature is a colored part, prepare a memory for storing the color information about what kind of color is featured, and the color stored there A region having the same color as the information is extracted from the image information, and the barycentric coordinates are output to the pointing information generation unit 4. Further, in this case, when the size of an area having a color is known, the size of the area is also output as auxiliary information to the pointing information generation unit 4. The color information to be stored in this memory is written in the memory in advance, or after the apparatus according to this embodiment is activated, a learning means for learning color information is activated, and the input image information is displayed on the image display unit. The user can select a region of the image to be characterized by a cursor, or select an area with the cursor, or display the input moving image information and the window on the image display unit 11 so that the color characteristic of the user is displayed. Operate your hands and instruments to enter the window, capture an image of the area specified by the window at the timing of key input, etc., obtain a partial image containing the characteristic color, and learn the color from there Store in memory. For learning colors, for example, if the image is expressed by three components of I (lightness), H (hue), and S (saturation), the following quadratic equation expression is prepared, and the specified partial image For example, color learning can be performed by estimating a parameter by performing least square estimation on a pixel value.

H = h0 + h1I + h2I2
S = s0 + s1I + s2I2
When a light emitting element is used as in the instrument of FIG. 4, an appropriate threshold value is provided, the image is binarized, the center of gravity of each area brighter than the threshold value is calculated, the feature coordinate value is calculated, and the pointing information generation unit 4 Output to.

  The pointing information generation unit 4 inputs time-series data of feature point coordinates from the feature point extraction unit 2, selects the coordinates of a plurality of feature points at two points in time from there, and the affine flow analysis unit 3 determines the motion parameters. Analyze and use the result to generate the information needed to move the pointer.

  Processing performed by the affine flow analysis unit 3 will be described in detail below.

The affine flow analysis unit 3 inputs the coordinates of the four feature points in the two images, and between the two images of the camera (observer) that captured the image of the object composed of the four feature points. Calculate the motion. In this case, an imaging system model in which the center projection as shown in FIG. 5 is used is considered. Here, a point at coordinates (X, Y, Z) in the three-dimensional space is projected onto a point (x, y) on the image plane at the focal length f. In this state, the observer side is translation at a rate _{{U 1, U 2, U} 3}, and has a rotational movement _{{Ω 1, Ω 2, Ω} 3}. Assuming that the feature point (x, y) input from the feature point extraction unit 2 is sufficiently close to the line-of-sight direction, the component of the moving speed V (x, y) on the image plane of the feature point (x, y) is represented by (u , V).

This velocity component is expressed by the motion parameter on the observer side. The relationship between the three-dimensional coordinates (X, Y, Z) and the speed parameter is

Therefore, as the coordinates X1, Y1, Z1 after movement

Is obtained. This is the projection relationship between the 3D coordinates and the points on the image plane,

The derivative of

Is assigned to

Get.

Do the same calculation for v,

It can be expressed as. The moving speed of the image can be divided into a component including scene information (Z) that depends on translational motion and a component that depends on rotational motion. Since the component that depends on the rotational motion only changes depending on the pixel position, it does not depend on the location or shape of the object. If the value does not appear, the change appearing on the image becomes small, so it is difficult to actually determine the parameters of the rotational motion. Therefore, a change due to the rotational motion is added to the translational motion as an error, and the calculation accuracy of the translational motion is also deteriorated. As a result, it has been difficult to accurately calculate the shape and motion of an object.

However, assuming that two feature points are projected at the same location on the image, and considering the difference in movement speed (Δu, Δv) between the two feature points (hereinafter referred to as motion parallax), The magnitude of motion parallax is

It is. However, Z ₁ and Z ₂ are Z coordinates of two feature points used for calculation of motion parallax. This motion parallax depends only on the distance to the object and the translational motion of the observer, and does not depend on the rotational motion of the observer. From this equation 12,

Thus, it can be seen that if x, y and U3 are sufficiently small, the direction of translational motion can be obtained from motion parallax. When U ₃ is not small, the direction of translational motion can be obtained from the ratio of U ₁ and U ₂ by substituting the motion parallax of a plurality of points with different coordinate values (x, y) into this equation. .

On the other hand, if the observer's motion is sufficiently smooth and the surface of the object being photographed is also sufficiently smooth, the velocity field of the image of Equation (11) is approximated by a linear equation in a small region. be able to. In other words, the image velocity field in the vicinity of a certain coordinate (x, y) on the image is calculated using affine transformation (primary transformation),

It can be expressed as Of these, 0 (x ² , xy, y ² ) represents a second-order nonlinear component, but this portion is considered to be sufficiently small and is ignored in subsequent calculations.

The first term [u0, y0] represents the translation of the image, and the 2 × 2 tensor of the second term represents the deformation of the shape of the image. The subscript of the second term indicates partial differentiation with the parameter indicated by the subscript. This 2 × 2 tensor of the second term is divided into a number of geometrically meaningful components as in FIG. Rotation (Curl) curlV on the image plane showing the change in orientation, isotropic change divV showing the change in scale (Divergence), Deformation of the image (in the direction of a certain axis while keeping the area constant) DefV indicating the magnitude of the change (the change that extends and contracts in the direction of the axis perpendicular thereto), and the deformation main axis μ that indicates the direction in which the image deformation extends. These feature amounts are obtained when the image speed at a certain coordinate (x, y) is V (x, y).

It is represented by Among these feature amounts, the values of divV, curlV, and defV are invariant feature amounts that do not change depending on how the coordinate system is taken in the image. The principal axis μ of deformation is a feature quantity that depends only on the direction of the axis of the coordinate system.

  As shown in FIG. 7, a feature point P on a certain image plane and three feature points in the vicinity thereof are considered. As already indicated, in a sufficiently small region, the speed on the image plane can be approximated by affine transformation (primary transformation) obtained from three feature points. Consider a virtual point P ′ that has the same coordinates as the point P and has a moving speed determined by affine transformation of the moving speeds of the other three points. The motion parallax between the actual point P and the virtual point P ′ is projected on the same coordinates as the point P, but is a difference in motion from a point having a different distance to the observer. Hereinafter, the motion parallax between the points P and P ′ will be referred to as virtual parallax. As shown in Equation (12), the motion parallax is not affected by the rotational motion of the observer, but depends only on the translational motion and the distance. From this, stable motion parameters and three-dimensional structure information are calculated. can do.

What kind of information can be obtained from this virtual parallax is obtained by applying the moving speed of the image (formula 11) to the affine transformation formula (14). Since it is assumed that the object being photographed is sufficiently far from the observer, the change in the distance between the three-dimensional coordinates on the object surface is very small compared to the distance from the observer to the object. Therefore, let λ be the distance from the image center to the object when f is 1, and change the distance to the object surface.

By normalizing Z, Z is normalized by a variable λ representing depth. As a result, the translational component of the observer and the parameters of the Affine transformation are

It is expressed as

If this result is substituted into the expressions representing the invariant feature values of Expressions (15) to (18),

It becomes. As can be seen from the equation, these parameters depend on the observer's movement, depth, and surface orientation. This can be rewritten using two vectors A and F so as not to depend on the coordinate system. A is a translational velocity vector parallel to the image plane, normalized by depth λ, as in the equation below. U is a translational motion vector, and Q is a unit vector in the line-of-sight direction.

F is a two-dimensional vector indicating the direction of the maximum gradient of the object surface, also normalized by the depth λ.

As shown in FIG. 8, F represents the tangent of the inclination σ of the object surface (the tangent of the angle formed by the line-of-sight direction and the normal of the object surface). The direction F represents an angle τ formed by the tangent plane and the x axis.

Using the vectors A and F having the above properties, rewriting the invariant feature expression above,

It is expressed. An angle μ indicating the main axis of image deformation is represented by an angle passing through the midpoint between A and F.

The information obtained by using the equations (34) to (37) includes ambiguity because the center projection is approximated by the weak-perspective projection. For example, the movement speed of an object cannot be distinguished from a small object moving near and a large object moving far, and there is ambiguity in size and speed. The distance tc until it hits the object when moving at speed

It will be expressed as Equation (36) cannot determine whether a nearby carved shallow object or a distant carved object is deep and includes depth ambiguity. From the value of this equation, the deformation of the image is It is impossible to distinguish between an object that has moved a lot (with a large | A |) and a surface with a small inclination (small | F |) or an object that has moved a small surface with a large inclination. Thus, by clarifying the portion where ambiguity exists, the remaining necessary information can be accurately obtained without being affected by noise.

  Next, processing performed by the affine flow analysis unit 3 will be described with reference to the flowchart of FIG.

  First, when three points are extracted and combined from the four input feature points, the three feature points that maximize the area of the region formed by connecting the three points are selected, and the selected three points are used as reference points. The remaining one point is set as a reference point (step ST101).

  Equation (14) is solved by substituting the motion speeds of the three reference points, and first-order approximated affine transformation parameters u0, v0, ux, uy, vx, vy are obtained. When the motion of the object is small and smooth, the affine transformation parameter is obtained by using the least square method using the position information of the image of the reference point in three frames or more (step ST102).

  Next, the virtual parallax of the motion speed of the virtual point interpolated by the affine transformation with the reference point is obtained. When it can be assumed that the object is sufficiently far from the camera and the translational motion U3 in the viewing direction is not large, the direction of this virtual parallax represents the direction θA of A. If not, θA = Δu / Δv is obtained by substituting Equation (13) for the virtual parallax of a plurality of points (step ST103).

  Using the equations (15), (16), (17), and (18) from the three reference points, the invariant feature amounts of curl, div, and def are obtained. These invariant feature amounts are obtained by adding the change caused by the object surface orientation and the movement on the image plane to the change caused by the translational or rotational movement of the object (step ST104).

  From the deformation main axis μ and the projection θA of the translational motion onto the image plane, the inclination τ of the plane determined by the three reference points is obtained using equation (37) (step ST105).

  From the equation (35), the expansion and contraction of the shape due to the relationship between the surface direction and the movement on the image plane is subtracted.

Using the values obtained so far, F · A = | defv | cos (τ−θA) is subtracted from Equation (35). The remaining component indicates a change in the scale of the image due to the movement of the object along the line-of-sight direction. From this, the time tc until the collision is obtained (step ST106).

  Subtract the influence of the direction of the surface and the movement on the image plane from Equation (34). When F × A = | defv | sin (τ−θA) is subtracted from Expression (34) using the values obtained so far, the remaining component is due to rotation around the line-of-sight direction between the object and the photographer. It becomes only a thing (step ST107).

  In this way, the affine flow analysis unit 3 can calculate the motion parameters of the observer that can be stably calculated from the image information, such as the translational motion direction θA, the scale change tc, and the rotation Ω · U around the line-of-sight direction. Is output to the pointing information generator 4 (step ST108).

  As described above, the pointing information generation unit 4 inputs time-series data of feature point coordinates from the feature point extraction unit 2, selects appropriate two feature point coordinates from the time point, and the affine flow analysis unit 3 selects them. The motion parameters are calculated and information necessary for indicating a three-dimensional space is generated. Hereinafter, this process will be described with reference to the flowchart 10.

  First, feature point coordinates (if there is auxiliary information, also auxiliary information) are input from the feature point extraction unit 2. The number of input feature points is n, and the coordinates are (xi, yi) (step ST201).

  Since the object to be imaged is moving, the feature points may be hidden by other portions and may not be visible, or the hidden feature points may appear. If the number of feature points is less than 4, nothing is done. If the number of feature points is 4 or more, the feature points extracted when the previous pointing information was generated and the feature points extracted this time Four feature points that are evenly positioned are selected (step ST202).

For the selected feature point, the movement distance ((xi−lxi) ² + (yi−lyi) ² ) from the coordinate value (lxi, lyi) when used for the previous pointing is calculated, and this distance is set as a certain threshold value. Compare. If there is auxiliary information such as the size of a feature, the threshold is determined using that value. If the selected feature point has not been used for pointing before, xi and yi are substituted into lxi and lyi. If there is at least one point having a distance equal to or greater than the threshold among the four moving distances, the subsequent processing is performed, and if all are equal to or less than the threshold, the process returns to step ST201 (step ST203).

  The four past coordinate values (lxi, lyi) and the current coordinate values (xi, yi) obtained in this way are input to the affine flow analysis unit 3 to calculate motion parameters (step ST204).

  The motion parameters calculated by the affine flow analysis unit 3 are parameters when it is assumed that the object is stationary and the observer (camera) is moving. When this is replaced with a value representing the motion of the object, the motion of the center of gravity is the translational motion of the object in the X and Y directions, tc representing the change in scale is the translational motion in the Z direction, and Ω · U is around the Z axis. A represents the ratio of the rotational motion around the X axis to the rotational motion around the Y axis. Each of these parameters is compared with a threshold value, and if there is a movement larger than the threshold value, a pointing event that causes the movement of the object indicated by the parameter by a certain amount is generated (step ST205). At that time, the movement direction of the pointing event is determined so that the movement of the pointer seen on the screen matches the direction of movement when a human sees his / her hand or an instrument used for pointing.

  The pointing event generated here is sent to the pointer display unit 8 and the gesture pattern matching unit 6. The motion parameters obtained by the affine flow analysis unit 3 are expressed by using the depth λ for the parameters that cannot be calculated unless the central projection is assumed, but the pointing information generation unit 4 uses the absolute motion of the object. To calculate the position and orientation by substituting the parameters expressed by relative values using λ into the central projection equation (11), and also outputs this information to the pointer display unit 6 (step ST206).

  The pointer display unit 8 is stored in the pointer model storage unit 12 in accordance with an instruction from the status switching unit 5 described later. For example, as shown in FIG. 11, a pointer 3D model whose direction is easily understood in three dimensions or a 3D model designated by the status switching unit 5 among the models stored in the 3D model storage unit 9 is selected. From the current position and orientation of the selected 3D model, graphics image information that has been translated and rotated according to the input pointing event is generated and output.

  As described above, the pointer model storage unit 12 stores the 3D model of the pointer and the current position and orientation. The 3D model storage unit 9 stores the 3D model currently displayed in the image and the position of the model. And the attitude is remembered.

  In the gesture pattern matching unit 6, a list of time series of the latest pointing events input from the pointing information generating unit 4, a pattern that is not interrupted by a keyboard input from the user, and a gesture pattern stored in the gesture pattern storage unit 7. Are compared with each other to determine whether or not the operation of the pointer by the user is a movement having some registered meaning. If there is a matching gesture pattern, the operation stored with the pattern is executed.

In the gesture pattern storage unit 7, the gesture patterns are stored in a table with a list structure as shown in FIG. One gesture is composed of a gesture pattern and a character string indicating an operation to be called when the gesture occurs. One gesture pattern is represented by a list of pointing events. One pointing event includes six translational motions {U ₁ , U ₂ , U ₃ } and rotational motions {Ω ₁ , Ω ₂ , Ω ₃ }. The parameters are represented by three types of symbols, + or − indicating that there is movement in the positive and negative directions, or 0 indicating that there is no movement. In FIG. 12, there are two successors indicating the next element of the list in the gesture pattern list, and there is one that constitutes a closed loop by itself. This is a mechanism that allows the same pointing event to be repeated in this closed loop. It is. The variable n next to the sucessor indicates that all four closed loops repeat the same number of times. In the example of FIG. 12, the gesture pattern indicates a square of an arbitrary size on the xy plane, and indicates that the operation / usr / bin / X11 / kt is activated by this gesture.

  The status switching unit 5 is operated by freely moving the pointer in the three-dimensional space displayed on the display. After the pointer operation state or one of the displayed models is designated by the pointer, the position of the model or Stores either the current pointer state of the model grasping state for which the posture is to be changed, and switches the pointer state according to an instruction from the user or an instruction from the pointer display unit. Change the settings.

  In the pointer operation state, the pointer display unit 8 is instructed to use the model stored in the pointer model storage unit, and the position and orientation of the pointer model are changed according to the generated pointing event. The pointing event is also input to the gesture pattern matching unit, the gesture is recognized, and if the gesture is included in the event sequence, the operation corresponding to the gesture is executed. The pointer state is switched to the model grasping state when an instruction by a user's keyboard input or gesture or the three-dimensional position of the pointer matches one position of the 3D model stored in the 3D model storage unit.

  In the model grasping state, the model stored in the 3D model storage unit 9 is displayed with the position and orientation changed. First, when entering the model grasping state, the instructed 3D model is taken out from the 3D model storage unit 9 and sent to the pointer display unit 8 to instruct it to be displayed in a different color so that it can be distinguished from other models. To do. Next, the model position and shape are input to the image composition unit 10, and the model is grasped by hand from the image information of the 3D model and the input image information, or the 3D model is inserted into a pointing device. The images that appear to flutter are combined and displayed on the image display unit 11. For the movement and rotation of the model, unlike the pointer operation state, information on the position and orientation calculated by applying to the central projection is used instead of the pointing event.

  In the image composition unit 10, first, a closed region surrounding a feature point is cut out from the image information input from the image input unit 1, thereby extracting a portion where a hand or a tool is shown. Next, the left and right sides of the extracted image are reversed so that the movement of the hand or instrument matches the actual movement and the movement direction on the image when viewed from the user. Based on the input information such as the position and shape of the 3D model and the coordinates of the feature point, processing such as parallel movement and scale change is performed on the image of the hand or the instrument, and the position of the feature point is converted to the 3D model graphics image Fit to the vertex. Thereafter, the graphics image of the model and the image of the hand or instrument are displayed in a translucent manner, and an image that seems to hold the model is synthesized and output to the image display unit 11 as shown in FIG.

  Another embodiment of the present invention will be described with reference to FIG.

  The moving image input unit 21 captures an image while the camera (observer) performs an unknown motion in the three-dimensional space, and transfers image information corresponding to the captured moving image to the feature point extraction unit 22. The shooting environment in which the moving image input unit 21 is shooting is basically a stationary environment, but may include a moving object.

  The feature point extraction unit 22 receives the time-series image information from the moving image input unit 21, and the image processing rapidly changes the brightness and color from the neighboring region, and the same point of the object is projected on the two images. A number of feature points that can be identified as being extracted are extracted, and the extracted feature points are input to the translation flow calculation unit 23.

  The translation flow calculation unit 23 compares the coordinates of the input feature points to form a network connecting the nearest four points, and an affine flow analysis unit for all combinations of the four nearest points A process similar to the process performed by 3 is performed to obtain a virtual parallax, and the motion parameter of the observer is calculated. Looking at the translational motion direction θA among the calculated motion parameters, this value indicates the direction in which the camera can translate relative to the imaging environment. In a static environment, any combination of four feature points shows the same value. Actually, since the center projection is limited to a narrow visual field range and approximated by affine transformation, the same value is shown by a combination of feature points in the vicinity. Therefore, only the translational motion direction θA is extracted from the combination of feature points distributed over the entire image, and a distribution diagram as shown in FIG. 15 is created and output. In addition, the arrow of FIG. 15 has shown the motion of the some point of each of two objects.

  The independent motion dividing unit 24 compares the translational motion direction θA of the combination of neighboring feature points in the translational motion direction flow diagram calculated by the translational flow calculation unit 23, and divides the region when the difference is larger than a certain threshold. To do. As a result, it is possible to extract a region indicating an object that is moving differently from the moving background image, such as a region surrounded by a solid line in FIG. Thereafter, the regions that are moving differently are removed, and the motion parameters other than θA are calculated from the remaining region indicating the background image, and the motion of the camera is obtained and output.

  Another embodiment of the present invention will be described with reference to FIG.

  The image input unit 31 inputs image information corresponding to an image obtained by photographing one object from a plurality of directions. The image information image input here may not be temporally continuous. In addition, the positional relationship of the observer when the object is photographed is unknown.

  The feature point extraction unit 32 performs image processing such as point extraction processing on the image information input from the image input unit 31, and corresponds to a large number of feature points whose brightness and color are rapidly changing from neighboring regions. Feature point information is extracted and output to the corresponding feature search unit 33. The extracted feature point information is superimposed on the input image information and displayed on the image display unit 36.

  The initial correspondence search unit 33 compares the feature point information among a plurality of images, and checks whether the same point on the object to be imaged is projected. First, a flag indicating non-correspondence is attached to all feature point information input from the feature point extraction unit 32. Next, correlation matching of small regions centering on feature points is performed between images, and feature point information having a correlation coefficient higher than a certain threshold is associated with each other, and the corresponding feature point information is indicated as corresponding. The flag is attached, and the feature point set information is output to the correspondence correction unit, that is, the correspondence feature update unit 37. The corresponding feature point information is displayed on the image display unit 36 so as to be superposed on the input image information so that it can be distinguished from unsupported feature point information by changing the color. It is displayed so that it can be understood which point information corresponds to which feature point information.

  The interface unit 34 receives the feature point set information including the feature points with the correspondence from the initial correspondence search unit 33, and corrects the correspondence between the feature points. If the correspondence created by the initial correspondence search unit 32 is sufficiently accurate and has few errors, it is possible to perform later processing without correcting the corresponding points.

  As a result of the initial correspondence search, a cursor controlled by a pointing device such as a mouse is displayed on the input image in which the feature points displayed on the image display unit 36 are displayed so that the user can select the feature points. When a feature point already having a corresponding flag is selected, the feature point and the corresponding feature point flag are changed to unsupported, and the corresponding relationship is canceled. When feature point information with an unsupported flag is selected one by one in a plurality of input image information, a flag indicating that correspondence has been established is attached to the feature point information, and those feature points Set correspondence between information. Also, even in an area without feature points, if there are pixels selected one by one in a plurality of images one after another, feature points whose correspondence is determined at the coordinates of the pixels are generated and the correspondence relationship is set. In addition to images and feature points, buttons for updating corresponding points, extracting the structure of an object, and synthesizing an image are displayed so that these can be selected by a pointing device. When the user selects the corresponding point update button, the feature point set information is passed to the corresponding feature update unit 37 to update the feature point correspondence. When the object structure extraction button is selected, a set of feature points is passed to the structure extraction unit to extract the three-dimensional structure of the photographed object. When the image composition button is selected, it is further asked where the image to be synthesized is to be synthesized, and the translational and rotational motion vectors up to the viewpoint for synthesizing the image are input and input to the image composition unit 35. The image, the set of feature points, the three-dimensional structure of the object extracted by the structure extraction unit, and the motion vector are passed, and the images are combined and displayed on the image display unit.

  The image display unit 36 displays a plurality of input images, feature points, a 3D model of the object structure extracted from the image, an image synthesized by changing the direction of the line of sight, and the cursor is overlaid thereon to display the coordinates on the image. And feature points can be specified.

  Corresponding feature update unit 37 receives feature point set information from interface unit 34 and associates unsupported feature points according to a new standard. First, from the inputted feature point set information, correspondence is determined, a feature point with a correspondence flag (this is designated as point A0) is selected, and a feature with an unsupported flag in the vicinity of the feature point A0 is selected. Point B is selected. The feature point B ′ corresponding to the feature point A0 in the vicinity of the feature point A0 ′ in another image is compared with the feature point B to determine whether or not they correspond. In comparison between B and B ′, if there are two or less corresponding feature points in the vicinity of both, as in the initial correspondence search, correlation matching of small regions centering on the feature points is performed, and the initial correspondence search is performed. It is determined whether or not the threshold is lower than the threshold used in the unit 33. As shown in FIG. 17, if there are three or more feature points corresponding to the vicinity of both B and B ′, matching is performed in consideration of image deformation. First, an affine transformation that transforms the triangles A0, A1, and A2 into triangles A0 ', A1', and A2 'is calculated. The small area centered on B is transformed by the calculated affine transformation, and correlation matching is performed between the deformed small area and the small area centered on B ′, and there are no more than two corresponding points in the vicinity. Judgment is made with the same threshold as in the case of no. Like the initial correspondence search unit 33, the feature points found in this way are attached with a flag indicating the correspondence to the feature points, and the feature points are connected in a correspondence relationship. This process is repeated to return to the interface unit 34 until there are no more corresponding feature points.

The structure extraction unit 38 receives a plurality of pieces of image information obtained by photographing an object of unknown shape and a set of feature points associated with these pieces of image information, and from these pieces of information, a three-dimensional shape model of the object Extract the texture pattern of the object surface and output it. First, a combination of four neighboring points is extracted from the feature point set information, and the four parameters are processed by the affine flow analysis unit 3 to calculate motion parameters. These four points are divided into three points constituting the triangle and one other point P as shown in FIG. The point P ′ is a virtual point that is in the same coordinates as the point P in one image and moves in the other image by affine transformation expressed by the movement of the other three points. This point P ′ is on a three-dimensional plane determined by three points in the three-dimensional space, and indicates a point projected on the same coordinates as the point P on the image plane. The size of this virtual parallax PP ′ is given by equation (12):

It is represented by However, Zp is the coordinate of the point p, and Zp1 is the Z coordinate of the point P ′. Here, the situation of FIG. 18 is assumed in which another feature point Q in the vicinity of the four points is added. Like the point P ′, the point Q ′ is a virtual point that moves by affine transformation determined by the movement of the other three points, and is also a point on a three-dimensional plane that is determined by three points in the three-dimensional space. Considering the two virtual parallaxes PP ′ and QQ ′, since they are close in the image and the distance from the observer to the object is sufficiently long, the ratio of the lengths of the two virtual parallaxes is (ZP −ZP ′) / (ZQ−ZQ ′), from which the ratio of the coordinate in the depth direction to the three-dimensional plane with P and Q can be obtained. This process is performed on combinations of feature points in all the vicinity to calculate a 3D model of the object in which the length in the depth direction of the object is expressed as a ratio to a certain length λ. After this, the 3D model of the object corresponding to a certain λ is displayed graphically, and the user adjusts the value of λ while looking at the 3D model, or finds the part where the plane intersects on the surface of the 3D model, A complete 3D model of the object is calculated, such as estimating the magnitude of λ so that the intersection angle is a right angle. When the magnitude of λ is calculated, the gradient of the triangular object surface surrounded by the three feature points in the vicinity on the image is obtained from the equation (31). Texture information can also be extracted.

  The structure extraction unit 38 outputs the 3D model and texture information of the object calculated in this way.

  The image synthesizer 35 deforms the input image using the three-dimensional structure of the input object and the motion vector to the viewpoint, and synthesizes an image when the object is viewed from a new viewpoint. The deformation of the image when the observer moves is expressed by the translational motion vector and the rotational motion vector of the observer, the gradient of the object surface, and the translational motion on the image plane by Expressions (34) to (36). . The motion of the observer and the translational motion on the image plane are calculated from the motion vector to the viewpoint, and the gradient of the object surface is obtained from the three-dimensional structure of the object. An affine transformation matrix representing the deformation can be calculated. First, an area where the target object is shown in the input image is divided into triangular patches separated by a straight line connecting feature points in the area. The above affine transformation is applied to each triangular patch image to create a new triangular patch image. An image of an object when viewed from a new line-of-sight direction is formed by joining the created patch images, and this is displayed on the image display unit.

1 is a block diagram of a moving image processing apparatus which is an embodiment of the present invention and is used for a human interface such as a model operation in a CAD system. FIG. The figure of the instrument of the 1st example used for pointing. The figure of the instrument of the 2nd example used for pointing. The figure of the instrument of the 3rd example used for pointing. The figure which shows the relationship between the three-dimensional coordinate and image coordinate in the imaging form by center projection. The figure which shows the invariant feature-value which can be extracted from an image. The figure explaining motion parallax. The figure explaining the expression method of the inclination of an object surface. The flowchart figure explaining operation | movement of an affine flow analysis part. The flowchart figure explaining operation | movement of a pointing information generation part. The figure which shows the 3D (three-dimensional) model of a pointer. The figure which shows the data structure of the gesture pattern memorize | stored in the gesture pattern memory | storage part. The figure which shows the image which overlap | superposed and superposed | combined a part of input image with a 3D model. FIG. 5 is a block diagram of a system for discriminating independent movement according to another embodiment of the present invention. Distribution diagram of translational motion direction. The block diagram of the moving-image processing apparatus applied to the system which is another Example of this invention, acquires a 3D model from several images, and synthesize | combines the image when an object is seen from another direction. The figure which shows the matching which considered the image deformation | transformation by affine transformation. The figure which shows the relationship between two virtual parallaxes.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Image input part, 2 ... Feature point extraction part, 3 ... Affine flow analysis part, 4 ... Pointing information generation part, 5 ... Status switching part, 6 ... Gesture pattern matching part, 7 ... Gesture pattern storage part, 8 ... Pointer Display unit, 9 ... 3D model storage unit, 10 ... Image composition unit, 11 ... Image display unit, 12 ... Pointer model storage unit, 21 ... Moving image input unit, 22 ... Feature point extraction unit, 23 ... Translation flow calculation unit, 24 ... Independent motion division unit, 31 ... Image input unit, 32 ... Feature point extraction unit, 33 ... Initial correspondence search unit, 34 ... Interface unit, 35 ... Image composition unit, 36 ... Image display unit, 37 ... Corresponding feature update 38, structure extraction unit

Claims (2)

Image input means for inputting image information corresponding to a plurality of images, movement information estimation means for estimating movement information on the observing side that captures images between the plurality of images, and a pointing event is generated by the estimated movement information And a pointer display means for displaying a 3D model of an object stored in advance by translational rotation according to the pointing information. A step of inputting image information corresponding to a plurality of images, a step of estimating motion information on the observation side that images images between the plurality of images, a step of generating a pointing event based on the estimated motion information, A moving image processing method including a step of displaying a 3D model of a stored object by translation and rotation according to pointing information.

Images