[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

EP2078419A2 - Segment tracking in motion picture - Google Patents

Segment tracking in motion picture

Info

Publication number
EP2078419A2
EP2078419A2 EP07844821A EP07844821A EP2078419A2 EP 2078419 A2 EP2078419 A2 EP 2078419A2 EP 07844821 A EP07844821 A EP 07844821A EP 07844821 A EP07844821 A EP 07844821A EP 2078419 A2 EP2078419 A2 EP 2078419A2
Authority
EP
European Patent Office
Prior art keywords
sequence
markers
pattern
actor
known pattern
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP07844821A
Other languages
German (de)
French (fr)
Other versions
EP2078419A4 (en
EP2078419B1 (en
Inventor
Demian Gordon
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Sony Pictures Entertainment Inc
Original Assignee
Sony Corp
Sony Pictures Entertainment Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp, Sony Pictures Entertainment Inc filed Critical Sony Corp
Publication of EP2078419A2 publication Critical patent/EP2078419A2/en
Publication of EP2078419A4 publication Critical patent/EP2078419A4/en
Application granted granted Critical
Publication of EP2078419B1 publication Critical patent/EP2078419B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/20Analysis of motion
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/20Analysis of motion
    • G06T7/246Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/251Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving models
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/20Analysis of motion
    • G06T7/246Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30196Human being; Person
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30241Trajectory

Definitions

  • the present invention relates generally to motion capture, and more particularly to segment tracking using motion marker data.
  • Motion capture systems are used to capture the movement of an actor or object and map it onto a computer-generated actor/object as a way of animating it. These systems are often used in the production of motion pictures and video games for creating a digital representation of an actor or object for use as source data to create a computer graphics ("CG") animation.
  • CG computer graphics
  • an actor wears a suit having markers attached at various locations (e.g., small reflective markers are attached to the body and limbs) .
  • Appropriately placed digital cameras then record the actor' s body movements in a capture volume from different angles while the markers are illuminated.
  • the system later analyzes the images to determine the locations (e.g., spatial coordinates) of the markers on the actor's suit in each frame.
  • the system By tracking the locations of the markers, the system creates a spatial representation of the markers over time and builds a digital representation of the actor in motion. The motion is then applied to a digital model in virtual space, which may be textured and rendered to produce a complete CG representation of the actor and/or the performance. This technique has been used by special effects companies to produce realistic animations in many popular movies.
  • Certain implementations as disclosed herein provide for methods, systems, and computer programs for providing segment tracking in motion capture.
  • a method as disclosed herein provides for segment tracking.
  • the method includes: applying a marking material having a known pattern to a surface; acquiring a sequence of image frames, each image frame of the sequence including a plurality of images of the known pattern covering the surface; deriving position and orientation information regarding the known pattern for each image frame of the sequence; and generating animation data incorporating the position and orientation information.
  • a system for segment tracking includes: an image acquisition module configured to generate a sequence of image frames, each image frame including a plurality of synchronized images of a known pattern disposed on a surface; and a segment tracking module configured to receive the sequence of image frames and generate animation data based on the known pattern disposed on the surface.
  • Figure 1 is a block diagram of a motion capture system in accordance with one implementation
  • Figure 2 shows a sample collection of markers according to an implementation of the present invention
  • Figure 3 is an illustration of a human figure with marker placement positions according to one implementation
  • Figure 4 presents a frontal view of a human body model equipped with markers
  • Figure 5 depicts a quartering back view of a human body model configured with markers
  • Figure 6 is a flowchart describing a method of segment tracking
  • Figure 7A illustrates a marker labeled or patterned to represent a capital letter A
  • Figure 7B illustrates a known pattern comprising a plurality of markers forming the letter A as a pattern
  • Figure 7C illustrates a marker comprising a literal expression of the letter A
  • Figure 8 is a flowchart depicting an example method of utilizing strips of marking material
  • Figure 9 is a functional block diagram of an implementation of a segment tracking system
  • Figure 1OB is a functional block diagram illustrating the computer system hosting the segment tracking system.
  • Figure 11 is a functional block diagram illustrating an implementation of a segment tracking module.
  • Certain implementations as disclosed herein provide segment tracking in motion capture. Implementations include using one or more known patterns of markers applied on actors and/or objects.
  • the marker (s) comprising a pattern are tracked as a group rather than individually.
  • the pattern can provide information, such as identification, position/translation, and orientation/rotation, which significantly aids marker tracking.
  • markers encoded with known patterns are applied to actors and/or objects, normally covering various surfaces of the actor and/or object. Identification, position/translation, and orientation/rotation of the actor and/or object are obtained by recording and digitizing images of the patterns.
  • the patterns of markers can be selected to mitigate "missing-marker" effects in motion capture during marker occlusions.
  • identifiable random patterns are used as "known patterns.”
  • the known (and random) patterns may be generated, for example, using materials including quantum nanodots, glow- in-the dark (i.e., fluorescent) material, tattoos, and virtually any visible, infra-red, or ultra-violet ink, paint, or material which can be applied in a sufficiently identifiable pattern.
  • the patterns may also include patterns of inherent features (e.g., moles or wrinkles) of the actor and/or objects.
  • a pattern comprises a plurality of markers or features coupled to the actor's body.
  • a pattern comprises a single marker (e.g., a marker strip) or feature.
  • the pattern can be applied or affixed on or around the surfaces of an actor' s limbs, hands, and feet.
  • centroid information relating to the pattern can be derived from the circular disposition of a pattern strip wrapped around a limb (i.e., appendage) .
  • FIG. 1 is a functional block diagram of a motion capture system 100 in accordance with one implementation.
  • the motion capture system 100 includes a motion capture processor 110, motion capture cameras 120, 122, 124, a user workstation 130, and an actor's body 140 and face 150 appropriately equipped with marker material 160 in a predetermined pattern. Although Figure 1 shows only thirteen markers 160A-160F, substantially more markers can be used on the body 140 and face 150.
  • the motion capture processor 110 is connected to the workstation 130 by wire or wirelessly.
  • the motion capture processor 110 is typically configured to receive control data packets from the workstation 130.
  • three motion capture cameras 120, 122, 124 are connected to the motion capture processor 110. Generally more than three motion capture cameras are required according to a variety of user- and animation- related needs and requirements.
  • the motion capture cameras 120, 122, 124 are focused on the actor's body 140 and face 150, on which markers 160A-160F have been applied.
  • markers 160A-160F The placement of the markers 160A-160F is configured to capture motions of interest including, for example, the body 140, face 150, hands 170, arms 172, legs 174, 178, and feet 176 of the actor.
  • markers 160A capture motions of the face 150
  • markers 160B capture motions of the arms 172
  • markers 160C capture motions of the body 140
  • markers 160D, 160E capture motions of the legs 174
  • markers 160F capture motions of the feet 176.
  • uniqueness of the patterns on the markers 160A-160F provide information that can be used to obtain identification and orientation of the markers.
  • the marker 160D is configured as a strip of pattern wrapped around a leg of the actor.
  • the motion capture cameras 120, 122, 124 are controlled by the motion capture processor 110 to capture synchronous sequences of two-dimensional ("2-D") images of the markers.
  • the synchronous images are integrated into image frames, each image frame representing one frame of a temporal sequence of image frames. That is, each individual image frame comprises an integrated plurality of simultaneously acquired 2-D images, each 2-D image generated by an individual motion capture camera 120, 122, or 124.
  • the 2-D images thus captured may typically be stored, or viewed in real-time at the user workstation 130, or both.
  • each individual marker data point in a first volumetric frame corresponds to a single marker placed on an actor's body 140.
  • a unique label is assigned to each such marker data point of the first volumetric frame.
  • the marker data points are then associated with corresponding marker data points in a second volumetric frame, and the unique labels for the marker data points of the first volumetric frame are assigned to the corresponding marker data points of the second volumetric frame.
  • the labeling (i.e., tracking) process is completed for the entire volumetric frame sequence, the marker data points of the first volumetric frame are thus traceable through the sequence, resulting in an individual trajectory for each marker data point.
  • Discrete markers are conventionally used to capture the motion of rigid objects or segments of an object or body.
  • rigid markers attached at an elbow and a wrist define the positions of each end of a forearm.
  • the motion of the forearm is thus modeled as a rigid body (e.g., a rod) with only the ends defined by the elbow and wrist markers.
  • a common twisting motion of the forearm is difficult to detect because a twist can be performed without substantially moving the wrist or elbow.
  • markers disposed in a pattern are used, allowing the motion capture processor 110 to track the pattern as a group rather than individually tracking markers. Because the pattern provides identification information, movement of the markers of one pattern with respect to another pattern can be computed. In one implementation, a pattern tracked in this way is reconstructed in each volumetric frame as an individual object having spatial position information. The object is tracked through the sequence of volumetric frames, yielding a virtual animation representing the various spatial translations, rotations, and twists, for example, of the part of the actor to which the pattern is applied.
  • one or more known patterns are printed onto strips 160D.
  • the strips 160D are then wrapped around each limb (i.e., appendage) of an actor such that each limb has at least two strips.
  • two strips 160D are depicted in Figure 1, wrapped around the actor's left thigh 178.
  • End effectors e.g., hands, feet, head
  • the printed patterns of the wrapped strips 160D enable the motion capture processor 110 to track the position and orientation of each "segment" representing an actor's limb from any angle, with as few as only one marker on a segment being visible.
  • the actor's thigh 178 is treated as a segment at the motion capture processor 110.
  • the "centroid" of the limb i.e., segment
  • a centroid may be determined to provide an estimate or model of the bone within the limb. Further, it is possible to determine orientation, translation and rotation information regarding the entire segment from one (or more if visible) markers and/or strips applied on the segment.
  • the motion capture processor 110 performs segment tracking according to techniques disclosed herein from which identification, positioning/translation, and orientation/rotation information is generated for a group of markers (or marked areas) . While a conventional optical motion capture system typically records only position for a marker, segment tracking enables the motion capture processor 110 to identify which marker (s) are being captured and to locate the position and orientation of the captured markers of the segment. Once the markers are detected and identified, position and orientation/rotation information regarding the segment can be derived from the identified markers. Confidence in the determinations of position and orientation information for the segment increases as more markers are detected and identified. Markers or marking material applied in known patterns (and identifiable random patterns) essentially encode identification and orientation information facilitating efficient segment tracking.
  • a known pattern of smaller markers constitutes a single marker or marking area.
  • a marker labeled or patterned as representing a capital letter A actually comprises six smaller markers A, B, C, D, E and F.
  • a known pattern comprises a plurality of markers or marking areas forming the letter A as a pattern, as shown in Figure 7B.
  • the eight dots forming the letter A function as a single, larger marker.
  • a marker can comprise a literal expression of the letter A, as shown in Figure 7C.
  • a common characteristic to the markers illustrated in Figures 7A-C is that they all have an identifiable orientation.
  • the marker can be rotated, but will remain identifiable as a letter A, which significantly enhances marker tracking efficiency because it will be readily trackable from frame to frame of captured image data.
  • an orientation may also be determined for each of the marker examples from frame to frame. If a marker 160B (see Figure 1) is coupled to an actor' s forearm, for example, it will therefore rotate substantially 180 degrees when the actor moves the forearm from a hanging position to a straight-up, overhead reaching position. Tracking the marker will reveal not only the spatial translation to the overhead position, but that the forearm segment is oriented 180 degrees differently from before. Thus, marker tracking is enhanced and improved using markers or groups of markers encoding identification and orientation information.
  • FIG. 2 shows a sample collection of markers according to an implementation of the present invention.
  • Each marker comprises a 6x6 matrix of small white and black squares.
  • Identification and orientation information is encoded in each marker by a unique placement of white squares within the 6x6 matrix. In each case, rotating the marker causes no ambiguity in terms of determining the identity and orientation of the marker, thus demonstrating the effectiveness of this scheme for encoding information. It will be appreciated that encoding schemes using arrangements other than the 6x6 matrix of black and white elements disclosed herein by example may also be implemented.
  • FIG 3 is an illustration of a human figure with marker placement positions according to one implementation.
  • the markers shown encode identification and orientation information using a scheme similar to that depicted in Figure 2. They are positioned substantially symmetrically, and such that each major extremity (i.e., segment) of the body is defined by at least one marker. Approximately half of the markers depicted are positioned on a surface of the body not visible in the frontal view shown, and instead include arrows pointing to their approximate occluded positions.
  • motion capture cameras 120, 122, 124 encompass typically a much larger plurality of cameras encompass a capture space in which the actor's body 140 and face 150 are in motion.
  • Figure 4 presents a frontal view of a human body model equipped with markers as described in Figure 3. As shown, only the markers on the forward-facing surfaces of the model are visible. The rest of the markers are partially or fully occluded.
  • Figure 5 depicts a quartering back view of the same human body model in substantially the same pose as shown in Figure 4. From this view, the frontally-placed markers are occluded, but many of the markers occluded in Figure 4 are now visible. Thus, at any given time, substantially all of the markers are visible to some subset of the plurality of motion capture cameras 120, 122, 124 placed about the capture space.
  • the marker placements on the 3-D model depicted in Figures 4 and 5 substantially define the major extremities (segments) and areas on the body that articulate motions (e.g., the head, shoulders, hips, ankles, etc.) .
  • the positions of the body on which the markers are placed will be locatable and their orientations determinable .
  • the segments of the body defined by the marker placements e.g., an upper arm segment between an elbow and a shoulder, will also be locatable because of the markers placed substantially at each end of that segment.
  • the position and orientation of the upper arm segment will also be determinable from the orientations derived from the individual markers defining the upper arm.
  • Figure 6 is a flowchart describing a method of segment tracking 600 according to an implementation.
  • a marking material with a known pattern, or an identifiable random pattern is applied to a surface, at 610.
  • the surface is that of an actor' s body
  • a pattern comprises a plurality of markers that is coupled to the actor's body.
  • a pattern comprises a single marker (e.g., a marker strip) that is coupled to the actor's body.
  • the pattern may also be formed as a strip 160D and affixed around the actor's limbs, hands, and feet, as discussed in relation to Figure 1.
  • Markers also include reflective spheres, tattoos glued on an actor's body, material painted on an actor's body, or inherent features (e.g., moles or wrinkles) of an actor.
  • patterns of markers can be applied to the actor using temporary tattoos .
  • a sequence of image frames is acquired next, at 620, according to methods and systems for motion capture described herein, and above in relation to Figure 1.
  • the image data are used to reconstruct a 3-D model of an actor or object equipped with the markers, for example.
  • this includes deriving position and orientation information of the marker pattern for each image frame, at 630.
  • Position information is facilitated by the unique identification information encoded in the markers provided by the pattern using characteristic rotational invariance, as discussed with respect to Figures 4 and 5.
  • Orientation information may be derived by determining the amount of rotation of the markers comprising the pattern. Marker rotations, and orientation in general, may be defined by affine representations in the 3-D space facilitating marker tracking.
  • an orientation can be represented by values representing the six degrees of freedom ("6DOF") of an object in 3-D space.
  • the orientation can have three values representing any displacement in position (translation) from the origin of the coordinate system (e.g., a Euclidean space), and three values representing angular displacements (rotations) relative to the primary axes of the coordinate system.
  • Animation data based on the movements of the markers are generated, at 640.
  • a virtual digital model is to be activated by the data derived from the movements of an actor captured in the image data.
  • the actor swings the leg according to a script.
  • the actor's legs are equipped with markers at each major joint, i.e., the hip, knee, and ankle, for instance.
  • the movements of the markers are determined and used to define the movements of the segments of the actor's leg.
  • the segments of the actor's leg correspond to the segments of the leg of the digital model, and the movements of the segments of the actor' s leg are therefore mapped to the leg of the virtual digital model to animate it.
  • a pattern may be formed as a strip 160D (see Figure 1) and affixed around an actor's limbs, hands, and feet. A centroid may then be determined from the circular disposition of the strip wrapped around the limb. Two such strips positioned at each end of the limb may then be used to determine a longitudinal centroid relative to the limb (segment) , approximating an underlying skeletal element, for example.
  • a flowchart depicting a method of utilizing strips of marking material 800 is provided in Figure 8. Strips of marking material with known patterns are applied to one of an actor's limbs, at 810, typically by wrapping it around the limb.
  • the wrapped strip of marking material is disposed in a substantially circular shape, centroids of the strips may be determined and used to express the movements of the body area wrapped with the strip. Further, the motion of a skeletal element of the limb can thus be approximated and used for skeleton modeling.
  • a sequence of image frames is acquired next, at 820, according to methods and systems for motion capture described above in relation to Figure 1.
  • the image data are used to reconstruct a 3-D model of an actor equipped with the markers, for example, in essentially the same manner as described in Figure 6, with respect to 620.
  • this includes deriving position and orientation information of the known pattern for each strip in each image frame, at 830, in essentially the same manner as described in Figure 6, with respect to 630.
  • Centroid information is derived for the actor' s limb equipped with one or more strips from the position and orientation information, at 840.
  • the centroids of two or more strips can be used to approximate the skeletal structure (i.e., a bone) within the actor's limb.
  • the movements of the centroids of the marker strips are then used to express the movements of the skeletal structure, and thus the movement of the limb.
  • Animation data based on the movements of the marker strips is generated, at 850.
  • the actor swings his leg according to a script.
  • the actor' s legs are equipped with strips of marker material wrapped around as described above. That is, for instance, a strip may be wrapped around the upper thigh and another wrapped around near the knee.
  • the movements of the centroids of the marker strips are determined and used to define the skeletal structure of the upper leg, and consequently a skeletal element within the upper leg of a virtual digital model corresponding to the actor.
  • the movements of the centroids can therefore used to animate the leg of the virtual digital model.
  • the markers are printed or formed using quantum nanodots (also known as "Quantum Dots," or "QDs”) .
  • QDs quantum nanodots
  • the system would be similar in configuration to the traditional retro-reflective system, but with the addition of an exciter light source of a specific frequency (e.g., from existing filtered ring lights or from another source) and narrow band gap filters placed behind the lenses of the existing cameras, thus tuning the cameras to the wavelength of light emitted by the QDs.
  • QDs are configured to be excited by light of a particular wavelength, causing them to emit light (i.e., fluoresce) at a different wavelength. Because they can emit light that has been quantum shifted up the spectrum from the excitation wavelength, the excitation light can be filtered from the cameras. This causes any light that falls outside of the particular emission spectrum for the QDs to be substantially blocked at the camera.
  • the QDs can be tuned to virtually any wavelength in the visible or invisible spectrum. This means that a group of cameras can be filtered to only see a specific group of markers and nothing else, significantly cutting down the workload required of a given camera, and allow the camera to work "wide open” in the narrow response range of the QDs.
  • the quantum nanodots are added to a medium such as ink, paint, plastic, temporary tattoo blanks, etc.
  • Figure 9 is a functional block diagram of an implementation of a segment tracking system 900.
  • An image acquisition module 910 generates image frames of motion capture image data, and a segment tracking module 920 receives image frames and generates animation data.
  • the image acquisition module 910 operates according to methods discussed above relating to the motion capture system 100 described in Figure 1.
  • the image frames comprise volumetric data including untracked marker data. That is, marker data exist in each frame as unlabeled spatial data, unintegrated with the marker data of the other frames.
  • the segment tracking module 920 operates according to methods and schemes described in relation to Figure 2 through Figure 9 above.
  • Figure 11 is a functional block diagram depicting an example configuration of a segment tracking module 920.
  • the segment tracking module 920 includes an identification module 1100, an orientation module 1110, a tracking module 1120, and a animation module 1130.
  • the identification module 1100 includes capability to identify markers having known patterns, such as the markers 160A-160F shown in Figure 1. In one implementation, the identification module 1100 performs pattern matching to locate known and random patterns of markers 160 from frame to frame. In another implementation, the identification module 1100 is configured to identify a single larger marker comprising a group of smaller markers, as discussed in relation to Figures 7A-7C. A marker 160 is designed to be identifiable regardless of its rotational state, and the identification module 1100 is configured accordingly to perform rotationally invariant identification of the markers 160. Once the markers 160 are uniquely identified, they are appropriately associated with the particular part of the actor's body, for example, to which it is coupled.
  • a marker 160B is associated with the actor's upper arm 172, markers 160D with the upper and lower thigh 178 (marker strips wrapped around the leg segment) , and markers 160C associated with the torso.
  • the identification information is then passed to the orientation module 1110 and the tracking module 1120.
  • the orientation module 1110 receives the identification information and generates orientation information.
  • the orientation information comprises 6DOF data.
  • the marker 160 is then analyzed in each frame to determine its evolving positional and rotational (i.e., affine) state.
  • affine evolving positional and rotational
  • a 3-D affine transformation is developed describing the changes in these states from frame to frame for each marker 160.
  • strip markers 160D are wrapped around each end of a segment of an actor's limb, such as the thigh 178 (see Figure 1), a centroid may be determined for that limb segment approximating the underlying skeletal structure (i.e., bone) .
  • the orientation information (6DOF, affine transformation) is passed to the animation module 1130.
  • the tracking module 1120 receives identification information and generates marker trajectory information. That is, the markers are tracked through the sequence of image frames, labeled, and a trajectory is determined.
  • labeling and FACS approaches are employed according to U.S. Patent Application 11/467,503, filed August 25, 2006, entitled “Labeling Used in Motion Capture", and U.S. Patent Application 11/829,711, filed July 27, 2007, entitled “FACS Cleaning in Motion Capture.”
  • the labeling information i.e., trajectory data
  • the labeling information i.e., trajectory data
  • the animation module 1130 receives orientation and labeling information and generates animation data.
  • marker positions and orientations are mapped to positions on a (virtual) digital character model corresponding to the positions on the actor at which the markers were coupled.
  • a segment defined by markers on the body of the actor is mapped to a corresponding part on the digital character model.
  • the movement of the centroid approximating the skeletal structure of the actor' s limb also models the movement of that part of the actor's body.
  • the transformations describing the movements of the markers 160 and related centroids are then appropriately formatted and generated as animation data for animating a corresponding segment of the digital character.
  • the animation module 1130 receives the animation information and applies it to the digital character, resulting in its animation.
  • the animation is typically examined for fidelity to the actor's movements, and for purposes of determining whether any, an how much, reprocessing may be required to obtain a desired result.
  • Figure 1OA illustrates a representation of a computer system 1000 and a user 1002.
  • the user 1002 uses the computer system 1000 to perform segment tracking.
  • the computer system 1000 stores and executes a segment tracking system 1090, which processes image frame data.
  • Figure 1OB is a functional block diagram illustrating the computer system 1000 hosting the segment tracking system 1090.
  • the controller 1010 is a programmable processor and controls the operation of the computer system 1000 and its components.
  • the controller 1010 loads instructions (e.g., in the form of a computer program) from the memory 1020 or an embedded controller memory (not shown) and executes these instructions to control the system.
  • the controller 1010 provides the segment tracking system 1090 as a software system. Alternatively, this service can be implemented as separate components in the controller 1010 or the computer system 1000.
  • Memory 1020 stores data temporarily for use by the other components of the computer system 1000.
  • memory 1020 is implemented as RAM.
  • memory 1020 also includes long-term or permanent memory, such as flash memory and/or ROM.
  • Storage 1030 stores data temporarily or long term for use by other components of the computer system 1000, such as for storing data used by the segment tracking system 1090.
  • storage 1030 is a hard disk drive.
  • the media device 1040 receives removable media and reads and/or writes data to the inserted media.
  • the media device 1040 is an optical disc drive.
  • the user interface 1050 includes components for accepting user input from the user of the computer system 1000 and presenting information to the user.
  • the user interface 1050 includes a keyboard, a mouse, audio speakers, and a display.
  • the controller 1010 uses input from the user to adjust the operation of the computer system 1000.
  • the I/O interface 1060 includes one or more I/O ports to connect to corresponding I/O devices, such as external storage or supplemental devices (e.g., a printer or a PDA) .
  • the ports of the I/O interface 660 include ports such as: USB ports, PCMCIA ports, serial ports, and/or parallel ports.
  • the I/O interface 1060 includes a wireless interface for communication with external devices wirelessly.
  • the network interface 1070 includes a wired and/or wireless network connection, such as an RJ-45 or "Wi-Fi" interface (including, but not limited to 802.11) supporting an Ethernet connection.
  • the computer system 1000 includes additional hardware and software typical of computer systems (e.g., power, cooling, operating system) , though these components are not specifically shown in Figure 1OB for simplicity. In other implementations, different configurations of the computer system can be used (e.g., different bus or storage configurations or a multi-processor configuration) .

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)
  • Image Analysis (AREA)

Abstract

Segment tracking in motion picture, including: applying a marking material having a known pattern to a surface; acquiring a sequence of image frames, each image frame of the sequence including a plurality of images of the known pattern covering the surface; deriving position and orientation information regarding the known pattern for each image frame of the sequence; and generating animation data incorporating the position and orientation information.

Description

SEGMENT TRACKING IN MOTION PICTURE CROSS-REFERENCE TO RELATED APPLICATIONS
This application claims the benefit of priority pursuant to 35 U. S. C. §119 of co-pending U.S. Provisional Patent Application No. 60/856,201, filed November 1, 2006, entitled "Segment Tracking in Motion Picture," the disclosure of which is hereby incorporated by reference.
This application further incorporates by reference the disclosures of commonly assigned U.S. Patent Application 10/427,114, filed May 1, 2003, entitled "System and Method for Capturing Facial and Body Motion"; U.S. Patent
Application 11/467,503, filed August 25, 2006, entitled "Labeling Used in Motion Capture"; U.S. Patent Application 11/829,711, filed July 27, 2007, entitled "FACS Cleaning in Motion Capture"; and U.S. Patent Application 11/776,358, entitled "Motion Capture Using Quantum Nano Dots," filed July 11, 2007, the disclosures of which are hereby incorporated by reference.
BACKGROUND
The present invention relates generally to motion capture, and more particularly to segment tracking using motion marker data.
Motion capture systems are used to capture the movement of an actor or object and map it onto a computer-generated actor/object as a way of animating it. These systems are often used in the production of motion pictures and video games for creating a digital representation of an actor or object for use as source data to create a computer graphics ("CG") animation. In a typical system, an actor wears a suit having markers attached at various locations (e.g., small reflective markers are attached to the body and limbs) . Appropriately placed digital cameras then record the actor' s body movements in a capture volume from different angles while the markers are illuminated. The system later analyzes the images to determine the locations (e.g., spatial coordinates) of the markers on the actor's suit in each frame. By tracking the locations of the markers, the system creates a spatial representation of the markers over time and builds a digital representation of the actor in motion. The motion is then applied to a digital model in virtual space, which may be textured and rendered to produce a complete CG representation of the actor and/or the performance. This technique has been used by special effects companies to produce realistic animations in many popular movies. SUMMARY
Certain implementations as disclosed herein provide for methods, systems, and computer programs for providing segment tracking in motion capture.
In one aspect, a method as disclosed herein provides for segment tracking. The method includes: applying a marking material having a known pattern to a surface; acquiring a sequence of image frames, each image frame of the sequence including a plurality of images of the known pattern covering the surface; deriving position and orientation information regarding the known pattern for each image frame of the sequence; and generating animation data incorporating the position and orientation information.
In another aspect, a system for segment tracking is disclosed. The system includes: an image acquisition module configured to generate a sequence of image frames, each image frame including a plurality of synchronized images of a known pattern disposed on a surface; and a segment tracking module configured to receive the sequence of image frames and generate animation data based on the known pattern disposed on the surface. Other features and advantages of the present invention will become more readily apparent to those of ordinary skill in the art after reviewing the following detailed description and accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS The details of the present invention, both as to its structure and operation, may be gleaned in part by study of the accompanying drawings, in which:
Figure 1 is a block diagram of a motion capture system in accordance with one implementation; Figure 2 shows a sample collection of markers according to an implementation of the present invention; Figure 3 is an illustration of a human figure with marker placement positions according to one implementation;
Figure 4 presents a frontal view of a human body model equipped with markers; Figure 5 depicts a quartering back view of a human body model configured with markers;
Figure 6 is a flowchart describing a method of segment tracking;
Figure 7A illustrates a marker labeled or patterned to represent a capital letter A;
Figure 7B illustrates a known pattern comprising a plurality of markers forming the letter A as a pattern;
Figure 7C illustrates a marker comprising a literal expression of the letter A; Figure 8 is a flowchart depicting an example method of utilizing strips of marking material;
Figure 9 is a functional block diagram of an implementation of a segment tracking system;
Figure 1OA illustrates a representation of a computer system and a user;
Figure 1OB is a functional block diagram illustrating the computer system hosting the segment tracking system; and
Figure 11 is a functional block diagram illustrating an implementation of a segment tracking module. DETAILED DESCRIPTION
Certain implementations as disclosed herein provide segment tracking in motion capture. Implementations include using one or more known patterns of markers applied on actors and/or objects. The marker (s) comprising a pattern are tracked as a group rather than individually. Thus, the pattern can provide information, such as identification, position/translation, and orientation/rotation, which significantly aids marker tracking. After reading this description it will become apparent to one skilled in the art how to practice the invention in various alternative implementations and alternative applications. However, although various implementations of the present invention will be described herein, it is understood that these embodiments are presented by way of example only, and not limitation. As such, this detailed description of various alternative implementations should not be construed to limit the scope or breadth of the present invention as set forth in the appended claims. According to implementations of the present invention, markers encoded with known patterns are applied to actors and/or objects, normally covering various surfaces of the actor and/or object. Identification, position/translation, and orientation/rotation of the actor and/or object are obtained by recording and digitizing images of the patterns. The patterns of markers can be selected to mitigate "missing-marker" effects in motion capture during marker occlusions. In another implementation, identifiable random patterns are used as "known patterns."
The known (and random) patterns may be generated, for example, using materials including quantum nanodots, glow- in-the dark (i.e., fluorescent) material, tattoos, and virtually any visible, infra-red, or ultra-violet ink, paint, or material which can be applied in a sufficiently identifiable pattern. The patterns may also include patterns of inherent features (e.g., moles or wrinkles) of the actor and/or objects.
In one implementation, a pattern comprises a plurality of markers or features coupled to the actor's body. In another implementation, a pattern comprises a single marker (e.g., a marker strip) or feature. The pattern can be applied or affixed on or around the surfaces of an actor' s limbs, hands, and feet. In one example, centroid information relating to the pattern can be derived from the circular disposition of a pattern strip wrapped around a limb (i.e., appendage) . By applying known patterns to the actors and/or objects and then recording their movements with cameras, it is possible to obtain not only position but also marker identity and spatial orientation.
Figure 1 is a functional block diagram of a motion capture system 100 in accordance with one implementation. The motion capture system 100 includes a motion capture processor 110, motion capture cameras 120, 122, 124, a user workstation 130, and an actor's body 140 and face 150 appropriately equipped with marker material 160 in a predetermined pattern. Although Figure 1 shows only thirteen markers 160A-160F, substantially more markers can be used on the body 140 and face 150. The motion capture processor 110 is connected to the workstation 130 by wire or wirelessly. The motion capture processor 110 is typically configured to receive control data packets from the workstation 130.
As shown, three motion capture cameras 120, 122, 124, are connected to the motion capture processor 110. Generally more than three motion capture cameras are required according to a variety of user- and animation- related needs and requirements. The motion capture cameras 120, 122, 124 are focused on the actor's body 140 and face 150, on which markers 160A-160F have been applied.
The placement of the markers 160A-160F is configured to capture motions of interest including, for example, the body 140, face 150, hands 170, arms 172, legs 174, 178, and feet 176 of the actor. In the implementation illustrated in Figure 1, markers 160A capture motions of the face 150; markers 160B capture motions of the arms 172; markers 160C capture motions of the body 140; markers 160D, 160E capture motions of the legs 174; and markers 160F capture motions of the feet 176. Further, uniqueness of the patterns on the markers 160A-160F provide information that can be used to obtain identification and orientation of the markers. The marker 160D is configured as a strip of pattern wrapped around a leg of the actor. The motion capture cameras 120, 122, 124 are controlled by the motion capture processor 110 to capture synchronous sequences of two-dimensional ("2-D") images of the markers. The synchronous images are integrated into image frames, each image frame representing one frame of a temporal sequence of image frames. That is, each individual image frame comprises an integrated plurality of simultaneously acquired 2-D images, each 2-D image generated by an individual motion capture camera 120, 122, or 124. The 2-D images thus captured may typically be stored, or viewed in real-time at the user workstation 130, or both.
The motion capture processor 110 performs the integration (i.e., performs a "reconstruction") of the 2-D images to generate the frame sequence of three-dimensional ("3-D," or "volumetric") marker data. This sequence of volumetric frames is often referred to as a "beat," which can also be thought of as a "take" in cinematography. Conventionally, the markers are discrete objects or visual points, and the reconstructed marker data comprise a plurality of discrete marker data points, where each marker data point represents a spatial (i.e., 3-D) position of a marker coupled to a target, such as an actor 140. Thus, each volumetric frame includes a plurality of marker data points representing a spatial model of the target. The motion capture processor 110 retrieves the volumetric frame sequence and performs a tracking function to accurately associate (or, "map") the marker data points of each frame with the marker data points of preceding and subsequent frames in the sequence.
For example, each individual marker data point in a first volumetric frame corresponds to a single marker placed on an actor's body 140. A unique label is assigned to each such marker data point of the first volumetric frame. The marker data points are then associated with corresponding marker data points in a second volumetric frame, and the unique labels for the marker data points of the first volumetric frame are assigned to the corresponding marker data points of the second volumetric frame. When the labeling (i.e., tracking) process is completed for the entire volumetric frame sequence, the marker data points of the first volumetric frame are thus traceable through the sequence, resulting in an individual trajectory for each marker data point.
Discrete markers are conventionally used to capture the motion of rigid objects or segments of an object or body. For example, rigid markers attached at an elbow and a wrist define the positions of each end of a forearm. When the forearm is moved, the motions of the elbow and wrist markers are tracked and resolved as described above in a sequence of volumetric frames. The motion of the forearm is thus modeled as a rigid body (e.g., a rod) with only the ends defined by the elbow and wrist markers. However, while translational movements of the forearm are easily resolved by analyzing the changes in spatial positions of the elbow and wrist markers, a common twisting motion of the forearm is difficult to detect because a twist can be performed without substantially moving the wrist or elbow. In one implementation, contrasting with the use of conventional discrete markers, markers disposed in a pattern are used, allowing the motion capture processor 110 to track the pattern as a group rather than individually tracking markers. Because the pattern provides identification information, movement of the markers of one pattern with respect to another pattern can be computed. In one implementation, a pattern tracked in this way is reconstructed in each volumetric frame as an individual object having spatial position information. The object is tracked through the sequence of volumetric frames, yielding a virtual animation representing the various spatial translations, rotations, and twists, for example, of the part of the actor to which the pattern is applied.
In one implementation, one or more known patterns are printed onto strips 160D. The strips 160D are then wrapped around each limb (i.e., appendage) of an actor such that each limb has at least two strips. For example, two strips 160D are depicted in Figure 1, wrapped around the actor's left thigh 178. End effectors (e.g., hands, feet, head), however, may be sufficiently marked with only one strip. Once captured, as discussed above, the printed patterns of the wrapped strips 160D enable the motion capture processor 110 to track the position and orientation of each "segment" representing an actor's limb from any angle, with as few as only one marker on a segment being visible. Illustrated in Figure 1, the actor's thigh 178 is treated as a segment at the motion capture processor 110. By wrapping a patterned strip 160D with multiple markers around a limb in substantially a circle, the "centroid" of the limb (i.e., segment) can be determined. Using multiple patterned strips 160D of markers, a centroid may be determined to provide an estimate or model of the bone within the limb. Further, it is possible to determine orientation, translation and rotation information regarding the entire segment from one (or more if visible) markers and/or strips applied on the segment.
In one implementation, the motion capture processor 110 performs segment tracking according to techniques disclosed herein from which identification, positioning/translation, and orientation/rotation information is generated for a group of markers (or marked areas) . While a conventional optical motion capture system typically records only position for a marker, segment tracking enables the motion capture processor 110 to identify which marker (s) are being captured and to locate the position and orientation of the captured markers of the segment. Once the markers are detected and identified, position and orientation/rotation information regarding the segment can be derived from the identified markers. Confidence in the determinations of position and orientation information for the segment increases as more markers are detected and identified. Markers or marking material applied in known patterns (and identifiable random patterns) essentially encode identification and orientation information facilitating efficient segment tracking. In one implementation, a known pattern of smaller markers constitutes a single marker or marking area. For example, a marker labeled or patterned as representing a capital letter A, as shown in Figure 7A, actually comprises six smaller markers A, B, C, D, E and F. In another implementation, a known pattern comprises a plurality of markers or marking areas forming the letter A as a pattern, as shown in Figure 7B. The eight dots forming the letter A function as a single, larger marker. Alternatively, a marker can comprise a literal expression of the letter A, as shown in Figure 7C. A common characteristic to the markers illustrated in Figures 7A-C is that they all have an identifiable orientation. In each case, the marker can be rotated, but will remain identifiable as a letter A, which significantly enhances marker tracking efficiency because it will be readily trackable from frame to frame of captured image data. In addition to identification, an orientation may also be determined for each of the marker examples from frame to frame. If a marker 160B (see Figure 1) is coupled to an actor' s forearm, for example, it will therefore rotate substantially 180 degrees when the actor moves the forearm from a hanging position to a straight-up, overhead reaching position. Tracking the marker will reveal not only the spatial translation to the overhead position, but that the forearm segment is oriented 180 degrees differently from before. Thus, marker tracking is enhanced and improved using markers or groups of markers encoding identification and orientation information.
Figure 2 shows a sample collection of markers according to an implementation of the present invention. Each marker comprises a 6x6 matrix of small white and black squares. Identification and orientation information is encoded in each marker by a unique placement of white squares within the 6x6 matrix. In each case, rotating the marker causes no ambiguity in terms of determining the identity and orientation of the marker, thus demonstrating the effectiveness of this scheme for encoding information. It will be appreciated that encoding schemes using arrangements other than the 6x6 matrix of black and white elements disclosed herein by example may also be implemented.
Figure 3 is an illustration of a human figure with marker placement positions according to one implementation. The markers shown encode identification and orientation information using a scheme similar to that depicted in Figure 2. They are positioned substantially symmetrically, and such that each major extremity (i.e., segment) of the body is defined by at least one marker. Approximately half of the markers depicted are positioned on a surface of the body not visible in the frontal view shown, and instead include arrows pointing to their approximate occluded positions. Referring to Figure 1, motion capture cameras 120, 122, 124 (representing typically a much larger plurality of cameras) encompass a capture space in which the actor's body 140 and face 150 are in motion. Even when the view of any of the markers is occluded to some subset of motion capture cameras 120, 122, 124, another subset will retain a view and capture the motions of the occluded markers. Thus, virtually all movements by an actor so equipped with markers can be captured using the systems described in relation to Figure 1, and tracked using segment tracking methods.
Figure 4 presents a frontal view of a human body model equipped with markers as described in Figure 3. As shown, only the markers on the forward-facing surfaces of the model are visible. The rest of the markers are partially or fully occluded. Figure 5 depicts a quartering back view of the same human body model in substantially the same pose as shown in Figure 4. From this view, the frontally-placed markers are occluded, but many of the markers occluded in Figure 4 are now visible. Thus, at any given time, substantially all of the markers are visible to some subset of the plurality of motion capture cameras 120, 122, 124 placed about the capture space. Also, the marker placements on the 3-D model depicted in Figures 4 and 5 substantially define the major extremities (segments) and areas on the body that articulate motions (e.g., the head, shoulders, hips, ankles, etc.) . When segment tracking is performed on the captured data, the positions of the body on which the markers are placed will be locatable and their orientations determinable . Further, the segments of the body defined by the marker placements, e.g., an upper arm segment between an elbow and a shoulder, will also be locatable because of the markers placed substantially at each end of that segment. The position and orientation of the upper arm segment will also be determinable from the orientations derived from the individual markers defining the upper arm.
Figure 6 is a flowchart describing a method of segment tracking 600 according to an implementation. A marking material with a known pattern, or an identifiable random pattern, is applied to a surface, at 610. In one implementation, the surface is that of an actor' s body, and a pattern comprises a plurality of markers that is coupled to the actor's body. In another implementation, a pattern comprises a single marker (e.g., a marker strip) that is coupled to the actor's body. The pattern may also be formed as a strip 160D and affixed around the actor's limbs, hands, and feet, as discussed in relation to Figure 1. Markers also include reflective spheres, tattoos glued on an actor's body, material painted on an actor's body, or inherent features (e.g., moles or wrinkles) of an actor. In one implementation, patterns of markers can be applied to the actor using temporary tattoos .
A sequence of image frames is acquired next, at 620, according to methods and systems for motion capture described herein, and above in relation to Figure 1. Once captured, the image data are used to reconstruct a 3-D model of an actor or object equipped with the markers, for example. In one implementation, this includes deriving position and orientation information of the marker pattern for each image frame, at 630. Position information is facilitated by the unique identification information encoded in the markers provided by the pattern using characteristic rotational invariance, as discussed with respect to Figures 4 and 5. Orientation information may be derived by determining the amount of rotation of the markers comprising the pattern. Marker rotations, and orientation in general, may be defined by affine representations in the 3-D space facilitating marker tracking. For example, an orientation can be represented by values representing the six degrees of freedom ("6DOF") of an object in 3-D space. Thus, the orientation can have three values representing any displacement in position (translation) from the origin of the coordinate system (e.g., a Euclidean space), and three values representing angular displacements (rotations) relative to the primary axes of the coordinate system.
Animation data based on the movements of the markers are generated, at 640. For example, a virtual digital model is to be activated by the data derived from the movements of an actor captured in the image data. During the performance, the actor swings the leg according to a script. The actor's legs are equipped with markers at each major joint, i.e., the hip, knee, and ankle, for instance. The movements of the markers are determined and used to define the movements of the segments of the actor's leg. The segments of the actor's leg correspond to the segments of the leg of the digital model, and the movements of the segments of the actor' s leg are therefore mapped to the leg of the virtual digital model to animate it.
As discussed in relation to Figure 6, at 610, a pattern may be formed as a strip 160D (see Figure 1) and affixed around an actor's limbs, hands, and feet. A centroid may then be determined from the circular disposition of the strip wrapped around the limb. Two such strips positioned at each end of the limb may then be used to determine a longitudinal centroid relative to the limb (segment) , approximating an underlying skeletal element, for example. A flowchart depicting a method of utilizing strips of marking material 800 is provided in Figure 8. Strips of marking material with known patterns are applied to one of an actor's limbs, at 810, typically by wrapping it around the limb. Because the wrapped strip of marking material is disposed in a substantially circular shape, centroids of the strips may be determined and used to express the movements of the body area wrapped with the strip. Further, the motion of a skeletal element of the limb can thus be approximated and used for skeleton modeling.
A sequence of image frames is acquired next, at 820, according to methods and systems for motion capture described above in relation to Figure 1. Once captured, the image data are used to reconstruct a 3-D model of an actor equipped with the markers, for example, in essentially the same manner as described in Figure 6, with respect to 620. In one implementation, this includes deriving position and orientation information of the known pattern for each strip in each image frame, at 830, in essentially the same manner as described in Figure 6, with respect to 630. Centroid information is derived for the actor' s limb equipped with one or more strips from the position and orientation information, at 840. As discussed above, the centroids of two or more strips can be used to approximate the skeletal structure (i.e., a bone) within the actor's limb. The movements of the centroids of the marker strips are then used to express the movements of the skeletal structure, and thus the movement of the limb.
Animation data based on the movements of the marker strips is generated, at 850. For example, during the performance, the actor swings his leg according to a script. The actor' s legs are equipped with strips of marker material wrapped around as described above. That is, for instance, a strip may be wrapped around the upper thigh and another wrapped around near the knee. The movements of the centroids of the marker strips are determined and used to define the skeletal structure of the upper leg, and consequently a skeletal element within the upper leg of a virtual digital model corresponding to the actor. The movements of the centroids can therefore used to animate the leg of the virtual digital model.
In one implementation, the markers are printed or formed using quantum nanodots (also known as "Quantum Dots," or "QDs") . The system would be similar in configuration to the traditional retro-reflective system, but with the addition of an exciter light source of a specific frequency (e.g., from existing filtered ring lights or from another source) and narrow band gap filters placed behind the lenses of the existing cameras, thus tuning the cameras to the wavelength of light emitted by the QDs. QDs are configured to be excited by light of a particular wavelength, causing them to emit light (i.e., fluoresce) at a different wavelength. Because they can emit light that has been quantum shifted up the spectrum from the excitation wavelength, the excitation light can be filtered from the cameras. This causes any light that falls outside of the particular emission spectrum for the QDs to be substantially blocked at the camera.
QDs can be tuned to virtually any wavelength in the visible or invisible spectrum. This means that a group of cameras can be filtered to only see a specific group of markers and nothing else, significantly cutting down the workload required of a given camera, and allow the camera to work "wide open" in the narrow response range of the QDs. In one implementation, the quantum nanodots are added to a medium such as ink, paint, plastic, temporary tattoo blanks, etc.
Figure 9 is a functional block diagram of an implementation of a segment tracking system 900. An image acquisition module 910 generates image frames of motion capture image data, and a segment tracking module 920 receives image frames and generates animation data. The image acquisition module 910 operates according to methods discussed above relating to the motion capture system 100 described in Figure 1. In one implementation, the image frames comprise volumetric data including untracked marker data. That is, marker data exist in each frame as unlabeled spatial data, unintegrated with the marker data of the other frames.
The segment tracking module 920 operates according to methods and schemes described in relation to Figure 2 through Figure 9 above.
Figure 11 is a functional block diagram depicting an example configuration of a segment tracking module 920. As shown, the segment tracking module 920 includes an identification module 1100, an orientation module 1110, a tracking module 1120, and a animation module 1130.
The identification module 1100 includes capability to identify markers having known patterns, such as the markers 160A-160F shown in Figure 1. In one implementation, the identification module 1100 performs pattern matching to locate known and random patterns of markers 160 from frame to frame. In another implementation, the identification module 1100 is configured to identify a single larger marker comprising a group of smaller markers, as discussed in relation to Figures 7A-7C. A marker 160 is designed to be identifiable regardless of its rotational state, and the identification module 1100 is configured accordingly to perform rotationally invariant identification of the markers 160. Once the markers 160 are uniquely identified, they are appropriately associated with the particular part of the actor's body, for example, to which it is coupled. As depicted in Figure 1, a marker 160B is associated with the actor's upper arm 172, markers 160D with the upper and lower thigh 178 (marker strips wrapped around the leg segment) , and markers 160C associated with the torso. The identification information is then passed to the orientation module 1110 and the tracking module 1120.
In the example configuration illustrated in Figure 11, the orientation module 1110 receives the identification information and generates orientation information. In one example, the orientation information comprises 6DOF data. The marker 160 is then analyzed in each frame to determine its evolving positional and rotational (i.e., affine) state. In one implementation, a 3-D affine transformation is developed describing the changes in these states from frame to frame for each marker 160. In the case where strip markers 160D are wrapped around each end of a segment of an actor's limb, such as the thigh 178 (see Figure 1), a centroid may be determined for that limb segment approximating the underlying skeletal structure (i.e., bone) . The orientation information (6DOF, affine transformation) is passed to the animation module 1130. The tracking module 1120 receives identification information and generates marker trajectory information. That is, the markers are tracked through the sequence of image frames, labeled, and a trajectory is determined. In one implementation, labeling and FACS approaches are employed according to U.S. Patent Application 11/467,503, filed August 25, 2006, entitled "Labeling Used in Motion Capture", and U.S. Patent Application 11/829,711, filed July 27, 2007, entitled "FACS Cleaning in Motion Capture." The labeling information (i.e., trajectory data) is passed to the animation module 1130.
The animation module 1130 receives orientation and labeling information and generates animation data. In one implementation, marker positions and orientations are mapped to positions on a (virtual) digital character model corresponding to the positions on the actor at which the markers were coupled. Similarly, a segment defined by markers on the body of the actor is mapped to a corresponding part on the digital character model. For example, the movement of the centroid approximating the skeletal structure of the actor' s limb also models the movement of that part of the actor's body. The transformations describing the movements of the markers 160 and related centroids are then appropriately formatted and generated as animation data for animating a corresponding segment of the digital character. The animation module 1130 receives the animation information and applies it to the digital character, resulting in its animation. The animation is typically examined for fidelity to the actor's movements, and for purposes of determining whether any, an how much, reprocessing may be required to obtain a desired result.
Figure 1OA illustrates a representation of a computer system 1000 and a user 1002. The user 1002 uses the computer system 1000 to perform segment tracking. The computer system 1000 stores and executes a segment tracking system 1090, which processes image frame data.
Figure 1OB is a functional block diagram illustrating the computer system 1000 hosting the segment tracking system 1090. The controller 1010 is a programmable processor and controls the operation of the computer system 1000 and its components. The controller 1010 loads instructions (e.g., in the form of a computer program) from the memory 1020 or an embedded controller memory (not shown) and executes these instructions to control the system. In its execution, the controller 1010 provides the segment tracking system 1090 as a software system. Alternatively, this service can be implemented as separate components in the controller 1010 or the computer system 1000.
Memory 1020 stores data temporarily for use by the other components of the computer system 1000. In one plementation, memory 1020 is implemented as RAM. In one implementation, memory 1020 also includes long-term or permanent memory, such as flash memory and/or ROM.
Storage 1030 stores data temporarily or long term for use by other components of the computer system 1000, such as for storing data used by the segment tracking system 1090. In one implementation, storage 1030 is a hard disk drive.
The media device 1040 receives removable media and reads and/or writes data to the inserted media. In one implementation, for example, the media device 1040 is an optical disc drive.
The user interface 1050 includes components for accepting user input from the user of the computer system 1000 and presenting information to the user. In one implementation, the user interface 1050 includes a keyboard, a mouse, audio speakers, and a display. The controller 1010 uses input from the user to adjust the operation of the computer system 1000.
The I/O interface 1060 includes one or more I/O ports to connect to corresponding I/O devices, such as external storage or supplemental devices (e.g., a printer or a PDA) . In one implementation, the ports of the I/O interface 660 include ports such as: USB ports, PCMCIA ports, serial ports, and/or parallel ports. In another implementation, the I/O interface 1060 includes a wireless interface for communication with external devices wirelessly. The network interface 1070 includes a wired and/or wireless network connection, such as an RJ-45 or "Wi-Fi" interface (including, but not limited to 802.11) supporting an Ethernet connection. The computer system 1000 includes additional hardware and software typical of computer systems (e.g., power, cooling, operating system) , though these components are not specifically shown in Figure 1OB for simplicity. In other implementations, different configurations of the computer system can be used (e.g., different bus or storage configurations or a multi-processor configuration) .
Various illustrative implementations of the present invention have been described. However, one of ordinary skill in the art will recognize that additional implementations are also possible and within the scope of the present invention. For example, known and identifiable random patterns may be printed, painted, and inked onto a surface of an actor or object. Further, any combination of printing, painting, inking, and tattoos, quantum nanodots, and inherent body features may be used to obtain a desired pattern .
It will be further appreciated that grouping functionalities within a module or block is for ease of description. Specific functionalities can be moved from one module or block to another without departing from the vention . Accordingly, the present invention is not limited tc only those embodiments described above.

Claims

CLAIMSWhat is claimed is:
1. A method, comprising: applying a marking material having a known pattern to a surface; acquiring a sequence of image frames, each image frame of the sequence including a plurality of images of the known pattern covering the surface; deriving position and orientation information regarding the known pattern for each image frame of the sequence; and generating animation data incorporating the position and orientation information.
2. The method of claim 1, wherein the marking material conforms to the surface.
3. The method of claim 1, wherein the known pattern is disposed on the marking material with quantum nano dots.
4. The method of claim 3, wherein the quantum nanodots are supported in a medium including an ink, a paint, and a plastic .
5. The method of claim 1, wherein the marking material includes a temporary tattoo.
6. The method of claim 5, wherein the temporary tatt includes a face tattoo.
7. The method of claim 6, wherein the face tattoo includes a plurality of separate tattoos.
8. The method of claim 1, wherein the marking material includes a reflective material.
9. The method of claim 1, wherein the marking material includes a fluorescent material.
10. The method of claim 1, wherein the surface includes a surface of a person.
11. The method of claim 10, wherein the surface of the person includes at least one of face, body, hand, foot, arm, and leg.
12. The method of claim 1, wherein the surface includes a surface of an object.
13. The method of claim 12, wherein the object includes at least one of a stage set and a stage prop.
14. The method of claim 1, wherein the pattern is a edetermined pattern.
15. The method of claim 1, wherein the pattern is a random pattern.
16. The method of claim 1, wherein said applying a marking material having a known pattern to a surface includes wrapping at least one strip onto which a known pattern is printed around an appendage of an actor.
17. The method of claim 16, wherein said deriving position and orientation information includes determining a centroid of the strip.
18. The method of claim 1, wherein said applying a marking material having a known pattern to a surface includes wrapping at least two strips onto which known patterns are printed around an appendage of an actor.
19. The method of claim 18, wherein said deriving position and orientation information includes determining a centroid of the appendage of the actor.
20. A system, comprising: an image acquisition module configured to generate a sequence of image frames, each image frame including a plurality of synchronized images of a known pattern disposed on a surface; and a segment tracking module configured to receive the sequence of image frames and generate animation data based on the known pattern disposed on the surface.
21. The system of claim 20, wherein said segment tracking module includes: an identification module configured to receive the sequence of image frames and generate identification information regarding the known pattern in each image frame of the sequence; an orientation module configured to receive the sequence of image frames and the identification information and generate orientation information; a tracking module configured to receive the identification information and generate marker trajectory information; and an animation module configured to receive the orientation information and the marker trajectory information and generate animation data.
22. A computer program, stored in a computer-readab] storage medium, the program comprising executable instructions that cause a computer to: acquire a sequence of image frames, each image frame of the sequence including a plurality of images of a known pattern covering a surface; derive position and orientation information regarding the known pattern; and generate animation data incorporating the position and orientation information.
EP07844821.4A 2006-11-01 2007-11-01 Segment tracking in motion picture Active EP2078419B1 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US85620106P 2006-11-01 2006-11-01
US11/849,916 US20080170750A1 (en) 2006-11-01 2007-09-04 Segment tracking in motion picture
PCT/US2007/083365 WO2008057957A2 (en) 2006-11-01 2007-11-01 Segment tracking in motion picture

Publications (3)

Publication Number Publication Date
EP2078419A2 true EP2078419A2 (en) 2009-07-15
EP2078419A4 EP2078419A4 (en) 2013-01-16
EP2078419B1 EP2078419B1 (en) 2019-10-30

Family

ID=39365244

Family Applications (1)

Application Number Title Priority Date Filing Date
EP07844821.4A Active EP2078419B1 (en) 2006-11-01 2007-11-01 Segment tracking in motion picture

Country Status (7)

Country Link
US (1) US20080170750A1 (en)
EP (1) EP2078419B1 (en)
JP (2) JP2011503673A (en)
KR (2) KR20090081003A (en)
AU (1) AU2007317452A1 (en)
CA (1) CA2668432A1 (en)
WO (1) WO2008057957A2 (en)

Families Citing this family (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101796545A (en) * 2007-09-04 2010-08-04 索尼公司 Integrated motion capture
US9767351B2 (en) 2009-01-15 2017-09-19 AvidaSports, LLC Positional locating system and method
US8330611B1 (en) * 2009-01-15 2012-12-11 AvidaSports, LLC Positional locating system and method
US8988240B2 (en) * 2009-01-15 2015-03-24 AvidaSports, LLC Performance metrics
CN102725038B (en) * 2009-09-15 2014-09-03 索尼公司 Combining multi-sensory inputs for digital animation
WO2014198629A1 (en) 2013-06-13 2014-12-18 Basf Se Detector for optically detecting at least one object
JP2015011480A (en) * 2013-06-27 2015-01-19 カシオ計算機株式会社 Image generation device, image generation method and program
JP2015039522A (en) * 2013-08-22 2015-03-02 セイコーエプソン株式会社 Rehabilitation device and assistive device for phantom limb pain treatment
WO2015091607A1 (en) * 2013-12-18 2015-06-25 Basf Se Target device for use in optical detection of an object
US9684369B2 (en) 2014-04-08 2017-06-20 Eon Reality, Inc. Interactive virtual reality systems and methods
EP3129111A4 (en) * 2014-04-08 2018-03-07 Eon Reality, Inc. Interactive virtual reality systems and methods
US9542011B2 (en) 2014-04-08 2017-01-10 Eon Reality, Inc. Interactive virtual reality systems and methods
DE102014106718B4 (en) * 2014-05-13 2022-04-07 Immersight Gmbh System that presents a field of view representation in a physical position in a changeable solid angle range
KR102397527B1 (en) 2014-07-08 2022-05-13 바스프 에스이 Detector for determining a position of at least one object
WO2016092451A1 (en) 2014-12-09 2016-06-16 Basf Se Optical detector
KR102496245B1 (en) 2015-01-30 2023-02-06 트리나미엑스 게엠베하 Detector for optical detection of one or more objects
US10955936B2 (en) 2015-07-17 2021-03-23 Trinamix Gmbh Detector for optically detecting at least one object
US10412283B2 (en) 2015-09-14 2019-09-10 Trinamix Gmbh Dual aperture 3D camera and method using differing aperture areas
WO2017132563A1 (en) * 2016-01-29 2017-08-03 Baylor Research Institute Joint disorder diagnosis with 3d motion capture
GB2551970B (en) * 2016-06-29 2021-10-27 Sony Europe Bv Determining the position of an object in a scene
EP3491675B1 (en) 2016-07-29 2022-11-16 trinamiX GmbH Optical sensor and detector for optical detection
KR102575104B1 (en) 2016-10-25 2023-09-07 트리나미엑스 게엠베하 Infrared optical detector with integrated filter
JP7241684B2 (en) 2016-10-25 2023-03-17 トリナミクス ゲゼルシャフト ミット ベシュレンクテル ハフツング detector for optical detection of at least one object
EP3542183A1 (en) 2016-11-17 2019-09-25 trinamiX GmbH Detector for optically detecting at least one object
US11860292B2 (en) 2016-11-17 2024-01-02 Trinamix Gmbh Detector and methods for authenticating at least one object
EP3564929A4 (en) * 2016-12-27 2020-01-22 Coaido Inc. Measurement device and program
US20180268614A1 (en) * 2017-03-16 2018-09-20 General Electric Company Systems and methods for aligning pmi object on a model
US11060922B2 (en) 2017-04-20 2021-07-13 Trinamix Gmbh Optical detector
CN110998223B (en) 2017-06-26 2021-10-29 特里纳米克斯股份有限公司 Detector for determining the position of at least one object
CN110741413B (en) * 2018-11-29 2023-06-06 深圳市瑞立视多媒体科技有限公司 Rigid body configuration method and optical motion capturing method
US11610330B2 (en) * 2019-10-08 2023-03-21 Samsung Electronics Co., Ltd. Method and apparatus with pose tracking
WO2024147629A1 (en) * 2023-01-03 2024-07-11 재단법인대구경북과학기술원 Pattern marker and method for tracking same

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2001063320A1 (en) * 2000-02-22 2001-08-30 Aman James A Method for representing real-time motion
US20020097245A1 (en) * 2000-12-27 2002-07-25 Il-Kwon Jeong Sensor fusion apparatus and method for optical and magnetic motion capture systems
US20040179013A1 (en) * 2003-03-13 2004-09-16 Sony Corporation System and method for animating a digital facial model
US20050083333A1 (en) * 2003-05-01 2005-04-21 Sony Corporation System and method for capturing facial and body motion
US20060055706A1 (en) * 2004-09-15 2006-03-16 Perlman Stephen G Apparatus and method for capturing the motion of a performer
US20060203096A1 (en) * 2005-03-10 2006-09-14 Lasalle Greg Apparatus and method for performing motion capture using shutter synchronization

Family Cites Families (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3510210A (en) * 1967-12-15 1970-05-05 Xerox Corp Computer process character animation
JP2893052B2 (en) * 1990-07-31 1999-05-17 株式会社エフ・エフ・シー 3D feature point coordinate extraction method
US5622187A (en) * 1994-09-30 1997-04-22 Nomos Corporation Method and apparatus for patient positioning for radiation therapy
JPH09153151A (en) * 1995-11-30 1997-06-10 Matsushita Electric Ind Co Ltd Motion generator for three-dimensional skeletal structure
US5828770A (en) * 1996-02-20 1998-10-27 Northern Digital Inc. System for determining the spatial position and angular orientation of an object
KR100269116B1 (en) * 1997-07-15 2000-11-01 윤종용 Apparatus and method for tracking 3-dimensional position of moving abject
JP2865168B1 (en) * 1997-07-18 1999-03-08 有限会社アートウイング Motion data creation system
JPH1196374A (en) * 1997-07-23 1999-04-09 Sanyo Electric Co Ltd Three-dimensional modeling device, three-dimensional modeling method and medium recorded with three-dimensional modeling program
US20050105772A1 (en) * 1998-08-10 2005-05-19 Nestor Voronka Optical body tracker
US6973202B2 (en) * 1998-10-23 2005-12-06 Varian Medical Systems Technologies, Inc. Single-camera tracking of an object
US6487516B1 (en) * 1998-10-29 2002-11-26 Netmor Ltd. System for three dimensional positioning and tracking with dynamic range extension
US7483049B2 (en) * 1998-11-20 2009-01-27 Aman James A Optimizations for live event, real-time, 3D object tracking
US6567116B1 (en) * 1998-11-20 2003-05-20 James A. Aman Multiple object tracking system
JP4794708B2 (en) * 1999-02-04 2011-10-19 オリンパス株式会社 3D position and orientation sensing device
JP3285567B2 (en) * 1999-11-01 2002-05-27 日本電信電話株式会社 Three-dimensional position measuring device and method, and recording medium recording three-dimensional position measuring program
KR100361462B1 (en) * 1999-11-11 2002-11-21 황병익 Method for Acquisition of Motion Capture Data
DE10025922A1 (en) * 2000-05-27 2001-12-13 Robert Massen Automatic photogrammetric digitization of bodies and objects
JP2002210055A (en) * 2001-01-17 2002-07-30 Saibuaasu:Kk Swing measuring system
JP2003106812A (en) * 2001-06-21 2003-04-09 Sega Corp Image information processing method, system and program utilizing the method
WO2003021635A2 (en) * 2001-09-05 2003-03-13 Rensselaer Polytechnic Institute Passivated nanoparticles, method of fabrication thereof, and devices incorporating nanoparticles
US20030076980A1 (en) * 2001-10-04 2003-04-24 Siemens Corporate Research, Inc.. Coded visual markers for tracking and camera calibration in mobile computing systems
US8137210B2 (en) * 2001-12-05 2012-03-20 Acushnet Company Performance measurement system with quantum dots for object identification
US7231063B2 (en) * 2002-08-09 2007-06-12 Intersense, Inc. Fiducial detection system
WO2004038657A2 (en) * 2002-10-22 2004-05-06 Artoolworks Tracking a surface in a 3-dimensional scene using natural visual features of the surface
US9177387B2 (en) * 2003-02-11 2015-11-03 Sony Computer Entertainment Inc. Method and apparatus for real time motion capture
US7218320B2 (en) * 2003-03-13 2007-05-15 Sony Corporation System and method for capturing facial and body motion
US7565004B2 (en) * 2003-06-23 2009-07-21 Shoestring Research, Llc Fiducial designs and pose estimation for augmented reality
JP2005258891A (en) 2004-03-12 2005-09-22 Nippon Telegr & Teleph Corp <Ntt> 3d motion capturing method and device
JP2005256232A (en) 2004-03-12 2005-09-22 Nippon Telegr & Teleph Corp <Ntt> Method, apparatus and program for displaying 3d data
EP1741059B1 (en) * 2004-04-26 2013-10-02 Siemens Aktiengesellschaft Method for determining the position of a marker in an augmented reality system
WO2005124687A1 (en) 2004-06-16 2005-12-29 The University Of Tokyo Method for marker tracking in optical motion capture system, optical motion capture method, and system
US7991220B2 (en) * 2004-09-01 2011-08-02 Sony Computer Entertainment Inc. Augmented reality game system using identification information to display a virtual object in association with a position of a real object
US7567293B2 (en) * 2006-06-07 2009-07-28 Onlive, Inc. System and method for performing motion capture by strobing a fluorescent lamp

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2001063320A1 (en) * 2000-02-22 2001-08-30 Aman James A Method for representing real-time motion
US20020097245A1 (en) * 2000-12-27 2002-07-25 Il-Kwon Jeong Sensor fusion apparatus and method for optical and magnetic motion capture systems
US20040179013A1 (en) * 2003-03-13 2004-09-16 Sony Corporation System and method for animating a digital facial model
US20050083333A1 (en) * 2003-05-01 2005-04-21 Sony Corporation System and method for capturing facial and body motion
US20060055706A1 (en) * 2004-09-15 2006-03-16 Perlman Stephen G Apparatus and method for capturing the motion of a performer
US20060203096A1 (en) * 2005-03-10 2006-09-14 Lasalle Greg Apparatus and method for performing motion capture using shutter synchronization

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of WO2008057957A2 *

Also Published As

Publication number Publication date
AU2007317452A1 (en) 2008-05-15
JP2011503673A (en) 2011-01-27
KR20090081003A (en) 2009-07-27
CA2668432A1 (en) 2008-05-15
EP2078419A4 (en) 2013-01-16
WO2008057957A2 (en) 2008-05-15
EP2078419B1 (en) 2019-10-30
WO2008057957A3 (en) 2008-09-25
JP2012248233A (en) 2012-12-13
US20080170750A1 (en) 2008-07-17
KR20120114398A (en) 2012-10-16

Similar Documents

Publication Publication Date Title
EP2078419B1 (en) Segment tracking in motion picture
US8330823B2 (en) Capturing surface in motion picture
CN101310289B (en) Capturing and processing facial motion data
EP2191445B1 (en) Integrated motion capture
Gall et al. Motion capture using joint skeleton tracking and surface estimation
US8384714B2 (en) Systems, methods and devices for motion capture using video imaging
KR101519775B1 (en) Method and apparatus for generating animation based on object motion
Ballan et al. Marker-less motion capture of skinned models in a four camera set-up using optical flow and silhouettes
EP1335322A2 (en) Method of determining body motion from captured image data
JP2003061936A (en) Moving three-dimensional model formation apparatus and method
CN111897422A (en) Real object interaction method and system for real-time fusion of virtual and real objects
AU2012203097B2 (en) Segment tracking in motion picture
CN101573959A (en) Segment tracking in motion picture
Liang et al. Hand pose estimation by combining fingertip tracking and articulated ICP
KR102075079B1 (en) Motion tracking apparatus with hybrid cameras and method there
Xing et al. Markerless motion capture of human body using PSO with single depth camera
Roodsarabi et al. 3d human motion reconstruction using video processing
Roudsarabi et al. Solving occlusion problem in 3D human motion reconstruction

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20080320

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC MT NL PL PT RO SE SI SK TR

RBV Designated contracting states (corrected)

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC MT NL PL PT RO SE SI SK TR

RBV Designated contracting states (corrected)

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC MT NL PL PT RO SE SI SK TR

RIN1 Information on inventor provided before grant (corrected)

Inventor name: GORDON, DEMIAN

RBV Designated contracting states (corrected)

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC MT NL PL PT RO SE SI SK TR

DAX Request for extension of the european patent (deleted)
RBV Designated contracting states (corrected)

Designated state(s): DE FR GB

A4 Supplementary search report drawn up and despatched

Effective date: 20121218

RIC1 Information provided on ipc code assigned before grant

Ipc: G06T 7/20 20060101AFI20121212BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20181030

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602007059436

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: H04N0005225000

Ipc: G06T0007246000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: G06T 7/246 20170101AFI20190429BHEP

INTG Intention to grant announced

Effective date: 20190529

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): DE FR GB

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602007059436

Country of ref document: DE

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602007059436

Country of ref document: DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20200731

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20220616

Year of fee payment: 16

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230606

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20231019

Year of fee payment: 17

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20231019

Year of fee payment: 17

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602007059436

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20240601

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20240601