US20110021317A1

US20110021317A1 - System and method for displaying anonymously annotated physical exercise data

Info

Publication number: US20110021317A1
Application number: US12/673,793
Authority: US
Inventors: Gerd Lanfermann
Original assignee: Koninklijke Philips Electronics NV
Current assignee: Koninklijke Philips NV
Priority date: 2007-08-24
Filing date: 2008-08-22
Publication date: 2011-01-27
Also published as: CN101784230A; JP2010536459A; WO2009027917A1; EP2185071A1

Abstract

The present invention relates to a method for displaying anonymously annotated physical exercise data to a person undertaking exercises. Based on physical exercise data, the physical exercise data is annotated at a physically separate annotation unit. At the location of the person, visual recordings of the person undertaking exercises together with synchronized annotation information are displayed to the person. A system for performing the method comprises a physical data processing unit (1), a display device (2), at least one posture recording device (3, 3′), a visual recording device (4), a data storage unit (5) and a physically separate annotation unit (6) in connection with the physical data processing unit (1), the connection being via an interconnected computer network (7).

Description

BACKGROUND OF THE INVENTION

The present invention relates to a system and method for displaying anonymously annotated physical exercise data to a person undertaking exercises.
Home rehabilitation exercises for persons suffering from a medical condition like a stroke or home training exercises for persons wishing to improve body motions like a golf swing can be recorded via sensors. The exercises can also be evaluated by a professional such as a physiotherapist or a golf instructor in order to give the person a direct feedback.
If the professional performing the review is not present during the exercise, video camera recordings could be sent to him. These recordings could be reviewed intuitively by the professional and the commented recordings could be understood intuitively by the person undertaking the exercise. However, these recordings, especially when sent away to a remote professional, could breach the privacy of the person. Furthermore, a completely automatic processing of such recorded images to provide meaningful feedback is a demanding task.
Alternatively, the sole transmission of data from the sensors would not violate the privacy of the person. In this respect, U.S. Pat. No. 6,817,979 B2 relates to a system and method which provide for interacting with a virtual physiological model of a user with the use of a mobile communication device. Physiological data associated with the user is acquired from the user. The physiological data is transmitted to the mobile communication device, preferably with the use of a wireless communication protocol. The methodology further involves using the mobile communication device to communicate the physiological data to a network server. The physiological data is integrated into the virtual physiological model of the user. The user can access data and depictions of the user developed from the physiological data.
By way of example, a user can create an avatar representative of the current physical state of the user. The user can adjust the avatar to change the appearance of the avatar to a more desired appearance. For example, the anatomical dimensions of the avatar can be changed to reflect desired waist, chest, upper arms and thigh dimensions. Given differences between the desired avatar features and present avatar features, various training, diet and related fitness recommendations can be developed to establish a training regimen most suited to help the user achieve the desired fitness goals. Physiological data is subsequently acquired and applied to the user's avatar, and compared to the desired avatar's data to determine if the training regimen is effective in achieving the desired fitness goals.
However, in general the interpretation of sensor signals in the frontend leads to difficulties on the part of the user. It is hard to relate to an abstract rendering of an artificial screen character.
Despite this effort accordingly there still exists a need in the art for a system and a method for displaying anonymously annotated physical exercise data to a person undertaking exercises.

SUMMARY OF THE INVENTION

To achieve this and other objects the present invention is directed to a method for displaying anonymously annotated physical exercise data to a person undertaking exercises, comprising the steps of:
a) gathering physical exercise data from a person undertaking exercises;
b) synchronously gathering visual recordings of the person undertaking exercises;
c) transmitting the physical exercise data to a physically separate annotation unit;
d) based on the physical exercise data, annotating the physical exercise data at the physically separate annotation unit;
e) transmitting the annotation information to a display and processing unit for review of the person undertaking exercises;
f) displaying the visual recordings of the person undertaking exercises together with synchronized annotation information to the person.

DETAILED DESCRIPTION OF THE INVENTION

Before the invention is described in detail, it is to be understood that this invention is not limited to the particular component parts of the devices described or process steps of the methods described as such devices and methods may vary. It is also to be understood that the terminology used herein is for purposes of describing particular embodiments only, and is not intended to be limiting. It must be noted that, as used in the specification and the appended claims, the singular forms “a,” “an” and “the” include singular and/or plural referents unless the context clearly dictates otherwise.
In the context of the present invention, the term anonymously annotated data denotes data where a third person performing the annotation has no knowledge about the identity of the person whose data he is annotating. In particular, the data does not allow for a recognition of the person. One way of achieving the anonymization is by assigning identification numbers to the data. Physical exercise data is data relating to movements or other exercises of a person.
The first two steps of the method describe how two different sets of information about the exercise of the person are gathered. Firstly, physical exercise data is gathered, for example by continuously monitoring sensor signals from the person. At the same time, visual recordings are gathered, for example by using a digital video camera. By synchronously gathering this data it is ensured that later on, a certain portion of the video stream can be attributed to a certain portion of the sensor signal stream and vice versa.
As the visual recordings and the physical exercise data are separate entities, the physical exercise data can then be transmitted to a physically separate annotation unit. The physical separation of the annotation unit provides for an anonymization of the data. At the annotation unit the physical exercise data can be processed into representations of the exercise for review by a third person. The physical exercise data can then be annotated. This includes automatic processing of the data, for example by detecting deviations from motion templates. Furthermore, the third person can include comments and suggestions to provide helpful feedback to the person performing the exercise. Afterwards, the annotation information is transmitted to a display and processing unit at the site of the person performing the exercise. Here, the annotation information is joined with the visual recordings. The recordings of the person undertaking exercises are then displayed to the person together with the synchronized annotation information. The synchronization provides for displaying the annotation at the correct time so the person can directly understand what has caught the attention of the reviewer or the automatic reviewing system.
In summary, with the method according to the present invention an exercise of a person can be reviewed anonymously and feedback can be given to the person. The anonymization allows for the sharing of professional resources, making the reviewing process more efficient. At the same time, when the person receives the feedback it is very clearly shown to him, via the visual recordings, which part of the exercise has prompted the feedback.
In one embodiment of the invention, at the physically separate annotation unit in step d) an avatar is calculated based on the physical exercise data. For the purposes of this invention, the term ‘avatar’ shall denote a computer-generated abstract rendering which represents the posture or motions of a person. In simple cases, the avatar may be a stick figure. In more sophisticated cases, the avatar may represent additional information like the pulse rate, the amount of sweating, muscle fatigue and the like. An advantage of using an avatar representation is that the avatar can be rotated on the screen of the annotation unit while representing the exercise. This enables the reviewer to choose the best viewing angle for assessing the exercise.
In a further embodiment of the invention step f) additionally comprises calculating an avatar and displaying the avatar synchronously with the visual recordings and the annotations to the person. In summary, the person will then see the visual recording of his exercise, the annotations and the avatar. This is advantageous as the avatar may depict more clearly the motions of the persons if they are obscured in the visual recording by baggy clothing or if they have not been recorded correctly on camera. Again, the avatar may be rotated to achieve the best viewing perspective. Another option is to provide multiple viewing angles with one or more avatars.
In a further embodiment of the invention transmitting the physical exercise data in step c) and transmitting the annotation information in step e) is undertaken via an interconnected computer network, preferably the internet. This allows a remotely located person to perform the review and the annotation. Suitable protocols can include those of the TCP/IP protocol.
In a further embodiment of the invention the physical exercise data from the person is selected from the group comprising motion data, posture data, electromyographic data, pulse rate, blood pressure, blood oxygen content, blood sugar content, severity of perspiration and/or respiratory rate. These data types either relate to the exercise itself, such as in the case of motion and posture data. Other data types relate to the overall condition or physical fitness of the person. Knowledge about this can give valuable insight into the effectiveness of rehabilitation or training measures. For example, it may be inferred whether the person is in the supercompensation phase after a training stimulus.
In a further embodiment of the invention the annotation information is selected from the group comprising visual information, audio signals and/or speech recordings. Visual information can be in the form of markings such as arrows pointing out a specific issue that are inserted into the images of the avatar. Additionally, small video clips can be inserted to show the correct execution of the exercise. Other visual information can be written comments or graphs showing statistics of data like electromyographic data, pulse rate, blood pressure, blood oxygen content, blood sugar content, severity of perspiration and/or the respiratory rate. This enables to assess the situation of the person performing the exercise at one glance. Audio signals can be simple beeps when a movement is no performed correctly. Recorded speech comments can be added by the reviewer when this is the simplest way of explaining an exercise.
The present invention is further directed towards a system for displaying anonymously annotated physical exercise data to a person undertaking exercises, comprising:

- a physical data processing unit;
- a display device in communication with the physical data processing unit;
- at least one posture recording device assigned to the person undertaking exercises and in communication with the physical data processing unit;
- a visual recording device in communication with the physical data processing unit;
- a data storage unit for storing and retrieving data from the physical data processing unit and the visual recording device; the data storage means being in communication with the physical data processing unit;
- a physically separate annotation unit in connection with the physical data processing unit, the connection being via an interconnected computer network.

In one embodiment of the invention the at least one posture recording device comprises a motion sensor on the person undertaking exercises, the sensor being selected from the group comprising acceleration sensors, inertia sensors and/or gravity sensors. The motion sensors can be worn on the body of the person on selected locations like upper arm, lower arm, upper leg, lower leg or torso. They can be commercially available highly integrated solid state sensors. The transmission of the sensor signals to the posture assessment unit can be undertaken via wire, wirelessly or in a body area network using the electrical conductivity of the human skin. After calculation of the person's posture the result can be present in the form of an avatar.
In a further embodiment of the invention the at least one posture recording device comprises an optical mark on the person undertaking exercises. The posture recording device then employs an optical tracking system for tracking the at least one optical mark. Based on the signals of the optical tracking system a representation of the person's posture is then calculated. The optical marks can be borne on the body of the person on selected locations like upper arm, lower arm, upper leg, lower leg or torso. The tracking of the marks can be effected with a single camera or a multitude of cameras. When a stereo camera is used, three-dimensional posture and movement data is generated. After image processing and calculation of the person's posture the result can be present in the form of an avatar.
It is also possible to combine several posture monitoring principles. For example, a combination of motion sensors and optical tracking may provide complementary data to better calculate the posture of the person.
A further aspect of the present invention is the use of a system according to the present invention claims for displaying anonymously annotated physical exercise data to a person undertaking exercises.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will become more readily understood with reference to the following drawing, wherein

FIG. 1 shows a system according to the present invention

FIG. 2 shows a synchronous overlay of visual recordings and an avatar representing physical exercise data

FIG. 3 shows a flowchart of a method according to the present invention

FIG. 4 shows modules for performing a method according to the present invention.

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a system according to the present invention for displaying anonymously annotated physical exercise data to a person undertaking exercises. As posture recording devices, the person has motion sensors 3 situated on his thighs and his ankles. Furthermore, optical marks 3′ are located on the wrist and the torso. Being physical exercise data, the signals of the motion sensors 3 are transmitted wirelessly to the physical data processing unit 1 where the raw sensor signals are processed into motion and posture data. A video camera 4 records the motions of the person. Furthermore, the physical data processing unit 1 performs optical tracking operations on the video stream of the camera 4 for identifying the position and the movement of the optical marks 3′. This is also processed into motion and posture data and complements the data obtained from the motion sensors 3.
The raw or processed sensor signals and positional information from the optical marks 3′ are stored in a data storage unit 5. Furthermore, the video stream of the person performing the exercise is also stored there. The data in the data storage unit 5 is stored together with an information about the time of recording. This makes it possible to correlate or synchronize the information, for example knowing which position as indicated by posture recording devices 3, 3′ corresponds to which frame of a video clip of the person performing the exercise.
Using an interconnected computer network such as the internet 7, the physical data processing unit 1 transmits the processed sensor 3 signals and positional information from the optical marks 3′ to a physically separate annotation unit 6. Temporal information is also transmitted. This annotation unit then calculates a visual representation such as an avatar from the received physical data. A physical therapist views the motion of the visual representation on his terminal 8 and comments sequences, thus performing the annotation. The annotation together with the time within the exercise when the annotation has been made is transmitted back to the physical data processing unit 1 at the location of the person undertaking exercises. Again, the transmission is achieved over an interconnected computer network such as the internet 7.
The physical data processing unit 1 then accesses the data storage unit 5 and retrieves the recorded data and video clips from the particular exercise that has been annotated. A movie sequence is generated for viewing by the person and displayed on display 2. In this case, the video stream of the person and an avatar calculated from the recorded data are shown simultaneously. At the appropriate time, the comments of the physical therapist are also displayed or voiced to the person.
FIG. 2 shows a synchronous overlay of visual recordings and an avatar representing physical exercise data. A person has been performing an exercise. Physical data representing his motions has been recorded and used for calculation of an avatar representation. The avatar's motion has been time-resolved and split into a stream of individual frames 20. Likewise, the person's movements have been recorded by a video camera. This video image sequence has also been time-resolved and split into a stream of individual frames 21. As the physical exercise data and the visual recordings have been gathered simultaneously, one common time line can be assigned to them. The time line in FIG. 2 beneath the frame streams arbitrarily begins at 4:16 minutes and ends at 4:21 minutes.
In the exercise of FIG. 2, the person starts with both of his arms stretched lowered. In the images, the left arm is kept stretched and raised along the coronal plane until the hand is above the person's head. The arm is kept in this position while the same movement is supposed to be performed with the right arm. At the time of 4:20, the person is not able to keep his right arm outstretched in the horizontal position. The arm is bent at the elbow. This makes it much easier to lift the arm so at this point no therapeutic benefit is gained. A physical therapist remotely reviewing the avatar frames 20 can then single out the frame at 4:20 minutes and add a visual or verbal comment. This comment, together with the information that it is to be shown at 4:20 minutes into the exercise, is transmitted to the person for future reviewal. At the person's location the annotation can then be combined with the visual recordings 21 so that the person can relate more directly to the exercise and contemplate his errors in performing it.
FIG. 3 shows a flowchart of a method according to the present invention. The first step 30 is to record the exercise a person is performing visually, using a camera, and via posture data, using sensors. The visual recordings are stored 31 and the posture recordings are transmitted to an annotation system 32. Using the annotation system, a person reviews the posture recordings and adds his comments and markers 33. These annotations are transmitted back to the patient system 34, wherein ‘patient’ denotes the person performing an exercise. On the patient side, the stored visual recordings are retrieved 35 and combined with the annotations 36 in order to give the person a comprehensive feedback that still does not compromise his anonymity.
FIG. 4 shows modules for performing a method according to the present invention to complement the depiction of a system in FIG. 1. A sensor receiver 40 receives signals from motion sensors or information from the tracking of optical marks. This sensor receiver 40 communicates its data to a movement transmission module 41. Synchronously with the sensor receiver 40, a camera 42 captures a video sequence of the person performing exercises. These video sequences are stored in a storage facility 43. The movement transmission module 41 transmits its data to a remotely located movement receiver 45. This is symbolized by barrier 44 separating the two sub-groups of modules.
The movement receiving module 45 passes the data on to a movement annotator 46 where the data is transformed into processible data and annotated by a reviewer. The annotation together with information on the temporal position of the annotation within the exercise is passed on to annotation transmission module 47. Aforementioned annotation transmission module 47 transmits the information to an annotation receiver 48 located at the sub-group of modules assigned to the person performing the exercise. This annotation information reaches a processing and overlay module 49 which accesses video sequences from the storage module 43 and combines the sequences with the annotation so that the annotation is present at the appropriate time of the video sequence. Finally, via a rendering module 50, the overlaid video sequence is displayed to the person who has performed the exercise.
To provide a comprehensive disclosure without unduly lengthening the specification, the applicant hereby incorporates by reference each of the patents and patent applications referenced above.
The particular combinations of elements and features in the above detailed embodiments are exemplary only; the interchanging and substitution of these teachings with other teachings in this and the patents/applications incorporated by reference are also expressly contemplated. As those skilled in the art will recognize, variations, modifications, and other implementations of what is described herein can occur to those of ordinary skill in the art without departing from the spirit and the scope of the invention as claimed. Accordingly, the foregoing description is by way of example only and is not intended as limiting. The invention's scope is defined in the following claims and the equivalents thereto. Furthermore, reference signs used in the description and claims do not limit the scope of the invention as claimed.

Claims

1. A method for displaying anonymously annotated physical exercise data to a person undertaking exercises, comprising the steps of:

a) gathering physical exercise data from a person undertaking exercises;

b) synchronously gathering visual recordings of the person undertaking exercises;

c) transmitting the physical exercise data to a physically separate annotation unit;

d) based on the physical exercise data, annotating the physical exercise data at the physically separate annotation unit;

e) transmitting the annotation information to a display and processing unit for review of the person undertaking exercises;

f) displaying the visual recordings of the person undertaking exercises together with synchronized annotation information to the person.

2. Method according to claim 1, wherein at the physically separate annotation unit in step d) an avatar is calculated based on the physical exercise data.

3. Method according to claim 1, wherein step f) additionally comprises calculating an avatar and displaying the avatar synchronously with the visual recordings and the annotations to the person.

4. Method according to claim 1, wherein transmitting the physical exercise data in step c) and transmitting the annotation information in step e) is undertaken via an interconnected computer network, preferably the internet.

5. Method according to claim 1, wherein the physical exercise data from the person is selected from the group comprising motion data, posture data, electromyographic data, pulse rate, blood pressure, blood oxygen content, blood sugar content, severity of perspiration and/or respiratory rate.

6. Method according to claim 1, wherein the annotation information is selected from the group comprising visual information, audio signals and/or speech recordings.

7. A system for displaying anonymously annotated physical exercise data to a person undertaking exercises, comprising:

a physical data processing unit (1);

a display device (2) in communication with the physical data processing unit (1);

at least one posture recording device (3, 3′) assigned to the person undertaking exercises and in communication with the physical data processing unit (1);

a visual recording device (4) in communication with the physical data processing unit (1);

a data storage unit (5) for storing and retrieving data from the physical data processing unit (1) and the visual recording device (4); the data storage means (5) being in communication with the physical data processing unit (1);

a physically separate annotation unit (6) in connection with the physical data processing unit (1), the connection being via an interconnected computer network (7).

8. System according to claim 7, wherein the at least one posture recording device (3, 3′) comprises a motion sensor (3) on the person undertaking exercises, the sensor being selected from the group comprising acceleration sensors, inertia sensors and/or gravity sensors.

9. System according to claim 7, wherein the at least one posture recording device (3, 3′) comprises an optical mark (3′) on the person undertaking exercises.

10. Use of a system according to claim 7 for displaying anonymously annotated physical exercise data to a person undertaking exercises.