AU2021107210A4

AU2021107210A4 - System, method and virtual reality device for assessing compliance of body movement

Info

Publication number: AU2021107210A4
Application number: AU2021107210A
Authority: AU
Inventors: Julie Ann Ward
Original assignee: Connect Qld Pty Ltd
Current assignee: Connect Qld Pty Ltd
Priority date: 2021-08-25
Filing date: 2021-08-25
Publication date: 2021-12-09
Anticipated expiration: 2029-08-25

Abstract

A Virtual Reality (VR) based computer implemented method for assessing compliance of a subject's body movement is described. The method comprises receiving data from one or more wearable sensors, the received data comprising time-series data received prior to the execution of the body movement and real-time data and displaying an ideal or target image for the body movement. The method also comprises overlaying on the displayed ideal or target image a real-time subject image simulating the body movement of the subject during the body movement to assess compliance. The real-time image may be constructed from the time-series data. A VR based computer system, non-transitory VR based computer program and a device for assessing compliance of a subject's body movement are also described. The ideal or target image may be correlated with the time-series data and/or the correlation may comprise an increased or decreased stringency or an increased or decreased overall body movement-based exertion. 3/5 100 3D Images of Activate Body comparison of user Choose Training Choose Instrucntr overlays oav r an instrctDashboard -- Incoratesa1PI Location (age and ethnicity) and aboe Go system with istructor and able to be pre-set datamils reproduced for clinician and user. 102 102 ~ 104,106 ! I Glass panes on door so t Future Lecturre start Closed Double can see through to the cu utiest Seminar Room Doors with Notice live lecture. board on it 2. Current lecture name, Lei 3. Number ofcurrent Users lecture 1 Set reminder 2. Invite User View current lectures Pas Lectures body sensors and check they are wehin 102 pre-set limits to commenceE lecture Past Lectures menu Choose location to P sea mediate (cinema layout) Str 7ro VWch Live Choose instructor HUD (age and ethnicity) IPause 2 Play 3. Skip back 10s 4. Change speed (0 75x. 125x) 5. Restart Lecture Live / in real time 10 6EY monitoring of body 102 for data relaxation I meditative response Questlos for ecture end Store data i n 108 dashboard, data available for clinician and user Printable version of content and 3D demonstrations to be available FIG. 1 C

Description

3/5

100 3D Images of Activate Body comparison of user Choose Training Choose Instrucntr overlays oav r an instrctDashboard -- Incoratesa1PI Location (age and ethnicity) and aboe Go system with istructor and able to be pre-set datamils reproduced for clinician and user.

102 102 ~ 104,106

! I Glass panes on door so t Closed Double can see through to the cu Future Lecturre start Seminar Room Doors with Notice live lecture. utiest board on it 2. Current lecture name, Lei 3. Number ofcurrent Users lecture 1 Set reminder 2. Invite User

View current lectures Pas Lectures body sensors and check they are wehin 102 pre-set limits to commenceE lecture Past Lectures menu

Choose location to P sea mediate (cinema layout)

Str 7ro VWchLive

Choose instructor HUD (age and ethnicity) IPause 2 Play 3. Skip back 10s 4. Change speed (0 75x. 125x) 5. Restart Lecture Live / in real time 10 6EY monitoring of body 102 for data relaxation I meditative response Questlos for ecture end

Store data i n dashboard, data 108 available for clinician and user Printable version of content and 3D demonstrations to be available

FIG. 1 C

SYSTEM, METHOD AND VIRTUAL REALITY DEVICE FOR ASSESSING COMPLIANCE OF BODY MOVEMENT FIELD OF THE INVENTION

[0001] The present invention relates to a system, method and Virtual Reality (VR) device for assessing compliance of body movement. More particularly this invention relates to a computer implemented VR system, method and a device for assessing compliance of a subject's body movement based on received data from one or more wearable sensor disposed on the subject and overlaying on a displayed ideal or target image a real-time subject image simulating the body movement of the subject.

BACKGROUND TO THE INVENTION

[0002] The provision of physical therapy is known to achieve beneficial results in the treatment of injuries, ageing, diseases and disorders and maintaining general good health. Often regular and ongoing treatment can achieve results that will not be achieved with infrequent or no treatment. The provision of physical therapy (physiotherapy) achieves beneficial results in the treatment of acute and chronic injury, the maintenance of health during aging and in the management of other chronic health conditions such as Parkinson's Disease, Rheumatoid Arthritis and similar. The travel to a clinician, the time, the cost involved and recently, infection-control issues such as, in an ongoing pandemic, are barriers to effective therapy.

[0003] US Patent Publication No. 2021/0085220, the publication of US Patent Application No. 17/117,602 to Tornier, Inc., discloses a computing system used to obtain motion data describing a movement of an appendage of a patient and determining, based on the motion data, a range of motion of the appendage. An extended reality visualization of the range of motion of the appendage superimposed on an image of the patient or an avatar of the patient is then generated, for display by an extended reality visualization device.

[0004] US Patent No. 9641805 to Microsoft Technology Licensing, LLC teaches a tele immersive environment providing interaction among participants of a tele-immersive session. A mirror functionality is provided for presenting a three-dimensional virtual space for viewing by a local participant. The virtual space then shows at least some of the participants as if the participants were physically present at a same location and looking into a mirror.

[0005] US Patent No. 9747722 to Relexion Health, Inc is directed to generating visual exercise or rehabilitation instruction in which user positional information is received from a user imaging system and then a first mirror image of the user positional information is generated, before a first instructional image having the same positional orientation as the first mirror image of the user positional information is also generated. Then a composite output is displayed which included the first mirror image of the user positional information and the first instructional image.

[0006] WO 2014/124002, the publication of International Patent Application No. PCT/US2014/014868, details custom software that provides off-sets between patient and normal subject range of motion capabilities to provide a visual representation of these differences.

[00071 WO 2013/022890, the publication of International Patent Application No. PCT/US2012/049857, to Gary and Mary West Wireless Health Institute, elucidates non-invasive motion tracking to augment patient administered physical therapy via a motion tracking apparatus, a display, and a computing platform coupled to the motion tracking apparatus and the display. A rehab knowledge database interacts with a rehab decision support module and various rehab exercises are provided, which interface with the user via an avatar and the motion capture system. When a user's body movement does not match the avatar's, onscreen audio and visual feedback is provided. A variety of specific body and or other features are used to track patient position and motion with a depth sensor and a camera array. Various rehab clinical tools may be incorporated into the system. The system may monitor the user's vital signs and/or activities to automatically detect an alert or alarm condition.

[0008] WO 2019/241799, the publication of International Patent Application No. PCT/US2019/037550, to Viribu VR Labs, LLC, details a virtual reality therapy system which includes a virtual reality headset, a garment such as a, vest, having at least one sensor detecting movement, and an electronic device. A virtual reality environment is projected to the virtual reality headset based upon data generated by the sensor of the garment.

[0009] WO 2019/173765, the publication of International Patent Application No. PCT/US2019/021439 to VR HEALTH LTD, explains the application of computer program products to physical therapy using Virtual Reality (VR) or Augmented Reality (AR), specifically, for guiding user motion for physiotherapy using a VR or AR headset. Movement of a user is tracked using motion sensor such as, hand sensors, sensors in the headset and additional sensor such as, torso sensors and a stereo camera. A compliance metric is determined such that when the compliance metric differs from a predetermined range, an adjustment is applied to the guided motion.

[0010] WO 2019/010545, the publication of International Patent Application No. PCT/AU 2018/050732 to Smileyscope Pty. Ltd., provides a virtual reality (VR) device, system configured to allow device control by an operator other than the wearer (i.e. a medical practitioner), to allow the operator to re-imagine a physical procedure such as, giving an injection or taking a blood sample.

[0011] Other technologies for providing remote physical therapy would be beneficial.

[0012] The reference to any prior art in this specification is not, and should not be taken as, an acknowledgement or any form of suggestion that the prior art forms part of the common general knowledge.

SUMMARY OF THE INVENTION

[0013] The present invention is directed to a system, method and VR device for assessing compliance and providing visual feedback to allow for the correction of body movement.

[0014] In a broad form the invention relates to a VR based computer implemented system, method and a device for assessing compliance of a subject's body movement based on received data from one or more wearable sensor disposed on the subject and overlaying on a displayed ideal or target image a real-time subject image simulating the body movement of the subject. The received data may comprise real-time data and time-series data.

[0015] In a first form, although it need not be the only or indeed the broadest form, the invention resides in a VR based computer implemented method for assessing compliance of a subject's body movement, the method comprising: receiving data from one or more wearable sensors disposed on the subject, the received data comprising time-series data received prior to the execution of the body movement and real-time data received during execution of said body movement; displaying an ideal or target image for the body movement; and overlaying on the displayed ideal or target image a real-time subject image simulating the body movement of the subject during the subject's execution of the body movement to assess compliance, the real-time image constructed from the time-series data.

[0016] In a second form, the invention resides in a computer system for assessing compliance of a subject's body movement, the system comprising: an input for receiving data from one or more wearable sensors disposed on the subject, the received data comprising time-series data received prior to the execution of the body movement and real-time data received during execution of said body movement; and a display for displaying an ideal or target image for the body movement and overlaying on the displayed ideal or target image a real-time subject image simulating the body movement of the subject during the subject's execution of the body movement to assess compliance, the real-time image constructed from the time-series data.

[0017] In a third form, the invention resides in a non-transitory computer program product comprising: a computer usable medium and computer readable program code embodied on said computer usable medium for assessing compliance of a subject's body movement, the computer readable program code comprising: computer readable program code devices (i) configured to cause a computing device to receive data from one or more wearable sensors disposed on the subject, the received data comprising time-series data received prior to the execution of the body movement and real-time data received during execution of said body movement; computer readable program code devices (ii) configured to cause a computing device to display an ideal or target image for the body movement; and computer readable program code devices (ii) configured to cause a computing device to overlay on the displayed ideal or target image a real-time subject image simulating the body movement of the subject during the subject's execution of the body movement to assess compliance, the real-time image constructed from the time-series data.

[0018] In a fourth form, the invention resides in a virtual reality device for assessing compliance of a subject's body movement, the device comprising: one or more wearable sensors for disposition on the subject, the one or more wearable sensors providing received data comprising time-series data received prior to the execution of the body movement and real-time data received during execution of the body movement; and a display for displaying an ideal or target image for the body movement and for overlaying on the displayed ideal or target image a real-time subject image simulating the body movement of the subject during the subject's execution of the body movement to assess compliance, the real-time image constructed from the time-series data.

[0019] According to any one of the above forms, the display may allow for correction of the body movement and/or provide visual feedback.

[0020] According to any one of the above forms, the assessment may also comprise assessment of accuracy of the body movement.

[0021] The time series data may be acquired over minutes, hours, days, weeks, months, or years before execution of the body movement. The time series data may be averaged over one or more time period such as, minutes, hours, days, weeks, months, or years.

[0022] According to any one of the above forms, the ideal or target image may be correlated with the time-series data. The correlation may comprise an increased or decreased stringency or an increased or decreased overall body movement-based exertion. The exertion maybe relate to a number of repetitions, sets of repetitions, presence or amount of additional weight or resistance, or range of motion. The correlation may be directly related to a health assessment made based on the time-series data set. The health assessment may be conducted using one or more of: automatically determined using one or more algorithm; clinician directed; or user directed.

[0023] According to any one of the above forms, the one or more wearable sensors may comprise any type of wearable sensor that can provide data useful to health monitoring, physical therapy and/or rehabilitation. The one or more wearable sensor may be comprised on a smartwatch or other proprietary wearable device or may be a standalone sensor. In a particular embodiment, the one or more wearable sensor comprises at least one movement sensor such as, an accelerometer, a magnetometer and a gyroscope, and EMG sensor and body temperature sensors or any combination thereof. In some applications a plurality of moment sensors are comprised.

[0024] The one or more wearable sensors 150 according to any one of the above forms may monitor one or more of: heart rate; one or more respiratory rate measure; blood pressure; body temperature; muscle tension; oxygen saturation; sleep; restful sleep; wakefulness; length of sleep; and type of sleep.

[0025] In one embodiment of any one of the above forms, receiving reciprocal real time data from one or more wearable sensors outside of the execution of the body movement may be comprised. The reciprocal real time data may be from one or more wearable sensor which does not directly detect musculo-skeletal movement such as, one or more of body temperature, blood pressure, pulse (heart rate), and breathing rate (respiratory rate), and electromyography (EMG), oxygen saturation and blood glucose level (BGL).

[0026] According to any one of the above forms, the one or more wearable sensors may comprise a contemporaneous subset and a time-series subset. The contemporaneous subset may provide the real time data. The time-series subset may provide the time series data.

The contemporaneous subset and the time-series subset may comprise the same subset of one or more wearable sensors 150 or a different subset of the one or more wearable sensors 150.

[0027] According to any one of the above forms, the received data may be stored in one or more database. The received data may be confidential. The received data may be used to obtain population or cohort combined data.

[0028] In one embodiment of any one of the above forms, one or both of the real-time subject image and the ideal or target image may comprise a 3D image.

[0029] In another embodiment of any one of the above forms, one or both of the real-time subject image and the ideal or target image may be dynamic or static.

[0030] In yet another embodiment of any one of the above forms, the displayed real-time image may be modified using one or more model of weight gain and/or obesity such as, on one or more joint, muscle or tendon.

[0031] In still another embodiment of any one of the above forms, the assessment may further comprise a measurement of one or more parameter in the ideal or target image and a measurement of the one or more parameter in the real-time subject image and displaying a calculated difference.

[0032] In still another embodiment of any one of the above forms, the time-series data may be used to assess a subject health parameter such as, measured subject anxiety, stress, activity, sleep measured subject apparent age and/or measured subject energy-level. The measurement may be based on one or more measured change in time. The measured change in time may comprise a measurement using one or more data set averaged over a time period.

[0033] According to any one of the above forms, the pre-set and received data may be monitored against one or more stop and/or go parameter. The one or more stop and/or go parameter may be personalised and/or an alarm may be provided for outside the pre-set/ predetermined levels.

[0034] In one embodiment of any one of the above forms, the overlayed real-time subject image may be compared to received data to provide an indication of increased muscle tension and one or more system impacted by stress such as, increased respiratory rate and/or increased heart rate and increased body temperature.

[0035] In yet another embodiment of any one of the above forms, the body movement, is a musculo-skeletal movement such as, a cough, a pulmonary clearance technique, a puffer use, or use of an associated pulmonary or respiratory aid.

[0036] In still another embodiment of any one of the above forms, an image modelled on the subject avatar modified to display an impact of a subject health parameter in real time may be displayed. The one or more affects may be assessed using the received data. The subject health parameter may comprise one or more of measured subject anxiety, measured subject apparent age and/or measured subject energy-level, stress, activity level and sleep.

[0037] Further aspects and/or features of the present invention will become apparent from the following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order that the invention may be readily understood and put into practical effect, reference will now be made to embodiments of the present invention with reference to the accompanying drawings, wherein like reference numbers refer to identical elements. The drawings are provided by way of example only.

[0039] Figures 1A; 1B; IC; andID are component parts of a flowchart showing one embodiment of a method according to the invention.

[0040] Figures 2A and 2B show one embodiment of a personal computing device, computer system and computer processor according to the invention.

[0041] Skilled addressees will appreciate that elements in the drawings are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the relative dimensions of some elements in the drawings may be distorted to help improve understanding of embodiments of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[0042] The present invention relates to a system, method and VR device for assessing compliance and accuracy of body movement. As will be elucidated below, in one embodiment, this invention relates to a computer implemented system and method and a device for assessing compliance and accuracy of a subject's body movement based on received data from one or more wearable sensor disposed on the subject and overlaying on a displayed ideal or target image a real-time subject image simulating the body movement of the subject allowing for visual feedback and correction of said movement.

[0043] Conveniently, the method may be implemented with a personal computing device in a user's house or any other desirable location the user can take their portable personal computing device to. This key advantage of the present invention provides much needed flexibility and may aid, or at least encourage, observance of physical therapy or exercise routines and increased self-management of chronic health conditions and better health outcomes.

[0044] Figures 1A; IB; and IC show one embodiment of a computer implemented method 100 for assessing compliance of a subject's body movement. Figure 1A is continuous with Figure IB, with the bottom box of Figure 1A connected with an arrow to the top box of Figure 1B. Figure IC is continuous with Figure IB, with the top, second top and lower boxes of Figure 1B connected with arrows to the top, second top and third top box of Figure IC.

[0045] As shown in Figure 1A, an initial set up is conducted for new users, which may include a test for motion sickness and a tutorial.

[0046] Figures 1A andlB illustrate that method 100 comprises receiving 102 data from one or more wearable sensors 150 (not shown) disposed on the subject. The received data may comprise time-series data received prior to the execution of the body movement and real-time data received during execution of the body movement.

[0047] Turning to Figure IC, method 100 is shown to comprise displaying 104 an ideal or target image for the body movement and overlaying 106 on the displayed ideal or target image a real-time subject image simulating the body movement of the subject during the subject's execution of the body movement to assess compliance. The steps of displaying 104 and overlaying 106 may be conducted concurrently, so that they are presented to a user as one step 104, 106.

[0048] One of the advantages of the present invention is that the time series data may be acquired over minutes, hours, days, weeks, months, or years . The longer the duration of use of method 100, the greater and more powerful the stored time-series data set is. Just one of the benefits of the stored time series data is that it may be averaged over one or more time period such as, minutes, hours, days, weeks, months, or years.

[0049] Another illustration of this advantage of the invention is that the ideal or target image may be correlated with the time-series data. The correlation may comprise an increased or decreased stringency or an increased or decreased overall body movement based exertion. The exertion maybe relate to a number of repetitions, sets of repetitions, presence or amount of additional weight or resistance, or range of motion. The correlation may be directly related to a health assessment made based on the time-series data set. The health assessment may be conducted using one or more of: automatically determined using one or more algorithm; clinician directed; or user directed.

[0050] The one or more wearable sensors 150 may comprise any type of sensor that can provide data useful to health monitoring, physical therapy and/or rehabilitation. From the teaching herein, a skilled person is readily able to select suitable sensors for use in the invention. The one or more wearable sensors 150 may be comprised on a smartwatch or other proprietary wearable device or may be a standalone sensor. In one example, the one or more wearable sensor 150 comprises at least one movement sensor such as, an accelerometer, a magnetometer and a gyroscope, EMG sensor; body temperature sensor or any combination thereof.

[0051] The one or more wearable sensors 150 may also monitor one or more of: heart rate; one or more respiratory rate measure; blood pressure; body temperature; muscle tension; oxygen saturation; sleep; restful sleep; wakefulness; length of sleep; and type of sleep.

[0052] From the teaching herein a skilled person is readily able to select a suitable type and number of wearable sensors 150. For example, the number of movement sensors, and their location and relative density, may be selected based on the type of movement being executed.

[0053] Although not limited thereto, the body movement is generally a musculo-skeletal movement. The invention may find particular application to musculo-skeletal body movements such as, a cough; a pulmonary clearance technique; a puffer use; or use of an associated pulmonary or respiratory aid.

[0054] Method 100 may comprise receiving reciprocal real time data from one or more wearable sensors 150 outside of the execution of the body movement. The reciprocal real time data may be from one or more sensor 150 which does not directly detect musculo skeletal movement such as, one or more wearable sensor 150 measuring one or more of body temperature, blood pressure, pulse (heart rate), breathing rate (respiratory rate), and electromyography (EMG), oxygen saturation and blood glucose level (BGL).

[0055] The one or more wearable sensors 150 may comprise a contemporaneous subset and a time-series subset. The contemporaneous subset may provide the real time data. The time-series subset may provide the time series data. The contemporaneous subset and the time-series subset may comprise the same one or more wearable sensors 150 or a different subset of one or more wearable sensors 150.

[0056] As shown in Figure IC, the received data may be stored 108 in one or more database 292. The received data may be stored on a confidential basis. Advantageously, the received data may be used to obtain population or cohort combined data to be used in health population planning.

[0057] To aid understanding, one or both of the real-time subject image and the ideal or target image may be displayed as a 3D image.

[0058] Another advantage of the present invention is that one or both of the real-time subject image and the ideal or targe image may be displayed either dynamically or as a static image.

[0059] As a motivational or educational tool, the displayed real-time image may be modified using one or more model of weight gain and/or obesity such as, on one or more joint, muscle or tendon to highlight changes to the users health if such changes occur.

[0060] The assessment may further comprise a measurement of one or more parameter in the ideal or target image and a measurement of the one or more parameter in the real-time subject image and displaying a calculated difference. This can be measured over time to show improvement or deterioration.

[0061] Yet another advantage of the present invention is that the time-series data may be used to assess a subject health parameter such as, measured subject anxiety, measured subject apparent age and/or measured subject energy-level. The measurement may be based on one or more measured change in time. The measured change in time may comprise a measurement using one or more data set averaged over a time period.

[0062] To limit the risk of injury or undesirable exertion, the received data may be monitored against one or more stop and/or go parameter, which may be optionally personalised and/or an alarm provided to indicate times outside pre-set data ranges.

[0063] Advantageously, the overlayed real-time subject image may be compared to received data to provide an indication of increased muscle tension and one or more system impacted by stress such as, increased respiratory rate and/or increased heart rate.

[0064] Yet another advantage of the present invention is that an image modelled on the subject avatar modified to display an impact of a subject health parameter in real time may be displayed. The one or more affects may be assessed using the received data. The subject health parameter may comprise one or more of measured subject anxiety, measured subject apparent age and/or measured subject energy-level.

[0065] Also provided is a computer system 200 for assessing compliance and accuracy of a subject's body movement, the system comprises an input for receiving data from one or more wearable sensors 150 disposed on the subject, the received data comprising time- series data received prior to the execution of the body movement and real-time data received during execution of said body movement

[0066] System 200 further comprises a display 214 for displaying an ideal or target image for the body movement and overlaying on the displayed ideal or target image a real time subject image simulating the body movement of the subject during the subject's execution of the body movement to assess compliance and movement accuracy.

[0067] One example of a computing device and computer system of the invention is shown in FIGS. 2A and 2B.

[0068] Computer system 200 suitable for use in the present invention is shown in Figs. 2A and 2B. In the embodiment shown computer system 200 comprises a personal computing device 201 comprising input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, an external hard drive 227, and a microphone 280; and output devices including a printer 215, a display device 214 and loudspeakers 217. In some embodiments video display 214 may comprise a touchscreen.

[0069] A Modulator-Demodulator (Modem) transceiver device 216 may be used by the personal computing device 201 for communicating to and from a communications network 220 via a connection 221. The network 220 may be a wide-area network (WAN), such as the Internet, a cellular telecommunications network, or a private WAN. Through the network 220, personal computing device 201 may be connected to other similar personal devices 290 or server computers 291 or database 292. Where the connection 221 is a telephone line, the modem 216 may be a traditional "dial-up" modem. Alternatively, where the connection 221 is a high capacity (e.g.: cable) connection, the modem 216 may be a broadband modem. A wireless modem may also be used for wireless connection to network 220.

[0070] The personal computing device 201 typically includes at least one processor 205, and a memory 206 for example formed from semiconductor random access memory (RAM) and semiconductor read only memory (ROM). The personal computing device 201 also includes a number of input/output (I/O) interfaces including: an audio-video interface 207 that couples to the video display 214, loudspeakers 217 and microphone 280; an I/O interface 213 for the keyboard 202, mouse 203, scanner 226 and external hard drive 227; and an interface 208 for the external modem 216 and printer 215. In some implementations, modem 216 may be incorporated within the personal computing device 201, for example within the interface 208. The personal computing device 201 also has a local network interface 211 which, via a connection 223, permits coupling of the personal computing device 201 to a local computer network 222, known as a Local Area Network (LAN).

[0071] As also illustrated, the local network 222 may also couple to the wide network 220 via a connection 224, which would typically include a so-called "firewall" device or device of similar functionality. The interface 211 may be formed by an Ethernet circuit card, a Bluetooth wireless arrangement or an IEEE 802.11 wireless arrangement or other suitable interface.

[0072] The I/O interfaces 208 and 213 may afford either or both of serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standards and having corresponding USB connectors (not illustrated).

[0073] Storage devices 209 are provided and typically include a hard disk drive (HDD) 210. Other storage devices such as, an external HD 227, a disk drive (not shown) and a magnetic tape drive (not shown) may also be used. An optical disk drive 212 is typically provided to act as a non-volatile source of data. Portable memory devices, such as optical disks (e.g.: CD-ROM, DVD, Blu-Ray Disc), USB-RAM, external hard drives and floppy disks for example, may be used as appropriate sources of data to the personal computing device 201. Another source of data to personal computing device 201 is provided by the at least one server computer 291 through network 220.

[0074] The components 205 to 213 of the personal computing device 201 typically communicate via an interconnected bus 204 in a manner that results in a conventional mode of operation of personal computing device 201. In the embodiment shown in FIGS. 2A and 2B, processor 205 is coupled to system bus 204 through connections 218. Similarly, memory 206 and optical disk drive 212 are coupled to the system bus 204 by connections 219. Examples of personal computing devices 201 on which the described arrangements can be practiced include IBM-PC's and compatibles, Sun Sparc stations, Apple computers; smart phones; tablet computers or like a device comprising a computer module like personal computing device 201. It is to be understood that when personal computing device 201 comprises a smart phone or a tablet computer, display device 214 may comprise a touchscreen and other input and output devices may not be included such as, mouse pointer device 203; keyboard 202; scanner 226; and printer 215.

[0075] Fig. 2B is a detailed schematic block diagram of processor 205 and a memory 234. The memory 234 represents a logical aggregation of all the memory modules, including the storage device 209 and semiconductor memory 206, which can be accessed by the personal computing device 201 in Fig. 2A.

[0076] The methods of the invention may be implemented using personal computing device 201 wherein the methods may be implemented as one or more software application programs 233 executable within personal computing device 201. In particular, the steps of the methods of the invention may be effected by instructions 231 in the software carried out within the personal computing device 201.

[0077] The software instructions 231 may be formed as one or more code modules, each for performing one or more particular tasks. The software 233 may also be divided into two separate parts, in which a first part and the corresponding code modules performs the method of the invention and a second part and the corresponding code modules manage a graphical user interface between the first part and the user.

[0078] The software 233 may be stored in a computer readable medium, including in a storage device of a type described herein. The software is loaded into the personal computing device 201 from the computer readable medium or through network 221 or 223, and then executed by personal computing device 201. In one example the software 233 is stored on storage medium 225 that is read by optical disk drive 212. Software 233 is typically stored in the HDD 210 or the memory 206.

[0079] A computer readable medium having such software 233 or computer program recorded on it is a computer program product. The use of the computer program product in the personal computing device 201 preferably effects a device or apparatus for implementing the methods of the invention.

[0080] In some instances, the software application programs 233 may be supplied to the user encoded on one or more disk storage medium 225 such as a CD-ROM, DVD or Blu Ray disc, and read via the corresponding drive 212, or alternatively may be read by the user from the networks 220 or 222. Still further, the software can also be loaded into the personal computing device 201 from other computer readable media. Computer readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and/or data to the personal computing device 201 or computer system 200 for execution and/or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, DVD, Blu-ray Disc, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the personal computing device 201. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of software application programs 233, instructions 231 and/or data to the personal computing device 201 include radio or infra-red transmission channels as well as a network connection 221, 223, 334, to another computer or networked device 290, 291 and the Internet or an Intranet including email transmissions and information recorded on Websites and the like.

[0081] The second part of the application programs 233 and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon display 214. Through manipulation of, typically, keyboard 202, mouse 203 and/or screen 214 when comprising a touchscreen, a user of personal computing device 201 and the methods of the invention may manipulate the interface in a functionally adaptable manner to provide controlling commands and/or input to the applications associated with the GUI(s). Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilising speech prompts output via loudspeakers 217 and user voice commands input via microphone 280. The manipulations including mouse clicks, screen touches, speech prompts and/or user voice commands may be transmitted via network 220 or 222.

[0082] When the personal computing device 201 is initially powered up, a power-on self test (POST) program 250 may execute. The POST program 250 is typically stored in a ROM 249 of the semiconductor memory 206. A hardware device such as the ROM 249 is sometimes referred to as firmware. The POST program 250 examines hardware within the personal computing device 201 to ensure proper functioning, and typically checks processor 205, memory 234 (209, 206), and a basic input-output systems software (BIOS) module 251, also typically stored in ROM 249, for correct operation. Once the POST program 250 has run successfully, BIOS 251 activates hard disk drive 210. Activation of hard disk drive 210 causes a bootstrap loader program 252 that is resident on hard disk drive 210 to execute via processor 205. This loads an operating system 253 into RAM memory 206 upon which operating system 253 commences operation. Operating system 253 is a system level application, executable by processor 205, to fulfill various high level functions, including processor management, memory management, device management, storage management, software application interface, and generic user interface.

[083] Operating system 253 manages memory 234 (209, 206) in order to ensure that each process or application running on personal computing device 201 has sufficient memory in which to execute without colliding with memory allocated to another process. Furthermore, the different types of memory available in the personal computing device 201 must be used properly so that each process can run effectively. Accordingly, the aggregated memory 234 is not intended to illustrate how particular segments of memory are allocated, but rather to provide a general view of the memory accessible by personal computing device 201 and how such is used.

[084] Processor 205 includes a number of functional modules including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes called a cache memory. The cache memory 248 typically includes a number of storage registers 244, 245, 246 in a register section storing data 247. One or more internal busses 241 functionally interconnect these functional modules. The processor 205 typically also has one or more interfaces 242 for communicating with external devices via the system bus 204, using a connection 218. The memory 234 is connected to the bus 204 by connection 219.

[085] Application program 233 includes a sequence of instructions 231 that may include conditional branch and loop instructions. Program 233 may also include data 232 which is used in execution of the program 233. The instructions 231 and the data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending upon the relative size of the instructions 231 and the memory locations 228-230, a particular instruction may be stored in a single memory location as depicted by the instruction shown in the memory location 230. Alternately, an instruction may be segmented into a number of parts each of which is stored in a separate memory location, as depicted by the instruction segments shown in the memory locations 228 and 229.

[086] In general, processor 205 is given a set of instructions 243 which are executed therein. The processor 205 then waits for a subsequent input, to which processor 205 reacts by executing another set of instructions. Each input may be provided from one or more of a number of sources, including data generated by one or more of the input devices 202, 203, or 214 when comprising a touchscreen, data received from an external source across one of the networks 220, 222, data retrieved from one of the storage devices 206, 209 or data retrieved from a storage medium 225 inserted into the corresponding reader 212. The execution of a set of the instructions may in some cases result in output of data. Execution may also involve storing data or variables to the memory 234.

[087] The disclosed arrangements use input variables 254 that are stored in the memory 234 in corresponding memory locations 255, 256, 257, 258. The described arrangements produce output variables 261 that are stored in the memory 234 in corresponding memory locations 262, 263, 264, 265. Intermediate variables 268 may be stored in memory locations 259, 260, 266 and 267.

[088] The register section 244, 245, 246, the arithmetic logic unit (ALU) 240, and the control unit 239 of the processor 205 work together to perform sequences of micro operations needed to perform "fetch, decode, and execute" cycles for every instruction in the instruction set making up the program 233. Each fetch, decode, and execute cycle comprises: (a) a fetch operation, which fetches or reads an instruction 231 from memory location 228, 229, 230; (b) a decode operation in which control unit 239 determines which instruction has been fetched; and (c) an execute operation in which the control unit 239 and/or the ALU 240 execute the instruction.

[089] Thereafter, a further fetch, decode, and execute cycle for the next instruction may be executed. Similarly, a store cycle may be performed by which the control unit 239 stores or writes a value to a memory location 232.

[090] Each step or sub-process in the methods of the invention may be associated with one or more segments of the program 233, and may be performed by register section 244 246, the ALU 240, and the control unit 239 in the processor 205 working together to perform the fetch, decode, and execute cycles for every instruction in the instruction set for the noted segments of program 233.

[091] One or more other computers 290 may be connected to the communications network 220 as seen in Fig. 2A. Each such computer 290 may have a similar configuration to the personal computing device 201 and corresponding peripherals.

[092] One or more other server computer 291 may be connected to the communications network 220. These server computers 291 respond to requests from the personal computing device 201 or other server computers to provide information.

[093] Method 100 may alternatively be implemented in dedicated hardware such as one or more integrated circuits performing the functions or sub functions of the described methods. Such dedicated hardware may include graphic processors, digital signal processors, or one or more microprocessors and associated memories.

[094] It will be understood that in order to practice the methods of the invention as described above, it is not necessary that the processors and/or the memories of the processing machine be physically located in the same geographical place. That is, each of the processors and the memories used in the invention may be located in geographically distinct locations and connected so as to communicate in any suitable manner.

Additionally, it will be understood that each of the processor and/or the memory may be composed of different physical pieces of equipment. Accordingly, it is not necessary that a processor be one single piece of equipment in one location and that the memory be another single piece of equipment in another location. That is, it is contemplated that the processor may be two pieces of equipment in two different physical locations. The two distinct pieces of equipment may be connected in any suitable manner. Additionally, the memory may include two or more portions of memory in two or more physical locations.

[095] To explain further, processing as described above is performed by various components and various memories. It will be understood, however, that the processing performed by two distinct components as described above may, in accordance with a further embodiment of the invention be performed by a single component. Further, the processing performed by one distinct component as described above may be performed by two distinct components. In a similar manner, the memory storage performed by two distinct memory portions as described above may, in accordance with a further embodiment of the invention, be performed by a single memory portion. Further, the memory storage performed by one distinct memory portion as described above may be performed by two memory portions.

[096] Further, various technologies may be used to provide communication between the various processors and/or memories, as well as to allow the processors and/or the memories of the invention to communicate with any other entity, i.e., so as to obtain further instructions or to access and use remote memory stores, for example. Such technologies used to provide such communication might include a network, the Internet, Intranet, Extranet, LAN, an Ethernet, a telecommunications network (e.g., a cellular or wireless network) or any client server system that provides communication, for example. Such communications technologies may use any suitable protocol such as TCP/IP, UDP, or OSI, for example.

[097] Also provided is a non-transitory computer program product comprising a computer usable medium and computer readable program code embodied on said computer usable medium for assessing compliance of a subject's body movement, the computer readable program code comprising computer readable program code devices (i) configured to cause a computing device to receive data from one or more wearable sensors 150 disposed on the subject, the received data comprising time-series data received prior to the execution of the body movement and real-time data received during execution of said body movement; and computer readable program code devices (ii) configured to cause a computing device to display an ideal or target image for the body movement; and computer readable program code devices (ii) configured to cause a computing device to overlay on the displayed ideal or target image a real-time subject image simulating the body movement of the subject during the subject's execution of the body movement to assess compliance.

[098] Also provided is a device for assessing compliance of a subject's body movement, the device comprises one or more wearable sensors 150 for disposition on the subject, the one or more sensors 150 providing received data comprising time-series data received prior to the execution of the body movement and real-time data received during execution of the body movement; and a display for displaying an ideal or target image for the body movement and for overlaying on the displayed ideal or target image a real-time subject image simulating the body movement of the subject during the subject's execution of the body movement to assess compliance and accuracy of movement.

Examples

[099] The following non-limiting examples illustrate the invention. These examples should not be construed as limiting: the examples are included for the purposes of illustration only. The Examples will be understood to represent an exemplification of the invention.

[0100] The invention is exemplified with reference to the non-limiting exemplars of: Chronic back pain; arthritis; COPD (chronic obstructive pulmonary disease) and/or asthma; and anxiety and depression and/or mental health. Chronic backpain

[0101] Software will be written and/or one or more wearable sensors 150 provided, to allow the one or more wearable sensors 150 to send real time data regarding length of time sitting, length of time standing and sustained postures throughout the day. This will be able to identify lifestyle factors that can be altered to manage on going pain.

[0102] Software will also be written and/or one or more wearable sensors 150 provided, to identify anxiety and depression associated with the increased focus on pain indicated by altered heart rate and respiratory rate measures, BP, body temperature and muscle tension.

[0103] Software will further be written, or one or more wearable sensors 150 provided, to identify in real time the changes to this anxiety with use of VR guided meditation within the program - measures of heart rate, respiratory rate, BP, body temperature and muscle tension.

[0104] Additional software will be written, and/or one or more wearable sensors 150 will be provided, to allow the user avatar and exercise instructor avatar to be compared or mirrored, to ensure exercises are done correctly, with the correct posture, the correct range of movement of joints and the correct number of repetitions.

[0105] All this data will be able to be reproduced for users records and use by health professionals involved in their care. It will show the association between postural, lifestyle and the anxiety measures and the focus on pain via graphs and charts. 3D images of the user's avatar will be available to identify the areas of increased muscle tension and systems impacted by stress such as increased respiratory rate and increased heart rate. Arthritis

[0106] Software will be written to show, using 3D image of a user avatar, the impact of weight gain and obesity on the affected joints. COPD/Asthma

[0107] Software will be written and/or one or more sensors 150 provided to allow the one or more wearable sensors 150 to measure and record respiratory rate, heart rate and increased use of pulmonary muscles which can all indicate some level of respiratory distress.

[0108] Software will also be written and/or one or more wearable sensors 150 will be provided to allow one or more wearable sensors 150 to measure and record oxygen saturation levels.

[0109] Additional software will be written and/or one or more wearable sensors 150 will be provided to incorporate a stop/go alert system for these measures. If the data is outside of a pre-determined limit, then an alert will be sent to the user and a medical review suggested.

[0110] Additional software will be written and/or one or more wearable sensors 150 will be provided to to allow the user avatar to mirror the instructor avatar with exercise program to ensure correct technique and speed of exercise.

[0111] Software will be written and/or one or more wearable sensors 150 will be provided to to allow 3D images to be available of the user avatar, which will allow the identification of areas of increased muscle use and showing increased respiratory rate as it occurs.

[0112] In this specification, the terms "comprises", "comprising" or similar terms are intended to mean a non-exclusive inclusion, such that an apparatus that comprises a list of elements does not include those elements solely, but may well include other elements not listed.

[0113] Throughout the specification the aim has been to describe the invention without limiting the invention to any one embodiment or specific collection of features. Persons skilled in the relevant art may realize variations from the specific embodiments that will nonetheless fall within the scope of the invention.

Claims

CLAIMS The Claims defining the invention are as follows:

1. A VR based computer implemented method for assessing compliance of a subject's body movement, the method comprising:

receiving data from one or more wearable sensors disposed on the subject, the received data comprising time-series data received prior to the execution of the body movement and real-time data received during execution of said body movement;

displaying an ideal or target image for the body movement;

overlaying on the displayed ideal or target image a real-time subject image simulating the body movement of the subject during the subject's execution of the body movement to assess compliance, the real-time image constructed from the time-series data.

2. A VR based computer system for assessing compliance of a subject's body movement, the system comprising:

an input for receiving data from one or more wearable sensors disposed on the subject, the received data comprising time-series data received prior to the execution of the body movement and real-time data received during execution of said body movement; and

a display for displaying an ideal or target image for the body movement and overlaying on the displayed ideal or target image a real-time subject image simulating the body movement of the subject during the subject's execution of the body movement to assess compliance, the real-time image constructed from the time-series data.

3. A non-transitory VR based computer program product comprising:

a computer usable medium and computer readable program code embodied on said computer usable medium for assessing compliance of a subject's body movement, the computer readable program code comprising:

computer readable program code devices (i) configured to cause a computing device to receive data from one or more wearable sensors disposed on the subject, the received data comprising time-series data received prior to the execution of the body movement and real-time data received during execution of said body movement; computer readable program code devices (ii) configured to cause a computing device to display an ideal or target image for the body movement; and computer readable program code devices (ii) configured to cause a computing device to overlay on the displayed ideal or target image a real-time subject image simulating the body movement of the subject during the subject's execution of the body movement to assess compliance, the real-time image constructed from the time-series data.

4. A virtual reality device for assessing compliance of a subject's body movement, the device comprising:

one or more wearable sensors for disposition on the subject, the one or more wearable sensors providing received data comprising time-series data received prior to the execution of the body movement and real-time data received during execution of the body movement; and

a display for displaying an ideal or target image for the body movement and for overlaying on the displayed ideal or target image a real-time subject image simulating the body movement of the subject during the subject's execution of the body movement to assess compliance, the real-time image constructed from the time-series data.

5. The method, system, computer program product or virtual reality device according to any one of the claims 1 to 4 wherein the ideal or target image is correlated with the time series data and/or wherein the correlation comprises an increased or decreased stringency or an increased or decreased overall body movement-based exertion.

FIG. 1A 100 1/5

FIG. 1B 2/5