WO2003079184A2

WO2003079184A2 - Monitoring software and method

Info

Publication number: WO2003079184A2
Application number: PCT/GB2003/001127
Authority: WO
Inventors: Bernard Harvey Gaus; Callum Thomas Peter Kennedy
Original assignee: Avoca Systems Limited
Priority date: 2002-03-18
Filing date: 2003-03-18
Publication date: 2003-09-25
Also published as: GB0206339D0; AU2003212537A8; GB0306183D0; GB2390916A; AU2003212537A1; WO2003079184A3

Abstract

Disclosed is a computer program providing an interface between a first application, which communicates with a user via a user interface, and a second application on a windowing computer having an operating system, the program comprising a monitoring software module for monitoring messages generated by the operating system for messages associated with the window control for said one application and, in response to such a message being detected, using the operating system to obtain from the user interface the value of the control for use by the second application. There is also disclosed a computer program comprising a monitoring software module operable to monitor messages generated by a windowing computer operating system for messages for filling one or more pre-determined criteria, the criteria including at least the messages associated with a particular window control, the program preferably further comprising queryable storage means for storing a value associated with the window control, and updating means associated with the monitoring software module for updating the value stored by the queryable storage means upon receipt of a message which is associated with a change in the value associated with the window control.

Description

Monitoring Software and Method

Field of the invention

The invention relates to the field of monitoring input by a user and output from a computer. In particular it relates to software and a method for extracting or acting on information input by a user, or displayed to a user, with diverse applications.

Background to the invention

A common computing problem is enabling systems using diverse software and hardware implementations to work together. For example, at present time, different elements of the British National Health Service use a variety of different software packages, and different hardware installations to store information about patients. Individual general practitioners, consultants and departments have their own legacy systems. As with many computer systems, it is desirable to find a way to enable these systems to be used together enabling, for example, the provision of uniform access to patient records.

In the software industry, there are a number of methods of dealing with diverse software and hardware which are in common use. For instance, most modern computer programs are written with an application programming interface (API), that enables other applications to retrieve information from them and/or write information to them. Unfortunately, this facility is not present in many legacy systems.

Where an application does not have an appropriate API, a computer programmer will usually consider whether they could directly access a database used by that application. This is typically carried using a language such as Structured Query Language (SQL). However, it is not always possible to access an underlying database. For example, the underlying database may be remotely located, encrypted, or not contain all the information that is needed.

A further known technology used when there is no API, and direct access to an underlying database is not practical, is so called screen-scraping technology. Screen scrapers are used in networks and emulate clients and send instructions to a server, monitoring the data that is returned to them, extracting relevant data from, for example, ASCII text intended for screen presentation.

However, screen-scraping technology is also of only limited applicability. Screen scraping is typically used in a batch mode and is not generally suitable for real time use; furthermore, one can only retrieve information in the form in which the server transmits it. Yet further, screen-scraping is difficult to implement with applications running in a modern windowing operating system, such as Microsoft^® Windows^® NT, XWindows, MacOS^® or PalmOS^®.

One method of retrieving information from a computer system is to use a so-called keylogger program. Keylogger programs store all information typed in to a computer by a user. It is possible to use such programs to establish information which has been input by a user into an application. Unfortunately, such programs are impractical to use in a windowing operating system where the user (or an automatic action) may switch input from one window to another. The resulting stream of keypresses is of little practical value.

Summary of the Invention

According to the first aspect of the present invention, there is provided a computer program for providing an interface between a first application, which communicates with a user via a user interface, and a second application on a windowing computer having an operating system, the program comprising a monitoring software module for monitoring messages generated by the operating system for messages associated with a window control for said first application and, in response to such a message being detected, using the operating system to obtain from the user interface the value of the control for use by the second application.

Thus the software module can obtain the users input and commands in respect of the first application, and relays these for the second application to use. However, only inputs or commands associated with window controls are obtained. For example, the software module could obtain text input into a dialogue box only when the computer generates a message indicating that the user is no longer inputting or editing text in that box. For example, the user may have pressed the tab-key to move the text input/edit cursor to a different field or may have clicked on the "OK" button displayed on a dialogue box. Thus, by selecting which types of message are to trigger interrogation of a user interface, the invention can be configured to avoid unnecessary processing of user inputs.

According to a second aspect of the present invention there is provided a computer program comprising a monitoring software module operable to monitor messages generated by a windowing computer operating system for messages fulfilling one or more predetermined criteria, the criteria including at least that the message is associated with a particular window control, the program preferably also comprising queryable storage means for storing a value associated with the window control, and updating means associated with the monitoring software module for updating the value stored by the queryable storage means upon receipt of a message which is associated with a change in the value associated with the window control.

Advantageously, this means that not only can user input be monitored, but that it can be given a context. Messages preferably include messages relating to user input events, such as key presses and mouse operations. An application can therefore, by querying the queryable storage means, find out what information has been input selected or retrieved by a user. In contrast to a keylogger, the stored value is associated with a known window control. Preferably, the queryable storage means can persist, even once the window control has been deleted. This enables an application to access information recently input by a user even if that information is no longer stored anywhere else on that computer.

Window controls are controls which relate to elements of a user interface in a windowing environment and have properties. Example window controls include text boxes, scroll bars, list boxes, progress bar controls etc. which are found in various windowing operating systems. Example sets of window controls include the Windows 2000^® common controls and Windows^® user controls available from Microsoft^® (Redmond, WA). Window controls are often referred to in the art as "widgets" .

The value associated with a window control depends on the type of window control and may be a property of the window control. For example, it may be a text string equal to the current value of the text property of a text box control. Furthermore, the value associated with the window control may be a value which is not a property of the window control itself but requires deduction. For example, a queryable storage means might record the time at which a window control was created. Window controls do not typically include as a property the time at which they were created but, by storing this information in a queryable storage means, an application such as a logger of time spent working on an application may be readily built. Other examples values associated with window controls include: the application which created the control and the value of a check box (established by monitoring check and uncheck messages)

It is to be understood that in this context, the term 'value' is a qualitative or quantitative reference to a property. Thus a character string will have a value which is unique to that string and from which the string can therefore be deduced.

A queryable storage means may be associated with a plurality of window controls, in which case it may be adapted to store a value associated with the last of the plurality of window controls to change. The plurality of window controls need not be in existence at the same time. For example, the plurality of window controls may be "Save" dialogue boxes generated in a plurality of different circumstances by one or more applications. In the preferred embodiment, the monitoring software module is also adapted to generate one or more events upon detecting a message fulfilling one or more predetermined criteria associated with that message, the criteria including at least that the message is associated with one or more particular window controls. Preferably, these one or more events are generated in a monitor application.

Thus, the monitor application receives events enabling it to respond to specific user events in specific contexts. The monitor application may respond by querying a storage means. For example, an event may be generated every time a windows message BUTTONUP is generated on a control of class CBTN (button), with the caption: OK, having been created by a specified application, whereupon the monitor application can carry out a intervention, for example displaying a dialogue box stating: "are you sure?"

Preferably, the monitoring software module is embedded within an ActiveX^® control embedded within the monitor application, thereby enabling an event driven monitor application which is COM+ aware, such as one written in the Visual Basic^® development system, Delphi^®, Microsoft^® Access database environment or Microsoft^® Word for Windows^®, to receive events generated by monitoring software module, and also to query, through an ActiveX^® interface, the queryable storage means.

An event may also be generated by the monitor application itself. For example, the monitor application may generate an event every time it is selected from a toolbar. An example application of this would be a time logging program, which, when selected from a tool bar, queries a queryable storage means which has monitored the time that each application on the system has been opened and then queries a second queryable storage means which has recorded the most recently opened application, and can therefore deduce the amount of time the user has spent on the application which they previously had activated. Preferably, the queryable storage means are also writable to by either or both applications and controls. Preferably also, the updating means includes computer software to set a property of a window control associated with a specific queryable storage means.

For example, in response to an event that the user has brought up a specified dialogue box, the monitor application may retrieve some information, or make a calculation, and then write to the observer, the value of text or numeric data which should be input into a particular text box control. The updating means will then amend the value of the window control to correspond to this new value. This enables a monitor application to be written which can retrieve information or make calculations and automatically set the input which should be made by the user.

In a more detailed example, the user of a hospital administration computer program brings up a dialogue box asking for the user to input the name of a patient who is checking in to a hospital, the consultant they are to see, the type of operation for which they are being admitted and the length of time they are likely to stay. The monitoring software module determines that the user has input each of the above data items except the length of time they are likely to stay and then generates an event in a monitor application. The monitor application then calculates the expected length of time that the patient will stay and writes that to the queryable storage means, whereupon the updating means writes the length of time to the control which specifies the dialogue box. Thus it is possible to automate one aspect of data entry without having to customise the hospital administration computer program.

The monitoring software module may store data retrieved from queryable storage means. For example, it may monitor a form completed by a user, and store that data in a remote database. Preferably, the queryable storage means also stores properties of window controls which were generated by an application. Therefore, it is possible to store both input and output of third party applications, providing another method to record data from third party or legacy systems. Preferably, the monitoring software module monitors a messaging queue. Preferably, the operating system is Microsoft^® Windows^®. Preferably, the monitoring software module uses a hook to retrieve messages from the windows messaging queue. Preferably also the monitoring software module then returns the messages to the windows messaging queue. Preferably, the CBT (computer based training) hook in Microsoft^® Windows^® is used.

Preferably also, the monitoring software module is operable to monitor messages on the computer for messages relating to the creation of new windows controls. This enables the monitoring software module to learn the identifier of a windows control. Preferably, the one or more predetermined criteria include data which can be used to identify the window control when it is created, and, by monitoring the creation of new windows controls, the monitoring software module can determine the identifier of a window control for use with those predetermined criteria. An example criterion is the text displayed by a window control. Preferably, however, the criteria include the window control's parent window control and, optionally, its parent window control and so forth, all the way back to the desktop.

For example, in Microsoft^® Windows^®, each window control is allocated an identifier property, hWnd, when it is created. hWnd is different each time. Thus, a window control cannot be specified in advance by its hWnd property. However, if a new window control having a specific parent window control is created, the hWnd property of the new window can be stored and the monitoring software module can thereafter identify messages relevant to the window control with that hWnd property. The predetermined criteria may include information about other properties of the window control as its parent window control may not be enough to identify it uniquely.

Importantly, a specific event generated by the monitoring software module may be associated with more than one set of predetermined criteria. For example, a single type of event (intended to indicate the user has started work on a document) may be generated by the monitoring software module in response to the user selecting NEW from the FILE menu or OPEN from the FILE menu in Microsoft^® Word for Windows^®. Preferably, the software module is operable to intercept all messages placed to be acted upon by said first application, those messages not associated with window controls relevant to the second application being returned to the queue for onward processing, whilst messages which are relevant to the data to be captured for the second application cause the module to capture said data before being returned to the queue.

Preferably, the monitoring software module resides in the operating system kernel, where it can operate quickly with minimal effect on the apparent speed of the computer on which it is executed.

According to a further aspect of the present invention, there is provided computer software comprising training software means, which, when executed on a computer that is further executing a first software application which has window controls associated therewith, is user activatable to select a window control associated with the first software application, and to establish the identity of the selected window control.

Preferably, the computer program further comprises user interface means through which a user can associate selected window control with one or more actions. Preferably, the user interface means allows the user to relate a window control to one or more events to be generated in response to specific events related to the said window control.

Thus, the training software means can be used to identify the window controls that are generated during the execution of the first application, and to generate events in the computer programs of the first or second aspect, in response to specific events generated in relation to the identified window control, when the first application is executed.

Preferably, a window control is identified by specifying a hierarchy of window parameters that define a hierarchy of window controls. A window control within a hierarchy of window controls is associated with a parent or owner window control (or the desktop). Windows within the hierarchy are identified by their parameters, so that the computer program according to the first or second aspect of the present invention can identify when the said window control, or a window control equivalent thereto is created during execution of the first application, even though a particular window control will generally have a different window handle every time it is created.

Detailed description of an example embodiment

An example embodiment of the present invention will now be illustrated with reference to the following Figures in which:

Figure 1 is a schematic diagram of a workstation; and

Figure 2 is a schematic diagram of software running on a computer.

Figure 1 is a schematic diagram of a workstation 1 comprising a computer 2 and a screen 4 showing the desktop 6 of a Microsoft^® Windows^® operating system 7 installed on the computer 2. Microsoft^® Windows^® displays are made up of various window controls, known in the art as "widgets" , for example a window 8 and various user interface elements 10, such as text boxes, scroll bars, menus, buttons and the like. The user may interact with the computer 2 using user input peripherals including a keyboard 12 and mouse 14.

Figure 2 is a schematic diagram of software running on the computer 2. A monitor application 20 and a third party application 22 are both able to interact with software and hardware devices through the operating system 7.

The operating system 7 maintains information about window controls 26. The operating maintains a message queue 28 which stores messages 30 relevant to window controls 26. Windows controls 26 capture interface actions and post corresponding messages to the message queue 28 until they reach the head of the queue whereupon they are processed by the third party application 22. A monitoring software module 100 is installed in the operating kernel. The monitoring software module 100 is written in a low-level language such as C+ + and makes data available to an ActiveX^® control 24 which is embedded within the monitor application 20.

The system is adapted to enable the monitoring software module 100 to monitor messages 30. This is achieved by use of Windows hooks, which are a mechanism by which a function can intercept events before they reach the application 22. The monitoring software module 100 is attached to the CBT (Computer Based Training) Hook which was chosen as it provides a broad range of messages. Thereafter, the monitoring software module 100 is called in response to all CBT hook messages. The response of the monitoring software module 100 is discussed below.

The module 100 stores a list of observers and triggers against which each intercepted message in turn is tested to determine whether the value of the message is to be stored in queryable storage means (also part of the module 100) or whether an intervention or data capture operation is to be performed before the message is returned to the queue 28, and hence before the application 22 can act on the message.

Each observer stored in the module 100 contains class defining data that includes the "widget route" from the desktop to the windows widget being monitored. The widget route is a list of a set of window parameters that uniquely and recursively identifies a window and its parent in a route that leads from the desktop to each child window or widget until the route reaches the destination window or widget.

For example, the following table shows the data that is always available when a window is created in Microsoft^® Windows^®. Any or all of this data, together with the data contained in the structures may be used to identify Windows on a Widget Route.

To optimise recognition of the creation of windows that are on a widget route, the module 100 maintains a record of the windows handles (hWnd) of the relevant windows as they created along a path. A windows handle is a number created by the operating system at the time of creation of a window that uniquely identifies that particular window.

The observer also includes the windows handle for a given widget, a list of messages that trigger events, the events they trigger and a list of dead events and the name of the observer.

For ease of reference, the last window in a windows route, i.e. the window for which the observer is created, will be referred to as a destination window. The information enabling the observers to be set up in the module 100 are stored on a disk file.

When the module 100 is activated, it reads the disk file into its memory and checks the "widget route" of every new window created. If the window being created is a destination window then a corresponding observer is activated and the windows handle (created at window creation time) captured and applied to the observer. When the window is subsequently destroyed, the observer is destroyed too.

Multiple incidences of windows created (e.g. two copies of notepad running simultaneously) cause the software module to initiate multiple incidences of the observer. The observers are used as "data sources" and each watches the message queue 28 for messages that affect the current display value of the underlying widget (e.g. the text in a text box).

The triggers stored in the module are set up in a similar fashion, and cause the module to perform a pre-set function when specific messages are posted from associated widgets to the queue 28. The module checks all the messages that originate from the widgets which might provide a trigger message and, on detecting such a message, performs the tasks trained by the user as explained below. Examples of trigger messages are the user clicking the "OK" button on a window or closing the window. The events are implemented through the ActiveX control 24. The ActiveX control has properties that also allow the monitor application 20 to query or set the value for each observer.

In use, messages read from the queue 28 and are checked against the observers/triggers 32 to determine what actions if any needs to be taken.

Messages can result in the creation or destruction of observers, the updating of observer value information, an event or multiple events being fired (i.e. initiated) and, optionally, the return of a message to the queue. The return of a message is optional because the firing of an event may cause the message to be replaced by an alternative message in the other queue. Thus, for example, a user inputting a "minimised window" command into the application 22 could find that the application 20 overrides this command and, for example, provides the user with another prompt or a message.

All messages in the queue 28 are checked against the following criteria. 1) Is a message a "window request"? If it is, is the parent (identified by each hWnd) part of a widget route. If so, is the new window still on the widget route? If the new window is the end of the path a module creates an observer and labels it with the hWnd of the created window. If it is not the end of a path, then the module 100 makes a note of the hWnd as a marker on the widget route to simplify the above mentioned test on the parent.

2) Does the hWnd match one of the existing observers 32? If not, the module returns the message to the queue 28.

3) Is the message a "destroy window" request? Is so, the module 100 destroys the associated observer. Otherwise the module ascertains whether the hWnd marker is on a widget route. If it is, the module replaces the marker with its parent.

4) The module 100 checks whether the message is a "data modification" message. If it is, the module then obtains a new "value" of the window and updates the observer 32.

5) The module also checks whether the message is in the list of messages that have already been defined (in training the module) as an event for this observer. If it is, then the event is initiated.

6) The module also checks whether the message is in the list of messages that have already been defined as a "dead end message" from any given observer. If it is not, then the message is returned to the queue.

Reference 2 denotes the "updating means" which is used to update a stored value relating to an observer when the value is changed by the receipt of an associated message in the queue 28. The program may also include a training application which is used to teach the module 100 which controls needs to be monitored and what events (if any) need to be performed in response to a given input via the application 22. When setting up the module in this way, the user, first starts the training application and then minimise it. The user then starts the application that the data agent needs to learn about and navigates through the application to a screen containing widgets of interest. When the computer screen is showing those widgets, the user activates the training application which changes the cursor shown on the computer screen to a cross hare. The user then clicks the cross-hair cursor on each widget of interest (the module then highlighting the widget to show which widget is working with) and the user completes a dialogue box describing what the user wants to do with that widget. The dialogue box can produce a list of possible events to be associated with a given input.

One of the new messages which can be inserted into the message queue by module 100 is a command for the operating system to read the text typed into a input window by the user and pass that data to the monitor application 20 (in a format suitable for the monitor applications).

Claims

1. A computer program for providing an interface between a first application, which communicates with a user via a user interface, and a second application on a windowing computer having an operating system, the program comprising a monitoring software module for monitoring messages generated by the operating system for messages associated with a window control for said first application and, in response to such a message being detected, using the operating system to obtain from the user interface the value of the control for use by the second application.

2. A computer program comprising a monitoring software module operable to monitor messages generated by a windowing computer operating system for messages fulfilling one or more pre-determined criteria, the criteria including at least that the message is associated with a particular window control, the program further comprising queryable storage means for storing a value associated with the window control, and updating means associated with the monitoring software module for updating the value stored by the queryable storage means upon receipt of a message which is associated with a change in the value associated with the window control.

3. A computer program according to Claim 2, wherein the queryable storage means can persist once the window control whose associated value it stores, has been deleted.

4. A computer program according to Claim 2 or Claim 3, wherein the value associated with the window control is not a property of the window control, but is deduced by the computer program in relation to the window control.

5. A computer program according to any one of Claims 2 to 4, wherein the queryable storage means is associatable with a plurality of window controls.

6. A computer program according to Claim 5, wherein the queryable storage means is adapted to store a value associated with the window control of the plurality of window controls which most recently changed.

7. A computer program according to any one of Claims 2 to 6, wherein the queryable storage means are writable to by either or both applications and controls.

8. A computer program according to any one of Claims 2 to 7, wherein the updating means includes computer software to set a property of a window control associated with a specific queryable storage means.

9. A computer program according to any one of Claims 2 to 8, wherein the monitoring software module stores data retrieved from queryable storage means.

10. A computer program according to any one of Claims 2 to 9, wherein the monitoring software module is operable to monitor messages on the computer for messages relating to the creation of new windows controls.

11. A computer program according to any one preceding Claim, wherein the monitoring software module is adapted to generate one or more events upon detecting a message fulfilling one or more pre-determined criteria associated with that message, the criteria including at least that the message is associated with one or more particular window controls.

12. A computer program according to Claim 11, wherein the one or more events generated upon detecting a message for filling one or more pre-determined criteria are generated in a monitor application.

13. A computer program according to Claim 12, when dependent on Claim 2, wherein the monitor application is operable to respond to one or more events by querying a storage means.

14. A computer program according to Claim 12 or Claim 13, wherein an event is generated by the monitor application.

15. A computer program according to any one of Claims 11 to 14 wherein the one or more pre-determined criteria include data which can be used to identify the window control that is created, and, the monitoring software module monitors the creation of new window controls and thereby determines the identifier of a window control for use with those pre-determined criteria.

16. A computer program according to any one of Claims 11 to 15, wherein a particular event generated by the monitoring software module may be associated with more than one set of pre-determined criteria.

17. A computer program according to any one preceding Claim, wherein the monitoring software module monitors messages in a queue.

18. A computer program according to any one preceding Claim, wherein the software module is operable to intercept all messages placed in a queue to be acted upon by said first application, those messages not associated with windows controls relevant to the second application being returned to the queue for normal processing, whilst messages which are relevant to the data to be captured for the second application cause the module to capture said data before being returned to the queue.

19. A computer program according to any one preceding Claim wherein the monitoring software module resides in the operating system kernel of the computer.

20. Computer software comprising training software means, which, when executed on a computer that is further executing a first software application which has window controls associated therewith, is user activatable to select a window control associated with the first software application, and to establish the identity of the selected window control.

21. The computer program of Claim 20 further comprising user interface means through which a user can associate a selected window control with one or more actions.

22. A computer program according to Claim 21 wherein the user interface means allows the user to relate a window control to one or more events to be generated in response to specific events related to the said window control.

23. A computer program according to any one of Claims 20 to 22, wherein a window control is identified by a hierarchy of window parameters defining a hierarchy of window controls.