WO2005006171A1

WO2005006171A1 - Control area selection in a computing device with a graphical user interface

Info

Publication number: WO2005006171A1
Application number: PCT/GB2004/003008
Authority: WO
Inventors: Graham Phillip Oldfied
Original assignee: Symbian Software Limited
Priority date: 2003-07-10
Filing date: 2004-07-12
Publication date: 2005-01-20
Also published as: GB0415559D0; EP1646932A1; GB0316122D0; GB2404127A; GB2404127B; JP2007527046A

Abstract

The control areas (e.g. keys and buttons) in a GUI of a touch screen based computing device represented in a colour mask stored on the device; the colour mask has regions exactly corresponding in shape and size to each of the control keys or buttons; each region is completely filled with a unique colour. In use, when a X,Y co-ordinate pair for a particular contact/touch point is generated, it is straightforward and very fast to extract the pixel colour that is present at the same X,Y co-ordinates of the colour mask. It is also then very fast and straightforward to establish the control area associated with that colour by matching the pixel colour to a table that stores each control area together with its associated colour in the colour mask.

Description

CONTROL AREA SELECTION IN A COMPUTING DEVICE WITH A GRAPHICAL USER INTERFACE

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to a method of control area selection in a computing device with a graphical user interface, particularly a device with a touch screen.

2. Description of the Prior Art

A common problem in Symbian OS and other GUI (graphical user interface) based operating systems for computing devices is determining what control area in a GUI has actually been selected by a user, e.g. what particular control key or button is meant to be selected when a user taps a touch sensitive display screen. When a stylus taps the display screen, the touch position is translated into an X and Y co-ordinate. Touch screens typically comprise an ITO layer overlying a display (such as a LCD) and are pressure sensitive. When pressure is applied to the ITO overlay, the top flexible surface is locally deformed so that it makes contact with a stiff /solid base layer. An analogue voltage level across the ITO coating on the inside surface layer is interpreted by the device to give X, Y coordinates for the pressure point.

The X, Y co-ordinate is then delivered by the OS to the current window that fell under the co-ordinate. The window will have a number of control areas and each will typically have to be interrogated to find out which one falls under the co-ordinates.

Figure 1 shows a UI control design that one may want to implement to allow user input. The control in Figure 1 defines seven control areas that the user can select; these are Up / Down / Left/ Right/ Confirm / Yes / No. To represent these control areas in the OS, each is defined very approximately as a rectangle. For example the 'Up' control would be recorded as a rectangle with its top left co-ordinate pair as X 68, Y 4 and its bottom left co-ordinate pair as X 111, Y 29, as shown in Figure 2. Figure 2 shows all seven rectangles for the seven different control areas.

Given that each of the seven selectable control areas is loosely defined as a rectangle, we can now determine which area was selected by a pen /stylus. When the user touches the display screen overlying a given control area, X and Y co-ordinates where the pen touched are generated. The device then examines each rectangle to see if the X, Y co- ordinates fall within the bounds of that rectangle. A typical algorithm to determine if a X Y co-ordinate pair falls within the bounds of a rectangle is: 1. Is the User selected X co-ordinate greater than a control area's top left X coordinate. 2. Is the User selected X co-ordinate less than a control area's bottom right X co-ordinate. 3. Is the User selected Y co-ordinate greater than a control area's top left Y coordinate. 4. Is the User selected Y co-ordinate less than a control area's bottom right Y co-ordinate. If all the above conditions are true then we know that the user has selected that control area.

This algorithm must be run across every rectangle to discover which control area the user has selected. Given the complexity of many control designs and the consequent large number of rectangles, this can be a relatively time consuming processes. Another disadvantage is that the only portions of control areas that you can capture are rectangles; this means graphically intensive control designs, such as control buttons with complex shapes, can be hard to capture pointer events from accurately. Figure 3 shows a region where the user selected the 'down' control; this event would be missed as it is outside the bounds of the 'down' rectangle. Whilst its possible to capture user input with rectangles, there are therefore 3 main drawbacks:

• Hit areas have to be rectangular. Arbitrary shaped controls cannot be selected accurately.

• An algorithm examining 2 pairs of co-ordinates must be run across every hit area.

• It is time consuming to change the design of the control as all of the co-ordinate pairs that describe the hit rectangles must be recalculated.

SUMMARY OF THE INVENTION

In a first aspect, a method of establishing which control area shown on a display of a computing device has been selected by a user, in which each of several different selectable control areas is associated with one of a set of unique colours in a colour mask stored in device memory and not shown on the display, the colour mask being made up of regions that each correspond to one of the control areas and are each coloured in one of the unique colours; comprises the steps of: (a) generating a set of co-ordinates for a contact location on the display; (b) retrieving the colour mask colour at that set of co-ordinates; (c) establishing the control area which is associated with the same colour as the retrieved colour.

Hence, instead of making a loose rectangular approximation to a control key or button, the present invention requires a colour mask with a region preferably exactly corresponding in shape and size to that control key or button to be created and stored in memory; the region is completely filled with a unique colour. Any other control keys or buttons visible at the same time are given corresponding uniquely coloured regions in the colour mask. The device will hence store a set of different colour masks, each corresponding to a different control key/button arrangement. Some colour masks may be burned in device ROM (e.g. for GUI control designs that are meant always to be available); others may be part of application software loaded into the device by an end- user, where the colour masks relate to GUI control designs specific to that application software. The colour masks are not displayed on the screen at all and are hence entirely 'virtual'.

In use, when a XN co-ordinate pair for a particular contact/ ouch point is generated, it is straightforward and very fast to extract the pixel colour that is present at the same N co-ordinates of the appropriate colour mask. From there, it is also very fast and straightforward to establish the control area associated with that colour. For example, imagine a control design in which an icon is displayed in the center of the screen. The icon is a complex shape, such as a flower. If the flower is selected, then an imaging application opens. There are no other control areas on the same screen. Then, a colour mask in exactly the shape of the flower is created when designing the device or the imaging application; the internal parts of the flower are given a colour, say blue, in the mask. The rest of the colour mask if black. The colour mask is stored in device memory; it is not displayed on screen at all.

When the device is in use, the flower icon is displayed. The user touches any part of the flower: the touch co-ordinates are generated and then used to retrieve the colour in the colour mask at the same co-ordinates. If the colour mask colour is blue, then the device compares this to the possible colour s (blue and black) using a table which also associates each colour with a control area (e.g. blue = flower; black = no function). It can then rapidly determine that the flower has been selected. If the colour mask colour is black, then the device knows that the user has touched the screen outside of the flower icon. If the control design is complex, examining all the different rectangles using a prior art approach would be complex and time consuming. With the colour mask approach of the present invention, we just compare the different colour areas. This is fast and computationally efficient; especially valuable advantages in a resource constrained computing device such as a smattphone or other kind of mobile telephone.

The term 'computing device' covers any kind of computing device that a person interacts with by selecting control areas that are shown on a display. As such, the term covers PDAs, mobile telephones, dedicated devices such digital audio players, remote control units, etc. The term 'control area' should be expansively construed to cover any object which a user would 'select' in some way (e.g. choose in order to: (a) input data, such as text/ numeric data; (b) initiate an action; (c) control or manipulate a function; or (d) to identify data which is to be manipulated). It therefore covers keys or tiles from a keyboard (such as letter or number keys), a control button or slider, a jog dial, a joystick, or a roller button etc. The term 'control area' also covers items that do not correspond with objects as such; in particular, it covers any icon or image which a user might select. BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be described with reference to the accompanying drawings, in which:

Figure 1 shows the GUI control design appearing on part of a computing device screen; the control design has seven different control areas.

Figure 2 shows the prior art rectangle based approach to establishing which of the seven control areas has been selected by a user; Figure 3 shows how the prior art does not cope well with complex control areas;

Figure 4 shows a colour mask as used in the present invention; there are seven different control regions, each the exact same shape and size as a corresponding control area in the GUI control design that appears on the display;

Figure 5 shows how the device stores the original control design bitmap, the colour mask and a table of colours linked to control areas;

Figure 6 is a flow chart of the key steps in the present invention;

Figure 7 shows a different GUI control design with the same kinds of control areas as the Figure 1 GUI control design;

Figure 8 shows how this different GUI control design can be simply implemented by using a different colour mask.

DETAILED DESCRIPTION

The present invention will be described with reference to an implementation that can run on the Symbian OS operating system from Symbian Limited, London, United Kingdom. Symbian OS is designed for smartphones and other mobile telephones: the limited processing power available on these kinds of devices makes the present invention especially valuable.

Colour Mask Solution

The colour mask solution replaces the multiple rectangle hit area approach with a colour mask stored in memory. The colour mask is a direct representation of the control areas we are intending to capture input from, with different regions corresponding to each different control area. Creating the mask is simply a case of getting the original bitmap of the control areas and then filling in the corresponding colour mask regions that we require capturing user input from with a solid colour.

In Figure 4 we can see a colour mask representing the control in Figure 1. Any colour mask region that we do not want to count as user input, we colour black. We can then mark the colour mask regions we are interested in different colours, for example Figure 4 shows red is confirm, yellow is left, blue is right, purple is down etc.

In SymbianOS, each colour is stored as 32 bit unsigned integer with the format BB GG RR (Blue Green Red) so primary red woμld be OxOOOOff, green OxOOffOO and blue would be OxffOOOO. We store in device memory a table of the known region colours as member variables associated with a reference to a corresponding control area, as shown in Table 1 below.

Table 1 Const TUint KYesAreaColour = 0xf0ff05;/ /Cyan Const TUint KNoAreaColour = 0x80c5f6; //beige Const TUint KUpAreaColour = OxOOffDO; //Green Const TUint KDownAreaColour = 0xf773d0; //purple Const TUint KLeftAreaColour = Ox03faf7; //yellow Const TUint KRightAreaColour = OxffOOOO; //Blue Const TUint KConfirmAreaColour = OxOOOOff; //Red Const TUInt KblacklgnoreArea = 0x000000; //Black

The colour mask is also stored in device memory as a member variable.

The way we use the colour mask is as follows. If the user selects the 'left' control area by tapping on it with a stylus, an application would receive an event with the co-ordinates x 58, y 41 defining the stylus contact position. The program establishing the control area which has been selected then asks for the pixel in the colour mask at location x 58 y 41. As noted earlier, the colour mask is not actually shown on the display but exists only in memory: the colours deployed in the colour mask bear no relationship to the colours shown on the display.

In Symbian OS, all bitmaps are represented in memory as collections of pixels, (pixels are unsigned integers that represent the colour in the same format as we stored our known colours). Symbian OS and other GUI based Operating Systems provide a quick method to retrieve a pixel from a bitmap in memory and it is this functionality we use to retrieve the pixel at x58 y41.

In this case, the pixel at X58 y41 would be retrieved from the colour mask as TUint 0x03faf7 (corresponding to the colour yellow). The program would then use as statement as follows:

Void UserSelectedControl(Tint XselectedCoordinate, Tint YselectedCoordiante)

{ const TUint selectedPixel = MyOffScreenMask->GetPixel(XselectedCoordinate, YselectedCoordiante ); switch(selectedPixel)

{ case KblacklgnoreArea: //black //No control area selected we can ignore event break case KyesAreaColour: DoYesActionO; break; case KNoAreaColour: DoNoActionO; break; case KUpAreaColour: DoUpActionO; break; case KdownAreaColour: DoDownActionQ; break; case KleftAreaColour: DoLeftActionO; break; case KtightAreaColour: DoRightActionO; break; Default: break;

}; } In this case the pixel value 0x03faf7 would be matched to the KLeftAreaColour variable (0x03faf7) as shown in Table 1 (i.e. the program matches the pixel colour yellow to the control area associated with that colour, namely the 'left' control area). Hence, the program would determine that the user has selected 'left'. The relevant part of the program statement is shown in bold. The key steps are summarised in Figure 6.

So long as each region in the colour mask has a unique value, any colour (which term also includes grey scale) combination can be used.

This approach is more efficient in terms of processing time as we are comparing one unsigned integer against integers for each of the different regions (which each have a 1:1 correspondence to a control area). If the control design is complex, examining all the different rectangles using a prior art approach would be complex and time consuming. With the colour mask approach we just compare the different colour areas.

Customisation

Another advantage of the colour mask approach is that the look and feel of the control design can be completely changed, by simply updating the mask and the original control bitmap.

Figure 7 shows a complete re-design of the original control (in Figure 1). This new design however still has exactly the same seven control areas as the original control design e.g. up / down /left / right etc.. This new design would be extremely difficult to represent with the rectangle approach as it contains many non rectangular hit areas.

To update this control design with the colour mask approach simply requires creating a new colour mask bitmap that represent the new control design. The new colour mask is shown in Figure 8 and could be created by somebody with little technical knowledge in a paint application. The only criteria is that exactly the same colour Values are used again e.g. 'Left' arrow control region is 0x03faf7 (yellow). With the new Control bitmap and colour mask added to the application no code changes are required. This is because the colour mask approach only needs to know which colours represent which control action (e.g. as shown in Table 1).

Claims

1. A method of establishing which control area shown on a display of a computing device has been selected by a user, in which each of several different selectable control areas is associated with one of a set of unique colours in a colour mask stored in device memory and not shown on the display, the colour mask being made up of regions that each correspond to one of the control areas and are each coloured in one of the unique colours; comprising the steps of: (a) generating a set of co-ordinates for a contact location on the display; (b) retrieving the colour mask colour at that set of co-ordinates; (c) establishing the control area which is associated with the same colour as the retrieved colour.

2. The method of Claim 1 in which the colour mask is obtained using a bit map of the control areas.

3. The method of Claim 2 in which a table of the set of unique colours is stored in device memory, together with a reference to each associated selectable control area.

4 The method of Claim 3 in which each of the unique colours in the table is represented as an unsigned integer.

5. The method of Claim 4 in which each of the unique colours in the colour mask is represented as an unsigned integer and the unsigned integer representing the colour at the set of co-ordinates is compared against each unsigned integer in the table until a match is found.

6. The method of Claim 5 in which, when a match is found, the corresponding selectable control area is then established using the table.

7. The method of Claim 1 in which a selectable control area can be any arbitrary shape so long as the colour mask region corresponding to that arbitrary shape can be filled with a single colour.

8. The method of Claim 2 in which the arrangement or design of the different selectable control areas is updatable to a different arrangement or design by altering the bit map of the control areas and the colour mask.

9. The method of Claim 8 in which altering the bit map of the control areas and the colour mask is performed using a paint application.

10. A computing device adapted to establish which control area shown on a display of the computing device has been selected by a user, in which each of several different selectable control areas is associated with one of a set of unique colours in a colour mask stored in device memory and not shown on the display, the colour mask being made up of regions that each correspond to one of the control areas and are each coloured in one of the unique colours; the device being adapted to: (a) generate a set of co-ordinates for a contact location on the display; (b) retrieve the colour mask colour at that set of co-ordinates; (c) establish the control area which is associated with the same colour as the retrieved colour.

11. The device of Claim 10 in which the colour mask is obtained using a bit. map of the control areas.

12. The device of Claim 11 in which a table of the set of unique colours is stored in device memory, together with a reference to each associated selectable control area.

13. The device of Claim 12 in which each of the unique colours in the table is represented as an unsigned integer.

14. The device of Claim 13 in which each of the unique colours in the colour mask is represented as an unsigned integer and the unsigned integer representing the colour at the set of co-ordinates is compared against each unsigned integer in the table until a match is found.

15. The device of Claim 14 which, when a match is found, establishes the corresponding selectable control area using the table.

16. The device of Claim 10 in which a selectable control area can be any arbitrary shape so long as the colour mask region corresponding to that arbitrary shape can be filled with a single colour.

17. Application software programmed to run on a computing device, in which the application software causes each of several different selectable control areas to be

^■ displayed on the device and comprises a colour mask, the colour mask being made up of regions that each correspond to one of the control areas and are each coloured in a unique colour and the colour mask being stored in device memory and not shown on the display.