Personalizing Interaction Focused on a User’s Interactive Experience and Potential

Modern computer systems are very powerful and multi-functional. A main goal is to give the opportunity to users to handle complex systems, to present an interface the user wants to see, and to deliver desirable user experiences. Conventional Human Computer Interaction (HCI) mainly focusing on creating interactions to solve common problems rather than extending each user’s individual characteristics (McCarthy and Wright 2004) and lead to positive experience (Rogers et al. 2011, Heibeck et al. 2014). One good example is Photoshop, which is designed to allow the users to complete a particular task, which can have up to hundreds of functionalities (Haynes and Crumpler 2001) trough standard interaction model like dropdown menu, tool bar, shrotcut and so on (Evening 2005). Unfortunately, we lack an interaction design system that truly supports user specific interaction. Langdon et al. write that there has been no effective way to build human product interaction that can adapt to end users’ individual mental models (Langdon et al. 2012). The way that has been used is of is through “ontologies”, yet these ontologies can be hard to build, especially for the very people that are expert in the fields whose knowledge is being captured but who are not experienced in the specialised “modelling” field. In other words, the meanings of interactions are different to convey to divers of end users comprehensively. For that reason, the users have to spend lots of time and efforts to get familiar with an interactive system in order to operate it effective such as finding an appropriate functionality to complete a particular task.

As a result, the majority of its functionalities are rarely being used (Constantine and Lockwood 1999, Følstad et al. 2012) and only a small group of users found the system useful (Marshall et al. 2013). A significant question is that why are computers hard to use? And how to reduce gaps emerged between human and computer during the interaction (Norman 2002, Norman 2007, Nakken 2014). A challenge for HCI is how to create Human Computer Interaction in a personal way that leads to the users’ desirable experience and extend their potentials.

We need therefore, a scheme of methods to “present” typical functionalities to user in an effective and helpful way while users are using the computer. This requires getting the computer to cooperate with what the user think of the system and getting it to present reactions in a meaningful way. To do a good job, the system will have to understand something about what the person is trying to do and what sorts of results will be most interesting to them. For this, we need to adapt user personal model to organize a domain and interact with tools; be able to model the dialogues between a human and the provided system.

This research project aims to construct a personalized interaction between the users and computer as a way to integrate the user’s experiences and mechanisms of the existing application in a specific interaction context (Costabile et al. 2006). Bellotti et al. (2002) points out the most important questions that must be overcome by every designer and researcher developing user-centered systems.

1. 1.

   How does the user address the system?

2. 2.

   How does the system show the user that it is attending?

3. 3.

   How does the system understand commands?

4. 4.

   How does the user get feedback on command execution?

5. 5.

   How can the user fix errors?

This research provides an additional perspective on interaction design based on both designers’ and users’ point of view by adapting computer interactive system based on users’ personal needs. One particular purpose for this research is going to investigate how to integrate the user’s personal experience and behaviour. In this research, by looking beyond common notions of interactions (formalized action), we can increase the possibilities to open a personalizing interaction design system based on a novel co-development, co-creation, and co-ownership interaction composition. These significant features of the novel interaction are represented in the following:

• Co-ownership: the Interaction reflects and responds to the users’ thinking, problem and experience dynamically.

• Co-development: the interaction allows users to redefine a special meaning for the interaction by mapping, complementing and integrating existing elements of interaction.

• Co-design: the interaction can be changed by the users based on their interaction experience by customizing a particular interactive system.

Structure of constructing personalized Human Computer Interaction (Color figure online)

Full size image

The goal of research is to construct a user-oriented interaction which is called personalized interaction to make computer system easy, efficient and pleasure to the users. At the end of project, we will create new interactive drawing system that can integrate human cognition and behaviour to improve the users’ experience by allowing the user to personalize the system. Experiments showed positive user experience plays a key component to produce successful interaction between human and computer (Thüring and Mahlke 2007). In addition, it requires the HCI not only solve a particular task taking place in a certain context, but also fulfil the users’ individual needs like emotional expression (see Fig. 1).

The project will be carried out in two steps. The first step is to allow computers be able to “see and hear what the user does before they can prove truly helpful” (Pentland 1996). It relays on knowing the type of user; the user’s interaction pattern and the user’s experience level with the software. The second step is to provide accordingly patterns with corresponding function according to the user’s interaction behaviour in shaping software to meet the broad, varied, rapidly changing needs of the users (Rogers et al. 2011).

We use a set of state-of-the-art machine learning techniques to intelligently “learn” from a user’s behaviour, making predictions of user’s experience level, its typical pattern of usage, and most importantly, to generate a collaborative and personalized interaction to assist the user to have desirable experience. The data of learning the users’ dynamic interaction behaviour will be employed to define interaction behaviour algorithm and form in a way that he/she want to cope with.

User interactions with the system does not possess a linear path, it can comprised many “loops” and “branches” between user and system. The Fig. 2 shows a an example of this phenomenon: As one can see that, although these interactions are occurring in a time sequence, however, such behaviour can be drastically different to traditional time-series modelling, such as First order Auto Regressive Model or Hidden Markov Model (Ruber 1989) which assumes a linear progression of observations across time. For this reason, the modelling of complex time-sequence based interactions is of a challenging problem in machine learning research and needed to be studied closely in this project.

The sequence of interactions between user and a system

Full size image

The traditional time series modelling typically assumes the presentence of a series of latent nodes, for example, the syllables in speech recognition, which them can not be directly observed, but only can be inferred from the observation sequence themselves. In the setting of this project, such latent nodes may refer to the “user Intentions”, which may take on both discrete and/or continuous state space. Here we illustrate the graphical model for a traditional Hidden Markov Model: (Fig. 3)

The graphical model for a traditional Hidden Markov Model

Full size image

Part of the goal of this project from a machine learning perspective is to find ways to adept the traditional HMM to that of the complex user interaction sequences. If we were try to naively apply them directly, we end up of having the following probabilistic graphical model: (Fig. 4)

The graphical model for our proposed model

Full size image

Having stated this model, we must overcome the several scientific challenges in the realm of machine learning:

1. 1.

   If we choose the user intention as a discrete value, then, what is the number K, i.e., how many states should be present? What would be an optimal number in this setting

2. 2.

   How may one decide an appropriate likelihood function given that the observations is in a sequence of interactions between user and the system given a time window? The additional intricacy has been that the number of interactions at each time window may differ therefore, violating the traditional Hidden Markov Model in which the observations are generally of the same dimensionality (hence same family of distributions can be chosen).

In order to resolve these issues, we plan to employ a non-parametric approach, which is the state-of-the-art statistical method in which the number of states as well the parameters associated with each state can be inferred from the data itself. In particular, we will utilise the methods similar to those described in (Fox 2010), where state transitions matrix can potentially go infinite. In order to solve this intractable model, we will use the latest inference work, such as Variational Bayes (Jordan 2010) to provide a faster and accurate computation.

This process is iterative, in such a way that the learning and reinforcements from the users’ feedback are updated in an iterative manner. As a result, a personalized Human Computer Interaction can be established as the users know their own context and needs better than anybody else, and they often have real-time awareness of shifts in their respective domains.

To evaluate usability of personalized HCI and the user’s experience we are going to use a well-known evaluation analysis method - usability testing to test two types of drawing system by completing some drawing tasks. One is commercial drawing system- Photoshop CS5, the other one is a prototype of new drawing system created by our team.

To construct a suitable environment for usability testing we need to solve practical issues including designing typical tasks, selecting target users, prepare the testing conditions, setting up a variety of tests and dealing with the ethical issues.

The tasks for the usability testing are designed to compare users’ experiences when they creating a drawing using the above two drawing systems. There are three drawing tasks:

• Draw a picture by using different drawing tools (pencil, pen, oil pen and crayon).

• Draw multiple pictures with different styles.

• Customize the drawing system (circle, rectangle and square) to product a personal drawing pattern.

Next, we need to select suitable users to evaluate the systems - people that somehow represent those that the product is designed for. For example, some products are targeted at specific types of users like seniors, children, novices, or experienced people. In our case, the product is a digital drawing system, so the specific user audiences are computer users who use drawing systems.

In addition, users’ prior experience with a particular classes of product are different so selecting a range of users with different backgrounds is important. For example, a group of people who are using the web for the first time are likely to express different opinions to another group with five years of web experience.

To gather a range of views from different perspectives, we chose two different groups of people: novice users and experienced users with different backgrounds. These two groups of users participated in both prototype studies.

To achieve a gender balance, in the evaluation study we employed an equal number of males and females aged is between 21 and 38, including three males and three females from different disciplines and living areas. In the Hi-Fi prototype study, we recruited 30 representative users from, including 16 males and 14 females. All of them have different study backgrounds and nationalities.

Finally, it is necessary to prepare the test conditions and set up user tests for two studies in a lab situation. For both of the prototype studies, it requires the testing environment to be controlled to prevent unwanted disturbances and noise that may distort the result. As the product we are going to test will mainly be used in an office situation and environment we establish a simulated working environment by setting up a usability testing laboratory to carry out the above user tests. The facilities that have been used in laboratory are different for each of the two user tests. For the first test, after the user experimented with the prototype, we asked them questions to capture their opinions through a semi-structured interview. For the prototype evaluation, we set up a video camera in the laboratory to record all the data that the user generated during operating the provided prototype of the interaction language and the Photoshop CS5. After the user testing, the users were asked to complete the questionnaire.

Harker defines usability as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use (Harker 1995). In industry area, a well-known deification of the usability is ISO 9241 which is composed of three parts: effectiveness, efficiency and satisfaction (Abran et al. 2003).

According to Nilsson’s definition of usability, usability is composed of five quality components (Nielsen 1994):

Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?

Efficiency: Once users have learned the design, how quickly can they perform tasks?

Memorability: When users return to the design after a period of not using it, how easily can they re-establish proficiency?

Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?

Satisfaction: How pleasant is it to use the design?

From the above definitions we can see that there are several criteria for usability, but that they mainly relate to two key aspects:

• Usability of product including efficiency, learnability effectiveness; and

• User experience of using the product in a particular situation.

The results of user studies should provide measures of comprehensive base criteria of useability including completion effectiveness, efficiency related to the artefact usability and user experiences. Usability testing is typically carried out in laboratory settings. We designed multiple questions built upon the criteria of usability to evaluate the usability of the new drawing system.

For user experience study, the participants are required to complete a set of drawing tasks by using two digital drawing systems (Photoshop CS5 and a prototype that is built upon ILDP). User satisfaction is assessed based on the users’ feedback on their experience with the systems such as satisfaction, fun and/or frustration by using a well-designed questionnaire.

The purpose of data analysis is to develop an understanding or interpretation that answers the basic question being asked. B. Kaplan and J.A. Maxwell give five purposes for using qualitative methods in evaluating computer information systems (Kaplan and Maxwell 2005):

1. 1.

   Understanding how a system’s users perceive and evaluate that system and what meanings the system has for them.

2. 2.

   Understanding the influence of social and organizational context on systems use.

3. 3.

   Investigating causal processes.

4. 4.

   Providing formative evaluation that is aimed at improving a program under development, rather than assessing an existing one.

5. 5.

   Increasing the utilization of evaluation results.

As we have described, the data gathering method employed in this research is to set up appropriate user tests to assess whether the product we develop is usable and fitting for the intended user population to achieve their objectives. We set up two different prototype studies of a drawing system to clarify our design method. One is usability study and the other one is user experience study.

Accordingly, we employ different data analysis techniques to evaluate the data from each of the two prototype studies. Normally, there are four basic techniques of qualitative data analysis: coding, analytical memos, and contextual and narrative analysis (Kaplan and Maxwell 2005). These methods help us to identify themes, develop categories and explore similarities and differences in the data, and relationships among them.

The coding is created by the users when they doing the usability study focuses on evaluating usability of the domain specific interaction language, which has been developed from the ISO (ISO 9241 Usability Standard) base criteria of task effectiveness and efficiency. For example, we code the participants’ performance when they were required to complete few specific tasks by using the new drawing system refer to its usability, effectiveness and efficiency in a specified context of use.

6.1 Analytical Memos

The other analysis method is analytical memos. We asked the participants different questions about their responses to carry out the user experience study. Through these questions we are able to have a deeper understanding of what the participants think about of using the interactive artefact come through the specific interaction. The close questions focus on getting feedback about a particular design feature, such as the way of interaction that the interactive artefact provided. The open question related to exploring the users’ individual experiences how well a product supports them to complete a particular task and what other supports are needed. At a result, the general questions (open) and specific questions (close) that contribute to bring comprehensive feedback for the evaluation goal are asked.

6.2 Contextual and Narrative Analysis

For the Hi-Fi prototype study, we use contextual and narrative analysis method to analysis the data that is generated from the observation and user satisfaction questionnaire. Through the Hi-Fi prototype study, we category and analysis the results of user testing to evaluate whether the participants enjoyed using the new drawing system to improve their interaction experience by constructing personalized interaction.

In the Hi-Fi prototype study, we observe user using the Hi-Fi prototype to exam whether the developing product meets users’ needs. We use video to capture everything that the users did during the usability testing including keystrokes, mouse clicks, and other interactions. Through the observation data, we can clarify that and analyse what users do and how long they spend on completing different tasks. It also provides insights into users’ affective reactions that related to the users’ experiences as satisfaction and frustration. Moreover, the user satisfaction questionnaire is used to clarify and deepen understanding of the users’ experiences.

We designed a specific questionnaire to evaluate users’ satisfaction with some specific features of the Hi-Fi prototype. As we mentioned before, the questionnaire that we are created in different levels: visceral, behaviour and affective (Norman 2002). And then, we compare outcomes of users testing performance on manipulating two digital drawing systems and asking the users about their opinions based on their interaction experience through a user satisfaction questionnaire. The first drawing system is a well-known drawing system-Photoshop CS5; the second one is the Hi-Fi prototype of drawing system. The produces qualitative data demonstrates how the different participants make sense of the interactive artefact.

Aims

• This project addresses the user’s individual problem of interacting with a particular software.

• Provide a suitable interaction pattern to the user in a right at right place based on a continuing machines learning process to Integrate multi-channel interactive computer interaction like pen/touch type interaction, gesture interaction under a human natural way.

Research outcomes: The expected outcomes include the following:

• Producing an interactive drawing system human computer interaction. This composes of two aspects: one is to build a personalized user interface which can continuously adapt to user’s behaviour, create individual user experience. The other one is construct natural interaction that be able to suggest uncommon, but needed function to users through multi-model interaction and automatically customize interfaces through multi-modal integration of functions.

• Providing theoretical and technical preparation and application to explore in-depth study of natural human-computer interaction techniques.

• Research demonstrations will be showcased at international national interactive arts exhibitions.

Part of research outcomes can be used into improving existing commercial system like Photoshop. That means we can cooperate with industry partners to explore variety of more suitable application systems with personalized interaction in many different areas.