Research on Aesthetics of Interaction of Mobile Context-Aware Services——A Case Study of Notification System

In the era of pervasive computing, the improvement of computing of mobile devices, the built-in of sensors, the development of wireless communication technologies and the advancement of algorithms have facilitated the convergence of computing, communication and sensing technologies. This enables mobile devices to acquire increasing contextual awareness power of data acquisition, analysis, predicting human behaviors, and performing actions [1]. In 1994, Bill Schilit et al. first proposed the word “Context-aware” [2], which defined context as “the objects and their change of the position, the person and the surrounding”. Context awareness emphasizes the device’s perception of contextual information and the feedback of system. Context awareness can be understood as the acquisition of contextual information about the environment, devices and the social state in which users are located through human-computer interaction or sensor technology, allowing computing devices to perform real-time analysis and reasoning, adaptively changing their behavior to provide appropriate service.

The mobility, portability, and instantly reachability of mobile devices allow them to be around users, and become aware of the changes of user context proactively, then provide explicit and implicit services based on user’s individual needs, allowing contextual information to be fully integrated with user tasks, therefore the natural interaction could be achieved. Context-aware services based on mobile terminals have become one of the most widely used intelligent systems [3].

The mobile context-aware service reduces the load of the user’s perception and analysis of information [4], while also improves the efficiency of operations. The shopping software being able to shorten the whole process of searching by recommending products of good quality to the users according to the records of clients’ searching and preferences could be a great example. On the other hand, actively obtaining user context information, notifying some certain information after analysis, or performing operations through implicit interaction may bring negative impacts, such as privacy violation, information overload, interruption, loss of control, and psychological stress to users [5].

The efficiency improvement, experience difference and negative influence of context-aware system and traditional human-computer interaction system are mainly caused by the change of human-computer roles and the change of interaction modes. The negative impact is primarily related to the part of the user experience that removes efficiency and usability, that is, the part related to “aesthetics of interaction.” This paper takes the notification system, one of the mobile context-aware systems, as an example, to analyze its interaction modes from the perspectives of active and passive, explicit and implicit, central and peripheral, and analyze the aesthetics of these interaction modes and the impact on user experience. Finally, implications on the interaction design of the mobile context-aware system were put forward, and the future researches of the mobile context-aware interaction were discussed.

The interaction mode of the mobile context-aware system can be decomposed and analyzed from the perspectives of explicit and implicit, active and passive, central and peripheral, which are intertwined and synergistically interact with the user and the system. Therefore, it is possible for the mobile context-aware system to implicitly and actively assume more tasks in human-machine collaboration, thereby reducing the input requirements to users and free users from some attention, so that the users could perform multiple tasks efficiently at the same time. At the same time, the distribution of explicit and implicit, active and passive, central and peripheral interactions should be paid attention to, to ensure that users are less likely for users to experience negative emotions such as interruption, anxiety, worry about privacy and loss of control.

2.1 Explicit and Implicit Interaction

Most traditional interaction process is imperative, in which the device user gives clear instructions through keyboard, mouse, touch, gesture and voice commands, then the smart device explicitly outputs the sensory calculation result to the user through visual, tactile, and auditory modes, so that the human-computer interaction is achieved, this method is called explicit interaction. When performing explicit interaction, users are able to clearly and unambiguously control the input, understand the output information and the stage of interaction, so that they have sufficient sense of control, and the phenomenon of loss of control and invasion of privacy can be avoided. At the same time, there are only a few processes that require the equipment to perform sensing and calculation, so the recommendation errors and interruptions caused by the system rarely occur.

In contrast, Schmidt defined implicit interaction as an invisible interaction. Users no longer care about the interaction process itself, nor do they think much about how to use the device or system, but rather the device or system actively and implicitly identify and understand user behaviors, and the interpreted information is applied to the human-computer interaction process [6]. Compared to explicit interaction, implicit interaction can bring solution to the situation where the users cannot effectively input nor deal with the output from devices due to limited processing resources or physiological constraints when people are using the mobile context-aware system.

The users would not have to divide their attention or simply deal with their secondary tasks with only peripheral attention, so that they could highly focus on the main tasks without interruption, cognitive burden, or information overload.

2.2 Active and Passive Interaction

In the traditional human-computer interaction system, basing on the self-context perception and analysis, the user actively initiates interaction with the smart device according to the operation target and the contextual condition. Although this kind of interaction has a high psychological load on the user and the efficiency of the operation is limited, however, the user has a real sense of control over the interaction process, and has a clear understanding of the information required by the system and its corresponding reasons, thus they are less concerned about the information security.

In the mobile context-aware system, the smart device may actively perform an operation or issue a suggestion or request for performing an operation after analyzing and processing data such as user preferences, attributes, likes and dislikes, and acquired user context information. On the one hand, active interaction can reduce the consumption of user attention by interactive behavior, and make it possible for users to use limited attention to simultaneously handle multiple tasks. On the other hand, the inopportune active reminders and active executions can cause disturbance and even make the user feel uncomfortable and panicked about the contextual information (involving privacy) acquired by the computing device.

2.3 Central and Peripheral Interaction

Transition from the use of central attention to interact, to the use of peripheral attention for interaction, is a chain reaction after transition from explicit interaction to implicit interaction, passive interaction of equipment to active interaction. In the traditional human-computer interaction mode, the presentation of information and the response of the user require the user to concentrate on. When a user is in a task that consumes a large amount of psychological resources, it is difficult to perform multitasking both in terms of ability and emotion.

In the process of interaction with the mobile context-aware service, the smart phone has the ability to actively and implicitly recommend or perform operations according to the context and user preferences (such as automatically adjusting the screen brightness, automatically modifying the time zone, and recommend songs for users depending on the song history and preferences. Peripheral attention or subconscious is enough for users to complete tasks. In some situations, even without the user’s attention, the computing device can fully complete the task. In this way, the consumption of the user’s psychological resources is reduced, and the unutilized psychological resources are likely to be assigned to other tasks, which means that the way of multi-tasking is likely to be favored by more users.

As a philosophical term, aesthetics is used to define beauty and is understood to judge whether it meets the feeling of pleasure by sensory knowledge. The aesthetics of interaction research on the aesthetic issues between human, product and material culture, which is centered on aesthetic experience, but is different from philosophy, aesthetics or industrial design theory. It is not confined to abstract philosophical concepts, or apparent and functional descriptions of product, but reveals the beauty and aesthetic laws in daily life from different levels, such as from abstraction to figuration, thus promoting the renewal and development of design concepts. Aesthetics of interaction is not just the beauty on the interface, but a psychological experience when users communicating with products. Designers transform the user’s experience of interacting with the product and the environment into corresponding physical and ideological pleasure, then reflect it on the design. When the design of the product satisfies this “beauty” requirement, the “beauty” product is naturally born.

In the field of human-computer interaction and interaction design, aesthetics became a research topic in the late 1990s. At that time Noam Tractinsky (1997) reproduces Kurosu and Kashimura’s study (1995) of the relationship between “objective” usability and subjective evaluation of usability and aesthetics [8, 9], after which he believed that the usability, as one of the interactive attributes, was related to the aesthetics which is one of the attributes of the graphical user interface. Since then, researchers have begun to explore how the visual aesthetics of the interface affects the user’s perception of the product and the interaction with the product ultimately [10]. Although this study introduced “beauty” into the field of human-computer interaction, it still saw aesthetics as the attributes of the interface (appearance) and interprets usability as the attributes of interaction (feelings). From the perspective of interaction design, this understanding seems to be narrow. In 2000, Djajadiningrat and his colleagues challenged this, and they believed that the field should shift its focus from “beautiful appearance” to “beautiful interaction (the appearance is part of it)”. “Aesthetics of use” should explore a “finer, closer interaction with the object to enhance social connections or everyday experiences.” [11].

Measurements of interaction design can be very limited if they only stay at the usability level that emphasizes efficiency. Nowadays, making a call can be done by pressing a button, a touch gesture, or even a voice input. The usability of these interactions may be different, but they are very different in many other ways. For example, judging whether a swipe gesture is appropriate or not, is not just an efficiency issue. Some new interactive technologies, such as touch, can provide a good or even excellent feeling and experience despite the lack of precision in the interaction. Although the interaction mode of shaking the mobile phone consumes more energy, it can make the user feel interesting and novel. As Djajadiningrat and others believe: sometimes the product is difficult to use, but the user still chooses to use it, probably because it is challenging, seductive, fun, surprise, and special, so that users have the experience of enjoyment [11]. For interaction design, in addition to efficiency and ease of use, there are many metric properties, which requires researchers to better understand the aesthetics of interaction in interaction design.

By reviewing and summarizing the relevant research literatures on aesthetics of interaction, and based on Hassenzahl’s combing of interactive aesthetic elements, including time, space, action-response interaction, autonomy, popularity, security, meaning, etc. [12], we have sorted out the framework of aesthetic elements suitable for analyzing the interaction mode of context-aware system (see in Table 1).

The aesthetics of the interactions in the context-aware system will be analyzed based on this aesthetic element framework. It can help designers analyze soft indicators of interaction design, not just hard indicators such as efficiency. Furthermore, the impact of the non-efficiency level generated by the current context-aware system can be solved, and suitable aesthetic and emotional entry points for the popularization of context-aware technology can be discovered.

In the mobile context-aware interaction design, on the basis of defining different scenarios, when performing various physical interaction behaviors in different scenarios, it should be guaranteed to analyze the user’s expected sensations and meanings, and then to retrospect the available interaction modes, interaction elements and so on. In addition, due to the dynamic and relevant features of the mobile context-aware system, such as active and implicit, it may have a greater impact on the related features of aesthetics of interaction such as user control, privacy protection, which should be paid special attention. Taking the notification system, one of the mobile context awareness systems as an example, the performance of the interaction mode in the aesthetics of interaction and its impact on the user experience are analyzed.

4.1 Notification System

In the era of massive information, users receive a lot of notifications on mobile devices every day. The interaction of the user with the notification system includes the management of the notification and the interaction with the pushed notification, wherein the notification opportunity, the notification push form, the notification content, the user’s operation of the notification, the personality characteristics of users are several key concerns.

When the user starts using the new phone (before starting to deal with the new notification system), or after maintaining a certain setting mode for a period of time, the settings of the notification system will be managed and adjusted. The user’s management of the notification includes management of timing, whether to open the notification, and the selection of notification push form. Usually, when the application is installed and opened for the first time, when an application notification is received and viewed, the notification of applications is managed. When the application is first opened is the management opportunity set by the application designer and developer. At this time, the user just uses the application for the first time, and may not have a certain understanding of the functions of the application and the content notified by the application, plus the flooding notifications nowadays have already caused users to be bothered. In addition to individual high-frequency applications and content-critical applications, most users will choose to turn off the notification of this new application at this time. When the user has used the application for a period of time, has a certain understanding of the application content, preferably has a certain usage viscosity for the application, or does some core operations (or related to certain notification content), the opening rate of notifications may be high if users are reminded to open it. If an application sends notifications to the user frequently, or the user notices that the notifications of an application are hardly noticed when viewing the notification bar, the user is most likely to choose to close the notification of the application or adjust the notification push modes of the application. iOS 12 has now added an entry to quickly manage the notification switch and adjust the push form in the notification center.

The timing of dispatching notifications by the traditional notification system mainly includes real-time notification according to trigger time of messages/calls, time according to user setting (alarm clock, to-do list, etc.), time according to application development operator. Different notification timings are related to the active-passive interaction mode between the user and the application, and also related to the content type of the notification, and may also connected with the personality characteristics of the user. The real-time notification according to the real-time message triggering and the notification set by the application development operator, do not take the user’s current situation into account. The rushed and even a lot of interruptions are very likely to disturb the user, cause the user’s original task to be interrupted and make users feel anxious or stressful. In this regard, the context-aware notification system considers the context information of users and the importance of the notification content, selects the corresponding notification opportunity (real time, delay, unified to appropriate time) and notification form.

The mainly types of notifications are shown as follows, including explicit and implicit, central and peripheral forms:

• Sound

• Vibration

• Indicator

• Screen pop-up

• Banner display

  • Temporary: the banner will appear at the top and disappear automatically

  • Ongoing: the banner will appear at the top and disappear only if users respond to it

• Display in the status bar

• Display in the notification bar

• App icon tag

Due to the influence of timing, content and method of notification, after the notification is pushed to the user’s mobile phone, the user will interact with it in the following forms [5]:

• Seen: Only see and no other operations. View in the notification bar along with other notifications, view on the lock screen.

• Dismissed: Carry on Dismiss Event or Notification Clear event When the notification bar is open for more than 5 s.

• Actions: The user performs any action when the notification bar is turned on.

• App Launched: User clicks on notifications within 2 s. The user opens the app which is associated with the clicked notification.

• Ignored: None of the above behaviors have occurred. Apply a notification cleanup after the time is over, or update the notification content.

The interaction between the user and the notification is related to whether the user or the application actively initiates the interaction, the time mode in which the user uses the application, the time period in which the user wishes to receive the notification, the importance of the notification content to the user, the personal characteristics of the user (such as concentration of attention, whether it is easy to be disturbed), and contextual factors. The user’s interaction with the application can be divided into the following categories according to different initiators: the user initiates interactions, such as taking a taxi, searching, sending messages, etc.; the user actively sets, then the device initiates actively, such as using an alarm clock, a stopwatch; the user actively sets, and the device reminds, then the user decides whether to initiate, such as snapping, fitness, learning, habit formation, etc.; the user does not set, but the device reminds the interactive content, then the user decides whether to initiate, such as an incoming call, a message and real-time news information and other content push; users do not actively set, devices push operational content to attract users, users decide whether to initiate, such as games, videos and other entertainment activities and shopping activities. In addition, the degree of participation of users in different application notifications at different time periods will vary greatly. First of all, for each application, the user will have a relatively regular usage time pattern, and if the user is reminded to use the application at the time he/she is accustomed to logging in to it, there may be a higher response rate to the notification. Secondly, time of users in every day has a division of work, entertainment, rest, etc. Users will be in different tasks or activities in different time periods, and they will have different attention performance and sensitivity to time, which affects the interaction between users and notifications. Among them, the attention situation, the sensitivity to time, the attitude to control, whether it is good at multitasking, sensitivity to privacy etc., are related to personal characteristics, which vary from person to person. At present, when application developers push notifications, there is no clear difference in importance of notifications. But in fact, the importance of different applications and different types of push are significantly different. As a general matter, the message of the task arrangement sent by the tutor is obviously much more important than the gossip news of a certain star that the information application pushes. Last but not the least, the context factor is the most important factor to be considered by the context-aware notification system, including user context, computing context, physical context, social context, etc. User equipment situation, sensor data, notification features, time features, and location features could be acquired and utilized in real application.

4.2 Analyze Aesthetic of Interaction in the Notification System

At present, the context-aware notification system combines explicit and implicit interactions, active and passive interactions, and central and peripheral interactions to provide users with an excellent interactive experience. This has improved operational efficiency and user satisfaction to a certain extent, but at the same time there are still some problems of limited cognitive resources, information overload, interruption, stress, loss of control, privacy and so on.

The interaction modes such as sound, vibration, indicator light and screen pop-up in the notification system are all explicit interactions that the device actively pushes or initiates the requests. From aesthetics of interaction aspect, they not only affect the user’s current task, but also may affect the user’s cognitive resource allocation, reduce task efficiency, cause the fragmentation of time, and even cause irritability. Interactions initiated by devices such as task reminders, requests, etc., are based on user behavior and context information. However, under the condition of limited cognitive resources, excessive notification messages may result in insufficient cognitive resources, information overload and psychological stress. The modes of displaying icons in the status bar, displaying icons in the history records and other peripheral interactions are used to notify non-important and non-urgent secondary tasks, which are comfortable and effective that can minimize the disruption and consumption of cognitive resources to users. Researchers have found that the distribution of psychological resources is different when users are in different processes of different tasks and interactions. When users engage in complex and difficult task operations, they need to invest more cognitive resources. At this time, the disturbance will cause competition for cognitive resources, which may lead the original task to be interrupted. Task recovery after this kind of interruption may also be more difficult. When engaging in a less difficult task or just completing a task and not starting the next task, the cognitive resources are occupied less, it is possible to process the new task while maintaining the original task state, or to process two items at the same time. In this way, interruptions can be avoided. The context-aware notification system can obtain the context information, analyze the situation, and select the push mode and push timing according to the content importance and context without much cognitive resource consumption and interruptions.

Another study found that mobile phone users receive an average of at least 60 notifications per day, and some even receive 600. If not managed properly, these notifications may disturb users at least 4 times per hour during working hours [5]. Information overload makes users to be immersed in massive amounts of data, which can cause problems such as cognitive overload, excessive psychological stress, and anxiety.

If the context-aware notification system and related middleware obtain context information without explicitly statement about the purpose of obtaining the context information, or the result of the analysis reveals the user’s privacy too much, the user may be given a feeling of privacy violation. If you help the user to make judgments and perform tasks too much, it will also make users feel out of control in addition to feeling that privacy is violated. Therefore, clear permission, timely privacy-related feedback, and proper task execution are required to gain users’ trust and reduce negative user experience.

There are many ways to enhance the interactive aesthetic performance of notification system from the perspective of user-centered and user experience enhancement. For example, the context-aware notification system can perceive the consumption of cognitive resources of users in different contexts, choose the appropriate way (ringing, vibration, visual reminders, etc.) in the appropriate time (non-meeting, leisure, etc.) to attract the attention of users and enhance user perception. Which means, notifying users at the optimal time to obtain the maximum user response rate with minimal interruption and cognitive resource consumption. Urgent and important events can be reminded by the device in an active and explicit manner. Unimportant, non-emergency (contents that are casual and entertaining) can be pushed during the user’s rest period or pushed in an implicit way.

The machine learning algorithm can be incorporated into the process of defining the importance of the content of the push content and the search for the appropriate push timing, thus the user’s usage patterns are defined through a large number of individual user data.

In the case of having to bother the user, it can be used as a smart assistant to remind and help the user to return to the original task. Such as suspend and maintain the state of the original task and remind user to recover by using timed reminder and forced recovery.

Before using the context data, it is necessary to clearly state the specific data obtained and its usage. After analyzing the context information, the appropriate feedback is also important to eliminate the user’s concerns about personal privacy and establish trust in the mobile context-aware system.

In the future, there are varies of topics can be explored, such as the human-computer collaboration model based on mobile context perception, the role orientation of the mobile context-aware system to the user, the relationship between the user’s temperament personality and the identity and participation of the context-aware system. As the ultimate goal, smarter and more natural services will be provided to users.