Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences

Users of today are well surrounded with many connected devices providing easy things and more than 5 billion people will be connected by 2020 [9]. Internet of Things is reshaping the connected experiences of user’s lives. With the evolution of natural mechanisms of interaction in future, users are expected to switch to these interaction methods to interact with the connected environments and devices. User’s choice of interaction is likely to depend upon the kind of environment, the task situation, the devices and the available interaction modalities and relationships among these factors. Therefore, in order to design solutions for such connected experiences of future, one has to consider the user’s preferences and interrelationship among these three components named as M3(Multi-Environment, Multi-Device and Multimodal).

The key challenge is to take holistic approach towards understanding interdependencies among these components and designing solutions for such complex scenarios. There have been lot of research in the past focused on multimodal interaction for particular task context and evaluation of certain modalities [2, 4]. Moreover, considering that these scenarios being futuristic, it is also difficult to design for such scenarios by adopting conventional user centred design approach. In [6], authors proposed a methodology to ideate for IoT services by placing probes in daily environments or bodystorming and Nieminen [7] described user-centered concept development process for emerging technologies in four phases while giving emphasis on technology findings.

Other than these methodologies, researchers have proposed frameworks based on meta-design approach [3] and analysis of relationship among services, spaces, and users [10] for smart home like scenarios. We applied activity based user-centred design methodology through use of visual stimuli to understand the relationships between M3 factors and task types using modality and device switching patterns of the participants. Also this helped us identify fundamental themes responsible for these relationships and dependencies. Through systematic analysis of qualitative data, we evolved the M3 design framework for Connected Car context. Our choice to explore Connected Car as a sample domain for our exploration is motivated out of its great potential. A forecast predicts more than 92 million vehicles with Internet connectivity by 2016 [1]. This framework can help designers contemplate holistically and come up with solutions without losing user’s concerns.

We conducted interactive user sessions for two contexts: Home and Car. For these activity based sessions, we involved a total of 21 participants; 11 participants (6M, 5F) were involved for Smart TV viewing (Home) context and 10 (7M, 3F) connected car context. All participants were in the age group of 21–35 yrs (Mean = 27, SD = 2.36) and they were screened on the basis of their prior experience of the contexts.

Contextual scenarios were created for both Smart TV viewing and connected car situations. We gave the following scenario of Smart TV viewing in home context to participants: “You are watching a movie on TV and along with that you are following a Cricket match. During commercial, you switch to Sports channel. You find that one of the players is leading the team to win the match. You post about this on Facebook to show your happiness.” The scenario involved four unit tasks: T1- Changing the channel, T2- Specify action of posting on Facebook, T3- Add text to the post and T4- Confirmation of posting. Similarly, in car context, we presented the following scenario to the participants: “While driving the car to your Home, you receive the message a friend that your common friend is in the town. You reply back her saying that you will reach there in 15 min. Then, you change the navigator destination to your friend’s place.” Here again, the scenario consisted of four unit tasks: T1- Giving command to open/read the message, T2- Replying to the message, T3- Opening navigator and T4- Changing route in the Car navigator.

In order to maintain consistency between both contexts and ensure the exhaustive variability in the nature of the tasks, all the four tasks in both contexts were categorized under three categories:

1. (i)

   T_a: Tasks that involved changing the value through multiple similar interactions- T1 in home context and T4 in car context

2. (ii)

   T_b: one touch/command tasks- T2, T4 in home context and T1, T3 in car context

3. (iii)

   T_c: composing task- T3 in home context and T2, T4 in car context.

Participants were familiarized with the scenarios using Wizard of Oz with the help of interactive prototype. During this familiarization phase, they were asked to complete the four tasks of the context. During this process, participants were encouraged to discuss their expected usage style and response from the systems and also how according to them, these expectations depend on various factors.

Later on, respondents were inquired about their order of priority of modalities and devices for each of the tasks by means of cards. Each card graphically represented a choice of combination of a device and a modality. The choices given in both contexts and the devices made available to the users are shown in Table 1. Participants were asked to arrange the cards in the order of priority for each of the four tasks one by one. In order to observe the effect of change in the devices on modality preference, each user session was conducted in two parts where additional device(s) were given to the participant in the second part. Participants were asked to think aloud during the activity and the entire user session was audio-recorded. Post each of the tasks we involved them in a qualitative interaction to understand the reasons for their choices.

Preference score for a particular device-modality combination was generated by summing reverse of the ranks given by user. Task wise preference scores for all device-modality pairs (only in part 2) have been shown in Fig. 1. From the analysis of these scores, following patterns were observed:

1. 1.

   In home context, unlike other tasks, smartwatch gesture was preferred most for T1 task (changing channels) because of the task features: high frequency and quickness. As changing channel is relatively frequent while watching TV (other 3 tasks were related to posting on Facebook), participants were fine with defining gestures for it.

2. 2.

   In T4 task in Home (confirm posting on Facebook), touch was preferred relatively more than the other tasks because participants reported as it is the last step, they wanted to be sure of the action and unintentional mistake would bring back to the initial step or something wrong would be posted.

3. 3.

   In car context, touch was preferred relatively more in the T4 task (change destination in the navigator) when compared to other tasks because there are lots of possibilities in terms of outcomes of a task which may lead to mismatch between result and intention of the users. Therefore, participants wanted to look at the screen and use touch (zoom and pan) for verification and preparation before driving the car.

4. 4.

   Smartwatch and Car-Dashboard were most preferred devices in Home and Car respectively. This shows that users are more likely to choose devices most accessible to them in order to minimize the distraction from primary task (even if other devices have advantages like larger screen size and more familiarity).

Task wise preference scores for all device-modality pairs in home and car environments

Full size image

We converted participants’ individual device and modality preferences into switching patterns and created Fig. 2. These switching patterns led us to our initial explorations in establishing the relationships among M3 aspects and task which is summarized pictorially in Fig. 3. Following paragraphs explain how these relationships were evolved:

Modality and device switching patterns

Full size image

Inter-relationship among M3 and task

Full size image

Environment:

It was observed that changing environment from Home to Car resulted in device preference change (watch to car dashboard) for all three types of tasks. At the same time, environment change resulted in change in modality preference for only one type of task (one step task). This suggests that environment is more crucial for device selection than modality selection (a > c in Fig. 3). It is also clear from Fig. 2 that participants showed more modality and device switching in home context compared to car.

Modality:

It is evident from Fig. 2 that modality-switching occurred lot more because of task change than device change i.e. for modality selection, task is more important factor than device (d > e). In [5] also, authors showed the dependence of modality on task type. As mentioned earlier, change in environment resulted in change in modality preference for one of the three types of tasks whereas modality was switched only 4 times out of total 84 (44 + 40) cases for device addition. It means environment is more important than device for modality preference (c > e).

Device and Task Type:

Moreover, device switching was observed for all 3 types of tasks with the change of environment. On the contrary, out of total 63 (33 + 30) cases of task change, only seven resulted in device-switching. This indicates environment as more influential factor for device preference than task (a > b). Lastly, it can be seen in Fig. 2 that in case of task change, frequency of modality switching is higher than device switching for both contexts i.e. task type affects modality preference more than the device preference (d > b).

Further qualitative analysis was done to identify intermediary factors behind the relationships described in previous section. This analysis resulted into M3 design framework.

5.1 Evolution of Framework

The audio recorded user statements were transcribed and affinity analysis was used to get meaningful categories out of these statements. First, each user-statement was coded by its inference or intention behind it. Then, these codes were generalized and grouped into themes and sub-themes. Through this exercise, we evolved 14 main themes and more than 60 sub themes. The user statements involving two or more of these themes were interpreted as instances of interdependencies among those themes and were inferred on the basis of cause-effect relationship. For instance, one of the users said, “if I use touch then I have to give my attention to both important as well as unimportant” This user statement was found to involve two themes of ‘visual distraction’ and ‘information required by the user’ and latter is causing the former in direct proportion. Through such patterns the interdependencies among all the themes were established systematically and a detailed flow of interrelationships was evolved. These themes and relationships were generalized and further clustered to evolve the M3 Design Framework.

5.2 Themes and Relationships

We here describe the M3 Design Framework through its 14 main themes. Characteristics of a task as a theme include the importance, urgency and frequency of a certain task. Continuity is about continuing a certain modality or device throughout an activity or while task-switching and how it had an effect on the remaining factors or themes. Distraction has different kinds of visual, audio or task switching disturbances or case of user’s attention requirement. Ease of Use refers to the convenience of a choice, number of steps and total time involved in performing a task with certain choices of devices and interaction modalities. Effort refers to any kind of pain or strain the user goes through in terms of physical, cognitive load or due to too many actions expected from the user in a short time. Familiarity refers to the user’s prior experience in using various available options like devices, interaction modalities in various tasks involved in different environments. The kind and amount of feedback from the device in terms of information and modality is considered in the theme of Feedback. Flexibility is about having a choice of device and modality for a certain task and of doing the same task in many ways. Flow of information among various themes included in the framework is split into information required by the user and those required by the system separately. In [8], researchers highlighted the hierarchy input and output modality and information in order to minimize distraction while driving a car. Further, Intuitiveness indicates how natural and similar, the modalities and the expected user actions are in affordance. Learnability is the extent to which certain modality or device is remembered by the users to perform a task in a particular environment. Device screen size, proximity of device from the user, position of device for certain user input, environmental factors and user patterns are included under Physical features. User Control includes user’s consent and intention in tasks/actions that involves user commands or decision. Additionally, for the connected car context Safety was also included as one of the key themes based on the input from the users.

All the themes in the framework were categorized as Cause, Effect and Mediators or Influencers as shown in Figs. 4 and 5. There exist direct and inverse proportionality relationships between Causes-Effects pairs. Direct proportionality refers to relationship where Effects show similar changes to the Cause while inverse proportionality refers to the Effect showing opposite changes to the Cause. This relationship was used to understand the effect from the initial Causes through the Mediators. This kind of proportionality was not applicable for few non quantifiable themes such as type of environment, type of feedback, etc.

M3 design framework-home

Full size image

M3 design framework-car and legend for frameworks

Full size image

6.1 Design Ideation

In this approach, framework is used to come up with multiple ideas for a pre-decided domain. Out of the 14 themes, the themes which would play role in forming the ideas particular to that domain are discussed and identified. Then, each of these themes is separately branched out to come up with their detailed aspects through mind-mapping. Different aspects are connected across and within themes in such a way that a series of links result in one category of ideas and through multiple such links, many categories of ideas can be generated.

For example, when goal is to come up with multiple ideas on ‘proactivity in car’ as a domain, appropriate themes are ‘information required/received by system and user’, ‘user control’, ‘feedback’, ‘task characteristics’ and ‘environmental factors’. All six themes are branched out to details through mind mapping (see Fig. 6(a)). Time or place for providing a particular type of info required by a user can be connected to a particular task characteristic. One example of such information can be petrol level which should be proactively informed to driver when its low (urgency as a task characteristic) and a petrol pump are nearby.

Flow in frameworks of the Applications: (a) design ideation, (b) comparing design alternatives, (c) solution detailing [C- Cause, M- Mediators, and E- Effect]

Full size image

6.2 Comparing Design Alternatives

This approach is used when designers have to choose which idea works better keeping user’s considerations in mind. In this case, flow goes from the cause side of the framework towards the effect side. First, the causes which are relevant for a particular idea are discussed and identified by the designers and each one is marked with one of the four values: High (H)/Medium (M)/Low (L)/User-dependent. Cause- effect and proportionality relationships are used to carry forward these values to mediators and effects (from mediators). If a mediator theme is affected by multiple causal themes, average of the values received from the causal themes is passed to the effect theme. This procedure is followed for all the ideas to be compared. Finally, different values received by an effect is compared across ideas to know which idea is better for that effect because these effects are the goals to be achieved through the idea.

For example, let’s take an example of comparison between movie ticket booking and e-booking in home context. The activity includes the primary task as watching television and doing the above two tasks when a movie trailer or product advertisement comes on TV. Take Fig. 4, for movie ticket booking, physical features and task characteristics act as user dependent factors, information required(H/M), frequency of interactions(M), number of steps(H), continuity of device(M) (these values are given on comparing both the examples) act as inputs or initial causes. These values have an effect on ‘physical/cognitive effort’ which has values of H/M, M and M, so the average value is taken and the final effect of ease of use is given a value M. Similarly, when it was applied for E-Booking, the final effect ‘ease of use’ had a value L and distraction had a value of H. So comparing these two shows that achieving Movie ticket booking is better and solutions should be detailed for E-Booking under ideation application (which is explained in the next section).

6.3 Solution Detailing

This happens when you have certain goal to achieve. This includes flowing from the effect to the cause side in the framework, marking the important and controllable mediators and causes according to how they should be tweaked for the necessary effects or goals. The designer jots down few questions first according to how the mediators should be and ideates at every theme solving these questions.

For example, let’s take an example of detailed ideation on travel blogging in car context while the user is driving. Take Fig. 5, our main goal is to increase safety and ease of use while blogging in a drive. Few important and controllable mediators/causes can be information and flexibility etc. among others. Questions like “how to lower the information required by the system/user?”, “How to increase the flexibility in the way of using a modality?” etc. have to be jotted down. The ideas in terms of modalities, devices and information should be provided to answer these questions e.g. giving predefined or auto-detected hashtags for a photo though voice to reduce the information input by the user. Similarly, ideas can be jotted at every theme for the required effect.

6.4 M3 Design Framework Used in a Design Iteration

We have applied this framework, in case of ‘information on the go’ for context of ‘Watching TV’. After generating a number of ideas on this domain, all of them were compared (shown in Sect. 6.2) and ‘movie ticket booking while watching TV’ was chosen for idea detailing part. Ideas like ‘Introducing contextual icon in the mobile for booing app w.r.t the movie trailer on the TV’, ‘Using voice input or biometric inputs for entering card details to reduce the number of steps’, ‘auto selection and introducing one step confirmation steps for the user to reduce distraction’ and related things were finalized while detailing for prototyping the movie ticket booking idea.

The M3 Design Framework presented in this paper can be used as a reference tool to design user’s connected experience encounters for futuristic domains like connected cars, smart homes, IoT. In this paper, we presented an activity based qualitative research methodology which we used in two contexts of ‘Smart TV viewing’ and ‘Connected Car’. As the activity involves a structure flow and visual stimuli, it provides the participants with ground to divulge their choices in such scenarios and more importantly, reasons behind the choices. Through device and modality switching patterns, we established relationships among M3 aspects and task type. We have also explored relative dependencies of one factor on another compared to third factor. In order to inform design decisions, it was essential to dive deeper and to identify intermediary factors behind the relationships. Affinity analysis of the qualitative data helped us to obtain 14 such intermediary themes and evolve M3 design Framework by making cause-effect relationships among these themes.

The M3 design Framework can be used as a reference (1) to aid brainstorming to generate multiple ideas, (2) to comparatively evaluate various design alternatives on the basis of user’s preferences, and (3) for solution detailing till the level of designing interactions and features. Additional application of the M3 design Framework can be formulation of questionnaires for usability testing of a prototype in which relevant effects and mediators can be considered as factors to be evaluated. The methodology can be used to generate frameworks for other contexts like office, public spaces etc. by providing context relevant tasks and visual stimuli during user-sessions. Therefore, along with M3 design framework for ‘Connected Car’ to produce design solutions for it, we propose the method to create similar frameworks for other contexts.

In future, we plan to extend the framework to include multi-user as fourth parameter. We will continue to generate these frameworks for smart home and other IoT contexts by incorporating quantitative approach as well.