UX and Usability on Smart TV: A Case Study on a T-commerce Application

For over 50 years, the television has been one of the main tools for home entertainment and information [1] in most Western countries. As “entertainment furniture” it has greatly influenced the home environment, setting the type of interaction and relations among the room’s elements. The greater influence occurred in the arrangement of the living room, where the television found a place of honor.

However, over the years the television has significantly changed, changing as well the type of interaction between the users and the medium itself. New devices (e.g. cable or satellite receiver, videocassette or digital video recorder, DVD player, audio amplifier) and controller (e.g. different kinds of remote controls, gesture recognition systems) were attached and integrated by television, up to the so-called Smart TV. Moreover, the succession of some technological innovations gave the television new looks and new capabilities: full colors images, lager screen sizes, less case thickness, LAN or Wi-Fi broadband connections, media sharing, smartphone connectivity (second screen), etc. A new offer in contents corresponds to the enhancement of the device. So, the Smart TV gives access to app stores, games, videos playback, social networks, web browser, streaming movies. However, the TV apps available today are not so much different from smartphone apps, while they should take advantage of the fact that they are located on a TV and make use of TV specific content [2].

In details, the Smart TV is a television including components for network connectivity, which make it able to communicate and to synchronize with other devices connected into the home network. Beside to Ethernet cable and Wi-Fi network card, a Smart TV generally consists of some other elements that make it similar to a personal computer, such as Operating System and USB ports, through which connecting various devices (e.g. keyboards) and interfaces, which greatly facilitate navigation and writing on the Internet.

However, Smart TV models differ in interaction modes and personalization features. The latest models, for example, allow an interaction through gesture or voice recognition, while a third more typical way of interaction is through remote control. A separate analysis should be done on the evolution of the remote control devices, since the lack of common guidelines on the implementation of the interfaces of these remote controls created an extremely confusing and varied set of devices [3].

In this paper the authors focus on the interaction between user and Smart TV through remote control. In detail, they test with the users an e-commerce (or more specifically, t-commerce) application on Smart TV, since there is a lack in the related literature about usability of Smart TV apps, due to the yet moderate diffusion of Smart TV devices and the recent interest in the debate about interaction guidelines for Smart TV. Moreover, the test of an application for buying products and services through television is useful as such type of services, i.e. e-commerce services, has been the drive motor for the main spread of Internet and are expected to give costumers a more complete purchase experience through Smart TV. So, in this paper the authors intend to contribute to the research on usable applications for Smart TVs by deducing some considerations from the usability test on a Smart TV application for T-commerce.

Smart TV devices are a recent product on the mass market and they look promising in increasing much more in the next years [4]. Indeed, although they present well-known schemes of human-computer interaction, such as “app” (largely used on smartphone, tablet, etc.) and web sites, there are substantial changes in the user interface (hereafter, UI) and in the user experience (hereafter, UX), due to the peculiarity of the Smart TV device. So, usability studies on Smart TV apps are needed in order to improve the quality of the UIs.

However, Smart TV is the last form of a series of similar technologies, often considered as the basis for its evolution, such as Digital TV, Connected TV, Interactive TV (iTV), Internet TV, Internet Protocol TV (IPTV), etc., and discussed in usability and UX related literature. First of all, Chorianopoulos [5] drew attention to the lack of specific UI principles for the unique characteristics of iTV applications. He sustained that the usability mentality of efficiency and task completion (effectiveness) may not be suitable for design and expert evaluation of iTV applications, since the latter gratifies entertainment needs and leisure activities in a relaxed domestic context, rather than productivity. So, he recommended two categories of design principles: the first refers to which are the most suitable features for interactive TV applications, the second to how to design UX that supports the novel features. However, applications that involves more information processing than enjoyment could be modeled after a list of high-level principles and generic design factors. However, Chorianopoulos mainly refers to a class of TV services, such as Electronic Program Guide (EPG), Digital Video Recorder, and applications related to the broadcast TV program. Miesler et al. [6] observed that, because of the rapid growth of Smart TVs, it has been hard to establish any standards concerning UI and interaction modes. However, the identification of user-centered design, as well as the consideration of UX, is an important precondition for service adoption. According to Shin et al. [4] Smart TV is considered a very promising connecting device among different home systems (computers, telephones, electricity, security, entertainment, etc.), but its contents and services are still limited and there are many usability problems to overcome, too. So, they examined consumer’s perceptions of Smart TVs and provided practical insights into developing a user-centered Smart TV interface in order to develop effective Smart TV services. In detail, designers should consider that although people turn increasingly to Smart TV for services they formerly got from other sources, however the expectations for those services change. So, the system characteristics impact on perception of enjoyment and usefulness. Moreover, perceived usability can influence behavioral intentions through attitude.

In general, many studies that concern the analysis of factors affecting consumers’ adoption of Smart TVs used the Information Technology Acceptance Model (TAM) or other derived models [4, 7, 8]. It emerged that a possible reason for low service usage of Smart TVs could be the insufficient service usability and UX, mostly due to the lack of dedicated contents, competing services, and high service quality. So, the development of specialized contents for Smart TVs is expected to strengthen the differentiation from other form of similar technologies and to provide stable service quality, promoting, as a consequence, the usage of Smart TV services [8]. Also well-designed functions are capital for the acceptance of the Smart TV by users. In detail, without a proper and convenient way to control Smart TV functions all other advanced features are useless [9].

With regard to this, in studying the usability of the Smart TV the presence of different interaction modes, mainly gesture recognition and remote control, which differently influenced the usability and UX of the Smart TVs services, has to be considered. The remote control is used to navigate between different types of content, more or less interactive, since Smart TV offers a greatest number of options to the user in respect of the traditional TV. Consequently, also the number of actions performed by remote control multiplies (in addition to the channel selection and the management of basic functions, there is the interaction with different apps). Nevertheless, in the past decade the user has been accustomed to this growing complexity, firstly because of the use of various remote controls to manage the home electronics connected to the TV (video recorder, home theater, etc.), and secondly because of the use of a universal remote control with an increasing number of buttons (in order to control all the features of the electronics mentioned above). In their study on usability and usage of iTV services, Bernhaupt et al. [10] revealed that users had difficulty in text-inputs with the remote control keypads and pushed arrow buttons more than the color buttons. Liang et al. [11] conducted a usability analysis of a Smart TV music service, but, identified some usability problems that can affect also other kind of services: system response lag, difficulty in using the remote control, possibility to exit the system by accident. In investigating the usability of remote controls in regards to the currently prevalent interfaces, Lim et al. [12] found that the general remote controller had a better performance in channel selection and simple navigation composed of low-depth tasks, while the motion remote controller facilitated the ubiquitous information communication between the TV and user (especially for more advanced functions such as web browsing and entertainment factors). Moreover, the motion remote controller reduced visual distraction and was relatively preferred in subjective ratings. Nilsson Helander [2] conducted an in-depth analysis about the different ways to interact with a Smart TV (i.e. remote control, keyboard, mouse, smartphone and tablet) during the performance of different activities (i.e. TV watching, texting, menu browsing, Internet browsing, video watching, gaming) by using Natural Goals, Operations, Methods and Selection Language Test (NGOMSL) usability model. Since the interaction through remote controller might turn to be clumsy, unnatural, and unsuitable to the requirements on functions and applications of Smart TV, Li et al. [13] proposed some design and usability parameters for a hand-wave Smart TV Control Mode through Kinect. In addition, Jeong et al. [14], considered as capital to minimize user efforts while interacting with multimedia contents. So they proposed a system able to improve the speed and accuracy of the contents search in Smart TV environment by inferring user’s search intent trough movement patterns.

The increasing market penetration of Smart TVs brings on these devices some services traditionally accessible through others, offering new possibilities to engage customers in a smarter way. In this sense, T-commerce (here considered as a new kind of market available for the user/customer through iTV or Smart TV applications) could create new business models [15]. According to McGuigan et al. [16], the television’s marketing capability can be optimally applied in T-commerce experience. Indeed, people can use the remote control to make purchases in a new kind of shop or directly interacting with advertisements [17]. According to Omar et al. [18], T-commerce is a tool in iTV advertising that purposely facilitates the purchase of services and goods at home using a remote control through TV, instead of a telephone, PC, or PDA. As a result, it will eliminate the constraints in terms of time and space always experienced by buyers in traditional commerce experience. Omar et al. state that since iTV, provides more vivid images, words, video, and audio, the TV audience is led to consider it more attractive than traditional TV or Internet.

In this section, the authors present the main findings from the usability study on the prototype of a Smart TV application (described in Sect. 3.1), focusing both on the description of the followed methodology (Sect. 3.2) and on the collected results (Sect. 3.3).

3.1 Description of the System

The prototype of the Smart TV application tested in the study described by the authors of this paper is part of a digital platform for electronic commerce designed for Samsung Smart TV and controlled through remote control. In detail, the platform includes a set of interactive channels for the individual merchants, who can customize contents (catalogues of products and services, company information, etc.), images, icons, menus, etc. of their own channel.

The platform provides to the final consumer a set of basic features to finalize the purchase: creating an account, adding and visualizing the products in the cart, paying the added products or services, adding one or more address, adding one or more payment card, checking the order history. So the user can surf among the different commercial channels adding the products he/she likes and then make a single purchase process.

3.2 Methodology of the Usability Test

The authors conducted a usability test on a Smart TV application for T-commerce in order to define and evaluate the usability problems of the app. Beside this, the test was also useful in order to study the general factors that may affect the interaction between user and Smart TV.

The performance of the application was evaluated against a set of criteria, defined according to the studies of Nielsen [19, 20], Norman [21], and Polillo [22]. In detail, the authors were mainly interested in evaluating: the suitability for the task (the dialogue supports the user in the effective and efficient completion of the task); the self-descriptiveness (each dialogue step is immediately comprehensible through feedback from the system or is explained to the user on request); the conformity with user expectations (the dialogue is consistent and corresponds to the user characteristics, such as task knowledge, education, experience, and to commonly accepted conventions); the error tolerance (despite evident errors in input, the intended result may be achieved with either no or minimal action by the user); the learnability (users can easily accomplish basic tasks the first time they use the system and the dialogue supports and guides the user in learning it); the subjective satisfaction (the system is pleasant to use). Generally, these criteria are applied to different kinds of devices, first of all, web sites and tablet or mobile app, while there is not a common specific methodology for evaluating the usability of T-commerce applications on Smart TV. However, since the system tested was an application as well and the mentioned criteria are general principles for Human-Computer Interaction, the authors considered them suitable for evaluating the usability of Smart TV apps, too.

The whole test was organized in 3 phases: filling the entry questionnaire, performing the required tasks, and filling the exit questionnaire.

So, first of all, the user completed an entry questionnaire consisting of a first part (identical for all) designed to know the participants habits in using digital devices (smartphone, PC, tablet, TV) and in purchasing products and services through them, and a second part that focuses on the use of Smart TV and on the purchasing of products and services through it. The authors prepared two version of the latter part, one for users that know what a Smart TV is (so they were asked to give the definition) and one for users that do not know what a Smart TV is (so they were given a definition).

Then, the authors conducted an in-person lab testing involving 10 users (5 male and 5 female) that, one by one, have to complete a purchasing process and some additional tasks by interacting with Smart TV by remote control. The group of participants was consistent with the age range of the main target audience of Smart TV (20-49 years old) [23]. Moreover, from the entry questionnaire it emerged that they also have a high educational level (they have at least bachelor’s degree). During each test session, the user was performing tasks and was thinking aloud to explain what he/she was doing, why and how he/she was feeling, as he/she did it (task walkthrough method). A scenario approach was adopted, instead of a list of specific tasks. Two test supervisors and observers introduced basic instructions of the usability test session and took notes regarding events, activities and thinking aloud that were occurring while the user was performing the test. In addition, a camera operator recorded the test session.

Lastly the participant completed an exit questionnaire designed to evaluate the usability of the application and the UX on the basis of the mentioned criteria. The questionnaire consisted of 4 5-point Likert scale questions, 2 open-ended questions, and 1 matrix question.

Since the task walkthrough usability inspection method was adopted, the authors did not measured metrics such as the task completion rate (effectiveness) and the task completion time (efficiency) in accomplishing the purchase through the Smart TV application. As a consequence, it was not possible to evaluate the usability according to the definition given by the ISO 9241-11 [24]. Instead, the authors evaluated how easy it is for new users to accomplish tasks with the system and noticed which usability problems emerged from each test session. Moreover, the whole interaction and UX with the system was studied by considering some general usability criteria and by observing the participant’s reactions (e.g. facial gestures, body language, tone of voice, verbatim comments, etc.) and expressed emotion while performing the required tasks.

3.3 Results of the Usability Test

In this paragraph, the authors present the main finding of the described usability study. In details: the first three sections show the results of the entry questionnaire and the following five sections present the results of the exit questionnaire.

Respondents’ Habits in Using Digital Devices. The 10 respondents used at least once the digital devices the authors investigated (smartphone, PC, tablet, TV). The most used device is the smartphone, whereas the less used one is the tablet. In detail, during a typical day, the use of smartphones and PCs decreases with the passing of hours, whereas the use of TVs and tablets increases.

Respondents also indicated the activities carried out more frequently through these devices. In details: smartphones are mainly employed to call, to send/receive SMS, and to listen music; PCs to access Internet for various activities; tablets to play; TVs to watch films/videos.

Respondents’ Habits in Purchasing Products and Services through Digital Devices. The PC is the most used device during almost all the different stages of a purchasing process, from the “search for information on products/services of interest” to the “evaluation of the purchased products/services”. The exception is for the “check of shipment status”, during which the smartphone is the most employed device. However, the “search of physical stores” is the task more frequently carried out through the smartphone. Finally, the TV is not used at any stages of the purchasing process. The respondents affirm that the PC meets their main needs during these processes (e.g.: comfort in visualizing information and in data entry). The PC is also the device most used for the impulse purchases.

Moreover, concerning the frequency of use of these devices during the purchasing processes, the most part of the participants declare to use the PC more than six times a year, the smartphone less than once a year or never, and tablets and TVs never.

Use of the Smart TV. 9 respondents out of 10 declare to know what a Smart TV is. The authors asked them to offer a definition of it. In general, they consider the Smart TV as a device with added features in relation to the traditional TV, making it similar to a PC. Considering that, the respondents identify as main feature of a Smart TV the Internet access, followed by the use of ad hoc applications that allow “to enjoy of various services”. 2 respondents point out that Smart TV are equipped with a specific OS.

7 of the 9 respondents who declare to know what a Smart TV is say they had already used a Smart TV. The remote control is the device employed for the interaction with this type of television; only 1 respondent declares to have used also a gesture recognition system. The most part (5 respondents) used the Smart TV to watch films/videos, 3 respondents to access to the Internet, and 3 respondents to use apps. The frequency of use of the Smart TV apps is low. Moreover, 2 respondents used them at home and 1 respondent in a store and during an event; the part of the day most affected is the evening. None of the 3 respondents who used Smart TV apps have purchased products/services through a TV app. The provided reasons are: “I do not have a Smart TV”, “I never thought of that”, and “I am not interested”.

The 4 respondents who used a Smart TV without interacting with app have not used them because they prefer other devices (e.g.: smartphone, PC, etc.) rather than Smart TV. They suppose that the interaction with these applications is uncomfortable.

Finally, the 3 respondents who never used a Smart TV declare that they don’t have this device.

Suitability for the Tasks. The participants in the usability test consider the app they analysed more focused on the technology rather than on the tasks to perform. This consideration emerges, for example, in the inability to save data useful to perform the consecutive tasks, in the lack of basic features (e.g.: retrieving the lost password), in requiring data not useful in order to complete a specific task, in the lack of visibility of central features, in the high number of steps to perform a task, etc.

The remote control is considered a not suitable and intuitive input device. It is considered complicated, uncomfortable, not ergonomic, laggard in the response on the screen, not controllable, not tolerant in case of a not totally correct position. The participants affirm that the buttons of the remote control were too many, difficult to push, and too close to one another (with the risk of pressing no correct buttons). Their position in the remote control is not totally intuitive, as well as the employed icons and labelling (e.g.: Fig. 1.a - “TTX”, used to switch on the keyboard letters/numbers or capital/small letters; Fig. 1.b - “PRE-CH” to delete letters), by obligating users to randomly push buttons. Moreover, some icons on the remote control are inconsistent with the operations required on the screen (e.g.: Fig. 1.c - “Press OK”, but the remote control has not an “OK” button, but a “sending” button) or they are similar to each other (e.g.: Fig. 1.d “source” button similar to the “sending” button).

Icons on some of the remote control buttons.

Full size image

Then, the remote control is considered too long. Indeed, the participants need to move from the top to the bottom of it, a sequence considered inconvenient.

The alphanumeric keyboard, through which data are entered (Fig. 2), resembles the keyboard of the feature phones. For this reason, it is considered not suitable to the Smart TV, since the interaction is hard (e.g.: inserting two letters on the same button). Moreover, during a data entry operation both the hand are used on the remote control, and users need at the same time to see both the screen and the remote control. So, the users require added/alternate input devices (e.g.: a physical keyboard, a controller, a smartphone, etc.), mainly based on gesture and voice recognition systems.

Alphanumeric keyboard

Full size image

The potential of the TV screen, used as output device, is not fully exploited. Also for this reason, the users tend to give more importance to the remote control rather than the contents on the screen (e.g. one of the respondents searches “add to cart” as a button of the remote control).

Self-Descriptiveness. The self-description of the system is low. In details, the users have little information about what they can do and how to move. One of the main problems of the system is to understand that, in order to add a product to her/his cart, he/she needs to register. Furthermore, the users do not receive feedback about specific actions or choices (e.g.: the stability of the chosen password) or they receive them too late (e.g.: the needed elements for registration).

Moreover, on the one hand, in some cases the selected interface elements are not clear (e.g.: the type of payment card, the channel, etc.); on the other hand, the selectable elements are not clear, or, if clear, users do not know how to select them.

The elements useful for navigation are not evident. Indeed, the menus at the bottom of the interface (Fig. 3) and the messages at the top are not very visible and, in some cases, the labelling is not present.

Menu at the bottom of the interface

Full size image

Finally, there is a lack not only of the aid to suggest users how to fill out a form or that help to fill it, but also of use guides and tutorials. It entails a random user interaction.

Conformity with User Expectations. The users pointed out a lack of conformity from different points of view: with familiar modes of interaction (e.g.: the users expect to be able to add a product to cart without be logged, in accordance with a traditional procedure of e-commerce); with data usually required (e.g.: during the registration phase or a purchase procedure); with familiar icons; with the expected position of specific icons (e.g.: “add a new card/address”, “sig in”); with the different modes to insert data in this application (letters are inserted only through the alphanumeric keypad whereas numbers are inserted both through the alphanumeric keypad and directly through the numeric keypad remote control; in some cases to insert data the form needs to be activated, in other cases not); with the modes used to refer to the same elements in this application (e.g.: “country” and “nation”); among remote control buttons and screen icons or commands. Moreover, the users understand with difficulty the labelling used on the remote control or on the screen (the “return” and “exit” button/icons are not clear).

Error Tolerance. The app is not tolerant to the errors. If, by mistake, the user presses a button instead of another, unexpected and irreversible things can happen (e.g.: exit the app). Moreover, the app presents a low possibility to avoid or retrieve the error. Finally, when it reports an error, it does not give additional and useful information (e.g.: during the registration phase).

Learnability. The participants declare that the prolonged use of the devices allows them to faster interact and to gradually understand the app mode of interaction. However, they have had many difficulties in understanding how to interact with the app (e.g.: the sequence of tasks to achieve a goal) and what data (and data format) are required to perform a task.

Satisfaction. Overall, the interaction with the Smart TV was unsatisfactory and hard, especially for the users who had never interacted with it. Their most frequent moods were: agitation, irritation, anger, frustration, fear, and boredom. Considering that, 2 of the participants abandoned the tasks they were doing. In addition, they were overly focused on understanding how to interact rather than carry out the task.

In general, the users consider that the Smart TV should communicate with other devices (e.g.: smartphone, PC, tablet) and that it is important to identify its real added value. Indeed, since the interaction with the Smart TV is considered complex, currently it is still uncompetitive compared with other devices. Figure 4 shows the devices that the participants would use as an alternative to the Smart TV in the different phases of a purchasing process. Inserting the shipping address and the registration process are considered the most inconvenient procedures on a Smart TV.

Exit questionnaire. Summary of the question: “Please indicate which of the following procedures you would have preferred to carry out through other devices”. Total: 10 participants.

Full size image

Moreover, the perception of the security of a Smart TV is low. Finally, the users consider as necessary added features: unique login for the different app, social activities, multimedia contents, etc.

In this paper, the authors present the main findings of a usability test on a Smart TV application for T-commerce. By evaluating the usability problems of the app, they also identified the general factors that may affect the interaction between user and Smart TV.

Mainly, the usability test participants consider the remote control an inappropriate input device. They point out that it is little ergonomic, its icons and labelling are not only difficult to understand but also risk to be inconsistent with the commands on the screen. These elements confuse the user and prevent him/her to interact effectively and efficiently with the Smart TV. On the contrary, the use of gesture/voice recognition systems could improve the interaction.

In conclusion, currently this device is considered uncompetitive with the others digital devices (smartphone, PC, tablet), both concerning the UX and usability, and the usefulness. Indeed, it is necessary to find a practical use of the Smart TV within a consistent multi-channel strategy, which represents an added value for the user (together with the traditional characteristics of the television, e.g.: home context, broadcasting contents, social use, etc.). Moreover, it is important to give the user more support in order to get him/her familiar with the device and with their modes of interaction.