Keywords

1 Introduction

Internet of Things (IoT) is a new paradigm that provides a variety of things/ objects capable of interacting and cooperating with each other, using the Internet [14, 15]. IoT technologies have been used to create smart environments such as smart cities, smart homes, and smart buildings.

Ubiquitous Computing (UbiComp) is one of the areas that are driving IoT [10, 23]. As a consequence, we believe that solutions developed by/for the UbiComp community can be applied in the IoT systems since both areas have characteristics in common. For instance, the context-awareness feature, which is the system’s capability of monitoring contextual information regarding the user, the system and the environment to provide relevant services, is an important characteristic presented in both areas [6, 7, 12, 13, 17, 20].

In our research lab, which is called GREat, Group of Computer Network, Software Engineering and SystemsFootnote 1, IoT applications have been developed to support professors and students in different everyday situations [4]. One of these applications is called GREatRoom [8], which provides a way to register the presence of people in a room, making possible to see who is in the place at a given time. Also, it allows sharing files among individuals in the same room.

GREatRoom uses beacons, Android mobile phones and cloud computing (i.e., Google Cloud) to perform its functionalities. The beacons send broadcast messages using Bluetooth [16], allowing the identification of new objects (smartphones) to a database in a Cloud, which in turn can notify the presence of new objects in a room. The interaction with the user occurs in the Android application, which allows the user to see who is in the room and what files are available to download.

GREatRoom aims to provide an efficient way to register people presence in events such as seminars and workshops, removing the need for manual signatures. However, the risk of users feeling overwhelmed is high, since this IoT application may notify a large number of new objects, such as mobile phones, in a short time. Therefore, we believe that the interaction quality evaluation of IoT applications becomes even more essential in the software development process than in both traditional and ubiquitous computing applications.

This work then aims to use solutions regarding the quality interaction evaluation that come from UbiComp to IoT [5, 6]. For that, we apply the work from [5], which defines a set of measures for the interaction quality evaluation of ubiquitous systems, in IoT systems. These measures help to verify if the application does not disturb users unnecessarily and if it supports user activities at the right moment and place, delivering the best service possible. We apply these measures in a case study that allows the identification of problems regarding user interaction. Therefore, a significant contribution of this work is the empirical results about using software measures from UbiComp area in an IoT application. Moreover, we also discuss there are specific characteristics of IoT systems that need to be considered and prioritized when doing such kind of evaluation, for example, the quality of the interaction between things in an IoT environment [14, 19].

The rest of this paper is organized as follows: Sect. 2 presents the main concepts about UbiComp, IoT and HCI Quality Evaluation. Section 3 presents the IoT application, GREatRoom. Section 4 presents the evaluation performed in GREatRoom, including results and discussion about the findings. Finally, Sect. 5 presents the conclusions and future work.

2 Background

2.1 UbiComp and IoT

The idea of UbiComp area is to make computing devices available everywhere, capable of sharing information between people and machines [25]. One of the primary goals of UbiComp is to simplify Human-Computer Interaction (HCI), by making the technology distributed, so that the user does not perceive their presence [11].

Nowadays, we are living a moving towards IoT, and Ubicomp is one of the areas that are leading up to IoT. The term Internet of Things, according to [1], appeared in 1999, where initially the idea was to use the new RFID technology in resource management. Over the past years, new definitions were added to the term to cover a greater number of applications. However, the main idea of having a context-aware technology without human intervention continues until today [9].

There are several IoT definitions in literature, and they have some specificities that differ them from one another. The definition adopted for this work was proposed by [22], which defines IoT as a global, self-configuring network infrastructure where protocols are used to communicate between things. These “things” are represented in both physical and virtual environments, and have identification, physical attributes, virtual personalities, and are seamlessly integrated into a network. In addition to being identifiable, objects/things can have various capabilities, such as processing power, storage, and communication technologies.

Besides, such objects may have ubiquitous computing features, such as context-awareness [9] and transparency [24], making possible to develop smarter applications and, consequently, the possibility of using UbiComp solutions in the IoT environment.

Moreover, Sundmaeker et al. (2010) define “things” as active participants in information where they can interact and communicate with each other and with the environment through the exchange of data and information about the context. Also, they can react autonomously according to the events that trigger actions and create services with or without direct human intervention [22].

UbiComp and IoT areas have also similar descriptions in the part of data sharing between machines to carry out a certain task, in the part of transparency with the end user, among other characteristics.

Therefore, due to these similarities, we believe we can use evaluation techniques and methods from UbiComp to evaluate IoT applications.

Table 1. Software measures for calmness characteristic. Adapted from [5]
Full size table

2.2 HCI Quality Evaluation

Applications from UbiComp has characteristics that completely change the way users interact with technology, and so impact on usability and user experience [20]. For example, a ubiquitous system has the context-awareness characteristic, which means that is capable of collecting contextual data and adapting behavior according to the data [17].

Moreover, these systems can be present in several devices, such as personal computers, smartphones, smart watches and others. Thus, they have the mobility characteristic, what makes it possible for the system to be working everywhere and available at anytime, as a consequence, this creates an increased risk that the user will feel disturbed by the system [18].

The work from [20] proposes a set of measures to evaluate the user interaction with ubiquitous applications regarding context-awareness and the work from [5] proposes a set of software measures (See Table 1) to the calmness characteristic. Calmness is the capability of the system to support user activities at the right time and place, delivering the best service possible [18]. We believe calmness has a great impact on user satisfaction and therefore also on acceptance of the ubiquitous application.

3 GREatRoom

3.1 Main Features

In the GREat research laboratory, there are often several presentations, such as lectures, mini-courses, thesis and dissertations defenses, among others. Such events usually take place in the seminar room and the meeting room. A necessity for such events is to register the presence of people who are attending the events in that room. Nowadays, this registration is done manually, through the collection of signatures. Therefore, in order to improve this activity, GREatRoom application was designed. The main feature of this application is then to provide a way to register the presence of people in a room, making possible to see who is in the room at a given time.

Fig. 1.
figure 1

GREatRoom - Initial screens

Full size image

Moreover, there is another need for people who participate in such events as follows. Typically, they present their work use files such as slides, documents, images, videos. After the presentation, they usually share these files with the interested people via email. However, this activity is not always successful, sometimes the presenter forgets to do so or even does not get all emails of the attendees, then people who are interested in the presentation are not always able to get the file. Thinking about this problem, another feature provided by GREatRoom is allowing the sharing of files among people in the same room. This feature is designed as follows: as soon as a person is registered in the room, he/she can make files available to others who must also be registered in the same room.

The application works as follows: (i) The user must log in using a Facebook account (Fig. 1); (ii) After performed login, the application automatically detects in which room the user is and then he-she is registered in it. In the example of Fig. 2(a), the user was registered in “Seminar Room”; (iii) Then, the application allows the user to access the file options of this room, as well as see the people who are in it (See Fig. 2(b)); and (iv) By accessing the persons, the user can visualize that two people besides him/her are in the room (Fig. 3(a)), also he/she can access five files (Fig. 3(b)).

Fig. 2.
figure 2

GREatRoom - Room

Full size image
Fig. 3.
figure 3

GREatRoom - Persons and Files

Full size image

3.2 Overview: Behavior and Technologies

GREatRoom allows the creation of IoT environments through the use of wireless labels (Beacons) distributed in places and objects. In this environment, devices collect user data and adapt their services to create a ubiquitous interactive environment. In this interactive environment, things (objects, places, and users) exchange information, perform processing, and generate information and services.

The interactions with the system are performed by a mobile device with operating system Android and communication Bluetooth. GREatRoom identifies the presence of users in the room and performs a check-in on the attendance list. After the identification, the user may have access to the content and information of a room. The idea is that the user has minimal interaction with the system and some actions are done transparently, ensuring such services are offered in the most natural way possible. However, the system will not perform person management or direct exchange of information between the users.

Figure 4 illustrates the flow of messages of GREatRoom, which uses the Publisher-Subscriber standard [21] to register a user in a room, which is identified by the beacon device that sends broadcast messages. From these messages, it is possible to find new objects in this room. The beacons have a high accuracy of device location, and this is a good point to determine the distance and calculate when a user is present in the room with a higher degree of assertiveness.

Fig. 4.
figure 4

GREatRoom overview

Full size image

The Publisher-Subscriber standard was used to perform event notifications on GREatRoom. This standard uses design principles to minimize the use of network bandwidth, to ensure reliability and to guarantee message delivery. These principles are important for the IoT paradigm, because mobile devices have limited resources [2]. In the GREatRoom environment, users have the role of Subscribers who receive notifications that are related to any object present in that room, and the objects are Publishers who will notify all their Subscribers when a change of state occurs.

3.3 Architecture

For the GREatRoom development, the reference architecture proposed by [3] was used. However, only the components for communication, service, and transformation of a physical entity in virtual entity were used. Figure 5 gives an overview of the GREat Room architecture. The description of each component belonging to the system is described as follows:

Fig. 5.
figure 5

GREatRoom architecture

Full size image

Application Layer is responsible for performing part of the data processing sent by the devices and for presenting a user interface.

IoT Service Layer provides a feature to make it accessible to other parts of the IoT system and can be used to obtain information from a feature of a sensor device or a storage resource connected to a network.

Virtual Entity Service Layer is responsible for manipulating entity services and providing access to an entity via operations that enable readings and/or updates of values of its attributes.

Communication Layer is responsible for the transmission of packets between the application and the objects of the environment using addressing and IDs. The arguments for packet transmission can be configured and include unicast/multicast addressing and access control. Another functionality is to obtain a locator from a given ID, which can be done internally based on a lookup table.

Device Layer represents the several devices that can communicate with the Great Room application, such as smartphones, air conditioners and smart lamps.

4 Evaluation

4.1 Planning and Execution

The execution of the evaluation was carried out in parallel with five users in two rooms. Each room had an IOS device with an application installed to simulate signals from a Beacon. At the beginning of the tests, users received training on the purpose of the application and its features. Also, users were instructed to use the system simulating a real situation in which they were in a meeting room.

Initially, each user logged in to use the features. Subsequently, the application allows the visualization of all places near the user and performs the presence of this user in an individual room according to the environment in which he is at the time of the test. After being present in a room the user can view the people and files linked to this same room, being possible to manipulate the files as well as delete a certain file, submit a new file to be shared, and download a file to your personal device.

We used two instruments for data collection in the evaluation: interaction logs and questionnaires. Interaction logs record in a file the contextual information collected, the adaptations made and their durations, and the failures that occurred during the use of the application. The questionnaire allows the collection of information about the perception of the user related to the quality of the interaction with the application.

4.2 Results

Table 2 presents the results from the GREatRoom evaluation. In general, the application showed satisfactory results with the following subjective measures: Availability Degree, Context-Awareness Degree, Relevancy Degree and Courtesy Degree highlights that the application has a good degree of calmness. This is important, because GREatRoom works when users are attending lectures, seminaries and others, then, it requires focus and, thus, it is important not to take the attention away from these activities.

Table 2. Results of the HCI evaluation of GREatRoom
Full size table

However, the following measures did not present good results: Adaptation Degree, and Adaptation Correctness Degree. It is interesting to highlight that although the application is good in recognizing the context correctly (Context Correctness = 98%) and took little time to adapt (1.28s), the adaptation degree (78%) and adaptation correctness (79%) are below the expected. That happened because the system presented some problems when the user positioned him/herself on the border between two rooms, where the system was not sure which room the user should assign. This result gives an idea of what can be improved about the distance calculation between one beacon and another. One of the solutions is to increase the number of beacons to perform the mapping of the room so that the synchronization between them can be made.

4.3 Discussion

The measures used to evaluate the application give us an insight into what can be improved in the application. Therefore, this study shows that measures for UbiComp are feasible to be applied in IoT applications. In general, by applying the same measures in ubiquitous [5] and IoT applications we could notice that: (i) Collecting the necessary data for the measures calculation requires the same effort; (ii) The questionnaires used to capture users’ opinion are adequate; and (iii) The results from the measures can be interpreted following the same procedures.

However, there are specific characteristics of IoT systems that need to be considered and prioritized when doing such kind of evaluation. This is because a thing in IoT sends data through the Internet to another thing, therefore, the latency, intermittency and reliability are big issues to the quality of the interaction [19]. This indicates that these characteristics should be investigated and also measured. The results of the measures should be analyzed together, considering that the values of one are impacted by the other. Moreover, new quality characteristics specific for IoT applications may also be required.

For example, in GREatRoom, when a new object (i.e., a user) enters the environment, an Internet message is sent to the Cloud, to update the list of people who are in that environment. If the application shows five persons in a room, and then a new user enters in that room, it can take a few minutes for the application be updated. During this time, the application of the users that were initially in the room shows five persons, and the application of the new user shows six persons. Therefore, the time it takes to synchrony the things can not be long, and then it is necessary to measure this synchronicity, characteristic not yet considered in UbiComp applications.

5 Conclusion and Future Work

This paper describes an evaluation of an IoT application using measures of ubiquitous computing systems. Since IoT applications have similar requirements to ubiquitous applications, we apply measures originally defined for ubiquitous systems to an IoT application, called GREatRoom. It was then possible to verify strengths and suggest improvements for this application based on the evaluation, and also raise research perspectives in the area of IoT systems evaluation. We believe that is possible to use good solutions from one area to the other and also discussed that in this paper.

As future work, we plan to distribute several beacons in different types of rooms of GREat to carry out the control of people who attend the different environments of the GREaT laboratory. With this intelligent environment deployed, it is possible to add functionalities to the GREat Room application, allowing the user interact with the objects present in the environment. Another future work is the verification of other quality characteristics from ubiquitous computing [7] and that could have similar aspects with IoT.