Assessing the Customer eXperience Based on Quantitative Data: Virtual Travel Agencies

Usability is a key concept in Human – Computer Interaction. It was discussed for decades, but its definition is still evolving. A widely accepted definition is the one provided by the ISO 9241-210 “the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [1]. It highlights that usability refers not only to software systems, but also to products and services.

User eXperience (UX) goes beyond the three generally accepted usability’s dimensions: effectiveness, efficiency and satisfaction. The ISO 9241-210 standard defines UX as a “person’s perceptions and responses resulting from the use and/or anticipated use of a product, system or service” [1]. Again, UX does not limit to software systems; it applies to products and services as well.

A broader concept of Customer eXperience (CX) is emerging. It addresses the growing emphasis on service design and the service science as discipline [2]. Service science is an interdisciplinary area of study focused on systematic innovation in service.

There are well established usability evaluation methods. Evaluating UX is more challenging and arguably overwhelming for newcomers. As CX is a wider concept, assessing CX is even more challenging than assessing UX. If usability is a subset of UX, and UX is a subset of CX, that means usability evaluation methods and UX evaluation methods are also able to evaluate some CX aspects. But how can we evaluate other (uncovered) CX aspects?

Web mining techniques offer valuable outcomes on CX. A common research approach nowadays is the opinion mining. It analyzes customers’ opinion (sentiment analysis), based on their (qualitative) feedback. An alternative approach is the web content mining, which may also offer interesting (complementary) results, based on quantitative data.

The paper analyzes quantitative data on customers’ opinion, freely available at virtual travel agencies’ websites. Two websites were used as case studies: www.tripadvisor.cl and www.hotelclub.com. Section 2 reviews the concepts of usability, UX, and CX. Section 3 presents the experimental results. Data relationships are identified and interpreted as CX outcomes. Section 4 points out conclusions and future work.

The current ISO 9241 defines usability as “the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [1]. It makes clear that usability refers not only to software systems, but also to products and services. The ISO/IEC 20000‑1 standard define a service as “means of delivering value for the customer by facilitating results the customer wants to achieve” [3]. The use of the term “customer” does not necessarily imply a financial relationship.

Literature refers to usability dimensions as “attributes”, “factors” or “goals”. Several aspects are recurrent in all usability definitions, as well as in ISO standards: effectiveness, efficiency, satisfaction, and context of use. As Bevan, Carter and Harker highlight, the ISO 9241 current approach directly relates usability to user and business requirements: effectiveness means success in achieving goals, efficiency means not wasting time and satisfaction means willingness to use the system. The standard is currently under review, and three main lessons have been learned since its first version, back in 1998: (1) the importance of understanding UX, (2) the “measurement-based” usability approach is not enough, (3) the need to explain how to take account negative outcomes that could arise from inadequate usability [4].

UX goes beyond the three generally accepted usability’s dimensions: effectiveness, efficiency and satisfaction. The ISO 9241-210 standard defines UX as a “person’s perceptions and responses resulting from the use and/or anticipated use of a product, system or service” [1]. Again, UX does not limit to software systems; it applies to products and services as well.

Even if most authors consider UX as an extension of the usability concept, some are still using the terms usability and UX indistinctly. There is a tendency to move from usability to UX; even the former “Usability Professionals Association” (UPA) was renamed as “User Experience Professionals Association” (UXPA) [5].

A broader concept of CX is emerging. It addresses the growing emphasis on service design and the service science as discipline [2]. Service science is an interdisciplinary area of study focused on systematic innovation in service. A compelling CX leads to enhanced customer attraction and retention. As UX extends the usability concept, CX extends the UX concept. Service science and CX may benefit from the adoption of lessons learned in usability engineering and UX design.

The concept of CX is not (yet) well understood and has no clear and consistent definition. CX is increasingly discussed, but is rarely defined [6]. Laming and Mason define CX as: “the physical and emotional experiences occurring through the interactions with the product and/or service offering of a brand from point of first direct, conscious contact, through the total journey to the post-consumption stage” [7].

CX includes a series of interactions between the customer and the company (or companies) that offer the product and/or service, called customer “touch-points”. Joshi resumes CX management as “the concentrated efforts made by an organization to improve the quality of the interactions between customer and the organization at various touch-points in a manner that is consistent and effective” [6]. He points out that CX management may lead to differential advantage for service organizations.

Gentile, Spiller and Noci consider CX as an evolution of the concept of relationship between the company and the customer [8]. They identify several CX dimensions: sensorial, emotional, cognitive, pragmatic, lifestyle, and relational. Nambisan and Watt also identify CX dimensions: pragmatic, hedonic, sociability, and usability [9].

Obviously, CX impacts the future relation that between the customer and the service/product’s provider. That is why is surprising that the CX is a relatively new concept; the academic research on CX is limited yet, and its application in marketing theory is quite recent [7]. Schmitt proposes experiential marketing as a new approach to traditional marketing [10]. Experiential marketers perceive customers not only as rational, concerned about functional features and benefits, but also as emotional human beings, concerned with achieving pleasurable experiences. Schmitt points out four key features of experiential marketing: (1) it focuses on CX, (2) it focuses on consumption as a holistic experience, (3) it considers customers as emotionally and rationally driven, and (4) it requires eclectic methods and tools.

Usability evaluation methods are usually classified as: (1) empirical usability testing, based on users’ participation [11], and (2) inspection methods, based on experts’ judgment [12]. Evaluating UX is more challenging and arguably overwhelming for newcomers. More than 80 UX evaluation methods are described by Allaboutux.org [13].

As CX is a wider concept, assessing CX is even more challenging than assessing UX. If we consider usability as a subset of UX, and UX as a subset of CX, that means usability evaluation methods and UX evaluation methods are also evaluating some CX aspects. Evaluating other CX aspects requires specific methods. A key indicator is the customer satisfaction. But CX is much more than one overall satisfaction score [7]; it should be assessed at least at each “touch-point” (instance of interaction between the customer and the product/service).

Web mining techniques offer valuable outcomes on CX. A common research approach nowadays is the opinion mining. It analyzes customers’ opinion (sentiment analysis), based on their (qualitative) feedback. An alternative approach is the web content mining, which may also offer interesting (complementary) results.

Chiou, Lin and Perng advocate for a “strategic evaluation” of virtual travel agencies’ websites [14]. Kim, Kim and Han study several virtual travel agencies’ websites, analyzing the attributes that determine users’ preferences for a particular website [15]; they identify as the most critical attribute finding low fares, followed by security. The study of Bernardo, Marimon and Alonso-Almeida shows that both functional quality and hedonic quality contribute to the perceived value of a virtual travel agency, but functional quality is more relevant than hedonics [16]. All three above mentioned studies use the term “online travel agency”.

Our research interest in virtual travel agencies initially focused on usability. We proposed a methodology to evaluate transactional websites [17]. We also developed a set of usability heuristics for transactional web applications [18]. Most of the case studies that we used were virtual travel agencies. We are also evaluating virtual travel agencies websites’ usability on a regular basis, with our undergraduate and graduate students; it gives us an important feedback for both researching and teaching. Later on we extended our research to UX, and recently to CX.

First we thought customers’ opinions available on virtual travel agencies’ websites could complement our previous findings on usability and UX. But we quickly realized that very few comments refer to the user interaction with the virtual travel agency’s website; instead they refer to the quality of the services acquired through the website. As researches usually focus on qualitative customers’ comments, we decided to take an alternative approach, focusing on quantitative data.

Two virtual travel agencies were used as case studies: www.tripadvisor.cl and www.hotelclub.com. Customers’ opinions are freely available at both websites. Quantitative and qualitative data are available. Customers’ quantitative perceptions on several dimensions, as well as their overall satisfaction, are given in a 1 (negative perception) to 5 (positive perception) scale.

The present study analyzes quantitative data on hotels located in Viña del Mar, one of the most popular tourist destinations in Chile. Data were extracted in November, 2015. As observations’ scale is ordinal, and no assumption of normality could be made, data were analyzed using nonparametric statistics tests.

3.1 Case Study: www.tripadvisor.cl

TripAdvisor is a popular platform that shares customers’ reviews and compares prices. It offers links to several virtual travel agencies. Travelers’ reviews are both qualitative (comments) and quantitative (numeric evaluation). Quantitative evaluation is made on the following dimensions:

• D0 – Overall rating,

• D1 – Location,

• D2 – Sleep quality,

• D3 – Rooms,

• D4 – Service,

• D5 – Value,

• D6 – Cleanliness.

Travelers are using for the 6 dimensions, as well as for the overall rating, a 5 points scale:

• 1 – Terrible,

• 2 – Poor,

• 3 – Average,

• 4 – Very good,

• 5 – Excellent.

We analyzed the Chilean version of TripAdvisor, specifically the “Hotels” section: www.tripadvisor.cl/Hotels. We found data on 44 hotels located in Viña del Mar, Chile. A total of 3097 reviews were extracted. Most of the reviews do not rate all dimensions, therefore reviews were filtered. We selected only the reviews that evaluate the overall satisfaction and all 6 above mentioned dimensions. 865 reviews met the criteria and were analyzed.

The Spearman ρ test was performed to check the hypothesis:

• H₀: ρ = 0, the dimensions D_m and D_n are independent,

• H₁: ρ ≠ 0, the dimensions D_m and D_n are dependent.

As Table 1 shows, there are positive correlations between all dimensions. In all cases, the correlations are significant, because the p-value is less than the chosen significance level (α = 0.05). Correlations are moderate to very strong.

• The overall rating (D0) is very strongly correlated with D3 (Rooms), is strongly correlated with D2 (Sleep quality), D4 (Service), D5 (Value), D6 (Cleanliness), and is moderately correlated with D1 (Location); we could assume that location influences less than other dimensions when assigning the overall rating.

• Location (D1) is moderately correlated with all other dimensions: D2 (Sleep quality), D3 (Rooms), D4 (Service), D5 (Value), and D6 (Cleanliness).

• Dimensions D2 (Sleep quality), D3 (Rooms), D4 (Service), D5 (Value), and D6 (Cleanliness) are strongly correlated.

Travelers are classified by TripAdvisor in 5 types, described below; the number of analyzed reviews associated to each type is also indicated:

• Families (230 reviews),

• Couples (452 reviews),

• Solo (34 reviews),

• Business (73 reviews),

• Friends (76 reviews).

The Kruskal–Wallis H test was performed to check the hypothesis:

• H₀: there are no significant differences between the opinions of different type of travelers,

• H₁: there are significant differences between the opinions of different type of travelers.

We used p-value ≤ 0.05 as decision rule.

The Kruskal–Wallis H test results (Table 2) indicate that there are significant differences between the opinions of different types of travelers only concerning dimension D6 (Cleanliness).

3.2 Case Study: www.hotelclub.com

HotelClub is a virtual travel agency oriented to hotels/accommodations. As in the case of TripAdvisor, travelers’ reviews are both qualitative (comments) and quantitative (numeric evaluation). Quantitative evaluation is made using a 5 points scale, from 1 (worst) to 5 (best), on the following dimensions:

• D0 – Overall rating,

• D1 – Amenities,

• D2 – Cleanliness,

• D3 – Hotel staff,

• D4 – Comfort,

• D5 – Location,

• D6 – Value.

HotelClub is less popular in Chile. Only 3 hotels in Viña del Mar have got more than 5 reviews, and were therefore selected for our study. A total of 27 reviews were extracted. All reviews include overall evaluations (D0) and rates on all dimensions (D1, D2, D3, D4, D5, and D6).

The Spearman ρ test was performed to check the hypothesis:

• H₀: ρ = 0, the dimensions D_m and D_n are independent,

• H₁: ρ ≠ 0, the dimensions D_m and D_n are dependent.

As decision rule we used p ≤ 0.05.

Table 3 shows the correlations between all dimensions. Only correlations marked with (*) are significant at level α = 0.05, and will be interpreted.

• The overall rating (D0) is strongly correlated with dimension D1 (Amenities), and is moderately correlated with dimension D6 (Value).

• Amenities (D1) is moderately correlated with dimensions D4 (Comfort) and D6 (Value).

• Cleanliness (D2) is strongly correlated with dimensions D3 (Hotel staff) and D4 (Comfort), and is moderately correlated with dimensions D5 (Location) and D6 (Value).

• Hotel staff (D3) is moderately correlated with dimension D4 (Comfort).

• Comfort (D4) is strongly correlated with dimension D6 (Value).

• Location (D5) is moderately correlated with dimension D6 (Value).

Travelers are classified by HotelClub in 6 types, described below. The number of analyzed reviews associated to each type is also indicated; 7 travelers did not specify the group they belong to:

• Business (no reviews),

• Couples (13 reviews),

• Families (4 reviews),

• Friends (no reviews),

• Singles (3 reviews),

• LGBT (no reviews).

The Kruskal–Wallis H test was performed to check the hypothesis:

• H₀: there are no significant differences between the opinions of different type of travelers,

• H₁: there are significant differences between the opinions of different type of travelers.

As three types of travelers have no associated reviews, the hypothesis may be checked only for the types Couples, Families and Singles. We used p ≤ 0.05 as decision rule.

The Kruskal–Wallis H test results (Table 4) indicate that there are no significant differences between the opinions of different the three indicated types of travelers (Couples, Families and Singles).

CX extends the UX concept beyond the use of interactive software systems, services or products. It focuses on service science as interdisciplinary area of study. Service science and CX may benefit from the adoption of lessons learned in usability engineering and UX design. Assessing CX is more challenging than assessing UX. CX is more than an overall customer satisfaction score; it should be assessed at least at each “touch-point” (instance of interaction between the customer and the product/service).

Our research interest in virtual travel agencies initially focused on usability. Later on we extended our research to UX, and recently to CX. Web mining techniques offer valuable outcomes on CX. As researches usually focus on qualitative customers’ comments, we decided to take an alternative approach, focusing on quantitative data.

We analyzed quantitative data on hotels located in Viña del Mar, one of the most popular tourist destinations in Chile. Two virtual travel agencies were used as case studies: www.tripadvisor.cl and www.hotelclub.com. Travelers’ quantitative perceptions on several dimensions, as well as their overall satisfaction, are freely available at both websites. Travelers give their opinion in a 1 (negative perception) to 5 (positive perception) scale. 865 reviews were extracted from www.tripadvisor.cl, but only 27 reviews were extracted from www.hotelclub.com.

The Kruskal–Wallis H test indicates that in the case of TripAdvisor there are significant differences between the opinions of different types of travelers only concerning dimension Cleanliness. There are no significant differences in the case of HotelClub.

The Spearman ρ test indicates that in the case of TripAdvisor there are positive correlations between all dimensions; correlations are moderate to very strong. It seems that travelers tend to evaluate uniformly all dimensions. Results are similar in the case of HotelClub, but only approximately half of the correlations are significant at level α = 0.05.

As future work, we will extend the study. We will first analyze travelers’ perception on hotels from other regions of Chile and Latin America. We intend to check if the preliminary conclusions are also valid in new contexts. We will then extend the research targeting other regions. We also intend to analyze data available at other virtual travel agencies’ websites.