1 Introduction

The technological advancement in virtual reality (VR) has changed the way people interact with technology, particularly at cultural heritage sites. As a result, there is a significant increase in technology that offers a richer presentation of objects, buildings, and other features of cultural heritage sites. Recently, many virtual galleries of static two-dimensional (2D) images of objects on the internet are being replaced with applications that offer panoramic views of real places. Panorama enables a person to visualise places by rotating them at any angle on the computer screen [1], while VR video, also known as panoramic stereoscopic video, is viewed through VR output devices [2]. Virtual tourism has transformed the tourism industry by providing virtual tours that provide two functions. Firstly, it provides a complementary source of information, and secondly, it presents artefacts through a unique approach. Thus, a virtual tour transforms cultural heritage by removing geographical and material boundaries. Tussyadiah et al. [3] and Huang et al. [4] indicated that using the VR-based information prior to visits had a positive impact on visitors. Furthermore, it aided in decision-making [5], increased perceived enjoyment [6], and created a positive attitude change towards a location (i.e. [3, 7]).

Museums and cultural spaces are embracing the use of mobile museum technologies [8]. While the use of mobile technologies at cultural heritage sites is not new, the advancement of smartphone technology has opened an avenue for an alternative platform to create virtual tours to improve visitors’ experiences using VR technology [4, 9,10,11,12,13]. Moreover, institutions such as museums, libraries, cultural heritage sites, and institutes of higher learning can use the technology to enhance visitor experience [14]. As a result, various mobile technologies are currently used in museums and cultural spaces to enhance user experience (UX) and emphasise visitor-centred learning in the museum. Examples of such technologies are QR codes [15], mobile applications [16,17,18,19], serious gaming [20,21,22,23], audio guides [24, 25], augmented reality (AR) [26,27,28,29], and virtual reality (VR) [11, 13, 30, 55, 56].

Sarawak Cultural Village (SCV) is a living museum located 35 kms from the city centre in a 14-acre tropical vegetation at the foothills of Mount Santubong. SCV was selected for this study because it displays the cultural heritage of Sarawak’s major ethnic communities by creating a village with each unique ethnic house that portrays their respective customs and traditions. SCV is also the venue for the internationally acclaimed World Harvest Festival and Rainforest World Music Festival. Currently, SCV has a website to provide information to potential visitors but is limited to a brief background of SCV and its different ethnic houses, location, promotional packages, and admission price. Hence, there is an opportunity to provide a better visualisation as a complementary source of information for their potential visitors.

This study aims to: (1) explore the use of a smartphone as a visualisation and information tool at a cultural heritage site (living museum), with a focus on the design, development, and evaluation of virtual reality (VR) SCV tour in the form of an application for Android smartphones; (2) assess the usability of 3D visualisation on smartphones as an information aid for SCV; (3) to determine if there are usability differences between two types of virtual tour applications (VR-version and non-VR version) in terms of gender, nationality, and prior experience with VR and/or SCV.

2 Literature review

2.1 VR/AR for tourism

Although Cheong [82] identified VR as a potential threat to travel and tourism in 1995 due to its ability to provide an enjoyable experience that might exceed people’s expectation or becoming a substitute to actual travel because VR vacation is cheaper and more convenient than actual travel, Guttentag [73] in 2010 posited that tourism sector could benefit from the rapid development of VR application. Guttentag [73] believed that VR could be exploited for museums and cultural heritage sites. He further added that VR could be used as a marketing tool to provide rich sensory information to prospective visitors of a tourist destination. A study by Sepanuik et al. [83] in Poland and Belarus also suggested that virtual tour is not an alternative to actual travel, but it can be used to plan for tourism activity. They also suggested that virtual tours and 360° panoramas are the most popular tools for virtual tourism. Moreover, experiences using different virtual tours such as a panoramic or 3D walk-through environment are more effective than a traditional brochure for promotional/marketing activity [74]. Furthermore, previous studies suggested that VR replicas motivate visitors to visit the real sites [78] and gain more knowledge, which enriched their visits [79]. Virtual museums also allow active engagement and involvement by visitors, thus contribute to the promotion of real museums [81]. Cultural heritage tourism is one of the areas where VR/AR technology is widely used, particularly in the past 5 years (i.e. [3, 75,76,77]).

A mobile application system using augmented reality (AR) has changed the way visitors explore a historical place. AR is a technology that superimposes virtual objects onto the real world. Virtual objects, such as images, videos, texts, and GPS data, are computer generated. Chen [17] explored the use of AR for Oslo’s city using a map application that has two views: a navigation view and an AR view. The navigation view shows the location of points of interests and the user’s orientation on the map while the AR view shows a picture of the place at a specific time in history when a user points his phone camera in the direction of an object (i.e. a building).

2.2 VR/AR for museum and cultural heritage sites

Museums and cultural heritage sites have started to exploit new visualisation techniques for displaying their collections to attract more visitors, particularly the two-dimensional (2D) or more sophisticated three-dimensional (3D) visualisation approaches. The visualisations are either in a multimedia format such as photographs, videos, dioramas, animations, and films [31], or models in the form of 3D modelling, image-based modelling and rendering, stereoscopic visualisation, and immersive VR and VR technologies [32] or use technologies such as terrestrial laser scanning and digital photogrammetry [33]. AR technology allows visitors to fully explore the museum contents without restrictions such as time, space, and language [34]. Additionally, VR can provide visitors with a personalised cultural heritage experience [35] and allow visitors access to restricted objects or heritage sites [30].

Many museums have utilised 2D still images and panorama photographic techniques to create a virtual tour of the exhibits by organising them in a sequence that simulates real visitors behaviour and engagement in a physical museum. Furthermore, advancement in multimedia frameworks such as QuickTime VR (QTVR) software by Apple Inc. and the advances in Web technology have made virtual tours available on the internet ([36, 37]). An example of such a museum is the Smithsonian National Museum of Natural History which provides visitors with a “virtual, self-guided, room-to-room walking tour of the whole museum” that is accessible online through computers and mobile phones [38].

Alternatively, a more advanced photograph-based visualisation technique such as panoramic images is created by mapping stitched adjacent photographs onto the inner surface of a cylinder. This technique displays a 360° panoramic view from the centre of the cylinder, allowing users to move left or right, and has an advantage over 2D images as it captures more space and depth. Besides cylindrical panoramic images, a lesser-used method is spherical mapping where a panoramic image is mapped onto the outside surface of a sphere and is applied to a single 3D object that can be viewed in all directions [1].

Tukiainen [39] was motivated by limited access to the Villa Rulludd museum during its renovation to design and develop panoramic photography of the late nineteenth-century mansion which is one of the premises of the Espoo City Museum. The web-based virtual tour allows visitors to view the museum through a 360° panoramic tour using a desktop computer browser or mobile devices. Similarly, Xu et al. [40] developed an indoor navigation system using 3D models to support localisation, navigation, and visualisation at Hubei Provincial Museum. Other researchers such as Purnama et al. [41] utilised panoramic photography at the Tugu Pahlawan (Heroes Monument) Surabaya museum as a continual effort to boost the number of visitors. However, while the monument and other surrounding hotspots displayed both texts and images, the images of the whole museum were displayed in a limited view. This problem was solved through a 360° panorama photography by developing a virtual tour for the Tugu Pahlawan and another point of interests in the vicinity such as the Patung Jeneral Sudirman and Lapangan Upacara Tugu Pahlawan.

The Museum of Fine Arts in Lyon took advantage of the multimedia capability of mobile devices to enhance visitors’ experience at the museum by introducing a cross-platform application called Musée des Beaux-Arts de Lyon [19]. The mobile application provides information about the museum, such as opening hours, prices, individual itineraries, and telephone bookings. Additionally, the mobile application can be used as a mobile guide when visiting the museum and allows free access to anyone to visit the museum virtually. Five self-guided tours are offered on a selection of works from the museum’s collection in high definition visuals, videos, text, and audio guides, and 360° virtual tours along with a 3D plan of the museum. However, to reduce the mobile application’s download size, other media such as HD videos and audio guides are downloaded separately.

Recently, the Smithsonian National Museum of Natural History deployed an AR mobile application for its Bone Hall exhibition. The Skins & Bone application is available for iPhone, iPad, and iPod download [18]. The application works by pointing the mobile device at the skeleton of an animal in the exhibit, and the application will give the visitors a visualisation of how the animal move and look when alive on the mobile device screen.

2.3 VR/AR and visitors’ experience

The recent proliferation of VR/ AR technology and its adaptation to tourism and cultural heritage sites provides a new dimension to the visitors’ experience. However, it poses a question of whether visitors are willing to use these technologies [5]. It is suggested that visitors’ behaviour towards the use of the technology is influenced by perceived usefulness, enjoyment, and attitude [5]. Furthermore, poorly designed applications may significantly affect their experience and discourage them from using these technologies during their visit.

Moreover, virtual museums developed using various technologies for different platforms (i.e. PDAs, handhelds, haptics, Web 3D, VR, AR, and mixed reality) make museum content more accessible and attractive to visitors and enhance the visitors’ experience [80]. For example, museums and cultural heritage sites could provide more access to their collections using these technologies in a more interactive, entertaining, and flexible way which was traditionally restricted due to the limited physical spaces. These technologies not only enable intuitive interactions and provide entertaining and learning experiences, but also provide a sense of immersion for visitors using various tracking devices (i.e. inertia and magnetic trackers), immersive displays (Head Mounted Display-HMD), and sensing technologies (i.e. data gloves). Han et al. [13] concluded that VR technology could also provide a reflective learning experience in a cultural heritage context.

More recently, a study by Chung et al. [34] concluded that AR could provide a better experience to visitors for cultural heritage tourism in Korea while preserving its integrity. They found that AR’s perceived advantage and aesthetics influenced visitor satisfaction with AR technology and subsequently affect visitors’ intentions.

Furthermore, AR/VR and mixed reality benefited the average visitors and could help visitors with special needs. For example, Letellier [84] developed a virtual tour for the remote, inaccessible area of Luang Prabang in Laos and argued that virtual tour can be the best option for visitors with health issues or difficulties to cope with environmental factors such as humidity and temperature. A study by Sheehy et al. [86] provided a good overview of research on AR for museum visitors with special needs such as sensory, or physically impaired visitors. They concluded that AR is used widely to provide a multi-sensory experience for visitors, with a varied focus for different target groups, for example, the affordances of such technology or different techniques for specific special needs. A recent framework on hearing-impaired mobile AR provides a good understanding of what should be included in the AR museum exhibits design for the hearing impaired [85]. They identified several key components for mobile museum AR such as aesthetics, interaction, interest, usability, satisfaction, motivation, curiosity, enjoyment, perceived control, self-efficacy, and focused attention.

3 Methodology

3.1 Design and development of the virtual tour

The virtual tour application for SCV used in this study is a native application running on Android, which is available for download without charge on Google Play Store. The virtual tour development used the Mobile Application Development Life Cycle (MADLC) [42], a relatively new framework for mobile applications development. The MADLC consists of seven main phases: identification, design, development, prototyping, testing, deployment, and maintenance.

In the first phase, requirements for the virtual tour application were collected and categorised. Visits to SCV were made to gather information about the various ethnic houses. During the visits, pictures, videos, and panoramic images of the ethnic houses were collected. Next, interview sessions were carried out with the staff of SCV to gather information on the artefacts and architecture of the ethnic houses. Information was also collected through the library and online search.

In the second stage, requirements and information from the first phase were translated into the application’s initial design. The integral part of the design phase is creating a storyboard for the user interface interaction, which describes the overall flow of the virtual tour application. Figure 1 shows the virtual tour application storyboard, while Fig. 2 shows the virtual tour application flowchart.

Fig. 1
figure 1

SCV virtual tour application storyboard

Full size image
Fig. 2
figure 2

System flowchart

Full size image

During this stage, a system design was also designed to visualise how the virtual tour application would interact with the VR headset. Figure 3 shows the system design for the virtual tour application. The controller continuously updates the system’s state based on data from the accelerometer and gyroscope to ensure it matches the orientation based on the user’s head position. Application data such as photospheres, in-object pictures, and in-app icons are stored in the smartphone’s memory, while the phone screen displays the scene view based on the users’ orientation.

Fig. 3
figure 3

System design

Full size image

In the development phase, the application is coded in two distinct stages: (1) to build the core functionalities of the application such as the visualisation component, navigation component, which connects panoramic images, and information component, and (2) to design and develop the user interface with the same look and feel for different Android smartphone models.

The interaction of the mobile application is a combination of touch input and gaze input. Unity’s Cardboard SDK is used for the implementation of the gaze input. The Cardboard SDK contains pre-programmed scripting to track head movement when used with the Google Cardboard. However, the script can be used without Google Cardboard while still retaining the tracking capabilities. This way, the user can look around in the panorama by physically pivoting on a spot. The SDK also has a pre-programmed reticle that acts as a pointer mouse. Touch input is implemented for tapping the buttons in the application when the reticle is aligned over a button.

The photospheres were collected using a Google camera on Samsung Galaxy Note 3 mounted on a tripod. Google camera allows capturing 360° panoramas and automatically stitches photospheres from the selected photos. In total, 42 photospheres were collected. However, only 18 photospheres were used in the application due to parallax error and photospheres’ corruption during the photospheres collection process. There were a few mismatches after the stitching process, especially objects near to the camera, resulting in blurred areas in the photospheres.

Adobe Photoshop CS 6 was used to fix the blurred areas by replacing them using the corrupted photospheres as shown in Fig. 4. Next, the tripod legs or other unwanted objects such as other visitors’ images were removed. The photospheres included the interior and exterior of each ethnic house as well as selected strategic points between houses. The strategic points are designed to help in the navigation to ensure that the jumps would not be too sudden or drastic between two ethnic houses.

Fig. 4
figure 4

Blurred area fixed using the corrupted photosphere

Full size image

Materials from the Google material library, as illustrated in Fig. 5, were used for navigation and POI buttons depending on the function.

Fig. 5
figure 5

Icons from Google material library

Full size image

A radio button icon is uniformly used for navigation and POI buttons, as shown in Fig. 6a. The function of the button is revealed when the reticle hovers over the button. If the button is navigational, the radio button changes to a walking man image, as illustrated in Fig. 6b At the same time, navigational information pops up to inform users where they are heading towards as long as the reticle and button are aligned. Otherwise, an information display box will appear when the reticle aligns with POI buttons, as shown in Fig. 6c. When users look away from the information dialogue box, it will disappear.

Fig. 6
figure 6

Left (radio button and reticle), middle (navigational button) right (information)

Full size image

The menu page, as shown in Fig. 7, is designed to be accessible from all scenes. The menu button is an expanding menu that includes a toggle switch between VR and non-VR (Fig. 8), a map, and information (a short tutorial on using the map).

Fig. 7
figure 7

Menu page and its behaviour when the reticle hovers over the button: Expanded menu (right)

Full size image
Fig. 8
figure 8

Toggle button and system modes: Mobile mode (a) and VR mode (b)

Full size image

The map offers users a shortcut to go to any houses. The map uses two different icons. A pin-drop icon shows other locations that users can go to, while the person-pin icon shows the current location. For example, Fig. 9 shows the map at the entrance of SCV. There are eight pin-drop icons that will take users to the houses when activated. The person-pin icon is at the entrance, which tells users that the current location is at the entrance.

Fig. 9
figure 9

The map with location pins at each ethnic house

Full size image

We followed the guidelines provided by Google Cardboard developers in designing the UX for VR and traditional 2D forms. These guidelines focused on the fundamental aspect of human perception and cognition.

In this study, the 3D space was developed using the Unity 3D game engine software, as illustrated in Fig. 10. Unity 3D allows the development of both 2D and 3D applications.

Fig. 10
figure 10

Unity 3D software

Full size image

Open-source Software Development Kit (SDK) was used in Cardboard’s application development, including the pre-programmed scripts (prefabs). Prefabs allow access to the smartphone gyroscope and accelerometer and states and continuously update the 3D world view based on this information [72]. Android SDK for Unity was used to develop the Android-based application with the *.apk extension. Unity 3D software allows for cross-platform development. During the prototyping phase, a prototype was tested using the simulator provided by Unity 3D. The prototype was tested on a range of mobile devices with variable screen size and versions of the Android operating system. The required changes were implemented during the development phase after analysing the feedback. When the final prototype was ready, usability testing was executed with end-users of the virtual guide.

3.2 Usability testing

Usability testing of the virtual tour was conducted during and after the development phase. During development, the evaluation and testing were done using Unity 3D’s simulator. After development, it was evaluated by end-users using the System Usability Scale (SUS) developed by Brooke [43]. The participants evaluated either the non-VR version or the VR version (with Google Cardboard). They were asked to fill a Google Form, which contained an informed consent form, demographic questions, SUS items, and an additional comment section on the virtual tour application. The participants were also asked if they owned a smartphone (yes/no) and their experience with smartphone technology. Moreover, they were asked to state whether they had any prior experience with virtual tours (yes/no), if they visited SCV before (yes/no), and if they experienced motion sickness while using the mobile application (yes/no).

3.3 Instrument

The System Usability Scale (SUS) is a widely used instrument to measure the perceived usability of various products and systems since it was first introduced by Brooke [43]. It was reported that 43 per cent of usability studies had utilised SUS [53]. In addition, a study conducted to evaluate SUS indicated that the SUS is indeed a valid and reliable tool for measuring usability [54]. Its analysis of SUS scores data collected over ten years showed that the SUS is a highly robust and versatile tool for usability professionals.

The SUS is a short questionnaire of 10 items with alternating positive and negative statements to avoid response biases with a five-point Likert score, ranging from strongly disagree to strongly agree. The SUS scoring requires all 10 items to be rated, or adjustment should be made to the unrated item(s). These items will be further analysed and calculated for the final score (0–100), and graded from A to F or using adjective ratings. Different grade rankings of SUS are presented in Bangor Kortum &Miller [45].

Prior to 2009, a SUS analysis was only focused on the perceived usability (unidimensional), but Lewis and Sauro [57] posited that the SUS is a bi-dimensional (usability and learnability sub-scales) measure. Since then, several studies were conducted on whether it should be treated as unidimensional or bi-dimensional (i.e. [58,59,60]). Borsci et al. [58] and Lewis and Sauro [59] proposed using a bi-dimensional analysis if the context of the study is suitable, and the participants are experienced users. The participants in our study had varied experiences with the smartphone, virtual tours, and the SCV before the experiments. Considering the diverse participants’ background, both unidimensional and bi-dimensional analysis of the SUS scale were conducted in this study. Furthermore, Tullis and Stetson [61] stated that SUS enables researchers to measure the perceived usability using a small sample size (between 8–12) when compared to other instruments (i.e. Questionnaire for User Interface Satisfaction-QUIS, Computer System Usability Questionnaire-CSUQ, Words and researchers’ instrument).

3.4 Participants

A convenience sampling technique was used to recruit 40 participants aged 20 to 45 years old, with varied backgrounds to participate in the usability testing. They were comprised of Malaysians and internationals either studying or working in Universiti Malaysia Sarawak. The internationals included nationals from Bangladeshi, China, Kyrgyzstan, Tajikistan, Pakistan, Nigeria, Cameroon, Morocco, Indonesia, Eritrea, Turkey, and the USA. They were approached by researchers and were asked if they would like to participate in this study. They were not offered any monetary or rewards for their contribution. The participants were divided into a non-VR version and a VR version evaluation group with an equal number of males and females. Since this study involved human participants, the university’s research ethics guidelines were followed throughout the research process.

3.5 Usability testing procedure

Below is the procedure of the study:

  1. 1.

    Participants recruitment: 40 participants were recruited to participate in the study;

  2. 2.

    Briefing session: Participants were briefed about the purpose of the study and the testing procedures;

  3. 3.

    Informed consent: Participants were asked and signed the consent form in the Google form. They were told that they could withdraw from the testing at any time without consequence;

  4. 4.

    Virtual Tour evaluation: Participants were given a demonstration on how to use the system and were asked to use the virtual tour for as long as possible;

  5. 5.

    Questionnaire: After using the virtual tour, participants were required to complete the Google form;

  6. 6.

    Debriefing session: The instructor answered any queries the participants had.

4 Result and discussion

4.1 Demographic analysis

A total of 40 participants took part in the study, all with varied years of experience using a smartphone: 22 participants (11 M and 11 F) had prior experiences with virtual tours, while 18 participants (9 M and 9 F) did not have experience with a virtual tour; regarding prior experience with SCV, a total of 22 (13 M and 9 F) participants had been to SCV, while 18 participants (11 F and 7 M) had not been to SCV; those who had been to SCV were 12 Malaysians and 10 international participants, while eight Malaysians and 10 international participants had not been to SCV.

In addition, 11 participants had both experiences with the virtual tour and SCV before the study, while seven participants had neither experience with a virtual tour nor SCV.

Seven participants (18.42%) felt motion sickness while using the virtual tour (we excluded two participants who failed to respond to the question). Six of them used the mobile version (non-VR version) of the application, whereas only one participant using the VR version experienced motion sickness. A total of 13 participants from the non-VR version group and 18 participants from the VR group did not experience motion sickness.

4.2 Individual SUS score for VR-version and non-VR-version

In general, the mean score of both versions of the mobile application for overall SUS score, usability score, and learnability score is more than 50. The SUS scores based on the virtual tour versions (non-VR or VR version) are summarised in Table 1.

Table 1 Comparison of mean scores between mobile and Google Cardboard version
Full size table

The mean scores for the overall SUS score, usability score, and learnability score are 68.80, 72.10, and 57.00, respectively, for the non-VR version, while for the VR version (Google Cardboard), the mean scores are 72.10, 75.50, and 58.40, respectively. This study shows that scores for both the non-VR version and the VR version are acceptable in terms of SUS [44]. According to the criteria in Bangor et al. [45], the overall SUS score and learnability score for the non-VR version are acceptable, whereas the usability score is good. An acceptable score suggests that the mobile application is usable but has some usability issues that need improvement. The VR version of the mobile application scored higher in all three components compared to the non-VR version. The VR version has a reasonable overall SUS and usability score, and satisfactory for learnability. A previous study indicated that a VR environment leads to higher satisfaction due to the novelty effect of using the product for the first time [62].

Although the difference in the scores is not significant, the SUS score does indicate a usability problem. However, SUS scores do not diagnose exactly what is the problem. According to Bangor et al. [45], the SUS is not biased against a particular user interface type. The lower difference in overall SUS scores could be caused by the VR version’s interaction design that uses a combination of gaze input and touch input instead of touch as the only input. The VR version uses gaze as the only interaction method. The satisfactory score for learnability for both versions suggests that users have a problem learning to use the application. Nielsen [46] describes learnability or “easy to learn” as the users’ ability to accomplish the task when using the system for the first time. During the experiment, it was observed that participants required further assistance. One of the participants in this study (a trained industrial usability evaluator) commented that “the system lacks descriptive information on how it should be started, operated, and ended”. Another participant commented that “it is very easy to get lost in the system, it would be great if you could add some marks that show where I have been, and where I have not reached yet”. This is echoed by another comment that “the map lacks the information which can tell me where I am about to go, please include caption above the house on the map”. These comments are similar to an issue in a study by Ariffin and Talib [63], which shows that the virtual tour lacks assistance during navigation. It is posited that natural representation and navigation ease are essential to design principles for VR [64]. In addition, Rautenbach et al. [65] highlighted that a lack of information (i.e. the environment models’ level) could lead to the inaccuracy of cognitive maps developed in a virtual experience. The design and development stage of the virtual tour was executed without any involvement from users. However, both usability testing and users’ feedback highlighted the issues hindering learnability, thus implying that participatory design should be considered in the design and development stage. Involving users throughout the design process may help root out usability issues that may escape the developers. As the users are ultimately the end-users of the mobile application, their feedback on the interaction method and the user interface design is crucial information in development. Developers can assess the users’ familiarity with the interaction method and other usability issues using this information. Previous studies also suggest that user involvement in the system development leads to system success [66], positive effects and user satisfaction [67], and improved interface usability [68].

4.3 SUS score between gender and nationality

In terms of gender, male participants scored higher for overall SUS score, usability, and learnability than female participants. The overall SUS score for male participants is 72.85 and 68.05 for females. The usability score is higher than the overall SUS score for males and females at 75.50 and 71.70, respectively. Both male and female participants scored the lowest in the learnability component at 62.05 and 53.35, respectively. However, within the group of participants evaluating the non-VR version of the application, male participants’ mean score is higher in the overall SUS score and usability component. Female participants within the group have a higher learnability score at 58.90 compared to male participants at 55.10. Table 2 shows a comparison of the mean SUS score according to gender in the non-VR version group.

Table 2 Comparison of mean scores according to gender
Full size table

The difference between genders in the usability of interactive systems such as VR-based applications, smart interface applications, and many others is still undetermined. Although a recent study by Wang, Tsai, Lu, and Wang [47] recorded a significant difference between gender for an instructional application based on Google Glass for mobile phone disassembly tasks, it was only for one part of the task. Furthermore, they indicated that male participants recorded a higher SUS score, similar to this current study. On the other hand, another study found no significant difference between genders for two types of interfaces provided to them [48]. Furthermore, this study also suggests that gender does not significantly affect the SUS score and further validated the results of previous research evaluating the perceived usability over a wide range of interface types in Bangor et al. [45]. In a different context, Orfanou, Tselios, and Katsanos [49] conducted a perceived usability evaluation on e-learning management systems found that gender did not affect the SUS scores. In a similar study, Othman [50] assessed users’ experience with a multimedia guide at cultural heritage sites using the Multimedia Guide Scale (MMGS) and discovered no gender difference in the mean scores. The results of both studies are the same despite using different evaluation instruments and different interface types.

The means of the scores based on nationality follow the same pattern as gender. Malaysians scored higher than internationals in all three components. The overall SUS score for Malaysians is 72.70, whereas the SUS score for internationals is 68.20. Malaysians and internationals scored the lowest for learnability at 65.20 and 50.20, respectively, and scored highest in usability at 74.55 and 72.65, respectively. Within the non-VR version group, Malaysians scored higher in all three components than internationals. Furthermore, the mean score for learnability among internationals is 45.10, which is unacceptable. Table 3 shows a comparison of the mean scores according to nationality.

Table 3 Comparison of mean scores according to nationality
Full size table

A cross-cultural study of on-site film-tourism experiences showed that nationalities played a significant role in the visitors’ experiences during visits to the theme park [69]. The findings suggested that cultural affiliation or nationalities have a significant impact on touristic experiences. This could be one reason for the similar pattern in this study in which Malaysian scored higher in SUS than internationals. Although we only measure the usability score with the VR and non-VR application, knowledge about local cultural heritage and the context of use could have contributed to this finding.

Further analysis should be made to validate this assumption by measuring and comparing the overall visitors’ experience of different nationalities with the overall experience of Malaysians. Furthermore, as suggested by Bevan and MacLeod [70], it is important to consider the context (the users, tasks, and environment) in the usability evaluation of a product. In addition, a special issue of interacting with a computer by Smith and Yetin [71] highlighted the importance of culture within the usability in which they further explained that culture is usually defined as “patterns of values, attitude and behaviours shared by two or more people” (p1).

4.4 SUS score for prior experience effects

Tables 4 and 5 show the comparison of SUS scores between participants with and without experience with a virtual tour, and SCV.

Table 4 Comparison of mean scores according to virtual tour experience
Full size table
Table 5 Comparison of mean scores according to prior visit to SCV
Full size table

The SUS scores for participants with virtual tour experience are at least five points higher than those with no experience on all three components. Moreover, the mean score for learnability for participants with no experience is 45.39, which is below the acceptable score compared to the score of 67.77 for participants with virtual tour experience. Experience with a virtual tour is vital to learnability as there is a 22.38 points difference. Within the non-VR version of the application group, a similar pattern is found. Participants who had used virtual tours on any platform, in general, scored higher on overall SUS score, usability, and learnability scores. Similarly, there is a 14.95 points difference in the learnability score between participants with and without virtual tour experience. This is similar to a study by Sauro [51] on website usability, which reported higher SUS scores with prior experience. Researchers also reported a significant effect of experience on the SUS scores of e-learning management systems [49]. A more recent study also suggested that participants with no prior experience relied more on the menu, took more time to complete given tasks, and generated more errors [52]. However, previous visits to SCV do not significantly affect the SUS score but shows a significant decrease in the usability score. These results suggest that experience with similar virtual tour software significantly impacts SUS scores rather than the physical experience of visiting SCV.

The result shows the opposite trend for participants who have visited SCV before the evaluation, as shown in Table 5. They generally scored lower on all three components than participants who have not visited SCV. Moreover, for those who have a prior visit, the learnability score is below the acceptable range at 45.20 compared to 68.80 for those who have never visited SCV before the evaluation. From the analysis, a prior visit to SCV has some effect on the participants to learn to use the mobile version of the application.

4.5 Test of normality

Due to the small sample size, further analysis of the normality test was made to the SUS score, usability score, and learnability score, to examine the score distribution to ensure that the sample data is drawn from a normally distributed population. Table 6 shows the value for skewness and kurtosis for the overall SUS score, which are 0.74 and −0.53, 1.48 and −0.85 for usability, and −0.74 and 1.51 for learnability, respectively. As skewness and kurtosis for the three variables fall within the ± 1.96 limit, the distributions are concluded to be normal.

Table 6 Skewness and Kurtosis values of SUS score, usability, and learnability
Full size table

4.6 Analysis of overall SUS scores between VR version and non-VR version

Further analyses were carried out to investigate the effect of the mobile application version on overall SUS score, usability, and learnability. Α two-tailed independent t test was performed and the results show that there is no difference between the non-VR version (M = 68.80, SD = 14.85) and VR version (M = 72.10, SD = 14.47) for overall SUS score where t(38) = −0.712, p = 0.481; between non-VR version (M = 71.70, SD = 12.69) and VR version (M = 75.50, SD = 14.41) for usability score where t(38) = −0.885, p = 0.382; and between non-VR version (M = 57.00, SD = 33.79) and VR version (M = 58.40, SD = 28.95) for learnability score where t(38) = −0.141, p = 0.889.

4.7 Analysis of SUS scores between gender, nationality, and prior experience for VR version and non-VR version

Analyses were also conducted to investigate SUS attributes in the context of gender, nationality, and prior experience across the two virtual tour versions. Α two-tailed independent t test was conducted to investigate the effect of gender on all three components. Results show that there is no significant difference observed in the males (M = 72.85, SD = 17.54) and females (M = 68.05, SD = 10.41) for overall SUS score, where t(38) = 1.04, p = 0.305, males (M = 75.50, SD = 15.31), and females (M = 71.70, SD = 11.58) for usability, where t(38) = 0.885, p = 0.382 and males (M = 62.05, SD = 33.97) and females (M = 53.35, SD = 228.05) for learnability score, where t(38) = 0.883, p = 0.383.

4.8 Analysis of SUS scores between nationality for VR version and non-VR version

In regard to nationality, there is no significant difference found between local Malaysians (M = 72.70, SD = 12.79) and internationals (M = 68.20, SD = 16.17) in the SUS score, where t(38) = 0.98, p = 0.335. No significant difference was discovered between Malaysians (M = 74.55, SD = 12.07) and internationals (M = 72.65, SD = 15.12) in the usability score, where t(38) = 0.44, p = 0.663 as well as between Malaysians (M = 65.20, SD = 26.69) and internationals (M = 50.20, SD = 33.91) in the learnability score, where t(38) = 1.55, p = 0.128.

4.9 Analysis of SUS scores for prior experience for VR version and non-VR version

Prior experience is also investigated in this virtual tour application’s evaluation. Prior experience in this study refers to an experience with virtual tour applications on other platforms and previous visits to SCV. Experience with similar virtual tour applications may influence the SUS scores as they are not first-time users. Similarly, prior experience with SCV may affect the scores as well.

Α two-tailed independent t test was performed to investigate the SUS score, usability, and learnability with no experience, and experience with other virtual tour application conditions. There is no significant difference found between with experience (M = 76.05, SD = 16.21) and no experience (M = 70.61, SD = 8.87) in the usability score, where t(38) = 1.35, p = 0.187. However, significant difference is found between with experience (M = 74.55, SD = 17.16) and no experience (M = 65.44, SD = 8.65) for SUS score, where t(38) = 2.17, p = 0.037, and between with experience (M = 67.77, SD = 32.17) and no experience (M = 45.39, SD = 25.37) in learnability score, where t(38) = 2.40, p = 0.021.

Similarly, an independent t test was conducted to compare the SUS score, usability, and learnability between participants with previous visits to SCV and participants without previous visits. Results show that there is a significant difference between with SCV experience (M = 69.36, SD = 12.48) and without SCV experience (M = 78.78, SD = 13.82) in the usability score where t(38) = −2.31, p = 0.027. There is no significant difference between with SCV experience (M = 66.59, SD = 14.85) and no SCV experience (M = 75.17, SD = 13.10) for SUS score, where t(38) = −1.92, p = 0.063, and between with SCV experience (M = 55.32, SD = 31.43) and no SCV experience (M = 60.61, SD = 31.25) in learnability score, where t(38) = −0.531, p = 0.598.

5 Conclusion

This study was carried out to develop a smartphone virtual tour application for SCV using MADLC. Two versions of the virtual tour were developed for Android smartphones: a non-VR and a VR version. This study utilised a smartphone camera and a free Google Camera photo-stitching application which is a low-cost tool to take moderately high-quality photospheres for visualisation purposes. Despite some issues related to the panoramic images during development, panoramic visualisation on the smartphone as an information aid is feasible and performs adequately.

The usability of the virtual tour according to the SUS score is at an acceptable level. However, some usability issues were indicated based on the usability and learnability components scores. Several lessons were learnt which could become guidelines for designing and developing a mobile-based VR application for cultural heritage sites, particularly living museums. First, users should be involved in the process to ensure that users’ requirements are met and addressed. Second, a wide range of metrics should be used in the evaluation. Although this is time-consuming and costly, it will provide a better understanding of the product. This study only measured the virtual tour’s usability; thus, the score does not indicate the overall user experience (UX) with the virtual tour. Although usability is important in accepting a new product, researchers should consider beyond usability and include “user experience”. Third, researchers should acknowledge the differences between user interfaces for VR applications and non-VR applications. Immersive and non-immersive VR applications could affect the users during the evaluation; for example, several participants reported that they had motion sickness while operating the applications. Evaluators should explain this possibility to the potential users and provide several breaks to reduce the effects. Fourth, it is important to consider conducting different types of evaluation at different stages of the design and development of the VR application. For example, heuristic evaluation (HE) or cognitive walk-throughs could be considered to refine the design before the development stage, and usability testing before deployment to real users.