[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Next Issue
Volume 7, November
Previous Issue
Volume 7, September
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 

Multimodal Technol. Interact., Volume 7, Issue 10 (October 2023) – 8 articles

  • Issues are regarded as officially published after their release is announced to the table of contents alert mailing list.
  • You may sign up for e-mail alerts to receive table of contents of newly released issues.
  • PDF is the official format for papers published in both, html and pdf forms. To view the papers in pdf format, click on the "PDF Full-text" link, and use the free Adobe Reader to open them.
Order results
Result details
Select all
Export citation of selected articles as:
16 pages, 2707 KiB  
Article
Developing Teams by Visualizing Their Communication Structures in Online Meetings
by Thomas Spielhofer and Renate Motschnig
Multimodal Technol. Interact. 2023, 7(10), 100; https://doi.org/10.3390/mti7100100 - 19 Oct 2023
Cited by 2 | Viewed by 2377
Abstract
This research pursues the question of how the computer-generated analysis and visualization of communication can foster collaboration in teams that work together online. The audio data of regular online video meetings of three different teams were analyzed. Structural information regarding their communication was [...] Read more.
This research pursues the question of how the computer-generated analysis and visualization of communication can foster collaboration in teams that work together online. The audio data of regular online video meetings of three different teams were analyzed. Structural information regarding their communication was visualized in a communication report, and then, discussed with the teams in so-called digitally supported coaching (DSC) sessions. The aim of the DSC is to improve team collaboration by discerning helpful and less helpful patterns in the teams’ communication. This report allows us to recognize individual positions within the teams, as well as communication structures, such as conversational turn taking, that are relevant for group intelligence, as other research has shown. The findings pertaining to the team members during the DSC were gathered via questionnaires. These qualitative data were then matched with the quantitative data derived from the calls, particularly social network analysis (SNA). The SNA was inferred using the average number of interactions between the participants as measured in the calls. The qualitative findings of the teams were then cross-checked with the quantitative analysis. As a result, the assessment of team members’ roles was highly coherent with the SNA. Furthermore, all teams managed to derive concrete measures for improving their collaboration based on the reflection in the DSC. Full article
Show Figures

Graphical abstract

Graphical abstract
Full article ">Figure 1
<p>Excerpt from a sample communication report.</p>
Full article ">Figure 2
<p>Airtimes and interaction diagrams of four meetings in case 1.</p>
Full article ">Figure 3
<p>Airtimes and interaction diagrams of four meetings in Case 2.</p>
Full article ">Figure 4
<p>Airtimes and interaction diagrams of four meetings in Case 3.</p>
Full article ">
25 pages, 22197 KiB  
Article
A Universal Volumetric Haptic Actuation Platform
by Patrick Coe, Grigori Evreinov, Mounia Ziat and Roope Raisamo
Multimodal Technol. Interact. 2023, 7(10), 99; https://doi.org/10.3390/mti7100099 - 17 Oct 2023
Cited by 2 | Viewed by 2408
Abstract
In this paper, we report a method of implementing a universal volumetric haptic actuation platform which can be adapted to fit a wide variety of visual displays with flat surfaces. This platform aims to enable the simulation of the 3D features of input [...] Read more.
In this paper, we report a method of implementing a universal volumetric haptic actuation platform which can be adapted to fit a wide variety of visual displays with flat surfaces. This platform aims to enable the simulation of the 3D features of input interfaces. This goal is achieved using four readily available stepper motors in a diagonal cross configuration with which we can quickly change the position of a surface in a manner that can render these volumetric features. In our research, we use a Microsoft Surface Go tablet placed on the haptic enhancement actuation platform to replicate the exploratory features of virtual keyboard keycaps displayed on the touchscreen. We ask seven participants to explore the surface of a virtual keypad comprised of 12 keycaps. As a second task, random key positions are announced one at a time, which the participant is expected to locate. These experiments are used to understand how and with what fidelity the volumetric feedback could improve performance (detection time, track length, and error rate) of detecting the specific keycaps location with haptic feedback and in the absence of visual feedback. Participants complete the tasks with great success (p < 0.05). In addition, their ability to feel convex keycaps is confirmed within the subjective comments. Full article
Show Figures

Figure 1

Figure 1
<p>Overhead view of keycap device. The coordinates of read points are labeled as XY pixel coordinates. Exploratory area is within the blue dotted lines. Keys are placed within red boundary boxes.</p>
Full article ">Figure 2
<p>A perspective projection view of the keycaps.</p>
Full article ">Figure 3
<p>Keycaps exploratory regions as shown along the borders and convex/concave areas (on the <b>left</b>) and across the borders and flat areas (on the <b>right</b>).</p>
Full article ">Figure 4
<p>XY superpositions of twelve continuous patterns of exploratory tracks across three plastic keycaps.</p>
Full article ">Figure 5
<p>3D plots of continuous patterns of exploratory tracks across three plastic keycaps averaged over twelve trials.</p>
Full article ">Figure 6
<p>For (<b>A</b>–<b>F</b>): Continuous patterns of exploratory tracks across three plastic keycaps. Movement is focused along the borders and convex keycap profile (on the <b>left</b>) and across the borders and flat areas (on the <b>right</b>). Marker of designated start point is marked by the red circle. Path to be followed is marked by the blue line.</p>
Full article ">Figure 7
<p>(<b>Left</b>): Pressure force-sensor calibration nomogram. (<b>Right</b>): Grand average pressure force of light touch exploration for a given exploratory direction (blue arrows) of three keycaps averaged over thirty-six trials.</p>
Full article ">Figure 8
<p>Three-keycap software interface. Red and green frame mark feedback zones.</p>
Full article ">Figure 9
<p>The schematic diagram of the perspective projection view of the Universal Volumetric Haptic Actuation Platform.</p>
Full article ">Figure 10
<p>The rotation to translation mechanism of the Universal Volumetric Haptic Actuation Platform.</p>
Full article ">Figure 11
<p>Demonstration mockup of the Universal Volumetric Haptic Actuation Platform without (<b>Left</b>) and with (<b>Right</b>) Surface Go tablet on top.</p>
Full article ">Figure 12
<p>Twelve-keycap software interface for haptic rendering 3D features. Keycap layout and force feedback zones (red/green boundaries) are shown for illustration purposes only.</p>
Full article ">Figure 13
<p>A schematic software diagram for haptic rendering 3D features in a 12-keycap arrangement.</p>
Full article ">Figure 14
<p>Instructional sketches used to help instruct users on the task to be performed.</p>
Full article ">Figure 15
<p>Flowchart of research procedure.</p>
Full article ">Figure 16
<p>Top view of the participant exploring the Microsoft Surface GO touchscreen as instructed with eyes closed. Keycap layout is shown for illustration purposes only.</p>
Full article ">Figure 17
<p>View of the participant’s hand during exploration of the virtual keyboard of the Microsoft Surface GO touchscreen placed over the mock up of the Haptic Actuation Platform. Keycap layout is shown for illustration purposes only.</p>
Full article ">Figure 18
<p>Data of EPs collected on the task completion time, scanpath distances and error rate in two different modes of haptic feedback. The letter R refers to the red zone, while the letter G refers to the green zone. The number after the zone letter refers to the width in pixels of each zone. The blue bars are used for the R10 × G25 zones, while the green bar is used for the R40 × G100 zones. Where used, ave refers to average.</p>
Full article ">Figure 19
<p>Data collected at “thick” force feedback (on the <b>left</b>), at “thin” force feedback zone (on the <b>right</b>). aveTime refers to average time in seconds. aveTrLen refers to average track length. TrLenMedian refers to median track length.</p>
Full article ">Figure 20
<p>Scanpath comparison in two different modes of haptic feedback. The letter R refers to the red zone, while the letter G refers to the green zone. The number after the zone letter refers to the width in pixels of each zone.</p>
Full article ">Figure 21
<p>Confusion matrices of keycap exploration in two different modes of haptic feedback: thick on left, thin on right. Confusion matrix helps to show the deviation of how far the setected number of keycaps is from the assigned keycap position. Items are highlighted in an increasingly darker shade of green according to the amount of selected responses.</p>
Full article ">Figure 22
<p>Superposition of EPs (blue lines) and averaged over 7 trials per task of detecting 12 keycaps (cyan lines). Data collected at “thick” force feedback (on the <b>left</b>), at “thin” force feedback zone (on the <b>right</b>).</p>
Full article ">Figure 23
<p>2D plot of user EPs scanpaths averaged over 7 trials per task at detecting 12 keycaps (Each represented by a different color) in the absence of visual feedback with different haptic force feedback. Data collected at “thick” force feedback (on the <b>left</b>), at “thin” force feedback zone (on the <b>right</b>).</p>
Full article ">Figure 24
<p>3D plot of EPs averaged over 7 trials per task at detecting 12 keycaps in the absence of visual feedback. Data collected at “thick” force feedback (on the <b>left</b>), at “thin” force feedback zone (on the <b>right</b>).</p>
Full article ">Figure 25
<p>(<b>Left</b>): The subjective task load ratings according to Nasa TLX questionnaire. (<b>Right</b>): Nasa TLX radar plot. Each coloured line represents an individual participant’s response.</p>
Full article ">
19 pages, 8188 KiB  
Article
Immersive Unit Visualization with Augmented Reality
by Ana Beatriz Marques, Vasco Branco, Rui Costa and Nina Costa
Multimodal Technol. Interact. 2023, 7(10), 98; https://doi.org/10.3390/mti7100098 - 17 Oct 2023
Cited by 1 | Viewed by 2482
Abstract
Immersive Unit Visualization is an emergent form of visualization that arose from Immersive Analytics where, unlike traditional visualizations, each data point is represented by an individual visual mark in an immersive virtual environment. This practice has focused almost exclusively on virtual reality, excluding [...] Read more.
Immersive Unit Visualization is an emergent form of visualization that arose from Immersive Analytics where, unlike traditional visualizations, each data point is represented by an individual visual mark in an immersive virtual environment. This practice has focused almost exclusively on virtual reality, excluding augmented reality (AR). This article develops and tests a prototype of an Immersive Unit Visualization (Floating Companies II) with two AR devices: head-mounted display (HMD) and hand-held display (HHD). Results from the testing sessions with 20 users were analyzed through qualitative research analysis and thematic coding indicating that, while the HHD enabled a first contact with AR visualization on a familiar device, HMD improved the perception of hybrid space by supporting greater stability of virtual content, wider field of view, improved spatial perception, increased sense of immersion, and more realistic simulation, which had an impact on information reading and sense-making. The materialization of abstract quantitative values into concrete reality through its simulation in the real environment and the ludic dimension stand out as important opportunities for this type of visualization. This paper investigates the aspects distinguishing two experiences regarding data visualization in hybrid space, and characterizes ways of seeing information with AR, identifying opportunities to advance information design research. Full article
Show Figures

Figure 1

Figure 1
<p>Floating Companies (FLOC) visual coding.</p>
Full article ">Figure 2
<p>Warping of the vertical axis along views 2, 3, 4, and 5. The red dots represent the spheres in the visualization. In the second view (<b>a</b>), the spheres are distributed randomly in space; in the third view (<b>b</b>), the spheres are arranged vertically according to the average profit per employee (€) on a uniform axis; in the third view (<b>c</b>), there is a deformation of this axis, influencing the arrangement of the spheres as illustrated in the figure. In the last view (<b>d</b>), all the spheres are arranged at ground level.</p>
Full article ">Figure 3
<p>Photo montage reproducing the black and white background (Passthrough mode), as seen while using the FLOC II application with Oculus Quest 2.</p>
Full article ">Figure 4
<p>Immersive Unit Visualization showing mass shooting victims in virtual reality, by Ivanov, Danyluk and Willett (2018) [<a href="#B10-mti-07-00098" class="html-bibr">10</a>]. Reproduced with permission from Ivanov, A., CHI’18 Extended Abstracts; published by ACM, 2018.</p>
Full article ">Figure 5
<p>“DeathTolls Experience”, by Ali Eslami (2016) [<a href="#B12-mti-07-00098" class="html-bibr">12</a>]. Reproduced with permission from Ali Eslami, <a href="https://alllesss.com/DeathTolls-Experience" target="_blank">https://alllesss.com/DeathTolls-Experience</a> (accessed on 14 September 2023); 2016.</p>
Full article ">Figure 6
<p>AR visualization with location recognition for predicting sea level rise, proposed by Sarri et al. (2022) [<a href="#B14-mti-07-00098" class="html-bibr">14</a>]. Reproduced with permission from Froso Sarri, 2022 International Conference on Interactive Media, Smart Systems and Emerging Technologies; published by IEEE, 2022.</p>
Full article ">Figure 7
<p>Screenshot of transcript analysis using thematic coding in Atlas.ti. Most of the participants speech is in Portuguese—their mother tongue.</p>
Full article ">
33 pages, 10930 KiB  
Systematic Review
Evolution and Trends in Sign Language Avatar Systems: Unveiling a 40-Year Journey via Systematic Review
by Maryam Aziz and Achraf Othman
Multimodal Technol. Interact. 2023, 7(10), 97; https://doi.org/10.3390/mti7100097 - 16 Oct 2023
Cited by 2 | Viewed by 4248
Abstract
Sign language (SL) avatar systems aid communication between the hearing and deaf communities. Despite technological progress, there is a lack of a standardized avatar development framework. This paper offers a systematic review of SL avatar systems spanning from 1982 to 2022. Using PRISMA [...] Read more.
Sign language (SL) avatar systems aid communication between the hearing and deaf communities. Despite technological progress, there is a lack of a standardized avatar development framework. This paper offers a systematic review of SL avatar systems spanning from 1982 to 2022. Using PRISMA guidelines, we shortlisted 47 papers from an initial 1765, focusing on sign synthesis techniques, corpora, design strategies, and facial expression methods. We also discuss both objective and subjective evaluation methodologies. Our findings highlight key trends and suggest new research avenues for improving SL avatars. Full article
Show Figures

Figure 1

Figure 1
<p>Overview of the SL avatar development process.</p>
Full article ">Figure 2
<p>PRISMA flow diagram for selection process.</p>
Full article ">Figure 3
<p>Keyword co-occurrence network.</p>
Full article ">Figure 4
<p>Published sources per year.</p>
Full article ">Figure 5
<p>Distribution of papers across countries.</p>
Full article ">Figure 6
<p>Collaboration between the countries.</p>
Full article ">Figure 7
<p>Frequency of publication based on authors.</p>
Full article ">Figure 8
<p>Co-authorship map for selected studies.</p>
Full article ">Figure 9
<p>Frequency of studied sign languages.</p>
Full article ">
18 pages, 5627 KiB  
Systematic Review
Metaverse and Extended Realities in Immersive Journalism: A Systematic Literature Review
by Alberto Sanchez-Acedo, Alejandro Carbonell-Alcocer, Manuel Gertrudix and Jose Luis Rubio-Tamayo
Multimodal Technol. Interact. 2023, 7(10), 96; https://doi.org/10.3390/mti7100096 - 12 Oct 2023
Cited by 6 | Viewed by 3849
Abstract
Immersive journalism is a new form of media communication that uses extended reality systems to produce its content. Despite the possibilities it offers, its use is still limited in the media due to the lack of systematised and scientific knowledge regarding its application. [...] Read more.
Immersive journalism is a new form of media communication that uses extended reality systems to produce its content. Despite the possibilities it offers, its use is still limited in the media due to the lack of systematised and scientific knowledge regarding its application. This is a problem because it is a very powerful technology that changes the way audiences receive information and can be used both for new forms of storytelling that generate greater user engagement and for very sophisticated disinformation, which is why it is really important to study it. This study analyses articles published in the last 5 years that cover the use of extended technologies and the metaverse applied to immersive journalism. A systematic literature review applying PRISMA was carried out to identify literature within Web of Science, Scopus and Google Scholar (n = 61). Quantitative and qualitative analyses were conducted on the data collection techniques, the type of the data and the analysis techniques used. The results show a low level of methodological maturity, with research that is fundamentally descriptive and not very formalised, which limits the scope of its results and, therefore, the transfer of knowledge for its application in the configuration of new immersive journalistic products. The metaverse and extended technologies are considered independently and with distinct applications. It is concluded that research in this area is still in an initial exploratory and generalist stage that offers results that are not yet applicable to the promotion of this type of media format. Full article
Show Figures

Figure 1

Figure 1
<p>Methodological Process.</p>
Full article ">Figure 2
<p>Process of identifying studies using the PRISMA model.</p>
Full article ">Figure 3
<p>Number of articles per year of search (2017–2022).</p>
Full article ">Figure 4
<p>Stated hypotheses regarding the metaverse.</p>
Full article ">Figure 5
<p>Stated hypotheses regarding immersive technologies.</p>
Full article ">Figure 6
<p>Percentage of the number of articles by study discipline.</p>
Full article ">Figure 7
<p>Percentage of articles according to the object of the virtuality continuum from which they are studied.</p>
Full article ">Figure 8
<p>Percentage of items according to the techniques for the development of the experience.</p>
Full article ">Figure 9
<p>Percentage of items by professional sector to which they apply.</p>
Full article ">Figure 10
<p>Percentage of the number of articles by research type.</p>
Full article ">Figure 11
<p>Percentage of universe of study.</p>
Full article ">Figure 12
<p>Percentage of types of information collected.</p>
Full article ">Figure 13
<p>Percentage of articles by type of analysis.</p>
Full article ">Figure 14
<p>Percentage of items according to sampling technique.</p>
Full article ">Figure 15
<p>Percentage ranking of article conclusions.</p>
Full article ">
24 pages, 4347 KiB  
Review
Current Trends, Challenges, and Future Research Directions of Hybrid and Deep Learning Techniques for Motor Imagery Brain–Computer Interface
by Emmanouil Lionakis, Konstantinos Karampidis and Giorgos Papadourakis
Multimodal Technol. Interact. 2023, 7(10), 95; https://doi.org/10.3390/mti7100095 - 12 Oct 2023
Cited by 5 | Viewed by 3731
Abstract
The field of brain–computer interface (BCI) enables us to establish a pathway between the human brain and computers, with applications in the medical and nonmedical field. Brain computer interfaces can have a significant impact on the way humans interact with machines. In recent [...] Read more.
The field of brain–computer interface (BCI) enables us to establish a pathway between the human brain and computers, with applications in the medical and nonmedical field. Brain computer interfaces can have a significant impact on the way humans interact with machines. In recent years, the surge in computational power has enabled deep learning algorithms to act as a robust avenue for leveraging BCIs. This paper provides an up-to-date review of deep and hybrid deep learning techniques utilized in the field of BCI through motor imagery. It delves into the adoption of deep learning techniques, including convolutional neural networks (CNNs), autoencoders (AEs), and recurrent structures such as long short-term memory (LSTM) networks. Moreover, hybrid approaches, such as combining CNNs with LSTMs or AEs and other techniques, are reviewed for their potential to enhance classification performance. Finally, we address challenges within motor imagery BCIs and highlight further research directions in this emerging field. Full article
Show Figures

Figure 1

Figure 1
<p>Techniques of signal acquisition.</p>
Full article ">Figure 2
<p>BCI categorization based on user’s activity.</p>
Full article ">Figure 3
<p>Typical EEG BCI pipeline.</p>
Full article ">Figure 4
<p>Classification techniques.</p>
Full article ">Figure 5
<p>Machine learning architectures.</p>
Full article ">Figure 6
<p>Deep learning architectures.</p>
Full article ">Figure 7
<p>Hybrid deep learning architectures.</p>
Full article ">Figure 8
<p>Protocol for the retrieval of papers.</p>
Full article ">Figure 9
<p>Typical EEG CNN architecture [<a href="#B52-mti-07-00095" class="html-bibr">52</a>].</p>
Full article ">Figure 10
<p>AlexNet architecture [<a href="#B66-mti-07-00095" class="html-bibr">66</a>].</p>
Full article ">Figure 11
<p>Architecture of the ResNet [<a href="#B67-mti-07-00095" class="html-bibr">67</a>].</p>
Full article ">Figure 12
<p>Autoencoder architecture [<a href="#B84-mti-07-00095" class="html-bibr">84</a>].</p>
Full article ">
14 pages, 275 KiB  
Article
Safe City: A Study of Channels for Public Warnings for Emergency Communication in Finland, Germany, and Greece
by Sari Yli-Kauhaluoma, Milt Statheropoulos, Anne Zygmanowski, Osmo Anttalainen, Hanna Hakulinen, Maria Theodora Kontogianni, Matti Kuula, Johannes Pernaa and Paula Vanninen
Multimodal Technol. Interact. 2023, 7(10), 94; https://doi.org/10.3390/mti7100094 - 10 Oct 2023
Viewed by 2134
Abstract
Public warning systems are an essential element of safe cities. However, the functionality of neither traditional nor digital emergency warnings is understood well enough from the perspective of citizens. This study examines smart city development from the perspective of safety by exploring citizens’ [...] Read more.
Public warning systems are an essential element of safe cities. However, the functionality of neither traditional nor digital emergency warnings is understood well enough from the perspective of citizens. This study examines smart city development from the perspective of safety by exploring citizens’ viewpoints. It investigates people’s perceptions of the ways in which they obtain warnings and information about emergencies involving health risks. Data were collected in the form of focus group interviews and semi-structured interviews in Finland, Germany, and Greece. The results suggest that people place a lot of trust in their social network, receiving text messages, and their ability to use web-based search engines in order to obtain public warnings. The study discusses the challenges identified by citizens in the use of conventional radio and television transmissions and sirens for public warnings. Based on the results, citizens demonstrate informed ignorance about existing mobile emergency applications. Our results imply that it is not sufficient to build emergency communication infrastructure: the development of smart, safe cities requires continuous work and the integration of both hard and soft infrastructure-oriented strategies, i.e., technological infrastructure development including digitalisation and education, advancement of knowledge, and participation of people. Both strategic aspects are essential to enable people to take advantage of novel digital applications in emergency situations. Full article
22 pages, 1359 KiB  
Article
Identifying Which Relational Cues Users Find Helpful to Allow Tailoring of e-Coach Dialogues
by Sana Salman, Deborah Richards and Mark Dras
Multimodal Technol. Interact. 2023, 7(10), 93; https://doi.org/10.3390/mti7100093 - 2 Oct 2023
Cited by 2 | Viewed by 2166
Abstract
Relational cues are extracts from actual verbal dialogues that help build the therapist–patient working alliance and stronger bond through the depiction of empathy, respect and openness. ECAs (Embodied conversational agents) are human-like virtual agents that exhibit verbal and non-verbal behaviours. In the digital [...] Read more.
Relational cues are extracts from actual verbal dialogues that help build the therapist–patient working alliance and stronger bond through the depiction of empathy, respect and openness. ECAs (Embodied conversational agents) are human-like virtual agents that exhibit verbal and non-verbal behaviours. In the digital health space, ECAs act as health coaches or experts. ECA dialogues have previously been designed to include relational cues to motivate patients to change their current behaviours and encourage adherence to a treatment plan. However, there is little understanding of who finds specific relational cues delivered by an ECA helpful or not. Drawing the literature together, we have categorised relational cues into empowering, working alliance, affirmative and social dialogue. In this study, we have embedded the dialogue of Alex, an ECA, to encourage healthy behaviours with all the relational cues (empathic Alex) or with none of the relational cues (neutral Alex). A total of 206 participants were randomly assigned to interact with either empathic or neutral Alex and were also asked to rate the helpfulness of selected relational cues. We explore if the perceived helpfulness of the relational cues is a good predictor of users’ intention to change the recommended health behaviours and/or development of a working alliance. Our models also investigate the impact of individual factors, including gender, age, culture and personality traits of the users. The idea is to establish whether a certain group of individuals having similarities in terms of individual factors found a particular cue or group of cues helpful. This will establish future versions of Alex and allow Alex to tailor its dialogue to specific groups, as well as help in building ECAs with multiple personalities and roles. Full article
Show Figures

Figure 1

Figure 1
<p>Screenshot of interaction with Alex.</p>
Full article ">Figure 2
<p>One of Alex’s dialogues with color-coded relational cues (see <a href="#mti-07-00093-t001" class="html-table">Table 1</a>). A user either receives the empathic version with relational cues or the neutral one with text inside the brackets.</p>
Full article ">Figure A1
<p>Decision Tree classifier.</p>
Full article ">Figure A2
<p>Logistic Regression classifier.</p>
Full article ">Figure A2 Cont.
<p>Logistic Regression classifier.</p>
Full article ">Figure A3
<p>Input variables with their weightage in determining the target variable.</p>
Full article ">
Previous Issue
Next Issue
Back to TopTop