Enhancing Surgical Team Collaboration and Situation Awareness through Multimodal Sensing
Pages 716 - 720
Abstract
Surgery, typically seen as the surgeon’s sole responsibility, requires a broader perspective acknowledging the vital roles of other operating room (OR) personnel. The interactions among team members are crucial for delivering quality care and depend on shared situation awareness. I propose a two-phase approach to design and evaluate a multimodal platform that monitors OR members, offering insights into surgical procedures. The first phase focuses on designing a data-collection platform, tailored to surgical constraints, to generate novel collaboration and situation-awareness metrics using synchronous recordings of the participants’ voices, positions, orientations, electrocardiograms, and respiration signals. The second phase concerns the creation of intuitive dashboards and visualizations, aiding surgeons in reviewing recorded surgery, identifying adverse events and contributing to proactive measures. This work aims to demonstrate an innovative approach to data collection and analysis, augmenting the surgical team’s capabilities. The multimodal platform has the potential to enhance collaboration, foster situation awareness, and ultimately mitigate surgical adverse events. This research sets the stage for a transformative shift in the OR, enabling a more holistic and inclusive perspective that recognizes that surgery is a team effort.
References
[1]
Adriana Arza, Jorge Mario Garzón-Rey, Jesús Lázaro, Eduardo Gil, Raul Lopez-Anton, Conchita de la Camara, Pablo Laguna, Raquel Bailon, and Jordi Aguiló. 2019. Measuring acute stress response through physiological signals: towards a quantitative assessment of stress. Medical & Biological Engineering & Computing 57, 1 (Jan. 2019), 271–287. https://doi.org/10.1007/s11517-018-1879-z
[2]
Hamed Asadi, Sara Monfared, Dimitrios I. Athanasiadis, Dimitrios Stefanidis, and Denny Yu. 2021. Continuous, integrated sensors for predicting fatigue during non-repetitive work: demonstration of technique in the operating room. Ergonomics 64, 9 (Sept. 2021), 1160–1173. https://doi.org/10.1080/00140139.2021.1909753 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/00140139.2021.1909753.
[3]
Dan Bohus, Sean Andrist, Ashley Feniello, Nick Saw, Mihai Jalobeanu, Patrick Sweeney, Anne Loomis Thompson, and Eric Horvitz. 2021. Platform for Situated Intelligence. https://doi.org/10.48550/arXiv.2103.15975 Number: arXiv:2103.15975 arXiv:2103.15975 [cs].
[4]
Francisco Maria Calisto, Nuno Nunes, and Jacinto C. Nascimento. 2022. Modeling adoption of intelligent agents in medical imaging. International Journal of Human-Computer Studies 168 (Dec. 2022), 102922. https://doi.org/10.1016/j.ijhcs.2022.102922
[5]
Jackie S. Cha and Denny Yu. 2022. Objective Measures of Surgeon Non-Technical Skills in Surgery: A Scoping Review. Human Factors 64, 1 (Feb. 2022), 42–73. https://doi.org/10.1177/0018720821995319
[6]
Robin Champseix, Laurent Ribiere, and Clément Le Couedic. 2021. A Python Package for Heart Rate Variability Analysis and Signal Preprocessing. 9, 1 (Oct. 2021), 28. https://doi.org/10.5334/jors.305 Number: 1 Publisher: Ubiquity Press.
[7]
Nancy J Cooke, Renée J Stout, and Eduardo Salas. 2017. A Knowledge Elicitation Approach to the Measurement of Team Situation Awareness. In Situational Awareness (1 ed.), Eduardo Salas (Ed.). Routledge, 157–182. https://doi.org/10.4324/9781315087924-10
[8]
R D Dias, M C Ngo-Howard, M T Boskovski, M A Zenati, and S J Yule. 2018. Systematic review of measurement tools to assess surgeons’ intraoperative cognitive workload. British Journal of Surgery 105, 5 (March 2018), 491–501. https://doi.org/10.1002/bjs.10795
[9]
Vanessa Echeverria, Roberto Martinez-Maldonado, and Simon Buckingham Shum. 2019. Towards Collaboration Translucence: Giving Meaning to Multimodal Group Data. (2019).
[10]
Mica R. Endsley. 1995. Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors 37, 1 (March 1995), 32–64. https://doi.org/10.1518/001872095779049543 Publisher: SAGE Publications Inc.
[11]
Cole Etherington, Aya Usama, Andrea M. Patey, Chantal Trudel, Antoine Przybylak-Brouillard, Justin Presseau, Jeremy M. Grimshaw, and Sylvain Boet. 2019. Exploring stakeholder perceptions around implementation of the Operating Room Black Box for patient safety research: a qualitative study using the theoretical domains framework. BMJ Open Quality 8, 3 (July 2019), e000686. https://doi.org/10.1136/bmjoq-2019-000686 Publisher: British Medical Journal Publishing Group Section: Original article.
[12]
Aimee K. Gardner, Matthew Kosemund, and Joseph Martinez. 2017. Examining the Feasibility and Predictive Validity of the SAGAT Tool to Assess Situation Awareness Among Medical Trainees. Simulation in Healthcare 12, 1 (Feb. 2017), 17. https://doi.org/10.1097/SIH.0000000000000181
[13]
Atul A. Gawande, Michael J. Zinner, David M. Studdert, and Troyen A. Brennan. 2003. Analysis of errors reported by surgeons at three teaching hospitals. Surgery 133, 6 (June 2003), 614–621. https://doi.org/10.1067/msy.2003.169
[14]
P. D. Grantcharov, T. Boillat, S. Elkabany, K. Wac, and H. Rivas. 2019. Acute mental stress and surgical performance: Acute mental stress and surgical performance. BJS Open 3, 1 (Feb. 2019), 119–125. https://doi.org/10.1002/bjs5.104
[15]
Amer Haffar, Chad A. Krueger, Graham S. Goh, and Jess H. Lonner. 2022. Total Knee Arthroplasty With Robotic Surgical Assistance Results in Less Physician Stress and Strain Than Conventional Methods. The Journal of Arthroplasty 37, 6 (June 2022), S193–S200. https://doi.org/10.1016/j.arth.2021.11.021
[16]
Nour Aldeen Jalal, Tamer Abdulbaki Alshirbaji, Bernhard Laufer, Paul D. Docherty, Thomas Neumuth, and Knut Moeller. 2023. Analysing multi-perspective patient-related data during laparoscopic gynaecology procedures. Scientific Reports 13, 1 (Jan. 2023), 1604. https://doi.org/10.1038/s41598-023-28652-7 Number: 1 Publisher: Nature Publishing Group.
[17]
Lauren Kennedy and Sarah Henrickson Parker. 2019. Biofeedback as a stress management tool: a systematic review. Cognition, Technology & Work 21, 2 (May 2019), 161–190. https://doi.org/10.1007/s10111-018-0487-x
[18]
Aurelien Lechappe, Mathieu Chollet, Jérôme Rigaud, and Caroline G.L. Cao. 2021. Assessment of Situation Awareness during Robotic Surgery using Multimodal Data. In Companion Publication of the 2020 International Conference on Multimodal Interaction(ICMI ’20 Companion). Association for Computing Machinery, New York, NY, USA, 412–416. https://doi.org/10.1145/3395035.3425205
[19]
Dominique Makowski, Tam Pham, Zen J. Lau, Jan C. Brammer, François Lespinasse, Hung Pham, Christopher Schölzel, and S. H. Annabel Chen. 2021. NeuroKit2: A Python toolbox for neurophysiological signal processing. Behavior Research Methods 53, 4 (Aug. 2021), 1689–1696. https://doi.org/10.3758/s13428-020-01516-y
[20]
P. Mascagni and N. Padoy. 2021. OR black box and surgical control tower: Recording and streaming data and analytics to improve surgical care. Journal of Visceral Surgery 158, 3, Supplement (June 2021), S18–S25. https://doi.org/10.1016/j.jviscsurg.2021.01.004
[21]
Hossein Mohamadipanah, Kenneth H. Perrone, Katherine Peterson, Jay Nathwani, Felix Huang, Anna Garren, Margaret Garren, Anna Witt, and Carla Pugh. 2020. Sensors and Psychomotor Metrics: A Unique Opportunity to Close the Gap on Surgical Processes and Outcomes. ACS Biomaterials Science & Engineering 6, 5 (May 2020), 2630–2640. https://doi.org/10.1021/acsbiomaterials.9b01019 Publisher: American Chemical Society.
[22]
Victoriano Montesinos, Fabio Dell’Agnola, Adriana Arza, Amir Aminifar, and David Atienza. 2019. Multi-Modal Acute Stress Recognition Using Off-the-Shelf Wearable Devices. In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2196–2201. https://doi.org/10.1109/EMBC.2019.8857130 ISSN: 1558-4615.
[23]
Katarzyna Powezka, Allan Pettipher, Apit Hemakom, Tricia Adjei, Pasha Normahani, Danilo P. Mandic, and Usman Jaffer. 2022. A Pilot Study of Heart Rate Variability Synchrony as a Marker of Intraoperative Surgical Teamwork and Its Correlation to the Length of Procedure. Sensors 22, 22 (Jan. 2022), 8998. https://doi.org/10.3390/s22228998 Number: 22 Publisher: Multidisciplinary Digital Publishing Institute.
[24]
Lauren Schiff, Ziv Tsafrir, Joelle Aoun, Andrew Taylor, Evan Theoharis, and David Eisenstein. 2016. Quality of Communication in Robotic Surgery and Surgical Outcomes. JSLS : Journal of the Society of Laparoendoscopic Surgeons 20, 3 (2016), e2016.00026. https://doi.org/10.4293/JSLS.2016.00026
[25]
Bertrand Schneider, Gahyun Sung, Edwin Chng, and Stephanie Yang. 2021. How Can High-Frequency Sensors Capture Collaboration? A Review of the Empirical Links between Multimodal Metrics and Collaborative Constructs. Sensors 21, 24 (Jan. 2021), 8185. https://doi.org/10.3390/s21248185 Number: 24 Publisher: Multidisciplinary Digital Publishing Institute.
[26]
Kevin Sexton, Amanda Johnson, Amanda Gotsch, Ahmed A. Hussein, Lora Cavuoto, and Khurshid A. Guru. 2018. Anticipation, teamwork and cognitive load: chasing efficiency during robot-assisted surgery. BMJ Quality & Safety 27, 2 (Feb. 2018), 148–154. https://doi.org/10.1136/bmjqs-2017-006701 Publisher: BMJ Publishing Group Ltd Section: Original research.
[27]
Venkatesh Sivaraman, Leigh A Bukowski, Joel Levin, Jeremy M. Kahn, and Adam Perer. 2023. Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems(CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–18. https://doi.org/10.1145/3544548.3581075
[28]
Kip Smith and P. A. Hancock. 1995. Situation Awareness Is Adaptive, Externally Directed Consciousness. Human Factors 37, 1 (March 1995), 137–148. https://doi.org/10.1518/001872095779049444 Publisher: SAGE Publications Inc.
[29]
Jacob O. Wobbrock and Julie A. Kientz. 2016. Research contributions in human-computer interaction. Interactions 23, 3 (April 2016), 38–44. https://doi.org/10.1145/2907069
[30]
Chuhao Wu, Jackie Cha, Jay Sulek, Chandru P. Sundaram, Juan Wachs, Robert W. Proctor, and Denny Yu. 2021. Sensor-based indicators of performance changes between sessions during robotic surgery training. Applied Ergonomics 90 (Jan. 2021), 103251. https://doi.org/10.1016/j.apergo.2020.103251
[31]
S Yule, R Flin, S Paterson-Brown, N Maran, and D Rowley. 2006. Development of a rating system for surgeons’ non-technical skills. Medical Education 40, 11 (2006), 1098–1104. https://doi.org/10.1111/j.1365-2929.2006.02610.x _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1365-2929.2006.02610.x.
[32]
S Yule, A Gupta, D Gazarian, A Geraghty, D S Smink, J Beard, T Sundt, G Youngson, C McIlhenny, and S Paterson-Brown. 2018. Construct and criterion validity testing of the Non-Technical Skills for Surgeons (NOTSS) behaviour assessment tool using videos of simulated operations. British Journal of Surgery 105, 6 (May 2018), 719–727. https://doi.org/10.1002/bjs.10779
[33]
Ting Zhang, Jing Yang, Nade Liang, Brandon J. Pitts, Kwaku O. Prakah-Asante, Reates Curry, Bradley S. Duerstock, Juan P. Wachs, and Denny Yu. 2020. Physiological Measurements of Situation Awareness: A Systematic Review. Human Factors: The Journal of the Human Factors and Ergonomics Society (Nov. 2020), 001872082096907. https://doi.org/10.1177/0018720820969071
[34]
Linxuan Zhao, Lixiang Yan, Dragan Gasevic, Samantha Dix, Hollie Jaggard, Rosie Wotherspoon, Riordan Alfredo, Xinyu Li, and Roberto Martinez-Maldonado. 2022. Modelling Co-located Team Communication from Voice Detection and Positioning Data in Healthcare Simulation. In LAK22: 12th International Learning Analytics and Knowledge Conference(LAK22). Association for Computing Machinery, New York, NY, USA, 370–380. https://doi.org/10.1145/3506860.3506935
[35]
Tian Zhou, Jackie S. Cha, Glebys Gonzalez, Juan P. Wachs, Chandru P. Sundaram, and Denny Yu. 2020. Multimodal Physiological Signals for Workload Prediction in Robot-assisted Surgery. ACM Transactions on Human-Robot Interaction 9, 2 (Feb. 2020), 1–26. https://doi.org/10.1145/3368589
Index Terms
- Enhancing Surgical Team Collaboration and Situation Awareness through Multimodal Sensing
Recommendations
Situation awareness in medical visualization to support surgical decision making
ECCE '10: Proceedings of the 28th Annual European Conference on Cognitive ErgonomicsMotivation -- To design information visualisation that in a decision support system for surgeons that aims to enhance situation awareness and hence improves performance.
Research approach -- A prototype of an Intra-operative Visualization System (IVS) ...
Self-improving situation awareness for human–robot-collaboration using intelligent Digital Twin
AbstractThe situation awareness, especially for collaborative robots, plays a crucial role when humans and machines work together in a human-centered, dynamic environment. Only when the humans understands how well the robot is aware of its environment can ...
Ontology-based situation awareness
The notions of ''situation'' and ''situation awareness'' have been formulated by many authors in various contexts. In this paper, we present a formalization of situations that is compatible with the interpretation of situation awareness in terms of ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2023
858 pages
ISBN:9798400700552
DOI:10.1145/3577190
Copyright © 2023 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 09 October 2023
Check for updates
Author Tags
Qualifiers
- Extended-abstract
- Research
- Refereed limited
Funding Sources
Conference
ICMI '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 453 of 1,080 submissions, 42%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 121Total Downloads
- Downloads (Last 12 months)69
- Downloads (Last 6 weeks)11
Reflects downloads up to 20 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format