Self-supervised Sim-to-Real Kinematics Reconstruction for Video-Based Assessment of Intraoperative Suturing Skills

Suturing technical skill scores are strong predictors of patient functional recovery following robot-assisted radical prostatectomy (RARP), but manual assessment of these skills is a time and resource-intensive process. By automating suturing skill scoring through computer vision methods, we can significantly reduce the burden on healthcare professionals and enhance the quality and quantity of educational feedbacks. Although automated skill assessment on simulated virtual reality (VR) environments have been promising, applying vision methods to live (‘real’) surgical videos has been challenging due to: 1) the lack of kinematic data from the da Vinci^® surgical system, a key source of information for determining the movement and trajectory of robotic manipulators and suturing needles, and 2) the lack of training data due to the labor-intensive task of segmenting and scoring individual stitches from live videos. To address these challenges, we developed a self-supervised pre-training paradigm whereby sim-to-real generalizable representations are learned without requiring any live kinematic annotations. Our model is based on a masked autoencoder (MAE), termed as LiveMAE. We augment live stitches with VR images during pre-training and require LiveMAE to reconstruct images from both domains while also predicting the corresponding kinematics. This process learns a visual-to-kinematic mapping that seeks to locate the positions and orientations of surgical manipulators and needles, deriving “kinematics” from live videos without requiring supervision. With an additional skill-specific finetuning step, LiveMAE surpasses supervised learning approaches across 6 technical skill assessments, ranging from 0.56–0.84 AUC (0.70–0.91 AUPRC), with particular improvements of 35.78% in AUC for wrist rotation skills and 8.7% for needle driving skills. Mean-squared error for test VR kinematics was as low as 0.045 for each element of the instrument poses. Our contributions provide the foundation to deliver personalized feedback to surgeons training in VR and performing live prostatectomy procedures.

This study is supported in part by the National Cancer Institute under Award Number 1RO1CA251579-01A1.

Authors and Affiliations

1. University of Southern California, Los Angeles, USA

   Loc Trinh, Tim Chu, Zijun Cui, Oscar Gomez & Yan Liu

2. Mimic Technologies Inc., Seattle, USA

   Anand Malpani

3. Department of Urology, Cedars-Sinai Medical Center, Los Angeles, USA

   Cherine Yang & Andrew Hung

4. Boston University Henry M. Goldman School of Dental Medicine, Boston, USA

   Istabraq Dalieh

5. Western University of Health Sciences, Pomona, USA

   Alvin Hui

Corresponding author

Correspondence to Andrew Hung .

Editors and Affiliations

1. Icahn School of Medicine, Mount Sinai, NYC, NY, USA, Tel Aviv University, Tel Aviv, Israel

   Hayit Greenspan

2. Emory University, Atlanta, GA, USA

   Anant Madabhushi

3. Queen’s University, Kingston, ON, Canada

   Parvin Mousavi

4. The University of British Columbia, Vancouver, BC, Canada

   Septimiu Salcudean

5. Yale University, New Haven, CT, USA

   James Duncan

6. IBM Research, San Jose, CA, USA

   Tanveer Syeda-Mahmood

7. Johns Hopkins University, Baltimore, MD, USA

   Russell Taylor

Reprints and permissions

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

Policies and ethics