[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Next Article in Journal
Mechanical Design, Manufacturing, and Testing of a Soft Pneumatic Actuator with a Reconfigurable Modular Reinforcement
Previous Article in Journal
Subtask-Based Usability Evaluation of Control Interfaces for Teleoperated Excavation Tasks
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Robotics
Volume 13
Issue 11
10.3390/robotics13110164
robotics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

HaptiScan: A Haptically-Enabled Robotic Ultrasound System for Remote Medical Diagnostics

by
Zoran Najdovski
1,*,
Siamak Pedrammehr
2,
Mohammad Reza Chalak Qazani
3,
Hamid Abdi
4,
Sameer Deshpande
5,
Taoming Liu
6,
James Mullins
1,
Michael Fielding
1,
Stephen Hilton
7 and
Houshyar Asadi
1
1
Institute for Intelligent Systems Research and Innovation, Deakin University, Waurn Ponds 3216, Australia
2
Faculty of Design, Tabriz Islamic Art University, Tabriz 5164736931, Iran
3
Faculty of Computing and Information Technology (FoCIT), Sohar University, Sohar 311, Oman
4
School of Engineering, Deakin University, Waurn Ponds 3216, Australia
5
ManuFutures, Deakin University, Waurn Ponds 3216, Australia
6
Superior Swift Technologies Pty Ltd., Melbourne 3122, Australia
7
Telstra Group Limited, Melbourne 3000, Australia
*
Author to whom correspondence should be addressed.
Robotics 2024, 13(11), 164; https://doi.org/10.3390/robotics13110164
Submission received: 9 September 2024 / Revised: 27 October 2024 / Accepted: 8 November 2024 / Published: 10 November 2024
(This article belongs to the Special Issue Development of Biomedical Robotics)
Browse Figures
Figure 1
<p>The graphical abstract representation of the proposed methodology in this research.</p> "> Figure 2
<p>(<b>a</b>) Haptically-Enabled Robotic Ultrasound Platform; (<b>b</b>) CAD model of the HaptiScan platform.</p> "> Figure 3
<p>The kinematics representation of Phantom Omni.</p> "> Figure 4
<p>Vectorial representation of Phantom Omni: (<b>a</b>) top view; (<b>b</b>) side view.</p> "> Figure 5
<p>UR5 robot model with the DH coordinate frames assignments.</p> "> Figure 6
<p>(<b>a</b>) Signostics Signos RT handheld ultrasound device [<a href="#B45-robotics-13-00164" class="html-bibr">45</a>], (<b>b</b>) ultrasound probe support mechanism with ATI Nano 17 sensor.</p> "> Figure 7
<p>UR5 robot model with the DH coordinate frames assignments.</p> "> Figure 8
<p>Teleoperation system scheme.</p> "> Figure 9
<p>The SimMechanics model of Phantom Omni.</p> "> Figure 10
<p>Time delay.</p> "> Figure 11
<p>Cartesian position and orientation of the slave manipulator.</p> "> Figure 12
<p>(<b>a</b>) Cartesian velocity of both manipulators; (<b>b</b>) Cartesian velocity error of the manipulators.</p> "> Figure 13
<p>(<b>a</b>) Joints’ angle and velocity of the master manipulator; (<b>b</b>) Joints’ angle and velocity of the slave manipulator.</p> "> Figure 14
<p>Force error observed during the teleoperation under varying time delays.</p> ">
Versions Notes

Abstract

:
Medical ultrasound is a widely used diagnostic imaging modality that provides real-time imaging at a relatively low cost. However, its widespread application is hindered by the need for expert operation, particularly in remote regional areas where trained sonographers are scarce. This paper presents the development of HaptiScan, a state-of-the-art telerobotic ultrasound system equipped with haptic feedback. The system utilizes a commercially available robotic manipulator, the UR5 robot from Universal Robots, integrated with a force/torque sensor and the Phantom Omni haptic device. This configuration enables skilled sonographers to remotely conduct ultrasound procedures via an internet connection, addressing both the geographic and ergonomic limitations faced in traditional sonography. Key innovative features of the system include real-time force feedback, ensuring that sonographers can precisely control the ultrasound probe from a remote location. The system is further enhanced by safety measures such as over-force sensing, patient discomfort monitoring, and emergency stop mechanisms. Quantitative indicators of the system’s performance include successful teleoperation over long distances with time delays, as demonstrated in simulations. These simulations validate the system’s control methodologies, showing stable performance with force feedback under varying time delays and distances. Additionally, the UR5 manipulator’s precision, kinematic, and dynamic models are mathematically formulated to optimize teleoperation. The results highlight the effectiveness of the proposed system in overcoming the technical challenges of remote ultrasound procedures, offering a viable solution for real-world telemedicine applications.

1. Introduction

Medical ultrasound is one of the crucial diagnostic imaging modalities that provide images in real-time. Compared to other primary imaging modalities, ultrasound offers several advantages: it has a substantially reduced cost, good image quality, greater transportability and portability, and does not involve harmful ionizing radiation [1]. Additionally, the 3D feature of ultrasound imaging has overcome several drawbacks that were evident in 2D imaging devices such as X-rays [2]. From a cost perspective, ultrasound systems are relatively affordable for most clinics and hospitals, even small ones in remote locations, making widespread use feasible. However, the barrier to general usability is the need for skilled personnel to acquire high-quality images, as the shortage of skilled and well-trained personnel is one of the most significant challenges preventing these facilities from utilizing these machines to serve nearby communities [3,4,5].
During the scan, sonographers must hold the ultrasound transducer firmly and move it within the field of interest on the patient’s body. They often need to adjust the probe’s position and angle its orientation to optimize the visualization of the target. Consequently, these actions can lead to unergonomic positioning for the sonographers. Additionally, they usually need to apply substantial force and torque to the patient’s body to get a clear ultrasound image. Burnett and Campbell-Kyureghyan [6] found that forces exceeding 50 N were sometimes required for scanning. The long-term effects of this repetitive work can cause occupational injuries, such as in the neck, shoulder, arm, and back, or lead to stress injuries for experienced sonographers [7].
In the past decade, the combination of robotics and teleoperation has made tele sonography an effective methodology for using ultrasound transducers to diagnose internal body injuries, diseases, and tumors from a remote distance [2,8,9,10,11]. This advancement means that patients do not need to travel long distances to receive an ultrasound scan; instead, they can access these services in nearby small clinics or communities, performed by an experienced and skilled sonographer located anywhere in the world. The results of their scans can be analyzed and processed at a single location. This approach also enhances operator comfort and removes the need for an ultrasound professional to be present at the examination place [9,12]. Additionally, the accuracy of ultrasound images obtained through robotic scanning is comparable to that of images acquired through manual scanning [13]. Real-time recognition is also another benefit of virtual scanning when it is AI-powered [14,15].
Telerobotic ultrasound systems typically consist of master and slave systems. The sonographer operates the master manipulator, such as a joystick device, mockup probe, or a haptic device, while the slave manipulator beside the patient in a remote location precisely follows the motion commands sent by the operator, mimicking the operator’s movements. Telerobotic ultrasound systems allow the operator to perform ultrasound procedures remotely, while haptically enabled robotic systems help improve the operator’s accuracy and control. Therefore, there is a need to develop precise control models in these systems to effectively manage time delays and ensure smooth, responsive movements of the ultrasound probe. This requires a comprehensive understanding of the system’s dynamic behavior. Investigating methods to achieve this understanding is a critical research issue, addressing a gap in the current literature.
This research presents the development of a haptically-enabled robotic system for remote ultrasound procedures (HaptiScan), utilizing a robotic manipulator and a haptic interface. Figure 1 illustrates the HaptiScan system’s architecture, showing the setup in the patient room and the doctor’s office. The diagram highlights key components like the ATI Nano 17 sensor, Phantom Omni haptic device, and safety features including patient discomfort monitoring and emergency stops, connected via a communication channel with delay. The HaptiScan integration allows operators to remotely perceive the interaction forces between the ultrasound probe and patient. The system employs the UR5 robot from Universal Robots, a commercially available manipulator certified as a collaborative robot compliant with industry safety standards, which is teleoperated over the internet via a commercially available haptic interface (Phantom Omni). A portable ultrasound probe and a force/torque sensor (F/T sensor) are integrated into the UR5′s end-effector. The F/T sensor measures the forces of interaction between the ultrasound probe and the patient’s abdomen. This force data, along with ultrasound information, is transmitted to the operators over the internet, assisting them in performing the teleoperated procedure while ensuring the manipulator operates within a predefined safe workspace. Additionally, stereovision is provided to enhance the operator’s perception of the manipulator and its environment.
Safety is a primary concern in this system. Therefore, safety algorithms are implemented to ensure that the slave manipulator interacts safely and physically with patients within a limited workspace. The HaptiScan system incorporates several safety features: patient discomfort monitoring, over-force sensing and response, motion control and workspace limiting, end-effector safety control, and emergency stops.
Furthermore, the analytical modeling of both the slave and master manipulators, including kinematic and dynamic formulations, is presented to provide a comprehensive understanding of the system’s dynamic behavior. The teleoperation control methodologies, such as control in joint space, bilateral control in Cartesian space, and a Jacobian-based control method in joint space, are mathematically formulated and implemented to optimize the control of ultrasound procedures over the internet. The proposed teleoperation control methodologies performance, with force feedback over the internet, is demonstrated through simulations using MATLAB Simulink’s SimMechanics models of the UR5 manipulator and the Phantom Omni. The simulation results illustrate the effectiveness of these methodologies in managing long-distance and varying time-delay conditions. The HaptiScan system represents a significant advancement in telerobotic ultrasound technology, integrating real-time force feedback, advanced safety mechanisms, and robust teleoperation control. These features address limitations observed in existing systems, such as limited safety features, lack of force feedback, and challenges in handling varying latency.
Related studies in the literature regarding telerobotic ultrasound systems are discussed in Section 2. The system design and integration are described in Section 3, including a review of the kinematic and dynamic models of the slave and master manipulators. The design of the control methods in joint and Cartesian spaces, and the teleoperation control system are presented in Section 4. The simulation study and results for verifying the proposed system are presented in Section 5, followed by the discussion and conclusions in Section 6.

2. Telerobotic Ultrasound System Literature Review

The research on tele sonography (tele-Echography) has gained significant interest from numerous researchers over the past decade. An overview and development of robotic ultrasound systems in medicine and telerobotic ultrasound systems are summarized by Priester et al. [16], Monfaredi et al. [17], Adams et al. [18], Jiang et al. [19], Hidalgo et al. [20], Shi and Rosario [21]. Meanwhile, several clinical studies have trialed telerobotic ultrasound systems in medicine. Swerdlow et al. [1] summarized these state-of-the-art research works and their potential medical applications. For example, Martinelli et al. [22] examined the abdominal aortas of 58 patients and detected a total of eight aneurysms using both telerobotic ultrasound and conventional ultrasound. Arbeille et al. [3,23,24] applied their telerobotic ultrasound system to successfully identify all organ systems in the abdominal area and for obstetric and cardiac applications. Zhang et al. [25] evaluated a 5G-based telerobotic ultrasound for thyroid examinations in a rural setting, demonstrating its feasibility and effectiveness compared to conventional ultrasound methods. He et al. [26] proposed a potential solution for delivering quality ultrasound services for breast examinations using a telerobotic ultrasound system in areas lacking experienced sonologists.
Initially, researchers focused on developing the mechanical design of lightweight, portable, and safe robotic manipulators for the slave manipulator [8,10,11,12,27,28,29,30,31,32]. Salcudean et al. [10] developed a six degrees-of-freedom (DOF) counterbalanced robot arm using a 4-bar parallel linkage structure for diagnostic ultrasound. A shared-control strategy allowed the robot to automatically adjust its configuration in response to motion changes from the sonographers. Vilchis et al. [11] presented a cable-driven and lightweight tele-echography robotic system (TER) working in master–slave mode. A cable-driven compliant structure with a wrist holding the ultrasound probe is connected by straps to four direct current motors. This non-rigid structure is located directly on the patient’s body and can perform translation movements through strap motions, with the wrist capable of orienting the probe and translating it along the probe axis via a serial four DOF structure.
Su et al. [16] proposed a fully autonomous robotic ultrasound system (FARUS) that integrates the roles of sonographer and radiologist into a single unit for scanning, employing reinforcement learning, and deep learning for optimal probe orientation and real-time segmentation. Mitsuishi et al. [27] presented a master–slave remote ultrasound system with force feedback. The slave manipulator was highly rigid, allowing for precise configuration control of the ultrasound probe. They implemented impedance control for force reflection. However, although these systems could control the slave manipulators to move the probe precisely, safety issues of these custom mechanical systems were not prioritized and remain a significant concern. The high-rigid slave manipulator [27] could potentially hurt patients in a shared workspace.
In recent years, researchers have used commercially available collaborative robots, designed for physical interaction with humans in a shared workspace, as slave manipulators. Some researchers have tried using KUKA LWR iiwa robots (KUKA Deutschland GmbH, Germany) for robotized ultrasound screening [32,33]. Kuhlemann et al. [33] used the force-sensitive lightweight KUKA LWR iiwa robot with probe contact pressure detection for non-invasive 4D live tracking. Virga et al. [32] developed a system using a KUKA robot for automating robotic ultrasound acquisitions in the procedure of abdominal aortic aneurysms. Other research groups have used UR5 robots as the slave manipulator in teleoperated robotic ultrasound systems [34,35]. Mathiassen et al. [34] developed a teleoperated robotic ultrasound using a collaborative robot with force feedback, but they did not test the teleoperation mode over network communication or assess the system’s usability when time delays over communication are present. Finocchi et al. [35] developed a collaborative force control algorithm with force application assistance on a UR5 robot carrying an ultrasound probe. Their proposed method can reduce the grip force exerted by sonographers when performing an ultrasound scan on a specific target. Ning et al. [36,37] also implemented a force-guided control method on a UR5 robot carrying an ultrasound probe. In their research, a reinforcement learning (RL) agent was constructed to adjust the ultrasound probe’s pose during the imaging process without needing prior environmental knowledge or space calibration. While systems like those developed by Mathiassen et al. [34] and Vilchis et al. [11] have contributed significantly to the field of telerobotic ultrasound, they lack comprehensive safety features and precise force feedback, which are critical for effective teleoperation. The HaptiScan system, by contrast, provides enhanced safety, real-time force feedback, and improved latency handling, setting it apart from existing solutions.

3. System Design and Integration

The HaptiScan platform, shown in Figure 2a, is developed to accommodate a UR5 robot as the slave manipulator, its controller, (three active and three passive) Phantom Omni haptic device (formerly SensAble Technologies, Woburn, MA, USA) as the master manipulator, and other essential components, most importantly, the patient. This platform is designed to be easily movable, allowing it to be transported to locations requiring emergency medical diagnostic procedures and controlled over the internet.
The following list describes each major system component:
  • Patient bed;
  • Force compliant probe, consisting of an ultrasound probe and a force/torque sensor;
  • Emergency stops (eStop);
  • UR5 robot manipulator;
  • 3D stereo camera;
  • Fold-up bed wings for easy transport;
  • Control cabinet, holding the robot controller, networking gear, local PC, and ATI Net F/T;
  • Adjustable and lockable wheels.

3.1. Master Manipulator

Phantom Omni is a six DOF haptic device, as shown in Figure 3. Its first three joints are actively actuated; however, the others are non-actuated and allow for passive wrist motion of a stylus. The mathematical modeling of the Phantom Omni, including forward and inverse kinematics, the Jacobian matrix, and dynamics, is developed in the following sections. Similar derivations were made by Cavusoglu et al. [38] and Jarillo-Silva et al. [39].
Forward kinematics:
The vectorial representation of the Phantom Omni is shown in Figure 3. Specifically, θ1, θ2 and θ3 are the joint angles for revolute joints 1, 2, and 3. L1 and L2 are the length of the links. L3 and L4 represent the workspace transformation vertical and horizontal offset between the end effector origin and the first joint.
The transformation matrix is computed after deriving its Denavit–Hartenberg (DH) parameters [40]. Thus, the end effector position, can be expressed as follows:
x m y m z m = − s θ 1 L 1 c θ 2 + L 2 s θ 3 L 1 s θ 2 − L 2 c θ 3 + L 3 L 1 c θ 1 c θ 2 − L 2 c θ 1 s θ 3 − L 4
where L1 = L2 = 133.35 mm, L3 = 23.35 mm, L4 = 168.35 mm. They are obtained from the physical parameter values of the device. It also should be noted that in the entire text of this paper, s and c represent sin and cos, respectively.
The Jacobian matrix J can also be obtained from (1), and expressed as follows:
J = − c θ 1 L 1 c θ 2 + L 2 s θ 3 L 1 s θ 1 s θ 2 − L 2 s θ 1 c θ 3 0 L 1 c θ 2 L 2 s θ 3 − L 1 s θ 1 c θ 2 − L 2 s θ 1 s θ 3 − L 1 c θ 1 s θ 2 L 2 c θ 1 c θ 2
Inverse kinematics:
The inverse kinematics problem of the Omni device can be solved by using a geometric approach given a desired end-effector position pm.
Figure 4a illustrates the top view of the Phantom Omni and thus θ1 can be computed by inspection as follows:
θ 1 = − tan − 1 ( x m ,   z m + L 4 )
Figure 4b also shows the side view, including the rotation motions of joint 2 and 3. Thus θ2 and θ3 can be expressed as follows:
θ 2 = γ + β
θ 3 = θ 2 + α − π 2
where α is the angle between two links, β is the angle between end effector and z axis.
Considering [ x m   x m − L 3   z m + L 4 ] T as the end point of the link 2, R and r can respectively be defined as follows:
R = x m 2 + z m + L 4 2
r = x m 2 + z m + L 4 2 + y m − L 3 2
Therefore, according to Figure 4b, the angles α, β, and γ can be obtained as follows:
α = cos − 1 L 1 2 + r 2 − L 2 2 2 L 1 L 2
β = tan − 1 2 y m − L 3 ,   R
γ = cos − 1 L 1 2 + r 2 − L 2 2 2 L 1 r
Dynamic model:
The Lagrange method is implemented to derive the dynamical equations of the robotic manipulators. The Lagrangian function L = K−P is based on the kinetic and potential energies, K and P. Thus, the Lagrangian differential equation can be written as follows:
M q q ¨ + C q , q ˙ q ˙ + G q = u
in which M q , C q , q ˙ and G q , respectively, denote the symmetric positive definite mass matrix, the Coriolis matrix including centrifugal terms, and the gravity vector. u is the input vector. Specifically for the Phantom Omni M q , C q , q ˙ , and G q can respectively be expressed as follows:
M ( q ) = k 1 + k 2 c 2 θ 2 + k 3 c 2 θ 3 + k 4 c θ 2 s θ 3 k 5 s θ 2 0 k 5 s θ 2 k 6 − 1 2 k 4 s θ 2 − θ 3 0 − 1 2 k 4 s θ 2 − θ 3 k 7
C q , q ˙ = − k 2 θ ˙ 2 s 2 θ 2 − k 3 θ ˙ 3 s 2 θ 3 − 1 2 k 4 θ ˙ 2 s θ 2 s θ 3 1 2 k 4 θ ˙ 3 c θ 2 c θ 3 − k 2 θ ˙ 1 s 2 θ 2 + k 5 θ ˙ 2 c θ 2 − 1 2 k 4 θ ˙ 1 s θ 2 s θ 3 − k 3 θ ˙ 1 s 2 θ 3 + 1 2 k 4 θ ˙ 1 c θ 2 c θ 3 k 2 θ ˙ 1 s 2 θ 2 + 1 2 k 4 θ ˙ 1 s θ 2 s θ 3 0 1 2 k 4 θ ˙ 3 c θ 2 − θ 3 k 3 θ ˙ 1 s 2 θ 3 + 1 2 k 4 θ ˙ 1 c θ 2 c θ 3 − 1 2 k 4 θ ˙ 2 c θ 2 − θ 3 0
G q = 0 k 2 c θ 2 + k 10 s θ 2 − π 2 k 9 s θ 3

3.2. Slave Manipulator

The UR5 robot is a well-known and widely used robotic manipulator in universities and industries. It has six DOF resulting from its six revolute joints. This research has investigated the kinematics and dynamics of the UR5 manipulator as presented in [41]. A comprehensive MATLAB model for UR5, including an accuracy evaluation of the models, was developed. Similar models of the UR5 have been developed by Hawkins [42] and Kufieta [43].
Forward kinematics:
The forward kinematics computes the transformation relationship from the robot base to the end effector based on the joint angle of each joint. The joints’ variables are regarded as system’s states, particularly in the context of control system design. The states for the UR5 are as follows:
q = θ s 1 , θ s 2 , θ s 3 , θ s 4 θ s 5 , θ s 6 T
where θ s i (i = 1 to 6) are respectively represented as follows: roll motion for the first joint, pitch motion for the second joint, pitch motion for the third joint, pitch motion for the fourth joint, roll motion for the fifth joint, and roll motion for the sixth joint; and their first- and second-time derivatives would be the generalized velocities and accelerations, respectively.
In this paper, the DH parameters are used to derive the forward kinematics of the UR5 [41]. The coordinate frame for each joint of this robot is illustrated in Figure 5.
To make the calculation of the DH parameters easier, some frames, like frames 1 and 2, share the same origin. The axis xi (i = 1 to 6) of each frame is directed vertically outward from the paper. The z axis of each frame indicates the rotational axis direction for the related joint. Put differently, θ s i represents the rotation about axis zi. All coordinate frames obey the right-hand rule. The derived DH parameters of the UR5 are listed in Table 1.
T i i − 1 is the general transformation matrix between link i−1 and link i, and using each set of DH parameters can be written as follows [41]:
T i i − 1 = c θ s i − s θ s i 0 α i − 1 s θ s i c α i − 1 c θ s i c α i − 1 − s α i − 1 − s α i − 1 d i s θ s i s α i − 1 c θ s i s α i − 1 c α i − 1 c α i − 1 d i 0 0 0 1
Additionally, T n 0 can be obtained by multiplying the transformation matrices between each pair of consecutive coordinate frames from the base to the end effector as follows:
T n 0 = T 1 0 × ⋯ T i i − 1 × T i + 1 i ⋯ × T n n − 1
For the UR5 robot, the representation of the T 6 0 matrix using the DH parameters specified in Table 1, can be obtained as follows:
T 6 0 = R 6 0 P 6 0 0 1
where the rotation matrix R 6 0 computes the end effector orientation angles with respect to the base frame, and the vector P 6 0 indicates the end effector position represented by the base coordinate frame.
Inverse kinematics:
The inverse kinematics of the UR5 robot has been investigated in [41] using a geometric approach.
The first step is to calculate θ s 1 using the position of the 5th joint as follows:
θ s 1 = α 1 + α 2 + π 2 = tan − 1 2 ( P 5 y 0 , P 5 x 0 ) ± cos − 1 d 4 R + π 2
There are generally two possible solutions for θ s 1 , corresponding to configurations where the shoulder is left or right.
Given the value of θ s 1 , θ s 5 can be obtained as follows:
θ s 5 = ± cos − 1 P x s 1 − P y c 1 − d 4 d 6
There are also two solutions for θ s 5 while the wrist is in/down or out/up.
The 6th joint is as follows:
θ 6 = tan − 1 2 y y c 1 − y x s 1 s 5 , x x c 1 − x y s 1 s 5
where s i represent the joint position in slave manipulator. One case is when s 5 = 0. In this scenario, joints 2, 3, 4, and 6 are parallel and coplanar, meaning the directions of their rotation axes are all the same. Consequently, the solution becomes undetermined.
The last three joints can be solved as an RRR planar robot, offering two potential configurations: either elbow up or down. Sometimes, there are no solutions when the distance to the 4th joint surpasses α 2 + α 3 , or is less than α 2 − α 3 . A singularity occurs when α 2 = α 3 and θ s 3 = π .
Dynamic model:
The dynamic equations for UR5 robot follow (11). To complete the dynamic model, M q , C q , q ˙ , and G q for UR5 robot can respectively be expressed as follows [44]:
M q = ∑ i = 1 n m i J v i T q J v i q + J ω i T q R i q I i R i T q J ω i q
C i , j q , q ˙ = ∑ i = 1 n 1 2 ∂ m i j ∂ q k + ∂ m i k ∂ q j − ∂ m k j ∂ q i   q ˙ k
G i q = ∂ P ∂ q i
where I i is the inertia tensor for the ith link. J ω i and J v i are respectively the angular and linear components of the Jacobian matrix J i . Then the robotic arm’s state-space equations would take the following form:
q ¨ = M − 1 q u − C q , q ˙ q ˙ − G q
M q is invertible. However, near singularities, such as at the workspace boundaries, its values can become very small, approaching singularity.

3.3. Force Compliant Ultrasound Probe

The Signostics Signos RT handheld ultrasound device [45], as shown in Figure 6a, is chosen for this project. The functions of this probe can be controlled by software, enabling the sonographer to directly modify and edit sonography functions at the teleoperation control station in real-time. This device allows for USB interfacing with a computer and supports device control through a software library interface.
Figure 6b shows the CAD design of the ultrasound probe support system that allows the UR5 robot to carry the handheld ultrasound probe. The ultrasound scan procedure requires the probe to maintain direct contact with patients under sufficiently large normal contact force for a clear scan. To prevent harm to patients from the rigid motions of the robotic arm and to ensure stable and consistent compliant contact, an ATI Nano 17 F/T sensor, Apex, NC, USA, as shown in Figure 6b, is mounted between the UR5 manipulator and the probe. This setup allows for accurate force measurement, enabling sonographers to remotely sense the real force feedback of the direct contact.

3.4. Graphical User Interface

Figure 7 presents the graphical user interface (GUI) developed to interface with the HaptiScan system. This interface provides the operator/sonographer with the ability to control the robotic manipulator and monitor various aspects of the system, such as applied forces, patient discomfort, and system connection status.
Below is a detailed list of the GUI features:
  • Tele-operation Start/Stop button;
  • Robot manipulator speed control slider;
  • Home button: to send the robot manipulator to the start position (away from the patient);
  • Run Mode button: activates the robot controller into a run state after being in a freedrive mode;
  • Freedrive button: activates freedrive mode, which sets the robot into a back-drivable state;
  • Stereo Vision button: triggers the Skype connection with the stereovision cameras at the robot end;
  • Robot manipulator shut down button;
  • Force levels graph: a real-time scrolling graph of force levels applied to the patient with the probe;
  • Log window: indicates the current state of the program and connections;
  • Force level bar graph: indicates force levels applied to the patient with the probe;
  • Discomfort level bar graph: indicates the level of discomfort from the patient through the hand interface;
  • Ultrasound probe data window: presents a real-time image stream of the ultrasound probe data;
  • Ultrasound probe Start/Stop button;
  • Ultrasound probe Gain control;
  • Ultrasound probe Depth control;
  • System status feedback window: provides robot mode status, state of the emergency stops, and network connection state.

3.5. Safety

UR5 is a commercial system built to be operational and safe to work with, in and around humans. In this project, several safety systems have been implemented to enhance the safety of the manipulator during teleoperation tasks.
The first method involves monitoring whether the normal force exerted by the ultrasound probe to the patient exceeds a predefined over-force threshold using data from the ATI Nano 17 F/T sensor. The second method involves checking the position commands from the operator to ensure they are within the predefined safe workspace. If the position command is within the safe workspace, the manipulator is controlled to move to that position. If the position command is outside the safe workspace, the manipulator is restricted to move only up to the boundary of the safe workspace. Additionally, the normal contact force exerted by the operator to the patient is continuously monitored. If this force exceeds the predefined threshold, a dynamic and adaptive motion boundary is constructed based on the interaction force between the robot and any object within the boundaries. This boundary limits the robot’s motion while it is in contact with a patient. Finally, a series of emergency stops are assembled around the robotic system to halt the robot and engage a soft E-STOP. This provides patients with some degree of control, ensuring they feel more relieved and safer, with the ability to enable the manipulator to free-drive mode and move it out of the way safely.

4. Teleoperation Control System Design

In a teleoperated robotic system, the operator interacts with the master manipulator and exerts force on it. Consequently, the master manipulator is displaced, and simultaneously, the slave manipulator mimics this displacement at the slave side. Meanwhile, information including force, position, and velocity of both manipulators is transmitted via the communication channel. As a result, the motion of the slave manipulator is influenced by this information and the environment. Figure 8 illustrates the scheme of the teleoperation system.
The dynamics of the master/slave systems are respectively described as follows:
M m q m q ¨ m + C m q m , q ˙ m q ˙ m + G m q m = u m + f h + f ^ e
M s q s q ¨ s + C s q s , q ˙ s q ˙ s + G s q s = u s + f e
where fh is the force imposed by the human on the master, fe represents the force applied on the slave by the environment, and f ^ e denotes the force exerted to the master manipulator’s motors reflecting the contact force experienced by the slave side from the environment.
During the transmission of information between both manipulators via the communication channel, communication delays are always present. For example, the position signal with a time delay on the slave side can be considered as follows:
q ^ m t = q m t − τ f t
In addition, the contact force from the environment is relayed from the slave to the master through the communication channel, introducing a communication delay. This contact force, sent to the master manipulator’s motors, can be expressed as follows:
f ^ e t = f e t − τ b t
When teleoperation is conducted over the Internet, the communication delays τ f t and τ b t could be highly irregular, creating additional challenges for the stability analysis of the teleoperator system.

4.1. Control Scheme in Joint Space

The control scheme of the force-reflecting teleoperation system includes local controllers on both the slave and master sides. The control equation of the master robot is as follows [46]:
u m = − M m q m Λ m q ˙ m − C m q m , q ˙ m Λ m q m + G m q m − K m q ˙ m + Λ m q m
in which K m and Λ m are symmetric positive definite matrices.
The control equation of the slave manipulator is as follows [46]:
u s = − M s q s ξ ˙ 2 + Λ s ξ ˙ 1 − q ˙ s + C s q s , q ˙ s ξ 2 + Λ s ξ 1 − q s + G s q s − K s q ˙ s − ξ 2 + Λ s q s − ξ 1
where ξ ˙ 1 and ξ ˙ 2 are subject to the following:
ξ ˙ 1 = ξ 2 + ω α 1 q ^ m − ξ 1
ξ ˙ 2 = ω 2 α 0 q ^ m − ξ 1
K s and Λ s are also symmetric positive matrices. α 0 and α 1 are positive constants, and ω > 0 is a constant.

4.2. Bilateral Control Scheme in Cartesian Space

In teleoperation systems, there are instances where the master and the slave possess different kinematics. In such cases, synchronization through joint angles is not possible, limiting bilateral control to Cartesian space only.
The end-effector velocities in Cartesian space are explained as X ˙ i = x ˙ i T ω i T T , which are related to the joint velocities q ˙ i through the Jacobian matrix J i q i , i.e.,:
X ˙ i = J i q i q ˙ i           and           i = m ,   s
For six DOF manipulators, the end effector position and orientation in Cartesian space are explained as x i ∈     R 3 and R i ∈ S O 3 [46]. The orientation error of the end effector between the master and slave at time t is defined as follows:
Δ R t = R s T t R m t
and thus, the time derivative of orientation error is as follows:
d d t Δ R t = Δ R t ω m − Δ R T t ω s
The goal is to achieve accurate tracking of the slave end effector to match the position and orientation of the master end effector. Additionally, the tracking errors must converge consistently, irrespective of the time delay. Simultaneously, the operator at the master side should perceive the forces exerted on the slave end effector.
Further analysis is based on the following assumptions:
  • Environment interaction and human dynamics are classified within a general passive systems class, i.e., for all t ≥ 0 ,     ∃     σ m ,     σ s     ∈     R + ,   s.t.
∫ 0 t X ˙ m T f h d δ ≥ − σ m ,             ∫ 0 t X ˙ s T f e d δ ≥ − σ m
2.
For simplicity in the analysis, gravitational forces are disregarded. Consequently, the dynamics equations are reformulated as follows:
M m q m q ¨ m + C m q m , q ˙ m q ˙ m = τ m − J m T q m f h
M s q s q ¨ s + C s q s , q ˙ s q ˙ s = τ s − J s T q s f e
3.
The time delays have an upper bound, i.e., for all t ≥ 0 ,     ∃     T f M ,     T b M   ∈     R < ∞ ,   s . t .     0 ≤ T f t ≤ T f M ,     0 ≤ T b t ≤ T b M .
4.
The time derivatives of time delays are limited. Particularly d T f t d t ≤ 1 and d T b t d t ≤ 1 .
The PD-like bilateral controller is written as follows:
τ m = J m T K d ζ b x ˙ s t − T b − x ˙ m ζ b R m T R s t − T b ω s t − T b ω m + K p x s t − T b − x m q ˜ R m T R s t − T b − D m d a m p X ˙ m
τ s = J s T K d ζ f x ˙ m t − T f − x ˙ s ζ f R s T R m t − T f ω m t − T f ω s + K p x m t − T f − x s q ˜ R s T R m t − T f − D s d a m p X ˙ s
where K d = d i a g k d x I k d ω I is a block diagonal matrix, I is the identity matrix, and k d x I and k d ω I are both positive; K p = d i a g k p x I k p ω I is also a block diagonal matrix with k p x > 0 and k p ω > 0 ; D m d a m p = d i a g d m x I d m ω I > 0 and D s d a m p = d i a g d s x I d s ω I > 0 are local damping matrices; ζ f and ζ b are dissipation factors, defined as ζ f 2 = 1 − T ˙ f t and ζ b 2 = 1 − T ˙ b t . 4 ensures the existence of ζ f and ζ b .

4.3. Jacobian-Based Control Scheme in Joint Space

To achieve satisfactory performance, a control strategy combines the control method in joint space with the bilateral control method in Cartesian space. The control strategy operates in joint space, utilizing the Cartesian position and velocity of the master robot to control the slave robot.
The primary objective is for the slave manipulator to accurately replicate the differential configuration of the master manipulator. This allows the slave manipulator to move correctly within the specified area. Jacobian matrix links Cartesian velocities to joint velocities. Therefore, changes in Cartesian space can be translated into corresponding changes in joint space. Consequently, the desired vector of joint velocities for the slave side q ˙ s * is determined by multiplying the pseudoinverse of the Jacobian J s + q s with the desired vector of Cartesian velocities X ˙ m which represents the command vector from master to slave, i.e.,:
q ˙ s * = J s + q s X ˙ m
q s * = ∫ q ˙ s *
Then, the control method in joint-space (30) to (33) is applied.
The singularity of the slave robot is a significant concern that could lead to instability. To address this issue, the damped least-squares inverse method is suggested to minimize both the solution error in joint velocities and the joint velocities magnitude [47,48]. The joint velocities are determined as follows:
J T J + λ 2 I q ˙ = J T ν
where q ˙ = J + ν . J + = J T J J T + λ 2 I − 1 is also the damped pseudoinverse Jacobian. The damping factor λ is a scalar that defines the relation between joint velocity and error. Additionally, the haptic feedback could give valuable information regarding the slave robot’s kinematic performance.
In this study, we have employed velocity control as opposed to position control to achieve smoother and safer teleoperation. This approach minimizes oscillations and ensures better responsiveness under varying network delays, making it suitable for remote ultrasound applications.

5. Simulation Model and Results

5.1. SimMechanics Model

SimMechanics models for the Phantom Omni and UR5 robot have been developed in the Mechanics Explorer of MATLAB Simulink to verify the proposed models and control methods.
Figure 9 illustrates the overall model of the Phantom Omni. Blocks such as Link1 and Base compute the transformation relations between the input and output.
The block representing the end-effector measures the configuration of the Phantom’s end effector using the required sensors.
The SimMechanics blocks for the dynamics of the UR5 have also been designed in MATLAB Simulink [41]. There are three types of blocks: Base, Joints, and Links. The Base blocks define the system’s environment (world frame) and its properties, including constants like gravity and the robot base configuration. The Joints block has two inputs: one for the SimMechanics physical connection between manipulator parts and the other for the torque signal generated by the controller. Two measured signals (position and velocity) partially represent the system’s states and are supplied to the controller. The Joints block output is connected to the Link block. The Link blocks contain blocks for rigid transformation and inertia properties specific to each link. Each Link block is connected to its corresponding joint and to the preceding Joint block via input ports numbered accordingly.

5.2. Comprehensive Simulation and Results

A comprehensive simulation has been conducted to demonstrate the efficacy and functionality of the proposed method. A sinusoidal function is employed as the desired velocity for the master, and a varying time-delay over a long distance is applied to test the controller robustness.
Considering the varying time-delay, depicted in Figure 10, non-constant Cartesian velocity (or position) for the slave robot is generated to respond to the master motion (Figure 11). As expected, sinusoidal behavior is observed in certain directions, while the responses remain stable.
In this simulation, a network latency of approximately 10 ms, as shown in Figure 10, was used to evaluate the performance of the teleoperation control methodologies. This latency reflects the performance of the system in an ideal, low-latency network environment which is increasingly being utilized in modern telemedicine settings with an established Quality of Service. These networks provide high bandwidth and minimal jitter, ensuring stable communication over long distances.
The decision to use a 10 ms latency assumed of a high-performance network that can support real-time teleoperation tasks. For the purposes of this study, we focused on validating the control methodology in conditions that would provide optimal system performance, demonstrating its robustness under stable network conditions with minimal delays.
Figure 12a displays the Cartesian velocity of both manipulators. The velocity errors converge to zero, as illustrated in Figure 12b.
Additionally, the angles and velocities of the joints of both manipulators are presented in Figure 13. Maintaining closed coupling between the environment and the operator in a teleoperation system is crucial. Transparency of the system involves two main aspects:
Firstly, ensuring position tracking between the slave and master mechanisms. The comparison of Cartesian velocities of both manipulators shown in Figure 12 indicates good velocity tracking in Cartesian space, despite a slight initial lag.
Secondly, accurately display the environment force exerted from the slave to the master. Figure 14 illustrates the force error observed during the teleoperation under varying time delays. While the maximum error reaches 2 N, this falls within the safety margin set for the system, considering the interaction dynamics of soft tissues and teleoperation delay variability. This error level does not compromise the quality of imaging or patient safety, as the control system is designed to prevent over-force conditions.
To provide a clearer perspective of the advancements introduced by the HaptiScan system, Table 2 presents a comparison with other leading telerobotic ultrasound systems, demonstrating superior performance in force feedback, safety measures, and latency handling.
Sometimes, adjustments like scaling up or down the position mapping between the slave and master or applying rate control are necessary for transparency, represented by mechanical impedance transmitted between the environment and the operator. Desired impedance at the slave side needs to be created, where bilateral impedance matching comprises operator impedance matching and environment impedance matching. However, transparency and stability are conflicting issues. Balancing optimal transparency of the system with stability is a key focus for future work.
Time delay poses significant challenges in teleoperation systems, affecting stability and performance. Long-distance and time-varying time delays (Figure 10) were added in the experiment, revealing robust and stable performance of the proposed bilateral controller. Latency has been considered in the teleoperation control system design. Extending teleoperation control to include packet loss and unknown variable delays is possible, requiring the project’s extension to the next phase with a primary focus on the stability of the system with optimal transparency.

6. Conclusions

This paper presents the development and implementation of the HaptiScan system, designed for remote ultrasound procedures. By integrating a UR5 robotic manipulator from Universal Robots and a Phantom Omni haptic interface, operators can remotely sense interaction forces between the ultrasound probe and the patient. This system facilitates remote ultrasound examinations and enhances operator perception and control through force feedback and stereovision capabilities.
Safety was a paramount concern in the design. To address this, several safety features were incorporated, including patient discomfort monitoring, over-force sensing and response, motion control, workspace limiting, end-effector safety controls, and emergency stop mechanisms. These features ensure that the system operates within safe parameters, protecting both the patient and the equipment.
This research included comprehensive mathematical modeling of both the slave and master manipulators. Detailed kinematic and dynamic formulations were provided to thoroughly understand the system’s behavior. Additionally, various teleoperation control methodologies were implemented and tested, including control in joint space, bilateral control in Cartesian space, and Jacobian-based control methods. These methodologies were optimized to ensure effective teleoperated ultrasound procedures over the internet, even under conditions of long-distance operation and varying time delays.
The simulation results, obtained through MATLAB Simulink’s SimMechanics models of the UR5 manipulator and the Phantom Omni, demonstrated the efficacy of the proposed control methodologies. The system’s performance under simulated time-delay conditions validated the approach, showcasing its potential for real-world application in remote medical diagnostics. The comparison with existing systems (see Table 2) confirms that the HaptiScan system offers improved safety, effective latency handling, and precise force feedback. These advancements make it a viable solution for long-distance remote ultrasound diagnostics, addressing critical challenges in telemedicine.
In conclusion, the developed HaptiScan system represents a significant advancement in remote ultrasound technology, combining safety, precision, and effective teleoperation capabilities. This work lays the foundation for future enhancements and practical deployment in telemedicine, aiming to improve access to ultrasound diagnostics and the overall quality of patient care.

Author Contributions

Z.N., S.P., M.R.C.Q., H.A. (Hamid Abdi) and S.D. supervision, methodology, administration, conceptualization, formal analysis, validation, investigation, writing—original draft. T.L., M.F., S.H., J.M. and H.A. (Houshyar Asadi) data curation, software, resources, writing—review, and editing. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported in part by Telstra Group Limited, VIC, Australia.

Data Availability Statement

Data will be made available on request.

Conflicts of Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

  1. Swerdlow, D.R.; Cleary, K.; Wilson, E.; Azizi-Koutenaei, B.; Monfaredi, R. Robotic arm-assisted sonography: Review of technical developments and potential clinical applications. Am. J. Roentgenol. 2017, 208, 733–738. [Google Scholar] [CrossRef] [PubMed]
  2. Fenster, A.; Downey, D.B.; Cardinal, H.N. Three-dimensional ultrasound imaging. Phys. Med. Biol. 2001, 46, R67. [Google Scholar] [CrossRef] [PubMed]
  3. Arbeille, P.; Ruiz, J.; Herve, P.; Chevillot, M.; Poisson, G.; Perrotin, F. Fetal tele-echography using a robotic arm and a satellite link. Ultrasound Obstet. Gynecol. 2005, 26, 221–226. [Google Scholar] [CrossRef] [PubMed]
  4. Shah, S.; Bellows, B.A.; Adedipe, A.A.; Totten, J.E.; Backlund, B.H.; Sajed, D. Perceived barriers in the use of ultrasound in developing countries. Crit. Ultrasound J. 2015, 7, 11. [Google Scholar] [CrossRef] [PubMed]
  5. LaGrone, L.N.; Sadasivam, V.; Kushner, A.L.; Groen, R.S. A review of training opportunities for ultrasonography in low and middle income countries. Trop. Med. Int. Health 2012, 17, 808–819. [Google Scholar] [CrossRef]
  6. Burnett, D.R.; Campbell-Kyureghyan, N.H. Quantification of scan-specific ergonomic risk-factors in medical sonography. Int. J. Ind. Ergon. 2010, 40, 306–314. [Google Scholar] [CrossRef]
  7. Magnavita, N.; Bevilacqua, L.; Mirk, P.; Fileni, A.; Castellino, N. Work-related musculoskeletal complaints in sonologists. J. Occup. Environ. Med. 1999, 41, 981–988. [Google Scholar] [CrossRef]
  8. Najafi, F.; Sepehri, N. A novel hand-controller for remote ultrasound imaging. Mechatronics 2008, 18, 578–590. [Google Scholar] [CrossRef]
  9. Abolmaesumi, P.; Salcudean, S.E.; Zhu, W.H.; Sirouspour, M.R.; DiMaio, S.P. Image-guided control of a robot for medical ultrasound. IEEE Trans. Robot. Autom. 2002, 18, 11–23. [Google Scholar] [CrossRef]
  10. Salcudean, S.E.; Zhu, W.H.; Abolmaesumi, P.; Bachmann, S.; Lawrence, P.D. A robot system for medical ultrasound. In Robotics Research; Hollerbach, J.M., Koditschek, D.E., Eds.; Springer: London, UK, 2000; pp. 195–202. [Google Scholar]
  11. Vilchis, A.; Troccaz, J.; Cinquin, P.; Masuda, K.; Pellissier, F. A new robot architecture for tele-echography. IEEE Trans. Robot. Autom. 2003, 19, 922–926. [Google Scholar] [CrossRef]
  12. Delgorge, C.; Courreges, F.; Bassit, L.A.; Novales, C.; Rosenberger, C.; Smith-Guerin, N.; Bru, C.; Gilabert, R.; Vannoni, M.; Poisson, G.; et al. A tele-operated mobile ultrasound scanner using a light-weight robot. IEEE Trans. Inf. Technol. Biomed. 2005, 9, 50–58. [Google Scholar] [CrossRef] [PubMed]
  13. Victorova, M.; Lau, H.H.T.; Lee, T.T.Y.; Navarro-Alarcon, D.; Zheng, Y. Comparison of ultrasound scanning for scoliosis assessment: Robotic versus manual. Int. J. Med. Robot. Comput. Assist. Surg. 2023, 19, e2468. [Google Scholar] [CrossRef] [PubMed]
  14. Li, K.; Xu, Y.; Liu, L.; Meng, M.Q.H. A virtual scanning framework for robotic spinal sonography with automatic real-time recognition of standard views. In Proceedings of the 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Mexico, Mexico, 1–5 November 2021; pp. 4574–4577. [Google Scholar]
  15. Su, K.; Liu, J.; Ren, X.; Huo, Y.; Du, G.; Zhao, W.; Wang, X.; Liang, B.; Li, D.; Liu, P.X. A fully autonomous robotic ultrasound system for thyroid scanning. Nat. Commun. 2024, 15, 4004. [Google Scholar] [CrossRef] [PubMed]
  16. Priester, A.M.; Natarajan, S.; Culjat, M.O. Robotic ultrasound systems in medicine. IEEE Trans. Ultrason. Ferroelectr. Freq. Control. 2013, 60, 507–523. [Google Scholar] [CrossRef] [PubMed]
  17. Monfaredi, R.; Wilson, E.; Azizi-Koutenaei, B.; Labrecque, B.; Leroy, K.; Goldie, J.; Louis, E.; Swerdlow, D.; Cleary, K. Robot-assisted ultrasound imaging: Overview and development of a parallel telerobotic system. Minim. Invasive Ther. Allied Technol. 2015, 24, 54–62. [Google Scholar] [CrossRef]
  18. Adams, S.J.; Burbridge, B.; Obaid, H.; Stoneham, G.; Babyn, P.; Mendez, I. Telerobotic sonography for remote diagnostic imaging: Narrative review of current developments and clinical applications. J. Ultrasound Med. 2021, 40, 1287–1306. [Google Scholar] [CrossRef]
  19. Jiang, Z.; Salcudean, S.E.; Navab, N. Robotic ultrasound imaging: State-of-the-art and future perspectives. Med. Image Anal. 2023, 89, 102878. [Google Scholar] [CrossRef]
  20. Hidalgo, E.M.; Wright, L.; Isaksson, M.; Lambert, G.; Marwick, T.H. Current applications of robot-assisted ultrasound examination. Cardiovasc. Imaging 2023, 16, 239–247. [Google Scholar] [CrossRef]
  21. Shi, R.; Rosario, J. Paramedic-performed prehospital tele-ultrasound: A powerful technology or an impractical endeavor? A scoping review. Prehosp. Disaster Med. 2023, 38, 645–653. [Google Scholar] [CrossRef]
  22. Martinelli, T.; Bosson, J.L.; Bressollette, L.; Pelissier, F.; Boidard, E.; Troccaz, J.; Cinquin, P. Robot-based teleechography: Clinical evaluation of the TER system in abdominal aortic exploration. J. Ultrasound Med. 2007, 26, 1611–1616. [Google Scholar] [CrossRef]
  23. Arbeille, P.; Capri, A.; Ayoub, J.; Kieffer, V.; Georgescu, M.; Poisson, G. Use of a robotic arm to perform remote abdominal telesonography. Am. J. Roentgenol. 2007, 188, 317–322. [Google Scholar] [CrossRef] [PubMed]
  24. Arbeille, P.; Provost, R.; Zuj, K.; Dimouro, D.; Georgescu, M. Teles-operated echocardiography using a robotic arm and an internet connection. Ultrasound Med. Biol. 2014, 40, 2521–2529. [Google Scholar] [CrossRef] [PubMed]
  25. Zhang, Y.Q.; Yin, H.H.; He, T.; Guo, L.H.; Zhao, C.K.; Xu, H.X. Clinical application of a 5G-based telerobotic ultrasound system for thyroid examination on a rural island: A prospective study. Endocrine 2022, 76, 620–634. [Google Scholar] [CrossRef] [PubMed]
  26. He, T.; Pu, Y.-Y.; Zhang, Y.-Q.; Qian, Z.-B.; Guo, L.-H.; Sun, L.-P.; Zhao, C.-K.; Xu, H.-X. 5G-based telerobotic ultrasound system improves access to breast examination in rural and remote areas: A prospective and two-scenario study. Diagnostics 2023, 13, 362. [Google Scholar] [CrossRef]
  27. Mitsuishi, M.; Warisawa, S.I.; Tsuda, T.; Higuchi, T.; Koizumi, N.; Hashizume, H.; Fujiwara, K. Remote ultrasound diagnostic system. In Proceedings of the 2001 ICRA. IEEE International Conference on Robotics and Automation, Seoul, Republic of Korea, 21–26 May 2001; Volume 2, pp. 1567–1574. [Google Scholar]
  28. Vieyres, P.; Poisson, G.; Courreges, F.; Smith-Guerin, N.; Novales, C.; Arbeille, P. A tele-operated robotic system for mobile tele-echography: The OTELO project. In Tele-Echography; Springer: Boston, MA, USA, 2006; pp. 461–473. [Google Scholar]
  29. Solazzi, M.; Frisoli, A.; Bergamasco, M.; Sotgiu, E.; PERCRO, S.S.S.A. Kinematic design of a gravity-compensated robot for ultrasound examination and assessment of endothelial dysfunction. In Proceedings of the 12th IFToMM World Congress, Besancon, Besançon, France, 17–21 June 2007. [Google Scholar]
  30. Koizumi, N.; Warisawa, S.I.; Nagoshi, M.; Hashizume, H.; Mitsuishi, M. Construction methodology for a remote ultrasound diagnostic system. IEEE Trans. Robot. 2009, 25, 522–538. [Google Scholar] [CrossRef]
  31. Najafi, F.; Sepehri, N. A robotic wrist for remote ultrasound imaging. Mech. Mach. Theory 2011, 46, 1153–1170. [Google Scholar] [CrossRef]
  32. Virga, S.; Zettinig, O.; Esposito, M.; Pfister, K.; Frisch, B.; Neff, T.; Navab, M.; Hennersperger, C. Automatic force-compliant robotic ultrasound screening of abdominal aortic aneurysms. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Republic of Korea, 9–14 October 2016; pp. 508–513. [Google Scholar]
  33. Kuhlemann, I.; Jauer, P.; Schweikard, A.; Ernst, F. Robotic system for ultrasound tracking in radiation therapy. Med. Phys. 2016, 43, 3672. [Google Scholar] [CrossRef]
  34. Mathiassen, K.; Fjellin, J.E.; Glette, K.; Hol, P.K.; Elle, O.J. An ultrasound robotic system using the commercial robot UR5. Front. Robot. AI 2016, 3, 1. [Google Scholar] [CrossRef]
  35. Finocchi, R.; Aalamifar, F.; Fang, T.Y.; Taylor, R.H.; Boctor, E.M. Co-robotic ultrasound imaging: A cooperative force control approach. In Proceedings of the Medical Imaging 2017: Image-Guided Procedures, Robotic Interventions, and Modeling, SPIE, Orlando, FL, USA, 11–16 February 2017; Volume 10135, pp. 270–280. [Google Scholar]
  36. Ning, G.; Chen, J.; Zhang, X.; Liao, H. Force-guided autonomous robotic ultrasound scanning control method for soft uncertain environment. Int. J. Comput. Assist. Radiol. Surg. 2021, 16, 2189–2199. [Google Scholar] [CrossRef]
  37. Ning, G.; Zhang, X.; Liao, H. Autonomic robotic ultrasound imaging system based on reinforcement learning. IEEE Trans. Biomed. Eng. 2021, 68, 2787–2797. [Google Scholar] [CrossRef]
  38. Cavusoglu, M.C.; Feygin, D. Kinematics and Dynamics of Phantom Model 1.5, Haptic Interface; Tech. Rep. UCB/ERL M01/15; EECS Department, University of California: Berkeley, CA, USA, 2001. [Google Scholar]
  39. Silva, A.J.; Ramirez, O.A.D.; Vega, V.P.; Oliver, J.P.O. Phantom omni haptic device: Kinematic and manipulability. In Proceedings of the 2009 Electronics, Robotics and Automotive Mechanics Conference (CERMA), Cuernavaca, Mexico, 22–25 September 2009; pp. 193–198. [Google Scholar]
  40. Craig, J.J. Introduction to Robotics: Mechanics and Control, 2nd ed.; Addison-Wesley Longman Publishing Co., Inc.: Boston, MA, USA, 1989. [Google Scholar]
  41. Kebria, P.M.; Al-wais, S.; Abdi, H.; Nahavandi, S. Kinematic and dynamic modelling of UR5 manipulator. In Proceedings of the 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Budapest, Hungary, 9–12 October 2016; pp. 004229–004234. [Google Scholar]
  42. Hawkins, K.P. Analytic Inverse Kinematics for the Universal Robots UR5/UR10 Arms; Tech. Rep.; Institute for Robotics and Intelligent Machines, Georgia Institute of Technology: Atlanta, GA, USA, 2013. [Google Scholar]
  43. Kufieta, K. Force Estimation in Robotic Manipulators: Modeling, Simulation and Experiments. Master’s Thesis, Norwegian University of Science and Technology, Trondheim, Norway, 2014. [Google Scholar]
  44. Kebria, P.M.; Khosravi, A.; Nahavandi, S.; Najdovski, Z.; Hilton, S.J. Neural Network Adaptive Control of Teleoperation Systems with Uncertainties and Time-Varying Delay. In Proceedings of the 2018 IEEE 14th International Conference on Automation Science and Engineering (CASE), Munich, Germany, 20–24 August 2018. [Google Scholar]
  45. Signostics Signos RT Handheld Ultrasound Device. Available online: http://www.signostics.com/SignosRT (accessed on 1 July 2024).
  46. Kebria, P.M.; Khosravi, A.; Nahavandi, S.; Shi, P.; Alizadehsani, R. Robust adaptive control scheme for teleoperation systems with delay and uncertainties. IEEE Trans. Cybern. 2020, 50, 3243–3253. [Google Scholar] [CrossRef] [PubMed]
  47. Kebria, P.M.; Khosravi, A.; Nahavandi, S.; Wu, D.; Bello, F. Adaptive Type-2 Fuzzy Neural-Network Control for Teleoperation Systems with Delay and Uncertainties. IEEE Trans. Fuzzy Syst. 2020, 28, 2543–2554. [Google Scholar] [CrossRef]
  48. Lawrence, D.A. Stability and transparency in bilateral teleoperation. IEEE Trans. Robot. Autom. 1993, 9, 624–637. [Google Scholar] [CrossRef]
Robotics 13 00164 g001
Figure 1. The graphical abstract representation of the proposed methodology in this research.
Figure 1. The graphical abstract representation of the proposed methodology in this research.
Robotics 13 00164 g001
Robotics 13 00164 g002
Figure 2. (a) Haptically-Enabled Robotic Ultrasound Platform; (b) CAD model of the HaptiScan platform.
Figure 2. (a) Haptically-Enabled Robotic Ultrasound Platform; (b) CAD model of the HaptiScan platform.
Robotics 13 00164 g002
Robotics 13 00164 g003
Figure 3. The kinematics representation of Phantom Omni.
Figure 3. The kinematics representation of Phantom Omni.
Robotics 13 00164 g003
Robotics 13 00164 g004
Figure 4. Vectorial representation of Phantom Omni: (a) top view; (b) side view.
Figure 4. Vectorial representation of Phantom Omni: (a) top view; (b) side view.
Robotics 13 00164 g004
Robotics 13 00164 g005
Figure 5. UR5 robot model with the DH coordinate frames assignments.
Figure 5. UR5 robot model with the DH coordinate frames assignments.
Robotics 13 00164 g005
Robotics 13 00164 g006
Figure 6. (a) Signostics Signos RT handheld ultrasound device [45], (b) ultrasound probe support mechanism with ATI Nano 17 sensor.
Figure 6. (a) Signostics Signos RT handheld ultrasound device [45], (b) ultrasound probe support mechanism with ATI Nano 17 sensor.
Robotics 13 00164 g006
Robotics 13 00164 g007
Figure 7. UR5 robot model with the DH coordinate frames assignments.
Figure 7. UR5 robot model with the DH coordinate frames assignments.
Robotics 13 00164 g007
Robotics 13 00164 g008
Figure 8. Teleoperation system scheme.
Figure 8. Teleoperation system scheme.
Robotics 13 00164 g008
Robotics 13 00164 g009
Figure 9. The SimMechanics model of Phantom Omni.
Figure 9. The SimMechanics model of Phantom Omni.
Robotics 13 00164 g009
Robotics 13 00164 g010
Figure 10. Time delay.
Figure 10. Time delay.
Robotics 13 00164 g010
Robotics 13 00164 g011
Figure 11. Cartesian position and orientation of the slave manipulator.
Figure 11. Cartesian position and orientation of the slave manipulator.
Robotics 13 00164 g011
Robotics 13 00164 g012
Figure 12. (a) Cartesian velocity of both manipulators; (b) Cartesian velocity error of the manipulators.
Figure 12. (a) Cartesian velocity of both manipulators; (b) Cartesian velocity error of the manipulators.
Robotics 13 00164 g012
Robotics 13 00164 g013
Figure 13. (a) Joints’ angle and velocity of the master manipulator; (b) Joints’ angle and velocity of the slave manipulator.
Figure 13. (a) Joints’ angle and velocity of the master manipulator; (b) Joints’ angle and velocity of the slave manipulator.
Robotics 13 00164 g013
Robotics 13 00164 g014
Figure 14. Force error observed during the teleoperation under varying time delays.
Figure 14. Force error observed during the teleoperation under varying time delays.
Robotics 13 00164 g014
Table 1. DH parameters of a UR5 robot represented by the corresponding coordinate frames in Figure 5.
Table 1. DH parameters of a UR5 robot represented by the corresponding coordinate frames in Figure 5.
iaiαdi θ s i
000--
10 π 2 0.089 θ s 1
20.42500 θ s 2
30.39200 θ s 3
40 π 2 0.109 θ s 4
50 − π 2 0.095 θ s 5
6--0.082 θ s 6
00 -
Table 2. Comparison of the HaptiScan system with existing telerobotic ultrasound systems.
Table 2. Comparison of the HaptiScan system with existing telerobotic ultrasound systems.
SystemForce FeedbackLatency HandlingSafety FeaturesTeleportation Range
HaptiScan (Developed System)Yes10 ms (Low Latency)Over force sensing, discomfort monitoring, motion control, emergency stopLong-distance (Stable)
Mathiassen et al. [34]YesUnstable in high latencyBasic force feedback and motion controlMedium-range
Vilchis et al. [11]NoNo interest in latencyNo specific safety features reportedShort-range
Kuhlemann et al. [33]YesNot explicitly tested for long distancesForce-sensitive manipulator with basic safety mechanismShort-range
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Najdovski, Z.; Pedrammehr, S.; Chalak Qazani, M.R.; Abdi, H.; Deshpande, S.; Liu, T.; Mullins, J.; Fielding, M.; Hilton, S.; Asadi, H. HaptiScan: A Haptically-Enabled Robotic Ultrasound System for Remote Medical Diagnostics. Robotics 2024, 13, 164. https://doi.org/10.3390/robotics13110164

AMA Style

Najdovski Z, Pedrammehr S, Chalak Qazani MR, Abdi H, Deshpande S, Liu T, Mullins J, Fielding M, Hilton S, Asadi H. HaptiScan: A Haptically-Enabled Robotic Ultrasound System for Remote Medical Diagnostics. Robotics. 2024; 13(11):164. https://doi.org/10.3390/robotics13110164

Chicago/Turabian Style

Najdovski, Zoran, Siamak Pedrammehr, Mohammad Reza Chalak Qazani, Hamid Abdi, Sameer Deshpande, Taoming Liu, James Mullins, Michael Fielding, Stephen Hilton, and Houshyar Asadi. 2024. "HaptiScan: A Haptically-Enabled Robotic Ultrasound System for Remote Medical Diagnostics" Robotics 13, no. 11: 164. https://doi.org/10.3390/robotics13110164

APA Style

Najdovski, Z., Pedrammehr, S., Chalak Qazani, M. R., Abdi, H., Deshpande, S., Liu, T., Mullins, J., Fielding, M., Hilton, S., & Asadi, H. (2024). HaptiScan: A Haptically-Enabled Robotic Ultrasound System for Remote Medical Diagnostics. Robotics, 13(11), 164. https://doi.org/10.3390/robotics13110164

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop