Keywords

1 Context

1.1 Mixed Reality

Microsoft HoloLens headsets [1] were recently released in France (December 2016). Such a headset uses several different sensors. It is a self-contained computer with Wi-Fi connectivity. These headsets have a semi-transparent visor on which 3D objects, called holograms, are projected; therefore, these holograms are superimposed on the user’s environment through the visor. A user can interact with these objects by headset-recognized gestures if the developed application has intended it (for example: aiming at an object to select it, then pinching the fingers and moving the hand to move this object), see Fig. 1. Moreover, the headset also scans the environment, and holograms can interact with it. Unity framework [3] with the appropriate software library MixedRealityToolkit-Unity (MRTK) [4], allow us to design interactive 3D graphics applications that can be deployed in this type of hardware (nevertheless with Unity and C#programming language skills). To design such an application, we must plan both the content to be displayed and all possible user’s interactions with the environment. More precisely, 3D content must be modeled with a 3D modeler software (for example, Blender [5]), then it must be imported inside the Unity application to be arranged in a 3D scene; then chosen interactions must be encoded (C#scripts in Visual Studio [6]). When the implementation of the application is finished inside Unity, we may deploy it in the HoloLens headset.

1.2 Medical Context

During surgical procedures, it may be necessary to insert one or several trocars between the patient’s ribs. This is the case, for example, with mini-invasive cardiac surgery to replace one or several heart valves assisted by 3D thoracoscopy: a trocar must be inserted before the camera is introduced. Most systems that provide visual assistance to trocar placement are complex and expensive. They are based either on robotics or magnetic tracking, and allow to track a precise positioning of surgical instruments and to provide an augmented visualization on a screen inside the operating room. Nevertheless, few operating rooms for this type of intervention own this type of hardware and most surgeons place and insert a trocar without visual assistance.

Fig. 1.
figure 1

HoloLens headset and user view [2].

Full size image
Fig. 2.
figure 2

CT-scan (CTChest sample – Slicer 3D [7], volume-rendering cardiac setting), volume-rendering MR generic in MITK [8].

Full size image

Minimally Invasive Surgeries (MIS). A large number of surgical techniques have evolved to minimally invasive surgical techniques (or MIS). One of the best known of MIS is laparoscopy in gynecology. Rather than making large openings in the patient, several mini-incisions are made. These techniques lead to faster recovery and shorter hospitalizations. The counterpart is that the surgeon no longer has a direct vision of the organs he has to handle. Some of these techniques require the use of trocars and in most cases, no visual assistance is offered.

About Incisions with Trocar. Many techniques for trocar insertion rely on the skills of clinicians (positioning relative to anatomical landmarks on the patient ...), and do not use visual assistance. In other words, trocars are often blind-inserted. See for example, the recent study about the initial placement of a trocar without any visual assistance in case of bariatric surgery [9]. Some clinicians have begun to study the impact of trocar placement, trocar diameter, hole size compared with subsequent intervention, healing, see for example a study in the case of chest surgery [10], laparoscopy [11].

We can find studies on the placement of trocar for minimally invasive surgery in the case of laparoscopy (measure of kinetic aspects of trocar insertion [12]), and with a surgical robot [13], see also the recent use of machine learning [14].

Minimally Invasive Cardiac Surgeries (MICS). In [15], authors review the evolution of various open heart cardiac surgery procedures (full sternotomy) to minimally invasive cardiac surgeries (MICS), these techniques were initiated following the growing interest of laparoscopic surgery. It is described the interest to avoid opening the thorax, to perform an extracorporeal circulation and to make instead some incisions for specific procedures (mini-thoracotomy) and if it is recommended for patients. These new approaches aim to reduce the complications associated with sternotomy, postoperative pain, and lead, as said before, to faster recovery and shorter hospitalizations.

For some centers that are equipped, some mitral valve replacement are assisted by video [16] or robot [17], see also [18].

Mixed Reality. In [19], the authors exploit HoloLens interactive commands for visualization (with no registration) of a 3D model of myocardial scar. In [20], a manual registration is made to superimpose 3D models (vascular tree) onto patient’s body, as it is the case for our experiment. We believe that this application is exploited rather for the visualization of data than for a precise registration (see wireframed box in Fig. 3 of [20], outside the patient’s body). In [21, 22], a project superimpose a patient’s vascular tree by registering it with an electromagnetic tracking system, several additional data are also displayed (orthogonal views of CT and angioscopies). In [23], we find the same guidance system for the alignment of a guide hole drilling guide (position and orientation) in case of hip surgery. The system requires a robotic arm, an additional camera and markers for guidance.

Fig. 3.
figure 3

Two points are marked in the CT-scan.

Full size image

Our Motivation. We have proposed a first software prototype of trocar guidance which exploits the mixed reality framework through the use of the Microsoft HoloLens headset. The surgeon first defines the position and direction of insertion using the CT scan in order to minimize the risk of damaging the patient’s internal organs during the trocar insertion then an interactive registration may help him or her during the intervention.

In this article, we describe the sequence of computing steps and the first obtained results about such kind of interactive registration.

2 Our Software Prototype

2.1 Preoperative Phase

The objective is to achieve a centered incision between a pair of patient’s ribs, while not damaging internal organs. The surgeon identifies in the patient’s CT-scan, Fig. 2, how he or she wishes to insert the trocar (position and orientation), by defining two points in the CT, Fig. 3. More precisely, the surgeon scrolls through the sections of the CT-scan, positions the first point on the patient’s surface between two ribs which corresponds to the insertion point of the trocar (Point1 in Figs. 3(a) and (c)), then positions the second point inside the patient’s body, in order to avoid damaging the patient’s organs when the trocar will be inserted (Point2 in Figs. 3(b) and (c)). In this way, he or she defines a virtual trocar, or the pair (position, orientation) of a virtual trocar, that will later assist him or her when inserting the actual trocar.

Fig. 4.
figure 4

(a) Torso and (b) ribs segmentation (Slicer).

Full size image
Fig. 5.
figure 5

(a) Torso mesh decimation (Blender), (b) torso and (c) ribs import into Unity.

Full size image
Fig. 6.
figure 6

Several possibilities of visualization of the 3D model of the virtual trocar (in red) according to torso and ribs transparency. (Color figure online)

Full size image

2.2 Preparation of the Application to Be Transferred into the Headset

Our application must be embedded inside the HoloLens headset. Patient-specific data (torso, ribs, position/orientation of a virtual trocar) must be available into the headset to help the surgeon during the surgical procedure.

Here are the steps that the surgeon (or one of his or her assistants) must perform to design this digital content:

  • (Step 1) to segment torso (Fig. 4(a)) and ribs (Fig. 4(b)) in Slicer software;

  • (Step 2) to import the torso mesh into Blender software, and to decimate the torso mesh (in order to reduce the data volume to be projected on the visor’s headset) (Fig. 5(a));

  • (Step 3) to import of both the torso (Fig. 5(b)) and ribs (Fig. 5(c)) in our Unity application respecting the spacing parameters of the DICOM file of the patient’s CT-scan. Then, our application computes the modeling of a 3D virtual trocar (in red) by using the two points previously marked by the surgeon (Fig. 6);

  • (Step 4) to deploy the application (with these data) inside the HoloLens headset. Note that it is also possible to transfer data to the application embedded inside HoloLens through the cloud, and not to deploy for each patient.

2.3 Our Embedded Software Inside Hololens Headset

Our application will propose to the user to interactively register patient’s data on the actual patient in the operating room. Details about this interactive registration are given in the next section.

Fig. 7.
figure 7

(a) Intermediate model for the interactive registration, (b) possibility to translate and to orient the intermediate model with the menu of our embedded application. (Color figure online)

Full size image

3 Interactive Registration

3.1 3D Model to Be Registered

It seems difficult to directly align the 3D model of the patient’s torso with the actual patient’s torso. Therefore, we have chosen to define an intermediate 3D model that will be registered onto the patient. This 3D intermediate model is constituted in this way: its origin corresponds with the origin of the DICOM file (in other words it corresponds to the CT-scan with the appropriate scales/spacing according to the CT-scan resolution), it contains the torso, the ribs, the two previously defined points. We also add three planes to this intermediate model (Fig. 7(a)):

  • the first plane (green plane) is the plane of the back of the patient which corresponds to the surface of the table,

  • the second plane (blue plane) is the one that corresponds to a virtual plane passing through the patient’s shoulders, thus the CT-scan must record this area,

  • the third plane (red plane) corresponds to the virtual plane between the patient’s torso and his left arm.

3.2 Registration Procedure

Our application is designed in such a way that this 3D intermediate model can be translated or orientated (Fig. 7(b)). In this way, the user of this application can position the intermediate model so as to first match it (translation and orientation) with the intervention table (green plane), then moves (translation and orientation) the model in such a way that it corresponds with the plane of the patient’s shoulders (blue plane), finally translates the model so that the red plane is between the torso and the patient’s left arm.

In practice, once this first rough registration of the intermediate model is made, it may be interesting not to display it longer, and to reveal only the torso and ribs (when the torso is totally transparent). Note that it is also possible to orient the first plane if the patient’s back is not fully over the table (this is the case for some surgical procedures).

Fig. 8.
figure 8

View through the headset by the surgeon: it is difficult to see the trocar due to the 3D models projection.

Full size image
Fig. 9.
figure 9

View through the headset by one surgeon’s assistant. (Color figure online)

Full size image

4 Results

4.1 Intraoperative Step

Different difficulties occur: a minimum distance (80 cm) from the helmet’s wearer to the first 3D object to be visualized is required, we modified it (near clipping plane property in an appropriate script of MRTK library). If the surgeon wears the helmet, it is very difficult for him or her to see the trocar even by adjusting the transparency of the models, Fig. 8. Therefore, one of his or her assistants must wear the headset and will help the surgeon to position the real trocar according to the materialization of the virtual trocar on the visor of the headset.

The assistant must first interactively register the intermediate model with the procedure described in the previous section. Then, he or she can remove the intermediate model and only see the 3d models of torso and trocar (Fig. 9(a)), or the set of torso in transparency, ribs and trocar (Fig. 9(b)) to better fit the interactive registration. Then, the assistant can begin guiding the surgeon in such a way that he or she first positions the tip of the trocar and then directs it (according to the virtual trocar in red) (Fig. 9(c)). The assistant must move around the surgeon to give the best fitting, Fig. 10.

Fig. 10.
figure 10

View through the headset by the surgeon’s assistant. Trocar alignment.

Full size image

4.2 First Results

Although this approach provides information that surgeons do not currently have, it is rather difficult to make a realignment with 3D data: the assistant must move many times to find the best fitting of position and orientation of the trocar, which is very difficult to do due both to the crowding of the operating room and of the 3D effect (see for example Fig. 9(c)). The transparency of the model has to be adapted many times in order to be able to superimpose the plans of the intermediate model or finally only the torso at best on the patient.

4.3 Future Works

In this paper, we study a first proof of feasibility to use HoloLens to assist cardiac surgeons to position trocar and it is difficult to evaluate any positioning or orientation accuracy. Although the order of magnitude is relatively large (a 4 to 5 cm space between ribs), it remains difficult to perform a precise interactive placement of the trocar.

We continue our work in order to facilitate the registration procedure by using a marker-based functionality recently provided by Unity (Vuforia library [24]).