KR20160138233A - Surgical system with haptic feedback based upon quantitative three-dimensional imaging - Google Patents

Abstract

A system for providing haptic feedback during a medical procedure comprising: a quantitative three-dimensional (Q3D) endoscope; A surgical instrument arranged to deform the tissue structure; A haptic user interface device configured to provide an indication of tissue structural deformations in response to information indicative of measurements of tissue structural deformations; And a processor configured to generate a Q3D model that includes information indicative of measurements of tissue structural deformation and to provide information to the haptic user interface device indicative of measurements of tissue structural deformations.

Description

TECHNICAL FIELD [0001] The present invention relates to a surgical system having a haptic feedback based on quantitative three-dimensional imaging,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates generally to a surgical endoscopic system having an associated image sensor, and more particularly to determining three-dimensional coordinates of a physical structure displayed on a surgical image.

The quantitative 3D (Q3D) vision provides numerical information about the actual physical (x, y, z) 3D coordinates of the target points in a real world scene. Quantitative 3D vision not only allows a person to obtain a three-dimensional perception of real-world scenes, but also obtains numerical information about physical dimensions of objects in the scene and physical distances between objects in the scene. In the past, several Q3D systems have been proposed that use time-of-flight related information or topological information to determine 3D information about a scene. Other Q3D systems used structured light to determine 3D information about the scene.

The use of flight time information is referred to as a " CMOS-compatible three-dimensional image sensor IC ", which uses pixel-by-pixel photodetectors Light Sensing Detectors < RTI ID = 0.0 > (< / RTI > US Pat. No. 6,323,942). Each detector is associated with an associated high speed counter (not shown) that accumulates clock pulses of a numerical value directly proportional to the flight time (TOF) for the system emitted pulse to be reflected from the object point and detected by a pixel detector focused at that point. (high speed counter). TOF data provides a direct digital measure of the distance to a point on an object that reflects the light pulse emitted from a particular pixel. In the second embodiment, the counter and high speed clock circuit are eliminated and instead have each pixel detector charge storage and electronic shutter. The shutter is opened when the optical pulses are emitted and then closed so that each pixel detector accumulates charge as a function of the return photon energy reaching the associated pixel detector. The amount of accumulated charge provides a direct measure of the reciprocal TOF.

The use of time lag information is referred to as " Apparatus and method for endoscopic 3D data collection ", and a light transmission mechanism for guiding a modulated measurement beam and a measurement beam to a region to be observed In addition to the optical imaging device for initiating and imaging the signal beam from the area to be observed on at least the phase-sensitive image sensor, a light transmission mechanism is disclosed in U. S. Pat. 8,262,559. A time delay, which may correspond to a difference in depth in the mm range, generates phase information that allows generation of an image representing depth and distance information.

U.S. Patent Application Publication No. 2012/0190923 entitled " Endoscope "for the use of structured light to determine the physical coordinates of objects in a visual image; Quot; An endoscopic 3D scanner based on structured light "published under the name of C. Schmalz et al. In Journal of Medical Image Analysis, 16 (2012) 1063-1072 have. Triangulation is used to measure surface topography. Structured light in the form of a projection beam, which can have a range of different color spectra, is incident on the surface and reflected. The reflected light is observed by the camera which is calibrated to use the reflected color spectrum information to determine the 3D coordinates of the surface. More specifically, the use of structured light generally includes illuminating a light pattern on a 3D surface, and determining a physical distance based on a variation pattern of light due to an outline of a physical object.

An imager array camera has been developed that includes a plurality of pixel arrays that can be used to compute scene depth information for pixels in a pixel array. High resolution (HR) images are generated from multiple low resolution (LR) images. A reference viewpoint is selected and the HR image is generated as seen at that point. The parallax processing technique uses an aliasing effect to determine the pixel correspondence points of a non-reference image for reference image pixels. Fusion and super resolution techniques are used to generate HR images from multiple LR images. See, for example, U.S. Patent No. 8,514,491 entitled " Capturing and Processing Images Using a Monolithic Camera Array with a Heterogeneous Imager "; &Quot; A system and method for determining depth from multiple views of a scene that includes aliasing using hypothesis fusion ", entitled " Systems and Methods for Determining Depth from a Scene That Includes Aliasing Using Hypothesized Fusion &2013/0070060;Quot; PiCam: An Ultra-Thin High Performance Monolithic Camera Array ", by K. Venkataraman et al.

1 is an illustration depicting details of a known imager sensor 180 in accordance with some embodiments. The image sensor 180 includes a sensor array 184. Each sensor in the sensor array includes a two-dimensional pixel array with at least two pixels in each dimension. Each sensor includes a lens stack 186. Each lens stack 186 has a corresponding focal plane 188. Each lens stack 186 generates a separate optical channel that resolves the image onto a corresponding pixel array disposed within its corresponding focal plane 188. The pixels act as light sensors, and each focal plane 188 with multiple pixels serves as an image sensor. Each sensor with its focal plane 188 occupies a certain area of the sensor array that is different from that of the sensor array occupied by the other sensors and focal planes.

FIG. 2 is an explanatory view showing a simple plan view of the known sensor arrangement 184 of FIG. 1, including the sensors labeled sensors S ₁₁ through S ₃₃ . The imager sensor array 184 is fabricated on a semiconductor chip to include a plurality of sensors S ₁₁ through S ₃₃ . Each of the sensors S ₁₁ through S ₃₃ includes a plurality of pixels (e.g., 0.32 mega pixels) and is connected to a peripheral circuit (not shown) that includes independent read control and pixel digitization. In some embodiments, the sensors S ₁₁ through S ₃₃ are arranged in a grid format as shown in FIG. In another embodiment, the sensors are arranged in a non-grid format. For example, the sensor may be arranged in an irregular pattern that includes a circular pattern, a zigzag pattern, a scattered pattern, or a sub-pixel offset.

Each individual pixel of the sensor 180 of FIGS. 1-2 includes a microlens pixel stack. Figure 3 is an illustration of a known micro lens pixel stack of the sensors of Figures 1-2. The pixel stack 800 includes a microlens 802 positioned over the oxide layer 804. Generally, below the oxide layer 804 there may be a color filter 806 disposed over the nitride layer 808, a nitride layer 808 is disposed over the second oxide layer 810, and a second oxide layer 810 are disposed on top of a silicon layer 812 that includes an active area 814 (typically a photodiode) of an individual pixel. The main role of the microlens 802 is to collect the light incident on its surface and to focus the light onto a small active area 814. [ The pixel opening 816 is determined by the diffusion of the microlenses.

Additional information regarding the above-described known image sensor array structure is provided in U.S. Patent No. 8,514,491 B1 (filed November 22, 2010) and U.S. Patent Application Publication No. US 2013/0070060 A1 (filed September 19, 2012).

In one aspect, a system is provided for providing haptic feedback during a surgical procedure. A quantitative three-dimensional (Q3D) endoscope is deployed to image the scene within its watch. The surgical instrument disposed within the watch is operable to deform the tissue structure within the watch. Wherein the haptic user interface device is configured to provide an indication of tissue structural deformations in response to information indicative of measurements of tissue structural deformations. One or more processors are configured to generate a Q3D model that includes information representative of measurements of tissue structural deformations and to provide information indicative of measurements of tissue structural deformations to the haptic user interface device.

In another aspect, a system is provided for providing haptic feedback during a medical procedure. A quantitative three-dimensional (Q3D) endoscope is deployed to image the scene within the watch. The surgical instrument disposed within the watch is operable to deform the tissue structure within the watch. One or more processors are configured to generate a Q3D model that includes information representative of measurements of tissue structural deformation. Based on the knowledge of the measured tissue deformation and tissue stiffness, one or more processors calculate or estimate the force exerted on the tissue by the device. The processor provides the haptic user interface device with information indicative of the applied force.

In another aspect, a system is provided for providing haptic feedback during a medical procedure. A quantitative three-dimensional (Q3D) endoscope is deployed to image the scene within the watch. The device disposed within the watch is operable to palpate the tissue structure within the watch. One or more processors are configured to generate a Q3D model that includes information indicative of measurements of tissue structure deformation during promotion. Based on the measured tissue deformation and knowledge of the applied force, one or more processors may calculate or estimate a measure of tissue stiffness. The processor provides the haptic user interface device with information indicative of the accelerated tissue stiffness.

BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the invention will be best understood from the following detailed description, which refers to the accompanying drawings, which are briefly described herein. It should be noted that according to standard practice in the industry, the various details are not shown proportionally. Indeed, the dimensions of the various details may optionally be increased or decreased for clarity of explanation. In addition, the specification may use reference numerals and / or letters repeatedly in various embodiments. Such repetitive use is for simplicity and clarity and does not affect the relationships between the various embodiments and / or configurations described in and of themselves.
1 is an explanatory diagram showing details of a known imager sensor array.
Figure 2 is an explanatory view showing a simple plan view of a known imager sensor array including the plurality of sensors of Figure 1;
Figure 3 is an illustration of a known microlens pixel stack
4 is an explanatory view showing a perspective view of a surgical scene through a viewer according to some embodiments.
5 is an exemplary block diagram of a remote operated surgical system for performing a minimally invasive surgical procedure using one or more mechanical arms, in accordance with some embodiments.
Figure 6 is an exemplary perspective view of a patient side system of the system of Figure 5 in accordance with some embodiments.
7A is an exemplary diagram of a first image acquisition system according to some embodiments.
7B is an exemplary diagram of a second image acquisition system in accordance with some embodiments.
8 is an exemplary block diagram illustrating control blocks associated with the first image acquisition system of FIG. 7A, and in operation, in accordance with some embodiments.
9 is an exemplary flow chart illustrating a process for determining a quantitative three dimensional location of a physical target, in accordance with some embodiments.
10 is an exemplary flow chart illustrating certain details of a process generally corresponding to the module of FIG. 9 for systematically selecting a target, in accordance with some embodiments.
Figure 11 is an exemplary sensor image that is arranged to have a clock that includes an exemplary three-dimensional physical world scene that includes a plurality of sensors and includes three objects, according to some embodiments. FIG.
12 is an illustration showing projection of a plurality of physical objects of FIG. 11 onto a plurality of sensors, in accordance with some embodiments.
13 is an illustration depicting the selection of a region of interest from within a real-world scene according to some embodiments.
14 is an illustration depicting details of the relative geometric offset of images projected in a plurality of sensors according to some embodiments.
15 is an explanatory diagram illustrating images projected to sensors of a specific example within a ROI that is shifted to the right to align with an image projected on a designated reference sensor within a region of interest (ROI), according to some embodiments.
16 is an explanatory view showing projection of a selected target point onto a plurality of sensors according to some embodiments.
FIG. 17 is an explanatory diagram illustrating a portion of an imager array including a plurality of sensors of FIG. 16 and a selected trellis point T disposed at a location within a physical space, in accordance with some embodiments.
FIG. 18 is an exemplary front view illustrating the projection of the currently selected target point T onto the plurality of image sensors of FIG. 16, in accordance with some embodiments.
Figure 19 illustrates an arrangement for multiple sensors of the currently selected target, as described above with reference to Figure 17, in accordance with some embodiments, and also a description of y-direction pixel offsets for candidate pixels in each sensor .
20 is an exemplary flow chart illustrating a first process for using Q3D information during a surgical procedure, in accordance with some embodiments.
21 is an illustration depicting menu selections displayed on a display screen in accordance with the process of FIG. 20, in accordance with some embodiments.
Figures 22A-22B are illustrations depicting specific details for receiving user input in accordance with the process of Figure 20, in accordance with some embodiments.
23 is an exemplary flow chart illustrating a second processor for using Q3D information during a surgical procedure, in accordance with some embodiments.
24 is an explanatory diagram illustrating menu selections displayed on a display screen in accordance with the process of FIG. 23, in accordance with some embodiments.
25 is an exemplary flow chart illustrating a process for using Q3D information to determine haptic feedback, in accordance with some embodiments.
26 is an explanatory view showing a tissue structure portion and a Q3D endoscope which are contacted by an end portion of a surgical instrument, according to some embodiments.
27A-27C are illustrations depicting a first embodiment of a sensory user interface (TUI) that functions as a shape display suitable for providing haptic feedback, in accordance with some embodiments.
28A to 28C are explanatory views showing a tissue structural part having a shape deformed by a force applied by an operation instrument.
29A-29C illustrate alternative embodiments of a sensory user interface (TUI) that functions as a shape display suitable for providing haptic feedback, in accordance with some embodiments.
30 is an illustration illustrating an alternative embodiment TUI mounted to a surgeon's finger, in accordance with some embodiments.
31 is a diagram illustrating an exemplary computational block configured to determine haptic feedback as a function of tissue surface deformation, in accordance with some embodiments.
32 is an exemplary flow chart illustrating a process executed using the computational block of FIG. 31, in accordance with some embodiments.
33 is an exemplary flow chart of a first haptic feedback process for use with the TUI of Figs. 27A-27C, in accordance with some embodiments.
Figure 34 is an exemplary flow chart of a second haptic feedback process for use with the TUI of an alternative embodiment of Figures 29A-29C, in accordance with some embodiments.
35 is an exemplary flow chart of a third process for controlling a force imposed on a selected tissue surface location, in accordance with some embodiments.
36a-36e are illustrations showing cross-sectional views of a sequence of surfaces of a tissue structure that show deformation of a tissue surface produced in response to forces exerted by an instrument, according to some embodiments.
37A-37E illustrate "instantaneous" variations of an example of a TUI pin top surface interface corresponding to exemplary tissue structural transformations shown in the section views of the constant sequence of Figs. 36A-36E, according to some embodiments Lt; RTI ID = 0.0 > 27A-27C < / RTI >
Figures 38A-38E illustrate an example of a " composite "variation of a TUI feedback surface corresponding to exemplary tissue structural deformations shown in cross-sectional views of the constant sequence of Figures 36A-36E, according to some embodiments. Sectional views of a certain sequence of the TUI of -27c.
Figures 39A-39E illustrate displacements of a sequence of one or more pins within the TUI of an alternative embodiment of Figures 29A-29C, in response to example modifications of the target tissue surface shown in Figures 36A-36E, according to some embodiments. Fig.

The following description is provided to enable those skilled in the art to make and use a surgical endoscope system that obtains quantitative three-dimensional (Q3D) information and generates tactile feedback based on the Q3D information. Various modifications to the embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the invention. Also, in the following description, numerous details are set forth for purposes of explanation. However, it will be appreciated that the present invention may be practiced without the use of such details. In some instances, well-known mechanical components, processes, and data structures are shown in block diagram form in order not to obscure the description with unnecessary detail. The same reference numerals can be used to represent different types of cities of the same item in different drawings. Flowcharts of the drawings referred to below are used to represent processes. A computer system may be configured to execute some of these processes. The modules in the flowcharts representing computer-implemented processes represent the configuration of a computer system according to computer program code for executing the operations described with reference to these modules. Accordingly, the invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features described herein.

Brief overview

According to some embodiments, an imager comprising a sensor array is associated with an endoscope. The image sensor array includes a plurality of sensors, and each sensor includes a pixel array. A portion of the endoscope is inserted into the human body cavity, and the target object in the image sensor array's eye is illuminated using a light source. The physical location and / or dimensions of the target object are determined based on the images of the target object projected onto each sensor of the array.

4 is an explanatory view showing a perspective view of a surgical scene through a viewer 312 according to some embodiments. A viewing system with two viewing elements 401R and 401L can provide a good 3D viewing perspective. Numerical values representing physical dimensions and / or positional information of the physical structure in the surgical scene are overlaid on the surgical scene image. For example, a numerical distance value "d_Instr_Trgt" is displayed between the device 400 and the target 410 in the scene.

Remotely Operated Medical System

Remote operation means operation of the machine at a certain distance. In a minimally invasive remotely operated medical system, the surgeon may use an endoscope including a camera to observe the surgical site within the patient's body. Stereoscopic images were obtained that enabled perception of depth during the surgical procedure. According to some embodiments, a camera system mounted on an endoscope and including an imager sensor array provides color and illumination data that can be used to generate three-dimensional images in addition to quantitative three-dimensional information.

5 is an exemplary block diagram of a remote operated surgical system 100 for implementing a minimally invasive surgical procedure using one or more mechanical arms 158, in accordance with some embodiments. Aspects of the system 100 include remote robotic and autonomous operating features. These mechanical arms often support the instrument. For example, a mechanical surgical arm (e.g., a central mechanical surgical arm 158C) may include an endoscope having a stereoscopic or three-dimensional surgical image acquisition device 101C, such as an endoscope associated with a Q3D image sensor array, Can be used. The mechanical surgical arm 158C may include a sterilizing adapter or clamp, clip, screw, slot / groove or other fastener for mechanically fixing the endoscope including the image capturing device 101C to the mechanical arm. Conversely, the endoscope with the image acquisition device 101C includes a physical contour and / or structure that is complementary to the physical contour and / or structure of the mechanical surgical arm 158C such that it is securely interfitted with the mechanical surgical arm 158C .

The user or operator O (typically a surgeon) performs a minimally invasive surgical procedure on the patient P by manipulating the control input device 160 at the master control console 150. The operator can view video frames of images of the surgical site within the patient's body through the stereoscopic display device 164 including the viewer 312 described above with reference to Fig. The computer 151 of the console 150 instructs the exercise of the endoscope surgical instruments 101A-101C to be controlled remotely via the control line 159 and uses the patient side system 152 (also called the patient side cart) Thereby realizing the movement of the devices.

The patient side system 152 includes one or more mechanical arms 158. Generally, the patient side system 152 includes at least three mechanical surgical arms 158A-158C (collectively referred to as a mechanical surgical arm 158) supported by a corresponding positioning setup arm 156. [ The central mechanical surgical arm 158C may support an endoscopic camera 101C suitable for acquisition of Q3D information for images in the camera's watch. The left and right mechanical surgical arms 158A and 158B can respectively support the supporting devices 101A and 101B capable of operating the tissues.

6 is an exemplary perspective view of a patient side system 152 in accordance with some embodiments. The patient side system 152 includes a cart column 170 supported by a base 172. One or more mechanical insertion surgical arms / links 158 are attached to one or more setup arms 156, which are part of the positioning portion of the patient side system 152, respectively. At approximately the center of the base 172, the cart column 170 includes a protective cover 180 that protects the components of the counterbalance subsystem and the pollutant isolation subsystem.

Except for the monitor arm 154, each mechanical surgical arm 158 is used to control the devices 101A-101C. Each of the mechanical surgical arms 158 is connected to a setup arm 156 and the setup arm 156 is then connected to the carriage housing 190 in one embodiment of the present invention. One or more mechanical surgical arms 158 are each supported by their respective setup arms 156 as shown in Fig.

The mechanical surgical arm 158A-158D may include one or more displacement transducers for generating raw uncorrected kinematics information to assist in initial acquisition by tracking of the tracking system and / or instruments, respectively, And / or a position sensor 185. Devices may also include displacement transducers, position lines and / or orientation sensors 186, in some embodiments of the invention. In addition, the one or more devices may include a marker 189 for facilitating acquisition and tracking of the device.

Additional information on a remotely operated medical system is provided in United States Patent Application Publication No. US 2012/0020547 filed September 30, 2011.

Endoscopic Imager System

7A is an explanatory diagram of a first endoscope having a first image capturing apparatus 101C according to some embodiments. The image acquisition device 101C includes an endoscope including a length 202 including a first end portion 204, a second end portion 206 and a tip portion 208 of the first end portion 204 do. The first end portion 204 is dimensioned to be inserted into the human body cavity. A sensor array 210 (not shown) comprising a plurality of image sensors is coupled to the tip portion 208 of the first end portion 204. According to some embodiments, each sensor in sensor array 210 includes a pixel array. The length portion 202 has a length sufficient to position the tip portion 208 sufficiently close to the target object in the body cavity so that the target object can be imaged by the imager sensor array 210. According to some embodiments, second end portion 206 may include a physical contour and / or structure (not shown) as described above to rigidly interlock with a mechanical arm (not shown). The length portion 202 also includes one or more electronic signal paths 212 for electronically communicating information with the imager sensor array 210. A light source 214 is arranged to illuminate the object to be imaged. According to some embodiments, light source 214 may be, for example, unstructured light, white light, color filtering light, or partially selective wavelength light. According to some embodiments, the light source 214 is located at the tip 208, and in another embodiment, is selectively positioned separately from the endoscope 101C.

7B is an explanatory diagram of a second endoscope having a second image acquisition system 101C 'according to some embodiments. The aspects of the second image acquisition system 101C 'which are basically the same as those of the first endoscope having the first image acquisition system 101C are denoted by the same reference numerals and will not be described again. An input portion for a light pipe input, such as a rod lens, is disposed in the tip portion 208 of the first end portion 204. The light pipe body extends into the length portion 202 to transmit the received image when the light pipe is input to the imager sensor array 210 physically displaced from the tip portion 208. [ In some embodiments, the imager sensor array 210 is displaced far enough away from the tip portion 208 so that the imager sensor array 210 is disposed outside the body cavity during observation of a subject in the body cavity.

FIG. 8 is an exemplary block diagram illustrating control blocks associated with a first endoscope 101C having the first image acquisition system 101C of FIG. 7A and in operation, according to some embodiments. The images acquired by the imager sensor array 210 are transmitted to the video processor 104 via the data bus 212 and the video processor 104 communicates with the controller 106 via the bus 105. The video processor 104 may include a camera control unit (CCU) and a video signal detector (VSD) board. The CCU programs or controls various settings of the image sensor 210, such as brightness, color scheme, white balance, and the like. The VSD processes the video signal received from the image sensor. Optionally, the CCU and the VSD are integrated into one functional block.

According to some embodiments, a processor system including one or more processors is configured to execute processor functions. In some embodiments, the processor system includes a plurality of processors configured to work together to perform the processor functions described herein. Accordingly, reference to at least one processor configured to perform one or more functions includes a processor system in which the functions can be executed by a plurality of processors that can be executed with only one processor or with one processor.

In one embodiment, controller 106, which includes a processor and a storage device (not shown), computes the physical quantitative 3D coordinates of points in the scene adjacent to tip 208 of length 202, The synthesized 3D scenes may be displayed on the 3D display 110. The synthesized 3D scenes may be displayed on the 3D display 110. For example, According to some embodiments, Q3D information is generated for a surgical scene, such as, for example, numerical indicia of the dimensions of the surface contour of an object in a surgical scene, or distance from an object in a surgical scene. As described in more detail below, numerical Q3D depth information can be used to annotate stereoscopic images of surgical scenes with distance information or surface contour information.

The data buses 107 and 108 exchange information and control signals between the video processor 104, the controller 106 and the display driver 109. In some embodiments, these elements may be integrated with the image sensor array 210 within the body of the endoscope. Alternatively, these elements may be distributed internally and / or externally of the endoscope. The endoscope is positioned through the cannula 140 to penetrate the body tissue 130 to provide visual access to the surgical scene including the target 120. Optionally, the endoscope and one or more devices may pass through a single opening (a single incision or a natural bore) to reach the surgical site. The target 120 may be an anatomical target, another surgical instrument, or any other aspect of a surgical scene within a patient's body.

The input system 112 receives the 3D visual representation and provides it to the processor 106. Input system 112 may include a storage device coupled to an electronic communication bus (not shown) that receives a 3D model, such as a CRT or MRI, from a system (not shown) that creates a 3D model. The processor 106 may be used, for example, to compute the intended alignment between the Q3D model and the 3D visual representation. More specifically, and without limitation, input system 112 may include a processor configured to establish an Ethernet communication connection between system 152 and an imaging system (not shown) such as an MRI, CT or ultrasound imaging system . Other imaging systems may be used. Other types of communication connections may be used, such as Bluetooth, Wi-Fi, optical communication, and the like. Optionally, the system 152 and the imaging system may be integrated into one larger system. The results of the alignment process may be stored in a storage device associated with the processor 106 if further manipulation of the external device is provided or indicated as shown in Fig.

Examples of Q3D information added to a scene image

Referring again to FIG. 4, FIG. 4 is an illustration depicting a perspective view of the viewer 312 of the master control console 150 of FIG. 5, according to some embodiments. In accordance with some embodiments, to provide a 3D perspective, the viewer 312 includes a stereoscopic image for each eye. As shown, the left side image 400L and the right side image 400R of the surgical site include the device 400 and the target 410 in the left viewfinder 401L and the right viewfinder 401R, respectively. The images 400L and 400R in the viewfinders may be provided by the left display device 402L and the right display device 402R, respectively. The display devices 402L and 402R may be a pair of a cathode ray tube (CRT) monitor, a liquid crystal display (LCD) or other type of image display device (e.g., plasma, digital light projection, etc.). In a preferred embodiment of the present invention, images are provided in color by a pair of color display devices 402L, 402R, such as a color CRT or color LCD. To support backward compatibility with existing devices, stereoscopic display devices 402L and 402R may be used with the Q3D system. Alternatively, the Q3D imaging system may be connected to an autostereoscopic display, such as a display that does not require the use of a 3D monitor, 3D TV, or 3D effect eyeglasses.

An observation system with two viewing elements 401R, 401L can provide a good 3D viewing perspective. The Q3D imaging system complements this observation with physical dimension information about the physical structure in the surgical scene. The stereoscopic viewer 312 used together with the Q3D endoscopic system can display Q3D information overlaid on a stereoscopic image of a surgical scene. For example, the numerical Q3D distance value "d_Instr_Trgt" between the device 400 and the target 410 may be displayed in the stereoscopic viewer 312, as shown in Fig.

A description of a video viewing system that can be used to overlay physical location and dimension information on a three-dimensional perspective of a surgical scene is disclosed in U. S. Patent Application Publication No. US 2012/0020547, filed September 30, 2011, Are provided in the paragraphs [0043] and the corresponding figures.

Quantitative three-dimensional physical information processing

9 is an exemplary flow chart illustrating a process for determining a quantitative three dimensional location of a physical target, in accordance with some embodiments. This process is described with reference to an endoscope having the image acquisition system 101C of the embodiment of Fig. The module 401 configures the controller 106 to acquire video data from the image sensor S _ij . Although the image sensor array 210 "images" the entire clock, different pixels in the image sensor array 210 and in different sensors may be illuminated by image projections from different object points in the clock I will understand. The video data may include, for example, color and light intensity data. Each pixel of each sensor may provide one or more signals that indicate the color and intensity of the image projected onto it. The module 402 configures the controller to systematically select targets in a selected region of interest within a physical world view. The module 403 configures the controller to start calculation of the target 3D coordinates (x, y, z) with the initial set values (x ₀ , y ₀ , z ₀ ). The algorithm then checks the coordinates for consistency by using image diversity data from all sensors (S _ij ) that receive the projected image of the target. The coordinate operation is refined in decision module 404 until an acceptable precision is reached. The determination module 404 also configures the controller to determine whether the currently computed physical location is sufficiently accurate. In response to determining that the currently computed position is not sufficiently accurate, control returns to module 403 to try another possible physical location. In response to determining that the currently computed position is sufficiently accurate, the module 405 configures the controller to determine whether the entire region of interest has been scanned. In response to determining that the entire region of interest has not been scanned, control returns to module 402 and another target is selected. In response to determining that the entire region of interest has been scanned, control proceeds to module 406, which configures controller 406 to assemble a three-dimensional model of the volume of interest image. The assembly of the 3D image of the target based on the three-dimensional information indicating the physical position of the target structures is known to those skilled in the art and need not be described here. Module 407 configures the controller to store the developed three-dimensional model using the physical position information determined for the plurality of targets for later review and manipulation. For example, a 3D model may be used later for surgical purposes, such as determining the size of an implant for a particular dimension of a patient's organ. In another example, when a new surgical instrument 101 is installed in the robotic system 152, recalling the 3D model to display on the display 110 to refer to the previous surgical scene for the new instrument May be required. Module 407 may also store the result of the alignment between the 3D visual representation and the Q3D model. Module 408 configures the controller to use physical position information determined for a number of targets to display a quantitative 3D view. One example of the Q3D view is the distance value "d_Instr_Trgt" shown in Fig.

Stereoscopic displays are known to produce three-dimensional illusions. However, an actual 3D display provides a three-dimensional image such as a holographic image or an image projected onto a curved surface. Generally, the 3D display allows the view to move to change the viewing perspective.

및

)(y 방향의 투영 시프트의 추정값들)이 다른 값들을 획득할 것이다. 아래에 제공되는 방정식에 따라,

및

는 픽셀 좌표[(Nx₁₁, Ny₁₁), (Nx₁₂, Ny₁₂), (Nx₁₃, Ny₁₃)]가 동일한 목표의 투영들로부터 발생하는 경우에는 동일해야 한다.10 is an exemplary flow chart illustrating certain details of a process, generally corresponding to module 402 of FIG. 9, in accordance with some embodiments. The module 402.1 configures the controller to acquire images of physical world scenes from all the sensors of the sensor array 210. The module 402.2 configures the controller to specify the region of interest from within the acquired scene. Module 402.3 configures the controller to search for the best match between scene images within the region of interest to identify pixel locations within different sensors that are illuminated by projection of the same target. As described below, best matching is achieved by shifting individual images from the sensor S _ij until it maximizes a two-dimensional cross-correlation function between the shifted image and the reference image ) Can be achieved. Reference picture may be a scene image received for example from a sensor (S _11). Module 402.4 configures the controller to identify candidate pixels illuminated by projection from the same target. The module 402.5 configures the controller to compute two or more pixel coordinates (N _x , N _y ) for the selected target to determine whether the candidate pixels are illuminated by projection from the same target. The determination module 402.6 determines whether the computed 2D pixel coordinate values indicate that the candidate pixels are illuminated by projection from the same target. The image diversity generated by observing the same scene with a plurality of sensors S _ij precisely identifies (N _x , N _y ) associated with a specific target in various individual images S _ij . For example, according to some embodiments, only three sensors (S _11, S ₁₂ and S _13), assuming a scenario simplicity only used, 2D pixel coordinates _{[(Nx 11, Ny 11)} , (Nx 12, Ny ₁₂ ), (Nx ₁₃ , Ny ₁₃ ) does not correspond to the projections of the same target onto [S ₁₁ , S ₁₂ and S ₁₃ ]

And

) (estimates of the projection shift in the y direction) will acquire different values. According to the equations provided below,

And

Should be the same in the case of occurrence from the pixel coordinates _{[(Nx 11, Ny 11)} , (Nx 12, Ny 12), (Nx 13, Ny 13)] is the projection of the same target.

및

가 대략 동일하지 않는 경우에는, 제어는 모듈(402.4)로 되돌아가, 센서 평면(S_ij) 상으로의 목표 투영들에 최상인 후보들을 정제(refining)한다. 상술한 바와 같이, 이상은 알고리즘의 단순화된 구현예일 뿐이다. 일반적으로, 도 10의 모듈(402.6)에 나타내진 바와 같이,

와

간의 차이의 놈(norm)은 모듈(402)이 그것의 반복(iteration)을 완료하기 위해 허용가능한 공차(

)보다 작아야 한다. 유사한 구속조건이 x 축의 대응하는 추정값

및

에 대해 충족되어야 한다. 연산된 2D 픽셀 후보값(N_x, N_y)이 후보 픽셀들이 동일한 목표로부터의 투영에 의해 조명된다는 것을 지시한다는 판정에 응답하여, 제어는 모듈(403)로 진행한다.

And

If not, control returns to module 402.4 and refines the best candidates for the target projections onto the sensor plane S _ij . As described above, the above is only a simplified implementation of the algorithm. Generally, as shown in module 402.6 of Fig. 10,

Wow

The norm of the difference between the tolerances of the module 402 to allow it to complete its iteration

). A similar constraint is applied to the corresponding estimate of the x-

And

. In response to determining that the computed 2D pixel candidate values (N _x , N _y ) indicate that candidate pixels are illuminated by projection from the same target, control proceeds to module 403.

It will be appreciated that each pixel acquires color and intensity information directly from a world scene. Also, according to the process, each pixel is associated with the coordinates (x, y, z) of the physical object in the world view projected onto the pixel. Thus, the color information, illumination information, physical location information, i.e., the location of the physical object onto which color and illumination are projected, may be associated with pixels in non-temporal computer-readable storage devices. Table 1 below illustrates this relationship.

Pixel identifier Color value Illumination value Position (x, y, z)

An example of determining Q3D information

Example of Projection Matching

FIG. 11 includes an array of sensors (S ₁₁ -S ₃₃ ) arranged to have a clock including an exemplary three-dimensional real-world scene including three exemplary objects, according to some embodiments, The sensor array 210 of FIG. As described above, each sensor S _{ij in the} array includes a two-dimensional pixel array with at least two pixels in each dimension. Each sensor includes a lens stack, which generates a separate optical channel that resolves the image onto a corresponding pixel array disposed within its focal plane. Each pixel acts as a light sensor, and each focal plane with a plurality of pixels serves as an image sensor. Each sensor (S ₁₁ -S ₃₃ ) with its focal plane occupies a certain area of the sensor array and the area of the sensor array occupied by the other sensors and focal planes. Suitable known image sensor arrays are disclosed in the aforementioned U.S. Patent No. 8,514,491 (filed on November 22, 2010) and U.S. Patent Application Publication No. US 2013/0070060 (filed Sep. 19, 2012).

According to some embodiments, the sensors include N _x and N _y . The total number of pixels in their x and y directions and the field of view angles ([theta] _x and [theta] _y ). In some embodiments, the sensor characteristics for the x and y axes are expected to be the same. However, in an alternative embodiment, the sensor has asymmetric x-axis and y-axis properties. Likewise, in some embodiments, all sensors will have the same number of pixels and the same viewing angle. The sensors are distributed over the sensor array 210 to be well controlled. For example, the sensors may be spaced apart by a distance, [delta], on the two-dimensional grid shown. The sensor arrangement pitch delta may be symmetrical or asymmetrical to the lattice.

In the embodiment shown in Figure 11, the sensors are sensors (S ₁₁ -S ₁₃₎ the upper row and the lower surface of the car, sensors (S ₂₁ -S ₂₃₎ is a middle row and the lower surface of the car, the sensor (S ₃₁ -S ₃₃ ) are arranged in the bottom row and the bottom surface is arranged in the square grid. Each sensor includes N columns of pixels and N rows of pixels. The rays generated by the light source, as indicated by the dashed lines, are reflected from each of the triangular first object, the spherical second object and the rectangular third object to each sensor of the imager array. For purposes of illustration, only the rays to the sensors in the upper row (S ₁₁ , S ₁₂ and S ₁₃ ) are shown. The light source may be, for example, non-structured white light or ambient light. Alternatively, the light source may provide light of a selected wavelength, such as within a visible light or infrared spectrum, or may be filtered to provide a selected wavelength (e.g., color) or a range of wavelengths (e.g., a range of colors) Can be divided. Rays will be appreciated that the reflection to the sensor (S ₂₁ -S ₃₃₎ from each of the target object as well. However, for simplicity of explanation, these other rays are not shown.

In accordance with modules 401 and 402.1, the sensors of sensor array 210 acquire images individually from a world view. Figure 12 is an illustration showing projections onto sensors S _ij (only S ₁₁ , S ₁₂ and S ₁₃ ) of three objects, according to some embodiments. Those skilled in the art will appreciate that the reflected rays incident on the sensors project images of objects within the clock. More specifically, the rays reflected from the objects in the clock that are incident on a number of different image sensors of the imager array receive a plurality of perspective projections of objects ranging from three-dimensional to two-dimensional, i.e., Resulting in different projections at each sensor. In particular, the relative positions of the projections of the objects shift from S ₁₁ to S ₁₂ , and from left to right when proceeding to S ₁₃ . The image sensor pixels illuminated by incident rays generate an electrical signal in response to incident light. Thus, for each image sensor, electrical signals of a certain pattern are generated by its pixels in response to reflected rays representing the shape and position of the image projection in the image sensor.

According to module 402.2, a region of interest is selected from a world scene. Fig. 13 is an explanatory diagram showing selection of a region of interest in a scene. Fig. In this example, both the triangular first object, the spherical second object, and the rectangular third object are within the selected region of interest. This step may be accomplished by receiving input from an operator, or it may be performed automatically, using a computer configured in some manner by software, or by a combination of operator input and automatic software controlled selection. For example, in some embodiments, the global scene may show an internal cavity of a human anatomical structure, and the objects may be internal organs or surgical instruments, or a portion thereof. The surgeon can receive real time visual images from within the lumen and view tissue areas of the human anatomical structure and a portion of the surgical instrument protruding in the body cavity. The surgeon may identify objects in the watch whose location information is to be determined through known techniques, such as telestration video markers. Optionally or in addition to such an operator request, an automated process such as an edge detection algorithm may be used to specify a region of interest (ROI).

In accordance with module 402.3, the best match is determined between scene images within the region of interest to identify the pixel locations of different sensors illuminated by the projections of the same target. 14 is an explanatory diagram illustrating additional details of the geometric offset geometry of images projected onto the sensors S ₁₁ , S ₁₂ and S ₁₃ , according to some embodiments. According to some embodiments, the image at the sensor S ₁₃ is considered a reference image and the projections of objects within the selected ROI are located within the sensor S ₁₂ for their position within the sensor S ₁₃ Lt; RTI ID = 0.0 > ( ₂₃ ) < / RTI > Similarly, the projection of the object in the selected ROI are offset to the right by an amount (σ ₁₃₎ of pixels in the sensor (S ₁₁₎ for their position within the sensor (S _13). Since the FOV observation axes of the sensors S ₁₂ and S ₁₁ are respectively offset to the right of the FOV observation axis of the sensor S ₁₃ (such as observation axes perpendicular to the planes of the sensors) S ₁₃ from the sensors S ₁₂ and S ₁₁ .

15 is an explanatory diagram showing projected images in the ROI in the sensors S ₁₁ and S ₁₂ shifted to the right to align with the projected images in the ROI in the sensor S ₁₃ , according to some embodiments. In the present example, the sensor S ₁₃ is designated to act as a reference sensor. It will be appreciated that other sensors may be selected for use in determining alignment and geometric dimensions. And identified in the given sensor, such as a projection can, for example sensors (S ₁₃₎ of the target object in the selected ROI, for example, the projection of the other sensors, such as sensors (S ₁₁ and S ₁₂₎ are, until they are aligned with the projection of the specified sensor Shifted. In this manner, corresponding projections of objects in the selected ROI can be identified in different sensors with their offsets to the position of projections in the designated sensor.

Amount within Specifically, for example, of the three exemplary object projected to the sensor (S ₁₂₎ is shifted in the right by an amount (σ ₂₃₎ pixels, the three exemplary object projected to the sensor (S ₁₁₎ (? ₁₃ ) pixels. In this description example, for the sake of simplicity, it is assumed that the projections are offset only in the y direction and not in the x direction, but the same principle applies to the x direction projection offsets between the sensors. This example also shows linear offsets, but one skilled in the art can apply to other transforms, for example rotation, to align projections having relative offsets in other sensors.

For example, according to some embodiments, a two-dimensional (2D) cross-correlation technique or principal component analysis (PCA) aligns projections in ROI within S _{13 with} projections in ROI within S ₁₂ , And can be used to align the projections in the ROI in S _{13 with} the projections in the ROI in S ₁₁ . In general, the intent is to best match or align images from sensors (S _ij ) with images from sensors designated as reference sensors. More specifically, the projected images within the ROI within S ₁₂ are shifted and cross-correlated with the projected images within the ROI within S ₁₃ until the highest correlation coefficient is achieved. Likewise, the projected images within the ROI within S ₁₁ are shifted and cross-correlated with the projected images within the ROI within S ₁₃ until the highest correlation coefficient is achieved. Thus, the alignment of the projections of the ROI determines the offset between the projection in the ROI in S ₁₃ and the projection in ROI in S ₁₂ , and determines the offset between the projection in ROI in S ₁₃ and the projection in ROI in S ₁₁ , ≪ / RTI > S ₁₁ and S ₁₂ ).

Example of candidate pixel selection and refinement

In accordance with module 402.4, candidate pixels are identified in different sensors that are illuminated by projections from the same target by the best matching process. Once the projections of objects in the ROI are identified in each of the sensors (S ₁₁ , S ₁₂ and S ₁₃ ) in the ROI, the physical (x, y, z) projections of each target point in the ROI can be determined for the array . According to some embodiments, for each of a plurality of target points in the ROI, one or more pixels in each sensor of a plurality of sensors illuminated by projections from a target point are identified. For each of these target points, the physical (x, y, z) target point position is determined based at least in part on the geometric relationships between the pixels disposed in different sensors determined to be illuminated by the projections from the target point.

(X, y, z) can be automatically selected by systematically traversing the ROI (e.g., from right to left with a certain step size and from top to bottom with a particular step size) A target point position can be determined for each selected target point. Since S ₁₁ and S ₁₂ are best matched against S ₁₃ , the transverse movement is performed within the region of interest to be shifted. Selecting a target includes identifying pixels in each sensor of the sensors (S ₁₁ , S _12, and S ₁₃ ) illuminated by the projection of the target. Thus, the candidate pixels in each sensor of the sensors S ₁₁ , S ₁₂ and S ₁₃ are identified as being illuminated by the projection of the selected target point.

In other words, in order to select the target point T, a pixel illuminated by the projection of the target point T is selected in each of the sensors S ₁₁ , S ₁₂ and S ₁₃ . It will be appreciated that the (x, y, z) physical location of target (T) is not known at the moment of its selection. In addition, the inaccuracy of the alignment process described above can lead to inaccuracies in the determination of which pixels in each sensor are illuminated by the projection of the selected target T. Thus, as described with reference to Figs. 17, 18 and 19, the determination of the pixels in each sensor of the sensors S ₁₁ , S ₁₂ and S ₁₃ illuminated by the projection of the currently selected target T An additional determination of accuracy is made.

및

가 모듈(402.5)의 일부로서 연산된다. 모듈(402.6)의 일부로서, 놈

이 연산되어, 허용가능한 공차(

)와 비교된다. 마찬가지로, x 축 픽셀 좌표들 및 위치 추정값이 연산되어 허용가능한 공차들과 비교된다. 모듈(402.6)의 조건이 충족되면, 프로세스는 모듈(403)로 진행한다. 그렇지 않으면, 프로세스는 목표 후보들을 정제하기 위해 모듈(402.4)로 되돌아간다.Continuing with the description of the example above, let us assume that the triangle first object is the currently selected target point. 16 is an explanatory diagram illustrating projections of selected triangular target points onto sensors S ₁₁ , S ₁₂ and S ₁₃ , according to some embodiments. From these projections, the 2D pixel coordinates [(Nx ₁₁ , Ny ₁₁ ), (Nx ₁₂ , Ny ₁₂ ), (Nx ₁₃ , Ny ₁₃ )] for the target T are determined. For simplicity, FIG. 16 shows only y-axis pixel coordinates. Using these 2D pixel coordinates, equations (402.5-1) and (402.5-2) are applied,

And

Is calculated as part of module 402.5. As part of module 402.6,

Is calculated so that the allowable tolerance (

). Likewise, the x-axis pixel coordinates and position estimate are computed and compared to acceptable tolerances. If the condition of module 402.6 is met, the process proceeds to module 403. Otherwise, the process returns to module 402.4 to refine the target candidates.

17, a selected triangular first object target point T disposed at a position (x, y, z) in a portion of the imager array and the physical space including the sensors S ₁₁ , S ₁₂ and S ₁₃ Respectively. The sensors in the imager array have a known spacing (δ _ij ) between them. The physical position spacing between S ₁₁ and S ₁₂ is δ ₁₂ , and the physical position spacing between S ₁₂ and S ₁₃ is δ ₂₃ . In some embodiments, the spacing between all sensors S _ij is the same structural specification as?. The sensor S _ij also has a known clock angle [theta].

As described above, in some embodiments, each sensor is configured as a 2D imaging device having pixels arranged in rows and columns of a square pattern. Alternatively, the pixels may be arranged in an irregular pattern including, for example, a circular pattern, a zigzag pattern, a scattered pattern or a sub-pixel offset. The angular and pixel characteristics of these elements may be the same or may be different for each sensor. However, these properties are assumed to be known. Although the sensors may be different, for simplicity of explanation, the sensors are assumed to be the same.

For the sake of simplicity, it is assumed that all sensors S _ij have N x N pixels. At a distance z from the sensor S ₁₁ , the N pixel width of the sensor is extended to the y-dimensional clock of the sensor S ₁₁ indicated by FOV ₁ . Similarly, in the sensor distance (z) from the (S _12), y-D clock of the sensor (S ₁₂₎ is indicated by FOV _2. Further, in the distance (z) from the sensor (S _13), y-D clock of the sensor (S ₁₃₎ is indicated by the length (FOV _3). The lengths FOV ₁ , FOV ₂ and FOV ₃ are superimposed on each other so that the sensors S11, S12 and S13 can measure the ternary sampling diversity 3 of the target T physically located at some (unknown) -way sampling diversity. Of course, if the sensors are configured identically, as assumed in this example, the lengths FOV ₁ , FOV ₂ and FOV ₃ will also be the same. Although the three lengths FOV ₁ , FOV ₂ and FOV ₃ all have the same size and are depicted as if they were stacked adjacent to one another for illustrative purposes, they are at some (unknown) distance (z) It will be understood that it is located on the same plane in that it is located.

Referring to Fig. 18, an exemplary front view of the projection onto the image sensors S ₁₁ , S ₁₂ and S _{13 of the} currently selected target point T is shown. For simplicity, it is assumed that the sensors include a geometric square pixel array of size N x N pixels. It is also assumed that the x coordinates of the target (T) projection are all the same. In other words, for the projections of the target T onto the sensors S ₁₁ , S ₁₂ and S ₁₃ , it is assumed that n _x1 = n _x2 = n _x3 . In order to simplify the explanation, it is also assumed that the geometric clock angle (?) Is the same when the horizontal direction and the vertical direction are the same, that is,? _X =? _Y. Those skilled in the art will know how to modify the process provided below to compute the x, y, and z physical coordinates of target (T) when any of the above assumptions are altered.

The single image of the target (T) is projected onto a single physical point in the image sensor of the sensor (S ₁₁₎ located on the geometric coordinates (n _x1, n _y1) in a plane of a (S _11). More specifically, the projection onto the sensor (S ₁₁₎ of the target point (T) is along the y-axis as seen from the origin point to the pixel n _y1 and _x1 are located in the n pixels along the x-axis. One image of the target T is projected onto one physical point in the sensor S ₁₂ located at the geometric coordinates n _x2 , n _{y2 in} the plane of the image sensor S ₁₂ . One image of the target T is projected onto one physical point in the sensor _S13 located at the geometric coordinates n _x3 and n _{y3 in} the plane of the image sensor S ₁₃ . It will be appreciated that the pixel location (n _xi , n _{y i} ) in each sensor is determined for the origin (0, 0) reference coordinates provided for the sensor. As shown in Fig. 17 or 19, a global coordinate system (x, y, z) is defined and used as a reference for the target. For example, the origin of this coordinate system may be located in the geometric center of the sensor (S ₁₁₎ (without limitation).

16 and 18, it can be seen that the y pixel distance of the projection of the target is different in each sensor. The projection of the currently selected target T is located at n _y1 pixels to the left of the origin at S ₁₁ . The projection of the currently selected target T is located at n _y2 pixels to the left of the origin at S ₁₂ . The projection of the currently selected target T is located at n _y3 pixels to the left of the origin at _S13 . As described above, to simplify the explanation, it is assumed that the projection of the target is located at the same x pixel distance from the origin in all three sensors.

19, the arrangement for the sensors S ₁₁ , S ₁₂ and S ₁₃ of the currently selected target T as described above with reference to FIG. 17 is shown, and also for the candidate pixels in each sensor and y-direction pixel offsets. The diagram of FIG. 19 provides a physical structure and an analysis framework for determining the (x, y, z) physical coordinates of the selected target point T. FIG. At a distance (z) (unknown) from the imager array plane, the y-directional clock for each sensor extends over the length denoted FOV _i . This length FOV _i corresponds to the maximum pixel width of the sensor, which in some embodiments is N pixels. Considering the operational assumption that the sensor has a clock that is symmetrical in the x and y directions, its length will be FOV _i also in the vertical direction along the x axis.

Recall that a candidate pixel selection is made based on a correlation process that may have an uncertainty that may result in inaccuracies in determining the physical location of the at least partially selected target. Thus, in accordance with some embodiments, an additional check of the accuracy of the target projection candidate selection is made as follows.

Examples of physical (x, y) positioning of targets and correctness checking of target projection candidates selection

According to module 402.5, two or more two-dimensional (N _x , N _y ) coordinate values are computed for the selected target to determine whether the candidate pixels are actually illuminated by projection from the same target. Based on the above assumptions, leaving the origin of the 3D coordinate system in the center of the sensor (S _11), Example imager array and the target (T) are selected in Figure 19 have the following relationship.

here:

N is the pixel dimension of the image sensors;

n _{x 1} is a position in the x direction expressed by the number of pixels from the origin of the S ₁₁ plane of the target point T;

n _y1 is a position in the y direction expressed by the number of pixels from the origin of the S ₁₁ plane of the target point T;

n _x2 is a position in the x direction represented by the number of pixels from the origin of the S ₁₂ plane of the target point T;

n _y2 is a position in the y direction expressed by the number of pixels from the origin of the S ₁₂ plane of the target point T;

θ is the clock angle.

When the same equations are executed using the sensors S ₁₁ and S ₁₃ , the following relation is obtained in consideration of the fact that the interval between S ₁₁ and S ₁₃ is 2δ.

here:

n _x3 is a position in the x direction expressed by the number of pixels from the origin of the S ₁₃ plane of the target point T;

n _y3 is a position in the y direction expressed by the number of pixels from the origin of the S ₁₃ plane of the target point T.

Thus, the determination of the physical x-coordinate of the selected target T can be determined based on equation (3) or (6). The determination of the physical y-coordinate of the selected target T can be determined based on equation (2) or (5). The determination of the physical z-coordinate of the selected target T can be determined based on equation (1) or (4).

_y)보다 더 크게 다를 경우에는, 정합 프로세스가 충분한 정밀도로 상이한 센서들 내에서의 투영들 간의 오프셋을 해소하는 데 실패하였고, 그 결과 후보 픽셀들이 동일한 목표(T)로부터의 투영들을 수취하지 못하므로 부합하지 못한다는 판정이 내려질 수 있다. 정합하기 위한 y 연산에 실패한 경우에는, 정합 프로세스의 또 다른 반복이 각각의 후보 픽셀이 선택된 목표에 대응되는 센서들 내의 후보 픽셀들의 개선된 선택을 이루려는 노력의 일환으로 실행될 수 있다. 상이한 센서들 상으로의 상이한 투시 투영들이 예컨대 시차 효과(parallax effect) 등으로 인해 다를 수 있기 때문에, 연산된 y 값들이 동일할 가능성이 적다는 것이 이해될 것이다. 따라서, 허용가능한 공차값이 의도된 적용처에 따라 규정된다. 수술 영상 적용처에 대해서는, 일반적으로 0.1 - 0.3 mm의

가 허용가능한 Q3D 정확도를 제공한다. 당업자는 본 발명의 기술사상에서 벗어나지 않고 다양한 허용가능한 공차 레벨들을 정할 수 있을 것이다.More generally, according to module 402.6, a determination is made whether the computed 2D coordinate values indicate that the candidate pixels are illuminated by projection from the same target. It will be appreciated that a more reliable determination of the physical (x, y, z) coordinates of the target T can be obtained through the use of two equations for each coordinate. For example, the y-coordinate for the target T may be determined using the positive formulas (2) and (5). The y coordinate values of the results computed using the two equations are compared to a specific allowable tolerance value (

_y ), the registration process fails to resolve the offset between projections in different sensors with sufficient precision, as a result of which the candidate pixels can not receive projections from the same target T It can be judged that it does not match. If the y operation to match fails, another iteration of the matching process may be performed as part of an effort to achieve an improved selection of candidate pixels in the sensors where each candidate pixel corresponds to the selected target. It will be appreciated that since the different perspective projections onto different sensors may differ due to, for example, the parallax effect, the calculated y values are less likely to be the same. Therefore, the allowable tolerance values are defined according to the intended application. For surgical imaging applications, it is usually 0.1-0.3 mm

Provides acceptable Q3D accuracy. Those skilled in the art will be able to determine various permissible tolerance levels without departing from the scope of the present invention.

Assuming the sensor symmetry of the x and y axis, one of ordinary skill in the art the determination of the same type by using the n _y1 instead of n _x1 using similar formulas with Formula (2) and (5) for the x-coordinate of the target (T) Lt; / RTI > Equations (3) and (6) can not be used in parts of modules 402.5 and 402.6 because they require knowledge of z coordinates. However, the nature of the modules 402.5 and 402.6 is to determine the correct target projections to the planes of the sensors S ₁₁ , S ₁₂ and S ₁₃ . To this end, equations (2) and (5) adjusted for the x and y axes are sufficient. As described below, the complete set of coordinates (x, y, z) is the computed fraction of modules 403 and 404.

Example of determining the physical z position of a target

_x) 내의 대략 동일한 값을 연산할 것이다.As shown in FIG. 19, according to modules 403 and 404, an initial estimate z ₀ for the z coordinate is used to start the computation process. This initial value is automatically determined according to the medical application. The medical application defines the intended world view to be visualized. The initial value z ₀ starts at the edge of the watch that is close to the home on the endoscope. Referring to Figure 8, for the Application The Q3D, including endoscopic surgery, z ₀ may for example 1-5mm gotil away from the distal end 208 of endoscope Q3D 202. These initial estimates are generally sufficient for this application because it is unlikely that any tissue or surgical instrument will be so closely related to Q3D endoscopy. Next, the value z ₀ is substituted into the expressions (3) and (6). Considering that the x coordinate of the target is unique, if z ₀ is the actual exact z coordinate of the target, equations (3) and (6)

_x ). &_lt; / RTI >

_x) 밖에 있으면, 반복(iteration)이 이어져, z에 대한 새로운 추정값 z₁이 시도된다. 일부 실시예에 따라, 새로운 추정값은 자동적으로 정해진다. 예컨대, z₁ = z₀ + D, 여기서 Δ는 반복 단계의 크기이다. 일반적으로, k번째 반복에서는 z_k = z_k-1 + Δ. 이 반복 프로세스는 조건 (7)이 충족될 때 중지된다. Δ가 작을수록 정확한 목표 좌표를 결정함에 있어서의 정확성의 증가를 낳지만, 프로세스를 완료하는 데 더 많은 연산 시간, 그에 따른 증가된 대기 시간도 필요로 할 것이다. 증가된 대기 시간은 수술 기기 운동과 조작하는 외과의에 의한 그것의 시각화 사이에 지연을 초래할 수 있다. 다시 말해, 외과의는 명령에 뒤처져 시스템을 지각할 수 있다. 20-30 cm 깊이의 수술 관찰 공간에 대해서는, 0.1-0.3 mm의 Δ가 충분할 수 있다. 물론, 당업자는 반복 프로세스를 완료하는 데 필요한 연산에 대해 Δ의 크기를 조정할 줄 알 것이다.If the computed values of equations (3) and (6)

_x ), the iteration continues and a new estimate z ₁ for z is attempted. According to some embodiments, the new estimate is automatically determined. For example, z ₁ = z ₀ + D, where Δ is the size of the iteration step. Generally, z _k = z _k-1 + Δ in the _kth iteration. This iterative process is stopped when the condition (7) is satisfied. Smaller? Results in an increase in accuracy in determining the correct target coordinates, but will require more computation time to complete the process, and thus increased latency. The increased waiting time can result in a delay between the surgical instrument movement and its visualization by the operating surgeon. In other words, the surgeon can lag behind the command and perceive the system. For a surgical observation space of 20-30 cm deep, a Δ of 0.1-0.3 mm may be sufficient. Of course, one of ordinary skill in the art would know how to adjust the magnitude of [Delta] for the operation required to complete the iterative process.

The above description has been simplified for the sake of explanation, and thus includes only three sensors S ₁₁ , S ₁₂ and S ₁₃ . In general, more sensors can be used to increase the accuracy of the Q3D coordinate operation and also to reduce the total number of iterations. For example, if more than three sensors, preferably a 3x3 sensor array, are used, methods such as steepest gradients can be used to determine trends in the direction of the estimated errors produced by modules 402.5 and 403 Can be employed to determine. Then, the iterative step size and orientation can be adjusted to match the progression towards the local extreme of the 3D error sphere back surface.

Endoscopic surgery guide by Q3D information

20 is an exemplary flow chart illustrating a first process 2000 for using Q3D information during a surgical procedure according to some embodiments. The computer program code configures the computer 151 to execute the process 2000. The module 2002 configures the computer to receive user input for selecting at least two objects within the surgeon's watch when looking into the viewer 312. Module 2004 configures the computer to display a menu on the computer console in response to receiving a user selection. The determination module 2006 configures the computer to determine whether a user input to the menu is received to indicate a distance. In response to determining that the user input is received to indicate the distance, the module 2008 configures the computer to display a numeric distance within the video image within the surgeon's watch. The determination module 2010 waits for a predetermined time interval for receipt of user input to select the distance indication and determines the determination module 2006 in response to the absence of user input within the "time out & And terminates the operation of the computer.

The determination module 2012 configures the system to determine whether the user input to the menu is received to enter proximity alarm limits. In response to determining that the user input is received to enter a proximity threshold, module 2014 configures the computer to use Q3D information to monitor proximity between two or more objects within the surgeon's watch. The determination module 2016 determines whether the proximity threshold has been exceeded. In response to determining that the proximity threshold has been exceeded, the module 2018 configures the computer to trigger an alert. Alarms may include visual queues such as sounds, blinking lights, locking of machine motion to avoid collisions, or other haptic feedback. In response to determining that the proximity threshold has not been exceeded, control returns to monitor module 2014 again. The decision module 2020 waits for a prescribed time interval for receipt of user input to enter a proximity threshold and responds to the absence of user input within a "time out" 2012 are terminated.

FIG. 21 is an illustration showing menu selections displayed on display screen 2012 in accordance with the process of FIG. 20, in accordance with some embodiments. The display screen 2102 includes an observation monitor associated with the computer 151. Alternatively, the display screen 2102 may include an area of the viewing elements 401R, 401L of the viewer 312. [ In response to user input, the module 2004 causes an indication of a menu 2104 that includes a first menu item "distance display" 2106 and a second menu item "proximity alarm setting" 2108. In response to a user input selecting the "Distance Display" menu item 2106, the module 2008 causes an indication of the Q3D distance between two or more objects. Referring again to Fig. 4, an indication of the Q3D distance "d_Instr_Trgt" between the displayed device 400 and the target using the module 2008 is shown. In response to a user input selecting the "Proximity Alarm Settings" menu item 2108, a "Distance Input" UI input (see FIG. 4), including a field in which a user can enter a proximity distance threshold 2110) is displayed. In one alternate embodiment (not shown), a default proximity threshold may be preset for all devices, and the user may use the menu of Figure 21, for example, You can change the value. In this alternative embodiment, the user may choose to choose a default threshold instead of entering a threshold. In some embodiments, the user can select both to display the distance and to set the proximity alarm.

22a-22b are explanatory diagrams illustrating certain details of receiving user input in accordance with the process of Fig. 20, in accordance with some embodiments. FIG. 22A illustrates an example of a target, such as a body tissue, that can be created using a video marker tool, such as telestration, or using a surgical console to manipulate the control input device 160 of FIG. The first highlighting areas 2202L and 2202R of the example of the first and second highlighting areas 410L and 410R. FIG. 22B shows a second highlighting area 2206L, 2206R of an example of device tips 400L, 400R that may be created using a video marker tool. In operation according to some embodiments, the user creates a first highlighting area 2202L, 2202R. Next, the user uses the video marker tool to create the second highlighting areas 2206L and 2206R of the example of the device tips 400L and 400R. It will be appreciated that the order in which the items are highlighted is not critical. The user then actuates a selector (shown in the figure) to enter a selection (e.g., presses the Enter key). The module 2002 interprets the received user input as the selection of the target images 410L and 410R and the device images 400L and 400R.

23 is an exemplary flow chart illustrating a second process 2300 for using Q3D information during a surgical procedure, in accordance with some embodiments. The computer program code configures the computer 151 to execute the process 2300. The module 2302 configures the computer to receive user input for selecting at least a subject within a surgeon's watch when looking into the viewer 312. For example, referring again to FIG. 22B, the user input is received using the video marker tool to generate the second highlighting areas 2206L, 2206R of the device tips 400L, 400R. A user input (not shown) is received (e.g., presses the Enter key) to activate a selector (not shown) for inputting a selection of images of device tips 400L and 400R.

Returning again to Figure 23, in response to receiving the user selection, the module 2304 configures the computer to display a menu on the computer console. The determination module 2306 configures the computer to determine whether the user input to the menu is received to rotate the image of the selected object. In response to determining that the user input is received to rotate the image, the module 2308 configures the computer to display the rotation of the image to show another three-dimensional perspective of the object. The determination module 2310 waits for a prescribed time interval for receipt of user input to rotate the image and determines whether the determination module 2306 is responsive to an inactivity of user input within a "time out & The computer is configured to end the operation.

FIG. 24 is an illustration showing menu selections displayed on the display screen 2402 in accordance with the process of FIG. 23, in accordance with some embodiments. The display screen 2402 includes an observation monitor associated with the computer 151. Alternatively, the display screen 2402 may include an area of the viewing elements 401R, 401L of the viewer 312. [ In response to the user input, the module 2304 causes an indication of a menu 2404 that includes a third menu item "left turn" 2406 and a fourth menu item "turn right" 2408. In response to a user input selecting one or the other of the third menu item or the fourth menu item 2406 or 2408, the module 2308 generates a rotation of the 3D model created and stored according to the module 407 of FIG. 9 . It will be appreciated that since the imager sensor array 210 has a limited overall clock, the amount of rotation can be limited to some degree, for example less than 30 degrees.

The haptic feedback provided by the Q3D endoscopic system

Haptics generally describe vibration and movement, as well as touch feedback, which may include kinesthetic (force) and skin (tactile) feedback. In manual minimally invasive surgery (MIS), surgeons feel the interaction of the device and the patient through the long shaft, which removes the tactile cue and mask force cue. In remote operated surgical systems, natural haptic feedback is largely eliminated because the surgeon no longer manipulates the device directly. One goal of the haptic technique in remotely operated minimally invasive surgery is not to feel that the surgeon is operating a remote device, but rather to provide "transparency" in which both hands feel as if they are in contact with the patient. Another way to explain this concept is haptic telepresence. Generally, this requires artificial haptic sensors on the patient side device to acquire haptic information and a haptic display on the surgical side to deliver the sensed information to the surgeon. The muscle kinesthetic or force feedback system generally measures or estimates the forces exerted on the patient by the surgical instrument and provides the resolved forces through the force feedback device to the hand. Tactile feedback can be used to provide information such as local tissue strain and pressure distribution across the tissue surface. Tactile feedback may include palpation, for example, where the surgeon uses the fingertips to detect tissue mechanical properties such as compliance, viscosity, and surface texture, which are indicators of tissue health. Lt; / RTI > Journal of Current Opinion in Urology, 2009, Vol. 19 (1), pp. Okamura Ai on 102-107. See Okamura A. M., "Haptic Feedback in Robotic Assisted Minimally Invasive Surgery (Okamura A.M.)".

The deformation of a tissue structure having a known stiffness can be determined as a function of the force applied thereto and the force applied to a tissue structure having a known stiffness can be determined as a function of the tissue structure deformation. The shape of the tissue structure is deformed during surgery in response to movement caused by the surgeon. Soft tissue deformations can be approximated in terms of linear elasticity and contact forces that cause tissue deformations can be approximated based on the resulting tissue deformations. In general, an organizational structure can be divided into a set of finite elements of similar size. Due to geometric irregularities in the anatomical models, decomposition of the tissue volume into tetrahedral elements may be a desirable approach. Thus, the configuration of the target tissue structure can be modeled by decomposing it into a set of polygonal (e.g., tetrahedral) elements.

In general, tissue deformations are related to forces according to the following equation:

[ K ] u = f (8)

Where [ K ] is a stiffness matrix, a symmetric quantity limited sparse matrix; u is the displacement field; f is an external force. The size of this matrix is 3N × 3N when N is the number of mesh vertices. Assuming at least some degree of tissue isotropy, the matrix K may be expressed as an NxN matrix.

More specifically, the external force applied to the surface of the deformable object based on the deformation of the object can be determined based on the following formula, in which the force required to obtain the displacement ( u *) of the deformable object can be determined.

(9)

(9)

을 최소화하기 위해 사용될 수 있다. 시스템을 푼 후에 얻어지는 [λ]의 값 λ_i는 변위 u _i = u _i *를 부과하기 위해 자유도(u _i )에서 가해질 것이 필요한 힘의 역원(opposite)과 같다. 여기에 그 전체 개시내용이 참조되는 학술지 IEEE Transactions on Visualization and Computer Graphics (Impact Factor: 1.9), 01/1999; 5:62-73, DOI: 10.1109/2945.764872에 실린 코틴 에스(Cotin, S.)등에 의한 "수술 시뮬레이션을 위한 연성 조직의 실시간 탄성 변형(Real-time elastic deformations of soft tissues for surgery simulation)" 참조. 조직 강성도 값들은 공지되어 있거나, 용이하게 결정될 수 있다. 예를 들어, 여기에 직접적으로 참조되는 Am. J. Roentgenology, 2002;178:1411-1417에 실린 맥나이트 에이. 엘.(McKnight A.L.) 등에 의한 "유방암의 MR 탄성초음파: 예비 결과(MR elastography of breast cancer: preliminary results)", NMR in Biomedicine, 2004;17:181-190에 실린 우프만 케이.(Uffmann K.) 등에 의한 "MR 탄성초음파에 의한 사지 골격 근육의 생체내 탄성 측정(In vivo elasticity measurements of extremity skeletal muscle with MR elastography)", IEEE Transactions on Biomedical Engineering, 2006;53:2129-2138에 실린 테이 비. 케이.(Tay B.K.) 등에 의한 "복강내 기관의 생체내 기계적 거동(In vivo mechanical behavior of intra-abdominal organs)"이 다양한 조직 구조부에 대한 조직 강성도의 보고를 제공한다. 이들의 데이터 및 그 밖의 데이터가 방정식 (8)에 의한 힘 연산에 적합한 행렬 [K]를 결정하는 데 사용될 수 있다.Here, [ K ] is a matrix composed of vectors ( e _i ) having 1 at the i-th position and 0 at the other positions. The system is created from the Lagrangian multiplication method, which is used to impose values that are specific to a pool of variables. Variance formulas and Lagrange multipliers

Can be minimized. The value of [λ] is obtained after the system loosen λ _i is equal to the degrees of freedom (u _i) inverse (opposite) of the force needed to be applied in order to impose the displacements u _i = u _i *. Herein, the entire disclosure is referred to in the journal IEEE Transactions on Visualization and Computer Graphics (Impact Factor: 1.9), 01/1999; 5: 62-73, DOI: 10.1109 / 2945.764872 by Cotin, S. et al., "Real-time elastic deformations of soft tissues for surgery simulation " The tissue stiffness values are known or can be readily determined. See, for example, Am. J. Roentgenology , 2002; 178: 1411-1417. Quot; MR elastography of breast cancer: preliminary results ", by McKnight AL, et al., Uffmann K., in NMR in Biomedicine , 2004; 17: 181-190. "In vivo elasticity measurements of extremity skeletal muscle with MR elastography", IEEE Transactions on Biomedical Engineering , 2006; 53: 2129-2138. &Quot; In vivo mechanical behavior of intra-abdominal organs ", by Tay BK et al., Provides a report of tissue stiffness for various tissue structures. These data and other data can be used to determine a matrix [K] suitable for force calculations according to equation (8).

25 is an exemplary flow chart illustrating a process 2500 for using Q3D information to determine haptic feedback, in accordance with some embodiments. The process 2500 may use one or more of the systems 152 and the Q3D endoscope system 101C. The system 152 of Fig. 5 or 6 computes the force vector f applied to the tissue by the device by the formula (8) or (9) with the Q3D endoscope system 101C of Fig. The stiffness matrix [K] may include known tissue stiffness values. Once computed, the force exerted on the tissue by the device may be displayed or used to drive the haptic interface device.

Referring to Fig. 25, the module 2502 acquires Q3D information of a surgical scene used to generate a Q3D model of a surgical scene. Examples of scenes include tissue structures and surgical instruments. Module 2504 identifies and registers surgical instruments within the scene. The module 2506 uses the Q3D information to determine the tissue displacement ( u ) around the device when the device is pressed against the surface of the tissue structure in the scene with a constant force. Module 2508 obtains the tissue stiffness matrix information [K] suitable for the tissue type of the tissue structure in the scene. The module 2510 uses the information of the tissue displacement information u and the tissue stiffness matrix determined to determine the force f . Module 2512 overrides the visual indication of the determined force f in the surgeon's watch within the surgical scene. Module 2514 actuates the haptic interface to provide the surgeon with haptic feedback indicative of the determined displacement u and the determined force f .

26 depicts a tissue structure 2602 that is touched by an end effector portion of a surgical instrument 2604 and a tissue structure 2602 that is configured to acquire Q3D image information about the contact between the tissue structure 2602 and the instrument 2604, And an endoscope 2606 having an image sensor array arranged so as to have a clock (FOV). It will be appreciated that the endoscope is also used to acquire stereoscopic visual images observable by a surgeon during surgery. During the remote operation procedure, the surgeon views the surgical scene through the stereoscopic display device 164 while manipulating the control input device 160 shown in FIG. 5 to manipulate the end effector portion of the device to contact the surface of the target tissue structure. do. The surgeon controls the movement of the device to impose a force on the tissue surface at a given location that causes a deformation of the tissue surface contour at that location.

The Q3D information is used to quantify the level of haptic feedback that will be provided to the surgeon to indicate strain and / or force applied by the device 2604 to the tissue surface 2608. More specifically, the dimensions of the displacement 2610 of the tissue surface 2608 determined using the Q3D information are used to generate the haptic feedback. Assuming that the tissue region of interest, or at least its subregion, has a substantially uniform stiffness, the larger the displacement strain, the greater the force the surgical instrument exerts. Using the displacement u measured by the Q3D endoscope system 101C of Fig. 8 for a known stiffness matrix K according to equation (8) The force vector f may be computed or approximated. Likewise, from the force vector f, the resistance provided by the tissue can be computed or approximated. This information can be useful for tissue palpation applications. Thus, in accordance with some embodiments, haptic feedback can be provided to the surgeon that exhibits greater strain and / or greater force or tissue ductility. Conversely, smaller displacement deformations may indicate that the surgical instrument is applying less force or that the tissue provides increased stiffness and, in some embodiments, may have smaller deformations and / or smaller forces or higher Haptic feedback indicating tissue stiffness can be provided to the surgeon.

The tissue structure portion 2602 may not have a uniform rigidity. Local sub-surface features that are internal to tissue structures such as tumors or blood vessels that have a stiffness that is different from the stiffness of the tissue structure may alter the local stiffness of the tissue structure. The local sub-surface structures are referred to herein as " anomalies. &Quot; The presence of a variant can affect local deformation deformations of the tissue structure in response to forces exerted by the instrument on the tissue surface location that is overlaid on the variant. That is, the displacement deformation of the tissue surface position overlaid with the subsurface anomaly in response to a constant magnitude of force is different from the displacement deformation at other tissue surface positions that are not overlaid with the surface deformation in response to forces of the same magnitude will be. Thus, as a result of this variation in tissue stiffness, the application of substantially the same force to different locations of the tissue structure can result in different deformations at such different locations. These different variations may indicate the presence of a sub-surface variant.

To address the variability of the tissue stiffness matrix K, the surgeon may have to tap the device for surrounding tissue structures known to be free of deformations. This is an achievable goal, as it is customary to identify the location of the deformity using pre-procedure images. Such an image can guide the surgeon toward a peripheral area free of deformations. When the device 2604 is tapped to the area containing the variant, the Q3D endoscope system 101C provides quantitative information on a smaller amount of displacement u than predicted. As a result, a relative map of the tissue stiffness can be obtained. Alternatively, if the device 2604 of Figure 26 comprises a force sensor (not shown), a tissue stiffness map may be calculated or approximated based on equation (8). The force sensor provides the force vector f , and the Q3D endoscope system 101C provides the displacement u . If f and u are measured at sufficiently many points in the region of interest, the elements of the stiffness matrix K can be computed or approximated. For example, assuming tissue isotropy and symmetry, the matrix K has individual elements up to N * (N + 1) / 2. Thus, the displacement u and force vector f must be measured up to N * (N + 1) / 2 positions to fully characterize the matrix K. [ According to some embodiments, haptic feedback is provided to the surgeon indicating different tissue structural displacements at different tissue surface locations.

Haptic user interface

As used herein, the term "haptic user interface" refers to a user interface that provides haptic information to a user. There are many different kinds of haptic interfaces. For example, as described herein, a shape display, also referred to as a tangible user interface, may serve as a haptic interface that provides haptic information indicative of the shape or contour of the object. The haptic user interface can provide haptic information, for example, via selective expansion and contraction of the pins or through mechanical stimulation through a variable oscillatory intensity. The haptic user interface can provide haptic information, for example, through electrical stimulation of varying intensity.

Figures 27A-27C are illustrations depicting a first embodiment of a haptic user interface in the form of a sensible user interface (TUI) 2702 that serves as a shape display suitable for providing haptic feedback, in accordance with some embodiments. The TUI 2702 includes a housing structure 2704 that receives a plurality of pins 2706 that can be selectively displaced outwardly from the housing. Figure 27A illustrates a first embodiment of a TUI with a plurality of pins that are outwardly displaced to produce a three-dimensional model 2708 of a visible tissue structure within a viewable surgical scene through the viewer 312 of Figure 4. [ The configuration of the pins shown in Figure 27A corresponds to an initial TUI state without feedback to support deformation of the tissue structure. The three-dimensional model 2708 of the tissue structure created by the first embodiment of the TUI 2702 in its initial state represents the tissue structure in a rested state without surface forces that deform its shape. 27B shows the TUI of the first embodiment in a state in which the pin 2706-1 at the first (x, y) position is displaced inward (downward in the drawing) by a first amount relative to the initial state position of the pin. Figure 27c shows the TUI of the first embodiment with the second pin 2706-2 inwardly displaced by a second amount different from the first amount for the initial position of the second pin.

28A to 28C are explanatory views showing a tissue structure portion 2802 having a shape deformed by a force applied by the surgical instrument 2804. Fig. Figure 28A shows a natural tissue structure 2802 in a rest state in which the device 2804 is positioned so as not to contact the tissue structure. Figure 28 shows a device 2804 in contact with the tissue structure at a first tissue structure location with sufficient force to cause a deformation 2806-1 of the tissue structure by a first amount of displacement. Figure 28c shows a device 2804 in contact with the tissue structure at a second tissue structure location with sufficient force to cause a deformation 2806-2 of the tissue structure by a second amount of displacement. 27A-27C, a first embodiment 2702 of the TUI includes an array 2706 of axially aligned pins arranged in a Cartesian (x, y) plane. Each pin is associated with a (x, y) position in the array. The pins are axially movable between a rest position and a rising displacement of variable degrees. (Tops) define a flat planar (x, y) surface in the housing 2704 when all the pins 2706 are in a rested state, i.e., without any axial displacement. Each pin 2706 is associated with its own actuator (not shown) such that each of the pins 2706 can be independently raised, lowered or lifted axially with controllably variable amounts of displacement. According to some embodiments, in the initial state, a plurality of pins are arranged to produce a three-dimensional model of the tissue structure 2708. Variations in pin displacements for the initial state three-dimensional model provide haptic feedback representing corresponding changes in the shape of the modeled tissue structure. The three-dimensional model generated using the TUI of the first embodiment is geometrically sized to allow the surgeon to interact with the model in a manner similar to the palpation of the actual tissue structure during the medical procedure. The shape of the TUI tissue model 2708 is modified in response to deformation of the surface of the tissue structure 2802 of Figs. 28A-28C, which occurs through contact with the device 2804 under the remote control of the surgeon, Provides haptic feedback indicative of deformation of the tissue structure viewed through the viewer 312 during the surgical procedure. Actuation, UIST '13, October 8-11, St. Louis. InForm: Dynamic Physical Affordances and Constraints through Shape and Object (inFORM) via Follmer S. et al. In Andrews, UK to provide haptic feedback Discloses a 2.5D shaped display using a shape display, and such a TUI device can be modified to function as described herein. For example, the TUI 2702 may be driven by the Q3D endoscope system 101C according to a stiffness matrix K computed or approximated as described above. As a result, the surgeon will be provided with at least relative information on the tissue stiffness and will be allowed to perform indirect or virtual tissue palpation.

As another example, the surgeon manipulates the surgical instrument to provide a substantially constant force at various locations on the tissue structure. Constant force application can be adjusted by using a sensor or visibility gauge or by setting the remotely operated manipulator to apply force until a threshold force level is sensed. Optionally, force application can be accomplished by using a cognitive feel of the surgeon. The Q3D endoscope acquires measurements of tissue location and tissue deformation at various tissue locations in response to a substantially constant force application. The relationship of equation (8) can be used to determine different tissue stiffnesses at different positions in response to a substantially constant force application, and these positions and stiffnesses can then be mapped to the haptic TUI. The haptic interface provides surgical feedback indicating different tissue stiffnesses at different locations to provide a virtual palpation output to the surgeon. The surgeon can then track the finger or hand along the path through the TUI and sense the varying stiffness along the path to facilitate the simulated tissue structure in a manner similar to the operation of the surgeon on the actual tissue.

According to some embodiments, a tissue stiffness matrix K or a stiffness map may be mapped to (x, y) positions of individual TUI pins 2706. [ That is, the physical (x, y) location of a given pin 2706 is mapped to a location on the surface of the tissue structure 2802. For the purpose of this example, the TUI pin 2706-1 at the first (x, y) position of Figure 27B is mapped to the first tissue structure location 2806-1 of Figure 28B, and the second (x, Assume that the TUI pin 2706-2 at position (y) is mapped to the second tissue structure location 2806-2 in Fig. 28C.

Assuming that the TUI 2702 has W x L pins, the Q3D model of surface tissue structure 2802 may be meshed on a W x L grid. Then, each node of the mesh will correspond to one TUI pin. Assuming that the Q3D model provides the actual _tissue W _tissue and length of _{tissue tissue} , the corresponding (x, y) coordinates of the pins with TUI position counts N _W and N _L are (N _W / W * W _tissue , N _L / L * L _tissue ). On the other hand, it can be, calculated the TUI position count of the TUI fin corresponding _{N X = x * W / W} tissue and N _Y = y * as L / L _tissue about the coordinate point (x, y) of known Q3D model. This mapping is stored in a non-transient computer readable storage device (not shown). The plurality of pins 2706 of the TUI 2702 are raised outwardly (upward in the figures) to create a tissue model 2802 or a three-dimensional model 2708 of the tissue surface profile of at least the region of interest of the tissue structure.

The surgical instrument is registered at the mapped tissue surface locations. That is, the kinematic location of the device 2804 to the mapped locations of the tissue surface of tissue structure 2802 is registered and updated during the surgical procedure. Registration can be accomplished, for example, by identifying a target device within the FOV of the Q3D endoscope using an identification algorithm similar to that described in Figures 9 and 10. Thus, due to the registration of the device 2804 and the mapping between tissue positions and TUI positions, each contact between the surface location of the device 2804 and the tissue structure 2802 can be mapped to one or more TUI pin locations have. The image sensor array 210 also acquires Q3D information that can be used to determine, for each contact between the device and the tissue surface, the deformation displacement of the tissue surface due to this contact. The Q3D information is then used to determine the amount of change in the outward displacement of one or more fins 2706 mapped to the touched tissue surface location in response to contact by the instrument.

For example, Figure 27A illustrates a TUI 2702 of the first embodiment in a rest state when the surgeon moved the instrument proximate the tissue structure but did not actually use the instrument to contact the tissue structure, as shown in Figure 28A. Of FIG. Figs. 27B and 28B show the first (x, y) position in which the feedback in the operation of the surgical instrument causing the deformation of the first tissue position 2806-1 shown in Fig. RTI ID = 0.0 > TUI < / RTI > pin 2706-1. The amount of inward displacement of the pin 2706-1 at the first (x, y) position is measured by the Q3D endoscopic system 101C, and as long as the first tissue position 2806-1 of Fig. 28B is displaced by the instrument As shown in FIG. Likewise, Figs. 27C and 28C show how the feedback in the operation of the surgical instrument 2804 causing the deformation of the second tissue location 2806-2 shown in Fig. 28C, x, y) position of the TUI pin 2706-2. The change in the outward displacement of the pin 2706-2 at the second (x, y) position is determined based on the second amount of displacement as much as the second tissue position 2806-2 of Figure 28C is displaced by the instrument .

Alternatively, the tissue deformations sensed as Q3D information may be registered in kinematic information identifying the posture of the device tip (e.g., kinematic information identifying the posture of the kinematic arm holding the device). The combined tissue deformation information and instrument tip position and orientation information are then mapped to a TUI to create a haptic interface that enables the surgeon to palpate the tissue structure being simulated.

29A-29C illustrate alternative embodiments of a haptic user interface in the form of a sensible user interface (TUI) 2902 that functions as a shape display suitable for providing haptic feedback, in accordance with some embodiments. An alternative TUI embodiment 2902 includes a housing structure 2904 that receives a plurality of axially aligned fins 2906 that can be displaced outward (upward in the drawing) by a variable amount in response to displacement of the tissue surface do. 29A shows an alternative TUI 2902 in a rested state in which no pin 2906 has been raised. Figure 29B shows an alternative TUI 2902 with the pin 2906 raised by a first amount. Figure 29c shows an alternative TUI 2902 with the pin 2906 raised by a second amount greater than the first amount. In general, the pin configuration of the TUI 2902 of the alternative embodiment of Figures 29A-29C is similar to that of the TUI 2702 of Figures 27A-27C. However, the alternative embodiment TUI 2902 is smaller and geometrically sized to mount on the fingertip of the surgeon. Examples of such alternative TUIs 2902 are known. Journal of Neuroscience Methods, 2007, Vol. 161 (1), pp. 62-74. "A Dense Array Stimulator to Generate Arbitrary Spatio-Temporal Tactile Stimuli" by Killebrew JH et al., &Quot; on the fly " ) Discloses a 400-probe stimulator for generating an arbitrary space-time tactile stimulus. Each probe includes an individual motor for controlling its axial displacement. Proceedings of ICRA in Robotics and Automation, 2000, IEEE , Vol. &Quot; A Compliant Tactile Display for Teletaction " by Moy G. et al., Which is incorporated herein by reference, discloses a one-piece pneumatically actuated tactile display molded from silicone rubber. "Remote Palpation Technology" by Howe, Robert D. et al., Published in IEEE Engineering in Medicine and Biology , May / June, 1995, Discloses a tactile display in which a shaped alloy wire is used as an actuator.

Similar to the TUI 2702 of Figures 27a-27c, the shape of the TUI surface represented by the heads of the pins 2906 of the TUI 2902 of the alternate embodiment of Figures 29a-29c is, for example, And changes in response to deformation of the tissue structure caused by contact with the device. However, unlike the TUI 2702 of FIGS. 27A-27C, in alternative embodiments of FIGS. 29A-29C, the pin locations of the TUI of the alternative embodiment are not mapped to tissue surface locations. Instead, the haptic feedback produced by the outward displacement of the pin 2906 of the TUI 2902 of the alternative embodiment can be viewed simultaneously via the stereoscopic display device 164 and is currently contacted by the surgical instrument under the control of the surgeon Lt; / RTI > to the location of the tissue surface. For example, a center point of the TUI 2902 is mapped to correspond to an instrument tip such that a changing haptic sensation occurs centered on the TUI 2902 as the instrument tip moves. The surgeon places the finger on the TUI 2702 to perceive the change. This alternative TUI (2902), the haptic feedback size provided by the calculation using the calculated or approximated tissue displaced or bangjeoksik (8) provided by the Q3D endoscope system (101C), or the approximate force vector (f) .

As an alternative example, the pins are initially displaced axially outward by a predetermined distance, and the fins are driven to attempt to maintain this distance. A default stiffness is set so that when the finger is pressed against the top of the pins, the finger slightly displaces the pins slightly while the pins are driven to return to their default level. Thus, the pins give a "squishy" feel. When the device is subjected to one or more changes in stiffness across the tissue, this change in stiffness will cause the tissue with increased stiffness to be less "clumsy" (pins displaced axially toward the finger) The stiffness tissue is delivered to the TUI so that it feels more "cramped" (the pins are displaced axially away from the finger). In this way, the instrument tip functions as a remote palpation instrument with a sense of urgency delivered to the TUI. Likewise, in one alternative embodiment, the fins are initially set to an axial displacement equal to the height of the TUI surface (see, e.g., Figure 29A), and the tissue with reduced stiffness is felt to be stiffer in the TUI (The pins are displaced axially towards the fingers).

30 is an illustration depicting an alternative embodiment TUI 2902 mounted to a surgeon's finger 3002, in accordance with some embodiments. When mounted, the TUI 2902 of an alternative embodiment presses the finger 2906 of the surgeon 2906 as the pin 2906 is displaced outward in response to tissue displacement of the tissue surface. According to some embodiments, different amounts of pin displacement represent different tissue displacements of the currently touched portion of the tissue surface or different amounts of force exerted by the instrument on the currently touched portion of the tissue surface.

Referring now to FIGS. 28A-28C and 29A-29C, FIG. 29A shows the surgical instrument 2804 moved proximally to the tissue structure 2802, as shown in FIG. 28A, And shows an example posture of the TUI 2902 of an alternative embodiment in a dormant state when not in use. Figure 29B illustrates a first tissue position 2806-1 of Figure 28B so that a plurality of pins 2906 provide a first magnetic level to the user contacting the pins, Of the TUI 2902 of the alternate embodiment, which is raised by a displacement amount determined based on the position of the pin 2906 of the TUI 2902. Figure 29c illustrates the second tissue position 2806-2 of Figure 28c so that a plurality of pins 2906 provide a second magnetic level to the user contacting the pins, Of the TUI 2902 of the alternate embodiment, which is raised by a displacement amount determined based on the position of the pin 2906 of the TUI 2902. It can be seen that the amounts of tissue displacement at the first and second tissue locations 2806-1 and 2806-2 are different and thus the amounts of TUI pin displacement in response to the first and second tissue displacement amounts are also different.

Those skilled in the art will know how to replace the use of pins with other types of haptic stimuli. For example, sub-threshold electricotactile stimulation may be provided instead of stimulating the surgeon's finger 3002 using a mechanical actuating pin that abuts the hand or finger grip. Such a system would use a safe level of current provided by the Q3D endoscope system 101C that is slightly higher than the human perception level, which is regulated by the tissue displacement u or the magnitude of the tissue force f . Other forms of such haptic stimulation are known. For example, IEEE Transactions on Biomedical Engineering , Vol. 39, No. 7, July 1992, Kaczmarek et al., &Quot; Maximum Dynamic Range Electrosthaphone Stimulation Waveform ", presented techniques and waveforms for a suitable electrotactile stimulation referenced herein as a whole. In an alternative embodiment, the operating surgeon may be provided with haptic feedback that may vibrate or provide operational resistance to the control input device 160 of FIG. The magnitude of the vibration or manipulation resistance will be adjusted according to the magnitude of the tissue displacement ( u ) or tissue force ( f ) as provided by the Q3D endoscope system 101C of Fig. Discloses a control input device having tactile feedback, the entire contents of which are incorporated herein by reference, such as those disclosed in U.S. Patent No. 6,594,552 B2, filed April 6, 2000, and U.S. Patent No. 8,561,473 B2, filed March 30, 2009 Suitable implementations are known such as the < RTI ID = 0.0 >

Conversion of Q3D information to haptic feedback

31 is a diagram illustrating an exemplary computation block 3102 configured to determine haptic feedback as a function of tissue surface deformation, in accordance with some embodiments. According to some embodiments, haptic feedback is provided through variable pin axial displacement. Alternatively, for example, electrical stimulation or vibration stimulation can be used to provide haptic feedback. Computer 151 of FIG. 5 may be configured to implement a computation block. The Q3D information collected using the image sensor array 210 includes information that includes an indication of the measured value of the vertical strain distance (X _{t i} ) at a particular time for an individual tissue surface location. The computational block receives as input information indicating the deformation distance X _ti and block 3102 generates one or more pin _{shift amounts} X _UIi as a function of the deformation distance X _ti as an output. Although Q3D endoscopes using image sensor arrays are discussed herein, other types of Q3D endoscopes can be used without departing from the spirit of the invention. For example, a Q3D endoscope using a time-of-flight sensor can be used with the same performance. U.S. Pat. No. 8,262,559 B2 (filed April 9, 2009), the entire contents of which is incorporated herein by reference, discloses at least one example of such an alternative Q3D endoscope.

32 is an exemplary flow chart illustrating a process 3200 performed using the computational block of FIG. 31, in accordance with some embodiments. In module 3202, the Q3D endoscope system calculates the 3D coordinates of the displacement u of the tissue surface 2608 of Fig. 26, for example, which is deformed by the instrument 2604. The 3D coordinate calculation is performed by the elements and modules described in Figs. 8-19. In module 3204, 3D coordinates of tissue displacement u are mapped to pin position counts and pin displacements. For example, if it is assumed that the tissue surface point (x, y) has a measured displacement ( u _xy ), the pin having the calculated position counts N _X and N _Y displaces its height by D _XY . The relationship between D and u can be proportional to the estimated extent of tissue displacement and the detailed structure of the TUI. For example, if the tissue is assumed to deform within a range of up to 20 mm, the relationship between D and u may be 1: 1. The TUI pin will then vary in height by 1 mm per 1 mm tissue displacement. If the surgical procedure is performed on a structure of increased stiffness, a smaller tissue displacement range is used. In this case, a range of up to 5 mm may be optimal. To provide the surgeon with sufficient haptic feedback resolution, the pin displacement D can be computed as 4 * u . That is, for a tissue displacement of 0.25 mm, the TUI pin varies in height by 1 mm. Those skilled in the art will know how to replace the use of the TUI haptic feedback interface with another type of haptic feedback interface. For example, if sub-threshold electric stimulation is used as a form of haptic feedback, the magnitude of the stimulation current is adjusted according to the proportional relationship described above. Alternatively, when the vibration of the control input device 160 is used as haptic feedback, the acceleration of vibration or the intensity of vibration can be adjusted according to the above-described proportional relationship. Generally, the flow chart module 3204 maps the displacement (u _xy ) to the magnitude of the feedback provided by the haptic interface used by the system 152 of FIG. 5 or 6. In module 3206, the process is repeated to cover all points of interest tissue surface contacted by device 2604 of FIG. In module 3208, the haptic interface is driven to provide feedback according to the computed map. The optional module 3210 may also provide visual formatted tissue displacement information on any display associated with the system 152.

Force feedback interface

Referring again to Figure 26, in accordance with some embodiments, a visual indicia 2602 may be used to provide visual feedback to the user indicating the magnitude of the force applied to the target tissue structure by the device, It can be seen that it is overlaid on the device. Mr. Rayleigh in The Journal of Thoracic and Cardiovascular Surgery, January 2008. "Effects of Visual Feedback on Robot Assisted Surgical Task Performance" by Reiley CE et al. Provide tactile feedback while tying surgical knots during a remote operation. And the use of visual markers. The visual indicator includes a color code marker overlaid on the image of the device within the surgical scene. The coloring of the marker changes with the magnitude of the force sensed by the force sensors. The marker is colored in yellow in response to the ideal magnitude of applied force. The marker is colored green in response to less than the ideal magnitude of the applied force. The marker is colored red in response to greater than the ideal magnitude of the applied force. In operation, the user can utilize the marker color as a guide to reposition the device as needed to achieve the ideal magnitude of the force. In response to seeing the green marker, the user can reposition the device to increase the magnitude of the force. In response to seeing the red marker, the user can reposition the device to reduce the magnitude of the force. This force feedback interface can be integrated into the system 152 that computes or approximates the magnitude of the force by processing the displacement information provided by the Q3D endoscope system 101C of Fig. 8, as described above. Optionally, other types of force feedback interfaces may be used. For example, the magnitude of the force exerted by the instrument may be displayed in the form of a bar graph or as a digital display on the viewer 312 side of FIGS. 22A and 22B.

The entire process using Q3D information to provide haptic feedback

Figure 33 is an exemplary flow chart of a first haptic feedback process 3200 for use with the TUI 2702 of Figures 27A-27C, in accordance with some embodiments. Module 3302 maps tissue surface locations to TUI locations and stores the mapping in a non-transitory computer-readable storage device. Module 3304 registers the surgical instrument at the tissue locations. Registration can be accomplished, for example, by identifying a target device within the FOV of the Q3D endoscope using an identification algorithm similar to that described in Figures 9 and 10. Module 3306 receives user input to select a tissue surface location and moves the device to align with the selected tissue surface location. Module 3308 receives user input to impose the selected force, and moves the device to contact the selected tissue location and impose the selected force. Module 3310 receives Q3D information indicating the deformation distance of the selected tissue surface in response to the imposed force. Module 3312 determines the amount of displacement of one or more pins mapped to a tissue surface location selected as a function of the indicated tissue deformation distance at the selected tissue location. Module 3314 performs displacement of one or more pins mapped to the selected tissue location by a determined amount. Decision module 3316 determines whether a new user input for selecting another tissue surface location is received. If such user input is not received, module 3318 generates a wait for a specified amount of time, and control then returns to decision module 3316 again. In response to receiving the user input to select a new tissue surface location, control proceeds to module 3306. [ Those skilled in the art will know how to replace the use of the TUI haptic feedback interface with another type of haptic feedback interface. For example, if less than a threshold electrical stimulus is used as one form of haptic feedback, the magnitude of the stimulation current is adjusted according to the steps described above. Alternatively, if the vibration of the control input device 160 is used as haptic feedback, the vibration acceleration or the vibration intensity can be adjusted according to the steps described above. Also, if a visual interface is used to provide haptic feedback, visual elements (e.g., visual indicia, bar graph, digital display, etc.) that provide feedback are adjusted according to the steps described above.

FIG. 34 is an exemplary flow chart of a second haptic feedback process 3400 for use with an alternative embodiment TUI 2902 of FIGS. 29A-29C, in accordance with some embodiments. It will be appreciated that the first haptic feedback process and the second haptic feedback process are similar. Module 3406 receives user input to select a tissue surface location and moves the device to align with the selected tissue surface location. Module 3408 receives user input to impose the selected force, and moves the device to contact the selected tissue location and impose the selected force. Module 3410 receives Q3D information representing the deformation distance of the selected tissue surface in response to the imposed force. Module 3412 determines the amount of displacement of one or more pins as a function of the indicated tissue deformation distance at the selected tissue location. Module 3414 performs displacement of one or more pins mapped to the selected tissue location by a determined amount. Decision module 3416 determines whether a new user input for selecting another tissue surface location is received. If such user input is not received, module 3418 generates a wait for a specified amount of time, and control then returns to decision module 3416 again. In response to receiving the user input to select a new tissue surface location, control proceeds to module 3406. [

35 is an exemplary flow chart of a third process for controlling a force imposed on a selected tissue surface location, in accordance with some embodiments. As described with reference to Fig. 26, a visual cue 2602 may be provided within the surgical scene to provide an indication of whether to change the force imposed on the tissue surface location. The third process may proceed in parallel with the first or second process. Module 3502 receives an indication of the force sensed using the sensor associated with the tip of the instrument and generates a visual indicia of the magnitude of the sensed force within the surgical screen display. The determination module 3504 determines whether a user input for changing the magnitude of the force imposed by the device is received. In response to determining that a user input to change the magnitude of the force has been received, the module 3506 alters the imposed force in accordance with the user input. In response to determining that no user input has been received to change the force, control returns to module 3502. [

Q3D-based haptic feedback for virtual tissue promotion

Figures 36a-36e are illustrations of cross-sectional views of a sequence of surfaces of a tissue structure 3602 showing deformation of a tissue surface 3608 generated in response to a force exerted by an instrument, according to some embodiments. More specifically, these figures illustrate the implementation of Q3D-based haptic feedback for virtual tissue palpation. In an exemplary sequence, the device 3604 contacts and exerts a force having a vertically downward component on the tissue surface 3608 of the tissue structure 3602. These figures represent the deformation of the tissue surface 3608 at a plurality of discrete positions in response to force. The Q3D endoscope 202 is shown positioned to acquire image information indicative of tissue deformation at each of a plurality of different tissue surface locations. The mold 3612 is shown embedded under the tissue surface 3608 in the tissue structure 3602. The tissue structure portion 3602 has a specific stiffness K _t . The mold release 3612 tissue structure has another unique stiffness K _a . Q3D can not be seen in the images acquired using the endoscope 202 because the heterogeneous tissue structure 3612 is below the surface of the target tissue structure. As described above, the stiffness matrix ( K _t and K _a ) can be known a priori. In this case, the system 152 will be configured to replace the matrix K _t with the matrix K _a as the device 3604 of FIG. 36A passes through the area containing the variant. As described above, the user or the system can be based on the preoperative image (pre-operative images) showing a release of the position on the tissue surface to identify a transition zone from _a K to K _t. Alternatively, a mechanism for determining or approximating the relative difference in stiffness between regions with and without moldings 3612 can also be incorporated into system 152 as described above. In all of these cases, the system 152 provides haptic feedback of direct, indirect, or relative magnitude with respect to the stiffness provided by the tissue or about the force applied by the instrument to the operator.

37A-37E illustrate "instantaneous" variations of an example of a TUI pin top surface interface corresponding to exemplary tissue structural transformations illustrated in the section views of the constant sequence of Figs. 36A-36E, according to some embodiments Sectional views of a certain sequence of TUI 2702 of Figures 27A-27C configured to illustrate. 37A, a TUI 2702 includes a surface structure 3706 having an outline that corresponds to and reflects deformations formed in the tissue structure surface 3608, according to some embodiments. More specifically, the TUI 2702 is configured to determine the magnitude of the force exerted by the device on the stiffness provided by the tissue structure 3602 or on the strength of the haptic feedback surface 3706 in response to tissue deformation measured by the Q3D endoscope. And a shape change haptic feedback surface structure 3706 that includes a plurality of pins 2706 that cooperate to alter the physical shape. According to some embodiments, the TUI shape modification surface 3706 includes a plurality of vertical fins 2706 collectively defining a TUI surface. Each pin 2706 is selectively raised and lowered to change the shape of the TUI surface structure. The vertical (axial) positions of the tops of the pins that make up the TUI surface are changed in response to changes in the tissue structure surface contour such that the TUI surface contours are in continuous alignment with the tissue structure contours. 37A-37E are "instantaneous " to incorporate the concept that they are present at substantially the same time as the presence of corresponding tissue structural deformations (i.e., the user does not experience any perceptible delay) ". The deformation of a given TUI occurs in response to the occurrence of a tissue structural deformation at the corresponding location and disappears in response to return of the tissue structural feature at its corresponding location to its natural shape. Alternatively, the TUI modification may correspond to the magnitude of the force exerted by the instrument during virtual stimulation or to the magnitude of the stiffness provided by the tissue. This alternative embodiment of the TUI is implemented using the same principles described in Figures 37A-37E.

36A shows a first cross-sectional view of the target tissue structure 3602 prior to contact by the device 3604. FIG. 37A shows a first cross-sectional view of the contour of the corresponding TUI surface 3706 before the device 3604 contacts the target tissue surface 3608. FIG. To simplify the following discussion, the external surface contour of the exemplary tissue structure 3602 is shown to be relatively flat. However, it will be appreciated that the tissue structure 3602 may have a naturally curved contour, and may have naturally irregular portions such as depressions and ridges.

Figure 36B shows the tissue structure 3602 being pressed into the tissue surface 3608 on the left side of the surface bottom mold 3612 structure so that the device 3604 applies a force to deform the first region 3601-1 of the tissue structure 3602. [ Lt; RTI ID = 0.0 > 3602 < / RTI > The depth of deformation at the point of contact between the device 3604 and the tissue surface at the first surface area 3606-1 is determined by at least the force (F _t1 ) applied on the surface at that point of contact, It is a function of the stiffness of the tissue surface. In the second sectional view of the example, the lowest point of deformation within the first region is the vertical distance X _t1 from the adjacent undisturbed region of the tissue surface 3608. The vertical distance X _t1 is measured by the Q3D endoscope system 101C in Fig. Next, the system 152 calculates the corresponding instrument force F _t1 as described above, or estimates the corresponding local tissue stiffness K _t1 . Figure 37b is a device 3604 of the tissue structure (3602) a first region (3606-1) to the amount _(t1 X) responsive to deformation by, TUI surface a first region (3706-1) in the 3706's Sectional view of the contour of the corresponding TUI surface 3706 with the pin retracted by a first amount X _UI1 . Alternatively, the same method, elements, or steps may be applied to modify the TUI surface in response to F _t1 or K _t1 . The deformation formed in the first region 3706-1 of the TUI surface corresponding to the first region 3606-1 of the target tissue surface has a shape corresponding to the deformation shape of the first region of the target tissue surface. The shape of the first area of the target surface has been restored to its natural contour. The depth of deformation X _UI1 in the first region of the TUI surface is a function of the depth of deformation (X _t1 ) of the target tissue surface at least in response to a force applied to the target tissue surface within the first region of the target tissue surface. Alternatively, it may be a function of the at least F or K _{t1 t1} depth (X _UI1) of the strain in the first surface region of the TUI.

Figure 36c shows a section 3602 of the tissue structure 3602 being pressed into the second area 3606-2 of the tissue surface above the subsurface structure 3612 to force the device 3604 to deform the target tissue surface. 3 section views. In this example, the depth of deformation at the point of contact between the device 3604 and the tissue surface 3602 is determined by at least the force exerted on the target surface at that point of contact, the target tissue surface at that point of contact, Is a function of the stiffness of both the subsurface structure below the point. In this example, it is assumed that the mold release 3612 has greater stiffness than the target tissue structure 3602. [ In this example, it is also assumed that the subsurface structure is sufficiently close to the target tissue surface such that its stiffness affects the depth of deformation of the target tissue surface in response to the force exerted by the apparatus. In the exemplary third cross-sectional view, the lowest point of deformation within the second region 3606-2 of the target tissue surface is the vertical distance (X _t2 ) from the adjacent non-ideal region of the target tissue surface. Figure 37c illustrates that the pin of the second region 3706-2 of the TUI surface 3706 is in contact with the second region 3706-2 in response to the device deforming the second region 3606-2 of the target tissue surface by a quantity X _t2 , Sectional view of the contour of the corresponding TUI surface 3706, which is retracted by the second X _UI2 . It can be seen that the first area 3606-1 of the target tissue surface has been restored to its natural shape in response to removal of the device force from it. Also, in response to the first tissue area being restored to its natural shape, the first area 3706-1 of the TUI surface is reverted to its rest shape pre-force. The deformation formed in the second area 3706-2 of the TUI surface corresponding to the second area 3606-2 of the tissue surface has a shape corresponding to the deformation shape of the second area of the tissue surface. The depth of deformation (X _UI2 ) in the second region of the TUI surface is a function of the deformation depth (X _t2 ) of the target tissue surface at least in response to a force applied to the target tissue surface within the second region of the target tissue surface. In accordance with the alternative, previously presented principles, techniques, components and steps, it may be a function of the at least F or K _{t2 t2} depth (X _UI2) of deformation of the second region of the TUI surface.

The stiffness in the second region 3606-2 of the tissue structure 3602 is greater than the stiffness in the second region 3606-2 in the first region 3606-1 of the target tissue structure Is higher than the stiffness of. That is, the lower surface deformation of the higher stiffness has the stiffening effect of the second region 3606-2 of the target tissue structure. As a result, the deformation depth (X _t1 ) of the target tissue surface in the first region 3606-1 is larger than the deformation depth (X _t2 ) of the target tissue surface in the second region 3606-2. Also, the depth (X _UI1 ) of the corresponding deformation in the first region 3706-1 of the TUI feedback surface, which is a function of the deformation depth (X _t1 ) of the target tissue surface in the first region 3606-1, Is greater than the depth of the corresponding deformation X _UI2 in the second region 3706-2 of the TUI feedback surface, which is a function of the deformation depth (X _t2 ) of the tissue surface in the second region 3606-2. However, if instead of the depth of deformation, the TUI changes its shape according to the local tissue stiffness (K _t2 ), the amount of deformation X _UI2 is _calculated as X _UI1 . Similarly, if the device force is mapped to the TUI 2702, the strain X _UI2 may be greater or less than X _UI1 , depending on the change in force from F _t1 to F _t2 .

Figure 36d shows a fourth cross-sectional view of tissue structure 3602 in which device 3604 is being pressed against a third region 3606-3 of tissue surface 3608. [ In the fourth section view of the example, the lowest point of deformation in the third region 3606-3 is the vertical distance X _t3 from the adjacent non-ideal region of the target tissue surface 3608. Figure 37d shows a fourth cross-sectional view of the contour of the corresponding TUI feedback surface 3706 generated in response to the device deforming the third region 3606-3 of the tissue structure portion by a quantity X _t3 . The deformation formed in the third region 3706-3 of the TUI feedback surface 3706 corresponding to the third region 3606-3 of the tissue surface 3608 causes deformation of the third region 3606-3 of the tissue surface 3606-3, As shown in Fig. In addition, the second region 3606-2 of the target tissue surface has been restored to its natural shape in response to removal of the device force therefrom. In addition, in response to the second target tissue region being restored to its natural shape, the second region 3706-2 of the TUI feedback surface 3706 has its pre-force rest shape ). Since the stiffness of the target tissue structure is substantially the same in both the first region and the third region of the target tissue surface, the depth of deformation X _t1 is substantially equal to the deformation depth X _UI3 . The depth of the deformation (X _UI1) is a function of the depth (X _t1) of the deformation, since the depth of the modified (X _UI3) is a function of the depth (X _t3) of the transformation, the depth (X _UI1) of deformation is strain Is substantially the same as the depth (X _UI3 ).

Figure 36E shows a fifth cross-sectional view of the target tissue structure 3602 in which the device 3604 is being pressed against the fourth region 3606-4 of the tissue surface 3608. [ In the fifth section view of the example, the lowest point of deformation within the fourth region 3606-3 is the vertical distance (X _t4 ) from the adjacent non-ideal region of the target tissue surface 3608. 37E shows a fifth sectional view of the contour of the corresponding TUI feedback surface 3706 generated in response to the device deforming the fourth region 3606-4 of the target tissue surface 3608 by the amount X _t4 Respectively. The third area 3606-3 of the target tissue surface has been restored to its natural shape in response to removal of the instrumental force therefrom. In addition, in response to the third target tissue region 3603-3 being restored to its natural shape, the third region 3706-3 of the TUI feedback surface has its pre-force dormant shape rest shape. The deformation formed in the fourth region 3706-4 of the TUI surface corresponding to the fourth region 3606-4 of the tissue surface 3608 is less than the deformation of the fourth region 3606-4 of the tissue structure 3602 And has a shape corresponding to the shape. The deformation depth X _UI4 in the fourth region 3706-4 of the TUI feedback surface 3706 is at least _equal to the deformation depth X _UI4 of the deformation of the target tissue surface in response to the force applied to the target tissue surface within the fourth region of the target tissue surface Is a function of depth (X _t4 ). For the cross-sectional views of the target tissue surface, the target tissue deformations (X _t1 , X _t3 and X _t4 ) are substantially the same, and the exemplary TUI surface deformations (X _UI1 , X _UI3, and X _UI4 ) are substantially the same.

Although the device 3604 is illustrated as generating deformations at discrete locations in a sequence on the tissue surface 3608, the user may be required to move the tissue surface in a smooth motion to allow the deformation front to move along the target tissue surface It can be made to move across. In that case, the individual positions shown in Figures 36A-36E represent the individual positions on the deformation face. Optionally, the user can cause the device to move from one location to the next location to a separate hop. Optionally, the system 152 may automatically move the devices to various locations under direct or indirect user control. Indirect user control refers to movement of devices through control by a programmed command in response to user initiation. Likewise, the system 152 may be implemented to fully automate the motion of devices in accordance with an algorithm determined by the programmed instructions. In the process of moving the device across the target tissue surface into individual motions of a sequence or to a smooth motion, the user contacts the TUI feedback surface 3706 to feel the contour of the TUI feedback surface 3706, A physical sense of change in shape and stiffness of the target tissue surface in response to application of the device force to the locations can be obtained. The amount of deformation in a given region of the TUI surface provides an indication of the stiffness of the corresponding region of the target tissue surface, since the amount of target tissue deformation is a function of the target tissue stiffness at the point at least partially the device force is applied. In particular, for example, the user may be able to modify the transformation between the deformation (X _UI1 ) at one of the second TUI positions and the deformation (X _UI1 , X _UI3, and X _UI4 ) at the other first, third and fourth TUI positions , Which may indicate to the user that the target tissue stiffness at the second target tissue location is different than the target tissue stiffness at the first, third, and fourth target tissue locations. As shown in Figs. 36A to 36E, this stiffness difference can indicate the presence of a sub-surface deformity not visible. Thus, this hypothetical tissue stimulation system and method provides haptic feedback on tissue strain, force exerted by the device, or local tissue stiffness.

38A-38E illustrate "composite" variations of an example of a TUI feedback surface 3806 corresponding to exemplary tissue structural deformations shown in cross-sectional views of the constant sequence of Figs. 36A-36E, according to some embodiments Sectional views of a certain sequence of the TUIs of Figs. 27A-27C. The TUI feedback surface deformations of Figures 38a-38e are referred to as "composite" to incorporate the notion that they are cumulative. At the end of individual target tissue deformations of a given sequence or at the end of the generation of the deformation front, each TUI surface deformation is subjected to different TUI surface deformations so that the accumulated TUI surface deformations provide one composite TUI surface comprising all individual TUI surface deformations. Lt; / RTI > For example, following the target surface deformations of the constant sequence of Figs. 36B-36E, the user contacts the contour of the TUI feedback surface 3806 to determine the different individual positions in response to application of the device force to different target tissue positions It is possible to obtain a physical sense of the difference in the change in the shape of the target tissue surface 3608 in the tissue. Differences in shape or depth at different locations of the TUI surface may indicate differences in target tissue surface stiffness at corresponding target tissue locations or forces exerted by the device at each tissue location. For example, the first TUI surface position 3806-1, the third TUI surface position 3806-3, and the fourth TUI surface position 3806-4 may have a positive (X _UI1 , X _UI3, and X _UI4 ), and the second TUI surface location 3806-2 is recessed by a different amount X _UI2 relative to the rest of the rest level of the TUI surface. When the device is assumed to have substantially the same strength as for all the tissue deformation, in the illustrated example, both because (X _UI1, X _UI3 and X _UI4) are substantially the same, the amount (X _UI1, X _UI3 and X the difference between _UI4) and Yang Yang (X _UI2) is is due to the difference in rigidity of the target tissue surface in the second region (3606-2), this may indicate a hidden surface and releasing (3612).

Figures 39A-39E illustrate one or more pins 2906 in the TUI 2902 of an alternative embodiment of Figures 29A-29C, in response to exemplary modifications of the target tissue surface shown in Figures 36A-36E, ≪ / RTI > The TUI 2902 providing a haptic interface includes an array of pins 2906 that can be selectively axially expanded so that one or more pins provide an indication of the stiffness of the tissue surface. In the example shown, the length of the pin extension is proportional to the stiffness, and the longer the pin extension is, the greater the stiffness is. Optionally, it can be increased to indicate that the axial drive force corresponds to a greater stiffness as described above. As described above, different regions of the tissue surface may have different stiffnesses, and different stiffnesses of these different regions may indicate the presence of hidden sub-surface variants. The initial pin displacement of the example shown in Fig. 39A represents an initial pin displacement of rest or inactivity by an amount (X _p0 ) corresponding to an example target tissue surface prior to contact by the device shown in Fig. 36A. Fig. 39B shows the first pin displacement by an amount X _p1 in response to the example deformation by the amount X _t1 at the first position 3606-1 on the tissue surface shown in Fig. 36B. Figure 39c shows the second pin displacement by an amount X _p2 in response to the example deformation by the amount X _t2 at the second position 3606-2 on the tissue surface shown in Figure 36c. Figure 39d shows the third pin displacement by an amount X _p3 in response to the example deformation by the amount X _t3 at the third position 3606-3 on the tissue surface shown in Figure 36d. Fig. 39E shows the fourth pin displacement by the amount X _p4 in response to the example deformation by the amount (X _t4 ) at the fourth position 3606-4 on the tissue surface shown in Fig. 36E.

Because the target tissue surface stiffness is the same at the first, third, and fourth tissue surface locations, the first, third, and fourth pin displacements ( _Xp1 , _Xp3, and _Xp4 ) are substantially the same. The second pin displacement X _p2 is greater than the remaining three pin displacements because the tissue surface stiffness is greater at the third tissue location 3606-2. Thus, the pin displacement can be achieved by a user of a certain sequence of contacts on the target tissue surface as shown by Figures 36a-36e, either through discontinuous movement of the device from one position to the next, And provides an indication of the target tissue surface stiffness when the device is controllably moved through the locations. The change in pin displacement when the device moves from one position to another indicates a change in the target tissue structural stiffness between the two positions. Alternatively, when the instrument force _f or the tissue stiffness K is mapped to the TUI, the displacement (X _{p1 - 4} ) depends on f and K and has values calculated according to the steps described above. Thus, this hypothetical tissue stimulation system and method provides haptic feedback on tissue strain, force exerted by the device, or local tissue stiffness.

Those skilled in the art will know how to replace the use of the TUI haptic feedback interface described above with other types of haptic feedback interfaces. For example, if a subthreshold electrotactic stimulus is used as one form of haptic feedback, the magnitude of the stimulation current is adjusted according to the steps described above. Instead of adjusting the displacement (X _UIi or X _pi ), the amplitude of the excitation current is changed accordingly. Alternatively, if the oscillation of the control input device 160 used as the haptic feedback, the vibration acceleration and the vibration intensity may be adjusted as a function of the tissue displacement (X _t), force (F _t) or stiffness (K _t). Also, if a visual interface is used to provide haptic feedback, visual elements (e.g., visual indicia, bar graph, digital display, etc.) that provide feedback are adjusted according to the steps described above.

In operation, the user may move the device across the target tissue surface in a discontinuous or continuous motion, for example up to several millimeters, or up to several centimeters, within a fraction of a millimeter, depending on the surgical procedure and tissue type, A small vertical force (relative to the target tissue surface) sufficient to deform the target tissue structure by a clinically safe amount that is large enough to trigger a counterattack providing tissue structure stiffness indication but not large enough to cause tissue damage You can visually observe the surgical scene when you are pressing down. For example, a force in the range of 5-40 g may be used during cardiac surgery. Smaller forces in the range of 1-30 g may be used during interventional procedures for the lungs. The process performed on a more rigid structure, such as a bone, may use a higher level of force in excess of 100 grams, as the case may be. When the device moves across the target tissue surface, the haptic interface provides a real-time indication of the target tissue structural strain, stiffness, or force exerted by the device (without any perceptible delay by the user). Thus, the user visually follows the current position on the target tissue surface, where the device places its force on the target, in real time as the device provides force to the given target tissue location.

As described above with reference to Figure 35, according to some embodiments, the Q3D endoscope system 101C measures tissue displacement and then computes or approximates the magnitude of the force imposed on the target tissue surface. In operation, during the stiffness determination, such as during a virtual acceleration process, the force applied to the target tissue surface is maintained within the safe range, as described above. More specifically, in accordance with some embodiments, when substantially equal forces are applied to each of the exemplary discrete positions on the target tissue surface shown in Figures 36b-36e, the tissue deformations represented by the haptic feedback interface information are local Inversely proportional to tissue stiffness. In some embodiments, the user controls contact between the device and the target tissue structure to adjust the force imposed on the target tissue structure by the device. Thus, a force sensor can be used to achieve a substantially constant normalized force on the target tissue structure throughout the entire stiffness determination. Optionally, the force can be controlled automatically or semi-automatically by the system 152. The system 152 may also provide an alert to the surgeon or surgeon depending on the level of device force determined from the Q3D displacement measurement. For example, in FIG. 35, in addition to providing a suitable visual index, when the force exerted by the device exceeds a certain safety threshold, the system 152 may provide an audible alarm or a visual or tactile alert .

The foregoing description and drawings of embodiments according to the present invention merely illustrate the principles of the present invention. It will thus be appreciated that various modifications may be made to the embodiments by those skilled in the art without departing from the spirit and scope of the invention as defined in the appended claims.

Claims (19)

A system for providing haptic feedback during a medical procedure,
The system comprises:
A quantitative three-dimensional (Q3D) endoscope operable to have a constant clock and to measure deformation of different locations of tissue structures disposed within the timepiece;
A surgical instrument operable to modify different positions of a tissue structure disposed within the Q3D endoscopic watch;
At least one processor configured to generate a Q3D model that provides a map of measurements of tissue structural deformation at different locations of the tissue structure; And
And a haptic user interface device configured to generate haptic feedback indicative of a map of measurements of tissue structural deformations in at least first and second distinct locations of the tissue structure. The method according to claim 1,
Wherein the haptic user interface device comprises a shape display. The method according to claim 1,
Wherein the haptic user interface device comprises an array of axially aligned fins. The method according to claim 1,
Wherein the haptic user interface device comprises a variable electrotactile stimulation feeder. The method according to claim 1,
Wherein the haptic user interface device comprises a surgical instrument manipulation control device. The method according to claim 1,
Further comprising an alert supplier configured to generate an alert in response to the process providing information indicative of a measured value of a tissue structural strain that is outside the clinically safe range. The method according to claim 6,
Wherein the alarm feeder comprises an audible alarm feeder, the alarm comprising an audible alarm. The method according to claim 6,
Wherein the alarm feeder comprises a visual display, and wherein the alarm comprises a visual alarm. The method according to claim 1,
A force sensor coupled to the surgical instrument to measure a force imposed on the tissue structure during deformation of the tissue structure; And
Further comprising a visual display configured to provide a visual indication of the force measured by the force sensor. The method according to claim 1,
Wherein the Q3D endoscope comprises an imager. The method according to claim 1,
Wherein the Q3D endoscope comprises a flight time sensor. A system for providing haptic feedback during a medical procedure,
The system comprises:
A quantitative three-dimensional (Q3D) endoscope operable to have a constant clock and to measure deformation at a location of a tissue structure disposed within the watch;
A surgical instrument operable to modify a position of a tissue structure disposed within the Q3D endoscope watch;
A haptic user interface shape display; And
Comprising at least one processor,
The haptic user interface shape display comprising:
Providing a three dimensional contour modeling at least a portion of the tissue structure;
And configured to provide an indication of a tissue structural deformation at a location on the tissue structure surface in response to information indicative of a measured value of the tissue structural deformation at the tissue structure site,
Wherein the at least one processor is configured to generate a Q3D model that includes information indicative of a measure of a tissue structural deformation at a location on the tissue structure surface and to provide information indicative of a measured value of the tissue structural deformation at the tissue structure location And to provide the haptic user interface shape display. A system for determining a force applied to a tissue during a medical procedure,
The system comprises:
A quantitative three-dimensional (Q3D) endoscope operable to measure a deformation of a tissue structure disposed within said timepiece, said timepiece having a predetermined clock;
A surgical instrument operable to modify a position of a tissue structure disposed within the Q3D endoscope watch; And
At least one processor configured to generate a Q3D model that includes information indicative of a measure of tissue structural deformation and to determine a force applied by the surgical instrument on the tissue structure based at least in part on the tissue deformation information, ≪ / RTI > 14. The method of claim 13,
Wherein the processor generates an output indicative of the determined force. 15. The method of claim 14,
Wherein the output drives the display or haptic feedback interface device. A system for providing virtual tissue stimulation during a medical procedure,
The system comprises:
A quantitative three-dimensional (Q3D) endoscope operable to measure a deformation of a tissue structure disposed within said timepiece, said timepiece having a predetermined clock;
A surgical instrument operable to modify a position of a tissue structure disposed within the Q3D endoscope watch; And
At least one of which is configured to generate a Q3D model that includes information indicative of a measure of tissue structural deformation, determine a tissue stiffness at the location, and provide a virtual stimulated output based at least in part on the determined tissue stiffness ≪ / RTI > 17. The method of claim 16,
Wherein the virtual facilitated output drives the display or haptic feedback interface device. 18. The method of claim 17,
Further comprising a processor configured to automatically drive the surgical instrument to a series of positions within the timepiece. 18. The method of claim 17,
Further comprising means configured to provide an alert in response to an increase or decrease to a hypothetical stimulation output or safety threshold exceeding a safety range.

Images