WO2019035206A1

WO2019035206A1 - Medical system and image generation method

Info

Publication number: WO2019035206A1
Application number: PCT/JP2017/029614
Authority: WO
Inventors: 井上　慎太郎
Original assignee: オリンパス株式会社
Priority date: 2017-08-18
Filing date: 2017-08-18
Publication date: 2019-02-21

Abstract

This medical system is provided with: a treatment tool; an endoscope which has an image capturing unit; a control device which generates a display image from an image captured by the image capturing unit; a display device which displays the display image; and an input device which selects medical equipment, wherein the control device measures a relative position from the endoscope to the treatment tool, and superposes, onto the display image, a virtual image of the medical equipment, which is at the relative position with respect to the treatment tool displayed on the display image and has a relative size, on the basis of the relative position.

Description

Medical system and image generation method

The present invention relates to a medical system and an image generation method using the medical system.

In laparoscopic surgery, a plurality of holes are made in the patient's abdomen etc., various medical devices such as an endoscope, forceps, and an electric scalpel are inserted, and observation of the affected area is confirmed by confirming an image captured by the endoscope. And the procedure is performed. This laparoscopic surgery can reduce the burden on the patient because it requires less incision area.

In laparoscopic surgery, the appropriate type of medical device is selected in accordance with the procedure, and the selected medical device is inserted into the abdominal cavity. For example, in abdominal wall hernia surgery in which the hernia gate formed in the abdominal wall is closed with mesh, the operator needs to select a mesh with the optimal size necessary and sufficient to close the hernia gate, and insert the selected mesh into the abdominal cavity is there.

The operator confirms the image captured by the endoscope, estimates the length and area of the contour of the tissue in the three-dimensional space, and selects a mesh of an optimal size. If the selected mesh is inserted into the abdominal cavity and the size of the mesh is found to be small relative to the hernia gate, the operator needs to remove the mesh from the abdominal cavity and reinsert the reselected mesh into the abdominal cavity . Depending on the experience and technique of the operator, there is a problem that it takes time for trial and error of mesh selection.

Patent Document 1 describes a telesurgery robot system capable of measuring the length and area of a contour of a tissue in a three-dimensional space from an endoscopic image of the tissue acquired by the endoscope. The telesurgical robot system can measure, for example, the distance between measurement points designated by a treatment tool, and superimpose the measured distance information on an endoscopic image for display. The operator who uses the telesurgical robot system can grasp the dimensions and the like in the three-dimensional space of the tissue shown in the endoscopic image.

US Patent No. 9155592

However, the telesurgical robot system described in Patent Document 1 requires procedures such as designation of measurement points in order to perform measurement in a three-dimensional space, and it is possible to rapidly measure dimensions etc. in the three-dimensional space. It was not. In addition, it has been difficult for the operator to intuitively determine the most appropriate type of medical device from the measurement results of dimensions and the like in the three-dimensional space.

In the light of the above circumstances, the present invention aims to provide a medical system capable of supporting selection of an appropriate type of medical device and an image generation method using the medical system.

In order to solve the above-mentioned subject, this invention proposes the following means.
A medical system according to the present invention includes a treatment tool, an endoscope having an imaging unit, a control device that generates a display image from a captured image acquired by the imaging unit, a display device that displays the display image, and medical treatment An input device for selecting a device, the control device measures a relative position from the endoscope to the treatment tool, and the control device reflects the display image based on the relative position A virtual image of the medical device at a relative position to the treatment tool and having a relative size is superimposed on the display image.

The image generation method according to the present invention is a medical system in which a display image is generated from a captured image acquired by an imaging unit of an endoscope, and a medical device selecting step of selecting a medical device using an input device; The medical device having a relative size relative to the treatment tool shown in the display image based on the relative position measuring step of measuring the relative position to the treatment tool and the relative position And a virtual image superimposing step of superimposing the virtual image on the display image.

According to the medical system and image generation method of the present invention, selection of an appropriate type of medical device can be assisted.

It is a figure showing the whole medical system composition concerning a first embodiment of the present invention. It is a hardware block diagram of the medical system. It is a figure which shows the example of whole structure of the control part of the medical system. It is a figure which shows the mode of laparoscopic surgery using the medical system. In the medical system, the display image on which the virtual image of the selected mesh A is superimposed is shown. In the medical system, the display image on which the virtual image of the selected mesh B is superimposed is shown. It is a figure which shows the mode of laparoscopic surgery which performs a large intestine detachment using the same medical care system. In the medical system, the display image on which the virtual image of the selected stapler A is superimposed is shown. In the medical system, the display image on which the virtual image of the selected stapler B is superimposed is shown. It is a figure which shows the whole structure of the modification of the medical system. It is a figure which shows the whole structure of the modification of the medical system. The medical system which concerns on 2nd embodiment of this invention WHEREIN: The display image on which the virtual image of the selected mesh A was superimposed is shown. In the medical system, the display image on which the virtual image of the selected mesh A is superimposed is shown. In the medical system, the display image on which the virtual image of the selected mesh A is superimposed is shown. In the medical system, the display image on which the virtual image of the selected stapler B is superimposed is shown. The endoscope of the medical system concerning a third embodiment of the present invention is shown. In the medical system, the display image on which the virtual image of the selected snare A is superimposed is shown. In the medical system, the display image on which the virtual image of the selected snare B is superimposed is shown.

First Embodiment
A first embodiment of the present invention will be described with reference to FIGS. In addition, in order to make a drawing legible, the dimension etc. of each component are adjusted suitably.

FIG. 1 is a diagram showing an entire configuration of a medical system 100 according to the present embodiment.
As shown in FIG. 1, the medical system 100 includes a treatment tool 1, an endoscope 2, a control device 3, a display device 4, and an input device 5. The medical system 100 is a system that supports a treatment to be performed by inserting the treatment instrument 1 or the endoscope 2 or the like from separate holes (openings) opened in an abdominal wall in laparoscopic surgery.

As shown in FIG. 1, the treatment tool 1 has an elongated insertion portion 10 that can be inserted into the abdominal cavity of a patient, and an operation portion 11 provided on the proximal end side of the insertion portion 10. The operator passes the insertion portion 10 through the trocar T which is punctured in the abdomen B of the patient and introduces the insertion portion 10 into the abdominal cavity. Depending on the type of treatment and the condition of the affected area, the operator may introduce a plurality of treatment tools 1 into the abdominal cavity.

As shown in FIG. 1, the insertion portion 10 has a treatment portion 12 at the tip end for treating an affected area of a patient. In the present embodiment, the treatment unit 12 is a gripping mechanism configured by a pair of gripping members 12 a.

The operation unit 11 is a member for operating the pair of gripping members 12a. The operation unit 11 has a handle, and moves the handle relative to the other portion of the operation unit 11 to open and close the pair of holding members 12 a of the treatment unit 12. The operator can operate the treatment unit 12 by holding the operation unit 11 with one hand.

FIG. 2 is a hardware configuration diagram of the medical system 100 excluding the treatment tool 1.
As shown in FIGS. 1 and 2, the endoscope 2 has an elongated rigid insertion portion 20 which can be inserted into the abdominal cavity of a patient, and a handle portion 21. As shown in FIG. 1, the operator passes the insertion portion 20 through the trocar T which is punctured in the abdomen B of the patient and introduces the insertion portion 20 into the abdominal cavity.

As shown in FIG. 1, the insertion unit 20 is provided at the tip thereof with an imaging unit 22 having a lens for imaging a situation in the abdomen of a patient and an imaging element. The insertion unit 20 introduced into the abdominal cavity is disposed at a position where the imaging unit 22 can photograph an affected area in the abdomen that is a treatment target. The imaging unit 22 may have an optical zoom or electronic zoom function.

The operator can change the position and the orientation of the imaging unit 22 of the endoscope 2 by holding the holding unit 21 and moving the endoscope 2.
The insertion portion 20 may further include an active bending portion that actively bends. By bending the active bending portion provided in a part of the insertion portion 20, the position and the orientation of the imaging unit 22 can be changed.

A control signal line for controlling the imaging unit 22, a transmission signal for transferring a captured image captured by the imaging unit 22, and the like are wired inside the hand holding unit 21.

The control device 3 has a control unit 33, as shown in FIG. The control device 3 receives the captured image captured by the imaging unit 22 of the endoscope 2 and transfers it to the display device 4 as a display image.

The control unit 33 is configured by a device (computer) provided with hardware capable of executing programs such as a CPU (Central Processing Unit) and a memory. The function of the control unit 33 can be realized as a function of software by the control unit 33 reading and executing a program for controlling the CPU.
Note that at least a part of the control unit 33 may be configured by a dedicated logic circuit or the like.
Furthermore, the same function can be realized by connecting at least a part of hardware configuring the control unit 33 with a communication line.

FIG. 3A shows an example of the overall configuration of the control unit 33. As shown in FIG.
As shown in FIG. 3A, the control unit 33 includes a CPU 34, a memory 35 capable of reading a program, a storage unit 36, and an input / output control unit 37. A program provided to the control unit 33 for controlling the operation of the control unit 33 is read into the memory 35 and executed by the CPU 34.

The storage unit 36 is a non-volatile storage medium storing the above-described program and necessary data. The storage unit 36 is configured of, for example, a ROM, a hard disk or the like. The program recorded in the storage unit 36 is read into the memory 35 and executed by the CPU 34.

The storage unit 36 executes, for example, virtual images or virtual images of “types” of medical devices introduced into the abdominal cavity, such as forceps, electric scalpel, stapler, snare, mesh, etc., and “types” of medical devices. And shape data for virtual image that can be generated. Further, with regard to medical devices having variations in size, virtual image shape data capable of generating a virtual image for each “size” or a virtual image for each “size” is also stored in the storage unit 36.
In the following description, virtual image or virtual image shape data is referred to as "virtual image data", and virtual image data for each "type" and "size" of a medical device is referred to as "medical device information". Name.

The input / output control unit 37 receives input data from the endoscope 2 and transfers the input data to the CPU 34 or the like. The input / output control unit 37 also generates data, control signals, and the like for the endoscope 2 and the display device 4 based on an instruction of the CPU 34.

The control unit 33 receives a captured image as input data from the endoscope 2 and reads the captured image into the memory 35. Based on the program read into the memory 35, the CPU 34 performs image processing on the captured image. The captured image subjected to the image processing is transferred to the display device 4 as a display image.

The control unit 33 has at least two types of image processing operation modes of a normal mode and a virtual image superposition mode. The operating mode is controlled by the program.
The control unit 33 operating in the normal mode performs image processing such as image format conversion, contrast adjustment, and resizing processing on the captured image to generate a display image.
In addition to the image processing in the normal mode, the control unit 33 operating in the virtual image superposition mode performs an image processing of superimposing a virtual image of a medical device described later on a display image.

Here, the control unit 33 is not limited to the device provided in one piece of hardware. For example, the control unit 33 is configured by separating the CPU 34, the memory 35, the storage unit 36, and the input / output control unit 37 as separate pieces of hardware, and then connecting the hardware through a communication line. You may Alternatively, the control unit 33 may realize the control unit 33 as a cloud system by separating the storage unit 36 and similarly connecting through the communication line.

The control unit 33 is other than the CPU 34, the memory 35, the storage unit 36, and the input / output control unit 37 shown in FIG. 3A, and further, those necessary to control the operation of the control device 3 are further included. You may have. For example, as shown in FIG. 3B, the control unit 33 may further include an image operation unit 38 that performs part or all of the image processing and image recognition processing performed by the CPU 34. By further including the image calculation unit 38, the control unit 33 can execute specific image processing and image recognition processing at high speed. Further, it may further include an image transfer unit for transferring the display image from the memory 35 to the display device 4.

The display device 4 is a device that displays the display image transferred by the control device 3. The display device 4 has a known monitor 41 such as an LCD display. The display device 4 may have a plurality of monitors 41. The display device 4 may include a head mounted display or a projector instead of the monitor 41.

The monitor 41 can also display a GUI (Graphical User Interface) image generated by the control device 3 as a GUI. For example, the monitor 41 can display a control information display of the medical system 100 and an alert display on the GUI for the operator.
In addition, when the control unit 33 needs information input from the operator, the display device 4 can also display a message prompting the user to input information from the input device 5 or a GUI display necessary for the information input.

The input device 5 is a device by which an operator inputs an instruction or the like to the control unit 33. The input device 5 has an input unit 51 as shown in FIG.

The input unit 51 is a device that selects an operation mode of image processing of the control unit 33, and selects a medical device to be superimposed and displayed as a virtual image on a display image. The input unit 51 may be configured by a switch or may be configured by a touch panel. The touch panel may be integrated with the monitor 41.

The input unit 51 may be configured by a keyboard or a mouse. In this case, the control unit 33 displays a list of operation modes and a list of medical devices on the monitor 41 as a GUI, and from the list displayed on the monitor 41, the operator inputs desired operation modes and operation of medical devices to the keyboard and mouse. It may be selected by The control device 3 acquires, from the input unit 51, selection information input by the operator.

The operator can use the input unit 51 to select the “type” of the medical device to be introduced into the abdominal cavity, such as forceps, electric scalpel, stapler, snare or mesh, from the medical device list. Also, for medical devices that have variations in size, "size" can also be selected.

Next, the operation of the medical system 100 and an image generation method using the medical system 100 will be described with reference to FIGS. 4 and 5 by taking laparoscopic surgery as an example in which a hernia gate formed in the abdominal wall is closed with mesh. . FIG. 4 is a view showing laparoscopic surgery using the medical system 100. As shown in FIG.

The operator provides a plurality of holes (openings) for placing the trocar T in the abdomen B of the patient, and punctures the trocar T in the holes. Next, as shown in FIG. 4, the operator passes the insertion portion 10 of the treatment tool 1 of the medical system 100 and the insertion portion 20 of the endoscope 2 through the trocar T punctured in the abdomen B of the patient, Introduce into the abdominal cavity.

The operator holds the handle 21 and moves the endoscope 2 so that the hernia gate H to be treated is included in the display image D as shown in FIG. Change the position or orientation of If the imaging unit 22 has an optical zoom or electronic zoom function, the function is adjusted. In the display image D shown in FIG. 4, the treatment unit 12 of the insertion unit 10 of the treatment tool 1 is imaged. Since the operation mode of the image processing of the control unit 33 is initially set to the “normal mode”, the virtual image of the medical device is not superimposed on the display image D.

Next, before introducing the mesh for closing the hernia gate H into the abdominal cavity, the operator examines the optimal mesh size necessary and sufficient for closing the hernia gate H. The operator sets the operation mode of the image processing of the control unit 33 to the “virtual image superposition mode”.

The control unit 33 in which the operation mode is set to the “virtual image superposition mode” displays on the monitor 41 a medical device list and a message prompting the user to select the “type” and “size” of the medical device from the input unit 51 Let
The operator operates the input unit 51 to select “mesh A” which is estimated to be the optimum size (medical device selection step).

The control unit 33 in which the operation mode is set to the “virtual image superposition mode” measures the three-dimensional relative position from the imaging unit 22 of the endoscope 2 to the treatment unit 12 of the treatment instrument (relative position measurement step).
The measurement of the three-dimensional relative position can be performed by a method appropriately selected from known methods. For example, methods such as position measurement by image processing (case 1), position measurement by stereo camera (case 2), position measurement by laser etc (case 3), position measurement by sensing (case 4), etc. can be used.

The three-dimensional relative position to the treatment unit 12 can be measured using the captured image captured by the imaging unit 22 (case 1). For example, the moving treatment unit 12 can be photographed, and the three-dimensional relative position can be measured from the movement amount of the image feature. If the dimensions of the treatment section 12 are known, the three-dimensional relative position can also be measured based on the dimensions. In addition, by providing a mark such as an optical marker on the treatment unit 12, the three-dimensional relative position can be easily measured.

By configuring the imaging unit 22 with a stereo camera, it is possible to measure a three-dimensional relative position from the imaging unit 22 to the treatment unit 12 using a parallax image of the stereo camera (case 2).

Laser, light, ultrasonic waves and the like are emitted from the imaging unit 22, and the three-dimensional relative position from the imaging unit 22 to the treatment unit 12 can be measured from the feedback information (case 3). The three-dimensional relative position to the treatment unit 12 may be measured by performing pattern projection from the imaging unit 22 and analyzing the pattern projection result.

In addition, a magnetic sensor or the like may be installed in the treatment unit 12 or the imaging unit 22, and the three-dimensional relative position from the imaging unit 22 to the treatment unit 12 may be measured from position information such as a magnetic sensor received by an external antenna. Yes (case 4).

Here, in the method of measuring the three-dimensional relative position shown in case 1 to case 4, it is desirable to be able to measure the three-dimensional relative posture of the treatment unit 12 in addition to the three-dimensional relative position of the treatment unit 12.

In addition, the control part 33 may parallelly implement a medical device selection process and a relative-position measurement process, and may implement either one first.

Next, the control unit 33 superimposes the virtual image of the selected medical device on the display image D based on the measurement result of the three-dimensional relative position (virtual image superposition step). FIG. 5A shows a display image D in which the virtual image VM1 of “mesh A” selected as the medical device is superimposed on the position where the treatment unit 12 appears. FIG. 5 (b) shows a display image D when the treatment tool 1 is moved.

The control unit 33 can set the transparency in the process of superimposing the virtual image VM1 on the display image D. By superimposing the virtual image VM1 on the display image D so that the virtual image VM1 is translucent, it is possible to confirm the display content previously displayed on the portion superimposed by the virtual image VM1.

The control unit 33 is a selected medical device having a relative size to the treatment unit 12 shown in the display image D based on the relative distance from the imaging unit 22 to the treatment unit 12. To generate a virtual image VM1 of The medical device information stored in the storage unit 36 is used to generate the virtual image VM1.

The display image D on which the virtual image VM1 is superimposed is displayed on the monitor 41. As shown in FIG. 5 (a), the operator can virtually see the situation where the treatment section 12 is holding "mesh A".

When the operator moves the treatment tool 1 and the position of the treatment unit 12 in the display image D is changed, the virtual image VM1 of the mesh A is superimposed on the position relative to the treatment unit 12 shown in the display image D Therefore, the position where the virtual image VM1 of the mesh A is superimposed is also changed. That is, the position where the virtual image VM1 of the mesh A is superimposed follows the position of the treatment unit 12 in the display image D.

The operator can move the treatment tool 1 to change the position of the virtual image VM1 of the mesh A in the display image D. The operator superimposes the virtual image VM1 of the selected mesh A in the vicinity of the hernia gate H by moving the treatment portion 12 to the vicinity of the hernia gate H formed in the abdominal wall, as shown in FIG. 5 (b). Can.

The virtual image VM1 of the mesh A is superimposed on the display image D so as to have a relative size with respect to the treatment unit 12 shown in the display image D. Therefore, the size of the virtual image VM1 of the mesh A superimposed on the display image D is substantially the same size as the real image of the mesh A in the display image in which the actual mesh A actually exists in the three-dimensional space. That is, the size of the virtual image VM1 of the mesh A follows the size of the treatment unit 12 in the display image D. For example, when the treatment unit 12 is at a position far from the imaging unit 22, the virtual image VM <b> 1 of the mesh A is small and superimposed as compared to the case where the treatment unit 12 is at a position near to the imaging unit 22.

The operator looks at the virtual image VM1 of the mesh A shown in the display image D, determines whether the selected mesh A has a size sufficient for closing the hernia gate H, the size etc. in the three-dimensional space It can be intuitively grasped without actually measuring it.

In the relative position measurement process, in addition to the three-dimensional relative position, when the three-dimensional relative attitude of the treatment unit 12 can be measured, the three-dimensional relative attitude can be used in the virtual image superposition process. The virtual image VM1 of the mesh A is superimposed on the display image D so as to have a posture relative to the treatment unit 12 shown in the display image D. For example, when the operator rotates the treatment tool 1 about the longitudinal axis, the mesh virtual image VM1 is generated and superimposed as an image rotated about an axis parallel to the longitudinal axis of the treatment tool 1 . By rotating the virtual image VM1 of the mesh, the dimensions and the like in the three-dimensional space can be grasped more intuitively.

FIG. 6A shows a display image D on which the virtual image VM2 of “mesh B” is superimposed when “mesh B” smaller in size than “mesh A” is selected as the medical device. FIG. 5 (b) shows a display image D when the treatment tool 1 is moved.

The operator moves the treatment section 12 to the vicinity of the hernia gate H formed on the abdominal wall to display the virtual image VM2 of the selected mesh B superimposed on the vicinity of the hernia gate H, as shown in FIG. 6B. be able to.
The operator can intuitively grasp that the selected mesh B does not have a necessary and sufficient size for closing the hernia gate H by looking at the virtual image VM2 of the mesh B shown in the display image D. it can.

Next, the operation of the medical system 100 and a method of generating an image using the medical system 100 will be described with reference to FIGS. 7 to 9, taking laparoscopic surgery in which the large intestine is detached using a stapler as an example. FIG. 7 is a view showing laparoscopic surgery in which a large intestine is detached using the medical system 100. As shown in FIG.

The operator holds the holding portion 21 and moves the endoscope 2 so that the display image D includes the cutaway portion P of the large intestine to be treated, as shown in FIG. The position and the orientation of the imaging unit 22 are changed. The treatment portion 12 of the insertion portion 10 of the treatment tool 1 is imaged in the display image D shown in FIG. 7. Since the operation mode of the image processing of the control unit 33 is initially set to the “normal mode”, the virtual image of the medical device is not superimposed on the display image D.

Next, the operator examines the size of the stapler necessary and sufficient for separating the separated portion P of the large intestine before introducing the stapler for separating the separated portion P of the large intestine into the abdominal cavity. . The operator sets the operation mode of the image processing of the control unit 33 to the “virtual image superposition mode”.

In the medical device selection step, the operator selects a stapler as the type of medical device, and further selects “stapler A” which is estimated to be the optimum size.

As shown in FIG. 8 in the virtual image superimposing step, based on the three-dimensional relative position from the imaging portion 22 of the endoscope 2 to the treatment portion 12 of the treatment instrument, which is measured in the relative position measuring step. The virtual image VS1 of "stapler A" is superimposed on the display image D.

The operator looks at the virtual image VS1 of the stapler A shown in the display image D, and instincts that the selected stapler A does not have the necessary and sufficient size to cut off the separated portion P of the large intestine. Can be grasped.

In the medical device selection step, the operator reselects the "stapler B" which is larger than the stapler A. The control unit 33 superimposes the virtual image VS2 of the "stapler B" on the display image D as shown in FIG. 9 in the virtual image superposition step.

The operator looks at the virtual image VS2 of the stapler B shown in the display image D, and instincts that the selected stapler B has a necessary and sufficient size to cut off the separated portion P of the large intestine. Can be grasped.

(Effect of the first embodiment)
According to the medical system 100 of the present embodiment, the operator looks at the virtual image of the medical device shown in the display image D and determines whether the selected medical device has a necessary and sufficient size in the three-dimensional space. Can be intuitively grasped without actually measuring dimensions and the like. The time taken for trial and error in the selection of medical devices can be reduced, and the selection of appropriate types of medical devices can be supported.

(Modification)
The first embodiment of the present invention has been described in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and design changes and the like within the scope of the present invention are also included. . In addition, the components shown in the above-described first embodiment and the modifications described below can be combined appropriately.

For example, in the above embodiment, the virtual image of the medical device is superimposed on the position where the treatment unit 12 appears in the display image D, but the superimposed position of the virtual image on the display image is not limited to the aspect of the above embodiment. In the case of using another treatment unit such as an electric scalpel, the virtual image may be superimposed on the position where the tip or side portion of the treatment unit is located.

For example, in the above embodiment, the superimposition of the virtual image on the display image is performed in the control unit 33, but the mode of the superimposition of the virtual image on the display image is not limited to the aspect of the above embodiment. For example, superimposition of a virtual image on a display image may be performed on a display device. The control unit 33 calculates information such as the superposition position and size necessary for superposition display, and the display device performs superposition on the display image of the virtual image based on the calculation result, and then the display image is displayed on the monitor 41. You may display it.

For example, although the display device 4 shown in the above embodiment is configured by the monitor 41, the display device 4 may be configured by a projector. A projector may be incorporated into the imaging unit 22 of the endoscope 2 and projection (projection mapping) of an image of a virtual image of a medical device may be performed by the projector in a real three-dimensional space. The operator can more intuitively select the most appropriate medical device. When the projector is incorporated into the imaging unit 22 of the endoscope 2, infrared pattern light may be irradiated from the projector to measure the three-dimensional relative position with the treatment unit or the like.

For example, although the endoscope 2 shown in the above embodiment is operated by the operator holding it directly, the form of the endoscope is not limited to the endoscope 2 of the above embodiment. For example, in a medical system 100B which is a modification of the medical system 100 shown in FIG. 10, the endoscope 2B includes an articulated robot arm 21B having a plurality of joints 23.
In the medical system 100B, the three-dimensional position of the imaging unit 22 at the tip of the endoscope 2B can be calculated from control information of the articulated robot arm 21B.

For example, although the treatment tool 1 shown in the above embodiment is held by the operator directly and operated, the form of the treatment tool is not limited to the treatment tool 1 of the above embodiment. For example, in a medical system 100C which is a modified example of the medical system 100 shown in FIG. 11, the treatment tool 1B includes an articulated robot arm 11B having a plurality of joints 13.
In the medical system 100C, the three-dimensional position of the treatment section 12 at the tip of the treatment tool 1B can be calculated from control information of the articulated robot arm 11B.

Second Embodiment
A second embodiment of the present invention will be described with reference to FIGS. 12-15. The present embodiment is different from the first embodiment in that depth measurement in a three-dimensional space shown in a display image is performed. In the following description, the same reference numerals are assigned to components common to those described above, and redundant description will be omitted.

The whole structure of the medical system 200 which concerns on this embodiment is the same as the medical system 100 which concerns on 1st embodiment. In the relative position measurement step, the control unit 33 of the medical system 200 adds to the three-dimensional relative position from the imaging unit 22 of the endoscope 2 to the treatment unit 12 of the treatment instrument, from the imaging unit 22 of the endoscope 2 The difference is that the "three-dimensional relative depth" to a tissue or the like (object) present in the three-dimensional space shown in the display image is also measured.

In the present embodiment, the virtual image shape data is always stored in the storage unit 36 of the control unit 33 of the medical system 200.

Next, an operation of the medical system 200 and an image generation method using the medical system 200 will be described with reference to FIGS. 12 to 15 by taking laparoscopic surgery in which a hernia gate formed in the abdominal wall is closed with a mesh as an example. . Only differences from the medical system 100 of the first embodiment will be described.

The operator sets the operation mode of the image processing of the control unit 33 to the “virtual image superposition mode”.
The operator operates the input unit 51 to select “mesh A” which is estimated to be the optimum size (medical device selection step).

The control unit 33 whose operation mode is set to the “virtual image superposition mode” measures the three-dimensional relative position from the imaging unit 22 of the endoscope 2 to the treatment unit 12 of the treatment tool as in the first embodiment (relative Position measurement process). In the relative position measurement step, the “three-dimensional relative depth” from the imaging unit 22 of the endoscope 2 to the tissue or the like present in the three-dimensional space shown in the display image is also measured.

The measurement of the three-dimensional relative depth can be performed by a method appropriately selected from known methods. For example, methods such as passive depth measurement by image processing (case P), active depth measurement by projection of a pattern or the like (case A), and the like can be used.

The three-dimensional relative depth to the treatment unit 12 can be measured using the captured image captured by the imaging unit 22 (case P). For example, the moving treatment unit 12 can be photographed, and the three-dimensional relative depth can be measured from the movement amount of the image feature. When the imaging unit 22 is configured by a stereo camera, it is possible to measure a three-dimensional relative depth from the imaging unit 22 to a tissue or the like by using a parallax image of the stereo camera.

Laser, light, ultrasonic waves and the like are emitted from the imaging unit 22, and the three-dimensional relative depth from the imaging unit 22 to the tissue and the like can be measured from the feedback information (case A). The three-dimensional relative depth to the tissue or the like may be measured by performing pattern projection of a striped pattern or a random pattern from the imaging unit 22 and analyzing the pattern projection result.

Next, the control unit 33 superimposes the virtual image of the selected medical device on the display image D based on the measurement results of the three-dimensional relative position and the three-dimensional relative depth (virtual image superposition step). 12 to 14 show the display image D on which the virtual image VM1 of "mesh A" selected as the medical device is superimposed.
Further, as shown in FIG. 12, the control unit 33 sets the display image D to a wire frame F indicating unevenness of a tissue or the like present in the three-dimensional space shown in the display image D based on the measurement result of the three-dimensional relative depth. Superimpose. The operator can easily grasp the unevenness of the tissue or the like present in the three-dimensional space shown in the display image D by the display image D on which the wire frame F is superimposed.

The control unit 33 controls the “three-dimensional relative position” from the imaging unit 22 to the treatment unit 12 and the “three-dimensional relative position” from the imaging unit 22 to the tissue etc. existing in the three-dimensional space shown in the display image Depth is measured in real time. Further, regarding the medical device on which the virtual image is superimposed, shape data for virtual image is stored in the storage unit 36.
Therefore, when the medical device on which the virtual image is superimposed is a real medical device, the control unit 33 determines whether the tissue or the like present in the three-dimensional space shown in the display image is touched (contact determination )It can be performed.

When the operator moves the treatment unit 12 near the hernia gate H formed in the abdominal wall, if the mesh A on which the virtual image is superimposed is actually present at the same three-dimensional position, it appears in the display image 3 When the control unit 33 determines that the tissue or the like existing in the dimensional space is in contact, as shown in FIG. 13, the control unit 33 superimposes a display indicating that the contact occurs on the display image D.

The operator sees the virtual image VM1 of the mesh A in the display image D and intuitively grasps whether the selected mesh A can block the hernia gate H without being obstructed by the obstacle. Can.

The control unit 33 causes the virtual image of the mesh A to contact a tissue or the like existing in a three-dimensional space when the treatment unit 12 approaches the vicinity of the hernia gate H formed in the abdominal wall as shown in FIG. It may be deformed and superimposed. The operator can more intuitively understand whether the selected mesh A can close the hernia gate H by looking at the virtual image VM1 of the mesh A deformed along the tissue or the like.

FIG. 15 shows a display image D on which the virtual image VS2 of the stapler B selected to perform the separation of the large intestine in the medical system 200 is superimposed.

When the operator moves the treatment unit 12 to the right on the display screen D, as shown in FIG. 15A, when the stapler B is an actual medical device, the control unit 33 determines the tissue positioned on the right. It is determined that the touch image is touched, and a display indicating that the touch occurs is superimposed on the display image D.

When the operator moves the treatment unit 12 to the left on the display screen D, as shown in FIG. 15B, when the stapler B is an actual medical device, the control unit 33 determines the tissue located on the left. It is determined that the touch image is touched, and a display indicating that the touch occurs is superimposed on the display image D. The display indicating that the contact occurs is superimposed on the place where it is determined that the contact occurs.

The operator looks at the virtual image VS2 of the stapler B shown in the display image D, and intuitively determines whether the selected stapler B can disconnect the large intestine without being obstructed by the obstacle around it. It can be grasped.

(Effect of the second embodiment)
According to the medical system 200 of the present embodiment, the operator looks at the virtual image of the medical device shown in the display image D and intuitively determines whether the selected medical device is obstructed by the obstacle. It can be grasped. The time taken for trial and error in the selection of medical devices can be reduced, and the selection of appropriate types of medical devices can be supported.

Third Embodiment
A third embodiment of the present invention will be described with reference to FIGS. The present embodiment is different from the first and second embodiments in that the endoscope is not rigid but flexible. In the following description, the same reference numerals are assigned to components common to those described above, and redundant description will be omitted.

The whole structure of the medical system 300 which concerns on this embodiment is the same as the medical system 100 which concerns on 1st embodiment. The difference is that the endoscope 2C has a flexible insertion portion.
Next, the operation of the medical system 300 and an image generation method using the medical system 300 will be described with reference to FIGS. 16 to 18 by taking EMR surgery as an example.

FIG. 16 shows an endoscope 2C of the medical system 300.
In the medical system 300, as shown in FIG. 16A, a treatment tool to which the local injection needle 12b is attached as the treatment unit 12 is inserted through the lumen provided in the endoscope 2C. The operator inserts the local injection material into the submucosa layer slowly by piercing the local injection needle 12b at the peripheral edge beyond the lesion to lift the lesion. After that, as shown in FIG. 16B, the treatment tool to which the snare 12c is attached as the treatment unit 12 is inserted through the lumen provided in the endoscope 2C. The operator uses the snare 12c to separate the lesion.
The operator needs to select the snare 12c of the optimum size and shape in accordance with the size and shape of the lesion. Depending on the experience and skill of the caster, trial and error of the selection of the snare takes time.

Therefore, the operator sets the operation mode of the image processing of the control unit 33 to the “virtual image superposition mode” when introducing the treatment tool to which the local injection needle 12b is attached as the treatment unit 12 into the body. The operator operates the input unit 51 to select “Snare A” which is estimated to be the optimum size (medical device selection step).

FIG. 17 shows a display image D in which the virtual image VW1 of the selected snare A is superimposed on the position where the local injection needle 12b appears in the medical system 300.
The operator intuitively grasps the virtual image VM1 of the snare A shown in the display image D and that the selected snare A has a necessary and sufficient size for the separation of the lesion. Can.

FIG. 18 shows a display image in which the snare B smaller in size than the snare A is selected in the medical system 300 and the virtual image VW2 of the snare B is superimposed.
The operator should intuitively understand that the selected snare B does not have a necessary and sufficient size for dissection of the lesion by looking at the virtual image VM2 of the snare B shown in the display image D. Can.

(Effect of the third embodiment)
According to the medical system 300 of the present embodiment, even in the case of using a flexible endoscope, the operator looks at the virtual image of the medical device shown in the display image D, and the selected medical device is necessary and sufficient. Whether it has a size can be intuitively grasped without actually measuring the dimensions and the like in the three-dimensional space. The time taken for trial and error in the selection of medical devices can be reduced, and the selection of appropriate types of medical devices can be supported.

Fourth Embodiment
A fourth embodiment of the present invention will be described. The present embodiment is different from the first embodiment to the third embodiment in that the medical device selection process is linked with the stock management system of the medical device. In the following description, the same reference numerals are assigned to components common to those described above, and redundant description will be omitted.

The whole structure of the medical system 400 which concerns on this embodiment is the same as the medical system 100 which concerns on 1st embodiment.

The control unit 33 in which the operation mode is set to the “virtual image superposition mode” displays on the monitor 41 a medical device list and a message prompting the user to select the “type” and “size” of the medical device from the input unit 51 Let
The control unit 33 is connected to an in-hospital inventory management system of medical devices, and can display the inventory status of each medical device according to the displayed medical device list.
The operator can select the medical device to be displayed as a virtual image after confirming the presence or absence of the stock. If you select a medical device that does not have an actual inventory, you can not actually use the medical device immediately, even if the selected medical device is the optimal size. If it is possible to display the stock status of medical devices, medical devices can be selected from the medical devices that can actually be used, reducing the time taken for trial and error in the selection of medical devices, and of appropriate types of medical devices. Can support selection.

The present invention can be applied to a medical system provided with an endoscope.

100, 100B, 100C, 200, 300, 400

medical system

1, 1B treatment tool 10 insertion unit 11 operation unit 12 treatment unit 12a grasping member 12b local injection needle 12c snare 2, 2B, 2C endoscope 20 insertion unit 21 holding hand Unit 22 Imaging unit 3 Control device 33 Control unit 35 Memory 36 Storage unit 37 Input / output control unit 38 Image operation unit 4 Display device 41 Monitor 5 Input device 51 Input unit

Claims

Treatment tools,
An endoscope having an imaging unit;
A control device that generates a display image from the captured image acquired by the imaging unit;
A display device for displaying the display image;
And an input device for selecting a medical device,
The control device measures the relative position from the endoscope to the treatment tool,
The control device superimposes, on the display image, a virtual image of the medical device having a relative size at a position relative to the treatment tool shown in the display image based on the relative position. Do,
Medical system.
The control device measures the relative posture of the treatment tool with respect to the endoscope,
The control device superimposes, on the display image, a virtual image of the medical device, which is a posture relative to the treatment tool shown in the display image, based on the relative posture.
The medical system according to claim 1.
The control device has at least two operation modes of a normal mode and a virtual image superposition mode.
The control device generates the display image in which the virtual image is not superimposed, when the operation mode is set to the normal mode.
The control device generates the display image on which the virtual image is superimposed, when the operation mode is set to the virtual image superposition mode.
The medical system according to claim 1.
The control device measures relative depth to an object shown in the display image;
If the medical device on which the virtual image is superimposed on the display image is a real medical device based on the relative position and the relative depth, the control device is an object shown in the display image. Determine if it will touch,
The medical system according to any one of claims 1 to 3.
The control device superimposes, on the display image, a wire frame indicating unevenness of an object shown in the display image based on the relative depth.
The medical system according to claim 4.
The control device stores medical device information of the selectable medical device, and displays the selectable medical device on the display device based on the medical device information.
The medical system according to any one of claims 1 to 3.
The medical device information includes inventory information of the medical device in a hospital.
The medical system according to claim 6.
The control apparatus superimposes the virtual image of the medical device on the display image such that the virtual image of the medical device is translucent.
The medical system according to any one of claims 1 to 7.
In a medical system that generates a display image from a captured image acquired by an imaging unit of an endoscope
A medical device selection step of selecting a medical device using an input device;
A relative position measuring step of measuring a relative position from the endoscope to the treatment tool;
Virtual image superimposing step of superimposing a virtual image of the medical device having a relative size at a position relative to the treatment tool shown in the display image based on the relative position on the display image And
Image generation method.
In the relative position measurement step, in addition to the relative position, the relative posture of the treatment tool with respect to the endoscope is measured.
The virtual image superimposing step superimposes, on the display image, a virtual image of the medical device which is in a posture relative to the treatment tool shown in the display image based on the relative posture.
The image generation method according to claim 9.
Has at least two operation modes: normal mode and virtual image superposition mode,
The medical device selection step, the relative position measurement step, and the virtual image superposition step are performed only when the operation mode is set to the virtual image superposition mode.
The image generation method according to claim 9.
The relative position measurement step measures relative depth to an object shown in the display image,
In the virtual image superimposing step, when the medical device on which the virtual image is superimposed on the display image is an actual medical device based on the relative position and the relative depth, an object shown in the display image To determine whether or not the touch occurs, and when it is determined that the touch occurs, a display indicating the touch is superimposed on the display image.
The image generation method according to claim 9.