TWM568113U - Airway model generation system and intubation assist system - Google Patents

Abstract

The disclosure provides an airway model generation system, including an endoscope device and a computing device, to establish a 3D model of a patient's airway by simultaneous localization and mapping (SLAM) technology. The disclosure also provides an intubation assist system to assist the medical staff in the intubation treatment by using the 3D models of the patients and machine learning techniques.

Description

Airway model generation system and intubation assist system

This case relates to an endoscope system, in particular to an airway model generating system and an intubation assisting system.

When a patient needs general anesthesia or first aid, and the patient is unable to breathe on his or her own, the patient is usually intubated. However, medical staff often rely on experience to perform intubation operations and may injure patients if inadvertently.

In view of this, the present invention provides an airway model generation system to establish a three-dimensional model of a patient's airway, and utilizes a three-dimensional model of a plurality of patients and machine learning techniques to provide an intubation assist system that assists the medical staff in intubating treatment.

The airway model generation system includes an endoscope device and a computer device. The endoscope device comprises a flexible tube, a camera module and a communication module. The photographic module is located at the front end of the flexible tube to capture multiple airway images of the flexible tube into the airway. The communication module is coupled to the camera module to send the airway images captured by the camera module. The computer device communicates with the communication module of the endoscope device to obtain the airway images transmitted by the communication module, and according to the airway images, uses a real-time positioning and map construction (SLAM) technology to establish a three-dimensional model of the airway.

The cannula assist system includes an endoscope device and a computer device. The endoscope device comprises a flexible tube, a camera module and a communication module. The photographic module is located at the front end of the flexible tube to capture a plurality of target airway images during the process of the flexible tube entering the target patient's target airway. The communication module is coupled to the camera module to send the target airway images captured by the camera module. The computer device comprises an input module, a storage module, a processing module and an output module. The input module is configured to receive a target patient data of the target patient. The storage module stores a patient database, the patient database includes an airway data and a pathological data corresponding to each patient, and each airway data includes a plurality of airway images and a three-dimensional model of the airway corresponding to the patient's one airway. The processing module inputs the pathological data and the three-dimensional model of the patients to a first learning model. The first learning model provides a first logic for evaluating a correlation of one or more feature values of the pathological data with a corresponding three-dimensional model of the airway, and inputting the target patient data to the first learning model to A logic finds a similar one of the three-dimensional models. The processing module further determines, according to the target airway image, a position of the front end of the flexible tube in the similar three-dimensional model to generate a guiding information according to the position, and the guiding information is output by the output module.

In summary, the embodiment of the present invention provides an airway model generating system, which can establish a three-dimensional model of a patient's airway during the intubation process. In addition, the embodiment of the present invention also provides an intubation assisting system, which constructs a three-dimensional airway model of each patient and a corresponding airway image, and inputs the learning model. Through machine learning to find out the correlation between pathological data and the three-dimensional model of the airway, and to find out the association between airway imaging and pathological data, it can assist the intubation operation of medical staff and remind them of possible diseases.

1 is a schematic structural diagram of an airway model generating system and an intubation assisting system according to an embodiment of the present invention. The airway model generation system and the intubation assistance system include an endoscope device 100 and a computer device 200. The airway model generation system will be described first below.

Referring to FIG. 1 and FIG. 2 together, FIG. 2 is a block diagram of an airway model generating system according to an embodiment of the present invention. The endoscope device 100 includes a flexible tube 110, a grip portion 120, a photographing module 130, and a communication module 140. The flexible tube 110 is coupled to the grip portion 120 for the medical personnel to hold the grip portion 120 to insert the flexible tube 110 into the airway of the patient. The front end of the flexible tube 110 is provided with a photographic module 130 for capturing an image in front of the flexible tube 110. Thus, during the process of the flexible tube 110 entering the patient's mouth and deep into the airway, the airway image can be captured continuously, intermittently, or triggered. The photography module 130 can include one or more photographic lenses, which can be photosensitive coupling elements (CCDs) or complementary metal oxide semiconductor (CMOS) image sensors. The communication module 140 can support wired communication technology or wireless communication technology. The wired communication technology can be, for example, low voltage differential signal transmission (LVDS), composite video broadcast signal (CVBS), etc., and the wireless communication can be, for example, wireless fidelity (WiFi). , wireless display (WiDi), wireless home digital interface (WHDI). The communication module 140 is coupled to the camera module 130 to transmit the captured airway image to the computer device 200.

The computer device 200 includes a processing module 210 and a communication module 220. The communication module 220 supports the same communication technology as the communication module 140 of the endoscope device 100, and is connected to the communication module 140 of the endoscope device 100 to obtain the airway image. The processing module 210 is coupled to the communication module 220 to establish a three-dimensional model of the air channel according to the airway image using SLAM technology. The processing module 210 is a processor with computing power such as a central processing unit (CPU), a graphics processing unit (GPU), and a visual processing unit (VPU). Processing module 210 can include one or more of the one or more processors described above.

In some embodiments, computer device 200 is a computing device.

In some embodiments, computer device 200 is comprised of a plurality of identical or different computing devices, such as a distributed computing architecture or a computer clustering technique.

The computer device 200 further includes a storage module 230, an input module 240, and an output module 250, which are respectively coupled to the processing module 210. The storage module 230 is a non-transitory storage medium for storing the airway image. The output module 250 can be an image output device, such as one or more displays, for displaying airway images. The input module 240 can be a human-machine interface, including a mouse, a keyboard, a touch screen, etc., for the medical staff to operate the computer device 200.

In some embodiments, the endoscope device 100 can also be equipped with a display (not shown) to display the airway image captured by the camera module 130.

In some embodiments, if the endoscope device 100 is equipped with a display, the computer device 200 may not be equipped with a display.

In some embodiments, different from the aforementioned endoscope device 100 and computer device 200 are two detachable individuals, the endoscope device 100 and the computer device 200 are integrated in the same electronic device.

Referring to FIG. 3, it is a flowchart of a method for generating an airway model according to an embodiment of the present invention, which is performed by the foregoing processing module 210 to implement the foregoing SLAM technology. First, the airway image stored in the storage module 230 is read and loaded (step S310). Next, the airway image is preprocessed to remove the noise region in the airway image (step S320). The noise area may be, for example, a mucosa, a bubble, or the like that affects image interpretation. In step S330, the complex feature points of the airway image are captured by the feature region detection algorithm. The feature region detection algorithm may be, for example, an algorithm such as an accelerated robust feature (SURF), a scale invariant feature transform (SIFT), or a directed BRIEF (ORB). Therefore, the moving direction and displacement of the flexible tube 110 can be converted according to the position and size change of the corresponding feature points on each airway image to reconstruct the three-dimensional model (step S340).

In some embodiments, the image captured by the camera module 130 having two lenses is available to the processing module 210 to perform a binocular vision SLAM calculus to reconstruct the three-dimensional model.

FIG. 4 is a block diagram of an airway model generating system according to another embodiment of the present invention. The difference from FIG. 2 is that the endoscope device 100 of the present embodiment further includes an inertial measurement module 150. The inertial measurement module 150 includes at least one inertial measurement unit 151 disposed on the flexible tube 110 (as shown in FIG. 1). The inertial measurement unit is used to obtain an inertial signal, such as an accelerometer, whereby the direction of movement and acceleration of the flexible tube 110 can be known. The inertial signal is transmitted to the computer device 200 through the communication module 140. Thus, the computer device 200 establishes a three-dimensional model of the airway using one of the aforementioned SLAM techniques, namely, a visual inertial odometer (VIO) technique, based on the inertial signal and the airway image.

In some embodiments, the inertial measurement unit 151 is evenly distributed along the long axis direction of the flexible tube 110. In other words, on the flexible tube 110, the inertial measurement unit 151 is disposed at a distance. Thereby, the bending deformation, the displacement direction and the displacement amount of each position of the flexible tube 110 can be known through the inertia signals of the inertial measurement units 151.

5 is a flow chart of a method for generating an airway model according to another embodiment of the present invention. The difference from FIG. 3 is that the endoscope device 100 (shown in FIG. 4) of the present embodiment further includes an inertial measurement module. 150. Therefore, after the computer device 200 obtains the feature points according to the foregoing steps S310 to S330, the moving direction and the displacement of the flexible tube 110 are converted according to the position and size change of the features on each airway image and the inertia signal, thereby reconstructing The three-dimensional model is derived (step S360). In addition, before the step S360, the inertia signal may be processed to filter out the noise of the inertial signal (step S350). Here, step S250 is not limited to being performed between step S330 and step S360, as long as it is performed before step S360. The manner of filtering out inertial signal noise may be, for example, a Kalman filter, a Gaussian filter, or a particle filter.

In some embodiments, the foregoing step S320, that is, pre-processing the airway image to remove the noise region in the airway image is to use a machine learning method to identify the noise region and remove it. In other words, the airway image of each patient can be input into a learning model, which can be selected from the categories of supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning, such as neural networks, random Forests, support vector machines (SVMs), decision trees, or clusters. The learning model is used to estimate the correlation between specific feature points in the airway image and the noise area, and to indicate the noise area in the airway image.

Next, the intubation assist system is described. The intubation assist system is used to assist the medical professional in correct operation when the intubation is performed on the current target patient, thereby avoiding an operation error and injuring the patient. Please refer to FIG. 1, FIG. 2, FIG. 4 and the foregoing description for the hardware composition of the intubation assisting system, and the detailed description thereof will not be repeated here.

Referring to Fig. 6, a schematic diagram of the operation of the intubation assisting system of an embodiment of the present invention is shown. Here, it is particularly noted that the storage module 230 of the computer device 200 stores the patient database. The patient database contains airway data 310 and pathology data 320 for each patient. The airway data 310 includes an airway image 311 and an airway three-dimensional model 312 that has been reconstructed by the foregoing methods. The pathological data 320 refers to the patient's rickets data, health check data, and the like. Before each intubation, the medical staff will input the target patient data (such as gender, height, weight, etc.) and/or medical record data of the current target patient through the input module 240, and the target patient data will be added. To the patient database. The input of this information can be done manually or by other means (such as reading files, reading wafers, accessing electronic medical records, etc.).

The processing module 210 inputs the pathological data 320 and the three-dimensional model 312 of the patient to the first learning model 330. The first learning model 330 can be selected from the group consisting of supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning, such as neural networks, random forests, support vector machines (SVMs), decision trees, or clusters. The first learning model 330 provides a first logic for evaluating the correlation of one or more of the feature values of the pathology data 320 with the corresponding three-dimensional model 312 of the airway. The first logic refers to calculating the probability of the airway three-dimensional model 312 corresponding to each patient or part of the patient based on the values, weights, and the like of the one or more feature values. In some embodiments, a plurality of representative airway model templates may also be generated according to the three-dimensional model 312 of the patients, and the first logic calculates a correspondence according to the values, weights, and the like of one or more feature values. The probability of each airway model template. For example, the performance of certain eigenvalues represents an airway type that is prone to difficult airway.

After the foregoing training, the processing module 210 inputs the target patient data to the first learning model 330 to find a similar one of the three-dimensional models 312 according to the first logic (that is, the highest probability one). Therefore, when the medical staff performs the intubation operation, the processing module 210 determines the front end of the flexible tube 110 through the aforementioned SLAM technology or VIO technology according to the airway image of the target patient (hereinafter referred to as the target airway image) or the inertial signal. A location in the similar three-dimensional model 312 to generate a navigational information based on the location. The navigation information is, for example, a guide to the direction. The output module 250 can display the navigation information through text, graphics, etc. through the display, or/and other forms of output, such as a speaker, and output the navigation information in other forms such as voice.

In some embodiments, the processing module 210 also inputs the airway image 311 of the patients to the second learning model 340. The second learning model 340 can be selected from the group consisting of supervised learning, unsupervised learning, semi-supervised learning, and enhanced learning, such as neural networks, random forests, support vector machines (SVMs), decision trees, or clusters. The second learning model 340 provides a second logic for evaluating the correlation of one or more of the feature values of the airway images 311 with at least one of the corresponding pathology data 320. The second logic refers to calculating the probability of suffering various diseases according to the values, weights, and the like of one or more characteristic values in the airway image 311. After training, the processing module 210 inputs the target airway image of the target patient to the second learning model to assess the probability of occurrence of one or more diseases in accordance with the second logic. The output module 250 can display the name of the disease that may be affected by text, graphic, etc. through the display, or/and other forms of output, such as a speaker, and output in other forms such as voice.

In summary, the embodiment of the present invention provides an airway model generating system, which can establish a three-dimensional model of a patient's airway during the intubation process. In addition, the embodiment of the present invention also provides an intubation assisting system, which constructs a three-dimensional airway model of each patient and a corresponding airway image, and inputs the learning model. Through machine learning to find out the correlation between pathological data and the three-dimensional model of the airway, and to find out the association between airway imaging and pathological data, it can assist the intubation operation of medical staff and remind them of possible diseases.

100‧‧‧Endoscope device

110‧‧‧ flexible tube

120‧‧‧ grip

130‧‧‧Photography module

140‧‧‧Communication Module

150‧‧‧Inertial Measurement Module

151‧‧‧Inertial measurement unit

200‧‧‧Computer installation

210‧‧‧Processing module

220‧‧‧Communication Module

230‧‧‧ storage module

240‧‧‧ input module

250‧‧‧Output module

310‧‧‧ airway data

311‧‧‧ airway imaging

312‧‧‧3D model

320‧‧‧Pathological data

330‧‧‧First Learning Model

340‧‧‧Second learning model

S310‧‧‧Steps

S320‧‧‧Steps

S330‧‧‧Steps

S340‧‧‧Steps

S350‧‧‧ steps

S360‧‧‧Steps

1 is a schematic structural view of an airway model generating system and an intubation assisting system according to an embodiment of the present invention. 2 is a block diagram showing an airway model generating system according to an embodiment of the present invention. FIG. 3 is a flow chart of a method for generating an airway model according to an embodiment of the present invention. 4 is a block diagram showing an airway model generating system according to another embodiment of the present invention. FIG. 5 is a flow chart of a method for generating an airway model according to another embodiment of the present invention. FIG. 6 is a schematic view showing the operation of the intubation assisting system according to an embodiment of the present invention.

Claims (8)

An airway model generating system comprises: an endoscope device comprising: a flexible tube; a photographic module located at a front end of the flexible tube to capture a plurality of the flexible tube into an airway And a communication module coupled to the camera module to transmit the airway images captured by the camera module; and a computer device communicatively connecting the communication module of the endoscope device to The airway images transmitted by the communication module are obtained, and a three-dimensional model of the airway is established by using the Instant Location and Map Construction (SLAM) technology according to the airway images. The airway model generating system of claim 1, wherein the endoscope device further comprises an inertial measurement module, the inertial measurement module comprising at least one inertial measurement unit to obtain at least one inertial signal, the computer device according to the At least one inertial signal and the airway images are created using a visual inertial odometer (VIO) technique to create a three-dimensional model of the airway. The airway model generating system of claim 2, wherein the at least one inertial measurement unit is evenly distributed along a long axis direction of the flexible tube. The airway model generating system of claim 2, wherein the computer device further filters out noise for the at least one inertial signal. The airway model generating system of claim 1, wherein the computer device comprises a processing module configured to perform the following steps: loading the airway images; and extracting the airways by using a feature region detecting algorithm The complex feature points of the image; and according to the position and size changes of the feature points of each airway image, the moving direction and displacement of the flexible tube are converted, and the three-dimensional model is reconstructed by SLAM technique. The airway model generating system of claim 5, wherein the processing module further preprocesses the airway images to remove noise regions in the airway images. An intubation assisting system comprising: an endoscope device comprising: a flexible tube; a camera module located at a front end of the flexible tube for capturing the flexible tube into a target airway of a target patient a plurality of target airway images in the process; and a communication module coupled to the camera module to transmit the target airway images captured by the camera module; and a computer device comprising: an input module, Receiving a target patient data of the target patient; a storage module storing a patient database, the patient database comprising an airway data and a pathological data corresponding to each patient, each of the airway data comprising a plurality of airways corresponding to the patient a three-dimensional model of the airway image and the airway; a processing module, the pathological data of the patient and the three-dimensional model are input to a first learning model, the first learning model providing a first logic for evaluating the Correlation of one or more feature values in the pathological data with the corresponding three-dimensional model of the airway, and inputting the target patient data to the first learning model to And determining, by the first logic, a similar one of the three-dimensional models, the processing module further determining, according to the target airway image, a position of the front end of the flexible tube in the similar three-dimensional model, according to the position Generating a guiding information; and an output module outputting the guiding information. The intubation assistance system of claim 7, wherein the processing module further inputs the airway images of the patients to a second learning model, the second learning model providing a second logic for evaluating the Correlation of one or more characteristic values in the airway image with at least one disease in the corresponding pathological data, and inputting the target airway images of the target patient to the second learning model to be based on the second logic Assess the probability of developing at least one disease.