CN118830920A - Navigation apparatus, navigation method, and readable storage medium - Google Patents

Abstract

The application provides a navigation device, a navigation method and a readable storage medium, wherein the navigation device receives an intraoperative image of a lung of an operation object acquired by an image device and comprises the following steps: inputting an intra-operative image of a lung of an operative subject acquired by an image device into a pre-trained recognition network, recognizing and outputting one or more first lung segments in the intra-operative image; based on the one or more first lung segments, one or more target lung segments are matched from a plurality of second lung segments included in the planned path; and determining a navigation path for guiding the image equipment according to the one or more target lung segments, so that the navigation precision and efficiency of the navigation equipment are improved.

Description

Navigation device, navigation method and readable storage medium

Technical Field

The present application belongs to the field of medical devices, and in particular, relates to a navigation device, a navigation method and a readable storage medium.

Background Art

During bronchoscopic biopsy or surgical treatment, it is difficult for doctors to operate the slender bronchoscope to reach the target lesion due to the multiple branches and complex structure of the bronchi, as well as the susceptibility to interference from tissue deformation and respiratory movement during the operation. An accurate and real-time bronchoscope navigation system is needed for guidance.

In the prior art, the patient's lung CT (Computerized Tomography) images are usually collected before surgery, and the patient's three-dimensional bronchial model is reconstructed. The doctor obtains reference information such as lesion detection, surgical path planning, and surgical operation plan through visual interactive operations. During the operation, electromagnetic and optical sensors are often attached to the end of the catheter, combined with the intraoperative images of the bronchoscope, to perform registration and positioning, and navigate the surgical path. However, due to interference from factors such as lung respiratory movement and deformation of the lumen squeezed by the end of the catheter, the position of the bronchoscope during the operation cannot be accurately located; the static three-dimensional bronchial tree model generated by preoperative CT reconstruction cannot dynamically feedback changes in lung morphology, resulting in a non-intuitive user field of view during navigation.

Therefore, how to improve accurate real-time navigation during surgery is a problem that needs to be solved at present.

Summary of the invention

The embodiments of the present application provide a navigation device, a navigation method and a readable storage medium, which can solve the problem of improving the navigation accuracy and efficiency of the navigation device.

In a first aspect, an embodiment of the present application provides a navigation device receiving an intraoperative image of a lung of a surgical subject acquired by an image device, including:

A processor, coupled to the imaging device, is configured to:

Inputting the intraoperative image into a pre-trained recognition network, recognizing and outputting one or more first lung segments in the intraoperative image;

Based on the one or more first lung segments, matching one or more target lung segments from a plurality of second lung segments included in the planned path;

A navigation path for guiding an imaging device is determined based on the one or more target lung segments.

The processor is configured to:

Obtaining identifications of one or more first lung segments and a plurality of second lung segments, wherein the identifications include locations, names, and/or identification numbers of the lung segments;

One or more target lung segments are obtained by matching the identifiers of the one or more first lung segments with the identifiers of the multiple second lung segments.

Optionally, the planned path is represented by identifications of one or more second lung segments.

Optionally, in determining a navigation path for guiding an imaging device according to one or more target lung segments, the processor is further configured to:

When one or more target lung segments include a target lung segment, forming a navigation path using the target lung segment; or,

When the one or more target lung segments include at least two target lung segments, one of the target lung segments is selected to form a navigation path.

Optionally, the processor is further configured to:

Extract the features of the lung segmental orifices in intraoperative images;

One or more first lung segments in the output intraoperative image are identified based on the features of the lung segmental opening and the feature set in the recognition network.

Optionally, the processor is further configured to:

Annotating preoperative images of the lungs of a training subject and/or a surgical subject to obtain annotation information;

The recognition network is obtained by training based on the preoperative images and annotation information.

Optionally, the processor is further configured to:

Inputting a preoperative image of the lung of a training subject and/or a surgical subject into a feature transfer network to obtain a preoperative image after style transfer;

Annotate the preoperative image after style transfer to obtain annotation information;

The recognition network is obtained by training based on the pre-operative images and annotation information after style transfer.

Optionally, the processor is further configured to:

Annotating preoperative images of the lungs of a training subject and/or a surgical subject to obtain annotation information;

Input the preoperative image into the feature transfer network to obtain the preoperative image after style transfer;

The recognition network is obtained by training based on the pre-operative images and annotation information after style transfer.

Optionally, the processor is further configured to:

Acquire an intraoperative image of the lung of the training subject, and annotate the intraoperative image of the lung of the training subject to obtain annotation information;

The recognition network is obtained by training based on the intraoperative images and annotation information.

Optionally, the processor is configured to:

obtaining preoperative images of the lungs of the surgical subject;

Determine the lung segment of the surgical subject from a preoperative image of the lung of the surgical subject based on a depth-first algorithm or a breadth-first algorithm;

A second lung segment included in the planned path is determined from the lung segments of the surgical subject according to the target position.

In a second aspect, an embodiment of the present application provides a navigation method, including:

Inputting an intraoperative image of the lung of the surgical subject acquired by an imaging device into a pre-trained recognition network, and identifying and outputting one or more first lung segments in the intraoperative image;

Based on the one or more first lung segments, matching one or more target lung segments from one or more second lung segments included in the planned path;

A navigation path for navigating an imaging device is determined based on the one or more target lung segments.

In a third aspect, an embodiment of the present application provides a surgical robot, comprising: an imaging device, a navigation device, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the imaging device is used to acquire intraoperative images of the lungs of the surgical subject in real time, and the navigation device is used to receive intraoperative images of the lungs of the surgical subject acquired by the imaging device, wherein the navigation device further comprises a processor, the processor being coupled to the imaging device, and implementing some or all of the steps of the navigation method when the processor executes the computer program.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which may store a program, and when the program is executed by a processor, implements part or all of the steps of the navigation method in the second aspect above.

In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a surgical robot, enables the surgical robot to execute part or all of the steps of the navigation method in the second aspect.

It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

Compared with the prior art, the embodiments of the present application have the following beneficial effects: through the embodiments of the present application, the intraoperative image of the lungs of the surgical subject captured by the imaging device is input into a pre-trained recognition network to identify and output one or more first lung segments in the intraoperative image; based on the one or more first lung segments, one or more target lung segments are matched from the multiple second lung segments included in the planned path; and a navigation path for guiding the imaging device is determined based on the one or more target lung segments, thereby improving the navigation accuracy and efficiency of the navigation device.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

FIG1 is a schematic diagram of a flow chart of a navigation method provided by an embodiment of the present application;

FIG2 is a schematic diagram of a first lung segment in a navigation method according to an embodiment of the present application;

FIG3 is a schematic diagram of a preoperative image after annotation in a navigation method according to an embodiment of the present application;

FIG4 is a schematic diagram of a target lung segment in a navigation method in an embodiment of the present application;

FIG. 5 is a schematic diagram of a planned path in a navigation method according to an embodiment of the present application.

DETAILED DESCRIPTION

In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and/or combinations thereof.

It should also be understood that the term “and/or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

The embodiment of the present application provides a navigation device, which can be used in a surgical robot. The surgical robot may further include an imaging device, which is used to collect intraoperative images of the lungs of the surgical object in real time. The navigation device receives the intraoperative images of the lungs of the surgical object collected by the imaging device. The navigation device also includes a processor, which is coupled to the imaging device and is configured to execute a navigation method. FIG1 is a schematic diagram of a navigation method provided in an embodiment of the present application, and the method includes:

Step S20: inputting the intraoperative image of the lung of the surgical subject acquired by the imaging device into a pre-trained recognition network, and identifying and outputting one or more first lung segments in the intraoperative image.

In the embodiments of the present application, the intraoperative image refers to an image obtained by an imaging device in a scene requiring navigation, and the scene requiring navigation includes an inspection or surgical treatment process, such as a bronchoscopic biopsy or cancer resection surgery, etc. The imaging device may be a medical endoscope, such as a bronchoscope, etc.

The recognition network extracts features that can distinguish each lung segment in the intraoperative image, especially the features of the lung segment opening, such as geometric shape, texture, color and other features. According to the features of the lung segment opening and the feature set in the recognition network, one or more first lung segments in the intraoperative image are identified and output. As shown in FIG2 , a schematic diagram of the first lung segment in the navigation method of an embodiment of the present application is shown. The left main bronchi (left main bronchi) 201 and the right main bronchi (right main bronchi) 301 are both identified first lung segments. In the embodiment of the present application, the lung segment identified based on the intraoperative image is referred to as the first lung segment. The number of first lung segments may be 0, 1 or more. The feature set refers to multiple feature parameters of the recognition network, which can be determined by network training.

Optionally, after identifying and outputting one or more first lung segments in the intraoperative image in step S20, the following steps may also be included:

Obtain identifications of one or more first lung segments and multiple second lung segments, wherein the identifications are used to distinguish each lung segment, and the identifications include the lung segment position, name and/or identification number of the lung segment. The lung segment position may indicate the spatial position of the lung segment in the lung, and the identification number may indicate the sequence number of the lung segment among all lung segments. After identifying the first lung segment of the intraoperative image, the recognition network may output an identification corresponding to the first lung segment.

The embodiment of the present application identifies the first lung segment and obtains the corresponding identifier, so as to uniquely determine which lung segment of the multiple second lung segments included in the planned path the first lung segment in the intraoperative image specifically corresponds to, thereby guiding the navigation path of the imaging device.

Optionally, in the navigation device provided in the embodiment of the present application, the processor is further configured to pre-train the recognition network, mainly including two situations:

(I) Training based on preoperative images

(II) Training based on intraoperative images

For the first case, the preoperative images of the lungs of the training object can be annotated first, and/or the preoperative images of the lungs of the surgical object can be annotated to obtain the annotation information. In the embodiment of the present application, the object of the medical operation is referred to as the surgical object. For example, the patient who is to undergo surgery can be the surgical object, and the patient who is to undergo bronchoscopy can also be the surgical object. The medical operation may include the aforementioned examination or surgical treatment. Optionally, before the medical operation is performed on the surgical object, the preoperative image is usually taken first, so the preoperative image can be used for training. Optionally, the training object in the embodiment of the present application may not be the surgical object, for example, the surgical object to be subjected to the medical operation is A, and the training object may be B, C and/or D. The preoperative image may include such as computed tomography (CT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and ultrasound, etc., which can be presented as a two-dimensional, three-dimensional or four-dimensional (such as time-based or rate-based information) image. A bronchial tree model can be established based on the preoperative image, and the slice image based on the bronchial tree model can reflect the actual basic geometric structure and texture characteristics at each corresponding real position. The slice images can be understood as images obtained by observing the bronchial tree model at various observation positions and angles using a virtual imaging device, such as a virtual endoscope. The number of slice images is sufficient for network training, and a portion of the slice images can be selected as a training set and a portion of the slice images can be selected as a test set.

Of course, sagittal images, coronal images or horizontal images based on the bronchial tree model may also be used, and these images may also reflect the actual basic geometric structure and texture characteristics at the corresponding real positions.

Among them, the preoperative image can be annotated by automatically identifying or manually selecting the lung segment opening. Exemplarily, the lung segment opening is automatically identified by using a preliminarily trained recognition network, or the lung segment opening is manually identified. The purpose of annotation is to form a learning sample for network training, so as to successfully complete the network training. By automatically or manually identifying the features in the preoperative image, the lung segment in the preoperative image is identified, and the identified lung segment is annotated, so as to establish an association between the features of the preoperative image and the lung segment. In order to characterize this association, it can include recording annotation information, and the annotation information includes the lung segment position, position information, circumscribed rectangle, lung segment name and/or identification number, etc. Among them, the annotation information and the aforementioned identification can include the same information, such as the lung segment name, identification number or lung segment position. Among them, the position information refers to the position of the lung segment in the preoperative image. Optionally, a certain part of the lung segment is selected to represent the position of the lung segment, for example, the position of the lung segment opening is used to represent the position of the lung segment. Among them, the lung segment opening is usually circular or elliptical. The circumscribed rectangle of the lung segment opening, the length and width of the circumscribed rectangle can be determined first, and a point in the circumscribed rectangle can be selected as the position of the lung segment, such as the upper left corner or upper right corner of the rectangle. For easy observation, the annotation information can also be displayed in the preoperative image. As shown in Figure 3, the annotated preoperative image includes the circumscribed rectangle of the lung segment opening and the identification number. The preoperative image containing the annotation information can be displayed on a display device, and the display device can be set on a surgical robot or other location.

Then, the recognition network is trained based on the preoperative images and annotation information. In the model training stage, the training images and annotation information are input into the recognition network, and the recognition network extracts the features of each lung segment, especially the features of the lung segmental opening, through convolution operations. It can be understood that network training is a gradual process. Based on the initial network architecture, the initial network is trained using preoperative images and annotation information, and continuously improved, so as to obtain a recognition network that can ultimately achieve the expected recognition effect.

Optionally, in order to increase the learning samples and improve the effect of network training, the preoperative images are enhanced in the embodiments of the present application, wherein the enhancement may include rotation, cropping, translation or scaling, etc. When the original preoperative images have been annotated, the enhanced preoperative images also have the corresponding annotation information, and thus can also be used as learning samples for network training. The recognition network can be obtained by training based on the enhanced preoperative images and the annotation information.

The embodiment of the present application performs network training using preoperative images of the lungs of a surgical subject or a training subject, thereby obtaining a pre-trained recognition network.

Optionally, in the navigation device provided in the embodiment of the present application, the processor is further configured to:

Inputting a preoperative image of the lung of a training subject and/or a surgical subject into a feature transfer network to obtain a preoperative image after style transfer;

Annotate the preoperative image after style transfer to obtain annotation information;

The recognition network is obtained by training based on the pre-operative images and annotation information after style transfer.

Among them, the feature transfer network is used to transfer the style of the preoperative image, that is, to render the preoperative image into the style of the intraoperative image, that is, to transfer the style of the virtual endoscopic image domain to the real endoscopic image domain, including two stages such as style transfer and geometric shape smoothing. In the style transfer stage, it is necessary to retain the geometric shape information to the greatest extent. The basic principles in the geometric shape smoothing stage include having the same image style around the pixels of the same geometric shape; limiting the global geometric feature offset of the generated image as much as possible. The style difference between the preoperative image and the intraoperative image may include color, local brightness or contrast. The feature transfer convolutional network renders the preoperative image according to the style effect of the image device, and does not change the geometric shape structure and texture of the preoperative image to the greatest extent. The feature transfer network can adopt a GAN (Generative Adversatial Net) network, including a generation function and a discriminant function. The feature transfer network can adopt a feature transfer network in the prior art, which is not specifically limited in the embodiment of the present application. For example, the preoperative image is black and white, and the preoperative image after style transfer is color, or the preoperative image is color, and the preoperative image after style transfer is black and white.

The implementation principle of annotating the preoperative image after style transfer is the same as the aforementioned preoperative image annotating by automatic recognition or manual selection of the lung segment orifice. For the sake of brevity of the entire article, it will not be repeated here.

In the embodiment of the present application, a pre-operative image of the lungs of a surgical subject or a training subject is input into a feature transfer network to perform style transfer and then annotate, and then training is performed based on the style-transferred image to obtain a pre-trained recognition network.

Optionally, in the navigation device provided in the embodiment of the present application, the processor is further configured to:

Annotating preoperative images of the lungs of a training subject and/or a surgical subject to obtain annotation information;

Input the preoperative image into the feature transfer network to obtain the preoperative image after style transfer;

The recognition network is obtained by training based on the pre-operative images and annotation information after style transfer.

In the embodiment of the present application, pre-operative images of the lungs of a surgical object or a training object are used for annotation and then input into a feature transfer network for style transfer, and then network training is performed based on the pre-operative images after style transfer, thereby obtaining a pre-trained recognition network.

For the second case, it is optional to first obtain the intraoperative image of the lungs of the training object, annotate the intraoperative image of the lungs of the training object, and obtain the annotation information. In medical practice activities, in some scenarios, the intraoperative image of the lungs of the training object can be obtained. For example, during the operation of the training object, the doctor can obtain the intraoperative image of the lungs of the training object and accumulate it for subsequent network training. In this case, since the network training input is the intraoperative image, it is closer to the intraoperative image to be identified in the actual application, so the recognition efficiency is high, but a large number of intraoperative images need to be collected for network training to improve the generalization recognition ability of the network. Therefore, it is difficult to quickly meet the requirements by only relying on the intraoperative image of the lungs of the surgical object for network training, and the operation time is urgent, and there is usually no sufficient time for network training. Similarly, the annotation of the intraoperative image can be performed by automatically identifying the preoperative image as before or manually selecting the lung segment orifice. The implementation principle is the same. For the sake of brevity of the whole text, it will not be repeated here.

Then, the recognition network is trained based on the intraoperative images and annotation information. In the model training stage, the intraoperative images and annotation information are input into the recognition network, which extracts the features of each lung segment, especially the features of the lung segment orifice, through convolution operations to complete the recognition of the lung segment.

The embodiment of the present application performs network training by using intraoperative images of the lungs of the training subject and corresponding annotation information, thereby obtaining a pre-trained recognition network.

Step S40: Based on one or more first lung segments, one or more target lung segments are matched from multiple second lung segments included in the planned path. The target lung segment is used to represent the lung segment included in both the intraoperative image and the planned path. For example, the target lung segment is marked based on the segment intersection of the first lung segment identified in the intraoperative image and the second lung segment in the planned path. FIG4 is a schematic diagram of the target lung segment, in which the right main bronchus 301 is a lung segment included in both the first lung segment and the second lung segment.

Optionally, since the identifier can uniquely identify the lung segment, the processor is further configured to match and obtain one or more target lung segments based on the identifiers of one or more first lung segments and the identifiers of multiple second lung segments.

Optionally, in the navigation device provided in the embodiment of the present application, the processor is further configured to determine a plurality of second lung segments included in the planned path, specifically including:

Step 1: Obtain preoperative images of the patient's lungs.

Step 2: Based on the depth-first algorithm or the breadth-first algorithm, the lung segment of the surgical subject is determined according to the preoperative image of the lung of the surgical subject.

As before, a bronchial tree model of the surgical subject is established based on the preoperative image of the surgical subject's lungs, and the bifurcation nodes on the skeleton centerline are identified by extracting the skeleton centerline in the bronchial tree model. The bifurcation nodes and the bifurcation nodes constitute a connecting segment, namely a lung segment, which can be specifically implemented by the prior art, and the embodiments of the present application do not impose any restrictions on this. In the embodiments of the present application, lung segment is a general term in the prior art. Generally speaking, the lung segments of most people are distributed in the same way, but slight differences are not ruled out. Therefore, for the accuracy of medical operations, the lung segments of each surgical subject are confirmed in the embodiments of the present application, that is, after identifying the bifurcation nodes, all lung segments are determined based on the principle that adjacent bifurcation nodes constitute lung segments.

At the same time, in order to identify different lung segments, an identifier is recorded for each lung segment, and the identifier may include the lung segment position and lung segment name of the first lung segment.

The identification of each lung segment is traversed using a depth-first algorithm or a breadth-first algorithm, thereby assigning a unique identification number to each lung segment. In the embodiment of the present application, the identification number is referred to as the Airway ID. At the same time, the identification number can also be used as an identification. The identification number can correspond one-to-one with the lung segment name and the lung segment position, and both are used to uniquely identify each lung segment. The one-to-one correspondence between the identification number and the lung segment name and the lung segment position can be stored. The storage form is not limited, as long as any one of them is known in the subsequent use process, the other can be found.

Step 3: According to the target position, determine the second lung segment included in the planned path from the lung segments of the surgical object. Specifically, according to all lung segments of the surgical object and the target position, determine the second lung segment, wherein the second lung segment is used to form the planned path.

The target position can be the position of the lesion, or it can be a position related to the medical operation customized by the doctor. When the third lung segment and the target position of the surgical object are known, a planning path to reach the target position can be planned. In the embodiment of the present application, the lung segment included in the planning path is called the second lung segment. The embodiment of the present application does not specifically limit the method for planning the path, and the shortest distance planning method or the shortest time planning method can be used. The planning path can be represented by the identification of the lung segment, and can be optionally represented by an identification number, a lung segment name, etc. For example, the planning path is 3-3 to 3-7 to 4-3 to 5-5 to 6-4, where 3-3, 3-7, 4-3, 5-5 and 6-4 are the identification numbers of the second lung segment. As shown in FIG5 , it is a schematic diagram of the planned path, in which the names of the lung segments in the planned path are identified: right main bronchi 501 (right main bronchi) to right intermediate bronchus 502 (intermediate bronchus) to right lower lobe bronchus 503 (inferior lobarbronchus) to right lower lobe outer base segment 504 (B9) to right lower lobe outer base segment subsegment 505 (B9b).

When the first lung segment can be identified in step S20, but the target lung segment cannot be matched in step S40, there may be two reasons for this: the first lung segment identified and output by the recognition network is wrong or has reached the wrong position. In order to eliminate the factor of the first lung segment error, the embodiment of the present application also includes:

Verify that the first lung segment is correct based on the bronchial tree structure.

Since the bronchial tree is a tree structure, it branches repeatedly step by step, becoming thinner and thinner, like a tree branch. It can be understood that the thinner branches are the child nodes of the thicker branches, and the thicker branches are the parent nodes of the thinner branches. The correct navigation path can go from the parent node to the child node, but the reverse is wrong. Therefore, by identifying whether the first lung segment conforms to the relationship between the parent node and the child node, it can be determined whether the first lung segment is correct. For example, according to the intraoperative image of the current frame, the 5-6 lung segment is identified, while the intraoperative image of the previous frame identifies the 6-2 lung segment, which obviously does not conform to the tree structure, so the first lung segment is wrong.

Optionally, verification of whether the first lung segment is correct may be performed after the first lung segment can be identified in step S20 , or after the target lung segment cannot be matched in step S40 .

The embodiment of the present application verifies whether the first lung segment is correct according to the tree structure of the bronchial tree, thereby eliminating the problem of incorrect recognition of the first lung segment, which is more conducive to successfully completing the path navigation.

Step S60: determining a navigation path for guiding the imaging device according to one or more target lung segments.

When there is no target lung segment, it indicates that the intraoperative image does not contain the next moving target, and the imaging device can continue to move straight; when a target lung segment is included, the target lung segment is the next moving target, and the imaging device moves toward the target lung segment, and the target lung segment is used to form a navigation path; when multiple target lung segments are included, one of the target lung segments can be selected as the next moving target according to the selection strategy, and the navigation path is composed of this target lung segment. It can be understood that the target lung segments identified and selected according to each intraoperative image are all part of the navigation path, and they together constitute the entire navigation path. Among them, the selection strategy may include selecting a target lung segment with a high probability of correct identification, or including selecting a target lung segment with a preceding identification number.

Optionally, the surgical robot may further include a display device on which real-time intraoperative images of the surgical site and/or catheter instrument captured by the imaging device, as well as navigation paths, etc., may be displayed, so as to provide more intuitive navigation guidance.

The embodiment of the present application realizes fast and accurate navigation by identifying the first lung segment in the intraoperative image, matching the target lung segment from multiple second lung segments included in the planned path based on the first lung segment, and then forming a navigation path according to the target lung segment.

It should be noted that other sorting schemes that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should also be within the protection scope of the present invention and will not be described in detail here.

An embodiment of the present application also provides a surgical robot, including an imaging device, a navigation device, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the imaging device is used to acquire intraoperative images of the lungs of the surgical subject in real time, and the navigation device is used to receive intraoperative images of the lungs of the surgical subject acquired by the imaging device, wherein the navigation device also includes a processor coupled to the imaging device, and the aforementioned navigation method is implemented when the processor executes the computer program.

Exemplarily, the computer program can be divided into one or more modules/units, one or more modules/units are stored in a memory and executed by a processor to complete the present invention. One or more modules/units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the surgical robot.

The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

The memory may be an internal storage unit of the surgical robot, such as a hard disk or memory of the surgical robot. The memory may also be an external storage device of the surgical robot, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the surgical robot. Furthermore, the memory may include both an internal storage unit and an external storage device of the surgical robot. The memory is used to store computer programs and other programs and data required by the terminal device. The memory may also be used to temporarily store data that has been output or is to be output.

The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

As used in the specification of this application and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if the described condition or event is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce the described condition or event is detected" or "in response to detecting the described condition or event" depending on the context.

In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

In the embodiments provided by the present invention, it should be understood that the disclosed devices/terminal equipment and methods can be implemented in other ways. For example, the device/terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated module/unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims (12)

1. A navigation device for receiving intra-operative images of a lung of a surgical object acquired by an image device, comprising:

a processor, coupled with the image device, configured to:

inputting the intra-operative image into a pre-trained recognition network, recognizing and outputting one or more first lung segments in the intra-operative image;

Based on the one or more first lung segments, one or more target lung segments are matched from a plurality of second lung segments included in the planned path;

A navigation path for guiding the image device is determined from the one or more target lung segments.

2. The navigation device of claim 1, wherein the processor is configured to:

Obtaining an identification of the one or more first lung segments and the plurality of second lung segments, wherein the identification comprises a location, a name, and/or an identification number of a lung segment;

and obtaining the one or more target lung segments according to the matching of the identification of the one or more first lung segments and the identification of the plurality of second lung segments.

3. The navigation device of claim 2, wherein the planned path is represented by an identification of the one or more second lung segments.

4. The navigation device of claim 1, wherein in the determining a navigation path for guiding the image device from the one or more target lung segments, the processor is further configured to:

When the one or more target lung segments include a target lung segment, composing the navigation path with the target lung segment; or alternatively

When the one or more target lung segments include at least two target lung segments, one of the target lung segments is selected to form the navigation path.

5. The navigation device of claim 1, wherein the processor is further configured to: extracting features of a lung segment opening in the intra-operative image;

The one or more first lung segments in the intra-operative image are identified and output based on the features of the lung segment ostium and a set of features in the identification network.

6. The navigation device of claim 1, wherein the processor is further configured to:

labeling the training object and/or the preoperative image of the lung of the surgical object to obtain labeling information;

training according to the preoperative image and the labeling information to obtain the identification network.

7. The navigation device of claim 1, wherein the processor is further configured to:

inputting preoperative images of the lungs of a training object and/or the operating object into a feature migration network to obtain preoperative images after style migration;

Labeling the preoperative image after style migration to obtain labeling information;

training according to the preoperative image and the labeling information after style migration to obtain the identification network.

8. The navigation device of claim 1, wherein the processor is further configured to:

labeling the training object and/or the preoperative image of the lung of the surgical object to obtain labeling information;

inputting the preoperative image into a feature migration network to obtain a preoperative image after style migration;

Training according to the preoperative image after style migration and the labeling information to obtain the identification network.

9. The navigation device of claim 1, wherein the processor is further configured to:

acquiring an intra-operative image of the lung of a training object, and labeling the intra-operative image of the lung of the training object to obtain labeling information;

training according to the intra-operative image and the labeling information to obtain the identification network.

10. The navigation device of claim 1, wherein the processor is configured to: acquiring a pre-operative image of a lung of the surgical object;

determining a lung segment of the surgical object from a pre-operative image of a lung of the surgical object based on a depth-first algorithm or a breadth-first algorithm;

A second lung segment comprised by the planned path is determined from the lung segments of the surgical object according to the target position.

11. A navigation method, comprising:

inputting the intra-operative image into a pre-trained recognition network, recognizing and outputting one or more first lung segments in the intra-operative image;

based on the one or more first lung segments, one or more target lung segments are matched from one or more second lung segments included in the planned path;

A navigation path for navigating the image device is determined from the one or more target lung segments.

12. Computer can the storage medium is read and the data is read, the computer readable storage medium stores a computer program, the method of claim 11, wherein the computer program is executed by a processor.