WO2024180593A1

WO2024180593A1 - Image processing device, image processing method, and storage medium

Info

Publication number: WO2024180593A1
Application number: PCT/JP2023/007007
Authority: WO
Inventors: 雅弘西光
Original assignee: 日本電気株式会社
Priority date: 2023-02-27
Filing date: 2023-02-27
Publication date: 2024-09-06
Also published as: WO2024180796A1

Abstract

This image processing device 1X comprises an acquisition means 30X, a detection means 31X, a first determination means 32X, and a second determination means 33X. The acquisition means 30X acquires an endoscopic image obtained by photographing a subject. The detection means 31X detects a lesion region, that is, a candidate region for a lesion in the subject in the endoscopic image, on the basis of the endoscopic image. The first determination means 32X determines whether or not the endoscopic image is suitable for determining a progression degree or an invasion depth of the lesion on the basis the size of the lesion region and/or the degree of reliability in terms of the likelihood of a lesion in the lesion region. The second determination means 33X determines the progression degree or the invasion depth on the basis of an endoscopic image which is determined as being suitable for determining the progression degree or the invasion depth.

Description

Image processing device, image processing method, and storage medium

This disclosure relates to the technical fields of image processing devices, image processing methods, and storage media that process images acquired during endoscopic examinations.

　Image processing systems that process images of the inside of organ lumens have been known for some time. For example, Patent Document 1 discloses a medical image processing device that detects lesion candidates from medical images and identifies the malignancy of the detected lesion candidates and the organs contained in the medical images.

JP 2021-083821 A

When judging the stage of progression (including the depth of invasion, the same applies below) from endoscopic images, many of the endoscopic images obtained in time series contain noise such as blur, shine, and splashes of water, and are not suitable for judging the stage of progression. Furthermore, images with little noise that capture areas that appear to be lesions are not necessarily suitable for judging the stage of progression.

In consideration of the above-mentioned problems, one of the objectives of the present disclosure is to provide an image processing device, an image processing method, and a storage medium that can appropriately determine the degree of progress.

One aspect of the image processing device is
An acquisition means for acquiring an endoscopic image of a subject;
a detection means for detecting a lesion area, which is a candidate area for a lesion of the subject in the endoscopic image, based on the endoscopic image;
a first determination means for determining whether or not the endoscopic image is suitable for determining the progression or depth of the lesion based on at least one of the size of the lesion area or the reliability of the lesion area with respect to the lesion-likeness;
a second determination means for determining the degree of progression or the depth of invasion based on the endoscopic image determined to be an image suitable for determining the degree of progression or the depth of invasion;
The image processing device has the following features.

One aspect of the image processing method includes:
The computer
An endoscopic image of the subject is acquired,
detecting a lesion area in the endoscopic image, the lesion area being a candidate area for a lesion in the subject based on the endoscopic image;
determining whether the endoscopic image is suitable for determining the progression or depth of the lesion based on at least one of the size of the lesion area or the reliability of the lesion area;
determining the degree of progression or the depth of invasion based on the endoscopic image determined to be an image suitable for determining the degree of progression or the depth of invasion;
An image processing method.

One aspect of the storage medium is
An endoscopic image of the subject is acquired,
detecting a lesion area in the endoscopic image, the lesion area being a candidate area for a lesion in the subject based on the endoscopic image;
determining whether the endoscopic image is suitable for determining the progression or depth of the lesion based on at least one of the size of the lesion area or the reliability of the lesion area;
The storage medium stores a program that causes a computer to execute a process for determining the degree of progression or depth of invasion based on the endoscopic image that has been determined to be an image suitable for determining the degree of progression or depth of invasion.

As an example of one effect of the present disclosure, it becomes possible to appropriately determine the degree of progression (including the depth of invasion).

1 shows a schematic configuration of an endoscopic examination system. 2 shows the hardware configuration of an image processing device. 4 is a diagram showing an overview of a progress determination process executed by the image processing device in the first embodiment; FIG. 4 is an example of a functional block of a progress determination process in the first embodiment. 1 shows a first display example displayed on a display device during an endoscopic examination. 13 shows a second display example displayed on the display device during endoscopic examination. 5 is an example of a flowchart illustrating an overview of a process executed by an image processing device during an endoscopic examination in the first embodiment. FIG. 13 is a schematic configuration diagram of an endoscopic inspection system according to a modified example. FIG. 11 is a block diagram of an image processing device according to a second embodiment. 13 is an example of a flowchart executed by the image processing apparatus in the second embodiment.

Below, embodiments of an image processing device, an image processing method, and a storage medium will be described with reference to the drawings.

First Embodiment
(1) System Configuration FIG. 1 shows a schematic configuration of an endoscopic examination system 100. As shown in FIG. 1, the endoscopic examination system 100 is a system that detects a lesion site, which is a site of a subject suspected of having a lesion, based on an image captured by an endoscope, judges the progression of the lesion at the detected lesion site, and presents the judgment result. Hereinafter, the "progression level" may refer to a comprehensive degree (grade) of progression of a lesion including the depth of invasion (degree of invasion), or may be the depth of invasion (degree of invasion) itself. The endoscopic examination system 100 mainly includes an image processing device 1, a display device 2, and an endoscope scope 3 that is connected to the image processing device 1 and is handled by an examiner such as a doctor who performs an examination or treatment.

The image processing device 1 acquires images (also called "endoscopic images Ia") captured by the endoscope scope 3 in a time series from the endoscope scope 3, and displays a screen based on the endoscopic images Ia on the display device 2. The endoscopic images Ia are images captured at a predetermined frame cycle during at least one of the processes of inserting or ejecting the endoscope scope 3 into the subject. In this embodiment, the image processing device 1 detects, from the time series of endoscopic images Ia, an endoscopic image Ia in which a region that is a candidate for a lesion site (also called a "lesion region") is present. The image processing device 1 then selects an image from the detected endoscopic images Ia that is suitable for determining the progression of the lesion, determines the progression of the lesion for the selected image, and presents information on the determination result. Furthermore, when an image suitable for determining the progression of the lesion cannot be obtained, the image processing device 1 outputs a suggestion for capturing such an image.

The display device 2 is a display or the like that performs a predetermined display based on a display signal supplied from the image processing device 1.

The endoscope 3 mainly comprises an operation section 36 that allows the examiner to perform predetermined inputs, a flexible shaft 37 that is inserted into the subject's organ to be imaged, a tip section 38 that incorporates an imaging section such as a miniature image sensor, and a connection section 39 for connecting to the image processing device 1.

The configuration of the endoscopic examination system 100 shown in FIG. 1 is an example, and various modifications may be made. For example, the image processing device 1 may be configured integrally with the display device 2. In another example, the image processing device 1 may be configured from multiple devices.

The subject of the endoscopic examination in the present disclosure may be any organ that can be examined endoscopically, such as the large intestine, esophagus, stomach, or pancreas. For example, endoscopes that are the subject of the present disclosure include pharyngeal endoscopes, bronchoscopes, upper gastrointestinal endoscopes, duodenoscopes, small intestinal endoscopes, colonoscopes, capsule endoscopes, thoracoscopes, laparoscopes, cystoscopes, cholangioscopes, arthroscopes, spinal endoscopes, vascular endoscopes, and epidural endoscopes. Examples of the pathology of the lesion site that is the subject of the endoscopic examination include (a) to (f) below.

(a) Head and neck: pharyngeal cancer, malignant lymphoma, papilloma (b) Esophagus: esophageal cancer, esophagitis, hiatal hernia, esophageal varices, esophageal achalasia, esophageal submucosal tumor, benign esophageal tumor (c) Stomach: gastric cancer, gastritis, gastric ulcer, gastric polyp, gastric tumor (d) Duodenum: duodenal cancer, duodenal ulcer, duodenitis, duodenal tumor, duodenal lymphoma (e) Small intestine: small intestine cancer, small intestine neoplastic disease, small intestine inflammatory disease, small intestine vascular disease (f) Large intestine: large intestine cancer, large intestine neoplastic disease, large intestine inflammatory disease, large intestine polyp, large intestine polyposis, Crohn's disease, colitis, intestinal tuberculosis, hemorrhoids

(2) Hardware Configuration Fig. 2 shows the hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, and a sound output unit 16. These elements are connected via a data bus 19.

The processor 11 executes predetermined processing by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memories that store information necessary for the processing of the image processing device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the image processing device 1, or may include a removable storage medium such as a flash memory. The memory 12 stores programs that enable the image processing device 1 to execute each process in this embodiment.

The memory 12 also stores lesion detection model information D1, which is information related to the lesion detection model, and progression determination model information D2, which is information related to the progression determination model. Details of the lesion detection model and the progression determination model will be described later. The memory 12 may also include any other information necessary for the image processing device 1 to execute each process in this embodiment.

The interface 13 performs interface operations between the image processing device 1 and an external device. For example, the interface 13 supplies the display information "Ib" generated by the processor 11 to the display device 2. The interface 13 also supplies light generated by the light source unit 15 to the endoscope scope 3. The interface 13 also supplies an electrical signal indicating the endoscopic image Ia supplied from the endoscope scope 3 to the processor 11. The interface 13 may be a communication interface such as a network adapter for communicating with an external device by wire or wirelessly, or may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), etc.

The input unit 14 generates an input signal based on the operation by the examiner. The input unit 14 is, for example, a button, a touch panel, a remote controller, a voice input device, etc. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also incorporate a pump or the like for sending water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

Next, we will explain the details of the lesion detection model and progression assessment model.

The lesion detection model is a machine learning model that generates an inference result regarding a lesion area corresponding to a disease to be detected in an endoscopic examination, and the parameters required for the model are stored in the lesion detection model information D1. For example, when an endoscopic image is input, the lesion detection model outputs an inference result regarding a lesion area in the input endoscopic image. In other words, the lesion detection model is a model that has learned the relationship between an image input to the lesion detection model and a lesion area in the image. The lesion detection model may be a model (including a statistical model, the same applies below) that includes an architecture adopted in any machine learning such as a neural network or a support vector machine. Representative models of such neural networks include, for example, Fully Convolutional Network, SegNet, U-Net, V-Net, Feature Pyramid Network, Mask R-CNN, DeepLab, and the like. When the lesion detection model is configured using a neural network, the lesion detection model information D1 includes various parameters such as the layer structure, the neuron structure of each layer, the number of filters and filter size in each layer, and the weights of each element of each filter.

Here, we will explain the specific aspects (first output aspect and second output aspect) of the inference results output by the lesion detection model.

In the first output mode, the lesion detection model outputs, as an inference result, a map indicating the reliability (also called "lesion reliability") that each unit area of the input endoscopic image is a lesion area. Hereinafter, the above map will also be called a "lesion reliability map." For example, the lesion reliability map is an image indicating the lesion reliability for each unit pixel (which may include subpixels) or for each pixel block separated by a predetermined rule. Note that the lesion reliability indicates that the reliability of an area is higher for an area with a higher lesion reliability. The lesion reliability map may be a mask image indicating lesion areas using binary values.

In the second output mode, the lesion detection model outputs an inference result indicating a bounding box indicating the extent of the lesion area in the input endoscopic image, and the confidence that the area surrounded by the bounding box is a lesion area. The confidence here is, for example, a confidence level that is a score indicating the degree of confidence output from the output layer of a neural network when the lesion detection model is configured using a neural network. Note that the above-mentioned mode of inference result output by the lesion detection model is one example, and various modes of inference results may be output from the lesion detection model.

The lesion detection model is trained in advance based on a pair of an input image conforming to the input format of the lesion detection model and correct answer data (in the above example, the correct lesion confidence map or bounding box) indicating the correct inference result that the lesion detection model should output when the input image is input. Then, the parameters of each model obtained by training are stored in memory 12 as lesion detection model information D1.

The lesion detection model may include a feature extraction model that extracts features from an endoscopic image, or may be a model separate from the feature extraction model. In the latter case, the lesion detection model is a model trained to output the above-mentioned inference result when a feature (tensor with a predetermined number of dimensions) output by the feature extraction model to which an endoscopic image is input is input.

The progression assessment model is a machine learning model that infers (classifies) the progression of a lesion indicated by a lesion area included in an input endoscopic image, and parameters required for the model are stored in the progression assessment model information D2. When an endoscopic image is input, the progression assessment model outputs an inference result indicating the progression of the lesion in the input endoscopic image (more specifically, a classification result indicating the class of the progression). In other words, the progression assessment model is a model that has learned the relationship between the image input to the progression assessment model and the progression of the lesion in the image. The progression assessment model may be a model (including a statistical model, the same applies below) that includes an architecture adopted in any machine learning such as a neural network or a support vector machine. When the progression assessment model is configured by a neural network, the progression assessment model includes various parameters such as a layer structure, a neuron structure of each layer, the number of filters and filter size in each layer, and the weight of each element of each filter.

The progress assessment model is trained in advance based on a pair of an input image conforming to the input format of the progress assessment model and correct answer data (i.e., the progress class that is the correct answer) that indicates the correct inference result that the progress assessment model should output when the input image is input. Then, the parameters of each model obtained by training are stored in memory 12 as progress assessment model information D2.

The endoscopic image input to the progression determination model may be the entire image of the endoscopic image Ia generated by the endoscopic scope 3, or may be an image cut out from the endoscopic image so as to include at least the lesion area detected by the lesion detection model (i.e., a partial image of the endoscopic image Ia). Furthermore, instead of the endoscopic image, the progress determination model may be input with the feature quantities of the image calculated by the lesion detection model or feature extraction model described above.

(3) Overview FIG. 3 is a diagram showing an overview of a progress determination process related to the determination of the progress degree executed by the image processing device 1 in the first embodiment.

First, the image processing device 1 detects a lesion area in each endoscopic image Ia obtained from the endoscope scope 3 at a predetermined frame period using a lesion detection model. Then, the image processing device 1 obtains an inference result regarding the lesion area in the endoscopic image Ia from the lesion detection model. In the example of FIG. 3, the image processing device 1 obtains, as the lesion detection result, either a lesion confidence map indicating the inference result of the lesion detection model based on the first output mode, or a bounding box indicating the inference result of the lesion detection model based on the second output mode. Here, in the lesion confidence map, pixels with higher lesion confidence are represented by a color closer to white. Also, for ease of explanation, the above-mentioned bounding box is displayed superimposed on the endoscopic image Ia input to the lesion detection model.

Here, the image processing device 1 does not perform a progression determination based on the progression determination model for an endoscopic image Ia in which a lesion area could not be detected. In the first output mode, the image processing device 1 regards a unit area (e.g., pixels) having a lesion reliability equal to or greater than a predetermined threshold (also referred to as the "first threshold") as a partial area constituting the lesion area, and determines that the lesion area could not be detected if no such unit area exists or if no connected area of the above-mentioned unit areas having a predetermined number of pixels or more exists. The above-mentioned first threshold is, for example, a default value stored in advance in the memory 12 or the like. In the second output mode, the image processing device 1 regards the area within a bounding box as the lesion area, and determines that the lesion area could not be detected if a bounding box cannot be obtained.

Next, the image processing device 1 judges whether each of the endoscopic images Ia in which the lesion area has been detected is an endoscopic image suitable for judging the degree of progression (also called a "progression suitability judgment"). In this case, the image processing device 1 judges the suitability of the degree of progression for the target endoscopic image Ia based on at least one of the size of the lesion area detected from the target endoscopic image Ia or the reliability of the detected lesion area.

If the image processing device 1 determines that the endoscopic image Ia in which the lesion area is detected is an endoscopic image suitable for judging the degree of progression, the image processing device 1 uses the endoscopic image Ia to judge the degree of progression of the lesion using a progression judgment model. This allows the image processing device 1 to automatically select an endoscopic image Ia suitable for the degree of progression of the lesion and judge the degree of progression of the lesion with high accuracy. On the other hand, if the image processing device 1 determines that the endoscopic image Ia in which the lesion area is detected is not an endoscopic image suitable for judging the degree of progression, it outputs a suggestion to capture an endoscopic image Ia suitable for judging the degree of progression using the endoscopic scope 3 via the display device 2 or the sound output unit 16. This allows the image processing device 1 to assist the examiner in operating the endoscopic scope 3 to capture an endoscopic image Ia suitable for judging the degree of progression, and promote the acquisition of an endoscopic image Ia suitable for judging the degree of progression.

(4) Functional Blocks Fig. 4 is an example of functional blocks of the progress assessment process in the first embodiment. The processor 11 of the image processing device 1 functionally has an endoscopic image acquisition unit 30, a lesion detection unit 31, a suitability assessment unit 32, a progress assessment unit 33, and an output control unit 34. Note that in Fig. 4, blocks between which data is exchanged are connected by solid lines, but the combination of blocks between which data is exchanged is not limited to this. The same applies to other functional block diagrams described later.

The endoscopic image acquisition unit 30 acquires the endoscopic image Ia captured by the endoscopic scope 3 via the interface 13 at predetermined intervals. The endoscopic image acquisition unit 30 then supplies the acquired endoscopic image Ia to the lesion detection unit 31 and the output control unit 34, respectively. Then, the respective processing units in the subsequent stages perform the processing described below, with the time interval at which the endoscopic image acquisition unit 30 acquires the endoscopic image Ia being set as a period.

The lesion detection unit 31 detects a lesion area in the endoscopic image Ia supplied from the endoscopic image acquisition unit 30 based on the lesion detection model information D1. In this case, the lesion detection unit 31 inputs the endoscopic image Ia to a lesion detection model constructed by referring to the lesion detection model information D1, and obtains an inference result of lesion detection output by the lesion detection model. The inference result of lesion detection may be a lesion reliability map based on the first output mode, or a set of a bounding box and lesion reliability based on the second output mode. When the lesion detection unit 31 determines that a lesion area has been detected, it supplies the lesion detection result corresponding to the inference result of the above-mentioned lesion detection together with the endoscopic image Ia to the suitability determination unit 32. In addition, the lesion detection unit 31 supplies the lesion detection result to the output control unit 34 regardless of whether or not a lesion area has been detected. Note that the lesion detection result when a lesion area has not been detected is, for example, information indicating that a lesion area has not been detected.

The suitability determination unit 32 performs a progression suitability determination, which is a suitability determination of whether the endoscopic image Ia supplied from the lesion detection unit 31 is suitable for determining the progression of the lesion. In this case, the suitability determination unit 32 acquires at least one of the size of the lesion area detected from the endoscopic image Ia or the reliability of the detected lesion area based on the inference result of the lesion detection, and performs a progression suitability determination based on at least one of the acquired size or reliability. Then, if the suitability determination unit 32 determines that the endoscopic image Ia is suitable for determining the progression of the lesion, it supplies the endoscopic image Ia to the progression determination unit 33. Also, if the suitability determination unit 32 determines that the endoscopic image Ia is not suitable for determining the progression of the lesion, it supplies a determination result (also called an "insuitability determination result") indicating that the endoscopic image Ia is not suitable for determining the progression of the lesion to the output control unit 34. Furthermore, even if the suitability determination unit 32 determines that the endoscopic image Ia is suitable for determining the degree of progression of the lesion, the suitability determination unit 32 may supply the output control unit 34 with a determination result indicating that the endoscopic image Ia is suitable for determining the degree of progression of the lesion (also referred to as a "suitable determination result").

Based on the progression determination model information D2, the progress determination unit 33 determines the progress of the lesion area included in the endoscopic image Ia that is determined by the suitability determination unit 32 to be suitable for use in progression determination (i.e., classifies the progress class to which the lesion area belongs). In this case, for example, the progress determination unit 33 inputs the endoscopic image Ia to a progress determination model configured by referring to the progress determination model information D2, and obtains the progress inference result output by the progress determination model. In this case, for example, the progress determination model outputs the most likely progress class (e.g., a grade representing the progress) and the confidence level (degree of confidence) for each progress class that is a candidate for classification as the progress inference result. Note that the progress determination unit 33 may input a partial image of the endoscopic image Ia including the lesion area or the feature amount of the endoscopic image Ia calculated by the lesion detection unit 31 to the progress determination model instead of the endoscopic image Ia. Then, the progress determination unit 33 supplies the progress determination result based on the above-mentioned progress inference result to the output control unit 34. The progress level determination result is information obtained by determining the progress level, and is, for example, information indicating the progress level class determined to be most likely and the degree of certainty of that class.

In addition, the progress determination unit 33 may determine the final progress to be output to the output control unit 34 based on a predetermined number of progress inference results based on a predetermined number of endoscopic images Ia, instead of determining the final progress to be output to the output control unit 34 based on a predetermined number of progress inference results based on two or more predetermined number of endoscopic images Ia based on the progress determination model. In this case, when a predetermined number of progress inference results output by the progress determination model are accumulated, the progress determination unit 33 determines the progress to be output to the output control unit 34 based on the accumulated predetermined number of progress inference results. In this case, for example, the progress determination unit 33 counts the predetermined number of progress inference results, and determines the progress class that is most frequently determined to be most likely (i.e., the class determined by majority vote) as the final progress. In this case, the progress determination result output by the progress determination unit 33 includes, for example, the class determined by majority vote and the average value of the certainty of the class (or a representative value other than the average value).

The output control unit 34 generates display information Ib based on the latest endoscopic image Ia supplied from the endoscopic image acquisition unit 30, the lesion detection result output by the lesion detection unit 31, and the progress determination result output by the progress determination unit 33. The output control unit 34 then supplies the generated display information Ib to the display device 2, thereby causing the display device 2 to display the latest endoscopic image Ia, the lesion detection result, and the progress determination result, etc. In addition, the output control unit 34 may control the sound output of the sound output unit 16 so as to output a warning sound or voice guidance, etc., to notify the user that a lesion area has been detected, based on the lesion detection result.

The output control unit 34 also outputs, via the display device 2 or the sound output unit 16, a suggestion for capturing an endoscopic image Ia suitable for judging the degree of progression using the endoscope scope 3, based on the inappropriate judgment result supplied from the appropriateness judgment unit 32. For example, the output control unit 34 outputs the above-mentioned suggestion when an inappropriate judgment result is generated a predetermined number of times or for a predetermined period of time in succession without generating a suitable judgment result.

In this case, in the first example of the suggestion, the output control unit 34 outputs information encouraging the user to move the shooting position closer to the lesion area in order to obtain an endoscopic image Ia with a larger magnified lesion area. For example, the output control unit 34 displays or outputs audio stating, "Please move the camera closer to the lesion."

In a second example of the suggestion, the output control unit 34 outputs information indicating a target range of the lesion area on the endoscopic image Ia. For example, the output control unit 34 displays a frame indicating the range in which the lesion area is preferably displayed, superimposed on the latest endoscopic image Ia. The range on the endoscopic image Ia in which the frame should be displayed may be a preset range stored in advance in the memory 12, etc., or may be a range in which at least one of the shape and size has been adjusted based on the lesion detection result. A specific aspect of the second example will be described in detail in the display example of Figure 6 described later.

In addition, if the output control unit 34 determines that an unsuitable judgment result has been generated consecutively without generating a suitable judgment result for a predetermined number of times or for a predetermined period of time even after the output of the suggestion, it displays or outputs audio information indicating that a progress judgment cannot be performed.

The output control unit 34 may output a suggestion based on the progress determination result output by the progress determination unit 33. For example, when the confidence level of the most likely progress class is less than a predetermined threshold, the output control unit 34 may output a suggestion based on the first or second example described above, regardless of the presence or absence of an inappropriate determination result.

Note that each of the components of the endoscopic image acquisition unit 30, the lesion detection unit 31, the suitability determination unit 32, the progress determination unit 33, and the output control unit 34 can be realized, for example, by the processor 11 executing a program. Also, each component may be realized by recording the necessary programs in any non-volatile storage medium and installing them as needed. Note that at least a portion of each of these components may not be realized by software through a program, but may be realized by any combination of hardware, firmware, and software. Also, at least a portion of each of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, a program consisting of each of the above components may be realized using this integrated circuit. Furthermore, at least a portion of each component may be configured by an ASSP (Application Specific Standard Production), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The above also applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

(5) Judgment of Progress Adequacy Here, the judgment of progress adequacy will be specifically described.

For example, when judging the suitability of the degree of progression based on the size of the lesion area, the suitability judgment unit 32 generates a suitable judgment result when the size of the lesion area is equal to or larger than a predetermined size, and generates an inappropriate judgment result when the size of the lesion area is less than the predetermined size. The above-mentioned predetermined size is, for example, a default value determined as the size necessary for accurate judgment of the degree of progression, and is stored in advance in the memory 12, etc. Furthermore, the size of the lesion area in the first output mode is, for example, the size (e.g., the number of pixels) of the connected area of unit areas having a lesion reliability equal to or greater than a first threshold value. Furthermore, the size of the lesion area in the second output mode is, for example, the size of a bounding box.

In another example, when the above-mentioned suitability judgment is made based on the reliability of the lesion area, the suitability judgment unit 32 generates a suitable judgment result when the reliability of the lesion area is equal to or greater than a predetermined threshold (also referred to as the "second threshold"), and generates an inappropriate judgment result when the reliability of the lesion area is less than the second threshold. Here, the reliability of the lesion area in the case of the first output mode is, for example, the average value, median, or other representative value of the lesion reliability of the unit areas that make up the lesion area. Also, the reliability of the lesion area in the case of the second output mode is, for example, the reliability associated with the bounding box. Also, the above-mentioned second threshold is, for example, a default value determined as the reliability required for accurate judgment of the degree of progression, and is stored in advance in the memory 12, etc. Note that the second threshold in the case of the first output mode is preferably set to a value equal to or greater than the first threshold that is compared with the lesion reliability of each unit area when determining whether or not it is a lesion area.

In yet another example, when the progress suitability is judged based on both the size and reliability of the lesion area, the suitability judgment unit 32 generates a suitable judgment result when the size of the lesion area is equal to or larger than a predetermined size and the reliability of the lesion area is equal to or larger than a second threshold. On the other hand, the suitability judgment unit 32 generates an inappropriate judgment result when the size of the lesion area is less than the predetermined size or the reliability of the lesion area is less than the second threshold.

Furthermore, preferably, the suitability determination unit 32 may change the criteria used to determine the suitability of the progress level based on the progress level determination result by the progress level determination unit 33 (i.e., information obtained by judging the progress level). The above-mentioned criteria correspond to at least one of a predetermined size to be compared with the size of the lesion area and a second threshold value to be compared with the reliability of the lesion area.

For example, the suitability determination unit 32 changes the criteria used to determine whether the progression is appropriate based on the confidence level for the progression class determined to be the most likely by the progression determination unit 33. In this case, for example, if the confidence level for the most likely progression class is less than a predetermined threshold, the suitability determination unit 32 changes the criteria used to determine whether the progression is appropriate to be stricter. In this case, the suitability determination unit 32 increases the predetermined size and/or the second threshold used as the criteria by a predetermined value or a predetermined percentage. As a result, the suitability determination unit 32 tightens the criteria for determining that the endoscopic image Ia is appropriate for determining the progression of the lesion, and promotes the determination of the progression using more carefully selected endoscopic images Ia. Note that the above-mentioned predetermined threshold, predetermined value, and predetermined percentage are, for example, default values and are stored in advance in the memory 12, etc. The predetermined size is an example of a "first criterion," and the second threshold is an example of a "second criterion."

In another example, the suitability determination unit 32 changes the criteria used to determine whether the progress level is appropriate, based on the degree of change over time in the progress class determined to be most likely by the progress level determination unit 33. In this case, for example, if the progress level determination results obtained from the time series for a predetermined number of times do not match the progress level class determined to be most likely, the suitability determination unit 32 determines that the progress level determination results are fluctuating (i.e., unstable) and changes the criteria used to determine whether the progress level is appropriate to be stricter. In another example, the suitability determination unit 32 tallyes up the most likely progress level class based on the progress level determination results obtained from the time series for a predetermined number of times, and if the proportion of the most frequent class is less than a predetermined proportion, changes the criteria used to determine whether the progress level is appropriate to be stricter.

5 shows a first display example displayed by the display device 2 during an endoscopic examination. The first display example shows an example of a display screen displayed by the display device 2 when the progression determination unit 33 generates a determination result of the progression (here, the invasion depth).

The output control unit 34 of the image processing device 1 outputs to the display device 2 display information Ib generated based on the latest endoscopic image Ia supplied from the endoscopic image acquisition unit 30, the lesion detection result output by the lesion detection unit 31, and the progress determination result output by the progress determination unit 33. The output control unit 34 transmits the display information Ib to the display device 2, thereby causing the display device 2 to display the above-mentioned display screen.

In the first display example shown in FIG. 5, the output control unit 34 of the image processing device 1 provides a real-time image display area 70, a lesion detection result display area 71, and an invasion depth determination result display area 72 on the display screen.

Here, the output control unit 34 displays a moving image representing the latest endoscopic image Ia in the real-time image display area 70. Furthermore, in the lesion detection result display area 71, the output control unit 34 displays the lesion detection result by the lesion detection unit 31. Note that at the time of displaying the display screen shown in FIG. 5, the lesion detection unit 31 has generated a lesion reliability map as the lesion detection result, so the output control unit 34 displays an image based on the lesion reliability map (here, a mask image indicating the lesion area) in the lesion detection result display area 71. Note that, when the lesion detection result indicates a bounding box, for example, the output control unit 34 displays an image in which the above-mentioned bounding box is superimposed on the latest endoscopic image Ia in the real-time image display area 70 or the lesion detection result display area 71.

The output control unit 34 may further display a text message in the lesion detection result display area 71 or the like to the effect that there is a high possibility that a lesion is present, and may output a sound (including voice) from the sound output unit 16 to notify the effect that there is a high possibility that a lesion is present.

Furthermore, the output control unit 34 displays the result of the progression (here, depth of invasion) judgment made by the progress determination unit 33 in the depth of invasion judgment result display area 72. Here, the progress determination unit 33 judges that the most likely depth of invasion class is "T3", and the output control unit 34 displays "T3" in the depth of invasion judgment result display area 72. The output control unit 34 may output the depth of invasion class judged by the progress determination unit 33 via the sound output unit 16.

In the first display example, the progression determination unit 33 determines the depth of invasion based on the endoscopic image Ia that is suitable for the progression of the lesion selected by the suitability determination unit 32. Therefore, the output control unit 34 can present the highly accurate depth of invasion determination result to the examiner.

FIG. 6 shows a second display example displayed by the display device 2 during an endoscopic examination. The second display example shows an example of a display screen displayed by the display device 2 when the suitability judgment unit 32 continuously generates unsuitability judgment results and the output control unit 34 suggests image capture. In the second display example shown in FIG. 6, similar to the first display example, the output control unit 34 of the image processing device 1 provides a real-time image display area 70, a lesion detection result display area 71, and an invasion depth judgment result display area 72 on the display screen. Here, the output control unit 34 displays a mask image based on the lesion detection result in the lesion detection result display area 71, similar to the first display example.

In the second display example, the output control unit 34 superimposes a frame 73 representing a target range (i.e., the target position and size) of a preferred lesion area in the endoscopic image Ia in the real-time image display area 70 as a suggestion for capturing an endoscopic image Ia suitable for judging the degree of progression using the endoscopic scope 3. In addition, since the output control unit 34 has not been able to judge the depth of invasion in the depth of invasion judgment result display area 72, instead of displaying the result of the depth of invasion, it displays a message urging the examiner to adjust the lesion area so that it falls within the frame 73. This allows the examiner operating the endoscopic scope 3 to operate the endoscopic scope 3 so that the outer edge of the lesion area shown in the lesion detection result display area 71 overlaps with the frame 73. In this case, the information shown in the lesion detection result display area 71 may be superimposed on the endoscopic image Ia in the real-time image display area 70.

Here, the output control unit 34 may determine at least one of the size and shape of the frame 73 based on the lesion detection result. For example, the output control unit 34 may increase the size of the frame 73 as the size of the lesion area detected by the lesion detection unit 31 increases. In another example, the output control unit 34 may recognize the shape of the lesion area detected by the lesion detection unit 31, and determine the shape of the frame 73 (e.g., the ratio of the length to the width) according to the recognized shape. In this case, for example, the output control unit 34 may approximate the lesion area detected by the lesion detection unit 31 with a figure such as an ellipse or a rectangle, and display the frame 73 according to the approximated figure.

As described above, according to the second display example, when an endoscopic image Ia suitable for determining the degree of progression (here, the depth of invasion) cannot be obtained, the image processing device 1 can suggest capturing an image and promote the acquisition of an endoscopic image Ia suitable for determining the degree of progression.

(7) Processing Flow FIG. 7 is an example of a flowchart outlining the processing executed by the image processing device 1 during endoscopic examination in the first embodiment.

First, the image processing device 1 acquires an endoscopic image Ia (step S11). In this case, the endoscopic image acquisition unit 30 of the image processing device 1 receives the endoscopic image Ia from the endoscopic scope 3 via the interface 13.

Next, the image processing device 1 performs lesion detection on the endoscopic image Ia acquired in step S11 (step S12). In this case, the image processing device 1 inputs the endoscopic image Ia to a lesion detection model configured with reference to the lesion detection model information D1, thereby acquiring a lesion detection result output from the lesion detection model.

Then, the image processing device 1 determines whether or not a lesion area has been detected from the endoscopic image Ia (step S13). If the image processing device 1 determines that a lesion area has been detected from the endoscopic image Ia (step S13; Yes), it performs a progress level determination for the endoscopic image Ia (step S14). In this case, the image processing device 1 performs the progress level determination based on at least one of the size and reliability of the detected lesion area. On the other hand, if the image processing device 1 determines that a lesion area has not been detected from the endoscopic image Ia (step S13; No), it proceeds to step S17 without performing a progress level determination or a determination of the progression level of the lesion.

If the endoscopic image Ia is determined to be an image suitable for progression assessment (step S15; Yes), the image processing device 1 assesses the progression of the lesion (step S16). In this case, for example, the image processing device 1 obtains an inference result of the progression output by the progression assessment model when the endoscopic image Ia is input to the progression assessment model constructed with reference to the progression assessment model information D2.

Then, the image processing device 1 displays on the display device 2 information based on the endoscopic image Ia obtained in step S11, the lesion detection result generated in step S12, and the progress assessment result generated in step S16 (step S17). If the image processing device 1 determines in step S13 that a lesion area has not been detected, it displays, for example, the endoscopic image Ia and information indicating that a lesion area has not been detected in step S17.

On the other hand, if it is determined that the endoscopic image Ia is not suitable for progression assessment (step S15; No), the image processing device 1 outputs a suggestion to capture an image suitable for progression assessment (step S19). Note that the image processing device 1 may execute step S19 only if it is determined in step S15 that the image is not suitable for progression assessment a predetermined number of times or for a predetermined period of time. Note that in this case, if the number of times or the period of time that the image is continuously determined in step S15 to be not suitable for progression assessment does not reach the above-mentioned predetermined number of times or predetermined period of time, the image processing device 1 performs a process of displaying the endoscopic image Ia and the lesion detection result without outputting a suggestion, for example.

Then, after step S17 or step S19, the image processing device 1 determines whether or not the endoscopic examination has ended (step S18). For example, the image processing device 1 determines that the endoscopic examination has ended when it detects a predetermined input to the input unit 14 or the operation unit 36. Then, if the image processing device 1 determines that the endoscopic examination has ended (step S18; Yes), it ends the processing of the flowchart. On the other hand, if the image processing device 1 determines that the endoscopic examination has not ended (step S18; No), it returns the processing to step S11.

(8) Modifications Next, preferred modifications of the first embodiment will be described. The following modifications may be applied in combination to the first embodiment.

(Variation 1)
The image processing device 1 may process the video composed of the endoscopic images Ia generated during the endoscopic examination after the examination.

For example, when an image to be processed is specified based on user input via the input unit 14 at any timing after the examination, the image processing device 1 sequentially performs the processing of the flowchart in FIG. 7 on the time-series endoscopic images Ia that constitute the specified image. Then, when the image processing device 1 determines in step S18 that the target image has ended, it ends the processing of the flowchart, and when the target image has not ended, it returns to step S11 and performs the processing of the flowchart on the next endoscopic image Ia in the time series.

(Variation 2)
The suitability determining unit 32 may perform the progression suitability determination based on a reliability calculated separately from the reliability output by the lesion detection model.

In this case, for example, when the lesion detection unit 31 detects a lesion area, the suitability determination unit 32 calculates the reliability based on the endoscopic image Ia including the detected lesion area. In this case, for example, when an endoscopic image is input, the suitability determination unit 32 calculates the reliability from the endoscopic image Ia using a model that outputs a reliability score that a lesion area exists in the input endoscopic image. In this case, the above-mentioned model is, for example, a model based on machine learning such as a neural network, and learned parameters are stored in advance in the memory 12 or the like. Note that the above-mentioned model may be a classification model that performs a binary classification of whether or not a lesion area exists. In this case, the suitability determination unit 32 acquires the confidence level corresponding to the class in which a lesion area exists, which is output from the above-mentioned classification model, as the above-mentioned reliability.

Even with this modified example, the suitability determination unit 32 can appropriately perform progress suitability determination.

(Variation 3)
The lesion detection model information D1 and the progression determination model information D2 may be stored in a storage device separate from the image processing device 1.

FIG. 8 is a schematic diagram of an endoscopic examination system 100A in a modified example. For simplicity, the display device 2 and the endoscope 3 are not shown. The endoscopic examination system 100A includes a server device 4 that stores lesion detection model information D1 and progression assessment model information D2. The endoscopic examination system 100A also includes multiple image processing devices 1 (1A, 1B, ...) that can communicate data with the server device 4 via a network.

In this case, each image processing device 1 refers to the lesion detection model information D1 and the progression determination model information D2 via the network. In this case, the interface 13 of each image processing device 1 includes a communication interface such as a network adapter for communication. In this configuration, each image processing device 1 can refer to the lesion detection model information D1 and the progression determination model information D2, as in the above-mentioned embodiment, and suitably execute processing related to determining the progression of the lesion. Note that the server device 4 may instead execute at least a part of the processing executed by each functional block of the processor 11 of the image processing device 1 shown in FIG. 4.

Second Embodiment
9 is a block diagram of an image processing device 1X according to the second embodiment. The image processing device 1X includes an acquisition unit 30X, a detection unit 31X, a first determination unit 32X, and a second determination unit 33X. The image processing device 1X may be composed of a plurality of devices.

The acquisition means 30X acquires an endoscopic image of the subject. The acquisition means 30X can be, for example, the endoscopic image acquisition section 30 in the first embodiment (including modified examples, the same applies below). The acquisition means 30X may instantly acquire an endoscopic image generated by the imaging section, or may acquire an endoscopic image generated in advance by the imaging section and stored in a storage device at a predetermined timing.

The detection means 31X detects a lesion area, which is a candidate area for a lesion in the subject in the endoscopic image, based on the endoscopic image. The detection means 31X can be, for example, the lesion detection unit 31 in the first embodiment.

The first determination means 32X determines whether or not the endoscopic image is suitable for determining the progression or depth of a lesion based on at least one of the size of the lesion area or the reliability of the lesion area as a lesion. The first determination means 32X can be, for example, the suitability determination unit 32 in the first embodiment.

The second determination means 33X determines the degree of progression or the depth of invasion based on the endoscopic image determined to be an image suitable for determining the degree of progression or the depth of invasion. The second determination means 33X can be, for example, the degree of progression determination unit 33 in the first embodiment.

FIG. 10 is an example of a flowchart showing the processing procedure in the second embodiment. The acquisition means 30X acquires an endoscopic image of the subject (step S21). Next, the detection means 31X detects a lesion area in the endoscopic image that is a candidate area for a lesion in the subject based on the endoscopic image (step S22). The first determination means 32X determines whether the endoscopic image is suitable for determining the progression or depth of the lesion based on at least one of the size of the lesion area and the reliability of the lesion area as a lesion (step S23). The second determination means 33X determines the progression or depth based on the endoscopic image determined to be suitable for determining the progression or depth (step S24).

According to the second embodiment, the image processing device 1X can accurately determine the progression or depth of invasion based on the selected endoscopic image.

In each of the above-described embodiments, the program can be stored using various types of non-transitory computer readable media and supplied to a computer, such as a processor. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., floppy disks, magnetic tapes, hard disk drives), optical storage media (e.g., optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R/Ws, semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory). Programs may also be supplied to computers by various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transient computer-readable media can supply programs to computers via wired communication paths such as electric wires and optical fibers, or wireless communication paths.

In addition, all or part of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following notes.

[Appendix 1]
An acquisition means for acquiring an endoscopic image of a subject;
a detection means for detecting a lesion area, which is a candidate area for a lesion of the subject in the endoscopic image, based on the endoscopic image;
a first determination means for determining whether or not the endoscopic image is suitable for determining the progression or depth of the lesion based on at least one of the size of the lesion area and the reliability of the lesion area with respect to the lesion-likeness;
a second determination means for determining the degree of progression or the depth of invasion based on the endoscopic image determined to be an image suitable for determining the degree of progression or the depth of invasion;
An image processing device comprising:
[Appendix 2]
The image processing device described in Appendix 1, wherein the first judgment means changes the criteria used to judge whether an image is suitable for judging the degree of progression or depth of invasion based on information obtained by judging the degree of progression or depth of invasion.
[Appendix 3]
The first determination means changes the criterion based on a degree of confidence for the class of the progression or depth of invasion determined by the second determination means.
[Appendix 4]
The first determination means changes the criterion based on a degree of change in the class of the progression or depth of invasion over time determined by the second determination means.
[Appendix 5]
The criterion is at least one of a first criterion related to the size of the lesion area and a second criterion related to the reliability.
[Appendix 6]
An image processing device as described in Appendix 1, having an output control means for outputting suggestions regarding the capture of the endoscopic image via a display device or audio output device when it is determined that the image is not suitable for determining the progression or depth of invasion.
[Appendix 7]
The image processing device according to claim 6, wherein the output control means outputs information indicating a target range of the lesion area on the endoscopic image.
[Appendix 8]
The image processing device according to claim 7, wherein the output control means determines at least one of a shape and a size of the area based on a detection result of the lesion area by the detection means.
[Appendix 9]
The image processing device according to claim 6, wherein the output control means outputs, as the suggestion, information encouraging the user to move the imaging position closer to the lesion area.
[Appendix 10]
The computer
An endoscopic image of the subject is acquired,
detecting a lesion area in the endoscopic image, the lesion area being a candidate area for a lesion in the subject based on the endoscopic image;
determining whether the endoscopic image is suitable for determining the progression or depth of the lesion based on at least one of the size of the lesion area or the reliability of the lesion area;
determining the degree of progression or the depth of invasion based on the endoscopic image determined to be an image suitable for determining the degree of progression or the depth of invasion;
Image processing methods.
[Appendix 11]
An endoscopic image of the subject is acquired,
detecting a lesion area in the endoscopic image, the lesion area being a candidate area for a lesion in the subject based on the endoscopic image;
determining whether the endoscopic image is suitable for determining the progression or depth of the lesion based on at least one of the size of the lesion area or the reliability of the lesion area;
A storage medium storing a program for causing a computer to execute a process for determining the degree of progression or depth of invasion based on the endoscopic image that has been determined to be an image suitable for determining the degree of progression or depth of invasion.

The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above-mentioned embodiments. Various modifications that a person skilled in the art can understand can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and amendments that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above cited patent documents and non-patent documents are incorporated by reference into this document.

Reference Signs List 1, 1A, 1B, 1X Image processing device 2 Display device 3 Endoscope 11 Processor 12 Memory 13 Interface 14 Input section 15 Light source section 16 Sound output section 100, 100A Endoscopy system

Claims

An acquisition means for acquiring an endoscopic image of a subject;
a detection means for detecting a lesion area, which is a candidate area for a lesion of the subject in the endoscopic image, based on the endoscopic image;
a first determination means for determining whether or not the endoscopic image is suitable for determining the progression or depth of the lesion based on at least one of the size of the lesion area and the reliability of the lesion area with respect to the lesion-likeness;
a second determination means for determining the degree of progression or the depth of invasion based on the endoscopic image determined to be an image suitable for determining the degree of progression or the depth of invasion;
An image processing device comprising:
The image processing device according to claim 1, wherein the first determination means changes the criteria used to determine whether an image is suitable for determining the degree of progression or depth of invasion based on information obtained by the determination of the degree of progression or depth of invasion.
The image processing device according to claim 2, wherein the first determination means changes the criteria based on the degree of confidence in the class of the progression or depth determined by the second determination means.
The image processing device according to claim 3, wherein the first determination means changes the criteria based on the degree of change over time of the progression or depth of invasion class determined by the second determination means.
The image processing device according to claim 3, wherein the criterion is at least one of a first criterion related to the size of the lesion area and a second criterion related to the reliability.
The image processing device according to claim 1, further comprising an output control means for outputting, via a display device or audio output device, a suggestion regarding the capture of the endoscopic image when the image is determined to be unsuitable for determining the progression or depth of invasion.
The image processing device according to claim 6, wherein the output control means outputs information indicating a target range of the lesion area on the endoscopic image.
The image processing device according to claim 7, wherein the output control means determines at least one of the shape and size of the range based on the detection result of the lesion area by the detection means.
The image processing device according to claim 6, wherein the output control means outputs, as the suggestion, information encouraging the imaging position to be brought closer to the lesion area.
The computer
An endoscopic image of the subject is acquired,
detecting a lesion area in the endoscopic image, the lesion area being a candidate area for a lesion in the subject based on the endoscopic image;
determining whether the endoscopic image is suitable for determining the progression or depth of the lesion based on at least one of the size of the lesion area or the reliability of the lesion area;
determining the degree of progression or the depth of invasion based on the endoscopic image determined to be an image suitable for determining the degree of progression or the depth of invasion;
Image processing methods.
An endoscopic image of the subject is acquired,
detecting a lesion area in the endoscopic image, the lesion area being a candidate area for a lesion in the subject based on the endoscopic image;
determining whether the endoscopic image is suitable for determining the progression or depth of the lesion based on at least one of the size of the lesion area or the reliability of the lesion area;
A storage medium storing a program for causing a computer to execute a process for determining the degree of progression or depth of invasion based on the endoscopic image that has been determined to be an image suitable for determining the degree of progression or depth of invasion.