[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

WO2020088328A1 - 一种结肠息肉图像的处理方法和装置及系统 - Google Patents

一种结肠息肉图像的处理方法和装置及系统 Download PDF

Info

Publication number
WO2020088328A1
WO2020088328A1 PCT/CN2019/112788 CN2019112788W WO2020088328A1 WO 2020088328 A1 WO2020088328 A1 WO 2020088328A1 CN 2019112788 W CN2019112788 W CN 2019112788W WO 2020088328 A1 WO2020088328 A1 WO 2020088328A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
polyp
target object
endoscopic
picture
Prior art date
Application number
PCT/CN2019/112788
Other languages
English (en)
French (fr)
Inventor
付星辉
孙钟前
尚鸿
章子健
杨巍
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Priority to EP19878480.3A priority Critical patent/EP3876188B1/en
Priority to JP2021512995A priority patent/JP7090836B2/ja
Publication of WO2020088328A1 publication Critical patent/WO2020088328A1/zh
Priority to US17/025,679 priority patent/US11468563B2/en
Priority to US17/885,361 priority patent/US11748883B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/50—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for simulation or modelling of medical disorders
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30028—Colon; Small intestine
    • G06T2207/30032—Colon polyp

Definitions

  • Embodiments of the present application relate to the field of computer technology, and in particular, to a method, device, and system for processing colon polyp images.
  • colon cancer ranks within the top five. The incidence of colon cancer is also high in North America and Europe. Colon cancer is a malignant tumor of the digestive tract that often occurs in the colon. Generally speaking, 50% of patients with advanced colon cancer will die from relapse and metastasis, and nearly 100% of patients with early colon cancer can be completely cured. Therefore, the prevention and treatment of colon cancer is necessary. However, early clinical colon cancer cannot be predicted by clinical symptoms.
  • sliding window refers to an endoscope video image frame, first from top to bottom, then from left to right Slide the image block), or manually mark the position of the polyp.
  • computer vision Computer Vision
  • a sliding window method is used to calculate whether each image block contains polyps on an endoscopic video image frame. Due to the large number of image blocks, the calculation amount is large and the real-time performance It does not meet the requirements.
  • the endoscope is controlled to move, the recognition results cannot be output in real time for the images collected in real time.
  • the real-time performance does not meet the requirements.
  • the endoscope is controlled to move, the recognition results cannot be output in real time for the images collected in real time.
  • the embodiments of the present application provide a method, a device and a system for processing a colon polyp image, which are used to find the position of a polyp in real time and determine the nature of the polyp, and improve the processing efficiency of the polyp image.
  • an embodiment of the present application provides a method for processing a colon polyp image, including:
  • a device for processing a colon polyp image uses a polyp positioning model to detect the polyp position of the endoscopic image to be processed, and locates a polyp image block from the endoscopic image.
  • the polyp image block includes: polyps in the endoscope The location area in the image;
  • the processing device of the colon polyp image uses a polyp property recognition model to perform classification detection on the polyp image block, and outputs a recognition result.
  • an embodiment of the present application further provides a device for processing a colon polyp image, including:
  • the position detection module is configured to perform polyp position detection on the endoscopic image to be processed using a polyp positioning model, and locate a polyp image block from the endoscopic image, the polyp image block includes: the polyp is in the endoscopic The position area in the mirror image;
  • the polyp classification module is configured to perform polyp type classification detection on the polyp image block using a polyp property recognition model, and output a recognition result.
  • the component modules of the colon polyp image processing device may also perform the steps described in the aforementioned first aspect and various possible implementations.
  • the component modules of the colon polyp image processing device may also perform the steps described in the aforementioned first aspect and various possible implementations.
  • an embodiment of the present application further provides a medical system, the medical system includes: an endoscope device and a colon polyp image processing device, between the endoscope device and the colon polyp image processing device Established a communication connection;
  • the endoscope device is configured to generate an endoscope video stream; and send the endoscope video stream to the colon polyp image processing device;
  • the colon polyp image processing device is configured to receive the endoscopic video stream from the endoscopic device; obtain the endoscopic image to be processed from the endoscopic video stream; use polyp positioning
  • the model performs polyp position detection on the endoscopic image to be processed, and locates a polyp image block from the endoscopic image.
  • the polyp image block includes: the position area of the polyp in the endoscopic image; using the nature of the polyp
  • the recognition model performs classification detection on the polyp image block to output the recognition result.
  • embodiments of the present application provide an image processing method, including:
  • the image processing device uses the target object positioning model to detect the target object position of the image to be processed, and locates the target object image block from the image.
  • the target object image block includes: the position of the target object in the image region;
  • the image processing device uses a target object property recognition model to perform classification detection of the target object type on the target object image block, and outputs a recognition result.
  • an embodiment of the present application provides a colon polyp image processing device, the colon polyp image processing device includes: a processor and a memory; the memory is configured to store instructions; the processor is configured to execute instructions in the memory, Causing the processing device of the colon polyp image to perform the method of any one of the preceding aspects.
  • the embodiments of the present application provide a computer-readable storage medium, in which instructions are stored in the computer-readable storage medium, and when it runs on a computer, the computer is caused to execute the method described in the above aspects.
  • the polyp positioning model is first used to detect the polyp position of the endoscopic image, and the polyp image block is located from the endoscopic image.
  • the polyp image block includes: the position area of the polyp in the endoscopic image.
  • a polyp property recognition model is used to classify and detect the polyp type on the polyp image block, and output the recognition result.
  • the polyp positioning model since the polyp positioning model is used to detect the position of the polyp, the polyp image block can be directly located from the endoscopic image.
  • the classification detection for the type of polyp is also performed on the polyp image block, without the need to The entire endoscopic image is performed, so the real-time performance meets the requirements.
  • the recognition result can be output in real time for the image collected in real time, improving the processing efficiency of the polyp image.
  • FIG. 1-a is a schematic diagram of a structure of a medical system provided by an embodiment of the present application.
  • FIG. 1-b is a schematic flowchart of a method for processing a colon polyp image provided by an embodiment of the present application
  • FIG. 2 is a schematic diagram of an endoscope image provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of an endoscope image provided by an embodiment of the present application as a qualified picture
  • FIG. 4 is a schematic diagram of an endoscope image provided by an embodiment of the present application as a picture with abnormal color tone and overexposure and underexposure;
  • FIG. 5 is a schematic diagram of an endoscope image provided by an embodiment of the present application as a blurred picture
  • 6-a is a schematic diagram of an endoscope image provided by an embodiment of the present application as a white light type picture
  • 6-b is a schematic diagram of an endoscope image provided by an embodiment of the present application as an NBI type picture
  • FIG. 7 is a schematic diagram of a polyp image block circled on an endoscopic image provided by an embodiment of the present application.
  • 8-a is a schematic structural diagram of a device for processing a colon polyp image provided by an embodiment of the present application
  • 8-b is a schematic structural diagram of another apparatus for processing a colon polyp image provided by an embodiment of the present application.
  • 8-c is a schematic structural diagram of another apparatus for processing a colon polyp image provided by an embodiment of the present application.
  • FIG. 8-d is a schematic diagram of a composition structure of a picture type identification module provided by an embodiment of the present application.
  • 8-e is a schematic structural diagram of a polyp classification module provided by an embodiment of the present application.
  • FIG. 9 is a schematic diagram of a composition structure of a terminal applied to a method for processing a colon polyp image provided by an embodiment of the present application.
  • FIG. 10 is a schematic diagram of a composition structure of a method for processing a colon polyp image provided by an embodiment of the present application applied to a server.
  • the embodiments of the present application provide a method, a device and a system for processing a colon polyp image, which are used to find the position of a polyp in real time and determine the nature of the polyp, and improve the processing efficiency of the polyp image.
  • An embodiment of a method for processing a colon polyp image of the present application can be specifically applied to a colon polyp image processing scene for an endoscopic video stream.
  • a recognition result is output, and the recognition result It can be used to assist doctors to find polyps in real time and judge the nature of polyps during endoscopy, and guide the doctor's next operation.
  • the medical system 10 includes: an endoscope device 20 and a colon polyp image processing device 30, an endoscope device 20 and a colon polyp image processing device There is a communication connection established between 30;
  • the endoscope device 20 is configured to generate an endoscope video stream; send the endoscope video stream to the colon polyp image processing device 30;
  • a colon polyp image processing device 30 is configured to receive an endoscopic video stream from the endoscopic device 20; obtain an endoscopic image to be processed from the endoscopic video stream; use a polyp positioning model for the endoscopic to be processed Mirror image for polyp position detection, locate the polyp image block from the endoscopic image, polyp image block includes: polyp position area in the endoscopic image; use polyp property recognition model to classify polyp type on the polyp image block Detection, output recognition results.
  • the medical system provided by the embodiments of the present application includes an endoscope device and a colon polyp image processing device, and the endoscope device and the colon polyp image processing device may be used to transmit an endoscope video stream in a wired or wireless manner
  • the endoscope device can take an image of the colon in the patient's body through the endoscope to generate an endoscope video stream.
  • the processing device of the colon polyp image uses a polyp positioning model to detect the position of the polyp, which can be directly from the endoscope image
  • the polyp image block is located, and the classification detection for the polyp type is also performed on the polyp image block, and does not need to be performed on the entire endoscopic image, so the real-time performance meets the requirements.
  • the endoscope is controlled to move, the real-time
  • the collected images can output the recognition results in real time, improving the processing efficiency of polyp images.
  • a method for processing a colon polyp image may include the following steps:
  • the processing device for colon polyp image uses the polyp positioning model to detect the polyp position of the endoscopic image to be processed, and locates the polyp image block from the endoscopic image.
  • the polyp image block includes: the position of the polyp in the endoscopic image region.
  • the endoscopic image to be processed may be an endoscope image obtained by the processing device of the colon polyp image from the endoscope video stream to a single frame, or may be a processing device of the colon polyp image from the inside
  • the endoscopic image of a single frame received by the speculum device.
  • After acquiring the endoscopic image of a single frame use a pre-completed polyp positioning model to detect the position of the polyp on the endoscopic image.
  • the polyp positioning model includes the trained network parameters.
  • the network parameters can detect which image areas on the endoscope image conform to the characteristics of the polyp, thereby determining that the position area conforming to the characteristics of the polyp is the polyp image block from the endoscope image to the circle in the embodiment of the present application, as shown in FIG. 7
  • FIG. 7 it is a schematic diagram of a polyp image block circled on an endoscopic image provided by an embodiment of the present application.
  • the polyp image block includes: the position area of the polyp in the endoscopic image.
  • the embodiment of the present application uses pre-completed
  • the polyp positioning model can quickly circle the polyp image blocks through the model detection to ensure that the polyp image blocks can be determined in real time after generating the endoscopic video stream to ensure that the classification detection of polyp types can be performed in real time.
  • the pre-trained polyp positioning model in the embodiments of the present application also needs to be divided into white light polyp positioning Model and NBI polyp positioning model.
  • the white light polyp positioning model is obtained as follows: the colon polyp image processing device uses a neural network algorithm to train the polyp position of the original polyp positioning model through the white light type picture training data;
  • the NBI polyp localization model can be obtained as follows: the colon polyp image processing device uses a neural network algorithm to perform polyp position training on the original polyp localization model through NBI-type image training data.
  • the training data for the white light type and the NBI type are first obtained in advance, that is, the white light type image training data and the NBI type image training data are obtained, and the polyp positioning model is pre-trained using a neural network algorithm.
  • a variety of machine learning algorithms can be used to complete the training.
  • the polyp localization model can be a deep neural network model, a recurrent neural network model, etc.
  • the polyp localization model can complete the training through the YOLO V2 algorithm.
  • the above step 101 processing device for colon polyp image uses the polyp positioning model to perform polyp position on the endoscopic image Detection, locate the polyp image block from the endoscopic image, including:
  • the endoscope image is a white light type picture
  • the NBI polyp positioning model is used to locate the polyp, and the NBI polyp image block is located from the endoscopic image.
  • the embodiments of the present application use the YOLO v2 algorithm to locate and detect polyps.
  • the principle and implementation of YOLOv2 are as follows.
  • the YOLOv2 is a joint detection and classification training method. Using this joint training method, the YOLO9000 model is trained on the COCO detection data set and the ImageNet classification data set, which can detect more than 9,000 types of objects. .
  • YOLOv2 Compared with YOLOv1, YOLOv2 has made many improvements, which also makes YOLOv2's performance significantly improved, and the speed of YOLOv2 is still very fast.
  • the YOLO V2 algorithm is an upgraded version of the YOLO algorithm. It is an end-to-end real-time target detection and recognition algorithm. The algorithm uses a single neural network to transform the target detection problem into extracting bounding boxes and category probabilities in the image. Back to the problem.
  • YOLO v2 algorithm adopts multi-scale training method and borrows Faster RCNN anchor box (anchor box) idea, while ensuring the detection speed, greatly improving the accuracy and generalization ability of model detection.
  • the YOLO v2 algorithm is applied to the polyp positioning task of the embodiment of the present application, the detection target is colon polyps, and the size setting of the anchor box is obtained by clustering the polyp training data.
  • transfer learning technology is used. Among them, transfer learning refers to applying mature knowledge in one field to other scenes, expressed in terms of neural network, which is the weight of each node in the layer-by-layer network Migrating from a trained network to a brand new network, rather than starting from scratch, does not require training a neural network for each specific task.
  • Imagenet data is an open source data set for image classification and object detection in the field of computer vision. Covering thousands of categories, the data volume is more than one million. Using model initialization parameters trained on a large number of data sets can better allow the model to converge to the global optimal solution.
  • white light type pictures and NBI type pictures will be distinguished.
  • the two types of images are very different in the appearance of polyps.
  • NBI type images can observe the blood vessel flow direction, and the color of blood appears black in NBI type images, so it is necessary to target white light image data and NBI image data separately.
  • Training polyp positioning model referred to as white light polyp positioning model and NBI polyp positioning model. Both of these polyp positioning models are trained using the method described above. The only difference is the training data of the model.
  • the training data of the white light polyp positioning model is a white light type picture
  • the NBI polyp positioning model is an NBI type picture.
  • the white light polyp positioning model is called to locate the polyp, otherwise, the NBI polyp positioning model is called to locate the polyp.
  • the circled polyp image block is output as the input of the polyp nature recognition model.
  • the method for processing a colon polyp image provided by an embodiment of the present application may further include the following step 100.
  • a device for processing a colon polyp image acquires an endoscopic image to be processed from an endoscope video stream.
  • the endoscope device may generate an endoscope video stream, the endoscope video stream includes a continuous endoscope image of multiple frames, the endoscope After the device generates the endoscopic video stream, the endoscopic video stream is sent to the colon polyp image processing device.
  • the colon polyp image processing device can receive the endoscopic video stream from the endoscopic device and the endoscopic video stream
  • the endoscopic image of a single frame is acquired in each frame, and the endoscopic image of each frame can be used to identify the position of the polyp and the type of polyp according to the method provided in the embodiments of the present application, so that the nature of the colon polyp in the endoscope video stream can be identified in real time
  • the doctor operates the endoscope to check the colon the position of the colon polyp in the video stream is located in real time and the nature of the polyp is determined. If the nature of the polyp is non-adenomatous polyp, the doctor can send the pathological examination without removing the polyp.
  • FIG. 2 it is a schematic diagram of an endoscopic image provided by an embodiment of the present application.
  • an endoscope image of one frame is extracted from the endoscope video stream.
  • the endoscope image is the colon image shown in the box.
  • the left side of the endoscope image is the endoscope parameters, and the endoscope parameter values can be set according to the actual scene.
  • the parameters of the endoscope have nothing to do with image processing. Therefore, after the endoscope video stream is collected, only the part of the colon image area can be retained.
  • step 100 obtains the endoscopic image to be processed from the endoscope video stream
  • step 100 obtains the endoscopic image to be processed from the endoscope video stream
  • a colon polyp image processing device extracts color features, gradient changes, and abnormal brightness features from the endoscopic image
  • the processing device of the colon polyp image determines whether the endoscopic image is a low-quality image according to the color characteristics, gradient change characteristics and abnormal brightness characteristics.
  • the low-quality images include: blurred pictures, abnormal color tones, overexposed and underexposed pictures, and low resolution pictures;
  • the processing device of the colon polyp image uses the polyp positioning model to detect the polyp position of the endoscopic image.
  • low-quality pictures can also be called low-quality pictures.
  • color features and gradients can be extracted The change feature and the abnormal brightness feature detect whether the endoscopic image is a low-quality picture based on the three extracted features.
  • the low-quality pictures defined in the embodiments of the present application include three types: blur, abnormal color tone, overexposure and underexposure, and low image resolution.
  • FIG. 3 it is a schematic diagram of the endoscope image provided by the embodiment of the present application as a qualified picture.
  • the two left and right pictures shown in Figure 3 are qualified pictures.
  • Qualified pictures refer to pictures other than blur, abnormal color tone, overexposure and underexposure, and low image resolution.
  • the endoscope images provided in the embodiments of the present application are schematic diagrams of abnormal color tone and overexposure and underexposure pictures. The color abnormality occurs in the left and right pictures shown in FIG. 4, so it is not Qualified pictures.
  • FIG. 3 it is a schematic diagram of the endoscope image provided by the embodiment of the present application as a qualified picture.
  • the two left and right pictures shown in Figure 3 are qualified pictures.
  • Qualified pictures refer to pictures other than blur, abnormal color tone, overexposure and underexposure, and low image resolution.
  • the endoscope image provided by the embodiment of the present application is a schematic diagram of a blurred picture. Both the left and right pictures shown in FIG. 5 are blurred, and therefore are unqualified pictures. Next, the specific recognition process of blurry pictures, abnormal color tones, overexposed and underexposed pictures, and low-resolution pictures will be separately illustrated.
  • the effective pixel area refers to the area after trimming the black, white, and black borders of the picture, as shown by the white frame in Figure 2 .
  • the black border trimming algorithm is mainly based on counting the gray value distribution of pixel values in each row or column. If the pixel values satisfying gray or black are greater than a certain percentage, the row or column should be cut off. If the effective area after cutting out the black border is less than a certain threshold, it is regarded as a low-resolution picture.
  • the threshold can be set according to the actual application.
  • the detection algorithm of fuzzy pictures is as follows:
  • the original image is defined as R, and the image P is obtained after a median filter with a pixel value of 3 * 3.
  • the endoscopic image that satisfies the above target tone matching result can be determined as an abnormal tone and an overexposed or underexposed picture.
  • the endoscopic video stream can be generated in multiple shooting modes, so the endoscopic image in the endoscopic video stream can have multiple picture types according to different shooting modes.
  • different types of polyp positioning models are required. For details, refer to the description in the subsequent embodiments.
  • Step 100 After obtaining the to-be-processed endoscopic image from the endoscopic video stream, the method provided by the embodiment of the present application further includes the following steps:
  • the processing device of the colon polyp image performs picture type recognition on the endoscopic image, and determines that the endoscopic image is a white light type image, or an endoscopic narrow band imaging (Narrow Band Imaging, NBI) type image.
  • NBI Endoscopic narrow band imaging
  • the endoscope image extracted from the video stream may also have different picture types, for example, the endoscope image may be a white light type picture, or the endoscope The mirror image may be an NBI type picture.
  • FIG. 6-a it is a schematic diagram of the endoscope image provided by the embodiment of the present application as a white light type picture.
  • the white light type picture refers to red (Red, R) and green (Green, G) imaged by a common light source 2.
  • Blue (B) (RGB for short) images are schematic diagrams of endoscope images provided by the embodiments of the present application as NBI type pictures.
  • the NBI type pictures are filtered out using filters
  • the broadband spectrum in the red, blue, and green light waves emitted by the mirror light source only leaves a narrow band spectrum for diagnosis of various diseases of the digestive tract.
  • picture type recognition is performed on the endoscope image to determine that the endoscope image is a white light type picture or an NBI type picture, including:
  • the training data for the white light type and the NBI type are first obtained in advance, that is, the white light type image training data and the NBI type image training data are obtained, and an image classification model is pre-trained using a neural network algorithm.
  • the image classification model A variety of machine learning algorithms can be used to complete the training.
  • the image classification model can specifically be a deep neural network model (Deep Neural Networks, DNN), a recurrent neural network model, etc.
  • the deep neural network model can be a densely connected convolutional network ( Densely Connected Convolutional Networks, DenseNet) etc.
  • the trained image classification model uses the trained image classification model to extract the color features of the blood vessel from the endoscopic image.
  • the color features of the blood vessel are the classification basis of the endoscopic image.
  • use the trained image classification model uses the trained image classification model to The values of the color features of blood vessels are classified, and the endoscopic image is obtained as a white light type picture or an NBI type picture.
  • the input of the image classification model is a qualified single-frame endoscopic image
  • the output of the image classification model is whether the endoscopic image is a white light type picture or an NBI type picture.
  • doctors actually perform endoscopy when they find suspicious polyps, they often use the NBI mode to diagnose the current pathology of polyps.
  • the pictures of NBI mode can show the direction of blood vessels more clearly.
  • Figure 6-a shows a white light type picture
  • Figure 6-b shows an NBI type picture.
  • the image classification model of the embodiment of the present application may use a densely connected convolutional network (DenseNet) to classify and identify picture types.
  • DenseNet densely connected convolutional network
  • other image classification networks may be used in the embodiment of the present application to achieve similar functions, but the recognition effect will be There is a certain degree of difference, which is not limited here.
  • the implementation of the image classification model can be transformed into an image classification problem.
  • the image classification algorithm used is DenseNet convolutional network.
  • the input image size of the network is 224 * 224, so the input original picture is first scaled down and scaled to a fixed 224 * 224 size.
  • it prefers lower-level feature combinations, such as blood vessel color, etc. so when designing the combination of depth and width of DenseNet structure, a wider and lighter mode is adopted.
  • the final network structure used is DenseNet-40, where 40 refers to the number of layers of the network.
  • the growth rate (growth-rate) is set to 48, and the compression ratio of the feature through the transition layer is 0.5, the effect is optimal.
  • the model structure is shown in Table 1 below:
  • the function implementation and execution process of each layer in DenseNet-40 can be determined according to the scenario.
  • the conv in the network layer contains three operations: batch normalization layer (batchnorm), activation layer (ReLU), and convolutional layer.
  • the processing device of the colon polyp image uses the polyp property recognition model to perform classification detection on the polyp image block, and outputs a recognition result.
  • the recognition result can output the most probable polyp type, and the recognition result can also output the polyp type under various confidence conditions, where the confidence level is determined after the prediction of the polyp nature recognition model, and the polyp image block contains each The credibility of the type of polyp.
  • the polyp property recognition model may perform a polyp property discrimination task, for example, through an image classification task, and the input is image data of a positioning frame output by the polyp positioning model.
  • the polyp image blocks circled on the endoscopic image are polyps detected by the polyp positioning model, and serve as input data for the polyp property recognition model.
  • the module output can be four categories of values (0,1,2,3), where 0 means that there is no polyp in this area is normal, 1 means non-adenomatous polyps, 2 means adenomatous polyps, 3 means adenocarcinoma, in addition, For normal, non-adenoma, adenoma, and adenocarcinoma, a confidence condition can also be set. If the output is 0, the judgment result of the polyp positioning model is corrected. This area has no polyps and is a normal area.
  • the processing device for the colon polyp image uses a polyp property recognition model to perform classification detection on the polyp image block, and the output recognition result includes:
  • the original polyp property recognition model is classified and trained by the polyp picture training data of different polyp types, and the trained polyp property recognition model is obtained;
  • the polyp property recognition model can use a variety of machine learning algorithms to complete training, such as the polyp property
  • the recognition model may specifically be a deep neural network model, a recurrent neural network model, etc.
  • the deep neural network model may be DenseNet or the like.
  • the polyp type feature is extracted from the polyp image block using the trained polyp property recognition model.
  • the polyp type feature is the classification basis of the polyp image block.
  • the trained polyp property recognition model is used to The values of polyp type characteristics are classified to obtain the recognition result.
  • the method provided by the embodiment of the present application further includes:
  • the processing device of the colon polyp image performs external expansion in the up, down, left, and right directions on the polyp image block occupied by the endoscopic image according to a preset image external expansion ratio to obtain the externally expanded polyp image block;
  • the processing device of the colon polyp image inputs the expanded polyp image block into the polyp property recognition model.
  • the polyp property classification task of the polyp property recognition model can be implemented by the DenseNet convolutional network algorithm, because the algorithm requires that the input image size must be consistent, and the polyp position output by the polyp positioning module varies in size.
  • the method used in the embodiments of the present application is to expand the polyp image block output by the polyp positioning model by a 10% proportional area up, down, left, and right to ensure that the framed area has certain contextual semantic information to assist Subsequent polyp property recognition models extract features, and the expanded area is directly normalized to the input size 224 * 224 required by the model.
  • a deeper DenseNet network can be used.
  • the final network structure used is DenseNet-121. Through network parameter tuning, the growth-rate is set to 24, and the compression ratio of the feature after the transition layer is 0.5, and the effect is optimal.
  • the model structure is shown in Table 2 below:
  • the method for processing polyp images requires about 100 milliseconds (ms) to process each frame of endoscopic image, which meets the real-time requirements, and the algorithm effect is compared with doctors of different levels, compared with top doctors The level is quite. Deployed to primary hospitals, it can help doctors find polyps and identify polyps in real time.
  • the doctor can help the doctor to find the polyp and judge the nature of the polyp in real time when performing the endoscopy. To prevent doctors from missing polyps, and to help doctors improve the accuracy of judging the nature of polyps. If it is judged as a high-confidence non-adenomatous polyp, the doctor can send the pathology without excision, which can reduce the doctor's operation time, further reduce the patient's high risk of complications and the cost of patient diagnosis and treatment, and reduce the burden of endoscopist and pathologist.
  • the polyp positioning model is used to detect the position of the polyp, and the polyp image block is located from the endoscope image.
  • the polyp image block includes: polyp endoscope image Location area in.
  • a polyp property recognition model is used to classify and detect the polyp type on the polyp image block, and output the recognition result.
  • the polyp image block can be directly located from the endoscopic image.
  • the classification detection for the type of polyp is also performed on the polyp image block, without the need to The entire endoscopic image is performed, so the real-time performance meets the requirements.
  • the recognition result can be output in real time for the image collected in real time, improving the processing efficiency of the polyp image.
  • a colon polyp image processing device 800 may include one or more processors and one or more memories storing program units, where the program units are composed of The processor executes, the program unit includes: a position detection module 801, a polyp classification module 802, wherein,
  • the position detection module 801 is configured to use the polyp positioning model to perform polyp position detection on the endoscopic image to be processed, and locate a polyp image block from the endoscopic image.
  • the polyp image block includes: the polyp is in the Location area in the speculum image;
  • the polyp classification module 802 is configured to perform polyp type classification detection on the polyp image block using a polyp property recognition model, and output a recognition result.
  • the apparatus 800 for processing a colon polyp image may further include: an image acquisition module 803,
  • the image acquisition module 803 is configured to acquire the endoscopic image to be processed from the endoscope video stream.
  • the apparatus 800 for processing a colon polyp image further includes:
  • the low-quality picture recognition module 804 is configured to extract the color feature, gradient change feature and brightness from the endoscope image before the position detection module 801 uses the polyp positioning model to detect the position of the polyp to be processed Abnormal features; judging whether the endoscopic image is a low-quality picture according to the color feature, the gradient change feature and the brightness abnormal feature Picture, low-resolution picture; when the endoscopic image is not the low-quality picture, trigger execution of the position detection module.
  • the apparatus 800 for processing a colon polyp image further includes:
  • a picture type recognition module 805 which is configured to determine the endoscope by performing picture type recognition on the endoscope image before performing position detection on the endoscope image to be processed using the polyp positioning model
  • the image is a white-light type picture, or an NBI type picture with endoscopic narrowband imaging.
  • the picture type identification module 805 includes:
  • the image classification model training unit 8051 is set to use a neural network algorithm to classify the original image classification model through white light type image training data and NBI type image training data to obtain the trained image classification model;
  • the blood vessel color feature extraction unit 8052 is configured to extract the blood vessel color feature from the endoscopic image using the trained image classification model
  • the picture classification unit 8053 is set to use the trained image classification model to classify the values of the blood vessel color features to obtain the endoscopic image as the white light type picture or the NBI type picture.
  • the polyp positioning model includes: a white light polyp positioning model and an NBI polyp positioning model;
  • the white light polyp positioning model is obtained by using a neural network algorithm to train the polyp position of the original polyp positioning model through the white light type picture training data;
  • the NBI polyp positioning model is obtained by using a neural network algorithm to train the polyp position of the original polyp positioning model through the NBI type picture training data.
  • the position detection module 801 is specifically configured to use the white light polyp positioning model for polyp positioning when the endoscopic image is the white light type picture.
  • a white light polyp image block is located on the endoscope image; when the endoscope image is the NBI type picture, the NBI polyp positioning model is used to locate the polyp, and the NBI polyp image is located from the endoscope image Piece.
  • the polyp classification module 802 includes:
  • Polyp property recognition model training unit 8021 is set to use neural network algorithm to carry out classification detection training on the original polyp property recognition model through polyp picture training data of different polyp types to obtain the trained polyp property recognition model;
  • a polyp type feature extraction unit 8022 is configured to extract the polyp type feature from the polyp image block using the trained polyp property recognition model
  • the polyp classification unit 8023 is configured to use the trained polyp property recognition model to classify the value of the polyp type feature and output a recognition result.
  • the polyp positioning model is used to detect the position of the polyp, and the polyp image block is located from the endoscope image.
  • the polyp image block includes: polyp endoscope image Location area in.
  • a polyp property recognition model is used to classify and detect the polyp type on the polyp image block, and output the recognition result.
  • the polyp image block can be directly located from the endoscopic image.
  • the classification detection for the type of polyp is also performed on the polyp image block, without the need to The entire endoscopic image is performed, so the real-time performance meets the requirements.
  • the recognition result can be output in real time for the image collected in real time, improving the processing efficiency of the polyp image.
  • the terminal may be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (Personal Digital Assistant, PDA), a sales terminal (Point of Sales, POS), an in-vehicle computer, etc. Taking the terminal as a mobile phone as an example:
  • the mobile phone includes: a radio frequency (Radio Frequency) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080 , And power supply 1090 and other components.
  • a radio frequency (Radio Frequency) circuit 1010 the radio frequency (Radio Frequency) circuit 1010
  • a memory 1020 includes: a radio frequency (Radio Frequency) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080 , And power supply 1090 and other components.
  • WiFi wireless fidelity
  • the structure of the mobile phone shown in FIG. 9 does not constitute a limitation on the mobile phone, and may include more or less components than those shown in the figure, or a combination of certain components, or a different component arrangement.
  • the RF circuit 1010 may be configured to receive and send signals during sending and receiving information or during a call. In particular, after receiving the downlink information of the base station, it is processed by the processor 1080; in addition, the designed uplink data is sent to the base station.
  • the RF circuit 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, and the like.
  • the RF circuit 1010 can also communicate with the network and other devices through wireless communication.
  • the above wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile (GSM), General Packet Radio Service (GPRS), and Code Division Multiple Access (Code Division) Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), E-mail, Short Message Service (SMS), etc.
  • GSM Global System of Mobile
  • GPRS General Packet Radio Service
  • CDMA Code Division Multiple Access
  • WCDMA Wideband Code Division Multiple Access
  • LTE Long Term Evolution
  • SMS Short Message Service
  • the memory 1020 may be configured to store software programs and modules, and the processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020.
  • the memory 1020 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, at least one function required application programs (such as a sound playback function, an image playback function, etc.), etc .; Data created by the use of mobile phones (such as audio data, phone books, etc.), etc.
  • the memory 1020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.
  • the input unit 1030 may be configured to receive input numeric or character information, and generate key signal input related to user settings and function control of the mobile phone.
  • the input unit 1030 may include a touch panel 1031 and other input devices 1032.
  • the touch panel 1031 also known as a touch screen, can collect user's touch operations on or near it (such as the user using any finger, stylus, or any other suitable object or accessory on the touch panel 1031 or near the touch panel 1031 Operation), and drive the corresponding connection device according to the preset program.
  • the touch panel 1031 may include a touch detection device and a touch controller.
  • the touch detection device detects the user's touch orientation, and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device and converts it into contact coordinates, and then sends To the processor 1080, and can receive the commands sent by the processor 1080 and execute them.
  • the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves.
  • the input unit 1030 may also include other input devices 1032.
  • other input devices 1032 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), trackball, mouse, joystick, and so on.
  • the display unit 1040 may be configured to display information input by the user or information provided to the user and various menus of the mobile phone.
  • the display unit 1040 may include a display panel 1041.
  • the display panel 1041 may be configured in the form of a liquid crystal display (Liquid Crystal) (LCD), an organic light emitting diode (Organic Light-Emitting Diode, OLED), or the like.
  • the touch panel 1031 may cover the display panel 1041, and when the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event, and then the processor 1080 according to the touch event The type provides corresponding visual output on the display panel 1041.
  • the touch panel 1031 and the display panel 1041 are implemented as two independent components to realize the input and input functions of the mobile phone, in some embodiments, the touch panel 1031 and the display panel 1041 may be integrated to Realize the input and output functions of the mobile phone.
  • the mobile phone may further include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors.
  • the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may close the display panel 1041 and / or when the mobile phone moves to the ear Or backlight.
  • the accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when at rest, and can be used to identify mobile phone gesture applications (such as horizontal and vertical screen switching, related Games, magnetometer posture calibration), vibration recognition related functions (such as pedometers, percussion), etc.
  • other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., are no longer here Repeat.
  • the audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone.
  • the audio circuit 1060 can transmit the received audio data converted electrical signal to the speaker 1061, which converts the speaker 1061 into a sound signal output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is converted by the audio circuit 1060 After receiving, it is converted into audio data, and then processed by the audio data output processor 1080, and then sent to, for example, another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.
  • WiFi is a short-range wireless transmission technology. Mobile phones can help users send and receive emails, browse web pages, and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access.
  • FIG. 9 shows the WiFi module 1070, it can be understood that it is not a necessary component of the mobile phone, and can be omitted as needed without changing the scope of the essence of the invention.
  • the processor 1080 is the control center of the mobile phone, and uses various interfaces and lines to connect various parts of the entire mobile phone, by running or executing software programs and / or modules stored in the memory 1020, and calling data stored in the memory 1020, the Various functions and processing data of the mobile phone, so as to monitor the mobile phone as a whole.
  • the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, where the application processor mainly processes an operating system, a user interface, and application programs, etc.
  • the modem processor mainly handles wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 1080.
  • the mobile phone also includes a power supply 1090 (such as a battery) for powering various components.
  • a power supply 1090 (such as a battery) for powering various components.
  • the power supply can be logically connected to the processor 1080 through a power management system, so as to realize functions such as charging, discharging, and power management through the power management system.
  • the mobile phone may also include a camera, a Bluetooth module, etc., which will not be repeated here.
  • the processor 1080 included in the terminal further has a process flow for controlling the execution of the above-mentioned colon polyp image processing executed by the terminal.
  • FIG. 10 is a schematic diagram of a server structure provided by an embodiment of the present application.
  • the server 1100 may have a relatively large difference due to different configurations or performances, and may include one or more central processing units (CPUs) 1122 (for example , One or more processors) and memory 1132, one or more storage media 1130 (for example, one or more mass storage devices) that stores application programs 1142 or data 1144.
  • the memory 1132 and the storage medium 1130 may be short-term storage or persistent storage.
  • the program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server.
  • the central processor 1122 may be configured to communicate with the storage medium 1130 and execute a series of instruction operations in the storage medium 1130 on the server 1100.
  • Server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158, and / or one or more operating systems 1141, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
  • operating systems 1141 such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
  • the processing method steps of the colon polyp image executed by the server in the above embodiment may be based on the server structure shown in FIG. 10.
  • the device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be The physical unit may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. Those of ordinary skill in the art can understand and implement without paying creative labor.
  • the technical solutions of the embodiments of the present application can be embodied in the form of software products that are essentially or contribute to the existing technology, and the computer software products are stored in a readable storage medium, such as a computer Floppy disk, U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or CD-ROM, etc., including several instructions to make a computer device ( It may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments of the present application.
  • the polyp positioning model is first used to detect the polyp position of the endoscopic image, and the polyp image block is located from the endoscopic image.
  • the polyp image block includes: the position area of the polyp in the endoscopic image.
  • a polyp property recognition model is used to classify and detect the polyp type on the polyp image block, and output the recognition result.
  • the polyp positioning model since the polyp positioning model is used to detect the position of the polyp, the polyp image block can be directly located from the endoscopic image.
  • the classification detection for the type of polyp is also performed on the polyp image block, without the need to The entire endoscopic image is performed, so the real-time performance meets the requirements.
  • the recognition results can be output in real time for the images collected in real time, improving the processing efficiency of polyp images,

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Primary Health Care (AREA)
  • Epidemiology (AREA)
  • Quality & Reliability (AREA)
  • Biomedical Technology (AREA)
  • Pathology (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Image Analysis (AREA)
  • Endoscopes (AREA)
  • Image Processing (AREA)

Abstract

一种肠息肉图像的处理方法和装置及系统，用于实时发现息肉位置并判断息肉的性质，提高息肉图像的处理效率。一种结肠息肉图像的处理方法，包括：使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测，从所述内窥镜图像上定位出息肉图像块，所述息肉图像块包括：息肉在所述内窥镜图像中的位置区域(101)；使用息肉性质识别模型在所述息肉图像块上进行息肉类型的分类检测，输出识别结果(102)。

Description

一种结肠息肉图像的处理方法和装置及系统
本申请要求于2018年10月31日提交中国专利局、申请号为201811287489.X、发明名称“一种结肠息肉图像的处理方法和装置及系统”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及计算机技术领域，尤其涉及一种结肠息肉图像的处理方法和装置及系统。
背景技术
当前，我国高发的恶性肿瘤类型中，结肠癌位居前5位以内。而结肠癌在北美和欧洲的发病率同样居高不下。结肠癌是常发于结肠部位的消化道恶性肿瘤。一般来讲，晚期结肠癌50％的患者将死于复发及转移，早期结肠癌接近100％患者可以完全治愈。因此，结肠癌的防治很有必要。但是通过临床症状无法预测出结肠早癌。
现有技术中在识别结肠息肉时，通常采用滑窗的方式来检测息肉图像(滑窗指的是在一个内窥镜视频图像帧中，采用先从上到下、然后从左至右的方式滑动图像块)，或者采用人工标注息肉位置的方式。在确定出息肉位置后，再使用计算机视觉(Computer Vision)提取方法，通过分类方法输出识别结果。
现有技术提供的上述方案中，对于滑窗的方式，在一个内窥镜视频图像帧上采用滑窗的方式计算每个图像块是否包含息肉，由于图像块很多，导致计算量大，实时性不符合要求，在内窥镜被控制移动时，对于实时采集到的图像无法实时的输出识别结果。对于人工标注的方式，实时性不符合要求，在内窥镜被控制移动时，对于实时采集到的图像无法实时的输出识别结果。
发明内容
本申请实施例提供了一种结肠息肉图像的处理方法和装置及系统，用于实时发现息肉位置并判断息肉的性质，提高息肉图像的处理效率。
本申请实施例提供以下技术方案：
一方面，本申请实施例提供一种结肠息肉图像的处理方法，包括：
结肠息肉图像的处理装置使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测，从所述内窥镜图像上定位出息肉图像块，所述息肉图像块包括：息肉在所述内窥镜图像中的位置区域；
所述结肠息肉图像的处理装置使用息肉性质识别模型在所述息肉图像块上进行息肉类型的分类检测，输出识别结果。
另一方面，本申请实施例还提供一种结肠息肉图像的处理装置，包括：
位置检测模块，被设置为使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测，从所述内窥镜图像上定位出息肉图像块，所述息肉图像块包括：息肉在所述内窥镜图像中的位置区域；
息肉分类模块，被设置为使用息肉性质识别模型在所述息肉图像块上进行息肉类型的分类检测，输出识别结果。
在前述方面中，结肠息肉图像的处理装置的组成模块还可以执行前述一方面以及各种可能的实现方式中所描述的步骤，详见前述对前述一方面以及各种可能的实现方式中的说明。
另一方面，本申请实施例还提供一种医疗系统，所述医疗系统包括：内窥镜装置和结肠息肉图像的处理装置，所述内窥镜装置和所述结肠息肉图像的处理装置之间建立有通信连接；其中，
所述内窥镜装置，被设置为生成内窥镜视频流；并将所述内窥镜视频流发送给所述结肠息肉图像的处理装置；
所述结肠息肉图像的处理装置，被设置为从所述内窥镜装置接收到所述内窥镜视频流；从所述内窥镜视频流中获取待处理的内窥镜图像；使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测，从所述内窥镜图像上定位出息肉图像块，所述息肉图像块包括：息肉在所述内窥镜图像中的位置区域；使用息肉性质识别模型在所述息肉图像块上进行息肉类型的分类检测，输出识别结果。
另一方面，本申请实施例提供一种图像的处理方法，包括：
图像的处理装置使用目标对象定位模型对待处理的图像进行目标对象位置检测，从所述图像上定位出目标对象图像块，所述目标对象图像块包括：所述目标对象在所述图像中的位置区域；
所述图像的处理装置使用目标对象性质识别模型在所述目标对象图像块上进行目标对象类型的分类检测，输出识别结果。
另一方面，本申请实施例提供一种结肠息肉图像的处理装置，该结肠息肉图像的处理装置包括：处理器、存储器；存储器被设置为存储指令；处理器被设置为执行存储器中的指令，使得结肠息肉图像的处理装置执行如前述一方面中任一项的方法。
另一方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，当其在计算机上运行时，使得计算机执行上述各方面所述的方法。
在本申请实施例中，首先使用息肉定位模型对内窥镜图像进行息肉位置检测，从内窥镜图像上定位出息肉图像块，息肉图像块包括：息肉在内窥镜图像中的位置区域。最后使用息肉性质识别模型在息肉图像块上进行息肉类型的分类检测，输出识别结果。本申请实施例中由于采用的是息肉定位模型来检测息肉位置，可以直接从内窥镜图像中定位出息肉图像块，针对息肉类型的分类检测也是在该息肉图像块上进行，而不需要在整个内窥镜图像上进行，因此实时性符合要求，在内窥镜被控制移动时，对于实时采集到的图像可以实时的输出识别结果，提高息肉图像的处理效率。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域的技术人员来讲，还可以根据这些附图获得其他的附图。
图1-a为本申请实施例提供的一种医疗系统的组成结构示意图；
图1-b为本申请实施例提供的一种结肠息肉图像的处理方法的流程方框示意图；
图2为本申请实施例提供的内窥镜图像的示意图；
图3为本申请实施例提供的内窥镜图像为合格图片的示意图；
图4为本申请实施例提供的内窥镜图像为色调异常及过曝欠曝图片的示意图；
图5为本申请实施例提供的内窥镜图像为模糊图片的示意图；
图6-a为本申请实施例提供的内窥镜图像为白光类型图片的示意图；
图6-b为本申请实施例提供的内窥镜图像为NBI类型图片的示意图；
图7为本申请实施例提供的在内窥镜图像上圈出的息肉图像块的示意图；
图8-a为本申请实施例提供的一种结肠息肉图像的处理装置的组成结构示意图；
图8-b为本申请实施例提供的另一种结肠息肉图像的处理装置的组成结构示意图；
图8-c为本申请实施例提供的另一种结肠息肉图像的处理装置的组成结构示意图；
图8-d为本申请实施例提供的一种图片类型识别模块的组成结构示意 图；
图8-e为本申请实施例提供的一种息肉分类模块的组成结构示意图；
图9为本申请实施例提供的结肠息肉图像的处理方法应用于终端的组成结构示意图；
图10为本申请实施例提供的结肠息肉图像的处理方法应用于服务器的组成结构示意图。
具体实施方式
本申请实施例提供了一种结肠息肉图像的处理方法和装置及系统，用于实时发现息肉位置并判断息肉的性质，提高息肉图像的处理效率。
为使得本申请实施例的发明目的、特征、优点能够更加的明显和易懂，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，下面所描述的实施例仅仅是本申请一部分实施例，而非全部实施例。基于本申请中的实施例，本领域的技术人员所获得的所有其他实施例，都属于本申请实施例保护的范围。
本申请的说明书和权利要求书及上述附图中的术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，以便包含一系列单元的过程、方法、系统、产品或设备不必限于那些单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它单元。
以下分别进行详细说明。
本申请结肠息肉图像的处理方法的一个实施例，具体可以应用于针对内窥镜视频流的结肠息肉图像处理场景中，通过本申请实施例对结肠息肉图像的处理，输出识别结果，该识别结果可用于辅助医生在做内窥镜检查时实时发现息肉并判断息肉的性质，指导医生下一步的操作。
本申请实施例还提供一种医疗系统，如图1-a所示，医疗系统10包括：内窥镜装置20和结肠息肉图像的处理装置30，内窥镜装置20和结肠息肉 图像的处理装置30之间建立有通信连接；其中，
内窥镜装置20，被设置为生成内窥镜视频流；并将内窥镜视频流发送给结肠息肉图像的处理装置30；
结肠息肉图像的处理装置30，被设置为从内窥镜装置20接收到内窥镜视频流；从内窥镜视频流中获取待处理的内窥镜图像；使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测，从内窥镜图像上定位出息肉图像块，息肉图像块包括：息肉在内窥镜图像中的位置区域；使用息肉性质识别模型在息肉图像块上进行息肉类型的分类检测，输出识别结果。
在本申请实施例提供的医疗系统中包括内窥镜装置和结肠息肉图像的处理装置，内窥镜装置和结肠息肉图像的处理装置之间可以采用有线或者无线的方式来传输内窥镜视频流，内窥镜装置可以通过内窥镜对患者体内的结肠进行图像拍摄，从而生成内窥镜视频流，结肠息肉图像的处理装置使用息肉定位模型来检测息肉位置，可以直接从内窥镜图像中定位出息肉图像块，针对息肉类型的分类检测也是在该息肉图像块上进行，而不需要在整个内窥镜图像上进行，因此实时性符合要求，在内窥镜被控制移动时，对于实时采集到的图像可以实时的输出识别结果，提高息肉图像的处理效率。
请参阅图1-b所示，本申请一个实施例提供的结肠息肉图像的处理方法，可以包括如下步骤：
101、结肠息肉图像的处理装置使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测，从内窥镜图像上定位出息肉图像块，息肉图像块包括：息肉在内窥镜图像中的位置区域。
在本申请实施例中，待处理的内窥镜图像可以是结肠息肉图像的处理装置从内窥镜视频流中获取到单个帧的内窥镜图像，也可以是结肠息肉图像的处理装置从内窥镜装置接收到的单个帧的内窥镜图像。在获取到单个帧的内窥镜图像之后，使用预先完成的息肉定位模型在该内窥镜图像上进 行息肉位置检测，该息肉定位模型中包括有训练完成的网络参数，通过该息肉定位模型的网络参数可以检测在内窥镜图像上哪些图像区域符合息肉的特征，从而确定出符合息肉的特征的位置区域为本申请实施例中从内窥镜图像向圈出的息肉图像块，如图7所示，为本申请实施例提供的在内窥镜图像上圈出的息肉图像块的示意图，息肉图像块包括：息肉在内窥镜图像中的位置区域，本申请实施例中使用预先完成的息肉定位模型，通过模型检测可以快速的圈出息肉图像块，以保证在生成内窥镜视频流后能够实时的确定出息肉图像块，以保证息肉类型的分类检测可以实时进行。
在本申请的一些实施例中，针对内窥镜图像的图片类型的不同，可以分为白光类型图片和NBI类型图片，因此本申请实施例中预先训练的息肉定位模型也需要分为白光息肉定位模型和NBI息肉定位模型。
其中，白光息肉定位模型，通过如下方式得到：结肠息肉图像的处理装置使用神经网络算法，通过白光类型图片训练数据对原始的息肉定位模型进行息肉位置训练；
NBI息肉定位模型，通过如下方式得到：结肠息肉图像的处理装置使用神经网络算法，通过NBI类型图片训练数据对原始的息肉定位模型进行息肉位置训练。
在本申请实施例中，首先针对白光类型和NBI类型分别预先获取训练数据，即获取到白光类型图片训练数据和NBI类型图片训练数据，使用神经网络算法预先训练出息肉定位模型，该息肉定位模型可以采用多种机器学习算法完成训练，例如该息肉定位模型具体可以是深度神经网络模型、循环神经网络模型等，例如该息肉定位模型可以通过YOLO V2算法完成训练等。
在本申请的一些实施例中，在息肉定位模型分为白光息肉定位模型和NBI息肉定位模型的实现场景下，上述步骤101结肠息肉图像的处理装置使用息肉定位模型对内窥镜图像进行息肉位置检测，从内窥镜图像上定位出息肉图像块，包括：
当内窥镜图像为白光类型图片时，使用白光息肉定位模型进行息肉定位，从内窥镜图像上定位出白光息肉图像块；
当内窥镜图像为NBI类型图片时，使用NBI息肉定位模型进行息肉定位，从内窥镜图像上定位出NBI息肉图像块。
在本申请实施例中需要在内窥镜图像中定位到具体的息肉位置，给下一步息肉性质识别提供输入数据。考虑到实时性的要求，本申请实施例采用YOLO v2算法对息肉进行定位检测。YOLOv2的原理以及实现如下，该YOLOv2是一种检测与分类联合训练方法，使用这种联合训练方法在COCO检测数据集和ImageNet分类数据集上训练出了YOLO9000模型，其可以检测超过9000多类物体。YOLOv2相比YOLOv1做了很多方面的改进，这也使得YOLOv2的性能有显著的提升，并且YOLOv2的速度依然很快。YOLO V2算法是YOLO算法的升级版，是一种端到端的实时的目标检测与识别算法，算法利用单个神经网络，将目标检测问题转化为提取图像中的边界框(bounding boxes)和类别概率的回归问题。相比于YOLO，YOLO v2算法采用多尺度训练方法及借用Faster RCNN锚框(anchor box)思想，在保证检测速度的同时，极大提升模型检测的精度及泛化能力。
该YOLO v2算法应用到本申请实施例的息肉定位任务中，检测的目标为结肠息肉，锚框(anchor box)的尺寸设置根据自有的息肉训练数据聚类得到。算法训练时，运用了迁移学习技术，其中，迁移学习是指将一个领域的已经成熟的知识应用到其他的场景中，用神经网络的词语来表述，就是一层层网络中每个节点的权重从一个训练好的网络迁移到一个全新的网络里，而不是从头开始，不需要每个特定任务训练一个神经网络。使用一个开源的大规模的有标注的数据集训练好的参数初始化，例如该数据集可以是Imagenet数据。Imagenet数据是计算机视觉领域有关图像分类和目标检测的开源数据集。涵盖成千上万中类别，数据量在百万以上。使用经过大量数据集训练的模型初始化参数，能更好的让模型收敛到全局最优 解。
在图像分类模型中，会区分出白光类型图片和NBI类型图片。这两类图像在息肉表观形态上有很大区别，其中，NBI类型图片可观测到血管流向，且血液的颜色在NBI类型图片中表现为黑色，因此需要分别针对白光图片数据和NBI图片数据训练息肉定位模型，简称为白光息肉定位模型和NBI息肉定位模型。此两种息肉定位模型均是采用上述介绍的方法来训练，唯一的区别在于模型的训练数据，白光息肉定位模型训练数据为白光类型图片，NBI息肉定位模型为NBI类型图片。算法流程中，当上一模块判断为白光类型图时，则调用白光息肉定位模型进行息肉定位，反之，则调用NBI息肉定位模型定位息肉。当有定位到息肉时，输出圈出的息肉图像块，以作为息肉性质识别模型的输入。
在上述步骤101之前，本申请实施例提供的结肠息肉图像的处理方法，还可以包括如下步骤100。
100、结肠息肉图像的处理装置从内窥镜视频流中获取待处理的内窥镜图像。
在本申请实施例中，医生操作内窥镜检查结肠时，内窥镜装置可以生成内窥镜视频流，该内窥镜视频流包括一段连续的多个帧的内窥镜图像，内窥镜装置生成内窥镜视频流之后，将内窥镜视频流发送给结肠息肉图像的处理装置，结肠息肉图像的处理装置可以从内窥镜装置接收到内窥镜视频流，从内窥镜视频流中获取单个帧的内窥镜图像，针对每个帧的内窥镜图像都可以按照本申请实施例提供的方法进行息肉位置以及息肉类型识别，从而可以实时识别内窥镜视频流中结肠息肉性质，在医生操作内窥镜检查结肠时，实时定位到视频流中的结肠息肉位置并判别息肉性质，若判别的息肉性质为非腺瘤息肉时，医生可以不用切除此息肉送病理检查。通过本申请实施例针对每个帧的内窥镜图像进行处理，可以辅助医生实时发现息肉，防止息肉漏诊，还可以辅助医生判别息肉性质,提升医生判别息肉准确率。后续步骤中将针对单个帧的内窥镜图像进行图像处理，以输出识 别结果，对于内窥镜视频流中其它帧的内窥镜图像的处理，也可以参阅前述的处理过程，此处仅作说明。
如图2所示，为本申请实施例提供的内窥镜图像的示意图。在生成内窥镜视频流之后，从该内窥镜视频流中提取到一个帧的内窥镜图像，在图2所示的图片中，内窥镜图像为方框内所示的结肠图像，内窥镜图像左侧为内窥镜的参数，可以根据实际场景来设置内窥镜的参数取值。该内窥镜的参数与图像处理无关，因此在采集到内窥镜视频流之后，可以只保留结肠图像区域部分。
在现有技术设计的算法中，需要人为的筛选掉低质量的噪音数据，但是现有技术的算法在实际生产环境下不能使用。人为的筛选掉低质量的噪音数据，导致设计的算法效果在理想环境下不错，但在实际场景中不可用。为解决该问题，在本申请的一些实施例中，步骤100从内窥镜视频流中获取待处理的内窥镜图像之后，本申请实施例提供的方法还包括如下步骤：
结肠息肉图像的处理装置从内窥镜图像上提取颜色特征、梯度变化特征及亮度异常特征；
结肠息肉图像的处理装置根据颜色特征、梯度变化特征及亮度异常特征判断内窥镜图像是否为低质量图片，低质量图片包括：模糊图片、色调异常及过曝欠曝图片、低分辨率图片；
当内窥镜图像不是低质量图片时，触发执行如下步骤101:结肠息肉图像的处理装置使用息肉定位模型对内窥镜图像进行息肉位置检测。
其中，低质量图片也可以称为低质图片。针对输入的视频流中的单个帧的内窥镜图像，判断是否为低质图片，若为低质图片，则直接过滤，不进行后续的模块识别。实际生产环境中，会存在大量的拍摄模糊及因肠道准备不充分导致的粪水图片，影响后续息肉定位及性质识别模块算法效果，基于此，本申请实施例中可以通过提取颜色特征、梯度变化特征及亮度异常特征，基于提取到的这三种特征来检测内窥镜图像是否为低质图 片。
本申请实施例定义的低质图片包含三类：模糊、色调异常及过曝欠曝、图像分辨率低。如图3所示，为本申请实施例提供的内窥镜图像为合格图片的示意图。图3所示的左右两幅图片均为合格图片，合格图片指的是除模糊、色调异常及过曝欠曝、图像分辨率低以外的图片。如图4所示，为本申请实施例提供的内窥镜图像为色调异常及过曝欠曝图片的示意图，在图4中所示的左右两幅图片中均出现颜色异常情况，因此为不合格图片。如图5所示，为本申请实施例提供的内窥镜图像为模糊图片的示意图，在图5所示的左右两幅图片中均出现模糊情况，因此为不合格图片。接下来针对模糊图片、色调异常及过曝欠曝图片、低分辨率图片的具体识别过程分别进行举例说明。
在识别低分辨率的图片时，主要是通过计算图片中有效像素面积来实现，有效的像素面积是指剪裁图片上下左右的黑边之后的面积，如图2中白色方框所框住的面积。剪裁黑边算法主要是通过统计每行或者每列像素值的灰度值分布。满足灰色或者黑色的像素值大于一定比例，则认为此行或者列应该剪除。剪除黑边之后的有效面积若小于一定的阈值则认为低分辨率图片，该阈值可根据实际应用自行设定。
模糊图片的检测算法思路如下：
(1)、对输入图像做一个标准差sigma＝2.5的高斯滤波，消除因图像采样时产生的摩尔纹。
(2)、定义原始图像为R，经过一次像素值为3*3的中值滤波之后得到图像P。
(3)、分别计算图像P与图像R的梯度，采用索贝尔(Sobel)边缘检测算子得到中值滤波图像梯度图G_P和原始图像梯度图G_R，G_P和G_R图像突出了图像边缘细节，对图像边缘进行增强。
(4)、计算G_P和G_R图像的相似度，例如可以采用分类模型的评 估方法F分数(F-Score)类似算法，来进行筛选。其中，越模糊的图像，G_P和G_R相似度越高。
最后可以根据G_P和G_R相似度确定内窥镜图像是否为模糊图片。
在色调异常及过曝欠曝图片的检测算法中，考虑到异常种类非常多，难以穷举。因此会建立色调合格拍摄正常的标准库文件。检测算法思路为：
(1)、将图像分为7*7的图像块，并获取其中9个图像块。
(2)、在HSV(Hue,Saturation,Value)空间下计算每个图像块的H、S、V。
(3)、以H和S为特征与标准图像的HS分别进行匹配，设定相似度阈值t，计算图像每个图像块是否与标准库相似。
(4)、将9份图像块的匹配度相似结果进行累积计数，匹配度大于阈值t的加1，当累积值>5时，则认为该图像为目标色调匹配图像，并返回检测结果为真(True)。
对于满足上述目标色调匹配结果的内窥镜图像可以确定为色调异常及过曝欠曝图片。
在本申请的一些实施例中，内窥镜视频流可以采用多种拍摄方式来生成，因此内窥镜视频流中的内窥镜图像可以根据拍摄方式的不同具有多种图片类型，在针对不同的图片类型进行息肉位置检测时需要采用不同的息肉定位模型，详见后续实施例中的说明。
步骤100从内窥镜视频流中获取待处理的内窥镜图像之后，本申请实施例提供的方法还包括如下步骤：
结肠息肉图像的处理装置对内窥镜图像进行图片类型识别，确定内窥镜图像为白光类型图片，或者内镜窄带成像(Narrow Band Imaging，NBI)类型图片。
其中，根据内窥镜视频流采用的不同拍摄方式，从该视频流中提取出 的内窥镜图像也可以有不同的图片类型，例如该内窥镜图像可以是白光类型图片，或者该内窥镜图像可以是NBI类型图片。如图6-a所示，为本申请实施例提供的内窥镜图像为白光类型图片的示意图，白光类型图片指的是采用普通光源成像的红(Red，R)、绿(Green，G)、蓝(Blue，B)(简称RGB)图像，如图6-b所示，为本申请实施例提供的内窥镜图像为NBI类型图片的示意图，NBI类型图片是利用滤光器过滤掉内镜光源所发出的红蓝绿光波中的宽带光谱，仅留下窄带光谱用于诊断消化道各种疾病。
进一步的，在本申请的一些实施例中，对内窥镜图像进行图片类型识别，确定内窥镜图像为白光类型图片，或者NBI类型图片，包括：
使用神经网络算法，通过白光类型图片训练数据和NBI类型图片训练数据对原始的图像分类模型进行分类训练，得到训练完成的图像分类模型；
使用训练完成的图像分类模型从内窥镜图像上提取到血管颜色特征；
使用训练完成的图像分类模型对血管颜色特征的取值进行分类，得到内窥镜图像为白光类型图片，或者NBI类型图片。
在本申请实施例中，首先针对白光类型和NBI类型分别预先获取训练数据，即获取到白光类型图片训练数据和NBI类型图片训练数据，使用神经网络算法预先训练出图像分类模型，该图像分类模型可以采用多种机器学习算法完成训练，例如该图像分类模型具体可以是深度神经网络模型(Deep Neural Networks，DNN)、循环神经网络模型等，例如该深度神经网络模型可以是稠密连接卷积网络(Densely Connected Convolutional Networks，DenseNet)等。预先收集到白光类型图片训练数据和NBI类型图片训练数据之后，然后通过白光类型图片训练数据和NBI类型图片训练数据进行模型训练后，输出训练完成的图像分类模型。
在图像分类模型的训练完成之后，使用训练完成的图像分类模型从内窥镜图像上提取到血管颜色特征，血管颜色特征是该内窥镜图像的分类依 据，最后使用训练完成的图像分类模型对血管颜色特征的取值进行分类，得到内窥镜图像为白光类型图片，或者NBI类型图片。
在本申请实施例中，图像分类模型输入的为合格的单个帧的内窥镜图像，该图像分类模型的输出为该内窥镜图像是白光类型图片还是NBI类型的图片。医生在实际操作内窥镜检查时，当发现疑似息肉时，往往会采用NBI模式来诊断当前息肉的病理分型。NBI模式的图片可以更清晰的显示出血管走向。如图6-a所示为白光类型图片，如图6-b所示为NBI类型图片。举例说明，本申请实施例图像分类模型可以采用稠密连接卷积网络(DenseNet)对图片类型进行分类识别，当然，本申请实施例中亦可采用其他图片分类网络实现类似功能，但是识别效果上会有一定程度的差异，此处不做限定。
图像分类模型的执行可以转化为一个图像分类问题。所使用的图像分类算法为DenseNet卷积网络。网络的输入图像大小为224*224，故首先对输入的原始图片进行放缩，缩放到一个固定的224*224大小。考虑到此图像分类模型的任务，更偏好于较低级的特征组合，如血管颜色等，故在设计DenseNet结构深度和宽度组合时，采用更宽更浅的模式。最终所使用的网络结构为DenseNet-40，其中，40指的是网络的层数，通过网络参数调优，增长率(growth-rate)设置为48，特征经过传输层(transition layer)压缩比为0.5，效果达到最优。模型结构如下表1所示：
Figure PCTCN2019112788-appb-000001
Figure PCTCN2019112788-appb-000002
需要说明的是，在上述表1所示的实施例中，DenseNet-40中各个层的功能实现以及执行过程可以根据场景来确定。另外，网络层中的conv包含三个操作：批量归一化层(batchnorm)、激活层(ReLU)、卷积层。
102、结肠息肉图像的处理装置使用息肉性质识别模型在息肉图像块上进行息肉类型的分类检测，输出识别结果。
在本申请实施例中，从内窥镜图像上圈出息肉图像块之后，接下来只 需要在该息肉图像块上，使用预先训练好的息肉性质识别模型进行息肉类型的分类检测，输出识别结果，该识别结果可以输出最大概率的息肉类型，该识别结果也可以输出各个置信度条件下的息肉类型，其中，置信度是在息肉性质识别模型的预测后，判断在该息肉图像块中包含各种息肉类型的可信程度。
在本申请实施例中，息肉性质识别模型可以执行息肉性质判别任务，例如通过一个图像分类任务来实现，输入是息肉定位模型输出的定位框的图片数据。如图7所示，在内窥镜图像上圈出的息肉图像块为息肉定位模型检测到的息肉，作为息肉性质识别模型的输入数据。模块输出可以是四个类别值(0,1,2,3)，其中0表示此区域没有息肉是正常的，1表示非腺瘤息肉，2表示腺瘤性息肉，3表示腺癌，另外，针对正常、非腺瘤、腺瘤、腺癌还可以分别设置一个置信度条件，若输出为0，则纠正息肉定位模型的判断结果，此区域并无息肉，为正常区域。
在本申请的一些实施例中，步骤102结肠息肉图像的处理装置使用息肉性质识别模型在息肉图像块上进行息肉类型的分类检测，输出识别结果包括：
使用神经网络算法，通过不同息肉类型的息肉图片训练数据对原始的息肉性质识别模型进行息肉类型的分类检测训练，得到训练完成的息肉性质识别模型；
使用训练完成的息肉性质识别模型从息肉图像块上提取到息肉类型特征；
使用训练完成的息肉性质识别模型对息肉类型特征的取值进行分类，输出识别结果。
在本申请实施例中，首先获取到不同息肉类型的息肉图片训练数据，使用神经网络算法预先训练出息肉性质识别模型，该息肉性质识别模型可以采用多种机器学习算法完成训练，例如该息肉性质识别模型具体可以是 深度神经网络模型、循环神经网络模型等，例如该深度神经网络模型可以是DenseNet等。预先收集到不同息肉类型的息肉图片训练数据之后，然后通过不同息肉类型的息肉图片训练数据进行模型训练后，输出训练完成的息肉性质识别模型。
在息肉性质识别模型的训练完成之后，使用训练完成的息肉性质识别模型从息肉图像块上提取到息肉类型特征，息肉类型特征是息肉图像块的分类依据，最后使用训练完成的息肉性质识别模型对息肉类型特征的取值进行分类，得到识别结果。
在本申请的一些实施例中，步骤102结肠息肉图像的处理装置从内窥镜图像上定位出息肉图像块之后，本申请实施例提供的方法还包括：
结肠息肉图像的处理装置按照预设的图像外扩比例，对息肉图像块在内窥镜图像上所占的息肉区域进行上下左右方向上的外扩，得到外扩后的息肉图像块；
结肠息肉图像的处理装置将外扩后的息肉图像块输入到息肉性质识别模型中。
在本申请实施例中，息肉性质识别模型的息肉性质分类任务，可以采用DenseNet卷积网络算法来实现，因算法要求输入的图片大小必须一致，而息肉定位模块输出的息肉位置大小不一。在构造算法输入数据时，本申请实施例采用的方法是：对息肉定位模型输出的息肉图像块，向上下左右外扩10％比例区域，以保证框住的区域有一定的上下文语义信息，辅助后续的息肉性质识别模型提取特征，外扩后的区域直接归一化到模型要求的输入大小224*224。考虑到任务的复杂性，可以采用更深的DenseNet网络。最终所使用的网络结构为DenseNet-121。通过网络参数调优，growth-rate设置为24，特征经过transition layer压缩比为0.5，效果达到最优。模型结构如下表2所示：
Figure PCTCN2019112788-appb-000003
最终，本申请实施例提供的息肉图像的处理方法，处理每一帧的内窥镜图像需要大约100毫秒(ms)左右，满足实时性要求，算法效果和不同水平的医生做对比，和顶级医生水平相当。部署到基层医院，可实时辅助 医生发现息肉及识别息肉。
在本申请实施例中，在医生做内窥镜检查时可实时帮助医生发现息肉并判断息肉的性质。防止医生漏诊息肉，同时帮助医生提升息肉性质判别准确率。若判断为高置信度的非腺瘤性息肉，医生可不用切除送病理，这样可减少医生的操作时间，进一步减少患者高并发风险及患者诊疗费用，减轻内镜医生负担及病理科医生负担。
通过以上实施例对本申请实施例的描述可知，首先使用息肉定位模型对内窥镜图像进行息肉位置检测，从内窥镜图像上定位出息肉图像块，息肉图像块包括：息肉在内窥镜图像中的位置区域。最后使用息肉性质识别模型在息肉图像块上进行息肉类型的分类检测，输出识别结果。本申请实施例中由于采用的是息肉定位模型来检测息肉位置，可以直接从内窥镜图像中定位出息肉图像块，针对息肉类型的分类检测也是在该息肉图像块上进行，而不需要在整个内窥镜图像上进行，因此实时性符合要求，在内窥镜被控制移动时，对于实时采集到的图像可以实时的输出识别结果，提高息肉图像的处理效率。
需要说明的是，对于前述的各方法实施例，为了简单描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本申请实施例并不受所描述的动作顺序的限制，因为依据本申请实施例，某些步骤可以采用其他顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一定是本申请实施例所必须的。
为便于更好的实施本申请实施例的上述方案，下面还提供用于实施上述方案的相关装置。
请参阅图8-a所示，本申请实施例提供的一种结肠息肉图像的处理装置800，可以包括一个或多个处理器，以及一个或多个存储程序单元的存储器，其中，程序单元由处理器执行，程序单元包括：位置检测模块801、息肉分类模块802，其中，
位置检测模块801，被设置为使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测，从所述内窥镜图像上定位出息肉图像块，所述息肉图像块包括：息肉在所述内窥镜图像中的位置区域；
息肉分类模块802，被设置为使用息肉性质识别模型在所述息肉图像块上进行息肉类型的分类检测，输出识别结果。
在本申请的一些实施例中，如图8-a所示，结肠息肉图像的处理装置800，还可以包括：图像获取模块803，
所述图像获取模块803，被设置为从内窥镜视频流中获取待处理的内窥镜图像。
在本申请的一些实施例中，请参阅图8-b所示，所述结肠息肉图像的处理装置800，还包括：
低质量图片识别模块804，被设置为所述位置检测模块801使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测之前，从所述内窥镜图像上提取颜色特征、梯度变化特征及亮度异常特征；根据所述颜色特征、所述梯度变化特征及所述亮度异常特征判断所述内窥镜图像是否为低质量图片，所述低质量图片包括：模糊图片、色调异常及过曝欠曝图片、低分辨率图片；当所述内窥镜图像不是所述低质量图片时，触发执行所述位置检测模块。
在本申请的一些实施例中，请参阅图8-c所示，所述结肠息肉图像的处理装置800，还包括：
图片类型识别模块805，被设置为所述位置检测模块801使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测之前，对所述内窥镜图像进行图片类型识别，确定所述内窥镜图像为白光类型图片，或者内镜窄带成像NBI类型图片。
在本申请的一些实施例中，请参阅图8-d所示，所述图片类型识别模块805，包括：
图像分类模型训练单元8051，被设置为使用神经网络算法，通过白光类型图片训练数据和NBI类型图片训练数据对原始的图像分类模型进行分类训练，得到训练完成的图像分类模型；
血管颜色特征提取单元8052，被设置为使用所述训练完成的图像分类模型从所述内窥镜图像上提取到血管颜色特征；
图片分类单元8053，被设置为使用所述训练完成的图像分类模型对所述血管颜色特征的取值进行分类，得到所述内窥镜图像为所述白光类型图片，或者所述NBI类型图片。
在本申请的一些实施例中，所述息肉定位模型，包括：白光息肉定位模型和NBI息肉定位模型；
其中，所述白光息肉定位模型，通过如下方式得到：使用神经网络算法，通过所述白光类型图片训练数据对原始的息肉定位模型进行息肉位置训练；
所述NBI息肉定位模型，通过如下方式得到：使用神经网络算法，通过所述NBI类型图片训练数据对原始的息肉定位模型进行息肉位置训练。
在本申请的一些实施例中，所述位置检测模块801，具体被设置为当所述内窥镜图像为所述白光类型图片时，使用所述白光息肉定位模型进行息肉定位，从所述内窥镜图像上定位出白光息肉图像块；当所述内窥镜图像为所述NBI类型图片时，使用所述NBI息肉定位模型进行息肉定位，从所述内窥镜图像上定位出NBI息肉图像块。
在本申请的一些实施例中，请参阅图8-e所示，所述息肉分类模块802，包括：
息肉性质识别模型训练单元8021，被设置为使用神经网络算法，通过不同息肉类型的息肉图片训练数据对原始的息肉性质识别模型进行息肉类型的分类检测训练，得到训练完成的息肉性质识别模型；
息肉类型特征提取单元8022，被设置为使用所述训练完成的息肉性质识别模型从所述息肉图像块上提取到息肉类型特征；
息肉分类单元8023，被设置为使用所述训练完成的息肉性质识别模型对所述息肉类型特征的取值进行分类，输出识别结果。
通过以上实施例对本申请实施例的描述可知，首先使用息肉定位模型对内窥镜图像进行息肉位置检测，从内窥镜图像上定位出息肉图像块，息肉图像块包括：息肉在内窥镜图像中的位置区域。最后使用息肉性质识别模型在息肉图像块上进行息肉类型的分类检测，输出识别结果。本申请实施例中由于采用的是息肉定位模型来检测息肉位置，可以直接从内窥镜图像中定位出息肉图像块，针对息肉类型的分类检测也是在该息肉图像块上进行，而不需要在整个内窥镜图像上进行，因此实时性符合要求，在内窥镜被控制移动时，对于实时采集到的图像可以实时的输出识别结果，提高息肉图像的处理效率。
本申请实施例还提供了另一种终端，如图9所示，为了便于说明，仅示出了与本申请实施例相关的部分，具体技术细节未揭示的，请参照本申请实施例方法部分。该终端可以为包括手机、平板电脑、个人数字助理(Personal Digital Assistant，PDA)、销售终端(Point of Sales，POS)、车载电脑等任意终端设备，以终端为手机为例：
图9示出的是与本申请实施例提供的终端相关的手机的部分结构的框图。参考图9，手机包括：射频(Radio Frequency，RF)电路1010、存储器1020、输入单元1030、显示单元1040、传感器1050、音频电路1060、无线保真(wireless fidelity，WiFi)模块1070、处理器1080、以及电源1090等部件。本领域技术人员可以理解，图9中示出的手机结构并不构成对手机的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。
下面结合图9对手机的各个构成部件进行具体的介绍：
RF电路1010可被设置为收发信息或通话过程中，信号的接收和发送，特别地，将基站的下行信息接收后，给处理器1080处理；另外，将设计上行的数据发送给基站。通常，RF电路1010包括但不限于天线、至少一个放大器、收发信机、耦合器、低噪声放大器(Low Noise Amplifier，LNA)、双工器等。此外，RF电路1010还可以通过无线通信与网络和其他设备通信。上述无线通信可以使用任一通信标准或协议，包括但不限于全球移动通讯系统(Global System of Mobile communication，GSM)、通用分组无线服务(General Packet Radio Service，GPRS)、码分多址(Code Division Multiple Access，CDMA)、宽带码分多址(Wideband Code Division Multiple Access,WCDMA)、长期演进(Long Term Evolution，LTE)、电子邮件、短消息服务(Short Messaging Service，SMS)等。
存储器1020可被设置为存储软件程序以及模块，处理器1080通过运行存储在存储器1020的软件程序以及模块，从而执行手机的各种功能应用以及数据处理。存储器1020可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序(比如声音播放功能、图像播放功能等)等；存储数据区可存储根据手机的使用所创建的数据(比如音频数据、电话本等)等。此外，存储器1020可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他易失性固态存储器件。
输入单元1030可被设置为接收输入的数字或字符信息，以及产生与手机的用户设置以及功能控制有关的键信号输入。具体地，输入单元1030可包括触控面板1031以及其他输入设备1032。触控面板1031，也称为触摸屏，可收集用户在其上或附近的触摸操作(比如用户使用手指、触笔等任何适合的物体或附件在触控面板1031上或在触控面板1031附近的操作)，并根据预先设定的程式驱动相应的连接装置。可选的，触控面板1031可包括触摸检测装置和触摸控制器两个部分。其中，触摸检测装置检测用户的触摸方位，并检测触摸操作带来的信号，将信号传送给触摸控制器；触摸控制器从触摸检测装置上接收触摸信息，并将它转换成触点坐标，再 送给处理器1080，并能接收处理器1080发来的命令并加以执行。此外，可以采用电阻式、电容式、红外线以及表面声波等多种类型实现触控面板1031。除了触控面板1031，输入单元1030还可以包括其他输入设备1032。具体地，其他输入设备1032可以包括但不限于物理键盘、功能键(比如音量控制按键、开关按键等)、轨迹球、鼠标、操作杆等中的一种或多种。
显示单元1040可被设置为显示由用户输入的信息或提供给用户的信息以及手机的各种菜单。显示单元1040可包括显示面板1041，可选的，可以采用液晶显示器(Liquid Crystal Display，LCD)、有机发光二极管(Organic Light-Emitting Diode,OLED)等形式来配置显示面板1041。进一步的，触控面板1031可覆盖显示面板1041，当触控面板1031检测到在其上或附近的触摸操作后，传送给处理器1080以确定触摸事件的类型，随后处理器1080根据触摸事件的类型在显示面板1041上提供相应的视觉输出。虽然在图9中，触控面板1031与显示面板1041是作为两个独立的部件来实现手机的输入和输入功能，但是在某些实施例中，可以将触控面板1031与显示面板1041集成而实现手机的输入和输出功能。
手机还可包括至少一种传感器1050，比如光传感器、运动传感器以及其他传感器。具体地，光传感器可包括环境光传感器及接近传感器，其中，环境光传感器可根据环境光线的明暗来调节显示面板1041的亮度，接近传感器可在手机移动到耳边时，关闭显示面板1041和/或背光。作为运动传感器的一种，加速计传感器可检测各个方向上(一般为三轴)加速度的大小，静止时可检测出重力的大小及方向，可用于识别手机姿态的应用(比如横竖屏切换、相关游戏、磁力计姿态校准)、振动识别相关功能(比如计步器、敲击)等；至于手机还可配置的陀螺仪、气压计、湿度计、温度计、红外线传感器等其他传感器，在此不再赘述。
音频电路1060、扬声器1061，传声器1062可提供用户与手机之间的音频接口。音频电路1060可将接收到的音频数据转换后的电信号，传输到扬声器1061，由扬声器1061转换为声音信号输出；另一方面，传声器 1062将收集的声音信号转换为电信号，由音频电路1060接收后转换为音频数据，再将音频数据输出处理器1080处理后，经RF电路1010以发送给比如另一手机，或者将音频数据输出至存储器1020以便进一步处理。
WiFi属于短距离无线传输技术，手机通过WiFi模块1070可以帮助用户收发电子邮件、浏览网页和访问流式媒体等，它为用户提供了无线的宽带互联网访问。虽然图9示出了WiFi模块1070，但是可以理解的是，其并不属于手机的必须构成，完全可以根据需要在不改变发明的本质的范围内而省略。
处理器1080是手机的控制中心，利用各种接口和线路连接整个手机的各个部分，通过运行或执行存储在存储器1020内的软件程序和/或模块，以及调用存储在存储器1020内的数据，执行手机的各种功能和处理数据，从而对手机进行整体监控。可选的，处理器1080可包括一个或多个处理单元；优选的，处理器1080可集成应用处理器和调制解调处理器，其中，应用处理器主要处理操作系统、用户界面和应用程序等，调制解调处理器主要处理无线通信。可以理解的是，上述调制解调处理器也可以不集成到处理器1080中。
手机还包括给各个部件供电的电源1090(比如电池)，优选的，电源可以通过电源管理系统与处理器1080逻辑相连，从而通过电源管理系统实现管理充电、放电、以及功耗管理等功能。
尽管未示出，手机还可以包括摄像头、蓝牙模块等，在此不再赘述。
在本申请实施例中，该终端所包括的处理器1080还具有控制执行以上由终端执行的结肠息肉图像的处理方法流程。
图10是本申请实施例提供的一种服务器结构示意图，该服务器1100可因配置或性能不同而产生比较大的差异，可以包括一个或一个以上中央处理器(central processing units，CPU)1122(例如，一个或一个以上处理器)和存储器1132，一个或一个以上存储应用程序1142或数据1144的 存储介质1130(例如一个或一个以上海量存储设备)。其中，存储器1132和存储介质1130可以是短暂存储或持久存储。存储在存储介质1130的程序可以包括一个或一个以上模块(图示没标出)，每个模块可以包括对服务器中的一系列指令操作。更进一步地，中央处理器1122可以设置为与存储介质1130通信，在服务器1100上执行存储介质1130中的一系列指令操作。
服务器1100还可以包括一个或一个以上电源1126，一个或一个以上有线或无线网络接口1150，一个或一个以上输入输出接口1158，和/或，一个或一个以上操作系统1141，例如Windows ServerTM，Mac OS XTM，UnixTM,LinuxTM，FreeBSDTM等等。
上述实施例中由服务器所执行的结肠息肉图像的处理方法步骤可以基于该图10所示的服务器结构。
另外需说明的是，以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。另外，本申请提供的装置实施例附图中，模块之间的连接关系表示它们之间具有通信连接，具体可以实现为一条或多条通信总线或信号线。本领域普通技术人员在不付出创造性劳动的情况下，即可以理解并实施。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到本申请实施例可借助软件加必需的通用硬件的方式来实现，当然也可以通过专用硬件包括专用集成电路、专用CPU、专用存储器、专用元器件等来实现。一般情况下，凡由计算机程序完成的功能都可以很容易地用相应的硬件来实现，而且，用来实现同一功能的具体硬件结构也可以是多种多样的，例如模拟电路、数字电路或专用电路等。但是，对本申请实施例而言更多情况下软件程序实现是更佳的实施方式。基于这样的理解，本申请实 施例的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在可读取的存储介质中，如计算机的软盘、U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述的方法。
综上所述，以上实施例仅用以说明本申请实施例的技术方案，而非对其限制；尽管参照上述实施例对本申请实施例进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对上述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。
工业实用性
在本申请实施例中，首先使用息肉定位模型对内窥镜图像进行息肉位置检测，从内窥镜图像上定位出息肉图像块，息肉图像块包括：息肉在内窥镜图像中的位置区域。最后使用息肉性质识别模型在息肉图像块上进行息肉类型的分类检测，输出识别结果。本申请实施例中由于采用的是息肉定位模型来检测息肉位置，可以直接从内窥镜图像中定位出息肉图像块，针对息肉类型的分类检测也是在该息肉图像块上进行，而不需要在整个内窥镜图像上进行，因此实时性符合要求，在内窥镜被控制移动时，对于实时采集到的图像可以实时的输出识别结果，提高息肉图像的处理效率、

Claims (23)

  1. 一种结肠息肉图像的处理方法，包括：
    结肠息肉图像的处理装置使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测，从所述内窥镜图像上定位出息肉图像块，所述息肉图像块包括：息肉在所述内窥镜图像中的位置区域；
    所述结肠息肉图像的处理装置使用息肉性质识别模型在所述息肉图像块上进行息肉类型的分类检测，输出识别结果。
  2. 根据权利要求1所述的方法，其中，所述结肠息肉图像的处理装置使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测之前，所述方法还包括：
    所述结肠息肉图像的处理装置从所述内窥镜图像上提取颜色特征、梯度变化特征及亮度异常特征；
    所述结肠息肉图像的处理装置根据所述颜色特征、所述梯度变化特征及所述亮度异常特征判断所述内窥镜图像是否为低质量图片，所述低质量图片包括：模糊图片、色调异常及过曝欠曝图片、低分辨率图片；
    当所述内窥镜图像不是所述低质量图片时，触发执行如下步骤:所述结肠息肉图像的处理装置使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测。
  3. 根据权利要求1所述的方法，其中，所述结肠息肉图像的处理装置使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测之前，所述方法还包括：
    所述结肠息肉图像的处理装置对所述内窥镜图像进行图片类型识别，确定所述内窥镜图像为白光类型图片，或者内镜窄带成像NBI类型图片。
  4. 根据权利要求3所述的方法，其中，所述结肠息肉图像的处理装置对所述内窥镜图像进行图片类型识别，确定所述内窥镜图像为白光类型图片，或者内镜窄带成像NBI类型图片，包括：
    使用神经网络算法，通过白光类型图片训练数据和NBI类型图片训练数据对原始的图像分类模型进行分类训练，得到训练完成的图像分类模 型；
    使用所述训练完成的图像分类模型从所述内窥镜图像上提取到血管颜色特征；
    使用所述训练完成的图像分类模型对所述血管颜色特征的取值进行分类，得到所述内窥镜图像为所述白光类型图片，或者所述NBI类型图片。
  5. 根据权利要求3或4所述的方法，其中，所述息肉定位模型，包括：白光息肉定位模型和NBI息肉定位模型；
    其中，所述白光息肉定位模型，通过如下方式得到：所述结肠息肉图像的处理装置使用神经网络算法，通过所述白光类型图片训练数据对原始的息肉定位模型进行息肉位置训练；
    所述NBI息肉定位模型，通过如下方式得到：所述结肠息肉图像的处理装置使用神经网络算法，通过所述NBI类型图片训练数据对原始的息肉定位模型进行息肉位置训练。
  6. 根据权利要求5所述的方法，其中，所述结肠息肉图像的处理装置使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测，从所述内窥镜图像上定位出息肉图像块，包括：
    当所述内窥镜图像为所述白光类型图片时，使用所述白光息肉定位模型进行息肉定位，从所述内窥镜图像上定位出白光息肉图像块；
    当所述内窥镜图像为所述NBI类型图片时，使用所述NBI息肉定位模型进行息肉定位，从所述内窥镜图像上定位出NBI息肉图像块。
  7. 根据权利要求1所述的方法，其中，所述结肠息肉图像的处理装置使用息肉性质识别模型在所述息肉图像块上进行息肉类型的分类检测，输出识别结果包括：
    使用神经网络算法，通过不同息肉类型的息肉图片训练数据对原始的息肉性质识别模型进行息肉类型的分类检测训练，得到训练完成的息肉性质识别模型；
    使用所述训练完成的息肉性质识别模型从所述息肉图像块上提取到息肉类型特征；
    使用所述训练完成的息肉性质识别模型对所述息肉类型特征的取值进行分类，输出识别结果。
  8. 根据权利要求1至4中任一项，或权利要求7所述的方法，其中，所述从所述内窥镜图像上定位出息肉图像块之后，所述方法还包括：
    所述结肠息肉图像的处理装置按照预设的图像外扩比例，对所述息肉图像块在所述内窥镜图像上所占的息肉区域进行上下左右方向上的外扩，得到外扩后的息肉图像块；
    所述结肠息肉图像的处理装置将所述外扩后的息肉图像块输入到所述息肉性质识别模型中。
  9. 一种结肠息肉图像的处理装置，包括一个或多个处理器，以及一个或多个存储程序单元的存储器，其中，所述程序单元由所述处理器执行，所述程序单元包括：
    位置检测模块，被设置为使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测，从所述内窥镜图像上定位出息肉图像块，所述息肉图像块包括：息肉在所述内窥镜图像中的位置区域；
    息肉分类模块，被设置为使用息肉性质识别模型在所述息肉图像块上进行息肉类型的分类检测，输出识别结果。
  10. 根据权利要求9所述的装置，其中，所述程序单元包括，还包括：
    低质量图片识别模块，被设置为所述位置检测模块使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测之前，从所述内窥镜图像上提取颜色特征、梯度变化特征及亮度异常特征；根据所述颜色特征、所述梯度变化特征及所述亮度异常特征判断所述内窥镜图像是否为低质量图片，所述低质量图片包括：模糊图片、色调异常及过曝欠曝图片、低分辨率图片；当所述内窥镜图像不是所述低质量图片时，触发执行所述位置检测模块。
  11. 根据权利要求9所述的装置，其中，所述程序单元包括，还包括：
    图片类型识别模块，被设置为所述位置检测模块使用息肉定位模型对待处理的内窥镜图像进行息肉位置检测之前，对所述内窥镜图像进行图片类型识别，确定所述内窥镜图像为白光类型图片，或者内镜窄带成像NBI类型图片。
  12. 根据权利要求11所述的装置，其中，所述图片类型识别模块，包括：
    图像分类模型训练单元，被设置为使用神经网络算法，通过白光类型 图片训练数据和NBI类型图片训练数据对原始的图像分类模型进行分类训练，得到训练完成的图像分类模型；
    血管颜色特征提取单元，被设置为使用所述训练完成的图像分类模型从所述内窥镜图像上提取到血管颜色特征；
    图片分类单元，被设置为使用所述训练完成的图像分类模型对所述血管颜色特征的取值进行分类，得到所述内窥镜图像为所述白光类型图片，或者所述NBI类型图片。
  13. 根据权利要求11或12所述的装置，其中，所述息肉定位模型，包括：白光息肉定位模型和NBI息肉定位模型；
    其中，所述白光息肉定位模型，通过如下方式得到：使用神经网络算法，通过所述白光类型图片训练数据对原始的息肉定位模型进行息肉位置训练；
    所述NBI息肉定位模型，通过如下方式得到：使用神经网络算法，通过所述NBI类型图片训练数据对原始的息肉定位模型进行息肉位置训练。
  14. 根据权利要求13所述的装置，其中，所述位置检测模块，具体被设置为当所述内窥镜图像为所述白光类型图片时，使用所述白光息肉定位模型进行息肉定位，从所述内窥镜图像上定位出白光息肉图像块；当所述内窥镜图像为所述NBI类型图片时，使用所述NBI息肉定位模型进行息肉定位，从所述内窥镜图像上定位出NBI息肉图像块。
  15. 一种医疗系统，所述医疗系统包括：内窥镜装置和结肠息肉图像的处理装置，所述内窥镜装置和所述结肠息肉图像的处理装置之间建立有通信连接；其中，
    所述内窥镜装置，被设置为生成内窥镜视频流；并将所述内窥镜视频流发送给所述结肠息肉图像的处理装置；
    所述结肠息肉图像的处理装置，被设置为从所述内窥镜装置接收到所述内窥镜视频流；从所述内窥镜视频流中获取待处理的内窥镜图像；并执行前述如权利要求1至8中任意一项所述的方法。
  16. 一种图像的处理方法，包括：
    图像的处理装置使用目标对象定位模型对待处理的图像进行目标对象位置检测，从所述图像上定位出目标对象图像块，所述目标对象图像块 包括：所述目标对象在所述图像中的位置区域；
    所述图像的处理装置使用目标对象性质识别模型在所述目标对象图像块上进行目标对象类型的分类检测，输出识别结果。
  17. 根据权利要求16所述的方法，其中，所述图像的处理装置使用目标对象定位模型对待处理的图像进行目标对象位置检测之前，所述方法还包括：
    所述图像的处理装置从所述图像上提取颜色特征、梯度变化特征及亮度异常特征；
    所述图像的处理装置根据所述颜色特征、所述梯度变化特征及所述亮度异常特征判断所述图像是否为低质量图片，所述低质量图片包括：模糊图片、色调异常及过曝欠曝图片、低分辨率图片；
    当所述图像不是所述低质量图片时，触发执行如下步骤:所述图像的处理装置使用目标对象定位模型对待处理的图像进行目标对象位置检测。
  18. 根据权利要求16所述的方法，其中，所述图像的处理装置使用目标对象定位模型对待处理的图像进行目标对象位置检测之前，所述方法还包括：
    所述图像的处理装置对所述图像进行图片类型识别，确定所述图像为白光类型图片，或者内镜窄带成像NBI类型图片。
  19. 根据权利要求18所述的方法，其中，所述图像的处理装置对所述图像进行图片类型识别，确定所述图像为白光类型图片，或者内镜窄带成像NBI类型图片，包括：
    使用神经网络算法，通过白光类型图片训练数据和NBI类型图片训练数据对原始的图像分类模型进行分类训练，得到训练完成的图像分类模型；
    使用所述训练完成的图像分类模型从所述图像上提取到指定对象颜色特征；
    使用所述训练完成的图像分类模型对所述指定对象颜色特征的取值进行分类，得到所述图像为所述白光类型图片，或者所述NBI类型图片。
  20. 根据权利要求18或19所述的方法，其中，所述目标对象定位模型，包括：白光目标对象定位模型和NBI目标对象定位模型；
    其中，所述白光目标对象定位模型，通过如下方式得到：所述图像的处理装置使用神经网络算法，通过所述白光类型图片训练数据对原始的目标对象定位模型进行目标对象位置训练；
    所述NBI目标对象定位模型，通过如下方式得到：所述图像的处理装置使用神经网络算法，通过所述NBI类型图片训练数据对原始的目标对象定位模型进行目标对象位置训练。
  21. 根据权利要求20所述的方法，其中，所述图像的处理装置使用目标对象定位模型对待处理的图像进行目标对象位置检测，从所述内窥镜图像上定位出目标对象图像块，包括：
    当所述图像为所述白光类型图片时，使用所述白光目标对象定位模型进行目标对象定位，从所述图像上定位出白光目标对象图像块；
    当所述图像为所述NBI类型图片时，使用所述NBI目标对象定位模型进行目标对象定位，从所述图像上定位出NBI目标对象图像块。
  22. 根据权利要求16所述的方法，其中，所述图像的处理装置使用目标对象性质识别模型在所述目标对象图像块上进行目标对象类型的分类检测，输出识别结果包括：
    使用神经网络算法，通过不同目标对象类型的目标对象图片训练数据对原始的目标对象性质识别模型进行目标对象类型的分类检测训练，得到训练完成的目标对象性质识别模型；
    使用所述训练完成的目标对象性质识别模型从所述目标对象图像块上提取到目标对象类型特征；
    使用所述训练完成的目标对象性质识别模型对所述目标对象类型特征的取值进行分类，输出识别结果。
  23. 根据权利要求16至19中任一项，或权利要求22所述的方法，其中，所述从所述图像上定位出目标对象图像块之后，所述方法还包括：
    所述图像的处理装置按照预设的图像外扩比例，对所述目标对象图像块在所述图像上所占的目标对象区域进行上下左右方向上的外扩，得到外扩后的目标对象图像块；
    所述图像的处理装置将所述外扩后的目标对象图像块输入到所述目标对象性质识别模型中。
PCT/CN2019/112788 2018-10-31 2019-10-23 一种结肠息肉图像的处理方法和装置及系统 WO2020088328A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP19878480.3A EP3876188B1 (en) 2018-10-31 2019-10-23 Colon polyp image processing method and apparatus, and system
JP2021512995A JP7090836B2 (ja) 2018-10-31 2019-10-23 結腸ポリープ画像の処理方法並びに、その装置、コンピュータプログラム及びシステム
US17/025,679 US11468563B2 (en) 2018-10-31 2020-09-18 Colon polyp image processing method and apparatus, and system
US17/885,361 US11748883B2 (en) 2018-10-31 2022-08-10 Colon polyp image processing method and apparatus, and system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811287489.XA CN109447973B (zh) 2018-10-31 2018-10-31 一种结肠息肉图像的处理方法和装置及系统
CN201811287489.X 2018-10-31

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/025,679 Continuation US11468563B2 (en) 2018-10-31 2020-09-18 Colon polyp image processing method and apparatus, and system

Publications (1)

Publication Number Publication Date
WO2020088328A1 true WO2020088328A1 (zh) 2020-05-07

Family

ID=65549487

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/112788 WO2020088328A1 (zh) 2018-10-31 2019-10-23 一种结肠息肉图像的处理方法和装置及系统

Country Status (5)

Country Link
US (2) US11468563B2 (zh)
EP (1) EP3876188B1 (zh)
JP (1) JP7090836B2 (zh)
CN (1) CN109447973B (zh)
WO (1) WO2020088328A1 (zh)

Families Citing this family (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109447973B (zh) * 2018-10-31 2021-11-26 腾讯医疗健康(深圳)有限公司 一种结肠息肉图像的处理方法和装置及系统
KR102236826B1 (ko) * 2018-11-30 2021-04-06 아주대학교산학협력단 기계학습용 의료 영상 데이터셋의 품질을 평가하는 방법 및 시스템
CN109871823B (zh) * 2019-03-11 2021-08-31 中国电子科技集团公司第五十四研究所 一种结合旋转框和上下文信息的卫星图像舰船检测方法
US11944262B2 (en) 2019-03-27 2024-04-02 Hoya Corporation Endoscope processor, information processing device, and endoscope system
CN110021020B (zh) * 2019-04-18 2022-03-15 重庆金山医疗技术研究院有限公司 一种图像检测方法、装置及内窥镜系统
CN111839429A (zh) * 2019-04-25 2020-10-30 天津御锦人工智能医疗科技有限公司 一种基于图像识别判断结肠镜检查是否完整的检测方法
CN111839445A (zh) * 2019-04-25 2020-10-30 天津御锦人工智能医疗科技有限公司 一种基于图像识别的结肠镜手术中窄带成像检测方法
CN111839428A (zh) * 2019-04-25 2020-10-30 天津御锦人工智能医疗科技有限公司 一种基于深度学习提高结肠镜腺瘤性息肉检出率的方法
CN110070938A (zh) * 2019-04-26 2019-07-30 河南萱闱堂医疗信息科技有限公司 一种辅助患者进行肠道准备的方法
CN110490856B (zh) 2019-05-06 2021-01-15 腾讯医疗健康(深圳)有限公司 医疗内窥镜图像的处理方法、系统、机器设备和介质
CN110189303B (zh) * 2019-05-07 2020-12-25 武汉楚精灵医疗科技有限公司 一种基于深度学习和图像增强的nbi图像处理方法及其应用
CN110363751B (zh) * 2019-07-01 2021-08-03 浙江大学 一种基于生成协作网络的大肠内视镜息肉检测方法
JP7116849B2 (ja) * 2019-08-16 2022-08-10 Hoya株式会社 内視鏡用プロセッサ、内視鏡システム、情報処理装置、プログラム及び情報処理方法
CN110738655B (zh) * 2019-10-23 2024-04-26 腾讯科技（深圳）有限公司 影像报告生成方法、装置、终端及存储介质
CN111178128B (zh) * 2019-11-22 2024-03-19 北京迈格威科技有限公司 图像识别方法、装置、计算机设备和存储介质
CN111105412B (zh) * 2019-12-30 2023-10-20 郑州大学 一种用于肠道息肉检测识别的智能辅助系统
CN111242920A (zh) * 2020-01-10 2020-06-05 腾讯科技（深圳）有限公司 一种生物组织图像检测方法、装置、设备及介质
US11782258B2 (en) 2020-01-21 2023-10-10 Aetherai Co., Ltd. Endoscopic inspection system and method thereof
CN112911165B (zh) * 2021-03-02 2023-06-16 杭州海康慧影科技有限公司 内窥镜曝光方法、装置及计算机可读存储介质
CN113139929B (zh) * 2021-03-17 2024-09-27 浙江工业大学 包含信息筛选与融合修复的胃肠道内窥镜图像预处理方法
CN113284146B (zh) * 2021-07-23 2021-10-22 天津御锦人工智能医疗科技有限公司 结直肠息肉图像的识别方法、装置及存储介质
CN113470026B (zh) * 2021-09-02 2021-11-05 北京字节跳动网络技术有限公司 息肉识别方法、装置、介质及设备
CN114022435A (zh) * 2021-10-29 2022-02-08 中国船舶重工集团公司第七0四研究所 基于机器视觉的管路泄漏识别系统建立方法
CN113822894B (zh) * 2021-11-25 2022-02-08 武汉大学 十二指肠胰头图像识别方法和十二指肠胰头图像识别装置
CN113935993B (zh) * 2021-12-15 2022-03-01 武汉楚精灵医疗科技有限公司 肠镜图像识别系统、终端设备及存储介质
CN114332019B (zh) * 2021-12-29 2023-07-04 小荷医疗器械(海南)有限公司 内窥镜图像检测辅助系统、方法、介质和电子设备
CN114782390B (zh) * 2022-04-29 2023-08-11 小荷医疗器械(海南)有限公司 检测模型的确定方法、息肉检测方法、装置、介质及设备
CN117204790B (zh) * 2023-09-12 2024-01-30 南京诺源医疗器械有限公司 一种内窥镜的图像处理方法及系统
CN118429371B (zh) * 2024-07-05 2024-09-20 中南大学湘雅二医院 一种基于自适应阈值的肠息肉轮廓分割方法、系统及设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050096712A1 (en) * 2003-10-06 2005-05-05 Klaus Abraham-Fuchs Method for automated localization of lesions in the gastrointestinal tract for treatment using laser light from an endorobot
CN103315699A (zh) * 2013-06-13 2013-09-25 福建师范大学 一种适用于巴雷特食管定位及识别的内窥系统
CN107730489A (zh) * 2017-10-09 2018-02-23 杭州电子科技大学 无线胶囊内窥镜小肠病变计算机辅助检测系统及检测方法
CN107895369A (zh) * 2017-11-28 2018-04-10 腾讯科技（深圳）有限公司 图像分类方法、装置、存储介质及设备
CN109447973A (zh) * 2018-10-31 2019-03-08 腾讯科技（深圳）有限公司 一种结肠息肉图像的处理方法和装置及系统

Family Cites Families (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7391893B2 (en) * 2003-06-27 2008-06-24 Siemens Medical Solutions Usa, Inc. System and method for the detection of shapes in images
CN101540000B (zh) * 2008-03-20 2011-07-27 中国科学院自动化研究所 基于纹理基元统计特性分析的虹膜分类方法
US20110301447A1 (en) 2010-06-07 2011-12-08 Sti Medical Systems, Llc Versatile video interpretation, visualization, and management system
CN102509112A (zh) * 2011-11-02 2012-06-20 珠海逸迩科技有限公司 车牌识别方法及其识别系统
EP3274915A1 (en) 2015-03-27 2018-01-31 Siemens Aktiengesellschaft Method and system for automated brain tumor diagnosis using image classification
WO2016161115A1 (en) * 2015-03-31 2016-10-06 Mayo Foundation For Medical Education And Research System and methods for automatic polyp detection using convolutional neural networks
US10861151B2 (en) * 2015-08-07 2020-12-08 The Arizona Board Of Regents On Behalf Of Arizona State University Methods, systems, and media for simultaneously monitoring colonoscopic video quality and detecting polyps in colonoscopy
EP3405908B1 (en) * 2015-09-10 2021-12-15 Magentiq Eye Ltd. A system and method for detection of suspicious tissue regions in an endoscopic procedure
CN106097335B (zh) * 2016-06-08 2019-01-25 安翰光电技术（武汉）有限公司 消化道病灶图像识别系统及识别方法
CN106127116B (zh) * 2016-06-16 2019-05-28 新浪网技术(中国)有限公司 一种图片类型识别的方法和装置
CN106530290A (zh) * 2016-10-27 2017-03-22 朱育盼 医学图像分析方法和装置
JP6751773B2 (ja) 2016-12-07 2020-09-09 オリンパス株式会社 画像処理装置及び画像処理装置の作動方法
CN106934799B (zh) * 2017-02-24 2019-09-03 安翰科技(武汉)股份有限公司 胶囊内窥镜图像辅助阅片系统及方法
CN107240091B (zh) * 2017-04-21 2019-09-03 安翰科技(武汉)股份有限公司 胶囊内窥镜图像预处理系统及方法
CN107256552B (zh) * 2017-06-14 2020-08-18 成都微识医疗设备有限公司 息肉图像识别系统及方法
WO2019003911A1 (en) * 2017-06-27 2019-01-03 Sony Corporation MEDICAL IMAGE PROCESSING APPARATUS, MEDICAL IMAGE PROCESSING METHOD, AND CALCULATION SYSTEM
CN108010021B (zh) * 2017-11-30 2021-12-10 上海联影医疗科技股份有限公司 一种医学图像处理系统和方法
CN108596237B (zh) * 2018-04-19 2019-11-15 北京邮电大学 一种基于颜色和血管的lci激光内镜下的结肠息肉分类装置
CN108615037A (zh) * 2018-05-31 2018-10-02 武汉大学人民医院（湖北省人民医院） 基于深度学习的可控胶囊内镜操作实时辅助系统及操作方法
WO2020012563A1 (ja) * 2018-07-10 2020-01-16 オリンパス株式会社 内視鏡装置、処理装置及び処理方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050096712A1 (en) * 2003-10-06 2005-05-05 Klaus Abraham-Fuchs Method for automated localization of lesions in the gastrointestinal tract for treatment using laser light from an endorobot
CN103315699A (zh) * 2013-06-13 2013-09-25 福建师范大学 一种适用于巴雷特食管定位及识别的内窥系统
CN107730489A (zh) * 2017-10-09 2018-02-23 杭州电子科技大学 无线胶囊内窥镜小肠病变计算机辅助检测系统及检测方法
CN107895369A (zh) * 2017-11-28 2018-04-10 腾讯科技（深圳）有限公司 图像分类方法、装置、存储介质及设备
CN109447973A (zh) * 2018-10-31 2019-03-08 腾讯科技（深圳）有限公司 一种结肠息肉图像的处理方法和装置及系统

Also Published As

Publication number Publication date
US11468563B2 (en) 2022-10-11
JP7090836B2 (ja) 2022-06-27
US11748883B2 (en) 2023-09-05
US20210004959A1 (en) 2021-01-07
EP3876188A1 (en) 2021-09-08
CN109447973A (zh) 2019-03-08
US20220392068A1 (en) 2022-12-08
EP3876188B1 (en) 2024-10-23
JP2021520980A (ja) 2021-08-26
EP3876188A4 (en) 2021-12-29
CN109447973B (zh) 2021-11-26

Similar Documents

Publication Publication Date Title
WO2020088328A1 (zh) 一种结肠息肉图像的处理方法和装置及系统
CN109886243B (zh) 图像处理方法、装置、存储介质、设备以及系统
US20210319560A1 (en) Image processing method and apparatus, and storage medium
WO2022036972A1 (zh) 图像分割方法及装置、电子设备和存储介质
CN107423699B (zh) 活体检测方法及相关产品
WO2020062969A1 (zh) 动作识别方法及装置、驾驶员状态分析方法及装置
WO2019052329A1 (zh) 人脸识别方法及相关产品
JP2016531362A (ja) 肌色調整方法、肌色調整装置、プログラム及び記録媒体
CN107889534B (zh) 用于在解剖手术期间检测手术用纱布的方法和系统
WO2019011098A1 (zh) 解锁控制方法及相关产品
CN107025441B (zh) 肤色检测方法及装置
JP2022548453A (ja) 画像分割方法及び装置、電子デバイス並びに記憶媒体
CN108154465A (zh) 图像处理方法及装置
CN114693593B (zh) 图像处理方法、装置、及计算机装置
CN109360222A (zh) 图像分割方法、装置及存储介质
CN110211086B (zh) 图像分割方法、装置及存储介质
WO2020114097A1 (zh) 一种边界框确定方法、装置、电子设备及存储介质
CN111557007B (zh) 一种检测眼睛睁闭状态的方法及电子设备
CN111724361B (zh) 实时显示病灶的方法及装置、电子设备和存储介质
CN107423663A (zh) 一种图像处理方法及终端
CN110363782B (zh) 一种基于边缘识别算法的区域识别方法、装置及电子设备
CN108171222A (zh) 一种基于多流神经网络的实时视频分类方法及装置
CN109785254A (zh) 图片降噪方法、图片降噪模型生成方法、终端及存储介质
CN110766631A (zh) 人脸图像的修饰方法、装置、电子设备和计算机可读介质
CN111738998A (zh) 病灶位置动态检测方法及装置、电子设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19878480

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021512995

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019878480

Country of ref document: EP

Effective date: 20210531