Classifications

• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H30/00—ICT specially adapted for the handling or processing of medical images
  • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data

Definitions

• This disclosure relates to a method and system for identifying and retrieving similar radiology images to a query image.
• a stack which corresponds to the orientation of images according to some form of Cartesian alignment, e.g. along a plane corresponding to each axis in three dimensions (this is how computed tomography (CT) images are organized).
• CT computed tomography
• each stack one finds the images ordered based on their position within a stack.
• heterogeneous annotations including a mix of enumerated classes (e.g., presence or a medical condition like pneumonia), numeric data (e.g. age, weight, body mass index, etc.), and free text (e.g. radiology reports), and other metadata.
• radiographic imagery chest X-rays, CTs, mammograms, ultrasound, etc.
• the reference information returned is useful only if it provides the clinician with medically relevant information for the decision they are trying to make.
• the clinician For example, for a person trying to identify whether a 21-year old female should get a follow-up CT given a chest X-ray that contains a pulmonary nodule or mass (something that would require a follow up), returning chest X-ray results of 21-year-old females without any nodules or masses are not as useful as returning chest X-rays with nodules and masses.
• one of the main challenges is to define a metric of similarity that is contextually tailored to the relevant clinical decisions being supported by a similar image search tool.
• each positive example indicated by the + sign
• has a corresponding negative example illustrated by the - sign
• all positive examples are closer to one another than they are to any negative examples.
• This simplified example gets even more complicated as one considers similarity on multiple dimensions of annotated information combined with the temporal and spatial components of radiographic medical information described above in the context of Figure 1 A.
• the present system for identifying and retrieving similar medical radiology images is motivated by several needs.
• Radiologists may be more familiar with how certain conditions look than others based on what they’ve seen previously.
• the distribution of cases can vary from one hospital to another. For instance, tuberculosis might be more prevalent in hospital A and quite rare in hospital B. There exists a long tail of rare conditions for which a high number of positives might be difficult to isolate for classification.
• This document proposes a solution to these issues by means of a system that features a combination of back-end design (software infrastructure), including fetchers and scorers, and modelling techniques implemented in the scorers which generate a similarity score that captures a plurality of similarity attributes (e.g., diagnostic, visual and patient) of the query image and a set of candidate similar radiology images, and the annotations (e.g., metadata or medical reports) associated therewith.
• back-end design software infrastructure
• fetchers and scorers and modelling techniques implemented in the scorers which generate a similarity score that captures a plurality of similarity attributes (e.g., diagnostic, visual and patient) of the query image and a set of candidate similar radiology images, and the annotations (e.g., metadata or medical reports) associated therewith.
• the system includes one or more fetchers receiving the query image and retrieving a set of candidate similar radiology images from a data store in the form of a library of ground truth annotated reference radiology images.
• the fetcher can take the form of a trained deep convolutional neural network, nearest neighbor algorithm based on a feature vector extracted from the image, or classifier.
• These candidate images may or may not already be associated with scores. For example, in one possible configuration, if the query images are already indexed, scores to similar images may be pre-computed and cached, and the fetcher may make use of pre-cached similar images to retrieve candidate similar images.
• the system further includes one or more scorers which receive the query image and the set of candidate similar radiology images and generate a similarity score between the query image and each candidate image.
• the score can be computed for example based on pre-computed embedding and a standard distance metric (e.g., cosine or
• Euclidean distance in an embedding space. For example, the scorer looks up the embedding of an image in a database and then uses a distance measure in the embedding space to determine how similar the query image is to the candidate similar radiology images.
• the system further includes a pooler which receives the similarity scores from the one or more scorers, ranks the candidate images (e.g., on the basis of acuteness/severity), and returns a list of the candidate images reflecting the ranking.
• a pooler which receives the similarity scores from the one or more scorers, ranks the candidate images (e.g., on the basis of acuteness/severity), and returns a list of the candidate images reflecting the ranking.
• the scorers implement a modelling technique to generate the similarity score that can capture similarity on many different attributes or axes (e.g., diagnostic, visual, patient, etc.). Diagnostic, visual and patient attributes are some of the many signals that could be important on specific axes of similarity, but these three are not meant to be an exhaustive list. A number of different modelling techniques are contemplated, and in a typical implementation where there are more than one scorer they will each use a different modelling technique that captures these different attributes of similarity (e.g., diagnostic, visual and patient).
• the fetcher can also use these different modelling techniques to retrieve similar medical images from the data store.
• Some of the modelling techniques include triplet loss, classification loss, regression loss, and object detection loss. Attention models may also be used which takes into account the additional regional information within an image, which allows us to consider one additional layer of hierarchy of the regions of interest within an image, i.e. , sub-image level metadata.
• triplet loss is a technique that handles heterogeneous data consistently in a way that notionally captures similarity. Specifically, suppose we have three images: a query image and two candidate images. If we know that we have a query image that is closer to one of the candidate images (the positive) than it is to the other (the negative), then we expect the distance between the extracted features between the positive pair (query and positive candidate) to be smaller than the distance between the query and negative candidate. The triplet loss is thus the difference between these two distances.
• the present document describes a variety of methods of calculating triplet loss (i.e., a distance metric between a query image and two candidate images), including patient and clinical metadata (including numerical data, e.g., BMI, age, weight, etc.), structured labels, a Hamming distance over a vector of classification labels based on medical reports, and the location of abnormalities within an image.
• triplet loss i.e., a distance metric between a query image and two candidate images
• patient and clinical metadata including numerical data, e.g., BMI, age, weight, etc.
• the general arrangement of fetchers, scorers, and a pooler allows for the processing and retrieving of similar radiology images on a scalable basis. Further, the use of different modelling techniques for similarity in the fetching and scorers allows for different aspects of similarity modelling to be combined to generate a set of similar medical images that provide diagnostically useful information to a user and that meet the needs of clinical applications of similar medical image search, particularly in the radiology context.
• the information that is returned to a user after performing the fetching, scoring and ranking of the similar images includes not just the similar images (and associated metadata), but also information that can be culled, inferred or aggregated from the result set of the similar images.
• the system includes a processing unit which performs the aggregation or inferring of data from the candidate similar images.
• the images can be returned not merely as a list of images but rather grouped together across common attributes that are useful for supporting a clinical decision. For example, images with certain misplaced foreign bodies (e.g., misplaced nasogastric tube) might be grouped separately from those images that are associated with a diagnosis of pneumothorax.
• misplaced foreign bodies e.g., misplaced nasogastric tube
• the groupings can involve the aggregation of relevant common text from radiology free text reports. For example, while there may not be a specific label indicating that an endotracheal tube is misplaced, we can aggregate together images that are associated with reports having common phrases that imply this condition to be present, for example reports having text entries“endotracheal tube at the level of the carina”,“endotracheal tube tip terminates in right main bronchus”, or“ET tube tip could be advanced a couple of centimeters for standard positioning.”
• relevant information is returned to the user from this set.
• fetchers, scorers and pooler are configured to perform the various functionality described above.
• a method for identifying and retrieving similar radiology images to a query radiology image.
• the query image is associated with annotations including metadata.
• the method includes a step a) of curating (i.e., developing and storing) a data store of ground truth annotated radiology images, each of the radiology images associated with annotations including metadata.
• the method includes a step b) of receiving the query image and retrieving a set of candidate similar radiology images from the data store.
• the method includes a step c) of generating a similarity score between the query image and each candidate similar radiology image using at least two different scorers.
• the at least two scorers implement a different modelling technique to generate the similarity score capturing a plurality of similarity attributes of the query image and the set of candidate similar radiology images and the annotations associated therewith.
• the method includes a step d) of ranking the candidate similar radiology images.
• the method further includes a step e) of returning a list of the candidate similar radiology images reflecting the ranking and aggregated information obtained from the annotations associated with the set of candidate similar radiology images.
• Figure 1 A is an illustration of a multi-stage hierarchy of medical information along with associated heterogeneous annotation data.
• Figure 1 B is an illustration of classifications of positive and negative examples by two different classifiers.
• Figure 2 is an illustration of a use case of the methods of this disclosure in which similar medical images to a query image are retrieved and used by a medical professional, e.g., radiologist or pathologist, to make certain findings regarding a patient associated with the query image.
• a medical professional e.g., radiologist or pathologist
• Figure 3 is a high level illustration of the workflow using the methods of this disclosure from a user perspective.
• Figure 4 is a block diagram of one possible configuration of a back-end design receiving a query image, and including a fetcher fetching candidate images, a plurality of scorers, and a pooler which receives the similarity scores from the scorers, ranks the candidate images (e.g., on the basis of acuteness/severity), and returns a list of the candidate images reflecting the ranking.
• Figure 5 is a block diagram of another possible configuration of a back-end design of Figure 4.
• Figure 6 is a block diagram of another possible configuration of a back-end design of Figure 4.
• Figure 7 is a block diagram of a portion of a back-end design showing a plurality of different scorers, each of which generate a similarity score between a query image and a set of candidate similar radiology images that captures diagnostic, visual and patient similarity attributes of the query image and the set of candidate similar radiology images including the annotations associated therewith.
• Figure 8 is an illustration of a display on a workstation used by the medical professional showing a query image and a set of similar radiology images retrieved, scored and ranked using the back-end design of Figure 4, 5, 6, or 7, as well as aggregated information from the set of similar radiology images in which the retrieved images are sorted or group by diagnosis.
• Figure 9 is an illustration of a display on a workstation showing a query image, a set of retrieved similar medical images, and a set of statistical plots showing various statistics associated with the retrieved similar images, such as the medications received, the occurrence of certain medical events over a period of days, and so forth.
• Figure 10 is an illustration of a display on a workstation showing a query image, and thumbnail images of retrieved similar images and showing the distribution of the similar images, in this case grouping by sex (left and right columns) and diagnosis (rows).
• Figure 11 is an illustration of a plot of an embedding of multitude of radiology images of the reference library in a multidimensional space. Each rectangular patch represents a single image in the reference library. The position of the images in the embedding is a factor of similarity in multiple axes.
• This document describes a computer-implemented system for identifying similar radiology images to a query image.
• the system can be considered a tool for assisting a medical professional such as a radiologist, ER doctor, or primary care physician in arriving at a diagnosis for a patient based on a radiology image of the patient, such as a chest X-ray, mammogram, or CT scan.
• the system provides diagnostically useful output information to a user based upon an input image.
• the general idea of how the system works is illustrated in Figure 2.
• a radiology image 200 is obtained, e.g., using conventional imaging equipment, and supplied to the system 202 of this disclosure.
• the image 200 is considered the query image; that is, the medical professional seeks to find similar images to the image 200.
• the query image 200 is associated with medical information, metadata, reports, etc. (collectively“annotations”).
• the system 202 returns a list of results 204 in the form of similar radiology images (SMILY, “similar medical image like yours”) obtained from a data store in the form of a reference library of ground-truth annotated radiology images of the type of the query radiology image 200.
• the results 204 will include not only the similar images but also the annotations associated therewith.
• the results will generally also be returned along with groupings and aggregated information, e.g., statistics, as will be explained in detail in Figures 3 and 8-10 later in this document.
• the medical practitioner then reviewing the query image 202 and the results 204 enters findings 206, typically diagnostic findings, in the medical record of the patient, for example in the form of a free-text report or structured note.
• One goal of the system of this disclosure is provide a tool to improve the decision-making task; the medical professional uses the results in addition to other diagnostic procedures and methods to generate the clinical findings.
• all findings for an image may not be clinically relevant to a specific action/plan; here, we are referring in Figure 2 to findings A, B and C as those findings that are clinically relevant.
• Figure 3 illustrates the method of operation of the system from a user perspective.
• a front end component 300 which provides a user interface to the medical professional on a conventional computer workstation (not shown).
• the front end component is entered from a PACS (picture archiving and communications) system, such as by activating a tab or icon within the PACS system, or which may be a custom design.
• the front end component provides a software interface, e.g., via a set of application programming interfaces (APIs) to a back end system in the form of a set of computer software modules which are shown in Figure 4, executing within a computer system that may be either local to the clinician or in a remote, cloud computing environment.
• the front end component provides options for a user to select a query image.
• the system executes a similar image search to the query image, as indicated at 302.
• the front-end system provides a display on the workstation of the query image 200 and the results 204.
• the results are shown in a summary format in Figure 3, but a variety of other formats are described later in conjunction with Figures 8-10 and the details may vary from that shown in Figure 3.
• the user has the option to click or select any one of the similar images in the results 204 (indicated by the arrow 303) and a detailed view of the similar image 304 that was selected is displayed.
• the interface of the front end component also includes tools to refine the search (indicated by the arrow 306) and similar images are retrieved based on the refinements that are specified. Such refinements could be specified by entry of text in a text box, by selecting only patients by certain age groups, smoker status, sex, diagnosis, or other criteria, or by selecting only a set of images associated with a particular diagnosis or condition.
• relevant information is returned to the user from this set. This would normally include not only the images themselves, but also metadata associated with each of the images like radiology reports, clinical decisions made (e.g., prescribing of antibiotics, diuretics), classification
• Figure 4 is a block diagram of one possible configuration of a back end 400 in the form of a set of software modules or objects which receive a query image and generate a list of results.
• the software modules are executed in a computer system having computing resources and processing units, e.g., a graphics processing unit, memory storing parameters of machine learning models, processing unit for calculating statistics, etc. as will be appreciated by those skilled in the art.
• computing resources and processing units e.g., a graphics processing unit, memory storing parameters of machine learning models, processing unit for calculating statistics, etc.
• the flow of requests and response are indicated by the thin and thick arrows as indicated by the legend in Figure 4.
• the objects in the back end can be roughly divided into two categories:
• Controller 402 an object that receives queries from outside the back end (e.g., the front end 300 of Figure 3) and orchestrates the Dispatchers 404, Fetcher 406 and Pooler 410 to generate a list of similar image results and ranking the results.
• the controller also constructs these objects from a configuration or initial state.
• Dispatcher 404 an object that distributes a query between several different fetchers 404 and scorers 406, then collates the results using a pooler 410.
• the dispatcher sends the candidate images and the queried image to a set of scorers in parallel, fetches the results, and passes the resulting scores to the pooler 410 for ranking.
• Fetcher(s) 406 - an object that receives a query image 200 and generates a set of candidate similar images by querying a data store (not shown in Figure 3) in the form of a library of ground truth annotated reference images, which may or may not already be associated with scores.
• a data store not shown in Figure 3
• Scorer(s) 408- an object that receives a query image and a set of candidate images and returns a similarity score between the query image and each candidate image.
• the scorers implement a modelling technique to generate the similarity score capturing a plurality of similarity attributes of the query image and the set of candidate similar radiology images and annotations associated therewith, such as diagnostic, visual and patient similarity. If there are multiple scorers, each implements a different modelling technique.
• Pooler 410 an object that receives scoring results from several different scorers or fetchers, collated by the dispatcher 404, and returns a single list of the combined results.
• the pooler ranks the candidate images (e.g., on the basis of acuteness/severity), and returns a list of the candidate images reflecting the ranking.
• the software architecture of Figure 4 provides the ability to combine different scoring techniques trained from different models together and combine them to produce a final ranking in a scalable manner.
• Figure 4 can be realized in other formats and arrangements of the basic building blocks or objects.
• Figure 5 illustrates one possible variation.
• the query image 200 is received by a dispatcher/pooler 502 which includes a fetcher 406 which retrieves a candidate set of similar images from a data store or repository 500.
• the set of images is sent to a dispatcher 404 which sends the image query and the candidate images to a scoring module 408 which includes three different scorers 408A, 408B and 408C.
• Each module 408A, 408B and 408C uses a different modelling technique to generate the similarity score for the candidate images.
• These modelling techniques each capture or take into account two or more similarity attributes between the query image and the set of candidate similar radiology images, and the associated annotations, such as patient, diagnostic, and visual similarity.
• attributes of similarity can be represented as coordinate axes in a multidimensional embedding space, see Figure 11 , where feature vectors of the image and associated annotations are used to plot the position of the query image and the candidate set of similar images in this feature space, and distance metrics or other types of modeling techniques described below are then used to generate similarity scores reflecting the similarity.
• the similarity scores and candidate set of similar images are then returned to the dispatcher 404 and then supplied to the pooler 410, which then ranks the candidate set of similar images using the scores.
• the pooler then returns the ranked images as results 204 (again, preferably with aggregation information, statistics, groupings, metadata, etc. as described in detail elsewhere).
• Figure 6 illustrates another possible configuration.
• the numbers 1 , 2, 3.1 , 3.2, etc. represent the order in which the objects 402, 404A, 406A, 408A, 410A, etc. are called.
• This embodiment features a controller 402, and a fetch dispatcher 404A which dispatches fetch requests to different fetchers 406A and 406B, each using different modeling techniques to identify a set of candidate similar images from a data store.
• the fetch results are pooled in a pooler 410 and then sent via controller 402 to a score dispatcher 404B, which dispatches the query image and candidate set of similar images to scorers 408A and 408B, each using different modelling techniques to generate similarity scores.
• fetch requests 3.1 and 3.2 can be computed in parallel, similarly for score requests 7.1 and 7.2.
• FIG 7 illustrates yet another configuration.
• a dispatcher/pooler 502 e.g., configured as per Figure 5, forwards candidate images and the query image to each of 5 different scorers 408A, 408B, 408C, 408D and 408E in parallel.
• Each of the scorers uses a different modelling technique to generate the similarity score that captures similarity attributes of the query image and the set of candidate similar radiology images and their associated annotations, such as for example diagnostic, patient and visual similarity.
• Each scoring module generates a score using a distance metric based on an embedding or projection of a feature vector of the query image into a multidimensional space, e.g. shown in Figure 11.
• Scoring module 408A uses embeddings from Regional Maximum Activations of Convolutions (R-MAC) on localized conditions. For further details see e.g. A Gordo et al., Deep Image Retrieval: Learning global representations for image search, arXiv.org [cs.CVj 1604.0132 (July 2016). Scoring module 408B uses embeddings obtained from supervised learning models. Scoring module 408C uses embeddings from triplet losses, explained below. Scoring module 408D uses embeddings from fine-grained image features, see. J. Wang, et al., Learning fine-grained image similarity with deep ranking,
• Scoring module 408E uses embeddings from a classifier with NCA (network component analysis).
• Figure 1 1 is an example of a plot of an embedding of multitude of candidate radiology images obtained from the reference library by the fetcher in a multidimensional space or embedding.
• the embedding consists of three axes. The axes represent different aspects of similarity, such as visual, diagnostic and patient.
• Each rectangular patch 1102A, 1102B, etc. represents a single radiology image. Images that are similar are clustered close to each other whereas dissimilar images are not. Similar medical images to a query image are found by projecting the query image feature vectors into the embedding of Figure 4 scoring the neighboring images by distance in the mu!tidimensionai space.
• the cluster of images 1104 containing image 1102A represents a group of images which are similar in all three axes to a query image 1106 indicated by the star.
• the query image was a chest X-ray positive for pneumothorax, patient was a smoker, etc. the query image would be positioned in the location of the star 408 and the images in the cluster 1104 would be scored lower (i.e., more similar) than for example the image 1102B which is further away.
• the fetcher receives the query image and retrieves a set of candidate similar radiology images from a data store in the form of a library of ground truth annotated reference radiology images.
• the data store can be curated, i.e., developed and maintained, by obtaining ground truth annotated radiology images from publicly available or private sources, or by obtaining images from public or private sources and adding the ground truth annotations with the use of trained readers.
• the fetcher can take the form of a trained deep convolutional neural network or classifier, optionally with filters, e.g. to exclude or include only some images for example those that are positive for a particular condition present in the query image.
• the fetcher can also include a function to first classify the query image (e.g., determine that it is positive for pneumothorax) and use that classification to filter the similar images to only those that have a ground truth annotation of pneumothorax.
• the fetcher could take several forms and could for example be configured in accordance with one of the following references, the content of which is incorporated by reference herein: C. Szegedy et al., Going Deeper with
• one or more of the fetchers could be configured as a precached fetcher.
• a pre-cached fetcher the similar candidate images for a given query image have been precomputed.
• the precomputing of similar images could be using any suitable technique.
• the fetchers can use various different modelling techniques to determine similarity of images, and such modelling techniques are described in more detail in the discussion of the scorers.
• Such modelling techniques can include triplet loss, classification loss, regression loss and object detection loss.
• the system uses one or more scorers which receive the query image and the set of candidate similar radiology images (identified by the fetcher) and generates a similarity score between the query image and each candidate image, using the image data as well as underlying annotations (image metadata, reports, patient information etc.) associated with the images.
• the score can be computed for example based on precomputed embedding and a standard distance metric (e.g., cosine or Euclidean distance) in the embedding space. For example, the scorer looks up the embedding of an image in a database and then uses a distance measure in the embedding space. See the discussion of Figure 11 , supra.
• the scorers implement a modelling technique to generate the similarity score that can capture similarity on many different axes (e.g., diagnostic, visual, patient, etc.)
• Diagnostic, visual and patient attributes are some of the many signals that could be important on specific axes of similarity, but these three are not meant to be an exhaustive list.
• a number of different modelling techniques are contemplated, and in a typical implementation where there are more than one scorer they will each use a different modelling technique that captures these different attributes of similarity (e.g., diagnostic, visual and patient).
• One configuration of the scorers develops various signals in parallel that capture diagnostic, visual and patient similarity.
• the output from these signals will either be image embeddings that captures the similarity signal or a similarity score for every candidate image.
• the scoring module is responsible for combining the various signals and for the final scoring and ranking the candidate images.
• Models to identify if two X-rays belong to same person or not A data set that includes longitudinal X-rays of a given patient gives us multiple images for the same person over time; use this to build a training set of same person vs not same person and the models can be trained over pairs or triplets to classify same person or not.
• GANs Cycle generative adversarial networks
• triplet loss which only requires some notion of ordering of images, thereby allowing us to deal with irregularly sampled data, hierarchy, and heterogeneous annotations/metadata
• regression loss i.e., regressing to report embeddings, which is one technique that allows us to combine the modalities of image and report data
• classification loss e.g., classification loss
• object detection loss e.g., through the use of attention models, which takes into account the additional regional information within an image, which allows us to consider one additional layer of hierarchy of the regions of interest within an image, i.e. sub-image level metadata.
• triplet Loss This is a technique, described in the literature, that allows us to handle our heterogeneous data consistently in a way that notionally captures similarity. Specifically, suppose we have three images: a query image and two candidate images. If we know that we have a query image that is closer to one of the candidate images (the positive) than it is to the other (the negative), then we expect the distance between the extracted features between the positive pair (query and positive candidate) to be smaller than the distance between the query and negative candidate. The triplet loss is thus the difference between these two distances. Thus, triplet losses are a way of comparing images by creating an ordering of some of the images, e.g. for a distance function D(.,.), saying that
• the Hamming Distance is one way to construct such an ordering, by saying that images that have more of the same conditions (similar medical conditions, similar demographic information, localizable abnormalities appear in the same region, etc.) are more similar than those that have fewer.
• condition is used loosely to capture both medical abnormalities as well as
• the Hamming Distance is not the only way to construct such an ordering. Some alternatives include: a) Images taken of the same patient that are closer in time are more similar than images taken of the same patient that are farther apart
• Chest X-Ray images with an associated radiology report text, projected to a common embedding space, are more similar to each other than the original X-ray with a radiology report associated with a different Chest X-ray image.
• Chest X-Ray images with a follow-up chest CT, projected to a common embedding space, are more similar to each other than the original chest X-Ray with a chest CT that followed up a different Chest X-ray.
• classification loss There are other methods for modelling similarity as alternative to triplet loss.
• classification loss can take several forms.
• Cross-entropy loss increases as the predicted probability diverges from the actual label. The details are known in the art and described in the literature, for example in the tutorial
• /(report) M g(image) + b, where M is a matrix, and b is a vector.
• g is a neural network
• the output dimension of g is the same as that of /, it turns out that this is equivalent to making M an identity matrix and b a zero vector, so given example pairs (reportj, imagej), we can solve a regression problem by minimizing for some loss, e.g. squared error: min_ ⁇ g ⁇ ⁇ sum_i (f(report_i) - g(image_i)) A 2.
• Object detection loss is another modelling technique for capturing similarity.
• a pneumothorax is found in the same part of a candidate image as the query, those images might be closer to one another.
• the existence, size, or location of elements within an image are important for determining similarity, e.g. the position of the carina and the tip of an ET tube to determine whether the ET tube is correctly placed, or the location and size of a pulmonary nodule, then we can formulate it as an object detection problem (object detection loss, e.g. intersection over union).
• Attention mechanisms give us the capability to do this.
• the technique of Integrated Gradients can be used, as an example of an attention mechanism.
• Attention mechanisms such as Integrated Gradients, are machine learning tools which basically identify those portions of the data set that contribute the most to the model predictions.
• IGj(image) is the integrated gradient w.r.t. the i th pixel, i.e., attribution for i th pixel; and V i s the gradients operator with respect to image,.
• the goal here is not only to use where in the image something occurs to inform similarity.
• the system of Figures 4-7 further includes a pooler 410 which receives the similarity scores from the one or more scorers 408, ranks the candidate images (e.g., on the basis of acuteness/severity), and returns a list of the candidate images reflecting the ranking.
• the pooler 410 thus pools and ranks the candidate images based on scores from the different scorers 408.
• a final ranking is done at the pooler 410, with intermediate rankings proposed by the scorers, e.g., based on the similarity scores. There can also be an implicit exclusion of certain images from the ranking based on the candidate set of images that are returned by the fetcher(s) 406.
• the final ranking can be a mix of objective measures like the Hamming distance and scores derived from subjective measures, e.g., what medical professionals actually consider to be similar images for the clinical context they are working in.
• Subjective measures could be used for a final comparison of different models or ranking methods. For instance, consider a set of query images q_1 ,...,q_N, and for each of these queries, we receive ranked images r_1 (q_i), r_2 ⁇ q_i),..., r_k(q_i) the top k images returned for query image qj.
• doctors and/or other medical professionals could indicate whether the ordering of r_1 (q_i), r_2(q_i),..., r_k(q_i) makes sense for image qj and how relevant they are. From these, one could compute scores for image pairs
• This option might fail to capture certain nonlinearities in when and how to weight the different scores from the scorers.
• the information that is returned to a user includes not just the similar images (and associated annotations, e.g. metadata, reports or excerpts thereof), but also information that can be culled, inferred or aggregated from the result set of the similar images, such as the statistics that can be computed from the query images.
• the images can be returned not merely as a list of images but rather grouped together across common attributes that are useful for supporting a clinical decision.
• Figure 8 there is shown a display on a workstation showing the query image 202 and the results 204, in the form of a multitude of similar images 304.
• the similar images 304 are grouped in rows, with a legend in the form of a diagnosis or fining associated with each row.
• Row 802 NGT (nasogastric tube) Correctly Placed, has a set of four images 304 that have that characterization.
• Row 804 has a legend Pneumonia.
• Row 806 has a legend Pneumothorax.
• images with correctly placed foreign bodies e.g., nasogastric tube
• images with correctly placed foreign bodies are grouped separately from those images that are associated with a diagnosis of pneumothorax and diagnosis of pneumonia.
• the groupings can involve the aggregation of relevant common text from radiology free text reports. For example, while there may not be a specific label indicating that an endotracheal tube is misplaced, we can aggregate together images that are associated with reports having common phrases that imply this condition to be present, for example reports having text entries“endotracheal tube at the level of the carina”,“endotracheal tube tip terminates in right main bronchus", or“ET tube tip could be advanced a couple of centimeters for standard positioning.” Attention mechanisms in the scorers can be used to identify portions of the free text reports, such as particular words or phrases, which contribute the most to the similarity score.
• the circles 808 indicate areas (patches of pixels) of the chest X-rays, with NG tubes placed correctly, which are colored with a contrasting color, e.g., red, to illustrate the areas in the image that the model weighted most heavily to generate a high score of similarity to the query image for this diagnosis.
• the circles 810 illustrate regions in the X-ray images in the“Pneumonia" row of images which are shown in a contrasting color to illustrate those patches or areas in the image that the model weighted most heavily to generate a high score of similarity to the query image for this diagnosis.
• Pneumothorax diagnosis has similar areas which are highlighted to indicate the greatest weight by the model.
• the attention models can also be applied to annotations associated with the similar images, such as the free text in the radiology reports associated with the images, and in Figure 8 the free text areas 812 below the images there can be words or phrases from the annotations highlighted to indicate that the attention model gave significant weight to such words or phrases in determining similarity to the query image.
• Figure 9 is an illustration of another display of a query image 202 and the results 204. This configuration emphasizes summarizing similar patient data over time, but still provides individual instances of similar images.
• the horizontal bars 900 identify similar conditions, with the counts (numbers) of similar images that were returned. For example, the first bar has a condition“pneumothorax” and a count of 13, the second bar has a condition of “pulmonary embolism” and a count of 31.
• the bar 902 on the right hand has a condition “change in line or tube placement” and an option (X) to remove the condition if it is not relevant to the diagnosis of the query image.
• the region 904 shows summary statics on a timeline for similar patients. This timeline shows what medications/events happened for similar patients before and after the most similar image.
• the trends in the timelines may highlight obvious interventions to make for the patient associated with the query image.
• the region 906 shows plots of other significant distributions. Gradual trends may indicate that correct diagnoses were initially missed, suggesting it could be useful to carefully consider this patient's case.
• the bars 908 can be clicked in which case a filtering operation occurs which filters the similar images to only those represented in that selected bar. Other aggregate statistics are shown in the region 910.
• the area 912 shows the most similar images to the query image.
• a load more icon 914 allows the user to load more images and a scroll bar allows the user to navigate down to the newly loaded images.
• Figure 10 shows another alternative display in which the similar images are grouped or clustered by diagnosis (rows) and by patient sex (columns).
• the number of thumbnail images in each cluster 1000 reflects either directly or proportionately the number of similar images that were returned.
• a user can select a cluster, e.g., female, plural infusion, cluster 1000A, and the similar images in that cluster are displayed, e.g., as shown in Figure 3 or Figure 8.
• relevant information is returned to the user from this set. This would normally include not only the images themselves, but also metadata associated with each of these images like radiology reports, clinical decisions made (e.g., prescribing of antibiotics, diuretics), classification
• the system of this disclosure could be deployed in a cloud environment in which the back end of Figures 4 -7 is implemented remotely (e.g., in the cloud, for example by a service provider that trained and developed the deep learning models used to fetch, score and rank similar images and curated the reference library of ground truth annotated radiology images).
• a query image set is sent over a computer network (e.g., the Internet) and the service provider returns the candidate similar images, with ranking, aggregated information, annotations, etc., to a front end implemented in a medical clinic, hospital or office where the query image was obtained, where a radiologist is considering the query image, or where a physician may be consulting with a patient in regards to the query image and planning further treatment.
• a computer network e.g., the Internet
• system of this disclosure could be implemented locally, i.e. , with the back end and associated computing resources, reference library, software modules and deep learning models located locally to the office, clinic, or hospital where the query image is obtained, or viewed on a workstation e.g. by a radiologist or a primary care physician.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Epidemiology (AREA)
• General Physics & Mathematics (AREA)
• Physics & Mathematics (AREA)
• Databases & Information Systems (AREA)
• Data Mining & Analysis (AREA)
• Radiology & Medical Imaging (AREA)
• General Engineering & Computer Science (AREA)
• General Health & Medical Sciences (AREA)
• Medical Informatics (AREA)
• Primary Health Care (AREA)
• Public Health (AREA)
• Medical Treatment And Welfare Office Work (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Apparatus For Radiation Diagnosis (AREA)

Applications Claiming Priority (1)

Publications (1)

Family

ID=63036479

Family Applications (1)

Country Status (4)

Families Citing this family (4)

Family Cites Families (15)

• 2018
  • 2018-07-11 EP EP18746507.5A patent/EP3785274A1/de active Pending
  • 2018-07-11 AU AU2018431593A patent/AU2018431593A1/en not_active Abandoned
  • 2018-07-11 JP JP2021500660A patent/JP7157232B2/ja active Active
  • 2018-07-11 WO PCT/US2018/041557 patent/WO2020013814A1/en unknown

Non-Patent Citations (4)

Also Published As

Similar Documents

Legal Events