DICOM Image ANalysis and Archive (DIANA): an Open-Source System for Clinical AI Applications

Background

Hospital medical imaging services rely on Picture Archiving and Communications Systems (PACS) as central medical imaging repositories. However, most PACS setups are optimized to support a single departmental workflow and do not readily accommodate new retrospective or prospective image analysis workflows for research or clinical purposes. Accessing a large collection of images from a PACS, as required for artificial intelligence (AI) modeling, often requires repetitive, manual labor to retrieve and post-process each of thousands of studies [1]. This challenge hinders multiple independent tasks. For example, the review of large datasets is contingent on the ability to assemble said large amount of data in the first place—it follows that manual assembly could serve as the rate-limiting factor. Other hindrance presents itself in the real-time application of clinical analytics as data can be difficult to obtain with manual methods in both a prospective and time-conscious manner; at best, imaging devices can be reconfigured to directly send desired data to specific hosts, but this requires coordination from different parties. Additionally, the administrative and operational complexity of managing the embedded protected health information (PHI) in medical imaging is another commonly cited difficulty in the use of electronic health records and medical images in research studies often before manual retrieval tasks can even begin [2].

In an effort to alleviate some of the aforementioned issues, academic institutions have moved towards alternative solutions including vendor neutral archives (VNAs) with the hope of providing an improved, universal image viewing experience by decoupling the archival part of PACS from proprietary dependencies [3]. However, the switch from a pure PACS to a VNA system is associated with significant data migration challenges [4]. For research purposes specifically, some institutions have attempted to leverage open-source public medical image informatics platforms such as XNAT, but such systems have challenges in setup and implementation, which may limit scalability [5, 6]. Integrating image data into clinical data warehouses has also been limited by the same issue of scalability, and also by the massive investment of resources required to establish a clinical data warehouse [7–9]. Due to the obstacles in handling medical data, researchers have begun to implement bespoke methods for image data handling focused on training and delivery of AI models. Other research has been exploring methods to optimize delivery of AI models and their results to clinical settings [10].

The DICOM Image ANalysis and Archive (DIANA) system was designed as a Python command line interface that interfaces with hospital PACS to automate common medical image data handling processes.¹ The development of modern computing frameworks such as container systems (e.g., Docker) and text databases (e.g., Splunk) all motivated the possibility of DIANA’s existence, and refinement of AI containers (e.g., Docker with Keras/Tensorflow) remains a constantly improving field of research [11, 12]. This work aims to describe and evaluate the core functionality of DIANA and its potential in improving both research and clinical practice in the context of big-data retrieval, analysis, and AI delivery for medical imaging research.

Methods

The DIANA system leverages several open-source tools including Docker, a container system that facilitates replicability, and Orthanc, an open-source DICOM service that serves as a mini-PACS capable of communicating with other PACS systems [13, 14].

At a high-level, DIANA was evaluated in the context of two core tasks:

• i.

  Automated, retrospective batch retrieval of specific studies from PACS with optional DICOM metadata PHI anonymization

• ii.

  Automated, prospective monitoring of PACS for incoming studies of interest for extension to pretrained, live AI pipelines and notification system

The bulk of the system’s functionality will be reviewed in the context of these tasks. Miscellaneous functionality, such as radiation dose monitoring, is presented in the context of the overall system. DIANA was run on a headless server with an Intel® Xeon® CPU E5-2637 v3 @ 3.50 GHz. All AI pipelines were run without graphics acceleration.

Results

Evaluation of time required to retrieve images through the batch retrieval pipeline yielded the results shown in Fig. 3. Total retrieval times ranged from 17.80 to 339.71 s and 128.18 to 5882.26 s for XR and CT studies, respectively. Mean per-image retrieval times were 1.12 ± 0.50 and 0.08 ± 0.01 s across XR and CT studies, respectively.

Preliminary assessment of BA measurement across 44 cases revealed a mean discrepancy of 9.25 ± 10.86 months for AI prediction compared to ground truth (radiologist reading). Mean DIANA latency of 9.04 ± 3.83 min was significantly lower than radiologist latency of 51.52 ± 58.9 min (p = 1.84*10^–5). BA results are summarized in Fig. 4.

For ICH cases, the DIANA AI captured 100% of ED head CTs completed during the study period. In the 626 studies included in analyses, DIANA demonstrated 92% sensitivity and 94% specificity in detecting presence or absence of ICH. The AUC for DIANA was 97% with optimal detection threshold at 50.7%. Mean latency for DIANA was 20.17 (median 17.6) ± 10.16 min. Primary and secondary latencies for radiologists were 65.62 (median 26.0) ± 110.39 and 91.67 (median 36.0) ± 157.82 min, respectively. Speed of interpretation for DIANA was significantly greater than radiologists’ (p = 1.28 × 10⁻²², 2.70 × 10⁻²⁷). ICH results are summarized in Table 1 and Fig. 5.

Discussion

In this work, DIANA demonstrated retrospective batch retrieval latencies that suggest efficiency and consistency in handling multi-image jobs. In general, the results demonstrate that pull times scale linearly with query burden.

The true value of such speeds is best appreciated in context. As an example of automation in action, a 90 x-ray cohort was pulled and anonymized via DIANA then transferred to and displayed from a research PACS in under 2 h. By comparison, manual labor is intensive, and from experience, it may take days to weeks to pull even a mere 100 studies.

The AI pipeline results demonstrate that DIANA can potentially integrate into existing workflow as an adjunct tool without hindering clinical work times. The AI pipelines ultimately aim to increase clinician confidence for the benefit of patients similar to other automated engineering solutions for medical image analysis problems [23–25]. In the context of clinical workflow, DIANA hopes to avoid obstacles in configuring modalities to send to specific hosts for post-processing through its comprehensive, institutional network presence. Likewise, the system aims to improve the accessibility of AI model predictions on live data through integration with ubiquitously accessible platforms.

Each pipeline’s results have their own nuances worth examining. A potential improvement for the BA estimation pipeline is to display the BA standard deviations for a patient on the Slack® channel based on gold standard literature tables. However, these standard deviations are read from the patient’s age by birthdate, not bone age. Since the institution limits DIANA’s access to patient chart data other than imaging, these metrics are not currently being automatically reported by the system described. Still, many of the patients assessed in Fig. 4 were teenagers, which put their bone age standard deviation at 10 months [20]. It is encouraging to see that even when the DIANA AI model and radiologist readings differed, the difference was almost always within two standard deviations of expectation.

For ICH, it is interesting to note that the average ICH read time by radiologists is about 1 h—much more than one might expect in an emergent setting. One potential explanation lies in the lumping of follow-up ED cases with acute patient scans. Acute scans may be more readily identifiable and would be expected to be read much more quickly, likely within minutes. Beyond this, the AI model used in this work demonstrated remarkable performance with respect to radiologist ground truth classification of bleed types. The AUC of 0.97 suggests that models such as this one have strong, timely potential in increasing clinician confidence in reads. This is further celebrated by the notion that there are always increasingly excellent models on the horizon.

It is important to note that the accuracy of latency calculations for radiologists in this work is limited by workflow realities. In the context of emergent cases such as ICH, it is entirely likely that the reading radiologist would have called the attending ED physician to report any acute findings. The practical latency in a case like this would be the time from imaging completion to the time of such phone call. However, given the challenge of tracking such reports, the closest institutional surrogate available was the time of initial radiologist report creation. When looking at latency with the endpoint defined as time of report completion, the longer times suggest another key component of workflow: radiologists are responsible for assessing innumerous other features in imaging studies beyond singular findings.

In this work, freely available models were integrated with DIANA and evaluated. However, there is no restriction on model integration beyond the abstraction of models to accept an arbitrary image-related input and provide an accessible output result. This opens the door for DIANA to be used in conjunction with commercial models, if desired, with subsequent ability to refine those models as prospective results are obtained and analyzed. With the flexibility of model choice, DIANA could improve delivery of AI models from diverse trades to clinical settings.

Creating PHI-sensitive interinstitutional datasets is a high priority in prospective clinical translational imaging research [26]. Overall, many healthcare environments could benefit from AI augmentation including automated screening for stroke on CT or magnetic resonance imaging, chest XR evaluation, and detection/grading of neoplastic pathology [27, 28]. Although AI training and deployment are obvious targets, there are also many other research and quality assurance applications for readily available imaging data and metadata.

For example, the radiation dose monitor has offered a comprehensive quality-assurance system by which annual national dose guidelines may be readily maintained. Prospective considerations could include meticulous monitoring of pediatric imaging radiation dose on account of the increased radiation morbidity and mortality associated with pediatric patients [29, 30].

DIANA strives to achieve maximum usability for clinicians and researchers while minimizing workflow disruptions that could influence acceptance. From the perspective of a healthcare system, there is room for continuous improvement in DIANA’s components. As an example, the present AI pipelines leverage Slack® as a prototype endpoint, substitutable for another endpoint. That said, other clinical environments may benefit more by leveraging an existing institutional, real-time notification system. Ideally, a system such as DIANA would integrate with electronic medical record (EMR) services as well—mPower serves as a partial EMR surrogate at the present institution. However, tighter integration with established clinical systems also shifts DIANA away from its strengths in rapid ad hoc use and into the domain of established commercial medical informatics products.

Anecdotally, once appropriate approvals were secured, a new DIANA system was integrated with a local clinic’s radiology PACS and downloading image data in a matter of hours. However, if anything, this demonstrated how disproportionately challenging it is to navigate bureaucratic approval to interface directly with PACS due to requirements for maintaining PHI confidentiality and minimizing potential impact on clinical IT workloads. Medical imaging is notoriously difficult to completely anonymize given the vast variability in DICOM header field utilization. From an operations standpoint, the risks associated with batch access to the hospital PACS are non-trivial. Extremely aggressive batch queries during high clinical volume times can result in significant system latency. And large batch queries can result in explosion of local or network file storage. Even dedicated resources can be overwhelmed by a single DIANA request that results in thousands of CT scans and hundreds of gigabytes of fulminant storage [31].

The final discussion this work will touch upon is in navigating some of the real-world challenges of deploying a system such as DIANA into enterprises that are understandably resistant to such tools. In making imaging data more accessible via DIANA, it was important to address the degree to which image PHI could be accessed by various users in the data retrieval process from PACS as well as the network location of the data at all points. In addressing these concerns, the DIANA research team worked closely with the institutional information security team to thoroughly explore these points. This collaborative discussion was streamlined through several means:

• i.

  Outlining system functionality and associated image PHI flow through system diagrams such as Fig. 1. Such depictions provide additional assurance to all parties that there are no invasive modifications required within PACS itself.

• ii.

  Review of storage locations of images (e.g., behind institutional firewall on the network, external machine) and possible degrees of PHI associated with images at all access points.

• iii.

  Manual comparison of random imaging studies and modalities post-anonymization via DIANA and prior institutional gold standard (e.g., manually “hand scrubbing” away PHI by technical expert) to, at minimum, establish anonymization parity between DIANA and original anonymization means. At the local institution, DIANA anonymization was even shown to be slightly more meticulous compared to traditional anonymization, with kudos to Orthanc’s capabilities.

By working through these valid concerns, the research team was able to bring a live DIANA setup to fruition. Establishment of this work aims to promote more wide-spread acceptance of such systems in the future.

Conclusion

DIANA empowers big data projects with image access and new routes of hospital communication that can extend AI applications within the academic medical center environment. Moreover, the facilitation of AI delivery to clinical settings could reduce workflow burdens on clinicians.

Funding

This research was supported by the department of diagnostic imaging at the local institution.

Availability of Data and Material

Any experimental data mentioned above is available upon request to authors.

Code Availability

The repository of interest is publicly available at https://github.com/derekmerck/diana2.

Declarations