Abstract
The aim of this study was to determine the feasibility of automated detection of adrenal nodules, a common finding on CT, using a newly developed search engine that mines dictated radiology reports. To ensure Health Insurance Portability and Accountability Act compliance, we utilized a preexisting de-identified database of 32,974 CT reports from February 1, 2009 to February 28, 2010. Common adrenal descriptors from 29 staff radiologists were used to develop an automated rule-based algorithm targeting adrenal findings. Each sentence within the free text of reports was searched with an adapted NegEx negation algorithm. The algorithm was refined using a 2-week test period of reports and subsequently validated using a 6-week period. Manual review of the 3,693 CT reports in the validation period identified 222 positive reports while the algorithm detected 238 positive reports. The algorithm identified one true positive report missed on manual review for a total of 223 true positive reports. This resulted in a precision of 91% (217 of 238) and a recall of 97% (217 of 223). The sensitivity of the query was 97.3% (95% confidence interval (CI), 93.9–98.9%), and the specificity was 99.3% (95% CI, 99.1–99.6%). The positive predictive value of the algorithm was 91.0% (95% CI, 86.6–94.3%), and the negative predictive value was 99.8% (95% CI, 99.6–99.9%). The prevalence of true positive adrenal findings identified by the query (7.1%) was nearly identical to the true prevalence (7.2%). Automated detection of language describing common findings in imaging reports, such as adrenal nodules on CT, is feasible.
Keywords: Data mining, Radiology information systems (RIS), Natural language, Processing, Computed tomography, Radiology reporting, Adrenal nodules, Negation algorithm, Unstructured radiology reports
Background
Adrenal nodules are commonly encountered on cross-sectional imaging, detected in up to 5% of CT examinations [1]. Approximately two third of these nodules can be diagnosed at initial presentation on CT and usually represent benign findings such as adenomas or myelolipomas [2]. The remaining one third of these nodules are indeterminate by imaging criteria but are also most likely benign in etiology, both among patients with no known malignancy as well patients who subsequently develop malignancy [3]. Despite this, subspecialty management guidelines recommend further testing to exclude either functional adrenal lesions or metastatic disease with additional laboratory testing up to 5 years following detection, repeat imaging at intervals ranging from 3 to 12 months following detection, and specialist referrals [4–8]. The lack of clear consensus among these guidelines poses a challenge both for radiologists and ordering physicians on how to best report and manage adrenal nodules. As imaging utilization increases nationally, the number of patients with adrenal nodules detected on cross-sectional imaging will increase correspondingly, and health care utilization associated with these findings will also grow.
Free text searches using natural language processing (NLP) have been utilized since the 1960s [9] in a variety of applications, including medicine [10–12] as well as in the field of radiology [13]. Previous works have demonstrated the ability to determine positive findings by using NLP to address negation within clinical reports [14–16].
We seek to determine if it is possible to automate detection of adrenal nodules on imaging reports at our institution using NLP in order to quantify their prevalence. Eventually this query can be used to perform a directed chart review evaluating the clinical significance of these nodules through follow-up, the results of which may help guide both radiologists and ordering physicians on the optimal reporting and management of adrenal nodules.
Methods
To ensure Health Insurance Portability and Accountability Act compliance, we utilized a preexisting de-identified database of radiology reports approved by our Institutional Review Board that is described in detail below.
Research Database
For data mining, we created a separate database containing 32,974 CT examinations of the chest, abdomen, thoracic spine, and lumbar spine over a 13-month period (February 1, 2009 to February 28, 2010). These examinations, including thoracic and lumbar spine CT studies, were selected as they all include the adrenal glands in the field of view. All CT examinations were performed at a single tertiary care institution located in a major US city with over 530 hospital beds and greater than 860 staff physicians. Reports of these examinations were identified with Pathology–Radiology Enterprise Search Tool (PRESTO) [12], a web-based search engine and data mining tool at our institution. PRESTO was used to mine our RIS (GE Centricity RIS-IC, Waukesha, WI, USA) and export anonymized report text along with meta-data into a separate MySQL [13] relational database. A PHP algorithm, in conjunction with regular expressions, was used to query the research database and identify positive reports during the iterative process of algorithm development, which is described below in detail.
Determination of Descriptors
In order to determine what language was commonly employed to describe adrenal findings, we randomly selected ten reports each from 29 staff radiologists in our chest and body sections who are responsible for reading all chest, abdomen, pelvis CT studies, as well as trauma thoracic and lumbar spine CT studies. The objective of this review was to determine commonly used adrenal descriptors in order to guide development of the query. Focus was placed on language that identified discrete adrenal findings. Positive descriptors identified through this search included in decreasing order: nodule (93%), adenoma (86%), mass (79%), lesion (45%), and myelolipoma (24%). Based on prior literature, descriptors such as “thickening” and “nodularity” when used in isolation were excluded due to their non-specific nature [17].
Following the test period, several terms were added to list of target positive terms used by the query including: adenomatous change, adenomatous hyperplasia, metastasis, metastases, and metastatic. The last three terms were included because our population included patients with malignancy, and these lesions met our criteria of true adrenal findings
Algorithm Development
A rule-based algorithm was created to search the free text of the radiology report in order to automatically identify discrete adrenal findings. A sentence-by-sentence search method was chosen in order to limit the false positives associated with searching the report as one large block of text. This approach also had higher specificity than a strict word proximity search method. Once the sentences were parsed, the algorithm would search each sentence for the term “adrenal.” If triggered, the algorithm would then search the same sentence for the singular or plural of one of the target positive terms (Table 1). Abbreviations were not included in the algorithm, as no common abbreviations associated with these descriptors are known. Similarly, spelling errors were not addressed. The combination of “adrenal” plus target positive term (in either order) resulted in the report being tagged as positive
Table 1.
Guidelines utilized in algorithm development
| Target positive terms | |
| Nodule(s) | |
| Mass(es) | |
| Lesion(s) | |
| Adenoma(s) | |
| Adenomata | |
| Myelolipoma(s) | |
| Adenomatous change | |
| Adenomatous hyperplasia | |
| Metastasis | |
| Metastases | |
| Metastatic | |
| Precondition negative terms | |
| Absence | |
| Cannot see | |
| Free of | |
| No | |
| No change | |
| Negative | |
| Not | |
| And without | |
| History section identifiers | |
| Starting terms | End terms |
| History | Technique |
| Clinical statement | Protocol |
| Indication | Methods |
| Clinical details | Procedure |
Subsequently, the algorithm evaluated each positive sentence for negation. We utilized a modified version of the NegEx algorithm [18] which was simplified to eliminate precondition negation terms that do not apply to our report texts such as “no radiographic evidence of.” The precondition negation terms in our algorithm are shown in Table 1. An example of a finding targeted by the negation filter would be: “No focal adrenal masses are seen.” In this case, the word “no” (precondition negation term) negates the positive finding of “adrenal masses.” Similar to the NegEx algorithm, the distance between the negation term and the target positive term was set to a maximum of five words.
Additionally, the algorithm ignored history section of the radiology report by disregarding text found between sets of starting terms and end terms (Table 1). This was done in order to reduce false positive reports where the only positive adrenal finding identified by the query was in the clinical history section.
Algorithm Testing and Validation Design
The algorithm was initially tested on a 2-week period and subsequently validated using a 6-week period. Both periods were selected from the 13-month study database using a random date generator. The validation period was selected as three sets of 2 weeks blocks for a total of 6 weeks as follows: February 15–28, 2009, August 17–30, 2009, and October 12–25, 2009. Findings from the manual review were compared to the algorithm.
Manual Review
Manual review was performed on all CT chest, abdomen, and pelvis reports identified in the 6-week validation period. Specifically, the comments and impression of each report were reviewed for the word adrenal by two researchers (JL and HZ). When identified, the report was searched for the presence of any positive adrenal descriptor including “nodule,” “lesion,” and “mass,” “adenoma,” and “myelolipoma.” Any report with a negation term was considered negative (e.g., “no adrenal nodule”). Any study where a discrepancy was coded between the two reviewers was settled by consensus. Interobserver agreement was calculated using a kappa statistic. The results of the manual review were compared to the results of the algorithm, and any discrepancies were re-evaluated by all authors. These results were used to determine true positives, false positives, true negatives, and false negatives from which precision and recall were calculated. In addition, we have reported sensitivity, specificity, positive predictive value, and negative predictive values to provide a clinical context to the performance of the algorithm.
Statistical Evaluation
True positives were defined as reports that both the algorithm and the manual review identified as containing a positive adrenal finding. False positives were defined as reports that the algorithm identified as containing a positive adrenal finding that were not detected by manual review. True negatives were defined as reports that neither the algorithm nor the manual review identified as containing a positive adrenal finding. False negatives were defined as reports that the manual review identified as containing a positive adrenal finding that were not detected by the algorithm. These definitions are illustrated in Table 2.
Table 2.
Definitions for statistical analysis
| Adrenal finding identified on manual review | Adrenal finding absent on manual review | |
|---|---|---|
| Adrenal finding identified on automated query | True positive | False positive |
| Adrenal finding absent on automated query | False negative | True negative |
Precision and recall were calculated using accepted definitions. Precision was defined as the number of true positives divided by total number of reports the algorithm identified as containing a positive adrenal finding (true positives plus false positives). Recall was defined as the number of true positives divided by the number of reports that manual review confirmed as containing a positive adrenal finding. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated using accepted definitions.
Results
A total of 3,693 CT reports, including 2,865 unique patients, were identified during the 6-week validation period. Manual review identified 222 positive reports. There were four cases of disagreement between the two manual reviewers (“Appendix”), which yielded a Cohen’s weighted kappa of 0.990 (95% confidence interval (CI) 0.981–0.999). The algorithm identified 238 positive reports. Of note, the algorithm detected one positive report that was initially missed on manual review but was found to be a true positive upon re-examination. This resulted in 223 true positive reports from 207 unique patients. A breakdown of the prevalence of positive descriptors identified by the algorithm in the total study population is provided in Table 3.
Table 3.
Prevalence of final list of positive adrenal descriptors identified by algorithm (among 3,826 radiology reports)
| Term | Total # of reports containing term | % of positive reports (n = 217) | % of total repots (n = 3,693) |
|---|---|---|---|
| Nodule | 87 | 40.09 | 2.36 |
| Nodules | 10 | 4.61 | 0.27 |
| Mass | 24 | 11.06 | 0.65 |
| Masses | 4 | 1.84 | 0.11 |
| Lesion | 41 | 18.89 | 1.11 |
| Lesions | 16 | 7.37 | 0.43 |
| Adenoma | 100 | 46.08 | 2.71 |
| Adenomas | 15 | 6.91 | 0.41 |
| Adenomata | 8 | 3.69 | 0.22 |
| Myelolipoma | 10 | 4.61 | 0.27 |
| Myelolipomas | 1 | 0.46 | 0.03 |
| Adenomatous change | 1 | 0.46 | 0.03 |
| Adenomatous changes | 2 | 0.92 | 0.05 |
| Adenomatous hyperplasia | 9 | 4.15 | 0.24 |
| Metastasis | 19 | 8.76 | 0.51 |
| Metastases | 7 | 3.23 | 0.19 |
| Metastatic | 21 | 9.68 | 0.57 |
There were 217 true positives, 21 false positives, 3,449 true negatives, and 6 false negatives. This resulted in a precision of 91% (217 of 238) and a recall of 97% (217 of 223). The sensitivity of the query was 97.3% (95% CI, 93.9–98.9%), and the specificity was 99.3% (95% CI, 99.1–99.6%). The PPV, or the probability that identification of an adrenal nodule by the algorithm was confirmed with identification of an adrenal nodule on manual review, was 91.0% (95% CI, 86.6–94.3%). The NPV, or the probability that the absence of identification of an adrenal nodule by the algorithm was confirmed with identification of an adrenal nodule on manual review, was 99.8% (95% CI, 99.6–99.9%).
The query identified 217 true positive from among 202 unique patients, which resulted in a calculated prevalence of 7.1% (202 of 2,865). This did not differ significantly from the true prevalence in our population of 7.2% (223 of 2,865). Analysis of the data excluding the true positive that was initially missed from the manual review did not reveal any significant change in precision, recall, sensitivity, specificity, PPV, or NPV.
Examples of false positive and false negative reports identified by the query are provided in Table 4. The most common cause of a false positive was due to the utilization of positive descriptor in a separate sentence from the word adrenal, which we refer to as a “positive finding detection error.” The most common cause of a false negative was what we refer to as a “complex sentence error” whereby multiple organs were described within the same sentence as the word adrenal whereby the algorithm failed to associate the positive descriptor (e.g., lesion) with the appropriate non-adrenal organ.
Table 4.
Examples of false negatives and false positive from final query
| False negatives (6 total) | False positives (21 total) | ||
|---|---|---|---|
| Report text | Explanation | Report text | Explanation |
| There is a 3 cm benign adrenal myolipoma. | Spelling error: Algorithm searches for “myelolipoma”, and this term was misspelled within the original report text | Metastatic pheochromocytoma of the right adrenal post adrenalectomy. | Error due to missing “History”: heading for the history section of the report. The report was labeled negative on manual review as no positive findings were reported in the body of the report |
| No adrenal masses are seen probable tiny right adrenal myelolipoma. | Missing punctuation leading to a sentence-based negation error (i.e., negated finding “no adrenal masses” is in the same sentence as the positive terms of “adrenal myelolipoma”) | The adrenal glands appear thickened but discrete mass. | Radiologist dictation error: There is a missing negation term “no” before “discrete mass”. Manual review interpreted this as negative |
| Up to five small bilateral pulmonary nodules are unchanged without evidence of new lung nodule including: 6 mm left lower lobe nodule image 17 3 mm left upper lobe nodule image 20 5 mm right lower lobe nodule image 26 3 mm left lower lobe nodule image 29 4 mm right lower lobe nodule image 34 Limited cuts into the upper abdomen demonstrate a stable 15 mm fat containing left adrenal nodule suspected adenoma which is partially imaged. | Missing punctuation leading to a sentence-based negation error. Sentence contains a positive finding (in bold), but this was negated by the negation algorithm due to italicized phrase, which contains “without” and “nodule”. A period after “image 34” would have allowed the algorithm to correctly identify this report as having a positive adrenal finding | The high attenuation masslike abnormality in the left adrenal probably represents an adrenal hematoma which is resolving. | Adrenal hematoma was considered as a negative finding for the purpose of this paper. However, this sentence was marked positive secondary to presence of “masslike” in association with “adrenal” |
| There is nodularity of the right adrenal gland and some thickening in the left. IMPRESSION: 2. Bilateral nodularity in the adrenal glands likely adenomatous/hyperplastic changes. | Positive finding detection error. Algorithm searches for “adenomatous changes” but was unable to identify “adenomatous/hyperplastic changes” due to insertion of “/hyperplastic” | The spleen pancreas both adrenal glands and both kidneys are unremarkable aside from a 7 mm low-density lesion in the lower pole of the left kidney which is too small to characterize but probably benign. | Complex sentence error. Multiple organs were described within a same sentence, and the algorithm failed to associate the positive term “lesion” with the appropriate organ (left kidney). Also, the term “unremarkable” was not detected by the algorithm for negation. A total of 18 cases had similar error. Some examples are shown below |
| Nodular thickening of the right adrenal gland measuring 2.2 × 1.7 cm on image 98 stable in size. The CT attenuation characteristics of this lesion are indeterminate (with an average density of 25 HU). The prior MRI tissue characteristics suggested an adenoma as the most likely diagnosis. | Positive finding detection error. Because the terms “adrenal,” “lesion,” and “adenoma” were in separate sentences, this was missed by the sentence-based detection algorithm | The liver spleen gallbladder biliary tree pancreas adrenal glands and kidneys are unremarkable with the exception of the low attenuating lesion in the interpolar aspect of the right kidney IV representing a small renal cyst. | Complex sentence error |
| There is continued nodular thickening of the left adrenal gland with a maximum diameter of 8 mm unchanged since prior study. The appearance is benign and may represent small adenoma or nodular hyperplasia. | Positive finding detection error. Similar to above where “adrenal” and “adenoma” were in separate sentences | The adrenal glands and kidneys are unremarkable except for bilateral low attenuation renal lesions which are subcentimeter in size likely small cysts. | Complex sentence error |
| FINDINGS: The liver spleen pancreas adrenal glands and kidneys are normal except for a subcentimeter cystic lesion in the right kidney which is too small to characterize. | Complex sentence error | ||
| There is a 10 × 17 mm low attenuation soft tissue mass adjacent to the lateral limb of the left adrenal gland that probably represents a lymph node that is increased in size from the prior exam. | Complex sentence error. The spatial relationship was not accounted for with our algorithm to determine that the “mass” is “adjacent” and not part of the adrenal gland | ||
Discussion
This algorithm demonstrates that it is feasible to automate the detection of language describing discrete adrenal findings using reports of cross-sectional imaging at our institution with high precision, recall, sensitivity, and specificity. While the algorithm has yielded promising outcomes, it can be further refined to decrease both the number of false positive and false negative reports. The overwhelming majority of false positives were caused by a complex sentence error. Specifically, these sentences generally listed a number of unaffected organs, such as “…liver spleen gallbladder biliary tree pancreas adrenal glands and kidneys are unremarkable” followed by a finding in one of the previously mention organs such as “…with the exception of the lesion in the interpolar aspect of the right kidney.” This type of error could be reduced with both more advanced NLP techniques as well as by incorporating other negation techniques used by programs like NegExpander [19]. In the previous example, an advanced algorithm would assign “lesion” to the appropriate organ, the right kidney, rather than the adrenal gland.
False negatives were usually due to a positive finding detection error where the description of an adrenal finding was spread over multiple sentences. This type of error is the biggest limitation of the sentence-by-sentence search methodology of the algorithm. In future iterations, this could be addressed by modifying the algorithm to either search adjacent sentences once “adrenal” has been identified or to use a proximity search window. Both of these options would allow for multi-sentence adrenal findings. However, they could also hypothetically lead to an increased false positive rate. Similarly, we could incorporate laterality detection into the algorithm to further reduce false negative rates.
Two cases identified by the query are notable. The first was a true positive case detected by the query but not identified on the manual review. Importantly, there was no significant change in precision, recall, sensitivity, specificity, PPV, or NPV when this case was included or excluded from the analysis. The second was a false positive case identified by the query due to the utilization of the term “mass like” in describing a “hematoma” (Table 4). Hematoma and hemorrhage were not included in the final list of positive descriptors because this diagnosis has a low prevalence and is strongly associated with acute trauma or multisystem failure requiring emergent follow-up rather potential follow-up [20], which was the stated goal of the query. Other cases of adrenal hematoma or hemorrhage within the final 6-week period were not identified by the query because of the absence of positive descriptors in the same sentence as this diagnosis. The inclusion of this term may be an area of potential refinement for the algorithm moving forward.
The true prevalence of adrenal nodules on imaging in our patient population (7.2%) is slightly higher than the 5% reported in the medical literature [2, 21]. This discrepancy most likely reflects the inclusion of adrenal findings such as metastases and patients with known malignancy in our study compared to these other articles, which excluded this diagnosis and patient population. Because the objective of our algorithm was to automate detection of true adrenal findings, we chose to include adrenal metastases. It is reassuring, however, that our prevalence of 7.2% does not exceed the prevalence of 9% reported in large-scale autopsy series [22].
Limitations of this study include a limited sample of 3,693 CT reports including 2,865 unique patients over a 6-week time period. The algorithm was tested on a 2-week sample during a single month. There may be other terms used to describe adrenal nodules at our institution that were not included in our limited sampling of ten reports per attending that could have increased the sensitivity of our algorithm. However, this seems unlikely from a clinical radiology perspective. Finally, we sampled almost all of the staff physicians in our chest and body radiology departments who are responsible for reading all chest, abdomen, pelvis CT studies, as well as trauma thoracic and lumbar spine CT studies rendering this algorithm sensitive to detect adrenal findings at our institution. As a result, this automated algorithm may have different precision, recall, sensitivity, and specificity at other institutions that may employ different phrases to describe adrenal findings.
Adrenal nodules are commonly encountered on cross-sectional imaging with CT, often incidentally. Most of these lesions can be diagnosed on initial presentation and represent benign diagnoses such as adenomas. However, subspecialty management guidelines recommend further testing to exclude either functional adrenal lesions or metastatic disease including additional laboratory testing up to 5 years following detection, repeat imaging at intervals ranging from 3 to 12 months following detection, and specialist referrals. The lack of clear consensus among these guidelines poses a challenge both for radiologists and ordering physicians on how to best report and manage adrenal nodules. As imaging utilization increases nationally, the number of patients with adrenal nodules detected on cross-sectional imaging will increase correspondingly, and health care utilization associated with these findings will also grow. Eventually this query can be combined with electronic medical record data searches in order to determine the clinical significance of these adrenal nodules through resultant follow-up and to automatically provide this follow-up to both radiologists and ordering physicians. These evidence-based data on large-scale patient populations could then be used to adjust the reporting and management of adrenal nodules in order to optimize patient care.
Conclusions
Automated detection of language describing common findings on imaging reports, such as adrenal nodules on CT, is feasible.
Appendix
Table
Table 5.
Phrases disagreed on during the manual review
| Phrase | Included/excluded | Reason |
|---|---|---|
| 10 mm nodule in the left adrenal image 25/6 is grossly stable from the 7/26/99 study image 12/3 likely an adenoma | Included | Missed by one reviewer |
| There is redemonstration of prominent right adrenal gland given the stability over 4 years likely represents small adenomas | Included | Missed by one reviewer |
| The high attenuation masslike abnormality in the left adrenal probably represents an adrenal hematoma which is resolving | Excluded | Hematoma was neither included in the original query nor considered an adrenal finding that merited clinical follow-up |
| There is minimal enlargement and mild nodularity of the lateral limb of both adrenal glands. The prior examination reports a history of biochemical abnormality consistent with primary hyperaldosteronism. Although the imaging appearance is not diagnostic, tiny adenomas may not be visible in this nondedicated study. At clinical discretion and if not previously performed, dedicated imaging with adrenal protocol may provided additional information | Excluded | No discrete finding is visualized on the current examination. The terms thickening and nodularity when used in isolation were excluded due their non-specific nature |
References
- 1.Kloos RT, et al. Incidentally discovered adrenal masses. Endocr Rev. 1995;16(4):460–484. doi: 10.1210/edrv-16-4-460. [DOI] [PubMed] [Google Scholar]
- 2.Song JH, Chaudhry FS, Mayo-Smith WW. The incidental adrenal mass on CT: prevalence of adrenal disease in 1,049 consecutive adrenal masses in patients with no known malignancy. AJR Am J Roentgenol. 2008;190(5):1163–1168. doi: 10.2214/AJR.07.2799. [DOI] [PubMed] [Google Scholar]
- 3.Song JH, Chaudhry FS, Mayo-Smith WW. The incidental indeterminate adrenal mass on CT (>10 H) in patients without cancer: is further imaging necessary? Follow-up of 321 consecutive indeterminate adrenal masses. AJR Am J Roentgenol. 2007;189(5):1119–1123. doi: 10.2214/AJR.07.2167. [DOI] [PubMed] [Google Scholar]
- 4.Grumbach MM, et al. Management of the clinically inapparent adrenal mass (“incidentaloma”) Ann Intern Med. 2003;138(5):424–429. doi: 10.7326/0003-4819-138-5-200303040-00013. [DOI] [PubMed] [Google Scholar]
- 5.Berland LL, et al. Managing incidental findings on abdominal CT: white paper of the ACR incidental findings committee. J Am Coll Radiol. 2010;7(10):754–773. doi: 10.1016/j.jacr.2010.06.013. [DOI] [PubMed] [Google Scholar]
- 6.Management of the Clinically Inapparent Adrenal Mass (Incidentaloma): National Institutes of Health State-of-the-Science Conference Statement February 4–6, 2002. 2002. Available from: http://consensus.nih.gov/2002/2002AdrenalIncidentalomasos021html.htm.
- 7.Zeiger MA, et al. The American Association of Clinical Endocrinologists and American Association of Endocrine Surgeons medical guidelines for the management of adrenal incidentalomas. Endocr Pract. 2009;15(Suppl 1):1–20. doi: 10.4158/EP.15.S1.1. [DOI] [PubMed] [Google Scholar]
- 8.Graham DJ, McHenry CR. The adrenal incidentaloma: guidelines for evaluation and recommendations for management. Surg Oncol Clin N Am. 1998;7(4):749–764. [PubMed] [Google Scholar]
- 9.Swanson DR. Searching natural language text by computer. Machine indexing and text searching offer an approach to the basic problems of library automation. Science. 1960;132:1099–1104. doi: 10.1126/science.132.3434.1099. [DOI] [PubMed] [Google Scholar]
- 10.Friedman C, Hripcsak G. Natural language processing and its future in medicine. Acad Med. 1999;74(8):890–895. doi: 10.1097/00001888-199908000-00012. [DOI] [PubMed] [Google Scholar]
- 11.Hripcsak G, et al. Unlocking clinical data from narrative reports: a study of natural language processing. Ann Intern Med. 1995;122(9):681–688. doi: 10.7326/0003-4819-122-9-199505010-00007. [DOI] [PubMed] [Google Scholar]
- 12.Sager N, et al. Natural language processing and the representation of clinical data. J Am Med Inform Assoc. 1994;1(2):142–160. doi: 10.1136/jamia.1994.95236145. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Langlotz CP. Automatic structuring of radiology reports: harbinger of a second information revolution in radiology. Radiology. 2002;224(1):5–7. doi: 10.1148/radiol.2241020415. [DOI] [PubMed] [Google Scholar]
- 14.Elkin PL, et al. A controlled trial of automated classification of negation from clinical notes. BMC Med Inform Decis Mak. 2005;5:13. doi: 10.1186/1472-6947-5-13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Huang Y, Lowe HJ. A novel hybrid approach to automated negation detection in clinical radiology reports. J Am Med Inform Assoc. 2007;14(3):304–311. doi: 10.1197/jamia.M2284. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Mutalik PG, Deshpande A, Nadkarni PM. Use of general-purpose negation detection to augment concept indexing of medical documents: a quantitative study using the UMLS. J Am Med Inform Assoc. 2001;8(6):598–609. doi: 10.1136/jamia.2001.0080598. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Benitah N, et al. Minor morphologic abnormalities of adrenal glands at CT: prognostic importance in patients with lung cancer. Radiology. 2005;235(2):517–522. doi: 10.1148/radiol.2352031708. [DOI] [PubMed] [Google Scholar]
- 18.Chapman WW, et al. A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform. 2001;34(5):301–310. doi: 10.1006/jbin.2001.1029. [DOI] [PubMed] [Google Scholar]
- 19.Aronow DB, Fangfang F, Croft WB. Ad hoc classification of radiology reports. J Am Med Inform Assoc. 1999;6(5):393–411. doi: 10.1136/jamia.1999.0060393. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Vella A, Nippoldt TB, Morris JC., 3rd Adrenal hemorrhage: a 25-year experience at the Mayo Clinic. Mayo Clin Proc. 2001;76(2):161–168. doi: 10.1016/S0025-6196(11)63123-6. [DOI] [PubMed] [Google Scholar]
- 21.Bovio S, et al. Prevalence of adrenal incidentaloma in a contemporary computerized tomography series. J Endocrinol Invest. 2006;29(4):298–302. doi: 10.1007/BF03344099. [DOI] [PubMed] [Google Scholar]
- 22.Hedeland H, Ostberg G, Hokfelt B. On the prevalence of adrenocortical adenomas in an autopsy material in relation to hypertension and diabetes. Acta Med Scand. 1968;184(3):211–214. doi: 10.1111/j.0954-6820.1968.tb02445.x. [DOI] [PubMed] [Google Scholar]