Prediction of lung cancer risk at follow-up screening with low-dose CT: a training and validation study of a deep learning method

Training and Validation Datasets

The datasets for training (NLST) and validation (PanCan) have been reported previously.^1,21 In brief, NLST recruited ever-smokers from 33 screening centers, 55 to 74 years of age, who had a history of at least 30 pack-years smoking history and last smoked within the previous 15 years. The median follow-up was 6·5 years. 25,097 participants received repeat annual LDCT screenings. The PanCan study recruited current and former smokers from 8 screening centers across Canada, 50 to 75 years of age, with a 6-year lung cancer risk of at least 2% as determined by the PanCan model, a precursor to the validated PLCO_M2012 model.²² The median follow-up was 5·5 years. Among the 2,350 PanCan participants who received follow-up screenings, 56 individuals were removed because they received investigational screenings before the next scheduled annual screening for suspicious lung nodules. Both datasets included demographics, radiology reports from the most recent follow-up LDCT scan (=“S2” scan) within two years from baseline screening and from the most recent prior scan (=“S1” scan) from all participants (supplement analysis samples).

Study Oversight

Investigators from three institutions, Johns Hopkins University (JHU), BC Cancer (BCC), and the National Cancer Institute (NCI), jointly conducted this double-blinded study using the PanCan study as a validation cohort.²¹ Blinded to the PanCan participant’s cancer outcome, the JHU team submitted ML risk scores to the NCI team. Blinded to the JHU team’s prediction, BCC provided the verified PanCan participant’s cancer outcomes to the NCI team. Both JHU team’s risk scores and BCC’s cancer outcomes were then locked by the NCI team who compared the JHU team’s prediction accuracy to the Lung-RADS³ and NELSON⁵ guidelines. After this evaluation was completed, both JHU team and BCC team were unblinded to the PanCan participants’ cancer outcomes and JHU team’s ML risk scores.

Deep Machine Learning Algorithm

The endpoint was the duration from follow-up S2 screening date to the last date the person was known to be free of pathologically confirmed lung cancer within three years after S2 scan date. Input predictors are listed in Table 1 and Table S1. Observations were censored at the end of year 3. The 3-year period was selected since our intention was to demonstrate the long-term predictive ability of ML algorithm and determine whether an annual screening interval is appropriate in the lowest-risk group. Limiting the follow-up period to three years also increase the probability that incident cancers within this period could be in-part predicted by CT scan findings. Similar to NLST, only non-calcified nodules with size ≥4mm in PanCan were included. Supplementary Table S1 shows how different radiographic variables used in NLST and PanCan were reconciled. Nodule diameter was computed as the average of the longest diameter and the longest perpendicular diameter as measured on a single transverse image (high-spatial frequency, lung “windows”). Nodule margin was dichotomized to spiculation or no spiculation. Attenuation was replaced by a variable with 4 categories: “ground-glass opacity”, “solid”, “semi-solid”, and “others”. Nodule location in PanCan data was converted to the corresponding NLST labels. The American College of Radiology Lung-RADS category was determined for each screen-detected nodule as described previously.²³ The highest Lung-RADS score for the S2 scan was assigned to each individual.

The diameter of a nodule that resolved at S2 was set to 1mm that was considered to be below the scanner detection limit; Similarly, if a nodule was seen in S2 but not in S1, its diameter in S1 was also set to 1mm. Nodule volume was estimated using V=πd³/6 where d is the nodule diameter, and nodule volume doubling time (VDT) in days was calculated for the largest non-calcified nodules from the S2 scan using formula $\text{VDT=}\frac{\text{D}{\text{log}}_{10}2}{3{\text{log}}_{10}(d_2/d_1)}$ where d₁ and d₂ are diameters at S1 and S2 screenings respectively, and D is the number of days between S1 and S2 scan dates. For new nodules with d₂>1mm, we set d₁=1mm. Since this VDT formula does not provide finite values for individuals who did not have any or new nodules with d₂>1mm, these individuals were assigned to an artificial VDT value of MaxVDT+1, where MaxVDT is the maximum of all finite VDT values (>36500). The reason is that larger VDT is considered to have lower risk, and such assigned value allowed us to compute AUC and to compare VDT with other predictors using all PanCan individuals. In the NLST training sample, since the same nodule may be labeled differently at different screenings, VDT cannot be calculated for individuals who had at least two nodules from the same lobe. Thus, VDT was only used to evaluate the ML’s performance on PanCan validation sample where nodule locations were consistently labeled. Two imaging variables unrelated to nodule characteristics were duration of emphysema and duration of cardiovascular disease; both were calculated by the number of days between the first date of reported abnormality (S1 or S2) and the S2 screening date. A new summary interval change variable, Nchg, was created using formula Nchg=∑_{(i: nodules in s2)} C_i ×365/D where the summation is computed over all non-calcified nodules with a size of at least 4mm detected on the S2 screening, and Ci is the number of nodule changes among the following: (1) increase in size, (2) increase in density, (3) new from S1.

The neural network used to develop ML predictor was built using multilayer perceptron (MLP). We used two MLP structures with two hidden layers each. The first one has sizes 5 and 2, and second one has sizes 51 and 8. The cross-entropy loss function with L₂ penalty parameter was used. Weights were optimized using quasi-Newton method and stochastic gradient-based method.²⁴. Following Faraggi and Simon’s suggestion,²⁵ rectified linear units function was used in the network prior to the final layer where survival random forest was used.²⁶ Since data were highly unbalanced, downsampling was used in the ensemble learning. The final output is a normalized continuous score ranging from 0 to 1.

Statistical Methods

This study was designed to analyze the residual survival time since the S2 screening date. Data were collected and analyzed based on conditional distribution that participants had received two CT screenings. The primary analysis was to compare the lung cancer prediction accuracy among three predictors (ML, Lung-RADS, and VDT) in the PanCan validation sample using time-dependent area under the ROC curve (AUC) at cut-off years 1, 2, and 3 after the S2 scan.^27,28 The AUC standard deviation and p-values used to compare two AUCs were computed using inverse probability of censoring weighted estimators²⁹ in 500 bootstrap simulations. Secondary analysis compared cancer incidence rates (=one minus Kaplan-Meier estimates) among high-risk and low-risk subgroups. The high-risk subgroups of ML, Lung-RADS and VDT were defined by >0·3, >3 (i.e., 4A/4B/4X), and <400 days respectively. The corresponding low-risk subgroups were defined by <0·1, <3, and >600 days respectively. Missing demographics were imputed using multiple imputation method. Missing nodule data were imputed using the last-observation-carrying-forward method.

An exploratory analysis was performed to compare the time-dependent positive predictive value (PPV) of all predictors under common sensitivity levels using PanCan dataset. The PPV was computed using Bays rule PPV(t)=(time-dependent sensitivity at time t)*(1-S(t)) /P(test positive) where S(t) is the Kaplan-Meier estimate of cancer-free probability at time t=1, 2, or 3 years.

Since mortality data were not used in ML algorithm development, exploratory survival analyses were performed to study whether ML algorithm detected lung cancers that were biologically more aggressive using the NLST dataset only as mortality data was not available in PanCan. Deaths not from lung cancer were censored. ML and Lung-RADS were compared by: (1) the mortality rate within high-risk and low-risk subgroups; (2) the added value of ML to Lung-RADS’ high-risk subgroup to detect aggressive cancers; (3) the added value of Lung-RADS to ML’s high-risk subgroup to detect aggressive cancers. In all analyses, the hazards ratio (HR) was used to compare subgroups, and two-sided p-values were reported from the log-rank test.

Role of funders

Funders have no role in the study. Huang, Lam, and Aka-Khattra have access to all data. Huang, Lam and Lin were responsible for the final decision to submit the manuscript.

Study Samples

Table 1 summarizes participant’s demographics and radiology reports. Compared to the NLST, PanCan participants were older at follow-up S2 screening, were more likely to be current smokers, and had more individuals with ≥4mm nodules at both S1 and S2 screenings. A total of 283 Lung-RADS high-risk nodules were found in 235 PanCan S2 scans.

Machine Learning Predictors

An online web-based tool at https://www.caced.jhu.edu is available to provide ML score and estimated cancer incidence probabilities within 3 years after the S2 scan date.

Primary analysis

In the NLST training sample, time-dependent AUC ± standard deviation values of ML are 0·99±0·003, 0·985±0·005, and 0·983±0·005 at years one, two, and three, respectively (Figure 1). They are higher than the corresponding AUC values 0·909±0·019, 0·856±0·022, and 0·811±0·024 from Lung-RADS (all p<0·001). The blinded ML prediction was originally performed on all 2,350 PanCan individuals. But final analysis excluded 56 individuals who received investigational screenings (see supplement appendix for details). Final AUC values from ML were 0·968±0·013, 0·946±0·013, and 0·899±0·017 (Figure 1). They are higher than the updated Lung-RADS AUC values of 0·944±0·016, 0·908±0·019, and 0·858±0·022 (p=0·202, 0·048, 0·028); and VDT AUC values of 0·830±0·030, 0·777±0·029, and 0·762±0·026 (all p<0·001).

Secondary analysis

Using Lung-RADS, 5%(1,333/25,097) of the NLST participants were classified as high-risk (4A/4B/4X) compares to 2%(392/25,097) using ML>0·3 criterion (Table 2 and Figure 2). Among those that were deemed high-risk, Lung-RADS identified 77%(274/358) and 66%(308/464) of all lung cancers diagnosed within 12 and 24 months respectively. The corresponding figures using ML were 87% (313/358) and 74·1%(344/464) (Table 2). In the PanCan study, 10%(235/2,294), 10%(221/2,294) and 8%(192/2,294) would be deemed high-risk by Lung-RADS, ML and VDT respectively. Among those that were deemed high-risk, Lung-RADS identified 89%(48/54) and 81%(58/72) of all lung cancers diagnosed within 12 and 24 months respectively. The corresponding figures were 94%(51/54) and 85%(61/72) with ML, and 57%(31/54) and 50%(36/72) with VDT (Table 2).

Using Lung-RADS, 93% (23,458/25,097) of the NLST participants were classified as low-risk (1 or 2). With ML≤0·1, 97%(24,234/25,097) are classified as low-risk. The proportion of low risk participants with lung cancer diagnosed within 24 months are 0·62%(145/23,458) and 0·21%(51/24,234) for Lung-RADS and ML≤0·1 respectively. For PanCan, the proportion of participants classified as low-risk would be 74%(1,698/2,294), 55%(1,266/2,294) and 90%(2,075/2,294) for Lung-RADS, ML≤0.1 and VDT respectively. The proportion of low-risk individuals diagnosed with lung cancer within 24 months are 0·53%(9/1,698), 0·16%(2/1,266) and 1·49%(31/2,075) respectively (Table 2).

Exploratory analyses

When the common sensitivity level was set between 0·85 and 0·90, using all 1,070 PanCan individuals who had at least one non-calcified nodule with size 4mm or above in their S2 scans, ML has higher time-dependent PPV than Lung-RADS and VDT in all three years (Figure S1).

Among all 353 and 327 lung cancers included in ML and Lung-RADS’ high-risk subgroups, 231 and 206 of them were in pathological stage I. Higher lung cancer mortality rates were observed from ML high-risk subgroups as compared to the Lung-RADS high-risk subgroup (supplementary Figure S2). ML low-risk subgroup had uniformly lower mortality rate compared to Lung-RADS low-risk subgroups. Within both Lung-RADS=4A/4B/4X and 4B subgroups, individuals with ML high-risk scores also had significantly higher mortality rates (HR=16·07 and 31·79, both p<0·001, Figure 3, A and B). However, Lung-RADS could not stratify the mortality rate within the ML high-risk subgroup (Figure 3, C and D).