The lung CT datasets selected in the present study were obtained from LIDC-IDRI (22) (https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI). The LIDC-IDRI dataset contained a total of 1,018 CT images of patients with relevant clinical information. These CT images were marked by four physicians to indicate the location of the lung nodules, the edge contour information, the degree of benign and malignant characteristics and the quantitation of different signs. Pulmonary nodules with different malignant phenotypes exhibited a number of morphological characteristics. In the LIDC-IDRI dataset, the malignant phenotype of the lung nodules was quantified by the physician using specific numbers and the quantification range was set to 1–5, according to the definition of dataset (14,16). The probability of malignancy was indicated as follows: i) Malignancy 1, high probability of being benign; ii) malignancy 2, moderate probability of being benign; iii) malignancy 3, indeterminate probability being benign; iv) malignancy 4, moderate probability of being malignant; and v) malignancy 5, high probability of being malignant.

As nodules in the lung parenchyma are generally small in diameter, the remaining parts of the CT image may affect the classification results; therefore, the lung nodule images were extracted according to the required annotations. The annotation file records comprised the edge information used by the doctor to mark the position of the nodule. Based on the edge information, the center position of the nodule was determined and a 64×64 pixel² image located at the center of the lung nodule was obtained as experimental data. The computer-aided diagnostic system automatically extracts the characteristics of the pulmonary nodules and assesses the malignant phenotype of the nodules, which improves the prediction efficiency. In the present study, the LIDC-IDRI dataset was used for model training. Initially, the 64×64 pixel² regions of interest (ROI) containing the pulmonary nodules were extracted according to the annotation file. The extracted ROI images of the pulmonary nodules were used as the input of the three models, and the corresponding pulmonary malignant phenotype of the nodule was extracted.

The autoencoder (23) is an unsupervised learning method that automatically maps input data into the hidden layers and reconstructs the output of the hidden layers to the same shape as the raw input data. It locates hidden features from specific inputs and extracts them to represent the original input. The process from the input to the hidden layers is known as encoding, whereas the process of reconstruction from the hidden layers is known as decoding. The difference between the raw and the reconstruction input data is the reconstruction error. The autoencoder assumes that the distribution representation of the hidden layers can capture the main factors of change within the data.

Following lung nodule extraction using the DAE, the Softmax function was used to classify the nodular malignancy, assuming the training sample {(x₁,y₁), (x₂,y₂), …, (x_n,y_n)}, where x_i is the lung nodule image data, and y_i is the corresponding malignancy score. The classification of a lung nodule by the Softmax function requires estimation of the probability corresponding to each malignancy score. The formula used to calculate the probability was:

where ω is a parameter used in the model. The degree of malignant phenotype corresponding to the probability value was selected as the predicted malignancy of the lung nodule image x_i. The process and feature extraction by Denoising Autoencoder (Fig. S1) are further described in Data S1.

CNN is a feedforward deep neural network, which consists of convolution operation. CNN calculates matching levels between the images and labels by extracting the feature representation of images (24–27). The deeper of network layers is, the stronger CNN representation is; however, it has been shown that the network degenerates as CNN increase in depth, and this increase results in a decrease in Acc. ResNet network is another typical CNN that adds a shortcut connection and an identity map to the network using residual learning (27). The extraction capability of network features is enhanced, and the network performance gradually improves as the network deepens (25).

In the present study, the sum of the cross-entropy loss function and the regularization loss function were used as the loss function of the residual network:

where y_i is the true label of the pulmonary nodule, ŷ_i is the prediction label for the pulmonary nodule, p is the regularization factor and θ, the network model parameter. The features extracted using ResNet-18 were also classified using the Softmax function. The process of feature extraction by ResNet-18 (Fig. S2) and the hyper-parameters of ResNet-18 (Table SI) are further described in Data S2.

The handcrafted features of the pulmonary nodules were used to classify the lung nodules. Due to the particularity of the medical images, only SF and TF were used to classify the lung nodules (28,29). The geometric parameter method was used to determine the shape of the lung nodules, and the Gray Level Co-occurrence Matrix was used to determine the texture of the nodules.

Following extraction of TF and SF, the extracted features were concatenated using a set of feature vectors. The multi-class machine learning method was selected to classify the features of the lung nodules. In the present study, the K-Nearest Neighbor (KNN) method was selected to classify the extracted handcrafted features. KNN is a method for classifying targets based on feature space training examples, and consists of two components, learning and classification. In the present study, five grades were used for the classification of the malignant phenotype and an additional five categories were employed as vectors. All candidate feature vectors were classified by KNN and divided into five categories, representing the five malignancy grades. The process and some typical handcrafted features (Fig. S3) are further described in Data S3

The three feature methods of unsupervised learning, supervised learning and handcrafted feature combination exhibited different classification abilities for the classification task, and different classification performances. If a single classifier is used alone, the generalization ability of the classifier may not be strong. Three classifiers were combined by certain rules and the combined model could make full use of the features extracted by the three methods. This approach may improve the Acc and generalization ability of the model and could decrease the risk of the model leading to local minimum points in the learning task during the training process. Fusion of the multi-classifiers resulted in cascade and parallel forms. The parallel mode adjusts the base classifiers into a parallel action. Therefore, in the present study, the three classifiers were combined in parallel.

Using parallel fusion, weighted voting was based on the error rate (30–32). The ensemble classification model can utilize the features of each classifier and further ensure flexibility between the different feature coefficients of each classifier (Table I).

In the present study, models were assessed based on accuracy (Acc), precision (Pre) and sensitivity (Sen). Acc is the correct proportion of the total sample, indicating the classification ability of the models. Pre is the positive predictive value, representing the proportion of true positives in the positive samples. Sen is the true positive rate, which is the proportion required to make a true positive prediction. Larger values indicate a better performance of classification. The calculation formula used were as follows:

The pulmonary nodule classification was defined by the following parameters: TP_i (true positive) indicated the probability that the malignancy i was classified as I; FN_i (false negative) indicated the probability that the malignancy i was not classified as I; FP_i (false positive) indicated the probability that a malignancy that was not i was not classified as I; and TN_i (true negative) indicated the probability that a malignancy that was not i was classified as i (i=1, 2, 3, 4 and 5). The detailed data description and some nodule samples (Fig. S4) are shown in Data S4. In order to determine sensitivity and accuracy trends, receiver operating characteristic (ROC) curves and area under the curve (AUC) values were generated (Fig. 2). The ROC curves comprehensively demonstrate the association between precision and sensitivity, while the AUC value is the area under ROC curves. The larger the AUC value, the better the classifier performance (32).

The proposed method exhibited an average Acc of 93.10%, a Pre of 83.85% and a Sen of 81.75% for identification of the malignant phenotype of the lung nodules in the test set (Table II).

During the training phase, 10-fold cross-validation was used to obtain the Acc of the three classifiers. ResNet-18 exhibited high Acc, whereas DAE exhibited notably stable Acc (Fig. 3). Although the Acc of the handcrafted features was relatively low, it could describe the specific morphological and TF of the pulmonary nodules. In order to validate these findings, an ablation experiment was performed by removing the single methods (Table III).

The present study further compared the performance of the proposed method based on the weighted voting classification method and similar classification methods (13,33–36) used under the same conditions (Table IV).

The results indicated that the Acc, Sen and AUC of the classification of the malignant phenotype of the pulmonary nodules were optimal in the methods used in the present study. The proposed method exhibited a higher classification performance regarding the pulmonary nodules, which could be used for their accurate assessment, thereby supplying an auxiliary suggestion for the judgment of the medical practitioner. The Pre of the method used in the present study was lower compared with the Multi-Crop CNN (MC-CNN) method proposed by Shen et al (34), as MC-CNN captures more prominent features of the nodule via multiple cropping strategies. However, MC-CNN reduces Sen to ensure Pre and does not improve the classification Acc of pulmonary nodules. Therefore, complex convolution networks may result in longer time periods.

In the present study, a number of common supervised CNN models were selected for comparison, namely, GoogleNet, VGGNet and SENet (24–26). The CNN models were compared using different pre-training processes with the ResNet-18 under the same conditions (Table V).