The information of lung cancer patients admitted to the First Affiliated Hospital, Zhejiang University School of Medicine (Hangzhou, China) who received 2–4 cycles (3 weeks per cycle) of adjuvant or neoadjuvant chemotherapy between January 2016 and February 2020 was extracted from a single-center EHR system in December 2020. The inclusion criteria were as follows: i) A clear pathological diagnosis of NSCLC; ii) a detailed medical history; iii) regular chemotherapy cycles; iv) detailed hematological indicators; and v) detailed records of interventions in patients with NSCLC with AEs. A total of 403 patients were ultimately included, totaling 2,062 admissions. The patient cohorts consisted of 310 men and 93 women with a median age of 62 years (age range, 32–78 years). The final cohort comprised 1,659 single chemotherapy cycles, of which 1,224 experienced grade ≥1 adverse events [Common Terminology Criteria for AEs (CTCAE version 5.0)]. And 45 characteristics potentially associated with chemotherapy-associated AEs were incorporated into the model. This yielded events-per-variable (EPV)=1,224/45=27.2, markedly above the traditional logistic-regression threshold of EPV and within the recommended range of EPV for moderate-complexity algorithms such as random forest (RF). The present study integrated multi-level data (2,062 patient characteristic records +1,659 chemotherapy session records). The use of longitudinal data analysis methods enhanced statistical power, with this intensive measurement design partially compensating for patient number limitations.

Response evaluation criteria in solid tumors (RECIST; version 1.1) were used to evaluate the clinical efficacy of chemotherapy. Follow-up was performed to evaluate AEs before each cycle of chemotherapy. The survival of patients who completed neoadjuvant chemotherapy and underwent surgery was followed up for 3 years by telephone and clinical re-examination, follow-up was performed every 3 months. All procedures in the present retrospective study involving human participants were performed in accordance with the Declaration of Helsinki (as revised in 2013). The patients were informed that the clinical information was stored by the hospital and potentially used for scientific research, and the need to obtain signed informed consent was waived by the Ethics Committee of The First Affiliated Hospital, School of Medicine, Zhejiang University, (Hangzhou, China). All patient cohorts included in the present retrospective study underwent standardized testing and treatment protocols. The relevant features and adverse reaction data were uniformly recorded in the EHR system. For the very few instances where data were incomplete, a complete-case analysis was performed and these patients were excluded from the study to maintain the integrity and robustness of the dataset.

ML algorithms were used to predict AEs in real-world patients with NSCLC who were receiving chemotherapy. The primary task for the algorithms was to predict the probability of the next severe AEs based on data from the present or previous chemotherapy characterization. The targeted AEs predicted in the present study were myelosuppression, low albumin (ALB) and hepatic impairment, with the judgment criteria based on the CTCAE (v5.0), published by the U.S. Department of Health and Human services.

The following chemotherapy-associated AE characteristics were extracted as potential predictors for risk prediction: i) Patient baseline characteristics [age, sex, history of hypertension, diabetes and tumor history, family tumor history, smoking status, drinking status, weight loss and body mass index (BMI)]; ii) tumor-related features (tumor location, tumor size, histology, grade and stage after surgery); iii) chemotherapy-related features (first-line treatment, chemotherapeutic agents and dose); iv) hematological indicators [white blood cells (WBC), neutrophils, lymphocytes, monocytes, hemoglobin (Hb), platelet (PLT), total protein (ToP), ALB, alanine aminotransferase (ALT), aspartate aminotransferase, creatinine, uric acid, triglyceride and cholesterol]; and v) clinical intervention characteristics (recombinant-human granulocyte colony stimulating factor, thymosin and reduced glutathione).

The original dataset included 45 features generated during hospitalization. A forward stepwise regression approach was employed based on logistic regression (LR) for feature selection. Specifically, at the beginning, univariate analysis was performed for each feature, and feature A with the best predictive performance for the outcome was selected and incorporated into the model. Multivariate analysis involving two features was subsequently performed on the basis of feature A and each of the remaining features, and feature B, included in the combination with the best predictive performance, was selected and added to the model. This process was repeated step by step to incorporate predictive factors until the model performance converged, at which point the addition of factors was stopped, resulting in a set of features beneficial for predicting AEs. The Shapley Additive Explanation (SHAP) methodology was used to evaluate the interpretability of the prediction models. Feature ranking was achieved through the calculation of SHAP values, with features being prioritized based on the mean absolute SHAP value for each. The integration of machine learning techniques with SHAP offers a clear and explicit interpretation of efficacy predictions.

ML is a general term for a class of methods that includes multiple algorithms with different technical principles, each of which may have different performance on a particular task. To select the optimal model, representative models of common ML algorithms were used, namely, RF, multilayer perceptron (MLP) and AdaBoost. For comparison, LR was also used, which is commonly used in clinical medical research, as a benchmark for performance comparison. The experiment was performed with five-fold cross-validation, and the receiver operating characteristic (ROC) curve, area under the ROC curve (AUC) value and calibration curve were used to evaluate the performance of the model.

An RF model is an integrated learning model that comprises a number of independent decision trees (also referred to as weak classifiers), each of which is trained on separate training data; thus, each tree independently predicts the type of a new sample. RF subsequently counts the results on the basis of the predictions of each decision tree and ultimately determines the specific type of a new sample and its corresponding probability on the basis of the majority vote classification and mean. The inclusion of numerous decision trees is the reason for the word ‘forest’ in the name. To avoid the lack of variability in the trained decision trees due to the inclusion of the same training data, RF uses a randomized strategy for selecting the training data. Specifically, for an ‘N’ number of samples of the original data, the model uses put-back sampling to randomly sample a set of ‘N’ from the original data to train a decision tree. The put-back sampling strategy ensures that the generated sample will cover ~63% of the original data; thus, the training data of each decision tree differ, and there is no significant homogeneity among the resulting decision trees.

An MLP is a type of basic neural network that can be divided into input, hidden and output layers according to its structure. The layers can be interconnected or not connected. After data are input into an MLP through the input layer, each node in the input layer inputs feature values in the data to the nodes in the next layer with specific weights. The input strength of the corresponding nodes in the next layer is the cumulative sum of the output nodes in the previous layer, which is then processed by a non-linear transformation function to continue to transfer information to the next layer. These steps are repeated until the prediction result is finally output at the output layer.

AdaBoost is an ensemble learning model, similar to an RF, that also uses numerous decision trees to complete classification tasks. The decision trees used by AdaBoost are not independent but are interrelated. Specifically, after the model has trained the first decision tree, the second decision tree focuses on the samples misclassified by the first decision tree, thus ensuring classification accuracy. The third decision tree focuses on samples misclassified by the former two decision trees. In the testing phase, after new samples are accepted, AdaBoost runs all the decision trees simultaneously and then calculates the average of their outputs with specific weights to obtain the final prediction results.

All the models used in the present study were provided by sklearn (22), and relevant statistical analysis was performed through SciPy (23). The specific experimental environment was conducted on a Lenovo computer (Lenovo) including an Intel Xeon E2520 (Mountain View; Intel Corporation), 32 GB of memory and two Nvidia Titan V graphics cards (NVIDIA Corporation). All original code has been deposited at GitHub (github.com/ZJU-BMI/cancer). Data are available from the authors upon reasonable request and with written permission; following the requirements of data supervision regulations, these data were not uploaded to a public platform.

Comparisons of categorical data between chemotherapy-related AEs in patients and clinical response to chemotherapy were performed using χ² or Fisher's exact test. Survival curves were estimated using the Kaplan-Meier method and analyzed using the log-rank test. Statistical analysis was performed using Prism 10.3 (GraphPad; Dotmatics) and SPSS software 25.0 (IBM Corp.), and P<0.05 was considered to indicate a statistically significant difference.

A total of 50 patients with NSCLC who completed neoadjuvant chemotherapy and underwent surgery were analyzed. CTCAE v5.0 was used to determine the extent of AEs in neoadjuvant patients. All the AEs were found to be grade 1 or 2. Our previous study revealed that clinical responses according to RECIST v1.1 could predict survival outcomes and an improved objective response was notably associated with improved overall survival (OS) (24). Patients who did not experience multi-AEs (two or more adverse events) had improved OS compared with those who experienced multi-AEs throughout the chemotherapy period (hazard ratio, 0.10; 95% CI, 0.02 to 0.45; Fig. 1A). Furthermore, multi-AEs were significantly associated with the efficacy of neoadjuvant chemotherapy (Fig. 1B). However, there was no significant association between the single AEs, including low ALB levels and hepatic impairment, and the objective response (Fig. 1C and E; Table SI).

The characteristics of the single chemotherapy treatments in the dataset are shown in Table I, and the baseline characteristics of the patients in the experimental dataset are shown in Table II. A total of four independent prediction models were developed and the performance of all models were evaluated and compared using ROC curves and AUCs. Among the proposed models, the RF model exhibited the best performance in both AE prediction tasks, followed by the LR model, and AdaBoost outperformed the MLP (Table III). When comparing the AUC, accuracy and precision between RF and the other three models, RF outperformed other models. The ROC curves exhibited a similar trend (Fig. 2), and the ROC curves of the RF model were consistently higher compared with those of the other three models for all three prediction tasks, implying that at any cutoff point, the RF model demonstrated improved results in terms of the true-positive rate and false-positive rate simultaneously. With respect to specific classification performance metrics, the accuracy, precision and recall rates of the RF model were consistently higher compared with those of the LR model.

As shown in Fig. 3, each model included in the present study exhibited several instances of performance degradation when the volume of training data increased in each prediction task; however, overall, a more pronounced performance of the four models was observed for the three prediction tasks after increasing the training data volume. With the exception of the LR model, no significant performance convergence was observed, indicating performance saturation for myelosuppression and liver impairment. Thus, we hypothesized that if more patient data were used, the models could achieve higher prediction accuracies. Again, the RF model exhibited the best performance among all four models in any training environment.

The effect of the number of incorporated features on the predictive performance of the models was assessed by individually selecting and overlaying the number of features. Each feature was numbered from 0 to 45, these and their corresponding meanings are shown in Table IV. The results revealed that ≤10 features (Table SII) significantly improved the performance (Fig. 4). With an increasing number of incorporated features, the models exhibited noTable overfitting (25), resulting in a decreased average AUC. The features in each predictive model of AEs were ranked based on importance as follows: i) Myelosuppression: 27, 35, 18, 44, 36, 4, 14, 34, 23, 17, 2, 45, 41, 40, 25, 39, 33, 11, 19, 30, 31, 21, 10, 38, 29, 6, 12, 0, 15, 7, 3, 5, 32, 8, 26, 20, 43, 13, 42, 9, 16, 37, 28, 22, 24, 1, 2; ii) low-ALB: 36, 23, 1, 26, 10, 30, 31, 35, 45, 34, 37, 9, 28, 29, 39, 6, 42, 19, 18, 20, 16, 4, 5, 21, 41, 33, 12, 40, 0, 15, 3, 7, 38, 43, 27, 44, 32, 17, 25, 13, 24, 2, 11, 22, 8, 14, 3; and iii) hepatic impairment: 38, 21, 16, 10, 25, 20, 12, 24, 17, 1, 22, 2, 43, 31, 6, 29, 4, 18, 9, 34, 14, 0, 32, 26, 37, 39, 36, 8, 23, 13, 45, 15, 28, 27, 44, 33, 5, 7, 30, 3, 42, 19, 35, 41, 11, 40.

Metric calibration is crucial for evaluating the accuracy of a model in predicting the probability of an AE occurring in an individual in the future. This reflects the extent to which the theoretical risk predicted by a model agrees with the observed risk. Issues were observed in the MLP and AdaBoost model calibrations because their calibration curves for predicting myelosuppression, low ALB levels and hepatic impairment differed markedly from the optimal values (Fig. 5). Comparatively, the calibration and optimization curves of the RF model had improved fit for the three prediction tasks, indicating that the probability of predicting the risk of side effects in patients suggested by the RF model represented the true value to a considerable extent. Conversely, the LR model was significantly under-calibrated for the myelosuppression side effect, whereas the other two models exhibited calibration degrees that were similar to that of the RF model.

To improve the transparency and interpretability of the model, the SHAP algorithm was used to elucidate the model's output. The SHAP value of each of the most important features on RF model output was calculated (Fig. 6). Based on the importance ranking derived from the average absolute SHAP values, the top five features (WBC, PLT, Weight, ToP, Hb) were identified as the most significant variables for predicting myelosuppression. It was demonstrated that ‘ToP’, ‘ALB’, ‘BMI’, ‘Age’ and ‘Hb’ were the five most influential features in predicting low-ALB. Additionally, ‘ALT’ was identified as the most significant variable for predicting hepatic impairment. Fig. 6B presents a violin plot for each feature, illustrating the association between the feature values and their corresponding SHAP values. The horizontal position indicates whether a particular feature value contributes to a higher or lower model prediction. The color gradient reflects whether the variable value is high (red) or low (blue) for a given observation. A larger absolute SHAP value indicates a stronger influence of that feature on the predictions of the RF-based model. Lower WBC and lower PLT were associated with a higher predicted probability of myelosuppression. And lower ToP were associated with a higher predicted probability of low-ALB. It was also observed that increases in the ALT had a positive influence, directing the prediction toward hepatic impairment. The SHAP algorithm was also used to elucidate the other model's output. (Fig. S1, Fig. S2, Fig. S3). For predictive tasks assessed using the LR model, ‘WBC’ and ‘PLT’ were found to be the two most important features in predicting myelosuppression, ‘ToP’ was the most important feature in predicting low-ALB and ‘ALT’ was identified as the most significant variable for predicting hepatic impairment (Fig. S1). For predictive tasks using AdaBoost, ‘WBC’, ‘ToP’ or ‘ALT’ were the most influential features in predicting myelosuppression, low-ALB and hepatic impairment, respectively (Fig. S2). For predictive tasks using MLP, ‘PLT’, ‘WBC’, ‘Hb’, ‘Top’, ‘Weight’ and ‘HTN’ were the six most influential features in predicting myelosuppression. It was also demonstrated that ‘ToP’, ‘ALB’, ‘BMI’, ‘Age’, ‘Hb’, ‘PLT’, ‘LYM’ and ‘LYM%’ were the eight most influential features in predicting low-ALB. Additionally, ‘ALT’, ‘DP’ and ‘Tumor histology’ were identified as the most significant variable for predicting hepatic impairment (Fig. S3).