Colorectal cancer (CRC) is the third most common malignancy worldwide and the second most common cause of cancer-associated mortality (1). After the locoregional cancer is treated, CRC often causes distant metastasis, with the liver being the most common site (2). Due to anatomical reasons, patients with rectal cancer (RC) have a higher risk of local recurrence compared with patients with colon cancer, and locally recurrent RC is associated with a poor prognosis (3–5). Therefore, various neoadjuvant therapy approaches have been attempted for stage II or stage III RC to prevent distant metastasis and local recurrence, including neoadjuvant chemotherapy (6–8), neoadjuvant chemoradiotherapy (9–11) and total neoadjuvant therapy (12–14).

Previous studies have indicated that tumor regression detected through imaging after preoperative treatment is associated with long-term prognosis in patients with RC (15,16). In a prospective cohort study, the tumor regression grade assessed through imaging was significantly associated with overall survival and disease-free survival (15). Various efforts have been made to explore the usefulness of serological or genetic biomarkers as predictors of the response to preoperative treatment, although such biomarkers are costly (17,18). The use of regular examinations to predict the response of preoperative treatment is desirable in terms of cost-effectiveness and objectivity. Colonoscopy is typically performed on nearly all patients suspected of having colorectal cancer before surgery to assess tumor size, depth and circumference, followed by biopsy to confirm the diagnosis (19,20). Patients undergoing preoperative treatment are assessed for treatment response by comparing endoscopic images taken before and after preoperative treatment. Colonoscopy images are thus among the most readily available tumor images.

Conventional artificial intelligence (AI) techniques and machine learning have major limitations in analyzing natural data (21). However, deep learning, which has emerged in previous years, excels in learning complex structures with high-dimensional data and has enabled significant advancements in various fields (21,22). Deep learning is applicable to medicine, and AI tools developed in collaboration with AI experts, specialized facilities, companies and physicians are gradually being implemented in practical settings (23–25).

The present research developed a deep learning prediction model to assess the treatment response of RC to neoadjuvant chemotherapy using endoscopic images taken during the initial examination. The prediction accuracy of the model was further investigated. To the best of our knowledge, no report has verified whether deep learning of colonoscopy images from initial examinations can predict the response of RC to neoadjuvant chemotherapy.

A total of 53 of the 71 patients were included in the present study. A total of 322 pre-treatment images from 43 patients who underwent neoadjuvant chemotherapy for advanced RC between January 2011 and March 2018 were included in the learning set (Table I; Fig. 1). A validation set was created that included 84 images obtained during the pre-treatment examination of 10 patients who underwent neoadjuvant chemotherapy between June 2018 and August 2019. All patients underwent total mesorectal excision with R0 resection 3 to 6 weeks after completing neoadjuvant chemotherapy. In the learning set, patients with PR-RC had more advanced pathological tumor invasion and pathological lymph node metastasis compared with that observed among patients GR-RC, reflecting the clinical efficacy of neoadjuvant chemotherapy (Table II).

The present study constructed a prediction model based on deep learning of pre-treatment images and clinical responses in the learning set. The accuracy of pre-chemotherapy deep learning model in predicting the response to chemotherapy in the validation set is shown in Table III. In the validation dataset, 49 images were taken from five patients who were clinically diagnosed as PR-RC, and 35 images were taken from five patients who were clinically diagnosed as GR-RC. When using the validation set, the sensitivity, specificity, positive predictive value, accuracy and area under the curve (AUC) of the model in predicting PR-RC status were 77.6% (38/49), 62.9% (22/35), 74.5% (38/51), 71.4% (60/84) and 0.713, respectively (Table II; Fig. 2).

Prediction errors where the model was unable to predict response occurred when the AI focused on areas other than the tumor (Fig. 3). The percentages of incorrect predictions where the AI focused on tumor area were 36.4% for PR-RC and 15.4% for GR-RC. The percentage of incorrect predictions where the AI focused on the normal mucosa was 54.5% for PR-RC and 46.1% for GR-RC (Table IV). The percentages of incorrect predictions focused on bleeding were 9.1% for PR-RC and 38.5% for GR-RC. AI focus on non-tumor areas was considered to be partly responsible for the incorrect AI predictions.

Finally, the present study constructed another prediction model based on deep learning of clinical responses and colonoscopy images of the same patient dataset after preoperative chemotherapy, and its accuracy was verified. The same learning and validation sets were used to test the model based on colonoscopy images after preoperative chemotherapy. The accuracy of the model is shown in Table SII. The post-chemotherapy model had a sensitivity of 40.5% (17/42), specificity of 56.1% (23/41), accuracy of 48.2% (40/83) and AUC value of 0.592 (Fig. S1C). The post-chemotherapy deep learning model had inferior predictive performance to the model based on pre-chemotherapy endoscopic images.

AI technology has achieved unprecedented success in various fields. Deep learning, which is a subset of machine learning that focuses on deep artificial neural networks (21,25), has been used in numerous areas of oncology, ranging from cancer detection and classification to the molecular characterization of tumors and their microenvironment (33,34), drug discovery (35) and prediction of distant metastasis (36). In the context of predicting the efficacy of preoperative treatment for RC, deep learning based on MRI images and hematoxylin-eosin (HE) staining images has been reportedly useful in predicting complete response to preoperative chemoradiotherapy (37). The colonoscopy that was the source of images in the current study was an examination conducted for nearly all patients with RC who received preoperative treatment, meaning it was relatively straightforward to implement a deep learning model using this modality in clinical practice.

The present study examined the ability of deep learning models based on colonoscopy images to predict the response of RC to preoperative chemotherapy. This study did not include the patients who were provided neoadjuvant radiation therapy or neoadjuvant chemoradiotherapy because the mechanisms of antitumor effects of chemotherapy and radiation therapy differ. The model based on colonoscopy images taken before preoperative chemotherapy was able to differentiate PR-RC from GR-RC with a sensitivity of 77.6% and accuracy of 71.4% with an AUC of 0.713. Furthermore, the present study created a program to visualize the parts of the image that the AI recognized and diagnosed using a color map (38–40). AI-based image recognition detects the location of the tumor as well as the information around the tumor, including the bleeding area, which might influence the prediction accuracy. Implementing a program that accurately recognizes the tumor surface may improve the accuracy of the prediction. The analyses in the study did not exclude patients who used molecularly targeted drugs. The present study also attempted to create another model using pre-treatment images except for the patients who used molecularly targeted drugs and the accuracy was assessed. The sensitivity and accuracy of the model that excluded the patient on targeted therapy were 71.4% (35/49) and 65.5% (55/84), respectively.

The current study defined MRI-detected clinical response as the outcome of deep learning models because MRI assessments of tumor regression grade are imaging markers that predict survival outcomes (15). Pathologic tumor regression grading is based on the percentage of fibrosis in the surgical specimen and correlates with survival at a greater statistical significance (15,16); however, by applying a similar approach with MRI, the present study was able to assess tumor response and predict the outcome of patients before surgery (15). Future validation studies are needed to determine the tumor response and survival outcome before chemotherapy using deep learning and endoscopic images.

To the best of our knowledge, the present study was the first to report on the utility of deep learning based on pretreatment endoscopic images to predict the treatment response to neoadjuvant chemotherapy. The current results had several important implications. The first was the possibility of improving the model by combining the modality with other modalities, such as MRI images and clinicopathological factors. Recently, deep learning prediction models have combined multiple modalities to predict the clinical efficacy of preoperative RC treatment. Predictive models combining multiple modalities significantly outperform predictive models involving individual modalities (37). Secondly, the application of deep learning to pretreatment endoscopic images could serve as a potential method for predicting the complete response to preoperative chemoradiotherapy. In the future, if deep learning models based on endoscopic images demonstrate improved predictive accuracy compared with models based on histological images, their significance must be further emphasized through future validation. Thirdly, pre-chemotherapy endoscopic images may be more useful compared with post-chemotherapy images to build models predicting the effect of chemotherapy. The present study revealed that the deep learning model based on pre-chemotherapy images was more accurate in predicting chemotherapy efficacy compared with that using post-chemotherapy images. Furthermore, the present study validated a combination model; however, the sensitivity (20.0%) and accuracy (40.0%) were so low that the model was not considered useful. The deep learning model based on pre-chemotherapy images may capture features of tumor surface structures that reflect chemotherapy responsiveness.

The present results were limited by the potential risk of overfitting due to the use of data from a single-center and small sample size. Additionally, the current results are limited by uncertain reproducibility due to the lack of prospective validation; therefore, additional validation is required.

In conclusion, deep learning based on endoscopic images may allow for enhanced accuracy when predicting the response of RC to neoadjuvant chemotherapy. There is a need to collect more images and investigate additional image features to improve prediction accuracy.