Machine learning‑based identification of key genes underlying sex differences in hepatocellular carcinoma and targeted drug screening

Hepatocellular carcinoma (HCC) shows a marked predominance in men, yet the molecular basis for this sex disparity remains unclear. The present study leveraged multi‑omics data and machine learning algorithms to identify key genes associated with sex‑specific differences in HCC and to screen for putative candidate compounds, aiming to provide new insights for sex‑specific therapy. The mRNA expression data of male and female patients with HCC and paracancerous tissues were obtained from the GEO and TCGA databases. To mitigate overfitting, data were partitioned into independent training and testing sets. Candidate genes were screened by differential expression analysis and weighted gene co‑expression network analysis. A total of four complementary algorithms, random forest, support vector machines, generalized linear models and extreme gradient boosting were used to identify key genes with high predictive capability. CYP17A1 and IRX3 were identified as the top differentially expressed core genes associated with HCC in men. Pan‑cancer analysis showed that CYP17A1 was lowly expressed in the majority of tumors, but significantly highly expressed in HCC, rectal adenocarcinoma and gastric cancer (P<0.001). Functional cell‑based assays showed that knockout of CYP17A1 inhibited the proliferation, migration and invasion ability of HCC cells (P<0.001). Immunohistochemistry showed that CYP17A1 protein expression was significantly increased in HCC tissues from male patients when compared with that in paracancerous tissues (P<0.001), whereas there was no significant difference in female patient tissues (P>0.05). Notably, while IRX3 was identified computationally, its functional role remains to be experimentally validated. Molecular docking predicted a potential interaction between the natural compound Saikosaponin A and the CYP17A1 protein, and cellular assays revealed that it dose‑dependently inhibits HCC cell malignant phenotypes. The present study suggests that CYP17A1 is associated with sex differences in HCC, potentially via the androgen signaling axis. Furthermore, IRX3 emerges as a novel hypothesis‑generating candidate gene. Finally, the findings of the present study highlight Saikosaponin A as a putative therapeutic candidate for male patients with HCC, warranting further target‑dependency investigations.