Optimizing Treatment Selection for Hepatocellular Carcinoma: Machine Learning Guidance in Choosing Liver Transplantation versus Surgical Resection

Highlight

– Liver transplantation (LT) generally offers better long-term survival than surgical resection (SR) for hepatocellular carcinoma (HCC), but donor scarcity and patient variability complicate treatment selection.
– A nationwide Korean cohort study developed separate machine learning (ML) models to predict 3-year overall survival (OS) following LT or SR and stratified patients into treatment-favorable groups.
– The ML models demonstrated robust predictive performance; counterfactual analyses suggest ML-guided treatment decisions could reduce mortality risk by over 50% compared to current clinical practice.
– External validation confirmed reproducibility, supporting potential clinical utility of ML-assisted personalized treatment selection for HCC.

Study Background and Disease Burden

Hepatocellular carcinoma (HCC) is a leading cause of cancer-related mortality worldwide, commonly arising in the context of chronic liver disease. Curative treatment options include surgical resection (SR) and liver transplantation (LT). LT generally confers superior long-term survival by removing both tumor and cirrhotic liver. However, limited donor organ availability, risk of waitlist dropout, and substantial patient heterogeneity—such as liver function, tumor burden, and comorbidities—render optimal selection between LT and SR challenging. Current clinical criteria, such as Milan or UCSF criteria, provide guidance but have limitations in individualized prognostication. Thus, a data-driven decision-support tool is needed to personalize treatment strategies, balancing survival benefit against resource constraints.

Study Design

This retrospective cohort study utilized national registry data from the Korea Central Cancer Registry, encompassing 3915 patients with HCC who received LT or SR as curative treatment between 2008 and 2018, forming the derivation cohort. LT and SR groups were characterized by distinct clinical profiles, with LT patients younger but exhibiting more advanced liver disease and poorer liver function than SR patients. Tumor characteristics also differed, with LT recipients having smaller but more numerous tumors.

An independent external validation cohort comprised 614 HCC patients treated at Seoul St Mary’s Hospital between 2009 and 2020.

Separate machine learning (ML) algorithms were developed for LT and SR patients to estimate 3-year overall survival (OS). The study compared several ML models, including support vector machines and gradient boosting (CatBoost). Key clinical variables included demographic data, liver disease severity indicators (e.g., cirrhosis, albumin, bilirubin), tumor size and number, and laboratory values.

Patients were stratified into high- and low-risk groups based on predicted OS for each treatment, defining LT-favorable and LT-nonfavorable subsets. A counterfactual analysis simulated survival outcomes if ML-guided treatment decisions had been applied versus actual clinical decisions.

Key Findings

The derivation cohort comprised 296 LT and 3619 SR patients, predominantly male (~80%), with median ages of mid-50s to late 50s. LT recipients had increased cirrhosis prevalence (26.4% vs 19.3%; P=.005), hepatic encephalopathy (6.8% vs 0.3%; P<.001), and ascites (19.9% vs 4.2%; P<.001). Liver function indices showed worse profiles in LT patients, including lower albumin (median 3.4 vs 4.2 g/dL), elevated bilirubin (median 1.4 vs 0.7 mg/dL), and prolonged INR.

Tumor characteristics showed LT candidates had smaller tumors (median 2.3 cm vs 3.2 cm; P<.001) but higher tumor counts (mean 1.6 vs 1.2; P<.001).

Regarding predictive performance, the support vector machine model showed the highest AUROC (area under the receiver operating characteristic curve) of 0.82 (95% CI, 0.78-0.86) in the LT cohort, while CatBoost demonstrated optimal AUROC of 0.79 (95% CI, 0.78-0.80) for the SR cohort, indicating good discrimination in survival prediction.

By stratifying patients into LT-favorable and LT-nonfavorable groups via the ML models, the authors identified individuals likely to benefit most from LT versus SR, accounting for individual clinical characteristics.

Counterfactual survival analysis estimated that ML-guided treatment decisions could reduce the hazard of death by 54% compared with observed clinical practice (hazard ratio [HR] 0.46; 95% CI, 0.42-0.50; P<.001). External validation in the independent cohort confirmed these results, supporting the model’s generalizability.

Expert Commentary

This study exemplifies how advanced machine learning can support nuanced, personalized decision-making in complex oncologic settings complicated by resource limitations. By integrating heterogeneous clinical variables and predicting survival for competing treatments, the ML models provide actionable stratification beyond traditional criteria.

While LT often yields superior outcomes, many patients are disadvantaged by limited donor availability or prohibitive comorbidities. The ML-based approach offers a framework to balance these constraints rationally, potentially optimizing organ allocation and improving overall survival at the population level.

Limitations include retrospective design, potential unmeasured confounders, and the need for prospective validation across diverse populations. The models are as good as the input data quality and may require recalibration in different ethnic or geographic contexts. Additionally, integrating molecular or imaging biomarkers could further enhance predictive accuracy.

Conclusion

This comprehensive nationwide cohort study introduces a machine learning-based decision-support model accurately predicting 3-year overall survival in hepatocellular carcinoma patients undergoing liver transplantation or surgical resection. Model-guided individualized treatment selection shows promise for improving survival outcomes by tailoring therapeutic approaches to patient-specific risks and benefits. This approach can complement existing clinical guidelines and help address the critical challenges of donor scarcity and patient heterogeneity. Future prospective studies and multicenter validations will be essential to confirm clinical utility and facilitate integration into routine practice.