Although surgical resection is the standard therapy for stage II/III colorectal cancer, recurrence rates exceed 30%. Circulating tumor DNA (ctNDA) detects molecular residual disease (MRD), but lacks spatial and tumor microenvironment information. Here, we develop a deep learning (DL) model to predict disease-free survival from hematoxylin & eosin stained whole slide images in stage II-IV colorectal cancer. The model is trained on the DACHS cohort (n = 1766) and validated on the GALAXY cohort (n = 1404). In GALAXY, the DL model categorizes 304 patients as DL high-risk and 1100 as low-risk (HR 2.31; p < 0.005). Combining DL scores with MRD status improves prognostic stratification in both MRD-positive (HR 1.58; p < 0.005) and MRD-negative groups (HR 2.1; p < 0.005). Notably, MRD-negative patients predicted as DL high-risk benefit from adjuvant chemotherapy (HR 0.49; p = 0.01) vs. DL low-risk (HR = 0.92; p = 0.64). Combining ctDNA with DL-based histology analysis significantly improves risk stratification, with the potential to improve follow-up and personalized adjuvant therapy decisions.