Gleaning rigorous clinical insights from radiology scans, surgical videos, and pathology slides provides a comprehensive patient snapshot for more informed decision-making. In this talk, I will present our broader effort to weave information from these complementary modalities/scales to improve patient outcomes. Specifically, for radiology scans, I will introduce our work on (1) creating clinically-interpretable radiomics for screening and treatment response prediction, (2) artifacts: friends or foe?, (3) physically-realistic breathing motion induction in static scans, and (4) clinically-deliverable radiation dose prediction using AI and large-scale optimization. For surgery, I will briefly talk about our pioneering work in analyzing minimally invasive surgical videos. I will conclude with our work in pathology which aims to bridge hematoxylin-and-eosin (H&E), immunohistochemistry (IHC), and next-generation multiplex image analysis for improved biomarker quantification.