A machine learning pipeline revealing heterogeneous responses to drug perturbations on vascular smooth muscle cell spheroid morphology and formation

Machine learning approaches have shown great promise in biology and medicine discovering hidden information to further understand complex biological and pathological processes. In this study, we developed a deep learning-based machine learning algorithm to meaningfully process image data and facilitate studies in vascular biology and pathology. Vascular injury and atherosclerosis are characterized by neointima formation caused by the aberrant accumulation and proliferation of vascular smooth muscle cells (VSMCs) within the vessel wall. Understanding how to control VSMC behaviors would promote the development of therapeutic targets to treat vascular diseases. However, the response to drug treatments among VSMCs with the same diseased vascular condition is often heterogeneous. Here, to identify the heterogeneous responses of drug treatments, we created an in vitro experimental model system using VSMC spheroids and developed a machine learning-based computational method called HETEROID (heterogeneous spheroid). First, we established a VSMC spheroid model that mimics neointima-like formation and the structure of arteries. Then, to identify the morphological subpopulations of drug-treated VSMC spheroids, we used a machine learning framework that combines deep learning-based spheroid segmentation and morphological clustering analysis. Our machine learning approach successfully showed that FAK, Rac, Rho, and Cdc42 inhibitors differentially affect spheroid morphology, suggesting that multiple drug responses of VSMC spheroid formation exist. Overall, our HETEROID pipeline enables detailed quantitative drug characterization of morphological changes in neointima formation, that occurs in vivo, by single-spheroid analysis. *Co-first authors: K. Vaidyanathan and C. Wang. **Co-corresponding authors: Y. Bae and K. Lee