Intensive Care Unit Clinical Pathway Support

Overview

Activity detection in Intensive Care Units (ICUs) is currently performed manually by trained personnel, primarily nurses, who log the activities as they occur. This process is both expensive and time consuming. Our goal is to design a system which automatically gives an annotated list of all activities that occurred in the ICU over the day. Overall, this system will reduce the monitoring workload of trained personnel, and lead to a quicker and safer recovery of the patient, while providing benefits such as activity-based costing.

Activity recognition in hospitals is a task that has not received much attention in the past. Some of the main reasons for this gap in research are the lack of sensors installed in hospitals and the difficulty in obtaining access to the relevant data due to its sensitive nature. Thanks to our partnering hospital, we have access to depth sensors installed in eight intensive care unit (ICU) rooms.

We are developing a computer vision system capable of automatically detecting the following activities:

  • Stage 1: patient getting out of bed, patient getting out of bed and walking, and a nurse performing oral care.
  • Stage 2: clinician performing ultrasound, x-ray, turning patient over in bed, and patient getting in/out of bed.
  • Stage 3: various patient mobility activities such as patient getting in/out of a bed/chair with or without assistance.

Once our system can successfully log the basic activities above, we plan to expand it to detect anomalies such as emergency situations. To do so, we could potentially use a dataset of simulations of different emergencies (e.g., patient falls on the floor).

Pilot Partnerships


We are partnering with SHC where we will evaluate the use of computer vision technology to characterize patient-oriented activities of patients admitted to the SCH ICU rooms that are equipped with the computer vision sensors. We aim to develop a new vision and machine learning algorithm that benefits from the amount and variety of real-world data of activities in the ICU to successfully develop motion activity models that perform well in clinical settings.


We have partnered with Intermountain's Healthcare Transformation Lab where we have deployed 3D depth sensors in eight ICU rooms. With the help of Intermountain, we are using live data streams to teach our computer vision algorithms to discern events of clinical relevance. Using multiple sensors per room, our artificial intelligence system is capable of full-room activity understanding.

People

Amit Kaushal, MD, PhD.

Clinical Assistant Professor

Amit Kaushal, MD, PhD is a Clinical Assistant Professor of Medicine (Stanford-VA) and Adjunct Professor of Bioengineering at Stanford University. Dr. Kaushal's work spans clinical medicine, teaching, research, and industry.

Ali Al-Rajhi, PhD, MPH

Research Project Manager

Dr. Ali Al-Rajhi has developed experience leading cross-functional teams in conducting clinical trials and publishing healthcare policy and systematic reviews. Dr. Al-Rajhi has worked in the fields of ophthalmology with the American Academy of Ophthalmology; and at the Stanford Center for Clinical Research (SCCR) overseeing trials in gastroenterology and hepatology. His interest is at the crossroads of data science and healthcare, where he currently splits his time with Stanford’s Clinical Excellence Research Center (CERC) and Stanford Partnership in AI-Assisted Care (PAC).

Tracy Terada

Research Operations Manager

Tracy is a 15+ year adminstrative veteran for the Stanford School of Medicine.  She started at the Lane Medical Library and is currently with the Clinical Excellence Research Center. 

 

Ehsan Adeli, PhD.

Scientist, Stanford AI Lab, Stanford Vision and Learning, Computer Science Department
Clinical Assistant Professor, Department of Psychiatry and Behavioral Sciences, Stanford School of Medicine

Alan Luo

Doctoral Student, Computer Science

Alan is a Ph.D. student in the Stanford Vision and Learning Lab, advised by Prof. Fei-Fei Li. His main research interests include weakly-supervised learning, transfer learning, and deep learning.

Shrinidhi Lakshmikanth

PAC Data Engineer

Interested in Machine Learning / Deep Learning. Working on cloud infrastructure for data collection.

Zane Durante

Doctoral Student, Computer Science

Zane is a first-year PhD student in Computer Science and is currently rotating with Prof. Fei-Fei Li. His research interests include self-supervised learning, multimodal signal processing, and AI for healthcare. Admitted Autumn 2021.

Russell Xie

Graduate Student, Computer Science

Russell is a master student in PAC and SVL. His primary research interests include vision-based activity recognition, medical image analysis, and deep learning.

Claire Zhang

Graduate Student, Electrical Engineering

Claire is a student with PAC. Her interests lie broadly in artificial intelligence, visual computing, and digital health.

Linden Sky Li

Undergraduate Student, Computer Science 

 

Scott Maran

Undergraduate Student, Computer Science - Joined April 2021

 

 

Eunice Yang

Undergraduate Student, Computer Science - Joined April 2021

Eunice is a member of PAC. Her main research interests include machine learning and computer vision, with a focus on applications in healthcare and medical AI.

Bingxian (Kevin) Chen

Graduate Student, Computer Science - joined April 2021

 

Publications

A Computer Vision System for Deep Learning-based Detection of Patient Mobilization Activities in the ICU

 

Serena Yeung*, Francesca Rinaldo*, Jeffrey Jopling, Bingbin Liu, Rishab Mehra, N. Lance Downing, Michelle Guo, Gabriel M. Bianconi, Alexandre Alahi, Julia Lee, Brandi Campbell, Kayla Deru, William Beninati, Li Fei-Fei and Arnold Milstein

Nature Partner Journals (NPJ) Digital Medicine
March 2019

Descriptive Analysis of ICU Patient Mobilization from Depth Videos

Laëtitia Shao*, Zaid Nabulsi*, Ruchir Rastogi*, Bingbin Liu, Francesca Rinaldo, Serena Yeung, N. Lance Downing, William Beninati, Arnold Milstein, Li Fei-Fei

Machine Learning for Health Workshop, Neural Information Processing Systems (NeurIPS); December 2018

3D Point Cloud-Based Visual Prediction of ICU Mobility Care Activities

Bingbin Liu*, Michelle Guo*, Edward Chou, Rishab Mehra, Serena Yeung, N. Lance Downing, Francesca Salipur, Jeffrey Jopling, Brandi Campbell, Kayla Deru, William Beninati, Arnold Milstein, Li Fei-Fei

Machine Learning for Healthcare (MLHC) Conference; August 2018

Vision-Based Prediction of ICU Mobility Care Activities using Recurrent Neural Networks

Gabriel M. Bianconi, Rishab Mehra, Serena Yeung, Francesca Salipur, Jeffrey Jopling, Lance Downing, Albert Haque, Alexandre Alahi, Brandi Campbell, Kayla Deru, William Beninati, Arnold Milstein, Li Fei-Fei

Machine Learning for Health Workshop, Neural Information Processing Systems (NIPS);   December 2017