2017 Individual Seed Projects

Mining Digital Life for Precision Prediction, Prevention & Early Detection

Byron Reeves, PhD

Paul C. Edwards Professor of
Communication
(650) 725-3033
The Irving Schulman, M.D. Professor of Child Health, Professor of Medicine (Stanford Prevention Research Center) and, by courtesy, of Epidemiology and Population Health
Kenneth Fong Professor and Professor of Bioengineering, of Genetics, of Medicine, of Biomedical Data Science, Senior Fellow at the Stanford Institute for HAI and Professor, by courtesy, of Computer Science

Nilam Ram, PhD

Professor, Human Development
and Family Studies, and Psychology
(814) 865-7038

  • Much of life is now digitized on personal screens including information about relationships, money, work, play, travel and much more. 
  • We are recording screenshots of digital personal life from laptops and smartphones, and identifying features related to health outcomes.
  • Screenomics is science of studying digital life experiences, exposures and actions through personal screenshots (the screenome) for their impact on thoughts, emotions, behaviors and health.
  • The screenshots that make up the screenome are recorded every 5 seconds that devices are turned on.
  • The images are encrypted, compressed and transmitted to Stanford servers for analysis.
  • We extract text and images from the screenshots and create searchable databases.
  • We currently have over 10M screenshots from ~400 people

  • We visualize the information obtained in the screenome and analyze the behavioral sequences for content and action patterns.
  • Examples above:
  • The color bars on the left are 1-week screenomes for 30 people; colors indicate the timing and material viewed.
  • The circle on the right shows switching across devices and among multiple types of content.  
  • Using machine learning and individual dynamic modeling, we search for the screenshot features and screenome actions, and combinations of them, that predict health outcomes. 
  • Examples above (from left):
  • Person searching for information about a suicide bridge.
  • Texting about bruises that won’t heal.
  • Exhibiting memory problems in communication with a spouse.
  • Asking about dark skin on the back of the neck and why it won’t wash off.