2017 Individual Seed Projects
Mining Digital Life for Precision Prediction, Prevention & Early Detection
Byron Reeves, PhD
Paul C. Edwards Professor of
The Irving Schulman, M.D. Professor of Child Health, Professor of Medicine (Stanford Prevention Research Center) and, by courtesy, of Epidemiology and Population Health
Kenneth Fong Professor and Professor of Bioengineering, of Genetics, of Medicine (General Medical Discipline), of Biomedical Data Science and, by courtesy, of Computer Science
Professor, Human Development
and Family Studies, and Psychology
- Much of life is now digitized on personal screens including information about relationships, money, work, play, travel and much more.
- We are recording screenshots of digital personal life from laptops and smartphones, and identifying features related to health outcomes.
- Screenomics is science of studying digital life experiences, exposures and actions through personal screenshots (the screenome) for their impact on thoughts, emotions, behaviors and health.
- The screenshots that make up the screenome are recorded every 5 seconds that devices are turned on.
- The images are encrypted, compressed and transmitted to Stanford servers for analysis.
- We extract text and images from the screenshots and create searchable databases.
- We currently have over 10M screenshots from ~400 people
- We visualize the information obtained in the screenome and analyze the behavioral sequences for content and action patterns.
- Examples above:
- The color bars on the left are 1-week screenomes for 30 people; colors indicate the timing and material viewed.
- The circle on the right shows switching across devices and among multiple types of content.
- Using machine learning and individual dynamic modeling, we search for the screenshot features and screenome actions, and combinations of them, that predict health outcomes.
- Examples above (from left):
- Person searching for information about a suicide bridge.
- Texting about bruises that won’t heal.
- Exhibiting memory problems in communication with a spouse.
- Asking about dark skin on the back of the neck and why it won’t wash off.