Learning Health Systems

According to Centers for Medicare & Medicaid Services (CMS) - “Quality improvement (QI) is the framework used to systematically improve care. Quality improvement seeks to standardize processes and structure to reduce variation, achieve predictable results, and improve outcomes for patients, healthcare systems, and organizations”¹. Lynn summarized QI as ‘a systematic approach to data collection for the purpose of achieving immediate improvements in processes and outcomes in specific healthcare settings’². Quality improvement is considered by Agency for Healthcare Research and Quality (AHRQ) as a ‘continuous effort that apply systematic framework with the goal to improve patient care’³. 

QI initiative should help establish measurable processes for delivery care and will typically require commitment from the entire organization. The ‘measurable processes’ could include changes or innovative implementations of interventions such as  artificial intelligence (AI)/ machine learning (ML) -based tools, clinical decision supporting tools, educational approach, workflow revisions, etc.

According to Agency for Healthcare Research and Quality (AHRQ)- “a learning health system as a health system in which internal data and experience are systematically integrated with external evidence, and that knowledge is put into practice. As a result, patients get higher quality, safer, more efficient care, and health care delivery organizations become better places to work.”⁶ Data collected for evidence generation is typically through the electronic health records (EHRs) within the organization/institution⁵

Quality improvement (QI) initiatives, as one of the key components of the LHS, share the similar goals with LHS, although the main focus is on implementing and evaluating pragmatic and innovative interventions within the institution. The effectiveness and impact of the QI initiatives are quantified and evaluated with data from EHRs. Evidences of effectiveness are generated via robust data science and statistical methodology.

There are a lot of considerations in operational sciences that are applied for executing QI interventions. PDSA cycles, standards for ‘plan–do–study–act cycles’, applies a standardized operation to implement QI intervention that is very often used in real-world settings. As defined by IHI and the figure on the right side- ‘PDSA cycles are iterative and well adapted for action-oriented learning. It often takes several PDSA cycles to detect or sustain QI. There are several QI tools that can be used to facilitate the QI process’⁴

Not necessary. According to Scoto et al⁵, there are 3 main goals for QI initiatives in learning health systems.

A. Understand the current status or predict the future situation of the health system

B. Understand whether a QI intervention(s) improves outcomes to be assessed

C. Understand why and how a QI intervention improve outcome(s)

Most of the quality improvement (QI) will be focus on goal B by implementing and evaluating pragmatic and innovative interventions. Such interventions could include innovative clinical interventions, for example, artificial intelligence (AI)/ machine learning (ML)-based tools, clinical decision supporting tools (Best Practice Alerts, BPA), as well as pragmatic interventions such as educational approaches, workflow revisions and clinical guideline revisions. However, before implementing an ‘intervention’, it is also critical to first generate internal knowledge to understand the current landscape within the (goal A). After an ‘intervention’ is successfully implemented and demonstrates its effectiveness through robust evaluation, it is equivalently important to understand the mechanism of the success. Questions like, which population, component(s) of the interventions have the most impact upon improving patients’ outcomes, any institutional and organizational characteristics ensures the success, any lessons learned for future improvement, should be investigated in quality improvement initiatives via either quantitative or qualitative methodologies.

Stanford IRB has its own route in its determinations of ‘quality improvement (QI) vs research⁷. Key considerations include differences in the purposes/goals of the study, whether it’s a hypothesis-based or not, imposed risk to the patients and final deliverables⁷ . Typically, a QI waiver could be obtained if the study is considered of low risk to the patients, whereas the ‘risk’ here is specifically referring to risk occurs the data generation/collection process. On the other hand, a lot of QI initiatives could also contain its own ‘research’ component(s) that might impose higher risk to patient vs the QI interventions that did not have direct influence on the regular patient care process, therefore a QI waiver is not guaranteed for all projects with a QI component. Scientifically speaking, the evaluation process of a QI initiative should be as robust as the research studies that ensures the reproducibility of the findings.

According to Centers for Medicare & Medicaid Services (CMS), ‘Quality measures are tools that help us measure or quantify healthcare processes, outcomes, patient perceptions, and organizational structure and/or systems that are associated with the ability to provide high-quality health care and/or that relate to one or more quality goals for health care.’¹² Those metrics could be used in quality improvement, public reporting, and pay-for-reporting programs for specific healthcare providers, thus very often be considered as the key primary and secondary outcomes for QI quantitative evaluations. Different entities had their QI metrics to quantify the quality for healthcare systems. When designing and reporting a quality improvement study, we considered the three major type of measures including outcomes measures, process measures, and balancing measures¹⁸. Outcome measure refers to patients’ health outcomes, process measure tracks the progress of implementing improvement, and balancing measure evaluates whether the designed improvement causes new problems in the systems. The main entities established measures that are used very often at Stanford Healthcare (SHC) and other institutions includes Centers for Medicare & Medicaid Services (CMS) CMS¹² and Agency for Healthcare Research and Quality (AHRQ)¹³.

The following study designs could be considered.

• Randomized clinical trial (RCT)

• Quasi-experimental study 

• Observational study

Randomized clinical trial (RCT) uses randomization to minimize confounding, therefore is considered as the ‘gold-standard’ design in research. Randomization could be performed on the unit of orders, hospital encounters, patients, providers, departments or even institutions depending the study setting.  Cluster randomization clinical trial is one type of randomized clinical trial with randomization on the unit of providers, departments and institutions. In the real-word clinical setting for assessing a quality improvement intervention, high cost, infeasibility in implementation and high risk of ‘spillover’ effect draws RCT from the ideal design. Observational studies should be also considered which might impose less risk to patients. Statistical methods used for the purpose of causal inference such as inverse probability weighting or matching by propensity score, could be applied in observational study design to deal with the concern of confounders. One of the key types of study design in quality improvement study is the quasi-experimental design. It utilizes statistical models such as difference-in-difference analysis, stepped-wedge trials, and interrupted time series, and allows researcher to incorporate a natural control group that alleviating the issue of residual confounding factors.

There are two main factors to be considered when determining whether randomization should be implemented or not- the potential risk of ‘spillover’ effect and the feasibility of randomization. 

‘Spillover’ effect in randomized clinical trials is defined as ‘interventions may benefit those in close physical or social proximity to intervention recipients who do not receive the intervention themselves. When such ‘spillovers’ are present in the same direction as direct effects on recipients, direct effects alone do not capture the full health impact and cost-effectiveness of an intervention.’⁸ Under the setting of quality improvement studies, spillover effect is more likely to happen comparing to classic randomized clinical trials. Masking or blinding will not be feasible under most quality improvement scenarios, a typical scenario when the ‘spillover’ effect happens. For example, in a quality improvement study setting where a new clinical decision tool was made available to clinical providers to facilitate their treatment decisions. If we rolled it out and randomized it on the patient level, which means providers participating the study would have a randomized group of patients who they were asked to treat with the new tool, while they were asked to treated the rest of their patients without the tool.  In this case, providers who were previously exposed to decision tool from previously randomized patients could be intentionally or unintentionally influenced by their previous knowledge from the tool.  Patients’ outcomes are therefore likely to impacted by those exposures that could subsequently lead to a biased estimate of the impact. 

For the example above, cluster randomization might be a good alternative design that minimizes the risk of spillover effects. However, cluster randomization trials require a substantial number of clusters (under the clinical contexts- providers, clinics, hospitals) to ensure balances between the comparison groups. If one or a couple cluster is very different from the other clusters (e.g. with larger volume of patients, higher % of sicker patients, etc), using cluster randomization might introduce imbalances between the groups. Operational-wise, cluster randomization is not always feasible in practice, for reasons such as unethical to randomize, or adding huge logistic burden to providers, or discouraging QI advocating providers by randomizing them all to the control group. Under those scenarios, we shall consider observational or quasi-experimental study designs as good alternatives.

The difference in difference (DID) design is a quasi-experimental research to study causal relationships where randomized controlled trials (RCTs) are infeasible or unethical.⁹ It originates from health policy researches that have been applied very often under the quality improvement (QI) setting. We called it as ‘quasi-experimental research design’ because by design, it aims to mimic randomized clinical trials by differencing out unmeasured confounders⁹. Data before and after the quality improvement intervention (pre-QI vs post-QI) should be collected under the settings (provider, clinic, hospital, etc) where an intervention is implemented (intervention group).  In addition, it’s critical to also collect data under the setting where the intervention has never been implemented (control group). The DID estimator is interpreted as the difference of the changes from pre-QI period to post-QI period, comparing the intervention group to the control group. Graphic evidence, appropriate statistical tests could be performed to evaluate whether the ‘control’ group we selected is a good control. 

In quality improvement initiatives, it’s very often for the investigators to think that only looking at the change before and after the quality improvement invention implementation is sufficient (which is also called the ‘pre-post’ design). This means that only the changes for the ‘invention’ group in the DID design will be examined. No control group is considered in the pre-post design.  One limitation of pre-post design is that it often fails to take into account of changes over time and seasonal changes of the outcomes. For example, when evaluating the impact of a best practice alert (BPA) for colorectal cancer screening rate, an improvement of screening rate before and after the BPA implementation is not necessary contributed to the BPA alone. It’s also likely there’s a new screening guideline was implemented nationally during the post intervention period that was driving the improvement. As a result, the overall screening rate could be rising for every healthcare system nationally, not only the institute that the BPA was implemented.  For the pre-post design, it’s insufficient to only tease out the differences by clinics’ and patients’ characteristics (e.g. using multivariate regression model) to conclude an ‘successful’ quality improvement intervention. The investigators are required to show evidence to suggest there’s no external interventions/policy changes that is driving the outcomes to improve over time, which could more easily addressed by incorporating a reliable control group via the DID design.

According to Bernal et al. “In an ITS study, a time series of a particular outcome of interest is used to establish an underlying trend, which is ‘interrupted’ by an intervention at a known point in time.”11 Interrupted time series is well fit for studies that 

• the ‘pre-’ and ‘post-’ QI intervention is well defined 

• the outcome is considered ‘short-term’, or is impacted immediately by the intervention 

• data is collected either in aggregated format by regular time intervals or on the patient level with accurate date stamps. 

The Bernal paper provided good tutorials and examples for implementing interrupted time series study design.  A contemporary control group could be implemented in interrupted time series study design, which makes it similar to the design of difference-in-difference (DID). However, difference-in-difference study design is only comparing average change before and after the intervention, while interrupted time series also compares the slope of the changes (how quickly the outcome changes). For studies that expand over a long period of time, an ‘average’ change might not be sufficient to represent the patterns over time, therefore interrupted time series study design that reveals how the outcomes change over time could add another layer of robustness. 

The mixed-method approach fits well under the framework of  ‘plan–do–study–act (PDSA)’ for quality improvement implementation.

Mixed method approach is referring to the study design that incorporates both quantitative and qualitative evaluations in a single study.  According to Creswell. ‘This type of a study enables a policy researcher to understand complex phenomena qualitatively as well as to explain the phenomena through numbers, charts, and basic statistical analyses.’¹⁰ More importantly, the quantitative results could be used to give guidance upon the qualitative analysis or vice versa. However, how they talk to each other has to be pre-specified in the study protocol. 

Yes. Similar to the other RCT and observational studies, it is always recommended to establish a study protocol, develop a statistical analysis plan and conduct appropriate power analysis or sample size calculation before implementing any quality improvement interventions. To be noted, power analysis is more often conducted instead of sample size calculation for most quality improvement studies. This is because data collected for quality improvement evaluations (mostly via electronic health record data) is based on feasible population within the quality improvement scope, there’s no pre-planned prospective patient enrollment involved like the randomized clinical trials. It’s always recommended to involve QSU members as early as possible during the study design stage.

The main data source for QI researches will be collected from the Electronic Health Records (EHRs) system. EHR system is huge data system with the following types of data used very often for quality improvement.

• Comprehensive data collected during regular clinical encounters for patient care, such as patient demographics, diagnosis, procedures, clinical and lab measurements, hospital encounters, medications, imaging reports and clinical notes.  
• Patient self-reported health-related data (e.g. surveys, mobile device data)
• Providers’ information and patients’ satisfactory data
• Process and balancing measures for quality improvement if interventions were implemented via EHRs, for example, best practice alert records, AI-based clinical risk scores, providers’ behaviors, etc.

Other than EHRs, data derived from EHRs or stored in an external system could be also used for quality improvement researches, which includes the followings.

• Administrative data outside EHRs, such as costs from the billing data. 

• Qualitative surveys sent to the patients or providers to collect feedbacks for quality improvement. For example, questionnaires sent to the patients to monitor the effectiveness of intervention implementation and unstructured interviews for the providers to assess the quality improvement process.

Per Centers for Medicare & Medicaid Services (CMS), electronic health records (EHRs) is ‘an electronic version of a patient’s medical history, that is maintained by the provider over time, and may include all of the key administrative clinical data relevant to that persons care under a particular provider, including demographics, progress notes, problems, medications, vital signs, past medical history, immunizations, laboratory data and radiology reports’14. Although very often used interchangeably, EHRs covers much more rich patient data vs Electronic Medical Records (EMRs), for example, patients’ self-report health-related data (e.g. mobile health data, data generated from digital health devices).

For Stanford Healthcare, EPIC system is used as the main interface system for data collection and documentations. It contains functions to generate data reports for clinical use or preventative care. EPIC is translated into a relational database, called Clarity, that contained most data elements from EPIC. 

In the meantime, Clarity database has been further built into its copy for research use, called STAnford Research Repository (STARR). STARR included most common data elements for research use, including patients’ demographics, hospital encounters, diagnosis, labs, procedures, orders and order sets, medical prescriptions, clinical notes and imaging notes, etc. There are currently 2 versions of the STARR that researchers could get access – one is the STARR in its originally designed relational database format, the other version is the STARR-OMOP that has been further standardized into Observational Medical Outcomes Partnership (OMOP) format. STARR could be assessed through self-service tool, or through customization via the Research Informatic Center (RIC) team15.  For STARR-OMOP, researchers could get its access from the Nero-GCP, ALTLAS cohort discovery tool, or most recent implemented Advanced Cohort Engine (ACE) 16. There are two copies of the STARR-OMOP available: STARR-OMOP-deid: de-identified version for pre-IRB approval and STARR-OMOP which contained personal health information (PHI) or personal identifiable information (PII) that required IRB approval16. Stanford Medicine Technology & Digital Solution (TDS) website provided detailed information on the scope and use cases for STARR and STARR-OMOP databases and the corresponding platform for data access17. 

QSU collaborates with Stanford Medicine Technology & Digital Solution (TDS), Stanford Medicine Research Informatics Center (RIC) to navigate the investigators and make best recommendations on the most appropriate version of STARR for their specific study cases. QSU also collaborates with the Stanford Research Computing Center (SRCC) to assist the researchers to set up the Nero-GCP account that will be used for statistical analysis using STARR-OMOP data.

EHRs is a dynamic system that is developed for the clinical documentations.  Different from databases designed for research uses when a data lock could be enforced, data cleaning in EHRs could an iterative process that ideally required simultaneously clinical validations and chart reviews along the process. Given its dynamic nature, there are special challenges in ensuring reproducibility when EHRs data is used. For example, data lock and data monitoring is almost infeasible in EHRs so that consistency over time is not guaranteed. Timestamps for data entry is theoretically available but are not incorporated in our current research databases such as STARR which makes recovering a past data pull or cohort identification an extremely challenging task. A single data element (such as weight) could be recorded in multiple places in the system with varied data quality, which means different analysts could derive the same variable from different sources and with inconsistent approaches if not pre-specified. At QSU, we emphasize the implementation of the reproducibility exercises by incorporating the following principles into a data management plan (DMP), a documented established during the study design and planning stage.

• Integrate data sources across multiple EHR domains and check for consistency
• Incorporate clinical validations and chart reviews into data validation process: consider design a special database using Stanford REDCap to store chart review nots and validations so that a history records of the decisions could be easily recovered
• Record time stamps for each data pull and chart review
• Investigate the missing data mechanism and consider appropriate method to handle missing data
• Understand that the data documentation is more prone to error than the research databases and establish back-up plan for using alternative variable/proxy measures that has fewer missingness and less erroneous 
• Consider implementing a ‘test’ data pull before study launch to assess the feasibility of the DMPz

EHR (Electronic Health Record) data standardization or harmonization refers to the process of creating consistent, standardized data definitions and formats for data stored in EHR systems. This ensures that the data is meaningful and can be easily shared, compared, and analyzed across different healthcare organizations and systems19. 

Data sharing and harmonization is one of the key considerations when designing a study leveraging EHRs data. It could help to enhance the generalizability of the findings and/or allows us to explore variability across institutes. Data standard such as Observational Medical Outcomes Partnership (OMOP) should be considered for data sharing and harmonization. The data management plan (DMP) of such studies should include guidelines and protocols that outline the following key processes.

• Standardized procedures in requesting relevant data element/tables from the existing EHRs and converting them into the proposed data standard, preferably with mock-up query codes provided (such as big-query or structure query language/SQL codes)

• Detailed documentation for potential discrepant decisions made that might contribute the variability in the data pull and cleaning

• Meta data documentation

• Clinical validation approach across multiple institutes

Investigators tend to think that quantitative and qualitative results from quality improvement studies could only be reported in classic quality improvement journals, such as American Journal of Medical Quality, BMJ Quality & Safety, BMJ Quality Improvement Reports, etc. Comparing to clinical journals, quality improvement journals are typically of much lower impact factors, therefore will not likely to achieve the impact that the investigators initially envisioned in terms of the ‘research’ component of quality improvement. It is actually worthwhile to report findings from quality improvement study in higher impact clinical journals, especially when patients’ outcomes (such as hospital utilizations indicators such as length of stay, readmissions and clinical outcomes such complications and mortality, etc) are the key or primary outcomes. 

Distinguished from research initiatives, one of most important goal of QI is to achieve higher institutional impact, so that the QI team has to obtain leadership’s buy-in and make a plan to further normalize the QI interventions into regular clinical practice.  This will often require the e study team to maintain seamless communications to the hospital leadership during the study implementation. For those communications, a statistician should be present to explain the meanings behind the numbers. To be noted, those communications might not be just one single instance at the end of the study, it also likely that the leadership would like to see the progress during the process. One key caution is to make sure the team is refrained from revealing any study endpoints/outcomes if it’s not planned ahead in the study protocol. To demonstrate the progress of the study, we could consider reporting the process measures (enrollment, intervention rollout, compliance) using graphic tools like run chats or statistical control chart.

1. CMS. Quality Measurement and Quality Improvement. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/MMS/Quality-Measure-and-Quality-Improvement-

2. Lynn J, Baily MA, Bottrell M, Jennings B, et al. Special report/the ethics of using quality improvement methods in health care. Hastings Center Report. July–August 2006. The Hastings Center; http://www.thehastingscenter.org/uploadedFiles/Publications/Special_Reports/using_qi_methods_to_improve_health_care_quality_safety.pdf. 

3. Agency for Healthcare Research and Quality (AHRQ)

4. Institute for Healthcare Improvement. QI Essentials Toolkit. Institute for Healthcare Improvement. QI Essentials Toolkit. http://www.ihi.org/resources/Pages/Tools/Quality-Improvement-Essentials-Toolkit.aspx.

5. Stoto, M., Oakes, M., Stuart, E., Savitz, L., Priest, E. L., & Zurovac, J. (2017). Analytical Methods for a Learning Health System: 1. Framing the Research Question. eGEMs, 5(1). 

6. AHRQ: learning health system. https://www.ahrq.gov/learning-health-systems/about.html

7. Stanford Office of Research Compliance FAQ: Quality Assessment & Quality Improvement (QA/QI) - different to research?. URL: https://researchcompliance.stanford.edu/panels/hs/for-researchers/faqs

8. Benjamin-Chung, J., Arnold, B. F., Berger, D., Luby, S. P., Miguel, E., Colford Jr, J. M., & Hubbard, A. E. (2018). Spillover effects in epidemiology: parameters, study designs and methodological considerations. International journal of epidemiology, 47(1), 332-347.

9. Wing, Coady, Kosali Simon, and Ricardo A. Bello-Gomez. "Designing difference in difference studies: best practices for public health policy research." Annu Rev Public Health 39.1 (2018): 453-469.

10. Creswell, John W. "Mixed-method research: Introduction and application." Handbook of educational policy. Academic press, 1999. 455-472.

11. Bernal, James Lopez, Steven Cummins, and Antonio Gasparrini. "Interrupted time series regression for the evaluation of public health interventions: a tutorial." International journal of epidemiology 46.1 (2017): 348-355.

12. CMS ‘Quality Measures’. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/QualityMeasures

13. AHRQ ‘Quality Indicators’. https://qualityindicators.ahrq.gov/

14. CMS ‘Electronic Health Record’. https://www.cms.gov/Medicare/E-Health/EHealthRecords#:~:text=An%20Electronic%20Health%20Record%20(EHR,progress%20notes%2C%20problems%2C%20medications%2C

15. Stanford Research Data Repository: https://starr.stanford.edu/self-service

16. Observational Medical Outcomes Partnership (Stanford Research Data Repository): https://med.stanford.edu/starr-omop/access.html#omop_deid_access

17. STARR resources and comparisons: https://med.stanford.edu/starr-omop/summary.html

18. Institute for Healthcare Improvement: Science of Improvement: Establishing Measures https://www.ihi.org/resources/Pages/HowtoImprove/ScienceofImprovementEstablishingMeasures.aspx, accessed 2/7/2023

19. Ehrenstein, V., Kharrazi, H., Lehmann, H., & Taylor, C. O. (2019). Obtaining data from electronic health records. In Tools and technologies for registry interoperability, registries for evaluating patient outcomes: A user’s guide, 3rd edition, Addendum 2 [Internet]. Agency for Healthcare Research and Quality (US).

Ladabaum, U., Shepard, J., Weng, Y., Desai, M., Singer, S. J., & Mannalithara, A. (2023). Computer-aided detection of polyps does not improve colonoscopist performance in a pragmatic implementation trial. Gastroenterology, 164(3), 481-483.

Ladabaum, U., Mannalithara, A., Weng, Y., Shaw, B., Olsen, E., Watkins, K., ... & Singer, S. (2023). "Beliefs and attitudes about artificial intelligence (AI) among colonoscopist participants in a pragmatic implementation trial of computer-aided detection (cade) of polyps that did not replicate the positive results of randomized trials". Gastrointestinal Endoscopy, 97(6), AB763-AB764.

Baum, E., Tandel, M. D., Ren, C., Weng, Y., Pascucci, M., Kugler, J., ... & Kumar, A. (2023). Acquisition of Cardiac Point-of-Care Ultrasound Images With Deep Learning: A Randomized Trial for Educational Outcomes With Novices. CHEST Pulmonary, 100023.

Bio, L. L., Weng, Y., & Schwenk, H. T. Antifungal stewardship in practice: Insights from a prospective audit and feedback program. Infection Control & Hospital Epidemiology, 1-5.

Chen, L., Glatt, E., Weng, Y., Kerr, P., & Lough, M. E. (2023). A Standardized Stir-up Regimen Shortens Phase I Recovery Time in PACU. Journal of PeriAnesthesia Nursing, 38(4), e26.

Ong’uti, S. K., Artandi, M., Betts, B., Weng, Y., Desai, M., Lentz, C., ... & Holubar, M. K. (2023). A quality-improvement approach to urgent-care antibiotic stewardship for respiratory tract infections during the COVID-19 pandemic: Lessons learned. Infection Control & Hospital Epidemiology, 1-6.

Gold, C. A., Scott, B. J., Weng, Y., Bernier, E., & Kvam, K. A. (2022). Outcomes of a Neurohospitalist Program at an Academic Medical Center. The Neurohospitalist, 12(3), 453-462.

Singhal, S., Krishnamurthy, A., Wang, B., Weng, Y., Sharp, C., Shah, N., ... & Hom, J. (2022). Effect of electronic clinical decision support on inappropriate prescriptions in older adults. Journal of the American Geriatrics Society, 70(3), 905-908.

Goldstein, R. L., Keppler, H., Pineda, N., Weng, Y., & Carlson, J. L. (2022). Implementation of an inpatient reproductive health screening consult service. Hospital Pediatrics, 12(3), 303-310.

Rohatgi, N., Weng, Y., Kittle, J., & Ahuja, N. (2021). Merits of Surgical Comanagement of Patients With Hip Fracture by Dedicated Orthopaedic Hospitalists. JAAOS Global Research & Reviews, 5(3).

Durstenfeld, A., Chen, C., Moss, J., Lopez, M., Pyke-Grimm, K., Weng, Y., ... & Kache, S. (2021). 1238: Do Targeted Quality Improvement Interventions Improve Pediatric Sepsis Bundle Compliance?. Critical Care Medicine, 49(1), 623.

Kumar, A., Weng, Y., Bentley, J., Almli, M., Hom, J., Witteles, R., ... & Kugler, J. (2020). Portable ultrasound device usage and learning outcomes among internal medicine trainees: a parallel‐group randomized trial. Journal of Hospital Medicine, 15(3), 154-159.

Rohatgi, Nidhi, Yingjie Weng, and Neera Ahuja. "Surgical comanagement by hospitalists: continued improvement over 5 years." Journal of Hospital Medicine 15.4 (2020): 232-235.

Rohatgi, N., Weng, Y., Bentley, J., Lansberg, M. G., Shepard, J., Mazur, D., ... & Hopkins, J. (2019). Initiative for prevention and early identification of delirium in medical-surgical units: lessons learned in the past five years. The American journal of medicine, 132(12), 1421-1430.

Tayyar, R., Kiener, M. A., Liang, J. W., Contreras, G., Rodriguez-Nava, G., Zimmet, A. N., Contag, C. A., Srinivasan, K., McIntyre, K., Subramanian, A., Shepard, J., Tompkins, L. S., Pinsky, B. A., & Salinas, J. L. (2023). Low infectivity among asymptomatic patients with a positive severe acute respiratory coronavirus virus 2 (SARS-COV-2) admission test at a tertiary care center, 2020–2022. Infection Control & Hospital Epidemiology, 45(2), 241–243. https://doi.org/10.1017/ice.2023.210.

Lu, B., Ha, D., Shen, S., Ferguson Toll, J., Kim, A., Kim, S., Mui, E., Deresinski, S., Holubar, M., & Alegria, W. (2024). Combination antifungal therapy for invasive mucormycosis in immunocompromised hosts: A single-center experience. Open Forum Infectious Diseases. https://doi.org/10.1093/ofid/ofae103

Oke, O., Sullivan, K. M., Hom, J., Svec, D., Weng, Y., & Shieh, L. (2024). Quality improvement project to reduce medicare 1-day write-offs due to inappropriate admission orders. BMC Health Services Research, 24(1), 204.

Hansen K, Jenkins E, Zhu A, Collins S, Williams K, Garcia A, Weng Y, Kaufman B, Sacks LD, Cohen H, Shin AY. A parental communication assessment initiative in the paediatric cardiovascular ICU. Cardiology in the Young. 2024:1-9. 

Dujari, S., Scott, B. J., Gold, C. A., Weng, Y., & Kvam, K. A. (2024). Education Research: Educational Outcomes Associated With the Introduction of a Neurohospitalist Program. Neurology® Education, 3(2), e200131. 

Watkins, K., Ladabaum, U., Olsen, E., Hoogerbrug, J., Mannalithara, A., Weng, Y., ... & Singer, S. (2024). Exploring human-artificial intelligence interactions in a negative pragmatic trial of computer-aided polyp detection. IGIE.