AI assists clinicians in responding to patient messages at Stanford Medicine

Stanford Medicine researchers have found that large language models can help draft responses to patient portal messages, reducing health care providers’ workload and alleviating burnout.

The artificial intelligence-generated drafts, which are reviewed and edited by a clinician before they are shared with the patient, help respond to clinical inquires, such as what to do about symptoms of a cold or side effects of a medication. In a study of the AI-generated responses, clinicians reported a reduction in everyday clerical burden and fewer feelings of burnout.

When the large language model GPT, which powered the draft responses, was introduced in late 2022, it ignited a wave of excitement and anticipation about the potential of artificial intelligence.

“It got everyone in medicine thinking, ‘Hey this tool is so great at creating language, how could it be useful to us?’” said Patricia Garcia, MD, clinical associate professor of medicine and the associate chief medical information officer at Stanford Health Care. “It didn’t take long for folks to think about writing messages, summaries, clinical notes — we all saw a lot of potential for anything that involves language content generation.”

This is an early demonstration of how integrating generative AI into health care workflows with a “human in the loop” can assist providers, said Michael Pfeffer, MD, Stanford Health Care’s chief information officer. “We’re always trying to find ways to have the electronic health record work with clinicians through automation. Already, clinicians are noting a reduction in cognitive burden — and the AI is only going to improve from here.”

A paper detailing the study published in JAMA Network Open on March 20. Garcia and Stephen Ma, MD, PhD, clinical informatics fellow, led the research, supported by the Stanford Department of Medicine and Stanford Technology and Digital Solutions team.

“While multiple published studies show potential promise for generative AI in health care, this is among the first clinical use to be rigorously evaluated — which is critical to assess real-world safety and usefulness,” said Christopher Sharp, MD, chief medical information officer at Stanford Medicine and the senior author of the study.

Stanford Medicine has been incorporating various AI tools into health care while ensuring patient safety and privacy, with guidance from the RAISE Health initiative.

Code-to-bedside medicine

Garcia and the team evaluated the responses generated by the large language model, which is integrated into electronic health records and complies with the Health Insurance Portability and Accountability Act.

Upon receipt of a message, the model generated a draft response to the patient’s message within seconds. Integration with the electronic health record allowed the draft responses to appear in the clinician’s inbox, where they could seamlessly review them, make any necessary edits and send the final response back to the patient. The responses were generated over the course of five weeks in July and August 2023 on behalf of 162 primary care and gastroenterology clinicians.

This was a serendipitous alignment where an amazing new technology came to fruition during a time when we needed a different, innovative solution. 

Stanford Medicine is one of the first to publish about their experience using a large language model to generate draft responses for patient messages and to enable the use of such a tool by a broader care team that includes nurses, advanced practice clinician and pharmacists.

“We know that in-basket messaging and responding to patients is a team sport,” Garcia said. “I think what makes this study particularly impactful is that it’s capturing what it’s like to deploy this tool in a way that would be practical and reflective of a real care environment.” She noted that it’s not just doctors who respond to patients — nurses, advanced practice clinician and pharmacists all work collectively to answer patient messages.

Alleviating burden

After the pilot period, Garcia and the team issued a survey to the clinicians, asking them to report on their experience. They reported that the AI-generated drafts lightened the cognitive load of responding to patient messages and improved their feelings of work exhaustion despite objective findings that the drafts did not save the clinicians’ time. That’s still a win, Garcia said, as this tool is likely to have even broader applicability and impact as it evolves.

“We know that clinicians and care teams have been under a lot of pressure, even before the pandemic, and health care just hasn’t been able to find a good solution for burnout,” Garcia said. “This was a serendipitous alignment where an amazing new technology came to fruition during a time when we needed a different, innovative solution. We’re already seeing a positive impact on relieving burden, and I think it’s just going to get more effective as we continue to iterate.”

Garcia added that Stanford Health Care is planning to expand the use of this tool to additional clinicians in the coming months.

For more news about responsible AI in health and medicine, sign up for the RAISE Health newsletter.

Register for the RAISE Health Symposium on May 14.