AI’s Rapid Rise in Medicine Sparks a Call for Guardrails

October 21, 2025 - by Rebecca Handler

Artificial intelligence has entered medicine with astonishing speed. Once confined to research labs, algorithms now help diagnose disease, predict sepsis, process insurance claims, transcribe clinic visits, and even chat with patients about their mental health. In the excitement about AI's potential to improve healthcare, it can be easy to forget how untested many tools still are.

A new report from the Journal of the American Medical Association (JAMA) brings that reality into focus. Drawing on a Summit convened by the journal involving more than sixty experts, including Stanford Medicine faculty Tina Hernandez-Boussard, PhD, Nigam Shah, MBBS, PHD, and Michelle Mello, JD, PhD, the report defines what responsible AI in health care should look like, maps out how deeply AI has penetrated health care, and describes how little structure exists to ensure it’s safe, fair, and effective.

The authors’ message is clear: while AI’s integration into medicine is inevitable, its benefits are not. Without a coordinated system for evaluating, regulating, and monitoring these tools, the technology could deepen inequities as easily as close them.

From the Clinic to the Cloud

AI’s potential reach now spans every layer of health care. In hospitals, algorithms can scan X-rays for signs of pneumonia, monitor vital signs for sepsis, and generate physician notes in real time. At home, consumers use mobile apps that analyze heart rhythms or skin lesions, while chatbots promise instant advice on anxiety, diet, or sleep. Behind the scenes, hospitals can deploy AI to manage operating room schedules, predict supply shortages, and optimize billing.

Some of these systems have FDA clearance as medical devices. Most do not. And even among those that do, few have been tested for what truly matters: whether they make people healthier. “These tools can have important health effects, good or bad, but those effects are often not quantified,” the authors write, citing the lack of regulatory standards and the difficulty of running controlled studies in messy, real-world environments.

So what makes the research so difficult? There are multiple factors. One is that there’s little appetite or resources for the kind of drawn-out studies typically done for new technologies. Another is that the technology moves faster than the science can follow. A sepsis-prediction algorithm can be updated weekly. A generative AI model can change daily. By the time a traditional clinical trial finishes, the tool being tested may no longer exist in the same form.

The Oversight Gap

In contrast to pharmaceuticals, AI in medicine has no unified regulatory pathway. The FDA reviews some algorithms that diagnose or treat disease, but its jurisdiction stops short of administrative software, wellness apps, and many clinical decision-support systems.

That leaves vast areas of health care untouched by federal oversight. A chatbot offering mental health support, for example, can reach millions of users without proving its safety or accuracy. Overwhelmingly, the AI tools subjected to FDA clearance are imaging tools – for example, an algorithm that helps detect spots on a mammogram. “Even when the FDA does have authority,” the report notes, “clearance does not necessarily require demonstration of improved clinical outcomes.”

In the absence of regulation, the marketplace has become the de facto testing ground. Hospitals and consumers are learning through trial and error. The problem, as former FDA Commissioner Robert Califf warns in the report, is that “no health system in the United States is currently capable of validating an AI algorithm once it’s in use.”

A System Built for Drugs, Not Data

Regulatory systems in medicine, such as the FDA’s processes, clinical trial structures, and oversight rules, were designed for products like drugs or medical devices, which are stable and unchanging once approved. A pill or implant behaves the same way no matter who takes it or where it’s used, so regulators can evaluate it before it reaches the market.

AI tools are completely different. They are dynamic and context dependent. They learn from new data, adapt over time, and behave differently depending on how clinicians use them or to which populations they’re applied. That makes them moving targets for evaluation. The usual premarket testing model doesn’t work because an algorithm that’s “safe and effective” today could change tomorrow once it starts learning from new users or new data.

A sepsis alert might perform flawlessly in one hospital but fail in another because of differences in staffing or workflow. A wearable heart monitor might succeed with tech-savvy patients but miss early warning signs in older adults.

Traditional clinical trials aren’t well suited to such moving targets. They’re expensive, slow, and rigid. By the time results emerge, the code may have changed a dozen times. What’s needed, the authors say, are more flexible methods: embedded “platform” trials, rapid A/B testing, and continuous monitoring systems that can adapt alongside the technology.