Duke’s Human Approach to AI in Health Care

By Megan Hujber

Before the first patient arrives and clinic hallways begin to fill, a physician at Duke Health opens an inbox crowded with refill requests, follow-up questions and messages that may signal something urgent. In hospitals across the country, this is where modern health care often begins: not at the bedside, but in the flood of digital information clinicians must sort before the day even starts.

Artificial intelligence increasingly promises to help. It can draft clinical notes, prioritize patient messages, flag risks and reduce administrative burden. But at Duke, the more important question is not what AI can do. It is whether it should be doing it at all.

That question connects the work of two Duke leaders helping shape how health care systems nationwide think about responsible AI adoption: Nicoleta Economou-Zavlanos and Michael Cary. One focuses on evaluating whether AI tools are safe, useful and ready for care. The other examines whether those same tools operate equitably across patient populations. Together, their work reflects a growing truth in medicine: innovation without oversight is not enough.

Economou-Zavlanos leads the Duke Health AI Evaluation & Governance Program and the Algorithm-Based Clinical Decision Support Oversight initiative. Her team starts with practical questions. “If a tool is designed to help prioritize patient messages or draft clinical notes, does it actually reduce workload without introducing errors?” she said. “If the answers are not clear, or if the tool creates more risk than benefit, we do not move forward.”

That philosophy is reshaping how AI enters health care. At Duke, systems are reviewed not only for technical performance, but for clinical value, safety, fairness, usability, transparency and regulatory compliance. A model that performs well in a demonstration still must prove itself on Duke’s local patient data and within Duke’s clinical workflows. Clinicians test whether outputs are understandable and whether the system supports care rather than complicating it.

For patients, that process may show up in subtle but meaningful ways. A nurse may be able to identify urgent messages faster. A physician may spend less time typing notes and more time talking with patients. Follow-up care may happen more efficiently because the technology was carefully chosen before it was deployed.

But even an efficient tool can fail patients if it reproduces inequities already embedded in health care systems. That is where Cary’s work becomes critical.

A Duke School of Nursing professor and AI Health Faculty Council member, Cary is dually trained as a health services researcher and applied data scientist. His work uses AI and machine learning to identify clinical algorithms that perpetuate racial and ethnic disparities, and to implement standards to reduce harm.

“Clinical algorithms can reinforce inequities in a few different ways,” Cary said. “One of the most common is through the data they’re trained on. If historical care patterns reflect disparities … the algorithm can learn those patterns and treat them as the norm.”

He points to a striking example: some algorithms have used health care spending as a proxy for illness severity. But because many marginalized populations historically had less access to care, lower spending did not necessarily mean lower need. It can also lead the algorithm to underestimate risk.

In everyday life, that could mean a patient who needs extra outreach most is never flagged for follow-up. It could mean fewer resources reaching rural communities or historically underserved neighborhoods. It could mean a risk score quietly steering care away from the people who need it most.

Cary argues that fairness cannot be judged by accuracy alone. “Equity isn’t just about prediction accuracy,” he said. “It’s about whether patients benefit equally from the care decisions those predictions support.”

That principle is increasingly relevant nationwide as hospitals, regulators and policymakers search for standards governing AI in medicine. Duke’s work offers a model. Economou-Zavlanos has contributed to national frameworks through leadership with the National Institutes of Health Bridge2AI Program and the Coalition for Health AI, helping develop guidelines for AI assurance in health care. Cary’s research adds another essential layer: systems must be monitored not only for safety, but for who benefits and who may be left out.

Both emphasize that approval is not the end of the story. AI systems can drift as populations change and workflows evolve. They require continuous monitoring, clear accountability and a workforce trained to question unexpected outputs.

The future of health care AI may not be defined by the smartest algorithm or the fastest software. It may be defined by whether institutions build systems worthy of trust.

At Duke, that future is already taking shape. Inboxes are sorted more safely, clinicians can be supported thoughtfully, and patients are able to be served more fairly because someone asked hard questions before turning the technology on.