AI Should Not Prescribe Alone: New Framework Calls for Pharmacist Oversight of Agentic Clinical Systems
Introduction
As artificial intelligence moves beyond simple chatbots and into systems that can independently generate medication orders, a fierce policy debate is unfolding across healthcare: should AI ever be allowed to prescribe alone? A new article published in Cureus argues that the answer is no — and proposes a risk-stratified pharmacy governance framework designed to keep pharmacists firmly in the loop as agentic clinical AI systems become more autonomous.
The Rise of Agentic AI in Healthcare
Agentic clinical AI refers to systems that do more than answer questions or provide decision support. These tools can actively initiate medication-related recommendations, order sets, or even full prescriptions, raising the stakes from passive assistance to clinical action. Unlike the conversational agents many clinicians are accustomed to, agentic systems can operate with a degree of independence that has regulators, pharmacists, and patient safety advocates deeply concerned.
Hospitals and health systems are beginning to pilot such technologies, drawn by the promise of streamlined workflows and reduced administrative burden. Yet the leap from generating suggestions to executing clinical decisions without a human in the middle demands a new kind of safety net — one that the authors of the Cureus piece argue must be built on pharmacist oversight.
The Risk-Stratified Pharmacy Governance Model
The central policy claim of the article is straightforward: artificial intelligence should not prescribe alone. To operationalize that principle, the authors outline a risk-stratified governance framework. Under this model, the level of pharmacist review would be matched to the risk profile of the medication, the complexity of the patient, and the clinical setting.
AI should not prescribe alone, especially in higher-risk medication contexts.
For low-risk, routine medications in stable patients, AI might be allowed to generate preliminary orders that are queued for a pharmacist’s lightweight sign-off. For high-alert drugs, complex polypharmacy regimens, or intensive care scenarios, the framework would mandate in-depth pharmacist verification before any order reaches the patient. This sliding scale of human oversight aims to balance efficiency with safety, preventing unchecked automation errors while avoiding a one-size-fits-all bottleneck.
Why AI Can’t Go It Alone: Core Safety Concerns
Several risks underscore the call for pharmacist-led governance. Medication errors — ranging from wrong-dose recommendations to dangerous drug interactions — top the list. Agentic AI, however sophisticated, can misinterpret context, overlook subtle contraindications, or generate inappropriate advice when faced with incomplete patient data. The article emphasizes that the challenge is not only about algorithm accuracy but also about accountability. When an AI system prescribes, who is legally and professionally responsible if something goes wrong? Without a pharmacist in the loop, the answer becomes dangerously murky.
In addition, real-world pharmacy workflows are messy. AI outputs must be validated against dynamic patient records, lab values, and clinical nuances that may not be fully captured in structured data. The authors of the Cureus piece caution that healthcare organizations could face institutional liability if they deploy agentic prescribing tools without rigorous, ongoing auditing and monitoring — something a governance framework would formalize. As the FDA continues to refine its approach to AI/ML-based software as a medical device, questions about clinical decision support versus autonomous prescribing are increasingly urgent.
Implications for Hospitals, Regulators, and the Profession
The debate extends far beyond a single journal article. It touches the very architecture of care delivery. Hospitals will need to establish clear policies for approving, deploying, and auditing agentic AI tools. Pharmacy governance stakeholders — including professional associations and medical informatics groups — will be called upon to define standards of practice. The WHO’s guidance on ethics and governance of AI for health similarly frames the need for human-centric oversight, reinforcing the message that autonomous clinical decisions must not be wholly delegated to machines.
For pharmacists, the framework reasserts their indispensable role as medication experts. Rather than being displaced by AI, they become essential gatekeepers, ensuring that technology augments clinical judgment instead of bypassing it. The proposed model also gives regulators a template for tiered oversight, recognizing that not all AI-driven medication decisions carry the same level of danger.
Patient safety remains the ultimate yardstick. As agentic AI systems inch closer to real-world clinical use, the article’s risk-stratified approach offers one possible path forward — one where innovation is welcomed but never at the cost of removing the trained human eye from the final check.
