AI Agents in Healthcare: Use Cases, Risks & Safeguards

On October 6, 2026, the U.S. Department of Health and Human Services named MedStar Health Research Institute as the recipient of a LEAP in Health IT award to accelerate agentic AI in clinical care. The project plans open-source middleware to orchestrate clinical follow-ups [1]. In August, the FDA asked for public input on how it might regulate generative AI-enabled medical devices, including agentic systems [2]. An award funds work. It is not evidence of patient outcomes.

AI agents in healthcare are moving beyond answering questions. They can coordinate multi-step work across systems, summarize records and route tasks for review. That promise comes with a sharper boundary than in office automation. This guide separates administrative support from clinical functions, shows where a qualified person stays accountable, and explains how to handle privacy, integration and monitoring before a pilot touches real workflows.

This article is general information, not clinical, legal or regulatory advice. Confirm your obligations with your compliance and clinical governance teams.

Key takeaways

“Agent” is not a regulatory category. Intended use and actual function decide whether FDA device rules, HIPAA or the EU AI Act apply.
Start with administrative and workflow-support tasks where a named person reviews every output before it affects care.
Connect agents to the EHR with their own identity, least-privilege FHIR scopes and a full audit trail.
The FDA’s August 2026 discussion paper is not guidance, but its risk logic is useful: how independently a function acts, and how severe an incorrect output could be [2].
Measure pilots on correction, escalation and omission rates against a baseline, and expand autonomy only when evidence supports it.

What are AI agents in healthcare, and what are they not?

The FDA describes agentic AI systems as generative AI systems that “autonomously plan and execute multi-step tasks, use external tools, or take actions across a sequence of steps” [2]. The ONC funding notice behind the 2026 awards uses a similar definition: systems that break complex objectives into subtasks and navigate workflows by interacting with tools and databases [3].

That separates an agent from three familiar tools:

A chatbot answers a question. It does not act in other systems.
A rules engine follows fixed logic written in advance.
Conventional clinical decision support (CDS) presents information or recommendations to a clinician, who decides.

“Agent” is not a regulatory classification. What matters is intended use and what the software actually does. The FDA’s January 2026 CDS guidance sets four criteria a software function must meet to fall outside the device definition, including that a health care professional can independently review the basis for a recommendation rather than rely primarily on it [4]. The FDA’s discussion paper notes that agents used for care coordination, documentation, outreach and workflow support may not be a focus of device oversight, but that some agentic systems could meet the device definition, for example if their actions control another medical device [2]. Make that call per use case, with regulatory counsel.

For the wider shift in care delivery, see our coverage of digital health trends. This guide stays on agents.

Healthcare use cases with a clear human owner

Group use cases by what the agent touches, and name a human owner for each before you build. ONC’s notice lists examples such as care coordination, record reconciliation, prior authorization and patient follow-up reminders [3]. Treat these as candidates, not proven results.

Administrative operations

Intake, scheduling, referral routing and prior authorization packet preparation. The agent gathers documents and drafts. Staff check and submit.

Clinical-workflow support

Record summaries before a visit, retrieval of guidelines or prior results, and draft documentation. A clinician reviews and signs. The agent never finalizes clinical content.

Care coordination

Follow-up tracking, gap-in-care reminders and approved patient outreach. Care managers set the outreach rules and handle escalations.

For each use case, document the user, the input data, the output or action, the reviewer and the escalation path. Do not design for autonomous diagnosis or treatment.

Use case	Data sensitivity	Agent action level	Reviewer	Evidence gate before scaling
Appointment scheduling	Low to moderate	Acts within set rules	Scheduling staff	Error and rebooking rate vs. baseline
Prior authorization prep	High	Drafts the packet	Authorization specialist	Correction rate, denial rate
Pre-visit record summary	High	Drafts the summary	Treating clinician	Omission rate on audited charts
Gap-in-care outreach	Moderate to high	Sends approved templates	Care manager	Escalation handling, complaints
Dosing or diagnosis decisions	Highest	No autonomous action	Clinician decides	Clinical governance and regulatory review

Other sectors face the same split between preparing work and approving it. Our guide to AI agents in finance covers the same pattern in another regulated sector.

How agents connect to EHRs and clinical workflows

Integration decides how much harm a mistake can do. Five controls carry most of the weight.

Own identity. Register the agent as its own application or service account, never a shared clinician login, so every action is attributable.
Least-privilege access. In FHIR integrations, SMART App Launch scopes let an app request specific permissions, such as read and search on one resource type, and HL7 encourages clients to request only the scopes they need [5]. A scheduling agent needs no access to progress notes.
Standard APIs. ONC required its funded agentic AI solutions to be compatible with FHIR and interoperable with third-party applications [3]. Prefer documented APIs over screen scraping.
Provenance. Every summary or draft should cite the records it used, so the reviewer can check the source in one click.
Audit trail and safe handoff. Log each retrieval, proposed action, approval, edit and override. When the agent meets uncertainty, it should hand off with context, not guess.

The diagram shows the pattern: request, scoped retrieval, proposed action, human review and a recorded outcome. ONC also expects humans to be able to override or modify AI recommendations easily, and asks developers to track when and why overrides happen [3]. Overrides are your earliest signal of where the agent is wrong.

Safety, privacy and governance safeguards

Put governance in place before the pilot, not after the first incident. Cover six areas.

Patient-data minimization

HIPAA’s minimum necessary standard generally requires covered entities to take reasonable steps to limit uses, disclosures and requests of protected health information to the minimum necessary for the purpose, with exceptions such as disclosures to a provider for treatment [6]. Apply the same discipline to prompts, retrieval and logs. Do not copy full records into a prompt or trace when a few fields will do.

Contracts and vendors

HHS guidance on HIPAA and cloud computing treats a provider that creates, receives, maintains or transmits electronic PHI on your behalf as a business associate that needs a business associate agreement (BAA), even if it stores only encrypted data [7]. Confirm the BAA covers the exact product, model endpoints and logging you use.

Human review and exception handling

ONC’s notice calls for escalation protocols that recognize when human judgment is needed, and for telling patients when they are interacting with AI [3]. Define who reviews, how fast, and what happens when no reviewer is available.

Validation by intended use

Validate against the specific workflow and population, not a general benchmark. The FDA’s discussion paper, which it states is not draft or final guidance, floats a two-axis view of risk: how independently a function acts, and the severity of harm from relying on an incorrect output [2]. The agency requested comments by October 19, 2026 [8]. Use the same logic internally even where device rules do not apply.

Monitoring and incident ownership

Name an owner for each agent. Monitor correction and override rates, and route incidents through your existing patient-safety reporting process.

EU deployments

Under the EU AI Act, AI systems that are medical devices requiring third-party conformity assessment under the EU medical device regulations are treated as high-risk [9]. After the AI Omnibus entered into force on July 27, 2026, the European Commission lists August 2, 2028 for high-risk rules on AI embedded in regulated products and December 2, 2027 for Annex III high-risk uses [10].

For the broader picture of where agent architectures are heading, see our overview of agentic AI trends.

How to evaluate a pilot before scaling

Measure the workflow, not the model. Set a baseline from the current process, then compare like for like on the same case mix.

Measure	What it tells you	How to collect it
Completion time	Time saved end to end, including review	Timestamps from request to recorded outcome
Correction rate	How often reviewers change the output	Compare draft with final version
Escalation rate	How often the agent hands off, and whether it should have	Handoff logs plus sampled review
Omission rate	Missing facts in summaries or packets	Audit a sample against the source record
User burden	Whether review work replaced or added to existing work	Short surveys, time sampling
Override reasons	Where and why people reject proposals	Structured override codes

Before you scale, require five things:

A limited pilot: one site, one workflow, a defined case population.
Documented failure modes, including actions the agent must never take.
A rollback plan that restores the manual process quickly.
Representative testing across the patient groups, languages and record types the workflow actually sees.
Joint clinical and operational sign-off on the results.

ONC’s funded project must publish an assessment of whether its AI-enabled workflow improves efficiency, accuracy or clinical outcomes compared with non-AI baselines, and whether it is ready for production [3]. Hold your own pilots to the same standard.

A practical deployment roadmap

Sequence the work so each stage produces evidence for the next.

Select the workflow: high volume, low consequence, clear owner.
Review risk: intended use, regulatory status, data flows, failure modes.
Prepare data and integration: identity, scopes, BAA, logging, retention.
Test in a sandbox on synthetic or de-identified data and historical cases.
Run a supervised pilot with full human review at first.
Monitor and expand with thresholds that trigger rollback.

Ask vendors four questions before you sign:

What actions can the agent take, and which require approval?
How are sources cited, and can reviewers open them?
What data is retained, where, for how long, and is it used for training?
How fast can you revoke the agent’s access, and what happens to tasks in progress?

Summary

AI agents in healthcare can take on intake, preparation, summarization and coordination work, but only inside a workflow where a named person reviews, approves or escalates. Treat intended use as the regulatory question, integrate with least-privilege access and full audit trails, and measure pilots against a real baseline.

Expand autonomy only when evidence and controls justify it. Your next step: pick one administrative workflow, name its human owner and write down the measures that would justify moving past a supervised pilot.

FAQs

What AI applications are used in healthcare?

Conventional AI applications include image analysis, risk scores and language tools that summarize or draft text. Agentic systems go further: they plan steps and use tools, for example to prepare a prior authorization packet, reconcile records or track follow-ups. Whether any application helps patients depends on evidence for its specific intended use and setting, not on the category.

Is ChatGPT for healthcare HIPAA compliant?

No AI product is HIPAA compliant on its own. Compliance depends on how a covered entity or business associate uses it: the product configuration, a business associate agreement where the vendor handles PHI, data flows, retention and access controls [7]. Do not enter patient data into consumer chat tools that lack those arrangements. Check the vendor’s current terms and your own obligations.

Can AI agents make clinical decisions?

Clinical decisions should stay with a qualified clinician. Under FDA guidance, software that supports clinicians can fall outside device rules only if, among other criteria, the clinician can independently review the basis for its recommendation [4]. Agents that direct treatment or act autonomously raise device and patient-safety questions that need regulatory review, validation for the intended use and clear accountability.