How to Take AI Agents in Healthcare From Pilot to Production

Picture a prior-authorization agent that has drafted requests in a six-week pilot, and the clinic’s staff like what it produces. Then comes the scale-up meeting. Compliance asks where patient data goes. The clinical lead asks who decided the drafts were correct. Security asks what happens when a referral note contains instructions for the agent.

The team brings an accuracy number, and it answers none of the three questions. Each reviewer needs a different kind of proof, none of it was collected while the pilot ran, and the meeting ends with a follow-up date instead of a decision.

You can build each reviewer’s evidence while the pilot still runs, simulate risky patients before launch, and roll out without restarting the review.

TL;DR

• Production sign-off in healthcare comes from three reviewers: the compliance officer, the clinical lead, and the security team.
• Compliance needs traces, PHI check results, deployment choices, and business associate agreements.
• The clinical lead needs clinician-written rubrics, expert labels, judges checked against those labels, and code checks for required fields.
• Security needs guardrails on inputs and outputs, prompt-injection test results, and least-privilege access.
• Simulate risky patient personas before launch, then roll out with online evals and shadow traffic.

Where AI agents in healthcare are landing first

AI agents in healthcare are software systems that use language models and tools to complete clinical or administrative tasks, with a person reviewing the output. Most teams start in workflows that already have a review step, so the agent drafts and a human approves. Four workflow types account for most early deployments.

Clinical documentation

Agents draft visit notes, discharge summaries, and referral letters for a clinician to edit and sign. The risk that shapes review is accuracy inside the record. A missed allergy or a wrong medication in a draft can carry into care if the clinician signs without catching it, so reviewers want proof that drafts stay faithful to the source.

Prior authorization and claims

Agents assemble prior authorization requests from the chart and the payer’s rules, or prepare claims for coding review. These workflows touch a lot of patient data and send it outside the organization. Reviewers focus on whether the agent pulls only what the request needs and whether every required field is present.

Patient intake and support

Agents handle scheduling, intake questions, and benefits questions over chat or phone. This is the first patient-facing step for many teams, and it brings new risks. A patient may ask for medical advice, describe an emergency, or share more personal information than the task needs, and the agent has to handle each one safely.

Medical information and literature search in life sciences

AI agents in life sciences draft documents, review literature, and prepare medical information responses for pharma teams. The output often goes to healthcare professionals, so claims need sources and approved wording. Reviewers here care about traceability, meaning every statement can be tied back to a document the agent actually retrieved.

Why healthcare agent pilots stall before production

A pilot is judged on usefulness: does the agent save staff time, and do people like its drafts? Production is judged on risk, by three reviewers who each ask a different question. None of them is asking how often the agent is right on average.

Reviewers usually meet once the pilot looks promising, which leaves little time to gather proof. If the evidence grows with the pilot, the first meeting becomes a walkthrough of artifacts the reviewers can check, and the follow-up questions shrink to the gaps that remain.

The fix is an evidence pack, a set of artifacts built for each reviewer while the pilot runs. Collecting it during the pilot is cheap, since the traffic is already flowing, while collecting it afterward means re-running weeks of work. Our guide to productionizing agentic applications covers the general version of this transition.

A few questions sit outside the pack, and each needs a named owner. If the agent recommends a diagnosis or treatment, check where it falls under FDA’s clinical decision support guidance, last revised in January 2026, since some decision support counts as a medical device.

Some states require a disclosure when generative AI writes patient-facing clinical messages that no clinician reviewed, so check the laws where your patients live. If the agent ships inside certified EHR software, the developer may also owe ONC’s decision support transparency requirements.

What the compliance officer needs to see

Compliance wants to know where protected health information flows and who can see it at each step. A compliance officer usually asks about four things first, and each one maps to an artifact you can bring.

Traces as part of the audit trail

The HIPAA Security Rule’s audit controls standard (45 CFR 164.312(b)) requires covered entities and business associates to record and examine activity in systems that contain or use electronic PHI. Traces can support those audit controls, though tracing alone does not make an agent compliant.

Record every agent, LLM, and tool span and attach eval scores on spans, traces, or whole sessions, so the record shows what happened and how each step was checked.

A compliance officer can sample real sessions and see the PHI checks that ran on each one, instead of taking the team’s word for it.

For the review itself, bring a small sample the compliance officer can read in about an hour. Pick sessions across workflows, including a few where a PHI check fired, and show what happened next. A caught problem proves the checks run, and clean runs alone cannot.

PHI checks with their results

PHI can show up at several points in an agent loop, and each point needs its own control and its own evidence.

The minimum necessary standard applies to the agent’s own retrieval from the chart, because that is a use of PHI, and the role-based access logic of 164.514(d) applies to it the same way it applies to staff.

It also applies to a prior authorization request, which is a payment disclosure to the payer. Data elements the HIPAA transaction standard requires are exempt, but optional elements and attached notes are not, so the agent should pull only what the request needs. Disclosures to a provider for treatment and to the patient are exempt.

Data handling and deployment choices

Compliance will ask where the system runs and where its data rests. Decide between a cloud region and self-hosting, set a retention period for traces, and write down who can view them. A one-page deployment diagram that shows each data store and who can access it answers most of these questions in a single artifact.

Retention deserves its own line in that diagram. Keep the access and audit log for as long as your policy requires, and treat content-bearing debug traces separately. One common pattern masks identifiers before storage and keeps raw traces for a short window.

Masked text is not de-identified under 164.514(b) unless it meets Safe Harbor or Expert Determination, so masked traces usually stay PHI and keep their access controls.

Vendor attestations

A vendor that creates, receives, maintains, or transmits PHI on your behalf is a business associate, and it needs a signed BAA before it receives real patient data, even during the pilot.

A pilot on properly de-identified data needs no BAA, but per HHS cloud computing guidance, a vendor that only stores encrypted data it cannot read still does.

HHS lists a few exceptions, such as a pure transmission conduit, so confirm each vendor’s status with your compliance team, and collect SOC 2 reports and other security attestations.

List every vendor in the agent’s path, including the model provider, the observability platform, the hosting provider, any speech or document-reading service, and their subcontractors. Note the BAA status and attestation dates for each one in a short table.

Our guide to HIPAA-compliant voice AI covers the vendor paperwork for a voice agent in more depth.

What the clinical lead needs to see

The clinical lead asks who decided this output is correct. The answer is clinical judgment captured in a form you can re-run, so quality stays measured after the pilot ends.

Rubrics clinicians wrote

Ask the clinicians who will rely on the agent to write the criteria in their own words. For prior authorization, that might be complete clinical rationale, correct diagnosis codes, and no promises about coverage. Criteria in the team’s own words are the ones the clinical lead will sign.

Keep each rubric item testable. Complete clinical rationale should name what complete means, such as the diagnosis, prior treatments tried, and why the requested service fits. Items written that precisely turn into labels clinicians agree on, and later into checks a judge or a code rule can run.

Expert labels in a review queue

Have clinicians label a sample of real drafts against the rubric in a review queue. Where two clinicians disagree, discuss the case and refine the rubric wording. The labeled sample becomes your ground truth, and it shows the clinical lead that their own team judged the output.

Judges checked against those labels

An LLM judge can score every draft, but trust it only where it agrees with the clinician labels. Run the judge on the labeled sample and report agreement per rubric item, as percent agreement or Cohen’s kappa, with the sample size and the threshold you agreed on with the clinical lead.

Report agreement for each rubric item separately. A judge might match clinicians closely on required sections and drift on clinical rationale, and only the per-item view shows that. Where agreement is weak, keep a clinician in the loop for that item and re-check the judge after each rubric change.

Slice the same scores by patient subgroup, such as language, age, sex, and payer. An overall average can hide a subgroup that fails.

If the agent supports clinical decisions, the Section 1557 rule (45 CFR 92.210) also applies to most organizations that receive HHS funding: they must identify tools that use race, sex, age, disability, or similar inputs and reduce the discrimination risk.

Built-in checks help too, such as a clinically inappropriate tone check for patient-facing text.

Code checks for required fields and structure

Some checks never need a model. A prior authorization draft either fills every required section or it does not, and a labeled medical record number either appears or it does not. A clinical lead can read deterministic checks like these line by line, and they run in milliseconds.

import re

REQUIRED_SECTIONS = ("Diagnosis code", "Requested service", "Clinical rationale", "Prescriber NPI")
MRN_PATTERN = re.compile(r"\bMRN\s*[:#-]?\s*\d{6,10}\b", re.IGNORECASE)

def check_prior_auth_draft(text):
    """Code checks a clinical lead can sign off on: every required section has a value, no labeled MRN."""
    missing = [s for s in REQUIRED_SECTIONS
               if not re.search(rf"^[ \t]*{re.escape(s)}[ \t]*:[ \t]*\S", text, re.IGNORECASE | re.MULTILINE)]
    leaked = MRN_PATTERN.findall(text)
    return {"passed": not missing and not leaked, "missing_sections": missing, "mrn_found": leaked}

draft = """Diagnosis code: E11.9
Requested service: continuous glucose monitor
Clinical rationale: A1c above target on two visits.
Prescriber NPI:
Patient ref MRN: 00123456"""
print(check_prior_auth_draft(draft))
# {'passed': False, 'missing_sections': ['Prescriber NPI'], 'mrn_found': ['MRN: 00123456']}

The section names are an example, so take yours from your own payer forms. The MRN pattern catches an MRN label followed by 6 to 10 digits, such as MRN: 00123456 or mrn#00123456.

It misses alphanumeric or hyphenated MRNs, other label styles, unlabeled numbers, and a spelled-out “medical record number”, which is why it runs alongside an LLM-based PII eval.

Our roundup of healthcare AI evaluation platforms compares tools for the model-based side.

What the security team needs to see

Security asks what stops someone from making the agent misuse patient data. The team checks guardrails, injection resistance, and tool access as separate risks.

Guardrails on inputs and outputs

Screen inputs and outputs for personal identifiers, including PII entities such as MEDICAL_RECORD for MRN-labeled numbers, SSN, and date of birth. Block topics the agent should never handle, and check outputs before they reach a patient or a payer. Show the security team the guardrail config and the actions it takes.

Prompt-injection testing

Healthcare agents read documents that other people wrote: referral notes, faxed records, payer letters. Any of them can carry instructions aimed at the agent, such as a note telling it to send the full chart to an outside address. Test these cases before launch and bring the results, including the payloads you tried.

Mix obvious and subtle payloads in the test set. Include a faxed note telling the agent to email the chart, a payer letter with hidden text, and a referral asking it to change a diagnosis code. The pass condition is the same for each: the agent treats the text as data and takes no action it asks for.

Access controls

Give the agent least-privilege tools. A prior authorization agent can read the chart, but it should never write orders. Set per-key permissions so each agent only reaches the tools its workflow needs, and limit who can view traces. The security team will want the tool permission list in writing.

Review those permissions every time the agent gains a tool. A new tool often arrives with broad default access, such as write rights the workflow never needs. Trim that access when the tool is added, before any trace shows the agent using it.

Simulate patients before the agent meets a real one

Some cases are too risky to wait for in production. Persona-based simulation lets you test them before launch. Run the agent against an anxious caller, a non-native speaker, and a caregiver calling for a parent, and check that each conversation stays safe and accurate from the first turn to the last.

Make each persona concrete. The anxious caller repeats the same question and asks whether a symptom is serious. The non-native speaker mixes languages and uses informal names for medications. The caregiver knows the patient’s history but is not the patient, which tests whether the agent checks who it may talk to.

Then add risky edge cases with a clear pass condition for each. A patient asks for a diagnosis or a dosage, and the agent should refuse and point to a clinician. A caller describes chest pain, and the agent should escalate at once. Someone asks about a minor’s records, and the agent should hand off to staff.

Write down the expected behavior before you run the test, so the result is pass or fail and nobody argues about it afterward. Each failed case becomes a fix and then a regression test. Our post on simulating healthcare AI agents compares tools for running these scenarios at scale.

Rolling out AI agents in healthcare with online evals and shadow traffic

Evaluation keeps running after sign-off. Run online evals on a sample of live sessions, with alert thresholds for the checks each reviewer cared about, and keep guardrails on every request, since a sample can miss a rare leak. If PHI checks start failing or rubric scores drop, the team hears about it before a patient complaint arrives.

Route each alert to the reviewer who owns it. A PHI check failure goes to compliance, a drop in rubric agreement goes to the clinical lead, and a guardrail spike goes to security. Each reviewer keeps watching the evidence they approved, without waiting for the next formal review.

Models change, and a swap should reuse the evidence you already have. Use shadow experiments on mirrored traffic to send a copy of production requests to the candidate model while users only see the current one, then compare its responses, latency, and cost.

Switch only if the candidate passes the checks the current model passed. Mirrored requests carry PHI, so the candidate’s provider needs a signed BAA before mirroring starts.

Write rollback criteria before launch, such as a drop in rubric agreement or any confirmed PHI leak, and agree on them with all three reviewers. When the numbers trip a criterion, the team rolls back without a meeting.

A confirmed leak also starts the breach process.

An impermissible disclosure is presumed a breach unless a risk assessment shows a low probability of compromise. Notice runs without unreasonable delay and no later than 60 calendar days after discovery, and a business associate must tell the covered entity. Name the incident-response owner next to the rollback criteria.

Building the healthcare evidence pack with Future AGI

For compliance, traceAI records agent, LLM, and tool spans, and Observe attaches eval scores at span, trace, or session level. PII Detection and Data Privacy Compliance evals run on those spans. You can self-host the Apache-2.0 core with Docker Compose or a signed Helm chart, or use Cloud.

Future AGI holds a SOC 2 Type II report and ISO 27001 certification. It is HIPAA compliant (BAA available; see the pricing page). Those documents belong in the vendor attestation folder, and your team still owns its own risk analysis and configuration.

For the clinical lead, custom evals turn the clinicians’ rubric into an LLM judge or a code check. Annotation queues collect clinician labels and export them to datasets, so judge agreement is easy to measure.

Clinically Inappropriate Tone is a built-in check for patient-facing text, and No Racial Bias, No Gender Bias, and No Age Bias evals support the subgroup review.

For security, Protect guardrails detect PII entities including MEDICAL_RECORD, SSN, and DOB. Protect runs on Future AGI’s managed service, so list Future AGI in the vendor table with its BAA even when the rest of the platform is self-hosted.

Before launch, Simulation runs persona-based conversations against the agent. During rollout, shadow experiments mirror a share of production requests to a candidate model, so you see how it responds before any user does.

Getting the prior-authorization agent signed off

Back to the scale-up meeting. This time the team brings compliance a trace sample with PHI check results, a deployment diagram, and the vendor table with signed BAAs. The clinical lead gets the clinician rubric, the labeled sample, judge agreement, and the required-field check results.

Security gets the guardrail config, the injection test results, the tool permission list, and the log of permission reviews.

Each reviewer now has evidence for their own question, so the meeting can end with a decision instead of a follow-up date. Start collecting it this week, from the sessions your pilot already handles.

Start your evidence pack: set up eval scores on your traces.