Prior authorization automation, for a health plan, is software that takes over the mechanical parts of reviewing a request: intake, eligibility, finding the policy, reading the clinical record against each criterion, asking providers for what is missing and drafting the notice, while nurses and physicians keep the decisions. The way to find out whether it works for a particular plan is not a demonstration on the vendor's examples but a replay of the plan's own closed requests, chosen to include the hard ones, against the plan's own medical policies, scored criterion by criterion against what its reviewers actually decided. Botminds offers exactly that with Prior Authorization Automation: the solution runs on a sample of the plan's past requests and policies before any live request touches it. The four-week plan below works for any product a plan is testing.
Two reasons make closed requests the right test. They come with an answer key: each one has a decision, a notice and, for some, an appeal outcome. And they include the cases that matter most, the denials later overturned, where the question is whether the evidence the appeal relied on was in the original record.
Four weeks from data to decision, on the plan's own requests
- Week 1 · data and policies in5 d
- Week 2 · replay the sample5 d
- Week 3 · nurses and physicians score5 d
- Week 4 · results and decision5 d
What is prior authorization automation for a health plan?
The phrase covers two different things, and a plan should know which it is buying. One is electronic prior authorization: the exchange of requests and decisions between provider and plan through standard transactions and, from 2027, the Prior Authorization API that CMS-0057-F requires Medicare Advantage, Medicaid and other covered payers to run (42 CFR 422.122(b)). That changes how a request arrives. The other is review automation: software that reads what arrived and prepares the decision. That changes what the reviewer does with it.
Products built for the provider's office fill forms and check status. A plan-side pilot tests the second kind: whether the software reads records against the plan's criteria accurately enough that a nurse can trust its citations, finds what is missing early enough to ask for it inside the clock, and leaves every decision with the right person. Prior Authorization Automation is that kind of product; its six agents do the reading and citing, and the UM nurse approves while a physician decides any denial.
How do you test prior authorization automation? Replay, then compare
Feed the software each closed request exactly as it originally arrived, with the records the plan had at the time, and let it produce its checklist. Then put each checklist beside the original decision and ask, criterion by criterion, whether they agree, and where they disagree, who was right. Nurses and physician reviewers judge the disagreements; the software does not grade itself.
The reason to insist on this is the state of the evidence. A systematic review published in September 2026 found 16 eligible studies of AI in prior authorization and concluded:
“Most studies reported technical metrics (F1, precision, accuracy, recall, area under the receiver operating characteristic curve), while few demonstrated operational gains, including faster decisions, lower denial rates, and reduced cost.”
Systematic review of AI in prior authorization, J Manag Care Spec Pharm 32(9), Sept 2026
A vendor's accuracy figure belongs in the first half of that sentence. The plan needs the second half, measured on its own requests, its own policies and its own reviewers' judgements. Nothing published can substitute for that, and a pilot on closed requests is the cheapest way to get it before any member is affected.
Which closed requests to pull: a sample that tells you the most
A random sample of last quarter's requests would be mostly easy approvals, which tell the plan little. Draw a stratified sample instead, weighted toward the cases where review effort and risk concentrate, and include the services the plan reviews most.
A suggested pilot sample of 200 closed requests, and what each group tests
| Group | Requests | What it tests |
|---|---|---|
| Approved, every criterion documented | 50 | Whether the agents tick each criterion from the right page, and how fast a nurse can confirm them. |
| Approved after a records request | 30 | Whether the agents name the document the nurse eventually asked for, on day 0 instead of later. |
| Denied and upheld on appeal, or not appealed | 40 | Whether contradictions and truly missing evidence are found and shown to a physician, not smoothed over. |
| Denied, then overturned on appeal | 40 | Whether the page the appeal relied on was already in the original record, and whether the agents found it. |
| Expedited | 25 | Whether the reading fits a 72-hour clock with weekends in it. |
| Long records (100 pages or more) | 15 | Whether citations stay accurate when the evidence is on page 140 of 180. |
The overturned denials are the heart of the sample. For each one the appeal file says what evidence decided it. If that evidence was a page already in the original request, the question for the pilot is simple: did the agents find it and cite it? A plan that finds this in a quarter of its overturned denials has found its business case. A plan that finds it in none has learned its overturns come from new evidence, and should spend its money on records requests, not reading.
What data do you need to automate prior authorization?
Less than an integration project, and more than most plans expect to gather. The pilot needs each request as it arrived and the policy edition in force on the date it was decided, which is the item plans most often cannot produce quickly.
The data a prior authorization pilot needs, and why
- The closed requests, as received
Request form or 278, and every clinical page in the order the provider sent it, including faxes received after the request with their arrival times.
- The medical policies, by edition
The criteria in force on each decision date, marked as Medicare coverage (NCD, LCD) or the plan’s internal criteria.
- The decisions and notices
Who decided, when, and the notice text, including the reason given for each denial.
- The appeal outcomes
For denials appealed: the reconsideration decision and what evidence it relied on.
- Eligibility and claims history for the sampled members
Enough to check coverage and to corroborate treatments the chart names, such as billed therapy visits.
- A business associate agreement and a minimum-necessary scope
The vendor handles protected health information for the plan; HIPAA requires satisfactory assurances and limits use to what the purpose needs (45 CFR 164.502).
The policy editions deserve a week of their own if needed. Medicare Advantage plans already have to produce some of this: a utilization management committee must review every utilization management policy, prior authorization included, at least once a year (42 CFR 422.137(d); the yearly cycle is laid out in medical policy updates), internal criteria used where Medicare's are not fully established must be published with their evidence (422.101(b)(6)), and since 2026 plans submit information on the internal coverage criteria they and their delegates use to CMS each year, the first submission due 30 April 2026 (CMS). A plan that can produce that submission can produce the policy list a pilot needs.
The four-week pilot, from first request to go-live
Week 1 is data. The plan exports the sample, the policies and the outcomes; the vendor loads the policies as criteria and confirms each one against its source. The useful output of this week is a policy library a nurse recognises: every criterion in the plan's own words, with its edition and whether it is Medicare's or the plan's.
The policy library: each criterion, its source, and how often requests arrive without it
Week 2 is the replay. Each request runs through the agents as if it had just arrived, with pages that came later added at their original times, so the timeline of the replay matches the timeline of the original case. Week 3 is scoring: nurses review every criterion the agents marked met or open, physician reviewers review every contradiction and every overturned denial, and each disagreement is recorded with who was right. Week 4 is results: the scorecard below, a list of the disagreements by cause, and a decision.
What results should a prior authorization pilot show?
Six measures, each set before the pilot starts so the result cannot be argued into a pass. The thresholds below are suggestions; the point is that the plan writes its own down in week 1.
A pilot scorecard: six measures, how to take each one, and where to set the bar
1
Cited ticks that are right
A nurse opens the page behind every criterion the agents marked met and judges whether the passage says what the criterion asks.
Set before week 1. A plan might ask for 98% or better, because a wrong tick is the error that leads to a wrong approval.
2
Open criteria that were truly open
For every criterion marked not found, a nurse checks whether the evidence was in the record after all.
Low single digits. Each miss is a potential insufficient documentation denial.
3
Overturned denials the agents would have caught
For each denial later overturned on appeal, did the agents find the page the appeal relied on in the original record?
No fixed mark. This is the number the business case rests on.
4
Contradictions found
Criteria where the agents quoted passages for and against, checked by a physician reviewer.
Every contradiction a physician would have wanted to see.
5
Records requests that named the right document
For requests that needed records, did the draft name the document the nurse eventually used?
Set by the plan. Compare with the provider replies in the original files.
6
Nurse minutes per request
Time from opening the checklist to a decision ready to sign, against the plan’s own measured baseline.
Measured, not assumed. The solution’s design target is 12 minutes for a request that arrives as a cited checklist.
The first two measures are the safety case: a wrong tick risks a wrong approval, and a wrong "not found" risks a wrong denial. The third is the business case. The last is the efficiency case, and it is the one most often measured badly. Measure the plan's own baseline in week 1 by timing nurses on a handful of the same kind of requests, rather than relying on anyone's industry figure; no published benchmark for plan-side review time exists.
What a pilot asks of the plan's own staff
Less clinical time than a quarter of normal reviewing, if the scoring is organised around citations. The arithmetic below is our rough planning estimate for the 200-request sample, with every assumption stated so a plan can replace it with its own.
Nurse scoring is the largest block. If the sampled policies average six criteria, the sample holds about 1,200 criterion findings. A cited finding is checked by opening the page and reading the highlighted sentence; at about a minute each, that is some 20 nurse hours across week 3, plus time for the open criteria, which need a second search of the record. Physician scoring covers the 80 denials and any contradictions; at around ten minutes a case, that is roughly 13 to 15 physician hours. Add a few hours from compliance to review the agents' limits and settings, and a few from the person who exports the data in week 1.
The point of setting this out is that it is small against the decision it supports, and that it scales with citations. A system whose output had to be checked by re-reading whole charts would need the hours of the original reviews again; a pilot of that kind tells a plan little it could not learn from its own audit sample.
Does prior authorization automation work with your own medical policies?
It should run on the plan's policies exactly as written, and a pilot is where to prove it. Three things to check. Each criterion should be loaded in the plan's wording, not paraphrased, so a nurse recognises it. The edition in force on each request's date should be the one applied, since policies change during a year and the plan's committee may revise them at any review. And Medicare Advantage requests should be read against Medicare's own coverage rules first, with the plan's internal criteria used only where those are not fully established.
Where a plan licenses commercial criteria products, the pilot also tests how the software holds them: as the plan's licensed text, applied criterion by criterion, with the same citations. If a policy cannot be broken into criteria that a passage could satisfy, that is a policy-writing problem the pilot will surface, and one worth fixing whatever the plan buys.
Going live after the pilot: what changes on day one, and what waits
A pilot answers whether the reading is accurate; go-live answers whether the plan's people and systems can use it. The usual sequence is a period of running live requests in parallel, with nurses deciding as before and checking the agents' checklist beside their own work, then switching one service line to checklist-first review while the rest stay as they are. Connections to the provider portal, fax lines, the X12 278 feed, eligibility, claims history and the notice system can come one at a time; a pilot needs none of them, because the closed requests are exported.
Records requests that name the document, and how providers answer them
What should not wait is the governance. Before the first live request, the plan should have set who may approve, who must sign adverse decisions, when supervisors are alerted before a clock runs out, and how records requests are sent; in the solution these are settings, versioned and recorded. The limits written into each agent are what the plan's compliance team should review in the same week. For the wider questions to put to vendors, including the 2027 API and price, see the twelve questions for prior authorization software, and for how the overturned denials in the sample translate into money, pricing a wrong denial.
Questions people ask
How long does it take to implement prior authorization automation?
The pilot described here takes four weeks once the data is exported, and its longest step is usually gathering policy editions. Going live depends on how many connections the plan wants on day one: portal, fax, X12 278, eligibility, claims and notices. Running a service line in parallel before switching is a common, cautious path.
Can you pilot prior authorization automation on past requests?
Yes, and it is the most informative test, because closed requests come with decisions, notices and appeal outcomes to compare against. The requests should be replayed as they originally arrived, with late pages added at their original times. Botminds runs Prior Authorization Automation on a sample of a plan's own past requests and policies in exactly this way.
How many requests should a prior authorization pilot include?
Enough to cover each group that matters: easy approvals, approvals after records requests, upheld denials, overturned denials, expedited requests and long records. Around 200, weighted toward overturned denials and long records, is a workable size for a mid-sized plan; the point is coverage of the hard cases, not statistical precision on easy ones.
Who should score a prior authorization pilot?
The plan's own nurses and physician reviewers, not the vendor. Nurses judge the criteria the software marked met or open; physicians judge contradictions and overturned denials. Each disagreement should be recorded with who was right and why.
What if the pilot finds the agents miss evidence?
Look at where. Misses concentrated in one policy often mean the criterion is written in a way no passage can satisfy, which is a policy fix. Misses spread across scanned pages point at document quality. Either way, a cited design shows the miss: the open criterion lists what was searched, so the gap is visible rather than hidden in a score.