Coverage criteria, read twice: do two reviewers reach one answer?

Coverage criteria are the conditions a health plan's medical policy sets before it will pay for a service: a diagnosis, a failed first treatment, a test result, a setting, a frequency. They exist so that the same request gets the same answer whoever reads it. In practice it often does not, because criteria leave room for judgment and reviewers fill it differently. The measure is simple: give two reviewers the same cases and count how often they agree beyond chance. Botminds' Medical Policy Search narrows the gap from the criteria side, handing each reviewer the edition in force on the date with every criterion quoted by section, so disagreements are about the patient, not about which text applies.

What are coverage criteria?

Coverage criteria are the testable conditions in a plan's medical policy, or in Medicare's national and local coverage determinations, that decide whether a service is covered for a member. For a Medicare Advantage plan they come in a fixed order: Medicare's NCDs, general coverage conditions in Traditional Medicare law, the local contractor's LCDs, and only where those are not fully established, the plan's own internal criteria (42 CFR 422.101(b)). Medicaid managed care plans write criteria within the state's definition of medical necessity; commercial and exchange plans write them under the plan document and state law.

Plans publish them as medical policies, coverage policies or clinical policy bulletins, and license others as commercial criteria sets. Whatever the name, each criterion should be something a reviewer can check in the record: present or absent, met or not met.

Are coverage criteria the same as medical necessity criteria?

They overlap but are not the same. Coverage criteria include benefit rules (is this a covered benefit at all, in this setting, at this frequency) and clinical conditions; medical necessity is the clinical judgment that the service is reasonable and necessary for this patient. Medicare Advantage rules keep both in one test: the plan must base medical-necessity determinations on the coverage criteria allowed by 422.101(b) and (c), on whether the service is reasonable and necessary under the Medicare statute, and on the member's own history, physician recommendations and clinical notes (42 CFR 422.101(c)(1)).

The distinction matters for agreement. A benefit rule is usually settled by reading it. Clinical conditions are where reviewers part, above all over the words in them that ask for judgment.

Why do reviewers disagree on medical necessity?

Because unaided clinical judgment is a noisy instrument, and criteria only partly steady it. The classic evidence comes from peer review of care quality. A JAMA review of the literature found agreement among physician reviewers “only slightly better than the level expected by chance”, and a 1994 meta-analysis of 21 findings from 13 studies put the weighted mean kappa at 0.31 (Goldman, Evaluation & the Health Professions). Explicit criteria do not cure it by themselves. A study at a Montreal cardiology service compared three utilization review tools, applied by trained reviewers, with a panel of three cardiologists: kappa was 0.45 for the best tool and 0.24 and 0.25 for the others (Kalant et al., CMAJ 2000).

The recent evidence comes from second reads. When HHS-OIG's physician reviewers re-read a sample of Medicare Advantage prior authorization denials from 2019, they found that 13% of the denied requests met Medicare's coverage rules; one of the two common causes was plan criteria not found in Medicare's rules, the other requests deemed short of documentation when the record already held enough (OEI-09-18-00260). Of the payment denials that met the rules, OIG traced most to “human error during manual claims-processing reviews (e.g., overlooking a document)” and to system processing errors. And when Medicare Advantage members or providers appealed a prior authorization denial in 2024, KFF found that most appeals, 80.7%, were partially or fully overturned (KFF, January 2026). Each overturn is a second reader disagreeing with the first.

Consistency is written into the rules

Reviewer agreement is not only a quality aim; three programs require it in their own words. Medicaid managed care contracts must require plans to “Have in effect mechanisms to ensure consistent application of review criteria for authorization decisions.” Employer plans' claims procedures must contain safeguards so that “the plan provisions have been applied consistently with respect to similarly situated claimants.” Network Medicare Advantage plans must make their decisions on utilization management and coverage consistent with their guidelines (42 CFR 422.202(b)(3)).

A mechanism to ensure consistency is something a plan should be able to show an auditor: a sample, two readers, a measure and what was done about the disagreements. The test should cover the plan's own policies as well as any licensed criteria set, because the plan's own drafting choices sit there.

How do you measure inter-rater reliability in utilization review?

Draw a sample of real cases, have two qualified reviewers decide each one independently against the same edition, and compute both the raw agreement and a chance-corrected statistic such as Cohen's kappa, overall and criterion by criterion. The table at the top of this page is the arithmetic: 82% raw agreement became a kappa of 0.46 once chance was removed. The criterion-level view is the useful one, because it shows which sentence of the policy produced the split.

A workable test design has six parts. Stratify the sample by policy and decision type, so a few high-volume services do not hide the rest. Give both reviewers the same record and the same edition, named by number and effective date. Ask for a decision on each criterion, not only on the request. Compute kappa per criterion and per reviewer pair. Have a physician adjudicate the splits and say which reading is right. Then fix the cause: retrain if a reviewer misread clear text; redraft if the text allowed both readings. Repeat it each year and after any major edition.

How are coverage criteria applied?

One criterion at a time, against the member's own record, on the edition in force on the governing date. A reviewer reads each criterion, looks for the documentation that meets it, and marks it met, not met or not applicable; for Medicare Advantage the determination must rest on the member's history, physician recommendations and clinical notes, not on a summary. When the plan expects to deny for medical necessity, the case goes to a physician or other professional with expertise suited to the service before the decision is issued (42 CFR 422.566(d)).

Each criterion matched to the record, on the edition in force on the date

What makes coverage criteria clear?

A clear criterion names one checkable condition, a time window, the evidence that proves it and the plan types it applies to. Most disagreements trace back to a few drafting habits: words that ask for judgment without a measure (‘conservative’, ‘recent’, ‘appropriate’), two conditions joined by ‘and/or’, a time window with no start point, and plan-type exceptions buried in a footnote. Heyward and colleagues, reading 2017 coverage policies of 45 commercial, Medicaid and Medicare Advantage plans for low back pain treatments, found that utilization management criteria “varied greatly with respect to which conditions and what quantity and duration of services were covered.” Variation between plans is a policy choice; vagueness inside one policy is a drafting defect.

Who writes a plan's coverage criteria?

The plan's medical policy team drafts them, and in Medicare Advantage the UM committee must approve them before use; commercial criteria sets are licensed from vendors and adopted by the same committee. The drafting is done by medical policy analysts working with physicians from the relevant specialty; network Medicare Advantage plans must develop their guidelines in consultation with contracting physicians. When a plan writes its own criteria for Medicare Advantage members, the evidence standard is set in regulation, and how licensed criteria sets fit that standard is its own question.

Authorship matters for agreement because the drafter is the person who can remove ambiguity. A reviewer who meets an unclear criterion has two bad choices: interpret it, or escalate every case. A policy team that hears about the split from the inter-rater test can fix the sentence once.

What changes when every criterion arrives with the passage that decides it

Many disagreements are not about judgment at all. One reviewer read last quarter's edition, another today's; one applied the commercial criterion to a Medicare Advantage member; one paraphrased a criterion and lost a word. Medical Policy Search removes those by construction: each criterion comes quoted from the edition in force on the date, with section and page, laid out by plan type, and a quote that does not match the source blocks the answer. What is left for the two reviewers is the patient.

The same criteria laid out by plan type, on the same date

The solution does not decide whether a criterion is met; the nurse and the physician reviewer do. What it changes is the starting point of the inter-rater test. With the edition, the plan type and the quote fixed, a split between two reviewers points straight at a sentence the policy team can redraft. How analysts learn to trust each answer is the subject of the guide to the medical policy analyst and AI; the place to see the dated answer itself is Medical Policy Search.

Questions people ask

What is a good kappa for utilization review?

There is no regulatory threshold. The published studies of review cited here, with kappas from about 0.24 to 0.45, were judged by their authors as poor to fair, or quite limited. Plans set their own target, usually by criterion set, and the more useful number is the criterion-level kappa, because it shows which sentence to fix.

How many cases does an inter-rater reliability test need?

Enough that each policy and decision type you care about is represented; a few dozen cases per reviewer pair is a common starting point, with more for high-volume services. Small samples give unstable kappas, so report the count alongside the statistic and repeat the test after any major edition.

Do the reviewer's credentials matter for agreement?

They matter for the decision. In Medicare Advantage, an expected adverse medical-necessity decision must be reviewed by a physician or other appropriate professional with expertise in the field of the service, including knowledge of Medicare coverage criteria (42 CFR 422.566(d)). Matching specialty to the case also reduces disagreements that come from unfamiliarity with the treatment.

Can a plan use different coverage criteria for different plan types?

Yes, within each program's rules. A Medicare Advantage member's criteria must follow Medicare's where they are fully established; a Medicaid member's may be no more restrictive than the state's definition of medical necessity; commercial members' follow the plan document. What a plan should not do is let a criterion meant for one plan type reach a member of another.

Where can I find Medicare's coverage criteria for a specific service?

In the Medicare Coverage Database, searched by keyword, code or document ID and narrowed by state. National criteria are in NCDs; local criteria are in the LCD of the contractor for the area, with codes in its billing and coding article. A Medicare Advantage plan must follow the LCDs of the contractors with jurisdiction where its services are covered.