APS medical summarization is the reading of an attending physician statement, and the hospital, lab, prescription and MIB records that come with it, into a summary a life underwriter can decide from: each impairment, its history, its latest values, and where the records differ from the application. Tools that do it with language models are now easy to demonstrate and hard to judge, because a summary that reads well can still miss an impairment or cite the wrong page. The test that settles it is a pilot on cases you have already decided, where your own nurses' abstracts and your underwriters' decisions are the answer key. This guide sets out that pilot for 40 closed cases, using Botminds' APS Summarization as the worked example and its fictional sample carrier's monthly audit as the model scorecard.
A 40-case pilot of APS medical summarization, in seven decisions
- Use decided cases only
The decision is made, the abstract exists, no applicant is affected by the result.
- Choose the 40 for difficulty, not convenience
Long faxed files, several sources, known non-disclosure, medical director referrals, post-issue audit files.
- Fix the scorecard before the first run
Citations found, impairments missed, differences found, lab values right, minutes in file.
- Write the stop rules in advance
One fabricated statement, or one missed class-changing impairment, stops the pilot for a fix and a re-run.
- Score blind where you can
Reviewers check the tool’s summary against the pages before seeing the original abstract.
- Re-run after any change
A changed prompt, model or setting is a new system; the score does not carry over.
- Have it signed by the people who own the risk
Chief underwriter, medical director, and whoever runs your AI governance program.
What is APS medical summarization, and what should a pilot prove?
It is the step between records arriving and an underwriter deciding, done by software instead of by a person reading front to back. A pilot should prove three things, in this order. That the summary is faithful: every statement is on the page it cites. That it is complete: nothing your nurse would have abstracted is missing. And that it is useful: the underwriter reaches the same decision in less time. A tool that wins on speed and loses on the first two has not passed.
The regulators' model guidance points at the same method. The NAIC's model bulletin on AI asks insurers to validate AI systems, and says validation can take the form of comparing performance on data available during development with performance after implementation, "measuring performance against expert review, or other methods." A closed-case pilot is expert review with the expert's work already done: the nurse's abstract and the underwriter's decision on each file.
Can we test AI on closed underwriting files?
Usually yes, under your existing confidentiality rules, but check the state privacy law that governs your files first. Insurance privacy statutes based on the NAIC's model limit what a carrier may disclose about an applicant. Virginia's, for example, says a carrier "shall not disclose any medical-record information or privileged information about an individual collected or received in connection with an insurance transaction unless the disclosure is with the written authorization of the individual", and then allows disclosure to a person performing a business or insurance function for the carrier, provided that person agrees not to disclose it further.
For a pilot that means a contract before data: the tool provider performs a function for you, agrees in writing not to redisclose, and runs in an environment you control or have assessed. Running decided files through a consumer chat assistant does not meet that test, a point taken further in can ChatGPT review medical records for an underwriting decision?.
How many cases for an underwriting AI pilot? Why 40 is enough for some questions and not others
Forty cases is enough to measure citation accuracy precisely and not enough to prove that rare misses never happen; plan for both. At about 38 citations per summary, as on the fictional sample case, 40 files give roughly 1,500 citations to check. In the sample carrier's September audit, 40 decided cases produced 1,644 citations, of which 1,637 were found on their page: 99.6%. On those counts the 95% confidence interval for the found rate runs from about 99.1% to 99.8% (our arithmetic, Wilson interval, treating citations as independent; they cluster within files, so the true interval is a little wider).
Rare events are different. If the tool misses no class-changing impairment in 40 files, statistics allows the true miss rate to be as high as about 7.5% per file (the rule of three: 3 ÷ 40). That is why the stop rule for misses is absolute and why the pilot is followed by a standing monthly audit rather than declared finished. Forty files buys a go or no-go on faithfulness; completeness keeps being measured.
Which 40 cases? A sample built to find the failures
Pick the files most likely to break a summary, because easy files flatter every tool. A random draw of decided cases will be dominated by short, clean files with one or two impairments, and a tool that handles those proves little. A stratified sample puts the weight where the errors live: long faxed files, many sources, quoted and repeated lab values, and cases where something in the records did not match the application.
A stratified sample of 40 decided cases for an APS medical summarization pilot
| Stratum | Cases | What it tests |
|---|---|---|
| Long files, over 250 pages, several sources | 10 | Sorting, duplicates, dating, and whether late pages are read as carefully as early ones |
| Files with a hospital stay or emergency visit | 8 | Whether events outside the primary care chart reach the summary and the application check |
| Cases where an amendment or questionnaire was ordered | 6 | Whether the differences your underwriters acted on are found, worded neutrally, with pages |
| Medical director referrals | 6 | Lab trends and timelines on the cases where a clinician rated the line, not a value |
| Post-issue audit files from accelerated business | 5 | Reading a file against an application that was issued without records |
| Poor scans: handwriting, smudged faxes, rotated pages | 5 | Whether unreadable pages are routed to a person rather than skipped |
Two practical rules keep the sample honest. Choose the files before anyone sees the tool's output, and record why each was chosen, so nobody can swap out a hard case after a bad result. And include files your best nurse found difficult; their original abstract, with its margin notes, is the most demanding answer key a pilot can have.
What metrics prove an APS summary is right?
Five, each with a clear denominator. A pilot that reports one blended accuracy number has hidden the failures that matter. The table gives the metric, how to count it on closed cases, and the line the sample carrier set or would set.
The scorecard from one month of a monthly audit: what APS medical summarization should report
99.6%
citations found on their page: 1,637 of 1,644, against a 99% target
40
decided cases sampled, of 412 decided that month
4 min 50 s
median agent read per file, median file 214 pages
22 min
median underwriter time in file, opening to decision or referral
Five metrics for an APS medical summarization pilot
| Metric | How to count it on a closed case | Pass line (example) |
|---|---|---|
| Citations found on their page | Open every cited page; is the statement there, with the same value, date and unit? | 99% or better, as the sample carrier’s target |
| Impairments missed | Compare the summary’s impairments with the nurse’s abstract and the decision; count any the summary lacks | Zero that would change the class |
| Application differences found | Compare flags with the differences the underwriter actually acted on (amendments, questionnaires) | All acted-on differences found; false flags counted separately |
| Lab values right | For every charted value: value, specimen date, unit, laboratory, page | No wrong value; unit and date errors counted and fixed |
| Minutes in file | Underwriter time from opening to decision, against the same underwriter’s time on similar files | Your own baseline; the sample median was 22 minutes |
Count unreadable pages separately. A tool that silently skips a page it cannot read has failed on completeness even if every citation it made is correct. In the solution, unreadable pages go to a person and the file shows that they did, so the pilot can count them.
What errors to expect, and which ones stop the pilot
Expect slips of location more than invention, and decide in advance which kind halts everything. The sample carrier's seven September corrections are a useful template for the error log, because each is a different failure: a value cited to the next page; a merged reading cited only to the later of its two pages; a sleep study date misread through a fax smudge; a weight cited to the exam that was actually the application answer; a blood pressure cited to a duplicate page instead of the original; a citation pointing at a cover sheet; an eGFR shown in the wrong unit.
Seven corrections in one month's audit, by kind
The stop rules should be short. Any statement that is not on any page stops the pilot: that is fabrication, and it is the failure a cited summary exists to prevent. Any missed impairment that would have changed the class stops it. Wrong dates and units do not stop it, but each must be fixed and the fix re-tested. The published research on clinical notes suggests why re-testing is not optional. In a 2025 study by authors employed by Tortus AI, a company that builds clinical documentation software, one change of method that was meant to help, extracting facts before writing the note, made results worse: "major hallucinations increased from 4 to 25" against the structured prompt it replaced.
The monthly audit view: every citation re-checked, every correction listed
How long does a summarization pilot take?
About six weeks of calendar for 40 cases, most of it human review rather than machine time. The reading itself is minutes per file. The weeks go to choosing and exporting the files, to reviewers checking every citation against its page, and to the sign-off. The plan below is ours, not a measured duration; a carrier with a ready export and dedicated reviewers can compress it.
A six-week plan for a 40-case APS medical summarization pilot
- Choose files, set scorecardWeeks 1–2
- RunWeek 3
- Blind citation reviewWeeks 3–5
- Compare and signWeeks 5–6
Settings belong in week 2, not week 3. The solution's impairment list, lab reference ranges, referral rules and authority levels are carrier settings, saved as versions; the pilot should run on the version you would go live with, so the score belongs to that version. If the settings change after the pilot, the re-run rule applies.
The settings a pilot should freeze: what is indexed, which ranges, who decides
Who signs off an underwriting AI pilot?
The people who own the risk the tool touches: the chief underwriter for the decisions, the medical director for the clinical reading, and the owner of the carrier's AI governance program for the controls. The NAIC model bulletin asks insurers to maintain a written program for AI systems that make or support decisions; as of 8 October 2026 the NAIC's map lists 29 jurisdictions that have adopted it, with California, Colorado, New York and Texas listed separately for their own insurance-specific rules (NAIC adoption map). What the bulletin requires is set out in our sister guide on NAIC AI bulletin requirements.
A bought tool does not move the duty. The bulletin's section on third parties expects due diligence so that decisions supported by a vendor's system meet the same legal standards as the insurer's own (what the NAIC AI bulletin requires). New York's 2024 circular says the same in fewer words: the insurer stays responsible for understanding any tool a vendor builds or runs for it. The pilot file, with its sample, scorecard, error log and signatures, is the first piece of that due diligence, and how the rules apply to medical record review specifically is the subject of AI medical record review in life underwriting.
What to do after the pilot passes
Keep the scorecard running. The pilot measured one version on 40 files; live work changes the mix of files every month. The sample carrier's answer is a monthly audit of 40 decided cases with every citation re-checked and every correction listed, signed by the underwriting auditor, which is the pilot repeated forever at a cost of a few reviewer days a month. Signing in the solution is blocked while any citation cannot be found on its page, so the live control and the audit check the same thing from two sides. The full sample case is on the APS Summarization page.
Questions people ask
What is an APS summarization tool?
Software that reads an attending physician statement and related records and writes a summary for the underwriter. The useful ones cite the page behind each statement, chart lab values over time and check the application against the records. APS Summarization does all three and leaves the rating to the underwriter.
What accuracy should an APS summary have?
Zero fabricated statements and zero missed class-changing impairments, plus a high citation rate. The fictional sample carrier targets 99% of citations found on their page and measured 99.6%, 1,637 of 1,644, in one month's audit.
How do you evaluate an APS summarization tool?
On your own decided cases, against your nurses' abstracts and your underwriters' decisions. Check every citation against its page, count missed impairments and differences, and time the underwriter. The NAIC model bulletin names measuring performance against expert review as one form of validation.
Is 40 cases enough?
For citation accuracy, yes: about 1,500 citations give a narrow interval. For rare misses, no: zero misses in 40 files still allows a true rate up to about 7.5% per file. That is why the pilot is followed by a standing monthly audit.
Does a vendor tool change who is responsible?
No. The NAIC bulletin expects insurers to make sure vendor-supported decisions meet the standards imposed on the insurer itself, and New York's 2024 circular says insurers retain responsibility for understanding vendor tools.