The fair test of an aerial imagery underwriting tool is a replay of referrals you have already closed, the ones where an inspector has been to the house. Give the tool each file as it stood before the inspection, let it outline the features and report its mismatches with the application, then score every line against what the inspector photographed. Count two errors apart from the rest, because they cause the worst outcomes: a capture of the wrong house, and a capture too old or too poor to judge a roof. A vendor's accuracy figure on its own test images says little about your book, your states and your inspection vendors; a replay on your closed referrals says a great deal. Botminds' Aerial Imagery Verification is built to be tested this way: it reads the imagery, inspections and permits a carrier already has, so a replay needs no new captures and no new visits.
The worked example below comes from a fictional sample file in the solution, a Maryland new-business referral for 1127 Wrenfield Court written by an invented carrier, Norland Mutual. Its people and figures are sample data, not a customer's test result.
The replay test on one page
Population
Every imagery referral closed last quarter that has an exterior inspection on file. New business and renewals, every state you write.
What the tool sees
The file as it stood before the inspection: the application, every capture dated before the visit, permits and assessor data.
What it does not see
The inspection report and photos, the underwriter’s notes and the decision. These are the answer key.
The truth
What the inspector photographed, plus a second underwriter’s ruling on every line where tool and inspector disagree.
What is scored
Each feature, each roof slope’s age and condition class, each capture’s parcel match and fitness, and the recommendation.
Counted apart
Wrong-house captures caught, and captures too old, too coarse or too cloudy to judge the roof caught.
What should an aerial imagery underwriting tool be tested on?
On your own referrals that an inspector has already settled. The inspection is the only independent account of the house in an imagery referral: the application is the applicant's, the capture is the tool's input, and the inspector stood there. A tool that agrees with your inspectors on last quarter's files will agree with them on next quarter's; a tool that only scores well on its vendor's benchmark has not yet met your houses.
A replay also tests what a demonstration cannot. Your inspection vendors write reports their own way. Your states have their own image-age rules. Your rating manual prices a pool or a solar array its own way. And your book contains the awkward files: a roof replaced in halves, a capture of the house next door, a pool filled in last year. Those are the files where tools disagree with people, and where the disagreements cost money or a hearing.
Carriers are already using such tools. In an NAIC survey of 194 homeowners insurers on 2022 data, limited to "more advanced" models, underwriting was among the most common uses of artificial intelligence or machine learning, reported at 47% against 54% for claims (NAIC survey memo). The survey does not say how many tested their tools on closed files first.
Building the replay set: which referrals, and what the tool may see
Take every imagery referral closed in the last full quarter that has an exterior inspection on file, new business and renewals, across every state you write. Do not pick the interesting ones; picking is how a test flatters a tool. If the quarter is small, take two.
Then freeze each file at the moment before the inspection. The tool may see the application, every capture dated before the visit, the permits and the assessor data. It may not see the inspection report, the photos, the underwriter's notes or the decision. Those are the answer key, and a tool that has seen them is grading its own homework.
The sample file shows why the freeze matters. On 1127 Wrenfield Court there are three captures: 14 August 2026, 3 April 2025 and 21 June 2023. The inspection was on 2 October 2026. All three captures predate it, so all three go into the replay, including the 2023 capture, which turns out to show the neighbour's house. A replay that quietly dropped old captures would never test whether the tool catches the wrong house.
The replay pipeline, from closed referral to adjudicated disagreement
The pipeline has five steps, and the tool under test runs only two of them. The others belong to the carrier's quality team, because a test whose truth is set by the vendor is not a test.
The replay pipeline: two steps for the tool, three for your people
A person decides
Select and freezeThe quality lead pulls every inspected imagery referral from the quarter and hides the inspection, notes and decision.
Agent
Run the toolThe tool reads each frozen file and reports features, roof age by slope, condition classes, capture fitness, parcel match and a recommendation.
Never: sees the inspection before it has answered
Agent
Line up the answersEach output line is set beside the inspector’s photo for the same attribute; agreements and disagreements are listed.
Never: resolves a disagreement in its own favour
A person decides
AdjudicateA second underwriter rules on every disagreement, from the photos, with the tool’s output and the inspector’s note side by side.
A person decides
Score and decideThe quality lead computes the measures below and decides whether the tool is fit for which uses.
Adjudication is the step most tests skip. When the tool says "pool" and the inspection says nothing, the inspector may have missed it, or the tool may have seen a blue tarpaulin. Neither the tool nor the inspector gets the benefit of the doubt. A second underwriter looks at the photos and rules, and the ruling is the truth for that line.
How to score features: pools, trampolines, solar arrays and outbuildings
Score features with two numbers, found and false. Recall is the share of features the inspector confirmed that the tool also found; precision is the share of features the tool reported that turned out to exist. A tool tuned to find every pool will report some that are not there, and every false pool is a question to an agent and a customer annoyed.
The measures for an aerial imagery underwriting tool test, and how to compute each
| Measure | Formula | Why it matters |
|---|---|---|
| Feature recall | Features found by the tool and confirmed ÷ features confirmed by the inspection or adjudication | Each missed pool or trampoline is premium short every year |
| Feature precision | Features found by the tool and confirmed ÷ features reported by the tool | Each false feature is a needless question to the producer |
| Roof age error | Tool’s age − inspector’s age, per slope, in years | Roofs are replaced in parts; a single age hides the old slope |
| Condition agreement | Slopes where tool and inspector agree on cosmetic vs bearing on the risk ÷ slopes scored | Cosmetic findings carry no action in several states |
| Wrong-house catch rate | Off-parcel captures the tool excluded ÷ off-parcel captures in the set | An image of the neighbour’s house should never be read |
| Unfit-capture catch rate | Too-old, too-coarse or too-cloudy captures the tool refused to judge ÷ such captures in the set | Maine reports complaints about images more than 18 months old |
| One-source adverse rate | Adverse recommendations resting on a single source ÷ adverse recommendations | Should be zero; most of the state rules we read call one image insufficient on its own |
Report recall and precision by feature, not pooled. Pools are large and blue and most tools find them; trampolines are small, round and often under trees; solar arrays are easy to see and hard to attribute, since the image cannot show whether they are owned or leased. On the sample file the capture found all four undeclared features the inspector photographed, which is one file, not a measure.
One replayed referral, line by line: what the nine attributes came to
AI roof condition assessment: score the slope, the age and the cosmetic call apart
Score the roof slope by slope, and score age and condition separately. A single roof score folds three different judgments into one number: how old each part of the roof is, whether its wear bears on the risk, and whether the picture was good enough to tell. When the number is wrong, nobody can say which judgment failed.
The sample roof is the case for splitting. The application says 2020. The capture shows two shingle tones. The inspector's photos show a south slope about six years old and a north slope 13 to 16 years old with moderate granule loss, and dark streaks on the north slope that are algae on intact shingles. A good tool reports "partly true" on age, "bears on the risk" on the granule loss and "cosmetic" on the streaks. A tool that reports "roof condition: poor" has blended a pricing fact, a risk fact and a cleaning job.
The cosmetic call is the one with legal weight. Connecticut's insurance department says "cosmetic roofing conditions such as natural discoloration and streaking do not support taking homeowner or dwelling fire nonrenewal action" (Connecticut notice). A tool that cannot tell algae from deterioration will generate exactly the nonrenewals that notice was written about, so condition agreement deserves its own line in the scorecard.
Count wrong-house captures on their own
Count every capture in the replay set that does not show the insured house, and check that the tool excluded each before reading anything from it. This error is rare and severe. Maine's insurance bureau has had complaints from homeowners judged on an image of another building (Maine Bulletin 483), and Tennessee counts the wrong structure among the ways imagery is misused (Tennessee Bulletin 25-03); the aerial imagery laws by state sets the three states' words side by side.
The test is mechanical: measure the offset between the capture's building footprint and the insured parcel, and compare it with a tolerance. On the sample file the 21 June 2023 capture's footprint sits 31 metres west of parcel 02-118-447 and matches 1125 Wrenfield Court, the house next door. With an 8-metre tolerance the parcel matcher marks it "Not this property", keeps it in the file and reads nothing from it. A tool that read the neighbour's gable roof as the Castellanos' would have dated the roof wrong and seen no pool.
A capture of the house next door, caught before it is read
Count stale and unreadable captures on their own
Count every capture that is too old, too coarse or too obscured to judge the roof, and check that the tool refused to judge from it. Delaware's insurance department reminds insurers that aerial imagery "has inherent limitations, such as image distortion, weather interference, and the inability to capture fine details, which may impact its accuracy and readability" (Delaware Bulletin No. 150). A tool should know when it is looking at one of those images.
Colorado's 2026 bulletin gives the fullest list of what to check. Insurers "should consider the combination of spatial resolution, orthorectification method and quality, elevation modeling, capture date relative to the relevant event or condition, and surface-only limitations of aerial imagery" (Colorado Bulletin B-5.57). It adds that imagery "sufficient for parcel-level identification or community risk assessment may not be sufficient to support decisions on property conditions". A replay should therefore score fitness by use: a capture can be good enough to count a pool and not good enough to judge shingles.
The thresholds a replay should test, as settings
The thresholds themselves are the carrier's to set and the test's to check. On the sample carrier, no capture older than 12 months (or the state's limit, if shorter) may support an adverse action, condition is read only from imagery at the resolution the carrier sets (7.5 centimetres by default), and a capture with more than 30% of the roof under cloud, shadow or trees cannot judge the roof. A replay counts how often each rule fired and whether the adjudicator agreed it should have.
The aerial imagery underwriting process after the test
A passing test changes the process, not just the vendor list. The aerial imagery underwriting process that the test supports is the one regulators describe: the image flags, a second source confirms, a person decides, and the rule that does all this is written down. Connecticut expects the guardrails to live in the carrier's filed underwriting guidelines; it names, as an example, a physical inspection or a contractor's report when an image does not unequivocally show material damage, and says it "expects that this type of guardrail be set forth in each insurer’s filed underwriting guidelines if aerial imagery is used" (Connecticut notice). Colorado says adverse actions derived from imagery tools should be consistent with the insurer's filed underwriting criteria and rate filings (Colorado Bulletin B-5.57).
So the scorecard feeds the guidelines. If the replay shows condition agreement is weak in one state's housing stock, the guideline for that state requires an inspection for every condition finding. If wrong-house catches are perfect, the guideline can say so and cite the test. Pricing every mismatch the test confirms is the subject of what a trampoline, a pool and a roof slope cost.
Pass marks to write into an aerial imagery tool test before it runs
- Every inspected referral from the quarter, not a sample the vendor chose
New business and renewals, every state, every inspection vendor.
- Files frozen before the inspection
The tool never sees the report, the notes or the decision before it answers.
- Every disagreement adjudicated by a second underwriter
From the photos, with the tool’s output and the inspector’s note side by side.
- Recall and precision reported per feature and per state
Pools, trampolines, solar, outbuildings; a pooled figure hides the weak one.
- Roof scored by slope, with age and condition apart
Partly true is an answer; “poor” is not.
- Wrong-house captures: all excluded before reading
Anything less than all is a finding to fix before go-live.
- Unfit captures: refused for condition, by use
A capture may count a pool and still be unfit to judge shingles.
- No adverse recommendation on one source
Count them; the target is zero.
- Every output line cites its image, photo or permit
If the adjudicator cannot open the evidence, neither can an examiner.
Where Aerial Imagery Verification fits in a test, and where it does not
Aerial Imagery Verification is the reconciliation layer, not the camera. It does not fly drones, capture or sell imagery, or produce a roof score of its own from pixels a carrier has not licensed. It reads the captures, inspection reports, permits and certificates a carrier already has, matches each capture to the parcel, outlines and measures what each shows, and sets every source beside the application for each attribute, with each line priced from the carrier's rating manual. The underwriter decides.
That makes it straightforward to replay. Its agents' normal run on a frozen file is step two of the pipeline; its every-source grid is step three. On the sample file the run took 3 minutes 40 seconds for five sources and three captures.
The line-up a replay scores: every attribute, every source
What a buyer should not take from this page is a promised score. We have shown one sample file, not a test result, and the measures above are the ones to compute on your own referrals. Integration with the imagery, inspection and permit sources you already license is the subject of insurance aerial imagery software, working with the captures you buy.
Questions people ask
What should aerial imagery underwriting guidelines say?
Which uses an image may support on its own, which need a second source, how old and how sharp an image must be, and what happens on a dispute. Connecticut expects such guardrails in the filed underwriting guidelines of any insurer using aerial imagery, and Colorado expects imagery-based adverse actions to match the filed underwriting criteria.
What does an AI roof inspection tool do for an insurance company?
It reads aerial captures of a roof and reports features and apparent condition: shape, material, tones, staining, debris, trees. What it cannot do is see under the surface; Colorado's bulletin lists "surface-only limitations" among the things insurers must weigh. Its output is a reason to look closer, which is why several states expect a second source before an adverse action.
What should a roof condition scorecard show?
Each slope separately, with its estimated age, its condition class (cosmetic or bearing on the risk), the capture it came from with date and resolution, and whether a second source agrees. A single number per roof hides the half-replaced roof and the algae streak, which are the two most common reasons a roof finding is wrong.
How many referrals does a replay test need?
Enough to measure each feature and state you care about. As a rough guide, with 200 inspected referrals and a measured recall of 95%, the 95% confidence interval is about plus or minus 3 points (1.96 × √(0.95 × 0.05 ÷ 200), our arithmetic). Rare errors such as wrong-house captures need every case you have, which is why the whole quarter goes in.
Which property underwriting tasks should a tool automate, and which not?
Automate the reading, matching, measuring and pricing; keep the decision, the signature and anything adverse with people. Maryland tells insurers that use AI to interpret aerial images to be familiar with its AI bulletin, 24-11, and the NAIC's model bulletin on AI expects governance over decisions that could harm consumers.