To merge duplicate clients in an agency system, find the records that describe the same insured, decide which values survive, merge them into one client with the history of each, and keep the merge reversible. The finding is mechanical; the deciding is not. Two records with the same address and phone but different tax IDs, a business and its trade name, a contractor set up once by the commercial desk and again by the workers' comp desk: careful people disagree about these, and a matcher trained on one person's answers inherits that person's habits. After an acquisition the problem doubles, because the seller's book has its own duplicates and also overlaps the buyer's. Botminds' Agency Acquisition Data Migration has a client matcher agent that standardises names, addresses, phones, emails and tax IDs, groups likely duplicates within and across both books and explains each match, but never merges two different tax IDs and never merges without a person's approval.
The way to know whether any matcher, or any team, is good enough is to measure agreement first. Two reviewers label the same sample of candidate pairs without seeing each other's answers; the share of pairs on which they agree, broken down by kind of pair, shows where the rules are clear and where they are not. The figure below is an illustration of what that test typically reveals.
Where two reviewers agree on 'same insured?', by kind of pair (illustrative)
How do you merge duplicate clients in an agency system?
In five steps: standardise, group, decide, merge, and keep the undo. Standardising puts every record in the same form, so "1420 N Broadway" and "1420 North Broadway Ave" compare equal and a tax ID typed as 482917305 matches 48-2917305. Grouping proposes clusters of records that probably describe one insured. Deciding is a person confirming the cluster and choosing, field by field, which value survives where the records differ. Merging creates one client that carries every policy, note and claim of the records it replaced. Keeping the undo means every original value stays in an archive and the merge can be reversed, with a reason.
The sample acquisition in the solution shows the scale. The client matcher grouped 401 records into 187 clusters, 38 of them reaching across both books. One cluster held three records for the same roofing contractor: one set up by the seller's commercial desk in 2009, one by its comp desk in 2016, and one by the buyer's surety team in 2023. Same tax ID once formatted, same location once the address was standardised, same phone on two of the three. The matcher scored them at 0.97 and listed where they differed for a person to choose: the mailing address, the main phone and the account manager.
A proposed merge shows its evidence and the values a person must choose
What makes two client records the same insured?
The same legal person or entity, and in insurance that usually means the same named insured with the same taxpayer identification number. Name, address, phone and email are evidence for it; none of them is proof. Two businesses can share an address, a phone and an owner, and still be two insureds with two sets of policies, two loss histories and two sets of obligations.
That is why the tax ID carries the most weight in the sample's match explanation, and why a mismatch on it is a stop, not a penalty. The other fields do different jobs. Location identifies the risk; a business can move and keep its tax ID. Names are noisy: "&" and "AND", "LLC" and no "LLC", a city appended in brackets. Phones and emails change hands. The matcher weighs all of them, but only a person can confirm that the records are one insured.
Personal lines add households. Two people at one address may be one household for servicing and two named insureds on their policies. The sample integration has a setting that merges spouses at one address into a household, which links their records without pretending they are one person.
Should you merge clients with different tax IDs?
No. Link them as related if the evidence says so, but keep them as separate clients. Different tax IDs usually mean different taxpayers, and the IRS's own rules show why identity can change while the name and address do not.
A sole proprietor who incorporates has to get a new employer identification number; one who simply changes the business name or location does not (IRS, When to get a new EIN). So the same contractor, at the same yard, under the same trade name, can legitimately appear in an agency's book under a Social Security number before incorporation and an EIN after. Those are two named insureds with two policy histories, and a claim from the first period belongs to the first one. The IRS list also runs the other way for corporations: a corporation that is "the surviving corporation after a corporate merger" keeps its EIN, while one that merges to "create a new corporation" needs a new one.
The sample integration treats "never merge two tax IDs" as a locked rule that the matcher cannot override. A person can still decide to link the two records as one relationship for servicing, which is often what the account manager wants; the records, their policies and their claims stay apart.
A matcher can be right about the business and wrong about the insured. The tax ID is where the law draws the line, so it is where the merge stops.
How do you handle DBA names in client matching?
Treat a trade name as a property of a client, not as a client of its own. A business doing business as another name keeps its legal name and its tax ID; the trade name is how it presents itself, and it is often what the seller's staff typed into the client field. The IRS says plainly that changing a business name alone does not require a new EIN.
In matching, that means comparing the trade name against both the legal name and any other trade names on file, and letting an identical tax ID settle the question when it is present. When the tax ID is missing on one side, a trade name match is a reason to look, not a reason to merge. Reviewers in an agreement test usually split on exactly these pairs, because one reads "Acme Roofing" and "Acme Roofing & Sheet Metal LLC" as obviously one company and the other wants to see a document. Settle it once in a written rule and apply it everywhere.
What is fuzzy matching for customer records?
Fuzzy matching, which statisticians call record linkage, scores how likely two records are to describe the same entity when no shared identifier settles it. A Census Bureau overview of the method puts its purpose simply: it "is used to identify duplicates when unique identifiers are unavailable", relying on names, addresses and other fields "that are typically not unique identifiers of entities" (Winkler, U.S. Census Bureau, 2006).
The classic model, from Fellegi and Sunter in 1969, makes three decisions, not two. Pairs above an upper threshold are matches; pairs below a lower one are non-matches; pairs in between go to people. The Census Bureau paper calls the middle band the region where no automatic decision is made and a clerk reviews the pair. That middle band is the honest design for client matching too, and it is the one the solution uses: proposals above 0.92 similarity go to a person, nothing merges on its own, and the tax ID rule sits outside the score altogether.
The same paper warns that companies are harder than people. "Business lists are typically more difficult to match than other types of lists such as person lists or agriculture lists because of the variants of names and the variants of addresses." An agency's commercial book is a business list, with the extra noise of desk-by-desk data entry, so it is where agreement between reviewers should be tested first. The payoff can be large: on the Census Bureau's own farm list for 1992, computerised matching left 2% duplication in the final file, against 10% in the file from five years earlier.
Two reviewers, one sample: measure agreement before you trust a matcher
Draw a sample of candidate pairs, have two people who know the book label each pair "same insured" or "not" without conferring, and measure their agreement overall and by kind of pair. The measurement tells you three things: whether your rules are clear enough to apply, which kinds of pair need a written rule, and what accuracy it is reasonable to expect from any matcher, since a machine cannot be checked against a truth that people cannot agree on.
Report two numbers. Percent agreement is the share of pairs labelled the same way. Cohen's kappa corrects that share for the agreement two people would reach by chance, using κ = (observed agreement − chance agreement) ÷ (1 − chance agreement). A review of the statistic in Biochemia Medica explains why both matter: percent agreement cannot "account for chance agreement", and kappa's conventional labels are generous, treating a kappa as low as 0.41 as moderate agreement; it adds that many texts set 80% agreement as the minimum acceptable (McHugh, 2012).
A worked agreement test on 200 candidate pairs (illustrative)
| Reviewer A | B says same insured | B says not the same | Total |
|---|---|---|---|
| A says same insured | 104 | 16 | 120 |
| A says not the same | 6 | 74 | 80 |
| Total for B | 110 | 90 | 200 |
From the counts to agreement and kappa
| Measure | Calculation | Result |
|---|---|---|
| Observed agreement | (104 + 74) ÷ 200 | 89% |
| Chance agreement | 0.60 × 0.55 + 0.40 × 0.45 | 51% |
| Cohen's kappa | (0.89 − 0.51) ÷ (1 − 0.51) | 0.78 |
Two practical points make the test worth doing. First, oversample the hard pairs: a random sample of an agency book is mostly obvious matches and obvious non-matches, which inflates agreement and teaches nothing. Stratify by kind of pair, as in the opening figure. Second, rerun the test after the rules are written. If agreement on trade-name pairs rises from the low eighties to the mid nineties once the rule exists, the rule works; if it does not move, the rule is not clear enough to follow.
Where reviewers disagree, and the rules that settle it
Disagreements cluster in a few kinds of pair, and each kind can be settled by a short written rule. The table below is a starting set; write your own from the disagreements in your test.
Match rules that settle the usual disagreements
| Kind of pair | Rule | Who decides |
|---|---|---|
| Same tax ID, different spelling or address | Same insured. Keep the legal name from the record with the tax ID document; choose the mailing address per field. | Analyst approves the merge |
| Different tax IDs, same address and phone | Not the same insured. Link as related if the owner is the same; never merge. | Analyst; the rule is locked |
| Legal name against trade name | Same insured only when the tax ID matches or a document ties the trade name to the entity. | Analyst, with the account manager if unsure |
| Tax ID missing on one record | Request the tax ID from a policy document before deciding; until then, keep apart. | Analyst |
| Seller's client already the buyer's client | Merge when the tax ID matches; the buyer's producer is told before approval. | Analyst, after the buyer's producer is told |
| Two people at one address (personal lines) | One household, two named insureds; link, do not merge people. | Account manager |
Can merged client records be unmerged?
They can if the merge kept what it replaced, and many merges in practice do not. A merge that overwrites the losing values and deletes the losing record is final; the only way back is from a backup and a re-key. A merge that keeps every original value, with who merged and when, can be reversed cleanly.
The solution records each merge with its source records and the values kept, and an undo needs a reason. In the sample, the roofing contractor's merge record names the three source records, says the legal name came from one, the mailing address from another and the office phone from the first, and confirms that every original value stays in the archive. That record is also what an auditor or an E&O carrier will ask for if a merged client later says a policy was mishandled.
Three records become one client, with the undo kept
Who should approve client merges?
The data migration analyst should approve merges within the seller's book, and merges that touch the buyer's existing clients should wait until the buyer's producer for that client has been told. The integration manager does not need to see each merge, but should see how many are waiting before signing the day-one load.
The reason for the second rule is commercial, not technical. A seller's client who is already a buyer's client has a relationship with someone at the buyer, often for a different line. Merging the records decides who services the combined account and who is paid on which renewals. In the sample, 38 of the 187 clusters crossed both books, mostly surety bonds and workers' comp written by the buyer's own teams. At sign-off, 35 clusters were still waiting; they loaded as the matcher proposed and approvals continued after day one, an accepted risk written onto the sign-off with its reason. Master data management for agencies covers how to keep one client record once the two books are combined.
Running a duplicate-client review on an acquired book
- Standardise before you match
Addresses, tax ID formats, legal suffixes and phone formats, or the matcher scores typing differences.
- Run the agreement test on a stratified sample
Two reviewers, blind, on every kind of pair; report percent agreement and kappa.
- Write a rule for every kind of disagreement
Then rerun the test to see whether the rules are clear enough to follow.
- Lock the tax ID rule
Different tax IDs are linked, never merged.
- Tell the buyer's producer before cross-book merges
The merge decides who services the account.
- Keep every merge reversible
Source records, values kept, who and when; an undo needs a reason.
Questions people ask
How do you find duplicate customers in a large agency system?
Standardise names, addresses, phones and tax IDs, then compare only records that share something, such as a tax ID, a postcode or a phone, rather than every record with every other. Record-linkage practice calls this blocking; it keeps the comparison count manageable without missing the likely pairs.
How many duplicate clients should we expect in an acquired book?
No public benchmark exists for agencies, so count your own. In the solution's fictional sample book, 3,412 client records became 3,198 clients after 214 records were merged in 187 clusters; books built desk by desk, with separate commercial and workers' comp teams, tend to carry more.
Is percent agreement or kappa the better measure?
Report both. Percent agreement is easy to read but ignores chance; kappa corrects for chance but its conventional labels are lenient. A review in Biochemia Medica notes that many texts set 80% agreement as the minimum acceptable.
Should duplicate clients be merged before or after go-live?
Before the day-one load wherever possible, in a staging copy. Merges left until after go-live are worked during renewals, by account managers who are also servicing clients, and every renewal processed on the wrong record adds to the clean-up.
What is a duplicate record?
Two or more records that describe the same real person or entity. In an agency system that usually means the same named insured entered more than once, under different spellings, by different desks, or once in each of two books after an acquisition.