ESG controversy score: two scorers, one incident, and the gap

An ESG controversy score rates how serious a company's involvement in an environmental, social or governance incident is, usually on a five- or ten-point scale built from the harm done, how directly the company was involved, how it responded and whether it keeps happening. No regulator defines the scale, so every data provider and many asset managers use their own, and the same incident can earn different scores. Severity is a judgment, and the measurable quality of a judgment is whether two competent scorers, working blind, reach the same answer. Botminds' ESG Risk Monitoring is an agentic solution that scores every serious incident twice, the second time without seeing the first, and sends the gaps to an analyst.

The score matters because policies key off it. In the solution's sample firm, a severity 4 incident in a screened fund must reach an analyst within four business hours, and severity 5 without credible remediation can lead to divestment. A score one level too low can keep a holding off a watchlist; one level too high can force a sale the fund later regrets. The sample firm, Quillfield Capital, and every company on this page except where a public source is cited are fictional.

What is an ESG controversy score?

It is a severity grade attached to a company because of one or more incidents: a pollution judgment, a mine collapse, a forced-labour finding, a bribery charge. Most scales combine four factors. Impact asks how much harm and to whom. Involvement asks whether the company caused it, contributed to it or is linked through a supplier or a former asset. Response asks whether it acknowledged, remedied and compensated. Recurrence asks whether this is the first incident or the latest in a pattern.

A company-level controversy score summarises those incident grades over a period. An incident-level severity score, which is what a fund's policy usually needs, grades one event. The difference matters: a company can carry a moderate overall score while one new incident in it requires a watchlist decision this week.

What does an ESG controversy score mean on a vendor's report?

It means the provider's analysts, applying the provider's methodology, graded the company's incidents on the provider's scale. Scales differ in direction as well as range, so the first thing to check is which end is bad and what a one-step change represents. The second is when the score was last updated, because controversy scores move with news and are refreshed on the provider's schedule.

The EU now treats some of these products as regulated. Its ESG ratings regulation, applying from 2 July 2026, treats as an ESG rating any opinion or score on a company's ESG profile or risk exposure that rests on a set methodology and a defined scale of rating categories, whatever the product is called (Regulation (EU) 2024/3005, Article 3; what the ratings rules require is covered separately). Raw "data on ESG controversies" is outside it, by the regulation's own recital. A controversy score with a methodology and rating categories may well be inside. Whether a given product is in scope is for its provider and supervisor; for a fund, the useful consequence is that the provider's methodology becomes something it can ask for.

Why do ESG controversy scores from different providers disagree?

Because each provider measures differently, covers different sources, weighs factors differently and updates on a different cycle. The best-known study of ESG rating divergence, by Berg, Kölbel and Rigobon, decomposed it into scope, weight and measurement and found measurement the largest part, along with a "rater effect" in which a provider's overall view of a company colours how it measures the details (Review of Finance, 2022). Christensen, Serafeim and Sikochi found that more corporate disclosure produces more rating disagreement, not less, and that raters disagree more about outcomes than about policies (The Accounting Review, 2022). A controversy is an outcome.

Timing adds its own gap. In the solution's sample, a district court's ruling against the nickel miner was public on a Monday evening, while the firm's data vendor still showed the company at its earlier category, last updated twelve days before. Neither number is wrong for the moment it was made; they describe different moments. EU supervisors have also observed managers' sustainability-risk analysis resting on "sectoral exclusion policies, controversy analysis and ESG ratings/scores" benchmarked against a data provider's rating (ESMA, June 2025), which imports the provider's disagreement with every other provider into the fund. How ratings are built, and why they diverge more broadly, is the territory of Botminds' ESG Issuer Research: why two raters score one company apart.

Severity is a judgment: score it twice, and blind

If two qualified people can disagree about a score, a single score carries an error nobody has measured. The remedy is borrowed from every field that grades by judgment, from medical diagnosis to exam marking: have a second grader score the same case independently, compare, and resolve the differences by a defined rule. Independence is the whole point. A second scorer who sees the first score anchors on it, and the agreement rate becomes meaningless.

ESG Risk Monitoring applies this to every incident of severity 3 or more. A Severity agent scores it from the four factors, citing each. A Second Opinion agent re-scores it on a different model without seeing the first score. If the two are within one level, the flow continues and both scores are kept with their rationales; if they differ by more than one level, the flow stops and the analyst sets the score from both rationales. The Severity agent may not score 5 on allegations alone, and below 75% confidence it hands the incident to a person rather than guessing.

ESG Risk Monitoring: when two controversy scores differ by two levels, a person decides

When the two scores differ: what the gap tells you

A large gap usually means the scorers disagree about a fact or a rule, not about arithmetic. In the sample above, a shipping company sold a bulk carrier that was later beached for scrapping, and two workers were injured. One agent scored 3, reading the sale as ending the company's involvement. The other scored 5, because the company's own policy commits it to safe recycling yards even when it sells through cash buyers, and because it was the second such beaching in 18 months. Both cited their sources. The question for the analyst is not which agent is cleverer but which reading the firm's policy supports.

The resolved disagreements are the most valuable output of the whole process. Each one is a case where the policy was unclear enough for two careful readers to part ways. Collected over a quarter, they show which clauses need rewriting, and the committee can settle the rule once instead of once per incident.

How do you measure agreement between two controversy scorers?

Start with the plain shares: how often the two scores are identical, one level apart, or further. Then correct for chance, because two scorers who both give most incidents a 2 will agree often without either being good. Jacob Cohen's kappa does that for categories (Educational and Psychological Measurement, 1960), and his weighted kappa extends it to ordered scales: plain kappa "treats all disagreements equally", while weighted kappa gives partial credit by how far apart the two grades are (Psychological Bulletin, 1968). On a severity scale, a 4 against a 5 is a smaller miss than a 2 against a 5, and the measure should say so.

No regulator sets a minimum. What matters is that the number exists, is measured on enough incidents in each severity band, and moves in the right direction after the policy is clarified. A firm that cannot state its agreement rate cannot say how much of a published controversy figure is signal.

A controversy score without a second, independent score is an opinion with a number on it. With one, the gap between the two becomes the firm's best measure of its own judgment.

What separates a 4 from a 5?

The evidence standard, in most policies. Severity 5 usually means grave harm, direct involvement, and a finding the company cannot easily reverse. The sample policy writes two guardrails into the scale: no 5 on allegations alone, and no 5 while a first-instance judgment can still be appealed. The miner in the river case stayed at 4 for that reason, with a watch condition set for any appeal by 2 November.

Markets draw a similar line. Studying stock reactions to corporate social responsibility news, Krüger found that "CSR news with stronger legal and economic information content generates a more pronounced investor reaction" (working-paper abstract, SSRN, 2013; the study was published in the Journal of Financial Economics in 2015). A court judgment carries more legal content than an accusation, and a severity scale that ranks them the same will misjudge both the harm and the financial risk. That reaction is also why severity belongs in a fund's sustainability-risk assessment, not only in its exclusion list; an ESG risk assessment framework built on decisions shows how the grade feeds the next step.

How to test a controversy scoring process before you rely on it

Test it on incidents whose right answer your own committee has already agreed. In ESG Risk Monitoring's design, the agents and their prompts are versioned and evaluated against a set of 1,860 past incidents labelled by the sample firm's committee, and every human override is recorded with its reason code and feeds the next evaluation. The same method works for a team of human analysts: give two of them the same 50 closed incidents, blind, and compare.

The second-opinion thresholds in ESG Risk Monitoring are written down

Agreement measures consistency, not timeliness: a process can score perfectly and still score a week late. The timing side, from a ruling's publication to a fund-mapped score, is traced minute by minute in a court ruling in 49 minutes, and the agents, settings and limits are described on the ESG Risk Monitoring page.

Questions people ask

What does an ESG controversy score mean?

It is a provider's or manager's grade of how severe a company's ESG incidents are, on its own scale. Check the direction of the scale, what each step represents, and when it was last updated before using it in a decision.

What is an ESG controversy exposure score?

Usually a portfolio-level figure: how much of a fund sits in companies with controversies above a given severity, weighted by holding size. It depends on the incident scores underneath it, so its reliability is their reliability. The PAI 10 indicator under the SFDR is a regulated cousin: the share of investments in companies involved in UN Global Compact or OECD Guidelines violations.

How is the severity of an ESG controversy scored?

Most scales combine impact, involvement, response and recurrence, and set an evidence standard for the top grades. In ESG Risk Monitoring's sample scale, no incident reaches 5 on allegations alone, and every factor is cited to the page it rests on.

Is a controversy score an ESG rating under EU law?

It can be. The EU ESG ratings regulation, applying from 2 July 2026, covers opinions or scores produced by a set method and graded on a fixed scale, whatever they are called, but excludes raw data on ESG controversies. A graded controversy product may therefore fall inside it while the incident feed behind it does not.

Should a fund use a vendor's controversy score or its own?

Use the vendor's as an input and keep your own decision. A vendor's score reflects its methodology and its update cycle; your fund's policy, prospectus and deadlines are yours, and supervisors judge the fund against them. Comparing the two, without copying either into the other, shows where each is ahead.