How good is AI at translation? A test for fund documents

How good is AI at translation? In the field's main annual public test in 2024, the best systems produced near-flawless English to Spanish, yet human translations were still in the top-ranked group for seven of the eleven language pairs, and the test deliberately avoided specialist text. For a fund manager that answer is incomplete in two ways. The documents that matter, the PRIIPs key information document, the factsheet and the shareholder notice, are specialist text, and the errors that matter in them, a transposed digit, a paraphrased warning, "trimmed" rendered as "sold", are exactly the ones general benchmarks score lightly. The honest way to know how good AI translation is for your documents is to measure it on them. Botminds' Fund Document Translation is built to be measured that way: its agents report four numbers on every cycle, and people approve each language.

The sample figures on this page come from the solution running on fictional data, a manager called Quillfield Capital with 64 cross-border funds.

How good is AI at translation? What the field's main public test found in 2024

Very good on general text, not yet better than people across the board. The Conference on Machine Translation runs the field's main shared task each year, and its 2024 findings carry the verdict in their title: "The LLM Era is Here but MT is Not Solved Yet" (Kocmi et al., WMT24). The organisers evaluated systems from 28 participants plus "8 different large language models (LLMs) and 4 online translation providers" on eleven language pairs, with professional annotators marking error spans in whole paragraphs rather than sentences.

Three findings matter for anyone choosing a tool. "Human references are in the winning cluster in 7 out of 11 language pairs", so even the best systems did not clearly beat professional translation in most pairs. Quality varied by direction: "the best systems for English→Spanish produced close to flawless translations making it the easiest language pair". And the automatic metrics vendors often quote were not reliable judges: "Automatic scores are biased", the organisers write, because a system tuned on a metric can top that metric without winning with human evaluators.

Why a general benchmark says little about a KID

Because the benchmark was designed to avoid documents like it. The WMT24 organisers chose news, social and user-generated content, speech, literary and educational text, "to be understandable for humans without specialist in-domain knowledge, thus not requiring specialized translators or human raters for evaluation". A key information document is the opposite: a fixed structure, regulated phrases, four performance scenarios and cost tables, and a three-page limit. A system's rank on news says little about how it handles "moderate scenario" in Polish or a summary risk indicator sentence in Greek.

The way quality is judged matters as much as the text. In a study of a claimed parity between machine and human translation, Läubli, Sennrich and Volk found that "human raters assessing adequacy and fluency show a stronger preference for human over machine translation when evaluating documents as compared to isolated sentences", and concluded that errors "which are hard or impossible to spot at the sentence-level become decisive" (EMNLP 2018). A fund document is read as a whole: a scenario table that contradicts its narrative, or a term rendered two ways on one page, is invisible in a sentence-by-sentence score. Any test of AI translation for fund documents should be judged at document level.

Is AI translation accurate on numbers? Not reliably, and the usual scores hide it

No, not reliably, and the problem is general rather than a quirk of one system. Wang and colleagues built behavioural tests for numbers and found "that numerical mistranslation is a general issue: major commercial systems and state-of-the-art research models fail on many of our test examples, for high- and low-resource languages" (Findings of ACL 2021). They name the stakes in finance directly, in the sentence below.

"Numerical errors made in financial document translation, e.g., an extra or omitted digit or decimal point, could lead to significant monetary loss."

Wang et al., 'As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translation', Findings of ACL 2021

The same paper explains why overall quality scores do not reveal the problem: a numerical error "only contributes a very minor penalty, as it is typically a single-token mistranslation". A translation can score well on a metric and still carry the wrong ongoing charge. The study tested systems available in 2021, and newer models are better at many things, but the measurement point stands: a test that does not check every figure separately will not tell you whether the figures are right.

Measure 1: figures, exact or wrong

Figure accuracy is binary per figure and should be measured on all of them. Count every figure, date, currency amount, percentage and identifier in the translated version, compare each with the English source and with the fund's own data, and record two kinds of failure separately: a wrong value, and a right value in the wrong format for the market. In Germany 12.4% must read 12,4 %, with a decimal comma and a space before the sign; in German-speaking Switzerland the same figure reads 12.4 %. A Swedish factsheet that writes a fund size as "1.912,3" where Swedish needs "1 912,3" has the right value and the wrong format.

A transposed figure in a French KID, and both readings with their sources

In the sample cycle the Numbers agent checked 39,804 figures against the source and the performance system and found six format or value problems, one of them in the English itself. That last one is a reason to measure against the fund's data and not only against the English: the Dutch factsheet of one fund carried 1.9% because the English did, while the performance system said 1.8%, so the source was wrong and every language would have repeated it. A pass mark for figures is zero released errors; the measure that varies is how many the checks catch before release.

Measure 2: regulated phrases, character for character

Regulated-phrase fidelity counts how many fixed phrases in the translation differ from the wording approved for that market, compared exactly. The PRIIPs Regulation itself prescribes some of them: the title "Key Information Document", and an explanatory statement that "shall read" a set text beginning "This document provides you with key information about this investment product. It is not marketing material." (PRIIPs Regulation, Art. 8). Others, such as a market's past-performance and currency warnings, are fixed by guidelines or by Compliance's approved wording.

Meaning-based checks and general quality scores both reward a good paraphrase, which is why this measure has to be exact. In the sample cycle, 9,368 regulated phrases were held to Compliance's approved wording and none was altered. A tool that cannot report this number on its own output has not been tested on it. How regulated phrases are held word for word in every market covers the difference between a glossary term and a locked phrase.

Measure 3: meaning drift, paragraph by paragraph

Meaning drift is measured by translating each paragraph back into English with an independent model and scoring how far it has moved from the original. It catches what the first two measures cannot: a verb that changes the trade, a weighting word that changes the view. In the sample the German factsheet's "trimmed" came back as "closed out", drift 0.62 against a threshold of 0.25; a French "underweight" came back as "excluded", 0.33. The score must be explained, not just reported, and it must be independent: round-trip translation through the same system was found to be "a poor predictor of quality" (Somers, 2005).

Report two numbers for this measure: the share of paragraphs above the threshold, and the share of those that the reviewer confirms as a real change of meaning. A tool that flags everything has a high catch rate and wastes the reviewer's day; one that flags nothing looks perfect until a regulator reads the Polish. The back translation process step by step covers how to calibrate the threshold.

Measure 4: reviewer edits per page

Reviewer edits per page measure what the checks left for people. After every automated check has run, count what the in-country reviewer changes, by type: meaning, terminology, style, layout. It is the measure closest to the reader's experience and the one most often skipped, because it requires reviewers to give a reason for each edit. In Fund Document Translation every decision carries a reason code, so the count falls out of the approvals.

This measure also tells you whether the reviewers trust the tool. Edits that cluster on terminology suggest a glossary gap that a rule can close; edits that cluster on style in one language suggest the market's style guide needs work; meaning edits after release should be zero. Over three or four cycles the trend matters more than the level.

AI translation quality assessment: judge whole documents, not sentences

A quality assessment for fund documents should be run on whole documents, by reviewers who read the target language, with errors marked rather than sentences rated. That follows directly from the research. The WMT organisers moved to document-level evaluation in 2024 and "no longer provide source texts segmented into individual sentences", keeping paragraphs intact; Läubli and colleagues showed that sentence-level judging hides the errors that separate good translations from adequate ones.

For a manager this means three practical choices. Test on complete KIDs and factsheets, not extracted paragraphs. Ask reviewers to mark each error and its type rather than give a score out of five. And keep the automated measures separate from the human one, so you can see whether a tool's figures and regulated phrases are right independently of whether a reviewer liked its style.

Is AI translation accurate enough for regulated fund documents?

On its own, no tool's output should be treated as accurate enough; with independent checks and a reviewer's approval, AI translation can meet the standard the law sets. That standard is a result, not a method: a KID translation must reflect the original faithfully and accurately (Art. 7(1)). Regulators expect firms to control AI output before it reaches investors. ESMA warns that text-generating AI can "hallucinate", writing things that sound accurate and are false, and expects "robust controls" to ensure accuracy "ex-ante" (ESMA, 30 May 2024).

Every figure in the market's format, checked against the performance system

That is how Fund Document Translation is designed. Its agents draft and check; four checks run on every version; anything that fails goes to the reviewer for that market; and a figure mismatch, an altered regulated phrase, a page overflow or a translated fund name always blocks release. The in-country reviewer approves the language, and Marketing Compliance releases it as a second person. The answer to "is it accurate enough" then becomes four numbers you can see each cycle rather than a vendor's assurance.

How to run the test on your own documents

The most informative test uses documents you have already approved, because you know what correct looks like. Take one recent cycle's released versions as the answer key, run the tool on the same English sources, and measure the four numbers against what you actually sent. No regulator or independent study publishes a pass rate for fund document translation, so your own documents are the only benchmark that counts.

Two results in such a test are worth more than any average. One is a figure that the tool changed silently, which should end the evaluation. The other is a meaning change that the checks did not flag and a reviewer found. Count both, and judge any tool, including Fund Document Translation, by them.

Questions people ask

What is the best AI for translation?

There is no single answer, and no independent ranking for fund documents. At WMT24 the leading system differed by language pair, human references were in the winning cluster in seven of eleven pairs, and the test domains excluded specialist text. Test the candidates on your own approved documents.

How do AI translation accuracy comparisons work?

Credible comparisons use professional annotators marking errors on whole documents, as WMT24 did, rather than automatic metrics alone, which the organisers found "biased". Ask any vendor whether its comparison was human or automatic, sentence or document level, and on which text domain.

What is an LLM translation benchmark?

A shared test set and evaluation method used to compare translation systems, including large language models. WMT's annual General Machine Translation task is one; in 2024, its nineteenth edition, it covered eleven language pairs and included translations from eight large language models and four online translation providers.

Is AI translation better than human translation?

Sometimes on general text, not consistently. In WMT24 human translations were still in the top group for most language pairs, and when documents rather than single sentences are judged, raters prefer human translation more strongly. For regulated fund documents the practical question is whether the checks and the reviewer catch what the machine gets wrong.

Can AI translate numbers correctly?

Not reliably without a separate check. Behavioural tests found numerical errors across major commercial and research systems, and the usual quality metrics barely penalise them. Every figure should be compared with the source and the fund's data in the market's number format before release.