A fuzzy match percentage measures the source, not the translation

Nobody publishes what the percentage means

Every translation memory tool puts a number on a match. A new sentence comes in, the tool finds the closest sentence it has translated before, and it shows you the old translation with a percentage beside it: 100%, 92%, 76%. Translators plan their day around that number, and clients pay for words in bands of it.

What the number means is a question people keep asking, and the honest answer is that nobody outside the vendor knows. A TM-Town explainer calls it “the million dollar question”, then says there is “no set method or formula for how to calculate a fuzzy match”. Tool makers “do not release their algorithms”, and a few tests show “each tool produces a different fuzzy match score for the same test example” ¹.

Researchers say the same thing more formally. A 2014 study of TM retrieval states that the algorithm used by commercial systems “is typically not disclosed”, and that the best of them is “widely believed to be based on edit distance” ². Believed, not documented. The number that translation work is priced by is a formula each vendor keeps to itself.

The same TM-Town page shows why the choice matters. Take Description of the Service and Definitions against Ownership of the Services and Marks. Count every word equally and three of the six words match, plus a near miss on Service and Services, which one plausible scheme turns into a 65% match. Ignore the small words and it becomes plain that the two sentences have nothing useful in common ¹. Same text, two defensible formulas, two different answers.

One match, three scores, and a choice of which to trust

Open-source tools let you look. OmegaT’s manual says each match shows three percentages. The first is calculated “with stemming based on the source language tokenizer, and ignoring tags and numbers”. The second drops the stemming and still ignores tags and numbers. The third is “calculated on the entire text, including tags and numbers”, and is “generally the lowest” ³.

So one pair of sentences has three honest scores, and the tool has to pick one to sort by. By default OmegaT sorts by the stemmed one, the one that cannot see numbers. The preferences page says that to get “more literal matches closer to 100%” you switch to full text. It also shows the five best matches above 30% by default ⁴. None of that is wrong. It is a set of choices, and a percentage from another tool is the result of other choices you cannot see.

The choice changes more than the number on screen. It changes which old sentence you are shown at all. Bloodgood and Strauss ran six similarity measures over the same memories, and had five crowd workers rate how helpful each retrieved match was. Word-level edit distance, their stand-in for the industry method, picked the same match as their best measure only 35.0% of the time on Chinese to English and 39.3% on French to English ². Most of the time, two reasonable formulas pointed at different sentences.

The score never reads the translation

All of these formulas share one property. They compare the new source sentence with the old source sentence. The old translation, the thing you are about to reuse, is not part of the calculation.

The same paper found this the hard way. One match scored high under their best measure and the raters called it unhelpful. Their explanation was that “the English translation was incorrect”. The authors conclude that a limitation of their measure “and all other TM metrics we’re aware of, is that they only consider the source side” ². A perfect score on the source says nothing about a wrong target.

Commercial tools do lower a match for reasons beyond the text, and it is fair to say how. memoQ applies penalties when a match comes from a memory “that we know to be unreliable”, from a translator the team considers unreliable, or from an alignment nobody fully reviewed ⁵. Those are judgments about where a translation came from. They are set in advance, per source, and they still never read what the translation says. We have written about the memory that holds two approved targets for one source; a penalty cannot tell which of the two is right either.

The edits that change meaning are the ones that score well

A similarity score rewards small edits, and some of the smallest edits in a language are the ones that reverse a sentence.

TM source    Your card will be charged today.TM target    Votre carte sera débitée aujourd'hui.New source   Your card will not be charged today.​Word-level edit distance: 1 insertion against 7 words, 1 - 1/7 = 86%

At 86% the match sits in the band memoQ calls high fuzzy, 85 to 95%, which it describes as matches where, in an average sentence, “normally there is a difference of one word” ⁵. That is exactly what happened. The difference is one word, and the old French promises the opposite of the new English. We made the same point about message keys: edit distance measures spelling, not meaning.

Retrieval can do this too, not only scoring. In the same study, a French sentence meaning Do not use during pregnancy and lactation was matched, at one setting of the new measure, to Can be used during pregnancy and lactation ². The paper uses the example to show how a length setting changes which match comes back, not as a failure. It still shows a similarity measure handing a translator a sentence that says the opposite.

Numbers are the other small edit. memoQ puts a match in its “nearly exact” 95 to 99% class when the source text is the same but “numbers, tags, punctuation marks and spaces might be different”. It will also “substitute numbers if it can”, and when it patches tags or terms into a match it will “boost the match rate”, marked with an exclamation point ⁵. That is helpful automation. It also means the percentage you see can be a score for a translation the tool has already edited, not for the one in the memory.

The people doing the editing know where this bites. In an experiment with 24 professional translators working on 85 to 94% matches from English to Spanish, Guerberof found the main problem in fuzzy matches was accuracy, and explained why: in that range the changes needed “are related to single words and therefore if this change is missed, there will be more probabilities of Accuracy errors” ⁶. The high bands are where the old translation looks almost right, which is where it is easiest to accept.

The bands price average effort, not risk

The percentage matters because it is priced. A TMS lets a manager set a net rate scheme, and “the number of units for translation (pages, words, or characters) decreases based on the selected net rate scheme”. Phrase’s own worked example charges 10% for a 100% match and 0% for a context match ⁷. The scheme sets a fraction of the full rate for each band in between.

Where the band edges came from is not written down anywhere we could find. Bloodgood and Strauss note that setting the threshold at 70% is widely done, “but again it remains something of a black-box as to why 70% ought to be the setting” ². Parra Escartín and Arcedillo describe the traditional practice of giving no discount below 75%, on “the general assumption that they do not yield any productivity increase” ⁸.

When someone timed the bands, the order did not hold. The same authors timed in-house translators on a real software user guide in memoQ, keeping eight of the ten after discarding two sets. Against translating from scratch, the average speed gain was:

The 95 to 99% band was slower than the band below it. The authors explain that “the vast majority of edits required in the 95–99% band involved dealing exclusively with inline tags”, which slowed translators “more than it would be expected” ⁸. The score saw a nearly finished sentence. The work was in the formatting the score counts least.

That is one study, one language pair, one document and a 95 to 99% band of only 21 segments, so it proves no general rule about that band. It does show what the bands are. They are a price list attached to a similarity score, and similarity predicts effort only on average. We have written about the newer metered pricing that bills for the words you touch. Fuzzy bands are the older meter, and they bill for how alike the English looks.

Why the score works anyway, when a person reads the diff

The strongest case for the fuzzy score is that, on average, it works, and the research says so.

In Guerberof’s study, translators were faster on 85 to 94% matches than when translating from scratch, and the segments they translated from scratch contained more errors ⁶. Parra Escartín and Arcedillo scored machine translation output with a fuzzy-style score and found the speed gain stopped at about 75%, which they note matches the industry’s traditional cut-off for memory matches ⁸. The bands are rough, and they are not arbitrary.

The same paper found something more telling. With the same amount of editing needed, translators were faster on memory matches than on machine translation. The authors put that down to the tool: in machine translation the translator has to find what needs fixing, whereas with a memory match the tool will “highlight these parts automatically” ⁸. In their setup, the editor showed the difference between the stored source and the new one on every memory match.

That is the real safeguard, and it is not the number. The number decides what gets shown first. The highlighted difference, read by a person who knows both languages, decides whether the old translation still holds. The score works well enough because it was never asked to do the second job.

Keep the number as a sort order, never as a verdict

Trouble starts when the second job quietly moves to the number. Every tool has settings that do this. OmegaT can insert the best fuzzy match above a threshold into the segment for you ³. memoQ’s pre-translation treats a 95 to 99% match as a good match by default ⁵. A pipeline that pastes the top match into a model’s prompt does the same thing with no screen at all. In each case the percentage has stopped sorting candidates for a reader and started approving them.

A few habits keep it in its place:

• Use the percentage to order and price work, never to skip reading. Whatever the score, a person or a check looks at what changed in the source.
• Check the changed part with rules, not with the score. If the diff touches a number, a tag, a placeholder or a negation, the match gets reviewed at any percentage. A number or a tag can be compared mechanically between the new source and the old target.
• Distrust high scores on short strings. A one-word change in a four-word button is a large part of its meaning, and the tools warn that their numbers get vague there.
• Do not compare percentages across tools. A leverage report from one vendor and a quote built on another’s analysis were computed with different formulas, so the same file can land in different bands.

The picture at the top is what a high fuzzy match looks like. Same size, same laces, set out side by side as a pair. Everything a quick look compares matches, and they are both for the same foot. The only way to find out is to try to wear them.

References

1. 1.Kevin Dias The Fuzziness of Fuzzy Matches TM-Town blog
2. 2.Bloodgood and Strauss, 2014 Translation Memory Retrieval Methods EACL 2014
3. 3.OmegaT Panes OmegaT 6.1 user manual
4. 4.OmegaT Preferences OmegaT 6.1 user manual
5. 5.memoQ Match rates from translation memories and LiveDocs corpora memoQ 12.5 documentation
6. 6.Guerberof Arenas, 2014 Correlations between productivity and quality when post-editing in a professional context Machine Translation 28(3-4), author's copy
7. 7.Phrase Net Rate Schemes (TMS) Phrase support
8. 8.Parra Escartín and Arcedillo, 2015 Living on the edge: productivity gain thresholds in machine translation evaluation metrics WPTP 2015, MT Summit XV