Lexical precision and semantic intent: what a translation memory should actually store

A memory that stores only strings cannot see a paraphrase. A memory that stores only meaning cannot see the difference between 10°C and 100°C. Neither one is a translation memory that could be shipped.

A figure pulling open a drawer, drawn in white line on a slate ground

The question a translation memory has to answer

A translation memory is asked one question, over and over: have we translated something like this before? Everything else it does is bookkeeping. The interesting word is “like”, and for about thirty years the industry had exactly one answer to it.

Two segments were alike if their characters were alike. A memory was a table of source strings with approved translations hanging off them, and retrieval was a string comparison. That definition was not a limitation anyone apologised for. It was the product. A memory that returns what a human approved last time, character for character, is a memory you can put in front of a regulator.

Dense sentence embeddings gave the field a second answer. Two segments are alike if a model that has read a great deal of both languages puts them near each other in a vector space. That definition finds things the first one cannot see at all, and it lets a memory answer questions the first one could not even be asked.

So there is a real architectural question underneath: should a translation memory store the string, the meaning, or both? The answer is both, which reads as a dodge until you look at what each one does when it is on its own. They do not fail in the same way, and that asymmetry is the whole design.

A lexical retriever fails by returning nothing. A semantic retriever fails by returning something confident and wrong. Those two failures have completely different costs, and a system that treats them as interchangeable scores has already lost the argument.

How fuzzy matching actually works

Given an incoming source segment $S_q$ and a candidate already in the memory $S_m$ , classical fuzzy matching counts the single-character operations needed to turn one into the other, then normalises that count against the longer of the two.

\text{match}(S_q, S_m) = \left(1 - \frac{\operatorname{lev}(S_q, S_m)}{\max(|S_q|, |S_m|)}\right) \times 100

The fuzzy match score. lev is the Levenshtein edit distance: insertions, deletions and substitutions, sometimes with transpositions.

That percentage is not a report, it is a control signal. In production workflows the thresholds route work. Above roughly 95% a segment can be applied with a light review. Between about 85 and 95 it reaches a translator as a proposal to post-edit. Below that it is frequently not shown at all, on the reasoning that a bad suggestion costs more attention than an empty box costs. Neural fuzzy repair took those same tiers and handed them to a model instead of a person¹.

Edit distance is far too slow to run against every unit in a memory holding millions, so it is almost never the first stage. An inverted index over tokens or character n-grams, scored with something like BM25, narrows the corpus to a few dozen candidates, and the expensive character-level comparison runs only on those. The whole thing is cheap because the index is sparse and the vocabulary is discrete.

What surface matching gets right

The reason lexical retrieval survived the entire deep learning era untouched is not that it is accurate. It is that it is legible. You can answer “why did it show me this” with a diff. A translator who disagrees with a match can see precisely which characters differ, and an engineer who gets a bug report about a bad proposal can reproduce the ranking on a laptop. Nothing about the retrieval is a judgement call.

It also cannot invent. A lexical retriever will never return a segment containing a part number that was not in a segment it stored, because it has no mechanism for producing text at all. It returns rows. In medical devices, contracts and anything with a regulated glossary, that property is not a nice-to-have. The approved phrasing is frequently contractual, and a system that paraphrases it has not been helpful, it has created a compliance problem.

And it is fast in a way that shapes what you can build. Sub-millisecond lookups across millions of translation units, on CPU, with no model to host and no vector index to keep warm. You can afford to query it on every keystroke.

Where surface matching falls apart

The vulnerability is structural rather than incidental. Lexical algorithms treat tokens as atomic symbols with no relationship to each other. Swap a word for its exact synonym, move a clause, or convert an active sentence to a passive one, and the score collapses while the meaning does not move at all.

text

The technician must inspect the valve dailyDaily inspection of the valve is required by the technician      same instruction, same domain, same approved translation on file      edit distance: 44 of 58 characters      fuzzy match: 24%, which is below every threshold that would show it

A translation that already exists, in the same domain, that the memory will never surface.

This is not a rare edge. Technical documentation is full of the same instruction written four ways by four authors across six years, and a lexical memory holds all four as unrelated entries. The recall problem it creates is invisible, which is what makes it expensive: nobody files a bug about a suggestion that never appeared.

It gets worse as the morphology gets richer. In Finnish, Hungarian or German, a single lexeme surfaces in a dozen inflected forms, and every one of them is a different string.

text

de   Die Datei wurde gespeichertde   Die Dateien wurden gespeichert      one lexeme, two inflections, 6 characters apart      a human sees one root; the matcher sees two unrelated strings

Edit distance has no concept of a root. Stemming helps and does not solve it, because the inflection is often exactly the part that carries the agreement.

Dense retrieval and the shared vector space

Semantic retrieval throws away the string comparison entirely. Both sides are pushed through an encoder into a continuous vector space, and similarity becomes geometry: the cosine of the angle between two points. Segments that mean the same thing land near each other whether or not they share a single token.

The encoders are trained in pairs on parallel data, with a contrastive objective that pulls a segment and its real translation together while pushing it away from every other target in the batch.

\mathcal{L} = -\sum_{i=1}^{n} \log \frac{\exp\!\big(\operatorname{sim}(E_\theta(x_i), E_\phi(y_i)) / \tau\big)}{\sum_{j=1}^{n} \exp\!\big(\operatorname{sim}(E_\theta(x_i), E_\phi(y_j)) / \tau\big)}

The standard bi-directional contrastive objective. E_theta and E_phi are the source and target encoders, sim is cosine similarity, and tau is a temperature that controls how sharply the model separates the true pair from the rest of the batch.

Models trained this way, LASER² and LaBSE³ being the two most widely deployed, place a sentence and its translation in the same region of one shared space across a hundred languages or more. Paraphrase, reordering and inflection stop mattering, because none of them move a sentence very far.

The genuinely new capability is not better recall on the same corpus. It is that the source-side key stops being necessary. A classical memory can only return a target if you match its paired source, so the memory is bounded by how much parallel data you own. Cross-lingual encoders let you query a target-language monolingual corpus directly with a source-language sentence^4,6. That turns every reviewed document you have in the target language into retrievable material, which for most teams is a far larger pile than their aligned memory.

Semantic drift, or the 100°C problem

Then you ship it, and find out what a continuous space costs.

Encoders place conceptually related terms near each other. That is the entire trick, and in a translation memory it is also a hazard, because “related” and “interchangeable” are not the same relation. Hydrochloric acid and sulfuric acid appear in nearly identical contexts, so an encoder puts them in nearly identical places. The retrieval is working exactly as designed when it offers you the wrong acid.

The pooling makes it sharper. A sentence embedding is a fixed-length vector produced by collapsing every subword representation into one, and in that collapse the high-frequency contextual tokens dominate the orientation of the result. Fine-grained tokens, which is to say precisely the numbers and proper nouns you cannot afford to get wrong, contribute very little to where the sentence lands. So Operating temperature must not exceed 10°C sits right next to Operating temperature must not exceed 100°C, and the cosine similarity reports a near-perfect match on a pair that differs by one order of magnitude on the one value that matters.

There is a result that puts a number on how bad this gets. When a retrieval-augmented model is fed matches from a memory whose domain does not fit the text being translated, quality does not merely fail to improve. It drops below what the same model produces with no memory at all⁵. A semantic retriever always returns its nearest neighbours, and when the corpus holds nothing genuinely relevant, its nearest neighbours are confident nonsense that the model then leans on.

A match still has to survive the parser

Everything above is the literature’s version of the problem, and it is stated in terms of sentences. Localization has an extra constraint that does not appear in it at all: the thing being retrieved is usually not a sentence. It is a small program.

We have written about this before: a user interface string carries interpolation placeholders, plural branches, select cases and markup, and every one of those is structure that a parser will execute rather than prose that a reader will interpret. That argument was about generation. It applies just as hard to retrieval, and the semantic retriever is the worse offender, for exactly the pooling reason above. A placeholder token is low-frequency, structurally load-bearing, and almost invisible to the embedding.

text

source   Deleted {count} files from {folder}  0.94   Removed {count} items from {folder}        both placeholders present  0.91   Deleted the file from your folder          both placeholders gone  0.89   Deleted {count} files from {dir}           {folder} silently renamed

Three candidates ranked by cosine similarity. The two the retriever likes least are the two that would compile.

The second candidate is a fine sentence and a broken proposal: applied, the count and the folder name never render. The third is worse, because it looks correct in review and fails only for the one caller that passes folder. Neither is caught by a similarity threshold, because neither is a similarity problem.

Which is why the placeholder set belongs in a retrieval pipeline as a gate rather than as a score. Before a match is eligible to be proposed at all, the placeholders in the candidate’s source have to correspond to the placeholders in the incoming segment, and a candidate that fails is dropped from the list rather than down-weighted or surfaced with a warning. A rank is a claim about relevance. Structural compatibility is not the kind of thing that survives being expressed as one: down-weight it and all you have decided is how many broken proposals to show, not whether to show them.

That rule is worth stating on its own, because it is the practical form the whole lexical-versus-semantic argument takes. Semantics decides what is worth showing. The string decides what is safe to ship.

Running both retrievers and fusing the ranks

Which is why nobody serious runs one retriever. The two halves have complementary failure modes, so the architecture that works runs both and reconciles them: a sparse lexical index for exactness and speed, a dense index for reach, and a merge step that has an opinion about which signal wins where.

Dimension	Lexical only	Semantic only	Both
Matching space	Discrete strings and character n-grams	Continuous vector space	Sparse inverted index beside a dense one
Paraphrase and inflection	Score collapses; the match is never surfaced	Handled natively	Dense retriever supplies recall the sparse one cannot
Numbers and named entities	Any difference is visible immediately	Pooled away; 10°C matches 100°C	Lexical comparison gates what the dense side proposes
Cross-lingual retrieval	Impossible without a paired source	Native, including target-only corpora	Dense retrieval opens the corpus, lexical checks constrain it
Cost per query	Microseconds, CPU only	Milliseconds, plus a vector index to host	Two lookups and a rerank
Explaining a result	A diff	A number nobody can argue with	A diff for the constraint, a number for the ranking

The point of the third column is not that it averages the first two. It is that each side is used for the thing the other cannot do.

The merge is where the design decisions actually live. Rank fusion across the two candidate lists is the cheap version. A cross-encoder rerank over the pooled candidates is the expensive and better one. And there is a third approach that pushes the reconciliation earlier, into the retriever itself: train the dense encoder with word-level matching objectives alongside the sentence-level contrastive one, so the embedding space is aware of surface overlap rather than abstracting away from it⁶. A retriever trained that way is less likely to hand you the wrong acid in the first place, though it does not remove the need for a hard structural gate on the way out.

Handing a match to the model

Retrieval is half the system. The other half is what the model does with what it was handed, and the integration strategies differ more than they first appear.

The simplest is data augmentation. Neural fuzzy repair concatenates the source with the target side of a retrieved match behind a reserved delimiter, then trains the model on those augmented inputs¹.

S_{\text{aug}} \;=\; S_q \;\Vert\; \langle \text{sep} \rangle \;\Vert\; T_m

No architectural change at all: the encoder's ordinary self-attention learns when to copy a fragment from the retrieved target and when to generate.

It works, and it scales badly. Attention is quadratic in sequence length, so concatenating several matches gets expensive quickly, and the matches attend to each other as freely as they attend to the source, which produces interference that has nothing to do with the translation.

Approach	Match reaches the model as	General cost
Concatenated (RAT-CAT)	Source and all top-k matches in one sequence through one encoder	Quadratic in total length, and matches attend to each other as noise
Separated (RAT-SEP)	Each match encoded on its own path, merged at the decoder	No inter-match interference, but no source-to-match alignment in the encoder either
Joint	Each match encoded alongside its own copy of the source	Keeps source-to-match alignment and drops the cross-match noise; multiplies encoder passes
In-context prompting	Matches supplied as exemplars in an LLM prompt, training untouched	Nothing to train, but sensitive to exemplar order and prompt shape

Controlling which things are allowed to attend to which is the substance of the RAT architecture work; the joint row is the design that came out of it.

The comparison of these pathways, and the joint design that keeps source-to-match alignment while eliminating the cross-match clutter, is the subject of the retrieval-augmented translation architecture work⁷. The in-context route is the one most teams building on hosted models will actually reach for, and it is worth knowing that it holds up: supplying translation memory matches as prompt exemplars, with no change to training at all, is a real technique rather than a shortcut⁸.

The system that got too good at trusting the memory

There is one result here that changed how we think about the whole pipeline, and it is not an architecture.

If you train a retrieval-augmented system on high-quality, in-domain matches, it learns that the retrieved segment is trustworthy. That is a correct inference about its training distribution and a catastrophic one about production, where the memory is sometimes wrong, sometimes stale and sometimes about a different product entirely. The model has never seen a bad match, so it has never learned to discount one.

The fix is almost insultingly simple: stop always giving it the best match. Sample from a wider pool during training, a few candidates drawn from the top ten rather than the single top-ranked one, so the model spends training seeing matches of every quality and has to learn which fragments are worth copying. Doing this recovers up to 5.8 BLEU when the memory at inference time has a domain mismatch, and the same model stays competitive with the baseline when the memory is relevant⁵.

Worth reading that carefully, because the number is easy to misquote. The 5.8 BLEU is not a general improvement over a non-shuffled system. It is how much of the loss you claw back in the case where the memory does not fit, which is the case that a benchmark built from a well-matched test set never shows you.

The general lesson has very little to do with BLEU. A system trained only on the happy path encodes the happy path as an assumption, and the assumption is invisible until the day the retrieval degrades. That is worth holding onto for the section below, because there is a way to make your memory degrade quietly and continuously without anyone deciding to.

Retrieving against your own output

A translation memory in a modern pipeline is not a static human artefact. Practically every system that pairs machine translation with a memory writes some of its output back, per language, so that the next document benefits from the last one. It is genuinely useful: it is how a memory stops being a record of what a human once did and starts being a record of how a given project says things.

It also means that some of what you retrieve is your own past output. Which is fine, until you put it next to the two findings above.

Semantic retrieval has a bias toward the plausible. Postfill writes the plausible back into the corpus. Retrieve from that corpus and the bias is now in your training signal as well as your proposals. Nothing about this is a bug in any individual component, and every component behaving correctly is exactly what makes the loop hard to see: the similarity scores go up over time, because the memory increasingly contains things that look like what the model produces.

The mitigation is not clever, it is bookkeeping. A memory that postfills has to record provenance on every unit, so that a human approval and a machine proposal are distinguishable at retrieval time and can be weighted differently, or the machine-origin tier excluded entirely for content where that matters. If your memory cannot answer “did a person sign off on this” for a given segment, you do not have a translation memory with a feedback loop, you have a feedback loop.

BLEU, COMET, and the number that actually matters

None of these decisions can be made without measurement, and the measurement has the same split running through it as the retrieval does.

Surface metrics, BLEU and TER and chrF, compare strings. They are direct proxies for post-editing effort and they punish a perfectly good translation for choosing a different valid synonym. Neural metrics, COMET being the one in general use, encode the source, the hypothesis and the reference, and predict a human quality judgement from the three⁹. They correlate with human preference far better, and they are the right default.

They also have a blind spot that lands precisely on the failure modes this whole post is about. Because COMET is optimising for semantic adequacy, it will award a high score to a fluent output that has dropped a number, mangled a named entity, or omitted a clause, and in some conditions to output in the wrong target language altogether. The metric agrees with the semantic retriever about what matters, which means it cannot catch the semantic retriever’s mistakes.

The fix that has been demonstrated is to put the surface signal back into the neural metric rather than running the two side by side and hoping someone reads both. Two variants exist: one fuses sentence-level BLEU and chrF scores into the model through a late fusion layer, and the other tags word representations as ok or bad using a Levenshtein subword alignment against the reference¹⁰. Both are attempts to make a neural metric notice a wrong digit.

Metric	Measures	Blind to	Operational role
BLEU	N-gram precision against a reference	Punishes valid paraphrase; survives severe semantic errors that keep the n-grams	A baseline, and tracking verbatim reproduction
chrF	Character n-gram precision and recall	No discourse or document awareness	Morphologically rich languages, without needing a tokenizer
COMET	Source, hypothesis and reference through a trained regression model	Fluent output that drops a number or an entity; sometimes the wrong language	The primary quality signal, and human preference correlation
COMET with surface features	The above, plus BLEU/chrF features or word-level ok/bad tags	Costs more per evaluation pass	A guardrail where a wrong digit is the expensive failure

For a translation memory specifically, there is a number none of these produce, and it is the one that settles arguments: how much of what you proposed did a human rewrite, broken down by which retrieval tier proposed it. A memory whose 95%-and-above tier gets heavily edited is not returning what it claims to return. A semantic tier that gets accepted as often as the exact-match tier has earned the right to be shown higher. Neither BLEU nor COMET can tell you that, because both are comparing against a reference translation rather than against what your translators actually did with your suggestion.

Both, and the boundary between them

So: the string, the meaning, or both. Both, but the interesting part is not the answer, it is where the boundary between them sits.

At the storage layer, every unit carries its surface text and a synchronised vector, indexed in both a sparse structure and an approximate nearest-neighbour one, along with enough provenance to say where it came from and who approved it.

At the retrieval layer, both indexes run, and their candidate lists are merged by fusion or by a reranker. The dense side is responsible for recall, including recall against target-language material that has no paired source at all. The lexical side is responsible for the constraints: entities, numbers, terminology, and in localization the placeholder and markup structure that decides whether a proposal can be applied without breaking a build.

At the generation layer, the model receives matches with an honest signal about how good they are, and has been trained on matches of varying quality so that it discounts a weak one instead of copying it.

The line running through all three layers is the same one. The semantic representation decides what is worth considering. The surface string decides what is allowed through. Systems that collapse those two roles into a single similarity score are not simpler, they have just moved the decision somewhere nobody can inspect it, which is the one place a translation memory should never put anything.

References

Bulté and Tezcan, 2019 Neural Fuzzy Repair: Integrating Fuzzy Matches into Neural Machine Translation ACL 2019
Artetxe and Schwenk, 2019 Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond arXiv:1812.10464
Feng, Yang, Cer, Arivazhagan and Wang, 2022 Language-agnostic BERT Sentence Embedding arXiv:2007.01852
Esplà-Gomis, Sánchez-Cartagena, Pérez-Ortiz and Sánchez-Martínez, 2022 Cross-lingual neural fuzzy matching for exploiting target-language monolingual corpora in computer-aided translation EMNLP 2022
Hoang, Sachan, Mathur, Thompson and Federico, 2022 Improving Robustness of Retrieval Augmented Translation via Shuffling of Suggestions arXiv:2210.05059
Bouthors, Crego and Yvon, 2025 Improving Retrieval-Augmented Neural Machine Translation with Monolingual Data arXiv:2504.21747
Hoang, Sachan, Mathur, Thompson and Federico, 2022 Improving Retrieval Augmented Neural Machine Translation by Controlling Source and Fuzzy-Match Interactions arXiv:2210.05047
Reheman, Zhou, Luo, Yang, Xiao and Zhu, 2023 Prompting Neural Machine Translation with Translation Memories arXiv:2301.05380
Rei, Stewart, Farinha and Lavie, 2020 COMET: A Neural Framework for MT Evaluation arXiv:2009.09025
Glushkova, Zerva and Martins, 2023 BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluation EAMT 2023