Can a system tell which strings need translating?

The string does not tell you what it is

Here are six string literals taken out of real codebases. Three of them need translating and three of them must never be touched. The text is all you get, which is the same thing a classifier gets.

"Times New Roman""*.txt""Tracked""user.role.admin""Command not implemented""SELECT name FROM survey WHERE id = ?"

Casing does not separate them. “Times New Roman” is title case, space delimited and made of dictionary words, and translating it changes which font the application asks for. Dictionary lookups do not separate them either: “Tracked” is an ordinary English past participle that happens to be a movement type in a war game, and it does need translating. A regex for technical shapes catches *.txt and user.role.admin, and then it also catches every product name and every acronym you were supposed to leave alone anyway, so it has told you nothing new.

The SQL is the one that ends the argument. A query is obviously not user-facing copy, until you find the ones that are printed on the screen. When the FSE 2010 study of this problem ran its tool over Lime Survey, six of its false positives were user-visible SQL queries². Visible to a user, made of words, and untranslatable.

So the question in the title has a short answer and a long one. The short answer is that no, a tool cannot tell just by reading the string, because the property being asked about is not a property of the string. The long answer is about what it is a property of, and what that costs you.

Most literals in a codebase are not for reading

Before any of the accuracy talk, the base rate, because it decides which kind of error you can afford and nobody quotes it.

The ICSE 2009 study that established this task counted every constant string in four Java applications and then established, from the versions those projects shipped after internationalizing, which ones had actually needed translating ¹. The ratios are not close to even.

Megamek is the honest case, because it is the big one. A hundred and ten thousand lines of Java, ten and a half thousand string literals, and five out of every six of them are not text anybody reads. Class names, map keys, file paths, protocol tokens, debug output, the whole sediment of a decade of a program talking to itself.

Now imagine a classifier that is right ninety percent of the time on each individual string, which sounds respectable. Run it over Megamek and it hands you roughly eight hundred and seventy false positives out of the eight thousand seven hundred literals that were never candidates. That is five times the number of real strings it missed, and somebody has to read all of them.

This is why per-string accuracy is the wrong number to advertise. On a task where the negative class is five times the positive one, the error that matters is the one you make on the majority.

What makes a string translatable is where it ends up

The technique that actually works stops looking at the string. It picks a set of methods it already knows put text in front of a person, treats every one of them as a sink, and then traces backwards through assignments, concatenations and returns to find every literal that can reach one ^1,3. Whether the string reads like English never comes up. The question is whether a value can arrive at a label.

On the four Java applications, that produces results a linter cannot approach.

Ten missed strings in Megamek. Not ten percent, ten strings. That is a better result than the developers of Megamek managed by hand: the study reported seventeen user-facing strings the tool found that the project had internationalized around and left in English, and the maintainers confirmed and fixed all seventeen¹.

The reason it beats reading the code yourself is the case where the string and the label are nowhere near each other. Megamek builds a unit description by appending a translated fragment to the return value of a helper.

the code  sBasic.append(Messages.getString("MechView.Movement"));  sBasic.append(entity.getMovementTypeAsString());​what getMovementTypeAsString returns  case TRACKED: return "Tracked";  case WHEELED: return "Wheeled";​what the German build renders  Bewegung: Tracked

Nothing about “Tracked” at its point of definition says user-facing. It sits in a switch, in a method that returns a string, in a class that is not a view. A person auditing that file has no reason to stop on it. Only the flow from that return to an append to a description that a window renders makes it a translatable string, and following that flow is exactly the thing a program is better at than a person.

Which sets up the claim this post is actually about. Translatable is not an attribute the literal has. It is a statement about the set of paths the value can take through the program, and the literal is one end of it.

The same technique, a different codebase, a different number

The same authors took the same technique to PHP web applications a year later. It fell apart.

The tool returned nearly three false positives for every string that needed translating. The paper says so in its own words, that without a way to separate user-visible constant strings from non-visible ones the previous approach is not suitable for web applications ².

Nothing about the technique changed. Nothing about the quality of the analysis changed. What changed is that in a PHP application almost every string reaches output, because the output is a document that the program prints. <div class= reaches the browser. So does </td>. So does the query whose result gets echoed. Every literal in the file is on a path to a screen, so a sink analysis says yes to all of them.

What repaired it is worth knowing too, because it was not a model and it was not semantics. It was a structural distinction: for a string emitted into HTML, is it inside a tag or outside one. Adding that removed 526 of the 534 false positives in Squirrel Mail, 380 of 398 in Lime Survey and 98 of 100 in Mrbs². One fact about position in a grammar, and the problem became tractable again.

The authors are also straight about where it still does not reach. Their approach could not handle applications that generate JavaScript on the fly, because you would have to analyse the syntax of a program that does not exist until runtime². That was 2010, and it describes most of what has been built since.

Two throwaway heuristics did most of the cleaning

There is one more result in the 2009 paper, reported almost in passing, and it is the most useful thing in it.

After the taint analysis produced its candidates, the tool applied a filter. The filter was two rules. Drop any constant string containing no letter character. Drop any string equal, ignoring case, to the name of the project ¹. That is the whole filter.

Two lines of logic, and Megamek’s false positives fall from 356 to 41. The last column is the part that matters: turning the filter off finds nothing new. It is pure noise removal, at no cost in coverage.

Hold that against the interesting part of the system. String-taint analysis is a real piece of program analysis, and on Megamek it was worth 575 strings: with the transmission tracking switched off, the misses went from ten to five hundred and eighty five¹. The analysis is what finds the strings. But what makes the output something a human being will actually sit down and read is a two-rule filter that any of us would have written in a minute and never mentioned in a paper.

Both halves are load-bearing and they do different jobs. It is the same shape as putting a deterministic check in front of a paid probabilistic one: the cheap rule is not a lesser version of the clever one, it is the thing that makes the clever one usable.

The two errors do not cost the same

Every tool in this space is tuned somewhere on a line between missing strings and over-reporting them, and the whole literature discusses that as a tradeoff between two numbers. It is not a tradeoff between two numbers. The two errors do different damage.

A missed string ships English into a French build. It is visible, embarrassing, reported by the first user who sees it, and fixed by adding a key. The RText study found “Line” and “Col.” sitting untranslated in the status bar; they were externalized eleven months later, and in the meantime the application worked ¹. Squirrel Mail had “Command not implemented” hardcoded in its SMTP handling for about three years ². Both are bugs. Neither is an outage.

A false positive is a different animal, because sending a string to be translated is a change to the program. The ICSE 2009 paper catalogues what its own false positives were, and the largest category in three of four subjects is strings that are visible on the screen and still must not be translated: file extensions, directory paths, font names¹. Sixty-one of the sixty-five in ArtOfIllusion. Thirty-five of the forty-one in Megamek. Translate a font name and the application asks the system for a typeface that does not exist. Translate an extension and the file dialog filters on a pattern that matches nothing.

The worst version of it is in the paper’s false negative analysis, and it runs the other way. A string constant compared against a value read back out of a widget is not something you may translate independently, because translating the constant while the widget returns translated text, or the reverse, breaks the comparison. The authors note that this category may result in run time errors¹. Every false negative in ArtOfIllusion and every one in Megamek was of this kind.

So the tool that was too conservative was conservative in exactly the right place. This is the argument we have made about translation being a code change, arriving one step earlier than usual. It is normally a claim about what happens after a translator edits a string. Here it is a claim about the extraction step: the act of nominating a literal for translation is itself the edit, and it can break the build before any translator has seen it.

Which gives you the tuning rule, and it is not symmetric. Missing a string costs you a bug report. Nominating the wrong one costs you a defect that no reviewer of the French file can see, because the French file looks perfectly reasonable. Tune for precision, and accept that something else has to find what you missed.

Which is why the shipped tools check so little

Look at what the tools people actually run do, with that asymmetry in mind, and their apparent timidity turns out to be the whole design.

Android’s HardcodedText check ships on by default in the standard toolchain and is the most widely deployed internationalization check in existence. It affects resource files: it reads XML layouts and flags a hardcoded attribute where a @string reference belongs⁴. It does not look at Kotlin or Java at all. Every string your code builds, formats, concatenates or returns from a helper is invisible to it, by construction.

eslint-plugin-i18next makes the same call and exposes the dial. Its no-literal-string rule defaults to jsx-text-only, which forbids plain text in JSX markup and nothing else. There is an all mode that validates every literal in the file, and alongside it a configuration surface for surviving that mode: allowed words by pattern, ignored components, ignored attributes, ignored callees, ignored object properties, ignored class properties, each with include and exclude lists⁵.

That option list is the honest artifact here. It is what the two-rule filter from the 2009 paper grows into once real projects run it, and it is maintained by hand, per repository, forever. Nobody built that because they enjoy configuration. They built it because turning the recall up on a real codebase produces the Squirrel Mail result, and the only way back is to enumerate your own exceptions.

So the mainstream tools are markup-only not because their authors could not manage a data-flow analysis, but because markup-only is the region where a check can be default-on and still be trusted. Precision by construction, coverage traded away deliberately, and the missed strings left for a person to notice.

The oldest tool in the field refuses to guess

The oldest tool in the field answers the question by refusing it.

xgettext has been extracting translatable strings since the early nineties and it has never classified a string in its life. It looks for calls to a keyword, and it extracts the argument. The default keyword set is a list of function names, gettext, ngettext, dgettext and their relatives, and you can add your own ⁶. Anything not wrapped in one of them is not extracted, however English it looks.

not marked, and therefore not extracted  printf("Command not implemented");​marked, and therefore extracted  printf(gettext("Command not implemented"));

There is an --extract-all flag that takes every literal in the file ⁶. It exists, it is documented, and essentially nobody runs it on a real project, because what it hands back is the Squirrel Mail result: every literal in the program, most of which are not text.

Read that design decision as an answer to this post’s question, because that is what it is. Thirty years ago the people who built the canonical extraction tool concluded that the classification problem was not worth solving, and that the developer should record the answer at the moment they already know it. The information exists exactly once, in the head of the person typing the literal, and it costs four characters to write it down.

Which is the position this blog keeps arriving at from different directions. A system cannot recover a fact that nobody wrote down. Every classifier in this post is doing archaeology on a decision that was already made and then discarded. The developer knew. They wrote “Tracked” meaning a word a player reads, and they wrote “user.role.admin” meaning a key, and the file records neither intention. Everything after that is inference over evidence that was thrown away.

The obvious objection is that marking is a convention and conventions are not kept, which is true and is the reason the linters exist. A convention is a request with a failure rate. But notice what the linter is doing in that arrangement. It is not classifying strings. It is checking that a mark is present where the markup grammar already proves one is needed, which is a decidable question, and leaving alone the region where the question is not decidable. The mark carries the fact, and the gate checks the mark.

The answer, and what it is an answer about

So, to what extent can automated tools tell user-facing strings from program identifiers?

On code that marks its strings, completely, and the tool does not need to be clever because there is nothing left to decide. On code that does not, well enough to be worth running and never well enough to trust unread. The published evidence is one line of work that reached ten missed strings out of 1,734 on a large Java application, and 137 to 290 percent false positives on PHP applications with the same analysis^1,2.

Both of those are the same tool. That is the finding. The number is a property of the codebase, and specifically of how far its literals travel and how much of its output is a document rather than a widget. A codebase where UI text goes straight into a component is easy. One where strings are assembled, cached, keyed and passed through helpers is hard, and it is hard in a way no model fixes, because the information the model would need is not somewhere else in the file. It was never written.

If you want to know how automatable your own extraction is, do not ask a vendor for their accuracy figure. Go and look at how your user-visible strings reach a screen. If most of them are literal children of components, a markup rule will find nearly all of them and you should turn one on today. If most of them are built somewhere and rendered somewhere else, no tool is going to give you a clean answer, and the useful work is upstream: mark them as you write them, and put a gate in front of the region where the mark can be checked.

The literal in the file does not know what it is for. The person typing it does, for about four seconds. Everything in this post is a consequence of not writing it down.

References

1. 1.Wang, Zhang, Xie, Mei and Sun, 2009 Locating Need-to-Translate Constant Strings for Software Internationalization ICSE 2009
2. 2.Wang, Zhang, Xie, Mei and Sun, 2010 Locating Need-to-Translate Constant Strings in Web Applications FSE 2010
3. 3.Wang, Zhang, Xie, Mei and Sun, 2013 Locating Need-to-Externalize Constant Strings for Software Internationalization with Generalized String-Taint Analysis IEEE Transactions on Software Engineering 39(4)
4. 4.Android lint checks HardcodedText: Hardcoded text googlesamples.github.io, custom lint rules reference
5. 5.eslint-plugin-i18next no-literal-string GitHub, rule documentation
6. 6.GNU gettext Invoking the xgettext Program GNU gettext manual