What is a good COMET score?
There is no passing grade: an untranslated copy of the source averaged 65.72 out of 100 in one public report, and the same checkpoint scored the same files 0.86 or 0.79 depending on the package version, so the only COMET number with a meaning is a difference between two systems scored the same way.

A COMET score has no passing grade
In February 2023 someone asked the COMET maintainers the question most teams get to eventually. They had 500 French sentences from TED talks and 500 Spanish sentences from parliament sessions, and the French set scored higher. Did that mean the French machine translation was better? And was there a way to read a COMET score “other than using it as a ranking system”? 1
Ricardo Rei, first author of the COMET paper, answered that you should avoid comparing scores between languages and even between domains, and that this is true of any MT metric, not just COMET. Asked again for an interpretation, he suggested plotting the distribution of scores for one language pair and one domain and looking at quantiles 1,2.
The official FAQ says it in one sentence: “While the raw score itself does not have a direct interpretation, it is useful for ranking translations and systems according to their quality.” It then adds that newer models such as wmt22-comet-da scale scores between 0 and 1, where “a score close to 0 indicates a translation that is no better than random chance” 3. Keep that second sentence in mind; the next section tests it.
The scale belongs to the checkpoint. The older default, wmt20-comet-da, is documented as unbounded and “typically” between -1 and 1. Its successor, wmt22-comet-da, is documented as between 0 and 1 4,5. A 0.8 from one is not a 0.8 from the other. A 2023 study that tried to use an unbounded COMET-DA model as a pass or fail filter found that, for one task, the best cutoff on its development data was -0.028, and that a correct translation of a Chinese flight query, “Sunday afternoon from Miami to Cleveland”, scored -0.1 6.
So the honest answer to “what is a good COMET score” is that there is no such thing. There is a good difference, under conditions most published numbers never state.
An empty translation does not score zero
If 0 meant “no better than random chance”, a translation that is not there at all would score close to 0. Zouhar and colleagues checked. They scored WMT23 test sets with wmt22-comet-da four ways: a real system’s output, a file of empty lines, fluent sentences in the right language picked at random from another test set, and the same random sentences with their words shuffled 7.
| Output scored | English to Chinese | Chinese to English |
|---|---|---|
| A real system (Online-A) | 0.862 | 0.803 |
| An empty line | 0.392 | 0.374 |
| A random fluent sentence | 0.434 | 0.350 |
| A random sentence, words shuffled | 0.419 | 0.347 |
The gap between real output and nothing is large, which is why COMET ranks systems well. But “nothing” sits at roughly 0.4, so the bottom of the scale is not where a non-translation lands. The authors also counted real translations that scored below an empty line for the same source. That happened rarely, 6 times out of 2,075 for English to Chinese, but it happened 7.
The output language is not checked either. The same team took a system’s English to Russian test set and swapped in that system’s Ukrainian output for the same English sources. The correct Russian scored 0.853. The Ukrainian scored 0.797. Random correct Russian sentences scored 0.463, and empty lines 0.316 7. Translating into the wrong, closely related language cost about 0.06. Answering in the right language with a sentence that meant something else cost about 0.39.
A copy of the source can do better still. In December 2023 a user reported that when their system returned the input unchanged, the copy could outscore a real attempt. For the Chinese source 没有可能, with the reference “not possible”, the untouched copy scored 92.40 and the translation “impossible” scored 90.15, on the 100-point scale the reporter printed. Leaving all 1,875 sentences of a test set untranslated averaged 65.72 8. Two of the researchers behind the study above replied: there is no explicit penalty for a hypothesis in the wrong language, so run language identification, and always run a string metric such as chrF or BLEU alongside COMET, because it is more sensitive to output in the wrong language.
None of this means COMET is broken. It learned from human judgments of real translations, and in the paper’s words it “has likely not seen empty hypotheses during training that would have received a score of 0 from a human annotator” 7. It does mean the number cannot tell you that a translation exists, is in the right language, or is a translation at all. Those are cheap checks, and they belong in front of the score.
The same files get a different score
The same checkpoint does not even give the same number twice. The COMET package requires Python 3.8 or later, and on Python 3.7 an upgrade quietly stops at the old 1.1.2 release. Scoring the same WMT23 output with the same wmt22-comet-da checkpoint under both releases gave this 7:
| Direction | unbabel-comet 1.1.2 | unbabel-comet 2.2.2 |
|---|---|---|
| English to Chinese | 0.911 | 0.862 |
| Chinese to English | 0.795 | 0.803 |
Two separate reports in February 2024 found the same thing: 0.86 against 0.79 on identical files, and 0.8618 against 0.7982 on another identical set, both with wmt22-comet-da. The second report drew the obvious conclusion: a result needs both the COMET model and the software version beside it 9,10.
Fixes move scores too. In April 2025 a release changed how XCOMET was loaded, and some example scores shifted by up to 10%. The maintainer fixed it within a day by updating the model files and the loader. In September 2025 another user found their scores had changed silently because a locally cached checkpoint predated that update while the code did not 11. Earlier, the reference-free wmt20-comet-qe-da kept its name while the model behind it was retrained to fix a training error: the same inputs scored 0.2254 under version 1 and 0.2046 under version 2. The maintainer’s explanation was that the error “did not affect correlations but the scores were weird” 12. Rankings survived; absolute values did not.
The research literature has the same gap. Of 417 papers checked, 50 gave no specific COMET model, which the authors put as “at least 12% of papers report COMET scores without specific model information” 7. A score without its checkpoint and version is a measurement nobody can repeat, including you next quarter.
A score gap is a probability, not a size
COMET earned its reputation for ranking: two systems, the same segments, the same references, the same checkpoint. Kocmi, Zouhar, Federmann and Post asked the question that matters for that job. How big does the gap have to be before human judges would agree with the ranking? They fitted the answer on ToShip23, a set of human-judged system pairs that is, in their words, “a magnitude larger than any publicly available data” 13.
| Agreement with humans | COMET-22 gap | Same gap, 0 to 1 scale |
|---|---|---|
| 50% (coin toss) | 0.03 | 0.0003 |
| 60% | 0.18 | 0.0018 |
| 70% | 0.35 | 0.0035 |
| 80% | 0.56 | 0.0056 |
| 90% | 0.94 | 0.0094 |
| 95% | 1.53 | 0.0153 |
wmt22-comet-da on a 0 to 1 scale, so a gap of 0.56 points in the paper is 0.0056 in the numbers you see 13,5.Read the table the right way round. A gap does not say how much better one system is. It says how likely human judges are to rank the pair the same way. On the 0 to 1 scale, 0.851 against 0.850 is close to a coin toss, and 0.86 against 0.85 is just over 90%. The authors guard their own numbers too: “we do not claim that +0.56 Comet22 yields 80% accuracy for all scenarios but rather that it is as accurate as +2.34 BLEU” 13. Across all 6,530 system pairs, COMET-22 ranked the pair the way humans did 78.6% of the time.
Statistical significance does not replace this. The FAQ rightly tells you to run comet-compare for a significance test 3. But Kocmi and colleagues found that the gap between two systems stays mostly constant as the test set grows while the p-value keeps falling, and the Pitfalls paper sums it up: significance “can be made arbitrarily high” with a large enough test set, which “tells little about whether this difference is meaningful to a human reader” 13,7. A significant gap of 0.0005 is a confident coin toss.
Averages across languages break the same way. The Pitfalls authors note that “the score range is inherently distinct for each translation direction” and that “there is no assurance that the scores are on a linear scale”, so the same gain in two directions is not the same improvement. They suggest reporting the number of wins per direction instead of one averaged score 7.
A per-segment cutoff borrows a reliability the score does not have
Everything above is about systems: averages over thousands of segments. The most common production use is the opposite: one string, one score, one decision about whether a person needs to look at it. We have described that cutoff before as a dial that routes work and spends money. The question here is whether a COMET score can carry it.
Moghe and colleagues tested exactly that. They turned metrics into pass or fail classifiers for whether a machine-translated input would still work in a downstream task: question answering, dialogue state tracking and semantic parsing. Their finding: “all metrics exhibit negligible correlation with the extrinsic evaluation of downstream outcomes.” The best thresholds were “inconsistent across language pairs and tasks for both bounded and unbounded metrics”, and they put part of the blame on the scale itself: neural metric scores “are not interpretable, in large part due to having undefined ranges” 6.
Perrella and colleagues tuned thresholds for separating good translations from bad ones and found they moved when the dataset changed and differed by translation direction: “Such differences in the optimal thresholds might suggest that metric scores have different meanings depending on the translation direction.” They conclude that “the characteristics of the development set play an important role in determining appropriate thresholds” 14.
A segment cutoff is therefore not a property of COMET. It is a property of COMET plus your language pair, your content, your reviewers’ idea of good, and the checkpoint and version that produced the number. That is the general rule from our post on typed model output applied to one metric: a score is evidence for a policy, and the threshold follows measured error per locale and per model version.
The cheapest errors in meaning are the hardest to see
Some errors move the score less than they should. Amrhein and Sennrich used COMET to pick the best of many sampled translations and found it kept choosing candidates whose numbers or names differed from the source and reference. Their conclusion: “COMET models are not sensitive enough to discrepancies in numbers and named entities”, and retraining on extra synthetic data did not fully remove the weakness. In fairness, they also note an earlier shared-task test in which COMET mostly preferred a correct translation over a copy with swapped numbers when it had a reference to compare against 15.
DEMETR measures this with minimal pairs: a translation and a copy that differs in one deliberate way, scored side by side. The figure is the share of pairs where the metric scored the original higher. Fifty percent is a coin toss 16.
| One change to the translation | COMET, with reference | COMET-QE, no reference |
|---|---|---|
| A number replaced | 80.4% | 53.2% |
| A word replaced with its antonym | 97.0% | 83.7% |
| The sentence negated | 99.4% | 83.7% |
| A named entity replaced | 98.8% | 85.4% |
With a reference, COMET catches flipped meaning almost every time, and a changed number still slips through about one time in five. Without a reference, a changed number is a coin toss. The paper’s own example is a tennis report where “Murray lost the first set” became “Murray won the first set”, and the reference-free score did not prefer the original 16. These are the edits that change meaning most cheaply, the same ones a fuzzy match percentage rewards. And a minimal pair is the easiest test there is: the original and its corrupted twin, side by side. A production string has no twin.
Notice the pattern, because it is the whole post in small. Given two versions of the same sentence, COMET nearly always prefers the right one. Given one version and asked for a number, it scores an untranslated copy at 65.72 out of 100.
The case for trusting the number
The strongest case for trusting COMET is strong. Kocmi and colleagues at Microsoft evaluated metrics against 2.3 million sentence-level human judgments covering 4,380 systems. COMET ranked 83.4% of 3,344 system pairs the way humans did, rising to 96.5% on pairs where the human judgments themselves differed significantly; BLEU managed 74.6% and 88.2%. Their first recommendation: “Use a pretrained metric as the main automatic metric; we recommend COMET.” They also report: “We do not see any drawbacks of the metrics when investigating various languages or domains, especially, for methods pretrained on human judgement” 17. Three years later, the Metrics Maze paper still recommends a COMET model, CometKiwi, as the main metric 13.
That is real, and nothing in this post argues for going back to BLEU. But look at what was validated: pairwise comparisons of systems on a shared test set. The same paper’s conclusion opens with “We show that metrics can use a different scale for different languages, so Pearson’s correlation cannot be used” 17. The best evidence for COMET is evidence for the comparison. None of it validates a bare 0.85.
How to read a COMET number
Read a COMET number the way it was built to be read:
- Refuse a bare score. A COMET number needs its checkpoint, package version, language pair, test set and references beside it. The Pitfalls authors released a small tool, sacreCOMET, that writes that signature for you 7.
- Compare like with like: the same segments, the same references, the same checkpoint and version. Count wins per language pair instead of averaging scores across them.
- Turn the gap into agreement with humans using a published table such as the one in Kocmi et al., and treat a significance test as necessary, not sufficient 13.
- If you need a per-segment cutoff, fit it on a reviewed sample of your own content for each language pair, and fit it again when the checkpoint, the package or the content changes.
- Put cheap checks in front of the score: empty output, output language, an untranslated copy, numbers and names against the source, and a string metric such as chrF alongside. Placeholders, markup and plural structure are a different gate altogether.
When an engine or a vendor is sold on a COMET number, ask the same questions in the same order: which checkpoint, which version, which test set, compared with what. COMET is a balance: it tells you which of two things weighs more. One pan on its own is not a reading.
References
- 1.clairehua1 Interpretation and comparison of COMET scores across several languages Unbabel/COMET issue #110, 2023
- 2.Rei, Stewart, Farinha and Lavie, 2020 COMET: A Neural Framework for MT Evaluation arXiv:2009.09025
- 3.Unbabel Frequently Asked Questions COMET documentation
- 4.Unbabel Unbabel/wmt20-comet-da Hugging Face model card
- 5.Unbabel Unbabel/wmt22-comet-da Hugging Face model card
- 6.Moghe, Sherborne, Steedman and Birch, 2023 Extrinsic Evaluation of Machine Translation Metrics arXiv:2212.10297
- 7.Zouhar, Chen, Lam, Moghe and Haddow, 2024 Pitfalls and Outlooks in Using COMET WMT 2024
- 8.ZeroneBo if tgt is same with src, the score is still high Unbabel/COMET issue #192, 2023
- 9.PinzhenChen Different scores from different COMET package versions 1.1.2 and 2.2.1 Unbabel/COMET issue #203, 2024
- 10.bhaddow Different versions of COMET code give different scores with the same model and date Unbabel/COMET issue #204, 2024
- 11.de9uch1 Inconsistent scores of XCOMET between v2.2.4 and v2.2.5 Unbabel/COMET issue #244, 2025
- 12.thammegowda v1.x and v2.x have different scores for wmt20-comet-qe-da model Unbabel/COMET issue #143, 2023
- 13.Kocmi, Zouhar, Federmann and Post, 2024 Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies ACL 2024
- 14.Perrella, Proietti, Huguet Cabot, Barba and Navigli, 2024 Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics EMNLP 2024
- 15.Amrhein and Sennrich, 2022 Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET arXiv:2202.05148
- 16.Karpinska, Raj, Thai, Song, Gupta and Iyyer, 2022 DEMETR: Diagnosing Evaluation Metrics for Translation arXiv:2210.13746
- 17.Kocmi, Federmann, Grundkiewicz, Junczys-Dowmunt, Matsushita and Menezes, 2021 To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation arXiv:2107.10821