A couple of years ago -three and half years- we at SAECULII YK (Japan, Tokyo) ran an experiment on machine translation accuracy.

The experiment was fairly straight forward. We took a sentence from an English news article and ran it through the Babel Fish Translation Service for a Japanese translation. The article is As Kindle Fire Faces Critics, Remedies Are Promised with the sentence:

The Kindle Fire, Amazon’s heavily promoted tablet, is less than a blazing success with many of its early users.

You can read more about the experiment here Feedback has varied, ranging from interesting demonstration of the (poor) accuracy of machine translation to claims of unfairness for singling out one machine translation vendor, and the technology is still in its infancy so it’s to be expected.

Overwhelmingly, though, most wanted to know where machine translation accuracy stands today. Many folks reason that since the demand for machine translation -the global machine translation market is expected grow 23.19% to USD6.9 billion by 2019- is increasing, accuracy must also be increasing (otherwise there would be no demand, right?). These are all valid points.

We decided to re-run this experiment using Google Translate.

Here’s how the original experiment works, which is replicated below:

• English to Japanese translation
  Run the article sentence through Google’s machine translation software.
  
  
• Japanese-to-English translation
  Why do we need to reverse the translation? If you don’t understand Japanese, there’s no way to verify the accuracy of the translated text. Therefore, translating the outputted Japanese text back into English will give you an accurate indication of what native Japanese speakers see.
  
  

The verdict of our original experiment is gibberish. In the intervening three and half years, not much has changed in terms machine translation accuracy...

Machine translation vendors regularly make claims of 70, 80 and even 85% accuracy. I’m sure you can see from the result above that that is simply not the case. And, therefore, by extension the accuracy of translation methodologies, such as translation clouds, translation crowdsourcing and post-editing machine translation (PEMT), which are all underpinned by machine translation is also called into question.

Do you think there will be a significant improvement in accuracy by the time 2019 comes around? Let us know what you think in the comments section below!

(Oh, and we’ll keep you posted when we re-run our experiment…in 2019.)

About the Author
Ivan Vandermerwe is the CEO of SAECULII YK, the owner of Tokyo based Translation Agency Japan Visit SAECULII for the latest professional case studies, articles and news on Japanese Translation Services

Copyright (C) SAECULII YK. All rights reserved. Reproduction of this article is permitted with inclusion of the "About the Author" reference as is (including text links, japan-translators.saeculii.com/), and this copyright information. Articles may not be altered without written permission from SAECULII YK.