Google Translate fares better on long sentences, Naver Papago better on slang

Posted on : 2017-02-13 17:12 KST Modified on : 2019-10-19 20:29 KST
Two online translation services have both shown significant improvement, and may take over rough draft phase of professional translation
Naver Translate
Naver Translate

The two leading artificial neural network translation services operating for Korean today are Google’s Google Translate and Naver’s Papago. With the help of a professional translator with seven years of career experience, the Hankyoreh attempted a brief assessment of the quality of their work. The verdict: Naver performed slightly better at translating new Korean coinages, while Google showed less of a decline in quality when translating longer sentences. Both services performed well on short sentence translation.

The Hankyoreh and translator Kim Jeong-gyu assigned three types of tasks: English-to-Korean translation, Korean-to-English translation, and coinages. It then ran a total of five randomly chosen words or sentences through Google Translate and Papago for each type, or 15 in total. We asked Kim how many points he would assign the results, where 100 represented the equivalent of a human professional’s work. Google came out slightly ahead with an average of 76.6 points to Papago’s 71.4.

The two programs were almost indistinguishable in their excellence with shorter sentences. While Google produced more natural-sounding results for some sentences and Papago for others, both were generally satisfactory. When given the sentence “Trump’s business advisory council is led by Stephen Schwarzman, chairman and chief executive of the private equity group Blackstone” to translate into Korean, the resulting Google translation received 95 points, rendering Schwarzman as “Syuwa Seuman” and Blackstone as “Black Stone.” Kim was particularly astonished at the programs’ care in placing the original English in parentheses alongside the translated versions of proper nouns like “Blackstone” and “Schwarzman.”

But for longer sentences, the programs still lost points for accuracy compared to humans.

“This was something more evident in the Naver engine,” Kim said.

In response, Naver explained that Papago is “still a beta service” and that “these [inaccuracies] can happen because we offer traditional statistical services rather than artificial neural network ones once it exceeds 200 characters.”

Papago appeared slightly stronger when it came to Korea‘s unique coinages, slang, and abbreviations from online and daily life. It translated the term “meokbang” as “food-eating broadcasts,” a more accurate rendering than Google’s “food.” But the term “gillyangi,” referring to a stray cat, was merely romanized by Papago and translated as “hunting” by Google; “geukhyeom,” meaning “extreme hatred,” was also romanized by Papago and rendered incorrectly as “superstition” by Google.

“Compared to past translation services, it’s astonishing,” said Kim.

“Professional translation duties involve an additional rough translation, an examination, and editing,” he added. “This is enough that you can imagine artificial intelligence taking over the initial rough draft work.”

By Kwon O-sung, staff reporter

Please direct questions or comments to [english@hani.co.kr]

 

button that move to original korean article (클릭시 원문으로 이동하는 버튼)

Related stories

Most viewed articles