AR language translation

AR Live Language Translation using Google Translate App

Today I went to the local store to buy something to eat. The problem was that many of the products have Japanese text and I don’t read and understand Japanese.

I was actually looking for a peanut sauce for the salad. When I reached the area where all the sauces were, I knew that there is no way I’ll be able to find that sauce if it existed at all. I then realized that I can use the Google Translate app on my Android. I have a live camera translation in AR. Which means that you see the translation on the exact location where the original Japanese text it. Now just that, it even tried the best to match the original text and background color and does that surprisingly well.

For example, if the original text is yellow on a black background, it will show the translated text in yellow on a black background. The overlay in augmented reality really makes a big difference, because you don’t get a mishmash of many words both in Japanese and English (or other languages depends which languages you are using).

The translation is of course far from being perfect. Now just that, not every text can be translated well from one language to another, in this case, Japanese to English, but also it continuously changes the text and it’s hard to pick up a clear fixed translation.

Having said that, I was able to pick up the words that do tell me about the product, and I was able to decide whether this is the product I was looking for or not. If I didn’t get a translation, I would turn the product and looked at the ingredients, trying to get an idea of what this product is.

Now, of course, Google Translate live AR camera feature can be used for translating any type of text like letters, signs, translation of videogames in a foreign language (e.g. if you want to play a Japanese game and don’t know Japanese, translate restaurant menus, etc.

Of course, this feature will get better and more accurate in time, but even now it’s better than not having any translation tool at all. You can ask people around you to help you out or use text translation which can explain what you are searching for. I actually did it today when I didn’t find out what I want and I didn’t want to use the AR translation camera app on all the products. So eventually I just came to the salesperson and he told me that the sauce I was looking for doesn’t exist, LOL.

I definitely think that a translation app in augmented reality is a must-have tool, but it has to improve. I just can’t imagine using this with AR glasses and instead of putting the carrying basket every time I need to translate a product, I can just look at the product with one hand and see a translation live on the headset’s display in augmented reality (AR).

just keep in mind that you can also capture an image and let Google show translations. It actually segments the words that it was able to recognize. You can then tap on each detect word/phrase to get a translation or just go with select all and get a translation of all the words/phrases that Google was able to recognize.

Of course, I prefer the live language translation an I liked how it augmented the translation beautifully on the product with the same text and background color. In some cases, it looked very authentic, like I was holding a product with English text and not in Japanese, but most of the time it was a bit chaotic seeing all that live translation happening in front of you. The other thing is that the text doesn’t stick or get fixed, and it’s continuously being updated in most part and it sometimes moves, the text font get larger, some text appears outside of the product regions, etc. I mean, you don’t get that seamless, clean translation that you are familiar with when using Google Translate Web black on white.

I would personally prefer that eventually, after maybe a few seconds, Google will decide on a final translation (maybe I should give a time limit) and just show a static final translation so I can read it clearly, instead of making changes to some of the text every few seconds or so. I assume this is done because I was moving the product. So even a slight move can cause a rescanning of the text, but I am not 100% sure about that.

Anyways, I wanted to share my experience with you and I actually do recomnebd tryig it out if you are a tourist in a foreign country with no understanding of the local language and need to do some translation in real time for any reason.

In the video below I review this live translation camera that augmented the translation on top of the original text and talk about my experience with it as a tourist in a foreign country (Japan) who enters a store and don’t understand the local language.