As many of you who already tried ARKit app might know, each ARKit app requires users to scan their environment to detect flat surfaces which ARKit can deploy its app on. In this article I want to talk my experience with scanning phase which is required when you start using any ARKit app.
An Basic Overview of ARKit’s Plane Detection
What ARKit does is analyzing the content of the scene and uses hit-testing methods to find real-world surfaces. Once ARKit is able to detect a flat surface in the image from the camera, it saves its position and size and the developer app can work with that data to put virtual objects into the scene.
I’ve been playing around with ARKit for quite some time as you can see from this blog. I’ve reviewed quite a lot of ARKit games and apps. I have to admit that the surface detection initialization process is very tiring and sometimes can be really annoying. In some occasions, if I would have needed to review the app, I would have probably uninstalled some of the apps after the first few tries.
I am using the iPad 9.7 2017 for my reviews because this is the only ARKit-compatible device I have right now. I am aware that the camera on this device isn’t that great. In fact, to get fast readouts of the scene, ARKit’s plane detection needs clear images and sometimes I do find that it’s hard to ARKit to locate a surface when I use it indoors. Outdoors on a clear day, it’s not a problem and it can detect the surfaces really fast.
Another thing that I’ve noticed is that even if ARKit detects a surface, it will still continue to monitor the surface over time and chance the plane anchor’s position even after it is placed to make sure it’s in the most accurate position. I’ve seen this in practice in many apps that allows you to deploy several 3D objects in the scene.You can clearly see that when you move, sometimes those objects move as well, even after they have been placed on a detected plane.
ARKit doesn’t use the advanced depth-sensing scene analysis technology that Tango uses. It can’t detect complex structures, just the geometry of flat surfaces. It doesn’t mean that the surface has to be completely flat without anything on it,
Now, when ARKit tried to continuously optimize surfaces for accurate positioning and scale, you might see jittering effect, where the objects seem to be moving slightly too many different directions over a short period of time.
As you can see in the video above, ARKit detects surfaces, but it doesn’t create a continuous connection between surfaces, like a large interconnected grid, although when you spend time scanning the area, there will be more surfaces that can overlay and make it look interconnected. I am not an ARKit developer, I just understand that from what I’ve seen in videos, what I’ve read on some online resourced and with my experience with games and apps that I’ve tried.
Now how all that affects the user experience?
The problem with this technology, in my opinion, is that it’s not super accurate. It relies not on IR beams that scan the area but on images from the camera. This means that the plane detection will work better on the iPhone X compared to my iPad because it has a camera with better low-light capabilities. What looks like a dark area, for some apps (like in Follow Me Dragon won’t necessarily be like that when played on the iPhone X.
This means that the user experience may vary based on the quality of the camera sensor, the lens aperture, the ISP and even the processing power of a specific iPhone or iPad model.
Another thing that I’ve noticed is that my iPad has a slightly narrower field of view compared to how I see the scene with my own eyes. This means that detecting surfaces in some indoor areas is harder in some cases just because I need sometimes to move backward so that the app can detect a surface I’m trying to play a game on. I can’t compare it to the iPhone because I don’t have an ARKit-powered iPhone device, but this was something that I could test while playing indoors and it was a bit problematic.
There is another thing. Some games require a larger surface to play on. I’ve seen some games that just won’t spawn unless I have a large area on which the game’s pre-defined size can be placed on. Some games allow you to resize the level, other don’t. This leads to a very inconsistent AR experience. I never know what surprises are waiting for me when I launch a game.
Ways to make the surface detection is less annoying
Just think about it for a second. When you launch a regular non-AR game, you just launch it and play it. You don’t need to think or worry about anything (Well, unless you have a very underpowered device and you try to play a highly-demanding 3D game on it), it just works. In ARKit it doesn’t just work, it can work well, very slowly, sometimes it can not work at all.
The dependency on the setting is something that will definitely cause issues with many users. It’s the responsibility of the developer to try to work with those limitations and make them as less annoying as possible.
Let me give you an example. I really don’t like the whole surface detection part, I think no one likes that. I’ve seen some developers trying to make it less painful by adding some sort of a fun interaction. For example, in one game, I can’t remember which one, the game told me to follow the butterfly with my device. It was a really nice butterfly and I enjoyed just following it with my iPad. What the game did behind the scene is scanning the scene for surfaces. This is a very smart idea and it’s encouraged to do so.
It’s much better than seeing a message that tells you to move the device slowly and seeing the app telling you that it scans for a surface.
Can this annoying surface detection feature be solved?
Let put some logic here. If there was an option, we wouldn’t even be talking about it. I assume that Apple prefers using this specific technology it uses because it requires only software implementation with specific hardware that many of its devices already have (A9 or faster processors). This means that developers will be motivated to develop apps for this platform because there are already tens of millions of users that can use ARKit straight away (after upgrading to iOS 11).
If Apple had developed a new hardware for its future phones, something similar to Tango, this means that much fewer people could have used it, which means smaller market and less motivation for developers to aim for that platform. We ‘ve already seen that Google had followed Apple’s steps with its own ARCore AR framework, which works similar to ARKit. As of the time of writing, ARCore is compatible with a limited selection of devices, including the Google Pixel, Pixel XL, Pixel 2 XL, Pixel 2 and Samsung Galaxy S8 (source: developers.google.com).
The reason Google prefer this solution is for the same reason as Apple, to make it more accessible rather than relying on forcing phone manufacturers to implement expensive new hardware that has implications on many other aspects of the device, like it’s size, internal design, etc.
As for now, developers will need to deal with those shortcomings and so do we, the users.
After spending a lot of time experimenting with many ARKit apps, I can say that there is certainly room for improvement, but the experience isn’t bad either. If you use the app in optimal conditions, you won’t even notice most of the issues that I’ve mentioned here. Even if you do, they won’t really hurt the experience too much that it will cause you to stop using the app. Of course, this varies between app and app, but just make sure that you are playing in a well-lit area and follow the guidelines for each app.
Another thing that I found really annoying that sometimes when the app loses focus, the apps requires a re-scan of the surface. It doesn’t remember the surface that you just scanned so when you launch the app, it asks you to re-scan it. Now, some apps keep the objects and their last positioning in the cache, but once the app gains focus you can see that ARKit start re-scanning for surfaces and those objects moves until the found a surface to be played on. Again, I am not an ARKit developer so I don’t know what is being done behind the scene, but some apps provide a good user experience, in others, it is just a catastrophe.
In time developers will optimize their app to make that initial surface detection process less painful and more fun and optimize their games to run better.