I’ve bee playing quite a lot of VR games in the past couples of years. When I start getting involved in AR, I was interested to know how what type of controllers and physical accessories will be used in AR to make user interaction with those AR experiences more exciting for users.
In this part 10 of “Designing Great AR Games: Tips & Suggestions” we’ll talk about game controllers and accessories. We see how we can benefit from them in our game with and without AR/MR glasses. We’ll look at what other companies are doing and how they tried to innovate in the field, the caveats of choosing this route and learn about some cheaper and affordable alternative to allow you to use accessories in your next AR game.
In VR like the HTC Vive and Oculus Rift, we have motion controllers, which give the user a direct control over either his own hand presentation or other virtual entities within the virtual space. Gear VR and Daydream also employ a hand controller which allow the user to both interact with the user interface of a certain app or game.
Having said that, having the physical connection enhances immersion and leads to a more realistic experience. When we interact with the real world, we do it using our senses: sight, hearing, smell, taste and, touch. This is how we are human beings perceive and interact with the world around us.
The more that particular XR experience is able to stimulate those senses in the virtual space, the more authentic and immersive the experience would be.
Augmented Reality and Mixed Reality have to follow the same paradigms. However, unlike VR, in AR and MR, users can see their own physical body and their view of the world is not completely obscured like in VR.
I’ve already seen various hand tracking technologies that were designed for AR and MR. These technologies were designed to allow users to natively interact with AR and MR interfaces and provide more natural control and manipulation of virtual content using hand gestures without the need for any external accessories. This is obviously a necessary accessibility requirement for many types of apps. Adding eye-tracking and voice control will further simplify the way we interact with virtual content in many types of AR and MR experiences.
Standard Gaming Controllers
Having said all that, when it comes to gaming, things take a different shift. Looks at the way people play games nowadays. The most natural way to interact with your in-game avatar whether playing in first or third-person perspective is using a controller or a mouse.
In mobile games, people use on-screen hand gestures, however, this type of interaction is useful for some games, while being very inaccurate and uncomfortable in many other types of games. This is why some FPS games employ automatic shooting, to lessen the amount of on-screen gesture interaction that the user needs to do. The main reason for that is that when you play a game on mobile, you usually play it with your thumbs, whether your other hands secure the hold of the mobile device itself. When using a physical gaming controller like the PS4 DualShock controller, players can use six fingers to control many types of buttons while still conforming to good ergonomics and comfortable gameplay experience.
Some companies have developed different accessories in order to make some mobile games like first-person shooters feel similar to playing them with standard controls by making detachable controls that mimic the feeling of gaming controllers. Nintendo did it with their Joycon controllers for its Nintendo Switch.
Physical game controllers are not going anywhere anytime soon and they actually do have many benefits, here are some of those.
- Accuracy and responsiveness – although hand recognition algorithms are becoming very accurate and responsive, controllers still more accurate, reliable and responsive, although this will change in the future as those technologies develop. Just think about a professional player that needs a super responsive reaction, right now nothing beats a keyboard/mouse or controller.
- Ergonomics – when playing a game with keyboard/mouse or controller, your hand can be in a comfortable relaxing state. For example, try to mimic holding a controller with your hands without a physical controller. You can see that your thumb muscles work harder. With a physical controller, the thumb rest on top of the cushioned controller joysticks.
- No need for Graphical UI overlay – when you use a controller, you don’t need to show the controls UI on the screen for the user to interact with. In a short come it comes as a second nature and the user can just hold the controller and focus on the actual game itself, rather than the graphical user interface
- Absorbing applied force – when playing games, we intend to sometimes apply extra force to buttons. For example, we push the button hard in order to ensure action or holding the controller very firmly in an intense combat action. The controller absorbs that force so that extra physical doesn’t hugely negatively affect usability. However, without a controller, this extra force would probably be channeled in a different way and be negatively affect the input accuracy and consistency if we use our hands for the same type of controls for example. Let alone that it can extra muscle work. The controller actually restraint that (sometimes) uncontrollable applied force.
- Haptic Feedback – provide a physical sense of touch by applying forces, vibrations or motions to the user. This mechanical stimulation can be used to physicalize virtual objects thus enhancing the touch communication channels that we are so familiar with in the real world, but what was missing in many VR interfaces and those solutions suppose to further lessen the gap between virtual and real.
Unlike standard controllers (which are also used in VR games as well), in VR, controllers were designed to match the medium, and this is why we have those motion controllers that are used to obtain accurate coordinates and orientation information, so the users’ hands can be faithfully represented in the virtual space.
We can also see some companies trying to innovate in this field and allow users to interface with AR and VR via wearable accessories like gloves for a more natural hands-free interaction even with haptic feedback (sensing glove). These are essentially wireless controllers that were designed as wearable hand machine interface for games and smart devices. These type of gloves are controllers for all intents and purposes but optimized for with AR and VR applications.
Playing using AR Glasses/Headset vs Mobile Device
Gaming accessories are obviously much more suitable when using AR glasses. The reason for this is simple, when you play AR games using your mobile device, your hands are occupied holding that device. Therefore there is no option to hold the second accessory in your hands. This is especially true when playing an AR game on the tablet compared to playing on a mobile phone.
If you look at the available AR accessories, most of them are AR guns where users can mount their phone onto the gun itself, thus releasing them from holding their phone.
Other experiences like Star Wars: Jedi Challenges rely on a proprietary AR headset to deliver its Lightsaber fighting experience.
If we look at the Merge Cube, you can see that this experience is best used with VR glasses or with mounting the device on the bundled plastic mobile stand. I think this is why the Merge Cube’s Achilles heel when it comes to AR.
All of that didn’t stop developers thinking of ways to bring unique AR experiences before AR glasses become widely spread. They are very much aware that this might take some time and until then, they can innovate in what they have right now, instead of waiting for tomorrow’s technology. In this time, they can get a good reputation in the AR field, gain experience, impress their investors and be ready for what’s coming up next.
As we get closer and closer to 2019, I think that developers might hold any adventures investments in AR accessories. MergeVR might be an exception because they build their business on Vr and AR accessories. Furthermore, in my opinion, when AR glasses come out, they will have no problem adapting to it as their experiences can be easily ported and play nicely with AR glasses as well.
Game Accessories For AR Games
AR accessories allow natural and comfortable interaction with your AR game, same as standard controllers work for console and PC console.
When thinking about your next AR game, you should certainly not rule out the use of gaming accessories. Imagine playing a first-person shooter game in AR, where you get out with your buddies outdoors and play against another group of people. How do you think a user would prefer playing it, with a gun accessory or holding his hand in a gun-holding posture?
Well, the answer can vary. For example, if a user plays an FPS game with AR glasses on, a gun controller will be a great choice. If the user plays it without AR glasses, obviously there is no place for an AR gun, unless it is mounted on top of the gun like MergeVR did with their AR/VR upcoming gun controller. Having said that, in general, a gun accessory will deliver a more immersive gameplay experience because it allows them to feel like they are holding a real gun.
In the future, we might see many different multi-purpose haptic gloves that will allow us to use them for different applications. For example, connected gloves that can be used to give you the sense like you are holding a gun, a ball or golf club depends on how they are configured and adjusted. The physical joint between the gloves is needed in order to assign different firm and secure holding properties that mimic hold of different type of physical objects.
You might get the impressions that physical controllers in games are mostly for first-person shooters. Although we can clearly see this popular in the FPS category, controllers are used in different other game genres.
One good example is the 2018 Auggie Awards’ winner for Best Game or Toy: Star Wars: Jedi Challenge.
This amazing AR experience includes a smartphone-powered Lenovo Mirage™ AR headset, a lightsaber controller, and tracking beacon. It allows users to get as close they can get to being a Jedi and fight the evil forces. Later on, The developer added 1-vs-1 local multiplayer lightsaber duels in the new “Versus mode”. There is no other technology that could bring you to that close to this type of sci-fi Star Wars physical encounter than Augmented Reality.
To make that experience possible, and due to the lack of consumer-ready and popular AR glasses, the developer had to release the experience with AR glasses as well. In the future, there won’t be a need for this, only for the physical lightsaber controller.
Ths physical Lightsaber controller helped to deliver an authentic and immersive AR experience. Having said that, when you develop your AR game, it doesn’t mean that you need to invest in expensive AR accessories.
The first reason is that we can expect to see many accessories that will become popular and be widely spread among mobile users. Who knows, maybe Apple will come up with its own unique accessory that you will be able to target your game for it.
Second, using other advantage technologies like Computer Vision, we can deliver AR experience using simple materials that can be found in almost every home. A great example of that is InstaSaber app. An ingenious app in my opinion. Its developer uses computer vision in order to attack a lightsaber effect to a folded piece of paper.
Just try to imagine using these type of advanced technologies to create your own sword fighting augmented reality game!
AR is evolving fast, and if you do a little research, you can see that there are some cheap and accessible DIY solutions to help you deliver AR experiences using cheap items. I personally don’t know why the developer of that app didn’t take it a step further and deliver a sword AR fighting game with vs AI and local multiplayer mode. This app could be so epic, especially at this time where this type of app can get you easily stand out from the rest.
You can even include your own cheap branded manufactured accessory made of paper or give short instructions how users can make them themselves at home.
I also wondered whether when using AR glasses, if a standard controller will be used. I think it will be the same as it is in room-scale VR, where players will benefit more from a dedicated controller for each hand to have a greater degree f hand freedom. This will help deliver a more natural interaction, especially for first-person based games, as well as offer more freedom in making it suitable for different type of interfaces. If there is a need for a standard controller, I think that companies can do what Nintendo has done, make an accessory where you can attack both controllers on each side.
No matter which way you choose, the goal is to make sure you deliver an entertaining and immersive AR experience that users will enjoy playing. If that accessory can help deliver a better experience, you might think of ways you can use it, even as an optional accessory to enhance the experience.
Physical gaming accessory controllers will continue to be a part of different gameplay experiences. It’s obviously not necessary for any app and right now most AR games don’t use or require any accessory controller to operate.
It’s also easier for a developer for an already existing widespread accessory using its SDK than the alternative, which means researching and making your own proprietary game accessor. Also if you have an accessory, users will still need to purchase it and that accessory needs to be delivered to them.
Looking at how many companies are into AR and VR game accessories, I have a good feeling that as AR continues to grow, we’ll get to see more gaming accessories coming into the market and some becoming widespread enough to convince developers to target their games for them.
The Merge Cube is a good example for that. A company who wanted to deliver a unique AR (and VR) experience and it did it with a very exciting accessory that helped deliver unique and exciting AR gameplay experiences like no other.
It doesn’t mean that the game must use that accessory, you can deliver a hybrid app that can work even without it. However, from what I can see, apps that are designed from the ground up for a particular accessory, turn to deliver more impressive results.
I predict that game accessories for AR and MR will become more common when AR glasses with good gaming capabilities are released, thus releasing our hands from holding a device when we play. Of course, there always be tons of great games that won’t need any accessories and some rely on hand gestures to deliver unique gameplay experiences. We’ve seen that in VR with spellcasting games and many other different types of games. If you want to get some inspiration for these type of games, you probably want to start looking at some VR games. I will write more about the good game design practices that developers can borrow from VR and can be beneficial for AR/MR as well.
I do hope that this article inspired you to think about if you can benefit from using accessories with your AR game or deliver a completely new experience based on a physical accessory.
If you find this guide useful, please don’t forget to share it with others. Thanks.