Mixed Reality responsive design

Mixed Reality Responsive Design Principles

Note: this is a draft (1.0) for developing responsive mixed reality experiences. It is incomplete, a work in progress and subject to change. When we develop apps and games to mixed reality, we are no longer developing for pre-defined boundaries (e.g. web page) but for unknown space with unknown size and structure. Although some apps will be developed in a fixed size with adjustable size functionality, many other apps would need to be designed to be adaptable to unlimited varied types of spaces in order to make the experience work well for the MR medium. By saying “well” I mean, taking advantage of the available space to deliver more immersive, exciting, engaging and visually impressive experiences that are fundamentally different than anything users have experienced before.

I write this article with game development in mind, but it still valuable for any type of app development, just keep that in mind.

Overview

I mentioned this topic when I wrote my first guide to augmented reality. Even in augmented reality, I really wanted to have the opportunity to play a game that takes advantage of the available pre-scanned surfaces. Well, some games did it, in a way that the game was designed from the ground up to take advantage of any sized game area. For example, in ARrrrgh, you are able to hide the chest on any scanned surface and let the user search for it. With AR Toys you are able to position game elements anywhere in the scanned surfaces and drive a car through it. In Lego AR Studio I could build a long lego train track, and in ARZombie I could define the play areas size to match the size of my room by placing four objects that defined each side of the rectangular gameplay area.

The goal was to make good use of the available space to deliver an exciting AR experience that feels open without pre-defined boundaries, or at least, take use as the available space as much as possible. Make the experience really feel that it is part of the real world, and not like a “boxed” self-contained one.

Now in mixed reality, like when developing for the Magic Leap One, things are quite similar, but now instead of just surfaces, the platform is able to detect and mesh the entire space (with some limitations of course). This includes everything that is still and not moving at a very fast pace, including walls, ceiling, any objects in the room and even a person who stand still in the space. That depth scan is transformed into a 3D mesh which is available for your application for collision, occlusion, and positioning of virtual objects in most part.

Keep in mind that because the device is used in many different places, each user is going to have a different mesh for the app to work with. If it’s a game, you can look at it as the landscape of a level in the game, and it’s will be completely different from user to user. One way to make sure a game can be played in any place just makes it self-contained and work in a pre-defined area.  For example, a 2x2m square space.

However, we are developing for mixed reality, and this is an opportunity to make something unique and take advantage of the spatial computing technology to deliver unique experiences that are unlike anything else.  I’ve seen that done in augmented reality many times, and those experiences were very poor. Now, of course, you can develop a board game and port it to mixed reality, nobody will prevent you from doing so, ins some cases, this might be the only way to port that experience and bring it in this faithful representation to MR. However, if you want to make the switch and develop an experience that will look and feel great on this medium, you need to think outside the box and different than what you are used to as with standard screen-bound games.

Mixed Reality Responsive Design

In my opinion, the most important thing when designing an experience for mixed reality is going to it with the right mindset, with an in-depth understanding of the platform’s core features and functionality.  Even when thinking of ideas, you’ll find out that many new ideas pop out when you start thinking with that specific mindset.

For example, yesterday I thought about two ideas for games and I shared them on my blog: Tactical CQC and Spatial RTS.  The design of those game ideas is based on the core understanding of the Magic Leap One’s spatial platform and its features. I spent time carefully reading the official documentation in order to understand how it works. With that knowledge, start coming ideas. If you read my articles on those two ideas, you can see that they are designed to take advantage of any type of space in both size and structure. But it’s designed for outdoor use due to technology restrictions that the Magic Leap One is bound to.

Why Even Bother With Responsive Design?

Before I jump into talking about responsive design in MR in-depth, you probably want to know why even bother with this type of design and why we want to allow the app to be used in large spaces in general?

  • Allow users to use the app in small spaces as well as large spaces without negatively impact the experience or making it unusable
  • Gives a virtual content better sense of presence, like it is aware of the surroundings
  • Allow depth cues to be seen more efficiently, leading to a better sense of depth and virtual object location in the real world space, especially considering the lack of RT shadows in the Magic Leap spatial platform.
  • The ability to use object and spaces for occlusion (e.g. hiding places for characters)
  • More space for the virtual object to travel on. For example, imagine a car racing game on a very small 1x1m than one that spans the entire room. You can enjoy accelerating, drifting in a much better way.
  • Encourage user movement and exploration of the environment, which leas to different play opportunities beyond just sated positioning, including standing and moving in space.
  • Option to deliver a large-scale experience, giving users a feeling of being part of something bigger than them
  • Making the user take part of an experience in first-person, not just being an observer onto a gameplay stage but actually take an actual part and be the actual playable character in the game (similar to VR). You are no longer sitting in front of the monitor, you are inside the game itself!

These are a few quick examples to show you why we want to make good use of the available space and why having a responsive design, making the app make use of that space, is so important.

Mixed Reality Responsive Design Core Design Principles (Draft)

Now let’s take a dive into the real deal and see how we can deliver great mixed reality experiences by utilizing a responsive design in our apps and games.

The Realistic Bowling Responsiveness Game Dilemma

I’ve played many AR games one of the biggest mistakes I saw people making is trying to make the game fit into different play areas by enlarging the entire game area. this had very bad implications. For example, reducing the game size made the character appear very small, especially long vertical game like a bowling game. Sometimes making the game larger didn’t allow you to comfortably interact with the characters, blocked your view of other gameplay elements, etc.

Bowling game in AR
For illustration purpose only. In reality, it was way worse because of the bowling game was way longer.

The bowling AR game was a good example and one that let me think about responsive design for AR games. So the thing was like that. You are able to spawn along realistic bowling track in your surroundings. It was fine when playing it in a large outdoor area. However, when I tried it in my house, the game just overlapped the table and everything in the front and it ruing the AR illusion, now it was just a huge track floating on everything in my room.

The good news is that I was able to resize the track. But that’s the thing when you resize the track, you resize the entire game, which means that now the balls and every object in the game becomes super tiny. Instead of a realistic looking bowling track, I see a thin line and I couldn’t even locate where the ball is.

To make that game work well for multiple environment sizes, the developer could either crop the end and move the camera alongside the track, but this is a big no-no in AR. The other option was to make (same as a breakpoint) like three different sized tracks and let the user choose which one to use. A small track, medium track or a realistic one. A third option was to make the track responsive. The part where the ball is remains the same size (like a header of a webpage), the platform on which the ball travels will be responsive and the other side where the pins are will be sized based on the distance from the player. You can then give the user an option to use a slider to choose the size.

If it was in mixed reality, we can automate the process and let the app decide of the default size based on the current location of the place in space and the scene around him or her. We can then allow the user to manually adjust it as well.

Of course, we can play with the setup and remove optional objects, like the table where the balls are positioned, remove it and just put the ball in the center of the screen.

When designing a responsive game in mixed reality, you want to have the same mindset. Try to think about what type of game objects should retain their original size and what others can be responsive in design. Obviously, if you play a blowing game on the magic leap one and want to maintain a realistic experience in a small room, you have a problem. You can make some objects ignore the mesh but then you lose immersion.

Suggestions (Draft)

I have a few suggestions on how to tackle some of the design issues of designing MR games with responsiveness in mind.

It obviously depends on the specific app/game you are developing but in general, you’ll probably have these type of entities that you’ll need to define their responsive behavior:

  • Free – A Free object (can be placed anywhere in the scene, can be a character in the game or a decoration element)
  • Relative – Objects relative to other objects (or the player) – objects that should remain a relative (not fixed)  position to other objects like
  • Fixed – Objects that should maintain a fixed distance from other objects (or player) or in a specific point in space
  • Stationary – Object that needs to maintain its exact position in space and never mode.
  • Container – does the game needs to be played in a pre-defined container or can it be played on any type of open space without strict boundaries. Can be a dummy object, something that isn’t displayed in the game but defined boundaries.

For every object you should ask the following questions:

  • Can it be replicated
  • Can it be resized (does it need to maintain its original size)
  • Can it be stretched (keeping original dimensions but attached to closeby objects to create a seamless continuous formation of the same object)
  • Does it support continuity (e.g. a road that can be continued to be drawn until a certain endpoint)
  • Is it “fluid” (expand or positioned over any surface)

How do any of these changes affect the gameplay?

It’s not just about Visuals

It’s important to understand that making a game design responsive isn’t just about visuals and location. For example, to make the game adapt to different types of spaces, we might consider adjusting the behavior of different characters in the game, for example: increasing their movement speed to maintain good experience in a large environment, making characters jump higher in the Y-axis, adding more enemies  to accommodate for the largest space so the game won’t appear empty and lifeless, etc.

These type of consideration are also part of the responsive design that should be taken into account so the game can maintain a great gameplay experience as it scales up or down in size and adapts itself to the playable environment.

Responsive Design by Example

Now let’s take some AR games and see how they can be fit into this responsive XR design paradigm.

CastleMustBeMine AR game:

Tower defense augmented reality mobile video game

Can it be expanded or replicated (duplications), can it be increased in size or not, is it fluid (can be span over different surfaces and objects),

A Tower Defense game in augmented reality. How we bring this type of game to mixed reality in a responsive design?

  • Castles – Relative (to the path)
    Can be resized – Yes (to some degree)
    Can be stretched – No
    Support continuity – No
    Fluid – Partially (can be positioned on any surface, but we can limit it just for horizontal surfaces)
  • Road – Free (can be positioned anywhere in the environment
    Can be replicated – No
    Can be resized – Yes (to some degree)
    Can be stretched – No
    Support continuity – Yes
    Is it fluid – Yes
  • Canons – Fixed (a fixed position from a tower, on top of the tower)
    Can be replicated – No
    Can be resized – Yes (to some degree,)
    Can be stretched – No
    Support continuity – No
    Is it fluid – No
  • Rocks – Free (use for decoration, except the cave which the characters are coming out from)
    Can be replicated – Yes
    Can be resized – Yes
    Can be stretched – No
    Support continuity – No
    Is it fluid – Yes

Based on our answers we can design the responsiveness of the object. For castles, we can resize them relative to the game size (we can maintain the original size of course). We can also create the game with the castles being located anywhere in space as they are categorized under “Fluid”. The road support continuity, so we can actually create a procedurally generated road throughout space. I personally might increase the speed of the characters based on a size of the play area, so it won’t take minutes until the characters reach the end of the road when playing in a large room. The rocks we can position anywhere in the room around the road or whenever we like, just make sure they don’t overlay other game objects. We can replicate them many times because they really just used for decoration purposes, nothing more.

The amount of which we stretch, replicate, and resize depends on the game design decision that we make and can be used as part of the algorithms we use to increase difficulty.

So we took a fixed size AR game and shown that we can design it so it can be played in a responsive design in mixed reality. Of course, there are many more design considerations, but this is a basic start and a way to demonstrate those design principles that I’ve written above.

Furthermore, if you add a level editor to your game, you can also create tools that conform to those responsive design practices.

Automatic vs Manual Responsiveness

When designing a responsive mixed reality app, you can make it so the level is created automatically while taking the available 3D mesh into account or/and give users the option to manually adjust or arrange things themselves.

What degree of control you give users depends on your specific type of application. Hainvg an automatic system will definitely require some hard coding work behind the scene but even in manual mode, you still need to apply some automatic monitoring of the manual adjustments that users are doing.

I think it’s a good practice to apply certain responsive design behaviors for each object that will define its responsive behavior and put limits on its “responsiveness behavior”. For example, how much an object can be stretched, enlarged, its max speed, etc.

I have no experience in procedurally generation level design, but I guess in some cases you might want to at least get familiar with this type of level design approach.

Another important thing to understand is that the responsiveness isn’t just something that applied before the game begins (in the level layer) but can apply to object behavior during the game itself. Object themselves can he dynamic responsive behavior.

I will talk more about this in Draft 2.0. You’ll also see how easy it is to come up with so many new ideas when you just start thinking about responsiveness in your apps.

To be continued

this is just a start (draft) and I definitely see many things change as I continue to explore the responsive design options.

I think, and I mentioned this earlier, that it’s very important to get into the app design with that new perspective and understanding. All the game ideas that I was thinking about started with that type of responsive design understanding. Trying to design a game that has no boundaries and can be played and adapt to any type of environment.

Example, my CQC game idea. A user scans the room, story and character’s location and behavior adapt to the available space. You scan two rooms, the hostage might be in the second room and not the first one, an enemy can hide behind a couch.

When we add responsiveness implementation, it’s important that we can keep a good track on how the change affects the experience, progression, visual integrity and so on. This can make responsive apps more complex to make compared to those who aren’t responsive. Making a wrong decision might lead to a bad experience or completely break it.

I will spend more time on this topic in the upcoming week. There are a lot of things to discuss and I need to spend a lot of time digesting all the information, all the loose corners, and so on. If you have things to add, by all means, feel free to do so. We are all exploring those uncharted territories and the goal is to help developers to create amazing mixed reality apps, one way or another. Thank you for reading.

Credit: Cabbibo for raising awareness on this topic.