Mixed reality app for kids

Reimagining a Great Mixed Reality Children’s App

Augmented Reality (AR) and Mixed Reality (MR) offers plenty of opportunities for creators and developers to share their creative ideas. With supplementary technologies like computer vision and machine learning, developers can develop apps that are more engaging for children. This helps to open a path for a more fun and useful way to deliver entertainment and educational content.

In this article, I’ll take you on a short trip that starts with looking at some of the technologies we have now, look a bit into the future of Mixed Reality glasses and see how those technologies can be combined to deliver a great MR app that children can benefit from. This is just an app that I thought of, one that I’m quite sure will find its way into upcoming MR glasses, like Magic Leap One for example, sooner or later.

Magic Leap One was actually the first product that inspired and ignites my ideas. When I imagine future ideas for AR, I try not to limit myself to current technologies and look to what near-future technologies have to offer.

Overview

I am sick today, so I couldn’t really do much, only think. When I don’t “Do AR”, I usually let my mind go to imaginary places, trying to foresee how the future of MR would be like, and what type of experiences we might experience.

So there I was, lying on my bed with my lovely cat. I remembered the time when my niece came to visit me, and the first thing she wanted to do is to play the Merge Cube adventure game called 57° North. Seeing how my niece enjoyed this app, made me think about what other types of apps children may like. This generation is born into the mobile era, an era where video games are one of the leading forms of entertainment and serve as a vital part of creating engagement as part of a learning process.

I remember when I was a child. Back then, the only way to learn and play was using physical toys. Still, observing the market today and reading articles online leaves me with a feeling that physical toys still have their place in our world and will still have their place in the future.

So what are the benefits of having a child interact with a physical toy versus a digital content, like a game for example?

Physical toys give the kids an option to explore how the toy works. Playing a physical game with parents of friends leads to a conversation that helps improve oral language development. When playing with other kids, this also helps children learn and develop social skills. Physical toys can help improve fine and gross motor development as well.

Of course, video games do have their own benefits, but for those type of games to really benefit the child, developers will need to find the right balance between the physical and digital elements of the game, so a child can benefit from both.

A good example of such a product is the Merge Cube. The Merge Cube is actually a fantastic product for kids because it allows children to enjoy some of the benefits of both digital and physical. However, one of the disadvantages of the Merge Cube experience in Augmented Reality, is that it required either holding a device or viewing the content when a mobile device is placed in front of the cube.

The experience was far more convenient when used alongside the Merge Virtual Reality headset. With the headset, your hands are free and there is no need to have a device act as a window to the experience, the phone’s screen resides inside the headset and covers the user’s field of view, but the camera projects the real world scene, so it leads to a more seamless and immersive experience.

This leads me to think about what’s the best way to develop an interactive app for kids using current and future technologies. I surely will prefer making such an app for a mixed reality glasses, but I might also consider having some sort of a physical toy that compliments the experience in a way that can benefit the child who uses it in ways that I’ve described above.

Reimagining an Educational Kid’s App in Mixed Reality

Magic Leap introduction to its Magic Leap One MR headset made me think about the available possibilities. I wondered how mixed reality technology can be used to enhance the type of MR experiences for kids, but also still maintaining some of the benefits that traditional imaginative physical play brings?

When I start thinking about and comparing physical games to virtual games, I immediately thought about the sense of touch. This lead me to read more about haptic technologies. Haptic technologies can be used to give the feeling of physicality in some degree. However, the more I’ve read, the more I realized that we are quite far from having advanced haptic technologies widely available in the consumer market, let alone having them compatible with upcoming MR platforms and more specifically designed for kids.

Developers are already working in this direction (mixing physical and virtual)

To understand the possibilities, we need to look at what we already have now, and what we are going to have in the near future.

One of the great examples of using AR alongside standard physical toys is the upcoming Lego AR app. This app was introduced at an Apple WWDC. That app uses augmented reality, alongside 3D object detection and shared AR (new to ARKit 2.0) to deliver a hybrid AR multiplayer gameplay experience. The app can detect existing lego structures, and combine virtual content with physical toys to deliver a fun collaborative gameplay experience in AR.

The main problem is like in many other mobile AR apps,  that the hands of the user aren’t completely free for interaction, both with the virtual and the physical content. The reason I said “completely”, is because the user is expected to use one hand to control the physical Lego toy, but then it puts more physical stress on the other hand that holds the device. Definitely not an optimal way to enjoy such experience, especially for children.

The World Waits for Mixed Reality headsets

I’ve seen the latest stream by Magic Leap, where they talked about their upcoming Magic Leap One MR headset. Magic Leap is aiming its MR headset for indoor used at first. It doesn’t mean that developers won’t develop apps that can be used outdoors, but at the first iteration, I think that it will be smart to develop apps that can be beneficial for indoor use. I do wonder how convenient this headset would be for a child to wear though. Still, I imagine it will be better than holding a mobile device for a long period of time. Furthermore, with hand movement and voice recognition, this headset opens up a whole new world of creative possibilities for apps.

In Mixed Reality, the virtual content can not just appear in your world, but also interact with it. This can be done using spatial mapping of the environment alongside advanced Computer Vision (CV) technologies. Developers will be able to trigger certain events based on the physical structure of the real world environment and objects within it.

One of these technologies is 3D object detection that was introduced in Apple’s ARKit 2.0 AR platform. This technology was used to accurately detect the physical lego structures in the Lego AR app that I’ve mentioned above. It’s used n AR, but will definitely be used in upcoming MR glasses as well.

3D object detection can be used to detect not just pre-scanned objects but many types of objects in the wild and with Deep Learning (DL), this will only get better and better.  The object detection algorithm will be first pre-trained with different deep learning models. This will help with automating labeling of indoor environments and as the user continues to use the app, the more accurate it will become.  This combination of appropriate environment representation, alongside object detection and Deep Learning, is used in robotic systems but can benefit virtual characters as well.

In my imaginary Mixed Reality app for children, a virtual character plays a significant role in the experience.

Enhancing Narrative Potential

The ability for a virtual character to respond and interact not just with the user, but with the environment significantly enhance the narrative potential. In other words, it helps create an immersive environment that invites user engagement.

By utilizing those advanced technologies, developers can create smart virtual characters with AI-enabled features. I imagine that like the movie A.I. Artificial Intelligence by Steven Spielberg, but instead of a physical robot, it’s going to be a virtual character.

Children will be able to have this friendly virtual friend that will continue to learn about them and their environment over time. That character can interact and react to different objects in the place where the child is using the app. It can ask questions and get answers using voice recognition. It can pull out cloud-based “activities” to interact with the user. Those “activities” can be games, quizzes and other educational and entertaining content tailored to the specific user based on Machine Learning (ML). The more the user interacts with the app, the more accurate it becomes.

Having a virtual character can lead to better engagement and helps create a good educational platform for the child to interact with.

Let me try to give you an example of how such app might work.

Imagine your kid sitting down on the floor, launching a mixed reality app. A cute virtual character appears it introduces itself to the user. A moment later, it asks the user to turn on the lights if the scene is dark. The character looks at the user and tells him: “That’s a nice a nice room, I like the blue carpet. I see you have a soccer ball, we might play sometime.”. A moment later, the character brings his own ball and asks the user if he or she wants to play a soccer game with him. This game is a pre-designed activity that was pulled out from the cloud based on the analysis of the environment.

A day later the kid meets his virtual friend again, and this time the virtual character surprises the kid by bringing him a cake for his birthday and a special interactive game of throwing cakes on each other.

The next day, the kid login and asks the virtual character for help with math, because he has a test next week. The VC brings up a math exercise app that matches this current math skill.

This is just a very simple example, but obviously, the possibilities are endless. I can definitely see some developers taking this approach and designing an app that will do just that. This also opens a place for adding physical toys that can further enhance the experience You might go to the toy store, and find a range of physical toys for that specific branded app.

This can be a really amazing experience for kids. When they come home and put their Mixed Reality glasses on, they know that that virtual character will remember them. Not just that, it can even remind them of things they should do, brings things from the past, and give personal suggestions. It’s like having a best friend, but one that can understand the child’s needs and can deliver personalized content that can be beneficial across different verticals.

Now, if you had multi-user activities,  and having your hands free for interaction with the virtual or complementary physical accessories, this becomes even more exciting.

Summary

I am very excited about the future of AR/MR. I am excited about seeing how the new generation adopts it, especially children. When I saw Magic Leap headset, I couldn’t ignore all the advantages it brings with it.

I am interested in children apps because I see the great potential of such device as a very smart educational tool when used alongside other technologies.

I see children encouraged to learn more using it and parents trusting certain apps to make the right choices for their children. To make such choices, an app should be there and grow with the child. This is why I imagined a virtual friend that the child would want to interact with on a daily basis. One that emotional bond is done, it paves a clear path for the other content to be delivered with hopefully a desire to engage with it.

I hope that this little story inspired some of you to deliver an app like this or something similar. I haven’t touched a lot the topic of education in AR. The last post I wrote was in March about new educational AR apps for iPhone and iPad.

The more I learn about current technologies, the more I see the importance of utilizing them in a meaningful way, especially when trying to educate the younger generation. We have a great opportunity here to make something exciting and different, and this what brought me to share this little story.