One topic that I was interested in learning more about is how the human eye is focusing on the virtual content that is projected on the ultra-thin near-eye displays (NED) in the Magic Leap One glasses. In other words, I try to understand how the Magic Leap One enables multiple plane focusing. How the headset can add that depth cue information to the light field to enable the brain to focus on virtual object in different differences like it usually does with real object in the natural environment.
I am on a journey to try to understand this advanced topic. Doing my first baby steps in that field. It’s very complicated, but I want to get some basic knowledge so I won’t be left in the dark. I know I’ll make many mistakes and might misunderstand ro interpret some of the information, but with your help and the web I might get more educated on this subject.
This is by no means a reference source, only logging my journey in trying to understand the topic of how the Magic Leap One works to tackle that focusing issue, allowing the brain to get depth cues so it can focus on objects located in different locations from the headset/user.
With virtual reality, there might be a vergence and accommodation mismatch due to the eyes focusing on a fixed physical focal plane (the screen), and therefore the user cannot focus on certain distances which can result in discomfort and other issues.
The Magic Leap One uses a different technology, utilizing ultra-thin transparent displays and transmit the virtual content light naturally to your eyes by combining it with the natural light coming from the environment.
I was interested to know how does the technology is capable of sending depth cues to the eyes in a natural form so the brain correctly interprets the distance of the object from the headset. I’ve heard that in arm’s touch length, a distance where we can actually touch things, this is where the accommodation cues, focus cues are most important.
The first suggested option I came across (seems outdated)
Using multiplicative two-layer modulation, there is an option to generate multiple different perspectives of the same 3D scene and those enter the pupils and different positions (source 10:50. I highly recommend watching the entire video). That video talks about technology for VR, but that type of reconstruction method can be used to mimic the process of the eye receiving the natural light field data that we get normally from observing the natural world, and bring it so it works with digitally projected content that appears on the displays. It’s creating a needed light field data that gives the needed depth cues for the eye to be able to focus on virtual objects in a natural form.
At 11:25 of that video, you can see that the system renders many different views of the scene, which then can be used to supply the light field with more perspectives (not just a subset as with a single layer display). It then projects a view of the virtual scene in different focus points. For example, the front panel represents front focus and the rear panel uses rear focus. The light field contains all the needed information that the brain needs to make that natural shift of focus between the two planes. The user can also have mid-focus within the light that exists between the two planes.
The goal is top to provide users with a more natural and realistic viewing experience and mitigate vergence and accommodation conflicts.
Vergence – the simultaneous movement of the pupils of the eyes toward or away from one another during focusing.
Accommodation (eye) – the process by which the vertebrate eye changes optical power to maintain a clear image or focus on an object as its distance varies.
Seems to be the system used in the ML1
This article on kguttag.com that was published on January 3, 2018, reveals an image showing a patent application the can shed some more light on how the magic Leap was able to achieve that same (or similar) effect in their ML1 headset.
Again, I am not an expert nor I’m a qualified person for this type of advanced topic, but I do my best to try to understand how things work with the little knowledge I get from various sources.
In the image, we can see several out-coupling optical elements that are stacked parallel to each other. Having said that, I also came across this new article on the same website that shows another patent scheme (Fig. 20). There you can see 6-layer diffractive waveguide. You can also see the structure of the layers with the world side cover window, eye side cover window and the 6 players between them. Each layer corresponds to different wavelengths (Red, Green. Blue) with each color, from my understanding, create light field data from either 1 meter or 3-meter focal depths. Which should allow the eye to focus on the subject at two different focal lengths, either close or far distance.
I guess this is the reason why this technology is designed for indoor use. To make it work for outdoor use, there might be a need for a technology that can generate light fields for many more focal depths, definitely more than 3 meters.
If I understand this correctly, having objects closer or farther away can lead to vergence and accommodation conflicts as the brain tries to adapt to conflicting cues derives from an unnatural simulation of vergence and accommodation of the display.
I have a long way to go until I am able to get a hold on how the technology works. I will continue to update this article once I get more information, so stay tuned.
Currently reading: Vergence-accommodation conflict on microsoft.com.