[VR] Next level hardware
Part 2: Smelling, hearing, sensing and moving
In the first part on next level VR we discussed graphics and displays. Both revolve around the theme of seeing. In this part we discuss the other senses.
Smelling and hearing
True immersion is not only about seeing, but also about smelling and hearing. Sadly, this is an underexposed topic. What if you are in VR concert and the sound is not good. What if you walk through a field of flowers and you don’t smell anything. This results in an incomplete immersion.
A lot of research for both still needs to be done, but there are already some prototypes. The Feelreal VR scent mask uses a cartridge with scents. Unfortunately this isn’t regarded safe by the FDA, so it didn’t reached the market yet:
There aren’t headphones that target specially for VR headsets. Such headphones must blend in with the headset, be light and the sound immersive. The best way you can do it is to go with a high-end headphone:
One thing important to VR is to have a sense where the sound comes from. For example when you are in a game you shouldbe able to recognize if the sound comes from left, right or from beyond yourself:
Again, Facebook Labs is researching this area. The research is about the sound around the hearer wearing a VR ( or AR) headset. The sounds can be enhanced or blocked:
Another option, which is maybe not so portable, but very lightweight and convenient is to not use headphones at all. Noveta Systems uses “Audio Beaming”. This gives the 3D sense of audio without needing headphones:
Sensing
Tracking
The visuals, scents and sounds are about how we view the virtual world. On the other side the headset must also find out how we move through the world. When we make a step, when we move our head, when we look at something. The first thing that increased this feeling of immersion was 6 degrees of freedom (6DoF). With help of sensors the headset tracks the body positions as forward/backward (surge), up/down (heave) and left/right (sway).
In the Ococulus Quest Facebook added experimental tracking of hands and fingers. This means you don’t need a controller for example to grap something in the virtual world. Or even use keyboard without a keyboard:
Kat Loco focuses not on hand tracking, but tries to do this with the feet:
The Chinese headset maker Pimax has extra modules for hand tracking, but also for eye tracking:
Especially eye-tracking is important for VR. The most important reason is that with eye-tracking the GPU needs to only render the parts in high resolution where you look at. This drops the GPU usage considerably (foveated rendering). There are also other reasons like increased security or more natural interactions. These reasons are stated in this article:
In a new protype of HP adds also heart and facetracking. Facetracking can for example be translated to your avatar so people see your facial expressions. The heart monitor gives extra information for fitness, science or business purposes.
Avatars
Often the virtual space is completely different than the physical room you are standing in. To translate facial expressions and movements to the virtual world, so that others can view you as a person in the virtual world, you need an avatar.
In its research labs Facebook tries to have a minimal hardware setup camera and setup to regenerate the body in VR:
Haptics
Haptics tries to give you a sense of touch or pressure of virtual objects. At the moment the Valve Index “knuckles” controllers give the best experience in the sense you forget there’s a controller in your hand.
There is a lot of research going on to give haptics through gloves or wrist gadgets.
Feeling with hands is the most obvious use case for VR. We feel not only with our hands ,but with our whole body. For this are whole suits developed:
Moving
Until now we talked about moving our body and the sense of touch. As a last topic we discuss movements like walking in a virtual world. Most headsets today use either a static way (for example Beat Saber or Playing tennis) or use teleporting technologies (for example Half Life: Alyx). These all are not really moving you body.
Of course you are mostly in a room with walland objects. Somehow, like with the display, there needs to be new kind of trick to let you believe you are actually moving. What trick works best and what trick is most practical?
Trick 1: Eye-tracking
The first trick is to let you walk in infinite circles within the boundaries you set up for your VR
Trick 2: Robot moving objects
This trick uses robots that move object around you. It could be combined with the eye-tracking trick. Then you can walk, but also feel real world objects like walls:
Trick 3: Treadmills
Like a rat walking forever in his treadmill, these are treadmills for humans. The treadmills were expensive until now, but new consumer products arrive on the market:
Trick 4: Boots
This trick let you wear boots that have some counter move. This let you walk forever like treadmills on one spot.
Trick 5: Exosuite
The last trick is to create a exosuite with haptics and hang that exoskeloton to a wall. So you are walking with this suite while actually hanging on a wall like a television.
Conclusion
It’s fascinating how creative startups and researchers are to find solutions for more immersive experiences. Before writing this blog, I didn’t know how much work is already done. Imagine how immersive it would be if a lot of these next level technologies come together.
