Immersive Tech Hardware

This is an opinionated tech hardware primer by Neil McDonnell.

  • Last updated October 2018

Virtual Reality (VR)

Virtual Reality tech has blossomed since 2012 thanks to a catalysing breakthrough from Oculus (who Facebook promptly bought). VR intervenes on your senses (via eyes and ears mainly, but not exclusively) to immerse you in a virtual environment. The sense of presence in the virtual environment that this can inspire in users is staggering, even with cartoonish or otherwise unreal environments.

Here are three key elements to a good experience in VR:

1. High refresh rate screens. With sub 60hz screens, simply looking around your virtual environment could cause quite significant nausea as there was a subtle delay between your head movement, and the world catching up. At 90hz or above, this is mostly eliminated.

2. Let the user move, but not too fast/suddenly. This is controversial for some, but I believe that the manner of movement in a VR experience is one of the biggest factors in a good experience. There is an immediate boost to immersion if people can take a step in VR by taking a real-world step. This grants the user 6dof (six degrees of freedom) in the virtual world, just as they have in the real world. That said, when movement in the virtual and real world are de-coupled, as in when you sit still in the real world, but on a virtual roller coaster in VR, nausea is almost guaranteed.

3. Interaction. If the user cannot engage with the environment, interacting with objects and such like, then the real powers of VR to immerse us somewhere else, are not being deployed.

All three depend on both the software and the hardware. For example, I know of no headset on the market that does not accomplish 1 for at least some apps, but problems arise when the headset, or the computer running it, are underpowered relative to the software being run. For movement (2), the interaction (3) the hardware needs to be capable of 6dof and have a controller, respectively, and the software needs to make the most of them.

The hardware splits along five broad types:

3dof. The cheapest and least capable devices are those which allow the user to look around but not move or (typically) interact. The most common such systems are the ‘phone-in-a-box’ systems such as the google carboard, Samsung gear, and equivalents. These harness the power of existing smartphones to give a basic VR experience. The Google Daydream is probably the best of the pack, with a comfortable, high quality ‘box’ for a compatible phone, and a good ‘wand’-style controller. (Cost: phone + £10 - £150)

Outside-in tethered. Systems which require a separate PC to run them currently deliver vastly higher quality experiences than the alternative ‘standalone’ variety. Of the tethered sort, the Oculus Rift was the first announced (though not the first to launch!). This system uses either 2 or 3 sensors attached to the PC to ‘watch’ the movement of the headset and (fantastic) controllers and co-ordinate that real movement with virtual movement. Since the sensor placement is critical to the veracity of the movement, this can be a somewhat fragile set-up requiring careful calibration. (Cost: approx. £350 plus gaming PC)

Inside-out tethered. The second sort of tethered system is that which dispenses with the sensors. Microsoft pioneered this technology on their Hololens platform, dubbed ‘Mixed Reality’, and they have licensed it out to a wide range of hardware manufacturers (Dell, Acer, HP, Samsung etc.). In these cases the headsets contain sensors that watch the world and triangulate their position in it. This is quicker to set up, and much more portable, but also much less stable at tracking position in a space than the outside-in rivals, especially if anchoring elements in the real world space (people!) move. The Samsung Odyssey is presently the archetype of this system, with a higher-than-Oculus resolution screen, and neat controllers. (Cost: £200 - £500 plus gaming PC)

Hybrid tethered. The Vive system is unique. It uses external beacons (not sensors) to cast an infrared pattern on the real world, and the headset is covered in sensors that detect the pattern, and calculate its relative position in relation to it. So it uses inside-out tracking, but with outside help. This system takes just a few minutes to set up, but needs stands or brackets for the sensors, so is not highly portable. It is the most stable tracking available, however, and several third party headsets that use the system have been developed. The Vive Pro is the top of the line here, I think. (Cost: £500 for Vive, £1299 for Vive Pro Kit, plus gaming PC)

Standalone. The holy grail for VR is a system that has the graphical power of the tethered system, and 6dof movement, but without the need for a PC. By necessity such a system would need to use inside-out tracking, and would need to supply all of its own power (i.e. not use mains via the PC). As of late 2018 there are a handful of attempts at this (Oculus GO, Vive Pre, Pico Neo the main ones) but they are all seriously compromised by their graphical power. The first serious breakthrough may be the Oculus Quest, due out in spring 2019, but I reckon that we remain a long-way from a Vive-standard experience in anything standalone – the power and graphic challenges remain fundamental.

Augmented Reality (AR)

Augmented Reality tech takes virtual elements, and integrates them with our perceptions of the actual world. You have almost certainly seen the rudimentary versions of this in SnapChat filters, in Google Translate’s camera app, or in the viral success of Pokemon Go. When we have a screen that relays a camera feed (as any smartphone can when framing a picture), basic augmented reality just requires layering some special effect onto the feed. In that sense it might seem like a much more straightforward, and much less profound technology than VR, but that is to confuse the basic applications of AR that we have seen to date with the enormously sophisticated AR that will be commonplace a decade from now.

Once again, splitting out the hardware options should prove illuminating.

Basic AR: This uses a screen and a camera to relay a feed, and some graphic overlay to adapt the image. The most basic versions of this are ones that are ‘blind’ to the topography in the real world, and so provide just a flat overlay (like clunky photoshop), but slightly more sophisticated versions use a known pattern in the world (QR code, perhaps) to help position the virtual content realistically in the scene. Notice that this needs a part of the world (the QR code) to be ready before the experience will work. (Most smartphones since 2014 can do Basic AR)

SLAM AR: Using surface-detection algorithms, the positioning of virtual elements in the scene can be achieved without the need for a known target (QR code). This liberates the AR process, but a balance is required to ensure that the graphic and power demands are not too excessive. Breakthroughs in SLAM saw Apple and Android race to release their AR engines (ARKit and ARCore respectively). Most high end phones from 2017 can make use of apps with these elements – if you use a phone tape-measure, you are using AR!

Context-aware AR: The holy grail is full context-aware AR that does not require any in-world trigger, and goes beyond SLAM in its ability to detect the geometry in a space. Key to this is understanding a scene well enough to know how a virtual object should be lit, and when it should be occluded, so that it seems genuinely part of the real world. This is unlikely to be achieved in a smartphone type format, and AR headsets are the focus of the industry at present. These headsets do/will contain a full array of sensors pointing in different directions to understand the space.

An AR headset the puts a screen between the user and the world is not much use. It would involve a huge amount of expensive and power-hungry processing to largely recreate the world as it actually is. In fact, it would re-create the real-world considerably less well than our eyes do on their own! This is why the real cutting edge for AR is transparent lenses that let you see the world as normal, except for the virtual elements in the scene. The main challenges faced by this technology are around producing a strong enough image to compete with ambient light, and to be able to cover a wide portion of the visual field with virtual entities (so they remain in the periphery even when you are not looking directly at them). The leading headsets today are Microsoft’s HoloLens (35 degree field of view, very expensive at £3k approx, only officially available to developers), and the recent, and much over-hyped, Magic Leap One (about 40 degree FOV, £2,300).