Immersive Tech Hardware
This is an opinionated tech hardware primer by Neil McDonnell.
Last updated December 2021
Virtual Reality (VR)
Virtual Reality tech has blossomed since 2012 thanks to a catalysing breakthrough from Oculus (who Facebook promptly bought). VR intervenes on your senses (via eyes and ears mainly, but not exclusively) to immerse you in a virtual environment. The sense of presence in the virtual environment that this can inspire in users is staggering, even with cartoonish or otherwise unreal environments.
Here are three key elements to a good experience in VR:
1. High refresh rate screens. With sub 60hz screens, simply looking around your virtual environment could cause quite significant nausea as there was a subtle delay between your head movement, and the world catching up. A widely-accepted minimum for a consumer unit is 72hz, and at 90hz or above, this is mostly eliminated.
2. Let the user move, but not too fast/suddenly. This is controversial for some, but I believe that the manner of movement in a VR experience is one of the biggest factors in a good experience. There is an immediate boost to immersion if people can take a step in VR by taking a real-world step. This grants the user 6dof (six degrees of freedom) in the virtual world, just as they have in the real world. That said, when movement in the virtual and real world are de-coupled, as in when you sit still in the real world, but on a virtual roller coaster in VR, nausea is almost guaranteed.
3. Interaction. If the user cannot engage with the environment, interacting with objects and such like, then the real powers of VR to immerse us somewhere else, are not being deployed.
All three depend on both the software and the hardware. For example, I know of no headset on the market that does not accomplish 1 for at least some apps, but problems arise when the headset, or the computer running it, are underpowered relative to the software being run. For movement (2), and interaction (3) the hardware needs to be capable of 6dof and have a controller, respectively, and the software needs to make the most of them.
The hardware splits along five broad types:
3dof. The cheapest and least capable devices are those which allow the user to look around but not move or (typically) interact. The most common such systems are the ‘phone-in-a-box’ systems such as the google carboard, Samsung gear, and equivalents. These harness the power of existing smartphones to give a basic VR experience. The Google Daydream is probably the best of the pack, with a comfortable, high quality ‘box’ for a compatible phone, and a good ‘wand’-style controller. (Cost: phone + £10 - £150)
Outside-in tethered. Systems which require a separate PC to run them currently deliver vastly higher quality experiences than the alternative ‘standalone’ variety (with one notable caveat, see Oculus Quest range below). Of the tethered sort, the Oculus Rift was the first consumer unit to be made widely available, and was launched in April 2016. This system uses either 2 or 3 sensors attached to the PC to ‘watch’ the movement of the headset and (fantastic) controllers and co-ordinate that real movement with virtual movement. Since the sensor placement is critical to the veracity of the movement, this can be a somewhat fragile set-up requiring careful calibration. (Cost: approx. £350 plus gaming PC - displaced in Spring 2019 by Oculus Rift S - see Inside-out tethered section below, both now discontinued)
Inside-out tethered. The second sort of tethered system is that which dispenses with the sensors. Microsoft pioneered this technology on their Hololens platform, dubbed ‘Mixed Reality’, and they have licensed it out to a wide range of hardware manufacturers (Dell, Acer, HP, Samsung etc.). In these cases the headsets contain sensors that watch the world and triangulate their position in it. This is quicker to set up, and much more portable, but also notably less stable at tracking position in a space than the outside-in rivals, especially if anchoring elements in the real world space (people, pets, furniture) move. The HP Reverb G2 is perhaps the high-bar of the mixed-reality systems, with a higher-than-Rift resolution screen, and decent controllers. The Rift S from Oculus displaced the original Rift, at a similar price point (£400), and offered superior resolution and tracking to the Microsoft-based systems of the time. With the discontinuation of the Rift S, the HP Reverb G2 is the standout option for most. (Cost: £500 plus gaming PC)
Hybrid tethered. The Vive system is unique. It uses external beacons (not sensors) to cast an infrared pattern on the real world, and the headset is covered in sensors that detect the pattern, and calculate its relative position in relation to it. So it uses inside-out tracking, but with outside help. This system takes just a few minutes to set up, but needs stands or brackets for the sensors, so is not highly portable. It is the most stable tracking available, however, and several third party headsets that use the system have been developed. For most enterprise or consumer users, the Vive Pro or Pro 2, or The Valvve Index for gaming, sit at the top of the line here, but there is an extraordinary outlier in the $5,995 Varjo XR-3 system, which boasts 'human-eye' resolution (akin to Apple's Retina). (Cost: £500 for Vive Cosmos, £1200 for Vive Pro Kit, plus gaming PC)
Standalone. The holy grail for VR is a system that has the graphical power of the tethered system, and 6dof movement, but without the need for a PC. By necessity such a system would need to use inside-out tracking, and would need to supply all of its own power (i.e. not use mains via the PC). As of late 2018 there were a handful of attempts in this direction (Oculus GO, Vive Pre, Pico Neo the main ones) but they were all seriously compromised by their graphical power. The first serious breakthrough device, however, is the Oculus Quest, which launched in spring 2019. Before use, I was sceptical that a standalone device, powered by a mobile-phone chip, could possibly replicate the experience you get when using a super-powered PC with a dedicated graphics card. (For comparison, the GeForce 1080Ti graphics card at the time cost in excess of £1,000 on its own, had its own cooling system, requires an increased power supply, and is about the size and weight of a hefty dictionary. The processor in the Quest is smaller than your fingernail and is powered by a mobile battery.) Technically, I was not wrong: the apps had been heavily 'optimised' for Quest, rendering them less detailed and less realistic in textures, reflections, and the like. From an experience point of view, however, I was wrong. The Quest somehow offered a superb user experience despite its technical limitations. Rather sooner than expected in 2020 Oculus released an updated, and much improved, Quest 2 and lowered the price. As of 2021, anyone can get the console-style experience of VR for just £300 without the need of any PC or other device. That is extraordinary. Anyone wishing to load on their own apps, or engage in enterprise-level applications, will need the enterprise edition, costing £800. The phrase "Game Changer" can scarcely be overused in any domain as much as it is in relation to VR, but the Quest range is deserving of that description. (Cost: £300 - £400 for Oculus Quest, consumer edition)
Augmented Reality (AR)
Augmented Reality tech takes virtual elements, and integrates them with our perceptions of the actual world. You have almost certainly seen the rudimentary versions of this in SnapChat filters, in Google Translate’s camera app, or in the viral success of Pokemon Go. When we have a screen that relays a camera feed (as any smartphone can when framing a picture), basic augmented reality just requires layering some special effect onto the feed. In that sense it might seem like a much more straightforward, and much less profound technology than VR, but that is to confuse the basic applications of AR that we have seen to date with the enormously sophisticated AR that will be commonplace a decade from now.
Once again, splitting out the hardware options should prove illuminating.
Basic AR: This uses a screen and a camera to relay a feed, and some graphic overlay to adapt the image. The most basic versions of this are ones that are ‘blind’ to the topography in the real world, and so provide just a flat overlay (like clunky photoshop), but slightly more sophisticated versions use a known pattern in the world (QR code, perhaps) to help position the virtual content realistically in the scene. Notice that this needs a part of the world (the QR code) to be ready before the experience will work. (Most smartphones since 2014 can do Basic AR)
SLAM AR: Using surface-detection algorithms, the positioning of virtual elements in the scene can be achieved without the need for a known target (QR code). This liberates the AR process, but a balance is required to ensure that the graphic and power demands are not too excessive. Breakthroughs in SLAM saw Apple and Android race to release their AR engines (ARKit and ARCore respectively). Most high end phones from 2017 can make use of apps with these elements – if you use a phone tape-measure app, you are using SLAM AR!
Context-aware AR: The holy grail is full context-aware AR that does not require any in-world trigger, and goes beyond SLAM in its ability to detect the geometry in a space. Key to this is understanding a scene well enough to know how a virtual object should be lit, and when it should be occluded, so that it seems genuinely part of the real world. This is unlikely to be achieved in a smartphone type format (i.e. mediated by a screen), and wearable AR headsets are the focus of the industry at present. These headsets do/will contain a full array of sensors pointing in different directions to understand the space.
An AR headset that puts a screen between the user and the world is not much use. It would involve a huge amount of expensive and power-hungry processing to largely recreate the world as it actually is. In fact, it would re-create the real-world considerably less well than our eyes do on their own! This is why the real cutting edge for AR is transparent lenses that let you see the world as normal, except for the virtual elements in the scene. The main challenges faced by this technology are around producing a strong enough image to compete with ambient light, and to be able to cover a wide portion of the visual field with virtual entities (so they remain visible in the periphery even when you are not looking directly at them). The leading headsets today are Microsoft’s HoloLens 2 (52 degree field of view, very expensive at £3,500k approx), and the recent, and much over-hyped, Magic Leap One (about 40 degree FOV, £2,300).