Extended Reality: Terminology, Development & Business Opportunities.

Like artificial intelligence and mentally controllable prostheses, the concept of virtual reality (VR) made its debut in science fiction, but it has now found its impact in real-world implementation.

As a matter of fact, with its broad applications, the technology has evolved into a series of variations along the way, each given a compelling name and a catchy abbreviation, including augmented reality (AR), mixed reality (MR), hybrid reality (HR), diminished reality (DR), substitutional reality (SR), cinematic reality (CR) and eXtended reality (XR).

In this article, we’ll first give you a simple introduction of all the terms above. Then, the potential prospects and business opportunities of the tech will be discussed. Hopefully, it can not only give you a grand picture about the current state of the technology but also give you some insights of how to apply it to your business.

Introducing the terminology

• Virtual Reality (VR):

In virtual reality, an individual will experience an utterly simulated environment.

The environment can be presented with 3D effect, and it’s most often realized by using a wearable device (i.e., a head-mounted display or HMD) that includes two tiny screens, one for each eye. The screens will be showing the same image from slightly different angles, and such trick will prompt the user to see stereo images.

Figure 1. A head-mounted display (HMD) or VR headset in colloquial language released by HTC, which weighs about 470 g (image source: XR Express TW).

As for the situations where 3D effect is not involved, VR can be achieved simply through a smartphone. By moving the phone around, one can explore a 360-degree environment just like peeking out from a small window. One application of such technique came from the company Sphero, who utilized it on their Star Wars toys, allowing players to explore the inside of a Star Wars spaceship with his or her phone. Another use case is CNN’s VR news, in which a report is presented as a 360º video so that the visitors can “feel” the news as if they were the man behind the camera (experience it yourself by clicking the link: https://edition.cnn.com/vr).

Because a HMD created specifically for virtual reality is usually expensive, some have found a way to show stereoscopic VR with a smartphone as well. Such magic is realized by splitting the phone’s screen into two halves, each displaying the same scene from different angles for different eye, respectively. The phone will then be placed in front of a special smartphone holder that resembles a HMD without any monitor. That way, people can experience 3D effect similar to using a traditional VR headset. Note that since the device said uses a smartphone as its stereo display, it can be dramatically cheaper and lighter comparing to a traditional head-mounted device.

Figure 2. A VR headset made by JetRock Industrial which uses a smartphone as its display. The whole device weighs only 87 g when not holding a phone and is foldable for portability.

• Augmented Reality (AR):

Unlike VR where everything in front of users’ eyes is artificial, AR is about adding simulated entities to the reality, and therefore can provide broader possibilities of application. With that being said, the interrelation between real and virtual objects in AR is seriously limited. Since the tech is achieved by putting a simulated layer on top of the real image, advanced interactions between actuality and fantasy, such as those involve depth (e.g., making a virtual t-rex show up from behind a sofa), is totally out of the question.

Such technology can be implemented by either a smartphone or a pair of AR glasses, and one of the most famous uses of this technology is the game, Pokémon Go.

Figure 3. A pair of AR glasses created by Epson (image source: Epson’s official site).

• Mixed Reality (MR) / Hybrid Reality (HR):

Mixed reality, sometimes hybrid reality, is a concept very similar to AR since both of them involve mixing the artificial objects with real images. As a matter of fact, some will even consider AR a subcategory of MR. However, instead of simply covering the reality with a simulation layer as in AR, MR allows far more complicated interrelation between virtual and real-life elements, and can therefore providing a more interesting interactive experience to the users.

Figure 4. The picture clearly demonstrates the difference between VR, AR and MR. In VR the whole scene is artificial, while in AR a simulated layer was added on top of the real image. As for MR, the virtual object can perform complex interaction with the entities from real scene.

Due to its power, MR is often equipped on the devices such as smartglasses. Two renowned examples of this are Magic Leap and Microsoft HoloLens.

• Diminished Reality (DR):

While AR and MR add virtual entities into a scene, diminished reality removes a particular target from the scene. This is done by first recording the background that does not contain the target, and then covering the area occupied by the target with the corresponding part of the background.

This technique can be utilized to create hollow-man or invisible-cape effect, such as the “cloaking device” effect the one can find on Apple’s Photo Booth app. Another application of DR is Adobe Cloak technology that was first introduced in Adobe Max 2017 conference.

Video 1. Adobe’s Project Cloak demonstrated on Adobe Max 2017.

• Substitutional Reality (SR):

The term, substitutional reality, was first presented by RIKEN (Japanese: rikagaku kenkyusho; English: Institute of Physical and Chemical Research) Brain Science Institute in Japan in a scientific context (click here to read the original paper).

In a typical paradigm of SR experiment, a participant was asked to wear a HMD. Then, he or she would be told that what they were seeing from the HMD was a live stream, while in reality the live stream and a pre-recorded panorama video were shown in an alternative manner secretly. The technique is designed to manipulate participants’ awareness of the reality or trigger confusions in time and spatial perception, which can be useful in studying one’s metacognitive functions or psychiatric disorders.

Other applications of the technique suggested by its creators include helping the procedure of psychotherapy and offering surreal experiences in an art performance.

• Cinematic Reality (CR):

In 2014, Magic Leap raised a fund of circa half a billion US dollars with the help from Google to develop a new technique which is now known as cinematic reality. Such technique involves extraordinary control over the lighting, quality and details of the simulated entities, so that they can look as realistic as possible when presented in a live condition. As a matter of fact, according to a short clip made by Magic Leap (see Video 2) for explaining what they hope to achieve, it seems that if mature, CR technology will not depend on any HMD sort of device to present virtual objects.

Video 2. A demonstration video from Magic Leap about what they hope to achieve with CR. Note that the whale and water in the video are special effects created in post-production, not a genuine CR effect.

Although the idea sounds pretty impressive, it’s obviously hard to be realized in practical. Thus, at least for now, Magic Leap seems to have moved their interest away from CR and focused mainly on MR.

• eXtended Reality (XR):

Although many names have been introduced, all of them can actually be categorized into one of the following three: VR, AR or MR. And the term extended reality, aka XR, is a general word coined for summarizing the three technologies.

Where are XR technologies being used?

According to a white paper of Taiwan XR industry published by XR Express TW, a project directed by National Development Council (NDC) in Taiwan for boosting XR-related businesses, the top four areas of XR applications in 2018 are gaming industry (23.0%), marketing (15.1%), video entertainment (14.5%) and education (11.8%). For more information, please refer to Figure 5, or download the white paper through this link.

Figure 5. Where XR technologies were used by Taiwanese companies in 2017 and 2018 (source: White Paper of Taiwan XR Industry, p.53).

Besides the categories mentioned above, XR has been applied to so many other fields as well. Below, we enumerate some applications that we deem impressive or interesting; hopefully they can ignite your imagination on how to use the tech:

• Building an avatar of your patients:

One can easily imagine that how medical field will be benefited if doctors can create an interactable digital twin of their patients from CT or MRI data, and that’s what Augmented Intelligence is attempting to fulfill. With the tech, doctors can better explain illness to patients and students, have a more efficient discussion with colleagues, or even simulate the surgery beforehand to prevent mistake from occurring.

• Using virtual reality in psychotherapy:

One of the most symbolic features of VR is immersion, and therefore it has great values for certain kinds of psychotherapy such as exposure therapy (to get people overcome their fear), PTSD treatment and helping people to fight substance abuses.

• XR assists job interviews:

Tech such as VR, AR and MR also have their roles in the recruitment process. Besides the obvious application of better knowing your candidates in a remote job interview, business such as Lloyds Banking Group utilizes XR to drop candidates into scenarios that they may have to deal with in the future. Also, a high-tech interviewing process involving XR can actually increase interviewees’ willingness to work in the organization; it not only signifies that the organization is constantly moving forward but also ensures candidates they will be working in an environment willing to embrace novelties.

The major challenges of XR development

Despite being potential, XR technology has its own problems. The following stats from the said white paper published by XR Express TW suggest some key obstacles in today VR and AR industries:

Figure 6. Bottlenecks of Taiwanese AR & VR industry (source: White Paper of Taiwan XR Industry, p.65).

As we can see from Figure 6, the major difficulties for AR and VR businesses are distinct. For AR, the biggest bottlenecks are lacking (quality) contents to use (30.6%) and low awareness of the tech from enterprises and consumers (20.6%; P.S., people are more familiar with VR than AR). As to VR, three things stand out, that is, the price of the require hardware is too high (26.4%), bad user experiences (24.2%) and, again, lacking (quality) contents to show (22.5%).

In addition to the obstacles listed above, XR technologies have other concerns that may hinder users from accepting them. The first concern is the potential dangers of them, especially VR. Since VR will shut users’ senses away from the actual world, people can hurt themselves when their headset is on (see a compilation for accidents involving VR; note that although it was presented as a joke, serious injuries might occur). And to prevent mishap from happening, many Taiwanese companies have started to ask people to sit on a chair while wearing a headset.

The second is about health. Since XR experiences have not yet been as prevalent as smartphones, it’ll be hard to tell whether the tech has significant impact on our vision or not. However, if in near future people were to use XR heavily each day (with devices such as smartglasses, there’s a chance), will that increase the odds of we getting myopia or glaucoma? Perhaps only time will tell.

The possible future for XR technology

In the last paragraph, let’s talk about some trends in XR industries that we’ve observed, and from there making a few inferences about where the industries may be heading to.

• AR will thrive, while VR is relatively restricted and MR is not there yet.

At least for Taiwanese companies, AR is deemed to have the best chance to prosper in the upcoming year (agreed by 48.9% of the businesses); VR comes second (29.8%), and MR the last (21.3%).

What are the reasons behind the stats? First, AR is the most popular technology among the three since it can achieve effects similar to MR (although not as great, but it’s enough for application) yet at the same time do not depend on pricy device as VR does. Also, AR is less immersing than VR; this not only makes AR much safer than VR but also help reducing potential dizziness.

Second, although MR is obviously the most powerful in the three, the techniques involve to make it possible are still immature. According to the XR Express TW’s white paper, most businesses believe that it’ll take at least another 2 to 3 years before MR really takes off.

• Creating XR materials will be easier and more intuitive.

As mentioned before, both AR and VR are suffering from not having sufficient or quality-enough materials for utilization. The causes of this phenomenon are several. For starters, the tech is still young, hence complete libraries or resources haven’t been built just yet. Second, people do not have the expertise to create the materials they need on their own.

Seeing the business opportunities behind, many companies have built a platform (such as Amazon Sumerian, MAKAR and so on) that allows one to create interactable XR elements without any coding ability. It should be safe to say that such service will only become more prevailing and mature in the future, making the technology more available to the general public.

• 5G will release the true power of XR.

While an individual can undergo an episode from another location through extended reality, 5G, which promises lower latency and faster data transmission, can conspicuously uplift such experience.

• XR will go from vision to other senses to create ultimate virtual experiences.

Today, most of the XR companies still focus on vision and audition. However, some have turned their attention to other senses. For instance, a technical group has made a cockpit-like chair that along with a VR headset is capable of providing its users a 4D experience while playing (see Video 3).

Video 3. A cockpit-like chair that provides more realistic virtual experiences in gaming.

In another example, an organization called NeuroDigital Technologies has developed a system to allow blind people to “see” artworks through virtual touchiness (see Video 4).

Video 4. Virtual touchiness is utilized to make blind people “see” artworks (source: Mashable).

In the future, we believe XR technologies will be extended to more and more senses (even enabling us to feel some non-human ones), and the hardware required more and more ignorable than what we have now, hence eventually providing us an ultimate virtual experience as vivid as the reality.

Footnote: Neurozo Innovation shares viewpoints, knowledge and strategies to help you succeed in your quest. If you have any question for us, please feel free to leave a comment below, or e-mail us at: neville@neurozo-innovation.com. For more articles like this, please join our free membership. Thank you very much for your time, and we wish you a wonderful day!