Extended Reality (XR) technologies have been advancing at exponential rates. Mark Hoffman Ph.D, Chief Research Information Officer at Children’s Mercy Research Institute, explains what they are and what they’re doing for healthcare.

As a visual learner, I appreciate the expression that “a picture is worth a thousand words”. While two-dimensional (2D) images, whether static or animated, have been the stock of visual learning, new technologies enable us to experience and interact with threedimensional (3D) visualizations in widely available and intuitive digital formats. After decades of foundational work, the past few years have witnessed a blossoming of a group of technologies broadly characterized as ‘Extended Reality’ (XR) and their purposeful application in healthcare.

Extended reality is the use of advanced visualization capabilities to enable users to better comprehend and interact with three dimensional images, whether for entertainment or work. While there are clear areas of overlap, there are three general categories of extended reality. With virtual reality (VR) the user wears a device that obscures the “real” world and displays sophisticated immersive visuals. The Oculus Rift is a good example of a VR platform. In augmented reality (AR) the virtual and “real” are blended. ‘Pokemon Go’ or ‘Harry Potter Wizards Unite’ are familiar smart phonebased examples of AR. Mixed reality (MR) builds on augmented reality by adding deeper layers of awareness of the environment. For example, mixed reality visors such as the Microsoft Hololens map the environment, identify flat surfaces and textures and use that information to influence interactions with the visualization. The user sees and hears the virtual in the context of their environment.

Extended reality technologies are being used to support healthcare from the perspective of the patient, provider, organizational administration and community. Patient applications include patient education and distraction. For example, the University of Sydney use VR to inform patients about the details of an upcoming radiation therapy procedure and found an increase in patient knowledge in the VR group compared to a control group. Lucille Packard Children’s Hospital uses the Hololens or Oculus Rift to distract children during painful or uncomfortable procedures. At Children’s Mercy we are using the Merge Cube to enhance patient and parent understanding of anatomy before a corrective surgical procedure. The Merge Cube is the size of a Rubik’s Cube and is covered with glyphs that are recognized by mobile applications. Merge enabled applications integrate the glyphs with a 3D view overlaid on the cube, which the user can move to reposition the image. Unlike other AR technologies, Merge cubes are priced for widespread access.

Provider applications of extended reality include general clinical education, targeted training, surgical planning and other purposes. The Anatomy Trainer developed by the Cleveland Clinic was one of the first Hololens applications and provides a detailed, interactive series of immersive modules. Other organizations have developed procedure training modules, including sophisticated versions that integrate with manikins or other tactile simulators that are overlaid with virtual imagery. Surgical planning and in a few instances, visualization during surgery, are very active areas of research and development for health applications of extended reality. During the COVID-19 crisis an additional provider use of extended reality has emerged – helping frontline care givers take brief respite into a virtual world. Providers wear a virtual or augmented reality device and can enter a calming simulation before returning to the clinic or emergency department.

Community applications of extended reality include the use of augmented reality games to promote positive behaviors such as walking and healthy eating habits. Shortly after the release of Pokemon Go, a number of studies demonstrated increased walking habits among users, though additional studies show a higher risk of distraction related injuries as well. So far, the use of advanced visualization has not been widely applied to support healthcare administration, despite the opportunity to integrate architectural visualizations with real time data or to use advanced interactive dashboards that are enhanced with augmented reality.

Extended reality also has applications in biomedical research. Our team began our mixed reality work by developing a data pipeline to incorporate small molecule and protein visualizations into the Hololens. Through this work, users can view a protein floating in front of them, can rotate the virtual structure and can zoom in and out or walk through the interior of the molecule.

There is significant need for research into the human factors design decisions that best promote improved comprehension during extended reality enabled interactions. Likewise, there is an important role for research into matching an extended reality modality to a particular clinical objective. There is significant potential for the integration of AI capabilities with extended reality technologies. The latest versions of these platforms include significant embedded AI but have variation in the extent to which they expose real time data streams from the sensors in the devices to AI developers. Ultimately, access to these real time data streams will enable the integration of AI, internet of things and extended reality to deliver fully integrated capabilities that enhance the delivery of healthcare.