Spatial computing is a technology that enables computers to blend in with the physical world in a natural way. Apple is not the first company to delve into the space, but it believes this will be the next big thing for computing. And the company is willing to bet on it with its new AR/VR headset, the Apple Vision Pro. Here’s our primer on spatial computing—and our thoughts on how it will change the way we interact with computers.

How Does Spatial Computing Work?

Spatial computing brings together the virtual and physical worlds in a seamless way. You can experience it through headsets such as the aforementioned Apple Vision Pro, Microsoft HoloLens, Meta Quest Pro, and Magic Leap. These devices display the real world and, at the same time, embed real objects into the scene in a way that appears three-dimensional. For example, you could place a virtual piece of furniture in your living room to see what it would look like before you buy it; or factory workers can overlay technical manuals onto machine they're building.

These devices rely on several technologies to mix the digital and physical worlds. They use computer vision, which processes data from cameras and other sensors and captures visual information about the environment, including the position, orientation, and movement of objects. With sensor fusion, they can combine data from multiple sensors, such as cameras and LiDARs , to create an accurate and comprehensive view of the environment. And they use spatial mapping to create a 3D model of the environment, which allows for more precise placement and manipulation of digital content

Apple Vision Pro headset

One of the key advantages of spatial computing is its ability to understand the depth of the environment. This allows for realistic and natural interactions with virtual objects, because they can be placed and manipulated in a way that corresponds to the physical world. For example, a virtual object can be placed on a table, moved around, or hidden behind other objects in a way that mirrors our real-world actions.

Spatial-computing devices have features that enable you to interact with virtual objects. Eye-tracking technology monitors your gaze, and handheld controllers and motion sensors enable you to manipulate virtual objects. For example, the Apple Vision Pro has an advanced eye-tracking system and hand-gesture-detection sensors that let you simply look at items to activate them, or to move them around by pinching your fingers or flicking your wrist.

Devices that employ spatial computing might also have speech-recognition features to support voice commands. These are especially useful within hands-on environments such as manufacturing, where you can’t use hand gestures or controllers. The Vision Pro, HoloLens, and Magic Leap support voice commands.

Magic Leap 2 headset

Spatial computing is related to both augmented reality (AR) and virtual reality (VR). AR means overlaying digital content onto the real world, typically using a phone or smart glasses. While AR can enhance the user's perception of an environment, it doesn’t embed digital content in 3D space or provide a sense of depth. VR, on the other hand, creates a fully immersive digital environment that replaces the user's physical surroundings. And mixed reality (MR) employs a blend of AR and VR.

How Can We Use Spatial Computing?

Spatial computing has the potential to revolutionize various industries by creating experiences and applications that were previously impossible. Here are some of the domains where spatial computing is applicable.

Gaming and Entertainment

Spatial computing creates immersive gaming and entertainment experiences that let you interact with virtual objects and characters in a natural and intuitive way. For instance, instead of a keyboard or joystick, you could use hand gestures or your gaze to control avatars and manipulate objects in a game. Spatial computing can also let you experience real-world events in a different way. With the Apple Vision Pro, you could watch an NBA game that lets you feel like you’re in a court-side seat. It also can record memories in a spatial format that you relive later in 3D.

Education and Training

Spatial computing can provide an engaging and interactive learning experience that enhances knowledge retention and skills acquisition. For example, medical students can use spatial computing to practice surgical procedures in a virtual environment that simulates real-world conditions. In science and engineering, students can use spatial computing to build and test virtual prototypes of machines or conduct experiments.

Architecture and Design

Spatial computing can enable architects to create, visualize, and modify designs in the real world without the need for physical prototypes. This can save time and cost and allow for more iterative design processes. Designers can also use spatial computing to create virtual prototypes of products and test their functionality and ergonomics in different physical settings.

Healthcare

Spatial computing can provide new ways to diagnose, treat, and monitor patients. For example, doctors could use spatial computing to overlay virtual screens and diagnostic information onto the real world, helping them to make more informed decisions. Or, a doctor could see a patient’s medical-imaging scans via a headset while they’re operating on that patient. Spatial computing could also help patients with physical or cognitive impairments to regain their independence by providing them with virtual assistants or rehabilitation exercises that are tailored to their needs.

In addition to these domains, spatial computing could have applications in fields such as retail, manufacturing, and transportation. For instance, manufacturers could use spatial computing to augment assembly lines with helpful virtual objects and information to optimize production processes.

The Future of Spatial Computing

Despite its appeal, spatial computing has had mixed success so far. One of the biggest challenges facing the field is the cost of hardware. The best spatial-computing devices cost thousands of dollars and are not accessible to everyone. That’s why they have yet to make their dent in the consumer space and have found limited success in niche enterprise applications where the boost in productivity justifies the cost.

In addition, the weight of the headsets, low resolution, and lack of comfort have been major concerns, as they can cause fatigue and motion sickness during extended use. Battery life is also a limiting factor: Most devices' batteries last just a few hours and require frequent charging. But companies are making progress in many of these areas. And as other industries have shown, the cost of sensors, displays, and hardware will gradually decrease as the field evolves and becomes more competitive.

The jury is still out on whether Apple’s entrance into spatial computing will be a failure or a defining moment for spatial computing. But Apple has a history of setting standards: Just consider the mouse, the graphical user interface, the iPod, and the iPhone. Although it's extremely expensive for now, the Vision Pro might trigger a momentum that will lead to new and innovative applications and devices for spatial computing.