Failing to properly handle occlusion in MR applications

What is Occlusion in Mixed Reality (MR)?

Occlusion in Mixed Reality (MR) refers to the phenomenon where virtual objects are blocked from the user’s view by physical objects in the real world. It is crucial for creating a realistic and immersive experience, as the MR system must correctly simulate how virtual objects interact with the real world, particularly when they are partially or fully obstructed by real-world objects.

For example:

In VR, objects can be occluded by walls, avatars, or other environmental elements.
In AR, virtual content (such as a digital chair) should appear behind real-world objects (like a table), and the AR system must account for this when displaying the scene.

Handling occlusion properly ensures that virtual objects interact with the physical environment in a way that feels natural. If this is not done correctly, the user may experience breaks in immersion, confusion, or visual inconsistencies.

Consequences of Failing to Handle Occlusion Properly

1. Inconsistent Visual Representation

When occlusion is not properly handled, virtual objects may appear in front of real-world objects or vice versa in situations where it’s impossible in the real world. This leads to broken immersion and can disrupt user experience.

Example: A virtual object floating in front of a real-world object that should be occluding it, like a chair appearing in front of a table that should block part of it.

2. Loss of Immersion

If virtual objects pass through real-world objects or vice versa without any realistic occlusion, the immersion breaks. It makes the virtual experience feel disconnected from reality, which can hinder user engagement and the effectiveness of the MR application.

Example: A virtual character walking through a real-world wall without any visual distortion, as if the wall is non-existent.

3. Unrealistic Interactions

The lack of proper occlusion can result in virtual objects incorrectly interacting with the real world. For instance, virtual objects might not be obscured when they should be, or they might incorrectly pop through physical objects when they shouldn’t.

Example: In AR, a virtual pet might be positioned behind a table but then appear in front of the table when the user moves around.

4. Confusion and Frustration

Users may become confused when they observe virtual objects behaving unrealistically in relation to the real world, leading to frustration. Inaccurate occlusion can make interactions feel unnatural, especially when users expect certain behaviors from the virtual objects.

Example: A virtual car placed in the real-world driveway may pass through the real-world garage door instead of being blocked by it.

Why Does Occlusion Fail in MR Applications?

Occlusion issues often arise due to limitations in tracking, sensing, or algorithms used in MR applications. Some of the primary reasons for failure in occlusion handling include:

1. Limited Depth Perception and Sensor Data

In MR, depth information (i.e., how far objects are from the user) is crucial to determine which objects should be in front of or behind others. Insufficient depth information from sensors can result in poorly simulated occlusion.

Example: A depth sensor like the one in Microsoft HoloLens or Magic Leap might fail to properly capture the distance of objects in the real world, leading to unrealistic virtual-to-physical object interactions.

2. Weak Object Detection and Tracking

MR applications often use cameras and sensors to detect the real-world environment. If the object detection or environmental tracking algorithms are insufficient, the system might fail to correctly detect the occluding objects or their boundaries.

Example: The software might fail to identify a real-world object like a table, which could lead to virtual objects being rendered in front of it.

3. Poorly Implemented Collision Detection

Collision detection algorithms help determine whether virtual objects should be obstructed by real-world objects. Incorrect or inadequate collision detection algorithms may fail to register when a virtual object should be hidden behind a physical one.

Example: A virtual object may intersect with a real-world object because the system didn’t accurately account for the collision zone or boundaries.

4. Inaccurate Environmental Mapping

For MR applications to handle occlusion properly, the environment must be accurately mapped in 3D. When environmental mapping is incomplete, outdated, or inaccurate, the application will fail to generate accurate occlusion information.

Example: If a real-world bookshelf is not accurately mapped, the MR system may not know when a virtual object should be blocked by it.

5. Lighting Conditions

Lighting can play a huge role in visual occlusion in MR. If lighting in the real world is inconsistent or overly reflective, it can be challenging for the MR system to differentiate between real and virtual objects.

Example: Bright lighting can interfere with depth sensing or visual tracking, leading to the incorrect rendering of occlusion.

6. Real-time Processing Limitations

Occlusion handling requires real-time processing of data from the environment and sensors. If the system does not have sufficient processing power or algorithm optimization, the occlusion rendering may be delayed or inaccurate.

Example: The MR system may show a virtual object in front of a real-world object in real-time, even though it should be occluded based on the current viewpoint.

Techniques and Technologies to Improve Occlusion Handling

1. Depth Sensors and LiDAR

To correctly simulate occlusion, MR systems need accurate depth data. This is achieved through depth sensors and LiDAR (Light Detection and Ranging), which provide precise measurements of the distance between the user and objects in the environment.

Example: Apple’s ARKit and Microsoft HoloLens use LiDAR scanners for improved depth sensing, enabling better occlusion handling.

2. Simultaneous Localization and Mapping (SLAM)

SLAM algorithms allow MR devices to map the environment in real-time and track the user’s position within that space. By creating a 3D model of the surroundings, MR applications can better understand which objects should be in front or behind others.

Example: SLAM technologies like Visual SLAM or LIDAR SLAM are used in AR applications to detect the real-world environment and correctly render occlusion.

3. Raycasting and Collision Detection Algorithms

Raycasting is a technique where a virtual ray is projected from the user’s viewpoint to determine whether objects are obstructed. This allows MR applications to simulate realistic occlusion by calculating if a virtual object should be blocked by a physical object.

Example: Unity and Unreal Engine provide built-in raycasting and collision detection systems that developers can use to handle occlusion correctly.

4. Improved Object Recognition and Tracking

Using AI-based object recognition and tracking algorithms can help MR systems better understand the real-world environment. These technologies can accurately detect the positions of real-world objects, allowing virtual content to react dynamically.

Example: Machine learning algorithms can be trained to recognize common objects in the environment, such as tables or chairs, to predict occlusion more accurately.

5. Lighting and Shadow Simulation

To improve occlusion handling in MR, it is important to simulate how virtual objects interact with real-world lighting and shadows. Real-time shadow casting and lighting models help ensure that virtual objects are realistically occluded by real-world objects.

Example: Using shadow maps in Unity or Unreal Engine, developers can simulate shadows from both real and virtual objects, improving the visual representation of occlusion.

✅ Best Practices for Handling Occlusion in MR Applications

Practice	Benefit
Integrate LiDAR or depth sensors	Provides accurate real-time depth information, improving occlusion accuracy.
Use SLAM for environment mapping	Ensures the system understands the layout of the physical world, allowing for better virtual object placement.
Implement raycasting for collision detection	Accurately simulates virtual-object occlusion based on real-world geometry.
Optimize object tracking and recognition algorithms	Enhances the system’s ability to detect real-world occluding objects and handle interactions realistically.
Simulate real-world lighting and shadows	Improves realism by creating correct shadow interactions between virtual and real objects.
Account for lighting conditions in the environment	Reduces errors due to reflections, shadows, and inconsistent lighting.

Real-World Example: AR Navigation App with Occlusion

Problem:

In an AR navigation app, a virtual arrow guiding users to a destination would pass through real-world obstacles like walls, doors, and furniture, causing confusion for the users.

Investigation:

The app relied on basic visual tracking but lacked accurate depth sensing and collision detection.
The virtual arrow was not occluded properly when the user moved around the environment.

Solution:

Integrated LiDAR depth sensing to map the environment accurately.
Implemented SLAM to track real-world objects and enable the system to calculate accurate occlusion.
Now, the virtual arrow stops behind walls and avoids passing through real-world obstacles.