BMW Augmented Reality Ride: The Future of In-Car Driving and Passenger Experiences

Updated: April 21 2025 16:23
While most car manufacturers focus on improving displays and digital dashboards, BMW's research team has been quietly working on something much more ambitious: fully functional AR glasses that transform how we experience driving and riding in cars. Let's dive into this fascinating innovation and its potential to revolutionize our automotive future.




BMW's exploration of augmented reality for vehicles didn't happen overnight. What began as a confidential research project more than 15 years ago has evolved into one of the most promising technological advancements in automotive history. Led by Tobias, an augmented reality software engineer at BMW Research, the team has been tackling a fundamental question: What if lightweight AR glasses become an everyday device like smartphones? How would that change our relationship with cars?

The journey began around 2008, just after BMW had introduced head-up displays in their vehicles. The research team started by developing their own AR headset prototypes, as commercial options like Google Glass weren't available until years later (2012). In 2015, BMW first revealed a glimpse of their AR vision through Mini (part of the BMW Group), showcasing potential applications at Auto Shanghai. However, this was just a stationary demonstration, not yet functional in moving vehicles.


The breakthrough came in 2019 when the team finally overcame the massive technical challenge of tracking AR content within a moving vehicle. By 2021, they had progressed to implementing world-locked tracking (allowing virtual objects to appear fixed to real-world locations), and in January 2024, BMW unveiled its fully functional "Augmented Reality Ride" concept at CES in Las Vegas—the world's first immersive AR glasses experience in a moving car.



To appreciate why BMW is pursuing AR integration, we need to understand how drastically car interiors have changed over the decades. Tobias highlighted this evolution by comparing the BMW 1500 from decades past with modern models.

The BMW 1500 featured a purely driver-centric interior—a steering wheel, clock, and tank level indicator—with nothing to distract from the driving task. Fast forward 40 years to the BMW M5, and the transformation is striking. Modern car interiors incorporate extensive technology to enhance safety, comfort, and enjoyment, with displays increasingly dominating the dashboard.


The head-up display (HUD) represented a significant advance when BMW introduced it over 20 years ago. Borrowed from aviation technology, it allows drivers to view critical information directly in their line of sight, reducing distraction as drivers can keep their eyes on the road. Today's BMW 5 Series features even larger, full-color HUDs showing navigation instructions and speed limit information.

However, despite improvements, traditional HUDs still have significant limitations—most notably their restricted field of view. No current technology can span the entire windshield, limiting the potential for truly immersive information display. This is where AR glasses present a game-changing opportunity.



According to BMW's research, AR glasses offer several compelling advantages over traditional in-car displays:

• Expanded field of view: Unlike fixed HUDs, AR glasses provide a massive, movable field of view that extends not just forward but to the sides as well
• Complementary technology: AR glasses won't replace HUDs but extend their capabilities, creating a seamless extension from the limited HUD area to the broader spatial environment
• Personalized content: Different passengers can enjoy different content simultaneously—navigation for the driver, work materials for one passenger, and entertainment for others
• Contextual placement: Information appears exactly where it's needed, not just on a fixed display
• Enhanced safety: Critical information appears directly in the driver's line of sight, reducing cognitive load
• True immersion: Entertainment and informational experiences can be truly immersive in ways traditional screens cannot match

Perhaps most intriguingly, AR glasses enable shared experiences while respecting individual preferences. BMW envisions scenarios where family members might initially use their glasses for different purposes (work, gaming, videos) but can then participate in shared experiences like watching movies together or playing multiplayer games—all while traveling in the same vehicle.


When BMW first approached AR glasses manufacturers around 2012-2013 with their vision for in-car use cases, they received disappointing news: existing technology simply couldn't support AR glasses in moving vehicles. This fundamental technical challenge became the focus of BMW's research for years to come.


Understanding why AR glasses fail in moving cars requires knowledge of how they normally function. AR glasses rely on a tracking-rendering pipeline to maintain the illusion that virtual objects exist in the real world. This process involves:

1. Sensors capturing head motion
2. Tracking algorithms calculating the device's position/orientation
3. Rendering the appropriate imagery
4. Displaying the rendered content

For immersion, this entire pipeline must complete in under 20 milliseconds (the "motion-to-photon latency"). Most AR glasses achieve this through sensor fusion, combining:

• Camera-based tracking: Outward-facing cameras observe the environment, but processing the images is computationally expensive and relatively slow (10+ milliseconds)
• IMU-based tracking: Inertial measurement units provide rapid acceleration and angular velocity data

The fundamental problem in moving vehicles is that these systems cannot distinguish between head motion and car motion. The IMU sensors detect all acceleration—whether from nodding your head or the car accelerating. Similarly, tracking cameras see both the car's static interior and the rapidly moving outside world, creating contradictory inputs.


Night driving presents additional challenges, as there's often insufficient illumination for the tracking cameras to function properly inside the vehicle. Even leading consumer AR/MR devices like Apple Vision Pro, Microsoft HoloLens, and Meta Quest Pro struggle in moving vehicles, despite their dedicated "travel" or "moving platform" modes designed primarily for slower-moving environments like airplanes and trains.

The challenge becomes even more complex when trying to place virtual objects in the real world outside the car (what BMW calls "world-locked assets"). This requires not just knowing where the glasses are relative to the car, but the car's precise position in the world.



After years of research, BMW developed an innovative approach to solve these complex tracking problems. Their solution involves:

1. Using driver-monitoring cameras already installed in BMW vehicles (with infrared illumination that works even at night)
2. Building a complete motion capture system within the car
3. Initially tracking markers on the glasses, similar to film industry motion capture
4. Eventually training neural networks to track unmarked glasses, using millions of training images
5. Creating a platform-independent C++ sensor fusion library that combines: IMU data from the glasses, vision tracking data from the car's cameras, car motion data, and high-accuracy GPS data

This breakthrough system allows the AR glasses to accurately distinguish between head movements and vehicle movements, correctly placing both vehicle-locked content (virtual objects that appear fixed within the car) and world-locked content (virtual objects that appear fixed to locations in the real world, like navigation arrows on specific roads).


The current implementation uses lightweight glasses from Xreal (formerly Nreal) connected to a smartphone, with Unity as the development platform. BMW specifically chose Unity for its:

• Ability to bring 3D scenes into cars easily
• Cross-platform compatibility
• Designer-friendly interface
• Rich asset ecosystem for rapid prototyping
• Low barrier to entry for new developers

BMW's AR implementation enables a range of innovative use cases for both drivers and passengers:

For Drivers:

• Immersive navigation: Navigation instructions appear directly on the road, with lane-precise guidance that shows exactly where to go
• Safety alerts: The system can highlight potential dangers like pedestrians crossing or upcoming hazards
• Traffic sign visualization: Speed limits and other important signs can be highlighted to ensure drivers don't miss critical information
• Enhanced parking assistance: Building on BMW's existing Parking Distance Control, AR visualizations show exactly how much space remains around the vehicle, with virtual "tiles" that appear both on the dashboard and in the real world around the car (with proper occlusion behind physical objects)

For Passengers:

• Immersive gaming: Simple but engaging games like "coin collector," where virtual coins appear along the road for collection
• Personal entertainment: Video screens can appear virtually positioned above the sun visor for comfortable viewing
• Work space: Multiple virtual screens for productivity without physical limitations

All these experiences are authentically immersive, with 3D content properly anchored to the real world rather than simply floating in space. The system accounts for occlusion (virtual objects being hidden by real objects when appropriate) and maintains proper positioning even during complex driving maneuvers.



While BMW's AR Ride concept is fully functional, several challenges remain before widespread adoption becomes possible:

Hardware Evolution: Current glasses used in the prototype weigh about 70 grams—significantly lighter than mixed reality headsets (600g) but still heavier than ideal. BMW hopes to see glasses evolve to below 50 grams, approaching the form factor of sunglasses.

Ecosystem Development: Just as smartphones connect seamlessly to vehicles through systems like Apple CarPlay or Android Auto, BMW envisions a future where consumers' personal AR glasses will integrate smoothly with their vehicles. This requires collaboration between automotive manufacturers, glasses manufacturers, and software developers.

Legal and Safety Considerations: Regulatory frameworks for driver use of AR glasses are still evolving. BMW is conducting extensive driver studies to ensure their implementation doesn't distract drivers and actively enhances safety rather than compromising it.

While BMW isn't planning to manufacture AR glasses themselves, they're establishing the technological foundation to integrate future consumer AR glasses into their vehicles. Their working prototype demonstrates the immense potential of this technology to transform how we experience travel.