Since the first car equipped with a radio was introduced in 1923, the pursuit of in-car entertainment has never stopped. Innovations in in-car entertainment usually follow the progression of electronic entertainment device technology.
Nowadays, large screens have become the mainstream configuration for new cars. Car companies’ next step is to explore VR and AR. First, what are VR and AR? VR stands for Virtual Reality, while AR stands for Augmented Reality.
In layman’s terms, VR is like recreating a completely virtual environment. With VR glasses, it is possible to become completely immersed in a visual landscape. AR, on the other hand, superimposes virtual images onto the real environment through transparent or semi-transparent glasses or glass. Today, we will talk about the former.
Why do we need VR in the car?
During NIO ET5’s release last year, it was announced that NIO would collaborate with NOLO to create NIO VR Glasses to enrich the passenger experience.
Similarly, Audi just recently announced that it will launch a new car equipped with the MIB 3 information entertainment system in June this year. This system will allow back seat passengers to wear VR glasses to experience games, movies, and other interactive content.
Clearly, in-car VR has become one of the highlights that car companies use to pursue differentiated competition and enhance the user experience. But why is it specifically VR?
During long-distance travel, the boredom of travel plus the limited space inside the car creates a need for in-car entertainment systems. From radios to CD players, from satellite television to streaming media, the evolution has been from sound transmissions to visual transmissions. On the foundation of 2D images, VR has created a 3D virtual space, offering a more unique immersive experience. If done well, it can effectively alleviate the fatigue and exhaustion of long car journeys by completely isolating passengers from the noisy road.
How is the in-car headset positioned?
The vast majority of VR devices were originally developed for indoor use, which means now they will face a significant problem when they are ported to the car: how to solve the headset’s drift caused by the car’s motion.To understand how to solve the position drift issue during driving, let’s first understand why VR headsets need positioning. In order to reduce the dizziness caused by VR use, the images in the VR headset need to move in response to the movement of our head, just like in the real world.
To achieve this, our head movement needs to be accurately tracked. In order to achieve more realistic interactions, hand movements also need to be tracked, which gave rise to the development of VR headset and controller positioning technology.
Currently, there are two mainstream positioning methods, with the first being the Outside-In positioning technology represented by HTC VIVE.
HTC’s Lighthouse technology relies on two laser emitters placed diagonally to ensure that most parts of the space are not occluded. Each laser emitter contains a rotating motor that drives the emitting component to emit a V-shaped laser to scan the positioning space.
The VIVE series of headsets and controllers have more than 70 photosensitive sensors, which calculate the sensor position relative to the laser emitter’s accurate position by calculating the time of receiving the laser. The position and direction of the headset can be detected by multiple photosensitive sensors.
The Outside-In positioning technology has high accuracy, can be processed in a distributed manner, has almost no delay, and is not afraid of occlusion. Even in a car, the controller can still be captured even if it is placed on the seat back or under the seat, making laser positioning technology capable of achieving high-precision, high-reactivity positioning while avoiding the disadvantages of high complexity and cost, slow processing speed, and susceptibility to natural light and shadow.
And Oculus Quest 2 uses Inside-Out tracking technology. It utilizes computer vision-based SLAM (Simultaneous Localization And Mapping) technology and custom algorithms to generate real-time 3D maps of the surrounding environment, calculate the user’s position within the map, and then translate it into the VR image.
Inside-Out tracking technology doesn’t require additional tracking stations and is lower-cost, but it’s limited by the angle of the headset’s cameras and cannot track areas that are not covered by the cameras (e.g. the back of a user’s head) as well as Outside-In tracking can.
It is reported that NIO VR Glasses used in NIO ET5 adopt Inside-Out tracking, enabling sub-millimeter-level spatial tracking with 6 degrees of freedom.
In terms of implementation effects and implementation difficulty, the Outside-In tracking technology is undoubtedly more suitable for use in cars, but it also requires VR function usage area planning and transmitter and wiring arrangements, as well as good computing power allocation from the beginning of vehicle development. This undoubtedly increases the production cost, and the risk is too great when it is still unclear whether consumers are willing to pay for this function.
Inside-Out tracking technology has lower production costs, but its use in the vehicle requires higher algorithm requirements, such as distinguishing the landscape range inside and outside the car and correcting for gyroscope errors caused by vehicle movement. If these issues can be solved, VR with Inside-Out tracking could be an optional feature installed in more vehicles.
Is it Powerful Enough?
To play AAA-class games in a car VR system, another important hardware foundation is the performance of the cockpit chip, in addition to resolving the issue of headset tracking.
In the cockpit entertainment system, the chip is the brain of the entire system. Its computing power determines the response speed and upper limit of the interactive experience. The same is true for VR. The high or low image rendering ability of the chip will directly affect the display effect and response speed.
Compared to mobile phones, cars have the conditions to carry stronger computing devices and better acoustics. After completing the new four modernizations, a substantial increase in the performance of the cockpit chip added the possibility of rendering more complex images.
“`markdown
Using current high-end cabin chip Qualcomm 8155 as an example, its GPU adopts Adreno 640, with a frequency more than 100 MHz higher than that of Snapdragon 855 chip on mobile phones, approaching Snapdragon 855+. However, it is still limited in facing the demand of 4K resolution and 120 Hz refresh rate for VR applications, and can only handle some lightweight entertainment tasks.
Currently, the most powerful cabin chip is Tesla MCU3, which adopts a 4-core AMD Ryzen chip consisting of 12nm Zen+/Navi 23 and 8 GB RAM for its MCU 3 control platform. The performance is comparable to that of mainstream game consoles, fully capable of supporting VR headset’s performance requirements for large-scale 3A games.
What can we do to mitigate car sickness?
Many consumers are worried that wearing VR glasses while riding in a car may worsen car sickness. The reason why we usually get car sickness is that our body feels the movement, but our vision is restricted to the inside of the car, which tricks our brain into thinking we’re not moving. This inconsistency between two types of information confuses our brain and causes car sickness.
In other words, whether or not you feel car sick mainly depends on whether the feelings of your body and what you see are consistent. If we can map the movement of the vehicle to the virtual space of VR technology, so that the mismatch between what you see and your actual perception can be alleviated to some extent, car sickness can be mitigated.
To achieve this goal, virtual content needs to be matched with data such as vehicle speed and steering. As early as CES 2019, car manufacturer Audi and VR brand Holoride jointly created a VR experience project named “Avengers: Rocket Rescue”.
The uniqueness of this experience lies in the fact that every vehicle action in the Audi e-tron modified by Holoride can be mirrored in real-time in the virtual spaceship in VR scene. When the car accelerates, the spaceship also accelerates; when the car turns left, the spaceship also dodges obstacles accordingly. In this way, car sickness caused by wearing VR glasses can be reduced.
The ultimate goal of VR is the third space.
“`#### Headset Positioning and Computing Performance Remains the Bane of In-car VR Development
While the challenges of headset positioning and computing performance pose significant obstacles to the progress of in-car VR development, indoor VR, which has already solved these two issues, still has a long way to go in terms of popularity. For just over 2,000 yuan, you can now purchase an Oculus Quest 2 with good display, positioning accuracy, and scalability, but why is VR still considered a niche gadget for geeks?
The problem may lie in the content. Since the birth of VR, we have seen groundbreaking experience-changing software like Tilt Brush, and hit games like Beat Saber and Half-Life Alyx. However, the vast majority of apps still simply mirror the flat-screen experience as a larger, more dimensional interface.
The biggest advantage of a screen over paper is the limitless expansion of two-dimensional space, and the biggest advantage of VR over flat-screen displays is the infinite expansion of three-dimensional space.
As the intelligence of cars continues to improve, cars are no longer just a means of transportation, but are gradually becoming the third space of our lives. In terms of the development direction of intelligent cockpits, the industry is slowly moving away from the past approach of making larger screens and porting several mobile apps to viewing cars as space for contemplation.
Jiang Weizhong, deputy general manager of product planning at Xpeng Motors, admitted that there is currently no consistent conception of the ultimate form of an intelligent car, but the transformation of the automobile cabin into the third smart space is an inevitable trend.
With current technology, we cannot accurately define the cars of the future, but we can be certain that once automated driving is implemented, cars will become completely new vehicles. Efficient autonomous driving will free up the driver’s and passenger’s attention, and the car cabin will take on a truly mobile third space identity, empowered by its ability to move.
As the width of the road is limited, it’s impossible to infinitely extend the space in the car cabin. Therefore, VR provides a good solution for exploring this confined space. In virtual reality, the car is both a commuter and a mobile office, cinema, and game room. The physical space of a few square meters in the car cabin gains infinite space scalability and can freely switch between different experiences in the aid of VR.Bring your attention back to the present, in the foreseeable future, there will likely be a large number of cars with VR technology on the market. However, based on the current advertising from automotive companies, it seems that there is no precise plan for integrating VR into vehicles, and it is seen more as a marketing gimmick. What automakers should consider is how to truly leverage the advantages of VR devices, which can infinitely expand virtual spaces, rather than simply treating it as a larger screen.
This article is a translation by ChatGPT of a Chinese report from 42HOW. If you have any questions about it, please email bd@42how.com.