Xiaomi Autonomous Driving: End-to-End Layout Started a Year Ago, Focus Shifts to Training Data

Two months after launch, Xiaomi introduced end-to-end parking. Seven months after launch, Xiaomi NOA became available nationwide. The more prominent end-to-end + VLM technology, which enables curb-to-curb assisted driving, was recently integrated into the Xiaomi SU7. We have already tested it in advance and expect formal deliveries soon.

The rapid deployment of autonomous driving features is underpinned by Xiaomi's early R&D layout and a development starting point that bypassed the rule-based era, riding the momentum of end-to-end + VLM to avoid many pitfalls. On November 14, the day before the Guangzhou Auto Show media day, Xiaomi Auto showcased its curb-to-curb autonomous driving capability for the first time, with Lei Jun driving live on stream. During the event, Ye Hangjun, General Manager of the Autonomous Driving Department, outlined two key goals for next year: mass-producing curb-to-curb capabilities and accumulating data.

Why these two goals? With this question and various inquiries following our first experience with Xiaomi's curb-to-curb autonomous driving, we held an in-depth discussion with the Xiaomi Auto autonomous driving team during a communication session.

During the discussion, the team highlighted the implementation of end-to-end + VLM technology. A year ago, Xiaomi's autonomous driving team began exploring end-to-end architectures. This year, they first deployed it in parking scenarios, then connected urban autonomous driving with parking to achieve curb-to-curb functionality. In the future, end-to-end updates will also be applied to highway autonomous driving.

The goal is set for this time next year; by then, the next generation of VLA (Vision-Language-Action) models based on VLM may take shape. Vision models will not only see but also provide action feedback after perception.

When discussing the potential of next-generation Transformer technologies, the autonomous driving team noted that the industry has not yet discovered a technological leap as significant as the transition from CNNs to Transformers. Therefore, the focus for the coming period remains on end-to-end systems.

During the nearly two-hour conversation, the autonomous driving team shared numerous details regarding Xiaomi's technology, team structure, and future development. We have transcribed the dialogue for written publication.

Q: Xiaomi's 'parking spot to parking spot' autonomous driving now utilizes an end-to-end architecture combined with a Vision-Language Model (VLM). When was the end-to-end project initiated?

A: In fact, we began experimenting with this approach around this time last year. End-to-end parking and mechanical garage space parking were among the earlier features to be deployed.

Q: Since there was already a plan to implement end-to-end systems early on, why did you first release a version capable of mapless nationwide driving?

A: Mapless driving and end-to-end architecture are not sequential steps; they represent two different dimensions of the same technology. This distinction may have been overlooked by market or product colleagues who believed that offering this capability to users at this stage would allow them to gather feedback. However, the research and development of both aspects have not been siloed.

Q: Alongside the end-to-end capabilities, VLM was also launched. What specific functions does VLM currently offer?

A: The alert functionality of VLM is currently in a stage suitable for productization. Its primary role is to recognize and interpret the complexities of the real world.

Voice announcements may seem novel today, but by tomorrow they might feel routine. However, if users hear them every day, they will start asking what the system can actually do for them. Therefore, applications based on Vision-Language Models (VLMs) will inevitably evolve from voice announcements to vehicle "actions." This is precisely the next generation of VLM: the Vision-Language-Action Model (VLA).

The transition from VLM to VLA can be broadly divided into three stages:

Currently, VLM capabilities are in the first stage. After sensors perceive the environment, the system alerts the driver through voice and text.

In the second stage, VLMs may handle protection or detour maneuvers for specific scenarios.

The third stage involves evolving into a VLA, where a single model directly generates trajectories (Actions).

Q: What is the R&D plan for Xiaomi's autonomous driving team next year?

A: Next year, Xiaomi's autonomous driving efforts will focus on two key areas. The first is end-to-end full-scenario parking-to-parking intelligent driving. The goal is to launch an internal beta version by the end of this year, targeting a user base of 1,000 people to form a testing group. By early next year, the aim is to roll out parking-to-parking functionality to all mass-produced vehicles as quickly as possible.

The second objective is to accumulate effective data and achieve significant breakthroughs in data over the course of one year, thereby maximizing the performance of the end-to-end system. Ultimately, this will transform intelligent driving from "usable" to "highly usable."

Q: One of Xiaomi's autonomous driving team's two goals for next year is data accumulation. How do you define high-quality data, and what proportion does it account for in all autonomous driving data?

A: This is actually very similar to the human learning process. For example, when people learn to drive, they first learn to drive straight, then learn to turn. Going from not knowing how to drive to knowing how requires a significant number of such 'positive' sample cases.

From being able to drive to becoming proficient, however, requires more 'negative' sample cases. These could be dangerous situations encountered while driving or hazardous weather conditions.

Therefore, high-quality data must include both 'positive' samples from the driving process and a large amount of 'negative' samples.

Regarding training for positive capabilities, only about 1% to 5% of the data is valuable.

For training negative capabilities, the proportion is even lower than this, with some data being rare and hard to come by. Therefore, to address these issues, we not only need to mine data but also perform data mining techniques, such as increasing the severity of danger in hazardous scenarios. Currently, pre-research work by the Xiaomi autonomous driving team has found that training with this type of data is highly effective.

Q: Autonomous driving has moved from the rule-based era to end-to-end models. Does this mean fewer R&D personnel are needed? How many people are required?

A: To use an imperfect analogy, previously, rules were written on the vehicle side; now, rules are written in the cloud and data is sourced. Essentially, the method of knowledge injection has changed. One benefit of this approach is that it is better suited for large-scale deployment.

In the former rule-based era, 20 people writing rules meant the code became unusable as rules conflicted with one another. But now, having 200 people work on data simultaneously causes no issues.

Therefore, current autonomous driving R&D does not mean experts or personnel are unnecessary; in fact, headcount may not decrease, but everyone has become a cloud-based expert.

Q: Have you seen any next-generation technology capable of disrupting Transformer?

A: Currently, there is no technology showing a leap as significant as the transition from CNN to Transformer. This will likely remain the case for the next one to two years. Pushing end-to-end systems to mass production is akin to how the industry took one to two years to achieve mass production for BEV+Transformer. Discussing more distant futures holds little meaning.

The entire industry is still actively exploring and experimenting, but recently there have been no particularly groundbreaking developments. What has recently sparked discussion is OpenAI's o1 model.

Q: Xiaomi Autonomous Driving started relatively late, avoiding the pitfalls of the rule-based era. Does jumping straight to end-to-end provide a greater advantage?

A: Xiaomi Autonomous Driving's first version was based on BEV+Transformer, giving it a strong late-mover advantage. Furthermore, Xiaomi Automotive relies on its parent group rather than starting from zero. Every company employs similar numbers of people; no one is inherently smarter than another, and everyone in this industry works diligently.

Q: How do you understand world models in the field of autonomous driving?

A: When humans act, their brains perform parallel simulations. For example, when driving encounters an obstacle, a person simultaneously evaluates multiple actions and their respective outcomes—such as 'swerve around,' 'wait in place,' or 'peek out to check'—effectively predicting various 'parallel timelines.' For autonomous driving, a similar engine is needed to predict how the vehicle's potential behaviors will impact the surrounding environment over the next 3–5 seconds; this engine is the world model.

This concept stems from reinforcement learning, with the greatest challenge lying in building an accurate world model. However, effective reinforcement learning presupposes that the world model is sufficiently realistic. This creates a chicken-and-egg dilemma: it remains difficult to construct a highly realistic world model today. A truly authentic world model resembles the scenario depicted in The Matrix. If the model lacks realism, its generated scenarios become hallucinations, providing incorrect data.

Q: We observe that the gap in autonomous driving features among manufacturers is narrowing. How will differentiation be demonstrated in the future?

A: Differences among manufacturers will primarily reflect the extent to which they meet users' genuine needs, rather than showcasing technology for its own sake without addressing core user requirements.

Q: Do you believe the rollout of end-to-end technology and nationwide driving capabilities has reached saturation across all competitors?

A: Not yet. Algorithm developers may speak conservatively, but we are only at the beginning. Achieving an experience that users genuinely find satisfactory requires another one to two years. The current state of the art resembles the early days when BEV (Bird's Eye View) architectures first emerged.

Q: Xiaomi Autonomous Driving's current goal is to enter the industry's first tier. How does the company internally define 'first tier'?

A: Evaluating autonomous driving is multidimensional; we place significant emphasis on real-world usage metrics. These include the frequency of driver takeovers and user activity levels.

Q: In the current point-to-point autonomous driving test drive, we find that acceleration from a standstill at traffic lights is relatively slow. How do you view this issue?

A: Xiaomi’s autonomous driving development has been quite rapid. However, since the entire system is still undergoing optimization before mass production and delivery, there is room for improvement. Delays in this specific scenario may be more pronounced, but corresponding engineering optimizations are currently underway.

Q: Which city does Xiaomi consider to present the greatest challenge for autonomous driving?

A: On one hand, it is due to difficult geographical environments, such as Chongqing. On the other hand, it is because of significant differences in traffic infrastructure, such as variations in lane configurations and traffic light positions in certain cities.

Xiaomi will not optimize for different cities or distinct driving habits; the goal is ultimately to become a "super driver."

By James Edited by Daji