Waymo Is No Longer the Best Autonomous Driving Enterprise in 2021, According to DMV Report

California Department of Motor Vehicles (DMV) has recently released the 2021 Autonomous Vehicle Disengagement Report with fresh data about a year’s autonomous driving tests, provided by various autonomous driving players.

The highly-anticipated disengagement rate (MPI), which evaluates the proficiency of autonomous driving technology, is one of the most significant indicators covered in the report.

Unsurprisingly, Chinese and American autonomous driving companies still dominate the top performers in the report.

Five out of the top 10 autonomous driving companies ranked in the report are from China.

Auto X, the Chinese autonomous driving player, unexpectedly dethroned previous champion Waymo, with the MPI of 50,108 miles and only one reported disengagement.

Criticism immediately follows the report’s release, with some autonomous driving company founders even come forward to state:

” The report is still being compiled on a question-and-answer basis. Ranking is not as practical as a real-life comparison.”

But what do the numbers say?

On February 10th, DMV officially released the 2021 Autonomous Vehicle Disengagement Summary Report, including detailed insights into a year’s worth of autonomous driving tests, such as testing distance, fleet size, and the human intervention rate, covering every aspect of autonomous driving performance.

MPI, the disengagement rate based on the testing distance and reported disengagements, which reflects the level of autonomous driving expertise, is also reported.

The higher the MPI distance, the lower the disengagement rate, and the better the level of autonomous driving is.

As expected, autonomous driving players from China and the United States dominate the top ten positions of MPI rankings.

Three out of the top five companies are Chinese autonomous driving companies, including Auto X, Didi Autonomous Driving, and WeRide.

Auto X impressively achieved a record-high MPI of 50,108 miles in the report and claimed the top spot, ending Waymo’s five-year winning streak.

Cruise, under General Motors, stands firmly in second place, achieving an MPI of 41,719 miles.

Didi ranks third with an MPI of 40,744 miles, and Argo ranks fourth with an MPI of 36,733 miles. WeRide ranks fifth with an MPI of 19,322 miles.The companies YF-Link, Pony.ai, Waymo, ZOOX, and Nuro rank from 6th to 10th place in terms of the miles per intervention (MPI) metric, with YF-Link and Pony.ai both breaking the 10,000-mile barrier while the latter three remain at the level of thousands of miles.

Regarding fleet size, both Waymo and Cruise have surpassed 100 vehicles, with Waymo deploying a staggering 693 test vehicles in California last year, far surpassing other players.

Total testing mileage is proportional to fleet size, and Waymo and Cruise also lead with absolute advantage in this regard. Waymo surpassed 2 million miles of testing last year, the only player with an annual testing mileage over a million miles, while Cruise also passed 800,000 miles of testing.

Overall, the top 10 players’ ranking has remained unchanged compared to last year. Nevertheless, the significant ranking changes of individual players within this short timeframe catch people off guard.

AutoX, ranked 3rd in 2020, saw its MPI performance exceed an impressive 20,000 miles and increase by over two times to over 50,000 miles within a year.

Didi has also achieved breakthrough growth in 2021, climbing straight to the third place from 7th in 2020, with MPI exceeding 40,000 miles, an increase of nearly eight times compared to the previous year when it had just passed the 5,000-mile threshold.

If there were no other variables, AutoX and Didi’s significant progress in autonomous driving technology this year is truly breathtaking.

There is also Waymo, which has experienced ups and downs. It held the top spot for five consecutive years and approached 30,000 miles of MPI in 2020 but plummeted to the eighth place with less than 8,000 miles in 2021.

According to normal logic, from a technical perspective, whether it’s algorithms or hardware, autonomous driving is getting better and better, and there is no such thing as going backwards.

Even if there is a setback, even if one accepts a wait-and-see attitude, it shouldn’t lead to such a drastic drop.

The contrast is truly unimaginable.As expected, the report was immediately questioned by industry experts. The critic, who is none other than the founder and CEO of Wenyuan Zhixing, Han Xu, suggested that the industry should hold an Olympic competition for autonomous driving. Everyone could compete against each other and see who ranked first.

The term “self-answering” can be seen as questioning the foundation of this autonomous driving report- objective reality.

Even Wenyuan Zhixing, founded by Han Xu, was on the report, and its data performance even ranked at the top. However, this should have been an opportunity for them to capitalize on the hot topic and create a buzz, right? Why did they choose to question it?

The controversy over California’s autonomous driving industry has been ongoing. California, as the birthplace of global autonomous vehicle testing, combined with the official background of the report, was initially considered as an authoritative report by the industry for measuring the level of autonomous driving.

But with the continuous advancement of autonomous driving testing, especially in the last two years, industry critics have become increasingly prominent.

For example, Hou Xiaodi, co-founder of TuSimple, publicly criticised the California autonomous driving report that it could not be considered as a reference for technical evaluation. Instead, it can only reflect the ethical standards of autonomous driving companies.

Chris Urmson, co-founder of Aurora, Kyle Vogt, co-founder of Cruise, and Han Xu, founder of Wenyuan Zhixing were also among those who expressed similar opinions.

As for the core of the criticism, there are primarily two aspects:

First, the generation of the Miles Per Intervention (MPI) score includes subjective factors. This is mainly due to the statistical problems of one of the MPI’s deduction bases – the disengagement frequency.

Disengagement refers to the situation where the autonomous driving system cannot handle an unexpected problem and the vehicle is taken over by humans. Literally, the more frequent the takeover, the shorter the time the autonomous driving system did its job, and the MPI score will reflect a lower level of autonomous driving technology.

However, the problem is that the factors leading to disengagement are complex, and the standards for self-recording by each company are different. It is not simply determined by the level of technical competence alone. For example, conservative companies will take over if autonomous driving violates road rules, while more aggressive companies will only take over just before potentially serious accidents happen.Obviously, radical styles result in increased disengagements and decreased MPI mileage, but this does not necessarily indicate poor technology for the companies with such styles.

There is also the uncertainty brought by road conditions.

On highways with few vehicles and simple road conditions, disengagements are infrequent, whereas there are many disengagements on urban and complex roads with heavy traffic.

However, as Chris Urmson has said, “the disengagement rates for driving one hundred million miles on a flat, dry area with no cars, no people, and no intersections cannot be compared with those for driving one hundred miles in cities like Pittsburgh with complex road conditions.”

Moreover, disengagement data generated by different road conditions are indiscriminately incorporated into the statistical process in the California DMV Autonomous Vehicle Testing Regulations, and thus the MPI data produced in this way cannot truly reflect the real level of autonomous driving.

Secondly, there are moral factors in the MPI statistical process.

The data in a company’s autonomous driving report is an active behavior, and the California DMV cannot control whether the data is truthful, and whether there is any concealment or underreporting. Therefore, it is only up to the testing subject to behave conscientiously.

If somebody wants the MPI mileage data to look good, they can underreport the number of disengagements, or find a road with simple conditions and few people in the suburbs, and run dozens of tests a day because the disengagement counting criteria does not make any difference.

That is why Hou Xiaodi bluntly stated that the California Autonomous Vehicle Test Reports can only evaluate the moral standards of autonomous driving companies.

Since MPI cannot objectively evaluate autonomous driving companies, are there any other standards for reference?

From testing to online operation, and then to realization of monetization, autonomous driving ultimately cannot do without commercialization.

Therefore, evaluating the value of autonomous driving companies, including technology value, based on the speed of commercialization landing is the best way.

Currently, autonomous driving players who have achieved commercialization landing globally are few and far between, except for specific segments such as low-speed logistics vehicles or closed venues.

On California’s ground, according to information from the California DMV, only Waymo and Cruise have obtained commercial testing licenses.

Turning back to China, the companies that have obtained formal commercial testing licenses are Baidu Apollo and Xiaoma Zhixing.

In addition, Wenyuan Zhixing and AutoX are conducting regular road tests and operations in Guangzhou and Shenzhen.

One more thingAccording to the official data from DMV, about 50 companies have been granted permission for autonomous driving testing in California up to now.

Currently, 24 companies are still conducting autonomous driving road tests and reporting data, which is 5 fewer compared to 2020.

Among them, 7 companies have been approved to conduct unmanned autonomous driving tests without safety drivers.

3 of them are domestic companies, Baidu Apollo, Auto X, and Pony.ai.

This autonomous driving report released by California DMV includes not only absolute data such as taking-over times and test mileage, but also the reasons for taking-over of all testing vehicles.

If you are interested, you can study it in detail.

However, in general, the reference value of this report has become weaker and weaker with the deepening of the implementation of autonomous driving.

On the one hand, the subjectivity is indeed too strong, and the autonomous driving leader Waymo is terrifyingly “lagging behind”.

On the other hand, practical application and commercialization are more important than theory.

What’s your opinion?

—End—

This article is a translation by ChatGPT of a Chinese report from 42HOW. If you have any questions about it, please email bd@42how.com.