Automatic Driving Simulation Testing: An Important Step for Commercializing Autonomous Vehicles

Author: Chen Kangcheng

Simulation testing is a crucial step in the commercialization of autonomous vehicles. The progress of simulation testing technology will determine the timeline for the deployment of autonomous driving technology.

This article is the fourth in the “Nine Chapters of Intelligent Driving” simulation series. The first three are “Understanding the Current State of Autonomous Driving Simulation Testing Technology in One Article“, “Analysis of Information on Startups in the Autonomous Driving Simulation Field“, and “Understanding Autonomous Driving Simulation Testing Scenarios and Scene Libraries in One Article“. The importance of simulation testing for autonomous driving is self-evident. I write these articles in the hope of stimulating others’ interest in and attention to the field of autonomous driving simulation testing.

The biggest challenge for autonomous driving systems at present is the difficulty of implementing them in practice. This is because the safety issues of autonomous driving systems have not been properly addressed. Addressing system safety issues requires a large amount of virtual simulation testing and real-world road testing. Simulation testing can greatly accelerate the testing and validation process of the system. Therefore, how to efficiently and reliably evaluate autonomous driving through simulation testing is the key to the commercialization of autonomous driving systems.

Next, let’s talk about the two major pain points of current autonomous driving simulation testing: the confidence level of the simulation testing and the coverage of simulation testing scenarios.

Confidence Level Uncertain

Confidence level of simulation software itself

Although many simulation platforms now have the ability to model sensors, vehicle dynamics, and traffic scenes, most simulation models are built under ideal conditions. There are no specific, quantifiable indicators for evaluating the confidence level of the results simulated by the simulator.

Taking simulated lidar as an example, the reflectivity of lidar is related to the distance of obstacles, the angle of the laser emission, and the physical material of the obstacles themselves. Lidar has a large detection range, emits a dense laser beam, and is influenced by multiple reflections and occlusions in the environment, making it difficult to simulate the reflected laser signal realistically. Most existing lidar models directly calculate the echo signal based on the laser reflectivity of each physical material. Such calculations inevitably lead to certain errors compared to the real-world echo signal.If we consider issues such as radar noise caused by sensor hardware or software, as well as environmental factors like rain, snow, water droplets, dust, etc. that impact radar performance, we see that these problems or phenomena become even more complicated to handle in simulation, especially for LIDAR simulation.

Confidence Issues with Simulation Reproduction and Generalization of Scenarios

Currently, the commonly used approach is to use a simulation generator to reproduce and generalize more virtual simulation testing scenarios based on real-world data. However, it remains unclear how well these testing scenarios match the actual environment.

What is scenario generalization? It is the process of extracting features and labels from the collected real-world scenario data, and then artificially recreating or deducing more reasonable scenarios based on the associated relationships between these scenario features and artificial experience. The end goal is to develop even more viable scenarios.

There are two issues with the generalization of actual scenarios:

First, it remains unclear whether the direction of generalization aligns with statistical meaning and testing needs. As it stands, self-driving companies have yet to establish statistical mapping between simulated scenarios and the real world.

Second, generalization loses some of its realism. For instance, in the case of dense traffic flow, if one change is made to a vehicle parameter, it will inevitably affect and spread to other vehicle parameters, ultimately making it difficult to reproduce this scenario through current generalization technology.

There will necessarily be differences between reproduction and generalization of virtual simulation environments and the real environment. These differences have the potential to significantly impact the testing results. Currently, however, there are no specific or quantifiable KPI indicators to evaluate the reliability of these testing scenarios.

Confidence Issues with Testing Result Evaluation Standards

Currently, traditional testing approaches still rely on hardware testing (including HIL hardware in the loop and VIL vehicle in the loop) or real-world road testing. Survey results show that for pure virtual simulation testing (such as MIL model in loop / SIL software in loop), many customers believe that the verified data has not met the high reliability requirements.

As one simulation professional once said in an article, “In self-driving simulation, it is very difficult to have a ‘parameter calibration’ process because ‘real experiments’ are too dangerous for safety supervisors to conduct and are difficult to execute. Therefore, it is difficult to adjust simulation parameters, and without calibrated parameters, it is hard to predict realistic outcomes, resembling a vicious circle. Without calibrated parameters, it’s hard to predict realistic results, and pure empirical simulation is hardly convincing or reliable.”Is it true that due to the unconnected feedback loop of “experiment-simulation-experiment”, it is difficult to judge the quality of the results of autonomous driving simulation tests, and it is inevitable that customers will have doubts about the authenticity of simulation results. So, how can we improve the confidence level of autonomous driving simulation testing systems? Although there is no perfect solution available yet, related companies have begun to adopt different approaches to enhance the confidence level of their simulation systems.

Tencent’s autonomous driving simulation platform, TAD-Sim, employs a hybrid game rendering and real data-driven method. It trains traffic flow AI models using large amounts of real road data, and then combines the game rendering engine technology to automatically construct interactive and realistic test scenarios.

Baidu uses an augmented reality-based autonomous driving simulation system – AADS. By using laser radar and high-definition cameras mounted on vehicles to scan street views, it acquires static scene images around the vehicle and dynamic trajectory data of traffic flow movement. Using these materials, its system applies augmented reality technology to create highly realistic simulation images directly, making the virtual scene created more close to the real scene.

51WORLD uses digital twin testing technology to increase the confidence level of simulation test results. That is, they use digital twin technology to construct a virtual scene model that is consistent with the real scene. During the real road test process, the real-time status data of the vehicle is mapped to the virtual scene in real-time through a vehicle-in-loop method. At the same time, the test data and evaluation results of the virtual scene are also fed back to the real world as an important basis for guiding and optimizing the decision-making of real vehicles in the real world.

Scene Coverage

“Corner Case” is Difficult to Exhaustively Cover

Now, another major pain point of autonomous driving simulation testing is how to build a high coverage level scene library (covering almost all “Corner Cases”). If such a perfect scene library exists, the system only needs to verify all the test cases covered in the scene library, and after meeting the requirements, it can reach the standard.

For autonomous driving simulation testing, the biggest challenge is to collect all the corner cases to cover different road environments, weather conditions, and traffic situations, which is almost impossible to achieve. In terms of collecting autonomous driving corner cases, Waymo has a longer mileage in real car road testing, and statistically, it encounters more corner cases (as of 2020, the accumulated mileage of simulation tests in their system exceeds 15 billion miles, and the accumulated mileage of actual road tests exceeds 20 million miles). Even so, its engineers constantly encounter new and endless long-tail scenarios to solve.The paper “Corner Cases for Visual Perception in Automated Driving: Some Guidance on Detection Approaches” analyzes the corner cases in visual perception for automated driving and proposes some methodologies. Based on the complexity of detecting “corner cases”, the paper divides them into five levels from easy to difficult:

a. Pixel level: caused by data errors, such as dirt on the windshield obstructing part of the camera’s field of view or glare from oncoming headlights at night, resulting in errors in the data collected by the camera.

b. Domain level: a whole offset in the observation of the world expressed by data, such as scenes covered in white snow during winter.

c. Object level: the presence of instances never seen before in the data, such as a camel on the road in a residential area.

d. Scene level: inconsistent scene patterns with expectations in a single frame of data, such as a tree fallen in the middle of the road after a strong windstorm or familiar objects appearing in an unusual location.

e. Scenario level: inconsistent scene patterns with expectations in a continuous stream of data, such as a pedestrian suddenly appearing in front of a stationary car, known as the “ghost head”.

Different methods need to be applied to different levels of corner cases. For relatively simple scene types in the first three levels, i.e., pixel level, domain level, and object level, feature elements can be extracted and then grouped into parameters to build a system. However, for the more complex levels of scene level and scenario level, which also have a large number of instances, it is difficult to enumerate them all completely. Therefore, the best solution is to further improve the system’s perception capability by evolving it from “perception” to “cognition” through deep learning methods, enabling the system to possess knowledge reasoning and generalization capability similar to that of humans.

Machine learning is an effective tool to solve the long tail problem of autonomous driving. The closed-loop process including data collection, labeling, training, and deployment on the vehicle can be achieved through machine learning technology, resulting in the continuous accumulation of cases and model improvement. However, machine learning models cannot solve all problems. To make up for the shortcomings of machine learning, a mixed system combining machine learning and non-machine learning can be used, where expert systems are employed.## Regional Characteristics of “Corner Cases”

The safety of using self-driving cars is undoubtedly the primary consideration for consumers. Similarly, for automakers, the ability to test vehicles to handle various “Corner Cases” is essential to ensure the safety of mass-produced vehicles on the road.

a. Different road environments: Different countries have varying road design standards, such as different road structures, traffic signs, and signal lights.

b. Different traffic conditions: In Chinese urban roads, mixed traffic of people and cars is common, and scenes of express delivery drivers and delivery riders driving alongside motor vehicles can be seen everywhere. However, this type of traffic scene is relatively rare in some countries in North America and Europe.

c. Different traffic rules: In China, red lights usually mean right turns are allowed, even at intersections without guiding lanes. However, in Germany, right turns are not allowed by default. Additionally, there are speed limits. The speed limit on highways in China is 120 km/h, but many roads in Germany have no speed limit, and driving at 200 km/h on such roads is normal. However, if someone drives like this in China, it is not only dangerous, but it may also harm innocent people.

d. Different driving habits: In Germany, the concept of the right of way is more strictly followed, and people generally drive according to the logic of the right of way. However, there is almost no such logic in Chinese traffic rules. At an intersection without traffic lights, especially at night, it is usually who dares to go first. There is also a bad phenomenon in China, which is that people tend to cut in line, especially when the traffic is heavy. The two cars on the left and right of yours may cut in front of you without using turn signals like ghosts.

Due to the differences in the driving environments of different countries and regions, the test scenarios’ data has strong regional characteristics. The extreme working conditions faced by test vehicles will also differ significantly in terms of content and quantity. Suppose an automated driving system that is entirely safe to test in the United States. In that case, there will inevitably be “Corner Cases” that have not been encountered before when it is tested in a country with a more complex traffic environment like China, and the safety of the vehicle will not be guaranteed.

If a company’s automated driving system wants to be commercialized and mass-produced in a country, it must first pass a local test scenario evaluation, which ensures that it can handle all the extreme working conditions locally.However, Chinese domestic simulation companies have the first-mover advantage of being “near the water platform” and are more likely to design and develop simulation testing software suitable for Chinese autonomous driving solutions. Firstly, they have a better understanding of the domestic road environment, traffic regulations and driving habits than foreign companies. Secondly, they are more likely to obtain Chinese road acquisition scene data first.

This article is a translation by ChatGPT of a Chinese report from 42HOW. If you have any questions about it, please email bd@42how.com.