Understanding the Current Status of Autonomous Driving Simulation Testing Technology in One Reading

Introduction

What is Autonomous Driving Simulation Testing

Virtual simulation technology is an indispensable technical means for automobile research and development, manufacturing, validation and testing, which can effectively shorten the development cycle of technology and products and reduce research and development costs; with the development of automobile intelligence and network connection, virtual simulation technology has a greater space for play, such as simulation testing and verification of autonomous driving systems; virtual simulation testing is a critical link to achieve high-level autonomous driving applications, and vehicles with autonomous driving functions must undergo a large number of virtual simulation tests and on-road tests before they can be commercialized.

The three testing stages for commercializing autonomous driving cars: simulation testing, closed-field testing, and open-road testing.

Autonomous Driving Simulation Testing: Mainly through mathematical modeling, a digital reproduction of the autonomous driving application scenario is carried out, and a system model that is as close to the real world as possible is established. The autopilot system and algorithm tests and verifications can be achieved directly through software simulation testing without the need for real vehicles.

Types of simulation testing include: model-in-the-loop simulation (MIL) – software-in-the-loop simulation (SIL) – hardware-in-the-loop simulation (HIL) – vehicle-in-the-loop simulation (VIL)

The Necessity of Autonomous Driving Simulation Testing

Issues with actual road testing：

According to research by RAND Corporation in the United States, an autonomous driving system requires at least 11 billion miles of driving data to test and verify the system and algorithms before it can be mass-produced. Therefore, it is difficult to rely solely on actual on-road testing to meet this goal, and actual on-road testing still faces the following problems:

• Long testing cycle, high cost and low efficiency
• Limited coverage of scene conditions, especially for “corner cases”, which are difficult to replicate
• The safety of on-road testing cannot be guaranteed for some extreme dangerous scenes

Advantages of simulation testing:

• The testing scenes can be flexibly configured with high scene coverage
• The testing process is safe, and “corner cases” can be reproduced for retesting
• It is possible to achieve automatic testing and cloud-accelerated simulation testing, which can improve testing efficiency and reduce testing costs

The relationship between the three:Simulation testing, enclosed site testing, and open road testing complement each other to form a closed testing loop, which promotes the development and establishment of automatic driving vehicles and standardized systems:

1) The results of simulation testing can be verified through enclosed site and open road testing.

2) Dangerous scenarios identified through road testing will be fed back into simulation testing, allowing for targeted adjustments of scenario settings and parameter space.

3) The final results of simulation and enclosed site testing are comprehensively evaluated, and the evaluation criteria and testing scenario library are continuously improved based on the evaluation results.

Important components of automatic driving simulation testing: scenario library, simulation platform, evaluation system; among them, the scenario library is fundamental, the simulation platform is core, and the evaluation system is critical. The three are closely coupled and mutually supportive: the construction of the scenario library requires guidance from the simulation platform and evaluation system; the development and evolution of the simulation platform requires support from the scenario library and evaluation system; and the establishment and improvement of the evaluation system also requires the existing scenario library and simulation platform as reference basis. In the following sections, we will discuss these three important aspects of scenario library, simulation platform, and evaluation system in detail.

Scenario Library

What are test scenarios and what are their core elements?

Definition of automatic driving test scenarios: The overall dynamic description of the interaction between an autonomous vehicle and its driving environment during a specified period of time, consisting of elements that are determined based on the expected functionalities of the autonomous driving vehicle. (Definition from: C-NCAP Group Standard – Functional Test for Autonomous Driving System Part 7 – Simulation Testing) In other words, scenarios can be viewed as an organic combination of driving situations and driving scenarios for autonomous driving vehicles. These scenarios are characterized by an unlimited abundance, extreme complexity, unpredictability, and impossibility to exhaust.

Elements of test scenarios: Elements of the test vehicle itself as well as external environmental elements; external environmental elements include: static environmental elements, dynamic environmental elements, traffic participants, meteorological elements, etc.

What is a scenario library? Where does the data for scenarios come from?## Definition of Scene Library

A database consisting of a series of automatic driving test scenarios that fulfill certain testing requirements. The scene library can complete the management of scene data and the bridging of scene testing engines, from the automatic generation, management, storage, retrieval, matching, to the final injection of testing tools.

The scene library contains four typical testing scenarios (classified by data sources): natural driving scenarios, dangerous operating conditions scenarios, standard regulatory scenarios, and parameter recombination scenarios.

Data Sources of Testing Scenarios

Testing scenario data mainly includes three parts: real data, simulated data, and expert experience.

1) Real Data: Real scenarios that have occurred in the real world, collected by sensors or saved in other forms, including natural driving data, traffic accident data, roadside unit monitoring data, closed-site test data, and open road test data.

2) Simulated Data: Mainly include driving simulator data and simulation data. The former is the scenario element information obtained through testing by using driving simulators while the latter is the scenario element information obtained through testing by automatic driving systems or vehicles on virtual simulation platforms.

3) Expert experience data: Scene element information summarized and refined based on simulation testing experience of experts, among which, standard regulations are the typical representatives of expert experience data.

![Sources of Testing Scenario Data
Note: The sources of testing scenario data are based on the Function Test of Automated Driving Systems Part 7: Simulation Test (Draft for Solicitation)].(https://upload.42how.com/article/image_20210707202750.png)

Scene Library Construction Process

Construction process:

• Determine the data storage method and standard for individual virtual scenes.
• Build a single automatic driving virtual test scenario.
• Select appropriate scenarios as part of the scene library based on feature tags in numerous virtual test scenarios.

The Closed Loop of Automatic Driving R&D Testing and Scene Library Construction

The construction of the testing scenario library can effectively promote the R&D test work of automatic driving, and the R&D test of automatic driving can provide feedback to the library, enriching the scene library.

Current Status of Scene Library Development in China

Analysis:## Introduction to Typical Scene Libraries in China

Table 1. Introduction of Typical Scene Libraries in China

1) China Automotive Data Company – Self-built Scene Library

Scope of Scene Library: Natural driving scene library (1228 types), functional safety scene library (110 types), V2X scene library, dangerous accident scene library (206 types), China-specific traffic regulations scene library (82 types), and expected functional safety scene library (70 types), etc.

a. Natural Driving Scene Library

Based on the dynamic and static element characteristics of different road topologies, traffic infrastructure, environmental conditions, and vehicle information in China, the database is classified and constructed from different dimensions and updated iteratively:

• Regional Coverage – Key cities such as Beijing, Tianjin, Shanghai, etc.
• Working Condition Coverage – Key areas such as highways, cities, rural areas, parking lots, etc.
• Environmental Coverage – Various weathers such as clear day, rainy day, snowy day, smog day, etc.
• Built functional scene library and logic scene library applicable for 24 ADAS tests

b. Functional Safety Scene Library

Developed based on scenario ODD element recombination, virtual simulation scene reproduction, and critical scenario simulation verification technologies.
Provided a quantitative evaluation plan with evidence for ASIL risk levels of ADAS system functionality safety hazards in compliance with ISO26262 standard.

Process for Creating the Functional Safety Scene Library:

1. Reference the international standard of ODD classification to arrange and recombine the 7 scenarios, such as driving areas, weather, and light, and 30 subclasses of scenarios to form a systematic functional scene library.
2. Based on the actual road data, adjust the relevant parameters of the above functional scene to obtain the corresponding logic scene, and finally build the driving scene and surrounding environment required in the simulation software.
3. Activate the ADAS function in the simulation verification software, conduct a simulation verification of electronic and electrical failures in the reproduction scene, evaluate the risks generated, calculate the severity (S), exposure level (E), and controllability (C) according to the formula, and finally determine the ASIL risk level in compliance with the ISO26262 standard.

c. V2X Scene Library- Building scenario library based on LTE-V terminal, DSRC terminal application layer development experience and scenario requirements in “Application Layer and Application Data Interaction Standard for Collaborative Intelligent Transport System Vehicle Communication System”

• Accumulating V2X scenario data from four aspects: data security, communication security, control security, and identity security
• Establishing V2X test scenario library creation process and forming 3500 test cases

2) China Automotive Engineering Research Institute – Self-built scenario library

Scenario library data sources: Standards and regulations, traffic accidents, artificial experience, and natural driving data.

Scenario library generation plan: Scenario data collection, scenario classification extraction, scenario data labeling, scenario clustering, scenario reconstruction, virtual scenario transformation, etc.

China Typical Scenario Library V2.0:

• Reference to the scene classification system of the German PEGASUS project and the simulation formats OpenDRIVE and OpenSCENARIO launched by ASAM
• Including 1000 cases of standard regulation scenarios, 3000+ cases of empirical scenarios, 50,000+ cases of reconstructed scenarios, and 150+ cases of accident scenarios

In December 2020, the “China Typical Scenario Library” was upgraded to V3.0 based on the V2.0 version:

• On the basis of version 2.0, three major categories of typical accident scenarios, autonomous driving accident scenarios, and expected functional safety scenarios are added
• Covering more than 20 autonomous driving functions, and the total number of OpenX-standard formatted scenario files exceeded 100,000
• In order to facilitate the development of enterprise autonomous driving technology, the scenario library is divided into four modules: basic, primary, intermediate, and advanced levels.

3) Baidu – Self-built scenario library

Covering scenario types:

• Standard regulation scenarios: meet basic standard regulatory requirements
• Dangerous working condition scenarios: ensure the safety of autonomous driving
• Capability evaluation scenarios: evaluate the autonomous driving capabilities

Number of typical scenario types covered: About 200

Covering: Different road types, obstacle types, road planning, traffic signals.Data Sources for Test Scenarios: Natural driving data, traffic databases, and artificial experience.

Test scenarios are based on different generation schemes: Logsim scenarios and Worldsim scenarios. Currently, there are a total of 220 Worldsim scenarios and 17 Logsim scenarios.

a. Logsim: Scenarios extracted from road test data that provide complex and variable obstacle behaviors and traffic conditions with high uncertainty.

b. Worldsim: Scenarios composed of artificially preset obstacle behaviors and traffic light states, which are simpler.

4) Tencent—Custom Scenario Library

Accumulated real driving data: As of 2020, over 500,000 kilometers of traffic scene data has been accumulated.

Types of covered scenarios include: vehicle collision avoidance, traffic compliance, behavioral ability, intersection vehicle collision avoidance with visibility impairment, collision warning, emergency braking, dangerous lane change, crossing at unsignalized intersections, and pedestrian crossing.

Number of typical scenario types: Around 1000.

Scenario generation scheme: Using Agent AI capability, various types of random driving scenarios can be generated freely.

Simulation Platform

Typical Architecture of Simulation Platform

The simulation platform generally includes simulation framework, physics engine, and graphics engine. Among them, the simulation framework is the core of the platform software, which supports sensor simulation, vehicle dynamic simulation, communication simulation, and traffic environment simulation.

1) Sensor simulation: It supports simulation of sensors including cameras, lidars, mmwave radars, GPS/IMU, etc.

2) Vehicle dynamic simulation: Based on multi-body dynamics modeling, the model simulates the posture and kinematics of the vehicle during the motion process by parameterizing multiple real components such as steering, suspension, braking, and I/O hardware interfaces.

3) Traffic scene simulation: It includes two parts: static scene restoration and dynamic scene simulation. Static scene restoration is mainly achieved through high-precision map and 3D modeling technology.

Dynamic scene simulation can be achieved by extracting real road test data through algorithms and combining them with existing high-precision maps to create complex traffic environments, or by generating complex traffic environments through setting relevant parameters based on the proportion of random traffic flows and using statistical methods.

For example, it can simulate extreme and dangerous situations that autonomous vehicles may encounter in the real world, from simulating severe weather conditions such as heavy rain and snow to poor lighting, to dangerous operations of surrounding vehicles, etc.

4) V2X simulation (communication simulation): It supports creating real or virtual sensor plugins, allowing users to create special V2X sensors. It can be used to test V2X systems and generate synthetic data for training.

Typical autonomous driving simulation platforms in China

Table 2. Summary of typical autonomous driving simulation platforms in China

1) Zhejiang TianXingJian Intelligent Technology – Panosim

Platform Type: Integrated simulation and testing platform for car autonomous driving technology and product development

Platform Features:

• Real-time online simulation support: MIL/SIL/HIL/VIL
• Support for development and testing of ADAS technology and products
• Support for development and testing of driving simulation experience and human-machine co-driving system
• Support for development and testing of perception, planning, decision-making, and control algorithms
• High-confidence vehicle dynamics model: benchmarked against CarSim, supports 27 degrees of freedom complex dynamics model

2) 51WORLD – 51Sim-One

Platform Type: Integrated simulation testing platform that covers the entire process of autonomous driving

Platform Features:

• Capable of simulating multiple sensors, traffic flow, intelligent agents, perception, and decision-making, and autonomous driving behavior training
• Scalable parallel distributed simulation architecture, deployable in standalone, private cloud, and public cloud environments
• Real-time online simulation support: HIL/DIL
• Autonomous driving digital twin evaluation testing
• Built-in library of scenarios and test cases
• Software platform provides Python, C++, ROS, OSI, and other API interfaces
• Support for multi-vehicle collaboration and interaction: the platform supports the integration of multiple testing systems, including automated and manual driving systems

3) Tencent – TAD Sim

Platform Type: Virtual-real hybrid online/offline simulation testing platform built on the Unreal Engine

Platform Features:

• Integration of gaming engine, 3D reconstruction technology, vehicle dynamics model, and virtual-real traffic flow
• Application of gaming technology can achieve the geometry, logic, and physics reconstruction of the scene
• A single system satisfies the full-stack algorithm usage requirements: can complete full algorithm closed-loop simulation testing verification from perception to decision-making and planning, to control
• A single system satisfies the MIL/SIL/HIL/VIL usage requirements, covering the entire automotive V-shaped development process
• Built-in high-precision map, supports simulation of national highways and expressways.4) Peidei (Shanghai) Technology Co., Ltd. – Pilot-D GaiA

Platform Type: A simulation testing platform developed based on German self-driving simulation core technology.

Platform Features:

• Provides high-fidelity environmental sensor simulation: including laser radar, mmWave radar, and camera simulation.
• Provides C++ and MATLAB interfaces for adaptation to different vehicles and systems.
• Provides three versions of software solutions: Standard version – stand-alone solution, NET version – cloud computing and interactive testing solution, RT version – provides HIL and VIL simulation tests.
• Supports interactive autonomous driving simulation: multiple hosts (vehicles) perform interconnected simulation testing in the same virtual scene.
• Provides driver simulation based on VR/MR/AR.

Typical Foreign Simulation Testing Platforms

Table 3. Summary of Typical Foreign Self-Driving Simulation Platforms

1) Siemens – PreScan

Type: A traditional car simulation platform based on physical modeling.

Platform Features:

• The simulation process is mainly divided into four steps: building the scene, adding sensors, adding control systems, and running the simulation.
• Can run in open loop, closed loop, offline and online modes.
• Supports importing high-precision maps in OpenDrive format.
• Supports joint simulation with third-party dynamic software such as CarSim, dSPACE ASM, VI-Grade, AmeSIM, etc.
• Supports joint simulation with other types of third-party software, such as Matlab & Simulink, MOMO & Logitech steering wheel, dSPACE/console, HIL tools (ETAS, dSPACE, Vector, OpalRT, etc.)
• Supports cloud-based deployment of large-scale simulations.

2) MSC Software – VTD

Type: A traditional car simulation platform.

Platform Features:

• Supports full-cycle development process from SIL to HIL and VIL.
• Supports joint simulation with third-party tools and plugins.
• The platform’s functions and storage support open formats: OpenDrive, OpenCRG, and OpenScenario.
• The simulation process is mainly divided into three steps: road network construction, dynamic scene configuration, and simulation operation.## German IPG – CarMaker

Type

A self-driving simulation platform based on traditional dynamics simulation

Platform features

• Support co-simulation with third-party software: ADAMS, AVLCruise, rFpro, etc.
• Closed-loop simulation system of road, traffic environment, vehicle and driver

IPG Road

• Simulate various forms, such as multi-lane, intersection, etc.
• Generate roadblocks of various shapes through GUI configuration, such as cones, cylinders, etc.

IPG Traffic

• Provide rich models of traffic objects, such as vehicles, pedestrians, road signs, traffic lights, road construction buildings, etc.

IPG Driver

• Provide a self-learning driver model
• Support import/export of high-precision maps
• Support importing map data from HERE HD Live Maps
• Support exporting map data in both ROAD5 and OpenDrive formats
• Support parallel execution of large test directories on high-performance computing (HPC) clusters
• Support running in Docker containers with good portability and scalability

NVIDIA – Drive Constellation

Type

A self-driving simulation platform based on Unreal Engine, consisting of two servers

Platform composition

a. The first server hardware consists of 8 NVIDIA RTX Turing GPUs

• Run the DRIVE Sim software to simulate sensor data (including cameras, millimeter-wave radar, lidar, IMU, and GNSS) and driving scene data on a self-driving vehicle

b. The second server hardware consists of an automatic driving vehicle target AI ECU

• Used to process simulated data transmitted from the first server, such as sensor data

Platform features

• High authenticity: As a real-time hardware-in-the-loop solution, it can test and verify the software and hardware systems that will be installed in self-driving vehicles in the future- Cloud Simulation Workflow: Execute autonomous driving tests in a seamless end-to-end workflow. The user can remotely access DRIVE Constellation and submit simulation scenarios to the cloud-based vehicle testing team, then visualize the tests and provide evaluation results at a faster speed.
• Scalability: The platform supports large-scale deployment in the data center and can run various simulations in parallel; and every mile of testing in the DRIVE Constellation platform includes events of interest (special conditions), which can be completed in a short time instead of months or years.
• Complete feedback loop: The first server runs the DRIVE Sim software, which simulates driving scenarios and sensor data generated on the autonomous vehicle. Then the simulated sensor data is sent to the second server for data processing, and the server outputs driving decision signals back to the first server, thus forming a simulation feedback loop.

5) Microsoft – AirSim

Type: Open-source simulation platform for drones and autonomous driving built on Unreal Engine.

Platform features:

• AirSim is a research project for simulating drones and autonomous driving built on Unreal Engine. It can create highly realistic traffic environments and simulate vehicles and sensors.
• Vehicle modeling testing: AirSim includes vehicle simulation, city road scenes and provides programmable API and plug-and-play code.
• Build rich scene quickly: AirSim provides detailed 3D city streetscape, including traffic lights, parks, lakes, construction sites, and other rich scenes.
• AI research platform: AirSim provides API interfaces in multiple languages, including C++ and Python. Users can use AirSim and many machine learning tools at the same time. It is mainly used to test end-to-end reinforcement learning algorithms for deep learning, computer vision, and autonomous driving.

6) Barcelona Autonomous University (jointly with Toyota Research Institute and Intel Lab) – CARLA

Type: Open-source platform based on Unreal Engine, adopting a server and multi-client architecture.

Platform features:

• An open-source simulator for designing, training, and validating autonomous driving systems in urban areas.
• It provides open-source digital data resources (including urban layout, buildings, and vehicles) to build autonomous driving testing and training scenarios.- RoadRunner, a road building software of Vector Zero, can be used to create scenes and high-precision maps for autonomous vehicles.
• The performance of three autonomous driving methods has been studied: traditional modular pipeline, end-to-end models trained through imitation learning, and end-to-end models trained through reinforcement learning.

7) LG Electronics – LGSVLSimulator

Type: An open-source autonomous driving simulation platform developed based on Unity game engine.

Platform features:

• The simulator supports ROS, ROS2, and Cyber RT, and provides integration with the Autoware and Baidu Apollo open-source autonomous driving platforms.
• Supports software-in-the-loop (SIL) and hardware-in-the-loop (HIL) testing for autonomous driving systems.
• Virtual scene construction:
  —— Provides a Python API that allows users to create the required virtual scenes by writing scripts.
  —— High-precision maps: Supports creating, editing, and importing/exporting of existing 3D environments’ high-definition maps.

Note: Import/export formats supported by high-precision maps are:

a. Supported import formats: Apollo5.0 HD map, Autoware Vector map, Lanelet2, and OpenDrive 1.4.

b. Supported export formats: Apollo 5.0, Lanelet2, and OpenDrive 1.4.

Domestic Technology Companies – Cloud Simulation Platform

Traditional simulation tests are generally single-machine simulation tests, but the commercial application of advanced autonomous driving systems requires the simulation and testing of a vast number of testing scenes. Massive data storage, computing, and processing have become a barrier on the road to autonomous driving research and development. Cloud platforms’ distributed architecture, accelerated computing capabilities, and high data computing power can solve this problem well, thereby greatly improving the efficiency of system research and development and verification testing. Therefore, cloud platform simulation will undoubtedly be the development trend of future autonomous driving simulation testing technology and will also be one of the core technical barriers for enterprises to develop autonomous driving.

Table 4. Domestic Technology Companies Cloud Simulation Platform Information Sorting

1) Baidu – Apollo Cloud Simulation Platform

Main applications:

• Internal use: Supports the development and iteration of the Apollo system.- External: Provide cloud-based decision system simulation services for Apollo ecosystem developers.

Platform features:

• Cloud simulation platform built on Baidu Cloud and Microsoft Azure
• Collaborate with Unity to develop virtual simulation environments based on the Unity engine
• Proposed an end-to-end autonomous driving simulation system: AADS system – data-driven traffic flow simulation framework + image rendering-based scene image synthesis framework

2) Huawei – Octopus Autonomous Driving Cloud Service

The platform provides the following three major services: data service, training service, and simulation service, which cover the entire lifecycle business of autonomous driving data, models, training, simulation, labeling, etc.

• Data service: data processing, data lake governance, data storage, etc.
• Training service: big data AI platform, model management, distributed training tasks, etc.
• Simulation service: scenario library, simulation system, evaluation system, etc.

Platform features:

• Decouple the four key elements – hardware, data, algorithms, and high-precision maps, to build an open platform driven by data as the core to drive the closed-loop iteration of autonomous driving
• Automated data processing: automatic data mining (the platform supports PB-level storage of data and billion-level data retrieval within seconds) and automated data labeling (which can save more than 70% of labeling labor costs)
• One-stop autonomous driving DevOps capability: a complete and closed-loop automation testing system is formed from code access, version management, to simulation and evaluation
• Large-scale parallel simulation: the system can simulate over 10 million virtual testing miles and over 200 evaluation metrics per day, supporting concurrent testing of 3,000 instances
• Real-scenario digital twins: typical road data collected from actual roads can be transformed into simulated scenarios and combined with high-precision maps to achieve over 95% scene restoration ability

3) Tencent – TAD Sim Cloud

Platform features:

• Distributed architecture, supporting deployment in Windows, Linux, and Web environments
• Accelerated testing of massive scenarios: the cloud platform supports parallel calculations of more than ten thousand scenarios, and the running time of 1000 scenarios is shortened to 4 minutes
• Accelerated testing of virtual city technology: It can load high-precision maps at the city level and deploy millions of traffic flow vehicles and thousands of autonomous driving main vehicles at the same time.- Developing a proprietary closed-loop simulation model: designed to include the full range of models such as vehicle dynamics, sensors, traffic flow, maps, and scene editing. The model also supports simulation standards like OpenX and OSI.

4) Alibaba – Hybrid Simulation Test Platform

Platform Features:

• Combining virtual and real simulation technology, the platform adopts real road test scenarios and cloud-based trainers.
• Solves the problem of reproducing extreme scenarios on a large scale: the platform can reproduce extreme scenarios within 30 seconds by arbitrarily changing extreme scenario variables.
• Solves the problem of insufficient data on extreme scenarios and low efficiency of autonomous driving AI model training.
• The daily virtual test mileage is over 8 million kilometers.

Simulation Test Evaluation System

Simulation Test Evaluation Dimension and Specific Evaluation Content

1) Two important evaluation dimensions: authenticity and effectiveness

a. Authenticity evaluation: the evaluation mainly targets the authenticity of the scenario library, and is divided into two aspects: realism of scenario information and realism of scenario distribution.

Realism of scenario information: During the construction of the scenario, it is necessary to accurately and reasonably render the static environmental elements (such as traffic facilities, roads, obstacles, etc.) in the virtual environment of the test scenario, dynamic environmental elements (such as dynamic indication facilities, communication environment information), participatory traffic elements (motor vehicles/non-motor vehicles, pedestrians, etc.), and meteorological environment elements (ambient temperature, lighting conditions, weather conditions).

Realism of scenario distribution: for parameter recombination scenarios, due to the modification of parameters by humans, scenes that do not exist in the real world may appear. Therefore, when editing scenes manually, it is necessary to refer to the parameter value range of real-world scenes to reasonably set the parameters of the parameter recombination scene.

b. Effectiveness evaluation

There is currently no unified effectiveness evaluation standard, and the current reference standards are as follows:

2) Specific evaluation content: simulation test evaluation itself and autonomous driving vehicle driving performance, driving coordination, standard matching, learning evolution, etc.

![Autonomous Driving Simulation Test Evaluation Project]### Example of Simulation Testing Evaluation Standards

Evaluation Methods: (Two)

• Function-based evaluation method: For specific autonomous driving functions, under the premise of given test scenarios, determine whether the tested autonomous driving vehicles can drive safely in the scenario
• Scene-based evaluation method: Place the test vehicle in the test scenario. Under the premise that the autonomous driving vehicle can drive safely, determine the complexity of the scene, such as the complexity of the surrounding environment or the complexity of the driving task.

Evaluation Indicators: (Two types)

• Pass/Fail evaluation indicators: Evaluate autonomous driving judgments and stipulate goals that must be achieved within the scenario. For example, safety, completeness, economy, human intervention, and other evaluation indicators that use a 0/1 quantization mechanism.
• Performance evaluation indicators: Characterize the degree to which autonomous driving vehicles achieve expected functions in specific test scenarios. For example, intelligence, comfort, anthropomorphism, and other evaluation indicators that use a difference value mechanism for quantization.

Reference: Evaluation standards refer to “Part 7 of Functional Testing for Autonomous Driving Systems: Simulation Testing (Draft for Solicitation of Opinions)”

Conclusion

Challenges facing the development of autonomous driving simulation testing

1) Construction of scenario library

• There is no unified data format standard, and cooperation among enterprises for scenario library construction is lacking, resulting in large duplicate investment in resources.
• Annotations, analysis, and extraction of scenario library data still rely heavily on manual labor. The efficiency of scenario library construction is low and the cost is high.
• The coverage of the scenario library is still not high enough to cover common traffic scenarios.
• How to balance the tradeoff between exhaustive coverage of all scenarios with test efficiency and accuracy.

2) Simulation platform

• Testing scenarios are limited, and standardized testing methods and virtual simulation testing applications are still relatively weak.
• There is currently no perfect autonomous driving simulation software that can perform simulation testing functions for all modules, and multiple simulation software is still needed for collaborative simulation. Different autonomous driving simulation software often requires the development of different simulation interfaces, which not only increases time costs but also increases the learning costs for engineers.
• The simulation environment lacks the reality and richness of physical reality in the real world, and the training effect is limited and scalability is poor.
• System simulation software also has its inherent weaknesses. For example, the simulation model needs to make clear assumptions about boundary conditions, but in reality, these boundary conditions may not be able to be identified or obtained accurately.
  -The accuracy of the sensor model, vehicle dynamics model, and road environment model will all affect the accuracy of simulation test results. Therefore, improving the credibility of simulation test results is currently a major challenge.

3) Test evaluation system.- The simulation software system architecture and scenario database construction methods adopted by different companies vary greatly, making it difficult to establish a unified standard for simulation testing evaluation.

• There is no unified standard for the evaluation of the authenticity and effectiveness of simulation software, and corresponding evaluation standards are still lacking for the learning and evolutionary aspects of vehicles.

Development Trends of Autonomous Driving Simulation Testing Technology

1) Cloud-based high-concurrency testing and accelerated testing are important directions for the future development of simulation testing and are one of the core competitiveness of autonomous driving simulation companies. The cloud platform has the ability to store, process and manage massive data, support large-scale simulation construction, and cover massive driving scenarios. It can effectively establish a city-level geographic information + virtual traffic flow information system, enabling cross-professional multi-user information interaction and sharing.

2) Functional safety scenario databases, V2X scenario databases, and expected functional safety scenario databases will be the focus of future scenario database construction. Different manufacturers adopt unified data format standards and jointly build a basic scenario database to form a common, portable scenario database.

3) Mixed traffic simulation testing will be an important research area in the future of autonomous driving simulation technology. The replacement of traditional cars by autonomous driving cars is not an immediate process and will inevitably coexist for a considerable period of time. Therefore, mixing traditional cars and autonomous driving cars will be a common traffic scenario. It is even necessary to establish a mixed traffic simulation model of autonomous driving vehicles under different penetration rates.

4) Digital twin technology will be a great help in the development of autonomous driving simulation testing and is an efficient tool for simulation testing. Digital twins create a model that is consistent with the real world in a virtual environment. The autonomous driving digital twin testing VRIL (Virtual Reality in the Loop) is the testing of real vehicles in a real testing environment that is mapped to a virtual testing environment to achieve synchronization of virtual and real states and to realize closed-loop real-time simulation testing of the entire digital twin system.

5) Currently, autonomous driving simulation testing mainly focuses on traditional passive-active safety and reliability testing verification, as well as R&D or iterative upgrading of perception, planning, decision-making, and control algorithms. However, each autonomous driving vehicle is not isolated and must be networked and exist as an intelligent node in the entire smart transportation system. Therefore, verification testing and evaluation in the direction of information security will inevitably be an important research direction for future autonomous driving simulation testing.

This article is a translation by ChatGPT of a Chinese report from 42HOW. If you have any questions about it, please email bd@42how.com.