Author: Aimee
Before autonomous driving vehicles go into mass commercial use, thorough functional safety and performance safety testing and validation must be conducted to ensure the safety of drivers and other traffic participants. The testing covers sensors, algorithms, actuators, human-machine interfaces, and more. By comprehensively considering the application function, stability and robustness of performance, as well as functional safety, expected functional safety, formal certification, etc., vehicle autonomy on the road can be ensured.
The entire testing process can be divided into simulation testing and real vehicle testing, with the former being an important means of developing and verifying autonomous driving systems. Autonomous driving simulation testing refers to the driving testing that an autonomous driving car algorithm undergoes in a virtual traffic environment created by computer simulation technology, which establishes a mathematical model for realistic static environments and dynamic traffic scenes.
Overview of Autonomous Driving Simulation Platform
The core of simulation testing is the testing model, which revolves around testing processes, simulation platforms, and model components, covering the entire process from product design validation, development validation, testing validation, and experience evaluation. The entire simulation testing platform consists of one main process, five major simulation testing categories, and three simulation models, covering the entire process of product design, development validation, testing validation, and experience evaluation. Here, we explain the content and key points of each testing phase from the perspective of simulation testing.
One process refers to the overall testing and validation conducted from the design domain, development domain, to the testing domain based on the ASPICE standard software development process. This includes the system design stage, which involves MIL simulation throughout the system analysis and design, detailed design, virtual component integration and testing and virtual system integration testing; during the software design stage, unit testing is achieved through continuous SIL simulation, and component integration and testing is optimized with HIL simulation, and system integration testing and system acceptance are optimized through VIL simulation.
The five categories of testing platforms cover the entire development process and require several steps, including software in the loop (SIL), hardware in the loop (HIL), vehicle in the loop (VIL), enclosed test site testing, and open road testing.# MIL (Model in the Loop) Testing
MIL testing is a model-in-the-loop simulation platform that is used in the concept stage of requirement analysis to implement algorithm development and verification optimization, and is a type of simulation testing in the early stages of development and modeling. Unlike Hardware-in-the-Loop (HIL) testing, which is a method used for code validation, MIL testing is a method used for design validation.
The MIL testing process verifies if the model meets the design functional requirements by inputting a series of test cases. MIL testing is the most critical because the MIL testing concept standard must originate from the functional requirements with no other references. The SIL/PIL test cases are often borrowed from the MIL test cases. If incorrect MIL test cases are used in this stage, even if subsequent tests pass, the error will still have an impact.
SIL (Software in the Loop) Testing
The purpose of SIL testing is to verify the behavioral consistency of autogenerated code and the model used for code generation. SIL testing can actually be viewed as an equivalent MIL test (the purpose of equivalent testing is to prevent errors in the code generation process). Actual code runs on an environment simulation platform, and code and model consistency is validated through model-generated code and algorithm analysis.
HIL (Hardware in the Loop) Platform
The HIL platform is a hardware-in-the-loop simulation platform used to test control systems’ hardware, low-level software, and application-level software. By connecting the test controller to an industrial computer with the model of the control object running on the computer, the electrical characteristics of the control object are simulated. It is essential to note that the HIL test object is tested in response to actual road conditions.
VIL (Vehicle in the Loop) Testing
VIL testing applies to the vehicle-in-the-loop simulation platform and improve the accuracy of tested controller performance results. Compared with vehicle testing, it is more cost-effective, reduces testing risks and enhances testing reproducibility. It bridges the gap between vehicle testing and hardware-in-the-loop testing and can be used for evaluating complex scenarios and human-machine interaction.
DIL (Driver in the Loop) and PIL (Processor in the Loop) Testing
In addition to the above commonly used simulation tests, capable automakers also conduct DIL and PIL tests. DIL testing is designed for simulation testing of driving simulators or related equipment to evaluate driver-in-the-loop testing, feature/HMI design, and subjective evaluation. PIL testing is running generated code on the target processor to discover potential bugs introduced by compilers through test results. Here, the PIL test is an equivalent test to the SIL test, but the SIL test runs on the compiler for Windows.
Three simulation models make up the autonomous driving simulation platform, including the scene/scene library, perception system, vehicle itself, and controller. Among them, the scene is the core of autonomous driving development, testing, and validation. Based on the data and records in the database, different traffic scenarios are defined to ensure system safety through simulation by bringing the defined traffic scenarios back to the associated development steps. In terms of perception system capability testing, different signal content is simulated according to the type of sensor, such as image signals output by camera sensors, point cloud signals output by LiDAR sensors, and positioning signals output by navigation sensors. At the same time, the corresponding “limitations” are simulated according to the characteristics and parameters of the sensors, such as the detection range of the sensor, the distortion and glare of the camera, the effects of different materials on the absorption of LiDAR light, and the impact of guardrails on millimeter-wave radar. The vehicle itself refers to its dynamic module, which generally refers to a real vehicle composed of different subsystems such as the body, powertrain system, suspension system, steering system, braking system, and wheels.
Cloud-based managed simulation testing platform
Traditional simulation testing is generally single-machine simulation testing, and there are many common simulation testing tools according to different autonomous driving development needs. They can generally be divided into the following categories: Traditional automakers tend to use commercial software in the automotive industry, teams that develop autonomous driving algorithms using robot platforms often use ROS/Gazebo, and teams skilled in computer graphics develop simulation tools based on game engines. The simulation needs of large-scale and complex traffic flow are met by software in the traffic planning field. In terms of scenario simulation, PreScan/Panosim/VTD are mainly used domestically, and in terms of vehicle dynamic applications, CarMaker/CarSim and other application software are mainly used, while ECUTEST is mainly used for testing management.
Regardless of which simulation tool is chosen, building a complete and highly accurate simulation testing platform is the ultimate goal. However, in the actual development and testing process, traditional single-machine testing faces the following issues:
The production and matching of high-precision maps cannot be effectively conducted, the establishment of highly realistic static scenes still cannot be met, the physical simulation usability of advanced perception sensors is still not correct, the simulation capabilities of the driver model and intelligent traffic body model are still not perfect, and the construction of dynamic scene libraries is still insufficient.Based on this, we propose a cloud-based simulation testing and development platform. As the cloud platform itself has massive data storage, processing, and management capabilities, it can effectively establish a city-level geographic information + virtual traffic flow information, which enables cross-disciplinary multi-user information interaction and sharing with good coverage, scalability, automation, and ultra-real-time capabilities. Combined with standalone simulation testing software, it can effectively make up for the shortcomings of standalone simulation testing software.
Currently, cloud simulation platforms are mainly used in high-end car brands in foreign countries, such as BMW using ANSYS, Audi mainly using Cognata, Ford using Quantum Signal, and Toyota mainly using Drive Constellation. In China, major applications include Baidu Cloud’s Apollo Simulator, Tencent Cloud’s TADSim, and Huawei Cloud’s Octopus. The main factors considered in the simulation terminal involving cloud platforms are cloud computing, road models, traffic models, environment models, sensor models, dynamic models, DOE analysis, and SOTIF. Four major cloud platform terminals need to be established in the cloud platform simulation test: the autonomous driving virtual simulation test cloud platform, the autonomous driving functional safety & expected functional safety analysis cloud platform, the simulation, site, road testing integrated management and sharing cloud platform, and the autonomous driving integrated simulation development/testing/management sharing cloud platform.
The focus of the four types of cloud platforms mentioned above mainly involves: how to effectively build a scene library? How to reconstruct data scenes based on network platforms such as V2X? How to set up a relevant simulation model library based on different control domains? How to establish a functional evaluation system and scene and virtual simulation evaluation methods?
Regarding the architecture design of the simulation testing cloud platform, it can be divided into two main areas (as shown below).
Firstly, establishing a scene library and cloud desktop management platform focusing on the client’s demands. The scene libraries mainly come from client-level demand sub-items, and the construction of scene libraries have been mentioned in the previous article, mainly involving the generation of corresponding scene concepts from customer demands, and the integration of software, components, hardware, and other aspects to generate corresponding scene verification setup plans.
Secondly, building a robust local data center as an effective online simulation service and high-performance computing cluster. The computing cluster is mainly distributed in three levels.i The bottom layer, which is the physical layer, serves as the cluster management node layer. It dynamically schedules algorithms and resource allocation, and is set as an administrative authority that can accommodate multiple people working collaboratively. However, CPU/GPU resource utilization may be limited due to physical hardware restrictions. To overcome this limitation, an all-containerized K8s solution and cluster manager joint operation are utilized to enable node expansion based on containers rather than hardware, providing greater convenience in expanding.
ii The middle layer, which is the simulation and testing layer, is the core of the entire cloud simulation architecture. It involves taking the original autonomous driving scenario library input and combining it with the sensor/vehicle dynamics model to simulate macro and micro traffic flows using a scene editor. This is followed by completion of software-level simulations and testing, such as MIL and SIL. Here, cloud platforms are utilized for computational acceleration, algorithm uploading, high-speed communication, and safety verification. During the automated testing phase, test case libraries are leveraged to generate test evaluation reports in bulk and analyze report results. This process involves integration with multiple testing platforms to conduct open and closed loop verification and achieve the architecture result of “one cloud, two ends”.
iii The top layer is the data layer, which manages raw data, parsed data, and scene data using basic data management tools. Next, data cleaning tools are utilized for visualization, fusion, and data conversion, followed by scene tagging and multi-dimensional dynamic data output using format processing tools. This builds a bridge between driving scenario data and the simulation platform.
To engage in true cloud simulation, it is essential to apply simulations to the cloud platform or build a simulation system on the cloud platform. This is actually a typical backend scenario reconstruction strategy. Since it involves cloud data management, it is necessary to use the connected vehicle network for data upload and download terminals. Vehicle data is stored on the cloud and pre-processed, and scene reconstruction is performed to create a scenario library for simulation testing and algorithm training.
Summary
The autonomous driving simulation testing platform is an essential means of providing platform integration and engineering services for intelligent networked automobiles. This approach helps to build a lightweight/fully stacked simulation testing tool chain and promote comprehensive application of high-level autonomous driving technologies. The cloud-based simulation process has significant advantages in simulation ability construction. It primarily lies in the ability to provide more data computation power and greater support with multiple scenario libraries covered under the same platform, saving time and effort and greatly improving simulation efficiency.
This article is a translation by ChatGPT of a Chinese report from 42HOW. If you have any questions about it, please email bd@42how.com.