42HOWAuto

Baidu's ApolloWorld Trains Apollo Go Robotaxis in a Virtual Environment

Yu Yue

Baidu has introduced ApolloWorld, a world model that generates camera and LiDAR imagery and alters traffic environments to train and validate its Apollo Go robotaxis. Wang Liang, chief architect of Baidu's Intelligent Driving Group (IDG), described the model as an "examiner" for verifying autonomous driving safety.

The image shows Wang Liang, chief architect of Baidu's Intelligent Driving Group (IDG) and chairman of the IDG Technology Committee, introducing the progress of ApolloWorld. He stands at the podium holding a microphone, with a large screen behind him displaying content on the "Impossible Triangle of Autonomous Driving Technology," including Safety, Intelligence, and Openness, and labeled with the "Triangle Universe of Autonomous Driving." The audience seating is also visible, with many people listening attentively. This image echoes the aforementioned disclosure of ApolloWorld's latest progress by Baidu, intuitively presenting the relevant introduction scene.

Baidu has built a virtual training ground for Apollo Go.

On November 3, Baidu disclosed the latest progress of its world model, ApolloWorld. It can generate images as perceived by cameras and LiDAR, and alter various traffic environments for the training and validation of autonomous driving.

Wang Liang, Chief Architect of Baidu's Intelligent Driving Group (IDG) and Chairman of the IDG Technology Committee, describes it as the "examiner" for validating autonomous driving safety. The R&D team can use it to repeatedly simulate risk scenarios, checking whether the driving model can brake in time, yield correctly, and complete the entire journey through continuous interactions, thereby exposing potential issues well before physical vehicle deployment.

In real-world operations, high-value scenarios can also be expanded into numerous variations through ApolloWorld, allowing the vehicle-side end-to-end foundation model ADFM to repeatedly practice and undergo testing within this environment.

During L4 fully driverless operations, relying solely on physical road tests is often insufficient to cover all corner cases. The world model enables this type of validation to be completed much faster. For Apollo Go, which is continuously expanding its operating areas, this approach is already accelerating its R&D and deployment efficiency.

End-to-end requires closed-loop validation

To understand ApolloWorld, one must first look at the changes implemented on Apollo Go vehicles.

In the past, autonomous driving systems typically divided perception, prediction, and decision-making and planning into separate modules. Engineers could optimize each module individually, but front-end recognition errors would propagate downstream; different models distributed the use of onboard computing power, requiring continuous coordination between the models and a vast array of hand-coded rules.

This image illustrates Baidu Apollo's traditional hierarchical architecture diagram of "Perception-Prediction-Decision Planning," presenting the module composition of past autonomous driving systems. The left side includes perception module elements labeled with "Road/Area/Size" and "Traffic Lights," along with multimodal fusion information; the center shows the process connection of "Prediction-Decision Planning," and the right side features the "Decision" and "Planning" modules. The diagram also highlights the architecture's characteristics, such as information error propagation, distributed use of vehicle-end computing power, and the need to coordinate models with manual rules, echoing the document's description of module splitting and connection issues in traditional autonomous driving systems.

Entering a new city presents more work for this system. For instance, traffic habits vary across different cities, and there are distinctions between left-hand and right-hand drive. If a new set of rules must be added for every such difference, adaptation efficiency will struggle to keep pace with business expansion.

Baidu's direction is the autonomous driving foundation model ADFM. It feeds video, LiDAR, navigation, and other information into the model, generating driving actions such as steering, acceleration, and deceleration through a unified feature representation. More driving experience enters the model via data, reducing information loss between traditional modules and enabling computing power to serve larger models.

This image shows the architecture diagram of ADFM, the "foundation model" for autonomous driving in Baidu Apollo's new generation AI Driver. The diagram labels the model's core modules, including the Scene Encoder and Action Decoder on the left, and the Action Decoder on the right (labeled with action categories: Steering, Acceleration). The model is used to process multiple types of driving-related information and serves as the foundation model supporting autonomous driving to generate driving actions such as steering, acceleration, and deceleration. This content corresponds to the document's relevant descriptions of Apollo technology reducing traditional information loss and optimizing model architecture.

However, the challenge here is that there is no human driver in the vehicle ready to take over at any moment.

Wang Liang used rock climbing as an analogy: L2 driver assistance is like rock climbing with a safety rope, while fully autonomous driving is like rock climbing without one. Architectural advancements must be accompanied by sufficient training and validation before entering actual operation.

One method for validating a new model is to re-input sensor data collected from real-world road tests into the model to see what driving decisions it would make in those scenarios; this method is called data playback. However, playback can only provide the environment exactly as it was recorded, and cannot simulate the reactions of surrounding vehicles to new actions.

This image displays Baidu Apollo's new end-to-end model simulation architecture, centered around the closed-loop logic of "Looping." The core content includes five modules: from left to right, the "World States" module for "Multimodal Data Generation," the "End-to-End AI Driver" module, the stage with the "Next Action" label for "Steering, Acceleration," the "World Model Simulator" module, and the real-time closed-loop interactive module linking "Video, LiDAR, Navigation" and generating "Next World States." This intuitively presents the architectural logic of ApolloWorld simulating surrounding reactions as driving actions change to achieve closed-loop verification, echoing the content introduced in the text about ApolloWorld solving the limitations of data playback and completing closed-loop verification for new models.

ApolloWorld, on the other hand, allows the test environment to change in response to driving actions: after the ADFM executes maneuvers such as steering, acceleration, and deceleration, ApolloWorld simulates the reactions of surrounding vehicles and pedestrians, generating new camera images and LiDAR point clouds. The ADFM then determines its next driving steps based on this new data, allowing the test to proceed continuously.

This gives Baidu the opportunity to conduct simulation validation for a wider variety of scenarios before hitting the road, supplementing real-vehicle testing. Only then, when actually operating on the road, can risks be reduced to a lower level.

Interactive multimodal simulation

Yang Ze, Distinguished Architect at Baidu IDG and Head of the World Model, explained that ApolloWorld models traffic participants such as vehicles and pedestrians as AI agents. The model integrates their historical states, road and traffic signal information, and the ego vehicle's actions to predict the next moves of all parties, then feeds the new states back into the model for continuous simulation.

This image is a related display interface of Baidu Apollo, titled "AI Agent-Driven World Deduction," corresponding to the document's content on ApolloWorld deducing the behavior of traffic participants based on AI agents and generating driving sensor data. The image demonstrates the AI agent deduction process; the left side shows the current state including different traffic agents and road information, the central core area is the multi-agent interactive deduction module, labeled with key features such as "Interactivity," "Letting Actions Change the Next Moment," and "Causality to Time Deduction." The right side presents the future world state obtained from the deduction, as well as the generated next-moment traffic state, clearly and intuitively presenting the core working logic of ApolloWorld.

The simulated traffic states must then be converted into sensor data readable by the driving model. ApolloWorld jointly generates camera images and LiDAR point clouds while maintaining consistency across different viewpoints and modalities.

Yang Ze disclosed that when running on a computing cluster, ApolloWorld can generate 24 frames of data per second, with each frame comprising 10 camera images and 4 LiDAR point clouds.

The image displays real-time capabilities related to Baidu's ApolloWorld. The blue background features the text "Gain Attribute 3 - Real-time" and "Low latency, the increase in world generation complexity does not affect model inference time, achieving high simulation test efficiency." The center highlights "24 frames per second," with "10 cameras (9 million RGB images)" and "4 LiDARs (300,000 points XYZ-1 point cloud)" labeled below. Closely related to the context, this image intuitively presents ApolloWorld's advantages in real-time generation, capable of producing 24 frames per second with sensor observations that conform to physical laws, impacting large-scale virtual testing efficiency.

This real-time generation capability determines whether the driving model can interact continuously with the environment and also impacts the efficiency of large-scale virtual testing.

Vehicle dynamics and sensor observations in the simulation must adhere to the laws of physics, and static structures such as roads and buildings must remain consistent over extended simulation runs.

According to Yang Ze, ApolloWorld can use street appearance and geometric data collected from operations as digital twin inputs to help the model maintain environmental structural stability. It then combines time and traffic flow to generate scenes, supporting continuous driving validation.

Simulating the entire journey

ApolloWorld can simulate the complete journey of Apollo Go from departure to arrival.

To complete a single order, an Apollo Go vehicle must continuously navigate through various intersections and road segments, where every acceleration, deceleration, and lane change can affect subsequent traffic conditions. By simulating the entire trip, ApolloWorld can evaluate these sequential driving decisions in context, testing whether the vehicle can continuously adapt to changing traffic and reach its destination smoothly.

The image shows a virtual ride route in Baidu's ApolloWorld. The pickup location is the Hong Kong SkyCity Marriott Hotel, with destinations including the Civil Aviation Department Headquarters, 11 Sky Shopping Mall, Airport Administrative Building, and parking lot. The route is marked with different colored lines, accompanied by icons for autonomous driving, LiDAR, millimeter-wave radar, and cameras, showing the perspectives of various sensors on different road sections. The top right corner has the "04 Virtual Ride" label. This image aligns with the context introducing ApolloWorld's simulation of entire trips to test the vehicle's ability to handle traffic changes, intuitively presenting the virtual ride route and sensor applications.

During testing, the starting point can be kept fixed while changing the destination, or the traffic density along the same route can be adjusted, allowing for comparisons of the driving model's performance and arrival times under different conditions.

This brings testing closer to real-world operational challenges: a route might be easily navigated when empty, but can the vehicle still complete the order safely and efficiently during morning and evening rush hours? If switched to a different route, what previously unverified interactions will it encounter?

Hazardous scenarios can also be introduced into this environment for repeated testing. Variations can be generated for situations such as an oncoming vehicle bypassing an obstacle and encroaching into the lane, a cyclist suddenly emerging from an obscured area, or a small animal darting into the roadway, allowing engineers to observe when the driving system brakes, how it maneuvers to avoid the hazard, and how it resumes normal driving once the danger has passed.

On real roads, a corner case is difficult to recreate multiple times. A virtual environment, however, can alter conditions around it, enabling limited real-world experience to cover a much wider range of possibilities.

This is also where Baidu's accumulated data proves invaluable. According to disclosures from Baidu, Apollo Go has been deployed in 28 cities worldwide, completing over 23 million trips. The total autonomous driving distance exceeds 350 million kilometers, of which fully driverless distance accounts for over 240 million kilometers.

The image displays content related to Baidu's ApolloWorld. The screen shows "Bringing safer and more convenient smart mobility experiences to global users," and lists multiple data points for Apollo Go: over 23 million mobility services completed globally, deployed in 28 cities, total autonomous driving mileage of 350 million kilometers, and fully driverless mileage of over 240 million kilometers. The background is a city night scene, with an Apollo Go vehicle parked in the center. Closely related to the context, this image intuitively presents Apollo Go's achievements in mobility services, city coverage, and driving mileage, providing a visual representation of the aforementioned data.

This mileage provides more than just roadway visuals; it also includes actual driving maneuvers, the reactions of other road users, and the distinct traffic habits across different cities. As ApolloWorld learns how the world changes, Apollo Go has already amassed a massive amount of data to support this.

Applying this data to simulation also requires addressing the discrepancies between simulated results and reality. Wang Liang noted that simulating details such as braking distances, lighting and shadows, and LiDAR reflections all require long-term optimization.

Real-world operations also continuously contribute to model iteration. According to the process described by Yang Ze, operational data from Apollo Go is used to train ApolloWorld, which then provides a training and validation environment for ADFM. The improved ADFM is deployed in real-world operations, and the newly accumulated data is used to further refine ApolloWorld.

Today, leading global L4 autonomous driving companies continue to expand. Waymo has expanded its service area to more cities across the United States, while newcomer Tesla is also expanding rapidly. Baidu's Apollo Go has already launched fully driverless commercial operations in Dubai and is conducting road tests in London, Hong Kong, and other locations.

Clearly, competition in L4 autonomous driving remains fierce.

To achieve safer autonomous driving, this competition will continuously increase the pressure on training and validation.

Baidu has launched ApolloWorld, aiming to shift more trial and error into the virtual environment and leverage existing operational experience to test a wider range of routes and traffic conditions. For the still-expanding Apollo Go, this investment aims to solve the challenge of reducing adaptation costs for new regions and shortening deployment cycles while thoroughly validating safety.