Tesla's End-to-End Autonomy: A Misunderstood Destiny

By 2024, "end-to-end" had become a buzzword in the autonomous driving sector.

The most direct catalyst was Tesla’s release of a test version of FSD V12 to general users in late January 2024. According to the release notes, FSD V12 upgraded the software stack for city street driving into a single end-to-end neural network trained on millions of video clips, replacing over 300,000 lines of C++ code.

Following this release, overseas media platforms saw a surge of testing videos, with many netizens expressing amazement and shock at its performance on urban roads, some even declaring it "the future."

Moreover, numerous professionals in the autonomous driving field have praised Tesla's FSD V12 performance.

Tesla’s moves in the end-to-end space also drew close attention across the Pacific; particularly within China’s smart driving sector, both industry players and public discourse have embraced the term, with some automakers eager to be first to market already incorporating end-to-end into their marketing narratives.

It is important to clarify that despite the hype surrounding end-to-end systems in autonomous driving, the industry remains in an exploratory phase regarding both technical development and commercial implementation. Against this backdrop, Tesla, as a pioneer in end-to-end autonomous driving, warrants focused attention.

So, what exactly has Tesla done regarding end-to-end systems?

Tesla's ChatGPT Moment

On May 16, 2023, following his keynote at Tesla’s annual shareholder meeting, Elon Musk gave an exclusive interview to the prominent U.S. financial media outlet CNBC.

During the discussion on Tesla AI, Musk stated that Tesla possesses immense capabilities in real-world AI, claiming it is far ahead of the competition, saying, 'I can't even name who is second.' When asked by the host about ChatGPT and generative AI, Musk added:

I believe Tesla will also experience a so-called 'ChatGPT moment,' whether it happens this year or next. This means that at some point, suddenly, 3 million Tesla vehicles will be able to drive themselves... then 5 million, then 10 million...

If we were to swap places, with Tesla building a large language model whose output is no weaker than ChatGPT, while Microsoft and OpenAI develop autonomous driving, and we exchange our respective tasks—

there is no doubt that we would win.

Given Elon Musk’s long-standing promotion of Tesla’s autonomous driving capabilities and delivery timeline, along with several missed deadlines, his comments on Tesla AI and autonomous driving in this interview did not initially spark significant controversy.

However, few noticed that a week before Elon Musk emphasized in this interview that "Tesla is approaching a ChatGPT moment," he had already mentioned the first major shift in Tesla's autonomous driving technology: FSD V12 is an end-to-end AI that takes image inputs and then outputs steering, acceleration, and braking actions.

In his words three days later, FSD V12 represents a complete AI system capable of "inputting video and outputting control" (FSD is fully AI from video in to control out).

So the question arises: When exactly did Tesla begin developing end-to-end systems?

In fact, according to Walter Isaacson's biography of Elon Musk, published officially in 2023, Tesla's journey toward end-to-end autonomous driving can be traced back to December 2022—clearly inspired by ChatGPT.

Specifically, late on December 2, 2022, Elon Musk spoke with Dhaval Shroff, an engineer on Tesla's Autopilot AI team.

A brief introduction to Dhaval Shroff's background is necessary here.

Dhaval Shroff is an accomplished student from India. He earned his undergraduate degree from Mengmaidaxue and later studied in the United States, obtaining a master’s degree in robotics from Carnegie Mellon University. In June 2014, he joined Tesla’s Autopilot team as an intern and was hired full-time in 2015, subsequently working on R&D and AI-related projects within the Autopilot team.

In November 2022, shortly after Elon Musk acquired Twitter, he needed staff to help address issues at the company and reached out to Dhaval Shroff for a meeting.

At the time, Elon Musk’s intention was actually to persuade Dhaval Shroff to leave Tesla’s autonomous driving team to work at Twitter. However, Dhaval Shroff wished to remain at Tesla and detailed the specifics of a neural network path planning project he was researching.

Indeed, Dhaval Shroff was already working on a cutting-edge autonomous driving project centered on designing a vehicle system capable of learning from human behavior. During this meeting, Dhaval Shroff told Elon Musk:

It’s like ChatGPT, but for cars. We process massive amounts of data on how real people behave in complex driving environments, then train computer neural networks to mimic that behavior... We no longer determine the correct driving route based solely on rules; instead, we use neural networks to determine the correct route.

In other words, it is an imitation of humans.

Ultimately, following this meeting, Dhaval Shroff was able to retain his position in Tesla’s autonomous driving team because Elon Musk became highly interested in the project—after all, in Elon Musk’s view, Tesla had become an artificial intelligence company, and he had already planned to hire a group of AI experts to compete with OpenAI.

With this, Dhaval Shroff and his team officially gained Elon Musk's support and began innovating within Tesla's autonomous driving technical framework. In Walter Isaacson's biography of Elon Musk, the project Shroff worked on was referred to as the "neural network planner."

Subsequent events proved that this project became a key turning point for Tesla's comprehensive shift toward end-to-end autonomous driving.

The "End-to-End" Approach Was Not Achieved Overnight

In fact, when discussing the term "end-to-end" within the context of autonomous driving frameworks, Tesla was not the first.

As early as August 2016, NVIDIA, which was then striving to enter the autonomous driving field, published an academic paper titled "End to End Learning for Self-Driving Cars." This paper introduced a deep learning algorithm designed for autonomous driving, which used convolutional neural networks (CNNs) to map images captured by the car's front-facing camera to the steering directions of the unmanned vehicle.

Actually, based on the introduction in the paper itself, the tasks this algorithm could perform were quite limited—for example, it could only learn to control the steering wheel, without considering path planning or speed. However, from a conceptual standpoint, it differed significantly from the modular autonomous driving technology frameworks at the time, which required separate modules for perception, detection, decision-making, and control, instead adopting an integrated approach.

Simply put, inputting images and outputting actions was indeed completely consistent with the philosophy of the "end-to-end" solution.

However, NVIDIA has only proposed the concept of "end-to-end" at the level of technical research. From the perspective of autonomous driving accompanying large-scale mass production in the automotive industry, end-to-end solutions have not been feasible for implementation for a long time—even Tesla, which is the most aggressive in the mass production and implementation of autonomous driving, is no exception.

So, how did Tesla step by step move toward "end-to-end"?

An industry expert engaged in autonomous driving algorithm research told us that although the concept of "end-to-end" appears novel and sophisticated, from the perspective of industry implementation, Tesla's adoption of end-to-end does not mean completely erasing previous FSD algorithm achievements to start over from scratch. Instead, it is highly likely that Tesla made structural adjustments to the algorithm framework based on previous algorithmic achievements.

In other words, Tesla's development of end-to-end was not achieved overnight.

For example, as early as Tesla AI Day in August 2021, Andrej Karpathy, then head of Tesla AI, officially introduced the perception algorithm architecture based on BEV + Transformer. In this case, Tesla's perception algorithm module had already become a fully neural-network-based version 2.0.

At the same time, according to the information presented at this AI Day event, Tesla had already optimized its Planning & Control module for software 2.0 in 2021. This means that Tesla had already attempted to incorporate neural network elements into the planning part (though not entirely).

By Tesla AI Day in October 2022, both the perception module and the planning and control module of Tesla's AI algorithm architecture were updated, but they remained relatively independent modules. Specifically:

In the perception module, Ashok Elluswamy, Tesla’s new head of AI, introduced the Occupancy Network within Tesla’s autonomous driving algorithm framework. Combined with the NeRF algorithm, it enables more robust 3D spatial perception on top of the BEV+Transformer perception framework.

In the planning and control module, Tesla leveraged the advantages of the Occupancy Network to rewrite its previous algorithms again. This revision incorporated neural networks and generative AI technologies (for trajectory prediction), but overall, the module still relied heavily on hand-coded rules—making it essentially a software 1.0 stack that uses software 2.0 code to address specific issues.

At this stage, although Tesla’s perception and planning-control modules remained relatively independent, their interconnection had become significantly tighter.

It is evident that throughout the construction of Tesla’s Autopilot software algorithm framework, while perception, planning, and control modules were relatively independent, they maintained a linked relationship. The planning and control modules would also upgrade or even be rewritten as the perception module evolved. By December 2022, the perception module had completed its evolution toward Software Version 2.0 based on neural networks, whereas the planning and control modules, due to their extreme complexity, still required substantial amounts of rule-based C++ code written by humans.

Against this backdrop, Dhaval Shroff’s mentioned neural network path planning project can be considered a critical step in Tesla’s autonomous driving journey toward “end-to-end” architecture.

It should be noted that, to date, Tesla has not publicly disclosed how it integrates autonomous driving algorithm modules such as perception, planning, decision-making, and control into a single large neural network architecture to achieve "end-to-end" processing. In fact, the company remains silent on the matter externally. Even if "end-to-end" enables the entire FSD algorithm framework to transition to software 2.0, it will not completely eliminate human-designed algorithmic rules by 100 percent.

Of course, there are skeptical voices suggesting that “end-to-end” may simply be a marketing term coined by Elon Musk.

Data: Tesla's Unique Advantage

For Elon Musk, backing Dhaval Shroff’s neural network path planning project was not a risk-free or effortless decision. In fact, pursuing another innovation in algorithms toward neural networks often means Tesla must invest corresponding time, data, and computing resources for trial and error.

Even within Tesla’s internal teams, some members doubted the success of this neural network path planning project.

Fortunately, Dhaval Shroff achieved a breakthrough in about half a year, proving to Elon Musk that this was the right direction.

One contributing factor was Tesla’s massive data advantage.

Drawing on the extensive experience of Tesla’s autonomous driving team in data processing, Dhaval Shroff’s neural network path planning project analyzed 10 million video clips from Tesla customer vehicles over several months from late 2022 to early 2023.

According to Dhaval Shroff, the videos selected for data training were carefully curated.

Its primary criterion is that human drivers in the video clips must be able to handle various scenarios well. Only such video data is included by Tesla for training and learning. Additionally, the training process requires diversity in video content so that the neural network can learn from a wide range of human driving scenarios.

To this end, Tesla recruited a large number of human annotators in Buffalo, New York, who evaluate video segments and assign scores. Per Elon Musk’s instructions, these annotators identify what an “Uber five-star driver” would do, and the corresponding videos are then used for data training.

According to a case Dhaval Shroff once presented to Elon Musk involving neural-network-based path planning: on a demonstration road littered with trash cans, traffic cones, and debris, guided by the neural network’s path planning, the vehicle was able to navigate around obstacles, cross lane lines, and even break rules when necessary.

This case excited Elon Musk.

In April 2023, in Palo Alto, where Tesla’s autonomous driving R&D team is based, Elon Musk experienced the Tesla autonomous driving software version based on the new neural network path planning technology for the first time. Joining him were team members including Ashok Elluswamy, Tesla’s AI head, and Dhaval Shroff.

During this test, team members explained to Elon Musk how FSD is trained using millions of video clips collected by Tesla users’ in-car cameras. They also told him that the resulting software stack is much simpler than traditional software stacks programmed with thousands of explicit rules.

As Dhaval Shroff put it, “It runs 10 times faster, allowing us to delete 300,000 lines of code.” This statement later became a common talking point for Elon Musk when promoting Tesla’s end-to-end autonomous driving solution.

During this test drive, Elon Musk was so pleased with one specific maneuver performed by the test vehicle that he whistled on the spot. He felt it was better than what he would have done himself. Following this experience, Musk announced that he had become fully convinced of the project's importance and believed that significant resources should be mobilized to advance it.

It must be clarified that at this stage, Tesla’s autonomous driving team had already identified a clear fact: neural networks require training on at least 1 million video clips to achieve good performance. If the training volume reaches 1.5 million clips, the system becomes highly effective.

Fortunately, given Tesla’s massive global vehicle fleet (amounting to millions of cars), the company is able to collect vast amounts of video data daily for training purposes.

As Ashok Elluswamy stated, Tesla possesses a significant advantage in data acquisition.

Beyond data, Tesla also holds a major edge in computing power. The company has not only purchased large quantities of NVIDIA GPUs for its data centers but is also continuously advancing its self-developed Dojo supercomputer project.

Coupled with the FSD Chip deployed in vehicles, Tesla has built an integrated hardware-software framework spanning from the cloud to the car. This represents a unique advantage for Tesla in the autonomous driving industry when implementing end-to-end systems.

From End-to-End to Large Model Thinking

Currently, Tesla’s exploration in the field of end-to-end systems has achieved numerous breakthroughs.

In August 2023, Elon Musk already hosted a live stream on social media discussing Tesla’s end-to-end autonomous driving test version (FSD V12 Beta). Judging from the broadcast content, Musk was full of confidence. During the autonomous driving process, the vehicle made only one mistake related to traffic lights, yet the results had already attracted the attention of the entire industry.

Several months later, as Tesla further trained its end-to-end autonomous driving system, FSD V12 achieved further breakthroughs and gradually began expanding its internal beta testing scope to Tesla employees in December 2023. By January 2024, FSD V12 was pushed to media personnel in January, and finally in February, Tesla pushed FSD V12 to ordinary users within a certain range.

Based on current feedback, FSD V12, which is based on neural networks, has gained more recognition in urban scenarios compared to FSD V11, which still relies on manual code for planning and control.

One case in point: On March 6, deep learning expert James Douma evaluated FSD V12 after experiencing it. He stated that compared to V11, human intervention would decrease by more than 100 times. This was not an incremental upgrade but a leap; he found its power impressive.

He also noted that a series of behaviors effectively realized through end-to-end path planning and control indicate that the Tesla team could achieve 'better than human' goals relying solely on more and better data and training. Elon Musk responded to this assessment, calling it a very accurate evaluation.

Of course, from the current situation, Tesla's exploration in the field of AI has clearly transcended the scope of 'end-to-end' itself.

In fact, at CVPR, an international top conference in computer science last year, Ashok Elluswamy, head of Tesla AI, delivered a keynote speech titled 'Foundation Models for Autonomy.' It explicitly mentioned that Tesla is building some foundation models, among which the Occupancy Network is an important component that has been integrated but does not exist as an independent entity.

More importantly, Ashok Elluswamy emphasized that a true Foundation Model is not a mechanical superposition of many small tasks, but rather one capable of generating spillover effects.

He stated that Tesla is attempting to build a more general World Model, which can predict the future, help neural networks learn autonomously, function like a neural network simulator, and even generate 3D spaces in an AI manner (performing highly consistent 3D transformations across all eight camera views simultaneously based on human instructions such as left and right turns).

It is precisely on this premise that Elon Musk repeatedly emphasized when Sora emerged in February 2024 that Tesla has already built generative AI that better conforms to the laws of the physical world.

At the conclusion of this presentation, Ashok Elluswamy stressed that Tesla’s ability to construct the aforementioned foundation models stems from two factors: its extremely massive data volume and its already powerful computing infrastructure. Although his purpose here was to recruit more talent to join Tesla, fundamentally speaking, vast amounts of data and immense computing power indeed constitute the important pillars for Tesla to build large models based on the real world.

He also explicitly stated that Tesla FSD is built not only for Car Robots but also for Humanoid Robots.

Also at this CVPR conference, another member of the Tesla AI team, Phil Duan, stated in his presentation that Tesla is building a highly diverse, high-quality dataset, which it uses to train a Foundation Model. In Tesla’s view, this will be its future path to empower autonomous driving and Embodied AI through the construction of large models.

Interestingly, according to a video released by Elon Musk in the second half of last year, the Tesla Optimus humanoid robot has already adopted the same end-to-end neural network as its autonomous driving system, demonstrating significant capability evolution.

Looking at the current landscape, Tesla’s exploration of AI has entered a new phase starting with end-to-end systems: using a single foundation video network to drive cars directly. From a technological paradigm perspective, through algorithmic changes in end-to-end approaches, Tesla has shifted its overall thinking toward large models similar to OpenAI’s GPT.

Final Thoughts

Reviewing the past decade, we find that the underlying driver of autonomous driving technology development is often the evolution of AI itself.

For example, starting with AlexNet in 2012, deep convolutional neural networks (CNNs) became the common algorithm for perception in autonomous driving over the following years. By 2020, after Transformers had already empowered natural language processing for several years, Transformer was introduced into Tesla’s autonomous driving perception algorithm architecture to address efficiency and power issues in 3D spatial perception.

Then, at the end of 2022 and early 2023, with the emergence of ChatGPT, Tesla shifted toward end-to-end autonomous driving under the inspiration of large model thinking, entering a development path where it builds more general next-generation autonomy systems by training a single foundation large model.

However, whether referring to today’s various large models or Tesla’s end-to-end system, the key component in their underlying algorithm architecture remains the Transformer algorithm born in 2017, which is unlikely to be replaced in the short term.

Looking back, it is hard to deny that Tesla was significantly influenced by OpenAI and ChatGPT along this path. Behind this lies Tesla’s adherence to Scaling Law at the business level. From this perspective, under the leadership of Tesla and Elon Musk, the speed at which AI development directly impacts autonomous driving development is becoming increasingly faster. Meanwhile, the relationship between autonomous driving and AI is growing tighter.

Some argue that when AI reaches artificial general intelligence, fully driverless autonomous driving will also become achievable.

Interestingly, few noticed that in June 2020, Andrej Karpathy, a founding member of OpenAI and then head of Tesla AI, already shared his vision for the development of GPT and autonomous driving on Twitter. His exact words were:

The ultimate form of Autopilot should involve feeding the contents of the Driver’s Manual (DMV Handbook) into a 'large multimodal GPT-10,' along with sensor data from the past 10 seconds, to guide the vehicle.

At that time, Andrej Karpathy was already closely following Transformer and GPT technologies and linking them to Tesla Autopilot. Looking back from today’s perspective, everything we see and anticipate in the field of autonomous driving had actually been foreseen and laid the groundwork for back then.

Written by: 308 Edited by: Daji