The Year of Car-mounted 8 Million Pixels Cameras

2021 was called the “Year of LiDAR” not only because the release of models such as NIO ET7, FAW Bestune Alpha S Hi version, XPeng P5, Great Wall Motors ORA R1 and IM L7 in the first half of the year raised the expectation of the forthcoming landing of LiDAR, but also because we witnessed the magic of LiDAR in the capital market.

However, compared with the lingering taste of LiDAR in PPT, the 8 million pixels camera next door has already achieved mass production.

The Year of Car-mounted 8 Million Pixels Cameras

The first one to taste the crab was the 2021 IDEAL ONE.

The price of the 2021 IDEAL ONE has increased by 10,000 yuan, leaving many old car owners with mixed feelings. The improvement of product strength seems to tell everyone that the concept of “early purchase, early enjoyment” is not entirely applicable to the intelligent electric vehicle industry that is still in a rapid development stage. But these are all unimportant, because the new IDEAL ONE has obtained the world’s first car-mounted 8 million pixels camera put into mass production.

How Clear Is the 8 Million Pixels?

For example, the most mainstream 4K TV now, “4K” refers to roughly 4 thousand small squares showing different colors horizontally on the TV screen, which are pixels (actually 3,840 of them). And the TV has 2,160 pixels vertically.

Multiplying the number of pixels horizontally and vertically results in 8.3 million pixels. This is where the 8 million pixels camera comes from.

Currently, the main cameras equipped with ADAS in most models use around 1.3 million pixels, equivalent to a resolution of 960p. The camera of XPeng P7 has 2 million pixels, providing 1080p resolution, but it is only as clear as the first generation of Full HD TV, which is only one-fourth as clear as 4K or 8 million pixels cameras.

The comparison in the figure below vividly shows the imaging effects of two cameras with the same FOV of 30° but different pixel numbers, 7.3 million and 1.3 million, respectively, at a distance of 160 meters. The latter is already blurry within this range, while the former can still distinguish the text in the banner, the speed limit numbers, and objects on the ground. Therefore, under the same conditions, a camera with a higher pixel number can allow you to see farther and smaller things.

Actual Experience

What Does 8 Million Pixels Bring to Ideal?

The FOV of the new Ideal camera is 120°, equivalent to a wide angle. Compared with the old version, which had a FOV of 52° and a pixel number of 1.3 million, the pixel number increased by about 5 times, and the FOV also increased more than twice its original width.

Since the increased pixel number is greater than the amplitude of the FOV, the new Ideal ONE can be said to have improved recognition ability throughout the full FOV. In actual use, the turning ability of the 2021 Ideal ONE has significantly improved, and it has successfully passed the elevated ramp section known as the A-level devil bend in our 42Mark project, wiping out past embarrassments.

At this point, some may say that the effective viewing distance of the 8 million pixel camera on the new ONE is 200 meters, while the effective viewing distance provided by the 1.2 million pixel Model 3 is 250 meters. Isn’t this a contradiction?

The situation is not that simple, but it is not that complicated either. The reason why Tesla can see farther with a 1.2 million pixel camera is that it uses three such cameras, each with a different focal length. The one with the longest focal length and narrowest FOV sees the farthest. Compared with Tesla’s three fans, the ideal field of view is a single, larger fan that can provide a wider FOV even at long distances.

Apart from the improvement of the camera performance, the updated Idean ONE also adds four corner radars. However, the better cornering performance is not solely due to the corner radars. The main role of corner radars is to detect vehicles on the lane change trajectory, but they cannot detect lane markings or the direction of the road. After all, not all places are single lanes with walls on elevated interchanges. The turning ability mainly depends on the amount of semantic information on the road that the camera can provide.

NIO starts with eleven cameras

If the Idean ONE, which uses a single 8 million-pixel front camera, to compete with or even surpass the mainstream low-pixel 3-front-view camera perception solutions, is seen as a cost-effective action, then NIO’s luxury approach of installing 11 8 million-pixel cameras on the ET7, the first sedan model expected to be delivered in Q1 next year, is aiming to completely enhance its perception capability. From the comparison chart released by NIO, the recognition distance of vehicles, pedestrians, and obstacles has increased threefold with the use of the 8 million-pixel camera.

High-pixel cameras have only advantages in user experience. Seeing farther means being able to see vehicle targets and vulnerable road users (VRUs) earlier, allowing for more preventive measures and a smoother assisted driving experience.

And in the NoA (Navigate on Autopilot) scene, which will gradually be opened up in the near future, a wider field of view means more accurate recognition of small obstacles and crossing VRUs on the road, especially for corner cases that mmWave radars cannot handle.

The technology behind high pixels

The car camera module consists of main components such as lens, image sensor (CIS), and image signal processor (ISP). The lens focuses the light onto the CIS, which generates image signals and inputs them into the ISP for further image processing.

The resolution of a camera is determined by the number of pixels on the CIS. However, the overall image quality also depends on the performance of the lens, the capability of the CIS itself, and the functionality of the ISP.

In terms of photo and video quality, mobile phone cameras can easily outperform any car cameras. However, car cameras have more demanding requirements than consumer electronics, as they need to maintain stable operation under complex conditions such as high and low temperatures, humidity, low light, and vibration. The lens material is also necessarily glass, rather than plastic.

Due to space and cost constraints, the size of the CIS cannot be infinitely increased. High pixelization theoretically means that the size of individual pixels decreases, resulting in reduced performance in low-light conditions. However, this is clearly not allowed to happen.

Let’s take a look at the flagship car CIS product from the industry-leading company Hikvision to illustrate the necessary capabilities of a car CIS.

The PureCel® technology can reduce pixel crosstalk and noise, achieving better color reproduction in low-light conditions.

Compared to Hikvision’s 1.3 megapixel car CIS with a pixel pitch of 4.2 microns and a dynamic range of only 120 dB, this 8.3 megapixel car CIS with a pixel pitch of only 2.1 microns achieves a high dynamic range (HDR) of 140 dB and also has LED flicker mitigation and lower nighttime noise.

What Is High Dynamic Range (HDR) Used For? Imagine coming out of a dark tunnel and your camera cannot clearly recognize important targets, such as lane lines, in these scenes. This will have a significant negative impact on the vehicle’s auxiliary driving control and driving safety.

For smartphone users, the flicker of LED is only a professional issue for your photos. However, for cars, it may cause errors in reading road information. Therefore, LFM (LED flicker elimination) is a necessary function for car-mounted cameras, although it is not commonly found on smartphone cameras.

CIS manufacturers need to master the abilities of spatial domain exposure and dual conversion gain for the above capabilities. Among car-mounted camera CIS manufacturers, Howei and Sony have advantages in this area of technology reserve.

High-pixel car-mounted cameras are inevitable

Television, display, mobile phone, and camera equipment are all entering the 4K era one after another. Obviously, cameras corresponding to 4K and higher image quality are becoming mainstream. The pursuit of high pixels in consumer electronics will also be reflected in the emphasis on production capacity.

Although the research and development progress of car-mounted cameras will inevitably lag behind that of consumer-grade cameras due to stricter vehicle regulation requirements and more complex functional requirements, the demand in the consumer market is pushing for the technological upgrading of the car-mounted market.

The increase in chip computing power and the bandwidth of the in-car network are also one of the driving forces for pixelization. In order to cope with the ET7’s perception system generating 8 GB of data per second, NIO also equipped it with 4 Orin chips, with a high computing power of 1012 TOPS. Connecting the logic before and after, we can find that Moore’s Law is also one of the driving forces for increasing pixels to some extent.

Li Xiang’s Idea

Li Xiang said when releasing the new version of the Ideal ONE:

As the price of lidar becomes cheaper and cheaper, the price of the camera is getting more and more expensive. An 8 million pixel camera costs four or five times as much, and after using up 8 million pixels, people will start looking at 20 million pixels, which will be more expensive. In four or five years, a 20 million pixel camera may be priced like lidar.In the future, the 20 million pixel camera and lidar may not be the same price, but the camera will still be a critical component of intelligent driving systems even after the lidar is put into mass production. Lidar, which can only provide shape information visually, cannot completely replace the visual information provided by the camera, which is currently the most important source of information in autonomous driving algorithms.

Tesla’s Position

Musk’s attitude towards lidar is “whoever uses it is a sucker and whoever doesn’t use it is dead,” while Tesla’s head of autonomous driving, Andrej Karpathy, expressed a more rational view.

Karpathy said Tesla’s biggest advantage is:

1. The top vertical integration capabilities among mass-produced vehicles. Tesla can collect driving data from cars sold in millions.
2. The newly released Dojo supercomputer cluster can automatically label massive amounts of data and train its neural networks for rapid iteration.

Karpathy believes that if this system continues to improve, Tesla may completely phase out lidar.

The End of Resolution

According to the theory of imaging expert Roger Clark, the resolution of the human eye can reach over 570 million. In comparison, the most visually capable car model currently known, the NIO ET7, has a total of only 88 million pixels for external vision, far from reaching the level of the human eye.

If the ultimate goal is to achieve human-like vision, the resolution of a single car-mounted camera will reach the level of 50 million pixels, which is equivalent to 10K according to the rules of high-definition television.

The intelligent travel ecosystem has just begun

Compared with the explosion of smartphones, the concept of autonomous driving was ten years late. The smartphone market has gradually entered a stock competition, but the autonomous driving market has just begun. According to Yole’s report, the market size of automotive camera module will reach 8.1 billion US dollars by 2025.

Only when all cars on the road can reach L5 will autonomous driving be a standard configuration without performance measurement. Obviously, the arms race of perceptual capabilities is a long and continuous process.

Obviously, the car camera industry chain is a hot track that investors should pay attention to. Considering safety factors, automakers tend to choose component manufacturers with mature technology and guaranteed quality. Therefore, it takes a longer certification cycle for car component manufacturers to enter the market system and the industry barriers are higher, so the head effect will be significant.

I don’t know if you have noticed that the recently launched Tesla FSD beta V9 requires high beam on when driving at night. Similarly, in our 42Mark test, we found that when the Tesla with AP turned on passes the elevated curve at night, the high beam will flicker from time to time to achieve the function of supplementary lighting.

Increasing pixels to improve resolution allows the car to see more clearly in the spatial dimension, while improving night vision is equivalent to allowing the car to see more clearly in the time dimension. Therefore, the comparison of full-time-domain image processing capabilities is the factor that changes the current market share pattern under high pixelization.

This article is a translation by ChatGPT of a Chinese report from 42HOW. If you have any questions about it, please email bd@42how.com.