“`

Author: Su Qingtao

A few months ago, I asked a friend who develops both LiDAR and 4D millimeter-wave radar a question: since you produce both types of products, how do you view the possibility of 4D millimeter-wave radar “replacing low-line-count LiDAR”?

My friend didn’t give me a direct answer but asked another question: Can we still see so-called “low-line-count LiDAR” now?

Suddenly, I realized that the existence of 16-line, 32-line, and even 64-line LiDARs is indeed becoming weaker. In fact, as the number of lines of LiDAR increases, 96-line or 128-line LiDARs may become “low-line-count” in the future.

So, will 96-line or 128-line LiDARs replace 16-line or 32-line LiDARs, or will 4D millimeter-wave radar do so? This is an interesting topic.

Jiuzhang IMa recently released its “4D Millimeter-wave Radar Report“——

Foreword:

The position of millimeter-wave radar in the configuration of automatic driving sensors is rapidly increasing.

As the level of automatic driving continues to improve, the degree of participation of automatic driving systems in the vehicle driving process is constantly increasing, and traditional millimeter-wave radar is becoming less and less capable. In order to meet the requirement of high-level automated driving systems that require full-target, full-scenario, all-weather coverage of sensing modules, millimeter-wave radar must move towards the “high-definition” direction.

The millimeter-wave radar with “high-definition” characteristics is called an imaging radar or 4D millimeter-wave radar. “4D” refers to the addition of high-dimensional data analysis of targets on the basis of the original distance, azimuth, and speed, which can achieve information perception in four dimensions of “3D + height.” The concept of “imaging” means that it has ultra-high resolution and can effectively resolve the contours, categories, and behaviors of targets.

Facing the same obstacle, the millimeter-wave radar can only receive a limited number of returning information points, and can only determine that there is an obstacle in front, while 4D millimeter-wave radar can receive dozens of times more returning information points to evolve into a high-density point cloud like LiDAR, and can further detect the shape of objects, and even identify objects by combining algorithms.

This means that compared to traditional millimeter-wave radar, the 4D millimeter-wave radar system can adapt to more complex road conditions, including identifying smaller objects, detecting partially obscured objects, detecting stationary objects and lateral-moving obstacles, etc.

And it is precisely thanks to these characteristics that the appearance of 4D mmWave radar has instantly elevated the “coolness factor” of mmWave radar.

Active players in the 4D mmWave radar market include traditional Tier 1 players such as mainland China, ZF, and Bosch, technology giants such as Waymo, Mobileye, and Huawei, as well as many start-ups such as Arbe, RoboSense, Valeo, Novelda Electronics, and Geoharmony.

mmWave Radar is Striving to Rise from Supporting Role to Lead Role

Since the second half of 2020, if a new mass-produced vehicle is about to be equipped with a LiDAR system, automakers will surely emphasize this “added value” in their promotions, particularly the name of the LiDAR supplier. By contrast, the mmWave radar, which has accompanied the automobile industry for decades, has not enjoyed such recognition.

From the perspective of automakers, a “star component” like LiDAR and its supplier can increase their brand value. By contrast, the mmWave radar is seen as “just too ordinary.” However, this situation is changing. For example, in March 2021, SAIC R brand promoted its new SUV ES33 as “equipped with ZF’s 4D mmWave radar PREMIUM.”

Behind this change is the fact that 4D mmWave radar, which has superior performance when compared to traditional mmWave radar, is striving to become a sensor that can “stand alone,” thus transforming mmWave radar from supporting role to lead role.

Compared to cameras, traditional mmWave radar has strong capabilities in ranging and speed measurement, and is not affected by weather or visibility. However, due to its limited resolution, it often fails to clearly outline obstacles ahead (especially small targets) and the contours of trees and curbs on both sides of the road.

By contrast, 4D mmWave radar has more antennas, higher angular resolution, speed resolution, and distance resolution, so it can more effectively resolve the target’s contour, category, and behavior without the participation of LiDAR. Therefore, it can more easily determine when to apply the brakes and when not to.

For example, an autonomous driving system needs to obtain information about the deviation of a target vehicle’s driving lane position from 200 meters in front of the vehicle, and the traditional front radar’s azimuth accuracy is about 0.3 degrees, resulting in a large position error for the target vehicle’s lane judgment in this scenario. By comparison, 4D mmWave radar’s azimuth accuracy can reach 0.1 degrees, 3 times that of traditional front radar azimuth, and can output more accurate lane deviation information to the decision-making system at 200 meters.For example, traditional millimeter wave radars have only a 20% chance of detecting hidden vehicles, while the 4D millimeter wave radar ARS 540 is said to have an 80% chance of doing so. In addition, ZF’s long-range 4D millimeter wave radar is said to be able to receive data points from about 10 pedestrians, even measuring the movement speed of these data points to analyze the motion trajectory of individual limbs, thereby identifying the direction of pedestrian movement.

Usually, millimeter wave radar targets rely on four dimensions, angle, height, distance and velocity, to work together. However, when two cars are traveling at the same speed and in the same direction (very close), they are actually in a “relative static” state, so the velocity dimension is invalid. At this time, if the resolution of the millimeter wave radar is not high enough, it is easy to regard these two cars as the “same car.”

Because the point cloud density of 4D millimeter wave radar is relatively high, even if the velocity dimension and distance dimension are invalid, the probability of “guessing” the target is still relatively high.

Moreover, different from common algorithms, the recognition accuracy is easy to be affected by the incompleteness of target object samples in the database. The 4D millimeter wave radar is more efficient in classifying target objects by increasing the point cloud density – reducing the time spent on calculation. Traditional millimeter wave radars + algorithms may need to scan many frames to identify weak obstacles, while a 4D millimeter wave radar may only need to scan 1-2 frames to complete the task.

There is another common scenario where “people and vehicles are not distinguished”: a person standing next to a parked vehicle.

The camera has no penetrating power and cannot see, while the traditional millimeter wave radar, due to its insufficient resolution and because the vehicle’s energy density and reflection intensity are higher than those of humans, the millimeter wave radar’s point cloud scanning on humans can easily be “absorbed” by the vehicle, resulting in the person being mistakenly identified as “part of the vehicle.” Similarly, when a small car is very close to a large car, the millimeter wave radar may mistake it for “part of the large car.”

Obviously, the person or small car that is “merged” is actually being ignored by the traditional millimeter wave radar, which is extremely dangerous. But the 4D millimeter wave radar has “high dynamic resolution” and can differentiate multiple obstacles with large differences in reflection intensity in the same environment, distinguishing between a large car and a small car, and that between cars and people. With clearer target resolution, the millimeter wave radar is more capable of supporting decision-making systems.Under a multi-sensor fusion solution, the 4D millimeter-wave radar can also “guide” the cameras and lidars to potential risk areas, greatly improving safety.

For example, detecting stationary objects is a challenge for traditional millimeter-wave radars as they cannot obtain height data without longitudinal antennas (although Bosch and Continental’s forward-facing millimeter-wave radars can output height data, their accuracy is insufficient). The traditional millimeter-wave radar finds it difficult to determine whether a stationary object ahead is on the ground or in the air, which results in confusion between low and small “obstacles” on the road (such as manhole covers, speed bumps, metal on the roadside, etc.) that do not require braking and high “aerial obstacles” (such as traffic signs, gantry frames, overpasses, etc.) that also do not require braking with static obstacles on the road (which require braking).

Therefore, if the traditional millimeter-wave radar is used as the primary sensor, it may lead to frequent false braking. To avoid false braking, the AEB algorithm decides to reduce the confidence weight of the millimeter-wave radar (even filter out the static obstacles) and rely more on visual perception. This is the truth behind the widely circulated “millimeter-wave radar cannot recognize static obstacles” statement.

However, the challenge of visual perception is that single-eye and three-eye cameras must first recognize (classify) the targets before detection, which not only requires good lighting but is also highly dependent on the target model library. The model library cannot exhaust all types, which means that many static obstacles become “neglected” in visual perception. As a result, it often happens that even though there are static obstacles ahead, the autonomous vehicle still crashes into them.

A friend who used to work as an ADAS algorithm engineer in a traditional car company told me that three years ago, the project he was working on was stuck on the “millimeter-wave radar cannot recognize static obstacles” issue. At that time, due to the physical limitations of the millimeter-wave radar, this problem was unsolvable, but the leader did not understand and demanded that he “must solve it.” In the end, the engineer was forced to resign.

But now, with the 4D millimeter-wave radar, the tragedy of engineers being forced to leave their jobs because they cannot solve the problem of recognizing static obstacles can be avoided. At least the probability of it happening will be greatly reduced because the “standard” longitudinal antenna (used specifically to measure height) of the 4D millimeter-wave radar provides data dimensions of vertical resolution. The return signals of obstacles in front are no longer roughly arranged on a two-dimensional ground but presented in a three-dimensional space. This helps to “treat differently” all kinds of static obstacles of different heights.In short, from the perspective of the decision-making system, compared to traditional millimeter-wave radar, the detection results of 4D millimeter-wave radar have higher confidence. Therefore, the decision-making system does not need to worry too much about frequent misbraking caused by path planning based on the perception results of 4D millimeter-wave radar. Therefore, the weight of 4D millimeter-wave radar can be ranked ahead of the camera in many functions.

What impact will the increase in weight of 4D millimeter-wave radar have on driving safety? Let’s imagine such a scenario: the car in front of the car in front has already braked, but the car in front hasn’t reacted and cannot brake in time. What will the “self-driving” car do?

Traditional millimeter-wave radar can penetrate the car in front and detect what happened to the car in front of it, but unfortunately, this detection result is often “ignored” by the decision-making system. On the other hand, as a sensor, the camera does not have penetration power and cannot see what happened to the car in front of the car in front. Therefore, in situations where the speed is relatively high, a chain collision is highly likely.

However, if the 4D millimeter-wave radar is used, the result will be different: due to its higher reliability than traditional millimeter-wave radar, the detection results of 4D millimeter-wave radar are highly valued by the decision-making system. Therefore, in the event that the car in front collides with the car in front of it due to its inability to brake in time, the “self-driving” car can brake in time because the decision-making system has received and trusted the warning of “danger ahead”.

With the assistance of 4D millimeter-wave radar, not only can the “self-driving” car avoid recklessly colliding with the car in front, but its braking action will also have a positive feedback effect on the car behind, which can prevent it from being hit from behind and even save the car behind and the car behind it.

This is an ability that even laser radar does not have (however, a development engineer of a certain new automaker said: “Although this method is theoretically feasible, it has not yet become a routine detection method in large-scale production projects”).

Last year, French market research institution Yole pointed out in the “2020 Radar Industry Situation Report: Manufacturers, Applications, and Technology Trends” that after the application scenarios have become more stringent, millimeter-wave radar is advancing towards 4D millimeter-wave radar that can more accurately describe the front and rear scenes of vehicles.

Currently, many companies are exploring the possibility of replacing low-line laser radar with 4D millimeter-wave radar.

Besides, DMS and vital sign monitoring systems in the cabin are also becoming important applications of 4D mmWave radar.

Tesla CEO Musk’s movement to “kill mmWave radar” in 2021 once made many people skeptical about the application of 4D mmWave radar in the autonomous driving market. However, in fact, when Musk denounced various drawbacks of mmWave radar, he mainly targeted traditional mmWave radar while he did not rule out the “high-precision mmWave radar.”

As early as October 2020, Musk mentioned the plan to use 4D imaging radar, and in September of this year, Tesla’s self-developed mmWave radar certification application to the FCC (Federal Communications Commission) has been approved. Due to confidentiality orders, the specific parameters and uses of this mmWave radar are unclear until December of this year. However, from the publicly available testing reports, it is learned that this is a 77 GHz radar with a 6-receiver and 8-transmitter system in antenna settings.

As of now in China, several car models priced between 250,000 and 400,000 yuan plan to install 4D mmWave radar on the front bumper. It is believed that more and more car companies will realize the value of 4D mmWave radar and eagerly bring it up at their future new product launch events.

Main players and technical routes of 4D mmWave radar

Currently, the main players in the 4D mmWave radar market include traditional Tier 1 companies such as Continental, ZF, Bosch, and Infineon, as well as autonomous driving solution providers like Waymo, Mobileye, and Huawei, and startup companies such as AkuSense, Arbe, Geoharmony, Chuhang Technology, and SenseTime.

Regarding the technical route that improves resolution by increasing the number of antennas, there are currently three main solutions: “cascade,” cascade + virtual aperture imaging, and integrated chip.### 1. Cascade

Cascade is the process of increasing physical antenna MIMO (the number of virtual channels obtained by multiplying the number of receiving antennas by the number of transmitting antennas) by cascading the 77G and 79G standard radar chips (MMIC chips) of companies such as Infineon, Texas Instruments, and NXP.

A cascade of two chips is to connect 2 chips of 3T4R, forming 6T8R. A cascade of four chips is to connect 4 chips of 3T4R, forming 12T16R, creating 192 virtual receiving channels, such as the mainland’s ARS 540. Navtech’s 18T24R product is a six-level cascade. Bosch, ZF, Waymo, and Huawei all use the cascade method.

The advantage of this solution is that the pre-development difficulty is low, so the time to market is relatively short. However, the disadvantages are increased volume, high cost, high power consumption (simultaneous operation of multiple chips will increase power consumption), insufficient signal-to-noise ratio (interference between multiple MMIC chips), algorithm adaptation, and other issues.

In addition, compared with traditional millimeter-wave radar, the multi-chip cascade scheme not only has a more complex antenna layout but also a much more complex layered structure of the PCB board. For example, if there are a total of six layers of boards, the materials and expansion coefficients of each layer of the board may be different, which may cause the board to warp and affect the energy utilization rate.

In addition, Jiuzhang IMa has learned that several 4D millimeter-wave radar manufacturers using the cascade technology route have encountered the technical problem of “intermediate frequency synchronization”-for example, in a four-level cascade, the 20G intermediate frequency signals of four chips need to be synchronized. Moreover, when two boards are pressed together, the yield cannot be improved.

Therefore, if the cascade scheme is adopted, it is relatively easy to make a demo based on the reference design and SDK provided by the chip manufacturer, but the threshold for mass production is high, and only those companies with extremely strong technical and engineering capabilities can do it well.

2. Cascade + Virtual Aperture Imaging Technology

Cascade + Virtual Aperture Imaging technology refers to using the existing chips to achieve high-multiplication virtual MIMO based on the cascade method, and then further virtualizing the antenna number by using a unique virtual aperture imaging software algorithm and antenna design. It successfully increases the angle resolution from 10 degrees directly to 1 degree based on the original physical antenna number.

The traditional radar waveform is single-frequency, repetitive, and non-adaptive. The only way to produce multiple waveforms is to increase the number of receiving antennas. The virtual aperture imaging waveform is adaptive phase modulation (frequency modulation + phase modulation + amplitude modulation). Each receiving antenna produces a different phase response at different times, and then the data is interpolated and extrapolated to create a “virtual aperture,” which greatly improves the angular resolution.

From public information, Autoliv is a representative manufacturer that uses this cascaded and virtual aperture imaging technology. Autoliv is positioned as a Tier 2 supplier and provides signal processing algorithms for 4D millimeter wave radar without doing hardware (hardware is provided by partners such as Hella). In October 2021, Autoliv’s independently developed 4D millimeter wave radar AI algorithm and AD4D millimeter wave radar technology were acquired by Aptiv.

The more virtual channels there are, the more complete the received signal is and the clearer the detection result. Xie Jianjun, head of Autoliv’s marketing department, said, “After using virtual aperture imaging technology, our single-chip can achieve the effect of four-stage cascade products of other companies in terms of resolution, and the two-stage cascade can achieve the effect of six-stage cascade of other companies.”

However, some people in the 4D millimeter wave radar industry questioned that this technology “does not comply with the laws of physics” in theory.

If someone says, “Although increasing the number of antennas by frequency modulation, amplitude modulation, and phase modulation can optimize the product performance in some areas, it is reasonable that software optimization cannot fully overcome the hardware’s inherent limitations. Because under the fixed point frequency, the number of electromagnetic waves cannot be increased by adjusting the software despite the way the electromagnetic waves are scattered, so if this part is dense, then that part is sparse.”

However, an engineer who has had close cooperation with Autoliv said, “In terms of principles, I don’t believe what Autoliv said is true, but the product testing shows that the angular resolution is indeed high, and the ranging accuracy of target objects within 50 meters can reach 0.1 meters.”

The barrier to virtual aperture imaging technology primarily lies in the layout of antennas and waveforms. The antenna layout mainly affects the size of the virtual aperture, while the waveform mainly determines the number of channels. In addition, increasing the number of antennas also imposes higher requirements on subsequent data processing capabilities.

Virtual antenna technology completely solves the problem that the automotive millimeter wave radar industry has been facing for decades, which is that only increasing the number of physical antennas can improve angular resolution. This technology greatly improves the angular resolution and keeps the cost at a reasonable level.### 3. Integrated Circuit

The so-called integrated circuit solution refers to integrating multiple transmitting and receiving antennas into a single ASIC chip to achieve the above functions. Currently, representative companies in this technology include Arbe, Uhnder, Vayaar, SteradianSemi, RFISee, etc. The most typical is the 4D millimeter-wave radar RFIC chip developed by Arbe company, which integrates 48 transmitters and receivers and has over 2300 virtual channels.

Integrated circuits can greatly reduce the size of 4D millimeter-wave radar and achieve the most advanced RF performance at the lowest cost per channel in the market. However, the implementation difficulty of the integrated circuit solution is also much higher than that of the cascade solution, and the main challenges are:

1. How to arrange so many antennas in a very small closed space;

2. How to overcome the problem of mutual interference between antennas;

3. How to reduce power consumption and heat dissipation;

4. How to improve the signal-to-noise ratio. If the signal-to-noise ratio cannot be improved, the effective detection distance is very short. (Uhnder claims that their integrated chip is a digital frequency modulation chip with anti-interference capabilities);

5. The chip solution is ASIC. Once the chip is taped out, the algorithm is fixed. Afterwards, the algorithm can only be adjusted for individual parameter configurations for specific scenarios, but cannot make major adjustments to the function.

The first few problems can be overcome by engineering techniques, while the last one is unsolvable.

The head of a certain 4D millimeter-wave radar manufacturer said that currently, 4D millimeter-wave radar is still a new product, and automakers have not yet used it on a large scale. Therefore, 4D millimeter-wave radar manufacturers cannot determine which parameter automakers will prefer. “Once the millimeter-wave radar manufacturer tapes out a certain version of the integrated chip, if the automaker finds that the algorithm of 4D millimeter-wave radar needs major changes during testing, Arbe will have to retape out the chip, which not only increases the cost, but also affects the progress of putting it on the car.”

The above-mentioned person believes that Arbe’s integrated chip solution may be most suitable for mass production after 5 years of 4D millimeter-wave radar being put on 20 automakers’ 100 car models, and the algorithm has been fixed. At this time, manufacturers can easily and quickly reduce costs and establish a moat by making integrated chip solutions. “We may also make integrated chip solutions in the future, but not now, because if we change the car model, the product may not be usable.”“`markdown
There is another “super material route”, but we won’t discuss it in this article as current research on super materials is still in the laboratory stage and commercialization is difficult to achieve in the short term.

Integration of Hardware and Software with Machine Learning

Even if the product is good, it can be difficult for car companies to “use” it.

As traditional millimeter-wave radar upgrades to 4D millimeter-wave radar, the ceiling of hardware has been broken, requiring higher algorithmic abilities from the system.

Compared with 3D millimeter-wave radar, the point cloud of 4D millimeter-wave radar has increased significantly. Therefore, it is a huge challenge to weed out false alarms or unnecessary point clouds and select the needed point clouds to be applied to the functional level.

Previously, since the hardware did not have the ability to detect small obstacles, the algorithm solved problems by “guessing”. Now that the hardware capabilities have improved, the algorithmic ability must also be upgraded from “guessing” to “analysis”, or else it would squander the improved hardware.

For example, how to separate similar targets, how to find failed scenarios, how to filter interference, and how to set an appropriate signal-to-noise ratio threshold (if the threshold is too high, small targets are easily leaked; if the threshold is too low, there may be errors) all require powerful software algorithmic capabilities.

But currently, most car companies do not have the millimeter-wave radar algorithmic capability.

A project manager from a millimeter-wave radar manufacturer said, “In projects that will be mass-produced from 2022 to 2023, most of them will basically use the results of millimeter-wave radar data processing. Most car companies still do not have the ability to truly use the point cloud of millimeter-wave radar.”

For a long time, millimeter-wave radar manufacturers have often provided integrated hardware and software solutions, where the algorithms are already integrated into the hardware. For car companies, millimeter-wave radar directly outputs sensing results, and they only need to fuse these results with the recognition results of other sensors.

Although car companies all claim to want to “decouple software and hardware” and hope to do the algorithms themselves, in reality, the barrier to entry for millimeter-wave radar algorithms is extremely high, and only a very small number of car companies can manage it. Therefore, for the majority of car companies, the integrated hardware and software solutions provided by manufacturers are still the preferred option.

Based on the results of research by JZ Intelligent Driving, currently the algorithms for processing 4D millimeter-wave radar data, such as ACC, AEB, BSD, LCA, are basically done by radar manufacturers. “Some OEMs are working on millimeter-wave radar algorithm development, but from the perspective of overall industry development, it is still quite difficult for OEMs to make it from point clouds to targets and then to functions.”“`
However, what embarrasses carmakers is that the algorithm of 4D mmWave radar is much more complex than that of traditional mmWave radar. Some manufacturers struggle to handle the algorithm themselves and can only deliver the hardware. According to a cooperation case between an international host factory and a German Tier 1, the host factory had to personally undertake the algorithm that they were not good at.

Why is it difficult for even the top Tier 1, who has accumulated decades of experience in traditional mmWave radar algorithms, to handle the algorithm of 4D radar?

One major difference is that traditional mmWave radar defines targets as “point targets”, while 4D mmWave defines targets as “extended targets”. Therefore, their signal processing and point cloud processing architectures are different.

Additionally, a technical manager of a 4D mmWave radar manufacturer said that for traditional mmWave radar algorithms, it only needs to do simple data clustering processing, while 4D mmWave radar’s algorithm needs to do target classification, and requires functions around AVP, HWP, and TJA, which are usually done by algorithm companies or hardware technology companies with strong algorithm expertise.

It is learned that when competing for a project of a domestic host factory with some foreign companies, the reason why a certain domestic mmWave radar manufacturer can quickly win is that, in addition to providing higher quality and quantity of point clouds, they can also provide 4D mmWave radar algorithms, which is also a key reason.

The difficulty in writing 4D mmWave radar algorithms has become an opportunity for BlueSpace.ai, a company that specializes in providing sensing software technology solutions for 4D radar, lidar, and other sensors.

Of course, mmWave radar manufacturers are not willing to hand over their “soul” to their partners. In fact, many companies in the 4D mmWave radar industry chain have planned that in the future, algorithm capabilities will be built as a core competitiveness. The introduction of machine learning algorithms is one of the brightest spots in this series of actions.

Previously in the report on “high intelligent vehicles”, an industry insider said: “In the past, because the radar resolution was very low, you couldn’t do any machine learning-related post-processing. Now, because 4D mmWave radar can produce point cloud data similar to lidar, machine learning can be used to train the radar perception system to identify objects, especially to help solve the edge detection problems that traditional radar cannot overcome.”

This trend has also been verified by more and more companies. For example, NXP has launched a vehicle-level AI tool kit, which, in addition to being applied to the traditional visual field, will also use a neural network to classify road users and obstacles based on its point cloud images for 4D mmWave radar.

According to the plans of radar chip manufacturers such as NXP and Texas Instruments, the next step for 4D millimeter-wave radar is to enhance its machine learning capabilities similar to camera algorithms.

From the beginning, Arbe has been thinking about how to combine signal processing and artificial intelligence on existing RF chipsets and digital signal processors (DSPs) to achieve real-time clustering, tracking, self-positioning, false target filtering, and target classification based on radar and radar + camera.

This AI algorithm can identify whether the detected object is a person instead of a tree and calculate its position within one second. It also integrates with other sensors in the camera and system to classify and match detected objects from multiple sensors.

According to “High-Tech Intelligent Cars”, Ampofo previously provided a series of data on the combination of 4D millimeter-wave radar and machine learning capabilities.

For example, for small objects or debris on the road, machine learning can increase the detection distance by more than 50% and track small objects within a 200-meter range. Compared with classical radar signal processing, machine learning reduces missed detections by 70%. Machine learning can also reduce the position error and target heading error by more than 50%, which means that the vehicle can better identify vehicles parked in other lanes, as well as stationary or slowly moving objects.

For another example, tunnels are a challenging environment for traditional millimeter wave radar-the tunnel walls are a huge reflective surface and may cause many return points, and may even exceed the radar’s target processing capabilities. However, Ampofo believes that machine learning can help vehicles understand when to enter tunnels, and can more accurately filter out noise in detection than classical methods. At the same time, it can better explain radar echoes in tunnels and other enclosed environments and classify sector and other targets.

In addition, Qualcomm does not produce millimeter-wave radar itself, but they claim that they can expand the performance of radar through deep learning on radar. For example, by using the “Radar Deep Neural Network” developed internally by Qualcomm, higher resolution and 3D scanning can be obtained from enhanced radar.

Therefore, the competitiveness of a 4D millimeter-wave radar will be largely affected by the deep learning algorithm capabilities of the manufacturers and partners.

Long and stony path: Challenges in 4D mass production applications

In late 2020, XPeng announced that it would use lidar on the P5, and other OEMs (both new and traditional OEMs) followed. It can be said that lidar has triggered an arms race among automakers. However, in the 4D millimeter-wave radar market, whether it is BMW’s cooperation with Continental or SAIC R-Brand’s cooperation with ZF, it seems to have little influence on the decisions of other automakers.According to research by JZ Smart Driving, only a few automakers will adopt 4D mmWave radar in their new models in the next year or two.

Why are most OEMs still cautious about 4D mmWave technology? What are the obstacles to its mass production? (The difficulties in product development have been introduced in Section II. Here, we focus on the challenges in its application.)

1. Technical and Engineering Challenges

(1) Multiple indicators need to meet the conditions simultaneously

According to a perception engineer at an OEM, many 4D mmWave radar manufacturers emphasize the advantages of their products in a single indicator such as ranging resolution, angle resolution, or velocity resolution. However, the significance of a single indicator to the final imaging is limited. “In fact, to achieve a good imaging effect with higher confidence, it is necessary to simultaneously improve distance resolution, angle resolution, and velocity resolution.”

(2) Front fusion is difficult to achieve

To fully leverage the technical advantages of 4D mmWave radar, front fusion with cameras is needed. There is a confidence issue with back-end fusion: if both sensors detect the target or one sensor detects the target, which one should we rely on? Therefore, front fusion is necessary. However, front fusion is not an easy task due to the following reasons:

A. For a long time, most OEMs have not “seen” the original data of mmWave radar, given the black-box mode of mmWave radar manufacturers which only provide integrated hardware and software. They do not really understand the characteristics of this data (the data format of mmWave radar is different from that of cameras), so there are too few people who truly understand the point cloud attributes of mmWave radar.

Nowadays, most companies are beginning to learn mmWave radar (5-6 years later than learning LiDAR), and even writing mmWave radar algorithms is difficult, let alone front fusion with cameras. Moreover, currently, there are not many 4D mmWave radar samples in the market, and downstream customers have few opportunities to study them.

B. 4D mmWave radar has multiple channels, with relatively large data size requiring high computing power for front fusion with vision. The computing power of the sensor is not sufficient-the memory of mmWave radar chips is limited, and processors need to handle FFT transformation, CFAR, filtering, etc., and cannot process too many point clouds. Therefore, if the point cloud density is relatively high, front fusion needs to be performed in the domain controller.

Currently, the new generation of AK 4D mmWave radar follows the algorithm + central domain control technology route. It is claimed that a single radar can achieve a resolution of 0.1°*0.1° and hundreds of thousands of points per second.

However, if the main control chip is placed in the domain controller, it will not only pose challenges to the centralized architecture of the high data rate and compression of the 4D mmWave radar, but also affect the detection accuracy due to the bandwidth and speed of the signal transmission between the antenna and processor. This means that the idea of ​​data calculation in the domain controller is also difficult to implement.

To solve the above contradiction, 4D mmWave radar manufacturers need to have a deep understanding of the central domain controller, or have a deep binding relationship with a domain controller manufacturer or chip manufacturer. AK can put the algorithm of 4D mmWave radar in the central domain controller, which has a lot to do with their acquisition by Ambarella – Ambarella’s domain control chip CV3 will take out a small piece to do all the signal processing of the mmWave radar (can handle 6 radars).

C. The calibration of 4D mmWave radar and camera is difficult before fusion – the 4D mmWave radar’s understanding of semantic information is not accurate enough, and its target classification is also inaccurate. In addition, the 4D mmWave radar has distance information, while the camera does not. Therefore, in the joint calibration of the two, how to handle confidence and reliability in the visual and 4D mmWave radar point cloud level, and in what situation the accuracy of which sensor is higher, is a big problem.

(3) EMC is difficult to pass

The electromagnetic compatibility (EMC) of 4D mmWave radar is difficult to pass. The key is to consider how to avoid interference from the outside world and how to combat interference from the outside world. Among them, interference from the outside world includes interference from objects outside the car and car radios inside the car, which are related to the emission frequency of electromagnetic waves (including EMI and EMF). The problem of EMC is generally difficult to discover in the early simulation stage, but can only be discovered in the experimental process.

(4) Immature testing equipment

Usually, car companies or Tier 1 will use radar simulators in the laboratory to test the hard conditions before road testing. 3D mmWave radar has mature testing equipment, but 4D mmWave radar currently has only been released by Schwartz. Although it was designed according to specifications during the design, there is no equipment to verify it.(5) Testing standards are not clear

In the past, testing for 3D millimeter-wave radar only required horizontal detection capability, while 4D millimeter-wave radar, with height information, will require increased testing in higher dimensions. However, standards and methods for height testing still need to be explored.

(6) Difficult to install

4D millimeter-wave radar is generally larger in size and has a different shape than traditional millimeter-wave radar, so the installation position is not easy to design.

2. Policy issues: Lack of national standards

Currently, there is a lack of national standards for 4D millimeter-wave radar, so many automakers are concerned that after installing on a large scale, it may not be usable.

For example, in sensitive areas such as road sections within one kilometer of an observatory, 77G millimeter-wave radar is not allowed to be turned on. However, if 4D millimeter-wave radar is on most of the time, it is also a problem to turn it off in these sensitive areas.

3. Commercial issues: Cost-effectiveness is not high enough

4D millimeter-wave radar is heterogeneous with LiDAR, and many engineers do not believe that it can replace LiDAR.

Even engineers who have worked in the 4D millimeter-wave radar manufacturer and automakers’ autonomous driving perception teams have candidly said, “From our test data, at this stage, 4D millimeter-wave radar not only cannot replace LiDAR, but its advantages compared with mature 3D millimeter-wave radar are not so obvious. I think 4D millimeter-wave radar will take at least 2-4 years to mature.”

The market director of a millimeter-wave radar manufacturer also said, “The imaging effect of 4D millimeter-wave radar is much worse than LiDAR, and it is completely unnecessary to compete with LiDAR with its weaknesses; 4D millimeter-wave radar cannot replace LiDAR in the short term, nor can it be used as the primary sensor.”

Although the product is not easy to use, the price is not low, and as a result, people have found that the current cost-effectiveness of 4D millimeter-wave radar is not high enough compared to other options.

The price information of 4D millimeter-wave radar is currently quite confidential, so we are not able to disclose too much in the article.

In conclusion: Need for survival or poetry and a distant goal?

According to a securities person who has long followed the autonomous driving industry-

Currently, start-ups on the 4D millimeter-wave radar track are all in the stage of desperately raising funds. Since last year, whether investors invest in you or not depends on whether you have mass production orders. Therefore, from the perspective of financing, they focus on the more easily mass-produceable 3D millimeter-wave radar, which is more advantageous than focusing on 4D millimeter-wave radar.However, in the battlefield of 3D millimeter wave radar, they still have to compete with companies like Bosch, Continental, and ZF, thus facing tremendous pressure. As a result, they may not have much energy and capital to develop 4D millimeter wave radar.

This article is a translation by ChatGPT of a Chinese report from 42HOW. If you have any questions about it, please email bd@42how.com.