Author: Jessie

Intelligent driving functions have gradually become recognized and accepted by people, thanks to their popularity. Of course, compared to the traditional functions of cars, we hope to see a state of mutual adaptation and achievement. In the development process of various intelligent driving functions, a greater consideration should be given to their impact on traditional driving functions of the vehicles, especially for those intelligent driving system functions that affect the basic functions of vehicles. This article takes several major related functions of intelligent driving system as an example to illustrate how the design of intelligent driving system functions can achieve the optimization of the entire vehicle control.

This series of articles will provide readers with detailed analysis of how to integrate these basic automobile functions into advanced intelligent driving functions, in order to offer more front-end design thinking and optimization strategies for ADAS function development.

Sentinel Mode

Sentinel Mode is a function that has been well-known in the field of intelligent automobiles, which was pioneered by Tesla. It works by continuously monitoring the vehicle through real-time panoramic cameras in the parking system. When suspicious behavior is detected, the vehicle can issue different degrees of warning behaviors, such as sounding an alarm, dialing the owner’s phone number, and turning on the alarm lights. At the same time, the suspicious behavior will be automatically recorded, saved, and uploaded, which effectively reminds the owner of a possible accident.

How to Define the Working State of Sentinel Mode?

The entire working premise of Sentinel Mode is to park and turn off the vehicle. Users can enable it through the audio-visual entertainment system or the mobile phone app. The Intelligent Cockpit Domain Control Unit calls the panoramic camera controlled by the Intelligent Driving Domain Control in the lowest power consumption state to transmit images. The entire working process involves the following elements:

Intelligent Driving Domain Control: Responsible for the integration of the panoramic camera. Since the panoramic camera is mainly used for parking control, it is directly connected to the intelligent driving domain control for real-time driving and parking assistance scenarios.

As shown in the figure above, the Sentinel mode works in the hardware system and the Sentinel mode software runs in the central driving domain control host software. The display port is displayed on the IVI assembly of the host’s peripheral or on the smartphone display screen. Overall, it is required that the link between the cabin domain control host and the smart driving hardware system must ensure that the video access must have low latency (such as less than 40ms transmission delay at a frame rate of 20fps). Secondly, after receiving the vehicle instructions, smart driving needs to efficiently turn on the camera and open the video recording port, and the execution efficiency or delay of its turn-on instruction cannot exceed 3s. Most importantly, there must not be timing asynchrony or timing disorder.

The tasks undertaken by each part are as follows.

1. Video Source Input:

The surround-view video input is the image outside the near vehicle body captured by the surround or panoramic camera and transmitted to the central smart driving domain control.

2. Video Data Processing:

For smart driving, it is necessary to convert the data from the panoramic camera into data formats that the host intelligent cabin domain control SOC can recognize. During this period, it is also necessary to maintain a certain video level format and meet the established maintenance timing. For smart driving, it is necessary to receive vehicle controller and other control instructions to turn on the panoramic camera, generate synchronization timestamps, and input them to the receiving end intelligent cabin domain control after the synchronization is completed.

3. Work Status Management:

The video transmission between smart driving and intelligent cabin domain control is generally conducted directly. Smart driving needs to provide feedback on the opening and closing status and abnormal states of the camera, and at the same time provide feedback on the working status of smart driving itself (such as Working, Standby, OFF, Sleep, and other states), and input the status to the intelligent cabin domain control after being forwarded through the gateway. The purpose is that the intelligent cabin domain control can effectively determine the rationality of the video input, the reasons for input failure, etc. based on the feedback of the status.

Driving Recorder DVR and Its Enhanced Version AVR

The driving recorder is actually a relatively simple traditional car function. Its startup process is also very simple. Why is it connected with ADAS related functions? It is because realizing the driving recorder in the intelligent driving system will be more practical for high-level functions. For example, the resolution of traditional driving recorders usually uses 1-2 million pixel cameras for driving recording, while for higher-configured intelligent driving functions, 8 million pixel high-definition front view cameras are usually reused as driving recorders, and of course, clearer videos can be recorded at this time. In addition, many cars with intelligent driving functions often put forward higher recording requirements for driving recorders, such as by including the panoramic camera video data in the entire driving record process to realize the overall video recording of the vehicle’s surroundings. In addition, in many scenarios, there are video capture and recording specifically for the driver inside the car.

It should be noted that for the advanced functions of video recording for vehicles with advanced autonomous driving, the following possible limiting factors need to be considered:

1) Regarding whether it is feasible to reuse the forward-facing camera as a driving recorder, it is necessary to focus on the clarity and completeness of the video recording of the driving recorder.

For example, under normal circumstances, the driving recorder needs to include the footage of the vehicle’s front cabin in the recorded video (because it mainly considers collision where the cabin cover is impacted). However, for the perception video required purely for ADAS functions, it is preferred not to have the cabin cover block the shooting of forward targets, as this may cause missed recognition. The potential conflict between these two requirements needs to be balanced in the design. Of course, from the perspective of the national standard for driving recorders, the requirement for the recorder’s video needs to be given more attention, and the blind zone in front of the camera can be compensated for by the front millimeter wave radar, front surround camera, and front LiDAR.

2) How to effectively encode and store high-definition multi-angle videos is also a key issue that needs to be considered for subsequent driving recorders.

For example, when considering multi-angle input for AVR video recording in addition to forward-facing video recording, the overall requirement for the storage card is greatly increased. Therefore, for intelligent driving domain control, a high video encoding ability is required. For instance, the encoding ability of many SOC chips only supports standard-definition H.264 video encoding, which is not suitable for video encoding with high-definition and multi-angles like this. For our chip selection, we need to support high-definition video compression ability, such as HEVC.

3) How to consider the allocation of computation power and power consumption for domain controllers during the driving record process also requires us to plan ahead.

It looks like a small video recording, which actually does not involve any actual perception processing. Can we assume that it does not consume any computing power or power consumption? Actually, no. As mentioned earlier in the Sentry mode, for the camera controlled by the intelligent driving domain controller, it is necessary to start the IIC control line to open and close the camera, and start the time synchronization module to put a timestamp before inputting the corresponding video source. All of this requires starting the chip and making power and computing power reservations. Of course, from the perspective of design optimization, it is entirely possible to separately start the part of the chip involved in video processing in the internal chips (such as the reduced schematic diagram below), so as to minimize power consumption and computing power.

Remote Real-Time Monitoring Function

As the name suggests, the remote real-time monitoring function mainly allows users to view real-time images of the inside and outside of the vehicle and its surroundings through the App when the vehicle’s status meets certain conditions. It also supports taking single photos. Its working principle is to start the cloud service through a mobile phone or iPad and call the in-car camera for cabin monitoring, while integrating panoramic imaging and driving recorder functions to monitor the real-time situation around the vehicle.

This function is more suitable for female drivers to find lost items (keys, mobile phones, bags, children, pets) in the car and to view the surrounding situation without approaching the car, thus ensuring a high degree of safety.

Therefore, the key elements involved in the remote monitoring function mainly include real-time monitoring, monitoring without blind spots, and low power consumption. The differences in the hardware planning of each camera module may affect the design of this function. For example, whether the driving recorder is reused with the front-facing camera may lead to considering whether to drive the front-facing camera module in the intelligent driving domain controller when designing this function. At the same time, there are also differences in the interaction between the entire image display system and the camera.

As shown in the figure above, developing the remote mobile monitoring function mainly involves the following key elements:

1) Real-time video calling:The remote monitoring of cameras is mainly called through the front camera video call (which can be either a separate driving recorder or a high-definition front detection camera reused by the intelligent driving system), the panoramic/reverse image call, and the in-car camera call. Before video capture, vehicle status queries (generally including whether the power is on, the door lock status, the gear status, etc.), control camera self-learning (configuration information, calibration status, video capture status), flow control status, etc., are required.

2) Video processing

The entire monitoring front end needs to include video collection, video desensitization, video splicing, image rendering (optional), time watermarking, etc. At the same time, based on the actual environment, manual remote supplementary lighting can be selected to enhance the video capture effect. If it is for in-car monitoring, many manufacturers may also use the flexible arrangement of cameras (such as electric DMS/OMS). At this time, remote information exchange with DMS/OMS (such as raising/lowering the camera) can also be added remotely. If data security is considered, the collected video is generally desensitized and transmitted back, and sometimes the camera video collection is closed directly through a compliant switch while driving.

3) Video viewing

If remote viewing is considered, it is necessary to use cloud-based information to check whether the vehicle is in the parking gear, whether the door locks are closed, whether the related cameras of the vehicle are occupied, and whether the entire wireless transmission link is open. If all of the above conditions are met, the corresponding video viewing switch can be activated, and different switch options can be used to view the video including forward, sideways, backward, and in-car images. Of course, if there is already a user in the car, the premise for remote users to start viewing is that the user must “confirm” the video viewing.

The following diagram shows the work timing diagram of the entire remote monitoring function:

Obviously, for the entire remote video monitoring function, in addition to the panoramic camera video information provided by the sentry mode, the intelligent driving system needs to choose whether to open and control the output of the front video stream based on the reuse status of the front camera and the driving recorder of the vehicle model. If it is purely a driving recorder, then there is no difference in the requirement of intelligent driving domain control compared to sentry mode. However, for a driving recorder that reuses the front camera, it is necessary to fully consider how to open the front camera for back-end video input at the lowest power and cost.“`
Due to the important role the forward camera plays in the entire ADAS system, the forward camera is usually placed at the most important SOC processing port in the intelligent driving domain control for processing. Therefore, the entire forward section can only be scheduled when the overall processing core module of the intelligent driving domain control is started. However, video surveillance and other applications are a type of basic service that is completely out of control of time (that is to say, the amount of time the user uses is completely out of control, and long-term use will result in a significant consumption of power). In the design process, we need to fully consider whether it can be designed to meet the overall requirements while saving power from the perspective of power consumption.

Summary

For the above image functions, since only the driving function of the camera is used in the simple opening and closing process of the camera for intelligent driving domain control, rather than using its high-power SOC for AI information processing. For image transmission, display and recording, it is often a long-term power consumption. Therefore, when designing intelligent driving domain control, it is necessary to consider how to significantly reduce its operating power consumption when the image function is activated, so as to enhance the endurance or standby ability of the entire vehicle.
“`

This article is a translation by ChatGPT of a Chinese report from 42HOW. If you have any questions about it, please email bd@42how.com.