Robotaxi: A Multidimensional Evaluation of Three Products

By Engineer X

Robotaxi, an important implementation scenario of autonomous driving technology, is gradually coming into public view as an autonomous driving taxi. As of the end of 2021, many companies such as SAIC, Baidu, Xiaoju Intelligence, Wenry Knowledge Mobility, Yuanrongqixing, T3 Traveling, Didi, and Auto X have started pilot commercial operations of their Robotaxi fleets.

For the Robotaxi products that are emerging like mushrooms after a rain, how do they really perform? What is the user experience across different dimensions? Is the autonomous driving technology exhibited by these companies mature?

At the beginning of the new year, armed with these questions, we entered the unmanned driving demonstration zone in Anting, Jiading, Shanghai, and carried out a multidimensional and comprehensive on-site evaluation of several Robotaxi products that have already been put into use.

For this evaluation, we selected three Robotaxi products: Xiangdao Travel (SAIC), Luobo Kuaipao (Baidu), and Xiaoma Zhixing (Xiaoju Intelligence). We conducted a comprehensive analysis of the product strengths of each player from multiple aspects including operation, mobile end, vehicle end, autonomous driving performance, human-machine interaction experience, and service experience.

The riding process of Robotaxi is the same as traditional ride-hailing services: users select pick-up and drop-off sites through their mobile phones, the system automatically dispatches the vehicle, after accepting the order, the vehicle automatically drives to the pick-up location, then opens the autonomous driving function, and finally takes the user to the drop-off point. After arriving at the drop-off point, the order is completed.

Currently, due to safety and regulatory considerations, each Robotaxi is equipped with a safety officer. However, the safety officer only takes over the vehicle in emergency situations and special scenarios. At other times, he/she only observes the surrounding environment.

Below are the specific evaluation contents and results.

Part 1: Operation

Currently, Xiangdao, Luobo, and Xiaoma all claim to operate in the Anting area of Jiading, but their actual operating ranges are different. As all Robotaxis only support fixed sites as pick-up and drop-off points, we can confirm the operating range of each Robotaxi through the distribution of the sites on the APP.

Xiangdao Robotaxi has the smallest operating range, only covering some prosperous areas of Anting Town and a small section of Boyuan Road demonstration area. The Robotaxi services of Luobo and Xiaoma cover almost the entire Anting area, including some urban areas of Jiading, with Xiaoma’s coverage being broader.

Regarding the distribution of stations, the three Robotaxi companies are mainly concentrated in the main urban area of Anting and the demonstration area of Boyuan Road. In addition, Luobu KuaiPao has more stations in Anting New Town, while Xiaoma Zhixing has set up many stations near Yongsheng Road in Jiading.

In terms of operating time, Xiangdao and Luobo are both operational 24/7, while Xiaoma only operates on weekdays.

During our experience and communication with the safety officers, we learned that the daily order volume of each Robotaxi company is in the tens, and now most orders are from regular Robotaxi users who use it as a daily commuting method, with fewer users purely for experiencing it. This also indicates that Robotaxi as a new way of travel is being accepted and increasingly becoming a habit for more people.

Part 2: Mobile App Experience

All three Robotaxi companies require operation through their own unique apps. Among them, Xiangdao integrates Robotaxi as a module into the “Xiangdao Travel” app because in addition to Robotaxi, Xiangdao Travel also has regular ride-hailing services, which Luobo and Xiaoma don’t have.

Coincidentally, all three apps use a green interface, which also reflects the meaning of green travel.

After entering the app, we started to reserve a vehicle. The interface for reserving a vehicle is similar in style to traditional ride-hailing apps, with maps and station prompts, and you can choose a pickup and drop-off station through the map or menu.

Currently, all 3 Robotaxi companies only support ordering at fixed stations located at bus stops and do not support ordering at any location.

After selecting the pickup and drop-off stations, the platform will dispatch the vehicle. During our actual experience, we reserved a total of 6 times, with each Robotaxi company reserving 2 times. Among them, 4 times were completed within half a minute, while 2 times took a long time to wait: one was at the Anting Metro Station where the passenger flow was high and there was a high demand for ride-hailing, which took Xiangdao 3 minutes to accept the order; the other was at a remote station on Boyuan Road, where Luobu KuaiPao took 15 minutes to accept the order, which is probably because it was too far from the densely populated area and there was a high demand for ride-hailing during the evening rush hour.Of course, since we don’t have a large number of orders, it is only a reference for a small sample size.

While waiting for the vehicle to arrive, the APP will display the current location of the vehicle, estimated arrival time, and vehicle information, which is consistent with traditional ride-hailing services. It’s worth noting that Enjoy offers the added feature of controlling the car’s air conditioning and music through the APP ahead of time, and users can also directly contact the safety officer via phone before the trip, which is a bonus point that neither of the other two companies currently offers.

In addition, Enjoy provides detailed information about the vehicles, including the time they have been in operation, total mileage, and total number of passengers carried. Interestingly, Enjoy references the concept of video game points and sets an AI level for each car that increases with the number of miles driven, which is another small bonus point.

Overall, the ride-hailing experience provided by the three APPs is very convenient, and the interfaces are relatively simple. Users who have used ride-hailing software before can quickly get the hang of it.

Part 3: Vehicle Configuration

Currently, the operating vehicles for the three Robotaxi companies are based on existing models and retrofitted with autonomous driving systems.

Enjoy uses SAIC’s Feiyun Marvel R model, Luobo retrofits the Hongqi EV EHS-3 model, and Xiao Ma chose the Lexus RX 450h for its fleet.

As for the hardware for autonomous driving, all three companies have equipped their vehicles with a rich array of sensors, both in terms of quantity and performance, which can be considered high-end.

It’s worth mentioning that all three Robotaxi companies have installed a rotating lidar on the roof of their cars. Enjoy utilizes one 128-line main lidar and two 32-line supplementary lidars. Luobo uses one 40-line lidar, and besides the two lidars on the roof, Xiao Ma also installs two auxiliary lidars on the side of the vehicle.

In addition, most of the primary perception cameras for Xiao Ma’s Robotaxi are integrated on the roof, along with one camera specifically for perceiving traffic lights.It can be said that the integrated perception system on the roof is the top priority of Robotaxi and an important feature of current Robotaxi, and it is naturally cost-prohibitive. If each company wants to fully deploy Robotaxi in the future, how to reduce the cost of this perception system will be a key and difficult point.

Part 4: Autonomous Driving Performance

On the day of the evaluation, the weather was sunny and well lit.

In order to accurately evaluate the performance of the three products, we chose the same starting point, and each went back and forth once.

The evaluation route includes two different types of roads: the first segment is in a busy section of Anting, with dense traffic, many non-motorized vehicles and pedestrians, complex traffic conditions, and various scenes. The second segment is on Boyuan Road in Anting, which is a smart driving public road test area in Anting. The road structure is high, the vehicle density is relatively small, the traffic is smooth, and there are fewer non-motorized vehicles and pedestrians.

The entire evaluation route (round-trip) is 11km long, passes through 14 intersections with traffic lights, involves 3 left turns, 1 right turn, and 1 road fork. The road is smooth, with no obvious congestion or construction sections. Through the entire route, the autonomous driving level of Robotaxi can be comprehensively evaluated.

Scene is the basis for the evaluation of autonomous driving, and in this experience, we evaluated the autonomous driving performance of different Robotaxi based on different categories of scene.

From the experience results, the overall performance of the three Robotaxi is generally satisfactory, basically achieving autonomous driving on the entire route, with only one manual takeover each for Enjoy and Luxcar, and full autonomous driving for Pony.ai.

The only manual takeover of Enjoy is understandable, as it belonged to a Corner Case scenario: when approaching the end point and preparing to park on the side of the road, the rightmost lane was full of cars with no opportunity to change lanes, and the vehicle had to drive while looking for an opportunity to change lanes. But it had already missed the end point. At this time, the safety officer intervened and accelerated to the front of the traffic and parked on the side of the road.

The manual takeover of Luxcar was somewhat unexpected: when the vehicle was about to reach the bus stop at the end point, it had not yet reached the drop-off point and was not eligible for passenger drop-off, but the vehicle prompted passengers to get off. After we reminded the safety officer, the safety officer intervened, driving forward for about 2m before reaching the platform and meeting the conditions for passengers to get off.

In addition, Enjoy encountered another Corner Case: when turning left at a crossroads and preparing to enter the lane, it encountered multiple congested cars that also wanted to enter this lane, creating a very chaotic situation. However, Enjoy Robotaxi waited intelligently, adjusted the vehicle position, gradually converged into the lane, and was not inferior to an experienced driver.“`markdown
Except for manual takeover and corner cases, the performance of the three Robotaxis in urban autonomous driving scenes is similar. The differences mainly lie in the default lane for cruising, lane change success rate, and braking comfort. In addition, there are some scenarios such as U-turns, left turn waiting areas, and intersection crossing that have a certain degree of randomness, which cannot all be reflected in this evaluation, and we hold a reserved attitude.

There are several points that impressed us:

1. All three companies have deeply integrated high-precision maps, so they do very well in reconstructing and positioning lane-level scenes.

2. Xiangdao cannot recognize left turn waiting areas and can only stop behind the stop line while waiting at red lights, which affects traffic efficiency. Rabo and Xiaoma cannot determine their recognition situation when they encounter vehicles in left turn waiting areas.

3. Xiaoma will actively adjust the vehicle attitude to avoid obstacles when encountering obstacles in front of them, while Xiangdao and Rabo are relatively conservative.

4. All three companies can recognize traffic signals and drive according to them, but Xiangdao and Xiaoma cannot recognize light flashing and countdowns, and Rabo did not encounter any uncertainties.

5. When the three Robotaxis go straight through an intersection, they will not slow down if the road ahead is clear, which violates traffic regulations.

For the specific performance of Robotaxis in autonomous driving mode, please refer to Table 5. Regarding data such as lane change success rate and takeover rate, more data support is needed for statistical significance. For such data, we can only draw conclusions based on the evaluation results of this 11 km section.

Part 5: Human-Machine Interaction Experience

Intelligent experience includes not only autonomous driving but also human-machine interaction experience.

Originally, we wanted to use the title “Intelligent Cabin Experience.” However, in reality, the three Robotaxis are still far from the “smart cabin,” and can only be considered as simple human-machine interaction.

In the process of autonomous driving, the three Robotaxis do not support voice interaction like XPeng and vibration warnings in emergencies. All interactions can only be completed through the passenger display in front, without demonstrating the expected level of intelligence in human-machine interaction.

Regarding scene reconstruction, the three Robotaxis have shown the basic traffic elements, including road environment, obstacles, pavement markings, traffic lights, etc., and also specially displayed the point cloud effect detected by LIDAR, which to some extent enhances the sense of technology.
“`However, the common problem is that the reconstruction of the scene is too simple. Except for the ground markings and traffic lights, the authenticity of other traffic elements is low, the models are single, and even the models of horses’ motor vehicles, two-wheeled vehicles, and pedestrians give people a “perfunctory” feeling. In addition, all three companies’ scene reconstructions have the problem of a single color scheme. Only Luobo has relatively rich color elements, but they are also limited.

There are some highlights in the display of Xiao Ma Robotaxi at the intersection: informing passengers of the current state and next action of the vehicle in classical Chinese, and using a blue guiding line to show the planning result of the turning path, resulting in a good experience.

Entertainment services are also part of human-machine interaction. The entertainment services of the three companies have been reduced, and Baidu even has no entertainment services. The entertainment services provided by Xiao Ma only have practical language selection, and other content is more like self-introduction and questionnaire surveys. Only Xiangdao keeps the air conditioning and music functions, but that’s it. We speculate that this is done to let passengers pay more attention to the autonomous driving process.

During the experience process, we found that besides the display screen in front of passengers, all three Robotaxis also have a screen in the central control area that can display more content. Xiao Ma’s central screen even outputs perception results before model rendering, as well as information on the movement speed of surrounding vehicles and obstacles. Unfortunately, the central control screen is for safety officers to view, and we must sit in the back seat as required by the government, so we can only see the screen in front of us in a normal sitting position.

Overall, the human-machine interaction experience of the current Robotaxis from the three companies is very ordinary, and for those who have experienced new power brands such as XPeng and NIO, there is almost no attraction.

Due to Luobo’s explicit statement that passengers are not allowed to take photos, we can only show the screen display effects of Xiangdao and Xiao Ma Robotaxi here.

Part 6: Service Experience

Robotaxi provides autonomous driving travel services. As an operator, the service level is naturally part of the product strength.

We experienced the services of the three Robotaxis from two aspects: riding experience and visiting the experience center.### Riding Experience

The riding experience mainly refers to the services that the operator provides to passengers. As a purely transportation tool, we believe that the riding experience is completely okay. Taking a taxi is relatively convenient, and the vehicle runs safely and steadily. Moreover, it is currently in the free stage. However, as the promotion and application demonstration of autonomous driving technology, we believe that there is a big difference among the three.

The attitude of Xiangdao is very open. The safety officer will answer questions and even take the initiative to chat with us about the current situation of Robotaxi vehicles in the Anting area. In case of deep questions that they could not answer, they will take the initiative to provide the address of the Xiangdao Robotaxi experience center for us to gain a detailed understanding.

The safety officer from Luobo told us that passengers are not allowed to take photos or videos under the Luobo’s rule. As for the Apollo experience center, we had to ask about it actively to get the information.

Like Xiangdao, Xiaoma allows photography, and their safety officers will also answer some of our questions. However, Xiaoma doesn’t have an experience center, and it is difficult for us to learn more about it.

After the real vehicle evaluation, we went to the Xiangdao Robotaxi and Baidu Apollo experience centers and got completely different results.

The Xiangdao experience center had beautiful ladies who warmly received us, introduced us to the exhibition hall and the operation of Xiangdao Robotaxi, and patiently answered all kinds of questions from us. We got all the information we wanted to know.

For the Baidu Apollo experience center, we didn’t even get to the gate. The security officer told us that they only serve customers who have made appointments, not individuals, and asked us to leave quickly. At the same time, the staff in the experience center rushed over as if facing a formidable enemy. We felt as if we had entered a military restricted area.

In terms of service experience, we believe that as operators of Robotaxi, while promoting and marketing themselves, they should also maintain an open-minded attitude, allowing the general public to learn more about autonomous driving technology and Robotaxi’s operation mode, and become true leaders and popularizers of autonomous driving. Instead of pretending to be mysterious, prohibiting photography, and refusing visitors.

In addition, safety officers, as important service providers of Robotaxi who directly interact with passengers, should provide more training to them regarding Robotaxi and autonomous driving knowledge, so as to answer their questions and satisfy their curiosity. Instead of simply telling passengers, “This car is driving autonomously, you see, I’m not intervening.”

Part 7: Conclusion

From this evaluation, we can easily find that with the expensive custom sensors and deep fusion of high-precision maps, all three Robotaxis can basically achieve fully autonomous driving in specific areas, and the safety and stability of the vehicles can be guaranteed.However, the current human-machine interaction of several Robotaxi companies is not very satisfactory. This is a key point to improve the user experience, and the difficulty is not high.

As for the mobile app, the three companies are mostly the same. After all, there are mature ride-hailing software as benchmarks, and users have also become accustomed to the style of traditional taxi-hailing software. Product homogeneity is not a major problem.

At present, Robotaxi is still in its early stages, with more and more companies entering the market, and the market size is gradually increasing. How to deal with more complex traffic scenarios, how to reduce vehicle and operating costs, how to increase the operating range (full coverage of high-precision maps), and how to attract customers with high-quality services and unique experiences are all problems facing Robotaxi players and are key factors in whether Robotaxi can be fully popularized.

Certainly, there are still various problems with Robotaxi, but as an exploration of the commercialization of autonomous driving technology, the future of Robotaxi is worth looking forward to.

This article is a translation by ChatGPT of a Chinese report from 42HOW. If you have any questions about it, please email bd@42how.com.