NIO IN? I'd Rather Call It an AI Day

At the 2024 NIO IN event, cars were barely visible; AI became the central protagonist. Highlights included the successful tape-out of a 5 nm autonomous driving chip, the full release of SkyOS, an AI capability upgrade for Banyan 3, the unveiling of the NWM (NIO World Model) for autonomous driving, and the official launch of the all-new NIO Phone. Hardware presence was limited to just one product: the new NIO Phone.

What is particularly interesting is that NIO founder, chairman, and CEO William Li took the stage twice. In one instance, he described the performance of the Shenji NX9031 as 'extraordinary' ('not just ordinary leading'). In another, he admitted they were 'far behind' regarding the NIO Phone's continued insistence on zero advertising and zero pre-installed apps—a notable highlight at this technology-focused launch event.

So, during the nearly three-hour NIO IN event, what is the critical role of AI? And compared with other domestic automakers, what distinct insights does NIO have regarding its AI capabilities?

01Successful Tape-Out of 5 nm Autonomous Driving Chip

Seven minutes into the opening of NIO IN, after reviewing the major developments over the past half year or so, William Li excitedly announced: 'NIO's self-developed smart driving chip, Shenji NX9031, has successfully completed tape-out!'

This chip is of great significance. Prior to this, no smart driving chip, whether domestic or international, had broken through to the 5 nm process node. Moreover, as NIO's first self-developed smart driving chip, the challenge was undoubtedly immense.

Shenji NX9031 was unveiled at the first NIO IN event last September. In just ten months, it has successfully completed tape-out, marking another significant step toward vehicle integration.

"Far ahead," William Li said on site.

William Li highlighted the image signal processing (ISP) capabilities of the Shenji NX9031. With both cameras boasting 8 megapixels, different ISPs produce vastly different imaging results. On everyday roads, an 8-megapixel camera powered by the Shenji NX9031 can see further.

In live photos from NIO IN, the Shenji NX9031’s ISP also preserved more details in low-light conditions, a capability that is crucial for autonomous driving.

William Li also revealed other technical details about the Shenji NX9031:

High-dynamic-range, high-performance ISP: It processes pixels at 6.25 gigapixels per second, features a 26-bit bit depth, and has a processing latency of less than 5 milliseconds.

More than 50 billion transistors.

A 32-core big.LITTLE CPU architecture with computing power of 615K DMIPS.

Memory bandwidth of 8,533 Mbps.

Transformer-based algorithms see a 6.5x performance increase; LiDAR-based algorithms improve by 4x; BEV-based algorithms improve by 4.3x.

Supports functional safety up to the ASIL-D level.

However, some parameters of this chip remain undisclosed. For instance, the NPU computing power—which is nearly synonymous with autonomous driving capability—was not given a specific figure at this launch event, though upgrade metrics for Transformer, LiDAR, and BEV algorithm performance were provided.

02SkyOS Full Release

This marks another continuation. At NIO IN last September, SkyOS·Tianshu and its architecture were officially unveiled, but not in a fully mass-production-ready state. It must be clarified that SkyOS is not an operating system; it offers different functionalities across various chip hardware platforms.

With SkyOS-L, the real-time operating system for vehicle components, now integrated into vehicles, NIO has fully released the complete SkyOS suite today.

The architecture diagram shows that at the base are chips, devices, and other in-vehicle hardware (such as autonomous driving sensors, doors, and windows), with SkyOS running above them.

SkyOS-H is a high-performance, high-reliability virtualization system designed for complex scenarios. It creates virtual sandbox environments while managing and scheduling the vehicle's heterogeneous high-compute resources, ensuring that upper-layer functions operate independently without interference, thereby serving as a secure foundation. In simple terms, this operating system divides a large house into separate rooms for different functions.

Leveraging the capabilities of SkyOS-H, end-to-end execution time for real-time tasks is reduced by 50%, multi-thread throughput increases by more than 40%, and virtual disk throughput improves by 15%.

One layer up, there are four operating systems: SkyOS-L, SkyOS-M, SkyOS-R, and SkyOS-C, each running on different hardware.

SkyOS-L is a lightweight OS tailored for MCU chips with low compute power, aiming to achieve higher real-time performance and reliability. Compared to AUTOSAR, SkyOS-L executes instructions with high, medium, and low priorities stably, whereas AUTOSAR demonstrates less accurate delivery capability for medium and low-priority instructions.

SkyOS-M, built on a microkernel architecture, runs on the central computing unit and primarily controls body, chassis, suspension, and other systems. The core advantage of its microkernel architecture is security isolation. NIO has achieved isolation across two types of CPUs, two levels of memory, five categories of I/O interfaces, and 25 core services. Additionally, beyond security isolation, it incorporates a safety mechanism featuring four layers of monitoring and three layers of recovery.

Compared to Linux, SkyOS-M is less affected by system issues and can recover more quickly. Data shows that SkyOS-M’s self-recovery time is just 5 milliseconds, which is 100 times faster than Linux.

SkyOS-R is primarily used for autonomous driving and has stringent end-to-end latency requirements. It aims to support various application scenarios while providing an operating system with high real-time guarantees. Compared to Linux distributions, it improves light-load end-to-end latency by 113 times and high-load end-to-end latency by 20 times.

SkyOS-C is a deeply customized Android-based operating system that powers the smart cockpit functions. It efficiently supports large-model-based applications and AI-driven experiences, including NOMI.

Above SkyOS lies its middleware, which is key to achieving layered decoupling and cross-domain integration. NIO has built more than 40 middleware components within this layer, functioning like a toolbox. To achieve layered decoupling, NIO abstracted over 1,600 atomic capabilities from the entire vehicle, making them available for invocation by various domains.

Given the complexity of so many interdependent atomic capabilities, NIO developed its own SOA framework and defined a high-performance cross-domain communication protocol called TOX (Talks-Over-X, where X stands for anything). This protocol applies to all network types and communication terminals.

As a result, TOX achieves high bandwidth and an MCU rate exceeding 20,000 pps. Its high-priority signal capacity is 30 to 50 times greater than that of traditional CAN buses, while its latency is reduced by 40% compared to SOME/IP. Additionally, the zero-packet-loss threshold has increased by 109%.

Additionally, NIO has established a data closed-loop mechanism, developed an end-to-end full-lifecycle R&D toolchain, and built a systematic safety framework.

03The Fully AI-Integrated Banyan 3

With SkyOS-C as a foundation, NIO’s immediate focus is the upcoming Banyan 3 intelligent system.

Banyan 3 is NIO's first intelligent system based on underlying AI capabilities, and for this reason, the background image of Banyan 3 was generated using diffusion technology and NOMI GPT.

The primary upgrade in Banyan 3 is the AI-powered smart chassis. "The true soul of a smart chassis today is its AI capability," said Li Tianshu, Vice President of Product Experience at NIO, during the event.

In Banyan 3, 4D Comfort Pilot has been upgraded to 4D All-Area Comfort Pilot. It can activate the 4D comfort chassis without requiring navigation to be turned on, features real-time calculation of bump indices displayed via visual line graphs, and sends the data to the mobile app after the trip concludes. This feature is available for all models equipped with CDC dynamic damping control, including the ES6 and EC6 even if they lack air suspension. However, the two ET5 variants currently do not support this function.

Banyan 3 adds a new driving mode—Rear Passenger Comfort Mode—that minimizes impact and motion sickness for rear-seat passengers, even if the driver’s technique is not flawless.

Inside the smart cockpit, NIO has partnered with QQ Yinyue to use AI tools that unlock 7.1.4 panoramic sound experiences for songs that do not natively support it; this feature currently supports over one million tracks.

The system also includes an Expert Equalizer Pro EQ. Later this year, NIO will launch an AI-generated EQ powered by NOMI GPT: after analyzing a song’s style, NOMI generates a tailored equalization profile, allowing users with no expertise in audio or music to enjoy improved sound quality.

Additionally, the hands-free karaoke feature has been upgraded: simply start singing, and the original vocals are automatically muted.

Native apps on the vehicle’s infotainment system remain limited, so NIO uses Web-to-Flux applications to deliver content and services nearly equivalent to native apps. Services such as Apple Music can now sync with the car and be controlled via NOMI.

NIO also announced integration with Tencent Mobility Services mini-programs today. The interface closely mirrors the mobile experience, enabling the use of various mini-programs—including lifestyle and food delivery services—directly on the vehicle’s screen.

For the in-cabin AI experience, NOMI is paramount. Over the past three months, users have interacted with NOMI GPT more than 15 million times. Moving forward, NOMI will continue to advance in three key areas: anthropomorphic interaction, becoming a capable assistant, and serving as an emotional companion.

Voice commands without wake words, boosting efficiency by 100%.

Support for multiple speakers: simultaneous speech and turn-taking are both supported.

Multiple instructions in a single sentence: users can issue dozens of commands in one breath.

NIO has also introduced NOMI Agents, a multi-agent system that reconstructs NOMI’s cognitive and task capabilities based on large language models. In summary, this brings six core capabilities:

NOMI Parking Assistant: Simply say, 'Help me find a parking lot that is easy to park in.' A parking lot map enabling space-level navigation will be launched in the fourth quarter of this year.

NOMI Guard: This feature allows NOMI to access the vehicle’s dashcam footage, summarize numerous guard events, and identify the most likely dangerous incidents. If a collision or scrape occurs, it can directly schedule maintenance services.

NOMI Voice Reservation for Ride-Hailing/Chauffeur Services.

NOMI Explore & Discover can serve as a child's encyclopedia, helping answer questions about what things are. It also recommends restaurants based on user needs, including partner restaurants offering benefits to NIO owners.

NOMI DJ generates personalized playlists and syncs with the ambient lighting.

NOMI Travel Memories performs aesthetic analysis of dashcam footage to automatically generate highlight videos.

Looking ahead, the core experience of future smart cockpits will be fully Agent-driven.

Meanwhile, NOMI’s emotional engine delivers impressive contextual intelligence, personalized intelligence, and emotional expression. For example, during a 30-minute family trip, the playlist is recommended according to the travel scenario. Personalized intelligence means NOMI’s intelligence belongs exclusively to you; it even remembers which hat it is wearing. Emotional expression enables more proactive interaction, such as singing a birthday song for you on your birthday.

According to the plan, Banyan 3 will begin recruiting its User Pilot Group next week, with an expected rollout by the end of August. Given the extensive content in Banyan 3, NIO will release it through multiple versions.

04The NIO World Model Is Here

In the field of autonomous driving, Ren Shaoqing, Vice President of NIO Autonomous Driving R&D, first introduced the latest progress. Currently, the open mileage for NIO's Full-domain Navigation has reached 3.899 million kilometers, and all users have driven a total of 1.112 billion kilometers. NIO’s vehicle-cloud total computing power stands at 287,100 POPS, with actual road verification mileage of 420 million kilometers. The monthly volume for sample analysis and mining reaches 220 million.

Recently, NIO launched its end-to-end AEB system, but this does not fully meet NIO’s vision. NIO aims to build a model that achieves spatial cognitive reconstruction similar to the human brain, as well as the ability to perform imaginative deduction based on temporal cognition. This is NIO’s World Model (NWM).

If the 2D/3D models from ten years ago were like rough sketches, then BEV and OCC are more detailed sketches, yet they still remain in the "sketch" stage.

NIO’s World Model (NWM), however, integrates more information. For example, it can imagine current weather conditions, which is an important condition for better autonomous driving performance.

NIO used data from over tens of millions of clips for learning to improve the realism of reconstruction. What is the practical application?

For instance, on congested roads, by providing video and vehicle status information to NWM, the vehicle can "imagine" thousands of possible road scenarios ahead, identifying a specific time point to overtake slower vehicles in front.

In NWM, 216 possibilities are predicted every 0.1 second, with the optimal solution selected after evaluation. As a generative model, NWM can take a 3-second driving video as a prompt to generate a 120-second video.

NWM also features closed-loop simulation testing capabilities and has fully tested and verified its performance in complex interaction scenarios.

NIO’s world model, end-to-end active safety model, along with all processing mechanisms and safety mechanisms, form NADArch2.0.

On the product side, features have been streamlined to two: Point-to-Point Navigation Assist 2.0, which will be available in the second half of this year; and Intelligent Safety Assist 2.0. End-to-end AEB is already live in Banyan 2.6.5, and end-to-end GOA will roll out successively in the second half of the year.

05William Li: The New NIO Phone Is Too Good to Not Be a Primary Device

After a series of technical explanations, the only hardware product at the entire launch event—the all-new NIO Phone—was introduced. At last September's NIO IN, NIO released the first-generation NIO Phone. William Li said that users have come to realize the NIO Phone is an indispensable part for NIO users.

Today, William Li pulled the all-new NIO Phone from his pocket and said, "I couldn't bear not to make it a main phone."

The new NIO Phone features a 6.82-inch deeply curved quad-edge display, known as the NIO Sky Dome screen. It achieves a bending depth of 724 micrometers, which poses significant manufacturing challenges. This greater curvature enhances the device's ergonomics and grip feel.

The display offers 2K resolution and is a deep-customization by NIO for Samsung. It has a pixel density of 510 PPI, a local peak brightness of 2,600 nits, and supports a dynamic refresh rate from 1 Hz to 120 Hz.

The thinnest part of the new NIO Phone measures 8.6 mm. The frame is made of matte aluminum alloy, and the back panel around the camera module also uses a matte finish. Compared to the glossy aluminum of the previous generation, this design is less prone to fingerprints.

The new NIO Phone introduces a new Nebula Red color, along with five other options. All body colors are derived from sky-inspired palettes that match NIO’s vehicle exterior colors.

In terms of performance, the all-new NIO Phone uses the Snapdragon 8 Gen 3 mobile platform and the Guyu cooling system, with improved gaming performance.

Regarding battery life, endurance time has increased by 10% compared to the previous generation. An emergency calling mode has been added, allowing for half an hour of talk time even when the battery level drops below 2%.

For imaging, the new NIO Phone uses a Sony LYT-T808 sensor and supports full-link Ultra HDR processing.

Furthermore, the new NIO Phone continues to adhere to a policy of zero system ads and zero commercial pre-installed applications.

William Li said: 'In terms of the number of ads and necessary pre-installed apps, we are far behind, always behind, and will remain behind.'

The new NIO Phone ships with SkyUI 2.0, and NOMI GPT is now available on the phone. Leveraging cross-device capabilities, the NOMI assistant on the phone and in the vehicle can be seamlessly integrated.

Several key capabilities stand out:

During a drive, phone calls generate call summaries that are saved to the phone.

Enhanced phone-vehicle connectivity upgrades entertainment features, allowing users to access Sky View more conveniently while maintaining safety standards.

NIO Link App has expanded to 21 applications, enabling high-quality audio sources from the phone to be experienced in the car.

Introducing the NIO Link in-car broadcast feature. During a live stream, users can seamlessly switch between the phone and the vehicle’s cameras as they exit and re-enter the car. The feed can toggle from broadcasting the person to showing the road view, and then to the interior cabin camera, with all transitions handled freely. Additionally, this feature supports video calls through other applications.

The new NIO Phone is available in three configurations: the Performance Edition (12 GB RAM + 512 GB storage) priced at RMB 6,499; the Flagship Edition (12 GB RAM + 1 TB storage) at RMB 6,899; and the EP edition (16 GB RAM + 1 TB storage) at RMB 7,499.

The new NIO Phone will be sold not only on the NIO App but also simultaneously on JD.com and Douyin. First-generation NIO Phone owners receive a RMB 500 voucher and can use up to 10,000 NIO Points toward payment. Shipments for the new model begin today.

06Final Thoughts

In 2024, both the smartphone and automotive industries are fully transitioning into the AI era. Without AI integration, a product or company is virtually synonymous with being left behind.

As one of China’s leading new-energy automakers, NIO appears to have made the deepest investments in AI. While many automakers are beginning to layout self-developed chips, NIO successfully taped out its 5 nm autonomous driving chip. As the industry grumbled about improving smartphone-vehicle connectivity, NIO launched the NIO Phone. Coupled with its advancements in operating systems and intelligent driving, NIO indeed holds a leading position in technological innovation.

All technical research and development ultimately serve a single purpose: selling cars effectively. In today’s fiercely competitive domestic automotive market, it is becoming increasingly difficult to sell high-end pure electric vehicles to consumers.

The surge in AI capabilities, rising demand for seamless vehicle-phone integration, and increasingly fierce competition in autonomous driving are compelling automakers to develop vehicles as AI terminals. This aligns precisely with the core focus of NIO IN.

I’ll leave you with a question: please share your thoughts in the comments. As a NIO owner, would you buy the new NIO Phone? If you’re not a NIO owner but are in the market for a new phone, would you choose the new NIO Phone?

Written by James Edited by Daji