Author | Yun Peng
Edit | Li Shuiqing
Zhidong reported on May 31 that today, the intelligent terminal industry is carrying out an unprecedented deep integration with generative AI technology represented by large models, and the mobile field represented by AI mobile phones and AI PCs, as the track most closely related to each of our lives, has emerged many new AI needs and scenarios.
Generative AI has brought new computing power challenges and storage challenges, and accelerated a series of technological iterations in hardware, software, algorithms, and ecology behind it.
As a provider of underlying computing solutions in the mobile field, Arm, a technology giant, has recently unveiled a series of new products and technologies closely related to the development of the generative AI industry.
Just yesterday, Arm held an in-person launch event in Beijing, China, to unveil its latest terminal computing subsystem (CSS), which combines the features of the Armv9 architecture and is built on the 3nm process node, including Arm's latest CPUs and GPUs. At the same time, Arm has also released the Kleidi library, a development tool tailored to developers.
The new CPU Cortex-X925 delivers a 41% increase in AI performance and significantly improves the responsiveness of on-device generative AI such as LLMs, while the Arm Immortalis-G925 delivers a 34% performance boost across multiple AI and ML networks.
It can be said that Arm's series of blockbuster technologies and new products are all aimed at AI.
It's worth mentioning that this is the first time Arm has delivered a physical implementation for Arm CPUs and GPUs, which makes it easier to build and deploy solutions based on Arm's architecture.
This time, Zhidong also had a face-to-face in-depth exchange with James McNiven, vice president of product management of Arm Device Division, to have an in-depth discussion on the opportunities and challenges of Arm in the era of generative AI, and to explore Arm's core advantages and in-depth thinking in terms of architecture, hardware, software, and ecology.
In the era of generative AI, the "energy efficiency" of computing has become the primary focus of the industry, and energy efficiency is the "DNA" of Arm. There is no doubt that Arm has become an important variable in the mobile smart terminal industry.
1. Generative AI has set off a surge in storage and computing demand, and Android CPUs are the main pillars
There is no doubt that now is a critical moment for the mobile intelligent terminal industry. The end-to-end implementation of generative AI has redefined the "intelligence" of smartphones, and in this critical moment, Arm is the "cornerstone" of computing. From mobile phones and laptops to wearables and digital TVs, Arm is delivering universal adaptation to the largest computing ecosystem.
The advent of generative AI will inevitably generate more computing needs and bring more new use cases, which is a good thing for Arm, chip manufacturers, and terminal manufacturers, but it is also a challenge for Arm, who need to continuously iterate on architecture, hardware, software, and other technologies to make generative AI run better on the Arm platform.
McNiven said that in response to the advent of generative AI, Arm has optimized Arm CPUs and GPUs for AI inference and training, continues to add support for some new instructions and data types in the architecture, and has spent a lot of time optimizing data streams to cope with the surge in memory traffic demand.
In the process of generative AI, heterogeneous computing has become the mainstream of the industry, and CPUs, GPUs, and NPUs need to work together with each other, but in fact, 70% of third-party Android ML workloads run on CPUs, so CPUs are still the core of computing in the AI era.
Second, the performance and energy efficiency of CPUs and GPUs have been improved, AI capabilities have been smashed, and new software tools have been implemented with AI
As Chris Bergey, senior vice president and general manager of Arm's Device Business Unit, said, the Arm platform adheres to the energy efficiency concept of Arm's DNA, and with the acceleration of the AI era, it is laying the foundation for the next wave of computing demand, which is also Arm's core competitiveness.
Arm Total Compute Solutions (TCS) are no strangers to us, and the launch of Arm Terminal CSS brings new CPUs and GPUs with improved performance, power efficiency, support for developer software, and AI capabilities, as well as the first physical implementation of Arm CPUs and GPUs.
It is understood that the Arm terminal CSS released this time is the fastest Arm computing platform to date, improving computing and graphics performance by more than 30% to cope with Android workloads in demanding real-world use cases, while its AI inference speed has increased by 59%, which can better handle AI, ML and computational vision workloads.
At the press conference, McNiven emphasized that Armv9 focuses on improving computing power in vector acceleration, ML and other fields, and adds AI-oriented features.
The new Arm Cortex-X925 is said to have achieved the highest year-on-year performance improvement since the launch of the Cortex-X series, with a 3nm process and a maximum frequency of 3.8GHz, and with microarchitecture improvements, its single-threaded performance is improved by 36% compared to the 4nm SoC used in flagship smartphones in 2023.
McNiven said that Arm wanted to make it more clear how different it was from its predecessor with a naming change — the Cortex-X925 is the CPU with the highest year-over-year IPC increase since the introduction of Cortex-X.
In terms of AI performance, Cortex-X925's token first response time has increased by 41%, which means that the responsiveness of device-side generative AI, such as large language models, has been significantly improved.
Specifically, Arm has focused on microarchitecture improvements, adopting the widest decoding and vectoring design to date, and achieving a 50% increase in TOPS count. At the same time, Arm retains instructions and data that pass in and out of the CPU through greater configurability and a larger private L2 cache size.
In terms of GPUs, the new Arm Immortalis-G925 GPU achieves a 37% performance improvement in various head mobile game applications and a 34% performance improvement across multiple AI and ML networks, in addition, the Arm Mali-G725 and Mali-G625 GPUs can cover more mobile smart terminal products, such as smart watches and XR wearables.
In terms of scalability, Arm's partners can leverage a range of different CPU and GPU options to design their own solutions for large-screen computing, smartphones, digital TVs, and wearables.
In terms of tools and software libraries, Arm debuted Arm Kleidi, which includes KleidiAI for AI workloads and KleidiCV for computer vision applications.
It is understood that KleidiAI is a set of computing cores for AI framework developers, allowing developers to easily get the best performance on Arm CPUs on various devices, and support key Arm architecture functions such as Neon, SVE2 and SME2.
KleidiAI's integration with popular AI frameworks such as PyTorch, Tensorflow, MediaPipe, and others accelerates the performance of key models such as Meta Llama 3 and Phi-3, and is also backwards and backwards compatible to ensure that Arm remains relevant to future market needs as more technologies are introduced.
In Arm's view, CPUs are the only computing engine that can ensure the future of the network, so Kleidi's solutions are all built for CPUs. Arm has been integrating KleidiAI into a variety of frameworks, from a universal framework that can run any network on any device to a task-specific point-to-point solution.
It is worth mentioning that the word Kleidi is derived from the Greek word meaning "key", which is the key to unlocking more performance for developers.
3. The wave of AI PCs is surging, and the growth of the WoA ecosystem is accelerating
It's worth mentioning that this time Arm highlighted the progress they have made in the Windows on Arm (WoA) ecosystem, which is also due in part to the rapid development of AI PCs.
McNiven made it clear that this year has been a fruitful year for the WoA ecosystem. In addition to Microsoft Office, Dropbox, Zoom, Adobe suite, more and more applications are becoming Arm native applications, especially Baidu, Bilibili, Chrome browser, iQiyi, Sogou, Tencent QQ Music and other head applications.
There are also a number of open-source tools for creators, such as the recently added Audacity, Blender, and OBS Studio, which integrate a large number of open-source libraries and developer tools.
Arm has partnered with Microsoft to grow the ecosystem by funding open source and releasing the Arm Performance Library for Windows. It is understood that for most users, they spend most of their time running Arm-native applications on Arm-based AI PCs.
In McNiven's view, the emergence of new use cases will bring new life to the market, whether it's in PC laptops, IoT, infrastructure or mobile phones, and Arm wants Arm's technology to be available wherever computing happens.
The rise of AI use cases brings ubiquitous computing, which is a great opportunity for both Arm and partners, and the ecosystem is critical for Arm.
Conclusion: To build a computing platform for future AI, Arm is one step ahead
It can be seen that Arm is combining cutting-edge CPU and GPU technology, production-ready physics implementation, and continuous software optimization, and Arm terminal CSS combined with Kleidi software to become an excellent platform for future AI computing.
Facing the future of generative AI, more innovations will inevitably emerge in the mobile intelligent terminal industry, and it remains to be seen how Arm will maintain its core competitiveness and continue to grow its ecosystem through technological innovation.