Under the wave of ultra-high definition, how does ARM Technology add "core" potential energy to the multimedia market?

In the digital era, ultra-high-definition video is accelerating the deep integration with a new generation of information technologies such as 5G, AI, VR, etc., giving rise to a large number of new applications and scenarios for smart homes, smart cars, industrial security and other fields.

According to data released by the China Electronic Information Industry Development Research Institute, as of the end of 2022, the scale of the mainland's ultra-high-definition video industry was about 3 trillion yuan, and it is expected that the scale of China's ultra-high-definition video industry will exceed 7 trillion yuan by 2025.

As the digital multimedia market enters the "fast lane", what new requirements do multimedia processors need to meet in various scenarios such as "device, edge, and cloud"? In view of the latest development trends in video encoding and decoding, display and other links, how should chip manufacturers adjust their layout? Recently, the reporter communicated with the relevant person in charge of ARM Technology.

"Linglong" VPU iteration: to meet the challenges of ultra-high-definition codec

According to Dong Feng, senior product manager of ARM Technology's VPU, massive multimedia data is coming from terminal fields such as mobile phones to cloud fields such as live broadcasting. And from HD to 4K to 8K, higher resolutions have also led to exponential growth in data.

In the process of video transmission and storage, how to show more picture details with the same or even smaller amount of data? Video compression technology becomes a "key". The latest international video coding standard in the industry is H.266, which can save about 50% of data traffic at the same definition compared to the previous generation standard, H.265. Based on this, H.266 can greatly improve the compression efficiency of 4K and 8K UHD video, thereby significantly reducing the transmission and storage cost of HD video.

Dong Feng further said that the "Linglong" V510/V710 VPU just released by ARM Technology is the first batch of VPU (video processor) in China that supports both H.266 standard encoding and decoding, and its single-core encoding and decoding performance can reach 4K60 frames per second. To meet the different needs of AI large model training in different scenarios such as "device, edge, and cloud", the "Linglong" V510/V710 VPU adopts a programmable processing architecture with multi-core and multi-format codec fusion, which can greatly save storage space and bandwidth costs, and provide multiple differentiated configurations such as 1-4 cores and 4-8 cores on demand, helping customers achieve flexible deployment.

From the perspective of generational improvement, compared with the previous generation "Linglong" V6/V8 VPU, the new generation "Linglong" V510/V710 VPU mainly achieves dimensional upgrading in three dimensions. The first is to double the encoding performance of H.264 and H.265, and the second is to improve the encoding quality of H.266 by 30%. The third is to achieve a 20% improvement in encoding quality compared to the veryslow grade of X265.

When it comes to the consideration of the proportion of coding quality improvement, Dong Feng told reporters, "Any improvement has a cost, according to our overall technical assessment, if we want to achieve a 50% quality improvement, the area needs to be increased by 10 times." From the perspective of IP, we hope to empower more customers, and the entire architecture is based on a basic platform to adapt to different scenarios. Based on this, we have evaluated that achieving a 20% area gain is the best, and we will optimize accordingly based on these considerations to achieve a balance between system performance, power consumption, and area. ”

"Linglong" DPU debut: flexibly meet the needs of multiple displays

In addition to transmission and storage, display is also an important part of video image processing. Ultra-high-definition, ultra-large screen, cross-screen linkage, multi-screen different display ...... Chai Weihua, senior product manager of DPU at ARM Technology, believes that the current multimedia display presents a series of new trends, and for chip manufacturers, it is not only necessary to provide key capabilities such as real-time processing, image quality enhancement, and data security according to diversified market needs, but also to consider how to support the entire SoC system more friendly from the perspective of IP design, so as to adapt to the rhythm of shorter and shorter chip time to market.

As the first DPU (display processor) developed by ARM Technology, how does the recently released "Linglong" D8/D6/D2 DPU have the above characteristics? On the one hand, it can cover all scenarios with high to low performance requirements, such as "Linglong" D8 for scenarios with high performance requirements, with a single-core performance of up to 8K60 frames per second, and a single IP in multi-screen linkage scenarios also supports up to 4 channels of 4K60 frames per second real-time display output.

On the other hand, Chai Weihua said, "The product architecture of the 'Linglong' DPU is based on a standardized and modular design, which can be flexibly programmed and configured as needed. The benefit of this is that it can be configured and software can be scheduled to allow hardware to work in parallel at the same time to achieve maximum performance. In addition to the architecture, the "Linglong" DPU team also considered rich applications in various scenarios during the design, and can support up to 16 layers of content input, effectively promoting the utilization and management of system resources. ”

In terms of system support, the "Linglong" D8/D6/D2 DPU is deeply synergistic with the Arm architecture, and has a built-in AFBC frame compression and decompression module, which can save bandwidth and power consumption to a greater extent while ensuring the high performance and high image quality of the overall system solution. The new product also integrates the TBU module in the Arm MMU-600/700, which can be specially optimized for various display scenarios and further reduce system access latency. In order to protect users' key display data, the "Linglong" DPU also supports the Arm TZMP architecture, realizing the integration of software and hardware, which can meet the needs of users for more advanced modes such as bank watermarking.

"Whether it's bandwidth optimization, memory management, or security requirements, we take the product into account from the very beginning." Chai Weihua said that the "Linglong" D8/D6/D2 DPU can provide ultra-high image quality display experience for smart terminals such as car cockpits, mobile phones, PCs, tablets, TVs, and wearable devices with flexible expansion, high efficiency and low latency, low system bandwidth and high data security, fully meeting the diversified display needs of intelligent application scenarios.

Rooted in the soil: Consolidate the technical foundation of the semiconductor industry

With the increasing popularity of intelligent terminals and the explosive growth of various scenario applications, users' demand for multimedia is becoming more and more fragmented. Zhou Hua, head of multimedia processor R&D at ARM Technology, said frankly that in addition to standard IP, many customers have their own ideas and want to add some of their own things to produce some differentiation on the basis of commonality. Therefore, the DPU and VPU of "Linglong" multimedia products are not aimed at a single market, and ARM hopes that the products will cover all market segments as much as possible, and build products that are both standardized and flexible, so that the IP can play the greatest value.

"Therefore, 'Linglong' multimedia products basically cover mainstream markets such as AIoT scenarios, automotive scenarios, client mobile terminal scenarios and cloud infrastructure scenarios, as well as emerging applications such as AI and smart cars. We will also make some expansions to integrate some unique elements into the system through open interfaces to meet the needs of customers in the domestic market for all aspects of multimedia processors. Zhou Hua said.

In fact, the release of the new "Linglong" VPU and DPU also adds an important piece of the puzzle to the product matrix of ARM Technology's self-developed business. Coupled with the previously released "Zhouyi" NPU, "Xingchen" CPU, and "Shanhai" SPU, this series of locally developed processors jointly pressed the "acceleration button" for the technological innovation and commercialization of the domestic semiconductor industry. According to the data, ARM Technology's self-developed business has authorized more than 220 local manufacturers, and the cumulative shipment of related chips has exceeded 500 million, injecting strong impetus into promoting the quality and upgrading of the domestic semiconductor industry.

It is worth mentioning that in July this year, ARM Technology also innovatively introduced the "Arm Technology Licensing Subscription" model, focusing on providing local chip designers with practical, flexible and leading solutions "Family Bucket". Based on "global standards, local innovation", ARM Technology has also further connected the Chinese market with the Arm global ecosystem, helping local manufacturers grasp new opportunities in the intelligent computing industry.

According to Zhou Hua, in the future, ARM Technology will continue to take root in the local market and strengthen the multimedia technology stack of the "Linglong" product line. By integrating cutting-edge AI processing units, the "Linglong" product line is expected to exert higher-end and efficient intelligent image processing capabilities in the fields of AI mobile phones, AI PCs, smart cars, and data centers.