In 2024, Nvidia will spend 45 billion to buy chips, what to do with domestic GPUs

Tencent Technology

2024-06-19 11:49Posted on the official account of Beijing Tencent News Technology Channel

"Core matters" semiconductor industry research and planning, this issue focuses on the analysis of global HBM chip production capacity in 2024, and exclusively publishes Tencent news, without authorization, please do not reprint.

文 / 腾讯科技特约作者 某厂后端产品PMO总监 Morris.Zhang

Some time ago, the well-known hardware website Tomshardware quoted sources familiar with the matter as saying that in order to ensure the supply of GH200 and H200 this year, Nvidia opened a budget of $1.3 billion and booked part of the HBM3e memory production capacity from Micron and SK hynix.

However, since there is no reasonable calculation of this set of data, the credibility of the budget of 1.3 billion needs to be confirmed, but it is certain that this budget is unlikely to buy out the global HBM production capacity in 2024.

Its value lies in the fact that it can help NVIDIA seize the window of opportunity for "monopoly computing power" in the first half of 2024 - the first product is marketized, and it can grab market share first.

In 2024, about 56 million HBM units will be shipped

In 2024, the total foundry capacity of SK hynix, Micron and Samsung will be expanded to 750,000 wafers, according to the current industry data, the yield rate of HBM3e is about 90%, each wafer can cut out about 750 particles, take an average of 10 layers of packaging, the total global HBM production capacity in 2024 is roughly 56 million (12 layers + 8 layers), but large-scale production capacity is concentrated in the second half of the year, and the proportion in the first half of the year is slightly smaller.

Based on the 2024 CoWoS packaging capacity, it is calculated:

In terms of GPU-HBM vertical packaging capacity, as of the fourth quarter of 2024, the total packaging capacity of global CoWoS is estimated to be about 300,000 wafers, including about 270,000 wafers from TSMC and about 40,000 wafers from Amkor, and because the process nodes of these wafer tape-outs are concentrated in 5nm and 3nm, the current yield is about 38%, and a single wafer can be cut into 30 GPU chips, that is, in the whole year of 2024, the global GPU products packaged by CoWoS will be built. The production capacity is about 9 million pieces. According to the standard computing of a single GPU logic chip with 6 HBM memory chips (AMD MI300 with 8 HBM), the global GPU in 2024 will have a demand for more than 54 million HBM memory (mainly 12 layers).

Small note: CoWoS is commonly known as 2.5D packaging, the full name is Chip on Wafer on Substrate, which is used to meet the interconnection of multiple computing chips and the stacking of memory chips under relatively ideal electrical rules.

Schematic diagram of CoWoS package, the upper red circle is CoW, and the lower red circle is oS Source: Network

At present, the channel unit price of 12-layer HBM particles is about $250+ per piece, and if the statement of the 1.3 billion budget is true, NVIDIA can only order 5.2 million pieces, accounting for only 1/10 of HBM's total annual production capacity.

In fact, according to the estimation of packaging data, NVIDIA has booked a CoWoS production capacity of more than 140,000 wafers in 2024, of which TSMC will receive 120,000 wafers as a "major supplier" and Amkor will receive 2-30,000 wafers as a "secondary supplier", corresponding to an overall production capacity of nearly 4.5 million GPUs.

According to the 1:6 ratio of each GPU logic chip and storage particle, that is, NVIDIA needs about 27 million HBMs throughout the year, based on the cost of $250 per chip, which means that Nvidia's annual purchase of HBM chips can be predicted to be $6.8 billion, far exceeding the $1.3 billion budget previously disclosed by the media.

Note: Consumer graphics cards such as the NVIDIA RTX series that use GDDR6 particles will naturally not be counted in the CoWoS production capacity; The above-mentioned NVIDIA production capacity refers specifically to the Hopper series and the B200 of the Blackwell architecture, so it is estimated that NVIDIA's HBM particle order demand in 2024 is 27 million.

Nvidia H100 physical picture, the logic chip in the middle, next to the 6 HBM chips Source: network

Behind HBM's tight production capacity

As of the fourth quarter of 2024, the scheduled production capacity of HBM particle packaging corresponding to GPU supercomputing products is about 9 million, combined with the expansion plan of Hynix/Samsung/Micron memory factories in 2025 with a total of nearly 60 million HBM (mainly 12 layers, slightly less 8 layers), these two supply data are consistent, and it also shows that the CoWoS and HBM production capacity in 2024-2025 is sufficient.

However, although there is no shortage of production capacity, but the above data is "annual" after all, in fact, a lot of production capacity will not be opened until the fourth quarter, and the scheduled production capacity is of course the sooner the better, timeliness is the key condition, the window of opportunity at the beginning of the first half of the year is more important, if the second half of the year only began to put into production, then the product into the channel will have to wait until the next year.

In addition, regarding the expansion of CoWoS production capacity, it currently mainly relies on the production capacity of TSMC and Amkor, such as Tailian Electric, GF and other production lines, although they can also do the front-end 65nm interposer, but due to the lack of advanced process coverage capabilities, that is, it is impossible to OEM the front-end advanced process logic chips and interposers, and it is impossible to achieve a one-stop CoWoS full stack.

Samsung plans to introduce a full-stack CoWoS package, which will be named I-CUBE/H-CUBE and compete with TSMC for orders; However, Samsung will not be able to open production capacity in 2024, and 2025 may be more beneficial to Samsung, as an IDM manufacturer that supplies HBM and CoWoS at the same time, its process characteristics and price advantages are obvious. (In the future, Samsung will also introduce the X-CUBE 3D package, collectively known as SAINT, which is Samsung's advanced packaging technology.) ）

In addition, if Intel's foundry business successfully achieves independent operation in 2025, its Intel Foveros packaging solution is also worth waiting and seeing.

At present, we see that HBM's global inventory and channel turnover are very tight, and the actual reason is that there is only one HBM3e supplier in the current period, which coincides with the peak demand for HBM brought about by the rapid development of large models, and then there is an imbalance between supply and demand, but this imbalance will slow down with the expansion of production capacity.

However, there is one thing to note, storage particles are a standard SKU, and there is no need to customize the particles themselves, so this category has the so-called spot market. Standard spot can be circulated through channels or distribution platforms, that is, if all the planned production capacity of SK hynix, Micron and Samsung is successfully opened in 2024, the inventory of agents in all regions of the world will be sufficient, and countless sub-channels/sub-agents can be resold unlimitedly.

Therefore, as long as the spot inventory is sufficient and the price trend is good, the spot market will always be able to get the goods, and it will evolve into the format of the DRAM spot channel, which is the industrial characteristic of memory chips.

This raises an interesting question, what if for special reasons a company is unable to place an order with the above three companies?

The spot market is a channel – specific companies can purchase HBM chips from the spot market, and then adapt the controller, I/O and logic packaging.

If HBM memory is still not available in the spot market, 2D-DRAM chips can also be purchased from the spot market, and then a low-spec HBM device can be stacked through the packaging process of TSV vertical through-hole and TCB hot-press bonding (hybrid bonding will be required for more layers stacking in the future).

Of course, the ultimate goal is to develop memory particles and realize the independent industrialization of HBM.

Diagram of H100 and MI300 packaging scheme Source: Network

The threshold of CoWoS: industrial chain and yield

Referring to the current GPU chip configuration, if there is no CoWoS package structure, HBM will not even be placed.

There are currently several categories of CoWoS capacity suppliers available worldwide:

其一是台积电 CoWoS；

The second is for TSMC to complete the fabrication of wafers and front-end interposers (i.e., the "CoW" part of CoWoS, stacking + interconnect), and then hand them over to its own packaging factories (such as idle InFO production capacity) or cooperate with third-party OSAT packaging factories, commonly known as outsourcing packaging factories, to complete the "on Substrate" part, that is, packaging on the substrate;

The third is that it can be entrusted to UMC and GF to produce the interposer, and then sent to the OSAT production line such as Amkor or ASE to complete the "WoS" part. However, as mentioned above, these two process nodes are basically around 65nm, and they cannot OEM advanced front-end advanced process logic chips and interposers, nor can they complete the one-stop CoWoS full stack;

The fourth is Samsung I-Cube/H-Cube and Intel Foveros, both of which can complete full-stack CoWoS delivery, but neither has yet opened production capacity;

Fifth, there are also some CoWoS capabilities in China, but almost all of them are CoW + WoS process docking, that is, the model of UMC, GF + Amkor, and ASE Group.

Compared to other manufacturing processes, CoWoS does not have an extremely cutting-edge technical threshold, the only key is that it ensures high yield in a high-scale process.

Because at the packaging level, if the product has a high defect/failure, then the HBM and other devices connected to the stack above will become irreparable losses, and only TSMC can meet the needs of manufacturers that can take into account higher process nodes and yields.

In terms of CoW+WoS production capacity alone, there is a lot of production capacity in the world (especially WoS manufacturers), but there are not many production capacity that can be applied to the process/yield of advanced computing chips.

The reasons for the above yield threshold are the process:

Taking WoS yield as an example, the difficulty is that the size of the interposer of the package should be strictly limited, that is, the area of the silicon interposer needs to be greater than the sum of the dimensions of two or more bare dies on it. However, as this size gets bigger and bigger, the fifth-generation CoWoS even supports a "2-way lithography stitching approach technology" that allows the size of the interposer to be expanded to 2500mm², and the process risk that comes with it is that on wafer, that is, when stacking silicon wafers, there are problems such as edge distortion and vertical convex joint angles, resulting in poor packaging and testing, while TSMC's CoWoS process has been running in for more than 10 years. Accumulate a lot of knowhow to get today's reliable high yield.

TSMC's external technology ppt, silicon interposer can reach 2500mm² Source: network

Although CoW+WoS is a reasonable division of labor in the industrial chain, it is necessary for the two processes to develop simultaneously to increase the yield of the two products.

At present, the advanced logic chips and interposers in mainland China are basically manufactured by SMIC, and then entrusted to OSAT professional packaging factories to complete WoS packaging and testing; If you can't get the production capacity of SMIC, you can also entrust UMC and GF foundry.

In the future, most of the world's 2.5D packaging will be a front-end + back-end cooperation model; The front-end wafer fab provides an interposer for CoW, and the back-end has a carrier board for WoS. In addition, CoWoS will also be applied to other scenarios, and most products involving AI-HPC will rely on CoWoS packaging in the future, and 2.5D/3D packaging is still more advantageous than OSAT at present.

epilogue

With the high demand for HBM memory in the AI-HPC industry, the product audience, value and market space of this device are all rising, and the unit price of HBM is now several times that of traditional DRAM and about five times that of DDR5.

As mentioned above, the channel unit price of 12-layer HBM particles in the current period has reached $250 per piece, which is slightly higher than that in 2023.

Another way to put it in perspective: At present, on AI-HPC computing chips, the total cost of six HBM3 chips with a capacity of 16GB is usually about $1,500, which is equivalent to $15.6/GB. Converted to H100 SXM5 boards, 6 HBM3 80GB, equivalent to $18.75/GB, accounting for about 50%+ of the material cost of the entire chip.

There is a set of data from Yole: the overall compound growth rate of advanced packaging in the next five years is 40%, of which 3D packaging is more than 100%; And in five years, nearly 40% of HBM will be based on hybrid bonding packaging.

Therefore, regardless of overseas or mainland manufacturers, the potential market space of the above-mentioned industrial directions "HBM, CoW, TSV/Hybrid Bonding" is broad, and it is expected that domestic manufacturers will have mature yields and the possibility of entering overseas markets. At the same time, driven by the boom in the AI-HPC market, HBM's share of the pan-DRAM market is expected to increase significantly, and HBM's share of the total capacity of the DRAM market is expected to rise from 2% in 2023 to 5% in 2024 and exceed 10% by 2025.

View original image 58K

• 

• 

• 

• 

• 

• 

• 

• 

• 

•