Amazon is developing a stronger AI chip: one that rivals Nvidia's GPUs for half the price

On Friday afternoon, inside Amazon's chip lab in Austin, Texas, six engineers tested a new server design that was kept under strict secrecy.

Amazon executive Rami · Sinnoh said during a tour of the lab on Friday that the server houses Amazon's artificial intelligence chips to compete with those of market leader Nvidia.

Amazon is developing its own processors to limit its reliance on expensive Nvidia chips (the so-called Nvidia tax) that power part of its Amazon Web Services' AI cloud business, which is the main growth driver.

Amazon hopes to help customers perform complex calculations and process large amounts of data at a lower cost with its self-developed chips. Its rivals Microsoft and Alphabet are doing the same.

Sinno, director of engineering at Annapurna Labs, part of Amazon's cloud business, said Amazon's customers are increasingly demanding cheaper alternatives to Nvidia.

Amazon acquired Annapurna Labs in 2015.

While Amazon's AI chip research and development is just getting started, its main chip, Graviton, which performs non-AI computing, has been in development for nearly a decade and is now in its fourth generation. AI chips Trainium and Inferentia are newer designs.

"So, in some cases, the price and performance can be increased by 40 to 50 percent, so the cost should be half of running the same model with Nvidia," said David Brown, vice president ·of compute and networking at AWS, Tuesday. ”

AWS sales, which account for nearly one-fifth of Amazon's overall revenue, soared 17% to $25 billion in the first quarter of this year compared to the same period last year. AWS controls about one-third of the cloud computing market, while Microsoft's Azure has about 25 percent.

Amazon said that during the recent Prime Day, the company deployed 250,000 Graviton chips and 80,000 custom AI chips to cope with the surge in activity on its platform.

Break Nvidia's chip encirclement

Amazon has been developing its own AI chips to reduce costs, which has also helped improve the profitability of Amazon Web Services (AWS). However, the e-commerce giant is working hard to develop AI chips that can rival Nvidia's standard chips.

Project migration issues, compatibility gaps, and low usage are some of the issues that are hindering the adoption of Amazon's AI chips. The situation also puts at risk the huge revenue that Amazon earns from its cloud business. According to Business Insider, Amazon's challenges were identified through confidential filings and sources familiar with the matter.

Trainium and Inferentia, top-of-the-line Amazon-designed chips, debuted late last year. The publication reports that last year, Trainium's adoption rate among AWS customers was only 0.5% of Nvidia's graphics processing units.

According to the report, Amazon evaluated the percentage of use of different AI chips through its AWS services in April 2024. Meanwhile, Inferentia has a slightly higher adoption rate at 2.7%. Inferentia is a chip designed for inference, an AI task that typically refers to the computational process in which the end consumer uses an AI model. The report refers to an internal document stating;

"Early customer attempts exposed some friction points and hindered adoption."

The above statement refers to the challenges faced by large cloud customers when transitioning to Amazon's custom chips. Nvidia's CUDA platform is considered more attractive to customers, which the report points to as a key reason.

AWS, the world's largest cloud service provider, is currently developing its self-developed computer chips to facilitate operations. Amazon sometimes flaunts its efforts with AI chips. However, the picture shown in the document is different from what the company expected.

Internal filings say the company is grappling with slow adoption, but Amazon's CEO has a different view. During the Q1 earnings call, Amazon CEO ·Andy Jassy said that demand for AWS chips is high.

"We have the widest selection of NVIDIA compute instances, but given its price/performance advantage over existing alternatives, the demand for our custom silicon, training, and inference is quite high."

Andy Jassy also mentioned early adopters of AWS silicon in a letter to investors, saying, "We already have multiple customers using our AI chips, including Anthropic, Airbnb, Hugging Face, Qualtrics, Ricoh, and Snap. Anthropic, meanwhile, is a completely different story, as Amazon is the startup's biggest supporter. The cloud computing giant has invested $4 billion in Anthropic, an investment agreement that requires the company to use AWS-designed silicon.

Amazon Web Services offers a wide range of processors, from Nvidia's Grass Hopper chips to AMD and Intel. Most of its earnings come from designing its own data center chips, which helps save costs by avoiding the need to buy GPUs from Nvidia.

Amazon launched its first AI chip, Inferntia, in 2018, but Nvidia is still leading the way in delivering solutions that are more widely adopted by different industries. AWS, Microsoft, and Google are Nvidia's largest customers. All of these giants rent GPUs through their cloud services.

In March, AWS CEO Adam Selipsku attended Nvidia GTC 2023. The two companies issued a joint statement focusing on their strategic collaboration in advancing generative AI.

"The deep collaboration between our two companies dates back 13 years when we jointly launched the world's first GPU cloud instance on AWS, and today we offer our customers the broadest range of NVIDIA GPU solutions."

Nvidia's platform, CUDA, is often favored by developers. Because Nvidia has spent years of time and effort creating it, and the industry has adopted it, it has made it easier for them to handle things. Amazon, on the other hand, still needs to solve this puzzle through trial and error.

Reference Links

https://www.channelnewsasia.com/business/amazon-racing-develop-ai-chips-cheaper-faster-nvidias-executives-say-4505146