Diversification of Chips: Training and Inference

NVIDIA is the current leader in the AI accelerator space, and most AI frameworks are built and optimized for NVIDIA hardware first before they are ported to other platforms. This concentration creates a significant power imbalance in the AI industry, as NVIDIA's supply capacity and allocation decisions dictate how fast AI labs and cloud providers can scale up, and each new batch of scarce chips that NVIDIA rolls out leaves its customers competing against one another at prices NVIDIA sets.

For most of the last decade, there were few AI accelerators besides NVIDIA GPUs that could run models at scale. Now, however, AMD is emerging as an alternative to NVIDIA GPUs, while Google, Amazon, and Microsoft have all developed their own AI accelerators (TPUs, Trainium, and Maia, respectively). Leading AI labs are adopting these platforms at an increasing rate, and multi-vendor setups are becoming more common among them, but adoption beyond the leading labs has been slow and has varied significantly by workload type.

AI workloads fall into two categories, training, which is the process of building new models, and inference, which is the process of using already trained models to generate outputs, and the two have quite different requirements. While inference requires an accelerator that handles individual requests efficiently and cheaply, training requires thousands of accelerators to operate in coordination, and inference can be further split into two phases with different accelerator requirements. As a result, diversity among AI accelerators will appear in inference well before it appears in training.

Synchronization in training

To train a model, a batch of text is fed through the model to predict the next token at each position in the batch, a loss function is computed, the gradient of the loss with respect to all the weights is computed through backpropagation, and an optimizer updates the weights, after which the process repeats for the next batch of text. Training a frontier-scale model is therefore a process of keeping tens of thousands of chips running non-stop for weeks or months.

Because no single accelerator has enough memory and compute to train a model on its own, models are parallelized across thousands of accelerators in three complementary ways [1][2]. In data parallelism, each group of accelerators stores a full copy of the model and trains on a different subset of the batch, and after the backward pass the groups perform an all-reduce to exchange gradients, so that all accelerators end up with the same averaged gradient.

In tensor parallelism, each weight matrix in each layer is split across multiple GPUs, each of which performs part of every matrix multiplication, and because these GPUs must exchange results several times per layer, tensor parallelism is usually used within a single server, where GPUs communicate at the highest speeds. In pipeline parallelism, the model is split into stages of consecutive layers, usually on different servers, which pass micro-batches to one another like an assembly line.

All of these parallelization methods require accelerators to communicate with each other constantly, and on NVIDIA hardware, collective operations such as all-reduce are implemented by NCCL (the NVIDIA Collective Communications Library) [3]. Within a server, GPUs communicate via NVLink, which has a bandwidth of about 900 GB/s per H100 and 1.8 TB/s per B200 [4][5], while traffic between servers travels over lower-bandwidth networks such as InfiniBand.

Since training is synchronous, each step only takes place once all accelerators have completed their computation and performed an all-reduce, which means that the throughput of the whole job is bottlenecked by the slowest accelerator, and if any one accelerator fails, the whole job stops. At frontier scale this happens often, and Meta's 405-billion-parameter Llama 3 training job, for instance, ran for 54 days on 16,384 H100 GPUs and saw 419 unexpected interruptions, about one every three hours [6]. About 58.7% of those interruptions were GPU-related, including faulty GPUs, NVLink failures, and HBM errors [7], and Meta kept over 90% of the training time effective thanks to automated failure detection and recovery systems.

Training at scale requires a lot of engineering to keep thousands of accelerators in sync, which is one reason why a mixed NVIDIA and AMD cluster is challenging. Since the two kinds of accelerators have different completion times, the faster ones will always end up waiting at each synchronization point, so the cluster pays for both the faster and slower accelerators but is only as fast as the slower ones. While it is theoretically possible to allocate more work to the faster accelerators, balancing that work over thousands of accelerators and three types of parallelism is a challenging scheduling problem.

Additionally, NCCL and RCCL, the collective communication libraries for NVIDIA and AMD GPUs respectively, are not designed to perform a single all-reduce over both types of accelerators. There have been recent efforts at mixed-vendor communication, such as Hu et al.'s training of 7- and 8-billion-parameter models on mixed AMD and NVIDIA clusters with up to 98% of the throughput of a pure NVIDIA system [8], and Moreh's collective communication library for mixed-vendor clusters [9], but both involve models much smaller than frontier-scale models.

Third, there is the numerical problem, since floating-point arithmetic is non-associative, meaning that summing the same numbers in a different order can produce a slightly different answer [10], and because different accelerators use different kernel implementations and different orderings of additions, the small errors that accumulate over millions of iterations can make training unstable.

Fourth, there are network issues, because NVIDIA's highest-speed interconnect, NVLink, allows NVIDIA GPUs to connect among themselves and, through NVIDIA's NVLink Fusion initiative, with a few other processors such as Amazon's forthcoming Trainium4 [11]. AMD GPUs are not part of this program, so NVIDIA and AMD GPUs in the same cluster must rely on the slower network between servers to talk to each other, and because tensor parallelism, the most communication-heavy type of parallelism, is unlikely to work across vendors, all of these factors point toward homogeneous clusters.

Non-NVIDIA homogeneous clusters do exist, as Google uses clusters of TPUs to train its models and Anthropic uses more than one million Amazon Trainium2 chips to train and serve Claude, including in Project Rainier [12]. Each such cluster is homogeneous and operated by an organization with a team of engineers who specialize in low-level programming for that specific accelerator, as Anthropic, for instance, has built its own Trainium kernels and collaborates with Annapurna Labs, the team at Amazon that designed the chip [13].

Clusters therefore differ from one another, with the same lab training one model on TPUs and another on GPUs, but it is rare for a single model to be trained across different types of chips, and only a few players can afford to maintain more than one type of training cluster.

Inference across accelerators

Inference is when the model is used to answer a question, for instance to write a chatbot reply, propose a code snippet, or make a choice for an AI agent, and during inference the model parameters do not change. Each request goes through two phases of computation that have very different compute profiles. In the prefill phase, the model computes all the tokens in the input prompt in parallel, which involves running very large matrix multiplications that saturate the accelerator and are typically compute-bound [14].

The next phase, decode, produces the output token by token, where every step of computation depends on the prior tokens and requires fetching model parameters from memory, making it memory-bound. For example, a 70-billion-parameter model is about 140 GB at 16-bit, and for every token produced in decode, the model parameters must be fetched to compute just about two floating-point operations per parameter.

An H100 could theoretically perform close to 1,000 trillion 16-bit floating-point operations per second but can only fetch about 3.35 TB of memory per second [4]. This means that to saturate its floating-point computation capacity, the H100 needs to perform about 300 floating-point operations per byte of data fetched; however, decoding one request performs about one. Batching, or running multiple user requests at once, helps a little, as it's possible to fetch the model parameters once and then produce one token per request in a batch of 64 concurrent requests, resulting in a 64x compute-to-bandwidth ratio (Scaling Book, JAX team at Google DeepMind [14]).

However, batching is limited by the key-value (KV) cache, a key component of inference serving that stores the intermediate attention values for the tokens already processed by the model so that they don't need to be recalculated for every token produced. The KV cache for a Llama 3 70B model is about 0.33 MB per token at 16-bit [6], which means an 8,000-token context will require about 2.6 GB and running 32 contexts in the same batch will require about 84 GB, which exceeds the H100's entire 80 GB of memory for just the KV cache before considering the model weights.

As such, the efficiency of the KV cache is a key consideration in modern inference system designs. For example, vLLM uses a PagedAttention mechanism that relies on operating-system paging techniques to minimize the cache's footprint [15]. Ultimately, decode is limited by the accelerator's memory bandwidth, which determines how fast parameters can be fetched, and memory capacity, which determines how many concurrent requests can share each fetch, and thus the cost of serving requests, rather than by the accelerator's peak compute performance.

Inference is therefore a good candidate for multi-accelerator systems, as inference queries are independent, so an inference provider can process some queries on NVIDIA GPUs and some on AMD GPUs without users noticing, and if one of the accelerators is slow or fails, only its queries are affected. Since decoding is memory-bound, an accelerator with more and faster memory at a lower cost than NVIDIA's chips can also be very good at inference without having the same compute power, and Table 1 summarizes the memory size and memory bandwidth of several modern accelerators.

Table 1. Memory size and peak memory bandwidth per chip for several AI accelerators.

The MI355X matches the B200 in memory bandwidth but has much more memory, which means a large model can fit on fewer chips with more memory left for the KV cache. In addition, prefill and decode have different needs, since prefill is compute-bound while decode is memory-bound, so a chip that is most cost-effective for prefill may not be the best choice for decode.

Patel et al. show with Splitwise, which runs prefill on one cluster of machines and decode on another while moving the KV cache between them, that decode can run efficiently on older A100 GPUs while H100s handle prefill, achieving up to 1.4 times the throughput at 20% lower cost [19]. Gimlet Labs goes even further, running prefill on NVIDIA B200s and decode on Intel Gaudi 3 accelerators within the same request [20], which requires the KV cache to be translated between the two vendors' formats in roughly 20 to 50 microseconds, and the authors estimate that this approach is around 1.7 times more cost-effective for typical workloads than relying on one vendor's hardware.

While training across vendors is still an open research problem, inference can already run across vendors within the same request, and hardware vendors are starting to take this into account. Google, for example, introduced Ironwood in 2025 as its first TPU "for the age of inference" [17], and in 2026 it split its eighth TPU generation into the TPU 8t for training and the TPU 8i for inference [21]. Microsoft developed Maia 200 purely for inference [18], and NVIDIA announced Rubin CPX, a GPU for the prefill step that allows prefill and decode to run on different NVIDIA chips [22], which shows that the trend toward specialized hardware is growing.

NVIDIA's software stack

NVIDIA dominates inference partly because of its software stack, CUDA, which consists of a programming language, a compiler from that language to GPU instructions, the cuBLAS linear algebra library, the cuDNN neural network library, the NCCL communication library, and the TensorRT model serving framework. Most developers work in PyTorch, which calls these libraries under the hood, and because CUDA has been NVIDIA's stack since 2006 [23], the majority of AI software has been built and tested on CUDA first, so every alternative accelerator needs a software stack of its own. AMD has ROCm, with a CUDA-like language called HIP [24], Google's TPUs use the XLA compiler, mostly through JAX [25], and Amazon's Trainium uses the Neuron SDK [26].

Of these, AMD's stack is the easiest to blend with NVIDIA's, since AMD's version of PyTorch has the same torch.cuda interface, so code written for a "cuda" device can run on AMD GPUs with little or no modification [27]. For CUDA-native code, AMD's HIPIFY tool automates much of the conversion to HIP [28], and Triton, OpenAI's Python-based kernel language, also works on AMD GPUs [29]. Any cloud provider can also buy AMD chips, unlike Trainium, which is only available on AWS and which Amazon has only recently begun discussing with other data centers [30], and unlike TPUs, which Google only began selling to select third-party data centers in 2026 [31].

Some platform differences still cannot be easily abstracted away, since code that uses NVIDIA libraries cannot be ported automatically, and high-performance kernels are often implemented in CUDA first and ported to other platforms only later, if at all. This matters less for inference than for training, because most inference on common open models happens through serving engines like vLLM, which runs on NVIDIA GPUs, AMD GPUs, Google TPUs, AWS Neuron chips, and Intel Gaudi [32].

Adoption by labs and neoclouds

Diversification is beginning in AI labs and neoclouds as inference rapidly increases its share of AI compute, and Deloitte expects inference to make up around two-thirds of AI compute in 2026, compared with one-third in 2023 and half in 2025 [33]. Epoch AI estimates that demand for tokens is growing roughly tenfold a year, whereas compute capacity grows three- to fourfold [34].

This is partly because AI workloads are changing, since a single chatbot response uses a few hundred tokens, while an agent working on a task for an hour, performing numerous document operations, API calls, and self-evaluation steps, may use orders of magnitude more, and because agents keep long contexts, they also increase the size of the KV cache and the memory intensity of the workload.

As demand for AI chips outstrips supply, any chip that can support a particular workload will suffice, and far more chips can serve inference than can train frontier models. The largest AI labs have already diversified their hardware, and Anthropic says it uses three chip types, Google's TPUs, Amazon's Trainium, and NVIDIA's GPUs, with access to up to one million TPUs [35]. My ballpark estimate is that around $402 billion of Anthropic's $518 billion in compute commitments are for non-NVIDIA chips.

OpenAI has gone even further, with contracts for 10 gigawatts of NVIDIA systems [36], 6 gigawatts of AMD GPUs [37], 10 gigawatts of custom accelerators co-developed with Broadcom [38], up to 750 megawatts of Cerebras inference capacity [39], and around 2 gigawatts of Trainium [40], so although NVIDIA remains OpenAI's main chip vendor, it is no longer its sole one.

Most neoclouds, including Nebius and CoreWeave, still offer only NVIDIA hardware, and Nebius says that 99% of its customer demand is for NVIDIA, while a CoreWeave executive has said that it is hard to focus on anything else when 99% of the market wants one thing [41]. Most smaller neoclouds also use only NVIDIA chips, as it is much easier for a small team to support a single chip type, which requires a single software stack, driver set, and spares inventory.

The neoclouds that support non-NVIDIA chips are primarily doing inference, as Crusoe has offered AMD GPUs since October 2024 [42] and is now using AMD's MI355X to scale its managed inference offering [43], and because Crusoe runs the models itself, its customers never have to interact with AMD's software. TensorWave raised $350 million in 2026 to build an all-AMD cloud [44], and Fluidstack has signed an agreement to host Google TPUs in its data centers, making it one of the first third-party cloud providers to do so [41][45].

General thoughts

NVIDIA's strongest position is in workloads that involve thousands of chips running together, such as large-scale training, which performs best on homogeneous chips connected by NVLink, orchestrated by NCCL, and supported by a mature software stack. NVIDIA is the industry leader in all of these aspects, so its main competition in training comes from custom chips that large companies can develop and support in-house. Inference is less demanding in all of these respects, as requests are independent, the decode phase is primarily memory-bound, the prefill and decode phases can use different hardware, and managed inference services never expose the chip to users.

NVIDIA's position over the next few years will therefore depend largely on how much AI compute is used for training. As inference makes up a larger fraction of AI compute, a smaller part of the market will depend on the strengths that favor NVIDIA, and neoclouds that switch to non-NVIDIA chips, starting with inference, will be less reliant on NVIDIA when its latest chips are in short supply.

For most neoclouds, the more immediate obstacle is operational, since serving popular open models on a second type of chip through engines like vLLM is already practical, while supporting a second software stack, driver set, and spares inventory remains a real cost for a small team. Managed inference services like Crusoe's reduce that cost by keeping the second chip invisible to customers, which makes them the most likely path for smaller neoclouds to diversify.

Several developments over the next year will show how quickly this shift happens, including whether more neoclouds announce AMD or TPU capacity, whether Google places TPUs in more data centers it does not own, whether Amazon begins selling Trainium outside AWS [30], whether tools for running the same code on different chips keep improving, and whether Nebius and CoreWeave still report that nearly all of their demand is for NVIDIA a year from now [41].

Find my more sporadic thoughts on this on twitter, including on how inference is becoming the largest market for AI compute, on why NVIDIA is central to training but less so to inference, on Anthropic relying mostly on chips that are not NVIDIA's, on the choice neoclouds face between staying with NVIDIA and buying other chips, on the end of many NVIDIA-only clouds, and on how prefill and decode use the same GPUs differently.

References

1. Shoeybi, M., et al. "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism." arXiv:1909.08053, 2019. https://arxiv.org/abs/1909.08053
2. Narayanan, D., et al. "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM." arXiv:2104.04473, 2021. https://arxiv.org/abs/2104.04473
3. NVIDIA. "NCCL Overview." NVIDIA Collective Communications Library Documentation. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/overview.html
4. NVIDIA. "NVIDIA H100 Tensor Core GPU." https://www.nvidia.com/en-us/data-center/h100/
5. Cornell Virtual Workshop. "B200 Memory and NVLink 5." https://cvw.cac.cornell.edu/gpu-architecture/horizon-gpus-blackwell-b200/b200_mem_nvlink5
6. Grattafiori, A., et al. "The Llama 3 Herd of Models." arXiv:2407.21783, 2024. https://arxiv.org/abs/2407.21783
7. Tom's Hardware. "Faulty Nvidia H100 GPUs and HBM3 Memory Caused Half of Failures During Llama 3 Training." 2024. https://www.tomshardware.com/tech-industry/artificial-intelligence/faulty-nvidia-h100-gpus-and-hbm3-memory-caused-half-of-the-failures-during-llama-3-training-one-failure-every-three-hours-for-metas-16384-gpu-training-cluster
8. Hu, J., et al. "Joint Training on AMD and NVIDIA GPUs." arXiv:2602.18007, February 2026. https://arxiv.org/abs/2602.18007
9. Moreh. "HetCCL: Accelerating LLM Training with Heterogeneous GPUs." Technical report, 2026. https://moreh.io/technical-report/hetccl-accelerating-llm-training-with-heterogeneous-gpus-260130/
10. Thinking Machines Lab. "Defeating Nondeterminism in LLM Inference." 2025. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
11. The Register. "Amazon to Fuse Nvidia's NVLink into Trainium4 Accelerators." December 2, 2025. https://www.theregister.com/special-features/2025/12/02/amazon-to-fuse-nvidias-nvlink-into-trainium4-accelerators/2071381
12. Anthropic. "Anthropic and Amazon Expand Collaboration for Up to 5 Gigawatts of New Compute." April 20, 2026. https://www.anthropic.com/news/anthropic-amazon-compute
13. Computerworld. "AWS and Anthropic Ink Deal to Accelerate Model Development, Enhance AI Chips." November 2024. https://www.computerworld.com/article/3611594/aws-and-anthropic-ink-deal-to-accelerate-model-development-enhance-ai-chips.html
14. Austin, J., et al. "All About Transformer Inference." How to Scale Your Model (the Scaling Book), Google DeepMind. https://jax-ml.github.io/scaling-book/inference/
15. Kwon, W., et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." arXiv:2309.06180, 2023. https://arxiv.org/abs/2309.06180
16. AMD. "AMD Instinct MI350 Series." https://www.amd.com/en/products/accelerators/instinct/mi350.html
17. Google. "Ironwood: The First Google TPU for the Age of Inference." April 2025. https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/ironwood-tpu-age-of-inference/
18. Microsoft. "Maia 200." January 2026. https://news.microsoft.com/maia-200/
19. Patel, P., et al. "Splitwise: Efficient Generative LLM Inference Using Phase Splitting." ISCA 2024. https://www.microsoft.com/en-us/research/wp-content/uploads/2023/12/Splitwise_ISCA24.pdf
20. Gimlet Labs. "Splitting LLM Inference Across Different Hardware Platforms." October 13, 2025. https://gimletlabs.ai/blog/multivendor-prefill-decode-disaggregation
21. Google. "Our Eighth Generation TPUs: Two Chips for the Agentic Era." April 2026. https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/eighth-generation-tpu-agentic-era/
22. The Register. "Nvidia's Context-Optimized Rubin CPX GPUs Were Inevitable." September 10, 2025. https://www.theregister.com/2025/09/10/nvidia_rubin_cpx/
23. NVIDIA. "CUDA Zone." https://developer.nvidia.com/cuda-zone
24. AMD. "ROCm Documentation." https://rocm.docs.amd.com/
25. OpenXLA. "XLA." https://openxla.org/xla
26. AWS. "AWS Neuron Documentation." https://awsdocs-neuron.readthedocs-hosted.com/
27. PyTorch. "HIP (ROCm) Semantics." https://docs.pytorch.org/docs/stable/notes/hip.html
28. AMD. "HIPIFY Documentation." https://rocm.docs.amd.com/projects/HIPIFY/en/latest/
29. AMD. "Kernel Development and Optimization with Triton." ROCm Documentation. https://rocm.docs.amd.com/projects/ai-developer-hub/en/v11.0/notebooks/gpu_dev_optimize/triton_kernel_dev.html
30. Data Center Dynamics. "Amazon Could Sell Trainium AI Chips to Data Centers, Report." June 2026. https://www.datacenterdynamics.com/en/news/amazon-could-sell-trainium-ai-chips-to-data-centers-report/
31. Data Center Dynamics. "Google to Sell TPUs to a 'Select Group of Customers' for Their Data Centers." April 2026. https://www.datacenterdynamics.com/en/news/google-to-sell-tpus-to-a-select-group-of-customers-for-their-data-centers/
32. vLLM. "vLLM: Fast, Memory-Efficient LLM Inference and Serving." https://vllm.ai/
33. Deloitte. "TMT Predictions 2026." Deloitte Insights, 2025. https://www.deloitte.com/us/en/insights/industry/technology/technology-media-and-telecom-predictions.html
34. Emberson, L., and Sevilla, J. "Is a Compute Crunch Coming?" Epoch AI, Gradient Updates, 2026. https://epoch.ai/gradient-updates/is-a-compute-crunch-coming
35. Anthropic. "Expanding Our Use of Google Cloud TPUs and Services." October 23, 2025. https://www.anthropic.com/news/expanding-our-use-of-google-cloud-tpus-and-services
36. NVIDIA. "OpenAI and NVIDIA Announce Strategic Partnership to Deploy 10 Gigawatts of NVIDIA Systems." September 2025. https://nvidianews.nvidia.com/news/openai-and-nvidia-announce-strategic-partnership-to-deploy-10gw-of-nvidia-systems
37. OpenAI. "AMD and OpenAI Announce Strategic Partnership to Deploy 6 Gigawatts of AMD GPUs." October 6, 2025. https://openai.com/index/openai-amd-strategic-partnership/
38. OpenAI. "OpenAI and Broadcom Announce Strategic Collaboration to Deploy 10 Gigawatts of OpenAI-Designed AI Accelerators." October 13, 2025. https://openai.com/index/openai-and-broadcom-announce-strategic-collaboration/
39. Data Center Dynamics. "OpenAI Signs $10 Billion Deal with Cerebras, with 750MW of Big-Chip Compute." January 2026. https://www.datacenterdynamics.com/en/news/openai-signs-10-billion-deal-with-cerebras-with-750mw-of-big-chip-compute/
40. Amazon. "OpenAI and Amazon Announce Strategic Partnership." SEC Form 8-K, Exhibit 99.1, February 2026. https://www.sec.gov/Archives/edgar/data/1018724/000110465926021050/tm267374d1_ex99-1.htm
41. Data Center Dynamics. "Nebius, Lambda, and CoreWeave Unlikely to Buy Google TPUs Any Time Soon." 2026. https://www.datacenterdynamics.com/en/news/nebius-lambda-and-coreweave-unlikely-to-buy-google-tpus-any-time-soon/
42. Crusoe. "How Crusoe First Virtualized AMD MI300X on Linux." June 18, 2025. https://www.crusoe.ai/resources/blog/crusoe-virtualize-amds-instincttm-mi300x-gpus-on-linux
43. TipRanks. "Crusoe Highlights Collaboration With AMD on Scalable AI Inference." August 4, 2026. https://www.tipranks.com/news/private-companies/crusoe-highlights-collaboration-with-amd-on-scalable-ai-inference
44. Data Center Dynamics. "AMD-Based AI Cloud TensorWave Secures $350M Series B Funding." June 2026. https://www.datacenterdynamics.com/en/news/amd-based-ai-cloud-tensorwave-secures-350m-series-b-funding/
45. Data Center Dynamics. "Google Offers Its TPUs to AI Cloud Providers, Report." 2025. https://www.datacenterdynamics.com/en/news/google-offers-its-tpus-to-ai-cloud-providers-report/