The Engines Behind Modern AI: A Clear Look at Today’s Chip Architectures
Why GPUs still rule, what TPUs and wafer-scale chips do differently, and why the “memory wall” matters more than raw FLOPs
In 2018, computer architecture legends John Hennessy and David Patterson stood on stage and declared that a new golden age of chip design was beginning. General-purpose CPUs had slowed dramatically. The future, they said, belonged to specialised architectures built for specific jobs — especially artificial intelligence.
Eight years later, that prediction has come true in spectacular fashion. We now have a real competition between very different approaches to building the silicon that trains and runs large language models. The clear frontrunner remains NVIDIA’s GPUs, but Google’s TPUs, Amazon’s Trainium, Cerebras’s wafer-scale engines, and (until recently) Groq’s Language Processing Units have all earned serious deployment.
Understanding the differences between these chips is no longer just for hardware engineers. The architecture choices determine how fast models respond, how much electricity data centres burn, and ultimately how capable the AI tools we use every day can become.
The Real Problem: Moving Numbers, Not Just Multiplying Them
Almost everything inside a modern transformer model — the architecture behind GPT, Claude, Gemini and most other large models — is matrix multiplication. Training a frontier model requires an almost unimaginable number of multiply-accumulate operations. The shape of those multiplications changes depending on the phase of work:
- Training and “prefill” (processing the full prompt) favour large matrix-matrix multiplies. These are compute-heavy.
- Decode (generating one token at a time) turns into matrix-vector multiplies. These are far more limited by memory bandwidth.
This creates the famous “memory wall.” Compute performance has scaled much faster than the speed at which data can be moved to the compute units. Every successful AI chip is ultimately a different strategy for winning the data-movement game.
The four questions that matter for any architecture are simple:
- Where does the data live?
- How does it get to the compute units?
- What do the compute units themselves look like?
- How do multiple chips talk to each other at scale?
NVIDIA GPUs: Programmability Wins
NVIDIA’s approach is deliberately general. A GPU is a massively parallel processor containing hundreds of Streaming Multiprocessors. Each SM contains CUDA cores for general work and specialised Tensor Cores for matrix multiplies. The entire system is programmed through CUDA, which has become the dominant software stack for AI.
Over successive generations (Volta → Ampere → Hopper → Blackwell → Rubin), NVIDIA has made the matrix units bigger and more independent while keeping them inside the same programmable model. Modern Tensor Cores can fire large matrix operations asynchronously so the rest of the warp can keep doing useful work (softmax, masking, loading the next tile). Memory movement has also been progressively automated with units like the Tensor Memory Accelerator.
The result is a chip that can train models, run inference, render graphics, and do scientific computing. That flexibility, combined with an unmatched software ecosystem, is why NVIDIA still dominates.
Systolic Arrays: Google TPU and Amazon Trainium
Google’s Tensor Processing Units and Amazon’s Trainium take a more specialised path. They use systolic arrays — grids of processing elements that rhythmically pass data through one another. This design is extremely efficient for the dense matrix multiplies that dominate transformers, especially during training and prefill.
The trade-off is reduced flexibility. These chips are excellent at the workloads they were designed for, which is why Google uses them heavily for Gemini and why Anthropic has committed to large numbers of both TPUs and Trainium chips. They are less convenient for rapidly changing research or unusual model architectures.
Wafer-Scale and Deterministic Designs
Cerebras takes the opposite extreme. Instead of cutting a silicon wafer into many individual chips, it builds one enormous processor the size of the entire wafer. This gives enormous on-chip memory and very high internal bandwidth, which helps with certain large models. Cerebras now serves inference for OpenAI.
Groq’s Language Processing Unit took a different bet: extreme determinism and simplicity. By removing much of the complex scheduling hardware found in GPUs, Groq achieved very low and predictable latency for inference. The company was later acquired by NVIDIA in a large deal, bringing that team and technology inside the GPU leader.
Scaling: The Next Battleground
Even the best single chip is not enough. Training and serving frontier models requires thousands of chips working together. There are two scaling regimes:
- Scale-up — connecting chips very tightly inside a server or rack with high-bandwidth links (NVLink, custom interconnects).
- Scale-out — connecting many racks across a data centre with somewhat higher latency networking.
Different architectures make different trade-offs here. Some optimise for enormous single-chip or single-package performance. Others focus on how cleanly they can be networked at massive scale.
Why This Matters Beyond the Data Centre
The architecture race is not abstract. It determines:
- How quickly new models can be trained
- How expensive it is to run them
- How low the latency can be for interactive applications
- How much electricity and cooling the industry will consume
For anyone building or using AI tools, the underlying silicon is becoming as important as the model weights themselves. The next few years will likely see continued NVIDIA dominance, growing use of specialised accelerators by the biggest labs, and experiments with even more exotic approaches (photonic, analog, neuromorphic).
The golden age of computer architecture that Hennessy and Patterson predicted is well underway. The chips that win will be the ones that best solve the memory wall while remaining practical to program and deploy at enormous scale.
