← All writing
Journal

Why Every AI Lab Is Building Its Own GPU

10 min read

Across the AI industry, labs are building custom silicon. You'll see billions spent on the design of chips, and accelerators these days are not commodity products but specialized, custom devices designed for a specific AI workload. If you were to ask me why I'm somewhat surprised, given how well NVIDIA has played this game and how much they've built out their software ecosystem, I'd say because it represents a change in how we think about AI performance today.

The supremacy of raw arithmetic power as a metric is losing steam as something of a science fiction dream. In current AI workloads, data movement has become a more pressing limitation on performance than raw calculation units. Custom silicon allows ownership of the memory hierarchy, KV-cache policy, precision, scheduling, and interconnect to be controllably engineered. On compatible workloads, this offers large gains in both latency and throughput, as well as energy efficiency and scaling, but these gains are real but not always guaranteed. The returns are most favorable when your workloads are somewhat predictable and if you can own the entire stack. Otherwise, the costs of the ecosystem and integration can eat up any benefit.

Matching Hardware to Transformer Dataflow

Large language models are based on the transformer architecture. These are big and necessarily have a lot of moving parts, supporting full activations, weights, and attention state. Memory hierarchy and its orchestration are an accelerator's most important elements, as it's responsible for storing and answering questions about the weights and attention state. A bit more specifically, on a standard GPU it may be necessary to move the full KV cache from the high bandwidth memory (HBM) into on-chip SRAM for each token, making KV loading a significant performance bottleneck.

Custom chips can achieve success without requiring better raw compute performance by reducing data movement either by increasing locality, reuse, or by changing the dataflow. The aim is not to increase compute per second, but to do the required computations with fewer unnecessary steps.

Improving Throughput-per-Watt

Savings in energy efficiency is one of the most popular reasons to build custom hardware. This is where some of the largest reported gains appear: custom designs often say they can achieve many times greater throughput-per-watt than even top-tier GPUs, while still achieving accurate model results. In some cases, for training-specific execution, we're seeing improvements in energy efficiency of 2-5x over prior generation designs. Inference-specific designs can do even better. For large fleets, it's hard not to recognize that these efficiency improvements have real impact on both costs and environmental footprint.

Controlling Latency and Tail Behavior

Custom inference accelerators can not only help speed up the delivery of responses but also leverage their deterministic performance and data movement over GPUs to improve latency. Custom systems have been reported to reduce latency 2-6 times over GPU baselines by altering the transformer dataflow. The deterministic nature of execution plays a role in tail latency as well. Tail latency refers to worst-case response times, which are important for both user experience and service-level agreements. When every millisecond counts, predictable performance can be as good as average speed.

Owning the Scaling Path

As models grow into hundreds of billions or trillions of parameters, the bottleneck moves from the chip core itself to interconnect infrastructure. Interconnect, routing, and physical topology are critical for large language model efficiency and for mixture-of-experts efficiency. "Owning a GPU" increasingly means owning the scaling path. When you own the interconnect, you're able to optimize the whole cluster, not just individual nodes.

Increasing Fleet Utilization

Newer designs are combining training and serving functionality. We're seeing reports of custom inference accelerators that can repurpose idle cycles for work that resembles training while still meeting inference tail-latency requirements. A combined architecture for training and serving is emerging, which could reduce the need for separate systems and increase utilization.

Where Custom Chips Beat Commodity GPUs

The strongest direct evidence for special-purpose hardware is in inference scenarios. Custom accelerators are reported to be 15x to 30x faster than the GPU and CPU baseline and 30x to 80x more throughput per watt. The winning formula appears to be special-purpose accelerators that give up unnecessary generality in favor of known operator and dataflow patterns. But don't miss the forest for the trees. Operator compatibility and memory behavior can dominate real-world results, particularly in edge deployments.

Gains in training are achieved by co-designing precision, sparsity, and scheduling. Custom training accelerators are reported to produce marked gains in energy efficiency, preserving accuracy. Sparse attention approaches have demonstrated savings in training time, and in some cases improved convergence speed as well. We lack clear evidence of direct and standardized comparisons with the best available GPU training stacks, but on the engineering principle, the evidence is sound. No conclusive universal quantitative superiority is proven.

Custom silicon works best when you have some understanding of your workloads. The benefits of custom silicon are still very conditional. Some custom hardware systems only report gains over commodity hardware at large batch sizes, while others explicitly target the low-latency, single-batch regime. In production serving scenarios the results vary with request distribution, SLA constraints, and traffic variability. When the workload is predictable enough to maintain hardware assumptions, we can reasonably expect specialization to be worth it.

The Main Engineering Advantages

The KV-cache and attention locality are both critical concerns. Increasingly, the limiting factor for Transformer serving is the movement of the KV-cache. The KV-cache can be 8 gigabits for Mistral 7B with batch size of 1. And that movement is expensive. Custom chips can be engineered to implement smarter KV-cache policies. For example, they can establish policies to ensure data accessed more frequently for the prompt and response is closer to the compute. Various software approaches, such as FlashAttention, PagedAttention, and grouped-query attention, all address this problem. But hardware assistance can greatly amplify their efficacy.

The strength of custom silicon is in its ability to jointly optimize between compute and data movement. By fusing operations, pruning token relationships in hardware, and computing closer to memory, you reduce traffic at the source. Compute efficiency and data-placement policy are now tightly coupled. Which gets us to the winner: a chip that optimizes both.

Interconnect and Cluster Design as First-Class Benefits

At scale training, the unit of optimization is not a single chip but a cluster. Communication can be as high as 91% of training time in some cases. Optimizing interconnect fabrics can offer massive improvements. For example, a recent optical interconnect offered 1.5x greater bandwidth and 3.2x faster total training time over 1024 GPUs. Another interconnect provided extremely high efficiency for pipeline and data parallelism at this scale.

The topology and scheduling of communication are as important as bandwidth. Matching communication groups to the physical layout of interconnects increased production training by more than 10% on clusters of more than 9,600 GPUs. By jointly optimizing flow scheduling, topology, and parallelism, one research team achieved over double the training speed per iteration compared to prior methods.

Taken together, the trend is clear: AI laboratories build their own GPUs not only to optimize the chip itself but to optimize the communication infrastructure surrounding them. By optimizing communication, they are beginning to see efficiency as the real determinant of training throughput. The chip is the first point of ingress into the cluster.

Hardware-Software Co-Design: The Real Moat

The repeated instances of evidence demonstrate hardware-software co-design as necessary, not optional. Hardware-aware training, automated hardware generation, compiler support, and simulation are all essential to jointly optimize latency, energy, area, accuracy, and deployment constraints. Some research systems jointly optimize over neural-architecture and configurable-hardware configurations while using cycle-accurate latency estimation. Others jointly optimize approximate multipliers and architectures, reducing both edge and latency by 40% on edge tasks.

The pattern is clear: The chip is only the story. At the engineering layer, the real advantage of having your own chip is that you control the entire loop including model assumptions, precision, kernel fusion, scheduling, and physical architecture. A lab that owns all of these can iterate faster and find optimizations that a chip vendor selling to many customers cannot find or cannot expose.

Why Custom AI Chips Are Hard to Ship

The remaining gaps in tooling and portability are probably the greatest hurdle. Software ecosystems still remain a major bottleneck. Retargetable compilers for new platforms still require platform-specific code, hardcoded instruction set descriptions, optimization implementations, and debugger support. In research literature, you can see explicit descriptions of retargetable compilers are "far from perfect." Porting CUDA to other platforms lacks the trust-level guarantee needed for ripeness of reuse. A lab building their own chip gets architectural control, but they absorb most of the compiler and tooling burden that GPU vendors already absorbed themselves.

End-to-end integration is important. Gross speedup at the chip layer does not guarantee speedup at service layer. For example, in deploying to an FPGA, data processing actually takes up only 0.34% of the end-to-end time, with input transfer taking up 98.3%. If your preprocessing, transfers, postprocessing, storage, or networking is the bottleneck, a faster accelerator does little to make a difference. Metrics across the entire stack matter more than just speed of the neural core.

Software and tools for architecture-specific systems can achieve peak performance, but they are not usable when using standard pytorch workflows. More general tools can help portability but may not achieve consistent speedups. Code portability does not guarantee performance portability. As you port different compilers and backends, your speedup can be slowed down 3x or more in certain kernels. Your real decision is not simply build or buy. More commonly it is a tradeoff between peak efficiency and ecosystem leverage.

The Bottom Line

AI labs are building custom GPUs because the limiting factor has shifted from raw arithmetic to data movement and from single chips to clusters. Custom silicon brings concrete and quantifiable benefits for watt-per-throughput, latency, memory hierarchy, and interconnect optimizations. But it is not without cost: You absorb the tooling burden, integration risk, and lose the benefit of a well-established software ecosystem.

The labs that will be successful treat hardware and software as a single co-designed system, not as separate layers. The chip is necessary but not sufficient. The real moat is the ability to manage everything from the architecture of a model down to the physical topology with a faster iteration pace than competitors.