A fast processor can spend its time waiting
A GPU can perform enormous numbers of calculations. Yet a language model may answer slowly because the data needed for those calculations is too large or too far away. The processor needs model weights, temporary results, and information from the current request. If those cannot fit in fast memory or move through it quickly enough, more raw computing power does not solve the bottleneck.
Think of a busy kitchen. Extra cooks help only if ingredients arrive and there is room to prepare them. A model’s weights are part of the recipe; memory capacity is the available workspace; memory bandwidth is how quickly ingredients move. This analogy has limits, but it makes one point clear: computing and data movement are separate resources.
Weights have to live somewhere
A model uses learned numerical weights at every stage of processing. A larger model generally needs more memory to hold them, though the exact footprint depends on number format and how it is stored. If the weights cannot fit in a GPU’s high-bandwidth memory, a system can distribute them across devices or move some data to slower memory. Either choice adds coordination or transfer costs.
Training needs still more working space. Besides weights, a training system retains intermediate activations, gradients, optimizer state, and batches of examples. Inference—the act of using a trained model—avoids some of those training costs but still needs to keep the model accessible while serving requests.
Capacity and bandwidth should not be confused. Capacity answers “How much can we keep near the processor?” Bandwidth answers “How much can we move per second?” A model can fit into memory and still run slower than expected if repeatedly reading its weights or other data uses the available bandwidth. Conversely, enormous bandwidth does not help if the model and active requests simply will not fit.
The conversation creates its own memory demand
Many text systems generate a response one token at a time. A transformer repeatedly compares the current position with information from earlier tokens. Without a shortcut, it would redo much of the same work as each token is added. A key-value cache stores intermediate attention information from previous positions so later generation can reuse it.
That cache is attached to a particular request. Give the model a longer document or conversation and it has more previous positions to keep available. Serve many users at once and each active request may bring a separate cache. The details vary by architecture, but the operational result is familiar: a long context can cost far more memory than a short question, and concurrency can fill the available space quickly.
Research on PagedAttention addresses this practical problem by organizing cache storage in blocks so an inference server wastes less memory on allocation and can serve more requests. The improvement is about efficient use of scarce memory; it does not make every prompt free or every long context equally fast.
When moving data dominates the math
An accelerator may perform a calculation quickly once its inputs arrive. The slower part can be fetching weights, moving attention data, or communicating between devices. FlashAttention research focuses on reducing expensive reads and writes between memory levels during attention. That is why an algorithm that performs a similar mathematical operation can still be substantially faster in practice: it changes where and how often data moves.
Putting a model across several GPUs introduces another boundary. The devices must exchange information during certain operations. Adding a device can increase total capacity and compute, but the benefit depends on how the model is partitioned and how fast the devices communicate. A slow interconnect can turn a promising scale-up into a waiting exercise.
This also explains why a faster chip specification does not translate directly into a faster assistant. Real latency depends on model size, request length, number of simultaneous users, batching, cache organization, and hardware layout. A short answer for one person and a long agent session involving tools impose different loads.
Why batching and context change the bottleneck
Batching can serve several requests in one pass, improving throughput while sometimes adding waiting time for an individual user. The provider must balance how many active conversations fit in memory against response latency. A short-prompt benchmark does not capture that operating choice.
Weight quantization and more efficient attention-memory management can reduce particular demands, with quality, engineering, or compatibility tradeoffs. To compare them with another GPU, first identify the measured limit: capacity, bandwidth, arithmetic, communication, or queueing. A change that addresses one limit can leave the others untouched.
The sensible performance question
When someone says a system “needs more GPUs,” ask what actually ran out. Was it compute, memory capacity, memory bandwidth, or communication between devices? Did requests become slower only when many users arrived, or only when conversations grew long? These clues point to different solutions.
Some improvements come from hardware. Others come from smaller numerical formats, better memory allocation, more careful batching, or attention algorithms that avoid unnecessary data transfers. Every choice can change speed, cost, and sometimes output quality. More processors can help, but only after the system can keep them fed.
