Key takeaway: A model too large for one GPU's memory splits across several in two fundamentally different ways. Tensor parallelism cuts each weight matrix into shards, so every GPU computes part of every layer in lockstep and must synchronize after every matrix multiply. Pipeline parallelism instead hands each GPU a whole block of layers, with no synchronization until activations cross a stage boundary. TP pays in network bandwidth, every layer. PP pays in idle GPU time, at every boundary. The two costs are why serving engines split model weights differently depending on what interconnect actually connects the GPUs.
The KV cache concurrency post assumed the model already fit on one GPU and asked how many users fit around it. This post asks the question one step upstream: what happens when the model itself doesn't fit on one GPU. The mixture-of-experts post called expert parallelism "a third parallelism axis, distinct from tensor and pipeline" without ever defining the first two. This is that definition.
What this post covers
- Why one GPU stops being enough, and the two orthogonal places to cut a transformer
- Tensor parallelism: splitting the math inside a layer, and the all-reduce tax it pays
- Pipeline parallelism: splitting whole layers, and the idle-time "bubble" it pays instead
- Why serving engines default to TP inside a node and PP across nodes
- The business "so what": what this decision costs, and what it doesn't fix
Splitting a model across GPUs, mechanistically
A GPU has a fixed VRAM budget: weights plus KV cache. Once a model's weights alone exceed that budget, no amount of batching or caching helps. The model has to be cut into pieces that live on different GPUs. There are two structurally different places to make that cut in a transformer:
- Across the math inside a layer: every attention and MLP block is a handful of large matrix multiplies. Split the matrices themselves across GPUs, and every GPU does a fraction of every operation.
- Across the sequence of layers: a transformer is a stack of identical blocks. Assign whole blocks to different GPUs, and each GPU does all of a smaller set of layers.
The first is tensor parallelism, formalized for transformers by Megatron-LM (Shoeybi et al., 2019, arXiv:1909.08053). The second is pipeline parallelism, formalized by GPipe (Huang et al., 2018, arXiv:1811.06965). Both were built for training; every major serving engine (vLLM, TensorRT-LLM, SGLang) adopted the same split for inference, exposed directly as --tensor-parallel-size and --pipeline-parallel-size flags.
Tensor parallelism: split the math, synchronize every layer
Megatron's trick is to split each weight matrix so the split is invisible to the math. An MLP block computes activation(X · W1) · W2. Split W1 by columns and W2 by rows across N GPUs, and each GPU can compute its shard of the intermediate result independently. The two matrix multiplies only produce the correct combined output if the partial results get summed back together first. That sum is an all-reduce: every GPU broadcasts its shard, every GPU adds them up.
Attention gets the same treatment, split by attention head instead of by matrix column. The result: two all-reduces in the forward pass per transformer layer, for every token, every request.
- That traffic has to move at NVLink speeds to stay invisible. Inside one node, GPUs share a switched fabric built for exactly this.
- Move that same all-reduce across a standard network link between nodes, and the synchronization cost usually dominates compute time: the GPUs spend more time waiting on the sum than computing.
So what: TP buys a bigger effective GPU (more VRAM, more FLOPs, seen as one unit) at the cost of communication that scales with shard count. That cost is why TP degree is capped by how many GPUs share one fast interconnect, not by how big the model is.
Pipeline parallelism: split the layers, pay in idle time
Pipeline parallelism sidesteps the all-reduce entirely. GPU 0 holds layers 1-10, GPU 1 holds layers 11-20, and so on. A token's activations flow through GPU 0's layers, then get handed to GPU 1 as a single tensor: one small transfer per stage boundary, not one all-reduce per matrix multiply.
The cost shows up differently. Run one request at a time through a naive pipeline and every stage but one sits idle while its neighbor works, a delay called the bubble. GPipe's fix was to split each batch into micro-batches, so stage 2 can be working on micro-batch 1 while stage 1 has already moved on to micro-batch 2, shrinking idle time to roughly (stages - 1) / microbatches of total run time.
Inference doesn't get micro-batches inside a single request: a token has to finish layer 10 before layer 11 can start. What fills the pipeline instead is concurrent request volume. With continuous batching keeping many requests in flight, different requests occupy different stages at the same instant, and the bubble closes the same way GPipe's did. Too little concurrent traffic, and PP GPUs sit idle waiting on a hand-off exactly like a training pipeline waiting on a micro-batch.
So what: PP's inter-GPU traffic is small and infrequent, one hand-off per boundary, not one all-reduce per operation, so it tolerates a slower link. That's the property that makes it the axis used to cross nodes.
Why the split lands where it does
Put the two costs side by side and the placement follows automatically: TP needs NVLink-speed synchronization every layer, so it's confined to GPUs sharing one fast fabric, typically one 8-GPU node. PP needs only a cheap hand-off at each boundary, so it's the axis that crosses nodes. NVIDIA's follow-up work on Megatron-LM (Narayanan et al., 2021, arXiv:2104.04473) formalizes this as 3D parallelism: tensor parallelism inside a node, pipeline parallelism across nodes, data parallelism across replicas of the whole stack. Expert parallelism, from the MoE post, layers on top as a fourth axis for sparse models: it shards which experts live where, not the matrices or the layer stack.
The business so-what
The buyer question this actually answers isn't "how many GPUs." It's which axis absorbs the split, and that answer is set by interconnect topology, not by the model card. Run tensor parallelism across a standard network link between nodes and the all-reduce tax can dominate latency regardless of how much compute you added. Get the axis right and the split is close to free; get it backwards and more GPUs make the response slower, not faster.
This is a compute-placement decision, full stop. It says nothing about whether the tokens moving through those shards are grounded in current, permissioned context. That's the layer beneath the model, and no parallelism strategy fixes a stale or ungoverned retrieval.
The bottom line
Tensor parallelism and pipeline parallelism cut a model in two different places and charge two different prices: TP pays in bandwidth, every layer, capped by how many GPUs share a fast fabric; PP pays in idle time, at every stage boundary, recovered only by keeping enough requests in flight. Most production deployments run both: TP inside a node, PP across nodes, matched to the interconnect they actually have, not the one they wish they had.
The correctness question sits one layer up from all of this: does the context reaching the model reflect the current, permissioned state of the world? That's what the Inherent Public API is built for: get started in the docs. Sizing a multi-GPU deployment and the topology math surprised you? DM Flow on X.
Next read: Mixture of Experts: Cheap to Compute, Expensive to Hold, the third parallelism axis and what it costs on top of TP and PP.
