Napkin math for tensor parallelism

If I hand you a model that is bigger than any single GPU you own, how do you decide what deployment shape will actually work? Which context length can you realistically serve? How many concurrent users can the memory hold? Which knob do you turn to trade one for the other? These sound like scheduling questions. They aren’t. Every one of them resolves to a memory calculation, and every calculation resolves to a handful of numbers you can read out of the model’s config file.

I got there the hard way. My first attempts at running Llama-3.3-70B on two A100 80 GB cards produced a pileup of errors, each one I “fixed” by tweaking a YAML field, each time convinced I was done. world size (2) is larger than the number of available GPUs (1). Then RuntimeError: Background writer channel closed, which turned out to mean “disk full.” Then a pod that got OOMKilled mid-load. Then finally the one that stopped me: 20.0 GiB KV cache is needed, which is larger than the available KV cache memory (5.63 GiB). Each error I patched. What I was actually doing was rediscovering, badly, the arithmetic that would have told me every one of those values up front.

So I stopped patching, opened the model’s config.json and the GPU’s spec sheet, and did the math on paper. Every value comes from those two documents. I wrote it up for one specific case, this model and this hardware, and built a calculator so you can do the same for yours. Then I instrumented the deployment so the dashboards could check the math against what actually ran.

Why TP=2 exists

Llama-3.3-70B has 70,553,706,496 parameters, per the model’s own metadata on Hugging Face. At bf16 that means 2 bytes per parameter, so the weight files add up to about 141 GB.

An A100 80GB has 80 GB of VRAM in vendor marketing units. In binary units, which is what CUDA actually reports and vLLM actually allocates against, that is closer to 74.5 GiB. Either way, 141 does not fit in 80. The deployment fails at load time.

Tensor parallelism is the fix. Not an optimization. For a model whose weights exceed a single GPU’s VRAM, TP is the answer on a single-node setup like mine. (Multi-node deployments would reach for pipeline parallelism as well, but that’s a story for another time.) --tensor-parallel-size 2 tells vLLM to shard the weight tensors across two GPUs. Every layer’s weight matrices get split down the middle, so each GPU holds half of all 80 layers rather than all of half of them. That is 141 / 2 = ~70.5 GB of weights per GPU (65.7 GiB in binary units, which is what CUDA reports). At ~88% of the A100’s usable VRAM, it fits. Barely.

Making this actually work required the Kubernetes side and the vLLM side to agree on the number 2, in three different places. This is the part that took a couple of tries. The finished deployment lives in gpu-vllm-tp-single-node .

The pod’s device request. nvidia.com/gpu: 2 in the deployment spec tells the kubelet to hand the pod two GPU device handles. If this stays at 1 (the default I copied from gpu-vllm/, my previous single-GPU experiment), vLLM sees exactly one GPU and crashes immediately with:

world size (2) is larger than the number of available GPUs (1)

The scheduler itself doesn’t know or care. Kubernetes gave the pod exactly what it asked for, and the pod model guarantees one thing. Both GPUs are on the same node, because a pod is only ever scheduled to one. Everything past that, nvidia.com/gpu: 2 can’t tell me. What’s the interconnect between them? Are they in the same NUMA domain, attached to the same group of CPU cores and their local memory, or does traffic between them have to cross domains? None of it is expressible in the request. All of it matters for whether TP will actually work well, once it’s running.

For both of those, nvidia-smi topo -m inside the running pod is the honest answer. On my hardware (Standard_NC48ads_A100_v4, two A100 80 GB cards on one VM), it reports:

     GPU0  GPU1  CPU Affinity  NUMA Affinity  GPU NUMA ID
GPU0  X    NV12  0-23          0              N/A
GPU1  NV12 X     24-47         1              N/A

NV12 is 12 bonded NVLinks between the two GPUs. NVLink is what makes TP=2 fast on this hardware. If the same table showed PHB, PIX, or SYS instead (PCIe host bridge, PCIe switch, or across NUMA), TP would still function but the all-reduce that runs after every layer would traverse a much slower link. Throughput would fall off a cliff. Note that the two cards sit in different NUMA domains, 0 and 1, with disjoint CPU affinity. That would matter a great deal if the all-reduce had to travel through the host. With NVLink between the cards, it doesn’t.

The reason I flag this at all is the Azure product page for the SKU describes it as an “A100 PCIe GPU.” That’s technically the form factor of the individual cards, but reading it, you would reasonably conclude that the two GPUs on this VM only talk over PCIe. They don’t. NVLink is there. The spec page is not lying, it’s just describing a different thing than the interconnect. This is the moment where reading the vendor page is not enough. Always verify with the hardware probe.

TP=2 gets the model to fit. What it doesn’t fix is everything downstream. I’d spent about 65 GiB of every GPU on weights alone. Every subsequent question about this deployment (what context length is realistic, how many concurrent requests can I run, whether a given request will get accepted) is now bounded by what’s left over on each GPU. Which is not much.

The 5.63 GiB budget decides everything

The first time I tried to start the pod after getting TP=2 to load correctly, vLLM refused. Not a crash. A refusal.

ValueError: To serve at least one request with the model's max seq len (131072),
20.0 GiB KV cache is needed, which is larger than the available KV cache memory (5.63 GiB).
Based on the available memory, the estimated maximum model length is 36896.

I read it twice. vLLM had done the sizing math before starting, and refused to lie about a config that couldn’t hold a single 128k-token request. Where did those numbers come from? All of them are in the model’s config.json and the hardware’s spec sheet.

Where 20.0 GiB comes from. The KV cache stores two vectors, K and V, per attention head, per layer, per token. For Llama-3.3-70B, config.json says:

  • num_hidden_layers: 80
  • num_key_value_heads: 8
  • head_dim: 128
  • torch_dtype: bfloat16 (2 bytes per element)

That last one, num_key_value_heads: 8, deserves a pause. This is grouped-query attention (GQA). The model has 64 attention heads for queries, but only 8 heads worth of K and V that all 64 queries read from. Without GQA, at 64 heads, the KV cache would be 8× larger per token. The whole “70B on two A100s” story only exists because GQA cut the cache down to something serviceable. GQA is not just an efficiency knob. It’s an enabler.

Under TP=2, those 8 KV heads get split. vLLM’s rule for how many KV heads land on each GPU is max(1, num_key_value_heads // tensor_parallel_size). So each GPU stores 4 KV heads. That’s the /TP in the formula below.

Per token, per GPU, at TP=2:

layers × 2 (K and V) × (kv_heads / TP) × head_dim × bytes_per_element
= 80 × 2 × (8 / 2) × 128 × 2 bytes
= 163,840 bytes
= 160 KiB per token per GPU

Now scale to a 128k request. That length isn’t a number I picked. config.json sets max_position_embeddings to 131,072, which is what 128k context means, and it’s what vLLM assumes it has to serve unless I say otherwise:

131,072 tokens × 160 KiB per token = 20 GiB

That’s vLLM’s 20.0 GiB, and it comes from four numbers in config.json for the per-token cost, one division by TP, and the context length the model ships with. No source code needed. Just arithmetic.

Where 5.63 GiB comes from. VRAM budget per GPU, straight from vLLM’s own startup log:

  • Total advertised: 80 GB decimal ≈ 74.5 GiB binary
  • Weights sharded to this GPU: 65.74 GiB (vLLM’s Model loading took 65.74 GiB)
  • CUDA graph capture: 1.25 GiB (Graph capturing finished, took 1.25 GiB)
  • Kernel workspace, framework state: another few hundred MiB
  • What’s left for KV: what vLLM measured at 5.63 GiB

Add those up. 65.74 + 1.25 + 5.63 = 72.62 GiB out of 74.5 GiB available. About 97% of physical VRAM in use. --gpu-memory-utilization defaults to 0.9, meaning vLLM will manage up to 90% of VRAM as its budget. But 90% is a ceiling on the pool vLLM controls, not a target. When the weights alone need 88% of usable VRAM, vLLM measures actual free memory during profiling and packs the KV cache into whatever slack remains, above its nominal cap. That is what puts real usage at ~97%.

That gap between “80 GB” and “74.5 GiB” bothered me the first time I saw it. GPU vendors advertise memory in decimal GB (10⁹ bytes). CUDA reports and vLLM allocates in GiB (2³⁰ bytes). 80 × 10⁹ divided by 2³⁰ is about 74.5, not 80. That’s a real ~7% you don’t get to spend, before you’ve done anything. Every calculation from here on is in GiB, because that’s what the deployment actually sees.

Why max_model_len = 32768? vLLM’s error said 36,896 tokens would fit for one request at max length. That is a limit for a single request. It leaves zero headroom for a second concurrent request at similar length. At more realistic average lengths (say 4k input plus 4k output = 8k), the same 5.63 GiB per-GPU budget holds four or five concurrent requests, which is a much more useful shape. I picked --max-model-len 32768 for the deployment, comfortable ceiling for any single request, enough headroom for a small batch, and no lying to callers about a 128k context I couldn’t actually serve.

Three misreads I owned before I understood any of this:

  • I thought “128k context” was a property of the model. It’s the length the model’s positional encodings were trained to handle. Whether you can actually serve at 128k depends on your GPU budget. On 2 × A100 80 GB with a 70B model at bf16, the answer is no, and not close. Your real ceiling is around 36k, and comfortable operation is closer to 32k. The model card and the deployable ceiling are two different numbers.
  • I thought GQA was a training-time detail. It’s the reason 70B is serviceable on this hardware at all. Multi-head attention with 64 KV heads would blow the KV budget by 8×. GQA is what turns “impossible” into “tight but fine.”
  • I thought more concurrent requests meant proportionally more VRAM. Approximately yes, until prefill dominates decode. At high concurrency, prefill batches consume activation memory that isn’t captured by the per-token KV formula. The clean per-token model breaks down at the top of the throughput curve, right where the calculator I built has a “tight” band before it turns red.

Which brings me to the calculator. Every number above is a formula and four inputs. That means you can play with it. Change the dtype from bf16 to fp8 and watch the weight budget halve, the KV budget balloon, and the concurrency at max length triple. Push num_key_value_heads up and watch the KV cost double per step. The point isn’t to check whether a specific model fits on a specific GPU. The point is to feel which lever moves what.

For a full-screen version, open it in a new tab.

Prove the math on the dashboard

The arithmetic makes predictions. Weights split evenly across two GPUs. KV cache fills as a working sum of active requests. num_requests_running settles at a level the KV budget allows. Prediction is cheap. What makes it useful is the loop that closes when the running deployment agrees.

The observability stack from the previous experiment gives you two lenses. The DCGM Grafana dashboard is the hardware view, with per-GPU utilization, memory used, power, and temperature. The vLLM project’s Grafana dashboard is the serving view, with vllm:num_requests_running, vllm:num_requests_waiting, vllm:kv_cache_usage_perc, plus TTFT and inter-token latency histograms and a panel that splits per-request time into prefill and decode. DCGM answers “is the hardware working?” vLLM answers “is the serving stack working?” You need both.

One small platform-engineering decision that pays off here. The vLLM ServiceMonitor uses namespaceSelector.any: true, matching any Service in any namespace that carries metadata.labels.app: vllm-server. That means gpu-vllm/, gpu-vllm-tp-single-node/, and any future gpu-vllm-*/ experiment I run gets scraped automatically without touching the observability layer. Documented so a reader building on top can see the intent, the observability stack is not for one deployment, it is a substrate for however many experiments come next.

To exercise the dashboards, I ran a load test through vllm bench serve :

vllm bench serve \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --base-url http://localhost:8000 \
  --dataset-name random \
  --random-input-len 256 --random-output-len 128 \
  --num-prompts 500 --request-rate 5

500 requests at 5 requests per second, 256-token prompts, 128-token responses. All 500 came back successfully in 112 seconds, though the server delivered 4.47 requests per second rather than the 5 I asked for. Three things to watch during the run.

Both GPUs move together. On the DCGM panel, GPU 0 and GPU 1 utilization climb in lockstep. If only one of them climbs, TP silently failed and one GPU is doing all the work. Both climbing means the weight shards are on both cards and every layer’s forward pass is actually parallel. This is the empirical proof that TP=2 is real, not just requested. It is also the check I would want in any production deployment, a single alert on “GPU 0 util - GPU 1 util > threshold” catches TP regressions before a human notices latency.

Scheduler state climbs and stabilizes. vllm:num_requests_running rises off the floor and settles at a level determined by decode speed and request rate. Here it sat around 110 and peaked at 116. vllm:num_requests_waiting stayed near zero the whole time, never above 6. A short queue usually means a server with headroom. This one didn’t have any, and the reason is on the next panel.

Cache utilization tells you when you have run out. vllm:kv_cache_usage_perc climbs as requests accumulate KV, and at 5 requests per second it climbed to 99.7% and stayed there. That is not the comfortable plateau I went looking for. The budget is 36,896 tokens. Each request arrives holding 256 prompt tokens and grows to 384 as it decodes, so about 96 of them fit at full length. The scheduler admitted 116. They fit on admission and stopped fitting as they grew.

What happens then is not a crash. vLLM preempts. It evicts a running request’s KV and recomputes it later when space frees up, which costs latency but never correctness.

Finding out it happened at all took work. The logs said nothing. The vLLM dashboard has twelve panels and none of them plot preemption. The counter exists, vLLM exports vllm:num_preemptions_total and Prometheus scrapes it, but nothing draws it. I only found the number by querying Prometheus by hand.

vllm:num_preemptions_total

136 preemptions across 500 requests. Every request succeeded and the queue never grew, so from the outside the deployment looked healthy. A dashboard is only as good as the questions someone thought to put on it.

Once several hundred requests have gone through, the histograms have enough samples to be interesting. TTFT separates hard, from 0.82s at p50 to 3.96s at p90 and 4.64s at p95. That spread is where preemption shows up. A request whose KV was evicted pays for the recompute before its first token. ITL shows the same tail, 98ms at p50 against 637ms at p99. And the prefill-vs-decode panel is not subtle. Across the run, 557 seconds went to prefill and 20,602 seconds to decode. Decode is 97% of all inference time, 37 times prefill. This workload is decode-bound at 128 output tokens, which is what the math would predict from a 70B model whose per-step cost is dominated by weight fetches from VRAM.

Before I had dashboards, when a request felt slow I would look at the vLLM logs and guess. With dashboards, guessing is not the move. Was the pod queueing? Look at Scheduler State. Was the GPU pinned? Look at DCGM. Was the KV cache filling toward the ceiling? Look at Cache Utilization. And when the panel you need doesn’t exist, the metric usually does. Guess less, measure more. This is the reflex you build after years of platform work, and it applies to inference the same way it applies to any service you own.

The loop is the point

Weights forced TP=2. TP=2 set the KV budget at ~5.63 GiB per GPU. That budget set max_model_len at 32k and set the concurrency ceiling at whatever the sum of active-request KV footprints could stack up to. Everything downstream of nvidia.com/gpu: 2 fell out of arithmetic on a handful of numbers in config.json and one spec sheet. The calculator lets you replay that arithmetic on the same hardware for any model you want to reason about.

Every Kubernetes field that took a scalar was hiding capacity information the scalar couldn’t say. The calculator gives that information back. The dashboards prove the calculator was right.

There are three things I deliberately left alone, because they each deserve their own story. Pipeline parallelism, which trades cross-node bandwidth for the ability to fit a model across separate hosts, not just separate GPUs on the same host. fp8 and other quantization, which the calculator’s dtype knob lets you explore in one click but which has its own arithmetic beyond the KV formula. Mixture-of-experts models, whose memory profile is dominated by activation not weight, and which need a different calculator entirely. Each of those is a different arc.

Further reading