Appendix A: VRAM arithmetic tables

Every serving and training decision in this book is an accounting problem against one number: the 16 GiB of GDDR7 on the RTX 5080 (the baseline machine). This appendix precomputes that accounting for every model and config the book uses, so that when a chapter says "the weights are $W$ GiB" or "this leaves $M_{kv}$ for cache" the number is already sitting here with its formula above it. Everything is regenerable: each table is preceded by the equation that produced it, so when a model revision or a dtype changes you re-run the arithmetic rather than trusting a stale figure.

Two conventions hold throughout, both inherited from Tensors, autograd, and number formats and KV cache arithmetic:

• GiB vs GB. Memory capacity is binary: $1\text{GiB}=2^{30}\text{byte}$. Vendor throughput and bandwidth are decimal: $1\text{GB}=10^9\text{byte}$. The card is 16 GiB of capacity; its bandwidth is ~960 GB/s. I keep the units explicit because mixing them is a ~7% error that hides exactly at the margin where things stop fitting.
• Measured vs derived. Every number here is derived arithmetic unless it carries the tag (measured on the baseline machine — record value, date, driver). Derived numbers are exact given their inputs; the inputs that must be confirmed on-device (activation overhead, allocator slack, real weight-file sizes) are flagged at each table.

Bytes per parameter, by dtype

The atom of every weight budget is numel × bytes_per_element, from Tensors, autograd, and number formats. For the quantized formats the effective figure includes the per-group scales, so it is not a clean power of two.

$$W_{\text{weights}}=N_{\text{params}}\times {\beta }_{\text{dtype}}\text{byte}$$

Weight budgets for the serving repertoire

$$W=N_{\text{params}}\times {\beta }_{\text{dtype}},W_{\text{GiB}}=\frac{W}{2^{30}}$$

Parameter counts are the published totals; verify against each model's config.json and safetensors index, since revisions move them.

*The ~10.4 GiB gpt-oss figure is the naive MXFP4-only estimate (all params at 0.53 byte). The shipped checkpoint is ~12-13 GiB because attention, embeddings, and the router are kept at higher precision; only the expert MLPs are MXFP4.

Per-token KV-cache bytes

From KV cache arithmetic, one token caches a K and a V vector per layer, each of width $H_{kv}\times d_h$:

$$\boxed{B_{\text{tok}}=2\times L\times H_{kv}\times d_h\times b}$$

with $L$ layers, $H_{kv}$ KV heads (GQA makes this smaller than the attention-head count), $d_h$ head dim, $b$ bytes per cached element (2 for BF16, 1 for FP8), and the leading 2 for keys and values. Pull $L$, $H_{kv}$, $d_h$ from config.json (num_hidden_layers, num_key_value_heads, head_dim).

Worked, BF16: Qwen3-8B $=2\times 36\times 8\times 128\times 2=147,456\text{byte}=144\text{KiB}$; Qwen3-14B $=2\times 40\times 8\times 128\times 2=163,840=160\text{KiB}$; gpt-oss $=2\times 24\times 8\times 64\times 2=49,152=48\text{KiB}$.

Total KV at representative context and concurrency

$$M_{\text{KV}}=N_{\text{seq}}\times S_{\text{ctx}}\times B_{\text{tok}},N_{\text{seq}}^{\max }=\lfloor \frac{M_{kv}}{S_{\text{ctx}}\times B_{\text{tok}}}\rfloor$$

Context and concurrency trade off on a hyperbola: double the promised context and you halve the seats. Per-sequence KV (one sequence at the given context):

Embedding + generation co-residency for RAG

The RAG chapter (8.1) adds a second model to the card: a local embedding model that turns chunks and queries into vectors, alongside the generation model that answers. This is a co-residency budget, and it is cheaper than it looks because an embedding model runs one forward pass and reads out a pooled vector; it never autoregressively decodes, so it grows no KV cache. Its whole resident cost is weights plus one prefill's transient activation (chapter 8.1 equation 1.6), with none of the linear-in-tokens KV growth that dominates a generation budget. That is why a small embedding model is a cheap lodger and not a second full tenant.

$$\underset{\text{generation}}{\underbrace{P_g{b}_g+M_{\text{KV}}^{(g)}}}+\underset{\text{embed weights}}{\underbrace{P_e{b}_e}}+M_{\text{ctx}}\le C,C\approx 15.5\text{GiB usable}$$

The generation side is the book's Qwen3-8B FP8: $P_g{b}_g\approx 7.7$ GiB of weights, a KV pool of ~3.5 GiB at --max-model-len 8192, and ~0.6 GiB of CUDA context, so it occupies ~11.8 GiB and leaves ~3.7 GiB for the embedding lodger. Every embedding model in the repertoire clears that headroom, so the choice is quality-per-byte, not fit (pick it with a small recall@k / MRR bake-off, per 8.1).

Decode-throughput ceilings (bandwidth budget)

Not VRAM but the other side of the same coin. From Prefill, decode, and the roofline, decode at batch 1 is memory-bound, so tokens per second cannot exceed bandwidth divided by bytes read per token:

$${\text{tok/s}}_{\max }=\frac{BW}{B_{\text{read}}},BW=9.6\times 10^{11}\text{byte/s},B_{\text{read}}\approx W_{\text{weights}}(\text{short ctx})$$

The gpt-oss $B_{\text{read}}$ is higher than the active-expert MLP bytes alone ($3.6\times 10^9\times 0.53\approx 1.9\times 10^9$): it adds the non-MXFP4 bytes read every token (attention, embeddings, and the router, all kept in higher precision) on top of the active-expert bytes, which is where the extra $\approx 0.6\times 10^9$ comes from.

These are ceilings, not predictions; the real measured tok/s sits under them by the kernel-overhead + sampling + KV-read gap the roofline lab plots. Record measured values with date and driver.

QLoRA training budget for a 4B policy

The training budgets are where the memory chapter's "4-16x inference" claim gets paid out. From LoRA and QLoRA, mathematically: the base weights are frozen in NF4 (no grads, no optimizer state); only the low-rank adapters are trainable, so optimizer state is tiny. The four accounts are base + adapters + optimizer + activations.

Base (frozen, NF4 + double-quant). $N_{\text{base}}\times {\beta }_{\text{NF4}}$ with ${\beta }_{\text{NF4}}\approx 0.516$: $$4.0\times 10^9\times 0.516=2.06\times 10^9\text{byte}=1.92\text{GiB}$$

Adapter trainable params. For LoRA rank $r$ on a linear of shape $d_{\text{out}}\times d_{\text{in}}$, the update $BA$ adds $r(d_{\text{in}}+d_{\text{out}})$ params. Summed over target modules $\{$q,k,v,o,gate,up,down$\}$ and $L=36$ layers of Qwen3-4B ($d_{\text{model}}=2560$, q width $4096$, kv width $1024$, MLP width $9728$) at $r=16$:

$$N_{\text{LoRA}}\approx 36\times 16\times [\underset{q}{\underbrace{(2560+4096)}}+\underset{k,v}{\underbrace{2(2560+1024)}}+\underset{o}{\underbrace{(4096+2560)}}+\underset{\text{MLP}}{\underbrace{3(2560+9728)}}]\approx 33\times 10^6$$

Adapter, grad, optimizer. Adapters in BF16, grads in BF16, AdamW keeps two FP32 moments $m,v$ plus an FP32 master copy (from Where memory goes: training vs inference, AdamW = 2 extra copies):

GRPO training budget on 16 GiB

GRPO (from GRPO and GRPO on 16GB) adds a generation phase to the QLoRA budget: for each prompt it samples a group of $G$ completions, scores them, and takes a group-relative advantage. So on top of the QLoRA static footprint you pay for (a) the KV cache of the generation rollouts and (b) a reference-policy forward for the KL term. With Unsloth the generation runs on the same 4-bit base, so there is no second copy of weights; the extra cost is KV and activations.

Generation KV. $G$ completions per prompt, each up to $S_{\text{gen}}$ tokens plus a prompt of $S_{\text{prompt}}$, at the 4B's $B_{\text{tok}}=144\text{KiB}$. I use the actual GRPO on 16GB config ($S_{\text{prompt}}=256$, $S_{\text{gen}}=768$, so $T=1024$) so this line matches that chapter's budget:

$$M_{\text{KV,gen}}=G\times (S_{\text{prompt}}+S_{\text{gen}})\times B_{\text{tok}}$$

For $G=8$, $S_{\text{prompt}}=256$, $S_{\text{gen}}=768$ ($T=1024$): $$8\times 1024\times 147,456=1.21\times 10^9\text{byte}=1.13\text{GiB}$$

This is the live KV floor; vLLM claims a larger slab (~2.5 GiB) sized by gpu_memory_utilization and pages completions into it. The table below counts the slab, and the 8-bit AdamW optimizer (Unsloth's default), so its total reconciles with GRPO on 16GB's ~6.9 GiB rather than the FP32-AdamW QLoRA table above.

Regenerating these tables

Every table above is parameters × formula. The one-file calculator that produces them (weights, KV, decode ceiling, QLoRA/GRPO budgets) lives with the KV lab (predict.py in KV cache arithmetic) and the roofline lab (ceiling.py in Prefill, decode, and the roofline). When a driver update, a vLLM release, or a model revision changes an input, re-run those scripts rather than editing a number here by hand, and re-stamp any measured overhead with its new date and driver.