Appendix F: Notation reference
The symbols the book's derivations use, grouped by domain and pinned to one meaning each. Where a symbol is genuinely overloaded across domains (the classic offender is , which the book uses for the bootstrap-resample count, the LoRA up-projection matrix, and an NF4 quantization block), the collision is flagged so a reader jumping between chapters is never guessing. Each table gives the symbol, its meaning as the book uses it, and the chapter where it is first introduced.
The book leans on a few standing conventions: vectors and matrices are as written in each chapter (no global bold/roman rule is imposed, because the source chapters set their own); is natural log throughout; expectations are over whatever the subscript names; and "byte" quantities use binary prefixes for capacity (GiB) and decimal for rate (GB/s), as in The hardware baseline and Appendix A.
Reinforcement learning
| Symbol | Meaning | First introduced |
|---|---|---|
| , | state (here, the prompt / partial generation at step ) | The RL problem |
| , | action (here, the next token or a full completion) | The RL problem |
| policy: the model's probability of action in state , parameters | The RL problem | |
| policy (model) parameters | The policy gradient theorem | |
| frozen reference policy for the KL penalty | GRPO | |
| , | reward; in RLVR, the verifiable checker's score | The RL problem / RLVR |
| , | return: (discounted) sum of future rewards from step | Value functions and Bellman equations |
| discount factor, | Value functions and Bellman equations | |
| , | state-value: expected return from state under | Value functions and Bellman equations |
| , | action-value: expected return taking in , then | Value functions and Bellman equations |
| advantage, | Actor-critic and GAE | |
| estimated advantage (GAE, or group-relative in GRPO) | Actor-critic and GAE | |
| GAE bias-variance interpolation parameter | Actor-critic and GAE | |
| temporal-difference residual, | Actor-critic and GAE | |
| probability ratio in PPO-clip | Trust regions: from TRPO to PPO | |
| PPO clip half-width, ratio clipped to | Trust regions: from TRPO to PPO | |
| KL-penalty coefficient in the RL objective | GRPO | |
| KL divergence from policy to reference | GRPO | |
| GRPO group size: completions sampled per prompt | GRPO | |
| GRPO group index, | GRPO | |
| the -th completion (output) in a group | GRPO | |
| the RL objective (expected return) being maximized | The policy gradient theorem |
is the PPO clip width here, but also names machine epsilon in Tensors, autograd, and number formats and appears as a small constant in RMSNorm. Context disambiguates: an RL objective versus a floating-point rounding bound. Likewise is the KL coefficient in RL but bytes-per-parameter in the quantization/VRAM tables (Appendix A); the RL chapters never mix the two.
Probability and statistics
| Symbol | Meaning | First introduced |
|---|---|---|
| , | expectation, over the distribution named in the subscript | The policy gradient theorem |
| variance | REINFORCE and variance reduction | |
| covariance | REINFORCE and variance reduction | |
| model's probability of | The language-modeling objective | |
| , | an estimator of a mean / parameter (hat = estimated from data) | The statistics of evals |
| eval sample size (number of problems / items) | The statistics of evals | |
| in pass@k, the number of sampled completions per problem | Metrics and their math | |
| number of bootstrap resamples | The statistics of evals | |
| 95% confidence interval (percentile bootstrap by default) | The statistics of evals | |
| significance level (e.g. 0.05) | The statistics of evals | |
| effect size (standardized mean difference, Cohen's ) | The statistics of evals | |
| , | standard deviation and its estimate | The statistics of evals |
| entropy of distribution | The language-modeling objective | |
| cross-entropy of relative to | The language-modeling objective |
is doubly booked and worth watching: it is the statistical effect size here, the head dimension / model dimension in the transformer tables below, and appears as a plain dimension in the roofline matmul argument. The subscript or the sentence resolves it; when a chapter needs both at once it writes for the head dimension explicitly.
Transformer / LLM dimensions
| Symbol | Meaning | First introduced |
|---|---|---|
| model (hidden) width | The transformer block | |
number of transformer layers (num_hidden_layers) | Where memory goes / KV cache arithmetic | |
| , | number of attention (query) heads | Attention from first principles |
number of key/value heads (GQA; num_key_value_heads) | KV cache arithmetic | |
head dimension (head_dim, or ) | Attention from first principles | |
| vocabulary size (context: transformer, not RL value) | Tokenization and embeddings | |
| , | sequence / context length in tokens | Prefill, decode, and the roofline / KV cache arithmetic |
| token position index | The language-modeling objective | |
| the token (or activation) at position | Tensors, autograd, and number formats | |
| , | logits (pre-softmax scores over the vocabulary) | The language-modeling objective |
| query, key, value matrices in attention | Attention from first principles | |
| Jacobian of layer (autograd) | Tensors, autograd, and number formats | |
| adjoint of activation , | Tensors, autograd, and number formats | |
| total parameter count of a model | Appendix A |
is overloaded three ways across the book: the RL state-value function , the attention value matrix , and the vocabulary size . These live in different parts (RL, attention, tokenization) and never appear in the same equation, but it is the collision most likely to trip a reader skimming across parts. is layers in the transformer tables and the scalar loss in the autograd derivation; again, disjoint contexts.
Inference, bandwidth, and the roofline
| Symbol | Meaning | First introduced |
|---|---|---|
| KV-cache bytes per token, | KV cache arithmetic | |
| bytes read from memory per generated token (decode) | Prefill, decode, and the roofline | |
| bytes per cached element (2 BF16, 1 FP8) | KV cache arithmetic | |
| VRAM pool available for the KV cache | KV cache arithmetic | |
| number of concurrent sequences | KV cache arithmetic | |
| , | weight footprint in bytes | Appendix A |
| arithmetic intensity, FLOPs per byte moved | Prefill, decode, and the roofline | |
| ridge-point intensity, | Prefill, decode, and the roofline | |
| memory bandwidth (~960 GB/s on the baseline card) | Prefill, decode, and the roofline | |
| peak compute throughput (FLOP/s) | Prefill, decode, and the roofline | |
| Prefill, decode, and the roofline | ||
| decode throughput ceiling, | Prefill, decode, and the roofline |
(lowercase) is bytes-per-cached-element in the KV formula; the quantization table below deliberately writes bit-width as , not , to keep the two apart. (uppercase) is triply booked, the bootstrap-resample count in the statistics table, the LoRA up-projection matrix, and an NF4 quantization block (Parameter-efficient fine-tuning). The KV chapter writes for element bytes and always states "2 for BF16, 1 for FP8" inline so the meaning travels with the symbol.
Number formats and quantization
| Symbol | Meaning | First introduced |
|---|---|---|
| sign bit (floating point); also the quantization scale (context) | Tensors, autograd, and number formats | |
| stored (biased) exponent field | Tensors, autograd, and number formats | |
| exponent bias (127 FP32/BF16, 15 FP16, ...) | Tensors, autograd, and number formats | |
| mantissa bit-count | Tensors, autograd, and number formats | |
| fractional part of the significand, | Tensors, autograd, and number formats | |
| unit roundoff, | Tensors, autograd, and number formats | |
| machine epsilon, | Tensors, autograd, and number formats | |
| spacing between representable values (1 ulp), | Tensors, autograd, and number formats | |
| stored integer code in integer quantization | Tensors, autograd, and number formats | |
| quantization scale, real | Tensors, autograd, and number formats | |
| zero-point (integer offset in asymmetric quantization) | Tensors, autograd, and number formats | |
| group/block size for group-wise scales (e.g. 128 AWQ, 32 MXFP4, 64 NF4); chapter 6.3 writes the NF4 block size as | Quantization: theory and formats | |
| bit-width of the quantized element (4, 8, 16) | Quantization: theory and formats | |
| effective bytes per parameter for a dtype (VRAM tables) | Appendix A |
This table is the reference the Quantization: theory and formats derivations and the Appendix A budgets both point back to. When those chapters write for a quantization scale, it is the same shown here, not the floating-point sign bit; the surrounding equation always makes clear which. The effective-bytes symbol is what turns a bit-width into the byte-per-parameter figures the VRAM tables multiply by .
Parameter-efficient fine-tuning (LoRA / QLoRA)
The LoRA/QLoRA symbols from LoRA and QLoRA, mathematically and GRPO on 16GB. This group collides hard with the RL and statistics tables above, by inheritance from the source chapters rather than by accident; every clash is flagged below.
| Symbol | Meaning | First introduced |
|---|---|---|
| LoRA rank: inner dimension of the low-rank update (collides with reward ) | LoRA and QLoRA, mathematically | |
| LoRA scaling in the update strength (collides with significance level ) | LoRA and QLoRA, mathematically | |
| LoRA down-projection, (Gaussian-initialized) | LoRA and QLoRA, mathematically | |
| LoRA up-projection, (zero-initialized) (collides with bootstrap count ) | LoRA and QLoRA, mathematically | |
| the low-rank weight update, | LoRA and QLoRA, mathematically | |
| (block) | NF4 quantization block size in elements (QLoRA uses 64); the quantization table above calls this | LoRA and QLoRA, mathematically |
This group is a collision minefield with the RL and statistics tables, and every clash is deliberate so the symbols match the source chapters. is the LoRA rank here but the reward in the RL table; is the LoRA scaling factor here but the significance level in the statistics table; and is triply booked, the LoRA up-projection matrix, the bootstrap-resample count (statistics table), and the NF4 block size the quantization table writes as . Chapter 6.3 sets these symbols, so the appendix follows it rather than renaming; the surrounding equation is always what disambiguates.