LoRA and QLoRA, mathematically

Full fine-tuning of a 4B model on a 16GB card is a non-starter, and the arithmetic says so before I even try: the weights alone in BF16 are 8 GiB, and the Adam optimizer state (two FP32 moments plus an FP32 master copy) is another four to six times the weight footprint, so I am tens of GiB over budget before a single activation is stored. LoRA and its quantized cousin QLoRA are the two ideas that make training on this hardware possible at all, and they are not hacks, they rest on a specific and testable hypothesis about the geometry of fine-tuning updates and a specific and clever data format for 4-bit weights. This chapter derives both: the low-rank reparameterization that turns a billion-parameter update into a few-million-parameter one, and the NF4 quantization that squeezes the frozen base into 4 bits without the usual catastrophe. Then it does the thing the whole book is about, a rank sweep measured with real eval deltas through the chapter 3.7 machinery, so the choice of rank stops being folklore and becomes a number with a confidence interval.

Theory

The low-rank update hypothesis

Fine-tuning changes a weight matrix $W_0\in {\mathbb{R}}^{d\times k}$ into $W_0+\Delta W$. The hypothesis behind LoRA is that for adapting a pretrained model to a downstream task, the update $\Delta W$ has low intrinsic rank, even though $W_0$ itself is full-rank. Intuitively, pretraining already learned the hard, high-dimensional structure of language; adapting it to "answer instructions" or "reason in this format" is a comparatively low-dimensional nudge, a rotation into a few task-relevant directions rather than a wholesale rewrite. If $\Delta W$ really lives near a rank-$r$ subspace with $r\ll \min (d,k)$, then I never need to represent the full $d\times k$ update, I can factor it.

Rank, alpha, and target modules

Three knobs follow from equation (6.3.1). Rank $r$ is the capacity of the update: how many independent directions $\Delta W$ can move in. Too small and the adapter underfits (it cannot express the task nudge); too large and you spend memory and risk overfitting a small dataset, with diminishing returns once $r$ exceeds the true intrinsic rank of the update. Common values are 8 to 64, and the rank sweep in the lab is exactly the experiment that tells you where the returns flatten for your task. Alpha $\alpha$ is the update strength via the $\alpha /r$ scaling; the common convention $\alpha =2r$ keeps the effective scale roughly constant as you vary rank, which is why the lab holds that ratio fixed so the sweep isolates capacity. Target modules is the set of weight matrices you attach adapters to. The attention projections (q_proj, k_proj, v_proj, o_proj) are the classic minimal set; adding the MLP projections (gate_proj, up_proj, down_proj) covers where most of a transformer's parameters and most of its task-specific capacity actually live, at more memory. For reasoning fine-tunes I target all seven, because the MLP is where a lot of the procedural knowledge sits.

NF4: quantizing the frozen base

LoRA shrinks the trainable footprint but I still hold the full $W_0$ in VRAM. QLoRA's second idea is to hold it in 4 bits instead of 16, using a format designed for the specific distribution neural-network weights actually follow. Pretrained weights are approximately zero-mean Gaussian, and a naive linear 4-bit grid (16 evenly spaced levels) wastes most of its levels in the tails where almost no weights live while under-resolving the dense center. NF4 (4-bit NormalFloat) fixes this by placing its 16 levels at the quantiles of a normal distribution, so each level is equally likely to be used, which is information-theoretically optimal for a Gaussian source.

The merge math, and when merging changes behavior

At inference I often want to fold the adapter back into the weights so there is no runtime overhead. From equation (6.3.2), the merge is exact in full precision:

$$W_{\text{merged}}=W_0+\frac{\alpha }{r}BA.\text{\hspace{1em}(6.3.6)}$$

If $W_0$ is stored in BF16, equation (6.3.6) is a lossless (to BF16 rounding) fold and the merged model is behaviorally identical to the adapter-on-base model. The subtlety is QLoRA. There, $W_0$ is stored in NF4, and the honest merge is $\text{quant}(W_0)+\frac{\alpha }{r}BA$: you must dequantize the base to BF16, add the BF16 update, and keep the result in BF16, because $\frac{\alpha }{r}BA$ has structure far finer than the NF4 grid and re-quantizing the sum to NF4 would round the delta away. So merging a QLoRA adapter changes the base dtype (NF4 to BF16), which quadruples the base footprint and is exactly what you want for a clean deployable checkpoint, but it means "merge then re-quantize to 4-bit" is not the same model as "keep the adapter separate on the 4-bit base," and I have watched people lose their whole fine-tune to that round-trip. Keep the adapter separate for iteration; merge to BF16 only when you freeze a release.

Tooling

PEFT implements equations (6.3.1)–(6.3.3) as a LoraConfig (rank r, lora_alpha, target_modules, dropout) wrapped around a base model; bitsandbytes implements the NF4 code of equations (6.3.4)–(6.3.5) with double quantization behind load_in_4bit=True and the bnb_4bit_* flags; and Unsloth fuses the two so the NF4 dequant and the LoRA matmul happen in one custom kernel rather than two eager ops, which is where its speed and its extra memory headroom come from. merge_and_unload() on a PEFT model performs equation (6.3.6). The reused evaluation piece is chapter 3.7's evalstats: after each rank's fine-tune I run the frozen thesis task suite and pass paired per-item scores to evalstats.bootstrap_paired_diff (and evalstats.mcnemar for the paired p-value) to get a delta-versus-base with a 95% bootstrap CI, so the sweep produces intervals, not point estimates.

Lab

The lab sweeps LoRA rank over $\{4,8,16,32,64\}$, fine-tunes the 4B QLoRA model at each rank on the same small reasoning-flavored set, evaluates each on the frozen thesis suite, and reports the eval delta versus the untuned base with bootstrap confidence intervals. The artifact is a CSV and a plot showing where accuracy stops improving with rank, which is the empirical version of "find the intrinsic rank of your task's update."

This is a uv project in the training environment. From the repo root:

uv init labs/lora-rank-sweep
cd labs/lora-rank-sweep
uv add "unsloth[cu124]" trl peft datasets matplotlib
# evalstats is the chapter-3.7 module, installed editable from the repo:
uv add --editable ../../packages/evalstats

"""LoRA rank sweep with eval deltas via the chapter-3.7 evalstats module.

For each rank in RANKS: QLoRA-SFT a 4B model, score it on the frozen thesis
suite, and compute the paired bootstrap delta vs the untuned base. Writes
artifacts/rank_sweep.csv and artifacts/rank_sweep.png.
"""
from pathlib import Path
import csv

import torch
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
from unsloth import FastLanguageModel

# chapter-3.7 machinery: paired bootstrap CI + McNemar over per-item scores.
import evalstats as es
# chapter-3.9 frozen suite: returns per-item 0/1 scores for a model.
from thesis_suite import score_model

MODEL = "unsloth/Qwen3-4B-Base"
RANKS = [4, 8, 16, 32, 64]
MAX_SEQ = 2048
OUT = Path("artifacts")
OUT.mkdir(exist_ok=True)


def train_one(rank: int):
    model, tok = FastLanguageModel.from_pretrained(
        model_name=MODEL, max_seq_length=MAX_SEQ, load_in_4bit=True, dtype=None,
    )
    model = FastLanguageModel.get_peft_model(
        model, r=rank, lora_alpha=2 * rank,   # hold alpha = 2r fixed
        lora_dropout=0.0,
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                        "gate_proj", "up_proj", "down_proj"],
        use_gradient_checkpointing="unsloth", random_state=3407,
    )
    ds = load_dataset("yahma/alpaca-cleaned", split="train[:3000]")
    # Prompt/completion message lists: SFTTrainer renders + masks the prompt.
    ds = ds.map(lambda ex: {
        "prompt": [{"role": "user", "content": ex["instruction"]}],
        "completion": [{"role": "assistant", "content": ex["output"]}]},
        remove_columns=ds.column_names)
    cfg = SFTConfig(
        output_dir=str(OUT / f"run_r{rank}"), max_seq_length=MAX_SEQ,
        completion_only_loss=True,
        per_device_train_batch_size=2, gradient_accumulation_steps=8,
        num_train_epochs=1, learning_rate=2e-4, warmup_ratio=0.03,
        lr_scheduler_type="cosine", bf16=True, logging_steps=25,
        report_to="none", seed=3407,
    )
    SFTTrainer(model=model, args=cfg, train_dataset=ds).train()
    FastLanguageModel.for_inference(model)
    return model, tok


def main() -> None:
    # Baseline scores once, from the untuned base model.
    base, tok = FastLanguageModel.from_pretrained(
        model_name=MODEL, max_seq_length=MAX_SEQ, load_in_4bit=True, dtype=None)
    FastLanguageModel.for_inference(base)
    base_scores = score_model(base, tok)          # list[int], per item
    del base
    torch.cuda.empty_cache()

    rows = []
    for rank in RANKS:
        model, tok = train_one(rank)
        tuned_scores = score_model(model, tok)     # paired, same item order
        # Estimate(point, ci_low, ci_high): paired bootstrap of the delta.
        est = es.bootstrap_paired_diff(tuned_scores, base_scores, level=0.95)
        # McNemar for the paired p-value on binary correctness.
        mc = es.mcnemar(tuned_scores, base_scores, exact=True)
        rows.append({
            "rank": rank,
            "base_acc": sum(base_scores) / len(base_scores),
            "tuned_acc": sum(tuned_scores) / len(tuned_scores),
            "delta": est.point, "ci_lo": est.ci_low, "ci_hi": est.ci_high,
            "p_value": mc.pvalue,
        })
        print(f"r={rank:3d}  delta={est.point:+.3f}  "
              f"95% CI [{est.ci_low:+.3f}, {est.ci_high:+.3f}]  p={mc.pvalue:.3f}")
        del model
        torch.cuda.empty_cache()

    csv_path = OUT / "rank_sweep.csv"
    with csv_path.open("w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        w.writeheader()
        w.writerows(rows)

    # Plot delta vs rank with CI whiskers.
    import matplotlib.pyplot as plt
    xs = [r["rank"] for r in rows]
    ys = [r["delta"] for r in rows]
    lo = [r["delta"] - r["ci_lo"] for r in rows]
    hi = [r["ci_hi"] - r["delta"] for r in rows]
    plt.errorbar(xs, ys, yerr=[lo, hi], marker="o", capsize=4)
    plt.axhline(0, color="grey", lw=0.8)
    plt.xscale("log", base=2)
    plt.xlabel("LoRA rank r"); plt.ylabel("accuracy delta vs base")
    plt.title("Rank sweep: eval delta with 95% bootstrap CI")
    plt.tight_layout(); plt.savefig(OUT / "rank_sweep.png", dpi=150)
    print(f"Artifacts: {csv_path.resolve()} and {(OUT/'rank_sweep.png').resolve()}")


if __name__ == "__main__":
    main()

Run it:

uv run python sweep.py

What you should see. The CSV and plot show the eval delta versus the untuned base rising with rank and then flattening, the empirical signature of the low-rank hypothesis: once $r$ exceeds the task's intrinsic update rank, more capacity buys nothing and the confidence intervals of adjacent ranks overlap. Typically the delta from $r=4$ to $r=16$ is real (CI excludes zero) and the delta from $r=16$ to $r=64$ is within noise (CIs overlap), which tells you $r=16$ is the honest choice for this task, more is just memory. All five fine-tunes fit in 16GB by the vram-budget above, with peak VRAM creeping up only slightly with rank because the adapter and its optimizer state grow linearly in $r$ but are tiny next to the frozen base. Record peak VRAM per rank and total sweep wall-clock (measured on the baseline machine, record value, date, driver). The headline artifact is a plot that turns "what rank should I use" from a Reddit argument into a measured curve with error bars.