When to burst

The previous two chapters were about how to burst: build the image, rent the box, keep the meter honest. This one is about whether to, which is the more important question and the one it is easiest to answer with your gut instead of arithmetic. The gut says "the H100 is faster, so use the H100." The arithmetic says something more useful: bursting is only rational when a job either does not fit on the 16GB card at all, or fits but would take so long locally that the deadline (not the dollar) forces your hand. Most jobs are neither, and the discipline of this chapter is knowing which is which before I spend a cent.

There is a real asymmetry to respect. The RTX 5080 is already paid for; its marginal cost per hour is electricity, call it pennies. The H100 costs its full on-demand rate per hour whether or not I am using it well. So local hours are nearly free and slow; rented hours are expensive and fast. The decision is never "which is better," it is "for this job, does the speedup buy back enough to justify the rate, given when I need the result." That is a break-even calculation, and once it is written down the answer is usually obvious.

The chapter builds that break-even model, sizes a 7-8B GRPO run on an H100 so I know what actually fits and why, and covers DDP and FSDP at reading level so the multi-GPU vocabulary is in place for the post-contract rig even though today's burst is a single H100.

Theory

Three reasons to burst, only two of them good

Strip the decision down. There are exactly three reasons a job leaves the baseline machine.

It does not fit. The model, its optimizer state, the KV cache for the rollouts, and the activations do not fit in 16GB even at the most aggressive quantization and offload I am willing to accept. This is a capability reason and it is unarguable: no amount of patience on the 5080 produces the result. A full 7-8B GRPO run with unquantized rollouts is the canonical example.
It fits but the deadline says no. The job would run locally, but the wall-clock time exceeds the time I have before a committee meeting, a Substack deadline, or my own patience threshold. This is a time reason and it is legitimate, but only after I have priced it.
The H100 is shiny. No.

The first two are the real gate. Everything below is machinery for deciding, in case (2), whether the deadline is worth the rate, and for confirming, in case (1), that the job genuinely does not fit rather than that I gave up on offload too early.

The break-even model

The core comparison is between doing the job locally for nearly free but slowly, and renting for a price but quickly. Let me define the quantities and then find the point where renting starts to make sense.

Local-hours vs rented-hours vs the deadline

Let a job require $W$ units of work (say, training steps, or token-generations for the rollouts). Define:

$s_{L}$ = local throughput on the 5080 (work per hour), and $s_{R}$ = throughput on the rented H100.
$r$ = rented rate in dollars per GPU-hour (cite current Lambda price, $/ GP U - h r, a so f t h er u n d a t e),$ g $= GP U co u n t o n t h ere n t e d in s t an ce . -$ p_L $= l oc a l ma r g ina l cos tp er h o u r (e l ec t r i c i t y), w hi c hi ss ma ll : o n t h eor d ero f t h e b o a r d^{'} s p o w er d r a wt im es t h e l oc a l$ /kWh $r a t e . -$ D $= t h e d e a d l in e, inh o u rs f ro mn o w, b y w hi c h I n ee d t h eres u lt . * * L oc a lw a ll - c l oc k * * i s$ T_L = W / s_L $a t cos t$ C_L = p_L \cdot T_L $. * * R e n t e d w a ll - c l oc k * * i s$ T_R = W / (s_R \cdot g) + t_{\text{ovh}} $, w h ere$ t_{\text{ovh}} $i s t h e b oo t / sy n c / t e a r d o w n o v er h e a df ro m t h e l a s t c ha pt er, a t cos t$ C_R = r \cdot g \cdot T_R $. * * G a t e 1, f e a s ibi l i t y . * * I f t h e j o b d oes n o t f i tl oc a ll y,$ s_L = 0 $(e q u i v a l e n tl y$ T_L = \infty $) an d t h e d ec i s i o ni s ma d e : b u rs t, b ec a u se l oc a l i s n o t an o pt i o n . S ki pt h eres t o f t h e ma t h . * * G a t e 2, d e a d l in e . * * I f i t d oes f i tl oc a ll y, re n t in g i so n l y * n ecess a ry * w h e n l oc a l mi sses t h e d e a d l in e :$ $T_{L} = \frac{W}{s _{L}} > D ⟹ local cannot make it; consider renting.$ $I f$ T_L \le D $, l oc a l mak es t h e d e a d l in e f or p e nni es an d re n t in g i s p u re w a s t e, n o ma tt er h o w m u c h f a s t er t h eH 100 i s . Sp ee d yo u d o n o t n ee d i s n o tw or t h p a y in g f or . * * T h e p r i ceo f t h e d e a d l in e . * * Wh e n l oc a l mi sses t h e d e a d l in e, re n t in g cos t s m e an e x t r a$ $Δ C = C_{R} - C_{L} = r g (\frac{W}{s _{R} g} + t_{ovh}) - p_{L} \frac{W}{s _{L}}$ $d o ll a rs t o m o v e t h e f ini s h l in e f ro m$ T_L $(t oo l a t e) t o$ T_R $(in t im e) . T h e q u es t i o ni ss im pl y w h e t h er t h eres u lt, d e l i v ere d b y$ D $in s t e a d o f$ T_L $, i s w or t h$ \Delta C $. T ha t i s aj u d g m e n t, b u t n o w i t i s aj u d g m e n t ab o u t a * s p ec i f i c n u mb ero fd o ll a rs f or a s p ec i f i c n u mb ero f h o u rss a v e d *, n o t a v ib e . * * A w or k e d e x am pl e (pl a ce h o l d er t h ro ug h p u t s; recor d t h ere a l o n es) . * * S u pp ose aj o bi s$ W = 1 $(o n e f u ll r u n), l oc a lw o u l d t ak e$ T_L = 30 $h, an d t h e d e a d l in e i s$ D = 12 $h, so l oc a l mi sses . O na s in g l eH 100 t h er u n t ak es$ T_R = 6 $h w a ll - c l oc kin c l u d in g$ t_{\text{ovh}} = 0.5 $h . Wi t ha pl a ce h o l d err a t e$ r = $3.00/\text{GPU-hr} $(c i t e t h ere a l c u rre n t L amb d a r a t e) an d$ g = 1 $:$ $C_{R} = 3.00 \times 1 \times 6 = $18.00,$ $v ers u s a l oc a l e l ec t r i c i t ycos t o f, s a y,$ p_L \approx $0.10/\text{h} \times 30,\text{h} = $3.00 $. S o t h e d e a d l in ecos t s ab o u t$ \Delta C \approx $15 $t o m o v e t h e f ini s h l in e f or w a r d b y$ T_L - T_R = 30 - 6 = 24 $h o u rs (o f w hi c h o n l y t h e l a s t 18, t h e g a p$ T_L - D $, i s t h e minim u m s a v in g a c t u a ll y n ee d e d t o hi t$ D $) . Wh e t h er t ha t i s a g oo d t r a d e d e p e n d se n t i re l yo n w ha tt h e d e a d l in e i s f or . Pl ug in t h e * m e a s u re d *$ s_R$ from the rented H100 (record value, date, driver) before trusting any absolute number here.

The model's punchline is the ordering of the gates. Feasibility first: does it fit. Then deadline: does local miss it. Only if both point to renting do I look at the dollar delta, and even then the delta is a decision input, not a verdict. The most common correct answer, for the small experiments that make up most of a thesis, is "run it locally overnight for free."

Sizing a 7-8B GRPO run on an H100

The clearest "does not fit" case is a full-precision GRPO run on a 7-8B model, so it is worth knowing what such a run needs, both to justify the burst and to size the instance. GRPO's memory footprint has an extra term that plain fine-tuning does not: it generates a group of rollouts per prompt and scores them, so a serving-shaped generation cost sits next to the training cost.

Rough memory budget for 7-8B GRPO

Take a 7.6B-parameter model as the example and account the big terms in bytes. The exact coefficients depend on optimizer and offload choices; this is the order-of-magnitude budget that tells me a single H100's 80GB is comfortable and the 5080's 16GB is not.

Weights (BF16): $7.6\times10^{9} \times 2,\text{byte} \approx 15.2,\text{GB} $. - * * G r a d i e n t s (BF 16) : * * an o t h er$ \approx 15.2,\text{GB} $i f t r ainin g f u llp a r am e t ers . - * * Opt imi zers t a t e (A d am, FP 32 m o m e n tp ai r + ma s t er w e i g h t s) : * * t h ec l a ss i c f a c t or i s$ \approx 12 $b y t es / p a r am f or f u ll f in e - t u nin g, so$ 7.6\times10^{9}\times 12 \approx 91,\text{GB} $, a l re a d y p a s t o n eH 100 o ni t so w n . T hi s i s t h e t er m t ha t f orces a c h o i ce . T h ec h o i ce t ha t mak es i t f i t i s * * L o R A / Q L o R A * * : f reeze t h e ba se w e i g h t s (n o g r a d i e n t s, n oo pt imi zers t a t eo n 7.6 Bp a r am s) an d t r ain s ma ll a d a pt ers . T h e n t h e h e a v yo pt imi zer t er ma ppl i eso n l y t o t h e a d a pt e r^{'} s f e wt e n so f mi ll i o n so f p a r am e t ers, co ll a p s in g t ha t$ 91,\text{GB} $t o w e ll u n d er a g i g ab y t e . Wha t i s l e f t o n t h eH 100 i sro ug h l y : ba se w e i g h t s (q u an t i ze d t o 4 - bi t f or Q L o R A,$ \approx 7.6\times10^{9}\times0.55 \approx 4.2,\text{GB} $, or BF 16 a t$ 15.2,\text{GB} $), a d a pt er g r a d s / s t a t e (s ma ll), a c t i v a t i o n s, an d t h e * * ro ll o u t K V c a c h e * * f or t h e GRPO g ro u p . - * * R o ll o u t K V c a c h e : * * GRPO s am pl es$ G $co m pl e t i o n s p er p ro m pt . K V c a c h esc a l es w i t h (l a yers$ \times $2$ \times $hi dd e n$ \times $co n t e x t$ \times $d t y p e - b y t es$ \times $co n c u rre n t se q u e n ces) . A g ro u p o f$ G = 8 $co m pl e t i o n s a t a f e w - t h o u s an d - t o k e n re a so nin g co n t e x t i s a ser v in g - s i ze d a ll oc a t i o n o f se v er a lGBt ha tpl ain SFT n e v er p a ys . S u mmin g t h e Q L o R A p a t h : ahan df u l o f GB o f q u an t i ze d ba se w e i g h t s + s ma ll a d a pt ers t a t e + se v er a lGB o f ro ll o u t K V c a c h e + a c t i v a t i o n s l an d sco m f or t ab l y in s i d e 80 GBw i t h roo m f or a re a l ba t c han d l o n g re a so nin g t r a ces . T h es am es u ma g ain s t 16 GB i s a co n s t an t f i g h tw i t h o ff l o a d an d t in y g ro u p s . T ha t g a p, co m f or t ab l eo n H 100 an d a g r in d o n t h e 5080, i s t h eco n cre t e " d oes n o t f i tw e ll " t ha t j u s t i f i es b u rs t in g a f u ll - p rec i s i o n, re a l - g ro u pGRPO r u n . R ecor d t h e a c t u a lp e akV R A M o n t h ere n t e d H 100 (m e a s u re d, w i t h d a t e an dd r i v er); t h e b u d g e t ab o v e i s w ha t I e x p ec tt osee, n o tw ha t I m e a s u re d . < / d i v >< / d i v > T h es i z in g a l soe x pl ain s * w h y * t h e 5080 d oess ma llGRPO a t a ll (P a r t V II) : s h r ink t h e m o d e l, q u an t i ze ha r d, c u tt h e g ro u p s i ze$ G$, shorten the context, and offload. Every one of those is a compromise the H100 lets me drop. So bursting a GRPO run is not just "faster," it is "the version of the experiment without the compromises," which is sometimes the only version worth putting in front of a committee.

DDP and FSDP at reading level

Today's burst is one H100 and needs neither of these, but the post-contract rig (the one I would buy if the thesis turns into something) is multi-GPU, and the vocabulary should be in place. Two ideas cover most of it.

DDP, Distributed Data Parallel, puts a full copy of the model on every GPU and gives each GPU a different slice of the batch. Each computes gradients on its slice, then all GPUs average their gradients (an all-reduce) so every copy takes the same step. DDP is simple and fast and scales throughput almost linearly, but it does nothing for memory: if the model plus optimizer state does not fit on one GPU, DDP does not help, because every GPU still holds the whole thing. DDP is the answer to "it fits, I just want it faster."

FSDP, Fully Sharded Data Parallel, attacks memory instead. It shards the parameters, gradients, and optimizer state across GPUs so no single GPU holds the full set. When a layer is needed for the forward or backward pass, its shard is gathered just-in-time from the other GPUs, used, then released. This trades communication for memory: you can fit models far larger than one GPU's RAM, at the cost of more inter-GPU traffic. FSDP (and ZeRO, its conceptual sibling) is the answer to "it does not fit on one GPU at all."

Note

The two are answers to different questions and mixing them up wastes money. DDP does not lower per-GPU memory, so renting a 2x or 4x instance with DDP to fit a model that overflows one GPU will not work, since every GPU still needs the whole model. FSDP lowers per-GPU memory but adds communication overhead, so using it when a single GPU already fits the model is paying a bandwidth tax for nothing. On a single rented H100, neither applies: it is one GPU, and the levers are quantization, LoRA, batch size, and group size, exactly as on the 5080 but with 5x the headroom. Reach for DDP/FSDP only when the instance actually has multiple GPUs and you have decided which problem, speed or fit, you are solving.

Tooling

The tool here is the break-even model itself, made runnable so the decision is a script, not a mood. It takes the job's parameters and today's Lambda rate and prints the two gates and, if both fire, the dollar delta. It deliberately refuses to recommend renting when local makes the deadline, because that is the discipline the whole chapter is defending.

"""Decide whether to burst, by the two-gate model.

Feasibility first (does it fit locally at all), then deadline (does local
miss it). Only if both point to renting do we price the delta. Throughputs
are inputs you measure; the Lambda rate is the current price on the run date.
"""
from __future__ import annotations
import argparse
from dataclasses import dataclass


@dataclass
class Job:
    work: float            # abstract units (e.g. runs, or step-count)
    s_local: float         # local throughput (work/hour); 0.0 => does NOT fit
    s_rented: float        # rented per-GPU throughput (work/hour)
    deadline_h: float      # hours from now until the result is needed
    rate_per_gpu_hr: float # cite current Lambda price, $/GPU-hr, run date
    gpus: int = 1
    overhead_h: float = 0.5   # boot+sync+teardown from the runbook
    local_power_usd_h: float = 0.10  # 5080 board draw x local $/kWh


def decide(j: Job) -> None:
    # Gate 1: feasibility.
    if j.s_local <= 0.0:
        t_r = j.work / (j.s_rented * j.gpus) + j.overhead_h
        c_r = j.rate_per_gpu_hr * j.gpus * t_r
        print("GATE 1: does NOT fit locally -> BURST (no local option).")
        print(f"  rented wall-clock ~ {t_r:.2f} h, cost ~ ${c_r:.2f}")
        return

    t_l = j.work / j.s_local
    c_l = j.local_power_usd_h * t_l
    print(f"GATE 1: fits locally. local wall-clock ~ {t_l:.2f} h, "
          f"cost ~ ${c_l:.2f} (electricity).")

    # Gate 2: deadline.
    if t_l <= j.deadline_h:
        print(f"GATE 2: local MAKES the deadline ({t_l:.2f} h <= "
              f"{j.deadline_h:.2f} h) -> RUN LOCAL. Renting is waste.")
        return

    t_r = j.work / (j.s_rented * j.gpus) + j.overhead_h
    c_r = j.rate_per_gpu_hr * j.gpus * t_r
    print(f"GATE 2: local MISSES the deadline ({t_l:.2f} h > "
          f"{j.deadline_h:.2f} h).")
    if t_r > j.deadline_h:
        print(f"  ...but rented also misses ({t_r:.2f} h). Rethink scope, "
              f"not hardware.")
        return
    print(f"  rented makes it: {t_r:.2f} h <= {j.deadline_h:.2f} h.")
    print(f"  price of the deadline: dC ~ ${c_r - c_l:.2f} "
          f"to save {t_l - t_r:.2f} h. Is that worth it? (judgment)")


if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--work", type=float, default=1.0)
    ap.add_argument("--s-local", type=float, required=True,
                    help="0 means it does not fit locally")
    ap.add_argument("--s-rented", type=float, default=5.0)
    ap.add_argument("--deadline-h", type=float, required=True)
    ap.add_argument("--rate", type=float, required=True,
                    help="current Lambda $/GPU-hr on the run date")
    ap.add_argument("--gpus", type=int, default=1)
    a = ap.parse_args()
    decide(Job(a.work, a.s_local, a.s_rented, a.deadline_h, a.rate, a.gpus))

Lab: run the decision for a real job

The artifact is a small decision record on disk: the parameters of an actual candidate job, the model's verdict, and my one-line rationale, saved next to the burst runbook so the "why did I rent" question has a written answer months later.

Three candidate jobs

# Use the REAL current Lambda rate for --rate (cite $/GPU-hr as of the run date).
RATE=3.00   # placeholder; replace with the day's price

# A) Small GRPO smoke experiment: fits locally, 3h, deadline 24h.
uv run python burst/should_i_burst.py \
    --work 1 --s-local 0.33 --s-rented 2.0 --deadline-h 24 --rate $RATE
# expect: GATE 2 -> RUN LOCAL (3h <= 24h). Renting is waste.

# B) Full 7-8B GRPO, real group, BF16 rollouts: does NOT fit 16GB.
uv run python burst/should_i_burst.py \
    --work 1 --s-local 0 --s-rented 0.17 --deadline-h 48 --rate $RATE
# expect: GATE 1 -> BURST (no local option).

# C) Big local job that misses a hard deadline: fits but 30h vs 12h.
uv run python burst/should_i_burst.py \
    --work 1 --s-local 0.033 --s-rented 0.17 --deadline-h 12 --rate $RATE
# expect: GATE 2 -> local misses; rented makes it; prints dC to decide.

The decision record

# Burst decision: full 7-8B GRPO, real group

Job:        Qwen 7-8B, GRPO, G=8, BF16 rollouts, ~2-4k ctx
Gate 1:     does NOT fit 16GB at real group/precision (see sizing budget)
Gate 2:     n/a (feasibility already forces the burst)
Verdict:    BURST, single H100 (80GB), QLoRA base, full group
Instance:   gpu_1x_h100
Rate used:  <cite current Lambda $/GPU-hr, run date>
Expected:   run wall-clock T_R = ____ h + overhead ____ h  (fill from runbook)
Measured:   peak VRAM ____ GB, run ____ h, cost $____  (H100; date; driver)
Rationale:  this is the no-compromise version of the Part VII experiment;
            the 16GB card can only do the shrunk-down version, which is not
            what goes in front of the committee.

What you should see

Running the three candidates prints three different verdicts from the same script, and that variety is the whole point: (A) fits and makes its deadline, so the tool refuses to rent and I run it overnight for pennies; (B) does not fit at all, so Gate 1 forces the burst with no dollar argument needed; (C) fits but misses a hard deadline, so the tool prints the dollar delta and hands the judgment back to me. The artifact on disk is burst/decisions/2026-07-22-full-grpo.md, a decision record whose "Expected" line I fill from the model before booting and whose "Measured" line I fill from the H100 afterward (peak VRAM, run hours, dollars, with date and driver). Months later, when I or the committee ask "why did this one experiment cost $18 when everything else was free," the answer is a file, not a memory: it did not fit on the 5080, here is the sizing budget that proves it, and here is what it actually cost. That written trail is what keeps the burst honest, and it is the reason the decision lives in a script and a record rather than in the moment's enthusiasm for a faster GPU.

flowchart TD
    A[Candidate job] --> B{Fits on 16GB 5080?}
    B -- no --> BURST[BURST: capability reason]
    B -- yes --> C{Local wall-clock <= deadline?}
    C -- yes --> LOCAL[RUN LOCAL: nearly free]
    C -- no --> D{Rented makes the deadline?}
    D -- no --> E[Rethink scope, not hardware]
    D -- yes --> F[Price dC; judge if worth it]

[RLHF] ch. 6, infrastructure notes

Lambert's infra chapter is the reading-level companion to this one. It lays out why post-training compute is dominated by generation (the rollouts), which is exactly the term that makes GRPO's memory budget balloon past 16GB and forces the burst, the KV-cache-for-a-group cost I sized above is his "generation dominates" observation in bytes. His treatment of data-parallel versus sharded training is the conceptual source for the DDP-vs-FSDP split here: DDP for throughput when it fits, sharding for fit when it does not. Read it to connect the cost model of this chapter (when renting is rational) to the systems reality of the last two (why serving and training stay separate processes, and why keeping the accelerator busy is the whole game).

Note

"When NOT to rent the H100." The honest version of a cloud-GPU post is mostly a list of times you should keep your laptop closed and let the cheap card grind overnight. Bursting is rational in exactly two cases, the job does not fit at all, or it fits but misses a hard deadline, and everything else is paying premium rates for speed you do not need. The useful artifact is a two-gate decision script that refuses to recommend renting when your own machine makes the deadline, plus a memory-budget back-of-envelope that shows precisely why a full-group GRPO run on a 7-8B model overflows 16GB but sits comfortably on an 80GB H100. A post that reframes cloud GPUs from "faster is better" to "faster is a purchase, and here is how to know if you should make it."

Evals as Rewards