Scorers as rewards

Here is the hinge the whole book turns on. In Part III I built scorers as measurement instruments: a scorer takes a model's completion and returns a number that says how good it was, and I was careful to make the good ones programmatic, reproducible, and cheap, precisely because I knew this chapter was coming. In Part V I derived GRPO, whose one external input is a reward: a number that says how good a completion was. Those are the same number. A verifier that can measure whether a reasoning trace reached the right answer can reward a policy for reaching it, and the only difference is which loop the number feeds. This chapter makes that identity real by taking the actual scorers from the thesis suite and wiring them into GRPOTrainer as reward functions. It is the load-bearing chapter of Part VII, because if this join is sloppy, every training result downstream is measuring something other than what I think, and if it is clean, the serve-evaluate-score-train-re-evaluate loop is finally a single closed circuit rather than four separate tools.

Theory

A scorer and a reward are the same function wearing two hats

Be precise about the claim, because "evals as rewards" is a slogan and I want the mechanism. A Part III scorer, in the Inspect sense, is a function from (a completed sample, its target) to a Score: a value, usually in $[0,1]$ or a CORRECT/INCORRECT label, plus metadata explaining the judgment and, critically, the log entry that makes the judgment auditable and re-scorable later. A GRPO reward function, in the TRL sense, is a function from (prompts, a batch of completions, and any passthrough columns) to a list of floats, one per completion. Strip away the framework packaging and both are the same core operation: parse a final answer out of a generated string, decide whether it matches the gold answer, return a number. The verifier at the center, the part that actually knows what "correct" means for this task, does not care whether its output is going to be averaged into an accuracy statistic or centered into a group-relative advantage.

That is why the discipline I imposed in Part III pays off exactly now. Because the thesis-suite scorers were written as pure, deterministic, side-effect-free verifiers (regex extraction, symbolic canonicalization, exact comparison) rather than as model-graded judges baked into an eval harness, I can lift the verifier out and call it in a training loop without dragging the whole harness along. The re-scorable log format from chapter 3.4 was not a nicety; it was the design decision that guaranteed the scorer's core was a standalone function I could reuse here. The alternative, a scorer entangled with Inspect's solver and dataset machinery, would force me to either run a full eval per training step (impossibly slow) or reimplement the verifier (two verifiers that drift apart, which is worse than none).

Sparse correctness, dense format, and why you need both

A pure correctness reward (1 if the final answer is right, 0 otherwise) is the honest signal, and on a hard reasoning task it is also a nearly silent one. Early in training the model gets almost everything wrong, so the reward is 0 for nearly every completion in a group, the group mean is near 0, and the group-relative advantage is near 0 for everyone. There is nothing to learn from a group where every member scored identically, because GRPO's signal is the spread within the group. This is the sparse-reward problem from Part V made concrete: correctness alone gives the policy no gradient until it stumbles into a correct answer by chance often enough for the group to differentiate.

The standard fix is to add a format reward: a small, dense signal that pays the model for structural compliance it can achieve long before it can reason correctly. Did it produce exactly one <think> block and one <answer> block? Is the answer parseable as the expected type? These are things a model learns in a handful of steps, and rewarding them does two useful things. First, it gives the group something to differentiate on early, so the advantage is non-zero and learning starts. Second, and less obviously, it makes the correctness reward computable at all: a verifier can only check an answer it can extract, so teaching the model to emit a well-formed <answer> tag is a precondition for the correctness signal ever firing. Format reward is scaffolding, and like scaffolding it is meant to become irrelevant once the building stands.

The danger, which is the entire subject of the next chapter, is that format reward is gameable in a way correctness is not. A model can max out a format reward without ever reasoning, and if the format reward is large relative to correctness, the policy will happily learn to produce beautifully-tagged nonsense. So the two rewards are not peers. Correctness is the objective; format is a small dense nudge toward being able to receive the objective. The scaling has to reflect that hierarchy, which is the next thing to get right.

Shaping and scaling: what the numbers should be

Reward shaping is choosing the magnitudes so the policy optimizes what I mean. TRL sums the reward functions, so if I return correctness in $\{0,1\}$ and format in $\{0,0.2\}$, a completion's total reward is in $\{0,0.2,1.0,1.2\}$, and the ordering encodes my intent: a correct, well-formatted answer (1.2) beats a correct but malformed one (1.0) beats a wrong but well-formatted one (0.2) beats wrong-and-malformed (0.0). That ordering is the thing to design deliberately, and the ratio matters more than the absolute values.

Beyond the format-versus-correctness split, shaping has a few more moves worth knowing. Graded correctness replaces the binary with partial credit where the task allows it (a proof that gets three of four steps, a numeric answer within tolerance), which densifies the objective itself rather than adding a proxy. Bounded penalties (a small negative reward for, say, exceeding a length cap or emitting no answer tag at all) discourage specific pathologies, but every penalty is a new surface to game, so I add them reluctantly and watch them. And normalization of the verifier's own output matters: if a graded verifier returns scores on $[0,100]$ and the format reward is $0.2$, the format term is numerically irrelevant, so I keep every component on a comparable scale before I reason about their ratios. The governing principle is that the reward is a specification of desired behavior written in the language of numbers, and every term I add is a clause in that specification that the policy will interpret adversarially.

Asynchronous scoring: the verifier is on the critical path

The last theory piece is a performance fact with a correctness consequence. A GRPO step generates $G$ completions per prompt and must score every one before it can compute advantages, so the verifier is called $G\times (\text{batch})$ times per step, synchronously blocking the optimizer. For a cheap verifier (a regex and an integer compare) this is free. But the interesting thesis-suite verifiers are not always cheap: symbolic canonicalization with a computer-algebra system, or a code-execution verifier that runs the model's output in a sandbox, can take tens or hundreds of milliseconds each, and $G\times \text{batch}$ of those in series can rival the generation time. The step is already generation-bound (last chapter); I do not want to make it verifier-bound on top.

The fix is to score the group concurrently. Because the verifiers are pure functions of independent completions, scoring them is embarrassingly parallel: a thread pool (for I/O-bound verifiers like a sandbox subprocess or a CAS call that releases the GIL) or an async gather runs all $G\times \text{batch}$ verifications at once and returns when the slowest finishes, turning a sum of latencies into a max. This is safe only because the verifiers are side-effect-free and deterministic, which is again the Part III discipline paying a dividend: a scorer that mutated shared state or depended on call order could not be parallelized this way. Two guardrails come with concurrency: a timeout per verification, so one pathological completion (an infinite loop in a code-execution task) cannot stall the whole step, with a timeout scored as 0; and, optionally, a cache keyed on the completion text, since a group sometimes contains duplicate completions and re-verifying identical strings is waste. The reward module in the lab builds all of this in, because "the verifier is on the critical path" is exactly the kind of thing that is invisible on the toy task and dominant on the real one.

Tooling

The tool here is not a new library, it is a small module of my own, rewards.py, that adapts the thesis-suite verifiers into TRL's reward-function interface. Its job is to be the single, thin, auditable seam between the Part III scoring world and the Part V training world, and everything I learned about the join goes into keeping that seam honest.

The verifier I am adapting is the SDA scorer from chapter 3.4. In Inspect it is wrapped as a @scorer returning a Score, but its core is a pure function I exposed deliberately: extract the <answer> content, canonicalize it, compare to the target, return a float and a reason. The reward module imports that function, not the Inspect wrapper, so training and measurement share one verifier with no reimplementation. Around it the module adds the three things GRPO needs that measurement did not: the TRL calling convention (a list-in, list-out batch function reading passthrough columns), the format reward as a separate dense component, and the concurrency-plus-timeout machinery so a slow verifier does not throttle the step.

Lab: train against the thesis suite's scorers

The artifact of this chapter is the reward module itself, rewards.py, because it is the reusable seam every subsequent training run in the book imports. The lab wires it into a GRPO run on a slice of the SDA suite and confirms, in MLflow, that the correctness and format rewards move the way the theory says they should. This is the first real training run of the thesis: same model and budget as the last chapter, but now the reward is the genuine SDA verifier rather than a toy adder.

Set up the project, reusing the Unsloth-pinned stack and adding the thesis suite and a symbolic library for canonicalization.

uv init labs/grpo-sda
cd labs/grpo-sda
uv add "unsloth" "unsloth_zoo" "vllm" "trl" "peft" "bitsandbytes" mlflow
uv add sympy                     # symbolic canonicalization in the verifier
uv add "thesis-suite @ ../../thesis-suite"   # the Part III scorers, as a local pkg
uv lock
export MLFLOW_TRACKING_URI=http://127.0.0.1:5000

The reward module. It imports the SDA verifier from the thesis suite (the same function chapter 3.4 measures with), wraps it for TRL, adds a dense format reward, and runs the verifiers concurrently with a per-item timeout.

"""Thesis-suite SDA scorers, adapted into GRPO reward functions.

This is the single seam between Part III measurement and Part V training.
Correctness reuses the EXACT verifier chapter 3.4 measures with; format is a
small dense shaping term; both run concurrently with a timeout so a slow
verifier does not throttle the step. Kept deliberately thin and auditable.
"""
from __future__ import annotations

import re
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FTimeout
from functools import lru_cache

# The pure verifier from Part III chapter 3.4. Same function used for
# measurement; imported here, not reimplemented, so the two never drift.
from thesis_suite import verify_sda_answer  # (response, target) -> bool

THINK_RE = re.compile(r"<think>.*?</think>", re.DOTALL)
ANSWER_RE = re.compile(r"<answer>(.*?)</answer>", re.DOTALL)

# --- shaping constants: ratios matter, not absolute scale (see derivation) ---
CORRECT_W = 1.0     # the objective
FORMAT_W = 0.2      # dense scaffold, kept well below the objective
NO_ANSWER = 0.0     # explicit policy: unparseable completion scores 0 correctness
VERIFY_TIMEOUT_S = 2.0


def _text(completion) -> str:
    """TRL hands conversational completions as message lists; pull the content."""
    if isinstance(completion, list):
        return completion[-1]["content"]
    return completion


@lru_cache(maxsize=8192)
def _cached_verify(completion_text: str, target: str) -> bool:
    """Cache identical (completion, target) pairs; verifiers are pure."""
    return bool(verify_sda_answer(completion_text, target))


def _verify_one(completion_text: str, target: str) -> float:
    m = ANSWER_RE.search(completion_text)
    if not m:
        return NO_ANSWER                      # cannot verify what has no answer
    try:
        return CORRECT_W if _cached_verify(completion_text, str(target)) else 0.0
    except Exception:
        return 0.0                            # a verifier that raises is a miss


def correctness_reward(prompts, completions, target, **kwargs):
    """Concurrent, timed correctness scoring via the shared SDA verifier."""
    texts = [_text(c) for c in completions]
    out = [0.0] * len(texts)
    with ThreadPoolExecutor(max_workers=min(16, len(texts) or 1)) as pool:
        futures = {pool.submit(_verify_one, t, tgt): i
                   for i, (t, tgt) in enumerate(zip(texts, target))}
        for fut, i in futures.items():
            try:
                out[i] = fut.result(timeout=VERIFY_TIMEOUT_S)
            except FTimeout:
                out[i] = 0.0                  # pathological completion -> miss
    return out


def format_reward(prompts, completions, **kwargs):
    """Dense scaffold: pay for exactly one well-formed think+answer structure."""
    out = []
    for c in completions:
        t = _text(c)
        ok = (len(THINK_RE.findall(t)) == 1
              and len(ANSWER_RE.findall(t)) == 1)
        out.append(FORMAT_W if ok else 0.0)
    return out


REWARD_FUNCS = [correctness_reward, format_reward]

A quick standalone check of the rewards before spending GPU time, which is the habit the earlier gotcha argued for.

"""Validate reward functions on hand-written strings before launching a run."""
from rewards import correctness_reward, format_reward

good = "<think>2 and 2</think><answer>4</answer>"
bad_fmt = "the answer is 4"
wrong = "<think>...</think><answer>5</answer>"
comps = [good, bad_fmt, wrong]
tgts = ["4", "4", "4"]

print("correctness:", correctness_reward(None, comps, tgts))  # [1.0, 0.0, 0.0]
print("format    :", format_reward(None, comps))             # [0.2, 0.0, 0.2]

uv run python check_rewards.py

Now the training script. It is the last chapter's script with the reward source swapped for the module above and the dataset drawn from the SDA suite.

"""GRPO on the SDA suite with the real thesis-suite verifier as reward."""
from unsloth import FastLanguageModel  # noqa: E402

import hashlib
from pathlib import Path

import mlflow
import torch
from trl import GRPOConfig, GRPOTrainer

from rewards import REWARD_FUNCS
from thesis_suite.data import load_sda_split   # Part III frozen v1.0 suite

MODEL = "unsloth/Qwen3-4B"
MAX_SEQ = 2048
LORA_R = 16
GROUP = 8


def main() -> None:
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name=MODEL, max_seq_length=MAX_SEQ, load_in_4bit=True,
        fast_inference=True, max_lora_rank=LORA_R, gpu_memory_utilization=0.16,
    )
    model = FastLanguageModel.get_peft_model(
        model, r=LORA_R,
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                        "gate_proj", "up_proj", "down_proj"],
        lora_alpha=LORA_R, use_gradient_checkpointing="unsloth",
    )

    # The TRAIN split only. The frozen v1.0 test split is never trained on;
    # it is what chapter 7.6 measures the delta against, with the same scorer.
    train_ds = load_sda_split("train")

    cfg = GRPOConfig(
        output_dir="outputs", num_generations=GROUP,
        per_device_train_batch_size=GROUP, gradient_accumulation_steps=2,
        max_prompt_length=512, max_completion_length=1024,
        learning_rate=5e-6, beta=0.02, epsilon=0.2, num_iterations=1,
        loss_type="dr_grpo", temperature=1.0, top_p=1.0,
        max_steps=500, logging_steps=1, save_steps=100,
        report_to="mlflow", run_name="grpo-sda-r16", seed=0,
    )

    torch.cuda.reset_peak_memory_stats()
    mlflow.set_experiment("p7-grpo")
    with mlflow.start_run(run_name="grpo-sda-r16"):
        mlflow.log_params({
            "model": MODEL, "lora_r": LORA_R, "group": GROUP,
            "beta": cfg.beta, "loss_type": cfg.loss_type,
            "correct_w": 1.0, "format_w": 0.2,
            "uv_lock": hashlib.sha256(Path("uv.lock").read_bytes()).hexdigest()[:12],
        })
        trainer = GRPOTrainer(
            model=model, processing_class=tokenizer,
            reward_funcs=REWARD_FUNCS, args=cfg, train_dataset=train_ds,
        )
        trainer.train()
        mlflow.log_metric("vram_peak_gib", torch.cuda.max_memory_allocated() / 2**30)
        model.save_lora("outputs/sda-adapter")
        mlflow.log_artifacts("outputs/sda-adapter", artifact_path="lora")
        print("adapter saved: outputs/sda-adapter")


if __name__ == "__main__":
    main()

uv run python train.py

What you should see

The artifact is rewards.py, the reusable seam, plus outputs/sda-adapter/, the LoRA adapter trained against the real SDA verifier, and an MLflow run under p7-grpo. In the MLflow UI the two reward curves tell the story the theory predicted. rewards/format_reward/mean should rise fast and saturate near its 0.2 ceiling within a few dozen steps, because structural compliance is easy and the dense signal drives it quickly. rewards/correctness_reward/mean should start near zero (early groups are almost all wrong, so the signal is sparse) and then, once format compliance makes answers reliably extractable, begin a slower climb as the policy actually gets better at the SDA reasoning. The shape to look for is format leading and correctness following, which is format-as-scaffolding working as designed. If correctness stays flat while format saturates, either the task is too hard for this model at this budget (the group never differentiates on correctness) or, more likely on a first run, the verifier is silently rejecting well-reasoned answers because of a canonicalization mismatch, which you debug by dumping a few completions the verifier scored 0 and checking them by hand against verify_sda_answer. The vram_peak_gib should again land near the last chapter's budget, a little higher for the longer max_completion_length=1024. Record the peak, the final reward means, and the step time with date and driver (measured on the baseline machine, record value, date, driver). The deep thing this run establishes is not the reward number, it is that the number came from the same function that will measure the result in chapter 7.6, so the loop is now genuinely closed: one verifier, two hats, no drift.