Tokenization and embeddings

A language model does not read text. It reads integers. Everything between the string you typed and the first matrix multiply inside the network is the job of the tokenizer and the embedding matrix, and this stage is where a surprising amount of eval reproducibility quietly lives or dies. If you run the same prompt through two different chat templates you can get two different integer sequences, two different loss values, and two different eval scores, without touching a single weight. So this chapter treats the input surface as a first-class object: how byte-pair encoding turns text into tokens, what the vocabulary size actually trades off, why a chat template is itself a tokenization decision and not a cosmetic string wrapper, and how the embedding matrix (and weight tying) turns those integers into the vectors the transformer actually consumes.

Theory

Why not just use characters, or words

There are two obvious ways to turn text into integers, and both are bad. Character-level tokenization gives a tiny vocabulary (a few hundred symbols) but makes sequences brutally long, and a transformer's compute scales with the square of sequence length, so long sequences are expensive exactly where you cannot afford it. Word-level tokenization gives short sequences but a vocabulary that is effectively unbounded (every proper noun, typo, and hashtag is a new word) and helpless in front of anything it never saw at training time, the dreaded out-of-vocabulary token. The whole field settled on a middle path: subword tokenization, where common words stay whole, rare words fracture into pieces, and truly novel strings fall back to bytes so that nothing is ever out-of-vocabulary. Byte-pair encoding is the dominant recipe for building that subword vocabulary, and it is what Qwen3, gpt-oss, and essentially every model I touch in this book use.

Byte-pair encoding, derived from a greedy compression rule

BPE started life as a compression algorithm, and it is easiest to understand as one. You begin with the most granular possible alphabet (raw bytes, so 256 base symbols, which guarantees every possible input is representable) and you repeatedly find the most frequent adjacent pair of symbols in your training corpus and merge it into a new single symbol. Do that $K$ times and you have learned $K$ merges and a vocabulary of $256+K$ tokens.

The consequence of equation (2.1)'s greedy, frequency-driven rule is that the resulting vocabulary is a mirror of the training corpus. Frequent English words become single tokens; a chemistry paper's jargon fractures into many pieces; a language underrepresented in the corpus tokenizes into far more tokens per sentence than English does, which is a real and measurable cost (more tokens means more compute and, in a fixed context window, less room). This is why "tokens per word" (the fertility of a tokenizer) differs across languages and domains, and why the same eval prompt can cost noticeably more tokens under one model's tokenizer than another's.

The vocabulary-size tradeoff, quantified

How big should the vocabulary be? This is a genuine tradeoff with costs on both sides, and it is worth making the two forces explicit rather than hand-waving "bigger is better up to a point."

Chat templates are tokenization, not decoration

Here is the point most tutorials skip and that matters most for evals. A modern instruction-tuned model was trained on conversations that were serialized into a specific string format using special tokens that mark the boundaries between roles. These special tokens (things like a begin-of-turn marker, role labels for system/user/assistant, and an end-of-turn marker) are real entries in the vocabulary with their own integer IDs, and the model learned that "an assistant reply begins right after this exact token sequence." The chat template is the (usually Jinja) function that takes your list of role-tagged messages and produces exactly that serialized string, special tokens and all, before BPE ever runs.

This means the chat template is part of the tokenization, and getting it wrong is not a style error, it is feeding the model a distribution it was never trained on. Concretely, a bare prompt like Solve: 2+2 and the same content wrapped as <|im_start|>user\nSolve: 2+2<|im_end|>\n<|im_start|>assistant\n are different token sequences, and the model's next-token distribution after each is different, because only the second one puts the model in the state it was trained to answer from. Two families disagree on the marker strings, on whether a system message is injected by default, on whether a trailing "generation prompt" (the dangling assistant\n that cues the model to start replying) is appended, and on whether reasoning models wrap a hidden thinking segment in their own special tokens. Every one of those choices changes the integer sequence, and therefore the logits, and therefore any eval score computed from them. When I insist later that evals pin the chat template, this is why: it is as load-bearing as the sampler settings.

From integers to vectors: the embedding matrix

Once text is a sequence of token IDs $[t_1,\dots ,t_L]$ with each $t_i\in \{0,\dots ,V-1\}$, the model's first learned operation is to look up a dense vector for each ID. That is the embedding matrix $E\in {\mathbb{R}}^{V\times d}$: row $t_i$ is the $d$-dimensional vector fed into the first transformer layer.

At the other end of the network, the model produces a hidden vector $h\in {\mathbb{R}}^d$ for each position and must turn it into a distribution over the $V$ possible next tokens. It does so with an output projection (the "unembedding" or lm_head) $U\in {\mathbb{R}}^{V\times d}$, computing logits $\ell =Uh\in {\mathbb{R}}^V$, one score per vocabulary entry, which the softmax then normalizes (that is the whole next chapter and the objective chapter). Both $E$ and $U$ are $V\times d$ tables that map between token-space and hidden-space, just in opposite directions.

Weight tying

Since $E$ maps tokens to vectors and $U$ maps vectors to token-logits, and both are $V\times d$, a natural question is whether they should be the same matrix (up to transpose). Weight tying sets $U=E$, so the output logits are $\ell =Eh$ and the input lookup is ${\mathbf{e}}_t^{\top }E$. The argument for it is partly parametric (it removes a full $V\times d$ matrix from the model, which for Qwen3-0.6B's $151,936\times 1024$ is $\approx 0.15$ billion parameters saved, a big fraction of a small model) and partly semantic (a token's input representation and the direction you must point a hidden state to predict that token are plausibly the same geometry). Small models tie almost always, because the embedding is a huge fraction of their parameters and the savings are decisive; large models sometimes untie, because at scale the extra flexibility of a separate output matrix is affordable and slightly helpful. Qwen3's smaller variants tie; whether a given checkpoint ties is a flag in its config.json (tie_word_embeddings), which I will read directly in the model-zoo chapter. Knowing whether tying is on tells you immediately whether the embedding and the lm_head are one tensor or two on disk, which is a memory fact you can check.

Tooling

The tool here is the Hugging Face tokenizers / transformers stack, and specifically the AutoTokenizer interface plus the apply_chat_template method. A loaded tokenizer bundles four things that map straight onto the theory: the merge rules and vocabulary (equation 2.1's learned artifact, usually a fast Rust-backed BPE), the special-token registry (the IDs for the role and boundary markers), the chat template (a Jinja string stored in tokenizer_config.json), and the decode direction (turning IDs back into text). The fast tokenizers are the same tokenizers library exposed to Python, which is why they are quick enough to sit in an eval inner loop.

The two calls that matter for reproducibility are tokenizer(text), which runs plain BPE and returns input_ids, and tokenizer.apply_chat_template(messages, ...), which first renders your role-tagged message list through the model's Jinja template into the exact training-time string and then tokenizes it. The important arguments are add_generation_prompt (whether to append the dangling assistant cue so the model knows to start replying, which you want for generation and not for scoring a completed conversation) and, for reasoning models, enable_thinking or a template-specific flag that controls whether the hidden reasoning segment's special tokens are emitted. Because the template lives inside the checkpoint, using AutoTokenizer.from_pretrained(model_id) is what guarantees you serialize conversations the way that specific model expects. Hand-rolling the string yourself is how you accidentally feed a Qwen model a Llama-shaped prompt and wonder why the eval regressed.

Lab

The goal is to make "chat templates are tokenization" undeniable by measuring it: I will take one fixed prompt, render and tokenize it under three different chat templates, and count exactly how the integer sequences differ. The artifact is a table on disk showing token counts and the specific special tokens each template injects, which is the evidence behind the reproducibility rule.

uv init labs/tokenization
cd labs/tokenization
uv add transformers torch

"""Tokenize one prompt under several chat templates and diff the results.

Artifact: artifacts/template_diff.json plus a printed comparison table.
"""
import json
from pathlib import Path

from transformers import AutoTokenizer

# Three instruction models with genuinely different chat templates.
MODELS = [
    "Qwen/Qwen3-0.6B",
    "meta-llama/Llama-3.2-1B-Instruct",
    "microsoft/Phi-3-mini-4k-instruct",
]

MESSAGES = [
    {"role": "system", "content": "You are a careful math tutor."},
    {"role": "user", "content": "What is 17 * 24? Think step by step."},
]

OUT = Path("artifacts")
OUT.mkdir(exist_ok=True)

def analyze(model_id: str) -> dict:
    tok = AutoTokenizer.from_pretrained(model_id)
    rendered = tok.apply_chat_template(
        MESSAGES, tokenize=False, add_generation_prompt=True
    )
    ids = tok.apply_chat_template(
        MESSAGES, tokenize=True, add_generation_prompt=True
    )
    special_ids = set(tok.all_special_ids)
    specials_used = [
        tok.convert_ids_to_tokens(i) for i in ids if i in special_ids
    ]
    return {
        "model": model_id,
        "vocab_size": tok.vocab_size,
        "n_tokens": len(ids),
        "n_special": sum(1 for i in ids if i in special_ids),
        "specials_used": specials_used,
        "rendered_prefix": rendered[:200],
        "first_20_ids": ids[:20],
    }

def main() -> None:
    results = [analyze(m) for m in MODELS]
    (OUT / "template_diff.json").write_text(json.dumps(results, indent=2))

    print(f"{'model':38} {'vocab':>8} {'tokens':>7} {'special':>8}")
    print("-" * 65)
    for r in results:
        print(f"{r['model']:38} {r['vocab_size']:>8} "
              f"{r['n_tokens']:>7} {r['n_special']:>8}")
    print()
    for r in results:
        print(f"{r['model']}")
        print(f"  special tokens injected: {r['specials_used']}")
        print(f"  rendered head: {r['rendered_prefix']!r}\n")

    counts = [r["n_tokens"] for r in results]
    print(f"Token-count spread for the SAME prompt: "
          f"min={min(counts)}, max={max(counts)}, "
          f"delta={max(counts) - min(counts)} tokens")
    print(f"Artifact: {(OUT / 'template_diff.json').resolve()}")

if __name__ == "__main__":
    main()

uv run python compare_templates.py

What you should see. The script writes artifacts/template_diff.json and prints a table where the same two-message prompt produces visibly different token counts across the three models, typically a spread of several tokens, driven entirely by how many special/boundary tokens each template injects and whether it prepends a default system frame. You will see genuinely different special-token strings in each row (Qwen's <|im_start|>/<|im_end|> family versus Llama's <|start_header_id|>/<|eot_id|> family versus Phi's <|system|>/<|end|> family), and the printed rendered head lets you read the exact serialized string each model expects. The closing line reports the token-count delta for identical content, which is the whole point: nothing about the meaning changed, yet the integer sequence the model sees did, and that is sufficient to move a loss or an eval number. Record the exact counts from your run (they depend on the checkpoint revisions you pulled) and keep the JSON as the receipt for why the eval config pins a tokenizer and template, not just a model name.