Tokenizers & Chat Templates

Module FT07 · Course 3 — LLM Fine-Tuning Masterclass

60 minutes · the capstone of Pillar 1 — the last gate between "clean data" and "correct tokens"

The model trains. The loss goes down. The checkpoint loads. And the model learned the wrong thing — because the tokens were wrong.

Pillar 1 — Data

Why FT07 is the silent-bugs module

The data pipeline is not complete when the JSONL is clean. It is complete when the tokens are correct.

FT00 thesis: fine-tuning steers behavior; it does not teach knowledge. Steering is only as good as the steering wheel — and the steering wheel is the tokenized data. A brilliant algorithm on mis-tokenized data steers precisely in the wrong direction.

The optimizer faithfully minimizes loss on the WRONG token stream. The loss curve gives no signal for these bugs — by design, they live below its resolution.

The top three silent bugs

The debugging checklist. Memorize the symptoms.

#	Bug	Symptom
1	Cross-family template misuse Llama-3 template on Qwen, or vice versa	Role boundaries fuzzy; assistant continues user turn; format drifts
2	EOS mishandling missing / wrong end-of-sequence token	Run-on generation to max length; loss explodes / NaN (Qwen SFT thread)
3	Packing without attention mask examples attend across boundaries	Trains fine, quality subtly degraded vs baseline — the quietest

All three are SILENT: the model trains, loss descends, output is wrong. None crashes the loop.

Bug 1 — Cross-family template misuse

Llama-3 convention

<|begin_of_text|>
<|start_header_id|>system<|end_header_id|>
...
<|eot_id|>

Qwen (ChatML) convention

<|im_start|>system
...
<|im_end|>

NOT interchangeable. A Qwen model trained on Llama-3 tokens sees unfamiliar header tokens where it expects <|im_start|>. Persona & format understanding degrades. Diagnosis: decode an example and read it.

Bug 2 — EOS mishandling

The EOS token tells the model when to stop. Every assistant turn during SFT must end with the family EOS.

Break	What happens
Missing	Stripped/dropped → model never learns to stop
Wrong	Generic `</s>` on a model whose EOS is `<\|im_end\|>`
Wrong place	Appended once per packed seq, not per turn

Public footprint: the HF forums Qwen SFT "loss exploding" thread repeatedly traces here. Fix: apply_chat_template() appends the correct EOS; verify tokenizer.eos_token_id == config.eos_token_id. For ChatML models, <|im_end|> is the EOS.

Bug 3 — Packing without attention mask

Packing multiple examples into one sequence is correct & standard — if and only if you do both:

3a — Mask cross-boundary attention

Example B must not attend to example A. Use document / variable-length attention (FlashAttention cu_seqlens / TRL position_ids).

3b — Mask loss on non-assistant tokens

System/user/role-scaffold are context, not targets. Set labels=-100 via return_assistant_tokens_mask=True.

The quietest bug. Trains fine, loss looks great, output plausible — quality only visibly degraded vs a correctly-packed A/B baseline. Invisible to the loss curve.

apply_chat_template() — always use it

The rule: never hand-concatenate role strings. Always use the model's apply_chat_template(). Prevents ~80% of bugs 1 & 2.

The inspection loop (2 minutes, before every run)

ids = apply_chat_template(conv, tokenize=True, return_assistant_tokens_mask=True)
print(tokenizer.decode(ids)) — read it
Confirm role tokens match the family; EOS appended
Confirm the assistant mask marks only assistant tokens
Then launch the full run

The number of teams that skip this step and ship a mis-tokenized model is the reason this module exists.

Base ≠ no template anymore. Modern base checkpoints (Qwen3 base) ship with ChatML tokens & a default template — the tokenizer is shared across base/instruct variants, and special tokens are real vocab entries with trained embeddings. base/instruct/chat (FT03) is about weights, not template presence. Always check tokenizer.chat_template regardless of checkpoint.

Train your own tokenizer vs reuse the base

Default: reuse the base. Train your own ONLY for grossly inefficient domain tokenization.

The four canonical "train your own" cases

DNA / protein sequences — tiny alphabet, fragmented into single chars
Chemistry SMILES — dense notation split into single tokens
Non-Latin scripts underrepresented in pretraining
Heavy domain code in unusual languages / esoteric DSLs

Symptom in every case: tokens-per-character is abnormally high. Measure before deciding. Training a new tokenizer commits you to continued pretraining, not just SFT.

Middle path (2025): efficient tokenizer adaptation (arXiv:2512.03989) — extend the merge table additively, respecting SentencePiece rules, preserving base embeddings.

Extending the vocab with domain tokens

When you need a handful of domain tokens (a citation marker, a tool-result tag) — not a new tokenizer:

tokenizer.add_tokens([...])
model.resize_token_embeddings(len(tokenizer)) — input emb and lm_head
WARM UP the new rows (short emb-only pass, or mean-init) — everyone forgets this
Re-save the tokenizer (ship with the adapter)

Anti-pattern: extending the vocab without warming up embeddings → new rows are random → noise injected into the loss → model behaves erratically on the new tokens. The warm-up is cheap insurance.

The debugging decision tree

Symptom → bug → fix

Symptom	Bug	Fix
Won't stop generating; runs to max len	2 (EOS)	`apply_chat_template` (append family EOS)
Loss explodes / NaN (Qwen SFT)	2 (EOS/template)	verify `eos_token_id == config.eos_token_id`
Role boundaries fuzzy; asst continues user	1 (cross-family)	use the family's native template
Trains fine, subtly degraded vs baseline	3 (packing)	document attn + mask non-asst loss
New domain tokens behave erratically	vocab	warm up new embeddings
Tool calls malformed / truncated	tool tpl	override `chat_template` explicitly

When in doubt: ENCODE → DECODE → READ. Every silent bug is visible if you decode one example and look at it.

The lab — "The Debugging Checklist"

CPU-only. Three deliberately-broken scripts — one per top bug.

Three broken scripts

Cross-family template — ChatML tokens on a Llama tokenizer
EOS stripped — model trained to never stop
Packing, all-ones mask — loss on every token

Your task

For each: decode the tokens, diagnose the bug, write the fix with apply_chat_template, confirm by re-inspecting.

No GPU, no training loop. The point: these bugs are diagnosable before you launch the run. The inspection loop is the skill. Calibrate on a known-good tokenizer, then break and fix each case.

What you can now do

State the thesis — tokenizers/templates are where most SILENT failures live — and defend it with the three bug classes.
Diagnose cross-family template misuse, EOS mishandling, and packing-without-attention-mask from their symptoms.
Explain why apply_chat_template() is mandatory, and distinguish Llama-3 vs Qwen role-token conventions.
Decide: train a new tokenizer, reuse the base, or extend the vocab — and execute the vocab-extension workflow with the warm-up.
Run the inspection loop — encode, decode, read — before every training run.

Next: FT08 — LoRA & QLoRA (Pillar 2 — PEFT)

The steering wheel is built. The tokens are verified. Now we build the adapter — the cheap, swappable steer.