{ "module": "FT07 — Tokenizers & Chat Templates", "course": "3 — LLM Fine-Tuning Masterclass", "version": "1.0.0", "duration_minutes": 40, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "Name the top three silent tokenizer/template bugs (the FT07 debugging checklist).", "options": [ "Wrong learning rate, missing weight decay, gradient clipping too aggressive.", "Cross-family template misuse, EOS mishandling, packing-without-attention-mask.", "Tokenizer vocab too small, embeddings frozen, optimizer wrong.", "Chat template missing Jinja, FlashAttention disabled, LoRA rank too low." ], "answer_index": 1, "rationale": "The three bug classes are: (1) cross-family template misuse — applying a Llama-3 template to a Qwen model; (2) EOS mishandling — missing or wrong end-of-sequence tokens during SFT; (3) packing-without-attention-mask — concatenating examples without masking cross-boundary attention or loss on non-assistant tokens. All three are SILENT: the model trains, loss descends, output is wrong." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "What are the role-token conventions of the Llama-3 and Qwen families respectively?", "options": [ "Both use ChatML (<|im_start|>/<|im_end|>).", "Llama-3 uses <|begin_of_text|>/<|start_header_id|>/<|eot_id|>; Qwen uses ChatML <|im_start|>/<|im_end|>.", "Llama-3 uses [INST]...[/INST]; Qwen uses <|endoftext|>.", "Both use <|start_header_id|>/<|end_header_id|>." ], "answer_index": 1, "rationale": "Llama-3 uses <|begin_of_text|>, then <|start_header_id|>/<|end_header_id|> to bracket the role name, and <|eot_id|> to end a turn. Qwen uses ChatML: <|im_start|> + role + newline, content, <|im_end|>. These are NOT interchangeable — mixing them is bug 1 (cross-family template misuse)." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "What is the FT07 rule on formatting chat data, and what single anti-pattern does it prevent?", "options": [ "Always hand-concatenate role strings with f-strings; prevents Jinja overhead.", "Always use apply_chat_template(); prevents hand-concatenation (which causes bugs 1 and 2).", "Always write your own Jinja template from scratch; prevents vendor lock-in.", "Always use a fixed system prompt; prevents role-token drift." ], "answer_index": 1, "rationale": "Always use apply_chat_template(). It reads tokenizer.chat_template and produces the family-correct token sequence (right special tokens, right role placement, right EOS). Hand-concatenation drops special tokens, uses wrong EOS, mismatches the family — directly causing bug 1 (cross-family) and bug 2 (EOS). This single rule prevents ~80% of these bugs." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "Your fine-tuned model generates fluent text but never stops — it runs to the max-length limit every time, producing run-on output and hallucinated follow-up turns. Loss looked healthy during training. Which FT07 bug is this, and what is the fix?", "options": [ "Bug 3 (packing); fix the attention mask.", "Bug 2 (EOS mishandling); the assistant turns were trained without EOS. Fix: use apply_chat_template (appends the family EOS) and verify the last token equals tokenizer.eos_token_id.", "Bug 1 (cross-family template); switch to the family's native template.", "Vocab extension without warm-up; warm up the new embeddings." ], "answer_index": 1, "rationale": "Run-on generation to max length is the canonical bug 2 symptom — the model was trained on assistant turns that never ended (EOS stripped or wrong). Loss looks healthy because the model IS learning something. Fix: use apply_chat_template (which appends the family EOS), and confirm the last token id equals tokenizer.eos_token_id. This is the most common single cause of 'my model won't stop generating' support threads." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "You load a Qwen2.5 checkpoint for SFT. Your data pipeline formats conversations by string-formatting: text = f'<|start_header_id|>{role}<|end_header_id|>{content}<|eot_id|>'. The model trains but role boundaries are fuzzy and the assistant sometimes continues the user's turn. Which bug, and the fix?", "options": [ "Bug 2 (EOS); append <|im_end|>.", "Bug 1 (cross-family template misuse); you applied Llama-3 header-id tokens to a Qwen (ChatML) model. Fix: delete the hand-concatenation and use apply_chat_template(), which emits <|im_start|>/<|im_end|> for Qwen.", "Bug 3 (packing); fix the attention mask.", "Vocab mismatch; train a new tokenizer." ], "answer_index": 1, "rationale": "You applied Llama-3's header-id tokens (<|start_header_id|>/<|end_header_id|>/<|eot_id|>) to a Qwen model that expects ChatML (<|im_start|>/<|im_end|>). This is bug 1 (cross-family template misuse), compounded by hand-concatenation (the anti-pattern). Fix: use apply_chat_template() which emits the correct ChatML tokens for Qwen. The role tokens are NOT interchangeable." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "You packed 4 conversations into each 2048-token sequence with a single all-ones attention mask and labels = packed[:]. The model trains and produces plausible output, but an A/B test shows it is quietly worse than a correctly-packed baseline. Which bug, and what are the TWO fixes?", "options": [ "Bug 1; switch template families.", "Bug 2; append EOS.", "Bug 3 (packing without attention mask). Fix (a): use document/variable-length attention (FlashAttention cu_seqlens / TRL position_ids) so example B cannot attend to example A. Fix (b): set labels=-100 on non-assistant tokens (system/user/role-scaffold) via return_assistant_tokens_mask=True.", "Vocab not extended; add domain tokens." ], "answer_index": 2, "rationale": "This is bug 3 with both sub-bugs. (a) Single all-ones attention mask lets conversation B attend to A — spurious cross-conversation correlations. Fix: document/variable-length attention. (b) labels = packed[:] computes loss on every token including user/system — model wastes capacity imitating the user. Fix: labels=-100 at non-assistant positions. The bug is SILENT: loss looks healthy, output plausible, only visible vs a correct baseline." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "You are fine-tuning on a protein-sequence domain. You tokenize a 200-character sample and it fragments into ~600 tokens (3 tokens per character). Per FT07, what does this tell you, and what is the correct response?", "options": [ "Tokenizer is fine; increase max sequence length.", "This is grossly inefficient domain tokenization (one of the four canonical cases). The correct response is to consider training a domain tokenizer — but note this commits you to continued pretraining, not just SFT. Measure tokens-per-character first to confirm.", "Switch to a Llama tokenizer; it handles proteins better.", "Disable BPE and use character-level tokenization for everything." ], "answer_index": 1, "rationale": "Abnormally high tokens-per-character (3:1 on proteins) is the diagnostic for grossly inefficient domain tokenization — one of the four canonical cases (DNA/proteins, SMILES, underrepresented scripts, unusual code). Training a domain tokenizer is justified, BUT it commits you to continued pretraining (the new merge rules have cold embeddings), not just SFT. Measure tokens-per-character first; for most domains the answer is 'reuse the base.'" }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "You add custom tokens ['<|citation|>', '<|ref_start|>'] to your tokenizer, call model.resize_token_embeddings(len(tokenizer)), and immediately launch SFT on your full task. The model trains but behaves erratically whenever it emits a citation token. What did you forget, and what is the fix?", "options": [ "You forgot to switch template families; reload the tokenizer.", "You forgot to WARM UP the new embedding rows (they are randomly initialized). Fix: do a short embedding-only training pass on domain text containing the new tokens (or mean-init the new rows from existing embeddings) before the full SFT.", "You forgot to append EOS; use apply_chat_template.", "You forgot to mask the loss; set labels=-100 on user tokens." ], "answer_index": 1, "rationale": "This is the 'vocab extended without warming up embeddings' anti-pattern. The new token rows are randomly initialized; training the full steering task while they are still random injects noise into the loss. Fix: warm up the new rows first — a short embedding-only pass (freeze the rest of the model) on domain text containing the new tokens, or mean-init from existing embeddings. This is the step everyone forgets." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "Your SFT data includes tool calls. After training, tool-call turns are sometimes truncated and the assistant mask appears to mark tool-response tokens as trainable. Per FT07, what is the issue and the correct action?", "options": [ "Bug 1; switch to a Llama tokenizer.", "The default apply_chat_template mishandles this tool-call scenario. The correct action is to decode several tool-call examples INCLUDING the assistant mask, verify the boundaries, and if wrong, override chat_template with a corrected version (never fall back to hand-concatenation).", "Bug 2; append more EOS tokens.", "Re-train the tokenizer from scratch." ], "answer_index": 1, "rationale": "Tool-calling is the area where the default apply_chat_template is most likely to need intervention. Known issues: multi-turn tool scenarios where a tool response contains text resembling a special token; EOS not appended after a tool-call turn; the assistant mask wrongly unmasking the tool RESPONSE (which is not assistant-generated). The fix is to decode several tool examples including the mask, verify, and override chat_template explicitly if wrong — never hand-concatenate." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why is bug 3 (packing without attention mask) the QUIETEST of the three top bugs, and what does that imply about how to detect it?", "options": [ "It crashes silently; detect via error logs.", "The model trains fine, loss looks great, output is plausible — quality is only visibly degraded vs a correctly-packed A/B baseline. It cannot be detected from the loss curve; it lives below the loss's resolution. Detection requires either an A/B against correct packing or decoding the packed sequence to inspect the attention/loss masks.", "It always produces NaN; detect via NaN checks.", "It only affects inference speed; detect via latency benchmarks." ], "answer_index": 1, "rationale": "Bug 3 is the quietest because NONE of its symptoms surface in the training loop — loss descends, checkpoint produces plausible output. The degradation is only visible against a correctly-packed baseline (A/B) or by inspecting the decoded packed sequence's attention and loss masks. This is why it is the most expensive: teams ship without knowing. The loss curve gives no signal by design; the bug lives below its resolution." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "A colleague claims: 'Base models never have chat templates, so we don't need apply_chat_template — we just train on raw text.' Why is this claim outdated and dangerous, per FT07?", "options": [ "It is correct; base models never have templates.", "It is outdated. Modern base checkpoints (e.g., Qwen3 base) ship WITH chat templates and special tokens in the vocab, because the tokenizer is shared across base/instruct variants. The base/instruct/chat distinction is about WEIGHTS, not template presence. Always check tokenizer.chat_template regardless of checkpoint type, and use apply_chat_template when formatting chat data.", "It is outdated because base models now ship pre-quantized.", "It is correct for Llama but wrong for Qwen." ], "answer_index": 1, "rationale": "The 'base = no template' assumption is outdated. Modern base checkpoints ship with chat templates and the special tokens are real vocab entries with trained embeddings (Qwen3 base includes ChatML tokens). The base/instruct/chat distinction (FT03) is about WEIGHTS, not template presence. Implication: the scaffolding tokens are not cold-started, and you must always check tokenizer.chat_template regardless of which checkpoint you loaded." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does the claim 'fine-tuning steers behavior; it does not teach knowledge' (FT00 thesis) imply that token-level bugs are disproportionately damaging compared to algorithmic or hyperparameter mistakes?", "options": [ "Because tokenizers determine what the model knows.", "Because steering is only as good as the steering wheel — and the steering wheel is the tokenized data. A brilliant algorithm (GRPO, DPO) on mis-tokenized data steers PRECISELY in the wrong direction. The tokens are the last transformation between your clean JSONL and the embedding layer; get them wrong and the entire steering operation is invalidated, no matter how good the optimizer.", "Because token bugs always cause crashes.", "Because hyperparameters matter less than people think." ], "answer_index": 1, "rationale": "The FT00 thesis ('steering not teaching') means the data IS the steering wheel. FT07 is the last gate between 'clean data' and 'correct tokens' — the embedding layer only sees tokens. A brilliant algorithm on mis-tokenized data steers precisely in the wrong direction: the optimizer faithfully minimizes loss on the WRONG token stream. This is why a 2-minute token inspection prevents more failures than a week of hyperparameter tuning." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "Two teams pack SFT data identically. Team A uses FlashAttention document-level attention + assistant-only loss masks. Team B uses a single all-ones attention mask + labels=packed[:]. Both report healthy training loss. Team A's model outperforms Team B's at inference. Why does the training loss NOT reveal the difference, and what does this teach about debugging?", "options": [ "Team B's loss is fake; their logging is broken.", "Both losses descend because both models ARE learning something — Team B's model just learned the WRONG thing (cross-example correlations + user imitation). The loss measures token prediction, not whether the attention boundaries or loss masks are correct. The bug is structurally invisible to the loss curve. Lesson: bugs 1/2/3 are diagnosed by DECODING tokens, never by reading the loss curve.", "Team A used a bigger LoRA rank; that is the difference.", "Team B's learning rate was wrong." ], "answer_index": 1, "rationale": "Both losses descend because both models learn SOMETHING — Team B's model learned the wrong thing: spurious cross-example correlations (no attention mask) and user imitation (loss on user tokens). The loss measures token-prediction error, not whether the attention/loss masks are semantically correct. The bug is structurally invisible to the loss curve. This is why FT07's discipline is encode-decode-read, not read-the-loss-curve." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team decides to 'improve' their Qwen SFT by post-processing apply_chat_template's output: they strip <|im_end|> because 'it looks redundant next to EOS,' then re-append the generic (id 2). Loss explodes to NaN. Trace this to the FT07 root cause and give the fix.", "options": [ "NaN is from learning rate; reduce it 10x.", "This is bug 2 (EOS mishandling). For ChatML-family models <|im_end|> IS the EOS (same token id); stripping it removes the model's stop signal, and re-appending (id 2) is an out-of-distribution token the model was not trained on. The HF forums Qwen loss-exploding thread traces to exactly this. Fix: do NOT post-process template output; use apply_chat_template as-is and verify tokenizer.eos_token_id == config.eos_token_id.", "NaN is from batch size; reduce it.", "Bug 1; switch to Llama." ], "answer_index": 1, "rationale": "This is the Qwen SFT loss-exploding pattern (bug 2). For ChatML models, <|im_end|> and EOS are frequently the SAME token id. Stripping <|im_end|> removes the stop signal; re-appending (id 2, the old T5/BART convention) is out-of-distribution for Qwen. The fix is to NOT post-process apply_chat_template's output and to verify tokenizer.eos_token_id == config.eos_token_id. This is the famous HF forums trace." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "The 2025 'efficient tokenizer adaptation' work (arXiv:2512.03989) offers a middle path between reusing the base tokenizer and training one from scratch. Why must it RESPECT the existing SentencePiece/BPE merge rules rather than naively retraining BPE on the domain corpus?", "options": [ "Because retraining BPE is faster.", "Because naively retraining BPE would INVALIDATE the existing merge table — every existing merge would shift, making the new tokenizer incompatible with the base model's trained embeddings (cold start for the WHOLE vocab). The method instead EXTENDS the merge table additively, adding only high-value domain merges so the base embeddings remain valid and no continued pretraining is needed.", "Because SentencePiece is deprecated.", "Because BPE cannot handle domain text." ], "answer_index": 1, "rationale": "Naively retraining BPE on the domain corpus invalidates the existing merge table — the merge ordering shifts, so the new tokenizer is incompatible with the base model's already-trained embeddings (the WHOLE vocab would cold-start, requiring continued pretraining). The adaptation method instead extends the merge table additively, adding only the high-value domain merges that compress the corpus most, preserving compatibility with existing embeddings. This is the middle path between reuse and retrain." } ] }