Diagrams — Module FT07: Tokenizers & Chat Templates

Module: FT07 — Tokenizers & Chat Templates Diagram count: 5 Tool: Mermaid (primary). Each diagram validated in Mermaid Live Editor.

Diagram 1 — The Top Three Silent Bugs and Their Symptoms

Type: Bug-symptom map Purpose: The debugging checklist on one diagram. Three bug classes, each with its tell-tale symptom. Reading the diagram: Each bug is a node; the symptom is the annotation. All three share the same root property — the model trains, loss descends, output is wrong.

flowchart LR
  subgraph Bugs["THE TOP THREE SILENT TOKENIZER/TEMPLATE BUGS"]
    direction TB
    B1["BUG 1 — Cross-family\ntemplate misuse\n(Llama-3 template on Qwen)"]
    B2["BUG 2 — EOS mishandling\n(missing / wrong EOS\nduring SFT)"]
    B3["BUG 3 — Packing without\nattention mask\n(examples attend across\nboundaries)"]
  end

  B1 -.->|symptom:| S1["Role boundaries fuzzy\nAssistant continues user turn\nFormat drifts at inference"]
  B2 -.->|symptom:| S2["Run-on generation\nLoss explodes / NaN\n(Qwen SFT thread)"]
  B3 -.->|symptom:| S3["Model trains fine\nQuality subtly degraded\nvs correctly-packed baseline"]

  style Bugs fill:#14141f,stroke:rgba(240,128,128,0.6),stroke-width:1.5px,color:#e4e4e8
  style B1 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
  style B2 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
  style B3 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
  style S1 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style S2 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style S3 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0

Diagram 2 — `apply_chat_template()` vs Hand-Concatenation

Type: Side-by-side comparison Purpose: Why the rule "always use apply_chat_template" exists. The hand-concatenated path drops special tokens, uses wrong EOS, mismatches the family. Reading the diagram: Two parallel paths from the same conversation. The template path produces correct tokens; the hand-concat path produces bug 1 and bug 2.

flowchart TD
  Conv["Conversation\n[system, user, assistant]"]
  Conv --> Tpl["apply_chat_template()\nread tokenizer.chat_template"]
  Conv --> Hand["HAND-CONCATENATION\nf'User: {x}\nAssistant: {y}'"]

  Tpl --> Correct["Correct token sequence\nfamily role tokens\n+ correct EOS\n+ assistant mask"]
  Correct --> Train["SFT trains correctly"]

  Hand --> Wrong["Missing special tokens\nwrong / no EOS\nmismatched family convention"]
  Wrong --> Bug1["BUG 1 — cross-family\n(role boundaries fuzzy)"]
  Wrong --> Bug2["BUG 2 — EOS mishandling\n(run-on / loss explode)"]
  Bug1 --> Fail["Silent failure"]
  Bug2 --> Fail

  style Conv fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Tpl fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Hand fill:#14141f,stroke:rgba(240,128,128,0.6),color:#f08080
  style Correct fill:#08080c,stroke:rgba(94,234,212,0.5),color:#82e0aa
  style Wrong fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
  style Train fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style Bug1 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
  style Bug2 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
  style Fail fill:#14141f,stroke:rgba(240,128,128,0.6),color:#f08080

Diagram 3 — The Tokenization Inspection Loop

Type: Loop / process Purpose: The two-minute discipline that prevents bugs 1, 2, and 3 before they cost GPU-hours. Reading the diagram: A loop. Encode one conversation, decode it, read it, check the assistant mask. Only launch the full run when all checks pass.

flowchart TD
  Start["Take ONE conversation\nfrom your dataset"]
  Start --> Encode["ids = apply_chat_template(\n  conv, tokenize=True,\n  return_assistant_tokens_mask=True)"]
  Encode --> Decode["print(tokenizer.decode(ids))"]
  Decode --> Q1{"Read it.\nRole tokens match\nthe family?\nEOS appended?"}
  Q1 -->|"No"| Fix1["Set correct chat_template\n(NEVER hand-concatenate)"]
  Fix1 --> Encode
  Q1 -->|"Yes"| Q2{"Assistant mask\nmarks ONLY\nassistant tokens?"}
  Q2 -->|"No"| Fix2["Fix the loss mask\n(system/user/tool masked)"]
  Fix2 --> Encode
  Q2 -->|"Yes"| Launch["LAUNCH the full run"]
  Launch --> Done["Train with confidence"]

  style Start fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Encode fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Decode fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Launch fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style Done fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style Q1 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style Q2 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style Fix1 fill:#08080c,stroke:rgba(240,168,104,0.6),color:#f0a868
  style Fix2 fill:#08080c,stroke:rgba(240,168,104,0.6),color:#f0a868

Diagram 4 — The Vocab-Extension Workflow

Type: Sequential workflow with the warm-up gate Purpose: How to add domain tokens correctly — and the warm-up step everyone forgets. Reading the diagram: Left to right. Add tokens → resize embeddings AND lm_head → warm up new rows → re-save tokenizer. Skipping the warm-up produces noise.

flowchart LR
  Add["1. ADD TOKENS\ntokenizer.add_tokens(\n['<|citation|>', ...])"]
  Resize["2. RESIZE\nmodel.resize_token_\nembeddings(len(tok))\n(input emb + lm_head)"]
  Warm["3. WARM UP\nnew rows\n(short emb-only pass,\nor mean-init)"]
  Save["4. RE-SAVE\ntokenizer (ship\nwith adapter)"]

  Add --> Resize
  Resize --> Warm
  Warm --> Save
  Save --> Train["SFT on full task"]

  Warm -.skip (anti-pattern).-> Skip["Random rows inject noise\ninto the loss\nmodel behaves erratically\non new tokens"]
  Skip --> Bad["Silent degradation"]

  style Add fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Resize fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Warm fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Save fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Train fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style Skip fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
  style Bad fill:#14141f,stroke:rgba(240,128,128,0.6),color:#f08080

Diagram 5 — The Debugging Decision Tree (Symptom → Bug → Fix)

Type: Decision tree Purpose: The whole module on one flowchart. Start from the symptom, land on the fix. Reading the diagram: Top-down. Match your symptom, follow to the bug, apply the fix.

flowchart TD
  Start["Symptom observed\nat training or inference"]
  Start --> S1{"Model won't stop\ngenerating; runs to\nmax length?"}
  S1 -->|"Yes"| B2a["BUG 2 — EOS missing/wrong\nFix: apply_chat_template\n(append family EOS)"]
  S1 -->|"No"| S2{"Loss explodes / NaN\n(Qwen SFT)?"}
  S2 -->|"Yes"| B2b["BUG 2 — template/EOS misconfig\nFix: verify tokenizer.eos_token_id\n== config.eos_token_id"]
  S2 -->|"No"| S3{"Role boundaries fuzzy;\nassistant continues\nuser turn?"}
  S3 -->|"Yes"| B1["BUG 1 — cross-family template\nFix: use family native template"]
  S3 -->|"No"| S4{"Trains fine, quality\nsubtly degraded vs\nbaseline?"}
  S4 -->|"Yes"| B3["BUG 3 — packing w/o attention mask\nFix: document/variable-length attn\n+ mask non-assistant loss"]
  S4 -->|"No"| S5{"New domain tokens\nbehave erratically?"}
  S5 -->|"Yes"| B4["Vocab extended, no warm-up\nFix: warm up new embeddings"]
  S5 -->|"No"| Decode["ENCODE → DECODE → READ\none example; the bug is\nvisible if you look"]

  style Start fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style S1 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style S2 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style S3 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style S4 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style S5 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style B1 fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style B2a fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style B2b fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style B3 fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style B4 fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style Decode fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8

Validation notes

All five diagrams use the course design system colors: #14141f panel fill, #5eead4 (--accent) for the correct path / primary nodes, rgba(94,234,212,0.4) for decision diamonds, #f08080 (--danger) for bug / anti-pattern nodes, #f0a868 (--warn) for fix / intervention nodes, #82e0aa (--ok) for the success path, #e4e4e8 / #9494a0 for text.
Paste each into Mermaid Live Editor to render. All use stable Mermaid syntax (flowchart, subgraph, dashed -.label.-> links) supported in current Mermaid (v10.4+).
For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live, inlined into reveal.js.

# Diagrams — Module FT07: Tokenizers & Chat Templates **Module**: FT07 — Tokenizers & Chat Templates **Diagram count**: 5 **Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live). --- ## Diagram 1 — The Top Three Silent Bugs and Their Symptoms **Type**: Bug-symptom map **Purpose**: The debugging checklist on one diagram. Three bug classes, each with its tell-tale symptom. **Reading the diagram**: Each bug is a node; the symptom is the annotation. All three share the same root property — the model trains, loss descends, output is wrong. ```mermaid flowchart LR subgraph Bugs["THE TOP THREE SILENT TOKENIZER/TEMPLATE BUGS"] direction TB B1["BUG 1 — Cross-family\ntemplate misuse\n(Llama-3 template on Qwen)"] B2["BUG 2 — EOS mishandling\n(missing / wrong EOS\nduring SFT)"] B3["BUG 3 — Packing without\nattention mask\n(examples attend across\nboundaries)"] end B1 -.->|symptom:| S1["Role boundaries fuzzy\nAssistant continues user turn\nFormat drifts at inference"] B2 -.->|symptom:| S2["Run-on generation\nLoss explodes / NaN\n(Qwen SFT thread)"] B3 -.->|symptom:| S3["Model trains fine\nQuality subtly degraded\nvs correctly-packed baseline"] style Bugs fill:#14141f,stroke:rgba(240,128,128,0.6),stroke-width:1.5px,color:#e4e4e8 style B1 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080 style B2 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080 style B3 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080 style S1 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0 style S2 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0 style S3 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0 ``` --- ## Diagram 2 — `apply_chat_template()` vs Hand-Concatenation **Type**: Side-by-side comparison **Purpose**: Why the rule "always use apply_chat_template" exists. The hand-concatenated path drops special tokens, uses wrong EOS, mismatches the family. **Reading the diagram**: Two parallel paths from the same conversation. The template path produces correct tokens; the hand-concat path produces bug 1 and bug 2. ```mermaid flowchart TD Conv["Conversation\n[system, user, assistant]"] Conv --> Tpl["apply_chat_template()\nread tokenizer.chat_template"] Conv --> Hand["HAND-CONCATENATION\nf'User: {x}\nAssistant: {y}'"] Tpl --> Correct["Correct token sequence\nfamily role tokens\n+ correct EOS\n+ assistant mask"] Correct --> Train["SFT trains correctly"] Hand --> Wrong["Missing special tokens\nwrong / no EOS\nmismatched family convention"] Wrong --> Bug1["BUG 1 — cross-family\n(role boundaries fuzzy)"] Wrong --> Bug2["BUG 2 — EOS mishandling\n(run-on / loss explode)"] Bug1 --> Fail["Silent failure"] Bug2 --> Fail style Conv fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8 style Tpl fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Hand fill:#14141f,stroke:rgba(240,128,128,0.6),color:#f08080 style Correct fill:#08080c,stroke:rgba(94,234,212,0.5),color:#82e0aa style Wrong fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080 style Train fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa style Bug1 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080 style Bug2 fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080 style Fail fill:#14141f,stroke:rgba(240,128,128,0.6),color:#f08080 ``` --- ## Diagram 3 — The Tokenization Inspection Loop **Type**: Loop / process **Purpose**: The two-minute discipline that prevents bugs 1, 2, and 3 before they cost GPU-hours. **Reading the diagram**: A loop. Encode one conversation, decode it, read it, check the assistant mask. Only launch the full run when all checks pass. ```mermaid flowchart TD Start["Take ONE conversation\nfrom your dataset"] Start --> Encode["ids = apply_chat_template(\n conv, tokenize=True,\n return_assistant_tokens_mask=True)"] Encode --> Decode["print(tokenizer.decode(ids))"] Decode --> Q1{"Read it.\nRole tokens match\nthe family?\nEOS appended?"} Q1 -->|"No"| Fix1["Set correct chat_template\n(NEVER hand-concatenate)"] Fix1 --> Encode Q1 -->|"Yes"| Q2{"Assistant mask\nmarks ONLY\nassistant tokens?"} Q2 -->|"No"| Fix2["Fix the loss mask\n(system/user/tool masked)"] Fix2 --> Encode Q2 -->|"Yes"| Launch["LAUNCH the full run"] Launch --> Done["Train with confidence"] style Start fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8 style Encode fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Decode fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Launch fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa style Done fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa style Q1 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8 style Q2 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8 style Fix1 fill:#08080c,stroke:rgba(240,168,104,0.6),color:#f0a868 style Fix2 fill:#08080c,stroke:rgba(240,168,104,0.6),color:#f0a868 ``` --- ## Diagram 4 — The Vocab-Extension Workflow **Type**: Sequential workflow with the warm-up gate **Purpose**: How to add domain tokens correctly — and the warm-up step everyone forgets. **Reading the diagram**: Left to right. Add tokens → resize embeddings AND lm_head → warm up new rows → re-save tokenizer. Skipping the warm-up produces noise. ```mermaid flowchart LR Add["1. ADD TOKENS\ntokenizer.add_tokens(\n['<|citation|>', ...])"] Resize["2. RESIZE\nmodel.resize_token_\nembeddings(len(tok))\n(input emb + lm_head)"] Warm["3. WARM UP\nnew rows\n(short emb-only pass,\nor mean-init)"] Save["4. RE-SAVE\ntokenizer (ship\nwith adapter)"] Add --> Resize Resize --> Warm Warm --> Save Save --> Train["SFT on full task"] Warm -.skip (anti-pattern).-> Skip["Random rows inject noise\ninto the loss\nmodel behaves erratically\non new tokens"] Skip --> Bad["Silent degradation"] style Add fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Resize fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Warm fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Save fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Train fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa style Skip fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080 style Bad fill:#14141f,stroke:rgba(240,128,128,0.6),color:#f08080 ``` --- ## Diagram 5 — The Debugging Decision Tree (Symptom → Bug → Fix) **Type**: Decision tree **Purpose**: The whole module on one flowchart. Start from the symptom, land on the fix. **Reading the diagram**: Top-down. Match your symptom, follow to the bug, apply the fix. ```mermaid flowchart TD Start["Symptom observed\nat training or inference"] Start --> S1{"Model won't stop\ngenerating; runs to\nmax length?"} S1 -->|"Yes"| B2a["BUG 2 — EOS missing/wrong\nFix: apply_chat_template\n(append family EOS)"] S1 -->|"No"| S2{"Loss explodes / NaN\n(Qwen SFT)?"} S2 -->|"Yes"| B2b["BUG 2 — template/EOS misconfig\nFix: verify tokenizer.eos_token_id\n== config.eos_token_id"] S2 -->|"No"| S3{"Role boundaries fuzzy;\nassistant continues\nuser turn?"} S3 -->|"Yes"| B1["BUG 1 — cross-family template\nFix: use family native template"] S3 -->|"No"| S4{"Trains fine, quality\nsubtly degraded vs\nbaseline?"} S4 -->|"Yes"| B3["BUG 3 — packing w/o attention mask\nFix: document/variable-length attn\n+ mask non-assistant loss"] S4 -->|"No"| S5{"New domain tokens\nbehave erratically?"} S5 -->|"Yes"| B4["Vocab extended, no warm-up\nFix: warm up new embeddings"] S5 -->|"No"| Decode["ENCODE → DECODE → READ\none example; the bug is\nvisible if you look"] style Start fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8 style S1 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8 style S2 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8 style S3 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8 style S4 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8 style S5 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8 style B1 fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa style B2a fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa style B2b fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa style B3 fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa style B4 fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa style Decode fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 ``` --- ## Validation notes - All five diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` (`--accent`) for the correct path / primary nodes, `rgba(94,234,212,0.4)` for decision diamonds, `#f08080` (`--danger`) for bug / anti-pattern nodes, `#f0a868` (`--warn`) for fix / intervention nodes, `#82e0aa` (`--ok`) for the success path, `#e4e4e8` / `#9494a0` for text. - Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`flowchart`, `subgraph`, dashed `-.label.->` links) supported in current Mermaid (v10.4+). - For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live, inlined into reveal.js.