Lab Specification — Module FT07: Tokenizers & Chat Templates

# conversation.py
CONVERSATION = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "What is 2 + 2?"},
    {"role": "assistant", "content": "Four."},
    {"role": "user", "content": "And 3 + 3?"},
    {"role": "assistant", "content": "Six."},
]

# inspect.py
from transformers import AutoTokenizer

def inspect_example(tokenizer_name, conversation, label=""):
    """The FT07 inspection loop: encode, decode, read, check the mask."""
    tok = AutoTokenizer.from_pretrained(tokenizer_name)
    print(f"\n{'='*70}")
    print(f"TOKENIZER: {tokenizer_name}   |   {label}")
    print('='*70)

    # 1. ENCODE using apply_chat_template with the assistant mask
    ids = tok.apply_chat_template(
        conversation,
        tokenize=True,
        add_generation_prompt=False,
        return_assistant_tokens_mask=True,   # requires transformers>=4.42
    )
    mask = ids.pop("assistant_masks") if isinstance(ids, dict) else None
    ids = ids["input_ids"] if isinstance(ids, dict) else ids

    # 2. DECODE the full sequence so we can READ it
    decoded = tok.decode(ids)
    print("\n--- DECODED (what the model sees) ---")
    print(decoded)

    # 3. CHECK the EOS
    last_id = ids[-1]
    eos_id = tok.eos_token_id
    print(f"\n--- EOS CHECK ---")
    print(f"Last token id:     {last_id}  ({tok.convert_ids_to_tokens([last_id])})")
    print(f"tokenizer.eos_id:  {eos_id}  ({tok.eos_token})")
    print(f"EOS present at end? {last_id == eos_id}")

    # 4. CHECK the assistant mask
    if mask is not None:
        masked_count = sum(1 for m in mask if m == 1)
        print(f"\n--- ASSISTANT MASK ---")
        print(f"Total tokens: {len(ids)}, loss-eligible (assistant): {masked_count}")
        # Show which tokens are unmasked (first 12 for brevity)
        print("Token-by-token (1 = train on this, 0 = mask):")
        for i, (tid, m) in enumerate(zip(ids, mask)):
            if i < 16:
                tok_str = tok.convert_ids_to_tokens([tid])[0]
                mark = "TRAIN" if m == 1 else "mask "
                print(f"  [{i:2d}] {mark}  id={tid:6d}  {tok_str!r}")
            elif i == 16:
                print(f"  ... ({len(ids) - 16} more)")
    return ids, mask

python -c "from inspect import inspect_example; from conversation import CONVERSATION; inspect_example('HuggingFaceTB/SmolLM2-135M-Instruct', CONVERSATION, 'CALIBRATION (good)')"

# broken_1_cross_family.py
from transformers import AutoTokenizer
from conversation import CONVERSATION

# BUG: We load a LLAMA tokenizer but format with a hand-written
# ChatML-style string intended for Qwen. The model will see tokens
# it does not understand at the role boundaries.
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")

# Hand-concatenation, ChatML-flavored, applied to a Llama tokenizer.
text = ""
for msg in CONVERSATION:
    text += f"<|im_start|>{msg['role']}\n{msg['content']}<|im_end|>\n"

ids = tok(text, return_tensors="pt")["input_ids"]
print("Decoded:", tok.decode(ids[0]))
print("Last 3 tokens:", tok.convert_ids_to_tokens(ids[0][-3:].tolist()))

# broken_2_eos.py
from transformers import AutoTokenizer
from conversation import CONVERSATION

tok = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")

# BUG: We strip trailing whitespace/special tokens from the template
# output AND we never append EOS. The model will learn to never stop.
ids = tok.apply_chat_template(
    CONVERSATION,
    tokenize=True,
    add_generation_prompt=False,
    truncation=True,
)
# The anti-pattern: manually truncating and "cleaning" the template output
# in a way that drops the trailing EOS.
ids = [i for i in ids if i not in {tok.eos_token_id, tok.convert_tokens_to_ids("<|im_end|>")}]

print("Last 3 tokens:", tok.convert_ids_to_tokens(ids[-3:]))
print("EOS in sequence?", tok.eos_token_id in ids)
print("Decoded (end):", repr(tok.decode(ids[-8:])))

# broken_3_packing.py
from transformers import AutoTokenizer
from conversation import CONVERSATION

tok = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")

# Two unrelated conversations.
CONV_A = [
    {"role": "user", "content": "Capital of France?"},
    {"role": "assistant", "content": "Paris."},
]
CONV_B = [
    {"role": "user", "content": "Best language for systems programming?"},
    {"role": "assistant", "content": "Rust."},
]

# BUG: We concatenate the two tokenized examples into one packed sequence
# with a SINGLE all-ones attention mask and a SINGLE all-ones loss mask.
# Example B attends to Example A, and we compute loss on the user/system
# tokens too.
ids_a = tok.apply_chat_template(CONV_A, tokenize=True)
ids_b = tok.apply_chat_template(CONV_B, tokenize=True)
packed = ids_a + ids_b
attention_mask = [1] * len(packed)          # BUG 3a: no cross-boundary masking
labels = packed[:]                           # BUG 3b: loss on every token (incl. user/system)

print("Packed length:", len(packed))
print("Attention mask (should have a boundary 0/1 split):", "all ones =", set(attention_mask))
print("Loss on user tokens? ", "YES (bug)" )
print("Boundary between A and B at index:", len(ids_a))

# Lab Specification — Module FT07: Tokenizers & Chat Templates

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT07 — Tokenizers & Chat Templates
**Duration**: 45–60 minutes (diagnosis + fix; CPU-only)
**Environment**: Python 3.10+. CPU is sufficient — the models are tiny (a sub-100M param toy or a small real checkpoint like `HuggingFaceTB/SmolLM2-135M`) and the lab is about *tokenization inspection*, not GPU throughput.

> **Setup (one time):**
> ```bash
> pip install "transformers>=4.46" torch
> ```
> No GPU. No TRL training loop. We simulate training failure by *inspecting the tokenized data* — which is exactly how you diagnose these bugs in production before you waste GPU-hours.

---

## Learning objectives

By the end of this lab you will have:

1. **Diagnosed the three canonical silent-bug classes** from their symptoms — cross-family template misuse, EOS mishandling, packing-without-attention-mask — *without running a real training loop*, by inspecting the tokens the way you would in production.
2. **Applied the FT07 inspection loop** (encode → decode → read) to each broken script and identified the specific misconfiguration.
3. **Written the corrected version of each script** and confirmed the fix by re-inspecting the decoded tokens and the assistant-token mask.
4. **Internalized the discipline**: every silent tokenizer bug is visible if you decode one example and look at it. The fix is almost always "use `apply_chat_template` correctly."

This lab is deliberately CPU-only and training-loop-free. The point is that the bugs are diagnosable *before* you launch the run — the inspection loop is the skill.

---

## The lab data

Every script uses the same three-turn conversation. Save it as `conversation.py` and import it in each exercise.

```python
# conversation.py
CONVERSATION = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "What is 2 + 2?"},
    {"role": "assistant", "content": "Four."},
    {"role": "user", "content": "And 3 + 3?"},
    {"role": "assistant", "content": "Six."},
]
```

A real training set would have thousands of these. We use one so you can read every token.

---

## The diagnostic helper

Save this as `inspect.py`. It is the only tool you need for the whole lab — it is the embodiment of the FT07 inspection loop.

```python
# inspect.py
from transformers import AutoTokenizer

def inspect_example(tokenizer_name, conversation, label=""):
    """The FT07 inspection loop: encode, decode, read, check the mask."""
    tok = AutoTokenizer.from_pretrained(tokenizer_name)
    print(f"\n{'='*70}")
    print(f"TOKENIZER: {tokenizer_name}   |   {label}")
    print('='*70)

    # 1. ENCODE using apply_chat_template with the assistant mask
    ids = tok.apply_chat_template(
        conversation,
        tokenize=True,
        add_generation_prompt=False,
        return_assistant_tokens_mask=True,   # requires transformers>=4.42
    )
    mask = ids.pop("assistant_masks") if isinstance(ids, dict) else None
    ids = ids["input_ids"] if isinstance(ids, dict) else ids

    # 2. DECODE the full sequence so we can READ it
    decoded = tok.decode(ids)
    print("\n--- DECODED (what the model sees) ---")
    print(decoded)

    # 3. CHECK the EOS
    last_id = ids[-1]
    eos_id = tok.eos_token_id
    print(f"\n--- EOS CHECK ---")
    print(f"Last token id:     {last_id}  ({tok.convert_ids_to_tokens([last_id])})")
    print(f"tokenizer.eos_id:  {eos_id}  ({tok.eos_token})")
    print(f"EOS present at end? {last_id == eos_id}")

    # 4. CHECK the assistant mask
    if mask is not None:
        masked_count = sum(1 for m in mask if m == 1)
        print(f"\n--- ASSISTANT MASK ---")
        print(f"Total tokens: {len(ids)}, loss-eligible (assistant): {masked_count}")
        # Show which tokens are unmasked (first 12 for brevity)
        print("Token-by-token (1 = train on this, 0 = mask):")
        for i, (tid, m) in enumerate(zip(ids, mask)):
            if i < 16:
                tok_str = tok.convert_ids_to_tokens([tid])[0]
                mark = "TRAIN" if m == 1 else "mask "
                print(f"  [{i:2d}] {mark}  id={tid:6d}  {tok_str!r}")
            elif i == 16:
                print(f"  ... ({len(ids) - 16} more)")
    return ids, mask
```

Run `inspect.py` against a known-good tokenizer first to calibrate your eye:

```bash
python -c "from inspect import inspect_example; from conversation import CONVERSATION; inspect_example('HuggingFaceTB/SmolLM2-135M-Instruct', CONVERSATION, 'CALIBRATION (good)')"
```

You should see: the `<|im_start|>`-style role tokens, EOS at the end, and the assistant mask marking only the assistant turns as trainable. That is what *correct* looks like. Memorize it.

---

## Exercise 1 — Cross-family template misuse (Bug 1)

### The broken script

```python
# broken_1_cross_family.py
from transformers import AutoTokenizer
from conversation import CONVERSATION

# BUG: We load a LLAMA tokenizer but format with a hand-written
# ChatML-style string intended for Qwen. The model will see tokens
# it does not understand at the role boundaries.
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")

# Hand-concatenation, ChatML-flavored, applied to a Llama tokenizer.
text = ""
for msg in CONVERSATION:
    text += f"<|im_start|>{msg['role']}\n{msg['content']}<|im_end|>\n"

ids = tok(text, return_tensors="pt")["input_ids"]
print("Decoded:", tok.decode(ids[0]))
print("Last 3 tokens:", tok.convert_ids_to_tokens(ids[0][-3:].tolist()))
```

### Your tasks

1. **Run** `broken_1_cross_family.py`. (If you do not have Llama-3 gated access, substitute any Llama-family instruct tokenizer you can load, or read the expected output below.)
2. **Diagnose**: Which of the three FT07 bugs is this? What specific misconfigurations are present? (There are at least two.)
3. **Inspect with the helper**: Run `inspect_example("meta-llama/Llama-3.2-1B-Instruct", CONVERSATION, "via apply_chat_template (correct)")` and compare the role tokens to the broken version.
4. **Write `fixed_1_cross_family.py`**: Re-tokenize using `apply_chat_template()` and confirm the role tokens match the Llama family (`<|begin_of_text|>`, `<|start_header_id|>`, `<|eot_id|>`).

### Expected diagnosis (what to look for)

- The broken script uses `<|im_start|>` / `<|im_end|>` — *Qwen's ChatML tokens* — on a Llama tokenizer. These are not Llama's role tokens. If they are even in the Llama vocab, they map to unrelated or `<unk>` embeddings; if not, they fragment into garbage BPE pieces. Either way the model's role-boundary understanding is corrupted.
- It also hand-concatenates instead of using `apply_chat_template()` — the root anti-pattern.
- The fix: delete the hand-concatenation; call `tok.apply_chat_template(CONVERSATION, tokenize=True)` and let the Llama template produce its native `<|start_header_id|>system<|end_header_id|>...<|eot_id|>` scaffolding.

---

## Exercise 2 — EOS mishandling (Bug 2)

### The broken script

```python
# broken_2_eos.py
from transformers import AutoTokenizer
from conversation import CONVERSATION

tok = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")

# BUG: We strip trailing whitespace/special tokens from the template
# output AND we never append EOS. The model will learn to never stop.
ids = tok.apply_chat_template(
    CONVERSATION,
    tokenize=True,
    add_generation_prompt=False,
    truncation=True,
)
# The anti-pattern: manually truncating and "cleaning" the template output
# in a way that drops the trailing EOS.
ids = [i for i in ids if i not in {tok.eos_token_id, tok.convert_tokens_to_ids("<|im_end|>")}]

print("Last 3 tokens:", tok.convert_ids_to_tokens(ids[-3:]))
print("EOS in sequence?", tok.eos_token_id in ids)
print("Decoded (end):", repr(tok.decode(ids[-8:])))
```

### Your tasks

1. **Run** `broken_2_eos.py`.
2. **Diagnose**: What symptom will this produce at inference? (Relate it to the FT07 debugging decision tree.)
3. **Inspect with the helper**: Run `inspect_example("HuggingFaceTB/SmolLM2-135M-Instruct", CONVERSATION, "EOS check")` and confirm the correct version *does* end in EOS.
4. **Write `fixed_2_eos.py`**: Re-tokenize *without* the EOS-stripping filter. Confirm the last token equals `tok.eos_token_id`. Then answer: what is the relationship between `<|im_end|>` and EOS for the ChatML family? (Hint: for ChatML models they are often the same token id — verify it.)

### Expected diagnosis (what to look for)

- The broken script actively removes the EOS / `<|im_end|>` token from the template output. The model is trained on assistant turns that *never end*. At inference it generates until max-length — the "won't stop generating" symptom, bug 2.
- The fix is to *not* post-process the template output. `apply_chat_template` appends the correct EOS for the family. Removing it is the bug.
- Bonus insight: for ChatML-family models (Qwen, SmolLM2-Instruct), `<|im_end|>` and EOS are frequently the *same token id* (the model's `eos_token_id` points at `<|im_end|>`). Confirm with `tok.eos_token`, `tok.eos_token_id`, and `tok.convert_tokens_to_ids("<|im_end|>")`.

---

## Exercise 3 — Packing without attention mask (Bug 3)

### The broken script

```python
# broken_3_packing.py
from transformers import AutoTokenizer
from conversation import CONVERSATION

tok = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")

# Two unrelated conversations.
CONV_A = [
    {"role": "user", "content": "Capital of France?"},
    {"role": "assistant", "content": "Paris."},
]
CONV_B = [
    {"role": "user", "content": "Best language for systems programming?"},
    {"role": "assistant", "content": "Rust."},
]

# BUG: We concatenate the two tokenized examples into one packed sequence
# with a SINGLE all-ones attention mask and a SINGLE all-ones loss mask.
# Example B attends to Example A, and we compute loss on the user/system
# tokens too.
ids_a = tok.apply_chat_template(CONV_A, tokenize=True)
ids_b = tok.apply_chat_template(CONV_B, tokenize=True)
packed = ids_a + ids_b
attention_mask = [1] * len(packed)          # BUG 3a: no cross-boundary masking
labels = packed[:]                           # BUG 3b: loss on every token (incl. user/system)

print("Packed length:", len(packed))
print("Attention mask (should have a boundary 0/1 split):", "all ones =", set(attention_mask))
print("Loss on user tokens? ", "YES (bug)" )
print("Boundary between A and B at index:", len(ids_a))
```

### Your tasks

1. **Run** `broken_3_packing.py`.
2. **Diagnose**: Why does this train "fine" (loss descends) but degrade quality? Name the two distinct bugs inside bug 3.
3. **Write `fixed_3_packing.py`**: Produce the *correct* packed representation. For a CPU-only lab you do not need real FlashAttention — simulate the two fixes:
   - **Attention mask**: produce a structure that records the document boundaries (a list of `(start, end)` spans, or per-token document ids). In production this becomes FlashAttention's `cu_seqlens` / TRL's `position_ids` packing.
   - **Loss mask**: produce a `labels` list where every non-assistant token is `-100` (the PyTorch `CrossEntropyLoss` ignore index). Use `return_assistant_tokens_mask=True` from `apply_chat_template` to get this per-example, then compose.
4. **Confirm**: In your fixed version, verify that (a) the two examples are tagged as belonging to different documents, and (b) the `labels` list has `-100` at every system/user/role-scaffold position.

### Expected diagnosis (what to look for)

- Bug 3a: the single all-ones `attention_mask` lets tokens of conversation B attend to conversation A. The model learns spurious cross-conversation correlations. In production the fix is document-level / variable-length attention (FlashAttention `cu_seqlens`, or TRL's packing with `position_ids`).
- Bug 3b: `labels = packed[:]` trains the model to predict *every* token, including system and user turns. The model spends capacity learning to imitate the user. The fix is `labels` with `-100` everywhere except assistant-generated tokens, derived from the `assistant_masks` returned by `apply_chat_template(return_assistant_tokens_mask=True)`.
- The combined symptom: training loss looks healthy, the checkpoint produces *plausible* output, but it is silently worse than a correctly-packed baseline. This is the quietest and most expensive of the three bugs.

---

## Deliverables

Submit `ft07-lab-report.md` containing, for each of the three exercises:

- [ ] **The broken script's output** (the decoded tokens, the EOS/mask check).
- [ ] **Your diagnosis**: which bug, the specific misconfiguration(s), and the symptom a user would observe at inference. Reference the FT07 debugging decision tree.
- [ ] **Your fixed script** (`fixed_1_*.py`, `fixed_2_*.py`, `fixed_3_*.py`) — runnable Python.
- [ ] **The fixed script's output** confirming the fix (correct role tokens / EOS present / loss mask correct).
- [ ] A 2–3 sentence reflection per exercise: why is this bug *silent* (the model trains, loss descends, no crash)?

Plus a final section: **The Inspection Discipline** — in 4–6 sentences, explain why the "encode → decode → read" loop catches all three bugs before a GPU-hour is spent, and why teams that skip it are the ones posting "my model won't stop generating" on the forums.

---

## Solution key

These are *defensible* diagnoses and fixes, not the only possible wording. A report that names the correct bug, cites the right symptom, and provides a working fix passes.

### Exercise 1 — Cross-family template misuse

- **Bug**: FT07 bug 1 (cross-family template misuse). Two misconfigs: (a) the script uses Qwen/ChatML tokens (`<|im_start|>`, `<|im_end|>`) on a Llama tokenizer, which does not recognize them as role boundaries; (b) it hand-concatenates instead of using `apply_chat_template()`.
- **Symptom**: role boundaries are fuzzy; the assistant may continue the user's turn; format drifts at inference.
- **Fix**: delete the hand-concatenation; call `tok.apply_chat_template(CONVERSATION, tokenize=True)`. The Llama template produces `<|begin_of_text|><|start_header_id|>system<|end_header_id|>...<|eot_id|>`. Confirm by decoding and checking for `<|start_header_id|>`.

### Exercise 2 — EOS mishandling

- **Bug**: FT07 bug 2 (EOS mishandling). The script strips the EOS / `<|im_end|>` token from the template output. The model is trained on turns that never end.
- **Symptom**: at inference the model generates until max-length — run-on text, hallucinated follow-up turns. This is the canonical "my model won't stop generating" trace. (The Qwen SFT loss-exploding variant is the same root cause with a different severity.)
- **Fix**: do not post-process the template output. Re-tokenize without the filter; confirm the last token id equals `tok.eos_token_id`. Note that for ChatML-family models, `<|im_end|>` and EOS are typically the same token id.

### Exercise 3 — Packing without attention mask

- **Bug**: FT07 bug 3 (packing without attention masking). Two sub-bugs: (a) single all-ones attention mask lets conversation B attend to conversation A; (b) `labels = packed[:]` computes loss on system/user tokens.
- **Symptom**: the model trains fine and produces plausible output but is silently worse than a correctly-packed baseline — the quietest degradation. Hard to detect without an A/B against a correct packing baseline.
- **Fix**: (a) record document boundaries (per-token document id list or `(start,end)` spans; in production this becomes FlashAttention `cu_seqlens` or TRL `position_ids` packing); (b) build `labels` with `-100` at every non-assistant position using `apply_chat_template(..., return_assistant_tokens_mask=True)`. Verify the `labels` list is `-100` at the system/user positions.

### Reflection (model answer)

Bugs 1, 2, and 3 are *silent* because none of them crashes the training loop. The loss descends on all three — the model is learning *something*, just not the right thing. The inspection discipline (encode → decode → read) catches them because every one of these bugs is *visible in the decoded token stream*: the wrong role tokens (bug 1), the missing EOS (bug 2), the boundary-less packed sequence and the all-on loss mask (bug 3). Teams that skip the inspection loop are the ones posting on the forums because the loss curve gives them no signal — by design, these bugs live below the loss curve's resolution. The two-minute decode is the cheapest insurance in the whole pipeline.

---

## Stretch goals

1. **The tool-call edge case.** Build a conversation that includes a tool call and a tool response. Run it through `apply_chat_template` for a model that supports tool calling (e.g., `Qwen/Qwen2.5-1.5B-Instruct` or `Herm`). Decode it and inspect the assistant mask. Is the tool *response* correctly masked out of the loss? (If not, you have found the FT07 tool-template edge case — fix it by overriding `chat_template`.)
2. **The vocab-extension warm-up.** Add a custom token `<|citation|>` to `SmolLM2-135M-Instruct`, resize the embeddings, and inspect the new row's norm before and after a short warm-up pass on toy data. Show that the unwarm'd row has a random norm while the warm'd row converges toward the mean embedding norm.
3. **Measure tokens-per-character.** Take a DNA sequence (e.g., `"ATGCGTACGTTAGCAT..."`, 200 chars) and a natural-language paragraph of the same length. Tokenize both with `SmolLM2-135M`. Compute tokens-per-character for each. This is the diagnostic that tells you whether you need a domain tokenizer (FT07 section 7.6).