"Question"	"Answer"	"Tag"
"State the FT07 module thesis in one sentence."	"Tokenizers and chat templates are where most SILENT training failures live: the model trains, loss goes down, but the model learns the wrong thing because the tokens were wrong. It is the capstone of Pillar 1 — the last gate between 'clean data' and 'correct tokens.'"	c3::ft07::recall
"Name the top three silent tokenizer/template bugs (the FT07 debugging checklist)."	"(1) Cross-family template misuse — applying a Llama-3 template to a Qwen model or vice versa. (2) EOS mishandling — missing or wrong end-of-sequence tokens during SFT. (3) Packing-without-attention-mask — concatenating examples without masking cross-boundary attention or loss on non-assistant tokens."	c3::ft07::recall
"What is the role-token convention of the Llama-3 family?"	"Llama-3 uses <|begin_of_text|> to start a document, then <|start_header_id|>/<|end_header_id|> to bracket the role name (system/user/assistant), and <|eot_id|> (end-of-turn) to end a turn. Header-id convention, not ChatML."	c3::ft07::recall
"What is the role-token convention of the Qwen family?"	"Qwen uses ChatML: <|im_start|> followed by the role name and a newline, then the content, then <|im_end|>. For ChatML-family models <|im_end|> and EOS are frequently the SAME token id."	c3::ft07::recall
"Why is apply_chat_template() mandatory and hand-concatenation an anti-pattern?"	"apply_chat_template reads tokenizer.chat_template and produces the family-correct token sequence: the right special tokens, the right role placement, the right EOS. Hand-concatenation drops special tokens, uses wrong EOS, and mismatches the family — directly causing bugs 1 and 2. It is the single rule that prevents 80% of these bugs."	c3::ft07::recall
"What is the FT07 inspection loop (the two-minute discipline that catches bugs 1, 2, 3)?"	"Take ONE conversation: (1) ids = apply_chat_template(conv, tokenize=True, return_assistant_tokens_mask=True); (2) print(tokenizer.decode(ids)) and READ it; (3) check EOS is appended and matches the family; (4) check the assistant mask marks only assistant tokens. Only then launch the full run. Every silent bug is visible if you decode one example."	c3::ft07::application
"What is bug 1 (cross-family template misuse) and what is its symptom?"	"Applying a template from one family to a model of another — e.g. a Llama-3 template on Qwen. The role tokens are NOT interchangeable; the model's persona/format understanding degrades. Symptom: role boundaries fuzzy, assistant continues user turn, format drifts at inference. Diagnosis: decode an example; do the special tokens match the family?"	c3::ft07::analysis
"What are the three ways bug 2 (EOS mishandling) breaks, and the common symptom?"	"(1) Missing EOS — stripped/dropped, model never learns to stop. (2) Wrong EOS — generic </s> on a model whose EOS is <|im_end|>/<|end_of_text|>. (3) EOS in the wrong place — appended once per packed sequence instead of per assistant turn. Symptom: run-on generation to max length; the 'my model won't stop generating' support thread."	c3::ft07::recall
"What is the famous public footprint of bug 2 (EOS mishandling)?"	"The Hugging Face forums thread where Qwen SFT loss 'explodes' (jumps to absurd values or NaN) repeatedly traces back to EOS and chat-template misconfiguration — the model is asked to predict tokens it cannot represent cleanly because turn boundaries are wrong. The fix is almost always apply_chat_template + verifying eos_token_id."	c3::ft07::recall
"What is bug 3 (packing-without-attention-mask), and why is it the quietest of the three?"	"When packing multiple examples into one sequence, you MUST (a) mask cross-boundary attention (FlashAttention document/variable-length attention) so examples don't attend across boundaries, and (b) mask loss on non-assistant tokens. It is the quietest because the model trains fine, loss looks great, output is plausible — quality is only visibly degraded vs a correctly-packed A/B baseline. A silent quality regression."	c3::ft07::analysis
"What are the two distinct sub-bugs inside bug 3 (packing)?"	"(a) Single all-ones attention mask lets example B attend to example A — spurious cross-conversation correlations. Fix: document/variable-length attention via FlashAttention cu_seqlens or TRL position_ids packing. (b) labels = packed[:] computes loss on every token including system/user/role-scaffold — model spends capacity imitating the user. Fix: labels with -100 at every non-assistant position (use return_assistant_tokens_mask=True)."	c3::ft07::analysis
"What does 'document-level' or 'variable-length' attention mean, and what problem does it solve?"	"It is FlashAttention's mechanism (via position_ids / cu_seqlens) that prevents tokens in one packed example from attending to tokens of another. Without it, packing forces a single all-ones attention mask and examples cross-contaminate (bug 3a). With it, you get the GPU efficiency of packing without the silent quality regression."	c3::ft07::recall
"Why must loss be masked on non-assistant tokens during SFT?"	"The model should only learn to PREDICT assistant tokens. The system prompt, user turns, and role scaffolding are context, not targets. Computing loss on every token (labels = packed[:]) trains the model to imitate the user — wasted capacity and silent quality degradation on the assistant behavior you actually care about (bug 3b). Use return_assistant_tokens_mask=True to build the mask."	c3::ft07::application
"True or false: base models never ship with chat templates. Explain."	"FALSE — the assumption is outdated. Modern base checkpoints increasingly ship with chat templates and special tokens (Qwen3 base includes ChatML <|im_start|>/<|im_end|> tokens and a default template). The tokenizer is shared across base/instruct variants. The base/instruct/chat distinction (FT03) is about WEIGHTS, not template presence. Always check tokenizer.chat_template regardless of checkpoint."	c3::ft07::analysis
"Why do Qwen3 base checkpoints ship with ChatML special tokens even though they are base (not instruct)?"	"Because the tokenizer is shared across the base and instruct variants of the same release, so the special tokens are present from the start — as real vocabulary entries with trained embeddings. Implication: the scaffolding tokens are NOT cold-started when you format SFT data; you are using embeddings the model already has. Do NOT assume base = no template."	c3::ft07::analysis
"What is the tool-calling template caveat from FT07?"	"The default apply_chat_template handles the common tool-call case, but NOT every case. Known issues: the default Qwen template can mishandle multi-turn tool scenarios where a tool response contains text resembling a special token; some templates don't append EOS after a tool-call turn; the assistant mask can wrongly unmask the tool RESPONSE (not assistant-generated). Rule: decode several tool examples including the mask; override chat_template if wrong."	c3::ft07::analysis
"When should you train your OWN tokenizer vs reuse the base? Give the four canonical train-your-own cases."	"Almost always REUSE the base. Train your own ONLY for grossly inefficient domain tokenization: (1) DNA/protein sequences (tiny alphabet, fragmented into single chars). (2) Chemistry SMILES strings (dense notation). (3) Non-Latin scripts underrepresented in pretraining. (4) Heavy domain code in unusual languages. Symptom that triggers it: tokens-per-character is abnormally high, sequence length balloons, VRAM multiplies."	c3::ft07::application
"What is the FT07 diagnostic that tells you whether you need a domain tokenizer?"	"Measure tokens-per-character on a representative domain sample. If it is abnormally high (sequence length balloons, VRAM multiplies, the model sees fragmented context), you have a tokenizer-fit problem. For steering-only tasks the answer is almost always 'reuse the base' — training a new tokenizer commits you to continued pretraining, not just SFT."	c3::ft07::application
"What is the vocab-extension workflow, and which step does everyone forget?"	"(1) tokenizer.add_tokens([...]). (2) model.resize_token_embeddings(len(tokenizer)) — resizes input embeddings AND lm_head. (3) WARM UP the new rows (short embedding-only pass, or mean-init). (4) Re-save the tokenizer (ship with adapter). The forgotten step is (3) — the new rows are RANDOMLY initialized and inject noise into the loss if you train the full task immediately."	c3::ft07::application
"Why must you resize BOTH the embedding matrix AND lm_head when extending the vocab?"	"The input embeddings map token id -> vector; lm_head maps vector -> token logits. If their row counts diverge (new tokens in embeddings but not lm_head), you get NaNs or garbage logits — a silent bug. For most architectures model.resize_token_embeddings handles both together; VERIFY both are resized. A mismatch is a classic silent failure."	c3::ft07::analysis
"What is the anti-pattern 'extending the vocab without warming up embeddings', and what is the symptom?"	"Adding domain tokens, resizing embeddings, and immediately training SFT on the full task. The new token rows are randomly initialized; training the steering task while the new embeddings are still random injects noise into the loss. Symptom: the model trains but behaves erratically on the new tokens. Fix: warm up new rows first (short emb-only pass or mean-init from existing embeddings)."	c3::ft07::analysis
"What is 'efficient tokenizer adaptation' (arXiv:2512.03989, 2025)?"	"A middle path between reusing the base and training from scratch. Carefully RE-TOKENIZE a domain corpus RESPECTING the SentencePiece/BPE merge rules already in the base tokenizer, adding only the high-value domain merges that compress the corpus most. Preserves compatibility with the base model's existing embeddings (merges are additive), avoiding the cold-start problem of a full retrain. Not naive BPE retrain (which invalidates existing merges)."	c3::ft07::recall
"Match the symptom to the bug: 'model won't stop generating; runs to max length'."	"BUG 2 — EOS missing or wrong. Diagnosis: decode the last tokens of an assistant turn; is the family EOS present? Fix: use apply_chat_template (appends correct EOS); re-tokenize. This is the most common single cause of 'my model won't stop generating' support threads."	c3::ft07::application
"Match the symptom to the bug: 'trains fine, loss looks great, but quality subtly degraded vs baseline'."	"BUG 3 — packing without attention mask. Diagnosis: check attention_mask and loss mask on a packed sequence. Fix: use document/variable-length attention (FlashAttention cu_seqlens / TRL position_ids packing); mask loss on non-assistant tokens (labels=-100). The quietest bug — only visible against a correct A/B baseline."	c3::ft07::application
"What is the single FT07 diagnostic principle that catches every silent tokenizer bug?"	"ENCODE, DECODE, READ. Every silent bug (wrong family, missing EOS, missing attention/loss mask) is VISIBLE in the decoded token stream of a single example. The discipline is not sophisticated — it is just routinely skipped. The loss curve gives no signal for these bugs by design; they live below its resolution. The two-minute decode is the cheapest insurance in the pipeline."	c3::ft07::analysis