Tokenizer How text becomes token IDs and back. Covers vocabulary representation, BPE merge algorithm, special tokens, chat templates. Model Byte-Pair Encoding (BPE) is the canonical tokenizer. Vocabulary is a map from token id (u32) to token string (bytes). Merges are an ordered list of `(left,…