Incremental Merge Tokenizer

Question

Language models usually work on sequences of token IDs instead of on raw characters. In the scheme used here, every token after the basic characters is built by gluing together two tokens that were defined before it. ...