TokEval: A Tokenizer Evaluation Suite
By Clara Meister · Paper · cs.CL
Language model tokenizers are typically selected with minimal evaluation, despite the fact that their design choices directly impact model capabilities. This can be partly attributed to a limited understanding of which tokenizer properties affect which aspects of downstream perfo