A comprehensive multilingual tokenizer benchmark covering 100+ human languages, programming languages, and mathematical expressions for evaluating LLM tokenization efficiency, compression, and Unicode handling.
multilingual nlp unicode ai tokenizer evaluation dataset text-processing bpe sentencepiece llm kitefishai tokenizer-benchmark
-
Updated
Apr 4, 2026 - Python