THE TOKENIZATION PREMIUM: MEASURING UZBEK TOKENIZATION OVERHEAD IN LARGE LANGUAGE MODELS AND EVALUATING THE 2026 ALPHABET REFORM

Authors

  • Xayrullo Ro’zimatov Millat Umidi University Author

Keywords:

Uzbek; tokenization; large language models; NLP; alphabet reform; low-resource languages; BPE; agglutinative languages.

Abstract

 Large language models process text as tokens, making tokenization efficiency important for computational cost and context capacity. This study examines Uzbek tokenization efficiency relative to English and other languages and evaluates whether the proposed 2026 Uzbek alphabet reform can reduce the observed overhead. Six widely used tokenizers are compared using a multilingual parallel corpus and a large Uzbek literary corpus. Latin Uzbek requires substantially more tokens than English across the evaluated tokenizers, while Cyrillic Uzbek shows an even greater overhead in some cases. Turkish generally requires fewer tokens than Uzbek under the same tokenizer, suggesting that agglutinative morphology alone does not explain the difference. The proposed alphabet reform produces only modest and tokenizer-dependent reductions in token counts. These findings indicate that orthographic reform can improve tokenization efficiency to a limited extent, but the larger Uzbek tokenization overhead remains primarily a broader tokenizer and language-resource problem.

References

1.OpenAI. API Pricing. — https://openai.com/api/pricing/ (accessed August 2026).

2.Sennrich R., Haddow B., Birch A. Neural Machine Translation of Rare Words with Subword Units // Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL). — Berlin, 2016. — Pp. 1715–1725. arXiv:1508.07909.

3.Kudo T. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates // Proceedings of the 56th Annual Meeting of the ACL. — Melbourne, 2018. — Pp. 66–75. arXiv:1804.10959.

4.Common Crawl Foundation. Statistics of Common Crawl Monthly Archives — Language Distribution. — https://commoncrawl.github.io/cc-crawl-statistics/plots/languages.html (accessed August 2026).

5.Petrov A., La Malfa E., Torr P., Bibi A. Language Model Tokenizers Introduce Unfairness Between Languages // Advances in Neural Information Processing Systems (NeurIPS). — New Orleans, 2023. arXiv:2305.15425.

6.Ahia O., Kumar S., Gonen H., Kasai J., Mortensen D. R., Smith N. A., Tsvetkov Y. Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models // Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). — Singapore, 2023. — Pp. 9904–9923. arXiv:2305.13707.

7.Rust P., Pfeiffer J., Vulić I., Ruder S., Gurevych I. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models // Proceedings of the 59th Annual Meeting of the ACL. — 2021. — Pp. 3118–3135. arXiv:2012.15613.

8.Kudo T., Richardson J. SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing // Proceedings of the 2018 EMNLP: System Demonstrations. — Brussels, 2018. — Pp. 66–71. arXiv:1808.06226.

9.Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A. N., Kaiser L., Polosukhin I. Attention Is All You Need // Advances in Neural Information Processing Systems (NeurIPS). — Long Beach, 2017. — Pp. 5998–6008. arXiv:1706.03762.

10.Conneau A., Khandelwal K., Goyal N., Chaudhary V., Wenzek G., Guzmán F., Grave E., Ott M., Zettlemoyer L., Stoyanov V. Unsupervised Cross-lingual Representation Learning at Scale // Proceedings of the 58th Annual Meeting of the ACL. — 2020. — Pp. 8440–8451. arXiv:1911.02116.

11.Devlin J., Chang M.-W., Lee K., Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding // Proceedings of NAACL-HLT. — Minneapolis, 2019. — Pp. 4171–4186. arXiv:1810.04805.

12.Gazeta.uz. Deputies pass law to modify Uzbek alphabet. — 9 July 2026. — https://www.gazeta.uz/en/2026/07/09/alphabet/ (accessed August 2026).

13.Presidential Decree of the Republic of Uzbekistan No. UP-6079 (PF-6079) “On the Approval of the Strategy ‘Digital Uzbekistan — 2030’ and Measures for its Effective Implementation.” — Tashkent, 5 October 2020.

14.Qahhor A. Anor (hikoya). — First published 1936. Text used in this study obtained from the ziyouz.uz digital library of Uzbek classical literature.

15.Hoshimov O’. Dunyoning ishlari (qissa va hikoyalar). — First pieces published 1981; complete collected edition Toshkent: Sharq nashriyoti, 2005. Text used in this study obtained from a scanned edition (60,390 words after OCR extraction and cleaning).

16.Kuriyozov E., Vilares D., Gómez-Rodríguez C. BERTbek: A Pretrained Language Model for Uzbek // Proceedings of SIGUL 2024. — 2024.

17.Salaev U. UzMorphAnalyser: A Morphological Analysis Model for the Uzbek Language Using Inflectional Endings. — arXiv:2405.14179, 2024.

18.Investigating linguistic errors in large language model generation of Uzbek text // Cogent Arts & Humanities. — Taylor & Francis, 2025. DOI: 10.1080/23311983.2025.2600519.

19.Safarov R. Unsupervised Morphological Segmentation for Hybrid Tokenization in Uzbek Language. — Under review as a conference paper at MathAI 2026, OpenReview, 2026. https://openreview.net/forum?id=cwUDQn6jt2.

Downloads

Published

2026-08-10