arXiv: cs.AI preprintsInternational11 November 2026
A Multi-Word Tokenizer for 204 Languages That Needs Fewer Tokens Than GPT-4o's, Including on English
This is an official announcement record
Firsthand records what arXiv: cs.AI preprints announced and links to the original. The wording below is theirs, not ours.
arXiv:2611.13059v1 Announce Type: cross Abstract: Languages of India need 1.6 to 14 times as many tokens as English on GPT-4o's tokenizer for the same content, which makes them more expensive and slower to serve. We train tokenizers from scratch on human-written text in about 200 languages. Havon v3 is a byte-level BPE tokenizer with 262,144 tokens and a pre-tokenizer that keeps combining marks with their letters and chunks scripts written without spaces. On FLORES-200 devtest it needs fewer tokens than GPT-4o's o200k_base on all 25 languages of India (median 0.74 of GPT-4o's count) and on 172
Read the official announcement
Opens arxiv.org
More from arXiv: cs.AI preprints
- Auditable Human-Agent Teams for Autonomous X-ray Experimentation11 November 2026
- EarthRecord: From Pixels to Land Change with Multimodal Agents over Four Decades of Satellite Imagery11 November 2026
- Train on prefill, monitor at decode: Using sparse autoencoders for topic monitoring11 November 2026
- LandslideFM: A Foundation Model Towards Unified Landslide Perception and Understanding11 November 2026
- Tracing Psychometric Inference in Large Language Models11 November 2026
This content is for informational purposes only and is not professional advice. Specifications, prices, plan tiers, and features change frequently and may differ from what is shown here; verify current details on the manufacturer's or company's official page before purchasing. Ratings are based on analysis of published documentation, not independent lab testing.