Multilingual Tokenization Challenge · AI Research Foundations

The challenge

Overview

Tokenization is the first step in how language models turn human language into something they can process. That makes tokenizer design a fundamental part of building a language model.

But not every tokenizer represents every language equally efficiently. In this challenge, you will build one tokenizer for six languages under a fixed vocabulary budget of 10,000 tokens.

Your goal is simple

Represent text using as few tokens as possible, while avoiding unknown tokens.

Any tokenizer design is welcome. This is an open-ended, beginner-friendly competition built for learning through experimentation.

Team formation

You may participate individually or form a team. If you enter as a team, it must include participants from at least two different nationalities.

What you take away

Skills you will gain

This challenge is designed to help you gain in-demand practical skills that are necessary for meaningful AI research. You keep these whether or not you top the leaderboard.

Tokenization
Train, tune, and compare BPE, WordPiece, Unigram, and byte-level tokenizers.
Multilingual NLP
Share one vocabulary across diacritics, rich morphology, and a non-Latin script.
Version control with GitHub
Fork a repository, work on a branch, and open your first Pull Request.
Evaluation and metrics
Interpret token fertility and unknown-token rate, and read what a single score hides.

No prior competition experience is assumed. The starter notebook walks through each step, and the submission checker tells you what to fix before you open a Pull Request.

The corpus

Dataset

The public corpus is balanced across six languages and contains the text you need to train and validate your tokenizer. It comes from a pinned Wikipedia snapshot, normalised to Unicode NFC and globally deduplicated (revision v1.0.0).

Supported languages
Language	Code	Writing system
English	`en`	Latin
French	`fr`	Latin
Hausa	`ha`	Latin
Swahili	`sw`	Latin
Yoruba	`yo`	Latin with diacritics
Amharic	`am`	Geʽez

How the splits work

Training set 240,000 rows: Used to build your tokenizer. There are 40,000 passages for each language.
Validation set 24,000 rows: Used to test ideas and improve your tokenizer. There are 4,000 passages for each language.
Test set Hidden: Kept private and used by the official evaluation pipeline to calculate leaderboard results.

A few examples

Example passages
Language	Text
English	Artificial intelligence is changing the world.
Yoruba	Ẹ̀kọ́ ṣe pàtàkì fún gbogbo ènìyàn.
Swahili	Teknolojia inabadilisha dunia.

Explore the dataset on Hugging Face

Starting points

Baselines

These two intentionally simple baselines show the central trade-off: shorter representations versus broader coverage.

Baseline 01

Word-level tokenizer

Each known word is represented by a single token. Words outside the vocabulary are represented using <UNK>.

This gives very short representations for known words, but it struggles when it encounters words it has never seen before.

Baseline 02

Character-level tokenizer

Text is represented using individual characters.

This provides much better coverage, but often requires many more tokens to represent the same text.

Baseline scores to beat
Baseline	Score	Key trade-off
Word-level	36.99	Very low fertility, but a high unknown-token rate
Character-level	5.68	Full coverage, but high fertility

A good subword tokenizer should beat both. Your job is to find something better.

What might you explore?

Try these, combine them, or develop your own idea. The challenge is intentionally open-ended.

How scoring works

Evaluation

Your tokenizer is scored on how efficiently it represents the four target languages, kept honest by an English and French guardrail and a check that it reconstructs text exactly. Two per-language metrics do the core work: how many tokens it needs, and how often it produces an unknown token.

Metric 01

Token fertility

F_l=number of tokens producednumber of words

Token fertility tells us how many tokens your tokenizer needs, on average, to represent one word.

150 tokens÷100 words=1.5

Lower fertility is better.

Metric 02

Unknown token rate

U_l=number of <UNK> tokensnumber of words

A tokenizer should ideally represent text without falling back to <UNK>. Unknown tokens are therefore heavily penalized.

Per-language result

Language score

S_l = fertilityF_l + unknown-token penalty100 U_l

The multiplier of 100 makes unknown tokens expensive. Even a small unknown-token rate can noticeably increase your score.

Effect of the unknown-token penalty
`<UNK>` rate	Added penalty
0%	+0.00
0.1%	+0.10
1%	+1.00
5%	+5.00
10%	+10.00

Target languages

Target score

The core objective averages the language scores for the four target languages: Hausa, Swahili, Yoruba, and Amharic.

S_target=S_ha + S_sw + S_yo + S_am4

Guardrail

English and French

English and French are not scored directly, but they keep your tokenizer honest: you cannot optimise only for the four target languages and ignore the rest. Each is allowed a fertility of up to 1.15× the average fertility of the target languages, and any excess is added to your score.

P_en/fr=∑_{en, fr} max(0, F_l − 1.15 F̄)

Losslessness

Reconstruction

A tokenizer should preserve the text it encodes. We check whether decode(encode(text)) exactly reconstructs the original text, and penalise any failures. This discourages approaches that shrink token counts by dropping or altering information.

P_{reconstruction}=3 (1 − reconstruction)

A tokenizer that reconstructs every example exactly receives no penalty. Unicode NFC, special tokens, and leading or trailing whitespace are ignored.

Official result

Complete score

Score = target languagesS_target + en / fr guardrailP_en/fr + reconstructionP_{reconstruction}

Lower is better.

Keep this in mind

Represent the text with as few tokens as possible, while avoiding unknown tokens.

From idea to entry

How to submit

The competition runs through GitHub. The process is designed to help you practise a real open-source workflow, even if this is your first Pull Request.

Rules and constraints

One tokenizer must support all six languages.
Maximum vocabulary size: 10,000 tokens.
Train using only the provided competition training data.
No pretrained tokenizer, external corpus, vocabulary, word list, embedding, or API — it must run offline.
Encoding must be lossless: decode(encode(text)) must restore the original text.
Any tokenizer approach is allowed.
The tokenizer must work without participant-written code during evaluation.
Evaluation must finish within 5× the character-level baseline runtime.
Your submission must pass the official submission checker.
You must also submit the notebook used to build your tokenizer. Omitting it means disqualification.
The hidden test set determines leaderboard performance.

Recommended starting point

Starter notebook

Use the guided Jupyter notebook to explore the dataset, build a first tokenizer, evaluate it on the validation set, and prepare your submission.

Open starter.ipynb on GitHub

Submission process

1
Build and export your tokenizer
Train it on the competition data and export it as tokenizer.json.
2
Run the submission checker
Confirm your tokenizer loads, stays within the 10,000-token limit, encodes all six languages, and reconstructs text.
3
Fork the repository and branch
Fork the competition repository and create a branch named submission.
4
Add your team folder
Place tokenizer.json and metadata.yml under submissions/your-team-name/.
5
Open a Pull Request
Push the submission branch and open a Pull Request to the official competition repository.
6
Add your notebook
Include the notebook that builds your tokenizer as notebook.ipynb before the deadline. This is required.

What happens next?

GitHub Actions automatically validates eligible submissions and updates the public leaderboard. If a check fails, the Pull Request shows what to fix. Your latest tokenizer on the submission branch is the one that gets evaluated.

Open the competition repository

Good to know

Frequently asked questions

Do I need a GPU?

No. Tokenizer training and evaluation should be possible on a normal laptop or a free Colab environment.

Can I use any tokenizer algorithm?

Yes. Word-level, character-level, BPE, WordPiece, Unigram, byte-level, hybrids, or your own approach are all allowed.

Can I use more than 10,000 tokens?

No. The maximum vocabulary size is 10,000 tokens.

What happens if my tokenizer produces <UNK>?

Your submission is still valid, but unknown tokens receive a large penalty in the evaluation score.

Does my tokenizer have to reconstruct the text exactly?

Yes. We check that decode(encode(text)) returns the original text, and any reconstruction failures add a penalty. Unicode NFC, special tokens, and leading or trailing whitespace are ignored.

Are English and French scored?

Not directly. The leaderboard averages the four target languages, but English and French act as a guardrail: if either exceeds 1.15× the average target fertility, the excess is added to your score.

Can I train on external datasets or use a pretrained tokenizer?

No. Use only the provided competition training data, with no external corpus, pretrained tokenizer, word list, embedding, or API. Your tokenizer must run offline.

Do I need to train a language model?

No. You are only building the tokenizer.

What should I submit?

Your exported tokenizer.json and a metadata.yml under submissions/your-team-name/, plus the notebook.ipynb used to build it. The notebook is required — omitting it results in disqualification.

Keep learning

Build the foundations first

This challenge builds on concepts introduced in the AI Research Foundations learning path.

Course 01

Build Your Small Language Model

Introduces the language model pipeline and where tokenization fits into building a language model.

Course 02

Represent Your Language Data

Explores how text is represented for language models, including tokenization and vocabulary construction.

Explore the learning path

Can you build a tokenizer that works efficiently for every language?