What are tokens?

A model cannot read text. It only works with numbers.

A language model works with numbers. It does not read text the way we do.

So your sentence must first be turned into numbers.

Before that happens, the text must be split into smaller pieces.

How the text is split affects cost, speed, spelling, and language support.

Why not use letters?

One idea is to treat every letter as a separate piece.

This keeps the vocabulary small. Any word can be built from letters.

But sentences become very long. A single page may contain thousands of pieces.

That makes processing slower and more expensive.

Letter based models exist. They are good at spelling, but less efficient for long text.

Why not use words?

Words seem like a better choice.

Each word already has meaning. Sentences also need fewer pieces.

But there are too many possible words.

There are names, slang, typos, hashtags, codes, and new words every day.

A fixed list of every possible word would never be complete.

Use chunks.

Modern models use something between letters and full words.

Common words may stay whole.

Rare words are split into smaller parts.

These parts are called tokens.

The tokenizer learns useful pieces from large amounts of text.

Type something and watch it split.

Common words often stay together.

Unusual words are more likely to split into smaller parts.

Try a name, a long word, a number, or some code.

This demo is a simple version of a real tokenizer.

This is why models struggle to count letters.

The model may not see every letter separately.

It sees tokens instead.

One token can contain several letters.

So counting letters can be harder than it looks.

Newer models are better at this, but tokenization still affects the task.

Tokens can make math harder too.

Numbers are split into tokens just like words.

The model may not see a number as ones, tens, and hundreds.

It may see strange chunks instead.

That makes exact calculations harder.

For important math, AI systems often use a calculator tool.

You pay for tokens.

AI providers usually charge by token.

You pay for the tokens you send.

You also pay for the tokens the model returns.

More tokens usually means a higher cost.

Some languages need more tokens.

Tokenizers do not handle every language in the same way.

Some languages need more tokens for the same sentence.

That can make them more expensive to process.

It can also make responses slower.

This matters when building AI products for people around the world.

The context window is measured in tokens.

A model can only process a limited amount of content at one time.

This limit is called the context window.

It is measured in tokens.

It is not measured in pages or words.

The same number of words can use different numbers of tokens.

Spaces and capitals can change the tokens.

Small text changes can create different tokens.

A word with a space before it may become a different token.

Capital letters can also change the split.

Punctuation can change it too.

This is one reason small prompt changes can sometimes change the answer.

Each token becomes a list of numbers.

A token ID is only a label.

The number itself has no meaning.

So the model turns each token into a long list of numbers.

This list is called an embedding.

Tokens split the text. Embeddings help the model represent meaning.

Spelling failures

The letters are discarded before the model starts. It is recalling what it has read about spelling, not looking.

Arithmetic failures

Digits split in ways that ignore place value, so the structure of a number is never reliably visible.

Your bill

Priced per token. The word-to-token ratio is the exchange rate between what you write and what you pay.

The language tax

English gets the efficient pieces. Other languages pay more for the same meaning, in money and in context.

Context limits

The window is tokens, not pages. Code and non-Latin scripts fill it several times faster than English prose.

Odd prompt sensitivity

A space or a capital changes the ids. Sometimes that is enough to change the answer.