Before an LLM Can Read Your Prompt, It Has to Break It Apart

Whenever you type something into ChatGPT, Claude, Gemini, or basically any LLM, there is a step happening before the model sees your prompt.

You write:

"The fox is running."

But the model doesn't actually receive:

"The fox is running."

It receives something more like:

[464, 1241, 318, 5632, 13]

These are token IDs.

And this step turning human text into these IDs is called tokenization.

This is one of those things that gets skipped way too quickly when learning LLMs.

People jump straight into:

• Attention
• Transformers
• Embeddings
• KV cache
• Flash Attention
• RLHF

But there is a very simple thing happening before all of that:

The model needs to turn text into numbers.

And there are several ways you could do that.

Let's build the intuition from scratch.

So... what exactly is a token?

A token is basically a piece of text that the model treats as one unit.

It doesn't necessarily mean a word.

For example:

"hello"

could be:

["hello"]

But something like:

"unbelievable"

could become:

["un", "believ", "able"]

Or even something completely different depending on the tokenizer.

The important thing is:

The tokenizer decides what pieces of text the model gets to see.

And that decision has a huge impact on the model.

So let's look at the different approaches.

Approach 1: Character-level tokenization

The most obvious idea is:

Why don't we just make every character a token?

Take:

hello

We could tokenize it as:

h e l l o

And assign each character an ID:

h → 10
e → 5
l → 12
o → 15

Very simple.

The intuition

Imagine you're building a vocabulary.

Instead of trying to memorize every possible word in English, you only need to know the alphabet.

That's extremely powerful because you basically can't encounter an unknown word.

If the user types:

xqzplm

you don't care.

You already know:

x
q
z
p
l
m

So from a vocabulary perspective, character-level tokenization is incredibly robust.

But there is a problem

Look at:

"the cat is sitting on the chair"

At the character level, this becomes something like:

t h e   c a t   i s   s i t t i n g ...

That's a lot of tokens.

And remember:

Transformers process sequences of tokens.

More tokens → longer sequence → more computation.

And there's another problem.

A character like:

t

doesn't tell you much.

But:

the

contains a lot more useful information.

The model now has to learn that:

t + h + e

forms a meaningful linguistic unit.

Then:

c + a + t

forms another.

So the model has to spend a lot of capacity reconstructing structure that we already know exists.

Pros

• Tiny vocabulary
• Almost no unknown-word problem
• Works with arbitrary text
• Very simple

Cons

• Huge sequence lengths
• Computationally expensive
• Individual tokens contain very little semantic information
• Model has to learn common patterns from scratch

So character-level tokenization is kind of like saying:

"I'll give the model the smallest possible Lego pieces and let it figure everything out."

Approach 2: Word-level tokenization

Then you might think:

Fine. Let's go in the opposite direction.

Instead of characters, let's make words tokens.

The fox is running

becomes:

["The", "fox", "is", "running"]

Much better!

Instead of 18-ish characters, we're down to 4 tokens.

And now each token carries much more information.

The model doesn't need to figure out that:

r + u + n + n + i + n + g

means:

running

It gets:

running

directly.

Sounds great.

Until you realize how ridiculous the vocabulary becomes.

The vocabulary explosion problem

Think about all the possible words.

English alone has a massive vocabulary.

And then you have:

run
running
runs
ran
runner
runners

Should every variation get its own token?

And what happens when someone writes:

TalalAhmedSuperLongRandomWord123

You probably don't have that entire thing in your vocabulary.

So now you need some kind of:

<UNK>

unknown token.

Which means you've gone from:

"The model knows every character"

to:

"The model has no idea what this word is."

That's a pretty bad tradeoff.

Pros

• Very short sequences
• Each token carries a lot of information
• Easy conceptually

Cons

• Massive vocabulary
• Unknown-word problem
• Poor handling of new words
• Morphology becomes awkward
• Different languages make this even harder

So we have two extremes:

Character
   ↓
Tiny vocabulary
Huge sequence

Word
   ↓
Huge vocabulary
Small sequence

And this is where things get interesting.

Approach 3: Subword tokenization

What if we don't choose between characters and words?

What if we allow the tokenizer to learn pieces of words?

For example:

"playing"

might become:

["play", "ing"]

And:

"unhappiness"

could become:

["un", "happi", "ness"]

Now we're getting something useful.

Common pieces get their own tokens.

Rare words can still be broken down into smaller pieces.

So we get something like:

Characters ←────────────→ Words
       ↑
    Subwords

This gives us a pretty nice middle ground:

• Vocabulary isn't insane
• Sequence isn't insanely long
• Unknown words can still be represented
• Common patterns can become single tokens

There are several subword algorithms.

One of the most important ones is BPE: Byte Pair Encoding.

The rest of this post walks through a simple byte-level BPE tokenizer implementation, step by step.

BPE: The basic idea

The intuition behind BPE is actually very simple.

Start with something extremely basic:

Every byte is its own token.

Then repeatedly ask:

Which pair of tokens appears together the most?

Merge that pair.

Then do it again.

And again.

And again.

Eventually, frequent combinations become their own tokens.

That's basically BPE.

Let's forget LLMs for a second

Suppose our entire dataset is:

low low low
lower lower
lowest

Initially, we're starting with small pieces.

For simplicity, imagine characters:

l o w
l o w
l o w
l o w e r
l o w e r
l o w e s t

Now count adjacent pairs.

We might find:

("l", "o") → very frequent
("o", "w") → very frequent
("w", "e") → somewhat frequent
("e", "r") → ...

Suppose:

("l", "o")

is the most frequent pair.

We merge it:

lo

Now our vocabulary contains:

l
o
w
e
r
s
t
lo

And our words become:

lo w
lo w
lo w
lo w e r
...

Now maybe:

("lo", "w")

becomes the most frequent pair.

Merge again:

low

Now:

low
low
low
low e r
low e r
low e s t

And maybe eventually:

low + er → lower

So the tokenizer has discovered useful chunks from the data.

Nobody manually told it:

"lower should be one token."

The training corpus told it.

That's the beauty of BPE.

What does the tokenizer look like?

Here's the core class. It holds the merge rules, the vocabulary, and optional special tokens:

class BPETokenizer:
    def __init__(self, pattern: str = GPT2_SPLIT_PATTERN, special_tokens=None):
        self.pattern = pattern
        self.pat = re.compile(pattern)
        self.merges: dict[tuple[int, int], int] = {}
        self.vocab: dict[int, bytes] = {i: bytes([i]) for i in range(256)}
        self.special_tokens: dict[str, int] = dict(special_tokens or {})
        self._inverse_special = {v: k for k, v in self.special_tokens.items()}
        self._cache: dict[str, list[int]] = {}

We start the vocabulary with 256 tokens:

self.vocab = {
    i: bytes([i])
    for i in range(256)
}

Why 256?

Because we're working at the byte level.

A byte can represent:

0 → 255

So we have 256 possible starting tokens.

This is slightly different from the simplified character example above.

We're actually saying:

"I don't even need to assume an alphabet. Give me raw UTF-8 bytes."

That's a very useful property.

For example:

hello

gets converted into bytes.

And something like:

你好

also gets converted into UTF-8 bytes.

So the base vocabulary can represent basically arbitrary text.

Step 1: Pretokenization

Before BPE starts merging things, the tokenizer first splits the text into chunks.

This is where this regex comes in:

GPT2_SPLIT_PATTERN = (
    r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+"""
)

This looks terrifying.

It isn't as bad once you break it down.

The idea is:

Don't run BPE over the entire raw document blindly. First split the text into sensible pieces.

This is called pre-tokenization.

def _pretokenize(self, text: str):
    return (m.group() for m in self.pat.finditer(text))

For example, we want things like:

"I'm running 123!"

to be separated into pieces that roughly correspond to:

"I"
"'m"
" running"
" 123"
"!"

The exact behavior comes from the regex.

Why is the regex important?

Because BPE is learning from frequency.

Imagine we didn't pre-split anything.

Then the tokenizer might happily start learning merges across completely different linguistic structures.

For example:

"hello world"

could eventually encourage merges involving:

"hello" + " "

or even larger sequences.

Sometimes that's useful.

But it can also make your vocabulary overly tied to the exact formatting of your training corpus.

The GPT-2 style regex provides some structure before BPE gets to work.

It basically says:

"Here are the kinds of chunks you should consider independently."

This regex comes from the tokenization approach used by GPT-2, with Unicode-aware categories such as:

\p{L}

for letters and:

\p{N}

for numbers.

That's one reason this uses the third-party Python regex package rather than the standard re module.

Python's normal re doesn't support these Unicode property escapes in the same way.

Step 2: Convert chunks into bytes

Once we have a chunk:

chunk.encode("utf-8")

we convert it into bytes.

So conceptually:

"hello"

becomes something like:

[104, 101, 108, 108, 111]

These are our initial tokens.

At this point:

hello

is basically:

h e l l o

except we're thinking in bytes rather than characters.

During training, every pretokenized chunk becomes a counted byte sequence:

words: Counter[tuple[int, ...]] = Counter()
for chunk, count in Counter(self._pretokenize(text)).items():
    words[tuple(chunk.encode("utf-8"))] += count

Step 3: Count every adjacent pair

Now comes the actual BPE algorithm.

For every word/chunk, we look at adjacent pairs:

pairs: Counter = Counter()
for word, count in words.items():
    for pair in zip(word, word[1:]):
        pairs[pair] += count

For:

hello

we get:

(h, e)
(e, l)
(l, l)
(l, o)

We do this across the entire training corpus.

And we count how often every pair appears.

So we might end up with:

(h, e) → 500
(t, h) → 1200
(i, n) → 900
...

Then:

best = max(pairs, key=lambda p: (pairs[p], p))

finds the most frequent pair.

This is the heart of BPE:

Find the most common neighboring pair.

Step 4: Merge it

Suppose the most frequent pair is:

(97, 98)

which corresponds to:

a + b

We create a new token:

new_token = 256 + i

So the first merge becomes:

256

and we store:

self.merges[best] = new_token

Meaning:

(a, b) → 256

And its actual byte representation is:

self.vocab[new_token] = (
    self.vocab[best[0]] +
    self.vocab[best[1]]
)

So:

a + b

becomes a new token representing:

ab

The merge itself walks a token sequence and replaces matching pairs:

@staticmethod
def _merge_pair(tokens: tuple[int, ...], pair, new_token: int) -> tuple[int, ...]:
    merged = []
    i = 0
    n = len(tokens)
    while i < n:
        if i < n - 1 and tokens[i] == pair[0] and tokens[i + 1] == pair[1]:
            merged.append(new_token)
            i += 2
        else:
            merged.append(tokens[i])
            i += 1
    return tuple(merged)

Then we do it again

This is the important part.

BPE isn't:

"Find the best vocabulary."

It's an iterative process.

Something like:

Start with bytes

        ↓

Count pairs

        ↓

Merge most frequent pair

        ↓

Count pairs again

        ↓

Merge most frequent pair

        ↓

Count pairs again

        ↓

...

The full training loop looks like this:

def train(self, text: str, num_merges: int, verbose: bool = False):
    self.merges = {}
    self.vocab = {i: bytes([i]) for i in range(256)}
    self._cache.clear()

    words: Counter[tuple[int, ...]] = Counter()
    for chunk, count in Counter(self._pretokenize(text)).items():
        words[tuple(chunk.encode("utf-8"))] += count

    for i in range(num_merges):
        pairs: Counter = Counter()
        for word, count in words.items():
            for pair in zip(word, word[1:]):
                pairs[pair] += count
        if not pairs:
            break

        best = max(pairs, key=lambda p: (pairs[p], p))
        new_token = 256 + i

        rebuilt: Counter[tuple[int, ...]] = Counter()
        for word, count in words.items():
            if pair_in_word(word, best):
                word = self._merge_pair(word, best, new_token)
            rebuilt[word] += count
        words = rebuilt

        self.merges[best] = new_token
        self.vocab[new_token] = self.vocab[best[0]] + self.vocab[best[1]]

    return self

If we perform:

num_merges=60

we do this 60 times.

So after training, we've learned 60 new tokens beyond the original 256 byte tokens.

The resulting tokenizer has:

256 base byte tokens
+
60 learned merge tokens

ignoring special tokens.

The really important thing: BPE learns merges, not words

This is something that is easy to miss at first.

We're not literally training a dictionary like:

hello → 1000
world → 1001
running → 1002

We're learning merge rules.

For example:

(h, e) → 256

(256, l) → 257

(257, l) → 258

(258, o) → 259

Which effectively allows:

h e l l o

to become:

hello

The tokenizer has learned how to construct larger pieces.

That's a much more interesting way to think about BPE.

But how do we tokenize new text?

Training is only half the story.

Suppose we've already learned these merges:

(h, e) → 256
(256, l) → 257
(257, l) → 258
(258, o) → 259

Now we encounter:

hello

We start with:

h e l l o

Then apply the learned merges.

Eventually:

hello

becomes:

[259]

But there is an important detail.

We can't just merge pairs randomly.

The tokenizer needs to follow the merge priority learned during training.

That's why encoding looks like this:

def _encode_chunk(self, chunk: str) -> list[int]:
    tokens = tuple(chunk.encode("utf-8"))
    while len(tokens) >= 2:
        pair = min(
            set(zip(tokens, tokens[1:])),
            key=lambda p: self.merges.get(p, float("inf")),
        )
        if pair not in self.merges:
            break
        tokens = self._merge_pair(tokens, pair, self.merges[pair])
    return list(tokens)

We're basically asking:

"Among the pairs currently available, which one was learned earliest?"

The earlier the merge was learned, the higher its priority.

This makes encoding deterministic.

Ordinary text encoding just pretokenizes, then encodes each chunk:

def encode_ordinary(self, text: str) -> list[int]:
    ids: list[int] = []
    for chunk in self._pretokenize(text):
        ids.extend(self._encode_chunk(chunk))
    return ids

Why not just use the most frequent pair again?

Because the frequencies during training and the merge order matter.

Imagine we learned:

A B → X
X C → Y

When encoding:

ABC

we need to first create:

X C

and then:

Y

So the tokenizer follows the merge rules in the order they were learned.

This is why self.merges is essentially the recipe the tokenizer learned during training.

What about decoding?

Eventually the LLM produces token IDs.

For example:

[259, 300, 42]

We need to turn those back into text.

That's what decode() does:

def decode(self, ids, errors: str = "replace") -> str:
    parts = []
    for t in ids:
        if t in self.vocab:
            parts.append(self.vocab[t])
        elif t in self._inverse_special:
            parts.append(self._inverse_special[t].encode("utf-8"))
        else:
            raise ValueError(f"unknown token id: {t}")
    return b"".join(parts).decode("utf-8", errors=errors)

Each token has its byte representation stored in self.vocab.

So we reconstruct:

token IDs
   ↓
bytes
   ↓
UTF-8
   ↓
text

That's why the tokenizer has both:

encode()
decode()

They are basically:

text → token IDs
token IDs → text

And then there are special tokens

There are certain things that aren't ordinary text.

For example:

<|endoftext|>

This isn't something we want BPE to break apart.

So we can register it separately:

tok.register_special_tokens({
    "<|endoftext|>": 100_000
})

Now when we encounter:

hello<|endoftext|>

the tokenizer recognizes hello normally, but <|endoftext|> as one special token:

def encode(self, text: str, allow_special: bool = True) -> list[int]:
    if not (allow_special and self.special_tokens):
        return self.encode_ordinary(text)

    splitter = "(" + "|".join(re.escape(s) for s in self.special_tokens) + ")"
    ids: list[int] = []
    for part in re.split(splitter, text):
        if not part:
            continue
        if part in self.special_tokens:
            ids.append(self.special_tokens[part])
        else:
            ids.extend(self.encode_ordinary(part))
    return ids

This is important because special tokens often tell the model something about the structure of the sequence, rather than representing ordinary language.

So what did we actually build?

At this point, the entire tokenizer is basically doing:

Raw text
   ↓
GPT-2 style pre-tokenization
   ↓
UTF-8 bytes
   ↓
BPE merge rules
   ↓
Token IDs
   ↓
LLM

And in the other direction:

LLM output token IDs
   ↓
Vocabulary lookup
   ↓
Bytes
   ↓
UTF-8 decoding
   ↓
Human-readable text

That little piece of code sitting before the Transformer is doing a surprisingly important job.

Why BPE is such a nice compromise

Let's compare the three approaches.

Method	Vocabulary	Sequence length	Unknown words
Character	Tiny	Huge	Basically no
Word	Huge	Small	Yes
BPE	Medium	Medium	Very rare

That's why subword tokenization became so popular.

BPE gives us a pretty nice middle ground.

Common things can become single tokens:

the
ing
tion
hello

while unusual things can still be decomposed.

For example, imagine the tokenizer has never seen:

hypermegafunctionalization

It doesn't necessarily need an <UNK> token.

It can fall back to smaller pieces:

hyper
mega
function
al
ization

or even smaller if necessary.

That is a huge advantage.

But BPE isn't perfect either

There are some interesting problems.

1. Tokenization isn't semantic

This is probably the biggest misconception.

BPE doesn't understand language.

It doesn't know that:

king

has a meaning.

It just knows that certain byte sequences occur frequently.

If:

"ing"

appears millions of times, it might become a token.

That's frequency, not understanding.

2. Token boundaries can be weird

You might assume:

"playing"

always becomes:

play + ing

But tokenization isn't necessarily linguistically clean.

It might become something like:

play + ing

or:

pla + ying

depending on what the tokenizer learned.

The tokenizer isn't trying to discover perfect morphemes.

It's trying to find useful compression of the training data.

3. Different languages can behave very differently

Languages with different writing systems, morphology, or scripts can get very different tokenization efficiency.

A tokenizer that is extremely efficient for English isn't necessarily equally efficient for every language.

This matters because:

more tokens
    ↓
longer context
    ↓
more computation
    ↓
potentially higher cost

So tokenization isn't just some preprocessing detail.

It directly affects how efficiently an LLM can process language.

And this is where the tokenizer becomes part of the model design

Once you see this, you realize something interesting.

An LLM doesn't really operate on:

words

It operates on:

tokens

And those tokens are created by a tokenizer.

Then each token ID gets converted into an embedding:

token ID
   ↓
embedding lookup
   ↓
vector
   ↓
Transformer

So the full pipeline is roughly:

"You are amazing"
        ↓
    Tokenizer
        ↓
[You, are, amazing]
        ↓
    Token IDs
        ↓
Embedding lookup
        ↓
Vectors
        ↓
Transformer
        ↓
Next-token prediction

And suddenly all those things you've been learning about — embeddings, attention, positional encoding, Transformers — sit after this seemingly boring little tokenizer step.

One last thing about the implementation

Libraries like Hugging Face's tokenizers are heavily optimized.

This implementation is intentionally simple.

For example, during training it basically does:

for every merge:
    count every pair
    find the best pair
    rebuild the corpus

That is conceptually clean, but it's not the fastest possible implementation.

A production tokenizer uses much more sophisticated data structures and optimizations.

But implementing the naive version is one of the best ways to understand BPE.

Because after writing it, the algorithm stops feeling like:

"Some magical tokenizer algorithm used by GPT."

and becomes:

"Oh... we're literally just counting neighboring pairs and repeatedly merging the most frequent one."

And that's basically BPE.

The entire algorithm in 15 seconds

If you had to explain BPE to someone at a whiteboard:

Start with bytes

        ↓

Find the most frequent adjacent pair

        ↓

Merge it into a new token

        ↓

Repeat

        ↓

Store the merge rules

        ↓

Use those rules to tokenize new text

That's it.

The cleverness isn't in some complicated neural network.

It's in realizing:

If certain pieces of text repeatedly occur together, why not turn them into one reusable token?

And that simple idea gives us a tokenizer that sits right at the entrance of the LLM.