Positional Embeddings: Why Attention Needs Order

We have talked about attention a lot: the dot product, self-attention, causal masking and heads.

There is one more ingredient that shows up everywhere in LLMs.

Positional embeddings.

Before explaining how they work, it is worth being clear about why we need them at all.

Attention does not know order

In attention, we build contextual representations by comparing tokens with the dot product. That comparison cares about content, not about where a token sits in the sentence.

There is a deeper reason this happens.

RNNs process tokens one after another. The hidden state walks through time, so order is baked into the computation.

Transformers do not work that way. They compute attention over the whole sequence in parallel. That is great for speed. It is also exactly why order disappears.

If you shuffle the tokens of your input, the set of attention scores is unchanged. Each score just moves to a different slot.

Attention is permutation-equivariant.

It sees a bag of tokens, not a sequence.

For example:

The dog bit the man.
The man bit the dog.

These two sentences contain the same tokens. Without some fix, a transformer literally cannot tell them apart.

So we need to do something to each token embedding so the model can remember position.

Attempt 1: just add the index

The first idea that comes to mind is almost too simple.

Take the token embedding, and add a position vector that is somehow based on the index.

Position 0 gets something like all zeros.
Position 1 gets ones.
Position 2 gets twos.
And so on.

In spirit:

$$e'_t = e_t + t$$

Theoretically, yes, the model now has a signal that differs by position.

Practically, this solution is unbounded.

If your context length is one million, the last positions carry huge numbers. Neural nets usually like weights and activations in a reasonably small range. Dumping values like $10^6$ into the residual stream is a good way to get unstable training and exploding gradients.

So "use the raw index" dies quickly.

Attempt 2: normalize by length

Okay, then normalize.

$$p_t = \frac{t}{L}$$

where $t$ is the position and $L$ is the sequence length.

Now everything sits in $[0, 1]$. Bounded. Looks better.

But watch what happens with two sentences of different lengths:

Hello world
Cat sat on table

For length 2:

$$\frac{1}{2},\; \frac{2}{2}$$

For length 4:

$$\frac{1}{4},\; \frac{2}{4},\; \frac{3}{4},\; \frac{4}{4}$$

The same conceptual "first token" gets a different value depending on how long the prompt is. Prompt size varies all the time. That makes this encoding brittle.

There are more problems:

• the signal is still basically a discrete step dressed up as a float
• it mainly encodes absolute position
• it does not naturally give you a clean notion of relative distance between two tokens

So this is not the right solution either.

What we actually want

From those failures we can write down the wishlist.

A good positional signal should be:

1. Bounded: values stay in a stable range
2. Continuous: smooth enough for gradient-based learning
3. Able to support relative positioning: "how far is token A from token B?" matters, not only "what is my absolute index?"

That is the design brief.

Attempt 3: just use sine

Sine is bounded and continuous. Perfect candidate.

$$e'_t = e_t + \sin(t)$$

Whatever position you pick, $\sin(t)$ always lands in $[-1, 1]$. No exploding values. Gradients stay happy.

But the bounded nature creates a new problem.

Sine is periodic. The wave goes up, comes down, and then repeats the same path forever:

$$\sin(t) = \sin(t + 2\pi k)$$

for any integer $k$.

So if position $5$ gets the value $\sin(5)$, then position $5 + 2\pi$ gets exactly the same number.

From the model's point of view, those two positions look identical.

That is a collision.

Think of it like a clock that only shows the second hand. After 60 seconds, the hand is back where it started. You cannot tell whether one minute passed or two, if that is your only signal.

A single sine is that second hand. Different times can share the same reading.

So "just add sine" is not enough.

Attempt 4: sine and cosine together

Until now we were adding a scalar: one number per position.

What if we add a small vector instead?

$$\left[\sin(t),\; \cos(t)\right]$$

Geometrically, that pair is a point on the unit circle. As $t$ increases, the point rotates around the circle.

Sine alone only tells you the height of the point. The left side and the right side of the circle can share the same height, so collisions are common.

Sine and cosine together give you height and horizontal location. That is the full phase of the angle. Inside one full turn, every position gets a unique pair.

That already helps a lot.

But there is still a limit.

After a full rotation ($2\pi$), you are back at the same point on the circle. So with one frequency, long sequences can still collide across periods.

And embeddings are not 2-dimensional. They might be 128, 512, 768, 1024...

So we need many frequencies, not one.

Sinusoidal positional encodings

This is the classic Transformer solution.

Instead of one wave, use many waves at different speeds, and pack them into the embedding dimensions.

For position $pos$ and embedding dimension index $i$:

$$PE_{(pos,\; 2i)} = \sin\left(\frac{pos}{10000^{2i / d}}\right)$$ $$PE_{(pos,\; 2i+1)} = \cos\left(\frac{pos}{10000^{2i / d}}\right)$$

where $d$ is the model dimension.

Even dimensions get sine. Odd dimensions get cosine. Each pair uses a different wavelength.

Then we add this vector to the token embedding:

$$x_{pos} = e_{pos} + PE_{pos}$$

Low-index dimensions oscillate quickly. High-index dimensions oscillate slowly. Together they give each position a unique fingerprint that stays bounded in $[-1, 1]$.

That multi-frequency trick is what finally attacks the periodicity problem from earlier.

One wave repeats. Many waves at different speeds almost never line up the same way twice inside a realistic context length.

Even if two positions agree on the fast channel, the medium and slow channels will usually disagree. The whole vector still looks different.

How to read that heatmap

The plot is the intuition made visible.

• Vertical axis: position in the sequence
• Horizontal axis: depth inside the embedding vector
• Color: value of the encoding, from about $-1$ (red) to $+1$ (blue)

Look at the left side.

Near low depth, the stripes flicker rapidly as you move down the positions. Those are the high-frequency dimensions. They change a lot from one token to the next. They are good at distinguishing nearby positions.

Now look further right.

As depth increases, the bands get wider and calmer. Those are the low-frequency dimensions. Over a short range of positions, they barely move.

That is not a bug. That is the design.

Why don't we see large changes farther to the right?

Because the denominator $10000^{2i/d}$ grows with $i$. The argument of sine and cosine advances more slowly. So for a fixed step in position, a high-depth dimension changes only a little.

You can think of it like a mixed radix clock:

• fast dimensions = seconds
• slower dimensions = minutes
• slowest dimensions = hours

Nearby tokens look different on the fast channels. Distant structure still has stable signal on the slow channels.

And because relative offsets interact nicely with sine and cosine, the model can learn patterns like "two tokens apart" more naturally than with raw absolute indices.

Absolute, relative, and what came later

Sinusoidal encodings start as an absolute function of position. But the geometry of sine and cosine makes relative relationships easier to recover than the naive index hacks we tried earlier.

That is why this idea stuck in the original Transformer paper.

Later models took different routes:

• GPT-2 learned a position embedding table wpe and added it, the same way we add wte
• many modern LLMs use RoPE, which rotates queries and keys by position instead of adding a vector to the residual stream

Different mechanisms. Same job: give attention a notion of order.

The sinusoidal version is still worth understanding, because it is the cleanest place to see the requirements (bounded, continuous, multi-scale) without any learned table hiding the idea.

Takeaway

Attention alone is a bag-of-tokens machine.

Raw indices explode. Length-normalized indices break across different prompt sizes. A single sine collides with itself.

Sinusoidal positional encodings fix the wishlist with a simple recipe:

many bounded waves, at many frequencies, added into the embedding

The heatmap is the whole story in one picture: busy on the left, calm on the right, unique down every row.

That is how a transformer remembers that the dog bit the man is not the man bit the dog.