Dot Product: The Magic Behind Attention

Look at a neural network and you will see this equation everywhere:

$$Wx + b$$

We remember this from regression.

What we were basically doing was taking our input $x$, multiplying it with a learnable parameter matrix $W$, and adding a bias $b$. During training, the model learns the values inside $W$ so that it can capture patterns in the data and eventually make useful predictions.

But here is something interesting.

The dot product is not some random mathematical operation that we only use in machine learning.

You will encounter it everywhere — linear algebra, calculus, probability, physics, finance, computer graphics, and many other fields.

Linear algebra itself is one of those subjects that keeps showing up underneath almost everything in machine learning.

So enough praising the dot product.

Let's understand why this simple operation becomes one of the most important ingredients of the Attention mechanism.

First, What Is Attention?

Before jumping into the mathematics, let's understand the problem we are trying to solve.

Consider this sentence:

The cat sat on the mat because it was tired.

Now pause here.

If I ask you:

What does "it" refer to?

You can probably answer immediately:

The cat.

But how did you figure that out?

Your brain didn't simply look at the word "it" in isolation.

It looked at the surrounding words and somehow figured out the relationship between "it" and "cat".

There are many words in the sentence, but some of them are more relevant to understanding "it" than others.

For example:

The   cat   sat   on   the   mat   because   it   was   tired
       ↑                                  ↑
       └──────────── relationship ────────┘

Your brain is effectively asking something like:

"Given the word it, which other words in this sentence should I pay attention to?"

And in this particular sentence, cat is highly relevant.

Now the interesting question:

Can we make a machine do something similar?

Can we give a model the whole sentence and let it figure out:

• which words are related?
• which words are important to each other?
• what context should each word look at?
• what information should one word use from another?

This is where Self-Attention comes in.

But First: What Is a Token?

If you have read the previous article on Byte Pair Encoding (BPE), you already know that before a neural network can process text, we need to convert the text into tokens.

Something like:

"The cat sat"

        ↓

["The", "cat", "sat"]

        ↓

[102, 8532, 421]

These numbers are token IDs.

But here is the problem.

Does the number 8532 actually tell us anything about the meaning of "cat"?

No.

The model doesn't look at 8532 and somehow know:

"Ah yes, this number means cat, and cat is an animal, and cat is related to dog..."

It's simply an identifier.

So we need something richer than a single integer.

We need a vector.

So What Is a Vector?

You probably learned something like this in school:

A vector is a list of numbers.

Or maybe:

A vector is a quantity with magnitude and direction.

Both are technically correct, but let's build a more intuitive understanding.

Imagine you are sitting in a room.

Look at the wall in front of you.

Now imagine that the entire wall has a grid of small hooks or nails, spaced one foot apart horizontally and vertically.

I give you a rubber band and tell you:

"Put one end at the origin and stretch it to the red point."

You might tell me:

"The point is 3 units horizontally and 4 units vertically."

We can represent that as:

$$v = [3, 4]$$

Now we know the position of that point.

But I can ask another question:

How long is this rubber band?

Well, using the Pythagorean theorem:

$$\sqrt{3^2 + 4^2} = 5$$

That length is called the norm or magnitude of the vector.

So our vector gives us information about both position and length:

Vector = [3, 4]

       position
          ↓
      (3, 4)

          +
          |
          |\
        4 | \
          |  \  ← vector
          |   \     length = 5
          |    \
          |_____\
             3

And this is where things get interesting.

A vector doesn't have to represent a physical position.

We can use a vector to represent features.

For example, imagine we somehow represented words using only two dimensions:

Now vectors aren't just telling us where something physically exists.

They are representing something about the properties or features of the thing.

And this is exactly the idea behind embeddings.

A word like "cat" can be represented by a high-dimensional vector:

$$\text{cat} = [0.21,\ -0.43,\ 0.72,\ \ldots]$$

Maybe another word "dog" has another vector:

$$\text{dog} = [0.18,\ -0.39,\ 0.69,\ \ldots]$$

These numbers don't individually mean something obvious to us.

But collectively, they encode useful patterns learned by the model.

Now Comes the Interesting Part: Dot Product

We have two vectors.

Let's say:

$$a = [3, 4]$$

and

$$b = [5, 2]$$

The dot product is simply:

$$a \cdot b = (3)(5) + (4)(2) = 15 + 8 = 23$$

That's it.

Multiply corresponding elements and add them together.

But why is this useful?

This is where the magic starts.

What Does a Dot Product Actually Tell Us?

There is a beautiful geometric interpretation of the dot product:

$$a \cdot b = |a||b|\cos(\theta)$$

Where:

• $|a|$ = magnitude of vector $a$
• $|b|$ = magnitude of vector $b$
• $\theta$ = angle between the two vectors

Now look at the important part:

$$\cos(\theta)$$

This tells us something about how aligned the two vectors are.

Same direction

The angle is close to $0^\circ$.

$$\cos(0^\circ) = 1$$

So the dot product is strongly positive.

Perpendicular

The angle is $90^\circ$.

$$\cos(90^\circ) = 0$$

So:

$$a \cdot b = 0$$

Opposite directions

The angle is $180^\circ$.

$$\cos(180^\circ) = -1$$

So the dot product becomes negative.

So we can roughly think of the dot product as asking:

"How much are these two vectors pointing in the same direction?"

But there is an important detail.

The raw dot product depends on both direction and magnitude.

For example, a huge vector pointing in the same direction can produce a much larger dot product than a tiny vector pointing in the same direction.

That's why, when we specifically want to measure directional similarity, we often normalize the vectors and use cosine similarity:

$$\text{cosine similarity} = \frac{a \cdot b}{|a||b|}$$

Now the magnitude is removed and we're mostly asking:

How aligned are these two vectors?

Let's Visualize It

Suppose we have:

$$a = [3, 4]$$

and:

$$b = [6, 8]$$

These two vectors point in exactly the same direction.

The second vector is basically just a scaled-up version of the first.

Their cosine similarity is:

$$1$$

because they are perfectly aligned.

Now imagine:

$$c = [-4, 3]$$

This vector is perpendicular to $a$.

Their dot product is:

$$(3)(-4) + (4)(3) = -12 + 12 = 0$$

The vectors are completely perpendicular.

And that is a very powerful property.

The dot product gives us a mathematical way of asking:

"How strongly is one vector aligned with another?"

And now we are getting very close to Attention.

From Dot Product to Attention

Remember our sentence:

The cat sat on the mat because it was tired.

Suppose the model is currently processing the word:

"it"

The model needs to figure out:

"Which other words should I pay attention to?"

This is where vectors become extremely useful.

Instead of thinking about words as just token IDs, the model represents them using vectors.

Then the Attention mechanism creates different representations called:

• Query
• Key
• Value

For the moment, don't worry too much about the names.

Think of the Query as:

"What am I looking for?"

And the Key as:

"What information do I contain?"

Then the model can use a dot product:

$$Q \cdot K$$

to calculate how strongly the query is aligned with each key.

If:

$$Q_{it} \cdot K_{cat}$$

is large, that means the model considers cat highly relevant to the query it.

If another word produces a smaller score, it receives less attention.

Eventually, these scores are converted into attention weights using softmax, and those weights determine how much information the model takes from each corresponding Value vector.

So, at a very high level:

Query
  │
  ├──── dot product ──── Key 1 → score
  ├──── dot product ──── Key 2 → score
  ├──── dot product ──── Key 3 → score
  └──── dot product ──── Key 4 → score

                 ↓
              Softmax
                 ↓
        Attention weights
                 ↓
       Weighted combination
             of Values

And suddenly that simple operation we learned in linear algebra:

$$a \cdot b$$

becomes one of the fundamental operations behind the Transformer architecture.

The Big Picture

This is why understanding the mathematics underneath neural networks matters.

At first glance, Attention can look like some complicated magical mechanism involving:

$$QK^{T}$$

Softmax.

Values.

Masks.

Multiple heads.

And a bunch of matrix multiplications.

But underneath all that complexity, one of the fundamental ideas is surprisingly simple:

Take vectors, compare them using their dot products, turn those scores into weights, and use those weights to decide what information to focus on.

That's the core intuition.

And once you understand vectors and dot products, the equations behind Self-Attention start looking much less scary.

In the next article, we'll take this one step further and actually break down Self-Attention from the ground up — what Query, Key, and Value actually mean, why we need all three, and how a model can use them to understand relationships between tokens in a sentence.

The rabbit hole continues.