Attention mathematically: why Q, K, and V?

Build attention from linear projections and matrix products to softmax and weighted aggregation.

Mathematics for AIIntermediate

Learning goal

Understand Q, K, V, attention scores, softmax, and weighted aggregation well enough to explain the full computation chain.

The input X is mapped through three parameter matrices. Q and K are used to construct scores; V carries the information that will be aggregated.

Attention projections

Q = X W_{Q}, K = X W_{K}, V = X W_{V}

Score matrix

S = Q K^{T}

Entry (i,j) measures the compatibility between query i and key j.

Scaled dot-product

S = \frac{Q K^{T}}{\sqrt{d_{k}}}

Scaling controls the magnitude of dot products as vector dimension grows, helping keep softmax numerically usable.

Attention weights

A = softmax (\frac{Q K^{T}}{\sqrt{d_{k}}})

Attention output

Z = A V

Each row of A sums to 1, so it can be interpreted as a distribution of weights over tokens. Z is the weighted combination of Value vectors.

HowAttentionWorks

Direct focus on attention mechanics.

HowTransformersWork

Continue from attention to Transformers.

HowLLMsWork

Connect the mathematics to language models.

A concept can be followed from equation to implementation.