Attention Is All You Need: reading the Transformer paper

The paper studies sequence transduction with an encoder-decoder architecture. Its central proposal is to remove recurrence and convolution from the core architecture and rely on attention mechanisms instead.

The central operation compares queries with keys, converts the scores into weights, and combines the values. Scaling by the square root of the key dimension keeps the score magnitudes better behaved as dimensionality grows.

Queries express what the current position is looking for, keys provide features to compare against, and values carry the information that is mixed after the attention weights are computed. Thinking in these three roles is often more useful than memorizing the letters.

Instead of forcing one attention operation to capture every relationship, the architecture uses several heads with their own learned projections, then combines the resulting representations.

• Position information is added because attention by itself does not impose recurrence-based order.
• Residual connections and layer normalization are used around the main sublayers.
• The original architecture contains feed-forward sublayers in both encoder and decoder blocks.
• The decoder uses masking so a position cannot attend to future target tokens during autoregressive generation.

The original paper is evidence for the architecture it proposed and the experiments it reported. It should not be read as a description of every later Transformer or modern LLM implementation. Distinguish the paper's design choices from changes introduced by later work.

HowLLMsWork in this site follows the same educational principle from the implementation side: make tokenization, attention, training, generation, and inference mechanics explicit enough to inspect rather than hiding them behind a high-level API.