FLAGSHIP PAPER WALKTHROUGH

Attention Is All You Need: a careful paper walkthrough

Introduction

The 2017 Transformer paper replaced sequence-aligned recurrence and convolution with attention in an encoder-decoder translation system. Reading it carefully shows both the enduring design and the parts that differ from modern decoder-only language models.

Learning goal: Trace the paper's architecture and evidence, reproduce its central attention equation in a tiny experiment, and separate reported claims from later interpretation.
Before you start: Encoder-decoder models, matrix shapes, softmax, residual connections, and the difference between training and autoregressive inference.

Lesson plan

Reconstruct the translation problem and the full encoder-decoder data path.
Work through scaled dot-product attention, multi-head projections, and positions.
Read the reported evidence with its scope, limitations, and later-model differences.

Download the tiny experiments · Read the primary paper

The problem

Paper claim. Vaswani and colleagues proposed the Transformer as a sequence-transduction architecture based on attention rather than sequence-aligned recurrent or convolutional layers. The original application was machine translation: encode a source sentence, then decode a target sentence autoregressively.

Our explanation. Earlier recurrent systems carried a hidden state from one position to the next. That creates a serial path through the sequence during training. The Transformer lets all source positions, and all known target positions, form their representations in parallel within a layer.

This does not make every part of generation parallel. During inference, an autoregressive decoder still emits the next token only after earlier output tokens are known. The paper's main parallelism advantage applies to training computations within a known source or shifted target sequence.

Paper context. The arXiv record shows the first submission on 12 June 2017. The current arXiv page is revision 7 from 2023. When this walkthrough says “the paper,” it refers to the architecture and experiments reported in that record, not every later system called a Transformer.

Work through a small example

Suppose the source has three tokens and the target prefix has two known tokens. The encoder produces three vectors. The decoder uses three attention operations for different purposes.

Encoder self-attention: each source position can read all three source positions.
Masked decoder self-attention: target position 0 can read only position 0; target position 1 can read positions 0 and 1.
Encoder-decoder attention: each decoder position queries all three encoder outputs.

Our explanation. “Attention-only” did not mean that every operation was attention. Each layer also used a position-wise feed-forward network, residual connections, and layer normalization. Embeddings, positional encodings, and an output projection were also essential.

The general rule

Scaled dot-product attention

Paper definition. Queries and keys have width d_k; values have width d_v. For matrices of many queries, keys, and values:

Attention(Q,K,V) = softmax(QK^T / √d_k)V

Our explanation. Each dot product measures compatibility. Softmax turns one query row into non-negative shares that sum to one. The final row is a weighted mixture of value vectors.

Paper motivation. If query and key components are independent with mean zero and variance one, their unscaled dot product has variance d_k. Dividing by √d_k keeps the score variance near one and avoids pushing softmax too easily into very small-gradient regions.

Multi-head attention

Paper design. The base model used h=8 heads, d_model=512, and d_k=d_v=64. Learned projections produced separate Q, K, and V spaces per head. The head outputs were concatenated and projected again.

Our explanation. Heads do not receive hand-written roles. Their different learned projections make multiple compatibility patterns possible. The paper argued this avoids some averaging limitation of one full-width head, but an attention map alone is not proof of a causal or linguistic role.

Order without recurrence

Paper design. Fixed sine and cosine positional encodings were added to token embeddings. The frequencies formed a geometric progression. The authors also tried learned positions and reported nearly identical development results in their ablation table.

Our explanation. Self-attention without a position signal is permutation-equivariant: reordering input rows simply reorders output rows. Position vectors break that symmetry. The paper hypothesized that sinusoids could help represent relative offsets and extrapolate beyond training lengths; this was a design reason, not a universal proof of extrapolation.

Implement and inspect

The tiny program implements the paper's equation directly and measures score variance for random queries and keys. It also constructs the fixed positional encoding.

def scaled_dot_product_attention(query, key, value, allowed=None):
    scores = query @ key.transpose(-2, -1) / math.sqrt(query.shape[-1])
    if allowed is not None:
        scores = scores.masked_fill(~allowed, torch.finfo(scores.dtype).min)
    weights = torch.softmax(scores, dim=-1)
    if allowed is not None:
        weights = weights.masked_fill(~allowed, 0.0)
    return weights @ value, weights

Run python examples/flagship/attention_paper_experiments.py. For random independent unit-variance components, the experiment should show unscaled score variance growing roughly with width while scaled variance stays near one.

score variance: width, unscaled, scaled
4    approximately 4     approximately 1
16   approximately 16    approximately 1
64   approximately 64    approximately 1
position encoding shape: (5, 8)

Tiny-experiment result. This supports the paper's variance calculation under its stated random-variable assumption. It does not show that every trained head has independent unit-variance components, nor does it compare translation quality.

Engineering checks

Check mask direction. In decoder self-attention, position i may use positions up to i, not after it. Apply the mask before softmax.
Check head reshape order. A tensor goes from (B,T,d_model) to (B,h,T,d_k). A reshape without the needed transpose can mix token and head axes.
Check score scaling. Divide by the square root of per-head key width, not model width.
Check residual dimensions. Every sublayer output must return to d_model before adding the residual.
Check the architecture version. The paper used layer normalization after residual addition. Many later systems use pre-normalization. They are not the same computation.
Check training versus inference claims. Parallel target-position training uses the known shifted target plus a causal mask. Free-running decoding remains sequential across generated tokens.

Go deeper

What the paper actually built

Paper specification. Both encoder and decoder used six layers in the base model. The feed-forward inner width was 2048. The decoder added encoder-decoder attention. The paper tied embedding and pre-softmax weights, multiplied embeddings by √d_model, used Adam with a 4,000-step warm-up schedule, dropout, and label smoothing.

Our explanation. A modern decoder-only language model removes the encoder and cross-attention, often changes normalization placement, position method, activation, and attention layout, and adds an inference cache. Those systems descend from the paper's ideas but are not the exact 2017 architecture.

Why self-attention was attractive

Paper analysis. For sequence length n and representation width d, the paper listed self-attention complexity as O(n²d), sequential operations as O(1) within a layer, and maximum path length between positions as O(1). A recurrent layer was listed as O(nd²) complexity with O(n) sequential operations and path length.

Our explanation. These expressions describe an architectural comparison under particular dimensions and operations. Dense self-attention's quadratic sequence cost becomes a major limitation for long contexts. “Constant path length” does not mean constant compute or memory.

Reported evidence

Paper result. The paper reported 28.4 BLEU for its big model on WMT 2014 English-to-German. Its result table lists 41.8 BLEU on English-to-French. The big models trained for 3.5 days on eight P100 GPUs; the base setup was reported as 100,000 steps, about 12 hours on the same GPU count.

Scope note. These are the authors' reported results under their datasets, tokenization, checkpoint averaging, beam search, and 2017 comparison set. This walkthrough has not rerun those experiments. A tiny CPU attention calculation cannot validate BLEU, training cost, or state-of-the-art status.

Paper ablations. The reported development experiments found one head worse than the best tested multi-head setting, too many heads also worse, reduced key width harmful, larger models better in the tested range, dropout useful, and learned positions close to sinusoidal positions. These are results from that setup, not laws for every task and scale.

Interpretation needs restraint

The appendix visualized heads that appeared related to long-distance dependencies, anaphora, and sentence structure. That is evidence about observed patterns. It does not establish that attention weights are complete explanations of model decisions.

The paper demonstrated translation and constituency parsing. It did not train a chat assistant, use a decoder-only next-token objective at modern scale, implement retrieval, or establish the behavior of today's long-context systems.

Practice and recap

Question: During teacher-forced training, the decoder receives target tokens [BOS, A, B] and predicts [A, B, EOS]. Which input positions may the representation at decoder position 1 attend to, and why can all three positions still be computed in one layer call?

Worked answer

Position 1 may attend to decoder inputs at positions 0 and 1: BOS and A. It must not read position 2, which contains B, because B is the target it is learning to predict at that step. The full shifted input is already known during training, so all position rows can be computed together while the causal mask blocks each illegal future connection.

Keep these five ideas

The paper introduced an encoder-decoder Transformer for sequence transduction.
Scaled dot-product attention uses QK^T/√d_k, softmax, then a value mixture.
Multi-head projections, position signals, feed-forward layers, residual paths, and normalization are all part of the model.
Training positions can be parallel while autoregressive generation remains sequential across tokens.
The paper's translation evidence and our tiny CPU checks answer different questions.

References and reading limits

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. arXiv:1706.03762, first submitted 2017.
The paper's accessible HTML rendering, used here to check equations, architecture settings, complexity claims, training details, and reported results.

This walkthrough is original teaching text. It paraphrases and attributes paper facts, and labels our explanations and experiments separately. It is not a substitute for the primary paper and does not claim to reproduce its full training run.