Understanding the Transformer

This article explains the landmark paper Attention Is All You Need by Vaswani et al. (2017), which introduced the Transformer architecture that powers GPT, BERT, and nearly every modern language model. All interactive numbers below are small toy values chosen for illustration, not weights from a trained model.

Key Takeaways

Introduction

How do you teach a model to relate any two words in a sentence, no matter how far apart they sit? For years the answer was to walk the sentence left to right and hope the signal survived the trip. The Transformer threw that assumption out.

Its single idea fits in one sentence: every token looks at every other token and decides how much to take from each. That operation is called attention. Stack it with a small per-token network, repeat a few times, and you have the Transformer.

The roadmap: why the old sequential models hurt, how text becomes vectors, attention as a soft lookup, the scaled formula, where queries, keys and values come from, multiple heads, masking, the full block, word order, and finally why it won.

The Sequential Bottleneck

Before the Transformer, sequence tasks such as translation were dominated by recurrent neural networks (RNNs), especially LSTMs and GRUs. An RNN reads one token at a time and carries a running summary, the hidden state:

In words: the state at position $t$ is computed from the previous state and the current token. Position $t$ cannot start until position $t-1$ has finished.

That chain causes two problems.

• No parallelism. Within one sentence the steps must run in order, so a GPU sits mostly idle on long inputs.
• Long-range forgetting. Information from word 1 must survive every step to reach word 50. Training adjusts weights using the gradient, a measure of how much the error changes when a weight moves slightly, and that signal must also travel back through every step, shrinking as it goes.

From Words to Vectors

A network cannot read letters, so the first step turns text into numbers. The sentence is split into tokens: whole words or common word pieces. Each token has an ID in a fixed vocabulary (the paper used about 37,000 shared tokens for English-German). Each ID then picks one row from a learned table, the embedding: a list of $d_{\text{model}} = 512$ numbers that the model is free to tune during training.

From here on, “a token” means its vector. Everything the Transformer does is arithmetic on this stack of vectors, one per position.

Attention as a Soft Lookup

Attention lets each token ask a question of the whole sentence. It works like a fuzzy dictionary lookup with three roles:

• the query: what the current token is looking for;
• a key for every token: what that token can offer, used for matching;
• a value for every token: the content handed over if it is chosen.

The query is compared with every key to give a score. Scores can be any real number, so they pass through softmax: raise $e$ to each score, then divide by the total. The results are positive and add up to 1, and a larger score always gets a larger share. These shares are the attention weights. The output is the values mixed in those proportions.

Try it on a classic example from Google’s 2017 announcement of the Transformer. Who does “it” refer to?

Two things to notice. Every word gets some weight: attention is a soft blend, not a hard pick. And “it” reaches “animal” six positions away in one step, with no chain of hidden states in between.

Scaled Dot-Product Attention

Now the precise version. Stack all queries into a matrix $Q$, all keys into $K$ and all values into $V$, one row per token. The paper computes attention for every query at once:

Read it right to left in four steps:

1. Score. $QK^T$ holds every dot product $q \cdot k$: multiply matching entries and add them up. A large dot product means the two vectors point the same way.
2. Scale. Divide every score by $\sqrt{d_k}$, where $d_k$ is the length of each key.
3. Normalize. Softmax each row, so each query’s weights add up to 1.
4. Blend. Multiply by $V$: each output row is the weighted mix of the value rows.

The interactive below runs exactly these steps for one query, three tokens and $d_k = 2$, small enough to draw on paper.

Drag the query onto the key for “the”, at $[1, 0]$. The scores are $1, 0.5, -1$; divided by $\sqrt{2}$ they become $0.71, 0.35, -0.71$; softmax gives weights $0.51, 0.36, 0.12$; and the output is $0.51 \cdot [3, 0.5] + 0.36 \cdot [1, 3] + 0.12 \cdot [0, 0] = [1.90, 1.34]$. Point the query toward “cat” and the cat row takes over. That is the whole mechanism; everything else in the Transformer repeats it at scale.

Why scale the scores?

Suppose each entry of $q$ and $k$ is a random number with mean 0 and variance 1. A dot product adds up $d_k$ such products, so its variance is $d_k$ and its typical size (standard deviation) is $\sqrt{d_k}$. With $d_k = 512$ scores spread over roughly $\pm 22$. Softmax of numbers that far apart gives almost all the weight to the largest one.

That is bad for learning. When one weight is nearly 1 and the rest nearly 0, nudging a score barely changes any weight, so the gradient through softmax is close to zero and the query and key matrices stop learning. Dividing by $\sqrt{d_k}$ brings the spread back to about 1 at every size.

Where Queries, Keys and Values Come From

So far the queries, keys and values were given. In the Transformer they all come from the same token vector $x$, multiplied by three learned matrices:

In words: one vector, three different learned views of it. $W^Q$ learns what a token should ask for, $W^K$ how it should advertise itself, and $W^V$ what it should hand over. None of these roles is programmed; they emerge from training.

When queries, keys and values all come from the same sequence, it is self-attention: the sentence reads itself. When the queries come from one sequence and the keys and values from another, it is cross-attention; the decoder uses it to read the source sentence during translation.

Multi-Head Attention

One attention pattern can only blend in one way per token. But “it” needs to know who it refers to, the verb needs its subject, and every word benefits from knowing its neighbour. Multi-head attention runs several attentions side by side, each with its own $W^Q$, $W^K$, $W^V$, so each can learn a different relationship.

In words: each head projects the tokens down to a small space, runs attention there, and the head outputs are glued together and mixed by $W^O$.

The arithmetic is what makes this cheap. The paper uses $d_{\text{model}} = 512$ and $h = 8$ heads, so each head works in $d_k = 512 / 8 =$ 64 dimensions. Eight heads of 64 cost about the same as one head of 512, so the model gets eight views of the sentence for the price of one.

Masking: No Peeking Ahead

The encoder reads a sentence that already exists, so every token may look at every other token. The decoder writes a sentence, one token at a time. When it predicts word 3 it must not see word 3 or anything after it, or training would teach it to copy the answer.

The fix is a causal mask. Before softmax, every score where the key comes after the query is set to $-\infty$. Since $e^{-\infty} = 0$, those positions get exactly zero weight, and softmax spreads the full weight over the positions that remain.

The mask also explains a phrase in the paper’s Figure 1: “outputs (shifted right)”. During training the decoder is fed the target sentence moved one place to the right, with a start symbol in front. Position $i$ then sees the target up to word $i-1$ and is trained to predict word $i$. Thanks to the mask, all positions train in parallel on one forward pass.

At inference time there is no target, so the decoder generates in a loop, feeding each prediction back in:

The Transformer Block

Attention mixes information across positions. Each block pairs it with a second sub-layer that works on each position on its own, and wraps both in two pieces of plumbing.

• A residual connection adds a sub-layer’s output back onto its input: $x + \text{Sublayer}(x)$. The sub-layer only has to learn a change to the vector, and the gradient has a direct path back through the addition, which keeps deep stacks trainable.
• Layer normalization (LayerNorm) rescales each token’s vector to mean 0 and variance 1, then applies a learned scale and shift. It keeps numbers in a steady range from layer to layer.

A useful picture is a residual stream: each token’s vector flows straight up through the stack, and each sub-layer reads from it and adds its result back.

The second sub-layer is a small two-layer network applied to every position separately:

In words: widen each token’s vector from 512 to 2,048 numbers, zero out the negatives (the ReLU, $\max(0, \cdot)$), and project back to 512. Attention moves information between tokens; the feed-forward network processes what each token has gathered.

Encoder and decoder

The paper stacks six blocks on each side.

• Encoder block: self-attention (no mask) + feed-forward. It turns the source sentence into one context-rich vector per source token.
• Decoder block: masked self-attention over the output so far, then cross-attention, then feed-forward. In cross-attention the queries come from the decoder and the keys and values come from the encoder’s final output. It is how the word being written looks up the relevant words in the source sentence.

A final linear layer and softmax turn each decoder output vector into a probability for every token in the vocabulary.

Positional Encoding

Attention treats its input as a set: shuffle the tokens and each output just moves with its token, unchanged. That is a problem, since “dog bites man” and “man bites dog” would look identical. So before the first block, the Transformer adds a positional encoding to each embedding, giving every position a distinct fingerprint.

Think of each pair of dimensions as a clock hand turning at its own speed. Fast hands change from one position to the next and pin down fine position; slow hands barely move and give coarse position. Reading all the hands together tells you exactly where you are, the way hours, minutes and seconds together give the time.

For any fixed offset $k$, $PE_{pos+k}$ is a rotation of $PE_{pos}$, a linear function of it. The authors chose sinusoids hoping this would make relative positions easy to attend to.

Why It Won

The paper compares one layer of each type on three costs, for a sequence of $n$ tokens with vectors of size $d$.

Darker cells mean more sequential work or a longer path. Self-attention wins both: no chain of steps, and any two tokens are one hop apart. The work column is a trade-off. When the sentence is shorter than the vector size ($n < d$, true for most 2017 translation sentences with $d = 512$), $n^2 d$ is smaller than $n d^2$.

The paper trained on WMT 2014 English-German (4.5M sentence pairs) and English-French (36M pairs) using 8 NVIDIA P100 GPUs. The base model trained in 12 hours, the big model in 3.5 days.

BLEU measures overlap with human reference translations; higher is better. Shading is relative within each column. A single big Transformer beat the best previous ensemble (several models averaged) on both language pairs, and did it with far less compute:

What Attention Heads Learn

Interactive D drew head patterns by hand. The paper shows real ones from its trained model. None of this structure is programmed; it emerges from training on translation.

The Lineage: One Block, Three Descendants

The paper shipped a full encoder-decoder for translation. What followed took the stack apart. The block turned out to be reusable on its own, and the field split along the seam between encoder and decoder.

The mask is the main difference. Remove the causal mask and every position sees the full sentence: that is BERT, built for understanding. Keep the mask so each position sees only its past, and the model can generate one token at a time: that is GPT. Same block, one switch flipped, the same switch as in Interactive E.