Inside Transformers, Part 1: Encoder, Decoder, and Attention

The Transformer was introduced as an encoder-decoder architecture for sequence-to-sequence tasks. The original design, described in Attention Is All You Need, uses attention instead of recurrence or convolution to connect tokens.

This series starts with that original design. Many current language models are decoder-only, so they use causal self-attention and generate one token at a time. The encoder-decoder Transformer is still the clearest place to learn the parts and how they fit together.

The original paper showed that the Transformer could process the positions of a training sequence in parallel inside each layer. Generation is different: a decoder must produce one target token, feed it back as context, and then produce the next. That difference between parallel training and autoregressive generation will matter throughout this series.

Before looking at the architecture, it helps to see where attention came from. Here are the other parts of the series:

• Inside Transformers, Part 2: Positional information
• Inside Transformers, Part 3: Scaled dot-product attention
• Inside Transformers, Part 4: Multi-head and masked attention
• Inside Transformers, Part 5: Layer normalization and residual connections

Attention mechanism brief history

Recurrent neural networks (RNNs) process a sequence step by step and carry a hidden state from one step to the next. Long Short-Term Memory networks (LSTMs) and Gated Recurrent Units (GRUs) add gates that control which information is kept. These models can represent long dependencies, but the sequential computation limits parallelism.

Sequence-to-sequence models introduced an encoder-decoder pattern for tasks such as translation. The encoder reads the source sequence and the decoder writes the target sequence. Early versions passed the source through one fixed-size context vector, which made long inputs difficult to represent.

In Neural Machine Translation by Jointly Learning to Align and Translate, Bahdanau, Cho, and Bengio used attention so each decoder step could form a context vector from the encoder’s sequence of representations. The decoder learned which source positions to weigh for its current output.

The Transformer replaced those recurrent connections with self-attention layers. Each token can read other permitted tokens in the same sequence, and the model can compute the positions of a training sequence in parallel. The mechanism and its masking rules are covered in the next parts.

Transformer architecture

The original Transformer has an encoder stack and a decoder stack. The encoder maps a source sequence to contextual representations. The decoder uses masked self-attention over the target prefix and cross-attention over the encoder output before producing the next target token.

1. Input and tokenization. A tokenizer maps text to integer token IDs. A token can be a word, a piece of a word, or punctuation.

2. Embedding. An embedding table maps each ID to a learned vector. The vector is the model’s starting representation for that token. It is not guaranteed to contain a human-readable meaning by itself; useful structure is learned during training.

3. Positional information. Self-attention by itself does not encode sequence order. The original Transformer adds a sinusoidal positional vector to each token embedding. Later models use other positional representations, including learned embeddings and rotary position embeddings.

4. Self-attention. Each position builds a weighted combination of value vectors from the positions it is allowed to read. In an encoder, a position can read the whole source sequence. In a decoder, a causal mask hides future target positions.

5. Multi-head attention. The model runs several attention heads with separate learned projections. Each head can represent a different relation, and their outputs are concatenated and projected back to the model width.

6. Position-wise feed-forward network. After attention, a small feed-forward network transforms each position independently. It does not mix information between positions.

7. Residual connection and layer normalization. The original paper wraps each sub-layer with a residual addition followed by layer normalization, written as LayerNorm(x + Sublayer(x)). Residual paths give later layers a direct route to the earlier representation and often make optimization easier. They do not guarantee that gradients cannot vanish.

8. Encoder and decoder stacks. The encoder repeats self-attention and feed-forward sub-layers. Each decoder layer repeats masked self-attention, cross-attention, and a feed-forward sub-layer. The original paper used six layers in each stack for its base configuration.

9. Output projection. The decoder’s final representation is mapped to one logit per vocabulary token. Softmax turns those logits into a probability distribution, and the next target token is selected from it during generation.

The original Transformer and a modern decoder-only language model share the attention calculation, but their stacks and masks differ. Part 2 adds positional information, and Parts 3 and 4 derive the attention operations in detail.