Inside Transformers, Part 2: Positional information

Part 1 introduced the original Transformer encoder-decoder model. This part focuses on one missing piece: self-attention compares vectors, but a comparison alone does not tell the model where each token appeared.

Absolute positional encoding

The original Transformer adds a fixed sinusoidal vector to each token embedding. For position pos and model width d_model, the paper defines

PE(pos, 2i)   = sin(pos / 10000^(2i / d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model))

Here, pos is the token position, i indexes pairs of dimensions, and d_model is the embedding width. The position is usually zero-indexed. Even dimensions use sine and odd dimensions use cosine.

The functions use different wavelengths, so nearby positions have related vectors while the model can still distinguish positions across the sequence. The original paper chose the sinusoidal version because it may allow the model to extrapolate to sequence lengths longer than those seen during training. The paper’s main result concerns the method in its translation experiments; it does not make positional encodings universally best.

For a small teaching example, suppose a tokenizer maps I love cats to [9, 27, 301] and the embedding width is 6. The token vectors below are illustrative values, not values learned by a particular model:

I     -> [ 0.1,  0.3,  0.9, -0.4,  0.5,  0.8]
love  -> [-0.3,  0.8, -0.6,  0.1,  0.7, -0.8]
cats  -> [ 0.2, -0.9,  0.6,  0.3, -0.4, -0.5]

For d_model = 6, the six positional components are

PE(pos, 0) = sin(pos / 10000^(0/6)) = sin(pos)
PE(pos, 1) = cos(pos / 10000^(0/6)) = cos(pos)
PE(pos, 2) = sin(pos / 10000^(2/6))
PE(pos, 3) = cos(pos / 10000^(2/6))
PE(pos, 4) = sin(pos / 10000^(4/6))
PE(pos, 5) = cos(pos / 10000^(4/6))

At pos = 1, this gives approximately

[0.841471, 0.540302, 0.046399, 0.998923, 0.002154, 0.999998]

With zero-indexed positions, I is at position 0, love is at position 1, and cats is at position 2. The displayed vector is therefore added element by element to the illustrative embedding for love; the vectors for positions 0 and 2 are added to I and cats. The resulting vectors enter the encoder or decoder layers.

Relative position methods

The original Transformer uses absolute sinusoidal encodings. Later work introduced relative position representations. Shaw, Uszkoreit, and Vaswani add learned representations for the distance between a query position and a key position. Other models use a scalar attention bias for that distance, while rotary position embedding rotates queries and keys so their dot product depends on the relative angle. These are related goals with different calculations, so “relative positional encoding” does not name one universal implementation.

Define the offset as r = j - i, where i is the query position and j is the key position. In a full sequence of length 3, with positions 0, 1, and 2, the possible offsets are

-2, -1, 0, +1, +2

For example, query position 2 and key position 0 have offset 0 - 2 = -2, while query position 0 and key position 2 have offset 2 - 0 = +2. A causal decoder cannot read the latter pair, but the offset still exists in the full position grid. Causality is a separate restriction on which keys a query may use.

A toy relative-position table might look like this:

r = -2 -> [ 0.2, -0.4,  0.3, -0.1,  0.5, -0.3]
r = -1 -> [ 0.1, -0.2,  0.3, -0.1,  0.2, -0.3]
r =  0 -> [-0.1,  0.3, -0.2,  0.4, -0.3,  0.1]
r = +1 -> [ 0.0,  0.2, -0.1,  0.3,  0.1, -0.2]
r = +2 -> [-0.2,  0.1,  0.2, -0.3,  0.0,  0.4]

These numbers are illustrative. A particular model may learn vectors, add scalar biases to attention scores, rotate query and key pairs, clip long distances, or combine position mechanisms. For a sequence of length 10, the full offset range under this convention is -9 through +9; with causal masking, a query at position i can use only offsets through 0.

Both absolute and relative methods give the attention calculation information about order. Part 3 derives the query, key, and value projections and explains why scaled dot-product attention divides by sqrt(d_k).