02.27 · UNIT 09 · Transformer architecture and BERT-family encoders · Lesson
Residual connections, normalization, and feed-forward layers
A transformer block adds attention and feed-forward updates through residual paths.
PLAIN-LANGUAGE INTRODUCTION
The idea
A transformer block adds attention and feed-forward updates through residual paths.
A worked example
Start with
x=[1,3]. Add attention update [0.5,0.5] to get [1.5,3.5]. Normalization gives about [−1,1].The input
A token tensor with shape (batch, sequence, width).
The result
A tensor with the same shape and updated token representations.
Why you use it
Residual paths preserve an easy route while sublayers learn changes.
A limit to remember
Normalization does not make every token vector identical.
Check your understanding
- Attention returns an update with the same shape as its input. Here, use
[0.5,0.5]. - The residual addition gives
[1,3]+[0.5,0.5]=[1.5,3.5]. - Layer normalization centers and scales features within this token. Ignoring epsilon, the result is
[−1,1]. - A feed-forward network then transforms each token position separately. Another residual path adds its update.
- Common mistake: normalizing across the batch axis changes examples using other examples.