02.22 · UNIT 07 · Transformer architecture and BERT-family encoders · Lesson

Residual connections, normalization, and feed-forward layers

A transformer block adds attention and feed-forward updates through residual paths.

PLAIN-LANGUAGE INTRODUCTION

What is this?

A transformer block adds attention and feed-forward updates through residual paths.

One simple example

Start with x=[1,3]. Add attention update [0.5,0.5] to get [1.5,3.5]. Normalization gives about [−1,1].

A token tensor with shape (batch, sequence, width).

A tensor with the same shape and updated token representations.

Residual paths preserve an easy route while sublayers learn changes.

Normalization does not make every token vector identical.

WORK THROUGH THE IDEA

Attention returns an update with the same shape as its input. Here, use [0.5,0.5].
The residual addition gives [1,3]+[0.5,0.5]=[1.5,3.5].
Layer normalization centers and scales features within this token. Ignoring epsilon, the result is [−1,1].
A feed-forward network then transforms each token position separately. Another residual path adds its update.
Common mistake: normalizing across the batch axis changes examples using other examples.