Tensor by Tensor

02.27 · UNIT 09 · Transformer architecture and BERT-family encoders · Lesson

Residual connections, normalization, and feed-forward layers

A transformer block adds attention and feed-forward updates through residual paths.

PLAIN-LANGUAGE INTRODUCTION

The idea

A transformer block adds attention and feed-forward updates through residual paths.

A worked example

Start with x=[1,3]. Add attention update [0.5,0.5] to get [1.5,3.5]. Normalization gives about [−1,1].

The input

A token tensor with shape (batch, sequence, width).

The result

A tensor with the same shape and updated token representations.

Why you use it

Residual paths preserve an easy route while sublayers learn changes.

A limit to remember

Normalization does not make every token vector identical.

Check your understanding

  1. Attention returns an update with the same shape as its input. Here, use [0.5,0.5].
  2. The residual addition gives [1,3]+[0.5,0.5]=[1.5,3.5].
  3. Layer normalization centers and scales features within this token. Ignoring epsilon, the result is [−1,1].
  4. A feed-forward network then transforms each token position separately. Another residual path adds its update.
  5. Common mistake: normalizing across the batch axis changes examples using other examples.
Open the detailed notes ↗