Tensor by Tensor

02.28 · UNIT 09 · Transformer architecture and BERT-family encoders · Lab

Lab: build and test every transformer block

A transformer-block lab proves shapes, masks, residuals, and gradients before stacking blocks.

PLAIN-LANGUAGE INTRODUCTION

The idea

A transformer-block lab proves shapes, masks, residuals, and gradients before stacking blocks.

A worked example

For three tokens, a causal attention matrix can be [[1,0,0],[0.4,0.6,0],[0.2,0.3,0.5]].

The input

A batch tensor and explicit padding or causal masks.

The result

Same-shaped block output plus test evidence for every rule.

Why you use it

Small tests reveal silent mask and shape errors early.

A limit to remember

A correct output shape does not prove a correct attention mask.

Check your understanding

  1. Use input shape (batch=1, sequence=3, width=4). The block must preserve this shape.
  2. The shown causal matrix has zeros above its diagonal. Future positions receive no weight.
  3. Each row sums to 1: 1, 0.4+0.6, and 0.2+0.3+0.5.
  4. Test residual identity with zeroed sublayer outputs. Test gradients for every intended parameter.
  5. Common mistake: a mask with reversed true and false meanings can expose every future token.
Open the detailed notes ↗