02.23 · UNIT 07 · Transformer architecture and BERT-family encoders · Lab
Lab: build and test every transformer block
A transformer-block lab proves shapes, masks, residuals, and gradients before stacking blocks.
PLAIN-LANGUAGE INTRODUCTION
What is this?
A transformer-block lab proves shapes, masks, residuals, and gradients before stacking blocks.
One simple example
For three tokens, a causal attention matrix can be [[1,0,0],[0.4,0.6,0],[0.2,0.3,0.5]].
What goes in?
A batch tensor and explicit padding or causal masks.
What comes out?
Same-shaped block output plus test evidence for every rule.
Why does it matter?
Small tests reveal silent mask and shape errors early.
What is it not?
A correct output shape does not prove a correct attention mask.
WORK THROUGH THE IDEA
See the idea in more detail
- Use input shape
(batch=1, sequence=3, width=4). The block must preserve this shape. - The shown causal matrix has zeros above its diagonal. Future positions receive no weight.
- Each row sums to
1:1,0.4+0.6, and0.2+0.3+0.5. - Test residual identity with zeroed sublayer outputs. Test gradients for every intended parameter.
- Common mistake: a mask with reversed true and false meanings can expose every future token.