02.28 · UNIT 09 · Transformer architecture and BERT-family encoders · Lab
Lab: build and test every transformer block
A transformer-block lab proves shapes, masks, residuals, and gradients before stacking blocks.
PLAIN-LANGUAGE INTRODUCTION
The idea
A transformer-block lab proves shapes, masks, residuals, and gradients before stacking blocks.
A worked example
For three tokens, a causal attention matrix can be
[[1,0,0],[0.4,0.6,0],[0.2,0.3,0.5]].The input
A batch tensor and explicit padding or causal masks.
The result
Same-shaped block output plus test evidence for every rule.
Why you use it
Small tests reveal silent mask and shape errors early.
A limit to remember
A correct output shape does not prove a correct attention mask.
Check your understanding
- Use input shape
(batch=1, sequence=3, width=4). The block must preserve this shape. - The shown causal matrix has zeros above its diagonal. Future positions receive no weight.
- Each row sums to
1:1,0.4+0.6, and0.2+0.3+0.5. - Test residual identity with zeroed sublayer outputs. Test gradients for every intended parameter.
- Common mistake: a mask with reversed true and false meanings can expose every future token.