02.25 · UNIT 07 · Transformer architecture and BERT-family encoders · Lesson
Encoder-only models and masked-token objectives
A BERT-style encoder predicts hidden tokens using visible context on both sides.
PLAIN-LANGUAGE INTRODUCTION
What is this?
A BERT-style encoder predicts hidden tokens using visible context on both sides.
One simple example
Input “cats [MASK] softly” has target “sleep”. The encoder can read “cats” and “softly”.
What goes in?
Token IDs with selected positions replaced by mask tokens.
What comes out?
Vocabulary scores at every position, with loss on chosen masked positions.
Why does it matter?
Bidirectional context helps learn useful token representations.
What is it not?
This objective does not train left-to-right text generation directly.
WORK THROUGH THE IDEA
See the idea in more detail
- BERT is an encoder-only transformer family. It can attend left and right within visible input.
- Replace “sleep” with
[MASK]in “cats sleep softly”. The target remains “sleep”. - The model receives “cats
[MASK]softly”. It returns one score per vocabulary item at the masked position. - Cross-entropy rewards a higher score for “sleep”. Other selected positions can use the same objective.
- Common mistake: calculate loss only on intended prediction positions, not ordinary visible tokens.