02.30 · UNIT 09 · Transformer architecture and BERT-family encoders · Lesson
Encoder-only models and masked-token objectives
A BERT-style encoder predicts hidden tokens using visible context on both sides.
PLAIN-LANGUAGE INTRODUCTION
The idea
A BERT-style encoder predicts hidden tokens using visible context on both sides.
A worked example
Input “cats
[MASK] softly” has target “sleep”. The encoder can read “cats” and “softly”.The input
Token IDs with selected positions replaced by mask tokens.
The result
Vocabulary scores at every position, with loss on chosen masked positions.
Why you use it
Bidirectional context helps learn useful token representations.
A limit to remember
This objective does not train left-to-right text generation directly.
Check your understanding
- BERT is an encoder-only transformer family. It can attend left and right within visible input.
- Replace “sleep” with
[MASK]in “cats sleep softly”. The target remains “sleep”. - The model receives “cats
[MASK]softly”. It returns one score per vocabulary item at the masked position. - Cross-entropy rewards a higher score for “sleep”. Other selected positions can use the same objective.
- Common mistake: calculate loss only on intended prediction positions, not ordinary visible tokens.