02.26 · UNIT 07 · Transformer architecture and BERT-family encoders · Lesson
BERT, RoBERTa, DistilBERT, DeBERTa, and ModernBERT
BERT-family encoders keep bidirectional attention while changing training, size, or attention details.
PLAIN-LANGUAGE INTRODUCTION
What is this?
BERT-family encoders keep bidirectional attention while changing training, size, or attention details.
One simple example
The same text “cats sleep” may produce different vectors in BERT, RoBERTa, DistilBERT, DeBERTa, and ModernBERT.
What goes in?
Text processed by the exact tokenizer required by one chosen checkpoint.
What comes out?
Contextual token vectors or task-head predictions.
Why does it matter?
Family differences affect speed, memory, token limits, and suitable tasks.
What is it not?
ModernBERT is one model family. It is not a name for every recent encoder.
WORK THROUGH THE IDEA
See the idea in more detail
- All five families produce token vectors that depend on surrounding text.
- BERT established masked-token pretraining. RoBERTa changed the training recipe. DistilBERT targets a smaller model.
- DeBERTa separates parts of content and position attention. ModernBERT changes several architecture and training choices.
- For “cats sleep”, use each checkpoint’s own tokenizer. Then compare the task output you actually need.
- Common mistake: mixing a tokenizer from one checkpoint with weights from another can change every input ID.