Modern NLP encoders

BERT, RoBERTa, DistilBERT, DeBERTa, and ModernBERT

Introduction

Pooling real values 2 and 4 with a padded value 100 gives a wrong mean near 35.33. Applying the padding mask gives the correct mean 3, showing that architecture comparisons still depend on basic tensor correctness.

Learning goal

Compare modern BERT-family encoders and choose correct padding-mask, pooling, and pretrained-checkpoint adaptation strategies.

Before you start

BERT-style masked prediction, attention masks, tensor shapes, pooling, and pretrained checkpoints.

Lesson plan

  1. Review the original BERT recipe and compare later families by design goal.
  2. Trace ModernBERT-related changes without treating newer architecture as automatically better.
  3. Apply padding-safe pooling and decide between adaptation and training from scratch.

Lesson map: model, tensors, choice

  1. See what a masked encoder learns.
  2. Trace input, hidden-state, and vocabulary-logit shapes.
  3. Compare five BERT-family designs.
  4. Handle padding and sequence pooling correctly.
  5. Choose pretrained use, continued pretraining, or training from scratch.

Plain-language start

Example. Cover one word: The patient took the [MASK] after lunch. Words on both sides help you guess the hidden token.

Definition. A BERT-style encoder turns every input token into a context-aware vector. The vector for bank can differ between “river bank” and “bank account”.

Key difference. A normal BERT encoder sees earlier and later unmasked context together. A left-to-right generator cannot see future tokens while predicting the next token.

Best fit. Encoders naturally support classification, token tagging, retrieval, and extractive question answering. Filling selected masks does not make BERT a convenient long-form generator.

The original BERT recipe

Tokenization. Text becomes subword token IDs. A rare word can split into familiar pieces instead of becoming one unknown item.

  1. Input IDs have shape (B,T).
  2. Embeddings produce (B,T,D), where D is model width.
  3. Self-attention mixes information across real token positions.
  4. The masked-language head produces vocabulary logits (B,T,V).
  5. Cross-entropy uses selected mask positions, not visible tokens or padding.

Training target. The original hidden token supplies its own answer, so pretraining can use unlabelled text. Original BERT also used a next-sentence-prediction objective.

token IDs (B,T)
  -> token and position representations (B,T,D)
  -> bidirectional encoder layers (B,T,D)
  -> masked-token logits (B,T,V)
  -> cross-entropy only at chosen mask positions

One family, different design goals

Compare variants along one axis at a time: training objective, parameter count, attention design, supported context, and intended deployment cost. A newer publication date alone does not choose the best model for your data and hardware.

ModelMain changeRead the claim carefully
BERTBidirectional masked pretraining plus fine-tuning. Common checkpoints use WordPiece and learned absolute positions.The original recipe also used next sentence prediction.
RoBERTaLonger training, larger batches, more data, changing mask patterns, longer sequences, and no next sentence prediction.Its main contribution is a stronger training recipe, not a new attention layer.
DistilBERTA smaller student learns from a larger teacher through knowledge distillation.The paper’s 40% smaller, 60% faster, and 97% retained-capability figures belong to its test setup.
DeBERTaDisentangled content and position attention, plus an enhanced mask decoder.This is an architecture change, unlike RoBERTa’s mainly training-recipe change.
ModernBERTRoPE, GeGLU, alternating local/global attention, and support for unpadding and Flash Attention.It is one specific newer encoder, not a generic name for every modern BERT.
ModernBERT checkpoint facts from the official model card

The base and large models were trained on two trillion tokens of mainly English and code. Their native context limit is up to 8,192 tokens.

  • Base: 22 layers and 149 million parameters.
  • Large: 28 layers and 395 million parameters.
  • Training used sequences up to 1,024 tokens first, then extended to 8,192.

Why the ModernBERT changes matter

Full-attention cost. Every token compares with every token, so the attention matrix grows roughly with T * T.

Limits remain. The checkpoint is mainly English and code. The 8,192-token window is finite, long inputs still cost more, and task quality must be measured.

Padding masks are part of correctness

Predict the pooled value: if real token features are 2 and 4 and a padded feature is 100, an unmasked mean is about 35.3. The masked mean is 3. The large difference shows why padding is a correctness issue, not only wasted computation.

Definition. Padding makes different-length examples into one rectangular (B,T) tensor. Padding is storage, not text.

Mask. An attention_mask normally uses one for a real token and zero for padding.

  1. Start with hidden states (2,5,768) and mask (2,5).
  2. Broadcast the mask to (2,5,1).
  3. Multiply, then sum across the token axis.
  4. Divide each row by its real-token count.
  5. Result: pooled vectors have shape (2,768).
# Small runnable demonstration; these are scalar token features.
hidden = [[2.0, 4.0, 8.0, 100.0], [3.0, 9.0, 99.0, 99.0]]
masks = [[1, 1, 1, 0], [1, 1, 0, 0]]

means = []
for row, mask in zip(hidden, masks):
    real = [value for value, keep in zip(row, mask) if keep]
    means.append(sum(real) / len(real))
print(means)

# Expected: [4.666666666666667, 6.0]

This code was tested as plain Python. It shows why averaging every padded position would be wrong: the artificial values 100 and 99 must not enter the sequence summary.

[CLS] versus mean pooling

Need. A classifier or retriever often needs one sequence vector, but the encoder returns one vector per token.

Decision rule. Follow the checkpoint’s model card and training objective. Raw BERT [CLS] vectors are not automatically good sentence embeddings.

Pretrained use versus training from scratch

RouteWhat it doesWhen to consider it
Use or fine-tune pretrainedLoads architecture and learned weights; optionally updates them for labels.Default when compute and labelled data are limited.
Continue pretrainingRuns more masked-language training on selected domain text.Domain wording differs. Compare against direct fine-tuning because narrow data can hurt.
Train from scratchStarts with random weights and performs full pretraining.A new language, tokenizer, license need, or architecture justifies the high cost.

API check. ModernBertConfig() creates random model values. from_pretrained("answerdotai/ModernBERT-base") loads checkpoint weights.

Optional Hugging Face masked-token example

Requirement. The official model card says ModernBERT support starts with transformers>=4.48.0.

Verification note. This downloads a large checkpoint, so it was not executed in this project’s CPU-only checks. The API and model ID follow the official model card.

# pip install "transformers>=4.48.0" torch
from transformers import pipeline

fill_mask = pipeline(
    task="fill-mask",
    model="answerdotai/ModernBERT-base",
    device=-1,  # CPU
)

for result in fill_mask(
    "Plants create [MASK] through photosynthesis.",
    top_k=3,
):
    print(result["token_str"], round(result["score"], 4))

This demonstrates the pretraining head, not a sentiment classifier or retrieval model. For classification, load or fine-tune a sequence-classification head. For token labels, use a token-classification head and align word labels with subword tokens. For retrieval, use an appropriately trained embedding checkpoint rather than treating fill-mask logits as document vectors.

Mistakes to avoid

  • Do not send padding without the matching attention mask.
  • Do not calculate masked-token loss at visible or padded positions.
  • Do not mix a checkpoint with a tokenizer from another model family.
  • Do not call random initialization “using BERT pretrained knowledge”.
  • Do not assume the longest supported context is always the fastest or best choice.
  • Do not choose pooling by habit; follow the checkpoint and downstream objective.

Solved practice

1. What are the shapes for batch 8, length 128, width 768, vocabulary 50,368?

Input IDs and the attention mask are (8,128). Encoder hidden states are (8,128,768). Masked-language logits are (8,128,50368). If 90 positions were selected for loss, gathered logits can be (90,50368) with targets (90,).

2. Why is mean pooling without a mask incorrect?

Padding positions are storage, not language. Including them changes the mean according to batch padding length. Masked mean pooling sums only real-token vectors and divides by the number of real tokens.

3. Which model would you always choose?

None. DistilBERT may suit a strict latency budget. ModernBERT may suit longer English or code input. A multilingual checkpoint may suit another language. Measure task quality, memory, latency, licensing, and data fit on your own validation set.

4. Does an 8,192-token window prove understanding across 8,192 tokens?

No. It means the model accepts that many tokens under its documented setup. Whether it uses distant information well depends on training, attention design, and task. Evaluate long-context behaviour directly.

Recap

Primary sources: BERT paper, RoBERTa paper, DistilBERT paper, DeBERTa paper, ModernBERT paper, the official Answer.AI ModernBERT model card, and Hugging Face ModernBERT documentation.