Lesson 12

Encoder-only models and masked-token objectives

Introduction

In 'the cat masked home,' the hidden word can use context on both sides. An encoder-only model learns this masked-token objective, then its contextual features can support tasks such as classification.

Learning goal

Build a tiny masked-token prediction exercise and trace how contextual encoder features transfer to downstream classification.

Before you start

Token IDs, attention, transformer blocks, logits, cross-entropy, and padding masks.

Lesson plan

  1. Contrast bidirectional masked-token prediction carefully with causal next-token sequence prediction.
  2. Trace token, hidden, and vocabulary-logit shapes through a tiny encoder.
  3. Apply loss only at masked positions, then reuse encoder features for classification.

Why hide tokens?

BERT learns context by predicting selected tokens that are hidden.

  1. Original text: the cat went home
  2. Hide a token: the cat [MASK] home
  3. Read both sides: use the cat on the left and home on the right.
  4. Predict the hidden token: compare the prediction with went.

Bidirectional context means reading both sides. This helps with tasks such as text classification, search, and name recognition.

BERT and GPT ask different questions

For this small BERT-style exercise, calculate the loss only at the selected masked positions.

Important terms

EncoderA transformer stack that makes context-aware vectors for input tokens.
Masked language modelingHide selected tokens and predict their original IDs.
BidirectionalUse earlier and later context together.
Special tokenA reserved item such as [MASK], padding, or a classification marker.
PretrainingLearn broad patterns from a large general dataset.
Fine-tuningUpdate a pretrained model for a smaller target task.

Follow the shapes

  1. Token IDs: (B,T) — B sequences, T positions.
  2. Token + position embeddings: (B,T,D) — D features per position.
  3. Encoder output: (B,T,D) — same shape, now with context.
  4. Prediction scores: (B,T,V) — V vocabulary choices per position.
  5. Loss: compare scores and targets only at masked positions.

Worked example: tiny masked encoder

This small CPU model teaches the data flow. It is not a useful pretrained BERT. Token ID 5 is reserved for [MASK]. Unlike GPT, the transformer receives no causal mask.

import torch
from torch import nn

class TinyMaskedModel(nn.Module):
    def __init__(self, vocabulary_size, maximum_length, model_size=12):
        super().__init__()
        self.token_embedding = nn.Embedding(vocabulary_size, model_size)
        self.position_embedding = nn.Embedding(maximum_length, model_size)
        layer = nn.TransformerEncoderLayer(
            d_model=model_size,
            nhead=3,
            dim_feedforward=24,
            dropout=0.0,
            batch_first=True,
        )
        self.encoder = nn.TransformerEncoder(layer, num_layers=2)
        self.language_head = nn.Linear(model_size, vocabulary_size)

    def forward(self, token_ids):
        length = token_ids.shape[1]
        positions = torch.arange(length, device=token_ids.device)
        x = self.token_embedding(token_ids)
        x = x + self.position_embedding(positions)
        encoded = self.encoder(x)
        return self.language_head(encoded)

torch.manual_seed(12)
model = TinyMaskedModel(vocabulary_size=6, maximum_length=5)
mask_id = 5
inputs = torch.tensor([[0, 1, mask_id, 3, 4]], dtype=torch.long)
logits = model(inputs)

print(tuple(inputs.shape))
print(tuple(logits.shape))

# Expected:
# (1, 5)
# (1, 5, 6)

The representation at position two can collect information from positions on both sides. The language head produces predictions at every position. For this simple example, we select the masked location.

Loss only where the answer was hidden

Count before calculating: if a sequence has six positions and two are masked, the masked-language loss should average two token losses. An average over all six positions gives extra influence to four easy or irrelevant terms.

original_token_id = torch.tensor([2])
masked_logits = logits[:, 2, :]
loss = nn.functional.cross_entropy(masked_logits, original_token_id)
loss.backward()

print(tuple(masked_logits.shape))  # Expected: (1, 6)
print(loss.ndim)                   # Expected: 0

A full data pipeline chooses several mask positions in each sequence and records their original IDs. Unmasked or padding positions are commonly assigned an ignore value in the target tensor so they do not affect cross-entropy. If the original token stays visible at the masked position, the task leaks its own answer.

A suspicious symptom of leakage is near-perfect training accuracy almost immediately. Inspect the actual model input at each scored position. The target token must be replaced or otherwise hidden there.

From pretraining to classification

A pretrained encoder already produces context-aware vectors. For text classification, a small output layer can map one pooled sequence vector to class logits. A common design uses a special first token whose encoded vector represents the sequence. Another simple choice is a masked mean over real token positions.

encoded_features = torch.randn(4, 12)  # four pooled examples
classification_head = nn.Linear(12, 3)
class_logits = classification_head(encoded_features)
labels = torch.tensor([0, 2, 1, 0])
classification_loss = nn.functional.cross_entropy(class_logits, labels)

print(tuple(class_logits.shape))  # Expected: (4, 3)
print(classification_loss.ndim)   # Expected: 0

In a real project, use a trusted pretrained checkpoint and its matching tokenizer rather than training a full BERT from random values. Begin with a small baseline, track validation metrics, and consider freezing the encoder before fine-tuning all layers.

Common mistakes

  • Do not use a GPT causal mask for ordinary BERT masked-token pretraining.
  • Do not calculate masked-token loss on padding and ordinary visible positions.
  • Use the exact tokenizer and special-token IDs expected by a pretrained checkpoint.
  • Padding still needs a padding mask so it does not become attention context.
  • BERT-style encoders are strong for understanding tasks but are not normally used to generate long text one token at a time.

Try it

A batch has eight sequences of length 20, model width 48, and vocabulary size 1000. State the encoder-output and masked-language-logit shapes. If there are 12 masked positions in total, what shape should the selected logits have before cross-entropy?

Reveal the worked answer

Encoder output is (8,20,48). The language head produces (8,20,1000). The selection of only the 12 masked positions gives logits of shape (12,1000) and targets of shape (12,).

Recap

BERT is an encoder-only transformer trained to reconstruct hidden tokens with context on both sides. Its input and encoder shapes are (B,T) and (B,T,D). A language head produces (B,T,V). Compute pretraining loss only at chosen masked positions. For practical classification, start from a pretrained encoder and add a small task head.

References: PyTorch TransformerEncoder and encoder layer documentation.