Cross-entropy and logits

Introduction

Train class scores with a loss that distinguishes uncertainty from confident mistakes.

Learning goal

Calculate softmax cross-entropy and use a numerically stable logits API.

Before you start

Class IDs, logarithms, exponentials, and model logits.

Lesson plan

  1. Normalize logits with softmax
  2. Read the target log probability
  3. Inspect stable batch losses

The problem

A classifier needs a training signal that distinguishes uncertain, confident-correct, and confident-wrong predictions. Accuracy cannot do this: moving the correct-class probability from 0.51 to 0.99 leaves accuracy unchanged.

Cross-entropy measures the negative log probability assigned to the observed class. Raw model outputs are logits: unrestricted scores before normalization.

Work through a small example

For logits [2,1,0], softmax exponentiates each score and divides by the sum. Subtracting one shared constant first leaves probabilities unchanged. The probabilities are approximately [0.665,0.245,0.090]. If the target is class 0, loss is -log(0.665)=0.408.

Three logits become probabilities and one target lossThree scalar logit circles two, one, and zero map through softmax to probabilities 0.665, 0.245, and 0.090. The target class zero probability maps to loss 0.408. 210 softmax .665.245.090target 0loss .408

If the same target receives probability 0.01, loss is 4.605. The logarithm strongly penalizes confident mistakes.

The general rule

For logits z, p_k=exp(z_k)/sum_j exp(z_j). For target class y, loss is -log(p_y). A batch loss commonly averages per-example losses.

Numerically stable implementations combine log-softmax and negative log likelihood using the log-sum-exp identity. Adding the same constant to every logit leaves probabilities unchanged. Subtracting the maximum logit avoids large exponentials.

Implement and inspect

import torch
import torch.nn.functional as F

logits = torch.tensor([[2.0, 1.0, 0.0], [0.0, 0.0, 0.0]])
targets = torch.tensor([0, 2])
losses = F.cross_entropy(logits, targets, reduction="none")
probabilities = logits.softmax(dim=1)

print([[round(v, 3) for v in row] for row in probabilities.tolist()])
print([round(v, 3) for v in losses.tolist()])

Expected probabilities are [[0.665,0.245,0.09],[0.333,0.333,0.333]]; losses are [0.408,1.099]. Logits shape is (B=2,C=3); targets shape is (2,) with integer class IDs.

Engineering checks

Go deeper

Cross-entropy is a proper scoring rule: in expectation, truthful conditional probabilities minimize it. Finite models, biased samples, regularization, and optimization error can still produce miscalibration.

Its gradient with respect to logit k is p_k - 1[k=y]. Every class receives a signal. Label smoothing replaces the one-hot target with a less extreme distribution, which changes both gradients and probability interpretation.

Practice and recap

Question: What is the loss when two classes have equal logits and either class is correct?

Worked answer

Softmax assigns each class probability 0.5. Loss is -log(0.5)=0.693. The absolute logit values do not matter when they are equal.

Cross-entropy trains relative logits, rewards probability on the target, and exposes confidence errors that accuracy hides.