Cross-entropy and logits

Introduction

Train class scores with a loss that distinguishes uncertainty from confident mistakes.

Learning goal

Calculate softmax cross-entropy and use a numerically stable logits API.

Before you start

Class IDs, logarithms, exponentials, and model logits.

Lesson plan

  1. Normalize logits with softmax
  2. Read the target log probability
  3. Inspect stable batch losses

The problem

A classifier needs a training signal. The signal must separate uncertain predictions, confident-correct predictions, and confident-wrong predictions. Accuracy cannot do this. A change of the correct-class probability from 0.51 to 0.99 leaves accuracy unchanged.

Cross-entropy measures the negative log probability assigned to the observed class. Raw model outputs are logits: unrestricted scores before normalization.

Work through a small example

For logits [2,1,0], softmax exponentiates each score and divides by the sum. Subtract one shared constant first, and the probabilities stay unchanged. The probabilities are approximately [0.665,0.245,0.090]. If the target is class 0, the loss is -log(0.665)=0.408.

Three logits become probabilities and one target lossThree scalar logit circles two, one, and zero map through softmax to probabilities 0.665, 0.245, and 0.090. The target class zero probability maps to loss 0.408. 210 softmax .665.245.090target 0loss .408

If the same target receives probability 0.01, loss is 4.605. The logarithm strongly penalizes confident mistakes.

The general rule

For logits z, p_k=exp(z_k)/sum_j exp(z_j). For target class y, the loss is -log(p_y). A batch loss usually averages the losses of each example.

Numerically stable implementations combine log-softmax and negative log likelihood with the log-sum-exp identity. Add the same constant to every logit, and the probabilities stay unchanged. Subtract the maximum logit to avoid large exponentials.

Implement and inspect

import torch
import torch.nn.functional as F

logits = torch.tensor([[2.0, 1.0, 0.0], [0.0, 0.0, 0.0]])
targets = torch.tensor([0, 2])
losses = F.cross_entropy(logits, targets, reduction="none")
probabilities = logits.softmax(dim=1)

print([[round(v, 3) for v in row] for row in probabilities.tolist()])
print([round(v, 3) for v in losses.tolist()])

The expected probabilities are [[0.665,0.245,0.09],[0.333,0.333,0.333]]. The losses are [0.408,1.099]. The logits shape is (B=2,C=3). The targets shape is (2,) with integer class IDs.

Engineering checks

Go deeper

Cross-entropy is a proper scoring rule: in expectation, truthful conditional probabilities minimize it. Finite models, biased samples, regularization, and optimization error can still produce miscalibration.

Its gradient with respect to logit k is p_k - 1[k=y]. Every class receives a signal. Label smoothing replaces the one-hot target with a less extreme distribution, which changes both gradients and probability interpretation.

Practice and recap

Question: What is the loss when two classes have equal logits and either class is correct?

Worked answer

Softmax assigns each class the probability 0.5. The loss is -log(0.5)=0.693. The absolute logit values do not matter when they are equal.

Cross-entropy trains relative logits, rewards probability on the target, and exposes confidence errors that accuracy hides.