PyTorch University Path · Core lesson

From logistic regression to nonlinear MLPs

Prerequisites: tensor shapes, matrix multiplication, gradients, and independent train/validation/test splits. Read data and evaluation first.

Download the complete CPU runner. Use --task logistic for a linearly separable problem and --task mlp for a nonlinear problem. Both use independent held-out data and a majority-class baseline.

Quick map

1. Map input features to one logit per class.
2. Train the logits with cross-entropy.
3. Find the limit of one linear boundary.
4. Add ReLU hidden layers for nonlinear boundaries.
5. Reuse the same logic with bag-of-words text features.

Classification answers a choice question

Regression predicts a number such as temperature. Classification chooses among named categories.

Consider a message represented by two measurements: how often it contains a question word and how often it contains an urgent word. A simple classifier might choose between class 0, ordinary, and class 1, urgent.

The model returns one score per class. These scores are called logits.

They can be positive or negative and do not need to sum to one. For a batch of 64 examples with two input features and two classes, input shape is (64, 2), weight shape is (2, 2), bias shape is (2,), and logit shape is (64, 2).

B = number of examples in a batch
D = number of features
C = number of classes

X has shape (B, D)
W has shape (D, C)
b has shape (C,)
logits = X @ W + b has shape (B, C)

PyTorch stores an nn.Linear(2, 2) weight as shape (2, 2), with output features first internally. You do not need to transpose it yourself. Calling the layer performs the correct operation.

Logistic regression draws one flat boundary

Despite its name, logistic regression is a classifier. In a two-class model, compare the two logits.

Their difference is a weighted sum of the input features. The prediction changes where that difference is zero.

With two features, that location is a straight line. With more features, it is a flat surface called a hyperplane.

class_1_score - class_0_score = a * x1 + b * x2 + c

decision boundary: a * x1 + b * x2 + c = 0

The runner's logistic task generates points around this rule: 1.4 * x1 - 0.9 * x2 + 0.2 > 0. A linear model is the right tool because one line separates the classes. Training should reduce loss and produce test accuracy well above the majority baseline.

from torch import nn

model = nn.Linear(in_features=2, out_features=2)
logits = model(features)                 # (B, 2)
loss = nn.functional.cross_entropy(logits, targets)  # one number

Do not apply softmax before cross-entropy. The loss function combines log-softmax and negative log likelihood in a numerically stable calculation.

For reporting probabilities after training, use logits.softmax(dim=1). For the predicted class, logits.argmax(dim=1) is enough because softmax preserves score order.

Cross-entropy rewards the correct class score

For one example, softmax converts logits into probabilities. Cross-entropy takes the negative logarithm of the probability assigned to the correct class.

A confident correct prediction has low loss. A confident wrong prediction has high loss.

The logarithm strongly penalizes confident mistakes.

probability of class k = exp(logit_k) / sum(exp(all logits))

loss for one example = -log(probability of the correct class)

If the correct-class probability is 0.8, loss is about 0.223. If it is 0.1, loss is about 2.303.

Accuracy treats both a 51% and a 99% correct prediction as one success. Cross-entropy sees the confidence difference, which is why validation loss can be a useful model-selection measure.

A boundary that one line cannot draw

Predict the failure: XOR labels (0,0) and (1,1) alike, while the other corners receive the other class. One straight line cannot place the two diagonal corners on one side and the remaining corners on the other.

Now place points near four corners: (-1, -1), (-1, 1), (1, -1), and (1, 1). Give the same label when the two signs match, and the other label when they differ.

This is an XOR-style arrangement. Opposite corners share a class.

No single straight line can place both matching-sign corners on one side and both different-sign corners on the other.

Training logistic regression longer does not fix a missing capability. Lowering the learning rate does not fix it either.

The model family cannot express the required boundary. This distinction matters in debugging: optimization asks whether training finds good parameters inside the model family; representation asks whether suitable parameters exist at all.

An MLP learns intermediate features

A hidden unit can detect one useful region, and another can detect a second region. The output layer combines those detections. This is the important change: the MLP learns intermediate features before drawing the final class boundary.

A multilayer perceptron, or MLP, stacks affine layers with nonlinear activation functions. The first layer can learn several boundaries.

ReLU then keeps positive values and replaces negative values with zero. Later layers combine these bent pieces into a nonlinear decision region.

hidden_1 = ReLU(X @ W1 + b1)
hidden_2 = ReLU(hidden_1 @ W2 + b2)
logits = hidden_2 @ W3 + b3

The runnable model uses widths 2 -> 16 -> 16 -> 2. For batch size 64, its important shapes are:

input features:       (64, 2)
first hidden values: (64, 16)
second hidden values:(64, 16)
class logits:        (64, 2)
targets:             (64,)

Without ReLU, three linear layers collapse into one linear transformation. Matrix products remain linear, so depth alone would not solve XOR. The activation is the step that changes the kinds of boundaries the network can represent.

model = nn.Sequential(
    nn.Linear(2, 16),
    nn.ReLU(),
    nn.Linear(16, 16),
    nn.ReLU(),
    nn.Linear(16, 2),
)

nn.Sequential passes the output of each item into the next. Think of it as function composition.

The linear layers own trainable tensors. ReLU owns no trainable parameters.

Calling model(features) runs the full chain.

Bridge to text: bag-of-words features

The same classifier does not care whether a feature came from geometry or language. A bag-of-words representation gives one feature to each vocabulary word.

The value may be a count, a binary present-or-absent flag, or a TF-IDF weight. If the training vocabulary has 5,000 words, a batch has shape (B, 5000).

A logistic text classifier is then nn.Linear(5000, C).

For a tiny example, use vocabulary ["great", "slow", "refund", "thanks"]. The message “great, thanks” becomes [1, 0, 0, 1].

“slow refund” becomes [0, 1, 1, 0]. The representation ignores word order, but it creates a useful bridge from raw text to supervised learning.

import torch
from torch import nn

# Vocabulary order: great, slow, refund, thanks
documents = torch.tensor([
    [1.0, 0.0, 0.0, 1.0],
    [0.0, 1.0, 1.0, 0.0],
    [1.0, 0.0, 0.0, 0.0],
])
labels = torch.tensor([1, 0, 1])
classifier = nn.Linear(4, 2)
logits = classifier(documents)

print(logits.shape, labels.shape)
# Expected: torch.Size([3, 2]) torch.Size([3])

Build the vocabulary from training text only. Unknown validation and test words can map to an unknown token or be ignored, depending on the design.

TF-IDF also fits document-frequency values on training documents only. These older representations remain valuable: they are fast, easy to inspect, and strong baselines for many topic or sentiment tasks.

BERT and ModernBERT use contextual token representations instead, so a word's vector can change with its surrounding words. That is more expressive, but it does not remove the need for clean splits, baselines, or error analysis.

An MLP can receive bag-of-words or TF-IDF features, but hidden layers are not automatically better. On sparse high-dimensional text, logistic regression is often a serious baseline. Use validation evidence, not model fashion, to justify extra complexity.

Train both tasks with the same loop

.venv-learning/bin/python examples/pytorch/build_supervised.py --task logistic
.venv-learning/bin/python examples/pytorch/build_supervised.py --task mlp

Expected output lists split shapes, the first and final epoch loss, the best validation loss, a majority test baseline, test accuracy, macro F1, and a confusion matrix. The exact measurements depend on the stated code and environment. The important evidence is that loss falls, each model beats its baseline, and the MLP handles the nonlinear held-out examples.

The runner initializes parameters after calling torch.manual_seed(404), shuffles training rows with a separate seeded generator, chooses the best in-memory state using validation loss, and touches test data only after selection. The model sees labels only through cross-entropy during training.

Failure drill: 50% on XOR

First verify the input and target shapes. Then overfit a very small batch with the MLP.

If the MLP cannot reach very low loss on 16 examples, inspect whether ReLU is present, gradients are cleared, backward() runs, and the optimizer owns model.parameters(). If the linear model fails but the MLP succeeds, the result demonstrates a representation limit rather than a broken loop.

Another common failure is using Sigmoid on two logits and then cross-entropy. Remove the sigmoid.

For two-logit cross-entropy, targets are integer IDs with shape (B,) and dtype torch.long. A separate one-logit binary setup uses binary cross-entropy instead; do not mix the two conventions.

Practice checks with worked answers

Quick recap

Logistic regression maps features directly to class logits and creates a linear decision boundary. It is simple, fast, and an important baseline for numeric and text features.

XOR reveals its limit because opposite corners require a nonlinear boundary. An MLP adds hidden affine layers and ReLU activations, allowing later layers to combine learned feature regions.

Cross-entropy trains either model from raw logits. The move from a two-number toy input to bag-of-words text changes the feature shape, not the central supervised-learning logic.

Next: optimization and generalization. Optional visual path: convolutional networks.