Nonlinear features and multilayer networks
Introduction
Add nonlinear hidden features when one linear decision boundary cannot represent the task.
- Learning goal
Prove that a small ReLU MLP can represent XOR and track every tensor shape.
- Before you start
Linear layers, ReLU, logits, and basic matrix multiplication.
Lesson plan
- Expose the XOR limit
- Construct hidden ReLU features
- Separate representability from training
The problem
One linear classifier draws one hyperplane. It cannot solve XOR. For XOR, (0,0) and (1,1) share one class, and (0,1) and (1,0) share the other.
Stacked linear layers without an activation do not help. Their matrix products collapse into one linear transformation. A nonlinear activation prevents that collapse.
Work through a small example
Define two hidden features: h1=ReLU(x1-x2) and h2=ReLU(x2-x1). Their sum is zero when the inputs match and one when they differ. A final threshold separates the XOR classes.
The general rule
An MLP layer computes H=activation(XW+b). For X shaped (B,D) and hidden width H, first-layer weights conceptually map D to H. PyTorch stores them as (H,D). The output layer maps hidden features to class logits.
ReLU computes max(0,x) elementwise. Piecewise-linear hidden units divide the input space into regions. Later layers combine them into nonlinear boundaries.
Implement and inspect
This code sets weights by hand to prove that a small MLP can represent XOR. It does not train those weights. nn.Sequential chains the listed functions in order. Each output goes into the next layer. torch.no_grad() lets us assign parameter values. Those assignments do not enter an automatic-gradient graph.
import torch
from torch import nn
x = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
model = nn.Sequential(nn.Linear(2, 2), nn.ReLU(), nn.Linear(2, 1))
with torch.no_grad():
model[0].weight.copy_(torch.tensor([[1., -1.], [-1., 1.]]))
model[0].bias.zero_()
model[2].weight.copy_(torch.tensor([[1., 1.]]))
model[2].bias.fill_(-0.5)
logits = model(x).squeeze(1)
print(logits.tolist())
print((logits > 0).long().tolist())
Expected logits are [-0.5,0.5,0.5,-0.5] and classes [0,1,1,0]. Input shape is (4,2), hidden shape (4,2), and output shape (4,1).
The parent integration lesson trains an MLP with cross-entropy and gradient-based updates. That step replaces hand-chosen parameters with weights learned from examples.
Engineering checks
- Overfit four XOR points before you test a larger dataset.
- Inspect the fraction of zero ReLU activations and gradient norms.
- Keep a linear baseline. Nonlinear capacity can overfit noise.
- Use raw final logits with the matching cross-entropy loss.
Go deeper
A sufficiently wide MLP can approximate continuous functions on compact regions. But existence does not guarantee efficient learning, good generalization, or sensible behavior outside the training support.
Width, depth, and activation affect expressivity and optimization. Deep compositions can reuse features well. But poor scaling can shrink or explode signals. Symmetries mean many parameter settings represent the same function. So the loss geometry is non-convex.
Practice and recap
Question: Why do two linear layers with no ReLU still make a linear classifier?
Worked answer
(XW1+b1)W2+b2 = X(W1W2) + (b1W2+b2). The products can be renamed as one weight and one bias. ReLU makes the transformation input-dependent, so this algebraic collapse no longer works.
An MLP learns nonlinear intermediate features. Use it when held-out evidence shows that one linear boundary limits the model.