Nonlinear features and multilayer networks

The problem

One linear classifier draws one hyperplane. It cannot solve XOR, where (0,0) and (1,1) share a class while (0,1) and (1,0) share the other.

Stacking linear layers without an activation does not help. Their matrix products collapse into one linear transformation. A nonlinear activation prevents that collapse.

Work through a small example

Define two hidden features: h1=ReLU(x1-x2) and h2=ReLU(x2-x1). Their sum is zero when inputs match and one when they differ. A final threshold separates the XOR classes.

The general rule

An MLP layer computes H=activation(XW+b). For X shaped (B,D) and hidden width H, first-layer weights conceptually map D to H; PyTorch stores them as (H,D). The output layer maps hidden features to class logits.

ReLU computes max(0,x) elementwise. Piecewise-linear hidden units divide input space into regions; later layers combine them into nonlinear boundaries.

Implement and inspect

This code hand-sets weights to prove that a small MLP can represent XOR. It does not train those weights. nn.Sequential chains the listed functions in order, sending each output into the next layer. torch.no_grad() lets us assign parameter values without adding those assignments to an automatic-gradient graph.

import torch
from torch import nn

x = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
model = nn.Sequential(nn.Linear(2, 2), nn.ReLU(), nn.Linear(2, 1))
with torch.no_grad():
    model[0].weight.copy_(torch.tensor([[1., -1.], [-1., 1.]]))
    model[0].bias.zero_()
    model[2].weight.copy_(torch.tensor([[1., 1.]]))
    model[2].bias.fill_(-0.5)

logits = model(x).squeeze(1)
print(logits.tolist())
print((logits > 0).long().tolist())

Expected logits are [-0.5,0.5,0.5,-0.5] and classes [0,1,1,0]. Input shape is (4,2), hidden shape (4,2), and output shape (4,1).

The parent integration lesson trains an MLP with cross-entropy and gradient-based updates. That step replaces hand-chosen parameters with weights learned from examples.

Engineering checks

• Overfit four XOR points before testing a larger dataset.
• Inspect the fraction of zero ReLU activations and gradient norms.
• Keep a linear baseline; nonlinear capacity can overfit noise.
• Use raw final logits with the matching cross-entropy loss.

Go deeper

A sufficiently wide MLP can approximate continuous functions on compact regions, but existence does not guarantee efficient learning, good generalization, or sensible behavior outside training support.

Width, depth, and activation affect expressivity and optimization. Deep compositions can reuse features efficiently, while poor scaling can shrink or explode signals. Symmetries mean many parameter settings represent the same function, making the loss geometry non-convex.

Practice and recap

Question: Why do two linear layers with no ReLU still make a linear classifier?

An MLP learns nonlinear intermediate features. Use it when held-out evidence shows that one linear boundary is an important limitation.