PyTorch University Path · Optional side lesson

Convolutional networks: locality and shared weights

Prerequisites: MLP classification, mini-batches, and train/validation/test evaluation. You may skip this lesson and continue directly to the transformer capstone.

Download the supervised CPU runner and use --task cnn. It creates noisy 8 by 8 images locally, downloads nothing, and writes no file unless a checkpoint path is explicit.

Quick map

1. Slide one learned kernel across local image regions.
2. Stack feature channels and track (B, C, H, W).
3. Reduce spatial size with pooling.
4. Map the final channel summary to class logits.
5. Verify the complete classifier on held-out generated images.

An MLP ignores image structure

Suppose an 8 by 8 grayscale image is flattened into 64 numbers. An MLP can classify it, but it treats the top-left pixel and its neighbor as merely two unrelated feature positions.

A line shifted one pixel becomes a different set of input coordinates. The network must learn many similar weights for patterns at different locations.

A convolution uses a small grid of weights called a kernel or filter. It slides this grid across the image and applies the same weights at every location.

This adds two useful assumptions. First, nearby pixels form local patterns.

Second, a useful local pattern can matter in more than one location. These assumptions are called locality and weight sharing.

The assumptions fit many images, but they are not laws. A convolution does not automatically understand objects, scale, lighting, or viewpoint. It is an architectural bias that often makes image learning more efficient.

Work through one kernel

Predict one output cell: place the kernel over the top-left input patch. Multiply matching entries, add the products, then add the bias. This one scalar becomes output row 0, column 0. Sliding right changes the input patch but reuses the same kernel values.

Take a one-channel image and a 3 by 3 kernel. At each output location, multiply the nine image values under the kernel by nine kernel weights, add those products, then add one bias. The result is one output number.

output[row, column] = bias
    + sum over kernel rows and columns(
        image[shifted row, shifted column] * kernel weight
      )

A vertical-edge kernel might assign negative weights on its left side and positive weights on its right side. Uniform regions produce values near zero, while a dark-to-light vertical change produces a strong response. During neural-network training, the kernel weights are learned rather than manually selected.

Stride is how far the kernel moves between output positions. Stride 1 visits every position.

Stride 2 skips every other position and reduces spatial size. Padding adds border values, usually zeros.

A 3 by 3 kernel with padding 1 and stride 1 preserves height and width.

output_size = floor((input_size + 2 * padding - kernel_size) / stride) + 1

input 8, kernel 3, padding 1, stride 1 gives output 8
input 8, kernel 3, padding 0, stride 1 gives output 6

Channels are learned feature maps

A grayscale image has one input channel. A color image normally has three: red, green, and blue.

A convolutional layer can produce many output channels. Each output channel has its own learned detector and bias.

Early channels may respond to simple edges or textures. Later channels combine previous channels into more task-specific patterns.

PyTorch image tensors use shape (B, C, H, W): batch, channels, height, width. The first layer in the runnable classifier receives (B, 1, 8, 8).

It has 8 output channels, kernel size 3, and padding 1. Its output is (B, 8, 8, 8).

convolution weight shape:
(output_channels, input_channels, kernel_height, kernel_width)

first layer weight shape:
(8, 1, 3, 3)

After ReLU, max pooling with kernel size 2 reduces each 2 by 2 region to its largest value. Height and width change from 8 to 4, while channels remain 8. The next convolution maps 8 channels to 12 channels and preserves the 4 by 4 spatial size.

input:                  (B, 1, 8, 8)
after first conv:       (B, 8, 8, 8)
after ReLU and pooling: (B, 8, 4, 4)
after second conv:      (B, 12, 4, 4)
after global average:   (B, 12, 1, 1)
after flatten:          (B, 12)
class logits:           (B, 3)

Pooling trades detail for compactness

Max pooling keeps the strongest activation in each local window. It reduces memory and makes small shifts less important.

It also discards exact location information. Too much early pooling can remove small objects or fine text.

Some modern CNNs replace pooling with strided convolutions, which learn how to reduce size.

The teaching network ends with adaptive average pooling to shape (1, 1). It averages every spatial location separately for each of the 12 channels.

This creates 12 numbers per image regardless of the current height and width. A final linear layer maps those values to three class logits.

Global averaging suits the generated task because orientation matters more than exact line position. It would be a poor choice if the label depended on whether an object was specifically in the top-left corner.

The tiny classifier task

The dataset contains three classes: vertical line, horizontal line, and diagonal line. Each 8 by 8 image starts with small random noise.

Vertical and horizontal lines appear at varied interior positions. Diagonal examples can slope in either direction.

Train, validation, and test tensors are generated with different seeds.

This task tests a real CNN property: shared local filters can recognize orientation across positions. It is still toy data.

The classes are balanced, the backgrounds are simple, and generated rules are much cleaner than real images. A high score does not establish performance on handwriting, medical images, or photographs.

model = nn.Sequential(
    nn.Conv2d(1, 8, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.MaxPool2d(kernel_size=2),
    nn.Conv2d(8, 12, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.AdaptiveAvgPool2d((1, 1)),
    nn.Flatten(),
    nn.Linear(12, 3),
)

The model returns logits, not probabilities. Cross-entropy compares logits shaped (B, 3) with integer targets shaped (B,). The same optimizer loop used for the numeric MLP works here because autograd follows convolution operations too.

Depth expands the receptive field

A unit's receptive field is the input region that can influence it. One 3 by 3 convolution sees a 3 by 3 area.

A second 3 by 3 convolution sees combinations of neighboring first-layer outputs, so it indirectly sees a larger part of the original image. Pooling expands this reach faster because one pooled position summarizes several positions.

This creates a hierarchy: pixels form edges, edges form motifs, and motifs can form object parts. That description is a helpful mental model, not a guarantee that every trained channel has a human-friendly meaning. Inspect activations and failure cases before assigning a story to a feature.

Bridge to larger architectures and language

A practical image network usually repeats convolutional blocks, increases channels as spatial size shrinks, and adds residual connections so gradients can move through deep stacks. Residual blocks learn a change to their input rather than a completely new representation.

Normalization layers stabilize activation scales. These ideas also appear in transformers, although attention replaces the local sliding kernel as the main mixing operation.

Convolutions can process sequences too. A one-dimensional convolution slides across neighboring token or audio representations shaped like (B, C, T).

It can detect local phrases or sound patterns. Unlike self-attention, its direct view is local and fixed by kernel size, though stacked layers expand the receptive field.

For an NLP-focused path, bag-of-words models, recurrent models, and transformers are the main course; this CNN lesson provides useful architectural contrast.

Run and verify the complete classifier

.venv-learning/bin/python examples/pytorch/build_supervised.py --task cnn
.venv-learning/bin/python examples/pytorch/build_supervised.py \
  --task cnn --checkpoint /tmp/tiny-cnn.pt
.venv-learning/bin/python -m unittest tests/test_supervised_builds.py

The first command writes nothing. The second writes and reloads a checkpoint because you supplied the path.

Expected output includes shapes for all three splits, training loss movement, the majority baseline, test accuracy, macro F1, and a 3 by 3 confusion matrix. The tests verify the image shape, class-logit shape, nonzero gradients, baseline improvement, and exact checkpoint output after reload.

Failure drill: wrong axis order

An image batch normally uses (batch, channels, height, width). A tensor shaped (batch, height, width, channels) may look familiar from another library. Print the shape and name every axis before changing the convolution layer.

Your tensor is probably shaped (B, H, W, C), a layout common in some image tools, while PyTorch Conv2d expects (B, C, H, W). For grayscale 8 by 8 images, require (B, 1, 8, 8).

A channel-last tensor can be converted with images.permute(0, 3, 1, 2). Print the shape before the model instead of guessing.

If shapes are correct but learning fails, try to overfit 12 images. Confirm targets are integer class IDs from 0 through 2.

Check that the final linear layer returns three logits. Remove excessive augmentation or dropout.

Inspect example images numerically or visually to verify the label generator.

If training succeeds but test results are poor, check whether train and test image creation differ unintentionally. In a real project, check subject, camera, time, and duplicate leakage. Adding layers before checking the data can hide the real problem.

Practice checks with worked answers

Quick recap and limits

A convolution scans local neighborhoods with shared weights. Multiple output channels learn different feature maps.

Padding, stride, and kernel size determine spatial output shape. Pooling reduces size and adds some tolerance to small shifts, while deeper layers gain a larger receptive field.

The runnable network joins these pieces into a full CPU training and held-out evaluation pipeline.

This side lesson is optional because the main curriculum is NLP-inclined. Its generated images make mechanics visible, but they do not measure real-world vision ability. The useful transfer is architectural thinking: track shapes, identify what information an operation mixes or discards, build a baseline, and evaluate on independent data.

Continue to the transformer capstone, or return to optimization and generalization.