PyTorch University Path · Core lesson

Datasets, batching, splits, and trustworthy metrics

Prerequisites: basic Python functions, tensor shapes, and the idea that an optimizer changes model parameters. Review tensors and the training loop if needed.

Download the complete supervised CPU runner and its correctness tests. The program generates local data, uses one CPU thread, and writes nothing unless you pass --checkpoint.

Quick map

1. Split before learning any data-dependent values.
2. Train with shuffled mini-batches.
3. Select settings with validation data.
4. Open the test set once, then inspect errors.

Start with a concrete decision

Imagine 1,500 short customer messages. Each message has a label: ordinary question or urgent problem.

A classifier must predict the label for future messages. If you train on all 1,500 messages and then report accuracy on those same messages, the number answers the wrong question.

It tells you how well the model remembers or fits known examples. It does not tell you how well it handles the next message.

A better plan is to reserve three independent groups. For example, use 900 messages for training, 300 for validation, and 300 for testing.

The training set supplies gradients. The validation set helps choose settings such as learning rate, model width, and number of epochs.

The test set stays closed until those choices are finished. This separation is an experimental rule, not a PyTorch feature.

training data: update parameters
validation data: choose the training recipe
test data: estimate final performance once

The downloadable runner follows this rule for generated numeric and image data. The same rule applies to text.

A message becomes features only after the split is decided. If you build a vocabulary or calculate word frequencies from all messages first, information from validation and test has already entered training.

Name every tensor shape

Suppose a numeric example has two features. A batch of 64 examples has feature shape (64, 2).

Its target tensor has shape (64,), because each example has one integer class ID. A two-class model returns logits with shape (64, 2).

A logit is an unrestricted class score. Cross-entropy turns these scores into a training loss, so the model should not apply softmax before torch.nn.functional.cross_entropy.

B = batch size
D = number of input features
C = number of classes

features: (B, D)
targets:  (B,) containing integers from 0 to C - 1
logits:   (B, C)

Images add channel and spatial axes. A grayscale image batch may be (B, 1, 8, 8).

Text representations vary. A bag-of-words batch can be (B, V), where V is vocabulary size.

Token IDs for a transformer can be (B, T), where T is sequence length. Shape names prevent vague reasoning: always ask what each axis means.

Independent does not only mean different rows

Predict the risk: one customer sends ten nearly identical messages. If eight go to training and two go to validation, does the split test a new customer? No. The row IDs differ, but the customer's writing pattern appears in both sets.

Two tensors can contain different rows but still leak information. Data leakage means the training process receives information that would not be available when predicting a genuinely new example.

Direct duplication is the clearest form. Near-duplicates also matter: two cropped versions of the same photograph, two messages from the same long conversation, or two time windows from one patient may be almost the same example.

Choose the split unit to match the real use. If the model will predict for new people, split by person, not by individual record.

If it will predict next month, train on earlier dates and test on later dates. If many messages belong to one conversation, keep the whole conversation in one split.

Random row splitting is suitable only when rows are truly exchangeable and independent enough for the question.

Preprocessing can leak too. Consider standardization:

standardized_feature = (feature - training_mean) / training_standard_deviation

Calculate the mean and standard deviation from training data only. Then reuse those fixed values for validation, test, and production inputs.

The same rule applies to a text vocabulary, TF-IDF document frequencies, missing-value replacements, and feature selection. Fit on training data; transform all splits.

Why mini-batches exist

A full-batch update uses every training example before changing parameters. It gives a stable gradient but can require too much memory.

A one-example update is cheap but noisy. A mini-batch is a middle choice.

With 900 examples and batch size 64, most batches contain 64 examples and the last contains the remainder. Do not silently discard the last batch unless equal batch sizes are required by a specific algorithm.

order = torch.randperm(train_x.shape[0], generator=shuffle_generator)

for indices in order.split(batch_size):
    optimizer.zero_grad(set_to_none=True)
    logits = model(train_x[indices])
    loss = torch.nn.functional.cross_entropy(logits, train_y[indices])
    loss.backward()
    optimizer.step()

Shuffling changes which examples meet in a batch. Use a separate seeded generator when reproducibility matters.

A fixed seed makes debugging easier because two code versions see the same order. It does not guarantee bit-for-bit equality on every device or every PyTorch version.

This course uses CPU and torch.set_num_threads(1) to reduce variation.

One step is one optimizer update. One epoch is one pass through the training set.

With 900 examples and batch size 64, one epoch has 15 steps because the final partial batch still counts. Confusing steps and epochs can accidentally train a model much longer or shorter than intended.

Accuracy is useful, but not sufficient

Accuracy is the fraction of correct predictions. If 99 of 100 transactions are normal, a model that always says “normal” gets 99% accuracy and catches no fraud.

Always compare against a simple baseline. The runner uses the majority class from training data as a baseline and measures that fixed choice on test data.

accuracy = correct predictions / all predictions

precision for class k = correct predictions of k / all predictions of k
recall for class k = correct predictions of k / all actual examples of k
F1 for class k = 2 * precision * recall / (precision + recall)

Precision asks whether positive predictions can be trusted. Recall asks whether actual positives were found.

Their importance depends on the application. A spam filter may value precision to avoid hiding useful mail.

A safety screen may value recall to miss fewer risky cases. Macro averaging calculates the metric separately for each class and then gives every class equal weight.

This is helpful when classes have different sizes.

A confusion matrix preserves the error pattern. Rows in this course are actual classes and columns are predicted classes. The following runnable example has three correct predictions out of four:

import torch

actual = torch.tensor([0, 0, 1, 1])
predicted = torch.tensor([0, 1, 1, 1])
matrix = torch.zeros((2, 2), dtype=torch.long)

for truth, guess in zip(actual.tolist(), predicted.tolist()):
    matrix[truth, guess] += 1

print(matrix)
print((actual == predicted).float().mean().item())

# Expected output:
# tensor([[1, 1],
#         [0, 2]])
# 0.75

Use validation without turning it into training data

Validation data does not create gradients, but repeated decisions can still overfit it. Suppose you try 200 model variants and select the one with the best validation score.

Some of that score may be luck specific to the validation set. The final test set gives one cleaner check after selection.

If you look at test results and then change the model, the test set has become another validation set. You need a new test set for an honest final estimate.

The runner keeps the best parameter state in memory according to validation loss. It then restores that state and evaluates the test split once.

A lower validation loss does not always mean higher accuracy, but cross-entropy notices confidence as well as correctness. That makes it a useful selection signal.

The program reports accuracy and macro F1 so you can inspect a different view of the final behavior.

model.eval()
with torch.no_grad():
    validation_logits = model(validation_x)
    validation_loss = torch.nn.functional.cross_entropy(
        validation_logits, validation_y
    )

model.eval() switches layers such as dropout to evaluation behavior. torch.no_grad() disables gradient recording. They solve different problems, so use both during ordinary evaluation.

Run the complete experiment

.venv-learning/bin/python examples/pytorch/build_supervised.py --task logistic
.venv-learning/bin/python -m unittest tests/test_supervised_builds.py

Expected output includes all three feature shapes, first and final training loss, best validation loss, majority baseline accuracy, test accuracy, macro F1, and a confusion matrix. Exact values belong to the stated code, seed, and PyTorch version.

Do not copy a number into a report unless you ran that configuration. The tests check shapes, gradients, metric orientation, held-out performance above the baseline, and exact checkpoint reload behavior.

Failure drill: the suspiciously perfect model

Before celebrating 100% validation accuracy, compare it with the task difficulty and baseline. Then search for duplicates, target-derived features, preprocessing fitted on all data, and groups split across sets. An unexpectedly strong score is evidence to investigate.

First, search for duplicates across splits. Next, inspect features.

A source filename, review score, or post-resolution status may directly encode the label. Check whether the vocabulary and TF-IDF statistics were fitted before splitting.

Finally, check grouping: messages from one conversation or one author may appear in both training and validation.

The repair is not “add dropout.” Rebuild the split around the real prediction unit, remove features unavailable at prediction time, fit preprocessing on training data only, and rerun the baseline. Regularization cannot repair an invalid experiment.

Practice checks with worked answers

Quick recap

An honest supervised experiment begins before the model. Define the future use, split at the correct unit, and fit every learned preprocessing step on training data only.

Use mini-batches for efficient updates, validation data for choices, and test data for one final estimate. Compare with a simple baseline.

Report metrics that match the cost of errors, and keep the confusion matrix close enough to inspect. A held-out score is evidence about one data-generating process; it is not a promise about every real-world condition.

Local implementation: supervised build runner. Continue with logistic regression and MLP classification.