Classification metrics and baselines

Introduction

Measure which errors a classifier makes instead of relying on one average accuracy.

Learning goal

Calculate confusion counts, precision, recall, F1, and threshold effects.

Before you start

Binary classes, predicted scores, and train-validation-test separation.

Lesson plan

  1. Build the confusion matrix
  2. Calculate class-sensitive metrics
  3. Choose thresholds on validation data

The problem

Accuracy gives one average. It can hide failure on a rare class. If 99 transactions are normal and one is fraud, always predicting normal gives 99% accuracy and zero fraud recall.

For a chosen positive class, a true positive is correctly predicted positive. A false positive is a negative example incorrectly flagged. A false negative is a missed positive. These counts form a confusion matrix.

Work through a small example

Suppose ten cases contain four actual positives. The model flags five cases, three correctly. Then TP=3, FP=2, FN=1, and TN=4.

Binary classification confusion matrixA two by two matrix shows three true positives, two false positives, one false negative, and four true negatives, each as labelled scalar circles. predicted classactual class negativepositivenegativepositive TN 4FP 2FN 1TP 3

Precision is TP/(TP+FP)=3/5=0.60. Recall is TP/(TP+FN)=3/4=0.75. F1 is their harmonic mean, 2PR/(P+R)=0.667. Accuracy is (TP+TN)/10=0.70.

The general rule

Precision answers, “Among positive predictions, how many were right?” Recall answers, “Among actual positives, how many were found?” Choose the emphasis from error costs, not habit.

A probability becomes a class after applying a threshold. Lowering the threshold usually raises recall and lowers precision. Select the threshold on validation data. Applying many thresholds to test data and choosing the best leaks test information.

For multiple classes, macro averaging computes a metric per class and averages equally. Micro averaging pools all decisions, so large classes dominate. Neither is universally correct.

Implement and inspect

import torch

truth = torch.tensor([1, 1, 1, 1, 0, 0, 0, 0, 0, 0])
score = torch.tensor([.9, .8, .6, .4, .7, .55, .3, .2, .1, .05])
pred = (score >= .5).long()

tp = int(((pred == 1) & (truth == 1)).sum())
fp = int(((pred == 1) & (truth == 0)).sum())
fn = int(((pred == 0) & (truth == 1)).sum())
precision = tp / (tp + fp)
recall = tp / (tp + fn)
print(tp, fp, fn)
print(f"{precision:.2f} {recall:.2f}")

Expected output is 3 2 1, followed by 0.60 0.75 for precision and recall.

Both tensors have shape (10,). Scores are floating point; labels and predictions are integer category IDs. The comparison creates a Boolean tensor before conversion to integers.

Engineering checks

Go deeper

Ranking metrics and threshold metrics answer different questions. Area under a precision-recall curve summarizes many thresholds, while operational precision and recall describe one chosen threshold. Calibration asks whether predictions near 0.8 are correct about 80% of the time; strong ranking does not imply calibrated probabilities.

Some metrics, especially precision and accuracy, depend on prevalence. Precision can fall when positives become rarer even if class-conditional behavior stays unchanged. Recall, specificity, and area under the ROC curve can stay fixed under that same class-conditional behavior. Report the evaluation population and confidence intervals when decisions matter.

Practice and recap

Question: A screen finds 18 of 20 risky cases and flags 30 cases total. Calculate precision and recall.

Worked answer

TP=18, FN=2, and FP=12. Precision is 18/30=0.60. Recall is 18/20=0.90. The system finds most risky cases but produces twelve false alarms.

Metrics encode different error costs. Preserve the confusion counts, choose thresholds on validation data, and interpret results for the population being measured.