Classification metrics and baselines

Introduction

Measure which errors a classifier makes instead of relying on one average accuracy.

Learning goal

Calculate confusion counts, precision, recall, F1, and threshold effects.

Before you start

Binary classes, predicted scores, and train-validation-test separation.

Lesson plan

  1. Build the confusion matrix
  2. Calculate class-sensitive metrics
  3. Choose thresholds on validation data

The problem

Accuracy gives one average. It can hide failure on a rare class. Suppose 99 transactions are normal and one is fraud. A model that always predicts normal gets 99% accuracy and zero fraud recall.

Choose one positive class. A true positive is a positive example that the model predicts as positive. A false positive is a negative example that the model flags as positive. A false negative is a positive example that the model misses. These counts form a confusion matrix.

Work through a small example

Suppose ten cases contain four actual positives. The model flags five cases, and three of them are correct. Then TP=3, FP=2, FN=1, and TN=4.

Binary classification confusion matrixA two by two matrix shows three true positives, two false positives, one false negative, and four true negatives, each as labelled scalar circles. predicted classactual class negativepositivenegativepositive TN 4FP 2FN 1TP 3

Precision is TP/(TP+FP)=3/5=0.60. Recall is TP/(TP+FN)=3/4=0.75. F1 is their harmonic mean, 2PR/(P+R)=0.667. Accuracy is (TP+TN)/10=0.70.

The general rule

Precision answers one question: of all positive predictions, how many are correct? Recall answers another question: of all actual positives, how many does the model find? Choose the emphasis from error costs, not from habit.

A threshold turns a probability into a class. A lower threshold usually raises recall and lowers precision. Select the threshold on validation data. If you apply many thresholds to test data and choose the best, you leak test information.

For multiple classes, macro average computes a metric for each class. Then it averages the metrics equally. Micro average pools all decisions, so large classes dominate. Neither method is always correct.

Implement and inspect

import torch

truth = torch.tensor([1, 1, 1, 1, 0, 0, 0, 0, 0, 0])
score = torch.tensor([.9, .8, .6, .4, .7, .55, .3, .2, .1, .05])
pred = (score >= .5).long()

tp = int(((pred == 1) & (truth == 1)).sum())
fp = int(((pred == 1) & (truth == 0)).sum())
fn = int(((pred == 0) & (truth == 1)).sum())
precision = tp / (tp + fp)
recall = tp / (tp + fn)
print(tp, fp, fn)
print(f"{precision:.2f} {recall:.2f}")

The expected output is 3 2 1. Precision and recall follow as 0.60 0.75.

Both tensors have shape (10,). Scores are floating point. Labels and predictions are integer category IDs. The comparison creates a Boolean tensor, then the code converts it to integers.

Engineering checks

Go deeper

Ranking metrics and threshold metrics answer different questions. The area under a precision-recall curve summarizes many thresholds. Operational precision and recall describe one chosen threshold. Calibration asks a question: are predictions near 0.8 correct about 80% of the time? Strong ranking does not imply calibrated probabilities.

Some metrics, especially precision and accuracy, depend on prevalence. Precision can fall when positives become rarer, even if class-conditional behavior stays unchanged. Recall, specificity, and area under the ROC curve can stay fixed under that same behavior. When decisions matter, report the evaluation population and confidence intervals.

Practice and recap

Question: A screen finds 18 of 20 risky cases and flags 30 cases total. Calculate precision and recall.

Worked answer

TP=18, FN=2, and FP=12. Precision is 18/30=0.60. Recall is 18/20=0.90. The system finds most risky cases but produces twelve false alarms.

Metrics encode different error costs. Preserve the confusion counts. Choose thresholds on validation data. Interpret the results for the measured population.