Logistic regression
Introduction
Map numeric features through one linear score and sigmoid to make a binary decision.
- Learning goal
Explain logits, probabilities, thresholds, and the limit of a linear boundary.
- Before you start
Dot products, tensor shapes, binary labels, and validation data.
Lesson plan
- Calculate a linear logit
- Apply sigmoid and a threshold
- Interpret the decision boundary
The problem
Binary classification needs a model that maps features to two class scores. Logistic regression calculates a weighted sum. It then applies the sigmoid function. Sigmoid maps any real score into (0,1).
Despite the name, logistic regression is a classifier. The model is a useful baseline. A linear model can solve the task. In that case, a larger network adds cost without better held-out evidence.
Work through a small example
Represent a message by x=[positive_words, negative_words]=[2,1]. Choose weights w=[1,-1] and bias b=0. The logit is z=x dot w+b=1. Sigmoid gives 1/(1+exp(-1))=0.731.
At threshold 0.5, the model predicts class 1. The threshold is a decision rule. It is not part of logistic regression training. Change the threshold on validation data when error costs differ.
The general rule
For a batch X shaped (B,D), weights have shape (D,). Logits z=Xw+b have shape (B,). The boundary is the set where probability equals 0.5, or Xw+b=0. It is a line in two dimensions and a hyperplane in higher dimensions.
Sigmoid is monotonic, so the boundary stays linear. More training cannot represent XOR or another pattern that is not linearly separable.
Implement and inspect
import torch
x = torch.tensor([[2.0, 1.0], [0.0, 2.0]])
w = torch.tensor([1.0, -1.0])
b = torch.tensor(0.0)
logits = x @ w + b
probabilities = torch.sigmoid(logits)
predictions = (probabilities >= 0.5).long()
print(logits.tolist())
print([round(p, 3) for p in probabilities.tolist()])
print(predictions.tolist())
Expected output is logits [1.0,-2.0], probabilities [0.731,0.119], and predictions [1,0]. Input shape is (2,2). Every row receives one scalar logit.
Engineering checks
- Standardize large continuous features. Use training statistics only.
- Compare the model with a majority-class baseline. Inspect class-level errors.
- Use logits directly with
binary_cross_entropy_with_logits. - Check probability calibration before you treat 0.8 as an 80% empirical frequency.
Go deeper
With cross-entropy and L2 regularization, binary logistic regression has a convex objective. Any local optimum is global. But separable unregularized data can drive weight norms upward without a finite minimum.
Coefficients describe changes in log-odds while other features stay fixed. An increase of feature j by one adds w_j to log-odds. Correlated features make individual coefficients unstable, even when predictions stay stable.
Practice and recap
Question: For x=[3,1], w=[0.5,-1], and b=0, find the logit and class at threshold 0.5.
Worked answer
The logit is 3*0.5 + 1*(-1)=0.5. Sigmoid is about 0.622, so the predicted class is 1. The decision depends only on the sign of the logit at threshold 0.5.
Logistic regression learns one linear boundary and probability scores. The scores still need held-out calibration checks.