Logistic regression

The problem

Binary classification needs a model that maps features to a probability-like score for two classes. Logistic regression starts with a weighted sum, then uses the sigmoid function to map any real score into (0,1).

Despite its name, it is a classifier. Its simplicity makes it a valuable baseline: if a linear model solves the task, a larger network may add cost without better held-out evidence.

Work through a small example

Represent a message by x=[positive_words, negative_words]=[2,1]. Choose weights w=[1,-1] and bias b=0. The logit is z=x dot w+b=1. Sigmoid gives 1/(1+exp(-1))=0.731.

At threshold 0.5, this predicts class 1. The threshold is a decision rule, not part of logistic regression training; change it on validation data when error costs differ.

The general rule

For a batch X shaped (B,D), weights have shape (D,), and logits z=Xw+b have shape (B,). The boundary where probability equals 0.5 is Xw+b=0, a line in two dimensions and a hyperplane in higher dimensions.

Sigmoid is monotonic, so the boundary remains linear. Training longer cannot represent XOR or another genuinely nonlinearly separable pattern.

Implement and inspect

import torch

x = torch.tensor([[2.0, 1.0], [0.0, 2.0]])
w = torch.tensor([1.0, -1.0])
b = torch.tensor(0.0)
logits = x @ w + b
probabilities = torch.sigmoid(logits)
predictions = (probabilities >= 0.5).long()

print(logits.tolist())
print([round(p, 3) for p in probabilities.tolist()])
print(predictions.tolist())

Expected output is logits [1.0,-2.0], probabilities [0.731,0.119], and predictions [1,0]. Input shape is (2,2); every row receives one scalar logit.

Engineering checks

• Standardize large continuous features using training statistics only.
• Compare with a majority-class baseline and inspect class-level errors.
• Use logits directly with binary_cross_entropy_with_logits.
• Check probability calibration before treating 0.8 as an 80% empirical frequency.

Go deeper

With cross-entropy and L2 regularization, binary logistic regression has a convex objective. Any local optimum is global, although separable unregularized data may drive weight norms upward without a finite minimizer.

Coefficients describe changes in log-odds while other features are fixed: increasing feature j by one adds w_j to log-odds. Correlated features make individual coefficients unstable even when predictions remain stable.

Practice and recap

Question: For x=[3,1], w=[0.5,-1], and b=0, find the logit and class at threshold 0.5.

Logistic regression learns one linear boundary and calibrated-looking scores that still require held-out calibration checks.