Logistic regression

Introduction

Map numeric features through one linear score and sigmoid to make a binary decision.

Learning goal

Explain logits, probabilities, thresholds, and the limit of a linear boundary.

Before you start

Dot products, tensor shapes, binary labels, and validation data.

Lesson plan

  1. Calculate a linear logit
  2. Apply sigmoid and a threshold
  3. Interpret the decision boundary

The problem

Binary classification needs a model that maps features to two class scores. Logistic regression calculates a weighted sum. It then applies the sigmoid function. Sigmoid maps any real score into (0,1).

Despite the name, logistic regression is a classifier. The model is a useful baseline. A linear model can solve the task. In that case, a larger network adds cost without better held-out evidence.

Work through a small example

Represent a message by x=[positive_words, negative_words]=[2,1]. Choose weights w=[1,-1] and bias b=0. The logit is z=x dot w+b=1. Sigmoid gives 1/(1+exp(-1))=0.731.

Two scalar features feed one logistic neuronPositive count two and negative count one flow through weights one and minus one. They enter a circular scalar logit of one. Sigmoid then produces probability 0.731. positive2negative1 × 1× −1z = 1sigmoid0.731

At threshold 0.5, the model predicts class 1. The threshold is a decision rule. It is not part of logistic regression training. Change the threshold on validation data when error costs differ.

The general rule

For a batch X shaped (B,D), weights have shape (D,). Logits z=Xw+b have shape (B,). The boundary is the set where probability equals 0.5, or Xw+b=0. It is a line in two dimensions and a hyperplane in higher dimensions.

Sigmoid is monotonic, so the boundary stays linear. More training cannot represent XOR or another pattern that is not linearly separable.

Implement and inspect

import torch

x = torch.tensor([[2.0, 1.0], [0.0, 2.0]])
w = torch.tensor([1.0, -1.0])
b = torch.tensor(0.0)
logits = x @ w + b
probabilities = torch.sigmoid(logits)
predictions = (probabilities >= 0.5).long()

print(logits.tolist())
print([round(p, 3) for p in probabilities.tolist()])
print(predictions.tolist())

Expected output is logits [1.0,-2.0], probabilities [0.731,0.119], and predictions [1,0]. Input shape is (2,2). Every row receives one scalar logit.

Engineering checks

Go deeper

With cross-entropy and L2 regularization, binary logistic regression has a convex objective. Any local optimum is global. But separable unregularized data can drive weight norms upward without a finite minimum.

Coefficients describe changes in log-odds while other features stay fixed. An increase of feature j by one adds w_j to log-odds. Correlated features make individual coefficients unstable, even when predictions stay stable.

Practice and recap

Question: For x=[3,1], w=[0.5,-1], and b=0, find the logit and class at threshold 0.5.

Worked answer

The logit is 3*0.5 + 1*(-1)=0.5. Sigmoid is about 0.622, so the predicted class is 1. The decision depends only on the sign of the logit at threshold 0.5.

Logistic regression learns one linear boundary and probability scores. The scores still need held-out calibration checks.