Weight initialization
Introduction
Start neural parameters at scales that break symmetry and keep signals usable.
- Learning goal
Connect fan-in scaling to Xavier and Kaiming initialization and verify activations.
- Before you start
Variance, linear layers, activation functions, and gradients.
Lesson plan
- Break hidden-unit symmetry
- Match scale to fan-in and activation
- Measure signal and gradient statistics
The problem
A neural network needs parameter values before its first update. Identical values for every hidden unit preserve symmetry. Identical units receive identical gradients and stay copies.
Random values break symmetry, but scale matters. Repeated multiplication by small weights shrinks activations and gradients. Large weights can amplify them or push sigmoid and tanh units into saturated regions.
Work through a small example
A neuron sums fan_in inputs. Assume three conditions. The inputs are independent and zero-mean. The weights are independent and zero-mean. The inputs and weights are independent of each other. Then unit input variance and weight variance s^2 give pre-activation variance of about fan_in*s^2. A choice of s^2=1/fan_in keeps this near one.
ReLU sets roughly half of symmetric inputs to zero. Kaiming initialization compensates with weight variance near 2/fan_in.
The general rule
Xavier initialization balances input and output widths. People commonly pair it with tanh-like activations. Kaiming initialization accounts for the gain of a rectifier. People commonly pair it with ReLU. These arguments preserve variance under approximate independence and distribution assumptions.
Initialization gives optimization a reasonable starting signal scale. It does not guarantee convergence, prevent overfitting, or repair an unsuitable architecture.
Implement and inspect
import torch
from torch import nn
torch.manual_seed(4)
x = torch.randn(4096, 100)
layer = nn.Linear(100, 80, bias=False)
nn.init.kaiming_normal_(layer.weight, nonlinearity="relu")
pre = layer(x)
post = torch.relu(pre)
print(tuple(layer.weight.shape))
print(round(x.std().item(), 3))
print(round(pre.std().item(), 3))
print(round(post.std().item(), 3))
The weight shape is (80,100). Expected standard deviations for input, pre-activation, and post-ReLU are approximately 1.00, 1.41, and 0.82 in this seeded CPU example. Small differences across supported environments are normal.
Engineering checks
- Set and record a seed. Test your conclusions across several seeds.
- Measure activation and gradient distributions layer by layer.
- Match initializer gain to the actual activation function.
- Do not accidentally reinitialize a loaded pretrained model.
Zero biases are usually safe. Random weights already break hidden-unit symmetry. A zero value for every weight is the harmful case.
Go deeper
For ReLU, the Kaiming argument preserves the second moment E[x²] approximately, not the variance after the activation. ReLU outputs have a nonzero mean. So their standard deviation can be about 0.82 even when the second moment is preserved. Correlations, residual branches, normalization, attention, and finite width also violate the simplest derivation. Deep networks can need architecture-specific scaling.
Dynamical isometry asks whether input-output Jacobian singular values remain near one, a stronger condition than stable activation variance. It helps explain why two initializers with similar variances can produce different optimization behavior.
Practice and recap
Question: A ReLU layer has fan_in=200. What Kaiming standard deviation does the simple rule suggest?
Worked answer
Variance is 2/200=0.01. Standard deviation is sqrt(0.01)=0.1. This is a scale argument, not a claim that every sampled layer has exactly that empirical standard deviation.
Initialization should break symmetry and keep signals usable. Verify the actual network. Do not trust a formula without measurements.