MODULE 01 · LESSON 06
Entropy
Entropy measures uncertainty. KL divergence compares two probability distributions.
PLAIN-LANGUAGE INTRODUCTION
The idea
Entropy measures uncertainty. KL divergence compares two probability distributions.
A worked example
Use coin probabilities
p=[0.75,0.25] and q=[0.5,0.5]. Then H(p)=0.811 bits and KL(p||q)=0.189 bits.The input
Probability lists whose values add to 1.
The result
Uncertainty for one list or extra coding cost between two lists.
Why you use it
These measures describe uncertain targets and mismatched model probabilities.
A limit to remember
KL divergence is not symmetric. Swapping p and q can change it.
Check your understanding
- A probability states how often an outcome is expected. Here, heads has
0.75. Tails has0.25. - Entropy uses
H(p) = −Σ p log₂ p. The symbolΣmeans add one term per outcome. - For
p, entropy is about0.811bits. The fair distributionqhas1bit. - Using
qfor outcomes frompcosts1bit. The extra cost is1 − 0.811 = 0.189bits. - Common mistake: KL is not an ordinary distance. Its direction matters.