MODULE 01 · LESSON 06

Entropy

Entropy measures uncertainty. KL divergence compares two probability distributions.

PLAIN-LANGUAGE INTRODUCTION

What is this?

Entropy measures uncertainty. KL divergence compares two probability distributions.

One simple example

Use coin probabilities p=[0.75,0.25] and q=[0.5,0.5]. Then H(p)=0.811 bits and KL(p||q)=0.189 bits.

Probability lists whose values add to 1.

Uncertainty for one list or extra coding cost between two lists.

These measures describe uncertain targets and mismatched model probabilities.

KL divergence is not symmetric. Swapping p and q can change it.

WORK THROUGH THE IDEA

A probability states how often an outcome is expected. Here, heads has 0.75. Tails has 0.25.
Entropy uses H(p) = −Σ p log₂ p. The symbol Σ means add one term per outcome.
For p, entropy is about 0.811 bits. The fair distribution q has 1 bit.
Using q for outcomes from p costs 1 bit. The extra cost is 1 − 0.811 = 0.189 bits.
Common mistake: KL is not an ordinary distance. Its direction matters.