Tensor by Tensor

MODULE 01 · LESSON 06

Entropy

Entropy measures uncertainty. KL divergence compares two probability distributions.

PLAIN-LANGUAGE INTRODUCTION

The idea

Entropy measures uncertainty. KL divergence compares two probability distributions.

A worked example

Use coin probabilities p=[0.75,0.25] and q=[0.5,0.5]. Then H(p)=0.811 bits and KL(p||q)=0.189 bits.

The input

Probability lists whose values add to 1.

The result

Uncertainty for one list or extra coding cost between two lists.

Why you use it

These measures describe uncertain targets and mismatched model probabilities.

A limit to remember

KL divergence is not symmetric. Swapping p and q can change it.

Check your understanding

  1. A probability states how often an outcome is expected. Here, heads has 0.75. Tails has 0.25.
  2. Entropy uses H(p) = −Σ p log₂ p. The symbol Σ means add one term per outcome.
  3. For p, entropy is about 0.811 bits. The fair distribution q has 1 bit.
  4. Using q for outcomes from p costs 1 bit. The extra cost is 1 − 0.811 = 0.189 bits.
  5. Common mistake: KL is not an ordinary distance. Its direction matters.
Open the detailed notes ↗