---

Kullback-Leibler Divergence

• Bregman Divergences
• ML1 - Kullback-Leibler divergence - how to compare probability distributions

Kullback-Leibler (KL) Divergence

The KL Divergence is a measure of how one probability distribution differs from another. It is sometimes called relative entropy.

Formally, for two probability distributions (true distribution) and (approximation), the KL divergence from to is:

For a continuos case:

The KL divergence measures how “surprise” you would be if you used instead of .

Properties:

• If and are identical then .

• Always non-negative:
• Asymmetric:

---

Use Cases in Machine Learning

• Variational Inference: approximate a complex posterior with a simpler by minimizing
• Generative Models: compare a model distribution to real data
• Regularization: Encourage a learned distribution to stay close to a prior

---

Intuition example

Imagine you have:

• = true probabilities of weather: sunny 0.7, rainy 0.3
• = your predicted probabilities: sunny 0.6, rainy 0.4

KL divergence measures how different your prediction (Q) is from reality (P).

• Large differences → higher KL divergence
• Small differences → lower KL divergence