Kullback-Leibler (KL) divergence, also known as relative entropy, is a measure of how one probability distribution diverges from a second, expected probability distribution. It is used in various fields such as machine learning, data science and information theory.
The KL-Divergence for two discrete probability distributions
In the context of continuous distributions, the KL-Divergence is defined using integrals instead of summation:
It's important to note that KL-Divergence is not symmetric. This means that
Here's a simple Python code snippet to compute the KL-Divergence between two probability distributions using the SciPy library:
from scipy.special import kl_div
import numpy as np
# Define two probability distributions
p = np.array([0.1, 0.2, 0.7])
q = np.array([0.2, 0.3, 0.5])
# Compute KL-Divergence
kl_divergence = kl_div(p,q).sum()
print(f"KL-Divergence: {kl_divergence}")
In this code snippet, p and q are NumPy arrays representing two discrete probability distributions. The kl_div function from the scipy.special module is used to compute the element-wise Kullback-Leibler divergence between these two distributions.
Finally, we sum up all the elements in the resulting array to obtain the total KL-divergence between the two distributions, p and q.