kl_divergence

Computes the Kullback-Leibler divergence from $Q$ to $P$.

What does it measure?

KL divergence answers: if I believe the world follows distribution $Q$, but it actually follows $P$, how many extra bits do I waste per observation?

It measures the information cost of using the wrong model. When $P=Q$, there is no cost: $D_{KL}(P\parallel Q)=0$. As $P$ and $Q$ diverge, the cost grows without bound.

KL divergence is not symmetric

$D_{KL}(P\parallel Q)\ne D_{KL}(Q\parallel P)$ in general. This asymmetry is intentional and meaningful: the cost of approximating $P$ with $Q$ is different from approximating $Q$ with $P$.

If you need a symmetric measure, use js_divergence instead.

Undefined for disjoint supports

If $P$ assigns positive probability to a value that $Q$ assigns zero probability, then $p(x){\log }_2(p(x)/q(x))\to \mathrm{\infty }$. kl_divergence returns Err(InfoError::UndefinedDivergence) in this case.

This is a fundamental property, not a library limitation. If your distributions may have different supports, use js_divergence, which is always finite.

Formula

$$D_{KL}(P\parallel Q)=\sum _{x}p(x){\log }_2\frac{p(x)}{q(x)}$$

The probabilities are estimated empirically from the input slices.

Signature

pub fn kl_divergence<T>(p: &[T], q: &[T]) -> Result<f64, InfoError>
where
    T: Eq + Hash

pub fn kl_divergence_unchecked<T>(p: &[T], q: &[T]) -> f64
where
    T: Eq + Hash

Parameters

Parameter	Description
p	Samples from the true distribution $P$
q	Samples from the reference (model) distribution $Q$

The slices do not need to have the same length — they are independent samples from each distribution.

Returns

$D_{KL}(P\parallel Q)$ in bits, or:

Error	When
InfoError::EmptyInput	Either slice is empty
InfoError::UndefinedDivergence	p contains a value absent from q

Examples

use entropium::{kl_divergence, InfoError};

// Same distribution → KL = 0
let p = vec![0, 0, 1, 1, 1];
assert_eq!(kl_divergence(&p, &p).unwrap(), 0.0);

// KL is not symmetric
let p = vec![0, 0, 0, 1, 1, 2];     // P(0)=1/2, P(1)=1/3, P(2)=1/6
let q = vec![0, 1, 2, 2, 2, 2];     // Q(0)=1/6, Q(1)=1/6, Q(2)=2/3
let kl_pq = kl_divergence(&p, &q).unwrap();
let kl_qp = kl_divergence(&q, &p).unwrap();
assert!((kl_pq - kl_qp).abs() > 1e-10);

// Disjoint support → error
assert_eq!(
    kl_divergence(&[0, 1], &[2, 3]).unwrap_err(),
    InfoError::UndefinedDivergence
);

// Sample sizes can differ
let p_large = vec![0u8; 1000].into_iter().chain(vec![1u8; 500]).collect::<Vec<_>>();
let q_small = vec![0u8, 0, 1];
let kl = kl_divergence(&p_large, &q_small).unwrap();

Practical uses

• Model evaluation: $D_{KL}(P_{\text{data}}\parallel P_{\text{model}})$ measures how well a model approximates the data distribution. Minimising this is equivalent to maximum-likelihood estimation.
• Variational inference: VI minimises $D_{KL}(q\parallel p)$ (note the reversed order), where $q$ is a tractable approximation and $p$ is the true posterior.
• A/B testing: compare the output distributions of two system versions to quantify how much they differ.
• Anomaly detection: compute $D_{KL}(P_{\text{live}}\parallel P_{\text{baseline}})$ over a rolling window; a spike signals a distribution shift.

Properties

Property	Statement
Non-negativity	$D_{KL}(P|Q)\ge 0$ (Gibbs' inequality)
Identity	$D_{KL}(P|Q)=0$ iff $P=Q$
Asymmetry	$D_{KL}(P|Q)\ne D_{KL}(Q|P)$ in general
Unbounded	$D_{KL}(P|Q)\to \mathrm{\infty }$ as supports diverge
Relation to cross-entropy	$D_{KL}(P|Q)=H(P,Q)-H(P)$