The f-divergence is defined as1

$$\operatorname{D}_f = \int f\left(\frac{p}{q}\right) q\mathrm d\mu,$$

where $p$ and $q$ are two densities and $\mu$ is a reference distribution.

Requirements on the generating function

The generating function $f$ is required to

• be convex, and
• $f(1) =0$.

For $f(x) = x \log x$ with $x=p/q$, f-divergence is reduced to the KL divergence

\begin{align} &\int f\left(\frac{p}{q}\right) q\mathrm d\mu \\ =& \int \frac{p}{q} \log \left( \frac{p}{q} \right) \mathrm d\mu \\ =& \int p \log \left( \frac{p}{q} \right) \mathrm d\mu. \end{align}

For more special cases of f-divergence, please refer to wikipedia1. Nowozin et al also provided a concise review of f-divergence2.

Published: by ;

Lei Ma (2021). 'f-Divergence', Datumorphism, 09 April. Available at: https://datumorphism.leima.is/cards/information/f-divergence/.

Current Ref:

• cards/information/f-divergence.md