Cross entropy is1 $$ H(p, q) = \mathbb E_{p} \left[ -\log q \right]. $$ Cross entropy $H(p, q)$ can …
The f-divergence is defined as1 $$ \operatorname{D}_f = \int f\left(\frac{p}{q}\right) q\mathrm …
The Jensen-Shannon divergence is a symmetric divergence of distributions $P$ and $Q$, $$ …
Shannon entropy $S$ is the expectation of information content $I(X)=-\log \left(p\right)$1, …
Mutual information is defined as $$ I(X;Y) = \mathbb E_{p_{XY}} \ln \frac{P_{XY}}{P_X P_Y}. $$ In …
The Fraser information is $$ I_F(\theta) = \int g(X) \ln f(X;\theta) , \mathrm d X. $$ When …
Fisher information measures the second moment of the model sensitivity with respect to the parameters.
The code function produces code words. The expected length of the code word is limited by the …