# Entropy

Maximum Entropy models makes least assumption about the data

Introducing latent variables to Boltzmann machine and restrict the connections within groups.

Cross entropy is1 $$H(p, q) = \mathbb E_{p} \left[ -\log q \right].$$ Cross entropy $H(p, q)$ can …

Shannon entropy $S$ is the expectation of information content $I(X)=-\log \left(p\right)$1, …

The code function produces code words. The expected length of the code word is limited by the …