Noise Contrastive Estimation: NCE

Noise contrastive estimation (NCE) objective function is¹

$$ \mathcal L = \mathbb E_{x, x^{+}, x^{-}} \left[ - \ln \frac{ C(x, x^{+})}{ C(x,x^{+}) + C(x,x^{-}) } \right], $$

where

• $x^{+}$ represents data similar to $x$,
• $x^{-}$ represents data dissimilar to $x$,
• $C(\cdot, \cdot)$ is a function to compute the similarities.

For example, we can use

$$ C(x, x^{+}) = e^{ f(x)^T f(x^{+}) }, $$

so that the objective function becomes

$$ \mathcal L = \mathbb E_{x, x^{+}, x^{-}} \left[ - \ln \frac{ e^{ f(x)^T f(x^{+}) } }{ e^{ f(x)^T f(x^{+}) } + e^{ f(x)^T f(x^{-}) } } \right]. $$

The function $f(\cdot)$ can be an encoder.