Noise Contrastive Estimation: NCE

Noise contrastive estimation (NCE) objective function is1

$$ \mathcal L = \mathbb E_{x, x^{+}, x^{-}} \left[ - \ln \frac{ C(x, x^{+})}{ C(x,x^{+}) + C(x,x^{-}) } \right], $$

where

  • $x^{+}$ represents data similar to $x$,
  • $x^{-}$ represents data dissimilar to $x$,
  • $C(\cdot, \cdot)$ is a function to compute the similarities.

For example, we can use

$$ C(x, x^{+}) = e^{ f(x)^T f(x^{+}) }, $$

so that the objective function becomes

$$ \mathcal L = \mathbb E_{x, x^{+}, x^{-}} \left[ - \ln \frac{ e^{ f(x)^T f(x^{+}) } }{ e^{ f(x)^T f(x^{+}) } + e^{ f(x)^T f(x^{-}) } } \right]. $$

The function $f(\cdot)$ can be an encoder.

Planted: by ;

L Ma (2021). 'Noise Contrastive Estimation: NCE', Datumorphism, 08 April. Available at: https://datumorphism.leima.is/cards/machine-learning/learning-theories/noise-contrastive-estimation/.