Noise Contrastive Estimation: NCE

#Learning Theory

Noise contrastive estimation (NCE) objective function is1

$$ \mathcal L = \mathbb E_{x, x^{+}, x^{-}} \left[ - \ln \frac{ C(x, x^{+})}{ C(x,x^{+}) + C(x,x^{-}) } \right], $$


  • $x^{+}$ represents data similar to $x$,
  • $x^{-}$ represents data dissimilar to $x$,
  • $C(\cdot, \cdot)$ is a function to compute the similarities.

For example, we can use

$$ C(x, x^{+}) = e^{ f(x)^T f(x^{+}) }, $$

so that the objective function becomes

$$ \mathcal L = \mathbb E_{x, x^{+}, x^{-}} \left[ - \ln \frac{ e^{ f(x)^T f(x^{+}) } }{ e^{ f(x)^T f(x^{+}) } + e^{ f(x)^T f(x^{-}) } } \right]. $$

The function $f(\cdot)$ can be an encoder.

  1. Liu2020 Liu X, Zhang F, Hou Z, Wang Z, Mian L, Zhang J, et al. Self-supervised Learning: Generative or Contrastive. arXiv [cs.LG]. 2020. Available:  ↩︎

Published: by ;

L Ma (2021). 'Noise Contrastive Estimation: NCE', Datumorphism, 08 April. Available at:

Current Ref:

  • cards/machine-learning/learning-theories/