Noise Contrastive Estimation: NCE

#Learning Theory

Noise contrastive estimation (NCE) objective function is1

$$ \mathcal L = \mathbb E_{x, x^{+}, x^{-}} \left[ - \ln \frac{ C(x, x^{+})}{ C(x,x^{+}) + C(x,x^{-}) } \right], $$

where

  • $x^{+}$ represents data similar to $x$,
  • $x^{-}$ represents data dissimilar to $x$,
  • $C(\cdot, \cdot)$ is a function to compute the similarities.

For example, we can use

$$ C(x, x^{+}) = e^{ f(x)^T f(x^{+}) }, $$

so that the objective function becomes

$$ \mathcal L = \mathbb E_{x, x^{+}, x^{-}} \left[ - \ln \frac{ e^{ f(x)^T f(x^{+}) } }{ e^{ f(x)^T f(x^{+}) } + e^{ f(x)^T f(x^{-}) } } \right]. $$

The function $f(\cdot)$ can be an encoder.


  1. Liu2020 Liu X, Zhang F, Hou Z, Wang Z, Mian L, Zhang J, et al. Self-supervised Learning: Generative or Contrastive. arXiv [cs.LG]. 2020. Available: http://arxiv.org/abs/2006.08218  ↩︎

Published: by ;

L Ma (2021). 'Noise Contrastive Estimation: NCE', Datumorphism, 08 April. Available at: https://datumorphism.leima.is/cards/machine-learning/learning-theories/noise-contrastive-estimation/.

Current Ref:

  • cards/machine-learning/learning-theories/noise-contrastive-estimation.md