Mutual Information

2 Contrastive Model: Context-Instance

#Self-supervised Learning #Contrastive Model #Mutual Information #Basics

In contrastive methods, we can manipulate the data to create data entries and infer the changes …

4 Contrastive Predictive Coding

#Self-supervised Learning #Contrastive Model #Mutual Information #Contrastive #Predictive Coding #Basics

Contrastive Predictive Coding, aka CPC, is an autoregressive model combined with InfoNCE loss1. …

4 Deep Infomax

#Self-supervised Learning #Contrastive Model #Mutual Information #Deep Infomax #Basics

Max Global Mutual Information Why not just use the global mutual information of the input and …

Mutual Information

#Information Theory #Mutual Information

Mutual information is defined as $$ I(X;Y) = \mathbb E_{p_{XY}} \ln \frac{P_{XY}}{P_X P_Y}. $$ In …