Deep Infomax
[[Mutual information]]
Mutual Information
Mutual information is defined as
Being a quantity that is notoriously hard to compute, mutual information
Local Mutual Information
To compare local features to the encoder output, we need to extract values from inside the encoder, i.e.,
The first step,
The objective is
Code
- rdevon/DIM: by the authors
- DuaneNielsen/DeepInfomaxPytorch: a clean implementation
- Devon2018 Devon Hjelm R, Fedorov A, Lavoie-Marchildon S, Grewal K, Bachman P, Trischler A, et al. Learning deep representations by mutual information estimation and maximization. arXiv [stat.ML]. 2018. Available: http://arxiv.org/abs/1808.06670
- Newell2020 Newell A, Deng J. How Useful is Self-Supervised Pretraining for Visual Tasks? arXiv [cs.CV]. 2020. Available: http://arxiv.org/abs/2003.14323
wiki/machine-learning/contrastive-models/deep-infomax
:wiki/machine-learning/contrastive-models/deep-infomax
Links to:L Ma (2021). 'Deep Infomax', Datumorphism, 09 April. Available at: https://datumorphism.leima.is/wiki/machine-learning/contrastive-models/deep-infomax/.