Generative Model: Auto-Encoder

#Self-supervised Learning #Generative Model #Auto-Encoder #Basics

Autoencoders (AE) are machines that encodes inputs into a compact latent space.

The simplest auto-encoder is rather easy to understand.

The loss can be chosen based on the demand, e.g., cross entropy for binary labels.

Notation: dot ($\cdot$)

We use a single vertically centered dot, i.e., $\cdot$, to indicate that the function or machine can take in arguments.

A simple autoencoder can be achieved using two neural nets, e.g.,

$$ \begin{align} {\color{green}h} &= {\color{blue}g}{\color{blue}(}{\color{blue}b} + {\color{blue}w} x{\color{blue})} \ \hat x &= {\color{red}\sigma}{\color{red}(c} + {\color{red}v} {\color{green}h}{\color{red})}, \end{align} $$

where in this simple example,

${\color{blue}g(b + w \cdot )}$ is the encoder, and
${\color{red}\sigma(c + v \cdot )}$ is the decoder.

For binary labels, we can use a simple cross entropy as the loss.

Code

See Lippe¹.

Lippe Lippe P. Tutorial 9: Deep Autoencoders — UvA DL Notebooks v1.1 documentation. In: UvA Deep Learning Tutorials [Internet]. [cited 20 Sep 2021]. Available: https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial9/AE_CIFAR10.html ↩︎

Planted: 2021-08-13 by L Ma;

References:

Lippe Lippe P. Tutorial 9: Deep Autoencoders — UvA DL Notebooks v1.1 documentation. In: UvA Deep Learning Tutorials [Internet]. [cited 20 Sep 2021]. Available: https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial9/AE_CIFAR10.html

Dynamic Backlinks to wiki/machine-learning/generative-models/autoencoder:

Variational Auto-Encoder

In an inference problem, $p(z\vert x)$, which is used to infer $z$ from $x$. $$ p(z\vert x) = …

L Ma (2021). 'Generative Model: Auto-Encoder', Datumorphism, 08 April. Available at: https://datumorphism.leima.is/wiki/machine-learning/generative-models/autoencoder/.