To measure the generalization, we define a generalization error,

\begin{align} \mathcal G = \mathcal L_{P}(\hat f) - \mathcal L_E(\hat f), \end{align}

where $\mathcal L_{P}$ is the population loss, $\mathcal L_E$ is the empirical loss, and $\hat f$ is our model by minimizing the empirical loss.

However, we do not know the actual joint probability $p(x, y)$ of our dataset $\{x_i, y_i\}$. Thus the population loss is not known. In machine learning, we usually use where we split our dataset into train and test dataset. We approximate the population loss using the test dataset.

Planted: by ;

L Ma (2020). 'Measures of Generalizability', Datumorphism, 11 April. Available at: https://datumorphism.leima.is/wiki/model-selection/measures-of-generalizability/.