Measures of Generalizability

#Model Selection #Generalizability

To measure the generalization, we define a generalization error,

$$ \begin{align} \mathcal G = \mathcal L_{P}(\hat f) - \mathcal L_E(\hat f), \end{align} $$

where $\mathcal L_{P}$ is the population loss, $\mathcal L_E$ is the empirical loss, and $\hat f$ is our model by minimizing the empirical loss.

However, we do not know the actual joint probability $p(x, y)$ of our dataset $\{x_i, y_i\}$. Thus the population loss is not known. In machine learning, we usually use cross validation Cross validation is a method to estimate the risk The learning problem posed by Vapnik:1 Given a sample: $\{z_i\}$ in the probability space $Z$; Assuming a probability measure on the probability space $Z$; Assuming a set of functions $Q(z, \alpha)$ (e.g. loss functions), where $\alpha$ is a set of parameters; A risk functional to be minimized by tunning “the handles” $\alpha$, $R(\alpha)$. The risk functional is $$ R(\alpha) = \int Q(z, \alpha) \,\mathrm d F(z). where we split our dataset into train and test dataset. We approximate the population loss using the test dataset.

Published: by ;

Lei Ma (2020). 'Measures of Generalizability', Datumorphism, 11 April. Available at:

Table of Contents

Current Ref:

  • wiki/model-selection/