SRM: Structural Risk Minimization

#Data #Loss

ERM Empirical risk $R$ is a measurement the goodness of fit based on empirical information. Empirical risk minimization minimizes the empirical risk to select a good model $\hat f$ out of all possible models $f$ in our hypothesis space for a dataset $\mathcal D$, $$ \hat f = \operatorname{argmin} R(f, \mathcal D). $$ Empirical Risk Example For example, the emprical risk can be represented by the negative log likelihood. A negative log likelihood (NLL) for a model $\theta$ of dataset $\mathcal D$ may lead to overfitting since ERM only selects the model to fit the train data well.

Though Regularized Risk is designed to attack the overfitting problem based on the empirical risk and the complexity of the model, the hyperparamter to determine the weight of the two composition is not easy to decide systematically.

There are two main categories of structural risk minimization method:

  • Cross-validation (CV),
  • Statistical Learning Theory (SLT).

Published: by ;

Table of Contents

Current Ref:

  • cards/machine-learning/learning-theories/