SRM: Structural Risk Minimization

[[ERM]] ERM: Empirical Risk Minimization In a [[learning problem]] The Learning Problem The learning problem proposed by Vapnik:1 Given a sample: $\{z_i\}$ in the probability space $Z$; Assuming a probability measure on the probability space $Z$; Assuming a set of functions $Q(z, \alpha)$ (e.g. loss functions), where $\alpha$ is a set of parameters; A risk functional to be minimized by tunning “the handles” $\alpha$, $R(\alpha)$. The risk functional is $$ R(\alpha) = \int Q(z, \alpha) \,\mathrm d F(z). $$ A learning problem … may lead to overfitting since ERM only selects the model to fit the train data well.

Though Regularized Risk is designed to attack the overfitting problem based on the empirical risk and the complexity of the model, the hyperparamter to determine the weight of the two composition is not easy to decide systematically.

There are two main categories of structural risk minimization method:

• Cross-validation (CV),
• Using Statistical Learning Theory (SLT).