Bias-Variance

GROWING This note is being developed. It is readible but changes are being applied.
wiki/machine-learning/basics/bias-variance.md Current Note ID: The unique ID of this note.
#Statistical Learning #Basics

Bias and Variance

Suppose f(X) is a perfect model that represents a “tight” model of the dataset (X,Y) but some irredicible error ϵ,

(1)Y=f(X)+ϵ.

On the other hand, we build another model using a specific method such as k-nearest neighbors, which is denoted as k(X).

Why the two models?

Why are we talking about the perfect model and a model using a specific method?

The perfect model f(X) is our ultimate goal, while the model using a specific method k(X) is our effort of approaching the ultimate model.

The bias measures the deficit between k(X) and the perfect model f(X),

Bias[k(X)]=E[k(X)]−f(X)

Zero bias means we are matching the perfect model.

E[g(X)]

E[g(X)] is the expectation of the function.

The variance of the model is a measurement of the consistency

Variance(k(X))=E((k(X)−E(k(X)))2)

The larger the variance, the more wiggly the model is.

Mean Square Error

Bias measures the deficit between the specific model and the perfect model. To measure the deficit between the specific model and the actual data point, we need the Mean Squared Error (MSE).

The Mean Squared Error (MSE) is defined as

(2)MSE(X)=E((Y−k(X))2).

This form of expected error can also be used to evaluate models, i.e., calculate expectations by varying models. A straightforward decomposition using equation (1) shows that we have three components in our expected error. There are several ways to derive the decomposition of the expected error. We only show one here.

ExpectedError(X)=E((Y−k)2)(3)=E((f+ϵ−k)2)(4)=E((f−Ek−(k−k)+ϵ)2)(5)=E((f−Ek)2+(k−Ek)2+ϵ2−(f−Ek)(k−Ek)+(f−Ek)ϵ−(k−Ek)ϵ)(6)=E((f−Ek)2)+E((k−Ek)2)+E(ϵ2)−2E((f−Ek)(k−Ek))+2E((f−Ek)ϵ)−2E((k−Ek)ϵ)(7)=(f−Ek)2+E((k−Ek)2)+E(ϵ2)(8)=Bias(k)2+Variance(k)+Irreducible Error

In this derivation, we’ve used several relations.

  1. We used E((f−Ek)2)=(f−Ek)2 becuase the term (f−Ek)2 is constant thus the expectation value is itself.
  2. We have dropped the terms in red. They are related to the fact that the irreducible error is required to be zero, E(ϵ)=0. If it is not zero then the model f(X) is not perfect.
    1. E((f−Ek)(k−Ek))=(f−Ek)E((k−Ek))=0.
    2. E((f−Ek)ϵ)=(f−Ek)E(ϵ)=0.
    3. E((k−Ek)ϵ)=0.

Bias-Variance Tradeoff

The more parameters we introduce in the model, it is more likely to reduce the bias. However, at some point, the more complexity we have in the model, the more wiggles the model will have. Thus the variance will be larger.

Free Parameters

Fermi once said,

I remember my friend Johnny von Neumann used to say, with four parameters I can fit an elephant, and with five I can make him wiggle his trunk.

There is a nice story about Dyson and Fermi behind this.

Planted: by ;

References:
  1. Understanding the Bias-Variance Tradeoff
  2. The Bias-Variance trade-off : Explanation and Demo
  3. The Element of Statistical Learning
  4. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An Introduction to Statistical Learning. In Springer Texts in Statistics. Springer Science & Business Media.
Similar Articles:
Confusion Matrix (Contingency Table)
Confusion Matrix It is much easier to understand the confusion matrix if we use a binary …
Unsupervised Learning: SVM
unsupervised learning: support vector machine
Linear Methods
linear methods
Machine Learning Overview
A brief overview of machine learning
wiki/machine-learning/basics/bias-variance Links to:
ERM: Empirical Risk Minimization
In a [[learning problem]] The Learning Problem The learning problem proposed by Vapnik:1 Given a …

L Ma (2019). 'Bias-Variance', Datumorphism, 06 April. Available at: https://datumorphism.leima.is/wiki/machine-learning/basics/bias-variance/.