Gini Impurity

Suppose we have a dataset $\{0,1{\}}^{10}$, which has 10 records and 2 possible classes of objects $\{0,1\}$ in each record.

The first example we investigate is a pure 0 dataset.

For such an all-0 dataset, we would like to define its impurity as 0. Same with an all-1 dataset. For a dataset with 50% of 1 and 50% of 0, we would define its impurity as max due to the symmetries between 0 and 1.

Definition

Given a dataset $\{0,1,\dots ,d{\}}^n$, the Gini impurity is calculated as

$$G=\sum _{i\in \{0,1,...,d\}}p(i)(1-p(i)),$$

where $p(i)$ is the probability of a random picked record being class $i$.

In the above example, we have two classes, $\{0,1\}$. The probabilities are

$$\begin{array}{}\text{(1)}& p(0)=& 1\text{(2)}& p(1)=& 0\end{array}.$$

The Gini impurity is

$$G=p(0)(1-p(0))+p(1)(1-p(1))=0+0=0.$$

Examples

Suppose we have another dataset with 50% of the values being 50%.

The Gini impurity is

$$G=p(0)(1-p(0))+p(1)(1-p(1))=0.5\ast 0.5+0.5\ast 0.5=0.5.$$

For data with two possible values $\{0,1\}$, the maximum Gini impurity is 0.25. The following chart shows all the possible values of the Gini impurity for two-value dataset.

For data with three possible values, the Gini impurity is also visualized using the same chart given the condition that $p_3=1-p_1-p_2$.