Centered Kernel Alignment (CKA)

Centered Kernel Alignment (CKA) is a similarity metric designed to measure the similarity of between representations of features in neural networks¹.

Definition of CKA

CKA is based on the [[Hilbert-Schmidt Independence Criterion (HSIC)]] Hilbert-Schmidt Independence Criterion (HSIC) Given two kernels of the feature representations $K=k(x,x)$ and $L=l(y,y)$, HSIC is defined as12 $$\mathrm{HSIC}(K,L)=\frac{1}{(n-1{)}^2}\mathrm{tr}(KHLH),$$ where $x$, $y$ are the representations of features, $n$ is the dimension of the representation of the features, $H$ is the so-called [[centering matrix]] Centering Matrix Useful when centering a vector around its mean . We can choose different kernel functions $k$ and $l$. For example, if $k$ and $l$ are linear kernels, … .

But HSIC is not invariant to isotropic scaling which is required for a similarity metric of representations¹. CKA is a normalization of HSIC,

$$\mathrm{CKA}(K,L)=\frac{\mathrm{HSIC}(K,L)}{\sqrt{\mathrm{HSIC}(K,K)\mathrm{HSIC}(L,L)}}.$$

Applications

CKA has Problems too

Seita et al argues that CKA is a metric based on intuitive tests, i.e., calculate cases that we believe that should be similar and check if the CKA values is consistent with this intuition². Seita et al built a quantitive benchmark².