How to Measure Similarities of Feature Representations

[emptymalei]

Centered Kernel Alignment (CKA) is a similarity metric designed to measure the similarity of between representations of features in neural networks¹.

Definition of CKA

CKA is based on the Hilbert-Schmidt Idependence Criterion (HSIC). Given two kernels of the feature representations $K=k(x,x)$ and $L=l(y,y)$, HSIC is defined as²¹

$$
\operatorname{HSIC}(K, L) = \frac{1}{(n-1)^2} \operatorname{tr}( K H L H ),
$$

where

• $x$, $y$ are the representations of features,
• $n$ is the dimension of the representation of the features,
• $H$ is the so-called centering matrix,

We can choose different kernel functions $k$ and $l$. For example, if $k$ and $l$ are linear kernels, we have $k(x, y) = l(x, y) = x \cdot y$. In this linear case, HSIC is simply $\parallel\operatorname{cov}(x^T,y^T) \parallel^2_{\text{Frobenius}}$

But HSIC is not invariant toisotropic scaling which is required for a similarity metric of representations¹. CKA is a normalization of HSIC,

$$
\operatorname{CKA}(K,L) = \frac{\operatorname{HSIC}(K, L)}{\sqrt{\operatorname{HSIC}(K,K) \operatorname{HSIC}(L,L)}}.
$$

Applications



CKA has problems too

Seita et al argues that CKA is a metric based on intuitive tests, i.e., calculate cases that we believe that should be similar and check if the CKA values is consistent with this intuition³. Seita et al built a quantitive benchmark³.

Footnotes

1. Kornblith S, Norouzi M, Lee H, Hinton G. Similarity of Neural Network Representations Revisited. arXiv [cs.LG]. 2019. Available: http://arxiv.org/abs/1905.00414 ↩ ↩² ↩³

2. Gretton A, Bousquet O, Smola A, Schölkopf B. Measuring Statistical Dependence with Hilbert-Schmidt Norms. Algorithmic Learning Theory. Springer Berlin Heidelberg; 2005. pp. 63–77. doi:10.1007/11564089_7 ↩

3. Seita D. How should we compare neural network representations? In: The Berkeley Artificial Intelligence Research Blog [Internet]. [cited 8 Nov 2021]. Available: https://bair.berkeley.edu/blog/2021/11/05/similarity/ ↩ ↩²