17.2. Similarity Measures

This section provides a review of popular similarity and dissimilarity measures in literature.

17.2.1. Introduction

17.2.1.1. Similarity Function

The following is an informal definition of a similarity function.

Definition 17.5 (Similarity function)

Let $\mathbf{X}$ be a dataset of $n$ data points sampled from the Euclidean space ${\mathbb{R}}^m$. Then a similarity function is some function $s:{\mathbb{R}}^m\times {\mathbb{R}}^m\to \mathbb{R}$ for any pair of data points $\mathbf{x},\mathbf{y}\in {\mathbb{R}}^m$ with a structure

$$s(\mathbf{x},\mathbf{y})=s(x_1,\dots ,x_d,y_1,\dots ,y_d)$$

which grows as the data points look similar to each other and satisfies the following properties

1. $0\le s(\mathbf{x},\mathbf{y})\le 1$.

2. $s(\mathbf{x},\mathbf{x})=1$ for every $\mathbf{x}\in {\mathbb{R}}^m$.

3. $s(\mathbf{x},\mathbf{y})=s(\mathbf{y},\mathbf{x})$.

Normally, we require that the similarity function is symmetric; i.e.,

$$s(\mathbf{x},\mathbf{y})=s(\mathbf{y},\mathbf{x}).$$

There are rare cases where asymmetric similarity functions have been used.

Example 17.6 (Similarity from metrics)

Let $d$ be a metric (distance function) defined on ${\mathbb{R}}^n$. Then we have for any $\mathbf{x},\mathbf{y},\mathbf{z}\in {\mathbb{R}}^m$,

1. $d(\mathbf{x},\mathbf{x})=0$.

2. $d(\mathbf{x},\mathbf{y})\ge 0$.

3. $d(\mathbf{x},\mathbf{y})=d(\mathbf{y},\mathbf{x})$.

4. $d(\mathbf{x},\mathbf{y})\le d(\mathbf{x},\mathbf{z})+d(\mathbf{z},\mathbf{y})$.

One way to induce a similarity function is:

$$s(\mathbf{x},\mathbf{y})=\exp (-d(\mathbf{x},\mathbf{y})).$$

We can see that

1. Since $\exp (t)\ge 0$ for every $t\in \mathbb{R}$, hence $s(\mathbf{x},\mathbf{y})\ge 0$.

2. Since $d(\mathbf{x},\mathbf{y})\ge 0$, hence $s(\mathbf{x},\mathbf{y})\le 1$.

3. Since $d$ is symmetric, hence $s$ is symmetric.

4. Since $d(\mathbf{x},\mathbf{x})=0$, hence $s(\mathbf{x},\mathbf{x})=1$.

Thus, $s$ is indeed a similarity function.

Note that a metric satisfies the triangle inequality. However, this property was not required for the similarity function induced from the metric. Thus, a metric imposes an additional structure which is more stringent than the needs for a similarity measure.

For example, the squared distance function $d^2$ satisfies the identity of indiscernibles, is nonnegative and symmetric. It doesn’t satisfy triangle inequality. However, we can construct a similarity function

$$s^{\prime }(\mathbf{x},\mathbf{y})=\exp (-d^2(\mathbf{x},\mathbf{y})).$$

17.2.1.2. Proximity Matrices

Definition 17.6 (Distance matrix)

Let $d$ be a metric on the space ${\mathbb{R}}^m$. Let $\mathbf{X}$ be a dataset of $n$ points $\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}$. Then the distance matrix for $\mathbf{X}$ is defined as

$$\begin{array}{r}{\mathbf{M}}_{\mathrm{dist}}(\mathbf{X})=\left[\begin{array}{cccc}0& d_{12}& \dots & d_{1n}\\ d_{21}& 0& \dots & d_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ d_{n1}& d_{n2}& \dots & 0\end{array}\right]\end{array}$$

where $d_{ij}=d({\mathbf{x}}_i,{\mathbf{x}}_j)$.

Definition 17.7 (Similarity matrix)

Let $s$ be a similarity function on the space ${\mathbb{R}}^m$. Let $\mathbf{X}$ be a dataset of $n$ points $\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}$. Then the similarity matrix for $\mathbf{X}$ is defined as

$$\begin{array}{r}{\mathbf{M}}_{\mathrm{sim}}(\mathbf{X})=\left[\begin{array}{cccc}1& s_{12}& \dots & s_{1n}\\ s_{21}& 1& \dots & s_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ s_{n1}& s_{n2}& \dots & 1\end{array}\right]\end{array}$$

where $s_{ij}=s({\mathbf{x}}_i,{\mathbf{x}}_j)$.

Definition 17.8 (Proximity matrix)

Let $\mathbf{X}$ be a dataset of $n$ points $\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}$. Then an $n\times n$ square matrix $\mathbf{M}$ where every $(i,j)$-th element is some measure of similarity or distance between ${\mathbf{x}}_i$ and ${\mathbf{x}}_j$ is known as a proximity matrix.

In other words, a proximity matrix is either a distance matrix or a similarity matrix.

Each entry in a proximity matrix is called a proximity index.

Remark 17.1 (Asymmetry in proximity)

There are several problems where a proximity metric may not be symmetric. For example, when the objects in the dataset are locations in a hilly area and the measure of the distance is the time taken in going from point A to point B, the time taken may be quite different when going uphill or downhill. Such problems do give rise to asymmetrical proximity matrices. However, we will normally be concerned with symmetric cases.

Definition 17.9 (Proximity graph)

Let $\mathbf{X}$ be a dataset of $n$ points $\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}$. Let $\mathbf{M}$ be a proximity matrix associated with the dataset $\mathbf{X}$. Then the corresponding proximity graph $\mathcal{G}$ is a weighted graph whose nodes/vertices are the data points ${\mathbf{x}}_i$ and the edges have weights equal to the proximity indices.

A symmetric proximity matrix leads to an undirected graph while an asymmetric proximity matrix leads to a directed graph.

17.2.1.3. Scatter and Covariance

Definition 17.10 (Scatter matrix)

Let $\mathbf{X}$ be a dataset of $n$ points $\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}$ sampled from the Euclidean space ${\mathbb{R}}^n$. Let $\overline{\mathbf{x}}$ be the arithmetic mean of the data points.

$$\overline{\mathbf{x}}=\frac{1}{n}\sum _{i=1}^n{\mathbf{x}}_i.$$

Then the scatter matrix for $\mathbf{X}$ is defined as

$${\mathbf{M}}_t(\mathbf{X})=\sum _{i=1}^n({\mathbf{x}}_i-\overline{\mathbf{x}})({\mathbf{x}}_i-\overline{\mathbf{x}}{)}^T.$$

Definition 17.11 (Statistical scatter)

The trace of the scatter matrix is known as the statistical scatter of the data set. It is given by

$$\mathrm{tr}({\mathbf{M}}_t(\mathbf{X}))=\sum _{i=1}^n({\mathbf{x}}_i-\overline{\mathbf{x}}{)}^T({\mathbf{x}}_i-\overline{\mathbf{x}}).$$

Recall that $\mathrm{tr}(\mathbf{A}\mathbf{B})=\mathrm{tr}(\mathbf{B}\mathbf{A})$.

Hence

$$\mathrm{tr}({\mathbf{M}}_t(\mathbf{X}))=\mathrm{tr}(\sum _{i=1}^n({\mathbf{x}}_i-\overline{\mathbf{x}})({\mathbf{x}}_i-\overline{\mathbf{x}}{)}^T)=\mathrm{tr}(\sum _{i=1}^n({\mathbf{x}}_i-\overline{\mathbf{x}}{)}^T({\mathbf{x}}_i-\overline{\mathbf{x}}))=\sum _{i=1}^n({\mathbf{x}}_i-\overline{\mathbf{x}}{)}^T({\mathbf{x}}_i-\overline{\mathbf{x}})$$

since this quantity is a scalar.

Let $\mathcal{C}=\{C_1,\dots ,C_k\}$ be a (hard) clustering (partition) of $\mathbf{X}$ into $k$ clusters. Then for a given cluster $C_j$, the within-scatter-matrix is given by

$${\mathbf{M}}_t(C_j)=\sum _{i\in C_j}({\mathbf{x}}_i-{\mathbf{z}}_j)({\mathbf{x}}_i-{\mathbf{z}}_j{)}^T$$

where ${\mathbf{z}}_j$ is the mean of the cluster $C_i$

$${\mathbf{z}}_j=\frac{1}{|C_j|}\sum _{i\in C_j}{\mathbf{x}}_i.$$

The within-cluster scatter matrix of the partition is given by

(17.2)$${\mathbf{M}}_w(\mathcal{C})=\sum _{j=1}^k{\mathbf{M}}_t(C_j)=\sum _{j=1}^k\sum _{i\in C_j}({\mathbf{x}}_i-{\mathbf{z}}_j)({\mathbf{x}}_i-{\mathbf{z}}_j{)}^T.$$

The within-cluster scatter is the trace of this matrix given by

(17.3)$$\mathrm{tr}({\mathbf{M}}_w(\mathcal{C}))=\sum _{j=1}^k\sum _{i\in C_j}({\mathbf{x}}_i-{\mathbf{z}}_j{)}^T({\mathbf{x}}_i-{\mathbf{z}}_j)$$

The between-cluster scatter matrix for the partition $\mathcal{C}$ is given by

$${\mathbf{M}}_b(\mathcal{C})={\mathbf{M}}_t(\mathbf{X})-{\mathbf{M}}_w(\mathcal{C}).$$

We shall denote the dataset after mean removal as

$$\tilde{\mathbf{X}}=\mathbf{X}-\overline{\mathbf{x}}$$

where the matrix vector subtraction is done according to the standard broadcasting rules.

The covariance matrix of the dataset is given by

(17.4)$$\mathrm{\Sigma }=\frac{1}{n}\tilde{\mathbf{X}}{\tilde{\mathbf{X}}}^T.$$

Sample covariance matrix is often used in statistics which is given by

(17.5)$$\mathbf{S}=\frac{1}{n-1}\tilde{\mathbf{X}}{\tilde{\mathbf{X}}}^T=\frac{n}{n-1}\mathrm{\Sigma }.$$

17.2.2. Common Proximity Measures

We look at some of the common proximity measures for numerical data.

17.2.2.1. Euclidean Distance

(17.6)$$d_2(\mathbf{x},\mathbf{y})=\sqrt{\sum _{i=1}^m(x_i-y_i{)}^2}=\sqrt{(\mathbf{x}-\mathbf{y}{)}^T(\mathbf{x}-\mathbf{y})}.$$

17.2.2.2. Squared Euclidean Distance

(17.7)$$d_2^2(\mathbf{x},\mathbf{y})=\sum _{i=1}^m(x_i-y_i{)}^2=(\mathbf{x}-\mathbf{y}{)}^T(\mathbf{x}-\mathbf{y}).$$

17.2.2.3. Manhattan Distance

This is also known as city-block distance

(17.8)$$d_1(\mathbf{x},\mathbf{y})=\sum _{i=1}^m|x_i-y_i|.$$

Manhattan segmental distance is a variant in which only some of the dimensions selected by a subset $P\subseteq \{1,\dots ,m\}$ are used.

(17.9)$$d_P(\mathbf{x},\mathbf{y})=\frac{1}{|P|}\sum _{i\in P}|x_i-y_i|.$$

17.2.2.4. Maximum Distance

This is also known as the Chebyshev distance.

(17.10)$$d_{max}(\mathbf{x},\mathbf{y})=\underset{i=1}{\overset{m}{max}}|x_i-y_i|.$$

17.2.2.5. Minkowski Distance

This is the generalization of Euclidean, Manhattan and Chebyshev distances for some $p\ge 1$.

(17.11)$$d_p(\mathbf{x},\mathbf{y})={(\sum _{i=1}^m|x_i-y_i{|}^p)}^{\frac{1}{p}}.$$

17.2.2.6. Mahalanobis Distance

As discussed in Example 17.5, Mahalanobis distance can address the issues related to the scale differences between different features and the correlation among features. This is given by

(17.12)$$d_{\mathbf{mah}}(\mathbf{x},\mathbf{y})=\sqrt{(\mathbf{x}-\mathbf{y}{)}^T{\mathrm{\Sigma }}^{-1}(\mathbf{x}-\mathbf{y})}.$$

Mahalanobis distance is invariant to nonsingular transformations. Let $\mathbf{C}$ be any $m\times m$ non-singular matrix.

Let $\mathbf{Y}=\{{\mathbf{y}}_1,\dots ,{\mathbf{y}}_n\}$ be a new dataset constructed by transforming the dataset $\mathbf{X}$ as

$${\mathbf{y}}_i=\mathbf{C}{\mathbf{y}}_i.$$

Then

$$d_{\mathbf{mah}}({\mathbf{x}}_i,{\mathbf{x}}_j)=d_{\mathbf{mah}}({\mathbf{y}}_i,{\mathbf{y}}_j).$$

17.2.2.7. Chord Distance

We normalize the data points to project them to the unit sphere in ${\mathbb{R}}^m$. Then we measure the length of the chord between the two points on the sphere. It is given by

(17.13)$$d_{\mathbf{chord}}(\mathbf{x},\mathbf{y})=\sqrt{2-2\frac{⟨\mathbf{x},\mathbf{y}⟩}{\Vert \mathbf{x}{\Vert }_2\Vert \mathbf{y}{\Vert }_2}}.$$

It is easy to see how this formula is arrived at. Assume $\mathbf{x},\mathbf{y}$ lie on the unit sphere. Then

$$⟨\mathbf{x}-\mathbf{y},\mathbf{x}-\mathbf{y}⟩=\Vert \mathbf{x}{\Vert }_2^2+\Vert \mathbf{y}{\Vert }_2^2-2⟨\mathbf{x},\mathbf{y}⟩=2-2⟨\mathbf{x},\mathbf{y}⟩.$$

When they are not normalized, we normalize them

$$2-2\frac{⟨\mathbf{x},\mathbf{y}⟩}{\Vert \mathbf{x}{\Vert }_2\Vert \mathbf{y}{\Vert }_2}.$$

Finally, for computing the length of the chord, we need to take the square root.

17.2.2.8. Geodesic Distance

Once the data points have been normalized to the unit sphere, one can calculate the shortest path (arc) between any two data points along the surface of the unit sphere. This is known as the geodesic distance. It is given by

(17.14)$$d_{\mathbf{geo}}(\mathbf{x},\mathbf{y})=\arccos (1-\frac{d_{\mathbf{chord}}(\mathbf{x},\mathbf{y})}{2}).$$

17.2.2.9. Cosine Similarity

The cosine similarity between two points is given by

(17.15)$$s_{\cos }(\mathbf{x},\mathbf{y})=\frac{|⟨\mathbf{x},\mathbf{y}⟩|}{\Vert \mathbf{x}{\Vert }_2\Vert \mathbf{y}{\Vert }_2}.$$

It is easy to see that cosine similarity is bound between $0$ and $1$, is symmetric and $s_{\cos }(\mathbf{x},\mathbf{x})=1$.

17.2.3. Proximity Between Clusters

Many clustering algorithms are hierarchical. In the context of agglomerative hierarchical clustering, one needs to merge clusters which are similar to each other in each iteration. Thus, it is imperative to have some measures of similarity or dissimilarity between clusters.

In this subsection, we shall consider two arbitrary clusters of the dataset $\mathbf{X}$ denoted by $C_1$ and $C_2$ where

$$C_1=\{{\mathbf{y}}_1,\dots ,{\mathbf{y}}_r\}\text{ and }{C}_2=\{{\mathbf{z}}_1,\dots ,{\mathbf{z}}_s\}.$$

Let the mean of $C_1$ be given by

$$\overline{\mathbf{y}}=\frac{1}{r}\sum _{i=1}^r{\mathbf{y}}_i.$$

Let the mean of $C_2$ be given by

$$\overline{\mathbf{z}}=\frac{1}{s}\sum _{i=1}^s{\mathbf{z}}_i.$$

17.2.3.1. Mean Based Distance

The mean based distance between two clusters is defined as

(17.16)$$d_{\mathbf{mean}}(C_1,C_2)=d(\overline{\mathbf{y}},\overline{\mathbf{z}}).$$

17.2.3.2. Nearest Neighbor Distance

(17.17)$$d_{\mathbf{nn}}(C_1,C_2)=\underset{1\le i\le r,1\le j\le s}{min}d({\mathbf{y}}_i,{\mathbf{z}}_j).$$

17.2.3.3. Farthest Neighbor Distance

(17.18)$$d_{\mathbf{fn}}(C_1,C_2)=\underset{1\le i\le r,1\le j\le s}{max}d({\mathbf{y}}_i,{\mathbf{z}}_j).$$

17.2.3.4. Average Neighbor Distance

(17.19)$$d_{\mathbf{ave}}(C_1,C_2)=\frac{1}{rs}\sum _{i=1}^r\sum _{j=1}^{s}d({\mathbf{y}}_i,{\mathbf{z}}_j).$$

17.2.3.5. Statistical Distance

(17.20)$$d_{\mathbf{stat}}(C_1,C_2)=\frac{rs}{r+s}(\overline{\mathbf{y}}-\overline{\mathbf{z}}{)}^T(\overline{\mathbf{y}}-\overline{\mathbf{z}}).$$

17.2.3.6. Lance-Williams Formula

Consider a step in an agglomerative hierarchical algorithm where clusters $C_i$ and $C_j$ are being merged into a new cluster $C$ and we wish to consider the distance between another cluster $C_k$ and the new cluster $C$.

The following formula provides a recursive definition such that we can compute the distance $d(C_k,C)$ using existing distance information.

(17.21)$$d(C_k,C_i\cup C_j)={\alpha }_{i}d(C_k,C_i)+{\alpha }_{j}d(C_k,C_j)+\beta d(C_i,C_j)+\gamma |d(C_k,C_i)-d(C_k,C_j)|$$

where $d$ is some distance function between two clusters.

Some common choices of the parameters ${\alpha }_i,{\alpha }_j,\beta ,\gamma$ are given in the following table.

Algorithm	${\alpha }_i$	${\alpha }_j$	$\beta$	$\gamma$
Single-link	$\frac{1}{2}$	$\frac{1}{2}$	$0$	$-\frac{1}{2}$
Complete-link	$\frac{1}{2}$	$\frac{1}{2}$	$0$	$\frac{1}{2}$
Group-average	$\frac{n_i}{n_i+n_j}$	$\frac{n_j}{n_i+n_j}$	$0$	$0$
Weighted group average	$\frac{1}{2}$	$\frac{1}{2}$	$0$	$0$
Centroid	$\frac{n_i}{n_i+n_j}$	$\frac{n_j}{n_i+n_j}$	$\frac{-n_i{n}_j}{(n_i+n_j{)}^2}$	$0$
Median	$\frac{1}{2}$	$\frac{1}{2}$	$-\frac{1}{4}$	$0$