4.9. Eigen Values

In this subsection, we discuss the eigenvalues and eigenvectors of square matrices. Much of the discussion in this section will be equally applicable to real as well as complex square matrices. We will use the complex notation mostly and make specific remarks for real matrices wherever needed. Of specific interest is the eigenvalue decomposition of real symmetric matrices.

4.9.1. Eigen Values

Definition 4.112 (Eigen value)

A scalar $\lambda$ is an eigen value of an $n\times n$ matrix $\mathbf{A}=[a_{ij}]$ if there exists a nonzero vector $\mathbf{x}$ such that

(4.9)$$\mathbf{A}\mathbf{x}=\lambda \mathbf{x}.$$

A nonzero vector $\mathbf{x}$ which satisfies this equation is called an eigen vector of $\mathbf{A}$ for the eigen value $\lambda$.

An eigen value is also known as a characteristic value, proper value or a latent value.

We note that (4.9) can be written as

$$\mathbf{A}\mathbf{x}=\lambda {\mathbf{I}}_n\mathbf{x}⟺(\mathbf{A}-\lambda {\mathbf{I}}_n)\mathbf{x}=\mathbf{0}.$$

Thus $\lambda$ is an eigen value of $\mathbf{A}$ if and only if the matrix $\mathbf{A}-\lambda \mathbf{I}$ is singular.

Definition 4.113 (Spectrum of eigen values)

The set comprising of eigen values of a matrix $\mathbf{A}$ is known as its spectrum.

Remark 4.13 (Uniqueness of eigen value for an eigen vector)

For each eigen vector $\mathbf{x}$ for a matrix $\mathbf{A}$, the corresponding eigen value $\lambda$ is unique.

Proof. We show this by contradiction.

1. Assume that for $\mathbf{x}$ there are two eigen values ${\lambda }_1$ and ${\lambda }_2$.

2. Then

   $$\mathbf{A}\mathbf{x}={\lambda }_1\mathbf{x}={\lambda }_2\mathbf{x}⟹({\lambda }_1-{\lambda }_2)\mathbf{x}=\mathbf{0}.$$

3. This can happen only when either $\mathbf{x}=\mathbf{0}$ or ${\lambda }_1={\lambda }_2$.

4. Since $\mathbf{x}$ is an eigen vector, it cannot be $\mathbf{0}$.

5. Thus ${\lambda }_1={\lambda }_2$.

Remark 4.14

If $\mathbf{x}$ is an eigen vector for $\mathbf{A}$, then the corresponding eigen value is given by

$$\lambda =\frac{{\mathbf{x}}^H\mathbf{A}\mathbf{x}}{{\mathbf{x}}^H\mathbf{x}}.$$

Proof. We proceed as follows:

$$\mathbf{A}\mathbf{x}=\lambda \mathbf{x}⟹{\mathbf{x}}^H\mathbf{A}\mathbf{x}=\lambda {\mathbf{x}}^H\mathbf{x}⟹\lambda =\frac{{\mathbf{x}}^H\mathbf{A}\mathbf{x}}{{\mathbf{x}}^H\mathbf{x}}.$$

since $\mathbf{x}$ is nonzero.

Remark 4.15

An eigen vector $\mathbf{x}$ of $\mathbf{A}$ for eigen value $\lambda$ belongs to the nullspace of $\mathbf{A}-\lambda \mathbf{I}$; i.e.,

$$\mathbf{x}\in \mathcal{N}(\mathbf{A}-\lambda \mathbf{I}).$$

In other words $\mathbf{x}$ is a nontrivial solution to the homogeneous system of linear equations given by

$$(\mathbf{A}-\lambda \mathbf{I})\mathbf{z}=\mathbf{0}.$$

We can put together eigen vectors of a matrix into another matrix by itself. This can be very useful tool. We start with a simple idea.

Lemma 4.50 (Matrix of eigen vectors)

Let $\mathbf{A}$ be an $n\times n$ matrix. Let ${\mathbf{u}}_1,{\mathbf{u}}_2,\dots ,{\mathbf{u}}_r$ be $r$ nonzero vectors from ${\mathbb{F}}^n$. Let us construct an $n\times r$ matrix

$$\mathbf{U}=\left[\begin{array}{cccc}{\mathbf{u}}_1& {\mathbf{u}}_2& \dots & {\mathbf{u}}_r\end{array}\right].$$

Then all the $r$ vectors are eigen vectors of $\mathbf{A}$ if and only if there exists a diagonal matrix $\mathbf{D}=\mathrm{diag}(d_1,\dots ,d_r)$ such that

$$\mathbf{A}\mathbf{U}=\mathbf{U}\mathbf{D}.$$

Proof. Expanding the equation, we can write

$$\left[\begin{array}{cccc}\mathbf{A}{\mathbf{u}}_1& \mathbf{A}{\mathbf{u}}_2& \dots & \mathbf{A}{\mathbf{u}}_r\end{array}\right]=\left[\begin{array}{cccc}{d}_1{\mathbf{u}}_1& d_2{\mathbf{u}}_2& \dots & d_r{\mathbf{u}}_r\end{array}\right].$$

Clearly we want

$$\mathbf{A}{\mathbf{u}}_i=d_i{\mathbf{u}}_i$$

where ${\mathbf{u}}_i$ are nonzero. This is possible only when $d_i$ is an eigen value of $\mathbf{A}$ and ${\mathbf{u}}_i$ is an eigen vector for $d_i$.

Converse: Assume that ${\mathbf{u}}_i$ are eigen vectors. Choose $d_i$ to be corresponding eigen values. Then the equation holds.

This idea will be extended further in building the eigen value decomposition of matrices.

Observation 4.5

Let $\mathbf{A}\in {\mathbb{R}}^{n\times n}$. Let $\lambda$ be a nonzero real eigen value of $\mathbf{A}$ and $\mathbf{x}\in {\mathbb{R}}^n$ be an eigenvector of $\mathbf{A}$ for $\lambda$. Then $\mathbf{A}\mathbf{x}$ and $\mathbf{x}$ are collinear.

In other words the angle between $\mathbf{A}\mathbf{x}$ and $\mathbf{x}$ is either $0^{\circ }$ when $\lambda$ is positive and is ${180}^{\circ }$ when $\lambda$ is negative. Let us look at the inner product:

$$⟨\mathbf{A}\mathbf{x},\mathbf{x}⟩={\mathbf{x}}^T\mathbf{A}\mathbf{x}={\mathbf{x}}^H\lambda \mathbf{x}=\lambda \Vert \mathbf{x}{\Vert }_2^2.$$

Meanwhile

$$\Vert \mathbf{A}\mathbf{x}{\Vert }_2=\Vert \lambda \mathbf{x}{\Vert }_2=|\lambda |\Vert \mathbf{x}{\Vert }_2.$$

Thus

$$|⟨\mathbf{A}\mathbf{x},\mathbf{x}⟩|=\Vert \mathbf{A}\mathbf{x}{\Vert }_2\Vert \mathbf{x}{\Vert }_2.$$

The angle $\theta$ between $\mathbf{A}\mathbf{x}$ and $\mathbf{x}$ is given by

$$\cos \theta =\frac{⟨\mathbf{A}\mathbf{x},\mathbf{x}⟩}{\Vert \mathbf{A}\mathbf{x}{\Vert }_2\Vert \mathbf{x}{\Vert }_2}=\frac{\lambda \Vert \mathbf{x}{\Vert }_2^2}{|\lambda |\Vert \mathbf{x}{\Vert }_2^2}=\pm 1.$$

4.9.2. Eigen Space

Definition 4.114 (Eigen space)

Let $\lambda$ be an eigen value for a square matrix $\mathbf{A}$. Then its eigen space is the nullspace of $\mathbf{A}-\lambda \mathbf{I}$ i.e. $\mathcal{N}(\mathbf{A}-\lambda \mathbf{I})$.

Remark 4.16 (Eigen vectors for an eigen value)

The set comprising all the eigen vectors of $\mathbf{A}$ for an eigen value $\lambda$ is given by

$$\mathcal{N}(\mathbf{A}-\lambda \mathbf{I})\setminus \{\mathbf{0}\}$$

since $\mathbf{0}$ cannot be an eigen vector.

Definition 4.115 (Geometric multiplicity)

Let $\lambda$ be an eigen value for a square matrix $\mathbf{A}$. The dimension of its eigen space $\mathcal{N}(\mathbf{A}-\lambda \mathbf{I})$ is known as the geometric multiplicity of the eigen value $\lambda$.

We can see that

(4.10)$$\dim (\mathcal{N}(\mathbf{A}-\lambda \mathbf{I}))=n-\mathrm{rank}(\mathbf{A}-\lambda \mathbf{I}).$$

4.9.3. Characteristic Polynomial

Remark 4.17 (Singularity of $\mathbf{A}-\lambda \mathbf{I}$)

A scalar $\lambda$ can be an eigen value of a square matrix $\mathbf{A}$ if and only if

$$det(\mathbf{A}-\lambda \mathbf{I})=0.$$

$det(\mathbf{A}-\lambda \mathbf{I})$ is a polynomial in $\lambda$ of degree $n$.

Observation 4.6 (Eigenvalues as zeros of a polynomial)

We can write the determinant as a polynomial:

$$det(\mathbf{A}-\lambda \mathbf{I})=p(\lambda )={\alpha }^n{\lambda }^n+{\alpha }^{n-1}{\lambda }^{n-1}+\cdots +{\alpha }^1\lambda +{\alpha }_0$$

where ${\alpha }_i$ depend on entries in $\mathbf{A}$. In this sense, an eigenvalue of $\mathbf{A}$ is a root of the equation

$$p(\lambda )=0.$$

It is easy to show that ${\alpha }^n=(-1{)}^n$.

Definition 4.116 (Characteristic polynomial)

For any square matrix $\mathbf{A}$, the polynomial given by

$$p(\lambda )=det(\mathbf{A}-\lambda \mathbf{I})$$

is known as its characteristic polynomial. The equation give by

$$p(\lambda )=0$$

is known as its characteristic equation. The eigen values of $\mathbf{A}$ are the roots of its characteristic polynomial or solutions of its characteristic equation.

Observation 4.7 (Factors of characteristic polynomial)

For real square matrices, if we restrict eigen values to real values, then the characteristic polynomial can be factored as

(4.11)$$p(\lambda )=(-1{)}^n(\lambda -{\lambda }_1{)}^{r_1}\dots (\lambda -{\lambda }_k{)}^{r_k}q(\lambda ).$$

The polynomial has $k$ distinct real roots. For each root ${\lambda }_i$, $r_i$ is a positive integer indicating how many times the root appears. $q(\lambda )$ is a polynomial that has no real roots. The following is true

$$r_1+\cdots +r_k+\mathrm{deg}(q(\lambda ))=n.$$

Clearly $k\le n$.

For complex square matrices where eigen values can be complex (including real square matrices), the characteristic polynomial can be factored as

(4.12)$$p(\lambda )=(-1{)}^n(\lambda -{\lambda }_1{)}^{r_1}\dots (\lambda -{\lambda }_k{)}^{r_k}.$$

The polynomial can be completely factorized into first degree polynomials. There are $k$ distinct roots or eigen values. The following is true

$$r_1+\cdots +r_k=n.$$

Thus, including the duplicates there are exactly $n$ eigen values for a complex square matrix.

It is quite possible that a real square matrix doesn’t have any real eigen values.

Definition 4.117 (Algebraic multiplicity)

The number of times an eigen value appears in the factorization of the characteristic polynomial of a square matrix $\mathbf{A}$ is known as its algebraic multiplicity. In other words, $r_i$ is the algebraic multiplicity for ${\lambda }_i$ in the factorization (4.11) and (4.12).

Theorem 4.119 (Geometric and algebraic multiplicities)

Let $\lambda$ be an eigen value of a square matrix $\mathbf{A}$. Then the geometric multiplicity of $\lambda$ is less than or equal to its algebraic multiplicity.

Corollary 4.16 (Distinct eigen values and multiplicity)

If an $n\times n$ matrix $\mathbf{A}$ has $n$ distinct eigen values, then each of them has a geometric (and algebraic) multiplicity of $1$.

Proof. The algebraic multiplicity of an eigen value is greater than or equal to 1. But the sum cannot exceed $n$. Since there are $n$ distinct eigen values, thus each of them has algebraic multiplicity of $1$. Further, geometric multiplicity of an eigen value is greater than equal to 1 and less than equal to its algebraic multiplicity.

Corollary 4.17

Let an $n\times n$ matrix $\mathbf{A}$ has $k$ distinct eigen values ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_k$ with algebraic multiplicities $r_1,r_2,\dots ,r_k$ and geometric multiplicities $g_1,g_2,\dots g_k$ respectively. Then

$$\sum _{i=1}^k{g}_k\le \sum _{i=1}^k{r}_k\le n.$$

Moreover if

$$\sum _{i=1}^k{g}_i=\sum _{i=1}^k{r}_i$$

then

$$g_i=r_i\mathrm{\forall }i=1,\dots ,k.$$

Remark 4.18 (Spectrum from roots of characteristic polynomial)

In the factorization (4.12) of the characteristic polynomial $p(\lambda )$, the set $\{{\lambda }_1,\dots ,{\lambda }_k\}$ forms the spectrum of $\mathbf{A}$.

Let us consider the sum of $r_i$ which gives the count of total number of roots of $p(\lambda )$.

$$m=\sum _{i=1}^k{r}_i.$$

With this there are $m$ not-necessarily distinct roots of $p(\lambda )$. Let us write $p(\lambda )$ from (4.11) as

$$p(\lambda )=(-1{)}^n(\lambda -c_1)(\lambda -c_2)\dots (\lambda -c_m)q(\lambda ).$$

where $c_1,c_2,\dots ,c_m$ are $m$ scalars (not necessarily distinct) of which $r_1$ scalars are ${\lambda }_1$, $r_2$ are ${\lambda }_2$ and so on. For the complex case, we have $q(\lambda )=1$.

We will refer to the set (allowing repetitions) $\{c_1,c_2,\dots ,c_m\}$ as the eigen values of the matrix $\mathbf{A}$ where $c_i$ are not necessarily distinct. In contrast the spectrum of $\mathbf{A}$ refers to the set of distinct eigen values of $\mathbf{A}$. The symbol $c$ has been chosen based on the other name for eigen values (the characteristic values).

4.9.4. The Zero Eigen Value

Lemma 4.51 ($0$ as an eigen value)

$0$ is an eigen value of a square matrix $\mathbf{A}$ if and only if $\mathbf{A}$ is singular.

Proof. Let $0$ be an eigen value of $\mathbf{A}$.

1. Then there exists $\mathbf{u}\ne \mathbf{0}$ such that

   $$\mathbf{A}\mathbf{u}=0\mathbf{u}=\mathbf{0}.$$

2. Thus $\mathbf{u}$ is a non-trivial solution of the homogeneous linear system $\mathbf{A}\mathbf{x}=\mathbf{0}$.

3. Thus $\mathbf{A}$ is singular.

Converse:

1. Assume that $\mathbf{A}$ is singular.

2. Then there exists $\mathbf{u}\ne \mathbf{0}$ s.t.

   $$\mathbf{A}\mathbf{u}=\mathbf{0}=0\mathbf{u}.$$

3. Thus $0$ is an eigen value of $\mathbf{A}$.

Lemma 4.52 (Eigenspace of the zero eigen value)

If a square matrix $\mathbf{A}$ is singular, then $\mathcal{N}(\mathbf{A})$ is the eigen space for the eigen value $\lambda =0$.

Proof. This is straight forward from the definition of eigen space (see Definition 4.114).

Remark 4.19

Clearly the geometric multiplicity of $\lambda =0$ equals $\mathrm{nullity}(\mathbf{A})=n-\mathrm{rank}(\mathbf{A})$.

4.9.5. Transpose

Lemma 4.53 (Eigen values of the transpose)

Let $\mathbf{A}$ be a square matrix. Then $\mathbf{A}$ and ${\mathbf{A}}^T$ have same eigen values.

Proof. The eigen values of ${\mathbf{A}}^T$ are given by

$$det({\mathbf{A}}^T-\lambda \mathbf{I})=0.$$

But

$${\mathbf{A}}^T-\lambda \mathbf{I}={\mathbf{A}}^T-(\lambda \mathbf{I}{)}^T=(\mathbf{A}-\lambda \mathbf{I}{)}^T.$$

Hence (using Lemma 4.33)

$$det({\mathbf{A}}^T-\lambda \mathbf{I})=det((\mathbf{A}-\lambda \mathbf{I}{)}^T)=det(\mathbf{A}-\lambda \mathbf{I}).$$

Thus the characteristic polynomials of $\mathbf{A}$ and ${\mathbf{A}}^T$ are same. Hence the eigen values are same. In other words the spectrum of $\mathbf{A}$ and ${\mathbf{A}}^T$ are same.

4.9.6. Powers and Inverses

Lemma 4.54 (Power rule)

Let $\mathbf{A}$ be a square matrix and $\lambda$ be an eigen value of $\mathbf{A}$. Let $p\in \mathbb{N}$. Then ${\lambda }^p$ is an eigen value of ${\mathbf{A}}^p$.

Proof. We prove this by mathematical induction.

1. For $p=1$ the statement holds trivially since ${\lambda }^1$ is an eigen value of ${\mathbf{A}}^1$.

2. Assume that the statement holds for some value of $p$.

3. Thus let ${\lambda }^p$ be an eigen value of ${\mathbf{A}}^p$ and let $\mathbf{u}$ be corresponding eigen vector.

4. Now

   $${\mathbf{A}}^{p+1}\mathbf{u}={\mathbf{A}}^p(\mathbf{A}\mathbf{u})={\mathbf{A}}^p\lambda \mathbf{u}=\lambda {\mathbf{A}}^p\mathbf{u}=\lambda {\lambda }^p\mathbf{u}={\lambda }^{p+1}\mathbf{u}.$$

5. Thus ${\lambda }^{p+1}$ is an eigen value for ${\mathbf{A}}^{p+1}$ with the same eigen vector $\mathbf{u}$.

6. With the principle of mathematical induction, the proof is complete.

Lemma 4.55 (Eigenvalue of the inverse)

Let a square matrix $\mathbf{A}$ be non singular and let $\lambda \ne 0$ be some eigen value of $\mathbf{A}$. Then ${\lambda }^{-1}$ is an eigen value of ${\mathbf{A}}^{-1}$. Moreover, all eigen values of ${\mathbf{A}}^{-1}$ are obtained by taking inverses of eigen values of $\mathbf{A}$; i.e., if $\mu \ne 0$ is an eigen value of ${\mathbf{A}}^{-1}$ then $\frac{1}{\mu }$ is an eigen value of $\mathbf{A}$ also. Also, $\mathbf{A}$ and ${\mathbf{A}}^{-1}$ share the same set of eigen vectors.

Proof. Since $\mathbf{A}$ is invertible, hence all its eigenvalues are nonzero.

1. Let $\mathbf{u}\ne \mathbf{0}$ be an eigen vector of $\mathbf{A}$ for the eigen value $\lambda$.

2. Then

   $$\mathbf{A}\mathbf{u}=\lambda \mathbf{u}⟹\mathbf{u}={\mathbf{A}}^{-1}\lambda \mathbf{u}⟹\frac{1}{\lambda }\mathbf{u}={\mathbf{A}}^{-1}\mathbf{u}.$$

3. Thus $\mathbf{u}$ is also an eigen vector of ${\mathbf{A}}^{-1}$ for the eigen value $\frac{1}{\lambda }$.

4. Now let $\mathbf{B}={\mathbf{A}}^{-1}$.

5. Then ${\mathbf{B}}^{-1}=\mathbf{A}$.

6. Thus if $\mu$ is an eigen value of $\mathbf{B}$ then $\frac{1}{\mu }$ is an eigen value of ${\mathbf{B}}^{-1}=\mathbf{A}$.

7. Thus if $\mathbf{A}$ is invertible then eigen values of $\mathbf{A}$ and ${\mathbf{A}}^{-1}$ have one to one correspondence (being reciprocal).

4.9.7. Invariant Subspaces

Definition 4.118 (Invariant subspace)

Let $\mathbf{A}$ be a square $n\times n$ matrix and let $\mathbb{W}$ be a linear subspace of ${\mathbb{F}}^n$. Then $\mathbb{W}$ is invariant relative to $\mathbf{A}$ if

$$\mathbf{A}\mathbf{w}\in \mathbb{W}\mathrm{\forall }\mathbf{w}\in \mathbb{W}.$$

In other words, $\mathbf{A}(\mathbb{W})\subseteq \mathbb{W}$ or for every vector $\mathbf{w}\in \mathbb{W}$ its mapping $\mathbf{A}\mathbf{w}$ is also in $\mathbb{W}$. Thus action of $\mathbf{A}$ on $\mathbb{W}$ doesn’t take us outside of $\mathbb{W}$.

We also say that $\mathbb{W}$ is $\mathbf{A}$-invariant.

Eigen vectors are generators of invariant subspaces.

Lemma 4.56 (Spans of eigenvectors are invariant)

Let $\mathbf{A}$ be an $n\times n$ matrix. Let ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_r$ be $r$ eigen vectors of $\mathbf{A}$. Let us construct an $n\times r$ matrix

$$\mathbf{X}=\left[\begin{array}{cccc}{\mathbf{x}}_1& {\mathbf{x}}_2& \dots & {\mathbf{x}}_r\end{array}\right].$$

Then the column space of $\mathbf{X}$; i.e., $\mathcal{C}(\mathbf{X})$ is invariant relative to $\mathbf{A}$.

Proof. Let us assume that $c_1,c_2,\dots ,c_r$ are the eigen values corresponding to ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_r$ (not necessarily distinct).

Let any vector $\mathbf{x}\in \mathcal{C}(\mathbf{X})$ be given by

$$\mathbf{x}=\sum _{i=1}^r{\alpha }_i{\mathbf{x}}_i.$$

Then

$$\mathbf{A}\mathbf{x}=\mathbf{A}\sum _{i=1}^r{\alpha }_i{\mathbf{x}}_i=\sum _{i=1}^r{\alpha }_i\mathbf{A}{\mathbf{x}}_i=\sum _{i=1}^r{\alpha }_i{c}_i{\mathbf{x}}_i.$$

Clearly $\mathbf{A}\mathbf{x}$ is also a linear combination of ${\mathbf{x}}_i$. Hence it belongs to $\mathcal{C}(\mathbf{X})$. Thus $\mathcal{C}(\mathbf{X})$ is invariant relative to $\mathbf{A}$ or $\mathcal{C}(\mathbf{X})$ is $\mathbf{A}$-invariant.

4.9.8. Triangular Matrices

Lemma 4.57 (Eigenvalues of triangular matrices)

Let $\mathbf{A}$ be an $n\times n$ upper or lower triangular matrix. Then its eigen values are the entries on its main diagonal.

Proof. We are given that $\mathbf{A}$ is triangular.

1. Then $\mathbf{A}-\lambda \mathbf{I}$ is also triangular with its diagonal entries being $(a_{ii}-\lambda )$.

2. Using Lemma 4.38, we have

   $$p(\lambda )=det(\mathbf{A}-\lambda \mathbf{I})=\prod _{i=1}^n(a_{ii}-\lambda ).$$

3. Clearly the roots of characteristic polynomial are $a_{ii}$.

Several small results follow from this lemma.

Corollary 4.18 (Eigenvalues of triangular matrices)

Let $\mathbf{A}=[a_{ij}]$ be an $n\times n$ triangular matrix.

• The characteristic polynomial of $\mathbf{A}$ is

  $$p(\lambda )=(-1{)}^n(\lambda -a_{ii}).$$

• A scalar $\lambda$ is an eigen value of $\mathbf{A}$ iff it is one of the diagonal entries of $\mathbf{A}$.

• The algebraic multiplicity of an eigen value $\lambda$ is equal to the number of times it appears on the main diagonal of $\mathbf{A}$.

• The spectrum of $\mathbf{A}$ is given by the distinct entries on the main diagonal of $\mathbf{A}$.

A diagonal matrix is naturally both an upper triangular matrix as well as a lower triangular matrix. Similar results hold for the eigen values of a diagonal matrix also.

Lemma 4.58 (Eigenvalues of diagonal matrices)

Let $\mathbf{A}=[a_{ij}]$ be an $n\times n$ diagonal matrix.

• Its eigen values are the entries on its main diagonal.

• The characteristic polynomial of $\mathbf{A}$ is

  $$p(\lambda )=(-1{)}^n(\lambda -a_{ii}).$$

• A scalar $\lambda$ is an eigen value of $\mathbf{A}$ iff it is one of the diagonal entries of $A$.

• The algebraic multiplicity of an eigen value $\lambda$ is equal to the number of times it appears on the main diagonal of $\mathbf{A}$.

• The spectrum of $\mathbf{A}$ is given by the distinct entries on the main diagonal of $\mathbf{A}$.

There is also a result for the geometric multiplicity of eigen values for a diagonal matrix.

Lemma 4.59 (Geometric multiplicity of eigenvalues for diagonal matrices)

Let $\mathbf{A}=[a_{ij}]$ be an $n\times n$ diagonal matrix. The geometric multiplicity of an eigen value $\lambda$ is equal to the number of times it appears on the main diagonal of $\mathbf{A}$.

Proof. The unit vectors ${\mathbf{e}}_i$ are eigen vectors for $\mathbf{A}$ since

$$\mathbf{A}{\mathbf{e}}_i=a_{ii}{\mathbf{e}}_i.$$

1. They are linearly independent.

2. Thus if a particular eigen value appears $r$ number of times, then there are $r$ linearly independent eigen vectors for the eigen value.

3. Thus its geometric multiplicity is equal to the algebraic multiplicity.

4.9.9. Similar Matrices

Some very useful results are available for similar matrices.

Lemma 4.60 (Characteristic polynomial of similar matrices)

The characteristic polynomial and spectrum of similar matrices is same.

Proof. Let $\mathbf{B}$ be similar to $\mathbf{A}$.

1. Then there exists an invertible matrix $\mathbf{C}$ such that

   $$\mathbf{B}={\mathbf{C}}^{-1}\mathbf{A}\mathbf{C}.$$

2. Now

   $$\begin{array}{rl}\mathbf{B}-\lambda \mathbf{I}& ={\mathbf{C}}^{-1}\mathbf{A}\mathbf{C}-\lambda \mathbf{I}\\ & ={\mathbf{C}}^{-1}\mathbf{A}\mathbf{C}-\lambda {\mathbf{C}}^{-1}\mathbf{C}\\ & ={\mathbf{C}}^{-1}(\mathbf{A}\mathbf{C}-\lambda \mathbf{C})\\ & ={\mathbf{C}}^{-1}(\mathbf{A}-\lambda \mathbf{I})\mathbf{C}.\end{array}$$

3. Thus $\mathbf{B}-\lambda \mathbf{I}$ is similar to $\mathbf{A}-\lambda \mathbf{I}$.

4. Hence due to Lemma 4.40, their determinant is equal.

5. In other words,

   $$det(\mathbf{B}-\lambda \mathbf{I})=det(\mathbf{A}-\lambda \mathbf{I}).$$

6. This means that the characteristic polynomials of $\mathbf{A}$ and $\mathbf{B}$ are same.

7. Since eigen values are nothing but roots of the characteristic polynomial, hence they are same too.

8. This means that the spectrum (the set of distinct eigen values) is same.

This result is very useful. Since if it can be shown that a matrix $\mathbf{A}$ is similar to a diagonal or a triangular matrix whose eigen values are easy to obtain then determination of the eigen values of $\mathbf{A}$ becomes straight forward.

Corollary 4.19 (Eigenvalues of similar matrices)

If $\mathbf{A}$ and $\mathbf{B}$ are similar to each other then

• An eigen value has same algebraic and geometric multiplicity for both $\mathbf{A}$ and $\mathbf{B}$.

• The (not necessarily distinct) eigen values of $\mathbf{A}$ and $\mathbf{B}$ are same.

Although the eigen values are same, but the eigen vectors are different.

Lemma 4.61 (Eigenvectors of similar matrices)

Let $\mathbf{A}$ and $\mathbf{B}$ be similar with

$$\mathbf{B}={\mathbf{C}}^{-1}\mathbf{A}\mathbf{C}$$

for some invertible matrix $\mathbf{C}$. If $\mathbf{u}$ is an eigenvector of $\mathbf{A}$ for an eigenvalue $\lambda$, then $C^{-1}\mathbf{u}$ is an eigen vector of $\mathbf{B}$ for the same eigen value.

Proof. We are given that $\mathbf{u}$ is an eigen vector of $\mathbf{A}$ for an eigen value $\lambda$.

1. Thus we have

   $$\mathbf{A}\mathbf{u}=\lambda \mathbf{u}.$$

2. Thus

   $$\mathbf{B}{\mathbf{C}}^{-1}\mathbf{u}={\mathbf{C}}^{-1}\mathbf{A}\mathbf{C}{\mathbf{C}}^{-1}\mathbf{u}={\mathbf{C}}^{-1}\mathbf{A}\mathbf{u}={\mathbf{C}}^{-1}\lambda \mathbf{u}=\lambda {\mathbf{C}}^{-1}\mathbf{u}.$$

3. Now $\mathbf{u}\ne \mathbf{0}$ and ${\mathbf{C}}^{-1}$ is non singular.

4. Thus ${\mathbf{C}}^{-1}\mathbf{u}\ne \mathbf{0}$.

5. Thus ${\mathbf{C}}^{-1}\mathbf{u}$ is an eigen vector of $\mathbf{B}$.

4.9.10. Linear Independence of Eigenvectors

Theorem 4.120 (Linear independence of eigenvectors with distinct eigenvalues)

Let $\mathbf{A}$ be an $n\times n$ square matrix. Let ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_k$ be any $k$ eigen vectors of $A$ for distinct eigen values ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_k$ respectively. Then ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_k$ are linearly independent.

Proof. We first prove the simpler case with 2 eigen vectors ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$ and corresponding eigen values ${\lambda }_1$ and ${\lambda }_2$ respectively.

1. Let there be a linear relationship between ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$ given by

   $${\alpha }_1{\mathbf{x}}_1+{\alpha }_2{\mathbf{x}}_2=\mathbf{0}.$$

2. Multiplying both sides with $(\mathbf{A}-{\lambda }_1\mathbf{I})$ we get

   $$\begin{array}{rl}& {\alpha }_1(\mathbf{A}-{\lambda }_1\mathbf{I}){\mathbf{x}}_1+{\alpha }_2(\mathbf{A}-{\lambda }_1\mathbf{I}){\mathbf{x}}_2=\mathbf{0}\\ ⟹& {\alpha }_1({\lambda }_1-{\lambda }_1){\mathbf{x}}_1+{\alpha }_2({\lambda }_2-{\lambda }_1){\mathbf{x}}_2=\mathbf{0}\\ ⟹& {\alpha }_2({\lambda }_2-{\lambda }_1){\mathbf{x}}_2=\mathbf{0}.\end{array}$$

3. Since ${\lambda }_1\ne {\lambda }_2$ and ${\mathbf{x}}_2\ne \mathbf{0}$, hence ${\alpha }_2=0$.

4. Similarly by multiplying with $(\mathbf{A}-{\lambda }_2\mathbf{I})$ on both sides, we can show that ${\alpha }_1=0$.

5. Thus ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$ are linearly independent.

6. Now for the general case, consider a linear relationship between ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_k$ given by

   $${\alpha }_1{\mathbf{x}}_1+{\alpha }_2{\mathbf{x}}_2+\dots {\alpha }_k{\mathbf{x}}_k=\mathbf{0}.$$

7. Multiplying by $\prod _{i\ne j,i=1}^k(\mathbf{A}-{\lambda }_i\mathbf{I})$ and using the fact that ${\lambda }_i\ne {\lambda }_j$ if $i\ne j$, we get ${\alpha }_j=0$.

8. Thus the only linear relationship is the trivial relationship.

9. This completes the proof.

For eigen values with geometric multiplicity greater than $1$ there are multiple eigenvectors corresponding to the eigen value which are linearly independent. In this context, above theorem can be generalized further.

Theorem 4.121 (Linear independence of eigenvectors from different eigen spaces)

Let ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_k$ be $k$ distinct eigen values of $\mathbf{A}$. Let $\{{\mathbf{x}}_1^j,{\mathbf{x}}_2^j,\dots {\mathbf{x}}_{g_j}^j\}$ be any ${\mathbf{g}}_j$ linearly independent eigen vectors from the eigen space of ${\lambda }_j$ where $g_j$ is the geometric multiplicity of ${\lambda }_j$. Then the combined set of eigen vectors given by

$$\{{\mathbf{x}}_1^1,\dots ,{\mathbf{x}}_{g_1}^1,\dots ,{\mathbf{x}}_1^k,\dots ,{\mathbf{x}}_{g_k}^k\}$$

consisting of $\sum _{j=1}^k{g}_j$ eigen vectors is linearly independent.

This result puts an upper limit on the number of linearly independent eigen vectors of a square matrix.

Lemma 4.62 (Maximum number of linearly independent eigenvectors)

Let $\{{\lambda }_1,\dots ,{\lambda }_k\}$ represents the spectrum of an $n\times n$ matrix $\mathbf{A}$. Let $g_1,\dots ,g_k$ be the geometric multiplicities of ${\lambda }_1,\dots {\lambda }_k$ respectively. Then the number of linearly independent eigen vectors for $\mathbf{A}$ is

$$\sum _{i=1}^k{g}_i.$$

Moreover if

$$\sum _{i=1}^k{g}_i=n$$

then a set of $n$ linearly independent eigen vectors of $\mathbf{A}$ can be found which forms a basis for ${\mathbb{F}}^n$.

4.9.11. Diagonalization

Diagonalization is one of the fundamental operations in linear algebra. This section discusses diagonalization of square matrices in depth.

Definition 4.119 (diagonalizable matrix)

An $n\times n$ matrix $\mathbf{A}$ is said to be diagonalizable if it is similar to a diagonal matrix. In other words there exists an $n\times n$ non-singular matrix $\mathbf{P}$ such that $\mathbf{D}={\mathbf{P}}^{-1}\mathbf{A}\mathbf{P}$ is a diagonal matrix. If this happens then we say that $\mathbf{P}$ diagonalizes $\mathbf{A}$ or $\mathbf{A}$ is diagonalized by $\mathbf{P}$.

Remark 4.20

$$\mathbf{D}={\mathbf{P}}^{-1}\mathbf{A}\mathbf{P}⟺\mathbf{P}\mathbf{D}=\mathbf{A}\mathbf{P}⟺\mathbf{P}\mathbf{D}{\mathbf{P}}^{-1}=\mathbf{A}.$$

We note that if we restrict to real matrices, then $\mathbf{U}$ and $\mathbf{D}$ should also be real. If $\mathbf{A}\in {\mathbb{C}}^{n\times n}$ (it may still be real) then $\mathbf{P}$ and $\mathbf{D}$ can be complex.

In other words, if $\mathbf{A}\in {\mathbb{R}}^{n\times n}$ is diagonalizable then both $\mathbf{P}$ and $\mathbf{D}$ must be real.

The next theorem is the culmination of a variety of results studied so far.

Theorem 4.122 (Properties of diagonalizable matrices)

Let $\mathbf{A}$ be a diagonalizable matrix with $\mathbf{D}={\mathbf{P}}^{-1}\mathbf{A}\mathbf{P}$ being its diagonalization. Let $\mathbf{D}=\mathrm{diag}(d_1,d_2,\dots ,d_n)$. Then the following hold

• $\mathrm{rank}(\mathbf{A})=\mathrm{rank}(\mathbf{D})$ which equals the number of nonzero entries on the main diagonal of $\mathbf{D}$.

• $det(\mathbf{A})=d_1{d}_2\dots d_n$.

• $\mathrm{tr}(\mathbf{A})=d_1+d_2+\dots d_n$.

• The characteristic polynomial of $\mathbf{A}$ is

  $$p(\lambda )=(-1{)}^n(\lambda -d_1)(\lambda -d_2)\dots (\lambda -d_n).$$

• The spectrum of $\mathbf{A}$ comprises the distinct scalars on the diagonal entries in $\mathbf{D}$.

• The (not necessarily distinct) eigenvalues of $\mathbf{A}$ are the diagonal elements of $\mathbf{D}$.

• The columns of $\mathbf{P}$ are (linearly independent) eigenvectors of $\mathbf{A}$.

• The algebraic and geometric multiplicities of an eigenvalue $\lambda$ of $\mathbf{A}$ equal the number of diagonal elements of $\mathbf{D}$ that equal $\lambda$.

Proof. From Definition 4.119 we note that $\mathbf{D}$ and $\mathbf{A}$ are similar.

1. Due to Lemma 4.40

   $$det(\mathbf{A})=det(\mathbf{D}).$$

2. Due to Lemma 4.39

   $$det(\mathbf{D})=\prod _{i=1}^n{d}_i.$$

3. Now due to Lemma 4.31

   $$\mathrm{tr}(\mathbf{A})=\mathrm{tr}(\mathbf{D})=\sum _{i=1}^n{d}_i.$$

4. Further due to Lemma 4.60 the characteristic polynomial and spectrum of $\mathbf{A}$ and $\mathbf{D}$ are same.

5. Due to Lemma 4.58 the eigen values of $\mathbf{D}$ are nothing but its diagonal entries.

6. Hence they are also the eigen values of $\mathbf{A}$.

7. We recall that

   $$\mathbf{D}={\mathbf{P}}^{-1}\mathbf{A}\mathbf{P}⟹\mathbf{A}\mathbf{P}=\mathbf{P}\mathbf{D}.$$

8. Now writing

   $$\mathbf{P}=\left[\begin{array}{cccc}{\mathbf{p}}_1& {\mathbf{p}}_2& \dots & {\mathbf{p}}_n\end{array}\right]$$

   we have

   $$\mathbf{A}\mathbf{P}=\left[\begin{array}{cccc}\mathbf{A}{\mathbf{p}}_1& \mathbf{A}{\mathbf{p}}_2& \dots & \mathbf{A}{\mathbf{p}}_n\end{array}\right]=\mathbf{P}\mathbf{D}=\left[\begin{array}{cccc}{d}_1{\mathbf{p}}_1& d_2{\mathbf{p}}_2& \dots & d_n{\mathbf{p}}_n\end{array}\right].$$

9. Matching columns, we see that

   $$\mathbf{A}{\mathbf{p}}_i=d_i{\mathbf{p}}_i\mathrm{\forall }i=1,\dots ,n.$$

10. Thus ${\mathbf{p}}_i$ are eigen vectors of $\mathbf{A}$.

11. Since $\mathbf{P}$ is invertible, hence its columns are linearly independent.

12. Hence all the eigenvectors of $\mathbf{A}$ are linearly independent.

13. Since the characteristic polynomials of $\mathbf{A}$ and $\mathbf{D}$ are same, hence the algebraic multiplicities of eigen values are same.

14. From Lemma 4.61 we get that there is a one to one correspondence between the eigen vectors of $\mathbf{A}$ and $\mathbf{D}$ through the change of basis given by $\mathbf{P}$.

15. Thus the linear independence relationships between the eigen vectors remain the same.

16. Hence the geometric multiplicities of individual eigenvalues are also the same.

17. This completes the proof.

So far we have verified various results which are available if a matrix $\mathbf{A}$ is diagonalizable. We haven’t yet identified the conditions under which $\mathbf{A}$ is diagonalizable. We note that not every matrix is diagonalizable. The following theorem gives necessary and sufficient conditions under which a matrix is diagonalizable.

Theorem 4.123 (Diagonalizability: necessary and sufficient conditions)

An $n\times n$ matrix $\mathbf{A}$ is diagonalizable by an $n\times n$ nonsingular matrix $\mathbf{P}$ if and only if the columns of $\mathbf{P}$ are (linearly independent) eigenvectors of $\mathbf{A}$.

Proof. We note that since $\mathbf{P}$ is nonsingular hence columns of $\mathbf{P}$ have to be linearly independent.

The necessary condition part was proven in Theorem 4.122. We now show that if $\mathbf{P}$ consists of $n$ linearly independent eigen vectors of $\mathbf{A}$ then $\mathbf{A}$ is diagonalizable.

1. Let the columns of $\mathbf{P}$ be ${\mathbf{p}}_1,{\mathbf{p}}_2,\dots ,{\mathbf{p}}_n$ and corresponding (not necessarily distinct) eigen values be $d_1,d_2,\dots ,d_n$.

2. Then

   $$\mathbf{A}{\mathbf{p}}_i=d_i{\mathbf{p}}_i.$$

3. Thus by letting $\mathbf{D}=\mathrm{diag}(d_1,d_2,\dots ,d_n)$, we have

   $$\mathbf{A}\mathbf{P}=\mathbf{P}\mathbf{D}.$$

4. Since columns of $\mathbf{P}$ are linearly independent, hence $\mathbf{P}$ is invertible.

5. This gives us

   $$\mathbf{D}={\mathbf{P}}^{-1}\mathbf{A}\mathbf{P}.$$

6. Thus $\mathbf{A}$ is similar to a diagonal matrix $\mathbf{D}$.

7. Hence $\mathbf{A}$ is diagonalizable.

8. This validates the sufficient condition.

A corollary follows.

Corollary 4.20 (Diagonalizability and linear independence of eigen vectors)

An $n\times n$ matrix is diagonalizable if and only if there exists a linearly independent set of $n$ eigenvectors of $\mathbf{A}$.

Now we know that geometric multiplicities of eigen values of $\mathbf{A}$ provide us information about the number of linearly independent eigenvectors of $\mathbf{A}$.

Corollary 4.21 (Diagonalizability and geometric multiplicities)

Let $\mathbf{A}$ be an $n\times n$ matrix. Let ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_k$ be its $k$ distinct eigen values (comprising its spectrum). Let $g_j$ be the geometric multiplicity of ${\lambda }_j$. Then $\mathbf{A}$ is diagonalizable if and only if

$$\sum _{i=1}^n{g}_i=n.$$

4.9.12. Real Symmetric Matrices

This subsection is focused on real symmetric matrices.

Following is a fundamental property of real symmetric matrices.

Theorem 4.124 (Existence of eigenvalues)

Every real symmetric matrix has an eigen value.

The proof of this result is beyond the scope of this book.

Lemma 4.63 (Orthogonality of eigenvectors for distinct eigenvalues)

Let $\mathbf{A}$ be an $n\times n$ real symmetric matrix. Let ${\lambda }_1$ and ${\lambda }_2$ be any two distinct eigen values of $\mathbf{A}$ and let ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$ be any two corresponding eigen vectors. Then ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$ are orthogonal.

Proof. By definition we have $\mathbf{A}{\mathbf{x}}_1={\lambda }_1{\mathbf{x}}_1$ and $\mathbf{A}{\mathbf{x}}_2={\lambda }_2{\mathbf{x}}_2$. Thus

$$\begin{array}{rl}& {\mathbf{x}}_2^T\mathbf{A}{\mathbf{x}}_1={\lambda }_1{\mathbf{x}}_2^T{\mathbf{x}}_1\\ ⟹& {\mathbf{x}}_1^T{\mathbf{A}}^T{\mathbf{x}}_2={\lambda }_1{\mathbf{x}}_1^T{\mathbf{x}}_2\\ ⟹& {\mathbf{x}}_1^T\mathbf{A}{\mathbf{x}}_2={\lambda }_1{\mathbf{x}}_1^T{\mathbf{x}}_2\\ ⟹& {\mathbf{x}}_1^T{\lambda }_2{\mathbf{x}}_2={\lambda }_1{\mathbf{x}}_1^T{\mathbf{x}}_2\\ ⟹& ({\lambda }_1-{\lambda }_2){\mathbf{x}}_1^T{\mathbf{x}}_2=0\\ ⟹& {\mathbf{x}}_1^T{\mathbf{x}}_2=0.\end{array}$$

Thus ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$ are orthogonal. In between we took transpose on both sides, used the fact that $\mathbf{A}={\mathbf{A}}^T$ and ${\lambda }_1-{\lambda }_2\ne 0$.

Definition 4.120 (Orthogonally diagonalizable matrix)

A real $n\times n$ matrix $A$ is said to be orthogonally diagonalizable if there exists an orthogonal matrix $\mathbf{U}$ which can diagonalize $\mathbf{A}$; i.e.,

$$\mathbf{D}={\mathbf{U}}^T\mathbf{A}\mathbf{U}$$

is a real diagonal matrix.

Lemma 4.64

Every orthogonally diagonalizable matrix $\mathbf{A}$ is symmetric.

Proof. We have a diagonal matrix $\mathbf{D}$ such that

$$\mathbf{A}=\mathbf{U}\mathbf{D}{\mathbf{U}}^T.$$

Taking transpose on both sides we get

$${\mathbf{A}}^T=\mathbf{U}{\mathbf{D}}^T{\mathbf{U}}^T=\mathbf{U}\mathbf{D}{\mathbf{U}}^T=\mathbf{A}.$$

Thus $\mathbf{A}$ is symmetric.

Theorem 4.125

Every symmetric matrix $\mathbf{A}$ is orthogonally diagonalizable.

We skip the proof of this theorem.

4.9.13. Hermitian Matrices

Following is a fundamental property of Hermitian matrices.

Theorem 4.126 (Existence of eigen values for Hermitian matrices)

Every Hermitian matrix has an eigen value.

The proof of this result is beyond the scope of this book.

Lemma 4.65 (Real eigen values for Hermitian matrices)

The eigenvalues of a Hermitian matrix are real.

Proof. Let $\mathbf{A}$ be a Hermitian matrix and let $\lambda$ be an eigen value of $\mathbf{A}$. Let $\mathbf{u}$ be a corresponding eigen vector. Then

$$\begin{array}{rl}& \mathbf{A}\mathbf{u}=\lambda \mathbf{u}\\ ⟹& {\mathbf{u}}^H{\mathbf{A}}^H={\mathbf{u}}^H\bar{\lambda }\\ ⟹& {\mathbf{u}}^H{\mathbf{A}}^H\mathbf{u}={\mathbf{u}}^H\bar{\lambda }\mathbf{u}\\ ⟹& {\mathbf{u}}^H\mathbf{A}\mathbf{u}=\bar{\lambda }{\mathbf{u}}^H\mathbf{u}\\ ⟹& {\mathbf{u}}^H\lambda \mathbf{u}=\bar{\lambda }{\mathbf{u}}^H\mathbf{u}\\ ⟹& \Vert u{\Vert }_2^2(\lambda -\bar{\lambda })=0\\ ⟹& \lambda =\bar{\lambda }.\end{array}$$

Thus $\lambda$ is real. We used the facts that $\mathbf{A}={\mathbf{A}}^H$ and $\mathbf{u}\ne \mathbf{0}⟹\Vert \mathbf{u}{\Vert }_2\ne 0$.

Lemma 4.66 (Orthogonality of eigenvectors for distinct eigenvalues)

Let $\mathbf{A}$ be an $n\times n$ complex Hermitian matrix. Let ${\lambda }_1$ and ${\lambda }_2$ be any two distinct eigen values of $\mathbf{A}$ and let ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$ be any two corresponding eigen vectors. Then ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$ are orthogonal.

Proof. By definition we have $\mathbf{A}{\mathbf{x}}_1={\lambda }_1{\mathbf{x}}_1$ and $\mathbf{A}{\mathbf{x}}_2={\lambda }_2{\mathbf{x}}_2$. Thus

$$\begin{array}{rl}& {\mathbf{x}}_2^H\mathbf{A}{\mathbf{x}}_1={\lambda }_1{\mathbf{x}}_2^H{\mathbf{x}}_1\\ ⟹& {\mathbf{x}}_1^H{\mathbf{A}}^H{\mathbf{x}}_2={\lambda }_1{\mathbf{x}}_1^H{\mathbf{x}}_2\\ ⟹& {\mathbf{x}}_1^H\mathbf{A}{\mathbf{x}}_2={\lambda }_1{\mathbf{x}}_1^H{\mathbf{x}}_2\\ ⟹& {\mathbf{x}}_1^H{\lambda }_2{\mathbf{x}}_2={\lambda }_1{\mathbf{x}}_1^H{\mathbf{x}}_2\\ ⟹& ({\lambda }_1-{\lambda }_2){\mathbf{x}}_1^H{\mathbf{x}}_2=0\\ ⟹& {\mathbf{x}}_1^H{\mathbf{x}}_2=0.\end{array}$$

Thus ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$ are orthogonal. In between we took conjugate transpose on both sides, used the fact that $\mathbf{A}={\mathbf{A}}^H$ and ${\lambda }_1-{\lambda }_2\ne 0$.

Definition 4.121 (Unitary diagonalizable matrix)

A complex $n\times n$ matrix $\mathbf{A}$ is said to be unitary diagonalizable if there exists a unitary matrix $\mathbf{U}$ which can diagonalize $\mathbf{A}$; i.e.,

$$\mathbf{D}={\mathbf{U}}^H\mathbf{A}\mathbf{U}$$

is a complex diagonal matrix.

Lemma 4.67

Let $\mathbf{A}$ be a unitary diagonalizable matrix whose diagonalization $\mathbf{D}$ is real. Then $\mathbf{A}$ is Hermitian.

Proof. We have a real diagonal matrix $\mathbf{D}$ such that

$$\mathbf{A}=\mathbf{U}\mathbf{D}{\mathbf{U}}^H.$$

Taking conjugate transpose on both sides we get

$${\mathbf{A}}^H=\mathbf{U}{\mathbf{D}}^H{\mathbf{U}}^H=\mathbf{U}\mathbf{D}{\mathbf{U}}^H=\mathbf{A}.$$

Thus $\mathbf{A}$ is Hermitian. We used the fact that ${\mathbf{D}}^H=\mathbf{D}$ since $\mathbf{D}$ is real.

Theorem 4.127

Every Hermitian matrix $\mathbf{A}$ is unitary diagonalizable.

We skip the proof of this theorem. The theorem means that if $\mathbf{A}$ is Hermitian then $\mathbf{A}=\mathbf{U}\mathrm{\Lambda }{\mathbf{U}}^H$ where $\mathrm{\Lambda }$ is a real diagonal matrix.

Definition 4.122 (Eigen value decomposition of a Hermitian matrix)

Let $\mathbf{A}$ be an $n\times n$ Hermitian matrix. Let ${\lambda }_1,\dots {\lambda }_n$ be its eigen values such that $|{\lambda }_1|\ge |{\lambda }_2|\ge \cdots \ge |{\lambda }_n|$. Let

$$\mathrm{\Lambda }=\mathrm{diag}({\lambda }_1,\dots ,{\lambda }_n).$$

Let $\mathbf{U}$ be a unit matrix consisting of orthonormal eigen vectors corresponding to ${\lambda }_1,\dots ,{\lambda }_n$. Then the eigen value decomposition of $\mathbf{A}$ is defined as

$$\mathbf{A}=\mathbf{U}\mathrm{\Lambda }{\mathbf{U}}^H.$$

If ${\lambda }_i$ are distinct, then the decomposition is unique.

Remark 4.21

Let $\mathrm{\Lambda }$ be a diagonal matrix as in Definition 4.122. Consider some vector $\mathbf{x}\in {\mathbb{C}}^n$.

$${\mathbf{x}}^H\mathrm{\Lambda }\mathbf{x}=\sum _{i=1}^n{\lambda }_i|x_i{|}^2.$$

Now if ${\lambda }_i\ge 0$ then

$${\mathbf{x}}^H\mathrm{\Lambda }\mathbf{x}\le {\lambda }_1\sum _{i=1}^n|x_i{|}^2={\lambda }_1\Vert \mathbf{x}{\Vert }_2^2.$$

Also

$${\mathbf{x}}^H\mathrm{\Lambda }\mathbf{x}\ge {\lambda }_n\sum _{i=1}^n|x_i{|}^2={\lambda }_n\Vert \mathbf{x}{\Vert }_2^2.$$

Lemma 4.68

Let $\mathbf{A}$ be a Hermitian matrix with non-negative eigen values. Let ${\lambda }_1$ be its largest and ${\lambda }_n$ be its smallest eigen values.

$${\lambda }_n\Vert \mathbf{x}{\Vert }_2^2\le {\mathbf{x}}^H\mathbf{A}\mathbf{x}\le {\lambda }_1\Vert \mathbf{x}{\Vert }_2^2\mathrm{\forall }\mathbf{x}\in {\mathbb{C}}^n.$$

Proof. $\mathbf{A}$ has an eigen value decomposition given by

$$\mathbf{A}=\mathbf{U}\mathrm{\Lambda }{\mathbf{U}}^H.$$

1. Let $\mathbf{x}\in {\mathbb{C}}^n$ and let $\mathbf{v}=U^H\mathbf{x}$.

2. Clearly $\Vert \mathbf{x}{\Vert }_2=\Vert \mathbf{v}{\Vert }_2$.

3. Then

   $${\mathbf{x}}^H\mathbf{A}\mathbf{x}={\mathbf{x}}^H\mathbf{U}\mathrm{\Lambda }{\mathbf{U}}^H\mathbf{x}={\mathbf{v}}^H\mathrm{\Lambda }\mathbf{v}.$$

4. From previous remark we have

   $${\lambda }_n\Vert \mathbf{v}{\Vert }_2^2\le {\mathbf{v}}^H\mathrm{\Lambda }\mathbf{v}\le {\lambda }_1\Vert \mathbf{v}{\Vert }_2^2.$$

5. Thus we get

   $${\lambda }_n\Vert \mathbf{x}{\Vert }_2^2\le {\mathbf{x}}^H\mathbf{A}\mathbf{x}\le {\lambda }_1\Vert \mathbf{x}{\Vert }_2^2.$$

4.9.14. Miscellaneous Properties

This subsection lists some miscellaneous properties of eigen values of a square matrix.

Lemma 4.69

$\lambda$ is an eigen value of $\mathbf{A}$ if and only if $\lambda +k$ is an eigen value of $\mathbf{A}+k\mathbf{I}$. Moreover $\mathbf{A}$ and $\mathbf{A}+k\mathbf{I}$ share the same eigen vectors.

Proof. We can see that

$$\begin{array}{rl}& \mathbf{A}\mathbf{x}=\lambda \mathbf{x}\\ ⟺& \mathbf{A}\mathbf{x}+\mathbf{k}\mathbf{x}=\lambda \mathbf{x}+k\mathbf{x}\\ ⟺& (\mathbf{A}+k\mathbf{I})\mathbf{x}=(\lambda +k)\mathbf{x}.\end{array}$$

Thus $\lambda$ is an eigen value of $\mathbf{A}$ with an eigen vector $\mathbf{x}$ if and only if $\lambda +k$ is an eigen vector of $\mathbf{A}+k\mathbf{I}$ with an eigen vector $\mathbf{x}$.

4.9.15. Semidefinite, Definite and Indefinite Matrices

4.9.15.1. Positive Semidefinite Matrices

Definition 4.123 (Positive semidefinite matrix)

A symmetric matrix $\mathbf{X}\in {\mathbb{S}}^n$ is called positive semi-definite if ${\mathbf{v}}^T\mathbf{X}\mathbf{v}\ge 0$ for all $\mathbf{v}\in {\mathbb{R}}^n$.

The notation $\mathbf{X}⪰\mathbf{O}$ means ${\mathbf{v}}^T\mathbf{X}\mathbf{v}\ge 0\mathrm{\forall }\mathbf{v}\in {\mathbb{R}}^n$ or the matrix $\mathbf{X}$ is positive semi-definite.

We define the set of symmetric positive semidefinite matrices as

$${\mathbb{S}}_{+}^n=\{\mathbf{X}\in {\mathbb{S}}^n|\mathbf{X}⪰\mathbf{O}\}.$$

“positive semidefinite” is often abbreviated as “p.s.d.”.

Theorem 4.128

Let $\mathbf{A}\in {\mathbb{R}}^{m\times n}$ be an arbitrary matrix. Then, ${\mathbf{A}}^T\mathbf{A}$ is a p.s.d. matrix in ${\mathbb{S}}^n$ and $\mathbf{A}{\mathbf{A}}^T$ is a p.s.d. matrix in ${\mathbb{S}}^m$.

Proof. We shall just prove this for ${\mathbf{A}}^T\mathbf{A}$.

1. We note that

   $$({\mathbf{A}}^T\mathbf{A}{)}^T={\mathbf{A}}^T\mathbf{A}.$$

   Thus, ${\mathbf{A}}^T\mathbf{A}$ is symmetric.

2. ${\mathbf{A}}^T\mathbf{A}\in {\mathbb{R}}^{n\times n}$. Since it is symmetric, hence ${\mathbf{A}}^T\mathbf{A}\in {\mathbb{S}}^n$.

3. Let $\mathbf{v}\in {\mathbb{R}}^n$.

4. Then,

   $${\mathbf{v}}^T{\mathbf{A}}^T\mathbf{A}\mathbf{v}=(\mathbf{A}\mathbf{v}{)}^T(\mathbf{A}\mathbf{v})=\Vert \mathbf{A}\mathbf{v}{\Vert }^2\ge 0$$

   where $\Vert \cdot \Vert$ is the norm induced by the dot product on ${\mathbb{R}}^n$.

5. Thus, ${\mathbf{A}}^T\mathbf{A}$ is p.s.d..

Theorem 4.129 (Non-negativity of the diagonal elements of p.s.d. matrices)

Let $\mathbf{A}\in {\mathbb{R}}^{n\times n}$ be positive semidefinite. Then, its diagonal elements are non-negative.

Proof. Let $\mathbf{A}\in {\mathbb{S}}_{+}^n$.

1. Then, for every nonzero $\mathbf{v}\in {\mathbb{R}}^n$, ${\mathbf{v}}^T\mathbf{A}\mathbf{v}\ge 0$.

2. In particular, this is true for standard unit vectors.

3. But ${\mathbf{e}}_i^T\mathbf{A}{\mathbf{e}}_i=A_{i,i}$.

4. Hence, $A_{i,i}\ge 0$ must be true.

Definition 4.124 (Square root of a positive semidefinite matrix)

Let $\mathbf{A}\in {\mathbb{S}}^n$ be a positive semidefinite matrix. The square root of $\mathbf{A}$, denoted by ${\mathbf{A}}^{\frac{1}{2}}$ is defined as follows.

Let $\mathbf{A}=\mathbf{U}\mathbf{D}{\mathbf{U}}^T$ be the eigenvalue decomposition of $\mathbf{A}$. Let $d_1,\dots ,d_n$ be the diagonal elements of $\mathbf{D}$. Let $\mathbf{E}=\mathrm{diag}(\sqrt{d_1},\dots ,\sqrt{d_n})$. Then,

$${\mathbf{A}}^{\frac{1}{2}}\triangleq \mathbf{U}\mathbf{E}{\mathbf{U}}^T.$$

The matrix $\mathbf{U}\mathbf{E}{\mathbf{U}}^T$ is known as the positive semidefinite square root.

The definition of $\mathbf{E}$ is justified since $d_i\ge 0$ for a p.s.d. matrix.

We can see that

$$ç{\mathbf{A}}^{\frac{1}{2}}=\mathbf{U}\mathbf{E}{\mathbf{U}}^T\mathbf{U}\mathbf{E}{\mathbf{U}}^T=\mathbf{U}\mathbf{E}\mathbf{E}{\mathbf{U}}^T=\mathbf{U}\mathbf{D}{\mathbf{U}}^T=\mathbf{A}.$$

4.9.15.2. Positive Definite Matrices

Definition 4.125 (Positive definite matrix)

A symmetric matrix $\mathbf{X}\in {\mathbb{S}}^n$ is called positive definite if ${\mathbf{v}}^T\mathbf{X}\mathbf{v}>0$ for all nonzero $\mathbf{v}\in {\mathbb{R}}^n$.

The notation $\mathbf{X}\mathrm{succ}\mathbf{O}$ means ${\mathbf{v}}^T\mathbf{X}\mathbf{v}>0\mathrm{\forall }\mathbf{v}\in {\mathbb{R}}^n,\mathbf{v}\ne 0$ or the matrix $\mathbf{X}$ is positive definite.

We define the set of symmetric positive definite matrices as

$${\mathbb{S}}_{++}^n=\{\mathbf{X}\in {\mathbb{S}}^n|\mathbf{X}\mathrm{succ}0\}.$$

“positive definite” is often abbreviated as “p.d.”.

Theorem 4.130 (Positivity of the diagonal elements of p.d. matrices)

Let $\mathbf{A}\in {\mathbb{R}}^{n\times n}$ be positive definite. Then, its diagonal elements are positive.

Proof. Let $\mathbf{A}\in {\mathbb{S}}_{++}^n$.

1. Then, for every nonzero $\mathbf{v}\in {\mathbb{R}}^n$, ${\mathbf{v}}^T\mathbf{A}\mathbf{v}>0$.

2. In particular, this is true for standard unit vectors.

3. But ${\mathbf{e}}_i^T\mathbf{A}{\mathbf{e}}_i=A_{i,i}$.

4. Hence, $A_{i,i}>0$ must be true.

Theorem 4.131 (Principal minors criterion)

A matrix $\mathbf{A}\in {\mathbb{S}}^n$ is positive definite if and only if the determinants of all the principal minors are positive.

In other words, $D_1(\mathbf{A})>0,D_2(\mathbf{A})>0,\dots ,D_n(\mathbf{A})>0$ where $D_i(\mathbf{A})$ denotes the determinant of the upper left $i\times i$ submatrix (the $i$-th principal minor).

4.9.15.3. Negative Semidefinite Matrices

Definition 4.126 (Negative semidefinite matrix)

A symmetric matrix $\mathbf{X}\in {\mathbb{S}}^n$ is called negative semi-definite if ${\mathbf{v}}^T\mathbf{X}\mathbf{v}\le 0$ for all $\mathbf{v}\in {\mathbb{R}}^n$.

The notation $\mathbf{X}⪯\mathbf{O}$ means ${\mathbf{v}}^T\mathbf{X}\mathbf{v}\le 0\mathrm{\forall }\mathbf{v}\in {\mathbb{R}}^n$ or the matrix $\mathbf{X}$ is negative semi-definite.

We define the set of symmetric negative semidefinite matrices as

$${\mathbb{S}}_{-}^n=\{\mathbf{X}\in {\mathbb{S}}^n|\mathbf{X}⪯\mathbf{O}\}.$$

“negative semidefinite” is sometimes abbreviated as “n.s.d.”.

4.9.15.4. Negative Definite Matrices

Definition 4.127 (Negative definite matrix)

A symmetric matrix $\mathbf{X}\in {\mathbb{S}}^n$ is called negative definite if ${\mathbf{v}}^T\mathbf{X}\mathbf{v}<0$ for all nonzero $\mathbf{v}\in {\mathbb{R}}^n$.

The notation $\mathbf{X}\prec \mathbf{O}$ means ${\mathbf{v}}^T\mathbf{X}\mathbf{v}<0\mathrm{\forall }\mathbf{v}\in {\mathbb{R}}^n,\mathbf{v}\ne 0$ or the matrix $\mathbf{X}$ is negative definite.

We define the set of symmetric negative definite matrices as

$${\mathbb{S}}_{--}^n=\{\mathbf{X}\in {\mathbb{S}}^n|\mathbf{X}\prec 0\}.$$

“negative definite” is sometimes abbreviated as “n.d.”.

4.9.15.5. Indefinite Matrices

Definition 4.128 (Indefinite matrix)

A symmetric matrix $\mathbf{A}\in {\mathbb{S}}^n$ is called indefinite if there exist $\mathbf{x},\mathbf{y}\in {\mathbb{R}}^n$ such that ${\mathbf{x}}^T\mathbf{A}\mathbf{x}<0$ and ${\mathbf{y}}^T\mathbf{A}\mathbf{y}>0$.

Indefinite matrices are neither positive semidefinite nor negative semidefinite.

Theorem 4.132 (Diagonal elements and indefiniteness)

Let $\mathbf{A}\in {\mathbb{S}}^n$. If the diagonal elements of $\mathbf{A}$ are both positive and negative, then $\mathbf{A}$ is indefinite.

Proof. Each diagonal element corresponds to ${\mathbf{e}}_i^T\mathbf{A}{\mathbf{e}}_i$ for a unit vector ${\mathbf{e}}_i$. If the diagonal elements are both positive and negative, then there exist unit vectors ${\mathbf{e}}_i$ and ${\mathbf{e}}_j$ such that ${\mathbf{e}}_i^T\mathbf{A}{\mathbf{e}}_i<0$ and ${\mathbf{e}}_j^T\mathbf{A}{\mathbf{e}}_j>0$. Thus, $\mathbf{A}$ is indefinite.

4.9.15.6. Eigenvalue Decomposition

Theorem 4.133 (Eigenvalue characterization theorem)

Let $\mathbf{A}\in {\mathbb{S}}^n$ be an $n\times n$ symmetric matrix.

1. $\mathbf{A}$ is positive definite if and only if all its eigenvalues are positive.

2. $\mathbf{A}$ is positive semidefinite if and only if all its eigenvalues are nonnegative.

3. $\mathbf{A}$ is negative definite if and only if all its eigenvalues are negative.

4. $\mathbf{A}$ is negative semidefinite if and only if all its eigenvalues are nonpositive.

5. $\mathbf{A}$ is indefinite if and only if at least one eigenvalue is positive and at least one eigenvalue is negative.

Proof. Let the eigenvalue decomposition of $\mathbf{A}$ be given by $\mathbf{A}=\mathbf{U}\mathbf{D}{\mathbf{U}}^T$.

We prove (1).

1. Let $\mathbf{v}\in {\mathbb{R}}^n$.

2. Then

   $${\mathbf{v}}^T\mathbf{A}\mathbf{v}={\mathbf{v}}^T\mathbf{U}\mathbf{D}{\mathbf{U}}^T\mathbf{v}={\mathbf{w}}^T\mathbf{D}\mathbf{w}=\sum _{i=1}^n{d}_i{w}_i^2.$$

   where $\mathbf{w}={\mathbf{U}}^T\mathbf{v}$.

3. Since $\mathbf{U}$ is nonsingular, hence ${\mathbf{v}}^T\mathbf{A}\mathbf{v}>0$ for every $\mathbf{v}\ne \mathbf{0}$ if and only if $\sum _{i=1}^n{d}_i{w}_i^2>0$ for every $\mathbf{w}\ne \mathbf{0}$.

4. Plugging ${\mathbf{e}}_i$ for $\mathbf{w}$, we see that $d_i>0$ is a necessary condition.

5. Also, if $d_i>0$ for every $i$, then, the sum is positive for every nonzero $\mathbf{w}$ since at least one $w_i\ne 0$. Hence, it is a sufficient condition.

Similar arguments apply for other statements.

4.9.15.7. Trace and Determinant

Corollary 4.22 (Trace and determinant of positive definite matrices)

Let $\mathbf{A}$ be a symmetric positive definite matrix. Then, $\mathrm{tr}(\mathbf{A})$ and $det(\mathbf{A})$ are positive.

Proof. Trace is the sum of eigenvalues. Determinant is the product of eigenvalues. If $\mathbf{A}$ is symmetric positive definite, then its eigenvalues are positive. Hence, trace and determinant are positive.

Corollary 4.23 (Trace and determinant of positive semidefinite matrices)

Let $\mathbf{A}$ be a symmetric positive definite matrix. Then, $\mathrm{tr}(\mathbf{A})$ and $det(\mathbf{A})$ are nonnegative.

4.9.16. Diagonally Dominant Matrices

Definition 4.129 (Diagonally dominant matrix)

Let $\mathbf{A}\in {\mathbb{F}}^{n\times n}$ with $\mathbf{A}=[a_{ij}]$.

1. $\mathbf{A}$ is called diagonally dominant if

   $$|a_{ii}|\ge \sum _{j,j\ne i}|a_{ij}|$$

   for every $i=1,\dots ,n$. In other words, the absolute value of the diagonal entry in a row is greater than or equal to the sum of absolute values of non diagonal entries in the row.

2. $\mathbf{A}$ is called strictly diagonally dominant if

   $$|a_{ii}|>\sum _{j,j\ne i}|a_{ij}|$$

   for every $i=1,\dots ,n$. In other words, the absolute value of the diagonal element is bigger than the sum of absolute values of all the off diagonal elements on that row.

Example 4.28 (Strictly diagonally dominant matrix)

Let us consider

$$\begin{array}{r}\mathbf{A}=\left[\begin{array}{cccc}-4& -2& -1& 0\\ -4& 7& 2& 0\\ 3& -4& 9& 1\\ 2& -1& -3& 15\end{array}\right].\end{array}$$

We can see that the strict diagonal dominance condition is satisfied for each row as follows:

$$\begin{array}{rlr}& \text{ row 1}:& |-4|>|-2|+|-1|+|0|=3\\ & \text{ row 2}:& |7|>|-4|+|2|+|0|=6\\ & \text{ row 3}:& |9|>|3|+|-4|+|1|=8\\ & \text{ row 4}:& |15|>|2|+|-1|+|-3|=6.\end{array}$$

Theorem 4.134 (Positive semidefiniteness of diagonally dominant matrices)

Let $\mathbf{A}\in {\mathbb{S}}^n$ be a real symmetric matrix.

1. If $\mathbf{A}$ is diagonally dominant whose diagonal entries are nonnegative then $\mathbf{A}$ is positive semidefinite.

2. If $\mathbf{A}$ is strictly diagonally dominant whose diagonal entries are positive then $\mathbf{A}$ is positive definite.

Proof. Assume that $\mathbf{A}$ is diagonally dominant with nonnegative diagonal entries.

1. For contradiction, assume that $\mathbf{A}$ is not positive semidefinite.

2. Then, there is an eigen value $\lambda <0$ and corresponding eigenvector $\mathbf{u}$.

3. Consider the absolute values of entries of $\mathbf{u}$ given by $(|u_1|,\dots ,|u_n|)$.

4. Let $i\in 1,\dots ,n$ denote the index of the largest absolute value entry of $\mathbf{u}$.

5. We also have $\mathbf{A}\mathbf{u}=\lambda \mathbf{u}$.

6. For the $i$-th row, we get the equality

   $$\begin{array}{rl}& \sum _{j=1}^n{A}_{ij}{u}_j=\lambda u_i\\ & ⟺\sum _{j\ne i}{A}_{ij}{u}_j=\lambda u_i-A_{ii}{u}_i\\ & ⟹\left|\sum _{j\ne i}{A}_{ij}{u}_j\right|=|\lambda -A_{ii}||u_i|\\ & ⟹|\lambda -A_{ii}||u_i|\le \left(\sum _{j\ne i}|A_{ij}|\right)|u_i|\le |A_{ii}||u_i|.\end{array}$$

7. Thus, $|\lambda -A_{ii}|=|A_{ii}-\lambda |\le |A_{ii}|$.

8. But $A_{ii}$ is nonnegative and $\lambda$ is negative, hence this reduces to $A_{ii}-\lambda \le A_{ii}$ or $\lambda \ge 0$.

9. We have arrived at a contradiction.

10. Thus, $\mathbf{A}$ must be positive semidefinite.

Now, assume that $\mathbf{A}$ is strictly diagonally dominant with positive diagonal entries.

1. By first part, it is clear that $\mathbf{A}$ is p.s.d..

2. We just need to show that all eigenvalues are positive. There are no zero eigenvalues.

3. For contradiction, assume that $0$ is indeed an eigenvalue of $\mathbf{A}$.

4. Let $\mathbf{u}\ne \mathbf{0}$ be corresponding eigenvector satisfying $\mathbf{A}\mathbf{u}=\mathbf{0}$.

5. Let $i\in 1,\dots ,n$ denote the index of the largest absolute value entry of $\mathbf{u}$.

6. Following the earlier argument

   $$\begin{array}{rl}|A_{ii}||u_i|& =\left|\sum _{j\ne i}{A}_{ij}{u}_j\right|\\ & \le \left(\sum _{j\ne i}|A_{ij}|\right)|u_i|\\ & <|A_{ii}||u_i|.\end{array}$$

7. This is impossible. Hence, $\mathbf{A}$ must be positive definite.

Strictly diagonally dominant matrices have a very special property. They are always non-singular. The following result is valid for both real and complex strictly diagonally dominant matrices.

Theorem 4.135

Strictly diagonally dominant matrices are non-singular.

Proof. For contradiction, suppose that $\mathbf{A}$ is strictly diagonally dominant and singular.

1. Then there exists a vector $u\in {\mathbb{C}}^n$ with $\mathbf{u}\ne \mathbf{0}$ such that

   $$\mathbf{A}\mathbf{u}=\mathbf{0}.$$

2. Let

   $$\mathbf{u}=\left(\begin{array}{cccc}{u}_1& u_2& \dots & u_n\end{array}\right).$$

3. We first show that every entry in $\mathbf{u}$ cannot be equal in magnitude.

4. For contradiction, let us assume that there exists $c>0$ such that

   $$c=|u_1|=|u_2|=\cdots =|u_n|.$$

5. Since $\mathbf{u}\ne \mathbf{0}$ hence $c\ne 0$.

6. We can write $u_j=c\exp (i{\theta }_j)$ for every $j=1,\dots ,n$.

7. Now for any row $i$ in $\mathbf{A}\mathbf{u}=\mathbf{0}$, we have

   $$\begin{array}{rl}& \sum _{j=1}^n{a}_{ij}{u}_j=0\\ ⟹& \sum _{j=1}^n{a}_{ij}c\exp (i{\theta }_j)=0\\ ⟹& \sum _{j=1}^n{a}_{ij}\exp (i{\theta }_j)=0\\ ⟹& -a_{ii}\exp (i{\theta }_j)=\sum _{j\ne i}{a}_{ij}\exp (i{\theta }_j)\\ ⟹& |a_{ii}|=|\sum _{j\ne i}{a}_{ij}\exp (i{\theta }_j)|\\ ⟹& |a_{ii}|\le \sum _{j\ne i}|a_{ij}|\end{array}$$

   by triangle inequality.

8. But this contradicts our assumption that $\mathbf{A}$ is strictly diagonally dominant.

9. Thus all entries in $\mathbf{u}$ are not equal in magnitude.

10. Let us now assume that the largest entry in $\mathbf{u}$ lies at index $i$ with $|u_i|=c$.

11. Without loss of generality we can scale down $\mathbf{u}$ by $c$ to get another vector in which all entries are less than or equal to 1 in magnitude while $i$-th entry has a magnitude of $1$.

12. In other words, $|u_i|=1$ and $|u_j|\le 1$ for all other entries.

13. From $\mathbf{A}\mathbf{u}=\mathbf{0}$ we get for the $i$-th row

    $$\begin{array}{rl}& \sum _{j=1}^n{a}_{ij}{u}_j=0\\ ⟹& -a_{ii}{u}_i=\sum _{j\ne i}{u}_j{a}_{ij}\\ ⟹& |a_{ii}{u}_i|=\left|\sum _{j\ne i}{u}_j{a}_{ij}\right|\\ ⟹& |a_{ii}|\le \sum _{j\ne i}|u_j{a}_{ij}|\le \sum _{j\ne i}|a_{ij}|\end{array}$$

    which again contradicts our assumption that $\mathbf{A}$ is strictly diagonally dominant.

14. Hence strictly diagonally dominant matrices are non-singular.

4.9.17. Gershgorin’s Theorem

We are now ready to examine Gershgorin’ theorem which provides very useful bounds on the spectrum of a square matrix.

Theorem 4.136 (Gershgorin’s circle theorem)

Every eigen value $\lambda$ of a square matrix $\mathbf{A}\in {\mathbb{C}}^{n\times n}$ satisfies

(4.13)$$|\lambda -a_{ii}|\le \sum _{j,j\ne i}|a_{ij}|\text{ for some }i\in \{1,2,\dots ,n\}.$$

Proof. The proof is a straight forward application of non-singularity of diagonally dominant matrices.

1. We know that for an eigen value $\lambda$, $det(\lambda \mathbf{I}-\mathbf{A})=0$

2. In other words, the matrix $(\lambda \mathbf{I}-\mathbf{A})$ is singular.

3. Hence it cannot be strictly diagonally dominant due to Theorem 4.135.

4. Thus looking at each row $i$ of $(\lambda \mathbf{I}-\mathbf{A})$ we can say that

   $$|\lambda -a_{ii}|>\sum _{j,j\ne i}|a_{ij}|$$

   cannot be true for all rows simultaneously.

5. In other words, this condition must fail at least for one row.

6. This means that there exists at least one row $i$ for which

   $$|\lambda -a_{ii}|\le \sum _{j,j\ne i}|a_{ij}|$$

   holds true.

What this theorem means is pretty simple.

1. Consider a disc in the complex plane for the $i$-th row of $\mathbf{A}$ whose center is given by $a_{ii}$ and whose radius is given by $r=\sum _{j\ne i}|a_{ij}|$ i.e. the sum of magnitudes of all non-diagonal entries in $i$-th row.

2. There are $n$ such discs corresponding to $n$ rows in $\mathbf{A}$.

3. (4.13) means that every eigen value must lie within the union of these discs. It cannot lie outside.

This idea is crystallized in following definition.

Definition 4.130 (Gershgorin’s disc)

For $i$-th row of the square matrix $\mathbf{A}$ we define the radius $r_i=\sum _{j,j\ne i}|a_{ij}|$ and the center $c_i=a_{ii}$. Then the set given by

$$D_i=\{z\in \mathbb{C}||z-a_{ii}|\le r_i\}$$

is called the $i$-th Gershgorin’s disc of $\mathbf{A}$.

We note that the definition is equally valid for real as well as complex matrices. For real matrices, the centers of disks lie on the real line. For complex matrices, the centers may lie anywhere in the complex plane.

Clearly there is nothing extraordinary about the rows of $\mathbf{A}$. We can as well consider the columns of $\mathbf{A}$.

Theorem 4.137 (Gershgorin’s circle theorem for columns)

Every eigen value of a matrix $\mathbf{A}$ must lie in a Gershgorin disc corresponding to the columns of $\mathbf{A}$ where the Gershgorin disc for $j$-th column is given by

$$D_j=\{z\in \mathbb{C}||z-a_{jj}|\le r_j\}$$

with

$$r_j=\sum _{i,i\ne j}|a_{ij}|$$

Proof. We know that eigen values of $\mathbf{A}$ are same as eigen values of ${\mathbf{A}}^T$ and columns of $\mathbf{A}$ are nothing but rows of ${\mathbf{A}}^T$. Hence eigen values of $\mathbf{A}$ must satisfy conditions in Theorem 4.136 w.r.t. the matrix ${\mathbf{A}}^T$. This completes the proof.