# Matrices II

## Contents

# 4.2. Matrices II#

This section deals with the concepts of vector spaces associated with matrices, span, rank, invertible matrices, similar matrices, gram matrices, pseudo inverses, traces and determinants.

## 4.2.1. Spaces Associated with a Matrix#

(Column space)

The *column space* of a matrix is defined as the vector space spanned by columns of the matrix.
Let \(\bA\) be an \(m \times n\) matrix with

Then the column space is given by

(Row space)

The *row space* of a matrix is defined as the vector space spanned by rows of the matrix.

Let \(\bA\) be an \(m \times n\) matrix with

Then the row space is given by

## 4.2.2. Rank#

(Column rank)

The *column rank* of a matrix is defined as the maximum number of columns which are linearly
independent.
In other words column rank is the dimension of the column space of a matrix.

(Row rank)

The *row rank* of a matrix is defined as the maximum number of rows which are linearly
independent. In other words row rank is the dimension of the row space of a matrix.

(Equality of row and column rank)

The *column rank* and *row rank* of a matrix are equal.

(Rank)

The *rank* of a matrix is defined to be equal to its column rank which is equal to its row rank.

(Rank bounds)

For an \(m \times n\) matrix \(\bA\)

(Zero rank matrix)

The rank of a matrix is 0 if and only if it is a zero matrix.

(Full rank matrix)

An \(m \times n\) matrix \(\bA\) is called *full rank* if

In other words it is either a full column rank matrix or a full row rank matrix or both.

(Rank of product of two matrices)

Let \(\bA\) be an \(m \times n\) matrix and \(\bB\) be an \(n \times p\) matrix then

(Full rank post multiplication)

Let \(\bA\) be an \(m \times n\) matrix and \(\bB\) be an \(n \times p\) matrix. If \(\bB\) is of rank \(n\) then

(Full rank pre multiplication)

Let \(\bA\) be an \(m \times n\) matrix and \(\bB\) be an \(n \times p\) matrix. If \(\bA\) is of rank \(n\) then

(Rank of a diagonal matrix)

The rank of a diagonal matrix is equal to the number of nonzero elements on its main diagonal.

Proof. The columns which correspond to diagonal entries which are zero are zero columns. Other columns are linearly independent. The number of linearly independent rows is also the same. Hence their count gives us the rank of the matrix.

## 4.2.3. Invertible Matrices#

We say that an \(m \times n\) matrix \(\bA\) is *left-invertible* if there exists
an \(n \times m\) matrix \(\bB\) such that
\(\bB \bA = \bI\).
We say that an \(m \times n\) matrix \(\bA\) is *right-invertible* if there exists
an \(n \times m\) matrix \(\bB\) such that \(\bA \bB= \bI\).

We say that a square matrix \(\bA\) is *invertible* when there exists another square matrix
\(\bB\) of same size such that \(\bA \bB = \bB \bA = \bI\).
A square matrix is invertible iff it is both left and right invertible.
Inverse of a square invertible matrix is denoted by \(\bA^{-1}\).

A special left or right inverse is the pseudo inverse, which is denoted by \(\bA^{\dag}\).

Column space of a matrix is denoted by \(\ColSpace(\bA)\), the null space by \(\NullSpace(\bA)\), and the row space by \(\RowSpace(\bA)\).

We say that a matrix is *symmetric* when \(\bA = \bA^T\),
*conjugate symmetric* or *Hermitian* when \(\bA^H =\bA\).

When a square matrix is not invertible, we say that it is *singular*.
A *non-singular* matrix is invertible.

(Invertible matrix)

A square matrix \(\bA\) is called *invertible*
if there exists another square matrix \(\bB\) of same size such that

The matrix \(\bB\) is called the *inverse* of \(\bA\) and is denoted as \(\bA^{-1}\).

(Invertibility of the inverse)

If \(\bA\) is invertible then its inverse \(\bA^{-1}\) is also invertible and the inverse of \(\bA^{-1}\) is nothing but \(\bA\).

(Invertibility of identity matrix)

Identity matrix \(\bI\) is invertible.

Proof. We can see that

(Linear independence of columns of invertible matrices)

If \(\bA\) is invertible then columns of \(\bA\) are linearly independent.

Proof. Assume \(\bA\) is invertible, then there exists a matrix \(\bB\) such that

Assume that columns of \(\bA\) are linearly dependent. Then there exists \(\bu \neq \bzero\) such that

a contradiction. Hence columns of \(\bA\) are linearly independent.

(Span of columns of invertible matrix)

If an \(n\times n\) matrix \(\bA\) is invertible then columns of \(\bA\) span \(\FF^n\).

Proof. Assume \(\bA\) is invertible, then there exists a matrix \(\bB\) such that

Now let \(\bx \in \FF^n\) be any arbitrary vector.

We need to show that there exists \(\ba \in \FF^n\) such that

\[ \bx = \bA \ba. \]But

\[ \bx = \bI \bx = \bA \bB \bx = \bA ( \bB \bx). \]Thus if we choose \(\ba = \bB \bx\), then

\[ \bx = \bA \ba. \]Thus columns of \(\bA\) span \(\FF^n\).

(Columns of invertible matrix as basis)

If \(\bA\) is invertible, then columns of \(\bA\) form a basis for \(\FF^n\).

Proof. In \(\FF^n\) a basis is a set of vectors which is linearly independent and spans \(\FF^n\). By Lemma 4.9 and Lemma 4.10, columns of an invertible matrix \(\bA\) satisfy both conditions. Hence they form a basis.

(Invertibility of transpose)

If \(\bA\) is invertible than \(\bA^T\) is invertible.

Proof. Assume \(\bA\) is invertible, then there exists a matrix \(\bB\) such that

Applying transpose on both sides we get

Thus \(\bB^T\) is inverse of \(\bA^T\) and \(\bA^T\) is invertible.

(Invertibility of Hermitian transpose)

If \(\bA\) is invertible than \(\bA^H\) is invertible.

Proof. Assume \(\bA\) is invertible, then there exists a matrix \(\bB\) such that

Applying conjugate transpose on both sides we get

Thus \(\bB^H\) is inverse of \(\bA^H\) and \(\bA^H\) is invertible.

(Invertibility of matrix product)

If \(\bA\) and \(\bB\) are invertible then \(\bA \bB\) is invertible.

Proof. We note that

Similarly

Thus \(\bB^{-1}\bA^{-1}\) is the inverse of \(\bA \bB\).

(Group of invertible matrices)

The set of \(n \times n\) invertible matrices under the matrix multiplication operation form a group.

Proof. We verify the properties of a group

[Closure] If \(\bA\) and \(\bB\) are invertible then \(\bA \bB\) is invertible. Hence the set is closed.

[Associativity] Matrix multiplication is associative.

[Identity element] \(\bI\) is invertible and \(\bA \bI = \bI \bA = \bA\) for all invertible matrices.

[Inverse element] If \(\bA\) is invertible then \(\bA^{-1}\) is also invertible.

Thus the set of invertible matrices is indeed a group under matrix multiplication.

(Rank of an invertible matrix)

An \(n \times n\) matrix \(\bA\) is invertible if and only if it is full rank i.e.

(Equality of rank of a matrix and its inverse)

The rank of an invertible matrix and its inverse are same.

## 4.2.4. Similar Matrices#

(Similar matrix)

An \(n \times n\) matrix \(\bB\) is *similar* to an \(n \times n\) matrix \(\bA\)
if there exists an \(n \times n\) non-singular matrix \(\bC\) such that

(Symmetry of similarity)

If \(\bB\) is similar to \(\bA\) then \(\bA\) is similar to \(\bB\). Thus similarity is a symmetric relation.

Proof. We have

Thus there exists a matrix \(\bD = \bC^{-1}\) such that

Thus \(\bA\) is similar to \(\bB\).

(Rank of similar matrices)

Similar matrices have same rank.

Proof. Let \(\bB\) be similar to \(\bA\). Then their exists an invertible matrix \(\bC\) such that

Since \(\bC\) is invertible hence we have \(\Rank (\bC) = \Rank(\bC^{-1}) = n\). Now using Lemma 4.4

and using Lemma 4.5 we have

Thus

(Similarity as equivalence relation)

Similarity is an equivalence relation on the set of \(n \times n\) matrices.

Proof. Let \(\bA, \bB, \bC\) be \(n \times n\) matrices.

\(\bA\) is similar to itself through an invertible matrix \(\bI\).

If \(\bA\) is similar to \(\bB\) then \(\bB\) is similar to itself.

If \(\bB\) is similar to \(\bA\) via \(\bP\) s.t. \(\bB = \bP^{-1} \bA \bP\) and \(\bC\) is similar to \(\bB\) via \(\bQ\) s.t. \(\bC = \bQ^{-1} \bB \bQ\) then \(\bC\) is similar to \(\bA\) via \(\bP \bQ\) such that \(\bC = (\bP \bQ)^{-1} \bA (\bP \bQ)\).

Thus similarity is an equivalence relation on the set of square matrices and if \(\bA\) is any \(n \times n\) matrix then the set of \(n \times n\) matrices similar to \(\bA\) forms an equivalence class.

## 4.2.5. Gram Matrices#

(Gram matrix)

The *Gram matrix* of columns of \(\bA\) is given by

(Frame operator)

The *frame operator* is the Gram matrix of rows of \(\bA\) is given by

Usually when we talk about Gram matrix of a matrix we are looking at the Gram matrix of its column vectors.

(Gram matrix and frame operators for real matrices)

For real matrix \(\bA \in \RR^{m \times n}\), the Gram matrix of its column vectors is given by \(\bA^T \bA\) and the frame operator for its row vectors is given by \(\bA \bA^T\).

Following results apply equally well for the real case.

(Linear dependence of columns and Gram matrix)

The columns of a matrix are linearly dependent if and only if the Gram matrix of its column vectors \(\bA^H \bA\) is not invertible.

Proof. Let \(\bA\) be an \(m\times n\) matrix and \(\bG = \bA^H \bA\) be the Gram matrix of its columns.

If columns of \(\bA\) are linearly dependent, then there exists a vector \(\bu \neq \bzero\) such that

\[ \bA \bu = \bzero. \]Thus

\[ \bG \bu = \bA^H \bA \bu = \bzero. \]Hence the columns of \(\bG\) are also linearly dependent.

Hence \(\bG\) is not invertible.

Conversely let us assume that \(\bG\) is not invertible.

Thus columns of \(\bG\) are dependent.

There exists a vector \(\bv \neq \bzero\) such that

\[ \bG \bv = \bzero. \]Now

\[ \bv^H \bG \bv = \bv^H \bA^H \bA \bv = (\bA \bv)^H (\bA \bv) = \| \bA \bv \|_2^2. \]From previous equation, we have

\[ \| \bA \bv \|_2^2 = 0 \implies \bA \bv = \bzero. \]Since \(\bv \neq \bzero\) hence columns of \(\bA\) are also linearly dependent.

(Linear independence of columns and Gram matrix)

The columns of a matrix are linearly independent if and only if the Gram matrix of its column vectors \(\bA^H \bA\) is invertible.

Proof. Columns of \(\bA\) can be dependent only if its Gram matrix is not invertible. Thus if the Gram matrix is invertible, then the columns of \(\bA\) are linearly independent.

The Gram matrix is not invertible only if columns of \(\bA\) are linearly dependent. Thus if columns of \(\bA\) are linearly independent then the Gram matrix is invertible.

Let \(\bA\) be a full column rank matrix. Then \(\bA^H \bA\) is invertible.

The null space of \(\bA\) and its Gram matrix \(\bA^H \bA\) coincide; i.e.,

Proof. Let \(\bu \in \NullSpace(\bA)\). Then

Thus

Now let \(\bu \in \NullSpace(\bA^H \bA)\). Then

Thus we have

The rows of a matrix \(\bA\) are linearly dependent if and only if the Gram matrix of its row vectors \(\bA \bA^H\) is not invertible.

Proof. Rows of \(\bA\) are linearly dependent, if and only if columns of \(\bA^H\) are linearly dependent.

There exists a vector \(\bv \neq \bzero\) s.t.

Thus

Since \(\bv \neq \bzero\) hence \(\bG\) is not invertible.

Converse: assuming that \(\bG\) is not invertible, there exists a vector \(\bu \neq \bzero\) s.t.

Now

Since \(\bu \neq \bzero\) hence columns of \(\bA^H\) and consequently rows of \(\bA\) are linearly dependent.

The rows of a matrix \(\bA\) are linearly independent if and only if the Gram matrix of its row vectors \(AA^H\) is invertible.

Let \(\bA\) be a full row rank matrix. Then \(\bA \bA^H\) is invertible.

## 4.2.6. Pseudo Inverses#

(Moore Penrose pseudo inverse)

Let \(\bA\) be an \(m \times n\) matrix.
An \(n\times m\) matrix \(\bA^{\dag}\) is called its
*Moore-Penrose pseudo-inverse* if it satisfies all of the following criteria:

\(\bA \bA^{\dag} \bA = \bA\).

\(\bA^{\dag} \bA \bA^{\dag} = \bA^{\dag}\).

\(\left(\bA \bA^{\dag} \right)^H = \bA \bA^{\dag}\) i.e. \(\bA \bA^{\dag}\) is Hermitian.

\((\bA^{\dag} \bA)^H = \bA^{\dag} \bA\) i.e. \(\bA^{\dag} \bA\) is Hermitian.

(Existence and uniqueness of Moore Penrose pseudo inverse)

For any matrix \(\bA\) there exists precisely one matrix \(\bA^{\dag}\) which satisfies all the requirements in Definition 4.42.

We omit the proof for this. The pseudo-inverse can actually be obtained by the singular value decomposition of \(\bA\). This is shown in Lemma 4.79.

(Moore Penrose pseudo inverse of a diagonal matrix)

Let \(\bD = \Diag(d_1, d_2, \dots, d_n)\) be an \(n \times n\) diagonal matrix. Then its Moore-Penrose pseudo-inverse is \(\bD^{\dag} = \Diag(c_1, c_2, \dots, c_n)\) where

Proof. We note that \(\bD^{\dag} \bD = \bD \bD^{\dag} = \bF = \Diag(f_1, f_2, \dots f_n)\) where

We now verify the requirements in Definition 4.42.

\(\bD^{\dag} \bD = \bD \bD^{\dag} = \bF\) is a diagonal hence Hermitian matrix.

(Moore Penrose pseudo inverse of a rectangular diagonal matrix)

Let \(\bD = \Diag(d_1, d_2, \dots, d_p)\) be an \(m \times n\)
*rectangular* diagonal matrix where \(p = \min(m, n)\).
Then its Moore-Penrose pseudo-inverse is an \(n \times m\) rectangular diagonal matrix
\(\bD^{\dag} = \Diag(c_1, c_2, \dots, c_p)\) where

Proof. We note that

is an \(n \times n\) matrix where

\(\bG = \bD \bD^{\dag} = \Diag(g_1, g_2, \dots g_n)\) is an \(m \times m\) matrix where

We now verify the requirements in Definition 4.42.

\(\bF = \bD^{\dag} \bD\) and \(\bG = \bD \bD^{\dag}\) are both real diagonal hence Hermitian matrices.

(Moore Penrose pseudo inverse of full column rank matrices)

If \(\bA\) is full column rank then its Moore-Penrose pseudo-inverse is given by

It is a left inverse of \(\bA\).

Proof. By Corollary 4.7 \(\bA^H \bA\) is invertible. First of all we verify that it is a left inverse.

We now verify all the properties.

Hermitian properties:

(Moore Penrose pseudo inverse of full row rank matrices)

If \(\bA\) is full row rank then its Moore-Penrose pseudo-inverse is given by

It is a right inverse of \(\bA\).

Proof. By Corollary 4.9 \(\bA \bA^H\) is invertible. First of all we verify that it is a right inverse.

We now verify all the properties.

Hermitian properties:

## 4.2.7. Trace#

(Trace of a square matrix)

The *trace* of a square matrix is defined as the sum of the entries on its main diagonal.
Let \(\bA\) be an \(n\times n\) matrix, then

where \(\Trace(\bA)\) denotes the trace of \(\bA\).

The trace of a square matrix and its transpose are equal.

Trace of sum of two square matrices is equal to the sum of their traces.

(Trace product rule)

Let \(\bA\) be an \(m \times n\) matrix and \(\bB\) be an \(n \times m\) matrix. Then

Proof. Let \(\bA \bB = \bC = [c_{ij}]\). Then

Thus

Now

Let \(\bB \bA = \bD = [d_{ij}]\). Then

Thus

Hence

This completes the proof.

(Trace triple product rule)

Let \(\bA \in \FF^{m \times n}\), \(\bB \in \FF^{n \times p}\), \(\bC \in \FF^{p \times m}\) be three matrices. Then

Proof. Let \(\bA \bB = \bD\). Then

Similarly the other result can be proved.

Trace of similar matrices is equal.

Proof. Let \(\bB\) be similar to \(\bA\). Thus

for some invertible matrix \(\bC\). Then

We used Lemma 4.29.

## 4.2.8. Determinants#

Following are some results on determinant of a square matrix \(\bA\).

(Determinant and scalar multiplication)

Determinant of a square matrix and its transpose are equal.

Let \(\bA\) be a complex square matrix. Then

Proof. We proceed as follows:

(Determinant product rule)

Let \(\bA\) and \(\bB\) be two \(n\times n\) matrices. Then

(Determinant of inverse)

Let \(\bA\) be an invertible matrix. Then

(Determinant power rule)

(Determinant of triangular matrices)

Determinant of a triangular matrix is the product of its diagonal entries; i.e., if \(\bA\) is upper or lower triangular matrix then

(Determinant of diagonal matrices)

Determinant of a diagonal matrix is the product of its diagonal entries; i.e., if \(\bA\) is a diagonal matrix then

(Determinants of similar matrices)

Determinant of similar matrices is equal.

Proof. Let \(\bB\) be similar to \(\bA\). Thus

for some invertible matrix \(\bC\). Hence

Now

We used Lemma 4.35 and Lemma 4.36.

Let \(\bu\) and \(\bv\) be vectors in \(\FF^n\). Then

(Determinant of perturbation of an identity matrix)

Let \(\bA\) be a square matrix and let \(\epsilon \approx 0\). Then