# The Euclidean Space

## Contents

# 4.7. The Euclidean Space#

This section consolidates major results for the real Euclidean space \(\RR^n\) as a ready reference. \(\RR^2\) (the 2-dimensional plane) and \(\RR^3\) the 3-dimensional space are the most familiar spaces to us.

\(\RR^n\) is a generalization in \(n\) dimensions.

\(\RR^n\) and Euclidean space)

(For any positive integer \(n\), the set of all \(n\)-tuples of
real numbers forms an \(n\)-dimensional vector space over \(\RR\)
which is denoted as \(\RR^n\).
It is sometimes called *real coordinate space*.

An element \(\bx\) in \(\RR^n\) is written as

where each \(x_i\) is a real number.

Vector space operations on \(\RR^n\) are defined by:

When equipped with the standard inner product and standard norm (defined below),
\(\RR^n\) becomes the *Euclidean space*.

(Standard basis)

\(\RR^n\) comes with the *standard*
ordered basis
\(\BBB = \{\be_1, \be_2, \dots, \be_n\}\):

An arbitrary vector \(\bx\in\RR^n\) can be written as

## 4.7.1. Inner Products#

(Standard inner product/ dot product)

The standard inner product (a.k.a. dot product) on \(\RR^n\) is defined as:

This makes \(\RR^n\) an inner product space.

The dot product is always a real number. Hence we have symmetry:

It is a real inner product.

\(\bQ\) inner product)

(Let \(\bQ\) be an \(n \times n\) real symmetric positive definite matrix. The \(\bQ\) inner product is defined as:

\(\bQ\) inner product reduces to standard dot product when \(\bQ = \bI\).

## 4.7.2. Norms#

We use norms as a measure of strength of a signal or size of an error. Different norms signify different aspects of the signal.

(Euclidean norm)

The *length* of the vector (a.k.a. *Euclidean norm*
or \(\ell_2\) norm) is defined as:

This makes \(\RR^n\) a normed linear space.

### 4.7.2.1. Angles#

(Angle)

The angle \(\theta\) between two vectors is given by:

### 4.7.2.2. \(\ell_p\) Norms#

In addition to standard Euclidean norm, we define a family of norms indexed by \(p \in [1, \infty]\) known as \(\ell_p\) norms over \(\RR^n\).

\(\ell_p\) norm)

(Let \(p \in [1, \infty]\). For any \(n \in \Nat\), the \(l_p\) norm denoted as \(\| \cdot \|_p : \RR^n \to \RR\) mapping any vector \(\bx \in \RR^n\) to a non-negative number is defined as:

We mention the special cases. \(p=1\) gives us:

\(p=2\) gives us:

which is same as the standard Euclidean norm.

\(p=\infty\) gives us:

We need to justify that \(\ell_p\) norm defined as above is indeed a norm. Before that, we state the Hölder’s inequality for the Euclidean space.

(Hölder’s inequality)

Let \(\bu, \bv \in \RR^n\). Let \(p \in [1, \infty]\) and let \(q\) be its conjugate exponent.

We have:

where \(\bu \bv\) denotes the element-wise multiplication given by:

Proof. If \(\bu = \bzero\) or \(\bv = \bzero\), then (4.4) follows immediately.

Now, consider the case where \(\bu \neq \bzero\) and \(\bv \neq \bzero\).

If \(p=1\), then \(q = \infty\). We have:

The same argument applies for the case of \(p=\infty\) and \(q=1\) too. For the case of \(p,q \in (1,\infty)\), we have:

We recall the Hölder's inequality for real numbers. For any \(n \in \Nat\), \(a_1, \dots, a_n \geq 0\), \(b_1, \dots, b_n \geq 0\), \(p, q\) being conjugate exponents, we have:

Let \(a_k = |u_k|\) and \(b_k = |v_k|\). Then

We are now ready to prove that \(\ell_p\) norm is indeed a norm.

\(\ell_p\) norms are norms)

(For any \(n \in \Nat\) and any \(p \in [1, \infty]\), the function \(\| \cdot \|_p\) as defined in Definition 4.95 is a norm.

Proof. [Positive definiteness] By definition, if \(\bu \neq \bzero\), then \(\| \bu \|_p > 0\). Similarly, \(\| \bu \|_p = 0\) implies \(\bu = \bzero\).

[Positive homogeneity] Let \(\alpha \in \RR\). If \(p < \infty\), then

For \(p=\infty\),

[Triangle inequality] Let \(\bu, \bv \in \RR^n\). If \(p=1\), then

If \(p=\infty\), then:

We are left with the case \(p \in (1, \infty)\).

By Hölder’s inequality:

Note that \((p-1)q = p\) since \(p,q\) are conjugate exponents.

Similarly:

Combining, we get:

Note that:

Thus, dividing both sides by \(\| \bu + \bv \|_p^{\frac{p}{q}}\), we get:

as desired.

### 4.7.2.3. \(\ell_2\) Norm#

As we can see from definition, \(\ell_2\) norm is same as the Euclidean norm.

So we have:

### 4.7.2.4. \(\ell_1\) Norm#

From above definition, we have

### 4.7.2.5. Quasi-norms#

In some cases it is useful to extend the notion of \(\ell_p\) norms to the case where \(0 < p < 1\).

In such cases norm as defined in Definition 4.95
doesn’t satisfy triangle inequality.
Hence it is not a proper norm function.
We call such functions as *quasi-norms*.

### 4.7.2.6. \(\ell_0\) “norm”#

Of specific mention is \(\ell_0\) “norm”. It isn’t even a quasi-norm. Note the use of quotes around the word norm to distinguish \(l_0\) “norm” from usual norms.

\(\ell_0\) “norm”)

(\(\ell_0\) “norm” is defined as:

where \(\supp(\bx) = \{ i : x_i \neq 0\}\) denotes the support of \(\bx\).

Note that \(\| \bx \|_0\) defined above doesn’t follow the definition in Definition 4.95.

Yet we can show that:

which justifies the notation.

\(\bQ\) norm)

(The \(\bQ\) norm induced by \(\bQ\) inner product is given by

### 4.7.2.7. Equivalence of Norms#

\(\RR^n\) is a finite dimensional vector space and all norms on \(\RR^n\) are equivalent. However, reaching this conclusion requires some hoofs to go through. Here is the roadmap.

We first establish that \(\ell_1\) and \(\ell_2\) norms are equivalent.

We then establish that \(\ell_2\) and \(\ell_{\infty}\) norms are equivalent.

We recall the Heine-Borel theorem for Euclidean metric and show that closed and bounded sets of \((\RR^n, \| \cdot \|_2)\) are compact.

We then take advantage of the fact that equivalent norms lead to same topologies (open, closed and compact sets) as well as bounded sets and show that closed and bounded sets of \((\RR^n, \| \cdot \|_1)\) are also compact.

We are then in a position to demonstrate that all norms on \(\RR^n\) are indeed equivalent.

\(\ell_1\) and \(\ell_2\) norms)

(Equivalence ofThe \(\ell_1\) and \(\ell_2\) norms on \(\RR^n\) are equivalent. In particular, for any \(\bx \in \RR^n\)

Alternatively,

Proof. By Cauchy-Schwarz inequality

where \(\bone\) is the vector of all ones. Thus,

Also,

Thus,

Thus, the two norms are equivalent.

\(\ell_2\) and \(\ell_{\infty}\) norms)

(Equivalence ofThe \(\ell_2\) and \(\ell_{\infty}\) norms on \(\RR^n\) are equivalent. In particular, for any \(\bx \in \RR^n\)

Alternatively,

Proof. Let \(\| \bx \|_{\infty} = \sup \{ |x_i|\}\).

Then, \(|x_i| \leq \| \bx \|_{\infty}\).

Thus, \(|x_i|^2 \leq \| \bx \|_{\infty}^2\).

Taking the sum \(\| \bx \|_2^2 \leq n \| \bx \|_{\infty}^2\).

Taking the square root \(\| \bx \|_2 \leq \sqrt{n}\| \bx \|_{\infty}\).

For the other side, we note that

Thus, \(\| \bx \|_{\infty} \leq \| \bx \|_2\). Thus, the two norms are equivalent.

\(\ell_1\), \(\ell_2\), and \(\ell_{|infty}\) norms)

(Equivalence ofThe \(\ell_1\), \(\ell_2\) and \(\ell_{\infty}\) norms on \(\RR^n\) are equivalent.

Proof. We proceed as follows:

By Theorem 4.112, \(\ell_1\) and \(\ell_2\) norms are equivalent.

By Theorem 4.113, \(\ell_2\) and \(\ell_{\infty}\) norms are equivalent.

By Theorem 4.58, equivalence of norms is an equivalence relation.

Hence, by transitivity, \(\ell_1\) and \(\ell_{\infty}\) norms are equivalent.

(Heine Borel theorem)

A subset of the normed linear space \((\RR^n, \| \cdot \|_2)\) is compact if and only if it is a closed and bounded set.

Proof. The distance metric induced by \(\| \cdot \|_2\) is the Euclidean distance.

This result is follows directly from Theorem 3.84.

\(\ell_1\) norm)

(Closed and bounded sets underA subset of the normed linear space \((\RR^n, \| \cdot \|_1)\) is compact if and only if it is a closed and bounded set.

Proof. We just need to show that if a set is closed and bounded in \((\RR^n, \| \cdot \|_1)\), then it is compact in \((\RR^n, \| \cdot \|_1)\).

The norms \(\| \cdot \|_1\) and \(\| \cdot \|_2\) are equivalent (Theorem 4.114).

Hence, the metrics induced by them are (strongly) equivalent (Theorem 4.59).

Thus, the open sets and closed sets in \((\RR^n, \| \cdot \|_1)\) and \((\RR^n, \| \cdot \|_2)\) are identical.

Hence, the compact sets in \((\RR^n, \| \cdot \|_1)\) and \((\RR^n, \| \cdot \|_2)\) are identical (Theorem 3.90).

Also the bounded sets in \((\RR^n, \| \cdot \|_1)\) and \((\RR^n, \| \cdot \|_2)\) are identical due to Theorem 4.57.

Now, let \(A\) be a closed and bounded set in \((\RR^n, \| \cdot \|_1)\).

Then, \(A\) is closed and bounded in \((\RR^n, \| \cdot \|_2)\).

But then by Heine Borel theorem, \(A\) is compact in \((\RR^n, \| \cdot \|_2)\).

But then, \(A\) is compact in \((\RR^n, \| \cdot \|_1)\) also.

(Equivalence of norms on the Euclidean space)

Let \(n \in \Nat\). All norms on \(\RR^n\) are equivalent.

Proof. The \(\ell_1\) norm \(\| \cdot \|_1 : \RR^n \to \RR\) is given by:

We shall show that any norm \(\| \cdot \| : \RR^n \to \RR\) is equivalent to \(\| \cdot \|_1 : \RR^n \to \RR\). Then, since norm equivalence is an equivalence relation (Theorem 4.59), hence all norms are equivalent.

In particular, if \(\| \cdot \|_a\) and \(\| \cdot \|_b\) are two different norms on \(\RR^n\), then \(\| \cdot \|_a \sim \| \cdot \|_1\) and \(\| \cdot \|_b \sim \| \cdot \|_1\) implies that \(\| \cdot \|_a \sim \| \cdot \|_b\) due to Theorem 4.59.

Towards this end, let’s show that any norm \(\| \cdot \|\) is indeed equivalent to \(\| \cdot \|_1\).

We first show that there exists a constant \(c_1 > 0\) such that

Let \(\{ \be_i \}\) be the standard basis for \(\RR^n\).

Let \(c = \max \{ \| \be_i \|, i=1,\dots, n\}\).

Then, for any \(\bx \in \RR^n\), we have

\[\begin{split} \| \bx \| &= \left \| \sum_{i=1}^n x_i \be_i \right \|\\ &\leq \sum_{i=1}^n \| x_i \be_i \| \\ &= \sum_{i=1}^n | x_i| \| \be_i \| \\ &\leq \sum_{i=1}^n | x_i| c \\ &= c \| \bx \|_1. \end{split}\]

We now show that there exists a constant \(c_2 > 0\) such that

Define a function \(g : (\RR^n, \| \cdot \|_1) \to \RR\) as

\[ g(\bx) = \| \bx \| \Forall \bx \in \RR^n. \]Then, for any \(\bx, \by \in \RR^n\),

\[ | g(\bx) - g(\by)| = | \| \bx \| - \| \by \| | \leq \| \bx - \by \| \leq c \| \bx - \by \|_1. \]Thus, \(g\) is Lipschitz continuous on the normed linear space \((\RR^n, \| \cdot \|_1)\).

Therefore, \(g\) is continuous.

Now, let \(S = \{\by \in \RR^n \ST \| \by \|_1 = 1 \}\).

\(S\) is a closed set in \((\RR^n, \| \cdot \|_1)\) since it is the boundary of the unit ball.

\(S\) is also bounded since \(\| \by \|_1 \leq 1\) for every \(\by \in S\).

Then, by Theorem 4.116 \(S\) is compact in \((\RR^n, \| \cdot \|_1)\).

Hence, due to Theorem 3.85 \(g\) attains a minimum value at some \(\by \in S\) over the compact set \(S\).

Let the minimum value of \(g\) over \(S\) be say \(m\) at some \(\by_0 \in S\).

Note that \(\bzero \notin S\) by definition since \(\| \bzero \| \neq 1\).

Thus,

\[ m = g(\by_0) = \| \by_0 \| > 0. \]Thus, for all \(\by \in S\), we have \(\| \by \| \geq m > 0\).

Now, for any nonzero \(\bx \in \RR^n\), the \(\ell_1\) normalized vector, \(\by = \frac{\bx }{\| \bx \|_1} \in S\).

But then

\[\begin{split} & \| \by \| \geq m\\ &\implies \left \| \frac{\bx }{\| \bx \|_1} \right \| \geq m\\ &\implies \| \bx \| \geq m \| \bx \|_1 \\ &\implies \| \bx \|_1 \leq \frac{1}{m} \| \bx \| \end{split}\]holds for every nonzero \(\bx \in \RR^n\).

Also, the inequality \(\| \bx \|_1 \leq \frac{1}{m} \| \bx \|\) is satisfied trivially by \(\bzero\).

We have shown that for \(c_1 = c > 0\) and \(c_2 = \frac{1}{m} > 0\)

holds true for every \(\bv \in \RR^n\).

Thus, the two norms \(\| \cdot \|\) and \(\| \cdot \|_1\) are indeed equivalent.

## 4.7.3. Distances#

(Euclidean distance)

Distance between two vectors is defined as:

This distance function is known as *Euclidean metric*.

This makes \(\RR^n\) a *metric space*.

## 4.7.4. General Euclidean Space#

We can generalize the definition of a Euclidean space to a more abstract case.

(General Euclidean space)

A finite dimensional real vector space \(\VV\)
equipped with an inner product \(\langle \cdot, \cdot \rangle\)
is called a *Euclidean space* if it is endowed
with the norm \(\| \cdot \| : \VV \to \RR\) given by

The norm induced by the inner product is known as
the *Euclidean norm*.

There are several properties emerging from this definition.

Let \(\VV\) be a Euclidean space.

The field of scalars is \(\RR\).

Assume that \(n = \dim \VV\).

The inner product is a real inner product.

\(\VV\) is isomorphic to \(\RR^n\).

If we choose a basis \(\BBB = \{ \be_1, \dots, \be_n \}\) for \(\VV\), then the coordinates for each vector \(\bv \in \VV\) form an element of \(\RR^n\).

This forms a direct bijective mapping between \(\VV\) and \(\RR^n\).

\(\| \bx \|^2 = \langle \bx, \bx \rangle\).

The Euclidean norm makes it a normed linear space.

Recall from Theorem 4.65 that a finite dimensional normed linear space is complete.

Thus, \(\VV\) is a Banach space.

\(\VV\) is an inner product space which is complete.

Hence \(\VV\) is also a Hilbert space.

\(\RR^n\) provides additional features like \(\ell_p\) norms. Corresponding norms can be induced on \(\VV\) by a coordinate mapping.

## 4.7.5. Complex Coordinate Space#

In this section we review important features of \(n\)-dimensional complex coordinate space \(\CC^n\) defined over a field of complex numbers.

(Complex coordinate space)

Let \(\CC\) denote the field of complex numbers.
For any positive integer \(n\), the set of all \(n\)-tuples of
complex numbers forms an \(n\)-dimensional vector space over \(\CC\)
which is denoted as \(\CC^n\) or \((\CC^n, \CC)\) and
sometimes called *complex coordinate space*
or “complex vector space*.

An element \(\bx\) in \(\CC^n\) is written as

where each \(x_i\) is a complex number.

Vector space operations on \(\CC^n\) are defined by:

\(\CC^n\) comes with the standard ordered basis \(\BBB = \{\be_1, \be_2, \dots, \be_n\}\):

We note that the basis is same as the basis for \(n\) dimensional real vector space (the Euclidean space).

An arbitrary vector \(\bx \in \CC^n\) can be written as

### 4.7.5.1. Sesquilinear Inner Product#

(Standard inner product)

Standard inner product on \(\CC^n\) is defined as:

where \(\overline{y_i}\) denotes the complex conjugate.
It is also known as the *sesquilinear inner product*.

This makes \(\CC^n\) an *inner product space*.
This satisfies the inner product rule:

### 4.7.5.2. Standard Norm#

(Norm)

The *length* of the vector (a.k.a. \(\ell_2\) norm) is defined as:

This makes \(\CC^n\) a *normed linear space*.

### 4.7.5.3. Standard Distance#

(Distance)

The standard distance between two vectors is defined as:

This makes \(\CC^n\) a *metric space*.
It reduces to the standard Euclidean distance between real vectors.

### 4.7.5.4. Norms#

In addition to standard norm, we define a family of norms indexed by \(p \in [1, \infty]\) known as \(\ell_p\) norms over \(\CC^n\).

\(\ell_p\) norm is defined as:

We can see that:

### 4.7.5.5. \(\ell_1\) Norm#

From the general definition of \(\ell_p\) norms, we have

We use norms as a measure of strength of a signal or size of an error. Different norms signify different aspects of the signal.

### 4.7.5.6. Quasi-Norms#

In some cases it is useful to extend the notion of \(\ell_p\) norms to the case where \(0 < p < 1\).

In such cases norm as defined in (4.6) doesn’t
satisfy triangle inequality, hence it is not
a proper norm function.
We call such functions as *quasi-norms*.

### 4.7.5.7. \(\ell_0\) “norm”#

Of specific mention is \(\ell_0\) “norm”. It isn’t even a quasi-norm. Note the use of quotes around the word norm to distinguish \(\ell_0\) “norm” from usual norms.

The \(\ell_0\) “norm” is defined as:

where \(\supp(\bx) = \{ i \ST x_i \neq 0\}\) denotes the support of \(\bx\).

Note that \(\| \bx \|_0\) defined above doesn’t follow the definition in (4.6). Yet we can show that:

which justifies the notation.

## 4.7.6. \(\CC^n\) over \(\RR\)#

We next consider the real vector space of complex coordinates.

(Real vector space of complex coordinates)

For any positive integer \(n\), the set of all \(n\)-tuples of complex numbers forms an \(n\)-dimensional vector space over \(\RR\) which is denoted as \((\CC^n, \RR)\)

An element \(\bx\) in \((\CC^n, \RR)\) is written as

where each \(x_i\) is a complex number.

Vector space operations on \((\CC^n, \RR)\) are defined by:

(Standard basis and dimension)

The standard basis for \((\CC^n, \RR)\) is given by \(\BBB = \{\be_1, \be_2, \dots, \be_n, \be_{n+1}, \dots, \be_{2n}\}\):

The dimension of \((\CC^n, \RR)\) is \(2 n\).

It is easy to see that \((\CC^n, \RR)\) is isomorphic to \(\RR^{2 n}\).

### 4.7.6.1. Bilinear Inner Product#

(Bilinear inner product)

The bilinear inner product on \((\CC^n, \RR)\) is defined as:

This inner product satisfies all the requirements
of a real inner product (Definition 4.72)
as shown in Example 4.25.
This makes \((\CC^n, \RR)\) a *real inner product space*.

### 4.7.6.2. Standard Norm#

\((\CC^n, \RR)\))

(Norm onThe *length* of the vector (a.k.a. \(\ell_2\) norm) is defined as:

This makes \((\CC^n, \RR)\) a *normed linear space*.
We note that the definition of the norm for both
\((\CC^n, \RR)\) and \((\CC^n, \CC)\) is identical.