7.6. Expectation

This section contains several results on expectation operator.

Any function \(g(x)\) defines a new random variable \(g(X)\). If \(g(X)\) has a finite expectation, then

\[ \EE [g(X)] = \int_{-\infty}^{\infty} g(x) f_X(x) d x. \]

If several random variables \(X_1, \dots, X_n\) are defined on the same sample space, then their sum \(X_1 + \dots + X_n\) is a new random variable. If all of them have finite expectations, then the expectation of their sum exists and is given by

\[ \EE [X_1 + \dots + X_n] = \EE [X_1] + \dots + \EE [X_n]. \]

If \(X\) and \(Y\) are mutually independent random variables with finite expectations, then their product is a random variable with finite expectation and

\[ \EE (X Y) = \EE (X) \EE (Y). \]

By induction, if \(X_1, \dots, X_n\) are mutually independent random variables with finite expectations, then

\[ \EE \left [ \prod_{i=1}^n X_i \right ] = \prod_{i=1}^n \EE \left [ X_i \right ]. \]

Let \(X\) and \(Y\) be two random variables with the joint density function \(f_{X, Y} (x, y)\). Let the marginal density function of \(Y\) given \(X\) be \(f(y | x)\). Then the conditional expectation is defined as follows:

\[ \EE [Y | X] = \int_{-\infty}^{\infty} y f(y | x) d y. \]

\(\EE [Y | X ]\) is a new random variable.

\[\begin{split} \EE \left [ \EE [Y | X ] \right ] &= \int_{-\infty}^{\infty} \EE [Y | X] f (x) d x\\ &= \int_{-\infty}^{\infty}\int_{-\infty}^{\infty} y f(y | x) f (x) d y d x\\ &= \int_{-\infty}^{\infty}y \left ( \int_{-\infty}^{\infty} f(x, y) d x \right ) d y \\ &= \int_{-\infty}^{\infty} y f(y) d y = \EE [Y]. \end{split}\]

In short, we have

\[ \EE \left [ \EE [Y | X ] \right ] = \EE [Y]. \]

The covariance of \(X\) and \(Y\) is defined as

\[ \Cov (X, Y) = \EE \left [ (X - \EE[X]) ( Y - \EE[Y]) \right ]. \]

It is easy to see that

\[ \Cov (X, Y) = \EE [X Y] - \EE [X] \EE [ Y]. \]

The correlation coefficient is defined as

\[ \rho \triangleq \frac{\Cov (X, Y)}{\sqrt{Var (X) Var (Y)}}. \]

7.6.1. Independent Variables

If \(X\) and \(Y\) are independent, then

\[ \EE [ g_1(x) g_2 (y)] = \EE [g_1(x)] \EE [g_2 (y)]. \]

If \(X\) and \(Y\) are independent, then \(\Cov (X, Y) = 0\).

7.6.2. Uncorrelated Variables

The two variables \(X\) and \(Y\) are called uncorrelated if \(\Cov (X, Y) = 0\). Covariance doesn’t imply independence.