Numerical Characteristics of Random Variables

This article introduces numerical characteristics of random variables such as expectation and variance.

Expectation

Definition

Discrete random variable

Let the probability distribution of the discrete random variable $X$ be $p_i = P\{ X = x_i \}$. If the series

$$\sum x_i p_i$$

If it converges absolutely, its value is called the expectation of $X$, written $EX$.

Continuous random variable

Let the density function of the continuous random variable $X$ be $f(x)$. If the integral

$$\int_{\mathbb{R}} xf(x) \text{d} x$$

When it converges absolutely, this value is the expectation of $X$, denoted $EX$.

Unified definition

Let the distribution function of the random variable $X$ be $F(x)$. If the Stieltjes integral

$$\int_{\mathbb{R}} x \text{d} F(x)$$

Absolute convergence makes this value the expectation of $X$, which we denote by $EX$.

Example where the expectation does not exist

Consider a discrete random variable $X$ with the following distribution:

$$P\left\{ X = (-1)^k \frac{2^k}{k} \right\} = \frac{1}{2^k}, \quad k = 1, 2, \cdots$$

Although the series $\sum x_i p_i$ converges to $- \ln 2$, it is not absolutely convergent, so the expectation of $X$ does not exist.

Now consider a continuous random variable $Y$ with the following density function:

$$f(y) = \frac{1}{\pi} \cdot \frac{1}{1 + y^2}, \quad y \in (-\infty, +\infty)$$

It is easy to verify that the expectation of $Y$ also does not exist.

Properties of expectation

Linearity

If the expectations of random variables $X, Y$ exist, then

• For any real numbers $a, b$, $E(aX + b) = a \cdot EX + b$.
• $E(X + Y) = EX + EY$.

Expectation of the product of random variables

If the expectations of random variables $X$ and $Y$ exist and $X$ and $Y$ are independent, then

$$E(XY) = EX \cdot EY$$

Note that independence is not a necessary condition in the property above.

Counterexample

Consider random variables $X$ and $Y$, where $X$ is uniformly distributed on $[-1, 1]$ and $Y = X^2$.

Converting between expectation and probability

For a random event $A$, consider its indicator function $I_A$:

$$I_A(\omega) = \begin{cases} 1, & \omega \in A \\ 0, & \omega \notin A \end{cases}$$

By definition, its expectation is $EI_A = P(A)$. This conversion is very common in practical applications.

Example

Suppose a sequence of length $n$, $\{ a_i \}$, is given, where $a_k$, with probability $p_k$, takes the value $k$ and, with probability $1 - p_k$, takes the value $0$. Consider how to find the expectation of $S = \sum_{i=1}^{n} a_i$.

Directly applying the definition requires finding the probability of every possible value of $S$, which is rather cumbersome, so the details are omitted here.

Alternatively, let $I_k$ be the indicator function of the random event $a_k = k$. Then

$$S = \sum_{k=1}^{n} k \cdot I_k$$

It is then easy to obtain

$$ES = E \left( \sum_{k=1}^{n} k \cdot I_k \right) = \sum_{k=1}^{n} k \cdot E[I_k] = \sum_{k=1}^{n} k \cdot p_k$$

Conditional distribution and conditional expectation

We have previously studied conditional probability; similarly, we can introduce the concept of conditional expectation.

Definition

For two random variables $X$ and $Y$, conditional on $Y = y$, the probability distribution (density function) of $X$ is called the conditional probability distribution (conditional probability density), denoted respectively by

$$P( X = x_i | Y = y ) \qquad f_{X|Y}(x|y)$$

Under this condition, the expectation of $X$ is called the conditional expectation, denoted by $E[X|Y=y]$.

Properties of conditional expectation

Many properties of conditional expectation follow from conditional probability and will not be elaborated here.

It is worth noting that $E[X | Y]$ is generally a function of the random variable $Y$, and this function is usually not linear. However, we do have

$$E[E[X|Y]] = EX$$

The equation above is called the law of total expectation.

Application

HDU 5984 Pocky

There is a Pocky stick of length $L$, which is randomly broken into two pieces each time. If the length of the piece on the right is no greater than $d$, stop; otherwise, repeat the process on the piece on the right. Find the expected number of repetitions.

Solution

Let $f(x)$ denote the expected number of repetitions for a piece of length $x$. The case $x \leq d$ is trivial.

When $x > d$, let the distance from the break point to the right endpoint be $k$. Clearly, $k \sim U[0, x]$, and the expected number of repetitions at this point is

$$g(k) = \begin{cases} 1, & k \leq d \\ 1 + f(k), & k > d \end{cases}$$

By the law of total expectation,

$$f(x) = Eg(k) = 1 + \frac{1}{x} \cdot \int_{d}^{x} f(t) \text{d} t$$

Solving the integral equation above and substituting the initial condition gives

$$f(x) = 1 + \ln \frac{x}{d}$$

Variance

Definition

Let a random variable $X$ have an existing expectation $EX$, and suppose that the expectation

$$E(X - EX)^2$$

also exists. Then its value is called the variance of the random variable $X$, denoted by $DX$ or $Var(x)$. The arithmetic square root of the variance is called the standard deviation, denoted by $\sigma(X) = \sqrt{DX}$.

Properties of variance

If the variance of a random variable $X$ exists, then

• For any constants $a, b$, $D(aX + b) = a^2 \cdot DX$.
• $DX = E(X^2) - (EX)^2$.

Covariance and correlation coefficient

In general, the equality $D(X + Y) = DX + DY$ does not hold. This naturally raises two questions:

• What exactly is the difference between $D(X + Y)$ and $DX + DY$?
• Under what conditions are $D(X + Y)$ and $DX + DY$ equal?

We introduce covariance to answer the first question.

Definition of covariance

For random variables $X, Y$,

$$E((X - EX)(Y - EY))$$

is called the covariance of $X$ and $Y$, denoted by $\operatorname{Cov}(X, Y)$.

Properties of covariance

For random variables $X, Y, Z$, we have

• $\operatorname{Cov}(X, Y) = \operatorname{Cov}(Y, X)$
• For any constants $a, b$, $\operatorname{Cov}(aX + bY, Z) = a \cdot \operatorname{Cov}(X, Z) + b \cdot \operatorname{Cov}(Y, Z)$

Covariance and variance are also related as follows:

• $DX = \operatorname{Cov}(X, X)$
• $D(X + Y) = DX + 2 \operatorname{Cov}(X, Y) + DY$

About covariance

You may notice that the properties of covariance are highly consistent in form with the properties of the inner product of vectors.

From the perspective of functional analysis, for a given probability space, all random variables on it form a linear space, while covariance is an inner product on this space, and the standard deviation is the norm induced by this inner product.

For the second question raised above, it is easy to see that $D(X + Y) = DX + DY$ if and only if $\operatorname{Cov}(X, Y) = 0$. A natural necessary condition is that $X$ and $Y$ are independent, because then

$$\operatorname{Cov}(X, Y) = E((X - EX)(Y - EY)) = E(X - EX) E(Y - EY) = 0$$

However, this condition is not sufficient. To describe the relationship characterized by $\operatorname{Cov}(X, Y) = 0$ between random variables $X$ and $Y$, we introduce the correlation coefficient.

Correlation coefficient

For random variables $X, Y$,

$$\frac{ \operatorname{Cov}(X, Y)}{ \sigma(X)\sigma(Y) }$$

is called the Pearson correlation coefficient of $X$ and $Y$, denoted by $\rho_{X,Y}$.

The Pearson correlation coefficient describes the strength of the linear association between two random variables. The larger $|\rho_{X,Y}|$ is, the stronger the linear association between $X$ and $Y$. It is not difficult to prove that $|\rho_{X,Y}| \leq 1$, and $|\rho_{X,Y}| = 1$ can occur only in the following two cases:

• If there are a real number $a$ and a positive real number $b$ such that $P(X = a + bY) = 1$, then $\rho_{X,Y} = 1$.
• If there are a real number $a$ and a negative real number $b$ such that $P(X = a + bY) = 1$, then $\rho_{X,Y} = -1$.

When $\rho_{X,Y} = 0$, we say that random variables $X$ and $Y$ are uncorrelated; in this case, there is no linear relationship between $X$ and $Y$.

“Uncorrelated” and “independent”

Two random variables being uncorrelated only means that there is no linear association between them; it does not mean that there are no other forms of dependence.

Therefore, the statement that two random variables $X, Y$ are uncorrelated is a necessary but not sufficient condition for them to be independent.

For the second question mentioned at the beginning of this section, we conclude that the necessary and sufficient condition for $\operatorname{Cov}(X, Y) = 0$ is that either one of $X$ and $Y$ is constant with probability $1$, or that $X, Y$ are uncorrelated.