Context

Next, let \(X\) be a random variable on \((\mathbb{R}, \mathcal{B}, \mathbb{P})\) , then the cumulative distribution function of \(X\) is given by

\[F(x) = \mathbb{P}(X \in (-\infty, x]). \tag{2.1}\]

We will assume that there are no point masses for \(F\) . The cumulative distribution function captures all of the distributional information about the random variable \(X\) ; e.g.,

\[\begin{aligned}\mathbb{P}(X \in (a, b]) &= \mathbb{P}(X \le b) - \mathbb{P}(X \le a) = F(b) - F(a), \\ \mathbb{P}(X > a) &= 1 - \mathbb{P}(X \le a) = 1 - F(a),\end{aligned}\]

and if \(X\) is symmetric about 0, for instance, and \(a > 0\) , \(\mathbb{P}(|x| > a) = 2 \cdot F(-a)\) . In this case we also have

\[\begin{aligned}F(0) &= \frac{1}{2}, \\ F(x) &= 1 - F(-x).\end{aligned}\]

Notice, too, that we may consider the impact of shift and scale on the cumulative distribution function; e.g., let \(Y = aX + b\) for \(a > 0\) and \(b \in \mathbb{R}\) , and let \(F_Y\) and \(F_X\) be the cumulative distribution functions of \(Y\) and \(X\) , respectively, then

\[\begin{aligned}F_Y(y) &= \mathbb{P}(Y < y) \\ &= \mathbb{P}(aX + b < y) \\ &= \mathbb{P}\left(X < \frac{y-b}{a}\right) \\ &= F_X\left(\frac{y-b}{a}\right).\end{aligned}\]

The case with \(a < 0\) is treated similarly.

We say that \(X\) has a probability density function, \(f: \mathbb{R} \to \mathbb{R}_{\ge 0}\) , if

\[F(x) = \int_{-\infty}^{x} f(s) ds. \tag{2.2}\]

At times we will simply refer to \(f\) as the density function or pdf of \(X\) . We note immediately that for \(f\) to be a density function, it is necessary that \(f\) is nonnegative, integrable, and satisfies

\[\int_{-\infty}^{\infty} f(s) ds = 1.\]

For \(X\) with a probability density function, the expected value of \(X\) , denoted \(\mathbb{E}(X)\) , is given by

\[\mathbb{E}(X) = \int_{-\infty}^{\infty} sf(s)ds. \quad (2.3)\]

Notice that we have not precluded infinite expectations so that \(\mathbb{E}(X) \in \mathbb{R} \cup \infty\) . We will often call the expected value of \(X\) the mean and denote it by \(\mu\) .

The variance of \(X\) , \(\text{Var}(X)\) , is found by integrating the squared distance from the mean according to the density \(f\) . That is,

\[\text{Var}(X) = \int_{-\infty}^{\infty} (s - \mu)^2 f(s)ds \quad (2.4)\]

if \(\mu\) is finite, otherwise \(\text{Var}(X)\) is infinite as well. Here again, then, we have not disallowed infinite variances. For reasons that will become clear shortly, we routinely denote the \(\text{Var}(X)\) as \(\sigma^2\) .

We may also define variance without resort to the density function of \(X\) , via

\[\text{Var}(X) = \mathbb{E}((X - \mu)^2), \quad (2.5)\]

which gives, after some manipulation, that

\[\text{Var}(X) = \mathbb{E}(X^2) - \mu^2.\]

Variance indicates some level of dispersion of the random variable \(X\) . The units of the mean and variance of \(X\) are not the same, however. Oftentimes we will want to construct statistics relating the mean to a dispersion metric. In this vein, the standard deviation, or volatility is defined as the square root of variance, \(\sigma\) .

The mean and standard deviation are often described as position and dispersion parameters, respectively. They are also known as the first and second moments of the distribution.

Considering once more the impact of shift and dilation to the expected value and variance of a random variable, \(X\) , we have that for any scalars \(a\) and \(b\) ,

\[\mathbb{E}(aX + b) = a\mathbb{E}(X) + b \quad (2.6)\]

\[\text{Var}(aX + b) = a^2 \text{Var}(X). \quad (2.7)\]

Each of these results may be quickly derived using the probability density function approach (the exercise is left to the reader). We shall see later that the expectation operator is linear in combinations of random variables, and the generalization of variance, covariance, is a bilinear form. Here we simply note that the mean is scale and shift sensitive, while variance is independent of changes in position.

We noted that both the mean and variance of a random variable, \(X\) , can be infinite. For a practical application, this is a disappointing feature. Especially, as we will see later, if we intend to understand risk via such metrics. An alternate approach is to look at the percentiles of the distribution of \(X\) . The (100 ·

\(p\) )th percentile of a random variable with cumulative density function \(F\) is the smallest number \(\pi_p\in\mathbb{R}\) satisfying \(F(\pi_p)=p\) . That is, \(\pi_p\) satisfies

\[\pi_p=\min\{\pi|F(\pi)=p\}.\] (2.8)

where we take the minimum of the set of all values satisfying our criterion. If \(F\) is invertible in a neighborhood about \(\pi_p\) , then we may solve for the \(p\) th percentile by solving \(\pi_p=F^{-1}(p)\) .

Alternative extensions to better capture the empirical distribution of returns have looked at higher moments. In particular, for a random variable \(X\) with mean \(\mu\) and variance \(\sigma^2\) , skew is defined as

\[\mathbb{E}\left(\frac{(X-\mu)^3}{\sigma^3}\right),\] (2.9)

and kurtosis is defined by

\[\mathbb{E}\left(\frac{(X-\mu)^4}{\sigma^4}\right).\] (2.10)

We will denote skew and kurtosis by \(\gamma\) and \(\kappa\) , respectively.

For symmetric \(X\) , skew is zero (consider the oddness of the integrand if a density is given). Generalizing this observation, skew measures a degree of asymmetry in the distribution of \(X\) ; viz., if the left tail of \(X\) has more (less) weight than the right, \(X\) has negative (positive) skew.

Kurtosis is a measure to compare tail densities across random variables. It is theoretically deficient in its usual application which often takes kurtosis as a measure of ‘tailedness.’ Even so, it provides a metric which accounts for extreme observations. This will be especially important when comparing empirical distributions to the densities that have wide application in theory and practice.

We proceed to a few of these example densities.

2.1.1 Normal Distribution

The normal density, or the density function for a normal random variable, is a two parameter function \(\phi_{\mu,\sigma^2}(\cdot)\) given by

\[\phi_{\mu,\sigma^2}(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(\frac{-(x-\mu)^2}{2\sigma^2}\right). \quad (2.11)\]

If a random variable, \(X\) , has the above density, we say that it is normal, and denote this by \(X \sim N(\mu, \sigma^2)\) . The cumulative distribution of a normal random variable will be denoted \(\Phi_{\mu,\sigma^2}\) .

The usage of \(\mu\) and \(\sigma\) above is not a coincidence.

Example 2.1.1. The expected value of \(X\) with density \(\phi_{\mu,\sigma^2}\) is \(\mu\) .

We apply the integral in (2.3) directly to see

\[\mathbb{E}(X) = \frac{1}{\sqrt{2\pi\sigma^2}} \int_{-\infty}^{\infty} x \exp\left(\frac{-(x-\mu)^2}{2\sigma^2}\right) dx,\]

a change of variables using \(u = \frac{x-\mu}{\sigma}\) gives

\[\begin{aligned} \mathbb{E}(X) &= \frac{1}{\sqrt{2\pi\sigma^2}} \int_{-\infty}^{\infty} (\sigma^2 u \exp(-u^2/2) + \sigma\mu \exp(-u^2/2)) du \\ &= \frac{1}{\sqrt{2\pi\sigma^2}} \left( \int_{-\infty}^{\infty} \sigma^2 u \exp(-u^2/2) du + \int_{-\infty}^{\infty} \sigma\mu \exp(-u^2/2) du \right). \end{aligned}\]

Now, the first integrand is odd, and hence integrates to zero over \(\mathbb{R}\) . Therefore

\[\begin{aligned} \mathbb{E}(X) &= \frac{1}{\sqrt{2\pi\sigma^2}} \int_{-\infty}^{\infty} \sigma\mu \exp(-u^2/2) du \\ &= \mu \cdot \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} \exp(-u^2/2) du. \end{aligned}\]

We know from calculus that a change of variables to polar coordinates is all it takes to show that

\[\int_{-\infty}^{\infty} \exp(-u^2/2) du = \sqrt{2\pi},\]

so that finally we arrive at

\[\mathbb{E}(X) = \mu.\]

The variance is found to be \(\sigma^2\) using integration by parts, and is left as an exercise.

The normal distribution is clearly symmetric about \(\mu\) , and hence the median is also \(\mu\) . When \((\mu, \sigma^2) = (0, 1)\) , we say that \(X\) is the standard normal; we will denote the probability density and cumulative distribution functions of the standard normal by \(\phi\) and \(\Phi\) , respectively, dropping the subscripts of the general case.

Example 2.1.2. Consider the random variable \(Y\) defined by

\[Y = \sigma X + \mu,\]

with \(X\) a standard normal random variable. Then \(Y\) is a normal random variable with mean \(\mu\) , and standard deviation \(\sigma\) .

We know immediately from (2.6) and (2.7) that \(\mathbb{E}(Y) = \mu\) and \(\text{Var}(Y) = \sigma^2\) . We should also verify that the cumulative density function of \(Y\) is in fact the normal cumulative density. We have

\[\begin{aligned}F_Y(y) &= \mathbb{P}(Y < y) \\&= \mathbb{P}(\sigma X + \mu < y) \\&= \mathbb{P}\left(X < \frac{y - \mu}{\sigma}\right) \\&= F_X\left(\frac{y - \mu}{\sigma}\right).\end{aligned}\]

Or that \(F_Y(y) = \Phi\left(\frac{y - \mu}{\sigma}\right)\) . This is actually sufficient to show that \(Y\) is normally distributed, but we utilize the density function, \(\phi\) , to make this more evident. We have that

\[\begin{aligned}F_Y(y) &= \Phi\left(\frac{y - \mu}{\sigma}\right) \\&= \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\frac{y - \mu}{\sigma}} \exp\left(\frac{-s^2}{2}\right) ds \\&= \left(\text{changing variables } x = \sigma s + \mu, \frac{1}{\sigma} dx = ds\right) \\&= \frac{1}{\sqrt{2\pi\sigma^2}} \int_{-\infty}^{y} \exp\left(\frac{-(x - \mu)^2}{2\sigma^2}\right) dx,\end{aligned}\]

where the density function is now clearly seen to be \(\phi_{\mu, \sigma^2}\) , and so \(Y \sim N(\mu, \sigma^2)\) .

The above technique is not restricted to normal random variables. For a random variables with finite mean and variance the transformation

\[X \mapsto \frac{X - \mu}{\sigma},\]

Table 2.1: Probability table for various standard deviation ranges for a normally distributed random variable

will always produce a zero mean and unit standard deviation random variable. Such normalization is often employed to create scaled variables in modeling.

The skew and kurtosis of a normal random variable are 0, and 3 respectively. The somewhat odd appearance of the number 3 in a math text leads some authors to define kurtosis as \(\kappa - 3\) . With this modification, a normal random variable has zero kurtosis. We do not adopt this convention.

The normal distribution is (for many reasons) the de facto reference distribution. And if not the normal distribution, its close cousin the log-normal distribution. This is in spite of its lack of applicability in some scenarios. In particular, if we recall David Viniar, the CFO of Goldman Sachs during the Quant Crisis from the introduction [15], we may ask ourselves what the likelihood of moves observed during that particular crisis would be under the assumption of normality.

In the case of normally distributed variables, we need only look at the number of standard deviations from the mean to determine likelihood. This is evidenced in the exponential in the probability density function in (2.3) where, in fact, the motivation for normalization is straightforward. Table 2.1.1 shows various probabilities obtained by calculating \(\Phi(k\sigma) - \Phi(-k\sigma)\) for various levels of \(k\) .

We may interpret some of this data in the following way: for a normally distributed random variable, seeing a four standard deviation observation is a 1 in 10,000 event. Looking at the table, it is clear that these probabilities are not linear in \(\sigma\) . A six sigma event, for instance, is a one in five hundred million event. Put in perspective, a six sigma event would occur one day every 1,388,455 million years. Five times longer than the appearance of anatomically modern humans in Africa.

Now, what about a 25 sigma event? The probability of such an event is \(6.10 \times 10^{-138}\) . In terms of years, this would occur one day in \(4.49 \times 10^{132}\) years. Here’s the thing: the universe is only 13.82 billion years old, so that you shouldn’t see this type of event but once across \(3.2 \times 10^{125}\) universes [16, 17].

Put another way, the surface area of the earth is 510 trillion \(\text{m}^2\) . Approximating a human hand at \(0.01 \text{ m}^2\) , it would be \(3.2 \times 10^{118}\) times more likely to catch a random ball dropped from space in the palm of your hand than to see a twenty five sigma event.

Returning to the cosmological scale, an upper bound for the number of particles in the universe is \(1.0e85\) . Thus you’d be more likely to pick a single

particle from the universe than to witness a 25 sigma event. Much, much, much more likely.

We mentioned in the first chapter that during the Quant Crisis, David Viniar, the CFO of Goldman Sachs, was quoted as saying they were observing 25-standard deviation moves, several days in a row. Connecting this statement to the probabilities just calculated is not done in a pejorative sense. More, it is likely that they had very carefully constructed a very stable and attractive product that was blown out by overcrowding and liquidations. Unfortunately we: 1) may find ourselves inherently referring to the normal distribution and its shortcuts of standard deviation explaining everything even when this is inappropriate; 2) tend to think linearly so that a 25 standard deviation feels like six times a four standard deviation move via some heuristics; and 3) do not fully understand the dynamics of equity returns, which are subject to crises and other exogenous effects.

The normal distribution, even with the above in mind, has extremely nice properties, not the least of which is that it is overwhelmingly the most likely distribution to assume if you are only given the mean and standard deviation of a random variable. The analogy would be that the normal distribution acts as our first linear approximation to the underlying distributions that we may never know. This does not preclude its use as a tool, but should give us a moment to reflect, especially if leverage is determined via risk estimates based on its use.

2.1.2 Log-normal Distribution

The density function for a log-normal random variable, is again a two parameter function \(ln_{\mu,\sigma^2}(\cdot)\) given by

\[ln_{\mu,\sigma^2}(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \frac{1}{x} \exp\left(-\frac{(\log(x) - \mu)^2}{2\sigma^2}\right), \quad (2.12)\]

and is clearly only defined on \(\mathbb{R}_+\) . A log-normal random variable with parameters \(\mu\) and \(\sigma^2\) is such that its logarithm is distributed as \(N(\mu, \sigma^2)\) , and we will denote it by \(LN(\mu, \sigma^2)\) . That is, \(Y\) is log-normal if \(\log(Y)\) is normal.

Example 2.1.3. In a manner similar to before we may obtain the cumulative distribution function for \(Y \sim LN(\mu, \sigma^2)\) ; viz.,

\[\begin{aligned} F_Y(y) &= \mathbb{P}(Y < y) \\ &= \mathbb{P}(\exp(X) < y) \\ &= \mathbb{P}(X < \log(y)) \\ &= F_X(\ln(y)), \end{aligned}\]

so that \(F_Y(y) = \Phi_{\mu,\sigma^2}(\ln(y)) = \Phi\left(\frac{\log(y)-\mu}{\sigma}\right)\) .

From here, we may also derive the density function for \(Y \sim LN(\mu, \sigma^2)\) as

\[\begin{aligned}F_Y(y) &= \Phi\left(\frac{\log(y) - \mu}{\sigma}\right) \\&= \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\frac{\log(y) - \mu}{\sigma}} \exp\left(-\frac{s^2}{2}\right) ds \\&= \left(\text{changing variables } s = \frac{\log(x) - \mu}{\sigma}, \frac{1}{\sigma \cdot x} dx = ds\right) \\&= \frac{1}{\sqrt{2\pi\sigma^2}} \int_0^y \frac{1}{x} \exp\left(-\frac{(\log(x) - \mu)^2}{2\sigma^2}\right) dx,\end{aligned}\]

which is exactly the density given in (2.12), as desired.

We shall see that the above technique generalizes, allowing us to derive density functions of random variables constructed from functions of random variables. We will call this a change of variables theorem – perhaps not surprisingly given the repeated application above.

The mean and variance of \(Y \sim LN(\mu, \sigma^2)\) are given by

\[\begin{aligned}\mathbb{E}(Y) &= \exp\left(\mu + \frac{1}{2}\sigma^2\right) \\Var(Y) &= \exp\left(2\mu + \sigma^2\right) \left(\exp\left(\sigma^2\right) - 1\right),\end{aligned}\]

and their derivations are left as exercises.

The log-normal distribution is terribly important, due in large part to the Black-Scholes option pricing formula which relies on the assumption of log-normality of future stock prices over any interval. For a stock price, a small increment in time \(\delta\) , and initial and terminal times \(t\) and \(T = t + \delta\) , respectively, \(\ln(S_T)\) is distributed as

\[\ln(S_{t+\delta}) \sim N\left(\ln(S_t) + \left(\mu - \frac{\sigma^2}{2}\right)\delta, \sigma^2\delta\right).\]

A related result is that so-called log returns,

\[r_t = \ln\left(\frac{S_{t+\delta}}{S_t}\right)\]

are normally distributed as

\[r_t \sim N\left(\left(\mu - \frac{\sigma^2}{2}\right)\delta, \sigma^2\delta\right).\]

2.1.4 Functions of Random Variables

In our previous examples, we encountered a few examples of random variables that were functions of random variables; e.g., \(Y = e^X\) and \(Y = \mu + \sigma X\) , for \(X\) with a known distribution. In determining the cumulative distribution and density functions of these transformed random variables, we began with a manipulation of the CDF to a known distribution and proceeded to analyze the probability density function and integration bounds obtained. We then used a change of variables to determine the density of the transformed variable. Here we formalize the procedure, developing the change of variable theorem.

Theorem 2.1.1. If \(g: \mathbb{R} \to \mathbb{R}\) is continuous and monotonically increasing and \(X \in \mathbb{R}\) is a random variable, then the random variable \(Y = g(X)\) defined by setting \(y = g(x)\) for every realization of \(X\) has the cumulative distribution function

\[F_Y(y) = F_X(g^{-1}(y)), \quad (2.14)\]

and if \(X\) has a density, then the density of \(Y\) is given by

\[f_Y(y) = \frac{dg^{-1}(y)}{dy} f_X(g^{-1}(y)). \quad (2.15)\]

Proof: We know by the inverse function theorem that a local inverse of \(g\) exists, hence for any \(y\) , we may proceed as above,

\[\begin{aligned} F_Y(y) &= \mathbb{P}(Y < y) \\ &= \mathbb{P}(g(X) < y) \\ &= \mathbb{P}(X < g^{-1}(y)) \\ &= F_X(g^{-1}(y)). \end{aligned}\]

From here we may relate the density function of \(X\) to the density function of \(Y\) . We get

\[\begin{aligned} F_Y(y) &= F_X(g^{-1}(y)) \\ &= \int_{-\infty}^{g^{-1}(y)} f_X(s) ds \\ &\quad \text{(changing variables } s = g^{-1}(u), \frac{dg^{-1}(u)}{du} du = ds, \text{ by the chain rule)} \\ &= \int_{-\infty}^{y} \frac{dg^{-1}(u)}{du} f_X(g^{-1}(u)) du \end{aligned}\]

as desired. Notice that we took advantage of the monotonicity of \(g\) in our change of bounds, with \(s = g^{-1}(y)\) implying \(g^{-1}(u) = g^{-1}(y)\) , and hence \(u = y\) .

We may also relax the original condition to require only that \(g\) is monotonic (and not necessarily increasing). Working through the proof in the same manner, but allowing for monotonically decreasing \(g\) gives

\[f_Y(y) = \left| \frac{dg^{-1}(y)}{dy} \right| f_X(g^{-1}(y)). \quad (2.16)\]

2.2 Multivariate Distributions

We may generalize our work to include vectors of random variables,

\[X = \begin{pmatrix} X_1 \\ X_2 \\ \vdots \\ X_N \end{pmatrix}.\]

The cumulative distribution function then also generalizes to the joint cumulative distribution function of \(X\) , a function \(F: \mathbb{R}^N \to [0, 1]\) which satisfies

\[F(x_1, x_2, \dots, x_N) = \mathbb{P}(X_1 \le x_1, X_2 \le x_2, \dots, X_N \le x_N) \quad (2.17)\]

We say that the multivariate random variable, \(X\) , has a density,

\[f: \mathbb{R}^N \to \mathbb{R}_+,\]

if we may write

\[F(x_1, x_2, \dots, x_N) = \int_{-\infty}^{x_1} \cdots \int_{-\infty}^{x_N} f(s_1, \dots, s_N) ds_N \cdots ds_1. \quad (2.18)\]

As before, we require that \(f\) be integrable, nonnegative, and integrate to one.

The expected value of the multivariate random variable \(X\) , \(\mathbb{E}(X)\) , is given component wise

\[\mathbb{E}(X) = \begin{pmatrix} \mathbb{E}(X_1) \\ \mathbb{E}(X_2) \\ \vdots \\ \mathbb{E}(X_N) \end{pmatrix}, \quad (2.19)\]

and the generalization of variance, covariance, denoted \(\text{Cov}(X)\) is determined by

\[\text{Cov}(X) = \mathbb{E}((X - \mu)(X - \mu)'), \quad (2.20)\]

where \(X \in \mathbb{R}^N\) , \(\mu = \mathbb{E}(X)\) , and expectation is again componentwise in \(\text{Cov}(X) \in \mathbb{R}^{N \times N}\) . We will denote the \((i, j)\) component of \(\text{Cov}(X)\) as \(\sigma_{ij}\) , with \(\sigma_{ii} = \text{Var}(X_i)\) .

We will often denote the covariance matrix by \(\Sigma\) , and we will spend considerable time analyzing its properties both from a theoretical as well as empirical point of view (and oftentimes settling on the intersection of the two).

Example 2.2.1. The expectation operator is linear; viz.,

\[\mathbb{E}(w_1X_1 + w_2X_2) = w_1\mathbb{E}(X_1) + w_2\mathbb{E}(X_2). \quad (2.21)\]

for random variables, \(X_1\) and \(X_2\) , and scalars \(w_1\) and \(w_2\) . This generalizes to

\[\mathbb{E}\left(\sum_{i=1}^{N} w_i X_i\right) = \sum_{i=1}^{N} w_i \mathbb{E}(X_i).\]

We prove the result assuming that there exists a joint density function, \(f(\cdot)\) , but this is not necessary. We have in this case

\[\begin{aligned}\mathbb{E}(w_1X_1 + w_2X_2) &= \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} (w_1X_1 + w_2X_2) f(x_1, x_2) dx_1 dx_2 \\ &\quad \text{(by the linearity of the integral operator)} \\ &= w_1 \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} X_1 f(x_1, x_2) dx_1 dx_2 + w_2 \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} X_2 f(x_1, x_2) dx_1 dx_2 \\ &= w_1 \mathbb{E}(X_1) + w_2 \mathbb{E}(X_2)\end{aligned}\]

as desired.

Example 2.2.2. The covariance matrix, \(\Sigma\) is positive semi-definite. We say that a matrix, \(A \in \mathbb{R}^{N \times N}\) , is positive semi-definite if for all \(w \in \mathbb{R}^N\) not identically zero,

\[w'Aw \ge 0, \quad (2.22)\]

with \(w'Aw = 0\) only when \(w \equiv 0\) .

Given a multivariate random variable, \(X \in \mathbb{R}^N\) , with \(\text{Cov}(X) = \Sigma\) , and a scalar vector \(w \in \mathbb{R}^N\) , the result follows by considering the variance of the random variable \(Y = w'X = \sum_{i=1}^N w_i X_i\) ; viz.,

\[\begin{aligned}\text{Var}(Y) &= \text{Var}(w'X) \\ &= \mathbb{E}\left((w'X - w'\mu)^2\right) \\ &= \mathbb{E}\left((w'(X - \mu))^2\right)\end{aligned}\]

where \(\mu = \mathbb{E}(X)\) . Continuing, we have by the linearity of the expectation operator that

\[\begin{aligned}\text{Var}(Y) &= \mathbb{E}(w'(X - \mu)(X - \mu)'w) \\ &= w'\mathbb{E}((X - \mu)(X - \mu)')w \\ &= w'\Sigma w.\end{aligned}\]

Now, we know from the definition of variance that \(\text{Var}(Y) \ge 0\) . The above relationship then implies \(w'\Sigma w \ge 0\) as well, proving the result.

We will make stronger claims about the covariance matrix, \(\Sigma\) , in future chapters, refining our assumptions about the composition of \(X\) to ensure that

\(\Sigma\) is positive definite; that is, inner products with nonzero vectors are strictly nonzero. The proof will be identical, with only a few comments added.

We may also consider linear combinations of the form \(BX\) for a multivariate random variable \(X\in\mathbb{R}^N\) and scalar matrix \(B\in\mathbb{R}^{M\times N}\) . We maintain linearity in expectation; viz.,

\[\mathbb{E}(BX)=B\mathbb{E}(X). \tag{2.23}\]

This can be shown directly since

\[BX=\begin{pmatrix} \sum_{i=1}^N b_{1i}X_i \\ \vdots \\ \sum_{i=1}^N b_{Mi}X_i \end{pmatrix}\]

so that

\[\mathbb{E}(BX)=\begin{pmatrix} \mathbb{E}\left(\sum_{i=1}^N b_{1i}X_i\right) \\ \vdots \\ \mathbb{E}\left(\sum_{i=1}^N b_{Mi}X_i\right) \end{pmatrix}=\begin{pmatrix} \sum_{i=1}^N b_{1i}\mathbb{E}\left(X_i\right) \\ \vdots \\ \sum_{i=1}^N b_{Mi}\mathbb{E}\left(X_i\right) \end{pmatrix}=B\mathbb{E}(X).\]

Similarly, \(\mathbb{E}(XB)=\mathbb{E}(X)B\) for scalar \(B\) with appropriate dimensions.

The covariance of a \(BX\) as above, \(Cov(BX)\) , is given by

\[Cov(BX)=BCov(X)B'. \tag{2.24}\]

This follows from the linearity of expectation since

\[\begin{aligned} Cov(BX) &= \mathbb{E}\left(\left(BX-\mathbb{E}(BX)\right)\left(BX-\mathbb{E}(BX)\right)'\right) \\ &= \mathbb{E}\left(B\left(X-\mathbb{E}(X)\right)\left(X-\mathbb{E}(X)\right)'B'\right) \\ &= B\mathbb{E}\left(\left(X-\mathbb{E}(X)\right)\left(X-\mathbb{E}(X)\right)'\right)B' \\ &= BCov(X)B'. \end{aligned}\]

Finally, if \(X\) and \(Y\) are univariate random variables taking their values in \(\mathbb{R}\) , we define

\[Cov(X,Y)=\mathbb{E}\left((X-\mu_X)(Y-\mu_Y)\right). \tag{2.25}\]

In terms of previous notation, then, if \(X\in\mathbb{R}^N\) , \(Cov(X_i,X_j)=\sigma_{ij}\) .

We may also consider the effect of arbitrary (continuous and injective) transformations on the distribution of a random variable, generalizing the change of variable theorem to the multivariate case.

Theorem 2.2.1. Let \(X\) be a multivariate random variable with \(X\in\mathbb{R}^N\) , and let

\[g:\mathbb{R}^N\rightarrow\mathbb{R}^N\]

be a one-to-one and continuous function. If \(f_X(\cdot)\) is the density of \(X\) , then the density of

\[Y=g(X)\]

is

\[f_Y(y) = f_X(g^{-1}(y)) \cdot \det(\nabla g^{-1}(y)) \quad (2.26)\]

where \(\nabla h\) denotes the Jacobian of a function \(h\) , and \(\det(\cdot)\) is the determinant function. The Jacobian, determinant, and change of variables in multivariate integration are discussed in the appendix.

The proof is exactly as before, accounting for the fact that we are working in \(\mathbb{R}^N\) and hence encounter the Jacobian rather than the derivative of the inverse of \(g\) .

The marginal distribution of \(X_j\) is denoted as in the univariate case as

\[F_j(x_j) = \mathbb{P}(X_j \le x_j)\]

and is given by

\[F(-\infty, \dots, x_j, \dots, \infty), \quad (2.27)\]

or in the case of densities, all upper bounds of integration are infinite except that related to the \(j\) th component, which is just \(x_j\) . When a density exists,

\[f_j(x) = \int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} f(s_1, \dots, s_{j-1}, x, s_{j+1}, \dots, s_N) ds_N \cdots ds_{j+1} ds_{j-1} \cdots ds_1 \quad (2.28)\]

is the \(j\) th marginal density.

2.2.1 Multivariate Normal Distribution

The multivariate normal density, or the density function for a multivariate normal random variable, is a two parameter function, \(\phi_{\mu,\Sigma}(\cdot)\) , given by

\[\phi_{\mu,\Sigma}(x)=\frac{1}{(2\pi)^{\frac{N}{2}}}\frac{1}{\det(\Sigma)^{\frac{1}{2}}}\exp\left(-\frac{1}{2}(x-\mu)'\Sigma^{-1}(x-\mu)\right) \quad (2.34)\]

for \(x\in\mathbb{R}^N\) .

As before, if a random variable, \(X\) , has the above density, we say that it is multivariate normal, and denote this by \(X\sim N(\mu,\Sigma)\) . The cumulative distribution of a normal random variable will be denoted as before by \(\Phi_{\mu,\Sigma}\) . We state without proof that in such a case

\[\begin{aligned}\mathbb{E}(X) &= \mu \\ \text{Cov}(X) &= \Sigma.\end{aligned}\]

Implicit in the definition is that \(\Sigma\) is invertible, and therefore, since \(\Sigma\) is a covariance matrix and hence positive semidefinite, we must have that \(\Sigma\) is positive definite.

We also note that linear combinations of jointly normal random variables are again normal. In the context of portfolio management, if the log returns, say, of various assets are assumed to be jointly normal, then a portfolio (that is a weighted sum) of these log returns will also be normal, with tractable mean and variance. Some care is being taken here to say ‘jointly normal.’ This is due to the fact that it is possible for marginal distributions to be normal without the joint distribution being multivariate normal. We are precluding this possibility.

We have in the general case the if \(X\) is multivariate normal with \(X\sim N(0,I)\) , for \(I\) the identity matrix, \(0\) a vector of zeros, and \(X\in\mathbb{R}^N\) , then for \(a\in\mathbb{R}^N\) and nonsingular \(B\in\mathbb{R}^{N\times N}\) , the random variable \(Y=a+BX\) is distributed as \(Y\sim N(a,BB')\) . We may employ the change of variables theorem for the multivariate case, but we choose to follow the same method as was seen in the univariate case. We have

\[\begin{aligned}F_Y(y) &= \mathbb{P}(Y < y) \\ &= \mathbb{P}(a+BX < y) \\ &= \mathbb{P}\left(X < B^{-1}(y-a)\right) \\ &= F_X\left(X < B^{-1}(y-a)\right).\end{aligned}\]

This gives that \(F_Y(y) = \Phi_{0,I}(B^{-1}(y-a))\) . Looking now at the formulation in terms of the density of \(X\) , we have

\[\begin{aligned}F_Y(y) &= \Phi_{0,I}(B^{-1}(y-a)) \\&= \frac{1}{(2\pi)^{\frac{N}{2}}} \int_{s \le B^{-1}(y-a)} \exp\left(-\frac{1}{2}s's\right) ds \\&= \text{(changing variables } x = Bs + a, \frac{1}{\det(B)}dx = ds) \\&= \frac{1}{(2\pi)^{\frac{N}{2}}} \frac{1}{\det(B)} \int_{x \le y} \exp\left(-\frac{1}{2}(B^{-1}(x-a))'(B^{-1}(x-a))\right) dx \\&= \frac{1}{(2\pi)^{\frac{N}{2}}} \frac{1}{\det(B)} \int_{x \le y} \exp\left(-\frac{1}{2}(x-a)'(B^{-1})'B^{-1}(x-a)\right) dx \\&= \frac{1}{(2\pi)^{\frac{N}{2}}} \frac{1}{\det(B)} \int_{x \le y} \exp\left(-\frac{1}{2}(x-a)'(BB')^{-1}(x-a)\right) dx\end{aligned}\]

where the density function is now clearly seen to be \(\phi_{a,BB'}\) , and so \(Y \sim N(a, BB')\) .

This example, too, indicates how we may simulate multivariate normal random variables. Given an arbitrary covariance matrix, \(\Sigma\) , and vector, \(\mu\) , if one can write \(\Sigma = \Lambda\Lambda'\) , then \(Y = \mu + \Lambda X\) , with \(X \sim N(0, I)\) is distributed as \(Y \sim N(\mu, \Sigma)\) .

In our proof, we required that \(B\) was invertible. A more general proof is available, resorting to so-called moment generating functions, but we do not cover these here. Suffice to say that the proof generalizes to the case \(Y = a + BX\) for \(a \in \mathbb{R}^M\) , \(B \in \mathbb{R}^{M \times N}\) , and \(X\) multivariate normal of dimension \(N\) .

Example 2.2.5. Many paragraphs have already been spent discussing the potential lack of applicability of the normal distribution for equity returns. We continue this arc, again looking at figure 2.2, where an estimated \(\hat{\mu}\) and \(\hat{\Sigma}\) are used to draw the contour lines of the multivariate normal density best fitting the data. To do this, contour lines are simply determined by

\[\mathcal{I}_c = \left\{x \mid (x - \hat{\mu})'\hat{\Sigma}^{-1}(x - \hat{\mu}) = c\right\} \quad (2.35)\]

for various values of \(c\) . The fact that these sets fix \(\phi_{\hat{\mu}, \hat{\Sigma}}\) is evident from the definition in (2.34).

Clearly there is, as in the univariate case, some reasonable fit, but extreme values and peakedness seem to still dominate, nonetheless.

2.2.2 Multivariate Log-Normal Distribution

The multivariate log-normal distribution is defined as before. We say that \(Y\) is a multivariate log-normal random variable if for some \(X \sim N(\mu, \Sigma)\) ,

\[Y = \exp(X)\]

or, equivalently, \(Y\) is log-normal if its logarithm is normal. In the above, exponentiation and logarithms are componentwise; viz., for \(X \in \mathbb{R}^N\) ,

\[\exp(X) = \begin{pmatrix} \exp(X_1) \\ \vdots \\ \exp(X_N) \end{pmatrix}.\]

The multivariate log-normal distribution is again described by two parameters, \(\mu\) , and \(\Sigma\) , and denoted \(Y \sim LN(\mu, \Sigma)\) . The density function for lognormal \(Y\) taking values in \(\mathbb{R}^N\) is given by

\[l_{\mu, \Sigma}(x) = \tag{2.36}\]

\[\frac{1}{(2\pi)^{\frac{N}{2}} \det(\Sigma)^{-\frac{1}{2}}} \exp\left(-\frac{1}{2}(\log(x) - \mu)' \Sigma^{-1} (\log(x) - \mu)\right) \cdot \prod_{i=1}^{N} \frac{1}{x_i}\]

for \(x \in \mathbb{R}_+^N\) .

Example 2.2.6. We may derive the density \(l_{\mu, \Sigma}\) by use of the change of variable theorem. For \(Y = \exp(X) = g(X)\) , with \(X\) having density \(\phi_{\mu, \Sigma}(\cdot)\) , the inverse of \(g\) is given by

\[g^{-1}(Y) = \begin{pmatrix} \log(Y_1) \\ \vdots \\ \log(Y_N) \end{pmatrix} = \begin{pmatrix} g_1^{-1}(Y_1) \\ \vdots \\ g_N^{-1}(Y_N) \end{pmatrix}.\]

To determine the Jacobian of \(g^{-1}(\cdot)\) , we must calculate the partial derivatives \(\frac{\partial g_i^{-1}}{\partial y_j}\) . From the above, these are exactly given by

\[\frac{\partial g_i^{-1}}{\partial y_j} = \begin{cases} \frac{1}{y_j} & \text{if } i = j \\ 0 & \text{otherwise.} \end{cases}\]

The Jacobian, \(\nabla g^{-1}\) , is therefore a diagonal matrix, whose determinant is found simply to be

\[\prod_{i=1}^{N} \frac{1}{y_i}.\]

We are now ready to derive the density function for \(Y\) :

\[\begin{aligned} l_{\mu, \Sigma}(y) &= \phi_{\mu, \Sigma}(g^{-1}(y)) |\det(\nabla g^{-1}(y))| \\ &= \phi_{\mu, \Sigma}(\log(y)) \prod_{i=1}^{N} \frac{1}{y_i} \end{aligned}\]

which is exactly (2.36).

The mean and covariance of \(Y \sim LN(\mu, \Sigma)\) are given as

\[\begin{aligned}\mathbb{E}(Y)_i &= \exp\left(\mu_i + \frac{1}{2}\sigma_i^2\right) \\ \text{Cov}(Y)_{ij} &= \exp\left(\mu_i + \mu_j + \frac{1}{2}(\sigma_i^2 + \sigma_j^2)\right) \exp(\sigma_{ij} - 1)\end{aligned}\]

where \(\sigma_{ij}\) denotes the \((i, j)\) component of the matrix, \(\Sigma\) as usual. We do not prove this result.

2.2.3 Multivariate Student \(t\) Distribution

The multivariate Student \(t\) distribution follows the same method of extension as the multivariate normal density. We say \(T\) is distributed as a multivariate Student \(t\) distribution with \(\nu \in \mathbb{N}_+\) degrees of freedom if \(T\) has density

\[\text{st}_{\mu, \Sigma; \nu}(x) = \frac{\Gamma\left(\frac{\nu+N}{2}\right)}{\Gamma\left(\frac{\nu}{2}\right)(\nu\pi)^{N/2} \det(\Sigma)^{1/2}} \left(1 + \frac{1}{\nu}(x - \mu)'\Sigma^{-1}(x - \mu)\right)^{-\frac{\nu+N}{2}}, \quad (2.37)\]

where \(\Gamma(\cdot)\) is the gamma function.

It may be shown that the marginals of the multivariate Student \(t\) are themselves univariate Student \(t\) with density \(\text{st}_{\mu_i, \sigma_i^2; \nu}(x)\) . As a result, when \(\nu > 1\) , then, \(\mathbb{E}(T) = \mu\) . Finally, for \(\nu > 2\) , \(\text{Cov}(T) = \frac{\nu}{\nu-2}\Sigma\) .

Example 2.2.7. Like the multivariate normal distribution, we may show that an affine transformation of \(X \in \mathbb{R}^N\) with \(X \sim St(\mu, \Sigma; \nu)\) ,

\[a + BX,\]

is again distributed as a Student \(t\) with \(\nu\) degrees of freedom, with

\[a + BX \sim St(a + B\mu, B\Sigma B'; \nu).\]

The proof follows the multivariate normal example, and the proof for nonsingular \(B\) is left to the reader as an exercise.

Much as in the case for multivariate normals, then, we have that portfolios of Student \(t\) distributed random variables with \(\nu\) degrees of freedom are again distributed as Student \(t\) with \(\nu\) degrees of freedom. Further, the isocontours of the Student \(t\) distribution are similarly defined.

2.3 Convergence Results and Estimators

Many are familiar with calculating the sample mean from observations \(\{x_i\}_{i=1}^N\) as

\[\hat{\mu} = \frac{1}{N} \sum_{i=1}^{N} x_i. \quad (2.38)\]

The case in which this is unassailably correct, however, is when each \(x_i\) is drawn from a distribution \(X_i\) , and \(\{X_i\}_{i=1}^N\) are iid. But the above is really an instance of a random variable; namely

\[M_N = \frac{1}{N} \sum_{i=1}^N X_i.\]

If the \(\{X_i\}_{i=1}^N\) are iid, with mean, \(\mu\) , and standard deviation, \(\sigma\) , we know from (2.21) that

\[\begin{aligned}\mathbb{E}(M_N) &= \mathbb{E}\left(\frac{1}{N} \sum_{i=1}^N X_i\right) \\ &= \frac{1}{N} \sum_{i=1}^N \mathbb{E}(X_i) \\ &= \mu.\end{aligned}\]

What we don't know—yet—is how \(M_N\) is distributed. That is, we know that the expected value is what we'd hope it to be, but we don't have an indication of just how big \(N\) should be to have a reasonable estimate of the mean, say.

In fact, \(N\) disappeared in our calculation above. What we would hope is that as \(N\) gets large, \(M_N\) is distributed more 'tightly' around \(\mu\) . Or, put another way, we hope that the probability that \(M_N\) is too far away from \(\mu\) as \(N\) gets large gets arbitrarily small. This second formulation is at the heart of the Weak Law of Large Numbers, and is descriptive of convergence in probability.

We say that a sequence of real valued random variables, \(\{X_i\}\) converges in probability to \(X\) if for any \(\epsilon > 0\) ,

\[\lim_{N \to \infty} \mathbb{P}(|X_N - X| > \epsilon) = 0. \quad (2.39)\]

Theorem 2.3.1. (Weak Law of Large Numbers) Let \(\{X_i\}\) be iid random variables taking values in \(\mathbb{R}\) , with \(\mathbb{E}(X_i) = \mu\) , and \(\text{Var}(X_i) = \sigma^2\) for each \(i\) . Then

\[M_N = \frac{1}{N} \sum_{i=1}^N X_i\]

converges in probability to \(\mu\) .

The proof resorts to two lemmas: Markov's Inequality and Chebyshev's Inequality. We prove each in turn and then return to prove 2.3.1.

Lemma 2.3.1. (Markov's Inequality) Let \(X\) be a nonnegative random variable, and \(a\) some positive constant. Then

\[\mathbb{P}(X \ge a) \le \frac{1}{a} \mathbb{E}(X).\]

Proof. We begin by introducing the indicator random variable

\[1_A(\omega) = \begin{cases} 1 & \text{if } \omega \in A \\ 0 & \text{otherwise.} \end{cases}\]

Since \(X\) is a positive random variable, we have

\[\begin{aligned} \mathbb{E}(X) &\ge \mathbb{E}(X \cdot 1_{X \ge a}) \\ &\ge a\mathbb{E}(1_{X \ge a}) \\ &= a\mathbb{P}(X \ge a). \end{aligned}\]

A geometric interpretation can help orient these steps. Recall that for \(f(x)\) the density of \(X\) , \(f(x) \ge 0\) for all \(x\) , and

\[\begin{aligned} \mathbb{E}(X) &= \int_0^\infty xf(x)dx \\ &= \int_0^a xf(x)dx + \int_a^\infty xf(x)dx \\ &\ge \int_a^\infty xf(x)dx \\ &\ge a \int_a^\infty f(x)dx \\ &= a\mathbb{P}(X \ge a). \end{aligned}\]

The latter proof is only shown for clarity, however, as the former is a stronger result, not requiring an assumption that \(X\) has a density, \(f\) .

Lemma 2.3.2. (Chebyshev's Inequality) Let \(X\) be a random variable taking values in \(\mathbb{R}\) with \(\mathbb{E}(X) = \mu\) and \(\text{Var}(X) = \sigma^2\) . Then

\[\mathbb{P}(|X - \mu| \ge a) \le \frac{\sigma^2}{a^2}.\]

Proof. Clearly, for positive \(a\) ,

\[\mathbb{P}(|X - \mu| \ge a) = \mathbb{P}(|X - \mu|^2 \ge a^2).\]

Now, \((X - \mu)^2\) is a nonnegative random variable, we may apply Markov's Inequality with \(\mathbb{E}((X - \mu)^2)\) and get

\[\mathbb{P}(|X - \mu|^2 \ge a^2) \le \frac{1}{a^2} \mathbb{E}((X - \mu)^2).\]

We know that \(\mathbb{E}((X - \mu)^2) = \sigma^2\) , so, putting these results together, we obtain

\[\mathbb{P}(|X - \mu| \ge a) \le \frac{\sigma^2}{a^2},\]

proving the result.

Notice that in both lemmas that proceeded, no distributional assumptions were used or needed. In fact, the existence of the mean and variance (with finiteness of the former) were the only statistical properties assumed.

We are now in a position to prove the Weak Law of Large Numbers. We again assume that \(\{X_i\}\) are a sequence of univariate iid random variables with mean \(\mu\) and variance \(\sigma^2\) . We have already shown that for

\[M_N = \frac{1}{N} \sum_{i=1}^{N} X_i\]

that \(E(M_N) = \mu\) . Using the fact that the variance of the sum of independent random variables is the sum of their variances, we also have

\[\begin{aligned} Var(M_N) &= Var\left(\frac{1}{N} \sum_{i=1}^{N} X_i\right) \\ &= \frac{1}{N^2} Var\left(\sum_{i=1}^{N} X_i\right) \\ &= \frac{1}{N^2} \sum_{i=1}^{N} Var(X_i) \\ &= \frac{1}{N^2} \sum_{i=1}^{N} \sigma^2 \\ &= \frac{1}{N} \sigma^2. \end{aligned}\]

Notice that the mean remained fixed while the variance, a measure of dispersion of \(M_N\) , scaled by \(\frac{1}{N}\) .

Using Chebyshev's Inequality, we have for any \(\epsilon > 0\) ,

\[\mathbb{P}(|M_N - \mu| \ge \epsilon) \le \frac{\sigma^2}{N} \frac{1}{\epsilon^2}.\]

For a fixed \(\epsilon\) , then, the right hand side goes to zero as \(N \to \infty\) , completing the proof.

The Weak Law of Large Numbers is presented here as a way in which to introduce the concept of estimators. Many of the quantities we take as granted (e.g., \(\mu\) and \(\Sigma\) in Modern Portfolio Theory) rely on estimated parameters in practice. As such, we are best served understanding their statistical properties. The Weak Law gives some indication about the distribution of \(\hat{\mu} = \frac{1}{N} \sum_{i=1}^{N} X_i\) , or at least gives a convergence result.

We might still want to know the actual distribution of \(M_N\) . This would of course be a stronger result, and requires convergence in distribution. We say that a sequence of real valued random variables \(\{X_i\}\) with distribution functions \(\{F_i(\cdot)\}\) , respectively, converges in distribution to \(X\) if

\[\lim_{N \to \infty} F_N(X) = F(X), \quad (2.40)\]

where \(F(\cdot)\) is the distribution function of \(X\) . Convergence in probability implies convergence in distribution in the sense that if real valued random variables \(\{X_i\}\) with distribution functions \(\{F_i(\cdot)\}\) converge in probability to \(X\) ,

\[X_N \to X\]

then

\[F_N \to F.\]

The Central Limit Theorem looks at the distribution of \(M_N\) and is a remarkable result. It states that for \(\{X_i\}\) iid real random variables with finite mean and variance, \(\mu\) and \(\sigma^2\) , respectively. Then the cumulative distribution function, \(F_N\) , of the random variable

\[Z_N = \frac{\sqrt{N}(M_N - \mu)}{\sigma}\]

satisfies

\[\lim_{N \to \infty} F_N(x) = \Phi(x). \quad (2.41)\]

That is, \(Z_N\) has a limiting normal distribution. This is regardless of the distribution of the \(X_i\) 's. The proof is outside the scope of our current work.

2.3.1 Estimators and Bias

An estimator of a given quantity is simply a rule for calculating that quantity based on observed data. If we denote the quantity being estimated as \(\theta\) and the estimator as \(\hat{\theta}\) , we define the bias of the estimator \(\hat{\theta}\) as

\[\text{Bias}(\hat{\theta}) = \mathbb{E}(\hat{\theta}) - \theta. \quad (2.42)\]

An unbiased estimator has zero bias and satisfies

\[\mathbb{E}(\hat{\theta}) = \theta.\]

We have already encountered an estimator of the mean, \(\mu\) , of a sequence of iid random variables, \(\{X_i\}\) , finding that

\[M_N = \frac{1}{N} \sum_{i=1}^{N} X_i\]

satisfied \(\mathbb{E}(M_N) = \mu\) immediately in our investigation. We now may state that the estimator \(M_N\) is an unbiased estimator of the mean.

Example 2.3.1. We next give an unbiased estimator of the variance of iid random variables, \(\{X_i\}\) , each with mean \(\mu\) and variance \(\sigma^2\) . Let

\[s_N = \frac{1}{N-1} \sum_{i=1}^{N} (X_i - \hat{\mu})^2, \quad (2.43)\]

with \(\hat{\mu} = \frac{1}{N} \sum_{i=1}^{N} X_i\) as before. Then

\[\mathbb{E}(s_N) = \sigma^2.\]

The term \(1/(N-1)\) is somewhat surprising on first pass. It will be slightly more intuitive when we encounter the estimate of variance again in ordinary least squares where degrees of freedom will be given more color as well. In the meantime, the current proof shows that we require \(N-1\) in the denominator to ensure that our estimator is unbiased.

Proof. Let \(X\) be distributed as each of the \(X_i\) , and look at

\[\mathbb{E} \left( \sum_{i=1}^{N} (X_i - \hat{\mu})^2 \right) = \sum_{i=1}^{N} \mathbb{E} \left( (X_i - \hat{\mu})^2 \right),\]

and notice that for each summand

\[\mathbb{E} \left( (X_i - \hat{\mu})^2 \right) = \mathbb{E}(X_i^2) - 2\mathbb{E}(X_i\hat{\mu}) + \mathbb{E}(\hat{\mu}^2),\]

so that the sum becomes

\[\begin{aligned} \mathbb{E} \left( \sum_{i=1}^{N} (X_i - \hat{\mu})^2 \right) &= \sum_{i=1}^{N} \mathbb{E}(X_i^2) - 2 \sum_{i=1}^{N} \mathbb{E}(X_i\hat{\mu}) + \sum_{i=1}^{N} \mathbb{E}(\hat{\mu}^2) \\ &= N\mathbb{E}(X^2) - 2 \sum_{i=1}^{N} \mathbb{E}(X_i\hat{\mu}) + N\mathbb{E}(\hat{\mu}^2) \\ &= N\mathbb{E}(X^2) - 2\mathbb{E}(N\hat{\mu}\hat{\mu}) + N\mathbb{E}(\hat{\mu}^2) \\ &= N\mathbb{E}(X^2) - 2N\mathbb{E}(\hat{\mu}^2) + N\mathbb{E}(\hat{\mu}^2) \\ &= N \left( \mathbb{E}(X^2) - \mathbb{E}(\hat{\mu}^2) \right) \end{aligned}\]

We look at \(\mathbb{E}(X^2)\) and \(\mathbb{E}(\hat{\mu}^2)\) in turn.

Now, we know that \(\text{Var}(X) = \mathbb{E}(X^2) - \mu^2\) , giving

\[\mathbb{E}(X^2) = \sigma^2 + \mu^2.\]

Looking at \(\mathbb{E}(\hat{\mu}^2)\) next, we require the variance and mean of \(\hat{\mu}^2\) and apply the same relationship. We have already established that

\[\begin{aligned} \mathbb{E}(\hat{\mu}) &= \mu \\ \text{Var}(\hat{\mu}) &= \frac{1}{N}\sigma^2. \end{aligned}\]

This gives that

\[\mathbb{E}(\hat{\mu}^2) = \frac{1}{N}\sigma^2 + \mu^2.\]

Putting the above together, we have

\[\begin{aligned}\mathbb{E}\left(\sum_{i=1}^N(X_i-\hat{\mu})^2\right) &= N\left(\mathbb{E}(X^2)-\mathbb{E}(\hat{\mu}^2)\right) \\ &= N\left(\sigma^2+\mu^2-\left(\frac{1}{N}\sigma^2+\mu^2\right)\right) \\ &= N\left(\frac{N-1}{N}\sigma^2\right) \\ &= (N-1)\sigma^2.\end{aligned}\]

Dividing both sides by \(N-1\) proves the result, and \(s_N\) is an unbiased estimator of the variance of the iid random variables \(X_i\) .

We have surreptitiously seen estimators in our work already. Whenever a fit of a distribution has been shown, we have estimated the mean and variance (and covariance in one case) of the underlying distributions using the above estimators. Implicit in this estimation is that the random variables under consideration are iid. Specifically, we repeatedly assumed, then, that daily log returns of the S&P are independent. This is actually less of a sure assumption than one might have guessed.

Amir Khandani and Andrew Lo [17] exhibit that a stock trading strategy that buys the last day’s losers and sells the last day’s winners is profitable – and their result is fairly robust within the context they study. If returns were independent, such a strategy should return something indistinguishable from zero. The fact that this is not the case implies that there is some dependence between yesterday’s returns and todays. A simple codification of the finding would be something like

\[r_t = \phi r_{t-1} + \epsilon_t\]

for the most extreme winners and losers each day in the market. The equation above is an example of an autoregressive model, and it says that far from being independent, there is a structural form between daily returns.

Before incorporating your new hedge fund in Delaware where you plan to exploit the strategy just exhibited, know this: it won’t work. Short time scale strategies that center around daily close prices are virtually impossible to execute for two reasons in particular: 1) a massive amount of each day’s trading volume is centered around the closing bell, so that achieving a fill on the exact close price is either very tricky or very expensive; and 2) a significant amount of the variation in stock prices comes between the close and the open (that is, when you can’t trade) so that waiting till the open to get your fills downgrades the strategy to random noise.

Knowing that the strategy isn’t viable in terms of execution does not negate its usefulness, however. For example, Lo and Khandani look at how this simplistic strategy did during the (wholly improbable) Quant Crisis we have alluded to previously – and it fared terribly. Performance over other periods was surprisingly stellar, and the crisis days in August 2007 were outliers for the strategy.

Their reversion strategy, then, provided a simple template to examine observations in the market through an interpretable lens.

While we won't present a solution to the apparent lack of independence of daily returns, we discuss these results to give some insight into the tension between estimation in practice, theory, and empirical findings.