Context

We begin by proving that \(s\) defined by

\[s^2 = \frac{1}{N-p} \sum_{t=1}^{N} \hat{\epsilon}_t^2 \quad (4.23)\]

is an unbiased estimator of \(\sigma\) , and that under the Gauss-Markov assumptions with added distributional requirement, \(\epsilon \sim N(0, \sigma^2 I)\) ,

\[\frac{N-p}{\sigma^2} s^2 \sim \chi_{N-p}^2 \quad (4.24)\]

where \(\chi_\nu^2\) is the Chi-Squared distribution with \(\nu\) degrees of freedom defined as the sum of squared iid standard normal variables; viz., we say \(W \sim \chi_\nu^2\) if

\[W = Z_1^2 + \cdots + Z_\nu^2 \quad (4.25)\]

with \(Z_i \sim N(0, 1)\) iid.

Proof. Let

\[\hat{\epsilon} = \begin{pmatrix} \hat{\epsilon}_1 \\ \vdots \\ \hat{\epsilon}_N \end{pmatrix}.\]

be the OLS estimate of \(\epsilon\) in (4.12). Since \(\mathbb{E}(\hat{\epsilon}) = 0\) , the covariance of \(\hat{\epsilon}\) is exactly

\[\text{Cov}(\hat{\epsilon}) = \mathbb{E}(\hat{\epsilon}\hat{\epsilon}')\]

We have, implementing a technique seen previously, that

\[||\hat{\epsilon}||^2 = \hat{\epsilon}'\hat{\epsilon} = \text{tr}(\hat{\epsilon}'\hat{\epsilon}).\]

Now by linearity we have that for random matrices, \(A\) , \(\mathbb{E}(\text{tr}(A)) = \text{tr}(\mathbb{E}(A))\) so that

\[\mathbb{E}(\text{tr}(\hat{\epsilon}'\hat{\epsilon})) = \text{tr}(\mathbb{E}(\hat{\epsilon}'\hat{\epsilon})).\]

And, as before,

\[\text{tr}(\mathbb{E}(\hat{\epsilon}'\hat{\epsilon})) = \text{tr}(\mathbb{E}(\hat{\epsilon}\hat{\epsilon}')).\]

So that

\[\begin{aligned} \mathbb{E}(||\hat{\epsilon}||^2) &= \text{tr}(\text{Cov}(\hat{\epsilon})) \\ &= \text{tr}(\sigma^2(I - H)). \end{aligned}\]

Since \(H\) is a projection into a \(p\) -dimensional space, there exists a basis such that

\[H = \begin{pmatrix} I_p & 0 \\ 0 & 0_{N-p} \end{pmatrix}.\]

This follows from the fact that the eigenvalues of \(H\) are zero or one, and that the rank of \(H\) is \(p\) . Clearly, under this basis, \(\text{tr}(H) = p\) . Finally, since \(\text{tr}(\cdot)\) is independent of the choice of basis, \(\text{tr}(H) = p\) under any basis. Similarly, \(I - H\) will necessarily have trace

\[\text{tr}(I - H) = N - p.\]

Therefore,

\[\begin{aligned} \mathbb{E}(\|\hat{\epsilon}\|^2) &= \sigma^2 \text{tr}(I - H) \\ &= \sigma^2(N - p), \end{aligned}\]

or,

\[\begin{aligned} \mathbb{E}\left(\sum_{t=1}^T \hat{\epsilon}_t^2\right) &= \mathbb{E}\left(\sum_{t=1}^T (y_t - \beta' x_t)^2\right) \\ &= \sigma^2(N - p), \end{aligned}\]

giving that the scaled sum of the squared residuals,

\[s^2 = \frac{1}{N-p} \mathbb{E}(\|\hat{\epsilon}\|^2)\]

is an unbiased estimator of \(\sigma^2\) . This result is independent of the distributional assumption of \(\epsilon\) ; i.e., it used only the Gauss-Markov assumptions.

If we further make the distributional assumption that \(\epsilon, \epsilon \sim N(0, \sigma^2 I)\) , we may show that

\[\frac{1}{\sigma^2} \|\hat{\epsilon}\|^2 \sim \chi^2_{N-p} \quad (4.26)\]

as in (4.24).

Let \(Q\) be the orthogonal change of basis matrix for the given matrix \(H\) such that

\[QHQ' = \begin{pmatrix} I_p & 0 \\ 0 & 0_{N-p} \end{pmatrix}.\]

Notice that this also implies that

\[Q(I - H)Q' = \begin{pmatrix} 0_p & 0 \\ 0 & I_{N-p} \end{pmatrix}.\]

For this \(Q\) , define

\[Z = \frac{1}{\sigma} Q\epsilon = \begin{pmatrix} Z_1 \\ \dots \\ Z_N \end{pmatrix}.\]

The key observation here is that \(\hat{\epsilon} = (I - H)\epsilon\) and that the simplified basis allows for a more tractable approach.

Under our last assumption, \(Z\) is a multivariate normal random variable since a linear combination of jointly normal random variables is jointly normal. The mean and covariance of \(Z\) are given by

\[\begin{aligned}\mathbb{E}(Z) &= \mathbb{E}\left(\frac{1}{\sigma}Q\epsilon\right) \\ &= \frac{1}{\sigma}Q\mathbb{E}(\epsilon) \\ &= 0,\end{aligned}\]

by linearity of the expectation operator, and

\[\begin{aligned}\text{Cov}(Z) &= \text{Cov}\left(\frac{1}{\sigma}Q\epsilon\right) \\ &= \frac{1}{\sigma^2}Q\text{Cov}(\epsilon)Q' \\ &= \frac{\sigma^2}{\sigma^2}QIQ' \\ &= I\end{aligned}\]

since \(QQ' = Q'Q = I\) . Hence

\[Z \sim N(0, I).\]

This also shows that the \(\{Z_t\}\) are iid. We may recover \(\epsilon\) as

\[\epsilon = \sigma Q'Z.\]

Since \(Q\) is orthogonal, we have that

\[\begin{aligned}||Q\hat{\epsilon}||^2 &= \hat{\epsilon}'Q'Q\hat{\epsilon} \\ &= \hat{\epsilon}'\hat{\epsilon} \\ &= ||\hat{\epsilon}||^2,\end{aligned}\]

giving

\[\begin{aligned}||\hat{\epsilon}||^2 &= ||Q\hat{\epsilon}||^2 \\ &= ||Q(I - H)\epsilon||^2 \\ &= ||Q(I - H)Q'Q\epsilon||^2 \\ &= \sigma^2||Q(I - H)Q'Z||^2 \\ &= \sigma^2 Z' \begin{pmatrix} 0_p & 0 \\ 0 & I_{N-p} \end{pmatrix}^2 Z \\ &= \sigma^2 \sum_{t=p+1}^{N} Z_t^2.\end{aligned}\]

As a result, we clearly have that \(\frac{1}{\sigma^2}||\hat{\epsilon}||^2\sim\chi_{N-p}^2\) as desired.

We may now return to the ratio in (4.22),

\[\frac{\hat{\beta}_i-\beta_i}{s\sqrt{c_{ii}}}.\]

We proved that under the full Gauss-Markov assumptions with additional distributional assumption for residuals,

\[\frac{\hat{\beta}_i-\beta_i}{\sigma\sqrt{c_{ii}}}\sim N(0,1).\]

We use this fact along with the result just obtained to decompose (4.22) as

\[\frac{\hat{\beta}_i-\beta_i}{s\sqrt{c_{ii}}}=\frac{\frac{\hat{\beta}_i-\beta_i}{\sigma\sqrt{c_{ii}}}}{\frac{s}{\sigma}}.\]

The result is a ratio of random variables with known distributions. In particular, in the numerator we have a standard normal random variable. The denominator is the square root of a \(\chi^2\) random variable with \(N-p\) degrees of freedom scaled by \(\frac{1}{N-p}\) .

Remarkably, a random variable

\[T=\frac{Z}{\sqrt{\frac{1}{M}X}} \tag{4.27}\]

with \(Z\sim N(0,1)\) and \(X\sim\chi_M^2\) and \(Z\) and \(X\) independent is distributed as a standard Student \(t\) random variable with \(M\) degrees of freedom as seen in (2.13); i.e., \(T\sim\text{St}(0,1;M)\) . We will abbreviate this notation as

\[T\sim t_M. \tag{4.28}\]

We do not prove the correspondence between our initial definition and that presented here, however.

The independence between \(\hat{\beta}_i\) and \(s^2\) has not been shown either. With the same assumptions as above we have that

\[\hat{\epsilon}=(I-H)\epsilon\] \[X(\hat{\beta}-\beta)=H\epsilon,\]

so that the covariance inner product between \(\hat{\epsilon}\) and \(X(\hat{\beta}-\beta)\) satisfies

\[(\hat{\epsilon},X(\hat{\beta}-\beta))=(I-H)\epsilon,H\epsilon)\] \[=(I-H)\sigma^2IH\] \[=\sigma^2(H-H^2)=0.\]

Under the normality assumption for \(\epsilon\) , this shows that \(\hat{\epsilon}\) and \(X(\hat{\beta}-\beta)\) are independent as well. Since we are also assuming that \(X\) is full rank,

\[\begin{aligned}0 &= (\hat{\epsilon}, X\hat{\beta}) ((X'X)^{-1}X')' \\&= (\hat{\epsilon}, \hat{\beta})\end{aligned}\]

as well, so that \(\hat{\epsilon}\) and \(\hat{\beta}\) are independent. Therefore \(s^2\) and the components of \(\hat{\beta}\) are independent as well.

4.3.1 Hypothesis Testing, Confidence Intervals, and Prediction Intervals

With the distribution of \(\frac{\hat{\beta}_i-\beta_i}{s\sqrt{c_{ii}}}\) now known (under the extended Gauss-Markov assumptions), we may construct symmetric confidence intervals for each \(\beta_i\) in turn. In particular, since \(t_{N-p}\) is symmetric about zero, a \(100(1-\alpha)\%\) confidence interval for \(\frac{\hat{\beta}_i-\beta_i}{s\sqrt{c_{ii}}}\) is given by

\[-t_{N-p;1-\alpha/2} \le \frac{\hat{\beta}_i - \beta_i}{s\sqrt{c_{ii}}} \le t_{N-p;1-\alpha/2}, \quad (4.29)\]

or

\[\hat{\beta}_i - s\sqrt{c_{ii}}t_{N-p;1-\alpha/2} \le \beta \le \hat{\beta}_i + s\sqrt{c_{ii}}t_{N-p;1-\alpha/2}, \quad (4.30)\]

where \(t_{N-p;1-\alpha/2}\) is the \(1-\alpha/2\) quantile of \(t_{N-p}\) . Clearly, with this choice,

\[\mathbb{P}\left(\left|\frac{\hat{\beta}_i - \beta}{s\sqrt{c_{ii}}}\right| \le t_{N-p;1-\alpha/2}\right) = 1 - \alpha.\]

We emphasize once more that this is a component-wise test.

A common issue in model development is determining whether a particular variable should be included in the model. As one approximation to the answer, we may approach the question by asking whether a particular \(\beta_i\) is likely to be zero. Necessarily this requires setting a specific confidence level. For example, for a fixed \(\alpha\) , we may determine if the interval

\[\left(\hat{\beta}_i - s\sqrt{c_{ii}}t_{N-p;1-\alpha/2}, \hat{\beta}_i + s\sqrt{c_{ii}}t_{N-p;1-\alpha/2}\right)\]

contains zero. If it does, there is evidence that the \(i\) th variable may be omitted from the model (relative to this particular confidence level \(\alpha\) ).

The preceding reasoning may be formalized into so-called hypothesis testing wherein a null hypothesis is formulated – and typically denoted by \(H_0\) – and statistical values are then measured. If the statistical values are extremely unlikely (relative to some threshold), we may reject the null hypothesis.

Example 4.3.1. We consider the null hypothesis

\[H_0: \beta_i = 0.\]

Under the null hypothesis,

\[b_i = \frac{\hat{\beta}_i}{s\sqrt{c_{ii}}} \sim t_{N-p}.\]

The probability of observing a value at least as large as \(|b_i|\) is exactly

\[p_0 = 2 \cdot (1 - S_{0,1;N-p}^{t-1}(b_i)).\]

We reject the null hypothesis at significance level \(\alpha\) if \(p_0 < \alpha\) . We call \(p_0\) the p-value of the test statistic associated with \(H_0\) .

Notice that we reject the null hypothesis exactly when

\[|b_i| > t_{N-p;1-\alpha/2},\]

which coincides with our previous observation that the confidence interval for \(\beta_i\) would in fact contain zero.

Falsely rejecting the null hypothesis when it is in fact true is called a type I error. When the extended Gauss-Markov assumptions obtain, the probability of a type I error is equivalent to the confidence level \(\alpha\) . It is not true that this holds generally.

Example 4.3.2. We return to examining ex post equity returns sorted by volatility quartiles. In our previous example we provided evidence that there is monotonicity in these ex post returns, but in a direction counter to that implied by CAPM. As such, we investigate constructing portfolios long in low trailing volatility and short in high volatility stocks.

At each month, we sort the same universe of 1,000 stocks considered previously by own volatility into quartiles, with quartile four holding the highest volatility names. At time \(t\) , let \(r_{1t}\) and \(r_{4t}\) be the average return over the next month \((t, t+1]\) for the lowest and highest quartile buckets, respectively. In addition, let \(\sigma_{1t}\) and \(\sigma_{4t}\) be the square root of the average trailing variance of the lowest and highest buckets.

We construct a portfolio with approximately equal long and short volatility by being long one unit of a portfolio of equal weighted names from quartile one, and short \(\frac{\sigma_{1t}}{\sigma_{4t}}\) units of the equally weighted names from quartile four. The return over the month post construction is given by

\[r_t = r_{1t} - \frac{\sigma_{1t}}{\sigma_{4t}} r_{4t}.\]

Over the period in our backtest, the average monthly return for this portfolio is 55 bps, annualizing to approximately 6.58%.

Regressing on a market proxy, we find the CAPM \(\alpha\) and \(\beta\) (ignoring the risk free rate) to be 0.60 and -0.0702, respectively. That is, the returns of the portfolio are generated without exposure to the market.

Further, we may consider the null hypothesis

\[H_0: \alpha = 0.\]

A confidence interval for \(\alpha\) at the confidence level 0.05 is found to be [0.0013, 0.0106] so that we may reject the null hypothesis at the 5% level.

In addition to examining confidence intervals (and making inferences based on these intervals), we may also construct confidence intervals about the mean response from the linear models considered. In particular, and again, for the original OLS model (4.12), OLS (least squares) estimates of \(\beta\) , \(\hat{\beta}\) , and assuming the Gauss-Markov with distributional assumptions obtains, an unbiased estimate of the variance of an input, \(x_t\) , is given by

\[\mathrm{Var}(\hat{y}_t) = \mathrm{Var}(\hat{\beta}'x_t) \quad (4.31)\]

\[= x_t'\mathrm{Cov}(\hat{\beta})x_t \quad (4.32)\]

\[= \sigma^2 x_t'(X'X)^{-1}x_t. \quad (4.33)\]

Following the same procedure as above, we find that a \(100 \cdot (1 - \alpha)\%\) confidence interval for \(\hat{\beta}'x_t\) is

\[\hat{\beta}'x_t \pm \left(s\sqrt{x_t'\mathrm{Cov}(\hat{\beta})x_t}\right)t_{N-p;1-\alpha/2}, \quad (4.34)\]

where, continuing our convention, we denote \(C = (X'X)^{-1}\) . The complete verification of this result, following that of (4.29), is left as an exercise.

The confidence interval just exhibited is both an estimate of the confidence interval of the mean and dependent on \(x = x_t\) being an observation in the derivation of \(\hat{\beta}\) . We may also consider the case of a new observation, or, equivalently, a prediction interval about an observation \(x\) as opposed to the mean response. This is in contrast with the previous result as we may no longer discard the variance of \(\epsilon\) . We have,

\[\begin{aligned} \mathrm{Var}(\hat{y}) &= \mathrm{Var}(\hat{\beta}'x + \epsilon) \\ &= x'\mathrm{Cov}(\hat{\beta})x + \sigma^2 \\ &= \sigma^2 x'(X'X)^{-1}x + \sigma^2, \end{aligned}\]

so that, in the usual manner, a \(100 \cdot (1 - \alpha)\%\) prediction interval for \(\hat{\beta}'x\) is

\[\hat{\beta}'x \pm \left(s\sqrt{1 + x_t'\mathrm{Cov}(\hat{\beta})x_t}\right)t_{N-p;1-\alpha/2}. \quad (4.35)\]

4.3.2 Submodel Testing

The ability to test the null hypothesis for the inclusion or exclusion of a particular variable is a powerful result. However, we may also want to construct null hypotheses about linear submodels of the full model specified in (4.12); i.e., we would like to consider the case that

\[\mathbb{E}(Y) \subseteq \mathcal{L}_0\]

for some (strict) subset \(\mathcal{L}_0\subset\mathcal{L}\) , where \(\mathcal{L}\) is the space spanned by the columns of \(X\) . Necessarily the dimension of \(\mathcal{L}_0\) will be less than \(\dim(\mathcal{L})\) . We will denote \(\dim(\mathcal{L}_0)\) by \(r\) in what follows.

Under the assumptions of the model, we have

\[\mathbb{E}(Y)=\mathbb{E}(X\beta+\epsilon)=X\beta,\] (4.36)

so that clearly

\[\mathbb{E}(Y)\subseteq\mathcal{L}.\]

From our previous work, ordinary least squares projects \(Y\) onto \(\mathcal{L}\) through the matrix \(H\) . For a fixed subspace \(\mathcal{L}_0\subset\mathcal{L}\) , we may consider a similar projection of \(Y\) onto \(\mathcal{L}_0\) by some matrix \(G\) , with

\[\hat{Y}_0=GY.\] (4.37)

Notice that since \(H\) projects into \(\mathcal{L}\) , \(GH=G\) and \(HG=G\) . As a result, \(Y-\hat{Y}\) and \(\hat{Y}-\hat{Y}_0\) are orthogonal. First note that

\[\begin{aligned}\mathbb{E}(Y-\hat{Y})&=\mathbb{E}((I-H)\epsilon)=0\\\mathbb{E}(\hat{Y}_0-\hat{Y})&=\mathbb{E}((H-G)\epsilon)=0.\end{aligned}\]

The covariance is therefore

\[\begin{aligned}\text{Cov}(Y-\hat{Y},\hat{Y}_0-\hat{Y})&=\text{Cov}((I-H)\epsilon,(H-G)\epsilon)\\&=(I-H)\text{Cov}(\epsilon)(H-G)'\\&=\sigma^2(I-H)(H-G)\\&=\sigma^2\left(H-H^2-G+HG\right)\\&=0.\end{aligned}\]

Assuming joint normality of \(\epsilon\) implies that \(Y-\hat{Y}\) and \(\hat{Y}_0-\hat{Y}\) are independent. As a result, \(||Y-\hat{Y}||^2\) and \(||\hat{Y}_0-\hat{Y}||^2\) are independent as well.

We next consider the null hypothesis

\[H_0:\mathbb{E}(Y)\subseteq\mathcal{L}_0.\] (4.38)

Under the null hypothesis, the expected value of \(Y\) lies in a space where we don't need all of the columns of \(X\) . In other words, some subset (or linear combination) of variables is sufficient to describe \(Y\) .

It also follows under \(H_0\) and the Gauss-Markov assumptions that \(\frac{1}{p-r}||\hat{Y}-\hat{Y}_0||^2\) is an unbiased estimator of \(\sigma^2\) , and further that under the extended Gauss-Markov assumptions,

\[\frac{1}{p-r}||\hat{Y}-\hat{Y}_0||^2\sim\chi_{p-r}^2\] (4.39)

The result follows considering the projection \(H-G\) (rather than \(I-H\) as in the preceding proof) and the change of basis matrix \(Q\) satisfying

\[Q(H-G)Q'=\begin{pmatrix} 0_{N-(p-r)} & 0 \\ 0 & I_{p-r} \end{pmatrix}.\]

The test statistic we will use to evaluate \(H_0\) will be

\[F=\frac{N-p}{p-r}\frac{||\hat{Y}-\hat{Y}_0||^2}{||Y-\hat{Y}||^2}\] (4.40)

We say (with clear motivation stemming from the results above) that a random variable constructed as the ratio of independent \(\chi^2\) random variables as in

\[F_{d_1,d_2}=\frac{\frac{1}{M}U}{\frac{1}{N}V}\] (4.41)

follows the F distribution with ( \(d_1, d_2\) ) degrees of freedom, where \(d_1\) and \(d_2\) are the degrees of freedom for \(U\) and \(V\) , respectively. Since under our assumptions, \(\frac{1}{\sigma^2}||Y-\hat{Y}||^2\sim\chi_{N-p}^2\) and \(\frac{1}{\sigma^2}||\hat{Y}-\hat{Y}_0||^2\sim\chi_{p-r}^2\) and independence has already been shown,

\[\frac{\frac{1}{\sigma^2(p-r)}||\hat{Y}-\hat{Y}_0||^2}{\frac{1}{\sigma^2(N-p)}||Y-\hat{Y}||^2}=\frac{N-p}{p-r}\frac{||\hat{Y}-\hat{Y}_0||^2}{||Y-\hat{Y}||^2}\sim F_{p-r,N-p},\]

giving that the test statistic in (4.40) may be tested against a known distribution. Notice that by orthogonality we have

\[||Y-\hat{Y}_0||^2=||Y-\hat{Y}||^2+||\hat{Y}-\hat{Y}_0||^2,\] (4.42)

or

\[||\hat{Y}-\hat{Y}_0||^2=||Y-\hat{Y}_0||^2-||Y-\hat{Y}||^2.\]

This gives an alternate formulation of the above as

\[\frac{N-p}{p-r}\frac{||\hat{Y}-\hat{Y}_0||^2}{||Y-\hat{Y}||^2}=\frac{N-p}{p-r}\frac{||Y-\hat{Y}_0||^2-||Y-\hat{Y}||^2}{||Y-\hat{Y}||^2}.\]

This refactoring is more than cosmetic as the ratio now only involves the terms \(||Y-\hat{Y}_0||^2\) and \(||Y-\hat{Y}||^2\) , the sum of the squared residuals from the submodel and full model, respectively; i.e., we have

\[\frac{N-p}{p-r}\frac{||\hat{Y}-\hat{Y}_0||^2}{||Y-\hat{Y}||^2}=\frac{N-p}{p-r}\frac{RSS_{\mathcal{L}_0}-RSS_{\mathcal{L}}}{RSS_{\mathcal{L}}},\]

where RSS is defined as the residual sum of squares of a given model. In particular, the full model residual sum of squares is given by

\[RSS_{\mathcal{L}}=||Y-X\hat{\beta}||^2=\sum_{t=1}^T\left(y_t-\hat{\beta}'x_t\right)^2.\]

We call the null hypothesis test constructed above an F test. As before, we reject the null hypothesis, \(H_0\) at confidence level \(\alpha\) if \(F>F_{p-r,N-p;1-\alpha}\) , where as before, \(F_{d_1,d_2,1-\alpha}\) is the \(100\cdot(1-\alpha)\) percentile of the F distribution with \(d_1\)

and \(d_2\) degrees of freedom. Notice that an \(F\) test is a one sided test, in contrast to the \(t\) test. An \(F\) test may be constructed whenever we have nested models, and gives a method for evaluating submodels; viz., if we do not reject the null hypothesis, then we may prefer the submodel giving rise to \(\mathcal{L}_0\) .

Example 4.3.3. Consider the model

\[\text{Model 1 } y_t = \beta_1 x_{1t} + \cdots + \beta_p x_{Nt} + \epsilon_t\]

and assume there are \(t = 1, \dots, T\) observations and that the extended Gauss-Markov assumptions hold. Let

\[H_0: \beta_1 = 2\beta_2.\]

Under \(H_0\) , we consider

\[\begin{aligned}\text{Model 2 } y_t &= (2\beta_2)x_{1t} + \beta_2 x_{2t} + \cdots + \beta_p x_{Nt} + \epsilon_t \\ &= \beta_2(2x_{1t} + x_{2t}) + \cdots + \beta_p x_{Nt} + \epsilon_t.\end{aligned}\]

Obtaining the RSS of both models (denoted by \(RSS_1\) and \(RSS_2\) , respectively) after this second formulation is now clear. As is the reframing of \(H_0\) in terms of a subspace of \(\mathcal{L} = \text{span}(X)\) . An \(F\) test at confidence level \(\alpha\) checks

\[(N-p)\frac{RSS_2 - RSS_1}{RSS_1}\]

against the value \(F_{1, N-p; 1-\alpha}\) . If the test statistic is greater than \(F_{1, N-p; 1-\alpha}\) , we reject the null hypothesis.

4.3.3 Variable Selection

In every case in OLS, a model with more input variables will fit better than one with less. However, additional regressors in the design matrix, \(X \in \mathbb{R}^{N \times p}\) , increases the chances of collinearity among the columns of \(X\) . In the worst case, this gives a singular \(X'X\) , but may also produce nearly singular matrices as well.

Let

\[X'X = Q\Lambda Q'\]

be the decomposition of \(X'X\) into its eigenvalues; i.e., \(\Lambda = \text{diag}(\lambda_1, \dots, \lambda_p)\) and \(Q\) orthonormal. The inverse of \(X'X\) is given

\[(X'X)^{-1} = Q\Lambda^{-1}Q'.\]

If one of the \(\lambda_i\) is close to zero we have that \(\Lambda^{-1}\) has a very large diagonal element. Therefore, in the case of collinearity in the columns of \(X\) , the covariance of \(\hat{\beta}\) given by \(\sigma^2 (X'X)^{-1}\) , is potentially very large. In other words, in the case of collinearity, the OLS estimates are less certain.

Hence some balance between the addition of variables for fit and estimation accuracy must necessarily be observed. In what follows we look at various step-wise selection procedures. In each case we let \(P\) be the maximum number of possible regressors. Notice that considering each possible model is computationally difficult due to the combinatoric nature of the problem. Instead we focus on building up a model one factor at a time, building a model by paring away factors from the \(P\) available, and a hybrid of these two methods.

Forward Selection

For a design matrix \(\tilde{X} \in \mathbb{R}^{N \times p'}\) , we define the partial correlation between \(\{y_i\}_{i=1}^N\) and \(\{w_i\}_{i=1}^N\) with respect to \(\tilde{X}\) to be the correlation of the estimated residuals \(\{\hat{\epsilon}_{yi}\}\) and \(\{\hat{\epsilon}_{wi}\}\) given by

\[\hat{\epsilon}_{yi} = y_i - \hat{y}_i\]

and

\[\hat{\epsilon}_{wi} = w_i - \hat{w}_i\]

where \(\hat{y}_i\) and \(\hat{w}_i\) are the OLS regression estimates of \(y_i\) and \(w_i\) , respectively, using the design matrix \(\tilde{X}\) .

In the forward selection algorithm, we initialize by calculating the correlation between \(\{y_i\}_{i=1}^N\) and each column of \(X\) , \(\{x_{ij}\}_{i=1}^N\) , for \(j = 1, \dots, P\) . This gives

a set of correlations \(\{\hat{\rho}_j\}_{j=1}^P\) . The first factor of the model is the regressor associated with the maximum correlation in this set.

Let \(\tilde{X}_P\) be the design matrix with all regressors included, and define \(\tilde{X}_{P-k}\) as the resulting matrix after removing the \(k\) regressors added to the model under the forward selection procedure. Similarly, let the design matrix of the model at this point be denoted \(X_k\) . To determine the next addition (should there be one), we look at the partial correlation between \(\{y_i\}_{i=1}^N\) and each column of \(\tilde{X}_{P-k}\) with respect to \(X_k\) .

A candidate model is then considered by adding the regressor with maximum partial correlation to the \(k\) factors already selected. Denote this new regressor by \(x^{k+1}\) . A partial \(F\) -test is then conducted at a fixed confidence level (usually 0.05 or 0.01) between the candidate model the model with \(k\) factors. If we fail to reject the null hypothesis

\[H_0: \beta_{k+1} = 0,\]

the procedure terminates and the model is fixed with \(k\) factors and design matrix \(X_k\) . Otherwise, the design matrix is updated to \(X_{k+1}\) by adding \(x^{k+1}\) and another regressor is considered.

Example 4.3.4. We have primarily considered equity returns in our modeling thus far. Further, we have only considered technical data; i.e., data coming from market pricing. Here we look at explaining the cross section corporate bond spreads through a model using a mixture of technical and so-called fundamental data. We consider four possible regressors:

• Debt to Market Cap: defined as the ratio of the sum of long and short term debt as reported by a company on its balance sheet to the most recent market cap (stock price \(\times\) common shares outstanding).
• Enterprise Value: The sum of
  • Market Cap: most recent stock price \(\times\) the number of common shares outstanding
  • Preferred Equity: the value of preferred equity reported on the company's balance sheet. Preferred equity offers a slightly senior position to equity holders, often with fixed dividend payments similar to a fixed income instrument
  • Total Debt: the sum of long and short term debt held by the company as reported on their balance sheet
  • Minority Interest: the value of the company's subsidiaries owned by minority shareholders as reported on the balance sheet

minus Cash and Short Term Investments.

Enterprise value is a proxy for a total valuation of the firm. If another company were to take over, common and preferred shares, minority interest, and debt would all have to be considered. Short term investments would be liquidated as cash, and cash is, well, cash.

• Equity Volatility: Standard deviation of 121 weeks of equity returns.
• CreditGrades Default Probability: Risky debt instruments have some probability that they will no longer pay their contractual obligations in the future – else everywhere there would be one common rate for borrowers of the same tenor, all of whom are perfectly creditworthy. Upon default, the assets of the firm are distributed amongst bondholders after liquidation. This portion of value recovered in liquidation is the recovery value. CreditGrades extended work by Merton (the so-called value of the firm) to incorporate balance sheet items and volatility to approximate a probability of default at the firm level. We will see these ideas fully formed in a later chapter. Here we take it as a reasonable input to describe spreads.

The correlation between these items and the corporate bond spreads we consider is given in the following table.

Immediately apparent is the high correlation between Equity Volatility and spreads. And, even more, the relationship between the CreditGrades default probability and equity volatility. We see, too, a positive relationship between spreads and leverage (Debt to Market Cap) and a slightly negative and near zero relationship to size as proxied by Enterprise Value.

The environment at the time this data was pulled was distressed, with great concern for the viability of many names in the energy sector.

We note, too, that the relationship between risk and compensation stands in stark contrast to the results for equities seen above where we saw compensation for lower risk. This first experience in bond pricing shows a hint of variation that holds true more generally in the credit markets.

We proceed to build a regression model out of the factors under consideration using the forward selection procedure. Using the notation above, we have \(P = 4\) , and

\[\tilde{X}_P = \begin{pmatrix} \text{DM} & \text{EV} & \text{VOL} & \text{CG} \end{pmatrix} \in \mathbb{R}^{N \times P}\]

with \(N = 413\) . We normalize each variable, and fix a confidence level of \(\alpha_c\) . Based on the correlation matrix above, we initialize with

\[X_1 = \begin{pmatrix} \text{VOL} \end{pmatrix}\]

which gives

\[\tilde{X}_{P-1} = \begin{pmatrix} \text{DM} & \text{EV} & \text{CG} \end{pmatrix} \in \mathbb{R}^{N \times P-1}\]

The iterative algorithm then proceeds. For each column of \(\tilde{X}_{P-1}\) – denoted by \(X_{P-1,j}\) for \(j = 1, \dots, P-1\) – we regress

\[X_{P-1,j} = \alpha_{j,1} + X_1\beta + \epsilon_{j,1}\]

and obtain estimates \(\hat{X}_{P-1,j}\) of \(X_{P-1,j}\) and estimated residuals

\[\hat{\epsilon}_{j,1} \in \mathbb{R}^N\]

for \(j = 1, \dots, P-1\) . We also regress \(Y = \text{spread to maturity}\) on \(X_1\) as

\[Y = \alpha_{Y,1} + X_1\beta + \epsilon_{Y,1}.\]

Again, we calculate estimated residuals of this model

\[\hat{\epsilon}_{Y,1} \in \mathbb{R}^N.\]

From these values, we calculate the partial correlation between each column of \(\tilde{X}_{P-1}\) and \(Y\) with respect to \(X_1\) as the correlations between \(\hat{\epsilon}_{j,1}\) and \(\hat{\epsilon}_{Y,1}\) . We choose as a candidate addition to the model the variable with highest partial correlation. Perhaps not surprisingly the next addition is the CreditGrades default probability.

We obtain a candidate model

\[Y = \alpha_{Y,1'} + X_1\beta + \epsilon_{Y,1'}.\]

where

\[X_{1'} = \begin{pmatrix} \text{VOL} & \text{CG} \end{pmatrix},\]

and conduct an \(F\) test with respect to the submodel consisting of \(X_1\) and intercept. The test statistic, \(f_{1'}\) is (ostensibly) distributed as \(F_{1,N-3}\) . If \(f_{1'}\) is less than the critical value of \(F_{1,N-3}\) determined by \(\alpha_c\) , then the iteration is terminated and the model is \(X_1\) with intercept. Otherwise, we add the factor to the model and update so that

\[X_2 = X_{1'}\]

and \(\tilde{X}_{P-2}\) is obtained by removing the newest addition to the model. In this case,

\[\tilde{X}_{P-2} = \begin{pmatrix} \text{DM} & \text{EV} \end{pmatrix}.\]

The iteration continues until the stopping condition is met or there are no more possible additions.

Here, Debt to Market Cap is added next, followed by the candidate addition of Enterprise Value which fails to reject the null hypothesis that the submodel is sufficiently explanatory.

Figure 4.1: Spread to maturity for bonds in the Merrill High Yield Index on 2/28/2016 against estimated values from a forward-selected model chosen.

Figure 4.1 shows the result of plotting the \(Y\) (spreads) against \(\hat{Y}\) as determined by the forward selection model. We expect to see, in a well-specified model, a linear relationship between the two with something close to homoscedasticity (uniform variance), allowing for the fact that estimated residual variances aren't constant as determined by (4.19).

Figure 4.2: Log spread to maturity for bonds in the Merrill High Yield Index on 2/28/2016 against estimated values from a forward-selected model chosen.

Based on the dispersion seen for higher spread values, we are motivated to consider a transformation of the dependent variable, in this case a log transform. We ensure positivity of yields (screening for issues in data transmission from the pricing vendor), and then carry out the same procedure as above with respect to log spreads. This time, the forward selection procedure picks out every available regressor. Figure 4.2 appears more tightly clustered around some linear relationship.

Backward Elimination

In the backwards elimination procedure, a confidence level \(\alpha\) is fixed and the initial design matrix consists of the full model with all \(P\) regressors. Denote this initial design matrix by \(X_0\) . The partial \(F\) -statistic for each regressor is then calculated, giving \(\{F_j\}\) , for \(j = 1, \dots, P\) . The minimum, \(F_-\) , of these \(F\) -statistics is compared to \(F_{1,N-P;1-\alpha}\) . If \(F_- > F_{1,N-P;1-\alpha}\) , the procedure terminates. Otherwise, we cannot reject that null hypothesis that the regressor associated with \(F_-\) should be omitted. The design matrix \(X_0\) is updated to \(X_1\) with this factor omitted. The procedure terminates with design matrix \(X_k\) if the associated \(F_-\) satisfies \(F_- > F_{1,N-(P-k);1-\alpha}\) .

Stepwise Regression

The two previous procedures may be utilized in tandem. For example, a forward selection procedure may be followed by backward elimination.

4.4 Generalized Least Squares

Throughout we have assumed that in

\[Y = X\beta + \epsilon\]

we had \(Cov(\epsilon) = \sigma^2 I\) . Here, we generalize to the case where \(Cov(\epsilon) = V\) for some matrix \(V \in \mathbb{R}^{N \times N}\) . Notice that this is equivalent – since all randomness in our model is derived from \(\epsilon\) – to \(Cov(Y) = V\) .

The general case is handled directly with the tools already in place. Throughout, we assume \(V \succ 0\) , and that the eigenvalues of \(V\) are unique. This implies that there exists an orthonormal change of basis matrix \(Q\) satisfying

\[QVQ' = \Lambda = \begin{pmatrix} \lambda_1 & & \\ & \ddots & \\ & & \lambda_N \end{pmatrix}\]

for diagonal matrix \(\Lambda\) . In what follows, we are guided by the goal of linearly transforming \(Y = X\beta + \epsilon\) via \(AY = AX\beta + A\epsilon\) so that

\[\begin{aligned} Cov(AY) &= Cov(A\epsilon) \\ &= ACov(\epsilon)A' \\ &= AVA' \end{aligned}\]

is a diagonal matrix. We will leverage the decomposition above. In essence, we seek a square root of \(V\) which we can invert.

For the matrix \(\Lambda\) above, define \(\Lambda^{\frac{1}{2}}\) as

\[\Lambda^{\frac{1}{2}} = \begin{pmatrix} \sqrt{\lambda_1} & & \\ & \ddots & \\ & & \sqrt{\lambda_N} \end{pmatrix}.\]

Clearly in this case we have that the inverse of \(\Lambda^{\frac{1}{2}}\) , which we denote by \(\Lambda^{-\frac{1}{2}}\) , is given by

\[\Lambda^{-\frac{1}{2}} = \begin{pmatrix} \frac{1}{\sqrt{\lambda_1}} & & \\ & \ddots & \\ & & \frac{1}{\sqrt{\lambda_N}} \end{pmatrix}.\]

Define \(P\) by

\[P = Q\Lambda^{\frac{1}{2}}Q' \tag{4.43}\]

for the same orthonormal change of basis matrix, \(Q\) , used for \(V\) . We have that

\[\begin{aligned} P^2 &= Q\Lambda^{\frac{1}{2}}Q'Q\Lambda^{\frac{1}{2}}Q' \\ &= Q\Lambda Q' \\ &= V. \end{aligned}\]

In addition, notice that \(P = P'\) . We may also show that \(P\) is positive definite. Lastly,

\[P^{-1} = Q\Lambda^{-\frac{1}{2}}Q'.\]

Looking back to our original regression equation, we have, then, that

\[P^{-1}Y = P^{-1}X\beta + P^{-1}\epsilon.\]

Defining

\[Y^* = P^{-1}Y \quad (4.44)\]

\[X^* = P^{-1}X \quad (4.45)\]

\[\epsilon^* = P^{-1}\epsilon, \quad (4.46)\]

the modified regression equation

\[Y^* = X^*\beta + \epsilon^* \quad (4.47)\]

satisfies

\[\begin{aligned} \text{Cov}(Y^*) &= \text{Cov}(\epsilon^*) \\ &= P^{-1}\text{Cov}(\epsilon)P^{-1} \\ &= P^{-1}VP^{-1} \\ &= P^{-1}(P^2)P^{-1} \\ &= I. \end{aligned}\]

We have then, that if \(E(\epsilon) = 0\) , (4.47) satisfies the Gauss-Markov assumptions (with \(\sigma^2 = 1\) ). Further, if the distributional assumption, \(\epsilon \sim N(0, V)\) obtains, then \(\epsilon^* \sim N(0, I)\) .

The projection matrix for (4.47) is

\[H^* = P^{-1}X(X'V^{-1}X)^{-1}X'P^{-1}. \quad (4.48)\]

The proof is left to the reader. As before, we have

\[H^*Y^* = X^*\hat{\beta}\]

with

\[\begin{aligned} \hat{\beta} &= (X^{*'}X)^{-1}X^{*'}Y^* \\ &= (X'V^{-1}X)^{-1}X'V^{-1}Y. \end{aligned} \quad (4.49)\]

Under the distributional assumption, we also have

\[\begin{aligned} \hat{\beta} &\sim N\left(\beta, (X^{*'}X)^{-1}\right) \\ &\sim N\left(\beta, (X'V^{-1}X)^{-1}\right). \end{aligned} \quad (4.50)\]

4.5 Collinearity and the Condition Number

Throughout this chapter we have required that the design matrix, \(X\) , be full rank, resulting in a nonsingular, \(X'X\) . We have also seen how the uncertainty in OLS parameter estimates is affected by this design matrix, specifically as the inverse, \((X'X)^{-1}\) is critical to defining the variance of these estimates. In this section we examine some features of the OLS estimate through the lens of linear algebra. In particular, we ask what impact small changes in \(Y\) might have in the estimate, \(\hat{\beta}\) .

Our approach will be via the matrix norm defined in (3.26) and a related measure called the condition number of a matrix. Consider the linear system of equations

\[Ax = b,\]

and assume for the moment that \(A\) is invertible.

For nonsingular matrices, we define the condition number, here fixing the matrix norm as \(||\cdot||_2\) as in (3.26), but noting that any norm may be used, as

\[\kappa(A) = ||A||_2 ||A^{-1}||_2. \tag{4.51}\]

Looking back at the system of equations, we have, for some perturbation in \(b\) , \(\Delta b\) , that there is some new solution \(x + \Delta x\) ; viz.,

\[A(x + \Delta x) = b + \Delta b.\]

We would like to understand how relative changes in \(b\) , \(\frac{||\Delta b||}{||b||}\) , translate to relative changes in \(x\) , \(\frac{||\Delta x||}{||x||}\) . Looking at the original system, we have

\[||Ax|| = ||b|| \le ||A||_2 ||x||,\]

and, similarly, the solution to the perturbed system must satisfy \(A\Delta x = \Delta b\) , so that

\[||\Delta x|| = ||A^{-1}\Delta b|| \le ||A^{-1}||_2 ||\Delta b||.\]

Putting these two inequalities together, we have

\[\begin{aligned} \frac{||\Delta x||}{||x||} &\le ||A||_2 ||A^{-1}||_2 \frac{||\Delta b||}{||b||} \\ &= \kappa(A) \frac{||\Delta b||}{||b||}. \end{aligned} \tag{4.52}\]

Interpreting this result, we see that relative changes in \(b\) scaled by the condition number of \(A\) provide an upper bound for the resulting relative change in \(x\) . This bound may be made tighter, and in doing so, one may produce examples where the inequality gives an equality; i.e., there is no lower bound. This is slightly beyond the scope of what we provide here, however.

Even so, (4.52) remains insightful. For, all things being equal, we might prefer a smaller condition number than a greater one. It is somewhat direct to

show that in the case of a nonsingular, \(A\) , we have \(\kappa(A) \in [1, \infty)\) , with \(\kappa(A) = 1\) indicating that in some basis, \(A\) is a scalar multiple of the identity.

We leave as an exercise to show that for nonsingular matrices,

\[||A^{-1}||_2 = \frac{1}{\min_{||v||=1} ||Av||},\]

thus implying that if a matrix is ‘close to singular’, then the condition number tends to infinity. This may be understood directly by resorting to an eigenvalue interpretation, where singularity is identified with zero valued eigenvalues. In our present work, we have then, that near-multicollinearity in the columns of \(A\) results in a large condition number, and solutions very sensitive to inputs.

In the case of ordinary least squares, the familiar setup

\[X\hat{\beta} = \hat{Y}\]

implies that

\[||\hat{Y}|| \le ||X||_2 ||\hat{\beta}||,\]

where as usual, \(\hat{\beta} = (X'X)^{-1}X'Y\) . In the case that \(\hat{Y}\) is perturbed by \(\Delta\hat{Y}\) , we have as above that

\[||\Delta\hat{\beta}|| \le ||X^\dagger||_2 ||\Delta\hat{Y}||,\]

where \(X^\dagger = (X'X)^{-1}X'\) . The matrix, \(X^\dagger\) , in the present case is called the Moore-Penrose pseudoinverse. The above gives that

\[\begin{aligned}\frac{||\Delta\hat{\beta}||}{||\hat{\beta}||} &\le ||X||_2 ||X^\dagger||_2 \frac{||\Delta\hat{Y}||}{||\hat{Y}||} \\ &= \kappa(X) \frac{||\Delta\hat{Y}||}{||\hat{Y}||},\end{aligned}\tag{4.53}\]

where we have defined the condition number, \(\kappa(X)\) , for non-square, but full rank matrices via \(X^\dagger\) . Notice that these definitions coincide for invertible (square) matrices, and that if we allow for non-singularity and rank-deficient matrices to have an infinite condition number, for matrices generally.

The condition number of the design matrix \(X\) , then, is a direct measure of the sensitivity of the parameter estimate, \(\hat{\beta}\) . We say that a matrix is ill-conditioned if its condition number is very large, and well conditioned otherwise. In the context of collinearity and ordinary least squares, then, we see that variable selection is critical, and a preference for orthogonality over similarity generally obtains. Conversely, and in practice, identifying that a particular design matrix is ill-conditioned signals to the practitioner that their choice of predictors is in need of culling as one or more inputs are near some vector in the span of the remaining variables.

Exercises

1. Show that for any sequences of random variables \(\{u_t\}\) and \(\{v_t\}\) ,

\[\sum_{t=1}^{T} u_t v_t = (T-1)\hat{\sigma}_{uv} + T\bar{u}\bar{v},\]

and

\[\sum_{t=1}^{T} u_t^2 = (T-1)\hat{\sigma}_u^2 + T\bar{u}^2.\]

2. Prove that the least squares estimate of \(\beta\) for the model

\[r_t - r_f = \alpha + \beta(m_t - r_f) + \epsilon_t\]

is exactly

\[\beta = \frac{\hat{\sigma}_{rm}}{\hat{\sigma}_m^2}.\]

3. Using the model in the preceding problem, show that \(\hat{\sigma}_{m\epsilon} = 0\) .
4. Let \(\{X_i\}_{i=1}^N\) be univariate normal random variables, and assume that the \(X_i\) 's are iid. Given observations \(\{x_i\}_{i=1}^N\) , show that the maximum likelihood estimator of variance is given by

\[\hat{s}_N^2 = \frac{1}{N} \sum_{i=1}^{N} (x_i - \mu)^2.\]

What is \(\mathbb{E}(\hat{s}_N^2)\) ?

5. For \(i = 1, 2\) , let \(\{r_{i,t}\}_{t=1}^{N_i}\) be iid with \(\text{Var}(r_i) = \sigma^2\) . Let

\[\hat{M}_i = \frac{1}{N_i} \sum_{t=1}^{N_i} r_{i,t}\]

be an estimator of \(\mu_i = \mathbb{E}(r_i)\) for \(i = 1, 2\) , and assume \(\hat{M}_1\) and \(\hat{M}_2\) are independent.

(a) What is \(\sigma_{12}^2 = \text{Var}(\hat{M}_1 - \hat{M}_2)\) ?
(b) Assume further that \(r_i \sim N(\mu_i, \sigma^2)\) , and let \(s_i^2\) be the unbiased estimator of variance shown previously. Let

\[s^2 = \frac{(N_1 - 1)s_1^2 + (N_2 - 1)s_2^2}{N_1 + N_2 - 2}.\]

i. What is \(\mathbb{E}(s^2)\) ?
ii. How is \(s^2\) distributed?

iii. Show

\[\frac{\frac{(\hat{M}_1 - \mu_1) - (\hat{M}_2 - \mu_2)}{\sigma_{12}}}{\sqrt{\frac{s^2}{\sigma^2}}} = \frac{(\hat{M}_1 - \mu_1) - (\hat{M}_2 - \mu_2)}{s\sqrt{\left(\frac{1}{N_1} + \frac{1}{N_2}\right)}}.\]

iv. How is this last ratio distributed?

v. Construct a 95% confidence interval for \(\mu_1 - \mu_2\) .

vi. Write down the confidence interval you would use to check the null hypothesis at confidence level \(\alpha = 5\%\) that the means are the same; i.e.,

\[H_0: \mu_1 = \mu_2.\]

6. Suppose \(\hat{\beta}_1\) and \(\hat{\beta}_2\) each minimize

\[\sum_{t=1}^{T} (y_{i,t} - \beta_i x_t)^2\]

for \(i = 1\) and 2, respectively, where \(y_i \in \mathbb{R}\) and \(x \in \mathbb{R}\)

\[f(\beta_1, \beta_2) = \sum_{t=1}^{T} (w_1 y_{1,t} + w_2 y_{2,t} - w_1 \beta_1 x_t - w_2 \beta_2 x_t)^2\]

Explain why the CAPM \(\beta\) of two portfolios is the weighted sum of the \(\beta\) 's.

7. Prove that (4.18):

\[\hat{\epsilon} = (I - H)\epsilon.\]

8. Let \(H\) and \(G\) be projections. Denote the space spanned by the columns of a matrix, \(A\) , by \(\text{span}(A)\) , and let \(\mathcal{L} = \text{span}(H)\) and \(\mathcal{L}_0 = \text{span}(G)\) . If \(\mathcal{L}_0 \subseteq \mathcal{L}\) , show that \(H - G\) is a projection.

9. Prove (4.39) under the extended Gauss-Markov assumptions.

10. Prove (4.42).

11. Verify the results regarding the cross-sectional performance of \(\beta\) . Expand the findings by performing a double sort with \(\beta\) and then correlation. In each case, report annualized returns, annualized volatility, and the ratio of the two for each quintile.

12. Using the same data as in the long-short volatility portfolio analysis, produce a portfolio which is \(\beta\) neutral at construction using even weight decile portfolios sorted on ex ante CAPM \(\beta\) (found using historical returns and including an intercept term, but ignoring the risk free rate). Report the average monthly return to your portfolio, its \(\alpha\) and \(\beta\) , and test the null hypothesis that \(\alpha = 0\) at the 0.05 confidence level.

13. Prove directly that \(P\) in (4.43) is positive definite by showing that for a given vector \(w \neq 0\) , \(w'Pw > 0\) .

14. The jackknifed residuals shown in (4.19) used the population variance of residuals, \(\sigma\) . We define the internally studentized residuals as

\[\tilde{\epsilon}_i = \frac{\hat{\epsilon}_i}{s\sqrt{1-h_{ii}}} \quad (4.54)\]

with \(s\) the unbiased estimator of \(\sigma\) given by (4.23). Using the results of the chapter, how is \(\tilde{\epsilon}_i\) distributed? How might you use these results to detect and remove outliers from a regression?

1. Verify (4.34) following the proof of (4.29).
2. Explain qualitatively the result of (4.35) and outline potential uses.
3. Prove (4.48).
4. Show that for a nonsingular matrix, \(A\) ,

\[||A^{-1}||_2 = \frac{1}{\min_{||v||=1} ||Av||}.\]

1. Prove that the \(||\cdot||_2\) is invariant under a change of basis. Conclude that the condition number of a square matrix with real eigenvalues is \(\frac{\max_i |\lambda_i|}{\min_i |\lambda_i|}\) , where \(i\) ranges over all eigenvalues of the matrix, and the convention of infinite condition number for nonsingular matrices is used.
2. Use a backward elimination procedure for the spread data discussed in this chapter and the four factors provided. Report the variables chosen at each stage along with associated test statistics and \(p\) values. Repeat the process for log spreads.
3. Use a forward selection followed by backward elimination procedure for the spread data discussed in this chapter and the four factors provided. Report the variables chosen at each stage along with associated test statistics and \(p\) values. Repeat the process for log spreads.
4. The file fama_french.txt contains monthly performance numbers for various portfolios: the market excess return (MARKET), a long-short portfolio on size (SMB), and a long-short value portfolio (HML). For each portfolio:

(a) Fit the regression model

\[r_t = \mu_1 \mathbf{1}_{\{t<\tau\}} + \mu_2 \mathbf{1}_{\{t\ge\tau\}} + \epsilon_t\]

with \(\tau = 12/31/1969\) (196912).

(b) Test the null hypothesis that \(\mu_1 = \mu_2\) at the 95% confidence level.