Skip to contents

Variance Estimation Testing Using A Simple Linear Specification

Mark Fredrickson, Ben Hansen

2023-03-30

Source: vignettes/LinearModelVarianceEstimation.Rmd
LinearModelVarianceEstimation.Rmd

The goal of this document is to consider a simple model that can be arranged as two separate regressions and relate the variance of the second stage coefficients to those that are fit in the entire model. With this information, we should be able to create tests of our variance estimation routines.

Combined model

Consider two sets of background variables and treatment assignments, xx and zz with dimension pxp_x and pzp_z, respectively, to be treated as nonrandom or fixed by conditioning. While we allow correlation between the variables, we will assume that the matrix (XZ)′(XZ)\begin{pmatrix} X & Z \end{pmatrix}' \begin{pmatrix} X & Z \end{pmatrix} has an inverse (I think most of these results would work with generalized inverses, but if does not exist it makes the dimesionality of the coefficients a little tricky).

Y=α′x+β′z+ϵ,E(ϵ)=0,Var(ϵ)=σ2Y = \alpha'x + \beta'z + \epsilon, \quad E(\epsilon) = 0, \text{Var}(\epsilon) = \sigma^2

Standard results give us

(α̂1β̂1)=(X′XX′ZZ′XZ′Z)−1(X′yZ′y) \begin{pmatrix} \hat \alpha_1 \\ \hat \beta_1 \end{pmatrix} = \begin{pmatrix} X'X & X'Z \\ Z'X & Z'Z \end{pmatrix}^{-1} \begin{pmatrix} X'y \\ Z'y \end{pmatrix} and that the variance of the estimators is σ2(X′XX′ZZ′XZ′Z)−1\sigma^2 \begin{pmatrix} X'X & X'Z \\ Z'X & Z'Z \end{pmatrix}^{-1} Results for blocked matrices (e.g., The Matrix Cookbook) give the variance for just β̂\hat \beta as Var(β̂1)=σ2[Z′Z−Z′X(X′X)−1X′Z]−1\text{Var}(\hat \beta_1) = \sigma^2 \left[Z'Z - Z' X (X'X)^{-1} X'Z\right]^{-1} Write H=I−X(X′X)−1X′H = I - X (X'X)^{-1} X', matrix that creates the residuals of the regression on xx alone. Then, Var(β̂1)=σ2[Z′HZ]−1\text{Var}(\hat \beta_1) = \sigma^2 \left[Z'H Z\right]^{-1}

The same β̂1\hat \beta_1 arises as the coefficient of a regression of HYHY on HZHZ (by the so-called Frisch-Waugh-Lovell Theorem). In consequence, β̂1=(Z′HZ)−1(Z′Hy).\hat \beta_1 = (Z'HZ)^{-1}(Z'Hy) .

Two regressions

Let α̂2\hat \alpha_2 and β̂2\hat \beta_2 be the estimators from first regression on YY on xx alone and then regressing Y−α̂2′xY - \hat \alpha_2'x, that is YHYH with HH as defined above, on ZZ. Standard results give

α̂2=(X′X)−1X′y\hat \alpha_2 = (X'X)^{-1} X' y and β̂2=(Z′Z)−1Z′(y−Xα̂2)=(Z′Z)−1Z′(y−X(X′X)−1X′y)=(Z′Z)−1Z′Hy\hat \beta_2 = (Z'Z)^{-1} Z' (y - X \hat \alpha_2) = (Z'Z)^{-1} Z'(y - X (X'X)^{-1} X' y) = (Z'Z)^{-1} Z' H y

As both ZZ and XX are taken to be nonrandom, we may pass between β̂1\hat \beta_1 and β̂2\hat \beta_2 via nonrandom linear transformations, as follows: β̂1=(Z′HZ)−1(Z′Z)β̂2;β̂2=(Z′Z)−1(Z′HZ)β̂1.\hat \beta_1 = (Z'HZ)^{-1}(Z'Z) \hat \beta_2;\quad \hat \beta_2 = (Z'Z)^{-1}(Z'HZ) \hat \beta_1 . Accordingly Cov(β̂1)=(Z′HZ)−1(Z′Z)Cov(β̂2)(Z′Z)(Z′HZ)−1 \operatorname{Cov}(\hat \beta_1) = (Z'HZ)^{-1}(Z'Z) \operatorname{Cov}(\hat \beta_{2})(Z'Z) (Z'HZ)^{-1} which may serve as a basis for tests.

TODO

  • Implement as tests.