16  Generalized least squares

References: Chapter 3. Gardini A. (2000).

16.1 Working hypothesis

The assumptions of the generalized least squares estimator are:

1. $\mathbb{E}\{y_i\mid {\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}=\mathbb{E}\{y_i\mid \mathbf{X}\}={\mathbf{x}}_i^{\mathrm{\top }}\mathbf{b}$.
2. $\mathbb{V}\{y_i\mid {\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}=\mathbb{V}\{y_i\mid \mathbf{X}\}={\sigma }_i^2$ with $0<{\sigma }_i^2<\mathrm{\infty }$.
3. $\mathbb{C}v\{y_i,y_j\mid {\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}=\mathbb{C}v\{y_i,y_j\mid \mathbf{X}\}={\sigma }_{ij}$.

Equivalently the formulation of the assumptions in terms of the stochastic component $\mathbf{e}$ are

1. $y_i={\mathbf{x}}_i^{\mathrm{\top }}\mathbf{b}+e_i$ for $i=1,\dots n$.
2. $\mathbb{E}\{e_i\mid {\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}=\mathbb{E}\{e_i\mid \mathbf{X}\}=0$.
3. $\mathbb{V}\{e_i\mid {\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}=\mathbb{V}\{e_i\mid \mathbf{X}\}={\sigma }_i^2$ with $0<{\sigma }_i^2<\mathrm{\infty }$.
4. $\mathbb{C}v\{e_i,e_j\mid {\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}=\mathbb{C}v\{e_i,e_j\mid \mathbf{X}\}={\sigma }_{ij}$

In this case the variance covariance matrix $\mathbf{\Sigma }$ is defined as in Equation 14.16 and contains the variances and the covariances between the observations.

16.2 Generalized least squares estimator

Proposition 16.1 ($\mathbf{\text{Generalized Least Squares (GLS)}}$)
The generalized least squares estimator (GLS) is the function ${\text{Q}}^{\text{GLS}}$ that minimize the weighted sum of the squared residuals and return an estimate of the true parameter $\mathbf{b}$, i.e.  $$\begin{array}{}\text{(16.1)}& {\text{Q}}^{\text{GLS}}(\mathbf{b})=\mathbf{e}(\mathbf{b}{)}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{e}(\mathbf{b})\text{.}\end{array}$$ Formally, the GLS estimator is the solution of the following minimization problem, i.e.  $$\begin{array}{}\text{(16.2)}& {\mathbf{b}}^{\text{GLS}}=\underset{\mathbf{b}\in {\mathrm{\Theta }}_{\mathbf{b}}}{\text{argmin}}\{{\text{Q}}^{\text{GLS}}(\mathbf{b})\}\text{.}\end{array}$$ Notably, if $\mathbf{X}$ and $\mathbf{\Sigma }$ are non-singular one obtain an analytic expression, i.e.
$$\begin{array}{}\text{(16.3)}& {\mathbf{b}}^{\text{GLS}}=({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{y}\text{.}\end{array}$$

Singularity of $\mathbf{X}$ or $\mathbf{\Sigma }$

The solution is available if and only if $\mathbf{X}$ and $\mathbf{\Sigma }$ are non-singular. In practice the conditions are:

1. $\text{rank}(\mathbf{\Sigma })=max=n$ for the inversion of $\mathbf{\Sigma }$.
2. $\text{rank}(\mathbf{X})=max=k$ and condition 1. for the inversion of ${\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}$.

Proof: Proposition 16.1

Proof. Let’s prove the optimal solution in Proposition 16.1. Developing the optimization problem in Equation 16.1: $$\begin{array}{rl}{\text{Q}}^{\text{GLS}}(\mathbf{b})& =\mathbf{e}(\mathbf{b}{)}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{e}(\mathbf{b})=\\ & =(\mathbf{y}-\mathbf{X}\mathbf{b}{)}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}(\mathbf{y}-\mathbf{X}\mathbf{b})=\\ & ={\mathbf{y}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{y}-2{\mathbf{b}}^{\mathrm{\top }}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{y}+{\mathbf{b}}^{\mathrm{\top }}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}\mathbf{b}\end{array}$$ In order to minimize the above expression, let’s compute the first derivative of ${\text{Q}}^{\text{GLS}}(\mathbf{b})$ with respect to $\mathbf{b}$ $$\frac{d{\text{Q}}^{\text{GLS}}(\mathbf{b})}{d\mathbf{b}}=-2{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{y}+2{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}\mathbf{b}\text{.}$$ Then, setting the above expression equal to zero and solving for $\mathbf{b}={\mathbf{b}}^{\text{GLS}}$ gives the solution, i.e. $${\mathbf{b}}^{\text{GLS}}=({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{y}\text{.}$$

16.3 Properties GLS

Theorem 16.1 ($\mathbf{\text{Aikten theorem}}$)
Under the following working hypothesis, also called Aikten hypothesis, i.e. 

1. $\mathbf{y}=\mathbf{X}\mathbf{b}+\mathbf{e}$.
2. $\mathbb{E}\{\mathbf{e}\}=0$.
3. $\mathbb{E}\{\mathbf{e}{\mathbf{e}}^{\mathrm{\top }}\}=\mathbf{\Sigma }$, i.e. heteroskedastic and correlated errors.
4. $\mathbf{X}$ is non-stochastic and independent from the errors $\mathbf{e}$ for all $n$’s.

The Generalized Least Square (GLS) estimator is $\mathbf{\text{BLUE}}$ (Best Linear Unbiased Estimator), where “best” stands for the estimator with minimum variance in the class of linear unbiased estimators of $\mathbf{b}$.

Proposition 16.2 ($\mathbf{\text{Properties GLS estimator}}$)

1. Unbiased: ${\mathbf{b}}^{\text{GLS}}$ is correct and it’s conditional expectation is equal to true parameter in population, i.e.  $$\begin{array}{}\text{(16.4)}& \mathbb{E}\{{\mathbf{b}}^{\text{GLS}}\mid \mathbf{X}\}=\mathbf{b}\text{.}\end{array}$$

2. Linear in the sense that it can be written as a linear combination of $\mathbf{y}$ and $\mathbf{X}$, i.e. ${\mathbf{b}}^{\text{GLS}}={\mathbf{A}}_{\text{x}}\mathbf{y}$, where ${\mathbf{A}}_{\text{x}}$ do not depend on $\mathbf{y}$, i.e.
   $$\begin{array}{}\text{(16.5)}& {\mathbf{b}}^{\text{GLS}}={\mathbf{A}}_{\text{x}}\mathbf{y}{\mathbf{A}}_{\text{x}}=({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\text{.}\end{array}$$

3. Under the Aikten hypothesis (Theorem 16.1) it has minimum variance in the class of the unbiased linear estimators and it reads: $$\begin{array}{}\text{(16.6)}& \mathbb{V}\{{\mathbf{b}}^{\text{GLS}}\mid \mathbf{X}\}=({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}\text{.}\end{array}$$

Proof: Proposition 16.2

Proof. The GLS estimator is correct. It’s expected value is computed from Equation 16.3 and substituting Equation 14.12, is equal to the true parameter in population, i.e.
$$\begin{array}{}\text{(16.7)}& \begin{array}{rl}\mathbb{E}\{{\mathbf{b}}^{\text{GLS}}\mid \mathbf{X}\}& =\mathbb{E}\{({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{y}\}=\\ & =\mathbb{E}\{({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}(\mathbf{X}\mathbf{b}+\mathbf{e})\}=\\ & =({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}\mathbf{b}+({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbb{E}\{\mathbf{e}\mid \mathbf{X}\}=\\ & =\mathbf{b}\end{array}\end{array}$$

Under the assumption of heteroskedastic and correlated observations the conditional variance of ${\mathbf{b}}^{\text{GLS}}$ follows similarly as for the OLS case (Equation 15.11) but with $\mathbb{V}\{\mathbf{e}\mid \mathbf{X}\}=\mathbf{\Sigma }$, i.e. $$\begin{array}{}\text{(16.8)}& \begin{array}{rl}\mathbb{V}\{{\mathbf{b}}^{\text{GLS}}\mid \mathbf{X}\}& =({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbb{V}\{\mathbf{e}\mid \mathbf{X}\}{\mathbf{\Sigma }}^{-1}\mathbf{X}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}=\\ & =({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{\Sigma }{\mathbf{\Sigma }}^{-1}\mathbf{X}({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}=\\ & =({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}=\\ & =({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}\end{array}\end{array}$$ where Equation 15.10 become a special case of Equation 16.8 where $\mathbf{\Sigma }={\sigma }_e^2{\mathbf{\text{I}}}_n$.

16.4 Alternative derivation

Let’s consider a linear model of the form $\mathbf{y}=\mathbf{X}\mathbf{b}+\mathit{\epsilon }$ and a transformation matrix ${\mathbf{T}}_{n\times n}$. Multiplying on both sides by $\mathbf{T}$, the model can be rewritten as follows: $$\begin{array}{rl}& \mathbf{T}\mathbf{y}=\mathbf{T}\mathbf{X}\mathbf{b}+\mathbf{T}\mathit{\epsilon }\\ & \Downarrow \Downarrow \Downarrow \\ & \tilde{\mathbf{y}}=\tilde{\mathbf{X}}\mathbf{b}+\tilde{\mathit{\epsilon }}\end{array}$$ The conditional mean of the transformed models reads as: $$\mathbb{E}\{\tilde{\mathbf{y}}\mid \tilde{\mathbf{X}}\}=\tilde{\mathbf{X}}\mathbf{b}$$ while it’s conditional variance $$\mathbb{V}\{\tilde{\mathbf{y}}\mid \tilde{\mathbf{X}}\}=\mathbb{V}\{\tilde{\mathit{\epsilon }}\mid \tilde{\mathbf{X}}\}=\mathbf{T}\mathbf{\Sigma }{\mathbf{T}}^{\mathrm{\top }}$$ The idea is to identify a transformation matrix $\mathbf{T}$ such that the conditional variance became equal to the identity matrix, i.e. $\mathbb{V}\{\tilde{\mathit{\epsilon }}\mid \tilde{\mathbf{X}}\}={\mathbf{\text{I}}}_n$. In this way it is possible to work under the Gauss-Markov assumptions obtaining an estimator with minimum variance. Let’s decompose the variance-covariance matrix (Equation 14.16) as $$\mathbf{\Sigma }=\mathbf{e}\mathbf{\Lambda }{\mathbf{e}}^{\mathrm{\top }}$$ where

• $\mathbf{\Lambda }$ is the diagonal matrix containing the eigenvalues.
• $\mathbf{e}$ is the matrix with the eigenvectors that satisfy the following relation, i.e. ${\mathbf{e}}^{\mathrm{\top }}\mathbf{e}=\mathbf{e}{\mathbf{e}}^{\mathrm{\top }}={\mathbf{\text{J}}}_n$.

Setting the transformation matrix as $\mathbf{T}={\mathbf{\Lambda }}^{-1/2}{\mathbf{e}}^{\mathrm{\top }}$ gives that the conditional variance is equal to 1 for all the observations, i.e.
$$\begin{array}{rl}\mathbb{V}\{\tilde{\mathit{\epsilon }}\mid \tilde{\mathbf{X}}\}& =\mathbf{T}\mathbf{\Sigma }{\mathbf{T}}^{\mathrm{\top }}=\\ & =({\mathbf{\Lambda }}^{-1/2}{\mathbf{e}}^{\mathrm{\top }})\mathbf{e}\mathbf{\Lambda }{\mathbf{e}}^{\mathrm{\top }}(\mathbf{e}{\mathbf{\Lambda }}^{-1/2})=\\ & ={\mathbf{\Lambda }}^{-1/2}\mathbf{\Lambda }{\mathbf{\Lambda }}^{-1/2}={\mathbf{\text{J}}}_n\end{array}$$ Moreover, the matrix $\mathbf{T}={\mathbf{\Lambda }}^{-1/2}{\mathbf{e}}^{\mathrm{\top }}$ satisfies the product: $$\begin{array}{}\text{(16.9)}& {\mathbf{T}}^{\mathrm{\top }}\mathbf{T}=\mathbf{e}{\mathbf{\Lambda }}^{-1/2}{\mathbf{\Lambda }}^{-1/2}{\mathbf{e}}^{\mathrm{\top }}=\mathbf{e}{\mathbf{\Lambda }}^{-1}{\mathbf{e}}^{\mathrm{\top }}={\mathbf{\Sigma }}^{-1}\end{array}$$ Finally, substituting $\tilde{\mathbf{X}}=\mathbf{T}\mathbf{X}$ in the OLS formula (Equation 15.3) and using the result Equation 16.9 one obtain exactly the GLS estimator in Equation 16.3, i.e.  $$\begin{array}{rl}\tilde{\mathbf{b}}& =({\tilde{\mathbf{X}}}^{\mathrm{\top }}\tilde{\mathbf{X}}{)}^{-1}{\tilde{\mathbf{X}}}^{\mathrm{\top }}\tilde{\mathbf{y}}=\\ & =({\mathbf{X}}^{\mathrm{\top }}{\mathbf{T}}^{\mathrm{\top }}\mathbf{T}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{T}}^{\mathrm{\top }}\mathbf{T}\mathbf{y}=\\ & =({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{y}\end{array}$$

16.5 Models with heteroskedasticity

16.5.1 Working hypothesis

The assumptions of the generalized linear model with heteroskedastic errors are:

1. $\mathbb{E}\{y_i\mid {\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}=\mathbb{E}\{y_i\mid \mathbf{X}\}={\mathbf{x}}_i^{\mathrm{\top }}\mathbf{b}$.
2. $\mathbb{V}\{y_i\mid {\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}=\mathbb{V}\{y_i\mid \mathbf{X}\}={\sigma }_i^2$ with $0<{\sigma }_i^2<\mathrm{\infty }$.
3. $\mathbb{C}v\{y_i,y_j\mid {\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}=\mathbb{C}v\{y_i,y_j\mid \mathbf{X}\}=0$

equivalently the formulation in terms of the stochastic component

1. $y_i={\mathbf{x}}_i^{\mathrm{\top }}\mathit{\beta }+e_i$ for $i=1,\dots n$.
2. $\mathbb{E}\{e_i\mid {\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}=\mathbb{E}\{e_i\mid \mathbf{X}\}=\mathbf{0}$.
3. $\mathbb{V}\{e_i\mid {\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}=\mathbb{V}\{e_i\mid \mathbf{X}\}={\sigma }_i^2$ with $0<{\sigma }_i^2<\mathrm{\infty }$.
4. $\mathbb{C}v\{e_i,e_j\mid {\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}=\mathbb{C}v\{e_i,e_j\mid \mathbf{X}\}=0$

For an heteroskedastic linear model the variance-covariance matrix of the residuals in matrix notation is written as: $$\underset{n\times n}{\mathbf{\Sigma }}=\left(\begin{array}{cccc}{\sigma }_1^2& 0& \dots & 0\\ 0& {\sigma }_2^2& \dots & 0\\ ⋮& ⋮& & ⋮\\ 0& 0& \dots & {\sigma }_n^2\end{array}\right)=\text{diag}\left(\begin{array}{c}{\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_n^2\end{array}\right)$$

Gardini A., Costa M., Cavaliere G. 2000. Econometria, Volume Primo. FrancoAngeli.