16  Generalized least squares

References: Chapter 3. Gardini A. (2007).

16.1 Working hypothesis

The assumptions of the generalized least squares estimator are:

1. The linear model approximate the conditional expectation, i.e. $\mathbb{E}\{Y_i\mid {\mathbf{x}}_i\}={\mathbf{x}}_i^{\mathrm{\top }}\mathbf{b}$.
2. The conditional variance of the response variable $Y$ depends on the observation $i$, i.e. $\mathbb{V}\{Y_i\mid {\mathbf{x}}_i\}={\sigma }_i^2$ with $0<{\sigma }_i^2<\mathrm{\infty }$ for all $i$ with $i=1,\dots ,n$.
3. The response variables $Y$ are correlated $\mathbb{C}v\{Y_i,Y_j\mid {\mathbf{x}}_i\}={\sigma }_{ij}$ for all $i\ne j$ and $i,j=1,\dots ,n$.

Equivalently the formulation of the assumptions in terms of the stochastic component $\mathbf{u}$ are

1. The residuals have mean zero, i.e. $\mathbb{E}\{u_i\mid {\mathbf{x}}_i\}=0$ for all $i$ with $i=1,\dots ,n$.
2. The conditional variance of the residuals depends on the observation $i$, i.e. $\mathbb{V}\{u_i\mid {\mathbf{x}}_i\}={\sigma }_i^2$ with $0<{\sigma }_i^2<\mathrm{\infty }$.
3. The residuals are correlated, i.e. $\mathbb{C}v\{u_i,u_j\mid {\mathbf{x}}_i\}={\sigma }_{ij}$ for all $i\ne j$ and $i,j=1,\dots ,n$.

In this case the variance covariance matrix $\mathbf{\Sigma }$ is defined as in Equation 14.15 and contains the variances and the covariances between the observations.

16.2 Generalized least squares estimator

Proposition 16.1 ($\mathbf{\text{Generalized Least Squares (GLS)}}$)
The generalized least squares estimator (GLS) is the function ${\text{Q}}^{\text{GLS}}$ that minimize the weighted sum of the squared residuals and return an estimate of the true parameter $\mathbf{b}$, i.e.  $$\begin{array}{}\text{(16.1)}& {\text{Q}}^{\text{GLS}}(\mathbf{b})=\hat{\mathbf{u}}(\mathbf{b}{)}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\hat{\mathbf{u}}(\mathbf{b})\text{.}\end{array}$$ Formally, the GLS estimator is the solution of the following minimization problem, i.e.  $$\begin{array}{}\text{(16.2)}& {\mathbf{b}}^{\text{GLS}}=\underset{\mathbf{b}\in {\mathrm{\Theta }}_{\mathbf{b}}}{\text{argmin}}\{{\text{Q}}^{\text{GLS}}(\mathbf{b})\}\text{.}\end{array}$$ Notably, if $\mathbf{X}$ and $\mathbf{\Sigma }$ are non-singular one obtain an analytic expression, i.e.
$$\begin{array}{}\text{(16.3)}& {\mathbf{b}}^{\text{GLS}}=({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{y}\text{.}\end{array}$$

Singularity of $\mathbf{X}$ or $\mathbf{\Sigma }$

The solution is available if and only if $\mathbf{X}$ and $\mathbf{\Sigma }$ are non-singular. In practice the conditions are:

1. $\text{rank}(\mathbf{\Sigma })=n$ for the inversion of $\mathbf{\Sigma }$.
2. $\text{rank}(\mathbf{X})=k$ and condition 1. for the inversion of ${\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}$.

Proof: Proposition 16.1

Proof. Let’s prove the optimal solution in Proposition 16.1. Developing the optimization problem in Equation 16.1: $$\begin{array}{rl}{\text{Q}}^{\text{GLS}}(\mathbf{b})& =\hat{\mathbf{u}}(\mathbf{b}{)}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\hat{\mathbf{u}}(\mathbf{b})=\\ & =(\mathbf{y}-\mathbf{X}\mathbf{b}{)}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}(\mathbf{y}-\mathbf{X}\mathbf{b})=\\ & ={\mathbf{y}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{y}-2{\mathbf{b}}^{\mathrm{\top }}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{y}+{\mathbf{b}}^{\mathrm{\top }}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}\mathbf{b}\end{array}$$ In order to minimize the above expression, let’s compute the first derivative of ${\text{Q}}^{\text{GLS}}(\mathbf{b})$ with respect to $\mathbf{b}$ $$\frac{d{\text{Q}}^{\text{GLS}}(\mathbf{b})}{d\mathbf{b}}=-2{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{y}+2{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}\mathbf{b}\text{.}$$ Then, setting the above expression equal to zero and solving for $\mathbf{b}={\mathbf{b}}^{\text{GLS}}$ gives the solution, i.e. $${\mathbf{b}}^{\text{GLS}}=({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{y}\text{.}$$

Proposition 16.2 ($\mathbf{\text{Two-stage derivation of GLS estimator}}$)
The GLS estimator in Equation 16.3 can be equivalently recovered as $${\mathbf{b}}^{\text{GLS}}=({\mathbf{X}}^{\mathrm{\top }}{\mathbf{T}}^{\mathrm{\top }}\mathbf{T}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{T}}^{\mathrm{\top }}\mathbf{T}\mathbf{y}\text{,}$$ where $\mathbf{T}={\mathbf{\Lambda }}^{-1/2}{\mathbf{e}}^{\mathrm{\top }}$ with $\mathbf{\Sigma }=\mathbf{e}\mathbf{\Lambda }{\mathbf{e}}^{\mathrm{\top }}$ and

• $\mathbf{\Lambda }$ is the diagonal matrix containing the eigenvalues of $\mathbf{\Sigma }$.
• $\mathbf{e}$ is the matrix with the eigenvectors of $\mathbf{\Sigma }$ that satisfies the following relation, i.e. ${\mathbf{e}}^{\mathrm{\top }}\mathbf{e}=\mathbf{e}{\mathbf{e}}^{\mathrm{\top }}={\mathbf{\text{J}}}_n$.

Moreover, the matrix $\mathbf{T}={\mathbf{\Lambda }}^{-1/2}{\mathbf{e}}^{\mathrm{\top }}$ satisfies the product: $$\begin{array}{}\text{(16.4)}& {\mathbf{T}}^{\mathrm{\top }}\mathbf{T}=\mathbf{e}{\mathbf{\Lambda }}^{-1/2}{\mathbf{\Lambda }}^{-1/2}{\mathbf{e}}^{\mathrm{\top }}=\mathbf{e}{\mathbf{\Lambda }}^{-1}{\mathbf{e}}^{\mathrm{\top }}={\mathbf{\Sigma }}^{-1}\end{array}$$

Proof: Proposition 16.2

Proof. Let’s consider a linear model of the form $$\mathbf{y}=\mathbf{X}\mathbf{b}+\mathbf{u}\text{,}$$ and let’s apply some (unknown) transformation matrix ${\mathbf{T}}_{n\times n}$ by multiplying on both sides, i.e. $$\begin{array}{rl}& \mathbf{T}\mathbf{y}=\mathbf{T}\mathbf{X}\mathbf{b}+\mathbf{T}\mathbf{u}\\ & \Downarrow \Downarrow \Downarrow \\ & \tilde{\mathbf{y}}=\tilde{\mathbf{X}}\mathbf{b}+\tilde{\mathbf{u}}\end{array}$$ In this context, the conditional expectation of $\tilde{\mathbf{y}}$ reads $$\mathbb{E}\{\tilde{\mathbf{y}}\mid \tilde{\mathbf{X}}\}=\tilde{\mathbf{X}}\mathbf{b}\text{,}$$ while it’s conditional variance $$\mathbb{V}\{\tilde{\mathbf{y}}\mid \tilde{\mathbf{X}}\}=\mathbb{V}\{\tilde{\mathbf{u}}\mid \tilde{\mathbf{X}}\}=\mathbf{T}\mathbf{\Sigma }{\mathbf{T}}^{\mathrm{\top }}\text{.}$$ The next step is to identify a suitable transformation matrix $\mathbf{T}$ such that the conditional variance became equal to the identity matrix (Equation 31.3), i.e.  $$\mathbb{V}\{\tilde{\mathbf{u}}\mid \tilde{\mathbf{X}}\}={\mathbf{\text{I}}}_n\text{.}$$ In this way it is possible to work under the Gauss-Markov assumptions (Theorem 15.1) obtaining an estimator with minimum variance.

A possible way to identify $\mathbf{T}$ is to decompose the variance-covariance matrix (Equation 14.15) as follows $$\mathbf{\Sigma }=\mathbf{e}\mathbf{\Lambda }{\mathbf{e}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}=\mathbf{e}{\mathbf{\Lambda }}^{-1}{\mathbf{e}}^{\mathrm{\top }}$$ where $\mathbf{\Lambda }$ is the diagonal matrix containing the eigenvalues and $\mathbf{e}$ is the matrix with the eigenvectors that satisfy the following relation, i.e. ${\mathbf{e}}^{\mathrm{\top }}\mathbf{e}=\mathbf{e}{\mathbf{e}}^{\mathrm{\top }}={\mathbf{\text{I}}}_{n,n}$.

Thus, for the particular choice of $\mathbf{T}={\mathbf{\Lambda }}^{-1/2}{\mathbf{e}}^{\mathrm{\top }}$, one obtain a conditional variance equal to 1 for all the observations, i.e.
$$\begin{array}{rl}\mathbb{V}\{\tilde{\mathbf{u}}\mid \tilde{\mathbf{X}}\}& =\mathbf{T}\mathbf{\Sigma }{\mathbf{T}}^{\mathrm{\top }}=\\ & =({\mathbf{\Lambda }}^{-1/2}{\mathbf{e}}^{\mathrm{\top }})\mathbf{e}\mathbf{\Lambda }{\mathbf{e}}^{\mathrm{\top }}(\mathbf{e}{\mathbf{\Lambda }}^{-1/2})=\\ & ={\mathbf{\Lambda }}^{-1/2}\mathbf{\Lambda }{\mathbf{\Lambda }}^{-1/2}={\mathbf{\text{J}}}_n\end{array}$$ where ${\mathbf{J}}_{n,n}$ reads as in Equation 31.2. Finally, substituting $\tilde{\mathbf{X}}=\mathbf{T}\mathbf{X}$ in the OLS formula (Equation 15.3) and using the result Equation 16.4 one obtain exactly the GLS estimator in Equation 16.3, i.e.  $$\begin{array}{rl}{\mathbf{b}}^{\text{GLS}}& =({\tilde{\mathbf{X}}}^{\mathrm{\top }}\tilde{\mathbf{X}}{)}^{-1}{\tilde{\mathbf{X}}}^{\mathrm{\top }}\tilde{\mathbf{y}}=\\ & =({\mathbf{X}}^{\mathrm{\top }}{\mathbf{T}}^{\mathrm{\top }}\mathbf{T}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{T}}^{\mathrm{\top }}\mathbf{T}\mathbf{y}=\\ & =({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{y}\end{array}$$

16.3 Properties GLS

Theorem 16.1 ($\mathbf{\text{Aikten theorem}}$)
Under the following working hypothesis, also called Aikten hypothesis, i.e. 

1. $\mathbf{y}=\mathbf{X}\mathbf{b}+\mathbf{u}$.
2. $\mathbb{E}\{\mathbf{u}\}=0$.
3. $\mathbb{E}\{\mathbf{u}{\mathbf{u}}^{\mathrm{\top }}\}=\mathbf{\Sigma }$, i.e. heteroskedastic and correlated errors.
4. $\mathbf{X}$ is non-stochastic and independent from the errors $\mathbf{u}$ for all $n$’s.

The Generalized Least Square (GLS) estimator is $\mathbf{\text{BLUE}}$ (Best Linear Unbiased Estimator), where “best” stands for the estimator with minimum variance in the class of linear unbiased estimators of $\mathbf{b}$.

Proposition 16.3 ($\mathbf{\text{Properties GLS estimator}}$)
1. Unbiased: ${\mathbf{b}}^{\text{GLS}}$ is correct and it’s conditional expectation is equal to true parameter in population, i.e.  $$\begin{array}{}\text{(16.5)}& \mathbb{E}\{{\mathbf{b}}^{\text{GLS}}\mid \mathbf{X}\}=\mathbf{b}\text{.}\end{array}$$

2. Linear in the sense that it can be written as a linear combination of $\mathbf{y}$ and $\mathbf{X}$, i.e. ${\mathbf{b}}^{\text{GLS}}={\mathbf{A}}_{\text{x}}\mathbf{y}$, where ${\mathbf{A}}_{\text{x}}$ do not depend on $\mathbf{y}$, i.e. $$\begin{array}{}\text{(16.6)}& {\mathbf{b}}^{\text{GLS}}={\mathbf{A}}_{\text{x}}\mathbf{y}{\mathbf{A}}_{\text{x}}=({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\text{.}\end{array}$$
3. Under the Aikten hypothesis (Theorem 16.1) it has minimum variance in the class of the unbiased linear estimators and it reads: $$\begin{array}{}\text{(16.7)}& \mathbb{V}\{{\mathbf{b}}^{\text{GLS}}\mid \mathbf{X}\}=({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}\text{.}\end{array}$$

Proof: Proposition 16.3

Proof. The GLS estimator is correct. It’s expected value is computed from Equation 16.3 and substituting Equation 14.11, is equal to the true parameter in population, i.e.
$$\begin{array}{}\text{(16.8)}& \begin{array}{rl}\mathbb{E}\{{\mathbf{b}}^{\text{GLS}}\mid \mathbf{X}\}& =\mathbb{E}\{({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{y}\}=\\ & =\mathbb{E}\{({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}(\mathbf{X}\mathbf{b}+\mathbf{u})\}=\\ & =({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}\mathbf{b}+({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbb{E}\{\mathbf{u}\mid \mathbf{X}\}=\\ & =\mathbf{b}\end{array}\end{array}$$

Under the assumption of heteroskedastic and correlated observations the conditional variance of ${\mathbf{b}}^{\text{GLS}}$ follows similarly as for the OLS case (Equation 15.12) but with $\mathbb{V}\{\mathbf{u}\mid \mathbf{X}\}=\mathbf{\Sigma }$, i.e. $$\begin{array}{}\text{(16.9)}& \begin{array}{rl}\mathbb{V}\{{\mathbf{b}}^{\text{GLS}}\mid \mathbf{X}\}& =({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbb{V}\{\mathbf{u}\mid \mathbf{X}\}{\mathbf{\Sigma }}^{-1}\mathbf{X}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}=\\ & =({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{\Sigma }{\mathbf{\Sigma }}^{-1}\mathbf{X}({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}=\\ & =({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}=\\ & =({\mathbf{X}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{X}{)}^{-1}\end{array}\end{array}$$ where Equation 15.10 become a special case of Equation 16.9 where $\mathbf{\Sigma }={\sigma }_{\text{u}}^2{\mathbf{\text{I}}}_n$.

Gardini A., Costa M., Cavaliere G. 2007. Econometria, Volume Primo. FrancoAngeli. https://cris.unibo.it/handle/11585/119378.