17  Restricted least squares

References: Chapters 3.7, 3.8, 4.2 Gardini A. (2007).

Let’s consider a generic uni-variate linear model with $k$-regressors, namely $$\mathbf{y}=b_1{\mathbf{X}}_1+\cdots +b_j{\mathbf{X}}_j+\cdots +b_k{\mathbf{X}}_k+\mathbf{u}=\mathbf{b}\mathbf{X}+\mathbf{u}\text{,}$$ and suppose that we are interested in testing whereas the coefficient $b_j$ is statistically different from a certain value $r$ known at priori. In this case the null hypothesis can be equivalently represented using a more flexible matrix notation, i.e.  $$\begin{array}{}\text{(17.1)}& {\mathcal{H}}_0:b_j=r⟺{\mathcal{H}}_0:{\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}=\mathbf{0}\text{,}\end{array}$$ where $${\underset{k\times 1}{\mathbf{R}}}^{\mathrm{\top }}=\underset{j\text{-th position}}{\left(\begin{array}{c}0\dots 1\dots 0\end{array}\right)}\text{.}$$ Hence, the linear restriction in Equation 17.1 can be written in matrix as $${\mathcal{H}}_0:{\underset{k\times 1}{\mathbf{R}}}^{\mathrm{\top }}\underset{k\times 1}{\mathbf{b}}-\underset{1\times 1}{\mathbf{r}}=\underset{1\times 1}{\mathbf{0}}⟺\underset{j\text{-th position}}{\left(\begin{array}{c}0\dots 1\dots 0\end{array}\right)}\left(\begin{array}{c}{b}_1\\ ⋮\\ b_j\\ ⋮\\ b_k\end{array}\right)-\left(\begin{array}{c}r\end{array}\right)=\left(\begin{array}{c}0\end{array}\right)\text{.}$$

17.1 Multiple restrictions

Let’s consider multiple restrictions, i.e.  $$\begin{array}{rlrl}{\mathcal{H}}_0:& (1)b_1-b_2=0& & b_1\text{ and }{b}_2\text{ has same effect}\\ & (2)b_3+b_4=1& & b_3\text{ plus }{b}_4\text{ unitary root}\end{array}$$ Let’s construct the vector for (1) (first column of $R$) and (2) (second column of $R$), i.e. $${\underset{2\times 4}{\mathbf{R}}}^{\mathrm{\top }}\underset{4\times 1}{\mathbf{b}}-\underset{2\times 1}{\mathbf{r}}=\underset{2\times 1}{\mathbf{0}}⟺\underset{\mathbf{R}}{\underbrace{\left(\begin{array}{cccc}1& -1& 0& 0\\ 0& 0& 1& 1\end{array}\right)}}\underset{\mathbf{b}}{\underbrace{\left(\begin{array}{c}{b}_1\\ b_2\\ b_3\\ b_4\end{array}\right)}}-\underset{\mathbf{r}}{\underbrace{\left(\begin{array}{c}0\\ 1\end{array}\right)}}=\left(\begin{array}{c}0\\ 0\end{array}\right)\text{.}$$

17.2 Restricted least squares

Proposition 17.1 ($\mathbf{\text{Restricted Least Squares (RLS) estimator}}$)
Let’s consider a linear model under the OLS assumptions and let’s consider a set of $m$ linear hypothesis on the parameters of the model taking the form $${\mathcal{H}}_0:{\underset{m\times k}{\mathbf{R}}}^{\mathrm{\top }}\underset{k\times 1}{\mathbf{b}}-\underset{m\times 1}{\mathbf{r}}=\underset{m\times 1}{\mathbf{0}}\text{.}$$ Therefore, the optimization problem became restricted to the space of parameters that satisfies the conditions. More precisely, the space ${\tilde{\mathrm{\Theta }}}_{\mathbf{b}}$, that is a subset of the parameter space ${\tilde{\mathrm{\Theta }}}_{\mathbf{b}}\subset {\mathrm{\Theta }}_{\mathbf{b}}$ where the linear constraint holds true, is defined as $${\tilde{\mathrm{\Theta }}}_{\mathbf{b}}=\{\mathbf{b}\in {\mathbb{R}}^k:{\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}=\mathbf{0}\}\text{.}$$ Hence, the optimization problem in Equation 15.2 is restricted to only the parameters that satisfy the constraint.

Formally, the RLS estimator is the solution of the following minimization problem, i.e.  $$\begin{array}{}\text{(17.2)}& {\mathbf{b}}^{\text{RLS}}=\underset{\mathbf{b}\in {\tilde{\mathrm{\Theta }}}_{\mathbf{b}}}{\text{argmin}}\{{\text{Q}}^{\text{OLS}}(\mathbf{b})\}\text{.}\end{array}$$ where ${\text{Q}}^{\text{OLS}}$ reads as in the OLS case (Equation 15.1). Notably, the analytic solution for ${\mathbf{b}}^{\text{RLS}}$ reads $$\begin{array}{}\text{(17.3)}& {\mathbf{b}}^{\text{RLS}}={\mathbf{b}}^{\text{OLS}}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^T({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r})\text{.}\end{array}$$

Proof: Proposition 17.1

Proof. In order to solve the minimization problem in Equation 17.2, let’s construct the Lagrangian (Equation 31.11) as $${\text{Q}}^{\text{RLS}}(\mathbf{b},\mathit{\lambda })={\text{Q}}^{\text{OLS}}(\mathbf{b})-2{\mathit{\lambda }}^{\mathrm{\top }}({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r})\text{.}$$ Then, one obtain the following system of equation, i.e.  $$\{\begin{array}{ll}{\partial }_{\mathbf{b}}{\text{Q}}^{\text{RLS}}(\mathbf{b},\mathit{\lambda })=-2{\mathbf{X}}^{\mathrm{\top }}\mathbf{y}+2{\mathbf{X}}^{\mathrm{\top }}\mathbf{X}\mathbf{b}-2\mathbf{R}\mathit{\lambda }=\mathbf{0}& (A)\\ {\partial }_{\mathit{\lambda }}{\text{Q}}^{\text{RLS}}(\mathbf{b},\mathit{\lambda })=-2({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r})=\mathbf{0}& (B)\end{array}$$ Let’s firstly solve explicitly $\mathbf{b}={\mathbf{b}}^{\text{RLS}}$ from (A), i.e.  $$\begin{array}{}\text{(17.4)}& \begin{array}{rl}{\mathbf{b}}^{\text{RLS}}& =({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}\mathbf{y}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}\mathit{\lambda }=\\ & ={\mathbf{b}}^{\text{OLS}}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}\mathit{\lambda }\end{array}\end{array}$$ and substitute the result in (B), i.e.  $$\begin{array}{rl}& {\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{RLS}}-\mathbf{r}=\mathbf{0}\\ & ⟹{\mathbf{R}}^{\mathrm{\top }}[{\mathbf{b}}^{\text{OLS}}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}\mathit{\lambda }]-\mathbf{r}=\mathbf{0}\\ & ⟹{\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}\mathit{\lambda }-\mathbf{r}=\mathbf{0}\\ & ⟹{\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r}=[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]\mathit{\lambda }\end{array}$$ Hence, it is possible to explicit the Lagrange multipliers $\mathit{\lambda }$ as: $$\begin{array}{}\text{(17.5)}& \mathit{\lambda }={[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r})\text{.}\end{array}$$ Finally, substituting $\mathit{\lambda }$ (Equation 17.5) in Equation 17.4 gives the optimal solution, i.e.  $${\mathbf{b}}^{\text{RLS}}={\mathbf{b}}^{\text{OLS}}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r})\text{.}$$ Note that if constraints hold true in the OLS estimate, ${\mathcal{H}}_0$ is true and therefore ${\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r}=\mathbf{0}$. Hence the RLS and OLS parameters are the same, i.e. ${\mathbf{b}}^{\text{RLS}}={\mathbf{b}}^{\text{OLS}}$.

Proposition 17.2 ($\mathbf{\text{Expectation RLS estimator}}$)
The RLS estimator (Equation 17.3) is correct for the true parameter in population $\mathbf{b}$ if and only if the restrictions imposed by ${\mathcal{H}}_0$ are true in population, i.e. expected value is computed as: $$\begin{array}{}\text{(17.6)}& \mathbb{E}\{{\mathbf{b}}^{\text{RLS}}\mid \mathbf{X}\}=\mathbf{b}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r})\text{,}\end{array}$$ where $\mathbb{E}\{{\mathbf{b}}^{\text{RLS}}\mid \mathbf{X}\}=\mathbf{b}$ only if the second component is zero, that happens only when ${\mathcal{H}}_0$ holds true and so ${\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}=\mathbf{0}$.

Proof: Proposition 17.2

Proof. Let’s apply the expected value on Equation 17.3 remembering that $\mathbf{X}$, $\mathbf{R}$ and $\mathbf{r}$ are non-stochastic and that ${\mathbf{b}}^{\text{OLS}}$ is correct (Equation 15.8). Developing the computations gives: $$\begin{array}{rl}\mathbb{E}\{{\mathbf{b}}^{\text{RLS}}\mid \mathbf{X}\}& =\mathbb{E}\{{\mathbf{b}}^{\text{OLS}}\mid \mathbf{X}\}-\mathbb{E}\{({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r})\mid \mathbf{X}\}=\\ & =\mathbf{b}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}\mathbb{E}\{{\mathbf{b}}^{\text{OLS}}\mid \mathbf{X}\}-\mathbf{r})=\\ & =\mathbf{b}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r})\end{array}$$ Hence ${\mathbf{b}}^{\text{RLS}}$ is correct if and only if the restriction holds true in population, i.e. $$\mathbb{E}\{{\mathbf{b}}^{\text{RLS}}\mid \mathbf{X}\}=\mathbf{b}⟺{\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}=\mathbf{0}\text{.}$$

Proposition 17.3 ($\mathbf{\text{Variance RLS estimator}}$)
The variance of the RLS estimator (Equation 17.3) $$\mathbb{V}\{{\mathbf{b}}^{\text{RLS}}\}=\mathbb{V}\{{\mathbf{b}}^{\text{OLS}}\}-{\sigma }_{\text{u}}^2\cdot ({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\text{.}$$ It is interesting to note that the variance of the RLS estimator is always lower or equal than the variance of the OLS estimator, in fact $$\mathbb{V}\{{\mathbf{b}}^{\text{RLS}}\}\le \mathbb{V}\{{\mathbf{b}}^{\text{OLS}}\}\text{.}$$

Proof: Proposition 17.3

Proof. In order to compute the variance of the RLS estimator, let’s apply the variance operator to @Equation 17.3, i.e.  $$\mathbb{V}\{{\mathbf{b}}^{\text{RLS}}\}=\mathbb{V}\{{\mathbf{b}}^{\text{OLS}}\}-\mathbb{V}\{({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}{\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}\}\text{.}$$ Let’s denote with ${\mathbf{R}}_{\mathbf{x}}$ the matrix $${\mathbf{R}}_{\mathbf{x}}=({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}{\mathbf{R}}^{\mathrm{\top }}\text{,}$$ and let’s bring it outside the variance, i.e.  $$\mathbb{V}\{{\mathbf{b}}^{\text{RLS}}\}=\mathbb{V}\{{\mathbf{b}}^{\text{OLS}}\}-{\mathbf{R}}_{\mathbf{x}}\mathbb{V}\{{\mathbf{b}}^{\text{OLS}}\}{\mathbf{R}}_{\mathbf{x}}^{\mathrm{\top }}\text{.}$$ Moreover, substituting the expression of the variance of ${\mathbf{b}}^{\text{OLS}}$ (Equation 15.10) one obtain $$\mathbb{V}\{{\mathbf{b}}^{\text{RLS}}\}=\mathbb{V}\{{\mathbf{b}}^{\text{OLS}}\}-{\sigma }_{\text{u}}^2\cdot {\mathbf{R}}_{\mathbf{x}}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}{\mathbf{R}}_{\mathbf{x}}^{\mathrm{\top }}\text{.}$$ Developing the matrix multiplication gives $$\begin{array}{rl}{\mathbf{R}}_{\mathbf{x}}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}{\mathbf{R}}_{\mathbf{x}}^{\mathrm{\top }}& =({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}=\\ & =({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\end{array}$$

17.3 A test for linear restrictions

Under the assumption of normality of the error terms, it is possible to derive a statistic to test the significance of the linear restrictions imposed by ${\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}=\mathbf{0}$. Let’s test the validity of the hull hypothesis ${\mathcal{H}}_0$ against its alternative hypothesis ${\mathcal{H}}_1$, i.e.  $${\mathcal{H}}_0:{\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}=\mathbf{0}\text{,}{\mathcal{H}}_1:{\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}\ne \mathbf{0}\text{.}$$ Under normality, the OLS estimate are multivariate normal, thus applying the scaling property one obtain that the distribution under ${\mathcal{H}}_0$ is normal, i.e.  $$\begin{array}{}\text{(17.7)}& {\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{RLS}}-\mathbf{r}\sim \mathcal{N}({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r},{\sigma }_{\text{u}}^2\cdot {\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R})\text{.}\end{array}$$ Thus, we can write the statistic $$\begin{array}{}\text{(17.8)}& {\text{W}}_m=\frac{1}{{\sigma }_{\text{u}}^2}\cdot ({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}{)}^{\mathrm{\top }}({\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{)}^{-1}({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r})\text{.}\end{array}$$

If we work under ${\mathcal{H}}_0$, then the mean in Equation 17.7 is zero, i.e.
$${\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{RLS}}-\mathbf{r}\underset{{\mathcal{H}}_0}{\sim }\mathcal{N}(0,{\sigma }_{\text{u}}^2\cdot {\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R})\text{.}$$ Recalling the relation (Section 32.1.1) between the distribution of the quadratic form of a multivariate normal and the ${\chi }^2$ distribution, then the test statistic $$\begin{array}{}\text{(17.9)}& {\text{W}}_m\stackrel{\text{d}}{\underset{{\mathcal{H}}_0}{\sim }}{\chi }^2(m)\text{,}\end{array}$$ has ${\chi }^2(m)$ distribution, with $m$ the number of restrictions.

Instead, under ${\mathcal{H}}_1$ the distribution of the linear restriction is exactly equal to Equation 17.7. Thus, applying property 4. in Section 32.1.1 one obtain that the test statistic is distributed as a non central ${\chi }^2(m,\delta )$, i.e. $$\begin{array}{}\text{(17.10)}& {\text{W}}_m\stackrel{\text{d}}{\underset{{\mathcal{H}}_1}{\sim }}{\chi }^2(m,\delta )\text{,}\end{array}$$ where the non centrality parameter $\delta$ reads $$\delta =\frac{1}{{\sigma }_{\text{u}}^2}\cdot ({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}{)}^{\mathrm{\top }}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r})>0\text{.}$$ As general decision rule ${\mathcal{H}}_0$ is rejected if the statistic in Equation 17.9 is greater than the quantile with confidence level $\alpha$ of a ${\chi }^2(m)$ random variable. Such critic value, denoted with $q_{\alpha }$ represents the value for which the probability that a ${\chi }^2(m)$ is greater than $q_{\alpha }$ is exactly equal to $\alpha$, i.e.  $$\mathbb{P}({\text{W}}_m>q_{\alpha })=\alpha \text{.}$$

Gardini A., Costa M., Cavaliere G. 2007. Econometria, Volume Primo. FrancoAngeli. https://cris.unibo.it/handle/11585/119378.