17  Restricted least squares

References: Chapter 4. Gardini A. (2000).

17.1 A general framework for linear restrictions

Let’s consider a generic uni-variate linear model with $k$-regressors, namely $$\mathbf{y}=b_1{\mathbf{X}}_1+\cdots +b_j{\mathbf{X}}_j+\cdots +b_k{\mathbf{X}}_k+\mathbf{e}=\mathbf{b}\mathbf{X}+\mathbf{e}\text{,}$$ and suppose that we are interest in testing if the $b_j$ coefficient is statistically different from a certain known value $r$. In this case the null hypothesis, that is ${\mathcal{H}}_0:b_j=r$, can be equivalently represented using a more flexible matrix notation, i.e.  $${\mathcal{H}}_0:b_j=r⟺{\mathcal{H}}_0:{\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}=\mathbf{0}\text{,}$$ where $${\underset{k\times 1}{\mathbf{R}}}^{\mathrm{\top }}=\underset{j\text{-th position}}{\left(\begin{array}{c}0\dots 1\dots 0\end{array}\right)}\text{.}$$ Hence, the linear restriction in matrix form reads explicitly as $${\mathcal{H}}_0:{\underset{k\times 1}{\mathbf{R}}}^{\mathrm{\top }}\underset{k\times 1}{\mathbf{b}}-\underset{1\times 1}{\mathbf{r}}=\underset{1\times 1}{\mathbf{0}}⟺\underset{j\text{-th position}}{\left(\begin{array}{c}0\dots 1\dots 0\end{array}\right)}\left(\begin{array}{c}{b}_1\\ ⋮\\ b_j\\ ⋮\\ b_k\end{array}\right)-\left(\begin{array}{c}r\end{array}\right)=\left(\begin{array}{c}0\end{array}\right)\text{.}$$

17.2 Multiple restrictions

Let’s consider a linear model of the form $$\mathbf{y}=b_1{\mathbf{X}}_1+b_2{\mathbf{X}}_2+b_3{\mathbf{X}}_3+b_4{\mathbf{X}}_4\text{,}$$ and suppose that the aim is to test at the same time the following null hypothesis, i.e.  $$\begin{array}{rlrl}{\mathcal{H}}_0:& (1)b_1-b_2=0& & b_1\text{ and }{b}_2\text{ has same effect}\\ & (2)b_3+b_4=1& & b_3\text{ plus }{b}_4\text{ unitary root}\end{array}$$ Let’s construct the vector for (1) (first column of $R$) and (2) (first column of $R$), i.e. $${\underset{2\times 4}{\mathbf{R}}}^{\mathrm{\top }}\underset{4\times 1}{\mathbf{b}}-\underset{2\times 1}{\mathbf{r}}=\underset{2\times 1}{\mathbf{0}}⟺\left(\begin{array}{cccc}1& -1& 0& 0\\ 0& 0& 1& 1\end{array}\right)\left(\begin{array}{c}{b}_1\\ b_2\\ b_3\\ b_4\end{array}\right)-\left(\begin{array}{c}0\\ 1\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right)\text{.}$$

17.3 Restricted least squares

Proposition 17.1 Let’s consider a linear model under the OLS assumptions and let’s consider a set of $m$ linear hypothesis on the parameters of the model taking the form $${\mathcal{H}}_0:{\underset{m\times k}{\mathbf{R}}}^{\mathrm{\top }}\underset{k\times 1}{\mathbf{b}}-\underset{m\times 1}{\mathbf{r}}=\underset{m\times 1}{\mathbf{0}}\text{.}$$ Therefore, the optimization problem became restricted to the space of parameters that satisfies the conditions. More precisely, such parameters will be in a subset of the parameter space, i.e. ${\tilde{\mathrm{\Theta }}}_{\mathbf{b}}\subset {\mathrm{\Theta }}_{\mathbf{b}}$, where the linear constraint holds true. Formally, the space ${\tilde{\mathrm{\Theta }}}_{\mathbf{b}}$ is defined as $${\tilde{\mathrm{\Theta }}}_{\mathbf{b}}=\{\mathbf{b}\in {\mathbb{R}}^k:{\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}=\mathbf{0}\}\text{.}$$ Hence, the optimization problem in Equation 15.2 is restricted to only the parameters that satisfy the constraint. Formally, the RLS estimator is the solution of the following minimization problem, i.e.  $$\begin{array}{}\text{(17.1)}& {\mathbf{b}}^{\text{RLS}}=\underset{\mathbf{b}\in {\tilde{\mathrm{\Theta }}}_{\mathbf{b}}}{\text{argmin}}\{{\text{Q}}^{\text{OLS}}(\mathbf{b})\}\text{.}\end{array}$$ where ${\text{Q}}^{\text{OLS}}$ reads as in the OLS case (Equation 15.1). Notably, the analytic solution for ${\mathbf{b}}^{\text{RLS}}$ reads $$\begin{array}{}\text{(17.2)}& {\mathbf{b}}^{\text{RLS}}={\mathbf{b}}^{\text{OLS}}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^T({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r})\text{.}\end{array}$$

Proof: Proposition 17.1

Proof. In order to solve the minimization problem in Equation 17.1, let’s construct the Lagrangian $L(x,\lambda )$, i.e. $$L(x,\mathit{\lambda })=f(x)-{\mathit{\lambda }}^{\mathrm{\top }}g(x)\text{,}$$ where $\mathit{\lambda }$ is the vector of the Lagrange multipliers. Minimizing $L(x,\mathit{\lambda })$ is equivalent to find the value of $x$ that minimize $f(x)$ under the constraint $g(x)=0$. In fact, it is possible to prove that the minimum is found as: $$\begin{array}{}\text{(17.3)}& \underset{x\in \chi }{\text{argmin}}L(x,\mathit{\lambda })⟹\{\begin{array}{l}(A){\partial }_{x}L(x,\mathit{\lambda })=0\\ (B){\partial }_{\mathit{\lambda }}L(x,\mathit{\lambda })=0\end{array}\end{array}$$ In the case of RLS estimate the Lagrangian reads: $$L(\mathbf{b},\mathit{\lambda })={\text{Q}}^{\text{OLS}}(\mathbf{b})-2{\mathit{\lambda }}^{\mathrm{\top }}({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r})\text{.}$$ Then, from Equation 17.3 one obtain the following system of equation, i.e.  $$\{\begin{array}{l}(A){\partial }_{\mathbf{b}}L(\mathbf{b},\mathit{\lambda })=-2{\mathbf{X}}^{\mathrm{\top }}\mathbf{y}+2{\mathbf{X}}^{\mathrm{\top }}\mathbf{X}\mathbf{b}-2\mathbf{R}\mathit{\lambda }=\mathbf{0}\\ (B){\partial }_{\mathit{\lambda }}L(\mathbf{b},\mathit{\lambda })=-2({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r})=\mathbf{0}\end{array}$$ Let’s explicit $\mathbf{b}={\mathbf{b}}^{\text{RLS}}$ from (A), i.e.  $$\begin{array}{}\text{(17.4)}& \begin{array}{rl}{\mathbf{b}}^{\text{RLS}}& =({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}\mathbf{y}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}\mathit{\lambda }=\\ & ={\mathbf{b}}^{\text{OLS}}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}\mathit{\lambda }\end{array}\end{array}$$ Let’s now substitute Equation 17.4 in (B), i.e.  $$\begin{array}{rl}& {\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{RLS}}-\mathbf{r}=\mathbf{0}\\ & ⟹{\mathbf{R}}^{\mathrm{\top }}[{\mathbf{b}}^{\text{OLS}}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}\mathit{\lambda }]-\mathbf{r}=\mathbf{0}\\ & ⟹{\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}\mathit{\lambda }-\mathbf{r}=\mathbf{0}\\ & ⟹{\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r}=[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]\mathit{\lambda }\end{array}$$ Hence, it is possible to explicit the Lagrange multipliers $\mathit{\lambda }$ as: $$\begin{array}{}\text{(17.5)}& \mathit{\lambda }={[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r})\text{.}\end{array}$$ Finally, substituting Equation 17.5 in Equation 17.4 gives the optimal solution, i.e.  $$\begin{array}{rl}{\mathbf{b}}^{\text{RLS}}& ={\mathbf{b}}^{\text{OLS}}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}\mathit{\lambda }=\\ & ={\mathbf{b}}^{\text{OLS}}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r})\end{array}$$ Note that if constraints hold true in the OLS estimate, ${\mathcal{H}}_0$ is true and therefore ${\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r}=\mathbf{0}$. Hence the RLS and OLS parameters are the same, i.e. ${\mathbf{b}}^{\text{RLS}}={\mathbf{b}}^{\text{OLS}}$.

17.4 Properties RLS

1. The RLS estimator is correct if and only if the restriction imposed by ${\mathcal{H}}_0$ is true in population. In fact, it’s expected value is computed as: $$\begin{array}{}\text{(17.6)}& \mathbb{E}\{{\mathbf{b}}^{\text{RLS}}\mid \mathbf{X}\}=\mathbf{b}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r})\text{,}\end{array}$$ and it is correct if and only if the second component is zero, i.e. if ${\mathcal{H}}_0$ holds true.

Correctness of RLS estimator

Proof. Let’s apply the expected value on Equation 17.2 remembering that $\mathbf{X}$, $\mathbf{R}$ and $\mathbf{r}$ are non-stochastic and that ${\mathbf{b}}^{\text{OLS}}$ is correct (Equation 15.8). Developing the computations gives: $$\begin{array}{rl}\mathbb{E}\{{\mathbf{b}}^{\text{RLS}}\mid \mathbf{X}\}& =\mathbb{E}\{{\mathbf{b}}^{\text{OLS}}\mid \mathbf{X}\}-\mathbb{E}\{({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r})\mid \mathbf{X}\}=\\ & =\mathbf{b}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}\mathbb{E}\{{\mathbf{b}}^{\text{OLS}}\mid \mathbf{X}\}-\mathbf{r})=\\ & =\mathbf{b}-({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{[{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}]}^{-1}({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r})\end{array}$$ Hence ${\mathbf{b}}^{\text{RLS}}$ is correct if and only if the restriction holds in population. $$\mathbb{E}\{{\mathbf{b}}^{\text{RLS}}\mid \mathbf{X}\}=\mathbf{b}⟺{\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}=\mathbf{0}\text{.}$$

17.5 A test for linear restrictions

Under the assumption of normality of the error terms, it is possible to derive a statistic to test the significance of the linear restrictions imposed by ${\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}=\mathbf{0}$. Let’s test the validity of the hull hypothesis ${\mathcal{H}}_0$ against the alternative hypothesis ${\mathcal{H}}_1$, i.e.  $${\mathcal{H}}_0:{\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r}=\mathbf{0}\text{,}{\mathcal{H}}_1:{\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r}\ne \mathbf{0}\text{.}$$ Under normality, the OLS estimate are multivariate normal, thus applying the scaling property one obtain that the distribution under ${\mathcal{H}}_0$ is normal, i.e.  $${\mathbf{R}}^{\mathrm{\top }}{\mathbf{b}}^{\text{OLS}}-\mathbf{r}\sim \mathcal{N}({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r},{\sigma }_e^2{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R})\text{.}$$ Applying the relation between the distribution of the quadratic form of a multivariate normal and the ${\chi }^2$ distribution from property 3 in Section 35.1.2, one obtain the statistic: $$\begin{array}{}\text{(17.7)}& {\text{T}}_m=({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}{)}^{\mathrm{\top }}({\sigma }_e^2{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{)}^{-1}({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r})\text{.}\end{array}$$ Under ${\mathcal{H}}_0$, ${\text{T}}_m$ is distributed as a ${\chi }^2(m)$, where $m$ is the number of linear restrictions, i.e. $$\begin{array}{}\text{(17.8)}& {\text{T}}_m\stackrel{{\mathcal{H}}_0}{\sim }{\chi }^2(m)\text{.}\end{array}$$

As general decision rule ${\mathcal{H}}_0$ is rejected if the statistic in Equation 17.8 is greater than the quantile with confidence level $\alpha$ of a ${\chi }^2(m)$ random variable. Such critic value, denoted with ${\chi }_{\alpha }^2(m)$ represents the value for which the probability that a ${\chi }^2(m)$ is greater than the value ${\chi }_{\alpha }^2(m)$ is exactly $\alpha$, i.e.  $$\mathbb{P}({\chi }^2(m)>x_{\alpha })=\alpha \text{.}$$ In this case the probability to have an error of type I, i.e. rejecting ${\mathcal{H}}_0$ when ${\mathcal{H}}_0$ is true is exactly $\alpha$.

Instead, by applying property 4. in Section 35.1.2, under ${\mathcal{H}}_1$ the statistic ${\text{T}}_m$ is distributed as a non central ${\chi }^2(m,\delta )$, i.e. $$\begin{array}{}\text{(17.9)}& {\text{T}}_m\stackrel{H_1}{\sim }{\chi }^2(m,\delta )\text{,}\end{array}$$ where the non centrality parameter $\delta$ is computed as: $$\delta =({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r}{)}^{\mathrm{\top }}({\sigma }_e^2{\mathbf{R}}^{\mathrm{\top }}({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}{)}^{-1}\mathbf{R}{)}^{-1}({\mathbf{R}}^{\mathrm{\top }}\mathbf{b}-\mathbf{r})\text{.}$$

Gardini A., Costa M., Cavaliere G. 2000. Econometria, Volume Primo. FrancoAngeli.