32  Linear Algebra

32.1 Vector multiplication

Consider a vector of this form,

$$\underset{n\times 1}{\mathbf{y}}=\left(\begin{array}{c}{y}_1\\ ⋮\\ y_i\\ ⋮\\ y_n\end{array}\right)\text{,}$$ then $$\underset{n\times 1}{\mathbf{y}}{\underset{n\times 1}{\mathbf{y}}}^{\mathrm{\top }}=\underset{n\times 1}{\mathbf{y}}\underset{1\times n}{\mathbf{y}}=\underset{n\times n}{\mathbf{y}}=\left(\begin{array}{ccccc}{y}_1^2& \dots & y_1{y}_i& \dots & y_1{y}_n\\ ⋮& \ddots & ⋮& & ⋮\\ y_i{y}_1& \dots & y_i^2& \dots & y_i{y}_n\\ ⋮& & ⋮& \ddots & ⋮\\ y_n{y}_1& \dots & y_n{y}_i& \dots & y_n^2\end{array}\right)\text{,}$$ and $${\underset{n\times 1}{\mathbf{y}}}^{\mathrm{\top }}\underset{n\times 1}{\mathbf{x}}=\underset{1\times n}{\mathbf{y}}\underset{n\times 1}{\mathbf{y}}=\sum _{i=1}^n{y}_i^2=\underset{1\times 1}{y}\text{.}$$

32.2 Matrix multiplication

Consider a matrix of this form,

$$\underset{n\times k}{\mathbf{X}}=\left(\begin{array}{ccccc}{x}_{11}& \dots & x_{1j}& \dots & x_{1k}\\ ⋮& \ddots & ⋮& & ⋮\\ x_{11}& \dots & x_{ij}& \dots & x_{ik}\\ ⋮& & ⋮& \ddots & ⋮\\ x_{n1}& \dots & x_{nj}& \dots & x_k\end{array}\right)\text{,}$$ then $$\underset{k\times k}{\mathbf{M}}={\underset{n\times k}{\mathbf{X}}}^{\mathrm{\top }}\underset{n\times k}{\mathbf{X}}=\left(\begin{array}{ccccc}\sum _{i=1}^n{x}_{i1}^2& \dots & \sum _{i=1}^n{x}_{i1}{x}_{ij}& \dots & \sum _{i=1}^n{x}_{i1}{x}_{ik}\\ ⋮& \ddots & ⋮& & ⋮\\ \sum _{i=1}^n{x}_{ij}{x}_{i1}& \dots & \sum _{i=1}^n{x}_{ij}^2& \dots & \sum _{i=1}^n{x}_{ij}{x}_{ik}\\ ⋮& & ⋮& \ddots & ⋮\\ \sum _{i=1}^n{x}_{ik}{x}_{i1}& \dots & \sum _{i=1}^n{x}_{ik}{x}_{ij}& \dots & \sum _{i=1}^n{x}_{ik}^2\end{array}\right)\text{,}$$ and $$\underset{k\times 1}{\mathbf{m}}={\underset{n\times k}{\mathbf{X}}}^{\mathrm{\top }}\underset{n\times 1}{\mathbf{y}}=\left(\begin{array}{c}\sum _{i=1}^n{x}_{i1}{y}_i\\ ⋮\\ \sum _{i=1}^n{x}_{ij}{y}_i\\ ⋮\\ \sum _{i=1}^n{x}_{ik}{y}_i\end{array}\right)\text{.}$$

32.3 Special matrices

32.3.1 Basis vector

$$\begin{array}{}\text{(32.1)}& {\mathbf{e}}_p^{\mathrm{\top }}=\left(\begin{array}{cccc}1& 0& \dots & 0\end{array}\right)\text{.}\end{array}$$ The standard basis vector of length p with 1 in the first position and 0 elsewhere.

32.3.2 Matrix of ones

In mathematics, a matrix of ones (also called an all-ones matrix) is a matrix where every entry equals 1, i.e. $$\begin{array}{}\text{(32.2)}& \begin{array}{rl}{\mathbf{J}}_2& =\left(\begin{array}{cc}1& 1\\ 1& 1\end{array}\right){\mathbf{J}}_3=\left(\begin{array}{ccc}1& 1& 1\\ 1& 1& 1\\ 1& 1& 1\end{array}\right)\\ {\mathbf{J}}_{3,2}& =\left(\begin{array}{cc}1& 1\\ 1& 1\\ 1& 1\end{array}\right){\mathbf{J}}_{2,3}=\left(\begin{array}{ccc}1& 1& 1\\ 1& 1& 1\end{array}\right)\end{array}\end{array}$$ In general, When two indices are provided, e.g., ${\mathbf{J}}_{3,2}$, the first indicates the number of rows and the second the number of columns. When only one index is given, e.g., ${\mathbf{J}}_3$, it denotes a square matrix of size $3\times 3$. Some basic matrix operations include:

• ${\mathbf{J}}_{n,1}\cdot {\mathbf{J}}_{1,n}={\mathbf{J}}_n$
• ${\mathbf{J}}_{1,n}\cdot {\mathbf{J}}_{n,1}={\mathbf{J}}_1=1$

J_n <- function(i, j){
  matrix(1, nrow = i, ncol = j)
}

32.3.3 Identity matrix

In linear algebra, the identity matrix of size $n$ is the $n\times n$ square matrix with ones on the main diagonal and zeros elsewhere, i.e.  $$\begin{array}{}\text{(32.3)}& {\mathbf{I}}_1=\left(\begin{array}{c}1\end{array}\right),{\mathbf{I}}_2=\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right),{\mathbf{I}}_3=\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right),{\mathbf{I}}_3=\left(\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right)\text{.}\end{array}$$

I_n <- function(i, j){
  diag(1, nrow = i, ncol = j) 
}

32.4 Determinant

The determinant is a scalar value that can be computed from a square matrix. It provides important information about the matrix, such as whether it is invertible, its volume-scaling factor in linear transformations, and the linear dependence of its rows or columns. For a matrix $\mathbf{A}\in {\mathbb{R}}^{n\times n}$, the determinant is denoted $det(\mathbf{A})$. Consider two matrices ${\mathbf{A}}_{n\times n}$ and ${\mathbf{B}}_{n\times n}$, then the determinant satisfies some properties.

1. Scalar: $adet(\mathbf{A})=a^{n}det(\mathbf{A})$ for $a\in \mathbb{R}$.
2. Transpose: $det({\mathbf{A}}^{\mathrm{\top }})=det(\mathbf{A})$.
3. Multiplication: $det(\mathbf{A}\mathbf{B})=det(\mathbf{A})det(\mathbf{B})$.
4. Inverse: $det({\mathbf{A}}^{-1})=\frac{1}{det(\mathbf{A})}$.
5. Rank: if
   • $det(\mathbf{A})\ne 0$ then $rank(\mathbf{A})=max=n$.
   • $det(\mathbf{A})=0$ then $rank(\mathbf{A})<max=n$.

The determinant of an $n\times n$ matrix can be computed by expanding along any row or column. Expanding along the i-th row gives: $$\begin{array}{}\text{(32.4)}& det(\mathbf{A})=\sum _{j=1}^n(-1{)}^{i+j}{a}_{ij}det({\mathbf{M}}_{ij})\text{.}\end{array}$$ where ${\mathbf{M}}_{ij}$ is the sub-matrix without the $i$-th row and the $j$-th column. For example considering a $2\times 2$ matrix, the determinant simplifies to a well-known formula: $$\begin{array}{}\text{(32.5)}& \mathbf{A}=\left(\begin{array}{cc}a& b\\ c& d\end{array}\right)⟹det(\mathbf{A})=ad-bc\text{.}\end{array}$$

Determinant of a $3\times 3$ matrix with Laplace recursion

Let’s consider a generic $3\times 3$ matrix, i.e. $$\mathbf{A}=\left(\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\\ a_{31}& a_{32}& a_{33}\end{array}\right)\text{,}$$ then, let’s fix the row $i=1$ and develop the Laplace expansion (Equation 32.4), i.e. $$\begin{array}{}\text{(32.6)}& \begin{array}{rl}det(\mathbf{A})& =(-1{)}^{1+1}{a}_{11}{\mathbf{M}}_{11}+(-1{)}^{1+2}{a}_{12}{\mathbf{M}}_{12}+(-1{)}^{1+3}{a}_{13}{\mathbf{M}}_{13}=\\ & =a_{11}{\mathbf{M}}_{11}-a_{12}{\mathbf{M}}_{12}+a_{13}{\mathbf{M}}_{13}\end{array}\end{array}$$ where the sub-matrices ${\mathbf{M}}_{11}$, ${\mathbf{M}}_{12}$ and ${\mathbf{M}}_{13}$ read explicitly as: $${\mathbf{M}}_{11}=\left(\begin{array}{cc}{a}_{22}& a_{23}\\ a_{32}& a_{33}\end{array}\right),{\mathbf{M}}_{12}=\left(\begin{array}{cc}{a}_{21}& a_{23}\\ a_{31}& a_{33}\end{array}\right),{\mathbf{M}}_{13}=\left(\begin{array}{cc}{a}_{21}& a_{22}\\ a_{31}& a_{32}\end{array}\right)\text{.}$$ Then, the determinant of $2\times 2$ matrices is easily computable as: $$\begin{array}{}\text{(32.7)}& \begin{array}{rl}& det({\mathbf{M}}_{11})=det\left(\begin{array}{cc}{a}_{22}& a_{23}\\ a_{32}& a_{33}\end{array}\right)=a_{22}{a}_{33}-a_{23}{a}_{32}\\ & det({\mathbf{M}}_{12})=det\left(\begin{array}{cc}{a}_{21}& a_{23}\\ a_{31}& a_{33}\end{array}\right)=a_{21}{a}_{33}-a_{23}{a}_{31}\\ & det({\mathbf{M}}_{13})=det\left(\begin{array}{cc}{a}_{21}& a_{22}\\ a_{31}& a_{32}\end{array}\right)=a_{21}{a}_{32}-a_{22}{a}_{31}\end{array}\end{array}$$ Finally, coming back to Equation 32.6 and substituting the result in Equation 32.7 one obtain: $$det(\mathbf{A})=a_{11}(a_{22}{a}_{33}-a_{23}{a}_{32})-a_{12}(a_{21}{a}_{33}-a_{23}{a}_{31})+a_{13}(a_{21}{a}_{32}-a_{22}{a}_{31})\text{.}$$

32.5 Trace

The trace of a square matrix $\mathbf{A}\in {\mathbb{R}}^{n\times n}$ is the sum of its diagonal elements. It is also equal to the sum of the eigenvalues ${\lambda }_i$ of $\mathbf{A}$, counted with algebraic multiplicity, i.e. $$\begin{array}{}\text{(32.8)}& \mathrm{tr}(\mathbf{A})=\sum _{i=1}^n{a}_{ii}=\sum _{i=1}^n{\lambda }_i\text{.}\end{array}$$

Some properties of the trace operator includes:

1. $\mathrm{tr}({\mathbf{A}}^{\mathrm{\top }})=\mathrm{tr}(\mathbf{A})$.
2. $\mathrm{tr}(\mathbf{A}+\mathbf{B})=\mathrm{tr}(\mathbf{A})+\mathrm{tr}(\mathbf{B})$.
3. $\mathrm{tr}(a\mathbf{A})=a\cdot \mathrm{tr}(\mathbf{A})$ for $a\in \mathbb{R}$.
4. $\mathrm{tr}({\mathbf{A}}^n)=\sum _{i=1}^n{\lambda }_i^n$ where ${\lambda }_i$ is the $i$-th eigenvalue of $\mathbf{A}$.
5. $\mathrm{tr}({\mathbf{A}}^{-1})=\sum _{i=1}^n\frac{1}{{\lambda }_i}$.