Linear Algebra · Book or PDF

Essential Student Algebra: Volume Four, Linear Algebra

T. S. Blyth & E. F. Robertson · First-year university · Study material generated by AI from this edition. Not affiliated with the publisher.

Chapter 7 · pp. 97–108

Chapter 7: Inner Product Spaces

This chapter asks when a linear map on an inner product space can be diagonalized using an orthonormal basis. The tools are projections onto perpendicular subspaces, adjoints, and a class of maps called normal maps. The chapter ends with positive maps, which behave like non-negative numbers.

Orthogonal sums and ortho-projections

Orthogonal direct sum. Take non-zero subspaces $V_1,\dots,V_n$ of an inner product space $V$ . They form an orthogonal direct sum if $V=\bigoplus V_i$ and each $V_i$ is orthogonal to the sum of all the other $V_j$ . This is an ordinary direct sum plus a perpendicularity condition. For example, $\mathbb{R}^3=\text{span}\{e_1\}\oplus\text{span}\{e_2,e_3\}$ is orthogonal, because $e_1$ is perpendicular to every combination of $e_2,e_3$ . The basic example is $V=W\oplus W^\perp$ . (p. 97)

Ortho-projection. Every projection $p$ splits $V$ as $\operatorname{Im}p\oplus\operatorname{Ker}p$ . It is an ortho-projection when $\operatorname{Ker}p=(\operatorname{Im}p)^\perp$ , which is the same as $\operatorname{Im}p=(\operatorname{Ker}p)^\perp$ . The split is then an orthogonal direct sum. Projecting $\mathbb{R}^2$ onto the x-axis along the y-axis is an ortho-projection. Projecting onto the x-axis along the line $y=x$ is not, because that line is not perpendicular to the x-axis. (p. 97)

Complements of a direct sum. In a finite-dimensional space, if $V=W\oplus X$ then also $V=W^\perp\oplus X^\perp$ . Taking complements swaps sums and intersections. So $W^\perp\cap X^\perp=(W+X)^\perp=\{0\}$ and $W^\perp+X^\perp=(W\cap X)^\perp=V$ . Together these give a direct sum. (p. 97)

Adjoint of a projection. Let $p$ be the projection on $W$ parallel to $X$ . Then $p^*$ is the projection on $X^\perp$ parallel to $W^\perp$ . First, $p^*$ is idempotent because $(p\circ p)^*=p^*\circ p^*$ , so it is a projection. Then $\operatorname{Im}p=W$ and $\operatorname{Ker}p=X$ give $\operatorname{Ker}p^*=W^\perp$ and $\operatorname{Im}p^*=X^\perp$ . (pp. 97–98)

Self-adjoint maps

A linear map $f:V\to V$ is self-adjoint if $f=f^*$ . Equivalently, $(f(x)\mid y)=(x\mid f(y))$ for all $x,y$ . In an orthonormal basis, a real self-adjoint map has a symmetric matrix. A complex one has a Hermitian matrix, which equals its conjugate transpose. (p. 98)

Projections. On a finite-dimensional space, a projection is an ortho-projection if and only if it is self-adjoint. If $p$ is an ortho-projection, then $p^*$ projects onto $(\operatorname{Ker}p)^\perp=\operatorname{Im}p$ along $\operatorname{Ker}p$ . That is the same image and kernel as $p$ , so $p^*=p$ . Conversely, if $p=p^*$ then $\operatorname{Im}p=\operatorname{Im}p^*=(\operatorname{Ker}p)^\perp$ . So orthogonality can be tested algebraically. (p. 98)

Testing a whole direct sum. A sum $V=\bigoplus V_i$ is orthogonal exactly when each projection onto $V_i$ along the others is self-adjoint. It is also orthogonal exactly when vectors from different summands are always orthogonal. For the second test, perpendicularity gives $\sum_{j\ne i}V_j\subseteq V_i^\perp$ . The dimensions agree, since $\dim\sum_{j\ne i}V_j=\dim V-\dim V_i=\dim V_i^\perp$ , so the two spaces are equal. (p. 98)

Ortho-diagonalization and spectral resolution

A map $f$ on a finite-dimensional inner product space is ortho-diagonalizable if some orthonormal basis consists of eigenvectors of $f$ . Equivalently, its matrix in some orthonormal basis is diagonal. For matrices, this means unitarily similar to a diagonal matrix (orthogonally similar in the real case). This is stronger than ordinary diagonalizability, which only needs some basis of eigenvectors. (pp. 98–99)

Characterisation. A map $f$ is ortho-diagonalizable if and only if $f=\sum_{i=1}^k\lambda_ip_i$ . Here the $\lambda_i$ are distinct scalars and the $p_i$ are non-zero self-adjoint projections with $\sum p_i=\mathrm{id}_V$ and $p_i\circ p_j=0$ for $i\ne j$ . (pp. 99–100)

Forward: let the $\lambda_i$ be the distinct eigenvalues and $p_i$ the projection onto the $i$ -th eigenspace along the others. These are self-adjoint because the eigenspaces are mutually orthogonal.
Reverse: the images of the $p_i$ turn out to be exactly the eigenspaces, and they are mutually orthogonal. Normalising vectors in them gives an orthonormal eigenbasis.

Spectral resolution. For an ortho-diagonalizable $f$ , writing $f=\sum\lambda_ip_i$ with the distinct eigenvalues and the orthogonal projections onto the eigenspaces is the spectral resolution. For $f=\text{diag}(2,2,8)$ it is $2p_1+8p_2$ . Here $p_1$ projects onto $\text{span}\{e_1,e_2\}$ and $p_2$ onto $\text{span}\{e_3\}$ . (p. 100)

Adjoint. If $f=\sum\lambda_ip_i$ , then $f^*=\sum\overline{\lambda_i}p_i$ , because $p_i^*=p_i$ and adjoints conjugate scalars. So $f^*$ is also ortho-diagonalizable. Since $p_i\circ p_j=0$ for $i\ne j$ and $p_i\circ p_i=p_i$ , both $f\circ f^*$ and $f^*\circ f$ equal $\sum|\lambda_i|^2p_i$ . Hence $f$ commutes with $f^*$ . (p. 101)

Normal maps

A map $f$ is normal if $f\circ f^*=f^*\circ f$ . A square matrix $A$ is normal if $AA^*=A^*A$ . The last paragraph above shows that normality is necessary for ortho-diagonalizability. Self-adjoint maps, unitary maps and rotation matrices are normal. The matrix $\begin{pmatrix}1&1\\0&1\end{pmatrix}$ is not. (p. 101)

Four useful facts about a normal $f$ :

Equal norms. $\|f(x)\|=\|f^*(x)\|$ for all $x$ . Indeed $(f(x)\mid f(x))=(x\mid f^*f(x))=(x\mid ff^*(x))=(f^*(x)\mid f^*(x))$ . So $f(x)=0$ exactly when $f^*(x)=0$ . (pp. 101–102)
Polynomials. For any polynomial $q$ , $q(f)$ is normal. Its adjoint is a polynomial in $f^*$ with conjugated coefficients, and polynomials in commuting maps commute. For example, $f-a\,\mathrm{id}$ is normal. (p. 102)
Image and kernel. $\operatorname{Im}f\cap\operatorname{Ker}f=\{0\}$ . If $z=f(y)$ and $f(z)=0$ , then $f^*(z)=0$ by fact 1. So $(z\mid z)=(f^*(z)\mid y)=0$ and $z=0$ . The nilpotent $\begin{pmatrix}0&1\\0&0\end{pmatrix}$ is not normal and violates this. (p. 102)
Projections. A projection is normal if and only if it is self-adjoint, so the normal projections are exactly the ortho-projections. For the hard direction, put $y=x-p(x)$ . Then $p(y)=0$ , so $p^*(y)=0$ , giving $p^*=p^*\circ p$ . Taking adjoints gives $p=p^*\circ p$ , hence $p=p^*$ . (p. 102)

Spectral theorems

Complex case. On a finite-dimensional complex inner product space, a map is ortho-diagonalizable if and only if it is normal. For sufficiency, the minimum polynomial splits into linear factors over $\mathbb{C}$ . A repeated factor $(X-a)^2$ would contradict $\operatorname{Im}\cap\operatorname{Ker}=\{0\}$ for the normal map $f-a\,\mathrm{id}$ . So $f$ is diagonalizable. Its spectral projections are polynomials in $f$ , hence normal, hence self-adjoint. (pp. 102–103)

Matrix form. A complex square matrix is unitarily similar to a diagonal matrix if and only if it is normal. Such a matrix is $A=UDU^*$ with $U$ unitary and $D$ diagonal. (p. 103)

Common mistake. It is false that every normal map on a real space is diagonalizable. The complex proof needs an algebraically closed field, and $\mathbb{R}$ is not one. A rotation matrix with entries $-\tfrac12$ and $\pm\tfrac{\sqrt3}{2}$ is normal. Its minimum polynomial $X^2+X+1$ has no real zeros, so it has no real eigenvalues. (p. 103)

Self-adjoint maps. On a complex space, $f$ is self-adjoint if and only if it is normal with all eigenvalues real. If $f$ is normal with real eigenvalues, then $f^*=\sum\overline{\lambda_i}p_i=\sum\lambda_ip_i=f$ . If $f=f^*$ , comparing spectral resolutions forces $\lambda_i=\overline{\lambda_i}$ . In particular, every Hermitian or real symmetric matrix has only real eigenvalues. For example, $\begin{pmatrix}2&1\\1&2\end{pmatrix}$ has eigenvalues 1 and 3. (p. 104)

Real case. On a finite-dimensional real inner product space, $f$ is ortho-diagonalizable if and only if it is self-adjoint. Equivalently, a real square matrix is orthogonally similar to a diagonal matrix if and only if it is symmetric. Over $\mathbb{R}$ , an ortho-diagonalizable map has real eigenvalues, so $f^*=f$ . Conversely, a symmetric real matrix, viewed as a complex self-adjoint map, has real eigenvalues. So its minimum polynomial splits over $\mathbb{R}$ into distinct linear factors. This is the spectral theorem for real symmetric matrices. (pp. 104–105)

Procedure for a real symmetric matrix $A$ . (p. 105)

Find the eigenvalues.
Find a basis of each eigenspace.
Orthonormalize each basis by Gram-Schmidt. This is only needed inside each eigenspace, because eigenspaces for different eigenvalues are already orthogonal.
Use all these vectors as the columns of an orthogonal matrix $P$ . Then $P^{-1}AP$ is diagonal.

In the text's $3\times3$ example the characteristic polynomial is $(X-2)^2(X-8)$ . Two orthonormal vectors span the eigenspace for 2, and one unit vector spans the eigenspace for 8. Together they give $P^{-1}AP=\operatorname{diag}(2,2,8)$ .

Positive maps

A complex-only fact. On a complex space, if $(f(x)\mid x)=0$ for all $x$ , then $f=0$ . Expanding $(f(y+z)\mid y+z)=0$ and $(f(iy+z)\mid iy+z)=0$ gives two equations. Together they force $(f(y)\mid z)=0$ for all $y,z$ . Over $\mathbb{R}$ this fails: a 90° rotation of $\mathbb{R}^2$ has $(f(x)\mid x)=0$ for all $x$ but is not zero. (p. 106)

Reality test. On a finite-dimensional complex space, $f$ is self-adjoint if and only if $(f(x)\mid x)$ is real for every $x$ . If $f=f^*$ , then $(f(x)\mid x)=(x\mid f(x))=\overline{(f(x)\mid x)}$ , so it is real. Conversely, if all these values are real, then $((f^*-f)(x)\mid x)=0$ for all $x$ , so $f^*=f$ . (p. 106)

Definitions. A map is positive (semi-definite) if it is self-adjoint and $(f(x)\mid x)\ge0$ for all $x$ . It is positive definite if it is self-adjoint and $(f(x)\mid x)>0$ for all non-zero $x$ . The identity is positive definite. A self-adjoint projection $p$ is positive, since $(p(x)\mid x)=\|p(x)\|^2\ge0$ . It is not positive definite unless $p=\mathrm{id}$ . (p. 106)

Equivalent descriptions. On a finite-dimensional space, these are equivalent: (1) $f$ is positive; (2) $f$ is self-adjoint with all eigenvalues $\ge0$ ; (3) $f=g^2$ for some self-adjoint $g$ ; (4) $f=h^*\circ h$ for some $h$ . (pp. 106–107)

(1)⇒(2): an eigenvector $x$ with eigenvalue $\lambda$ gives $0\le(f(x)\mid x)=\lambda(x\mid x)$ , so $\lambda\ge0$ .
(2)⇒(3): take $g=\sum\sqrt{\lambda_i}\,p_i$ from the spectral resolution.
(4)⇒(1): $h^*h$ is self-adjoint and $(h^*h(x)\mid x)=\|h(x)\|^2\ge0$ .

Unique square root. A positive map $f$ has exactly one positive square root $g$ with $g^2=f$ , and $g$ is a polynomial in $f$ . If $f=\sum\lambda_ip_i$ , then $g=\sum\sqrt{\lambda_i}\,p_i$ works. Any other positive root has the same eigenspaces as $f$ , with eigenvalues $\sqrt{\lambda_i}$ , so it equals $g$ . With Lagrange polynomials $P_i$ satisfying $P_i(\lambda_j)=\delta_{ij}$ , the polynomial $q=\sum\sqrt{\lambda_i}\,P_i$ gives $q(f)=g$ . (pp. 107–108)

Positive definite maps. A map is positive definite if and only if it is self-adjoint with all eigenvalues real and strictly positive. This holds if and only if $f=g^2$ for an invertible self-adjoint $g$ , and if and only if $f=h^*\circ h$ for an invertible $h$ . In particular, a positive definite map is invertible, since 0 is not an eigenvalue. For example, $\begin{pmatrix}2&0\\0&3\end{pmatrix}$ is positive definite. $\begin{pmatrix}1&0\\0&0\end{pmatrix}$ is only positive and is not invertible. (p. 108)

Gram matrices. A square matrix represents a positive map in an orthonormal basis (it is a Gram matrix) if and only if it is self-adjoint with all eigenvalues $\ge0$ . The matrix $\begin{pmatrix}2&1\\1&2\end{pmatrix}$ has eigenvalues 1 and 3, so it qualifies. $\begin{pmatrix}1&2\\2&1\end{pmatrix}$ has eigenvalue $-1$ , so it does not. (p. 108)

Check yourself

Why is projecting $\mathbb{R}^2$ onto the x-axis along the line $y=x$ not an ortho-projection? What does that say about the matrix of the projection?
Show that if $f$ is normal, then $f(x)=0$ exactly when $f^*(x)=0$ .
Explain why a rotation of $\mathbb{R}^2$ through 120° is normal but not ortho-diagonalizable over $\mathbb{R}$ .
Outline the steps for orthogonally diagonalizing a real symmetric matrix. Why is Gram-Schmidt only needed inside each eigenspace?
Is $\begin{pmatrix}1&2\\2&1\end{pmatrix}$ a positive map? Is $\begin{pmatrix}2&1\\1&2\end{pmatrix}$ positive definite? Justify each answer using eigenvalues.