Linear Algebra · Book or PDF

Essential Student Algebra: Volume Four, Linear Algebra

T. S. Blyth & E. F. Robertson · First-year university · Study material generated by AI from this edition. Not affiliated with the publisher.

Chapter 6 · pp. 79–96

Chapter 6: Dual Spaces

This chapter adds a way to measure lengths and angles to a vector space: the inner product. With it we can build orthonormal bases, find closest vectors, and see that every linear functional is an inner product with a fixed vector. That link between a space and its dual leads to adjoint maps, orthogonal complements and unitary matrices.

Inner products and their rules

An inner product on a complex vector space $V$ gives each pair $x,y$ a complex number $(x|y)$ . It must obey four rules: $(x+x'|y)=(x|y)+(x'|y)$ , $(\alpha x|y)=\alpha(x|y)$ , $(x|y)=\overline{(y|x)}$ , and $(x|x)>0$ unless $x=0$ . Conjugate symmetry makes $(x|x)$ real, so the positivity rule makes sense. A real inner product space is the same idea over $\mathbb{R}$ , without conjugation. (p. 79)

Conjugate symmetry also controls the second slot. The product is additive there, but scalars come out conjugated: $(x|\alpha y)=\overline{(\alpha y|x)}=\bar\alpha(x|y)$ . Also $(x|0)=0=(0|x)$ . Over $\mathbb{R}$ the product is linear in both slots. (pp. 79–80)

A common mistake is to think a complex inner product is linear in both arguments. It is not: $(x|iy)=-i(x|y)$ . Forgetting this gives sign errors when expanding $\|x+y\|^2$ or computing Fourier coefficients. (p. 80)

Examples of inner product spaces

On $\mathbb{C}^n$ the standard product is $(z|w)=\sum_{j} z_j\overline{w_j}$ . On $\mathbb{R}^n$ it is $\sum_j x_jy_j$ , which is the dot product when $n=2,3$ . For $z=(1,i)$ and $w=(i,1)$ : $(z|w)=1\cdot\bar i+i\cdot 1=-i+i=0$ . (p. 80)

Two more examples: continuous real functions on $[a,b]$ with $(f|g)=\int_a^b fg$ , where the integral replaces the sum, and matrices with $(A|B)=\mathrm{tr}(B^*A)$ . For $n\times n$ complex matrices the trace equals $\sum a_{ij}\overline{b_{ij}}$ , so this is the standard product on $\mathbb{C}^{n^2}$ . For real matrices $B^*$ is just $B^t$ . (pp. 80–81)

Norm, distance and two key inequalities

The norm is $\|x\|=\sqrt{(x|x)}$ and the distance is $d(x,y)=\|x-y\|$ . In $\mathbb{R}^2$ , $\|x\|^2=x_1^2+x_2^2$ , so these agree with Pythagoras. Also $\|x\|=0$ only when $x=0$ . (p. 81)

Scaling gives $\|\lambda x\|=|\lambda|\,\|x\|$ , because $(\lambda x|\lambda x)=\lambda\bar\lambda(x|x)$ . For example $\|(3+4i)x\|=5\|x\|$ . (p. 81)

Cauchy-Schwarz: $|(x|y)|\le\|x\|\,\|y\|$ . For $x\neq0$ put $z=y-\frac{(y|x)}{\|x\|^2}x$ , which is orthogonal to $x$ . Then $0\le\|z\|^2=\|y\|^2-\frac{|(x|y)|^2}{\|x\|^2}$ , and rearranging gives the result. Check with $x=(1,2)$ , $y=(3,1)$ : $5\le\sqrt5\sqrt{10}\approx7.07$ . (pp. 81–82)

Triangle inequality: $\|x+y\|\le\|x\|+\|y\|$ . Expand $\|x+y\|^2=\|x\|^2+2\,\mathrm{Re}(x|y)+\|y\|^2$ , bound $\mathrm{Re}(x|y)$ by $|(x|y)|$ , apply Cauchy-Schwarz to get $(\|x\|+\|y\|)^2$ , then take square roots. (p. 82)

Orthogonal and orthonormal sets

Vectors are orthogonal when $(x|y)=0$ . An orthogonal set has all distinct pairs orthogonal, and an orthonormal set is an orthogonal set of unit vectors. In $\mathbb{R}^2$ , orthogonal means $x_1y_1+x_2y_2=0$ , i.e. perpendicular lines. The standard bases of $\mathbb{R}^n$ and $\mathbb{C}^n$ are orthonormal, and so are the functions $1,\sin kx,\cos kx$ as an orthogonal set under $\int_{-\pi}^{\pi}fg$ (they need scaling to be unit length). (pp. 82–83)

To make an orthogonal set of nonzero vectors orthonormal, replace each $x$ by $x^*=x/\|x\|$ . The length becomes 1 and the direction is unchanged, so orthogonality stays. Example: $(3,4)\mapsto(3/5,4/5)$ . (p. 83)

Orthogonal sets of nonzero vectors are linearly independent. If $\sum\lambda_kx_k=0$ , take the product with $x_i$ . Every term vanishes except $\lambda_i\|x_i\|^2=0$ , so $\lambda_i=0$ . Hence $n$ orthogonal nonzero vectors in an $n$ -dimensional space form a basis. (p. 83)

Bessel's inequality and expansions

If $\{e_1,\dots,e_n\}$ is orthonormal, then $\sum_k|(x|e_k)|^2\le\|x\|^2$ (Bessel). Proof: let $z=x-\sum(x|e_k)e_k$ ; then $0\le\|z\|^2=\|x\|^2-\sum|(x|e_k)|^2$ . Equality holds exactly when $z=0$ . (pp. 83–84)

When the $e_k$ span $W$ , these statements are equivalent: $x\in W$ ; Bessel is an equality; $x=\sum(x|e_k)e_k$ ; and $(x|y)=\sum(x|e_k)(e_k|y)$ for all $y$ . The key step is that if $x=\sum\lambda_ke_k$ , taking the product with $e_j$ gives $\lambda_j=(x|e_j)$ . This gives a computable test for membership in the span. (pp. 83–84)

An orthonormal basis is an orthonormal set that is also a basis. Examples: the standard bases, and the matrices $E_{pq}$ (a single 1 in position $(p,q)$ ) under $(A|B)=\mathrm{tr}(B^*A)$ . (p. 84)

Relative to an orthonormal basis, three formulas hold: $x=\sum(x|e_k)e_k$ (Fourier expansion), $\|x\|^2=\sum|(x|e_k)|^2$ , and $(x|y)=\sum(x|e_k)\overline{(y|e_k)}$ (Parseval). The numbers $(x|e_k)$ are Fourier coefficients. For $x=(1,1,1)$ and the basis found below, they are $\sqrt2$ , $\sqrt2/\sqrt3$ , $1/\sqrt3$ , and $2+\tfrac23+\tfrac13=3=\|x\|^2$ . (pp. 86–87)

The Gram-Schmidt process

Given independent vectors $x_1,\dots,x_k$ , Gram-Schmidt produces an orthonormal list $y_1,\dots,y_k$ with the same span. Set $y_1=x_1^*$ . For each later $t$ , subtract the components along earlier vectors, $x_t-\sum_{i<t}(x_t|y_i)y_i$ , then divide by the norm. The subtracted vector is orthogonal to the earlier $y_j$ by construction, and each $y_t$ is a combination of $x_1,\dots,x_t$ . (p. 85)

Example in $\mathbb{R}^3$ with $(0,1,1),(1,0,1),(1,1,0)$ : $y_1=(0,1,1)/\sqrt2$ . Then $(1,0,1)-(0,\tfrac12,\tfrac12)=\tfrac12(2,-1,1)$ , giving $y_2=(2,-1,1)/\sqrt6$ . Finally the third vector minus its components is $\tfrac23(1,1,-1)$ , giving $y_3=(1,1,-1)/\sqrt3$ . Dropping scalar factors before normalising saves arithmetic. (p. 86)

Consequently every finite-dimensional inner product space has an orthonormal basis. Any orthonormal subset extends to one: extend it to an ordinary basis first, then apply Gram-Schmidt. The original vectors come out unchanged. (pp. 86–87)

Inner product isomorphisms

An inner product isomorphism is a vector space isomorphism $f$ with $(f(x)|f(y))=(x|y)$ . It preserves lengths, distances and orthogonality; a rotation of $\mathbb{R}^2$ is an example. (p. 87)

A linear map is such an isomorphism exactly when it sends an orthonormal basis of $V$ to an orthonormal basis of $W$ . One direction: $(f(e_i)|f(e_j))=(e_i|e_j)=\delta_{ij}$ . The other uses Parseval's identity in both spaces. (pp. 87–88)

Duality and adjoints

A conjugate transformation satisfies $f(x+y)=f(x)+f(y)$ and $f(\lambda x)=\bar\lambda f(x)$ . If it is also bijective it is a conjugate isomorphism. Over $\mathbb{R}$ these are just linear maps. An example is $x\mapsto\bar x$ on $\mathbb{C}^n$ . (p. 88)

In a finite-dimensional inner product space $V$ , the map $y\mapsto y^\flat$ , where $y^\flat(x)=(x|y)$ , is a conjugate isomorphism onto the dual space. It is injective because $y^\flat=0$ gives $(y|y)=0$ . It is surjective because any functional $f$ equals $x\mapsto(x|z)$ with $z=\sum\overline{f(e_k)}e_k$ . So every linear functional is an inner product with a unique vector. (pp. 89–90)

This gives the adjoint. For linear $f:V\to W$ there is a unique linear $f^*:W\to V$ with $(f(x)|y)=(x|f^*(y))$ . For $x\mapsto Ax$ on $\mathbb{C}^n$ it is $y\mapsto A^*y$ . (p. 90)

Rules: $(f+g)^*=f^*+g^*$ , $(\lambda f)^*=\bar\lambda f^*$ , $(h\circ f)^*=f^*\circ h^*$ , $(f^*)^*=f$ . Each follows by checking the defining identity and using uniqueness. The scalar is conjugated and the order reverses. (p. 91)

When $\dim V=\dim W$ , these are equivalent: $f$ is an inner product isomorphism; $f$ is an isomorphism with $f^{-1}=f^*$ ; $f\circ f^*=\mathrm{id}_W$ ; $f^*\circ f=\mathrm{id}_V$ . If $f^*f=\mathrm{id}$ then $(f(x)|f(y))=(x|f^*f(y))=(x|y)$ . So the inverse is found by taking the adjoint. (pp. 91–92)

Orthogonal complements and best approximation

The orthogonal complement of a non-empty set $E$ is $E^\perp=\{y:(x|y)=0\text{ for all }x\in E\}$ , always a subspace. In $\mathbb{R}^3$ , the complement of a plane through the origin is the perpendicular line through the origin. (p. 92)

If $W$ is a finite-dimensional subspace, $V=W\oplus W^\perp$ . For $x\in V$ take an orthonormal basis $e_i$ of $W$ , let $x'=\sum(x|e_i)e_i$ and $x''=x-x'$ . Then $x''\in W^\perp$ . The sum is direct because a vector $z$ in both has $(z|z)=0$ . (p. 92)

The closest vector in $W$ to $x$ is its component $a=\sum(x|e_k)e_k$ . Write $x=a+b$ with $b\in W^\perp$ . For $y\in W$ , $\|x-y\|^2=\|a-y\|^2+\|b\|^2\ge\|x-a\|^2$ , with equality only at $y=a$ . This is orthogonal projection. (p. 93)

Application: with $(f|g)=\int_{-\pi}^{\pi}fg$ , the functions $1,\sin kx,\cos kx$ ( $k\le n$ ) are orthogonal and span a $(2n+1)$ -dimensional space. The nearest trigonometric polynomial to $f$ is $\tfrac12a_0+\sum_{k\le n}(a_k\cos kx+b_k\sin kx)$ , where $a_k=\frac1\pi\int f\cos kx\,dx$ and $b_k=\frac1\pi\int f\sin kx\,dx$ . As $n\to\infty$ this gives the Fourier series, which converges to $f$ when $f$ is infinitely differentiable. (pp. 93–94)

In finite dimensions, $\dim W^\perp=\dim V-\dim W$ and $W^{\perp\perp}=W$ . Clearly $W\subseteq W^{\perp\perp}$ , and both have the same dimension. (p. 94)

For subspaces $A,B$ : $A\subseteq B$ implies $B^\perp\subseteq A^\perp$ ; $(A\cap B)^\perp=A^\perp+B^\perp$ ; $(A+B)^\perp=A^\perp\cap B^\perp$ . Taking complements reverses inclusion and swaps sums with intersections. (p. 94)

For linear $f$ : $\mathrm{Im}\,f^*=(\mathrm{Ker}\,f)^\perp$ and $\mathrm{Ker}\,f^*=(\mathrm{Im}\,f)^\perp$ . If $x=f^*(y)$ and $z\in\mathrm{Ker}f$ , then $(z|x)=(f(z)|y)=0$ . If $f^*(y)=0$ , then $(f(x)|y)=0$ for all $x$ . A dimension count turns the inclusions into equalities. (pp. 94–95)

Matrices of adjoints and unitary matrices

The adjoint (conjugate transpose) of $A=[a_{ij}]$ is $A^*$ with $[A^*]_{ij}=\overline{a_{ji}}$ . For $A=\begin{pmatrix}1&i\\2&3\end{pmatrix}$ , $A^*=\begin{pmatrix}1&2\\-i&3\end{pmatrix}$ . For real matrices it is the transpose. (p. 95)

If $A$ represents $f$ relative to ordered orthonormal bases, then $A^*$ represents $f^*$ relative to the same bases. This works because $a_{ij}=(f(v_j)|w_i)=\overline{(f^*(w_i)|v_j)}$ . Orthonormal bases are essential; otherwise the matrix of $f^*$ is generally not $A^*$ . (pp. 95–96)

A square matrix $U$ is unitary if $U^{-1}=U^*$ (orthogonal if real). $B$ is unitarily similar to $A$ if $B=U^*AU$ for some unitary $U$ . Unitary matrices represent inner product isomorphisms. Two matrices represent the same map in different orthonormal bases exactly when they are unitarily similar, which is an equivalence relation. (p. 96)

Check yourself

Show that $(x|\alpha y)=\bar\alpha(x|y)$ follows from the axioms, and compute $(x|iy)$ if $(x|y)=2+i$ .
Apply Gram-Schmidt to $(1,1,0)$ and $(1,0,1)$ in $\mathbb{R}^3$ with the standard product.
Why must an orthogonal set of nonzero vectors be linearly independent?
If $W$ is a plane through the origin in $\mathbb{R}^3$ , what are $W^\perp$ and $W^{\perp\perp}$ , and how are their dimensions related?
Find $A^*$ for $A=\begin{pmatrix}2&1+i\\0&-i\end{pmatrix}$ , and explain why orthonormal bases matter for the matrix of $f^*$ .