Linear Algebra

Orthogonal bases and projection onto a subspace

A plane through the origin and a point off it. Which point of the plane is closest, and is there a formula for it that needs no calculus?

You need

Dot product, length and orthogonality in n dimensionsLinear Algebra Basis and dimensionLinear Algebra

Unlocks

Least squares: the best approximate solution of Ax = bLinear Algebra The Gram-Schmidt processLinear Algebra

Where you are 3 in Orthogonality and least squares

Linear algebra6

Orthogonality and least squares3

Orthogonal bases and projection onto a subspace

Least squares: the best approximate solution of Ax = b The Gram-Schmidt process

Why this exists

Pirjo, Tuula and Kaija are surveying a climbing crag. In a frame with the base station at the origin, three perpendicular axes (the third pointing up) and coordinates in tens of meters, they model one steep rock face as the plane W = \operatorname{span}\{\mathbf{u}_1, \mathbf{u}_2\} with \mathbf{u}_1 = (1, 2, 2) and \mathbf{u}_2 = (2, 1, -2). A survey drone hovers at \mathbf{y} = (6, 3, 3). Where on the face is the drone closest, and how far away is it?

For a line through the origin the answer is known: drop a perpendicular, and the foot is \displaystyle \frac{\mathbf{y} \cdot \mathbf{u}}{\mathbf{u} \cdot \mathbf{u}}\mathbf{u} (Vector and scalar projection). For a plane, or a subspace of any dimension, minimizing a distance over every point sounds like calculus, but it is not needed. If the basis of W is orthogonal, the closest point is the sum of the projections onto the basis vectors, one at a time.

Definition

Throughout, W is a subspace of \mathbb{R}^n (Subspaces) and the dot product, length and orthogonality are those of Dot product, length and orthogonality in n dimensions.

A set of vectors \{\mathbf{u}_1, \dots, \mathbf{u}_k\} is an orthogonal set when every pair of different vectors in it is orthogonal: \mathbf{u}_i \cdot \mathbf{u}_j = 0 whenever i \neq j. An orthogonal basis of W is a basis of W that is also an orthogonal set. An orthonormal set is an orthogonal set of unit vectors, and an orthonormal basis is a basis that is an orthonormal set.

The orthogonal complement of W is the set of vectors orthogonal to every vector of W:

W^{\perp} = \{\mathbf{z} \in \mathbb{R}^n : \mathbf{z} \cdot \mathbf{w} = 0 \text{ for every } \mathbf{w} \in W\}.

If \{\mathbf{u}_1, \dots, \mathbf{u}_k\} is an orthogonal basis of W, the orthogonal projection of \mathbf{y} onto W is

\begin{aligned} \operatorname{proj}_W \mathbf{y} &= \frac{\mathbf{y} \cdot \mathbf{u}_1}{\mathbf{u}_1 \cdot \mathbf{u}_1}\mathbf{u}_1 + \dots \\ &\quad + \frac{\mathbf{y} \cdot \mathbf{u}_k}{\mathbf{u}_k \cdot \mathbf{u}_k}\mathbf{u}_k, \end{aligned}

written \hat{\mathbf{y}} for short. The vector \mathbf{z} = \mathbf{y} - \hat{\mathbf{y}} is the residual. Theorem 3 shows that \hat{\mathbf{y}} does not depend on which orthogonal basis is used. The projection of \mathbf{y} onto the zero subspace is \mathbf{0}.

Intuition

Take W = \operatorname{span}\{(2, 1)\}, a line in \mathbb{R}^2, and \mathbf{y} = (3, 4). Since \mathbf{y} \cdot (2, 1) = 10 and (2, 1) \cdot (2, 1) = 5, the projection is 2(2, 1) = (4, 2) and the residual is \mathbf{z} = (-1, 2), which is orthogonal to the line. The figure shows why that right angle makes \hat{\mathbf{y}} the closest point: from \mathbf{y} to any other point \mathbf{v} of the line is the hypotenuse of a right triangle whose legs are \mathbf{z} and \hat{\mathbf{y}} - \mathbf{v}, so it is longer than the leg \mathbf{z}.

Figure 1The point of W nearest \mathbf{y} = (3, 4) is \hat{\mathbf{y}} = (4, 2). The residual \mathbf{z} = (-1, 2) meets W at a right angle, and every other point of W, such as \mathbf{v} = (8, 4), is farther away, here 5 against \displaystyle \sqrt{5}.

In a larger subspace the residual must be orthogonal to the whole subspace. Each term of the formula removes the part of \mathbf{y} along one \mathbf{u}_i, and because the \mathbf{u}_i are mutually perpendicular, removing one part never puts back another. With a basis that is not orthogonal the parts overlap, and the overlap is counted twice (Example 7).

Results

Theorem 1 (orthogonal sets are independent). An orthogonal set of nonzero vectors in \mathbb{R}^n is linearly independent (Linear independence). So an orthogonal set of k nonzero vectors in a subspace W with \dim W = k is an orthogonal basis of W (Basis and dimension).

Theorem 2 (weights in an orthogonal basis). If \{\mathbf{u}_1, \dots, \mathbf{u}_k\} is an orthogonal basis of W and \mathbf{y} is in W, then

\mathbf{y} = c_1\mathbf{u}_1 + \dots + c_k\mathbf{u}_k \quad \text{with} \quad c_j = \frac{\mathbf{y} \cdot \mathbf{u}_j}{\mathbf{u}_j \cdot \mathbf{u}_j}.

Each weight is one dot product, with no system to solve (Coordinates in a basis and change of basis).

Facts about W^{\perp}. If W = \operatorname{span}\{\mathbf{w}_1, \dots, \mathbf{w}_k\}, a vector lies in W^{\perp} exactly when it is orthogonal to each \mathbf{w}_i. So W^{\perp} is the null space of the matrix whose rows are \mathbf{w}_1, \dots, \mathbf{w}_k (Kernel), and by Rank and nullity, \dim W + \dim W^{\perp} = n. Also W^{\perp} is a subspace, and W \cap W^{\perp} = \{\mathbf{0}\} (Exercises 9 to 11).

Theorem 3 (orthogonal decomposition). Let W have an orthogonal basis \{\mathbf{u}_1, \dots, \mathbf{u}_k\}. Every \mathbf{y} in \mathbb{R}^n can be written in exactly one way as

\mathbf{y} = \hat{\mathbf{y}} + \mathbf{z}, \qquad \hat{\mathbf{y}} \in W, \quad \mathbf{z} \in W^{\perp},

and \hat{\mathbf{y}} is \operatorname{proj}_W \mathbf{y} as defined above.

Theorem 4 (best approximation). For every \mathbf{v} in W other than \hat{\mathbf{y}} = \operatorname{proj}_W \mathbf{y},

\lVert \mathbf{y} - \hat{\mathbf{y}} \rVert < \lVert \mathbf{y} - \mathbf{v} \rVert.

So \hat{\mathbf{y}} is the point of W closest to \mathbf{y}, and the distance from \mathbf{y} to W is \lVert \mathbf{y} - \hat{\mathbf{y}} \rVert.

Theorem 5 (orthonormal bases and the projection matrix). If \{\mathbf{q}_1, \dots, \mathbf{q}_k\} is an orthonormal basis of W and U is the n \times k matrix with these columns, then U^{T}U = I_k and

\begin{aligned} \operatorname{proj}_W \mathbf{y} &= (\mathbf{y} \cdot \mathbf{q}_1)\mathbf{q}_1 + \dots + (\mathbf{y} \cdot \mathbf{q}_k)\mathbf{q}_k \\ &= UU^{T}\mathbf{y}. \end{aligned}

The n \times n matrix P = UU^{T} is the projection matrix onto W. It satisfies P^{2} = P and P^{T} = P (Exercise 22), and by Theorem 3 it is the same for every orthonormal basis of W.

Why it is true

Theorem 1. Suppose c_1\mathbf{u}_1 + \dots + c_k\mathbf{u}_k = \mathbf{0}. Dot both sides with \mathbf{u}_j. By the distributive property and the rule for scalars, the left side is c_1(\mathbf{u}_1 \cdot \mathbf{u}_j) + \dots + c_k(\mathbf{u}_k \cdot \mathbf{u}_j), and every term with i \neq j is 0 because the set is orthogonal. What is left is c_j(\mathbf{u}_j \cdot \mathbf{u}_j) = \mathbf{0} \cdot \mathbf{u}_j = 0. Since \mathbf{u}_j \neq \mathbf{0}, \mathbf{u}_j \cdot \mathbf{u}_j > 0, so c_j = 0. This holds for every j, so only the trivial combination gives \mathbf{0}. Then k independent vectors in a k-dimensional space form a basis.

Theorem 2. The same computation with \mathbf{y} in place of \mathbf{0}: dotting \mathbf{y} = c_1\mathbf{u}_1 + \dots + c_k\mathbf{u}_k with \mathbf{u}_j gives \mathbf{y} \cdot \mathbf{u}_j = c_j(\mathbf{u}_j \cdot \mathbf{u}_j), and dividing by the positive number \mathbf{u}_j \cdot \mathbf{u}_j gives c_j.

Theorem 3. Existence. Let \hat{\mathbf{y}} be the sum in the definition, which is in W because it is a combination of basis vectors, and let \mathbf{z} = \mathbf{y} - \hat{\mathbf{y}}. For each j, the orthogonality of the basis leaves one term of \hat{\mathbf{y}} \cdot \mathbf{u}_j:

\begin{aligned} \mathbf{z} \cdot \mathbf{u}_j &= \mathbf{y} \cdot \mathbf{u}_j - \frac{\mathbf{y} \cdot \mathbf{u}_j}{\mathbf{u}_j \cdot \mathbf{u}_j}(\mathbf{u}_j \cdot \mathbf{u}_j) \\ &= 0. \end{aligned}

So \mathbf{z} is orthogonal to each basis vector, hence to every combination of them, which is every vector of W. Uniqueness. Suppose also \mathbf{y} = \mathbf{w} + \mathbf{z}' with \mathbf{w} \in W and \mathbf{z}' \in W^{\perp}. Subtracting the two expressions gives \hat{\mathbf{y}} - \mathbf{w} = \mathbf{z}' - \mathbf{z}. The left side is in W and the right side is in W^{\perp}, both being subspaces, so this vector lies in W \cap W^{\perp} = \{\mathbf{0}\}. Hence \mathbf{w} = \hat{\mathbf{y}} and \mathbf{z}' = \mathbf{z}. So every orthogonal basis of W gives the same \hat{\mathbf{y}}.

Theorem 4. Let \mathbf{v} \in W with \mathbf{v} \neq \hat{\mathbf{y}}. Write

\mathbf{y} - \mathbf{v} = (\mathbf{y} - \hat{\mathbf{y}}) + (\hat{\mathbf{y}} - \mathbf{v}).

The first vector is \mathbf{z}, in W^{\perp}. The second is a difference of two vectors of W, so it is in W. They are orthogonal, and the Pythagorean theorem in \mathbb{R}^n gives

\lVert \mathbf{y} - \mathbf{v} \rVert^2 = \lVert \mathbf{y} - \hat{\mathbf{y}} \rVert^2 + \lVert \hat{\mathbf{y}} - \mathbf{v} \rVert^2.

The last term is positive because \hat{\mathbf{y}} \neq \mathbf{v}, so \lVert \mathbf{y} - \mathbf{v} \rVert^2 > \lVert \mathbf{y} - \hat{\mathbf{y}} \rVert^2. Taking square roots of these nonnegative numbers keeps the inequality.

Theorem 5. The (i, j) entry of U^{T}U is \mathbf{q}_i \cdot \mathbf{q}_j, which is 1 when i = j and 0 otherwise, so U^{T}U = I_k. Each \mathbf{q}_i \cdot \mathbf{q}_i = 1, so the denominators in the projection formula are 1. The vector U^{T}\mathbf{y} has entries \mathbf{q}_1 \cdot \mathbf{y}, \dots, \mathbf{q}_k \cdot \mathbf{y}, and multiplying it by U forms the combination of the columns with those weights (Matrix multiplication), which is the sum above.

Worked examples

Example 1

Day 1: an orthogonal basis of \mathbb{R}^3 and the drone’s coordinates

Show that \mathbf{u}_1 = (1, 2, 2), \mathbf{u}_2 = (2, 1, -2) and \mathbf{u}_3 = (2, -2, 1) form an orthogonal basis of \mathbb{R}^3, and write the drone’s position \mathbf{y} = (6, 3, 3) in this basis.

Solution

Check each pair: \mathbf{u}_1 \cdot \mathbf{u}_2 = 2 + 2 - 4 = 0, \mathbf{u}_1 \cdot \mathbf{u}_3 = 2 - 4 + 2 = 0 and \mathbf{u}_2 \cdot \mathbf{u}_3 = 4 - 2 - 2 = 0. The set is orthogonal and no vector is zero, so by Theorem 1 it is independent, and three independent vectors in the three-dimensional space \mathbb{R}^3 are a basis.

Each vector has \mathbf{u}_i \cdot \mathbf{u}_i = 9, so each has length 3. By Theorem 2 the weights are single dot products divided by 9:

\begin{aligned} c_1 &= \tfrac{6 + 6 + 6}{9} = 2, \\ c_2 &= \tfrac{12 + 3 - 6}{9} = 1, \\ c_3 &= \tfrac{12 - 6 + 3}{9} = 1. \end{aligned}

So \mathbf{y} = 2\mathbf{u}_1 + \mathbf{u}_2 + \mathbf{u}_3. Check: (2, 4, 4) + (2, 1, -2) + (2, -2, 1) = (6, 3, 3).

Example 2

Day 1: weights in \mathbb{R}^4, one at a time

The vectors \mathbf{v}_1 = (1, 1, 0, 1), \mathbf{v}_2 = (1, -1, 1, 0), \mathbf{v}_3 = (0, 1, 1, -1) and \mathbf{v}_4 = (1, 0, -1, -1) form an orthogonal set. Find the coordinates of \mathbf{x} = (3, 1, 2, 4) in this basis of \mathbb{R}^4.

Solution

The six pairwise dot products are all 0 (for instance \mathbf{v}_3 \cdot \mathbf{v}_4 = 0 + 0 - 1 + 1 = 0), and four nonzero orthogonal vectors in \mathbb{R}^4 are a basis by Theorem 1. Every \mathbf{v}_i \cdot \mathbf{v}_i = 3. Theorem 2 gives each weight on its own:

\begin{aligned} c_1 &= \tfrac{3 + 1 + 0 + 4}{3} = \tfrac{8}{3}, \\ c_2 &= \tfrac{3 - 1 + 2 + 0}{3} = \tfrac{4}{3}, \\ c_3 &= \tfrac{0 + 1 + 2 - 4}{3} = -\tfrac{1}{3}, \\ c_4 &= \tfrac{3 + 0 - 2 - 4}{3} = -1. \end{aligned}

Row reduction (Solving a system by row reduction: one, none or infinitely many solutions) gives the same answer, but here each weight is one dot product, found without the others. Check the second entry: \displaystyle \frac{8}{3} - \frac{4}{3} - \frac{1}{3} + 0 = 1.

Example 3

Day 1: the orthogonal complement of a plane in \mathbb{R}^4

Let W = \operatorname{span}\{(1, 1, 0, 1), (1, -1, 1, 0)\}. Find a basis of W^{\perp} and its dimension.

Solution

A vector \mathbf{x} = (x_1, x_2, x_3, x_4) is in W^{\perp} exactly when it is orthogonal to both spanning vectors, so W^{\perp} is the null space of

A = \begin{bmatrix} 1 & 1 & 0 & 1 \\ 1 & -1 & 1 & 0 \end{bmatrix},

whose reduced row echelon form is

\begin{bmatrix} 1 & 0 & \frac{1}{2} & \frac{1}{2} \\ 0 & 1 & -\frac{1}{2} & \frac{1}{2} \end{bmatrix}.

With x_3 and x_4 free, \displaystyle x_1 = -\frac{1}{2}x_3 - \frac{1}{2}x_4 and \displaystyle x_2 = \frac{1}{2}x_3 - \frac{1}{2}x_4. Taking x_3 = 2, x_4 = 0 and then x_3 = 0, x_4 = 2 gives the basis \{(-1, 1, 2, 0), (-1, -1, 0, 2)\}, so \dim W^{\perp} = 2 and \dim W + \dim W^{\perp} = 4.

This is the plane spanned by \mathbf{v}_3 and \mathbf{v}_4 of Example 2, which are orthogonal to \mathbf{v}_1 and \mathbf{v}_2; for instance \displaystyle \mathbf{v}_3 = \frac{1}{2}(-1, 1, 2, 0) - \frac{1}{2}(-1, -1, 0, 2).

Example 4

Day 2: the closest point on the face

Find the point of the face W = \operatorname{span}\{\mathbf{u}_1, \mathbf{u}_2\} closest to the drone at \mathbf{y} = (6, 3, 3), and the drone’s distance from the face.

Solution

The basis \{\mathbf{u}_1, \mathbf{u}_2\} is orthogonal (Example 1), so the projection formula applies:

\begin{aligned} \hat{\mathbf{y}} &= \frac{18}{9}\mathbf{u}_1 + \frac{9}{9}\mathbf{u}_2 \\ &= (2, 4, 4) + (2, 1, -2) = (4, 5, 2). \end{aligned}

The residual is \mathbf{z} = \mathbf{y} - \hat{\mathbf{y}} = (2, -2, 1). Check that it is in W^{\perp}: \mathbf{z} \cdot \mathbf{u}_1 = 2 - 4 + 2 = 0 and \mathbf{z} \cdot \mathbf{u}_2 = 4 - 2 - 2 = 0. So \mathbf{y} = (4, 5, 2) + (2, -2, 1) is the orthogonal decomposition of Theorem 3, and by Theorem 4 the closest point is (4, 5, 2) at distance \displaystyle \lVert \mathbf{z} \rVert = \sqrt{4 + 4 + 1} = 3, which is 30 meters.

Example 1 already held the answer: in \mathbf{y} = (2\mathbf{u}_1 + \mathbf{u}_2) + \mathbf{u}_3, the projection drops the \mathbf{u}_3 term.

Example 5

Day 2: best approximation in \mathbb{R}^4

With \mathbf{v}_1 and \mathbf{v}_2 as in Example 2, let W = \operatorname{span}\{\mathbf{v}_1, \mathbf{v}_2\} and \mathbf{b} = (2, 0, 3, 1). Find \operatorname{proj}_W \mathbf{b} and the distance from \mathbf{b} to W, and compare with the point \mathbf{v}_1 + \mathbf{v}_2 of W.

Solution

Since \mathbf{b} \cdot \mathbf{v}_1 = 2 + 0 + 0 + 1 = 3 and \mathbf{b} \cdot \mathbf{v}_2 = 2 - 0 + 3 + 0 = 5, with both denominators 3,

\hat{\mathbf{b}} = \mathbf{v}_1 + \tfrac{5}{3}\mathbf{v}_2 = \left(\tfrac{8}{3}, -\tfrac{2}{3}, \tfrac{5}{3}, 1\right).

The residual is \displaystyle \mathbf{z} = \left(-\frac{2}{3}, \frac{2}{3}, \frac{4}{3}, 0\right), and \displaystyle \mathbf{z} \cdot \mathbf{v}_1 = -\frac{2}{3} + \frac{2}{3} = 0, \displaystyle \mathbf{z} \cdot \mathbf{v}_2 = -\frac{2}{3} - \frac{2}{3} + \frac{4}{3} = 0. The distance is

\lVert \mathbf{z} \rVert = \sqrt{\tfrac{4 + 4 + 16}{9}} = \tfrac{2\sqrt{6}}{3} \approx 1.633.

The point \mathbf{v}_1 + \mathbf{v}_2 = (2, 0, 1, 1) is \lVert (0, 0, 2, 0) \rVert = 2 from \mathbf{b}, farther, as Theorem 4 promises. The Pythagorean split checks it exactly: \displaystyle \hat{\mathbf{b}} - (\mathbf{v}_1 + \mathbf{v}_2) = \frac{2}{3}\mathbf{v}_2 has squared length \displaystyle \frac{4}{3}, and \displaystyle \frac{24}{9} + \frac{4}{3} = 4.

Example 6

Day 2: the projection matrix of the face

Build an orthonormal basis of the face W = \operatorname{span}\{\mathbf{u}_1, \mathbf{u}_2\}, find the projection matrix P, and use it on the drone at (6, 3, 3).

Solution

Each \mathbf{u}_i has length 3, so \displaystyle \mathbf{q}_1 = \frac{1}{3}\mathbf{u}_1 and \displaystyle \mathbf{q}_2 = \frac{1}{3}\mathbf{u}_2 are orthonormal and span W. With these as the columns of U,

U = \frac{1}{3}\begin{bmatrix} 1 & 2 \\ 2 & 1 \\ 2 & -2 \end{bmatrix},

and U^{T}U = I_2 because \mathbf{q}_1 \cdot \mathbf{q}_2 = 0 and each \mathbf{q}_i \cdot \mathbf{q}_i = 1. The (i, j) entry of UU^{T} is the dot product of rows i and j of the bracketed matrix, divided by 9:

P = UU^{T} = \frac{1}{9}\begin{bmatrix} 5 & 4 & -2 \\ 4 & 5 & 2 \\ -2 & 2 & 8 \end{bmatrix}.

Then \displaystyle P(6, 3, 3) = \frac{1}{9}(36, 45, 18) = (4, 5, 2), the point found in Example 4. P is symmetric (The transpose and symmetric matrices), as Theorem 5 says. A second route gives the same matrix: \displaystyle \mathbf{q}_3 = \frac{1}{3}\mathbf{u}_3 spans W^{\perp}, and since \mathbf{y} = \hat{\mathbf{y}} + (\mathbf{y} \cdot \mathbf{q}_3)\mathbf{q}_3, the projection is P = I - \mathbf{q}_3\mathbf{q}_3^{T}, and \displaystyle I - \frac{1}{9}\mathbf{u}_3\mathbf{u}_3^{T} is the same P. With it, Kaija projects each logged drone position with one matrix product.

Example 7

Day 2: why the basis must be orthogonal

The vectors \mathbf{u}_1 = (1, 2, 2) and \mathbf{w} = (3, 3, 0) also span the face, since \mathbf{w} = \mathbf{u}_1 + \mathbf{u}_2. Add the separate projections of \mathbf{y} = (6, 3, 3) onto \mathbf{u}_1 and onto \mathbf{w}, and show the result is not the closest point. Then repair the basis.

Solution

The two separate projections are \displaystyle \frac{18}{9}\mathbf{u}_1 = (2, 4, 4) and \displaystyle \frac{27}{18}\mathbf{w} = \left(\frac{9}{2}, \frac{9}{2}, 0\right), with sum \displaystyle \left(\frac{13}{2}, \frac{17}{2}, 4\right). That point is in W, but \mathbf{y} minus it is \displaystyle \left(-\frac{1}{2}, -\frac{11}{2}, -1\right), whose dot product with \mathbf{u}_1 is \displaystyle -\frac{1}{2} - 11 - 2 = -\frac{27}{2} \neq 0. The residual is not in W^{\perp}, and the distance \displaystyle \sqrt{\frac{1 + 121 + 4}{4}} = \sqrt{31.5} \approx 5.61 is more than the true distance 3. The cause is that \mathbf{u}_1 \cdot \mathbf{w} = 9 \neq 0: both projections count the part of \mathbf{y} that lies along the overlap of the two directions.

To repair the basis, remove from \mathbf{w} its projection onto \mathbf{u}_1: \displaystyle \mathbf{w} - \frac{9}{9}\mathbf{u}_1 = (2, 1, -2) = \mathbf{u}_2. The new pair is orthogonal, spans the same plane, and gives (4, 5, 2) as in Example 4. Repeating that subtraction vector by vector is the The Gram-Schmidt process.

Boundaries

The projection formula needs an orthogonal basis of W. Otherwise the sum of the separate projections is in W but in general is not \operatorname{proj}_W \mathbf{y} (Example 7).
Theorem 1 needs every vector to be nonzero: an orthogonal set that contains \mathbf{0} is dependent. The converse fails too: an independent set need not be orthogonal.
UU^{T}\mathbf{y} is the projection only when the columns of U are orthonormal. Orthogonal columns that are not unit vectors need the denominators of the general formula. And U^{T}U = I_k does not make UU^{T} = I_n unless k = n, that is, unless W is all of \mathbb{R}^n.
W must be a subspace, so it passes through the origin. For a plane that does not, shift a point of it to the origin, project, and shift back.

Where this goes next

The The Gram-Schmidt process builds an orthogonal basis of any subspace, so \operatorname{proj}_W can be computed for every W. When A\mathbf{x} = \mathbf{b} has no solution, the best choice makes A\mathbf{x} the point of the column space closest to \mathbf{b}, its projection by Theorem 4; that is Least squares: the best approximate solution of Ax = b.

Exercises

Day 1: orthogonal sets and weights

In Exercises 2 and 3, \mathbf{a}_1 = (1, -1, 2), \mathbf{a}_2 = (3, 1, -1) and \mathbf{a}_3 = (-1, 7, 4). In Exercises 4 and 5, \mathbf{b}_1 = (1, 0, 1, 1), \mathbf{b}_2 = (0, 1, -1, 1) and \mathbf{b}_3 = (1, -1, -1, 0).

Is \{(1, 3, -1), (2, 0, 2), (3, -1, -3)\} an orthogonal set? Explain.
Show that \{\mathbf{a}_1, \mathbf{a}_2, \mathbf{a}_3\} is an orthogonal basis of \mathbb{R}^3, and write \mathbf{x} = (0, -4, 7) in this basis.
Find only the \mathbf{a}_3 weight of (2, 0, 5) in the basis of Exercise 2.
Check that \{\mathbf{b}_1, \mathbf{b}_2, \mathbf{b}_3\} is orthogonal, and find a vector \mathbf{b}_4 that extends it to an orthogonal basis of \mathbb{R}^4.
Using the basis of Exercise 4, find the coordinates of (2, 3, 1, 0).
Find W^{\perp} for W = \operatorname{span}\{(1, 2, -1)\} in \mathbb{R}^3, and give an orthogonal basis of it.
Find a basis of W^{\perp} for W = \operatorname{span}\{(1, 0, 2, 1), (0, 1, 1, -1)\} in \mathbb{R}^4, and its dimension.
Kaija records the direction of a fixed rope as \mathbf{c} = (3, 9, 0) in the survey frame. Write \mathbf{c} in the basis \mathbf{u}_1, \mathbf{u}_2, \mathbf{u}_3 of Example 1. Is the rope parallel to the face?

Day 1: reasoning

Prove that if \mathbf{z} is orthogonal to each of \mathbf{w}_1, \dots, \mathbf{w}_k, then \mathbf{z} is in W^{\perp} for W = \operatorname{span}\{\mathbf{w}_1, \dots, \mathbf{w}_k\}.
Prove that W^{\perp} is a subspace of \mathbb{R}^n.
Prove that W \cap W^{\perp} = \{\mathbf{0}\}.
Give an orthogonal set in \mathbb{R}^3 that is linearly dependent, and say which hypothesis of Theorem 1 it breaks. Then give a linearly independent set that is not orthogonal.

Day 2: projections and distances

Project \mathbf{y} = (-2, 4, 9) onto W = \operatorname{span}\{\mathbf{a}_1, \mathbf{a}_2\}, write \mathbf{y} = \hat{\mathbf{y}} + \mathbf{z}, and find the distance from \mathbf{y} to W.
Later the drone hovers at (0, 1, 8). Find the closest point on the face W = \operatorname{span}\{\mathbf{u}_1, \mathbf{u}_2\} and the distance in meters.
Project (1, 2, 3, 4) onto \operatorname{span}\{\mathbf{b}_1, \mathbf{b}_2\} and find its distance from that plane.
Use Exercises 4 and 5 to project (2, 3, 1, 0) onto \operatorname{span}\{\mathbf{b}_1, \mathbf{b}_2, \mathbf{b}_3\} with one subtraction.
Normalize \mathbf{a}_1 and \mathbf{a}_2, form U, and find the projection matrix P = UU^{T} onto \operatorname{span}\{\mathbf{a}_1, \mathbf{a}_2\}. Check it on the \mathbf{y} of Exercise 13.
Find the projection matrix onto the line spanned by (1, 1, 1, 1) in \mathbb{R}^4, and project (3, 1, 2, 6). What do the entries of the result have to do with the entries of the vector?
Let W = \operatorname{span}\{(1, 1, 0), (1, 0, 1)\} and \mathbf{y} = (0, 0, 1). Show that adding the separate projections onto the two spanning vectors does not give a residual in W^{\perp}. Then find \operatorname{proj}_W \mathbf{y} from an orthogonal basis of W.
For the drone of Exercise 14, Tuula suggests the point \mathbf{u}_1 = (1, 2, 2) of the face instead. Find its distance from the drone and confirm the Pythagorean split in the proof of Theorem 4.

Day 2: reasoning

Show that \operatorname{proj}_W \mathbf{y} = \mathbf{y} exactly when \mathbf{y} is in W, and \operatorname{proj}_W \mathbf{y} = \mathbf{0} exactly when \mathbf{y} is in W^{\perp}.
Using U^{T}U = I_k, prove that P = UU^{T} satisfies P^{2} = P and P^{T} = P. Explain P^{2} = P in words.
With an orthonormal basis \{\mathbf{q}_1, \dots, \mathbf{q}_k\} of W, show that \lVert \operatorname{proj}_W \mathbf{y} \rVert^2 = (\mathbf{y} \cdot \mathbf{q}_1)^2 + \dots + (\mathbf{y} \cdot \mathbf{q}_k)^2 \leq \lVert \mathbf{y} \rVert^2.
Show that if W has an orthogonal basis and W^{\perp} does too, then \operatorname{proj}_{W^{\perp}} \mathbf{y} = \mathbf{y} - \operatorname{proj}_W \mathbf{y}.

Answers

No. (1, 3, -1) \cdot (2, 0, 2) = 0 and (2, 0, 2) \cdot (3, -1, -3) = 0, but (1, 3, -1) \cdot (3, -1, -3) = 3 - 3 + 3 = 3.
The pairwise dot products are 3 - 1 - 2 = 0, -1 - 7 + 8 = 0 and -3 + 7 - 4 = 0, so Theorem 1 gives a basis. With \mathbf{a}_i \cdot \mathbf{a}_i = 6, 11, 66: \mathbf{x} = 3\mathbf{a}_1 - \mathbf{a}_2 + 0\mathbf{a}_3, from \displaystyle \frac{18}{6}, \displaystyle \frac{-11}{11} and \displaystyle \frac{0}{66}.
\displaystyle \frac{3}{11}, from \displaystyle \frac{-2 + 0 + 20}{66} = \frac{18}{66}.
\mathbf{b}_1 \cdot \mathbf{b}_2 = 0 - 1 + 1 = 0, \mathbf{b}_1 \cdot \mathbf{b}_3 = 1 - 1 + 0 = 0, \mathbf{b}_2 \cdot \mathbf{b}_3 = -1 + 1 + 0 = 0. Solving the three equations \mathbf{b}_i \cdot \mathbf{x} = 0 gives the multiples of \mathbf{b}_4 = (1, 1, 0, -1); any nonzero one works.
\displaystyle (2, 3, 1, 0) = \mathbf{b}_1 + \frac{2}{3}\mathbf{b}_2 - \frac{2}{3}\mathbf{b}_3 + \frac{5}{3}\mathbf{b}_4. Each \mathbf{b}_i \cdot \mathbf{b}_i = 3, and the dot products are 3, 2, -2, 5.
W^{\perp} is the plane x_1 + 2x_2 - x_3 = 0. One orthogonal basis is \{(1, 0, 1), (1, -1, -1)\}: both satisfy the equation and their dot product is 1 + 0 - 1 = 0.
Basis \{(-2, -1, 1, 0), (-1, 1, 0, 1)\}, dimension 2. The matrix with rows (1, 0, 2, 1) and (0, 1, 1, -1) is already reduced, so x_1 = -2x_3 - x_4 and x_2 = -x_3 + x_4.
\displaystyle \mathbf{c} = \frac{7}{3}\mathbf{u}_1 + \frac{5}{3}\mathbf{u}_2 - \frac{4}{3}\mathbf{u}_3, from the dot products 21, 15, -12 divided by 9. No: a vector parallel to the face lies in W, and the \mathbf{u}_3 weight is not 0.
Every \mathbf{w} \in W is c_1\mathbf{w}_1 + \dots + c_k\mathbf{w}_k, so \mathbf{z} \cdot \mathbf{w} = c_1(\mathbf{z} \cdot \mathbf{w}_1) + \dots + c_k(\mathbf{z} \cdot \mathbf{w}_k) = 0 by the distributive property and the rule for scalars.
\mathbf{0} \cdot \mathbf{w} = 0, so \mathbf{0} \in W^{\perp}. If \mathbf{z}_1, \mathbf{z}_2 \in W^{\perp} and c is a scalar, then (\mathbf{z}_1 + c\mathbf{z}_2) \cdot \mathbf{w} = \mathbf{z}_1 \cdot \mathbf{w} + c(\mathbf{z}_2 \cdot \mathbf{w}) = 0 for every \mathbf{w} \in W.
If \mathbf{x} is in both, then \mathbf{x} is orthogonal to every vector of W, including itself, so \mathbf{x} \cdot \mathbf{x} = 0 and \mathbf{x} = \mathbf{0}. And \mathbf{0} is in both, since both are subspaces.
\{(1, 0, 0), (0, 1, 0), \mathbf{0}\} is orthogonal and dependent, since 0(1, 0, 0) + 0(0, 1, 0) + 1\mathbf{0} = \mathbf{0}; it breaks “nonzero”. The set \{(1, 0, 0), (1, 1, 0)\} is independent but not orthogonal, since the dot product is 1.
\hat{\mathbf{y}} = 2\mathbf{a}_1 - \mathbf{a}_2 = (-1, -3, 5), from \displaystyle \frac{12}{6} and \displaystyle \frac{-11}{11}. Then \mathbf{z} = (-1, 7, 4), which is \mathbf{a}_3 and so lies in W^{\perp}, and the distance is \displaystyle \sqrt{66} \approx 8.12.
The closest point is \displaystyle 2\mathbf{u}_1 - \frac{5}{3}\mathbf{u}_2 = \left(-\frac{4}{3}, \frac{7}{3}, \frac{22}{3}\right), from dot products 18 and -15. The residual is \displaystyle \frac{2}{3}\mathbf{u}_3, of length \displaystyle \frac{2}{3} \cdot 3 = 2, so the drone is 20 meters from the face.
\displaystyle \frac{8}{3}\mathbf{b}_1 + \mathbf{b}_2 = \left(\frac{8}{3}, 1, \frac{5}{3}, \frac{11}{3}\right), from dot products 8 and 3. The residual \displaystyle \left(-\frac{5}{3}, 1, \frac{4}{3}, \frac{1}{3}\right) is orthogonal to \mathbf{b}_1 and \mathbf{b}_2, and the distance is \displaystyle \frac{\sqrt{51}}{3} \approx 2.38.
\displaystyle \left(\frac{1}{3}, \frac{4}{3}, 1, \frac{5}{3}\right). The span of \mathbf{b}_1, \mathbf{b}_2, \mathbf{b}_3 drops only the \mathbf{b}_4 term of Exercise 5, so the projection is \displaystyle (2, 3, 1, 0) - \frac{5}{3}(1, 1, 0, -1).
With columns \displaystyle \frac{1}{\sqrt{6}}\mathbf{a}_1 and \displaystyle \frac{1}{\sqrt{11}}\mathbf{a}_2, \displaystyle P = \frac{1}{6}\mathbf{a}_1\mathbf{a}_1^{T} + \frac{1}{11}\mathbf{a}_2\mathbf{a}_2^{T} is \displaystyle \frac{1}{66} times the matrix with rows (65, 7, 4), (7, 17, -28), (4, -28, 50). Then \displaystyle P(-2, 4, 9) = \frac{1}{66}(-66, -198, 330), which is (-1, -3, 5) as in Exercise 13.
\displaystyle \mathbf{q} = \frac{1}{2}(1, 1, 1, 1) and P = \mathbf{q}\mathbf{q}^{T} is the 4 \times 4 matrix with every entry \displaystyle \frac{1}{4}. The projection is (3, 3, 3, 3): every entry is the mean of the entries 3, 1, 2, 6.
The separate projections are \mathbf{0} and \displaystyle \frac{1}{2}(1, 0, 1); the residual \displaystyle \left(-\frac{1}{2}, 0, \frac{1}{2}\right) has dot product \displaystyle -\frac{1}{2} with (1, 1, 0). Replacing (1, 0, 1) by \displaystyle (1, 0, 1) - \frac{1}{2}(1, 1, 0) = \frac{1}{2}(1, -1, 2) gives the orthogonal basis \{(1, 1, 0), (1, -1, 2)\}, and \displaystyle \operatorname{proj}_W \mathbf{y} = 0 + \frac{2}{6}(1, -1, 2) = \left(\frac{1}{3}, -\frac{1}{3}, \frac{2}{3}\right).
\displaystyle \sqrt{38}, about 61.6 meters, from (0, 1, 8) - (1, 2, 2) = (-1, -1, 6). The split: \displaystyle \left(-\frac{4}{3}, \frac{7}{3}, \frac{22}{3}\right) - (1, 2, 2) = \left(-\frac{7}{3}, \frac{1}{3}, \frac{16}{3}\right) has squared length \displaystyle \frac{306}{9} = 34, and 4 + 34 = 38.
If \mathbf{y} \in W, then \mathbf{y} = \mathbf{y} + \mathbf{0} is a decomposition with pieces in W and W^{\perp}, so by uniqueness \hat{\mathbf{y}} = \mathbf{y}; conversely \hat{\mathbf{y}} is always in W. If \mathbf{y} \in W^{\perp}, every \mathbf{y} \cdot \mathbf{u}_i = 0, so every weight is 0; conversely \hat{\mathbf{y}} = \mathbf{0} makes \mathbf{y} = \mathbf{z} \in W^{\perp}.
P^{2} = U(U^{T}U)U^{T} = UI_kU^{T} = P, by the associative property of matrix products, and P^{T} = (U^{T})^{T}U^{T} = UU^{T} = P. In words: a vector already in W is its own projection, so projecting twice changes nothing.
The vectors (\mathbf{y} \cdot \mathbf{q}_i)\mathbf{q}_i are orthogonal, so the Pythagorean theorem, applied repeatedly, adds their squared lengths (\mathbf{y} \cdot \mathbf{q}_i)^2. And \lVert \mathbf{y} \rVert^2 = \lVert \hat{\mathbf{y}} \rVert^2 + \lVert \mathbf{z} \rVert^2 \geq \lVert \hat{\mathbf{y}} \rVert^2, since \hat{\mathbf{y}} \perp \mathbf{z}.
In \mathbf{y} = \hat{\mathbf{y}} + \mathbf{z}, the vector \mathbf{z} is in W^{\perp} and \hat{\mathbf{y}} is in W, which lies inside (W^{\perp})^{\perp} because each vector of W is orthogonal to each vector of W^{\perp}. So \mathbf{y} = \mathbf{z} + \hat{\mathbf{y}} is a decomposition for the subspace W^{\perp}, and by the uniqueness in Theorem 3, \operatorname{proj}_{W^{\perp}} \mathbf{y} = \mathbf{z} = \mathbf{y} - \operatorname{proj}_W \mathbf{y}.