Orthogonality and least squares

What this area is about

Perpendicular vectors, those with dot product 0, make distances easy. In \mathbb{R}^n, the point of a subspace W closest to a vector \mathbf{y} is the one whose difference from \mathbf{y} is perpendicular to W. That one fact turns an inconsistent system A\mathbf{x} = \mathbf{b}, with no exact solution, into a best approximate one.

How these fit together

Result. Orthogonal bases and projection onto a subspace gives the closest point: with an orthogonal basis of W, it is a sum of projections onto the basis vectors, one at a time.

Procedure. The Gram-Schmidt process turns any basis of a subspace into an orthogonal one, so that projection formula always applies.

Application. Least squares takes the \hat{\mathbf{x}} that brings A\hat{\mathbf{x}} as close to \mathbf{b} as possible, found from the normal equations A^T A\hat{\mathbf{x}} = A^T\mathbf{b}; when the columns of A are linearly independent, \hat{\mathbf{x}} is unique. Fitting a trend line to data is the standard use.

A path through it

Projection first, since both other lessons use it. Then Gram-Schmidt, which least squares relies on to know an orthogonal basis of the column space exists. Least squares comes last. The area assumes dot products and subspaces of \mathbb{R}^n, and the transpose from matrices.

Later, inner product spaces take length and angle to any vector space.