Math Core

Given a subspace $W$ and a vector $\mathbf{y}$ that is not in it, which vector in $W$ is closest to $\mathbf{y}$ ? The answer is the orthogonal projection of $\mathbf{y}$ onto $W$ : drop a perpendicular from $\mathbf{y}$ to $W$ . This one idea powers least-squares data fitting, the Gram-Schmidt process, and much of approximation theory.

Projecting onto a line

Start with a line $L = \operatorname{Span}\{\mathbf{u}\}$ , where $\mathbf{u} \neq \mathbf{0}$ . You want to split $\mathbf{y}$ into two pieces,

\mathbf{y} = \hat{\mathbf{y}} + \mathbf{z},

with $\hat{\mathbf{y}} = \alpha\mathbf{u}$ on the line and $\mathbf{z}$ orthogonal to $\mathbf{u}$ . The condition $\mathbf{z} \cdot \mathbf{u} = 0$ reads $(\mathbf{y} - \alpha\mathbf{u}) \cdot \mathbf{u} = 0$ , so $\mathbf{y} \cdot \mathbf{u} = \alpha(\mathbf{u} \cdot \mathbf{u})$ , which determines $\alpha$ .

Projection onto a line

The orthogonal projection of $\mathbf{y}$ onto $L = \operatorname{Span}\{\mathbf{u}\}$ is

\hat{\mathbf{y}} = \operatorname{proj}_L \mathbf{y} = \frac{\mathbf{y} \cdot \mathbf{u}}{\mathbf{u} \cdot \mathbf{u}}\,\mathbf{u}.

The component of $\mathbf{y}$ orthogonal to $\mathbf{u}$ is $\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}}$ .

The projection depends only on the line, not on which nonzero $\mathbf{u}$ you pick: replacing $\mathbf{u}$ by $c\mathbf{u}$ multiplies the fraction by $\tfrac{1}{c}$ and the vector by $c$ . Notice the fraction is exactly the weight formula from the previous lesson.

Worked example: Projection in the plane

Let $\mathbf{y} = (7, 6)$ and $\mathbf{u} = (4, 2)$ . Find the projection of $\mathbf{y}$ onto $\operatorname{Span}\{\mathbf{u}\}$ , write $\mathbf{y}$ as a sum of a vector on the line and a vector orthogonal to it, and find the distance from $\mathbf{y}$ to the line.

$\mathbf{y} \cdot \mathbf{u} = 28 + 12 = 40$ and $\mathbf{u} \cdot \mathbf{u} = 16 + 4 = 20$ , so

\hat{\mathbf{y}} = \frac{40}{20}(4, 2) = (8, 4), \qquad \mathbf{z} = \mathbf{y} - \hat{\mathbf{y}} = (-1, 2).

Check: $\mathbf{z} \cdot \mathbf{u} = -4 + 4 = 0$ . So $(7, 6) = (8, 4) + (-1, 2)$ .

The closest point on the line to $\mathbf{y}$ is $\hat{\mathbf{y}}$ , so the distance from $\mathbf{y}$ to the line is $\|\mathbf{z}\| = \sqrt{1 + 4} = \sqrt{5}$ .

ŷ = (8, 4) is the foot of the perpendicular from y = (7, 6) to the line through u. The dashed segment z = (−1, 2) is orthogonal to the line.Open in grapher →

Projecting onto a subspace

The same splitting works for any subspace $W$ of $\mathbb{R}^n$ , as long as you have an orthogonal basis for it.

The orthogonal decomposition theorem

Let $W$ be a subspace of $\mathbb{R}^n$ . Every $\mathbf{y}$ in $\mathbb{R}^n$ can be written uniquely as

\mathbf{y} = \hat{\mathbf{y}} + \mathbf{z}, \qquad \hat{\mathbf{y}} \text{ in } W, \quad \mathbf{z} \text{ in } W^\perp.

If $\{\mathbf{u}_1, \ldots, \mathbf{u}_p\}$ is any orthogonal basis of $W$ , then

\hat{\mathbf{y}} = \operatorname{proj}_W \mathbf{y} = \frac{\mathbf{y} \cdot \mathbf{u}_1}{\mathbf{u}_1 \cdot \mathbf{u}_1}\mathbf{u}_1 + \cdots + \frac{\mathbf{y} \cdot \mathbf{u}_p}{\mathbf{u}_p \cdot \mathbf{u}_p}\mathbf{u}_p.

In words: $\operatorname{proj}_W \mathbf{y}$ is the sum of the projections of $\mathbf{y}$ onto the separate basis lines. To see that $\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}}$ is orthogonal to $W$ , dot it with $\mathbf{u}_1$ . Orthogonality kills the cross terms, leaving $\mathbf{y} \cdot \mathbf{u}_1 - \dfrac{\mathbf{y} \cdot \mathbf{u}_1}{\mathbf{u}_1 \cdot \mathbf{u}_1}(\mathbf{u}_1 \cdot \mathbf{u}_1) = 0$ , and likewise for each $\mathbf{u}_j$ .

Two quick consequences: if $\mathbf{y}$ is already in $W$ , then $\operatorname{proj}_W \mathbf{y} = \mathbf{y}$ (this is the weight formula from the last lesson), and if $\mathbf{y}$ is in $W^\perp$ , then $\operatorname{proj}_W \mathbf{y} = \mathbf{0}$ .

Worked example: Projection onto a plane in R³

Let $\mathbf{u}_1 = (1, 1, 0)$ , $\mathbf{u}_2 = (1, -1, 2)$ and $\mathbf{y} = (3, 1, 5)$ . Write $\mathbf{y}$ as a vector in $W = \operatorname{Span}\{\mathbf{u}_1, \mathbf{u}_2\}$ plus a vector orthogonal to $W$ .

First check the basis is orthogonal: $\mathbf{u}_1 \cdot \mathbf{u}_2 = 1 - 1 + 0 = 0$ . Then

\hat{\mathbf{y}} = \frac{3 + 1}{2}\mathbf{u}_1 + \frac{3 - 1 + 10}{6}\mathbf{u}_2 = 2(1, 1, 0) + 2(1, -1, 2) = (4, 0, 4).

So $\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}} = (-1, 1, 1)$ . Check: $\mathbf{z} \cdot \mathbf{u}_1 = -1 + 1 = 0$ and $\mathbf{z} \cdot \mathbf{u}_2 = -1 - 1 + 2 = 0$ . The decomposition is

(3, 1, 5) = (4, 0, 4) + (-1, 1, 1).

Common mistake

The projection formula requires an orthogonal basis. If $\mathbf{u}_1 \cdot \mathbf{u}_2 \neq 0$ , adding up the one-line projections gives the wrong answer, and $\mathbf{y} - \hat{\mathbf{y}}$ won't be orthogonal to $W$ . Always check orthogonality first; if the basis isn't orthogonal, make it so with Gram-Schmidt (next lesson).

The best approximation theorem

Best approximation

Let $W$ be a subspace of $\mathbb{R}^n$ and $\hat{\mathbf{y}} = \operatorname{proj}_W \mathbf{y}$ . Then $\hat{\mathbf{y}}$ is the point of $W$ closest to $\mathbf{y}$ :

\|\mathbf{y} - \hat{\mathbf{y}}\| < \|\mathbf{y} - \mathbf{v}\| \quad \text{for every } \mathbf{v} \text{ in } W \text{ with } \mathbf{v} \neq \hat{\mathbf{y}}.

The proof is the Pythagorean theorem. For $\mathbf{v}$ in $W$ , write $\mathbf{y} - \mathbf{v} = (\mathbf{y} - \hat{\mathbf{y}}) + (\hat{\mathbf{y}} - \mathbf{v})$ . The first piece is in $W^\perp$ and the second is in $W$ , so they are orthogonal and

\|\mathbf{y} - \mathbf{v}\|^2 = \|\mathbf{y} - \hat{\mathbf{y}}\|^2 + \|\hat{\mathbf{y}} - \mathbf{v}\|^2,

which is larger than $\|\mathbf{y} - \hat{\mathbf{y}}\|^2$ unless $\mathbf{v} = \hat{\mathbf{y}}$ .

So the distance from $\mathbf{y}$ to $W$ is $\|\mathbf{y} - \hat{\mathbf{y}}\| = \|\mathbf{z}\|$ . In the plane example above, the distance from $(3, 1, 5)$ to $W$ is $\|(-1, 1, 1)\| = \sqrt{3}$ .

Projection matrices

With an orthonormal basis $\{\mathbf{u}_1, \ldots, \mathbf{u}_p\}$ the denominators are all $1$ , so $\operatorname{proj}_W \mathbf{y} = (\mathbf{y} \cdot \mathbf{u}_1)\mathbf{u}_1 + \cdots + (\mathbf{y} \cdot \mathbf{u}_p)\mathbf{u}_p$ . With $U = [\mathbf{u}_1 \ \cdots \ \mathbf{u}_p]$ , the vector of weights is $U^T\mathbf{y}$ , and combining the columns with those weights gives:

Projection with orthonormal columns

If the columns of $U$ are an orthonormal basis of $W$ , then $\operatorname{proj}_W \mathbf{y} = UU^T\mathbf{y}$ for every $\mathbf{y}$ .

Worked example: A projection matrix

Let $W$ be the line spanned by $\mathbf{u} = \left(\tfrac{3}{5}, \tfrac{4}{5}\right)$ , a unit vector. Find the matrix $P$ of $\operatorname{proj}_W$ and use it to project $\mathbf{y} = (5, 0)$ .

P = \mathbf{u}\mathbf{u}^T = \begin{bmatrix} 3/5 \\ 4/5 \end{bmatrix}\begin{bmatrix} 3/5 & 4/5 \end{bmatrix} = \frac{1}{25}\begin{bmatrix} 9 & 12 \\ 12 & 16 \end{bmatrix}, \qquad P\mathbf{y} = \frac{1}{25}(45, 60) = \left(\tfrac{9}{5}, \tfrac{12}{5}\right).

Check with the line formula: $\mathbf{y} \cdot \mathbf{u} = 3$ , and $3\mathbf{u} = \left(\tfrac{9}{5}, \tfrac{12}{5}\right)$ .

Tip

Every projection matrix satisfies $P^2 = P$ (projecting twice changes nothing) and $P^T = P$ . These are fast checks on a computed $P$ .

Practice

Practice 1

Find the orthogonal projection of $\mathbf{y} = (1, 7)$ onto the line spanned by $\mathbf{u} = (-4, 2)$ .

Enter a point like (2, -3)

Practice 2

Find the distance from $\mathbf{y} = (1, 7)$ to the line spanned by $\mathbf{u} = (-4, 2)$ . Give an exact answer.

Enter a number. Fractions like 3/4 and sqrt(2) are OK.

Practice 3

Let $\mathbf{u} = (1, 0, 1)$ and $\mathbf{y} = (2, -1, 5)$ . Find the component $\mathbf{z}$ of $\mathbf{y}$ orthogonal to $\mathbf{u}$ .

Enter a point like (2, -3)

Practice 4

Let $\mathbf{u}_1 = (1, 2, 2)$ , $\mathbf{u}_2 = (2, 1, -2)$ , $W = \operatorname{Span}\{\mathbf{u}_1, \mathbf{u}_2\}$ and $\mathbf{y} = (7, 2, -1)$ . Find the point of $W$ closest to $\mathbf{y}$ .

Enter a point like (2, -3)

Practice 5

With $W$ and $\mathbf{y}$ as in the previous problem, find the distance from $\mathbf{y}$ to $W$ .

Enter a number. Fractions like 3/4 and sqrt(2) are OK.

Practice 6

In $\mathbb{R}^4$ , let $\mathbf{u}_1 = (1, 1, 1, 1)$ , $\mathbf{u}_2 = (1, -1, 1, -1)$ and $\mathbf{y} = (3, 1, -1, 5)$ . Find the distance from $\mathbf{y}$ to $W = \operatorname{Span}\{\mathbf{u}_1, \mathbf{u}_2\}$ .

Enter a number. Fractions like 3/4 and sqrt(2) are OK.

Practice 7

The columns of the $4 \times 2$ matrix $U$ form an orthonormal basis for a subspace $W$ of $\mathbb{R}^4$ . Which matrix $P$ satisfies $P\mathbf{y} = \operatorname{proj}_W \mathbf{y}$ for every $\mathbf{y}$ in $\mathbb{R}^4$ ?