Projection Maps

Section 3.5 Projection Maps

There is a class of linear transformations that are very important and have a nice geometric interpretation called projection maps. Let’s look at an example in the \(xy\)-plane. Consider the point \((1,4)\) as shown below.

🔗

Figure 3.5.1. Projecting a point onto the \(x\)-axis.
🔗

If the point (or the vector) is projected onto the \(x\)-axis, then the result is the point \((1,0)\) found by taking the \(x\)-component of the point. In term of the vector, this is \([1\;\;0]^T\text{.}\)

🔗

We now look at projecting a vector \(\boldsymbol{u}\) onto a line (or a vector \(\boldsymbol{v}\)). We will derive this in the \(xy\)-plane, however the result will work for any vector space.

🔗

Figure 3.5.2. Projecting a vector onto another vector.
🔗

Let \(\boldsymbol{u}\) and \(\boldsymbol{v}\) be two vectors in \(\mathbb{R}^2\) as shown above. We seek the projection of \(\boldsymbol{u}\) onto \(\boldsymbol{v}\) and denote this as \(\text{Proj}_{\boldsymbol{v}} \boldsymbol{u}\text{.}\) The projection is a vector that is in the same direction as \(\boldsymbol{v}\text{.}\) The length will be explained below.

🔗

Any vector \(\boldsymbol{u}\) can be split into two parts, \(\boldsymbol{u}_{||}\) a vector in a direction parallel to \(\boldsymbol{v}\) and a vector \(\boldsymbol{u}_{\perp}\) which satisfies \(\langle \boldsymbol{u}_{\perp} , \boldsymbol{v} \rangle = 0\text{.}\)

🔗

\begin{equation} \boldsymbol{u} = \boldsymbol{u}_{||} + \boldsymbol{v}_{\perp}\tag{3.5.1} \end{equation}

🔗

where \(\boldsymbol{u}_{||} = \alpha \boldsymbol{v}\text{.}\)

🔗

Take the inner product of both sides of ((3.5.1)) with the vector \(\boldsymbol{v}\)

🔗

\begin{align*} \langle \boldsymbol{u}, \boldsymbol{v} \rangle \amp = \langle \boldsymbol{u}_{||} + \boldsymbol{u}_{\perp}, \boldsymbol{v} \rangle \\ \amp = \langle \alpha \boldsymbol{v}, \boldsymbol{v} \rangle + \langle \boldsymbol{u}_{\perp}, \boldsymbol{v} \rangle \\ \amp = \alpha \langle \boldsymbol{v},\boldsymbol{v} \rangle + 0 \end{align*}

🔗

and solving for \(\alpha\text{,}\)

🔗

\begin{equation*} \alpha = \frac{ \langle \boldsymbol{u}, \boldsymbol{v} \rangle}{\langle \boldsymbol{v}, \boldsymbol{v} \rangle} \end{equation*}

🔗

Therefore the projection vector

🔗

\begin{equation} \text{Proj}_{\boldsymbol{v}} \boldsymbol{u} = \frac{ \langle \boldsymbol{u}, \boldsymbol{v} \rangle}{\langle \boldsymbol{v}, \boldsymbol{v} \rangle} \boldsymbol{v}\tag{3.5.2} \end{equation}

🔗

and if needed the vector \(\boldsymbol{u}_{\perp}\) is

🔗

\begin{equation*} \boldsymbol{u}_{\perp} = \boldsymbol{u} - \text{Proj}_{\boldsymbol{v}} \boldsymbol{u} = \boldsymbol{u} - \frac{ \langle \boldsymbol{u}, \boldsymbol{v} \rangle}{\langle \boldsymbol{v}, \boldsymbol{v} \rangle} \boldsymbol{v} \end{equation*}

🔗

Although we derived this projection in \(\mathbb{R}^2\text{,}\) there is nothing about the projection formula to indicate it’s confined to \(\mathbb{R}^2\text{.}\) In fact, as we will see, the projection mapping is a linear transformation and applies to vectors in an inner product space.

🔗

Remark 3.5.3.

The projection of any vector \(\boldsymbol{u} \in \mathbb{R}^n\) onto a vector \(\boldsymbol{v} \in \mathbb{R}^n\) is given by

\begin{equation*} \text{Proj}_{\boldsymbol{v}} \boldsymbol{u} = \frac{\langle \boldsymbol{u},\boldsymbol{v} \rangle}{\langle \boldsymbol{v}, \boldsymbol{v} \rangle} \boldsymbol{v} \end{equation*}

and the vector perpendicular to \(\text{Proj}_{\boldsymbol{v}}\boldsymbol{u}\) that satisfies \(\boldsymbol{u} = \text{Proj}_{\boldsymbol{v}}\boldsymbol{u} + \boldsymbol{u}_{\perp}\) is given by

🔗

The example below finds the projection of vectors in \(\mathbb{R}^3\text{.}\)

🔗

Example 3.5.4.

Find the projection of \(\boldsymbol{u} = [1\;\;2\;\;3]^T\) onto the vector \(\boldsymbol{v}=[-2\;\; 0\;\; 4]\text{.}\)

🔗

Solution.

Since

\begin{align*} \langle \boldsymbol{u}, \boldsymbol{v} \rangle \amp = -2+0+12=10 \amp \langle \boldsymbol{v},\boldsymbol{v} \rangle \amp = (-2)^2 + 0^2 + 4^2 =20. \end{align*}

\begin{equation*} \text{Proj}_{\boldsymbol{v}} \boldsymbol{u} = \frac{10}{20} \boldsymbol{v} = \frac{1}{2} \begin{bmatrix} -2 \\ 0 \\ 4 \end{bmatrix} = \begin{bmatrix} -1 \\ 0 \\ 2 \end{bmatrix} \end{equation*}

🔗

Theorem 3.5.5.

Given a nonzero vector \(\boldsymbol{v}\) and a vector \(\boldsymbol{u}\) both in \(\mathbb{R}^n\text{,}\) the projection map in ((3.5.2)) is a linear transformation.

🔗

Proof.

We need to prove the two properties of linear transformations in Definition 3.4.1.

🔗

\begin{align*} \text{Proj}_{\boldsymbol{v}}(\boldsymbol{u} + \boldsymbol{w}) \amp = \frac{\langle (\boldsymbol{u}+\boldsymbol{w}), \boldsymbol{v} \rangle}{\langle \boldsymbol{v},\boldsymbol{v} \rangle} \\ \amp = \frac{\langle \boldsymbol{u},\boldsymbol{v} \rangle + \langle \boldsymbol{w},\boldsymbol{v} \rangle}{\langle \boldsymbol{v},\boldsymbol{v} \rangle } \\ \amp = \frac{\langle \boldsymbol{u},\boldsymbol{v} \rangle}{\langle \boldsymbol{v},\boldsymbol{v} \rangle } + \frac{\langle \boldsymbol{w},\boldsymbol{v} \rangle}{\langle \boldsymbol{v},\boldsymbol{v} \rangle } \\ \amp = \text{Proj}_{\boldsymbol{v}} \boldsymbol{u} + \text{Proj}_{\boldsymbol{v}} \boldsymbol{w} \end{align*}

🔗

and

🔗

\begin{align*} \text{Proj}_{\boldsymbol{v}} (\alpha \boldsymbol{u}) \amp = \frac{\langle (\alpha \boldsymbol{u}), \boldsymbol{v} \rangle}{\langle \boldsymbol{v},\boldsymbol{v} \rangle} \\ \amp = \alpha \frac{\langle \boldsymbol{u}, \boldsymbol{v} \rangle}{\langle \boldsymbol{v},\boldsymbol{v} \rangle} = \alpha \text{Proj}_{\boldsymbol{v}} \boldsymbol{u} \end{align*}

🔗

The example shown so far for projections have been onto a vector (which can be thought of as the line through the origin with that direction), but in general, one can project onto any linear space (more technically, any inner product space). Before showing the general projection, let’s consider the projection of a vector onto a plane. The following example shows this.

🔗

Example 3.5.6.

Find the projection of vector \([6,-2,3]^T\) onto plane given by \(x+2y+3z=0\text{.}\)

🔗

Solution.

First, let’s find two vectors in the plane, called \(P\text{,}\) that form an orthogonal basis (we will see why it’s helpful to have an orthogonal basis later). Recall that a normal vector to the plane is given by the vector \([1,2,3]^T\text{.}\) Two vector that are normal to it is \(\boldsymbol{u}_1=[3,0,-1]\) and \(\boldsymbol{u}_2=[0,-3,2]^T\text{.}\) (Note: these are two points in the plane). To form an orthonormal basis, we need to perform Gramm-Schmidt.

\begin{align*} \boldsymbol{v}_1 \amp = \boldsymbol{u}_1 \amp \boldsymbol{v}_2 \amp= \boldsymbol{u}_2 - \frac{\langle \boldsymbol{v}_1,\boldsymbol{u}_2 \rangle}{\langle \boldsymbol{v}_1, \boldsymbol{v}_1 \rangle} \boldsymbol{v}_1 \\ \amp = \begin{bmatrix} 3 \\ 0 \\ -1 \end{bmatrix}\amp \boldsymbol{v}_2 \amp = \begin{bmatrix} 0 \\ -3 \\ 2 \end{bmatrix} - \biggl(-\frac{1}{5}\biggr) \begin{bmatrix} 3 \\ 0 \\ -1 \end{bmatrix} \\ \amp\amp\amp = \begin{bmatrix} 3/5 \\ -3 \\ 9/5 \end{bmatrix} \end{align*}

and to make the second vector nicer, we’ll multiply by \(5/3\text{.}\) An orthogonal set in the plane is

\begin{equation*} \left\{ \begin{bmatrix} 3 \\ 0 \\ -1 \end{bmatrix} , \begin{bmatrix} 1 \\ -5 \\ 3 \end{bmatrix} \right \} \end{equation*}

To consider how to project the vector \(\boldsymbol{u} = [6,-2,3]^{\intercal}\) into the plane we will write

\begin{equation*} \boldsymbol{u} = \boldsymbol{u}_{\perp} + \boldsymbol{u}_{||} \end{equation*}

where \(\boldsymbol{u}_{\perp}\) is orthogonal to both \(\boldsymbol{v}_1\) and \(\boldsymbol{v}_2\) and we can write

\begin{equation*} \text{Proj}_{P}\boldsymbol{u} = \boldsymbol{u}_{||} = c_1 \boldsymbol{v}_1 + c_2 \boldsymbol{v}_2 \end{equation*}

So we next take the inner product of \(\boldsymbol{u}\) with both \(\boldsymbol{v}_1\) and \(\boldsymbol{v}_2\text{.}\)

\begin{align*} \langle \boldsymbol{u}, \boldsymbol{v}_1 \rangle \amp = \langle \boldsymbol{u}_{\perp} + \boldsymbol{u}_{||}, \boldsymbol{v}_1 \rangle \amp \langle \boldsymbol{u}, \boldsymbol{v}_2 \rangle \amp = \langle \boldsymbol{u}_{\perp} + \boldsymbol{u}_{||}, \boldsymbol{v}_2 \rangle \\ \amp = \langle \boldsymbol{u}_{\perp}, \boldsymbol{v}_1 \rangle + \langle \boldsymbol{u}_{||}, \boldsymbol{v}_1 \rangle \amp \amp = \langle \boldsymbol{u}_{\perp}, \boldsymbol{v}_2 \rangle + \langle \boldsymbol{u}_{||}, \boldsymbol{v}_2 \rangle \\ \amp = 0 + \langle c_1 \boldsymbol{v}_1 + c_2 \boldsymbol{v}_2 ,\boldsymbol{v}_1 \rangle \amp \amp = 0 + \langle c_1 \boldsymbol{v}_1 + c_2 \boldsymbol{v}_2 ,\boldsymbol{v}_2 \rangle \\ \amp = c_1 \langle \boldsymbol{v}_1, \boldsymbol{v}_1 \rangle + c_2 \langle \boldsymbol{v}_2, \boldsymbol{v}_1 \rangle \amp \amp = c_1 \langle \boldsymbol{v}_1, \boldsymbol{v}_2 \rangle + c_2 \langle \boldsymbol{v}_2, \boldsymbol{v}_2 \rangle \\ \amp = c_1 \langle \boldsymbol{v}_1, \boldsymbol{v}_1 \rangle + 0 \amp \amp = 0 + c_2 \langle \boldsymbol{v}_2, \boldsymbol{v}_1 \rangle \end{align*}

Therefore

\begin{align*} c_1 \amp = \frac{\langle \boldsymbol{u},\boldsymbol{v}_1 \rangle}{\langle \boldsymbol{v}_1, \boldsymbol{v}_1 \rangle} \amp c_2 \amp = \frac{\langle \boldsymbol{u},\boldsymbol{v}_2 \rangle}{\langle \boldsymbol{v}_2, \boldsymbol{v}_2 \rangle} \end{align*}

So in the case of projecting a vector onto a plane, \(P\text{,}\)

\begin{align*} \text{Proj}_{P} \boldsymbol{u} \amp = c_1 \boldsymbol{v}_1 + c_2 \boldsymbol{v}_2 \\ \amp = \frac{\langle \boldsymbol{u},\boldsymbol{v}_1 \rangle}{\langle \boldsymbol{v}_1, \boldsymbol{v}_1 \rangle} \boldsymbol{v}_1 + \frac{\langle \boldsymbol{u},\boldsymbol{v}_2 \rangle}{\langle \boldsymbol{v}_2, \boldsymbol{v}_2 \rangle} \boldsymbol{v}_2 \\ \amp = \frac{15}{10} \begin{bmatrix} 3 \\ 0 \\ -1 \end{bmatrix} + \frac{25}{35} \begin{bmatrix} 1 \\ -5 \\ 3 \end{bmatrix} = \frac{1}{14}\begin{bmatrix} 73 \\ -50 \\ 9 \end{bmatrix} \end{align*}

So the vector \(\frac{1}{14} [73,-50,9]^{\intercal}\) is the projection of the vector \([6,-2,3]^{\intercal}\) onto the plane \(x+2y+3z=0\text{.}\)

🔗

Subsection 3.5.1 Using Projections to solve Least-Squares Problems

A useful problem is to find the minimum distance between a point and line. For example consider the point \((1,4)\) and the line \(L: y= \frac{1}{2} x\) as shown in the figure below.

🔗

If we want to minimize the distance from the point \(P\) to the line \(L\text{,}\) we often minimize the square

Recall that the point that minimizes a function is the same point that to minimizes the square of the function. The reason for doing this it it gets rid of the square root and makes calculations easier to do.

of the distance from the point to the line or

\begin{equation*} g(x) = (x-1)^2 + \biggl(\frac{1}{2} x - 4\biggr)^2 \end{equation*}

and using techniques from calculus, this function is minimized when \(x=12/5\text{.}\) Looking at the plot, the \(y\)-value is \(12/5\) and the vector from \((12/5,6/5)\) to \((1,4)\) is perpendicular to the vector in the direction of \(L\) as shown below:

🔗

Figure 3.5.8. Find a point on a line that minimizes distance
🔗

In light of projections, we can reframe this problem. We are seeking the point on \(L\) closest to \(P\text{.}\) This can be found by projecting the vector \(\overrightarrow{OP}\) to the line \(L\) or

\begin{equation*} \text{proj}_{\boldsymbol{v}} \overrightarrow{OP} = \frac{\langle \boldsymbol{v}, \overrightarrow{OP} \rangle} {\langle \boldsymbol{v},\boldsymbol{v}\rangle} \boldsymbol{v} \end{equation*}

and in this case, with \(\boldsymbol{v} = [2\;\;1]^{\intercal}\) and \(\overrightarrow{OP}=[1\;\;4]^{\intercal}\text{,}\)

which is the same result as from Calculus.

🔗

Subsection 3.5.2 Projecting onto a Vector Space

We again generalize from projecting onto a single vector or set of vectors to a general vector space, \(V\text{.}\) First, to make things easier, we will use an orthogonal basis for \(V\text{,}\) call it, \(( \boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_n )\text{.}\)

🔗

Recall that a projection onto a vector is found by writing the original vector, \(\boldsymbol{u}=\boldsymbol{u}_{||} + \boldsymbol{u}_{\perp}\text{,}\) where \(\boldsymbol{u}_{||}\) is in \(V\) or

\begin{equation} \boldsymbol{u} = \boldsymbol{u}_{\perp} + \sum_{i=1}^n c_i \boldsymbol{v}_i\tag{3.5.3} \end{equation}

Finding the projection is analogous to finding the constants \(c_i\text{.}\) Take the inner product of ((3.5.3)) with \(v_k\text{,}\) the \(k\)th basis vector of \(V\) or

\begin{align*} \langle \boldsymbol{u}, \boldsymbol{v}_k \rangle \amp= \biggl\langle \boldsymbol{u}_{\perp} + \sum_{i=1}^n c_i \boldsymbol{v}_i , \boldsymbol{v}_k \biggr\rangle \\ \amp\amp\text{using properties of inner products}\\ \amp = \langle \boldsymbol{u}_{\perp}, \boldsymbol{v}_k \rangle + \sum_{i=1}^n c_i \langle \boldsymbol{v}_i, \boldsymbol{v}_k \rangle \end{align*}

since \(\boldsymbol{u}_{\perp}\) is chosen to be orthogonal to the vector space and since the basis of \(V\) is orthogonal,

\begin{equation*} \langle \boldsymbol{u}, \boldsymbol{v} \rangle = c_k \langle v_k, v_k \rangle \end{equation*}

Therefore

\begin{equation*} c_k = \frac{\langle \boldsymbol{u}, \boldsymbol{v}_k \rangle}{\langle \boldsymbol{v}_k, \boldsymbol{v}_k \rangle} \end{equation*}

and thus we can make the following statement about projecting any element onto a subspace:

🔗

Remark 3.5.9.

Let \(\boldsymbol{u}\) be an element in a vector space \(S\) and \(( \boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_n )\) be an orthogonal basis of a subspace \(V\text{.}\) The projection of \(\boldsymbol{u}\) onto \(V\) can be written:

\begin{equation*} \text{Proj}_V (\boldsymbol{u}) = \sum_{i=1}^n c_k v_k \end{equation*}

where

\begin{equation*} c_k = \frac{\langle \boldsymbol{u}, \boldsymbol{v}_k \rangle}{\langle \boldsymbol{v}_k, \boldsymbol{v}_k \rangle} \end{equation*}

Note: if the basis of \(V\) is also orthonormal, then \(c_k = \langle \boldsymbol{u}, \boldsymbol{v}_k \rangle\text{.}\)

🔗

Example 3.5.10.

Find the projection of \(\sin \pi x\text{,}\) an element of \({\cal C}^{(0)}\) onto \({\cal P}_5[-1,1]\)

🔗

Solution.

First, we need a orthogonal basis of the subspace and we found this above in Example 3.3.14. These are the first four Legendre polynomials (scaled to eliminate fractions) or

🔗

\begin{equation} (1,x,3x^2-1,5x^3-3x)\tag{3.5.4} \end{equation}

🔗

Next, find all of the values of \(c_k\) for \(k=0,1,2,3\)

🔗

\begin{align*} c_0 \amp = \frac{\langle \sin \pi x, 1 \rangle }{\langle 1, 1 \rangle} = \frac{\int_{-1}^1 \sin \pi x \, dx}{\int_{-1}^1 1^2 \, dx} = 0 \\ c_1 \amp = \frac{\langle \sin \pi x, x \rangle }{\langle x, x \rangle} = \frac{\int_{-1}^1 x \sin \pi x \, dx}{\int_{-1}^1 x^2 \, dx} = \frac{2/\pi}{2/3} =\frac{3}{\pi} \end{align*}

🔗

Similarly, it can be shown that

🔗

\begin{align*} c_2 \amp = 0 \\ c_3 \amp = \frac{7(\pi^2-15)}{\pi^3} \end{align*}

🔗

The projection then is the sum

🔗

\begin{equation*} c_0 p_0 + c_1 p_1 + c_2 p_2 + c_3 p_3 \end{equation*}

🔗

where \(P_k\) is the \(k\)th Legendre polynomial as in ((3.5.4)). A plot of this is

🔗

Figure 3.5.11. Find a projection of \(\sin x\) onto \({\cal P}_3\)
🔗

which looks quite sine-like. Notice that if we plot the absolute difference between the function and the projection that we get

\begin{equation*} |\sin \pi x -\sum_{i=0}^3 c_k P_k(x)| \end{equation*}

🔗

Figure 3.5.12. Difference between \(\sin x\) and its projection.
🔗

This plot shows that except for near the ends of the interval, the projection (or an approximation is within 1 decimal places.)

🔗

This last example shows the power of projections. If we are using functions as elements of vector spaces, then a projection of a function onto a vector space (using the span of a vector space), is the closest function in the vector space to the original function. In Chapter 6, we will do this with trigonometric function and is called Fourier Series.

🔗

Prev Top Next