Inner Product Spaces

Section 3.3 Inner Product Spaces

In Subsection 1.6.7, we examined the dot product between two vectors. Recall that we can write the dot product between two vectors $\vec{u}$ and $\vec{v}$ in $\mathbb{R}^n$ is

🔗

\begin{equation*} \vec{u} \cdot \vec{v} = u_1v_1 + u_2v_2 + \cdots +u_nv_n \end{equation*}

🔗

We used the dot product for finding the angle between two vectors and we’ll show that there are many other applications that the dot product later in the text. In this section, we extend that to any vector space and call this an inner product space.

🔗

Definition 3.3.1.

The vector space $V$ is called an inner product space if for every pair of vectors $\vec{u}$ and $\vec{v}$ in $V$ there is a unique number $\langle \vec{u}, \vec{v} \rangle\text{,}$ called the inner product of $\vec{u}$ and $\vec{v}$ such that for all $\vec{u}, \vec{v}$ and $\vec{w}$ in $V$ and $a, b \in \mathbb{R}$ then the following properties are satisfied:

🔗

$\displaystyle \langle \vec{u}, \vec{v} \rangle = \langle \vec{v}, \vec{u} \rangle$
🔗

🔗
$\langle a \vec{u} + b \vec{v}, \vec{w} \rangle = a \langle \vec{u}, \vec{w} \rangle + b \langle \vec{v}, \vec{w} \rangle\text{.}$
🔗

🔗
$\langle \vec{u}, \vec{u} \rangle \geq 0$ and equals 0 if and only if $\vec{u} = \vec{0}\text{.}$
🔗

🔗

🔗

Example 3.3.2.

Show that $\mathbb{R}^n$ is an inner product space with

🔗

\begin{align*} \vec{u} \amp= \begin{bmatrix} u_1 \\ u_2 \\ \vdots \\ u_n \end{bmatrix} \amp \vec{v} \amp = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix} \end{align*}

🔗

and

🔗

$\langle \vec{u}, \vec{v} \rangle = \vec{u}^T\vec{v}\text{.}$

🔗

Solution.

First, the set $\mathbb{R}^n$ is a vector space. Next, we need to show that the three properties are satisfied for $\langle \vec{u}, \vec{v} \rangle = \vec{u}^T \vec{v}\text{.}$

🔗

\begin{align*} \langle \vec{u}, \vec{v} \rangle \amp = \vec{u}^T \vec{v} \end{align*}

since the transpose of a number is itself

\begin{align*} \amp = (\vec{u}^T \vec{v})^T \end{align*}

using the properties of transposes in Subsection 2.2.6.

\begin{align*} \amp = \vec{v}^T (\vec{u}^T)^T = \vec{v}^T \vec{u} = \langle \vec{v}, \vec{u} \rangle. \end{align*}

so the first property is satisfied.

🔗

🔗
\begin{align*} \langle a \vec{u} + b \vec{v}, \vec{w} \rangle \amp = (a \vec{u} + b\vec{v})^T \vec{w} = (a \vec{u}^T + b \vec{v}^T) \vec{w} \\ \amp = a \vec{u}^T \vec{w} + b \vec{v}^T \vec{w} = a \langle \vec{u}, \vec{w} \rangle + b \langle \vec{v}, \vec{w} \rangle. \end{align*}

🔗

🔗
\begin{equation*} \langle \vec{u}, \vec{u} \rangle = \vec{u}^T \vec{u} = u_1^2 + u_2^2 + u_3^2 + \cdots + u_n^2 \end{equation*}

🔗

which satisfies $\geq 0\text{.}$ The only time that this quantity equals 0, is when $u_1=u_2 = \cdots =u_n$ or $\vec{u} = \vec{0}\text{.}$
🔗

🔗

🔗

The next example shows that the notion of an inner product is not limited to the dot product of vectors in $\mathbb{R}^n\text{.}$

🔗

Example 3.3.3.

Show that $\mathcal{P}_n$ on the interval $[0,1]$ is a inner product space with

🔗

\begin{equation*} \langle p,q \rangle = \int_0^1 p q \, dx \end{equation*}

🔗

for any $p, q \in \mathcal{P}_n\text{.}$

🔗

Solution.

We have seen above that $\mathcal{P}_n$ is a vector space and limiting this to the interval $[0,1]$ does not change that fact. We need to show that the inner product defined as an integral satisfies the three properties of inner products.

🔗

\begin{equation*} \langle p, q \rangle = \int_0^1 pq \, dx = \int_0^1 q p \, dx = \langle q, p \rangle. \end{equation*}

🔗

🔗
Let $r$ also be in $\mathcal{P}_n$ and $a,b \in \mathbb{R}\text{.}$
🔗

\begin{align*} \langle ap+bq,r \rangle \amp = \int_0^1 (ap+bq)r \, dx = \int_0^1 (apr+bqr) \,dx \\ \amp = a \int_0^1 pr \, dx + b \int_0^1 qr \, dx = a \langle p,r \rangle + b \langle q,r \rangle \end{align*}

🔗

🔗
\begin{equation*} \langle p, p \rangle = \int_0^1 p^2 \, dx \end{equation*}

🔗

and this is greater than or equal or zero and can be shown that equals 0 if and only if $p(x)\equiv 0\text{.}$
🔗

🔗

🔗

Subsection 3.3.1 Vector Norms and distance

Definition 3.3.4.

Let $\vec{u}$ be an element of an inner product space. The norm of the vector $\vec{u}$ is given by

🔗

\begin{equation*} || \vec{u} || = \sqrt{\langle \vec{u},\vec{u} \rangle} \end{equation*}

🔗

Note: if $\vec{u} \in \mathbb{R}^n$ then the vector norm is the length of the vector.

🔗

Definition 3.3.5.

The distance between vectors $\vec{u}$ and $\vec{v}$ denoted $d(\vec{u},\vec{v})$ and is defined as

🔗

\begin{equation*} d(\vec{u},\vec{v})=||\vec{u}-\vec{v}||. \end{equation*}

🔗

Note: if $\vec{u}$ and $\vec{v}$ are in $\mathbb{R}^n\text{,}$ then the distance function is the standard distance function where the vectors are considered to be points.

🔗

Lemma 3.3.6.

The distance $||\vec{u}-\vec{v}||=0$ if and only if $\vec{u}=\vec{v}\text{.}$

🔗

Proof.

$\Longrightarrow$ Let $||\vec{u}-\vec{v}||=0\text{,}$ then $\langle \vec{u}-\vec{v},\vec{u}-\vec{v} \rangle=0$ By definition, this is only zero if $\vec{u}-\vec{v}=\vec{0}\text{,}$ therefore $\vec{u}=\vec{v}\text{.}$

🔗

$\Longleftarrow$ If $\vec{u}=\vec{v}\text{,}$ then $||\vec{u}-\vec{v}|| = ||\vec{0}||=0\text{.}$

🔗

Except for the example of the inner product as an integral, all of the other discussion in this section has been about vectors in $\mathbb{R}^n\text{,}$ and this is mainly because it is the canonical vector and inner product space and some of the interpretations of length of vectors don’t make a lot of sense when talking about polynomials or other functions.

🔗

However, distance does have some interpretation with functions. Lemma 3.3.6 can be used to show when two functions are equal (at least on the interval of the integration), but can also be useful to determine when they are close to equal (when the integrals are close to one another). This will be useful in future chapters.

🔗

Subsection 3.3.2 Angles between vectors

If $\vec{u}$ and $\vec{v}$ are vectors in $\mathbb{R}^2$ then we saw in Subsection 1.6.7 that

🔗

\begin{equation*} \cos \theta = \frac{\langle \vec{u},\vec{v} \rangle}{||\vec{u}||\, || \vec{v}||} \end{equation*}

and thus the angle between the vectors can be found.

🔗

This notion generalizes to any vectors in an inner product space, $V\text{.}$ Most helpful, two vectors meet at a right angle if $\langle \vec{u},\vec{v}\rangle=0$ and it is said that if this holds then $\vec{u}$ and $\vec{v}$ are orthogonal. The next section talks about how an entire set of vectors can be orthogonal .

🔗

Subsection 3.3.3 Orthonormal sets of vectors

Definition 3.3.7.

Let $\{\vec{v}_1, \vec{v}_2, \ldots, \vec{v}_n\}$ each be elements of an inner product space, $V\text{.}$ The set is called an orthonormal set if

🔗

\begin{equation*} \langle \vec{v}_i, \vec{v}_j \rangle = 0 \qquad \text{if $i \neq j$} \end{equation*}

🔗

and

🔗

\begin{equation*} ||\vec{v}_i|| = \sqrt{\langle \vec{v}_i, \vec{v}_i \rangle} = 1 \end{equation*}

🔗

If only the first condition holds, the set is called orthogonal.

🔗

Example 3.3.8.

Show that

🔗

\begin{align*} \vec{v}_1 \amp= \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \amp \vec{v}_2 \amp = \begin{bmatrix} 0 \\ 1/\sqrt{2} \\ 1/\sqrt{2} \end{bmatrix}, \amp \vec{v}_3 \amp = \begin{bmatrix} 0 \\ 1/\sqrt{2} \\ -1/\sqrt{2} \end{bmatrix} \end{align*}

🔗

is an orthonormal set where each vector is an element of $\mathbb{R}^3\text{.}$

🔗

Solution.

We need to show that the three inner products between the vectors are zero

🔗

\begin{align*} \langle \vec{v}_1, \vec{v}_2 \rangle \amp = (1)(0) + (0)(1/\sqrt{2}) + (0)(1/\sqrt{2}) = 0 \\ \langle \vec{v}_1, \vec{v}_3 \rangle \amp = (1)(0) + (0)(1/\sqrt{2}) + (0)(-1/\sqrt{2}) = 0 \\ \langle \vec{v}_2, \vec{v}_3 \rangle \amp = (0)(0) + (1/\sqrt{2})(1/\sqrt{2}) + (1/\sqrt{2})(-1/\sqrt{2}) = \frac{1}{2} - \frac{1}{2} = 0 \end{align*}

🔗

and that the norms of each vector is 1.

🔗

\begin{align*} ||\vec{v}_1|| \amp = \sqrt{(1)^2 + (0)^2 + (0^2)} = 1 \\ ||\vec{v}_2|| \amp = \sqrt{(0)^2 + (1/\sqrt{2})^2 + (1/\sqrt{2})^2} = \sqrt{0 + \frac{1}{2} + \frac{1}{2}} = 1 \\ ||\vec{v}_3|| \amp = \sqrt{(0)^2 + (1/\sqrt{2})^2 + (-1/\sqrt{2})^2} = \sqrt{0 + \frac{1}{2} + \frac{1}{2}} = 1 \end{align*}

🔗

Note: this set of vectors is also a basis for $\mathbb{R}^3\text{.}$

🔗

Example 3.3.9.

Show that

🔗

\begin{equation*} \{\sin x, \sin 2x, \sin 3x\} \end{equation*}

🔗

form an orthogonal set on $[-\pi,\pi]$ with the inner product taken to be the definite integral on $[-\pi,\pi]\text{.}$

🔗

Solution.

Note: the following integrals will come up in the next few chapters a number of times and the technique shown below is helpful for such integrals. This also shows that even though in most cases in this text so far, that using complex numbers helps out tremendously.

🔗

Recall that

🔗

\begin{align*} \sin x \amp = \frac{e^{ix} - e^{-ix}}{2i} \amp \cos x \amp = \frac{e^{ix} + e^{-ix}}{2} \end{align*}

🔗

and using this we find the integrals of all pairs of functions on $[-\pi,\pi]$

🔗

\begin{align*} \langle \sin x, \sin 2x \rangle \amp = \int_{-\pi}^{\pi} \sin x \sin 2x \, dx = \int_{-\pi}^{\pi} \frac{e^{ix} - e^{-ix}}{2i} \frac{e^{2ix} - e^{-2ix}}{2i}\,dx \\ \amp = \frac{1}{-4} \int_0^{\pi} (e^{3ix} -e^{ix} -e^{-ix} + e^{-3ix} ) \, dx \\ \amp = -\frac{1}{4} \biggl( \frac{e^{3ix}}{3i} - \frac{e^{ix}}{i} - \frac{e^{-ix}}{-i} + \frac{e^{-3ix}}{-3i} \biggr) \biggr\vert_{-\pi}^{\pi} \\ \amp = -\frac{1}{4} \biggl( \frac{e^{3i\pi}}{3i} - \frac{e^{i\pi}}{i} - \frac{e^{-i\pi}}{-i} + \frac{e^{-3i\pi}}{-3i} \biggr) \\ \amp \qquad +\frac{1}{4} \biggl( \frac{e^{-3i\pi}}{3i} - \frac{e^{-i\pi}}{i} - \frac{e^{i\pi}}{-i} + \frac{e^{3i\pi}}{-3i} \biggr) = 0 \end{align*}

🔗

and as it can be seen that all term made with the substitution $x=\pi$ cancel with terms made with the substitution $x=-\pi\text{.}$

🔗

The other two inner products are similar. In fact it can be shown (and we will later) that

🔗

\begin{equation*} \langle \sin kx, \sin \ell x \rangle = \int_{-\pi}^{\pi} \sin kx \sin \ell x \, dx = 0 \end{equation*}

🔗

if $k \neq \ell\text{.}$

🔗

We saw the following theorem applied to vectors in $\mathbb{R}^n$ in Subsection 1.6.10. It was a consequence of the triangle inequality. We again generalize this to the inner product.

🔗

Theorem 3.3.10. Cauchy-Swartz Inequality.

If $\vec{u}$ and $\vec{v}$ are elements of an inner product space then

🔗

\begin{equation*} | \langle \vec{u}, \vec{v} \rangle| \leq || \vec{u} ||\, ||\vec{v}|| \end{equation*}

🔗

Proof.

If $\vec{v}=\vec{0}\text{,}$ then the inequality is satisfied. Assume $\vec{v} \neq \vec{0}$ and define

🔗

\begin{equation*} \lambda = \langle \vec{v},\vec{v} \rangle^{-1} \langle \vec{u},\vec{v} \rangle. \end{equation*}

🔗

By definition of the inner product,

🔗

\begin{align*} 0 \leq\amp \langle \vec{u} - \lambda \vec{v}, \vec{u} -\lambda \vec{v} \rangle \end{align*}

🔗

Using properties of the inner product,

🔗

\begin{align*} 0 \leq \amp \langle \vec{u}, \vec{u} \rangle - 2 \lambda \langle \vec{u}, \vec{v} \rangle + \lambda^2 \langle \vec{v},\vec{v} \rangle \\ = \amp \langle \vec{u},\vec{u} \rangle - 2 \langle \vec{v},\vec{v} \rangle^{-1} \langle \vec{u},\vec{v} \rangle \langle \vec{u}, \vec{v} \rangle + \bigl(\langle \vec{v},\vec{v} \rangle^{-1} \langle \vec{u},\vec{v} \rangle \bigr)^2 \langle \vec{v},\vec{v} \rangle.\\ = \amp \langle \vec{u}, \vec{u} \rangle - \langle \vec{v},\vec{v} \rangle^{-1} \langle \vec{u},\vec{v} \rangle ^2 \end{align*}

🔗

Multiply through by $\langle\vec{v},\vec{v}\rangle$

🔗

\begin{align*} \amp \langle \vec{u},\vec{u} \rangle \langle \vec{v},\vec{v} \rangle - \langle \vec{u},\vec{v} \rangle^2 \end{align*}

🔗

This can be rearranged to get:

🔗

\begin{equation*} \langle \vec{u},\vec{v} \rangle^2 \leq \langle \vec{u},\vec{u} \rangle \langle \vec{v}, \vec{v} \rangle \end{equation*}

🔗

and taking the square root, you get the desired result.

🔗

The Cauchy-Swartz inequality has numerous uses (which is not unexpected for a theorem named after such famous mathematicians). One such is to show that the angle between two vectors in $\mathbb{R}^n$ and actually in any inner product space is defined.

🔗

Recall that above in this section, the angle between two vectors was defined as the angle $\theta$ that satisfies

🔗

\begin{equation*} \cos \theta = \frac{\langle \vec{u}, \vec{v} \rangle}{||\vec{u}||\, ||\vec{v}||} \end{equation*}

🔗

and solving for the inner product,

🔗

\begin{equation*} \langle \vec{u}, \vec{v} \rangle = ||\vec{u}||\, ||\vec{v}|| \cos \theta \end{equation*}

🔗

and using the Cauchy-Swartz Inequality,

🔗

\begin{equation*} |\langle \vec{u}, \vec{v} \rangle | = \bigl\vert ||\vec{u}||\, ||\vec{v}|| \cos \theta \bigr\vert \leq ||\vec{u}||\, ||\vec{v}|| \end{equation*}

🔗

which results in

🔗

\begin{equation*} |\cos \theta| \leq 1 \end{equation*}

🔗

which is satisfied for all $\theta\text{.}$

🔗

Subsection 3.3.4 The Gram-Schmidt Orthogonalization Process

We saw above that an orthonormal set of vectors has the property that any two different vectors in the set are orthogonal (inner product of zero) and have norm of one. Often, if we have a set of vectors (say a basis), they aren’t orthonormal, but it is advantageous to have another set that is. The Gram-Schmidt algorithm creates an orthonormal set of vectors from a set of linearly independent vectors.

🔗

We first see the idea from a pair of vectors in the following example.

🔗

Example 3.3.11.

Find a orthonormal set of vectors that span the same set as $\{\vec{u}_1,\vec{u}_2\}\text{,}$ where

🔗

\begin{align*} \vec{u}_1 \amp= \begin{bmatrix} 2 \\ 0 \end{bmatrix}, \amp \vec{u}_2 \amp = \begin{bmatrix} 1 \\ 1 \end{bmatrix} \end{align*}

🔗

Solution.

There are many ways of doing this, but the following will always work. First, let’s start by finding a unit vector in the same direction as $\vec{u}_1\text{.}$ We denote this as $\hat{v}_1$ and this can be found by dividing $\vec{u}_1$ by its length:

🔗

\begin{equation*} \hat{v}_1 = \frac{1}{||\vec{u}_1||} \vec{u}_1 = \frac{1}{2} \begin{bmatrix} 2 \\ 0 \end{bmatrix} = \begin{bmatrix} 1 \\ 0 \end{bmatrix} \end{equation*}

🔗

Next, we will form a new vector $\vec{v}_2$ which is a linear combination of $\hat{v}_1$ and $\vec{u}_2$ and has the property that it is orthogonal to $\vec{v}_1$

🔗

\begin{equation*} \vec{v}_2 = c_1 \hat{v}_1 + c_2 \vec{u}_2 \end{equation*}

🔗

and we can take $c_1=1$ without loss of generality. with

🔗

\begin{align*} 0 \amp = \langle \vec{v}_2, \hat{v}_1 \rangle \\ \amp = \langle \hat{v}_1 + c_2 \vec{u}_2, \hat{v}_1 \rangle \\ \amp = \langle \hat{v}_1 \hat{v_1} \rangle + c_2 \langle \vec{u}_2, \hat{v}_1 \rangle \end{align*}

🔗

and solving for $c_2\text{,}$

🔗

\begin{align*} c_2 \amp = -\frac{\langle \vec{u}_2, \hat{v}_1 \rangle } {\langle \hat{v}_1 \hat{v_1} \rangle} \end{align*}

🔗

and since $\vec{v}_1$ is a unit vector

🔗

\begin{equation*} = -\langle \vec{u}_2, \hat{v}_1 \rangle = -1 \end{equation*}

🔗

\begin{align*} \amp \vec{v}_2 \amp = \hat{v}_1 - \vec{u}_2 = \begin{bmatrix} 1 \\ 0 \end{bmatrix} - \begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 0 \\ -1 \end{bmatrix} \end{align*}

🔗

Now since $\vec{v}_2$ is already a unit vector, normalization is not needed. If it were, then dividing by its length would create a unit vector. A orthonormal set that spans the set is

🔗

\begin{equation*} \left\{ \begin{bmatrix} 1\\0 \end{bmatrix}, \begin{bmatrix} 0 \\ -1 \end{bmatrix} \right\} \end{equation*}

🔗

If there are more than two vectors in the set, then the process is similar in that iteratively, one creates a new vector that is orthogonal to all previous vectors and then normalizes each one.

🔗

Remark 3.3.12.

The Gram-Schmidt Orthogonalization algorithm is the following. Consider a set of vectors $\{\vec{u}_1,\vec{u}_2, \ldots, \vec{u}_n\}$ which is the basis for some vector space.

🔗

Let $\vec{v}_1=\vec{u}_1\text{.}$
🔗

🔗
Let $\vec{v}_2 = \vec{u}_2 - \dfrac{\langle \vec{v}_1,\vec{u}_2 \rangle}{\langle \vec{v}_1,\vec{v}_1 \rangle} \vec{v}_1$
🔗

🔗
Let $\vec{v}_3 = \vec{u}_3 - \dfrac{\langle \vec{v}_2,\vec{u}_3 \rangle}{\langle \vec{v}_2,\vec{v}_2 \rangle} \vec{v}_2-\dfrac{\langle \vec{v}_1,\vec{u}_3 \rangle}{\langle \vec{v}_1,\vec{v}_1 \rangle} \vec{v}_1\text{.}$
🔗

🔗
And so on with

\begin{align*} \vec{v}_k \amp = \vec{u}_k - \frac{\langle \vec{v}_{k-1},\vec{u}_k \rangle}{\langle \vec{v}_{k-1},\vec{v}_{k-1} \rangle} \vec{v}_{k-1}-\frac{\langle \vec{v}_{k-2},\vec{u}_k \rangle}{\langle \vec{v}_{k-2},\vec{v}_{k-2} \rangle} \vec{v}_{k-2} \\ \amp \qquad \qquad \cdots - \frac{\langle \vec{v}_2,\vec{u}_k \rangle}{\langle \vec{v}_2,\vec{v}_2 \rangle} \vec{v}_2-\frac{\langle \vec{v}_1,\vec{u}_k \rangle}{\langle \vec{v}_1,\vec{v}_1 \rangle} \vec{v}_1 \\ \amp = \vec{u}_k - \sum_{i=1}^{k-1} \dfrac{\langle \vec{v}_i, \vec{u}_k \rangle}{\langle \vec{v}_i,\vec{v}_i \rangle} \vec{v}_i \end{align*}

🔗

🔗

🔗

The resulting set of vectors $\{\vec{v}_1, \vec{v}_2, \ldots, \vec{v}_n\}$ are orthogonal. If the set also needs to be orthonormal, for each vector $\vec{v}_k$ divide through by its length.

🔗

Proof of Gram-Schmidt Orthogonalization.

To prove Gram-Schmidt orthogonalization works, we will prove that $\vec{v}_k$ is orthogonal to $\vec{v}_i$ for all $i \lt k\text{.}$ We will show this by induction. First show that $\vec{v}_2$ is orthogonal to $\vec{v}_1$ :

🔗

\begin{align*} \langle \vec{v}_2 , \vec{v}_1 \rangle \amp = \biggl\langle \vec{u}_2 - \dfrac{\langle \vec{v}_1,\vec{u}_2 \rangle}{\langle \vec{v}_1,\vec{v}_1 \rangle} \vec{v}_1, \vec{v}_1 \biggr\rangle \\ \amp = \langle \vec{u}_2, \vec{v}_1 \rangle - \dfrac{\langle \vec{v}_1,\vec{u}_2 \rangle}{\langle \vec{v}_1,\vec{v}_1 \rangle} \langle \vec{v}_1, \vec{v}_1 \rangle \\ \amp = \langle \vec{u}_2, \vec{v}_1\rangle - \langle \vec{v}_1, \vec{u}_2 \rangle = 0 \end{align*}

🔗

Now assume that $\vec{v}_k$ is orthogonal to all other $\vec{v}$’s or $\langle\vec{v}_k, \vec{v}_i \rangle =0$ for $i=k-1,k-2,\ldots, 2,1\text{.}$ Now we will prove that $\vec{v}_{k+1}\text{,}$ given by Gram-Schmidt is orthogonal to all other $\vec{v}$’s. For all $j=1,2,\ldots,k\text{,}$

🔗

\begin{align*} \langle \vec{v}_{k+1}, \vec{v}_j \rangle \amp = \biggl \langle \vec{u}_{k+1} - \sum_{i=1}^{k} \dfrac{\langle \vec{v}_i, \vec{u}_{k+1} \rangle}{\langle \vec{v}_i,\vec{v}_i \rangle} \vec{v}_i , \vec{v}_j \biggl \rangle, \\ \amp = \langle \vec{u}_{k+1}, \vec{v}_j \rangle - \sum_{i=1}^k \dfrac{\langle \vec{v}_i, \vec{u}_{k+1} \rangle}{\langle \vec{v}_i,\vec{v}_i \rangle} \langle \vec{v}_i, \vec{v}_j \rangle \end{align*}

🔗

The only term that is not apparantly nonzero in the sum is when $i=j$

🔗

\begin{align*} \amp = \langle \vec{u}_{k+1}, \vec{v}_j \rangle - \dfrac{\langle \vec{v}_j, \vec{u}_{k+1} \rangle}{\langle \vec{v}_j,\vec{v}_i \rangle} \langle \vec{v}_j, \vec{v}_j \rangle \\ \amp = \langle \vec{u}_{k+1}, \vec{v}_j \rangle -\langle \vec{v}_j, \vec{u}_{k+1} \rangle = 0 \end{align*}

🔗

Therefore $\vec{v}_{k+1}$ is orthogonal to all other $\vec{v}$’s and thus the basis consisting of $\vec{v}$ form an orthonormal set.

🔗

Example 3.3.13.

Find a orthogonal set of vectors than spans the same subspace as

\begin{equation*} \left\{ \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}, \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix}, \begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix} \right\} \end{equation*}

🔗

Solution.

Let $\vec{u}_1, \vec{u}_2$ and $\vec{u}_3$ be these three vectors. We now use Gram-Schmidt orthogonalization to find an orthogonal set.

🔗

\begin{equation*} \vec{v}_1 = \vec{u}_1 = \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix} \end{equation*}

🔗

🔗
\begin{equation*} \vec{v}_2 = \vec{u}_2 - \frac{\langle \vec{u}_2, \vec{v}_1 \rangle}{\langle \vec{v}_1, \vec{v}_1 \rangle} \vec{v}_1 = \begin{bmatrix} 1 \\ 1\\ 0 \end{bmatrix} - \frac{1}{2} \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix} = \begin{bmatrix} 1/2 \\ 1\\ -1/2 \end{bmatrix} \end{equation*}

🔗

🔗
\begin{align*} \vec{v}_3 \amp = \vec{u}_3 - \frac{\langle \vec{u}_3, \vec{v}_2 \rangle}{\langle \vec{v}_2, \vec{v}_2 \rangle} \vec{v}_2- \frac{\langle \vec{u}_3, \vec{v}_1 \rangle}{\langle \vec{v}_1, \vec{v}_1 \rangle} \vec{v}_1\\ \amp = \begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix} - \frac{1/2}{3/2} \begin{bmatrix} 1/2 \\ 1 \\ -1/2 \end{bmatrix}- \frac{1}{2} \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix} = \begin{bmatrix} -2/3 \\ 2/3 \\ 2/3 \end{bmatrix} \end{align*}

🔗

🔗

🔗

And the set $\{\vec{v}_1,\vec{v}_2,\vec{v}_3\}$ is now orthogonal (check it).

🔗

Note: there are many other orthogonal sets that span this subspace. For example, the standard basis does this or if we would have shuffled the original three vectors, so this is not unique.

🔗

Although the example shown here uses vectors in $\mathbb{R}^3\text{,}$ Gram-Schmidt can be done on any set of vectors from an inner product space. This is very common for polynomials and will show an example with this next. Also, in Chapter 6, we will also show this with trigonometric functions.

🔗

Example 3.3.14.

Use Gram-Schmidt orthogonalization to find a orthogonal set that spans $\mathcal{P}_3[-1,1]\text{,}$ the set of all cubic functions on the interval $[-1,1]$ with inner product

🔗

\begin{equation*} \langle p, q \rangle = \int_{-1}^1 p(x) q(x) \, dx \end{equation*}

🔗

Solution.

To begin, we need a set of vectors (polynomials) on the set and we can take the standard basis $\{1,x,x^2,x^3\}\text{.}$ Call these $p_0,p_1,p_2,p_3$ and we’ll use Gram-Schmidt to find a set $\{q_0,q_1,q_2,q_3\}$ that is orthogonal.

🔗

It is very helpful to recall that $x^n$ is an odd function if $n$ is odd. Also $\int_{-1}^1 x^n \,dx =0$ if $n$ is odd.

\begin{align*} q_0 \amp= p_0 = 1, \\ q_1 \amp= p_1 - \frac{\langle p_1,q_0 \rangle}{\langle q_0,q_0 \rangle } q_0 \\ \amp = x - \frac{\int_{-1}^1 x \cdot 1 \, dx} {\int_{-1}^1 1 \cdot 1 \, dx} 1 \end{align*}

🔗

and since $x\cdot 1= x$ is odd, the integral in the numerator is 0, so $q_1=x$ and

🔗

\begin{align*} q_2 \amp = p_2 - \frac{\langle p_2,q_1 \rangle}{\langle q_1,q_1 \rangle } q_1 - \frac{\langle p_2,q_0 \rangle}{\langle q_0,q_0 \rangle } q_0 \\ \amp = x^2 - \frac{\int_{-1}^1 x^2 \cdot x \, dx}{ \int_{-1}^1 x \cdot x \, dx} - \frac{\int_{-1}^1 x^2 \cdot 1 \, dx}{\int_{-1}^1 1 \cdot 1 \, dx} 1 \end{align*}

🔗

and $x^2\cdot x=x^3$ is odd, so the first integral is 0 and also $\int_{-1}^1 1 \,dx = 2\text{,}$

🔗

\begin{align*} \amp = x^2 - \frac{x^3/3 \bigl\vert_{-1}^1}{2} 1 = x^2-\frac{1}{3} \\ q_3 \amp = q_3 - \frac{\langle p_3,q_2 \rangle}{\langle q_2,q_2 \rangle } q_2 - \frac{\langle p_3,q_1 \rangle}{\langle q_1,q_1 \rangle } q_1 - \frac{\langle p_3,q_0 \rangle}{\langle q_0,q_0 \rangle } q_0 \\ \amp = x^3 - \frac{\int_{-1}^1 x^3 (x^2-1/3) \,dx}{\int_{-1}^1 (x^2-1/3)^2 \, dx } \bigl(x^2-\frac{1}{3} \bigr) - \frac{\int_{-1}^1 x^3 \cdot x \, dx}{\int_{-1}^1 x \cdot x \, dx} x - \frac{\int_{-1}^1 x^3 \cdot 1 \,dx }{\int_{-1}^1 1 \cdot 1 \, dx} \end{align*}

🔗

and the integrals in the numerators of the 2nd and 4th terms are 0 due to the functions being odd, so

🔗

\begin{align*} \amp = x^3 - \frac{ x^5/5 \bigr\vert_{-1}^1}{x^3/3 \bigr\vert_{-1}^1} x \\ \amp = x^3 - \frac{3}{5} x \end{align*}

🔗

There an orthogonal set of cubic polynomials that span $\mathcal{P}_3[-1,1]$ is

🔗

\begin{equation*} \{1,x,x^2-\frac{1}{3},x^3-\frac{3}{5}x \} \end{equation*}

🔗

and these are the first four Legendre Polynomials. Also, they are a basis of $\mathcal{P}_3[-1,1]\text{.}$

🔗

Prev Top Next