Orthogonal sets

Finding the coordinates of a vector in a basis usually means row reducing an augmented matrix. But if the basis vectors are mutually perpendicular, like the standard basis, each coordinate comes from a single inner product. That is why orthogonal bases are the preferred bases throughout applied math, from signal processing to statistics.

Orthogonal sets

To check a set of $p$ vectors, you check all $\binom{p}{2}$ pairs. For three vectors that is three inner products.

Orthogonal sets are independent

Here is why. Suppose $c_1\mathbf{u}_1 + \cdots + c_p\mathbf{u}_p = \mathbf{0}$. Take the inner product of both sides with $\mathbf{u}_1$. Every term $c_j\,\mathbf{u}_j \cdot \mathbf{u}_1$ with $j \neq 1$ vanishes, leaving

$$c_1(\mathbf{u}_1 \cdot \mathbf{u}_1) = 0.$$

Since $\mathbf{u}_1 \neq \mathbf{0}$, $\mathbf{u}_1 \cdot \mathbf{u}_1 > 0$, so $c_1 = 0$. The same argument with $\mathbf{u}_2, \ldots, \mathbf{u}_p$ shows every weight is $0$.

So the set in the example above is a basis for $\mathbb{R}^3$: three independent vectors in a 3-dimensional space. An orthogonal basis for a subspace $W$ is a basis of $W$ that is also an orthogonal set.

Coordinates in an orthogonal basis

The same trick of "dot with $\mathbf{u}_j$" also computes coordinates. If $\mathbf{y} = c_1\mathbf{u}_1 + \cdots + c_p\mathbf{u}_p$, dotting with $\mathbf{u}_j$ kills every term but one: $\mathbf{y} \cdot \mathbf{u}_j = c_j(\mathbf{u}_j \cdot \mathbf{u}_j)$.

No row reduction, and each weight is computed independently of the others.

Orthonormal sets

With an orthonormal basis every denominator is $1$, so the weights are simply $c_j = \mathbf{y} \cdot \mathbf{u}_j$. The standard basis $\{\mathbf{e}_1, \ldots, \mathbf{e}_n\}$ is the most familiar example. To turn an orthogonal set of nonzero vectors into an orthonormal one, normalize each vector; scaling doesn't affect orthogonality.

Normalizing the basis from the examples gives the orthonormal basis

$$\tfrac{1}{\sqrt{3}}(1, 1, 1), \quad \tfrac{1}{\sqrt{6}}(1, -2, 1), \quad \tfrac{1}{\sqrt{2}}(1, 0, -1).$$

Matrices with orthonormal columns

Let $U = [\mathbf{u}_1 \ \cdots \ \mathbf{u}_p]$. The $(i, j)$ entry of $U^TU$ is row $i$ of $U^T$ times column $j$ of $U$, which is $\mathbf{u}_i \cdot \mathbf{u}_j$. So the columns are orthonormal exactly when those entries are $1$ on the diagonal and $0$ elsewhere.

The proof is one line: $(U\mathbf{x}) \cdot (U\mathbf{y}) = (U\mathbf{x})^T(U\mathbf{y}) = \mathbf{x}^TU^TU\mathbf{y} = \mathbf{x}^T\mathbf{y}$. Taking $\mathbf{y} = \mathbf{x}$ gives the length statement. Since angles are computed from inner products and lengths, $U$ preserves angles too, and in particular it sends orthogonal vectors to orthogonal vectors.

A square matrix with orthonormal columns is called an orthogonal matrix. For such a matrix, $U^TU = I$ means $U^{-1} = U^T$: the inverse costs nothing to compute. Rotations and reflections of $\mathbb{R}^n$ are orthogonal matrices. (The name is traditional; "orthonormal matrix" would be more accurate.)

Practice

Practice 1

Which set is orthogonal?

A$\{(1, 2), (2, 1)\}$B$\{(1, 1, 0), (0, 1, 1), (1, 0, 1)\}$C$\{(2, -1, 1), (1, 1, -1), (0, 1, 1)\}$D$\{(1, 0, 1), (1, 1, 0), (-1, 1, 1)\}$

Check

Practice 2

Find $k$ so that $\{(1, k, 2), (3, -1, k)\}$ is an orthogonal set.

Your answer

Enter a number. Fractions like 3/4 and sqrt(2) are OK.

CheckShow hint

Practice 3

The vectors $\mathbf{u}_1 = (2, 1)$ and $\mathbf{u}_2 = (-1, 2)$ form an orthogonal basis for $\mathbb{R}^2$. Find the weights $(c_1, c_2)$ with $(4, 7) = c_1\mathbf{u}_1 + c_2\mathbf{u}_2$.

Your answer

Enter a point like (2, -3)

CheckShow hint

Practice 4

The vectors $\mathbf{u}_1 = (1, 2, 2)$, $\mathbf{u}_2 = (2, 1, -2)$ and $\mathbf{u}_3 = (2, -2, 1)$ form an orthogonal basis for $\mathbb{R}^3$. Find the weights $(c_1, c_2, c_3)$ that express $\mathbf{y} = (2, 1, 7)$ in this basis.

Your answer

Enter a point like (2, -3)

CheckShow hint

Practice 5

The vectors $(1, 1, 1)$ and $(1, -1, 0)$ are orthogonal. Find $a$ and $b$ so that $(a, b, 1)$ is orthogonal to both, and enter the vector $(a, b, 1)$.

Your answer

Enter a point like (2, -3)

CheckShow hint

Practice 6

$U$ is a $5 \times 3$ matrix with orthonormal columns and $\mathbf{x} = (1, 2, -2)$. Find $\|U\mathbf{x}\|$.

Your answer

Enter a number. Fractions like 3/4 and sqrt(2) are OK.

CheckShow hint

Practice 7

$Q = \begin{bmatrix} 3/5 & -4/5 \\ 4/5 & 3/5 \end{bmatrix}$ is an orthogonal matrix. Find the $(1, 2)$ entry of $Q^{-1}$.

Your answer

Enter a number. Fractions like 3/4 and sqrt(2) are OK.

CheckShow hint

Practice 8

Which statement is always true?

AA square matrix with orthogonal columns is an orthogonal matrix.BIf $U$ has orthonormal columns, then $(U\mathbf{x}) \cdot (U\mathbf{y}) = \mathbf{x} \cdot \mathbf{y}$ for all $\mathbf{x}$ and $\mathbf{y}$.CEvery orthogonal set is linearly independent.DIf $U$ has orthonormal columns, then $UU^T = I$.

Check